| openstackgerrit | Merged zuul/nodepool master: Log error message on request handler removal on session loss https://review.opendev.org/c/zuul/nodepool/+/777689 | 00:08 |
|---|---|---|
| *** tosky has quit IRC | 00:12 | |
| *** ianychoi__ is now known as ianychoi | 00:34 | |
| clarkb | fungi: well and when we switched to using docker containers the playbooks got updated to do docker-compose down which just sends a sigint and the process dies immediately | 00:35 |
| clarkb | prior to that we used stop which is less graceful than graceful but more graceful than what docker-compose down does | 00:35 |
| corvus | mostly the difference is when the build directory cleanup happens; stop=as the process shuts down; docker-compose down=when we start it up again | 00:39 |
| *** wuchunyang has joined #zuul | 00:54 | |
| *** wuchunyang has quit IRC | 00:58 | |
| *** hamalq has quit IRC | 01:10 | |
| *** ikhan has quit IRC | 01:48 | |
| *** jhesketh_ is now known as jhesketh | 01:49 | |
| *** saneax has joined #zuul | 02:42 | |
| *** saneax has quit IRC | 02:43 | |
| *** EmilienM has joined #zuul | 02:46 | |
| *** ricolin has joined #zuul | 03:15 | |
| *** zbr has joined #zuul | 03:50 | |
| *** tflink_ is now known as tflink | 03:56 | |
| *** dmsimard has quit IRC | 04:08 | |
| *** dmsimard has joined #zuul | 04:09 | |
| *** jfoufas1 has joined #zuul | 05:20 | |
| *** evrardjp_ has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| openstackgerrit | Merged zuul/zuul master: Document zuul-executor graceful https://review.opendev.org/c/zuul/zuul/+/777699 | 05:50 |
| tobiash | corvus: regarding the failing nodepool-functional-k8s: the problem is not nodepool but the pod fails to start with backoff pulling image | 06:59 |
| tobiash | there is not much more information than the pod cannot pull docker.io/fedora:28 | 07:02 |
| tobiash | presumably due to rate limiting | 07:02 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Use different image in nodepool-functional-k8s https://review.opendev.org/c/zuul/nodepool/+/777719 | 07:09 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Make nodepool-functional-k8s more docker registry friendly https://review.opendev.org/c/zuul/nodepool/+/777721 | 07:16 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/c/zuul/zuul/+/685354 | 07:21 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Allow refreshing changes in canMerge() check https://review.opendev.org/c/zuul/zuul/+/767084 | 07:21 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Check cycle items are mergeable before reporting https://review.opendev.org/c/zuul/zuul/+/743450 | 07:21 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Make reporting asynchronous https://review.opendev.org/c/zuul/zuul/+/691253 | 07:21 |
| *** sassyn has joined #zuul | 07:21 | |
| sassyn | Hi All, Good Morning. Happy Friday.... | 07:22 |
| sassyn | I have an issue with my Zuul system which I can't find a solution for. Sometimes when a job is getting into zuul, I can see that the web management get the job. Clicking on the Job also show the black web console. However the job start after few min. So for example The first line in the web console log is 10:00 and the job only start at 10:06. Why? | 07:24 |
| sassyn | any idea? | 07:24 |
| *** icey_ is now known as icey | 07:43 | |
| *** yoctozepto5 has joined #zuul | 08:26 | |
| *** yoctozepto has quit IRC | 08:28 | |
| *** yoctozepto5 is now known as yoctozepto | 08:28 | |
| *** yoctozepto6 has joined #zuul | 08:32 | |
| *** yoctozepto has quit IRC | 08:33 | |
| *** yoctozepto6 is now known as yoctozepto | 08:33 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Remove superfluous flushes and queries from SQL reporter https://review.opendev.org/c/zuul/zuul/+/752664 | 08:36 |
| avass | sassyn: I think that could be a delay between when an executor accepts a job and it actually starts the job or something along those lines | 08:44 |
| avass | sassyn: so the executor would still need to clone and setup everything before it will start running ansible and show any output | 08:44 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Report executor stats per zone https://review.opendev.org/c/zuul/zuul/+/740448 | 08:58 |
| *** harrymichal has joined #zuul | 09:06 | |
| *** samccann has quit IRC | 09:37 | |
| *** samccann has joined #zuul | 09:37 | |
| *** SaifAddin has joined #zuul | 09:41 | |
| SaifAddin | Hi, got a cicd zuul job running that starts on a server using .yaml files. It is running an Ansible job with some specific Ansible inventory that cannot have network defaults. Is there a way to add through YAML, a file with a list of variables to be used for the job? | 09:44 |
| SaifAddin | note: I can't run zuul from command-line as it runs from the server. From command line this could be addressed with extra-vars with the -e command (and pass a file), but I can't do that here. Is there any equivalent from YAML? | 09:44 |
| tobiash | sassyn, avass: that depends on how long the executor takes to prepare all the repos it needs into the job workspace. | 09:55 |
| tobiash | sassyn: also, how many executors do you have and which storage backend? | 09:55 |
| tobiash | also depending on available ram etc executors pause themselves to prevent overloading. In this case job requests get queued if all executors pause themselves | 09:56 |
| tobiash | sassyn: you can see such a behavior e.g. here: https://grafana.opendev.org/d/5Imot6EMk/zuul-status?orgId=1&from=1614312602527&to=1614327553547 | 09:57 |
| tobiash | look at the graphs 'executors' and 'executor queue' | 09:58 |
| avass | SaifAddin: you can load varaibles during runtime instead, that could be easier. | 10:01 |
| SaifAddin | Thanks avass but it is a long list of variables (with hostnames, ports, usernames), and don't want to put the entire list as extra-vars keywords | 10:02 |
| SaifAddin | is it possible somehow to pass an entire .yaml file as extra vars? | 10:02 |
| avass | SaifAddin: I mean load the file with ansible with include_vars or similar | 10:11 |
| avass | but there's no way to point zuul to a yaml file with variables it should use afaik | 10:13 |
| SaifAddin | oh:( I wish that worked through zuul | 10:15 |
| *** mugsie_ is now known as mugsie | 10:16 | |
| *** jangutter has joined #zuul | 10:37 | |
| *** jangutter_ has quit IRC | 10:39 | |
| *** jangutter_ has joined #zuul | 10:40 | |
| *** jangutte_ has joined #zuul | 10:43 | |
| *** jangutter has quit IRC | 10:44 | |
| *** jangutter_ has quit IRC | 10:45 | |
| *** yoctozepto8 has joined #zuul | 10:47 | |
| *** yoctozepto has quit IRC | 10:47 | |
| *** yoctozepto8 is now known as yoctozepto | 10:47 | |
| *** yoctozepto1 has joined #zuul | 10:54 | |
| *** yoctozepto has quit IRC | 10:55 | |
| *** yoctozepto1 is now known as yoctozepto | 10:55 | |
| *** iurygregory has quit IRC | 11:13 | |
| *** tosky has joined #zuul | 11:25 | |
| *** yoctozepto2 has joined #zuul | 11:32 | |
| *** yoctozepto has quit IRC | 11:33 | |
| *** yoctozepto2 is now known as yoctozepto | 11:33 | |
| *** harrymichal has quit IRC | 12:46 | |
| *** yoctozepto4 has joined #zuul | 12:46 | |
| *** harrymichal has joined #zuul | 12:46 | |
| *** yoctozepto has quit IRC | 12:46 | |
| *** yoctozepto4 is now known as yoctozepto | 12:46 | |
| *** jangutte_ is now known as jangutter | 13:17 | |
| *** zbr9 has joined #zuul | 13:20 | |
| *** zbr has quit IRC | 13:22 | |
| *** zbr9 is now known as zbr | 13:22 | |
| *** zbr0 has joined #zuul | 13:43 | |
| *** zbr has quit IRC | 13:46 | |
| *** zbr0 is now known as zbr | 13:46 | |
| *** iurygregory has joined #zuul | 13:49 | |
| *** jangutter_ has joined #zuul | 13:55 | |
| *** jangutter has quit IRC | 13:59 | |
| *** zbr3 has joined #zuul | 14:03 | |
| *** zbr has quit IRC | 14:05 | |
| *** zbr3 is now known as zbr | 14:05 | |
| *** zbr7 has joined #zuul | 14:08 | |
| *** zbr has quit IRC | 14:10 | |
| *** zbr7 is now known as zbr | 14:10 | |
| fungi | zuul is designed to get variables from basically two sources: static inclusion in job configuration, and dynamically through pipeline triggers | 14:32 |
| fungi | though the ansible which zuul runs can get variables set through additional means | 14:33 |
| *** zbr4 has joined #zuul | 14:33 | |
| fungi | for example a pre-run playbook could fetch them based on some heuristic and set them in cached facts for other playbooks to consume in the same build | 14:34 |
| *** zbr has quit IRC | 14:35 | |
| *** zbr4 is now known as zbr | 14:35 | |
| SaifAddin | Thanks fungi , any leads onto how to set the cached facts? I was trying with pre-run playbook but I think the variables are not passing to the next job. How can I put them in this "cached fact"? | 14:37 |
| fungi | SaifAddin: here's an example: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L16-L18 | 14:47 |
| fungi | use set_fact and make cacheable true in the fact | 14:47 |
| fungi | it's a basic ansible feature | 14:48 |
| fungi | the executor should maintain a fact cache, and provide those facts to later playbooks within the same build | 14:48 |
| *** zbr4 has joined #zuul | 14:52 | |
| *** zbr has quit IRC | 14:53 | |
| *** zbr4 is now known as zbr | 14:53 | |
| SaifAddin | Thanks fungi , I'll be trying this. Totally skipped it. I hope that works. | 15:04 |
| *** zbr5 has joined #zuul | 15:12 | |
| *** zbr has quit IRC | 15:14 | |
| *** zbr5 is now known as zbr | 15:14 | |
| fungi | if anyone's interested on providing input, the cd foundation's interoperability sig has been discussing "policy-driven ci/cd" and is looking for more case studies/examples: https://lists.cd.foundation/g/sig-interoperability/topic/policy_driven_ci_cd/80929157?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,80929157 | 15:17 |
| SaifAddin | So Far no luck. Still getting undefined variables | 15:17 |
| SaifAddin | Following kind of that example | 15:17 |
| fungi | SaifAddin: are you accessing them as facts? | 15:17 |
| SaifAddin | :O | 15:17 |
| SaifAddin | normal {{ .... }} | 15:18 |
| SaifAddin | ahh got it | 15:19 |
| *** tosky has quit IRC | 15:19 | |
| SaifAddin | mmh, unfortunately not feasible. I don't want to change all variables access in the project to fetch from catched | 15:19 |
| *** tosky has joined #zuul | 15:20 | |
| corvus | cached facts are accessed as normal ansible variables | 15:21 |
| SaifAddin | By what I read in the docs, fact variables are accessed as `{{ ansible_facts['nodename'] }}` | 15:22 |
| fungi | well, i meant you weren't trying to access them via the zuul variable namespace or something | 15:22 |
| fungi | you just access them by name as a top level variable | 15:23 |
| fungi | if you're setting fact "foo" cacheable in a playbook and then trying to refer to {{ foo }} in another later playbook and it's acting like it's not set, then maybe something is wrong with the fact caching on the executor | 15:25 |
| corvus | SaifAddin: in the test you're doing, is that using zuul, or are you running ansible manually/locally? | 15:26 |
| SaifAddin | What I did was creating a Playbook, that calls a role. In the Role I put it like: | 15:26 |
| SaifAddin | ``` | 15:26 |
| SaifAddin | - name: setting_cache_variables | 15:26 |
| SaifAddin | set_fact: | 15:26 |
| SaifAddin | ansible_connection: local | 15:26 |
| SaifAddin | other_variables_foo: bar | 15:26 |
| SaifAddin | cacheable: true | 15:26 |
| SaifAddin | ``` | 15:26 |
| SaifAddin | And this playbook I ran in a pre-run calling this Playbook. | 15:27 |
| SaifAddin | From the actual run, at some point in one of the tasks I did {{ ansible_connection}} but then I got that ansible_connection was undefined | 15:27 |
| SaifAddin | My test runs through zuul. With Ansible this would work with just extra vars -e or something like that. | 15:27 |
| SaifAddin | The problem is that I need to pass extra variables to the Ansible job | 15:28 |
| SaifAddin | (and they are too many and don't want to list them as extra vars individually, but have them in a file) | 15:28 |
| SaifAddin | EDIT: I think I had messing up something. But I think this actually works (checking...) | 15:30 |
| *** zbr8 has joined #zuul | 15:30 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 15:31 |
| *** zbr has quit IRC | 15:33 | |
| *** zbr8 is now known as zbr | 15:33 | |
| *** jangutter has joined #zuul | 15:36 | |
| corvus | SaifAddin: that should work for "other_variables_foo". but it may not work for ansible_connection -- i don't think ansible will override an inventory variable from a fact cache. | 15:39 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 15:40 |
| SaifAddin | Alright, will keep that in mind! | 15:40 |
| *** jangutter_ has quit IRC | 15:40 | |
| SaifAddin | That is actually for me, I did not want to override inventory variables | 15:40 |
| SaifAddin | Extra question (thanks all btw). Couldn't I also consider include_vars (Ansible), coming from a variable defined in zuul (with the path to it) | 15:41 |
| SaifAddin | Extra question (thanks all btw). Couldn't we also consider include_vars (Ansible), coming from a variable defined in zuul (with the path to it) | 15:41 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 15:42 |
| corvus | SaifAddin: yes -- i think avass recommended something like that | 15:46 |
| SaifAddin | Got it. I didn't realize that I could pass the path through zuul. That's my plan B now. Will see | 15:50 |
| SaifAddin | Thanks!! | 15:50 |
| corvus | SaifAddin: ah yep. with include_vars you can use a relative path, or if you need to construct an absolute path, you may find some of the zuul variables useful: https://zuul-ci.org/docs/zuul/reference/jobs.html#zuul-variables | 15:52 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 15:52 |
| corvus | SaifAddin: and you can do things like 'vars: {include_path: "{{ zuul.work_dir }}/path/to/file"}' in the job definition (ie, you can use jinja templating in the string, zuul will ignore that and pass it through to ansible to do the templating). | 15:53 |
| SaifAddin | Perfect, before plan B | 15:56 |
| SaifAddin | I am stuck on plan A. The documentation says it should be cacheable: yes. The example shared above said cacheable: true | 15:56 |
| corvus | SaifAddin: in yaml they are the same | 15:57 |
| SaifAddin | Dang, then I still have variables not coming through xD. The set_fact + cacheable .. aghh | 15:57 |
| *** harrymichal has quit IRC | 15:57 | |
| SaifAddin | Good, making progress. Yep, starting to make sense | 16:01 |
| *** jfoufas1 has quit IRC | 16:02 | |
| *** zbr2 has joined #zuul | 16:11 | |
| *** zbr has quit IRC | 16:13 | |
| *** zbr2 is now known as zbr | 16:13 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 16:15 |
| *** zbr6 has joined #zuul | 16:15 | |
| clarkb | guillaumec: note that zuul supports python3.6+ only | 16:16 |
| clarkb | guillaumec: you shouldn't need to support 3.5 | 16:17 |
| corvus | gear might though | 16:17 |
| *** zbr has quit IRC | 16:17 | |
| *** zbr6 is now known as zbr | 16:17 | |
| clarkb | yup, but I think if you always do PROTOCOL_TLS from the 3.6+ side that will work for gear on the remote | 16:18 |
| clarkb | basically the zuul code doesn't need to fallback to handle python3.5 specific ssl lib oddities aiui | 16:19 |
| clarkb | 3.5 can speak tls 1.0, 1.1, and 1.2. The weirdness is in the python itself to select the one you want | 16:19 |
| *** SaifAddin has quit IRC | 16:24 | |
| *** icey has quit IRC | 16:33 | |
| *** icey has joined #zuul | 16:34 | |
| *** zbr has quit IRC | 16:38 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 16:51 |
| avass | tobiash, corvus: dockerhubs rate limit seem to be popping up now and then as a problem. maybe it would actually be worth it for opendev to host their own registry? | 16:52 |
| avass | there's also the option of setting up a pull-through cache i suppose | 16:52 |
| guillaumec | clarkb, that's why: "-min python version is 3.6, only keep PROTOCOL_TLS" , more explicit now | 16:53 |
| fungi | avass: we've talked about it. last time we looked the existing solutions weren't very flexible (e.g. downtime required to prune old entries, needing to maintain a list somewhere of every image which any job might use, still need something to do the copying of images from public registries, et cetera) | 16:54 |
| clarkb | but also we don't want to be a registry of record. or at least i don't want to be. I think that presents its own set of unique challenges. I'd rather we continue to mirror/cache what is on an upstream registry | 16:55 |
| fungi | maybe with the rate limits starting to hit people harder, someone might have written/improved some options for self-hosted dockerhub mirrors | 16:55 |
| fungi | but yes, that's also a problem... the way our infrastructure is designed, if we set up a dockerhub proxy/mirror it's likely random people on the internet will start pulling images from us instead of dockerhub to get around the rate limits | 16:56 |
| fungi | so any solution we design would need to take that into account | 16:56 |
| clarkb | another option is to try quay. They claim to not rate limit | 16:57 |
| avass | yeah that could indeed be a problem | 16:58 |
| fungi | are the images we need available from quay? i guess the one which came up today was the fedora image, so it's probably there | 16:58 |
| clarkb | fungi: I don't think the python ones are, but we could build our python-builder and python-base images and push them to quay using the bases found on docker hub periodically | 17:00 |
| clarkb | then zuul and opendev images would just pull what we've pushed. (theoretically, I've not tested any of that) | 17:00 |
| fungi | but we probably also don't want users pulling our python-base image instead of the official one, in case there's a python vulnerability and we're behind building/fetching new python-base or something | 17:01 |
| fungi | better would be to convince the upstream maintainers for our image dependencies to also upload to quay | 17:02 |
| clarkb | fungi: well that is how it works already | 17:02 |
| clarkb | if a user pulls zuul, they will pull our python-base, and the version of the python image that python-base is based on (not the latest python image) | 17:02 |
| fungi | er, yeah sorry, i meant the python image | 17:03 |
| clarkb | we wouldn't publish python directly, it would just be a layer under python-base | 17:03 |
| fungi | right, but we would want the python image to be published to quay by its maintainers, rather than us copying it into quay and ending up with users consuming that | 17:04 |
| fungi | in case we're behind on copying or whatever (not just zuul users, i'm talking about anyone who finds there's a copy of the python image on quay and decides relying on that's a good idea to get around dockerhub's rate limits) | 17:05 |
| corvus | tobiash: ftr re the k8s nodepool job -- my additional concern was that nodepool threw null dereference exceptions during the pod launch failure | 17:05 |
| clarkb | right, what I'm saying is that we wouldn't copy it into quay as a separate thing. It would only be part of our images that we publish | 17:05 |
| tobiash | corvus: oh, have to look again, I thought there were just the unclean shutdown messages | 17:06 |
| fungi | clarkb: but how does that solve our dockerhub rate limit problem then, if we need to obtain the python image from dockerhub? | 17:06 |
| clarkb | fungi: we don't it would be fetched from quay, but not as a directly addressable layer aiui | 17:06 |
| clarkb | fungi: we would be doing a layer squash essentially | 17:07 |
| corvus | tobiash: oh, i assumed not due to the NoneType exeption being attached to the "pod failed" exception with "During handling of the above exception, another exception occurred:" | 17:07 |
| fungi | clarkb: but how does it get into quay for us to get it from there? | 17:07 |
| clarkb | fungi: we would publish python-builder and python-base to quay | 17:07 |
| tobiash | corvus: afaik those exceptions are spammed in many test cases | 17:07 |
| fungi | clarkb: python-base needs the python image though, right? | 17:07 |
| fungi | wouldn't we still need to get the python image from somewhere? | 17:08 |
| fungi | or are we building it from source? | 17:08 |
| corvus | tobiash: hrm. it's a bit hard to untangle. :) | 17:08 |
| tobiash | after shutdown of the zk connection due to dangling threads wanting to write to zk | 17:08 |
| clarkb | fungi: we are not building from source. The upload to quay should take the layers for the python image and include them in the upload to python-base/python-builder if they are not already there | 17:08 |
| tobiash | when running single tests locally I always see such exceptions | 17:08 |
| corvus | fungi: we rebuild python-base less frequently than zuul/nodepool. so the problem exists, but the frequency is reduced. | 17:08 |
| clarkb | fungi: when we make that push to docker hub docker hub says I have these layers already and it is a noop | 17:08 |
| clarkb | when we push to quay, quay will accept them and tie them to python-base/python-builder | 17:09 |
| corvus | tobiash: yeah, i'm familiar with that, i just thought it was related, but maybe that's just an artifact of the call stack | 17:09 |
| corvus | fungi: in other words with clarkb's suggestion, we only need to be able to build python-base every X days in order to support building zuul every X minutes. | 17:10 |
| corvus | er Y. those are independent variables :) | 17:10 |
| fungi | clarkb: yeah, what i was getting at is we still need to get the python image from somewhere, and if not from dockerhub then where are we getting it? but as corvus says maybe less of a problem because we don't do that step very often | 17:10 |
| corvus | is fedora on quay? | 17:11 |
| tobiash | corvus: look at https://review.opendev.org/c/zuul/nodepool/+/777719 ;) | 17:11 |
| fungi | corvus: i assumed it might be due to the red hat connection, but i haven't looked | 17:11 |
| corvus | tobiash: yes that is the change i was going to write, thanks :) | 17:12 |
| corvus | tobiash: lol one of the functests failed | 17:12 |
| tobiash | there is a followup that changes to busybox but I think I broke the config | 17:12 |
| clarkb | fungi: right we would only need to talk to docker hub when rebuilding python-base/python-builder | 17:13 |
| corvus | tobiash: comment on 777721 | 17:14 |
| corvus | what is this setuptools_rust thing? https://zuul.opendev.org/t/zuul/build/2219649bf70845c698b93b1ee1edf54b | 17:14 |
| clarkb | corvus: it doesn't seem like there are "official" images on quay.io but I may be missing them | 17:14 |
| clarkb | however, there is fedora/fedora which is maybe functionally the same thing | 17:15 |
| corvus | clarkb: https://review.opendev.org/777719 from tobiash looks pretty officialish? | 17:15 |
| corvus | yeah | 17:15 |
| corvus | like, i doubt redhat is going to let anyone squat the fedora namespace on quay :) | 17:15 |
| fungi | corvus: pyca/cryptography recently added rust bindings, and needs setuptools_rust if being built from sdist | 17:15 |
| clarkb | when they publish new releases they tend to push the sdist first | 17:16 |
| corvus | fungi: why did that hit that error in that job? | 17:16 |
| *** hamalq has joined #zuul | 17:16 | |
| clarkb | corvus: ^ because of that typically | 17:16 |
| corvus | fungi, clarkb: oh, so that was a race with a release? | 17:16 |
| fungi | corvus: that log shows trying to consume cryptography as an sdist, so it will end up wanting setuptools_rust which is defined in a pyproject.toml which needs fairly new pip to parse | 17:17 |
| clarkb | we rely on their wheels for the jobs to function, but when they release they don't always push wheels first | 17:17 |
| clarkb | we could add all the build deps to the jobs and have them handle the sdist scenario. Or possibly convince pip that using a wheel for the previous version is preferably to sdist for latest | 17:17 |
| fungi | https://pypi.org/project/cryptography/#history shows 3.4.6 was uploaded 10 days ago | 17:18 |
| corvus | hrm. we're looking at jobs that failed this morning | 17:18 |
| fungi | and that build was from today, yeah | 17:18 |
| clarkb | in that case I wonder if pypi's bad index caching is back | 17:18 |
| clarkb | and it is serving us an index without the wheels in it | 17:18 |
| clarkb | oh or we don't know how to use abi3 wheels | 17:19 |
| fungi | that ran on bionic | 17:19 |
| fungi | so probably older pip with no abi3 support, right | 17:19 |
| clarkb | corvus: if it is the abi3 issue then upgrading pip in the job should address it | 17:20 |
| tobiash | corvus: thanks | 17:20 |
| fungi | cryptography recently stopped publishing per-cpython-version wheels and switched to publishing abi3 wheels instead which can cover multiple cpython versions. unfortunately needs fairly new pip which knows to look for those | 17:20 |
| * fungi refreshes his memory on which pip release added abi3 support | 17:21 | |
| fungi | pip 20.0 added abi3 support, january 2020 | 17:22 |
| guillaumec | is there a way to execute some sort of parent's playbook after a job child's pre-run and before job child's run ? (some sort of post-pre-run, or pre-run-post ^^) | 17:22 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Make nodepool-functional-k8s more docker registry friendly https://review.opendev.org/c/zuul/nodepool/+/777721 | 17:22 |
| tobiash | fungi: yeah, we fell over this a couple of weeks ago at various places | 17:23 |
| avass | guillaumec: no, we usually solve that by making the jobs configurable enough. but that depends on what you need to achieve of course | 17:23 |
| tobiash | fungi: hence the change https://review.opendev.org/c/zuul/nodepool/+/775017 | 17:23 |
| tobiash | but that failed three times in gate with three errors | 17:24 |
| fungi | pip 10.0.0 is what added support for build dependencies in pyproject.toml, btw. so basically latest cryptography can't be installed at all with pip<10 | 17:24 |
| fungi | and the python3-pip in ubuntu-bionic is 9.0.1 | 17:24 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 17:28 |
| corvus | fungi, clarkb, tobiash, avass: summary: tobiash has some quick fixes in progress to start using quay for one image in func test jobs. we could continue that approach by moving zuul and python-base and -builder into quay as well. that would eliminate a lot of the docker image pulls in most of our builds. | 17:37 |
| fungi | sounds like a reasonable approach | 17:37 |
| tobiash | corvus: would you then also publish zuul in quay or do that still on docker hub? | 17:37 |
| avass | sounds good. probably a good idea to announce the move to quay when it comes to that | 17:37 |
| corvus | however, zuul/nodepool publishing location is more or less a public api, so we should weigh that... | 17:38 |
| corvus | (python-base/builder too, but more of an "internal" api i guess) | 17:38 |
| corvus | if we're going to move zuul/nodepool, i think we should seriously consider whether we want to move it to quay or to registry.zuul-ci.org | 17:38 |
| corvus | (can we cname registry.zuul-ci.org to quay and have that work?) | 17:39 |
| tobiash | corvus: cnaming will throw ssl exceptions | 17:39 |
| corvus | i wonder if some kind of http redirect would work (i dunno what the auth system would think about that) | 17:40 |
| fungi | is there a reason not to publish to both dockerhub and quay? | 17:40 |
| corvus | fungi: we might be able to do that without doing any pulls for builds... | 17:41 |
| fungi | yeah, i'm oblivious to a lot of the nuances there | 17:41 |
| corvus | my main concern would be that ^ first of all (make sure we get the benefit of moving outside the rate cap) and second, user confusion, and third, double our potential problem area | 17:42 |
| fungi | i guess "publish to x" is not as trivial as just uploading some files | 17:42 |
| tobiash | are pushes to dockerhub also rate limited or just the pulls? | 17:42 |
| corvus | tobiash: these would be authenticated at least | 17:42 |
| tobiash | ah, so no rate limit problem there likely | 17:42 |
| corvus | yeah, we'd have a larger cap | 17:42 |
| corvus | but i still worry about a future where we make a release and find that dockehub failed and quay succeeded | 17:42 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 17:42 |
| tobiash | so we could switch the build chain to quay and keep publishing to dockerhub for now (and migrate later to quay if we want to) | 17:43 |
| tobiash | I think we should decide on dockerhub xor quay for publishing | 17:43 |
| corvus | tobiash: yeah, i think so. i'm in favor of migrating though, so if we do both, consider it temporary. | 17:43 |
| corvus | tobiash: ++ | 17:43 |
| tobiash | I wonder if we should do some deprecation/update notice about the new image location then | 17:44 |
| tobiash | like from release x zuul/nodepool images will be published on quay | 17:44 |
| corvus | yeah, i'd be fine with a cutover and not publishing to both. i think if we leave the old images in place that's sufficient. | 17:45 |
| tobiash | x could be 4.1 then or of changing the image is considered a breaking api 5.0 | 17:45 |
| corvus | i think 4.1 would be fine | 17:45 |
| tobiash | ++ | 17:45 |
| clarkb | corvus: I think we should publish to both docker hub and quay to start with python-base and python-builder | 17:59 |
| clarkb | for zuul and nodepool I think cutting over is fine, but the python-* images have a number of users now aiui | 17:59 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: [DNM] Support ssl encrypted fingergw https://review.opendev.org/c/zuul/zuul/+/777761 | 18:00 |
| corvus | clarkb, tobiash, fungi: a *really* quick local test suggests that 301 redirects might work | 18:02 |
| corvus | so if we wanted to do a cutover, and only do it once, we might consider setting up a static server to redirect to docker/quay. we could move at will and even self-host in the future if we wanted. | 18:03 |
| corvus | at least -- i think my test is sufficient to say we should try it for real on a server with ssl certs, etc. :) | 18:04 |
| fungi | sounds like not a bad idea at all | 18:09 |
| *** tjgresha has joined #zuul | 18:13 | |
| *** tjgresha has quit IRC | 18:16 | |
| *** tjgresha has joined #zuul | 18:22 | |
| openstackgerrit | Merged zuul/nodepool master: Upgrade pip before installing zuul https://review.opendev.org/c/zuul/nodepool/+/775017 | 18:30 |
| *** jangutter_ has joined #zuul | 19:03 | |
| *** jangutter has quit IRC | 19:07 | |
| tjgresha | question regarding check pipeline success / failure comments on patches | 19:13 |
| avass | we are now running zuul 4 :) | 19:13 |
| tobiash | yay | 19:17 |
| tjgresha | nice! | 19:18 |
| corvus | tjgresha: feel free to just go ahead and ask :) | 19:21 |
| avass | I get a rpcfailure when trying to dequeue a buildset in a timed pipeline :(, don't think that's new however | 19:21 |
| avass | zuul.rpcclient.RPCFailure: Ref refs/heads/develop does not belong to project "..." | 19:22 |
| tjgresha | k .. didnt wanna steal the zuul4 thunder for just a minute | 19:22 |
| tobiash | thundering zuul | 19:23 |
| tjgresha | excellent band name | 19:23 |
| corvus | avass: try using a more or less qualified project name? | 19:23 |
| avass | tjgresha: if anything I stole the attention from you :) | 19:23 |
| tobiash | avass: I think I've dequeued changes from periodic pipelines in the past, but that was always a fiddling with the ref names | 19:24 |
| tobiash | looking forward to the dequeue button | 19:24 |
| corvus | avass: either example.com/org/project or org/project or project | 19:24 |
| avass | corvus: worked, canonical name didn't and that's what was generated from zuul-changes.py | 19:24 |
| corvus | avass: line 1186 of scheduler if you want to change both of those to canonical_name :) | 19:25 |
| avass | corvus: ++ | 19:25 |
| corvus | (er, unsure about that line #, but it's _doDequeueEvent) | 19:25 |
| avass | would it be possible to merge enqueue and enqueue-ref? feels a bit inconsistent when dequeue can handle both :) | 19:26 |
| corvus | avass: i think they're different based on the different data (like old/new ref etc). they could probably be merged, but half the options would be irrelevant depending on which you were doing | 19:28 |
| corvus | (er, i meant old/new sha) | 19:28 |
| tjgresha | So Intel_Zuul is not leaving verification comments in Nova, Neutron, but it is in the ci-sandbox - same gerrit connection in my zuul.conf -- it is triggering, running, completing successfully on gerrit events on these repos | 19:30 |
| clarkb | tjgresha: are you trying to leave +/-1 votes? | 19:31 |
| clarkb | I think projects like nova, neutron etc restrict who is allowed to do that | 19:31 |
| corvus | something in tjgresha's commets are rendering them invisible to me | 19:32 |
| corvus | like some unicode character or something | 19:32 |
| tjgresha | using webchat maybe to do with it? | 19:32 |
| clarkb | corvus: there is maybe some weird whitespacing | 19:32 |
| corvus | i see them on eavesdrop, but i'm getting missing lines in my irc window | 19:32 |
| corvus | tjgresha: i saw that comment about webchat :) | 19:33 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Update _doDequeueEvent to check canonical name https://review.opendev.org/c/zuul/zuul/+/777816 | 19:33 |
| corvus | it's probably my fault for not having a sufficiently unicode-aware irc client. sorry about that, and if i ignore you, it's not intentional! | 19:34 |
| tjgresha | vpn'd into lab at Intel where our 3rdparty CI runs, changing proxy back and forth makes webchat easier sometimes.. | 19:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Update _doDequeueEvent to check canonical name https://review.opendev.org/c/zuul/zuul/+/777816 | 19:34 |
| avass | updated commit message | 19:34 |
| clarkb | tjgresha: typically when that happens it is because you don't have sufficient permissions to vote on the project. Intead they jus want you to leave a comment without hte +/-1 | 19:35 |
| tjgresha | we had them -- maybe there was an account cleanup that happened... will check with owners | 19:36 |
| tjgresha | thanks for advice :] | 19:38 |
| clarkb | tjgresha: you can check the projectname-ci group membership in gerrit to see who is allowed to vote +/-1 | 19:40 |
| clarkb | nova for example only allows three accounts to do it | 19:40 |
| tjgresha | there is a 3rdparty-ci group | 19:45 |
| fungi | but it's not used by nova's acl. also this discussion should probably be taking place in #opendev (or #openstack-infra or #openstack-nova) | 19:48 |
| *** ikhan has joined #zuul | 19:50 | |
| tjgresha | fungi - will move my questions there now that i know its the likely cause | 19:58 |
| *** tjgresha has quit IRC | 22:17 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support enqueuing behind circular dependencies https://review.opendev.org/c/zuul/zuul/+/777843 | 22:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support enqueuing behind circular dependencies https://review.opendev.org/c/zuul/zuul/+/777843 | 22:59 |
| *** sduthil has quit IRC | 23:37 | |
| *** hamalq has quit IRC | 23:40 | |
| corvus | clarkb: I carried your +2 over what I'm pretty sure is a merge conflict patch update on https://review.opendev.org/752079 (tobiash: i left some +2 comments on that) | 23:44 |
| clarkb | ok | 23:45 |
| clarkb | er not sure where those extra spaces came from | 23:45 |
| fungi | i'm wracking my brain trying to figure out how to get the related AnsibleJob context in https://review.opendev.org/777441 | 23:52 |
| fungi | the errors logged by that codepath do include an event and build id, so there must be some way to figure it out | 23:53 |
| clarkb | fungi: that innerUpdateLoop runs in a separate thread as a gearman worker pulling off update jobs | 23:57 |
| clarkb | fungi: I think you may need to return an indicator via gearman to the job that requested the update that the job should be aborted as the job context isn't known in that separate thread if I read this right | 23:58 |
| clarkb | only the gearman git update request is known but not more than that | 23:58 |
| fungi | weird then that the log call there actually reports an event/build | 23:58 |
| clarkb | oh that isn't gearman its an internal python queue | 23:59 |
| corvus | how about we retry the task? | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!