| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-container-image: support sibling copy https://review.opendev.org/697936 | 00:00 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix up siblings copy https://review.opendev.org/697614 | 00:00 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-container-image: support sibling copy https://review.opendev.org/697936 | 00:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix up siblings copy https://review.opendev.org/697614 | 00:14 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fixes tox log fetching when envlist is set to 'ALL' https://review.opendev.org/696531 | 00:18 |
| *** irclogbot_0 has joined #zuul | 00:24 | |
| *** irclogbot_0 has quit IRC | 00:40 | |
| *** jamesmcarthur has joined #zuul | 00:40 | |
| *** sgw has quit IRC | 01:11 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** irclogbot_0 has joined #zuul | 01:28 | |
| *** irclogbot_0 has quit IRC | 01:48 | |
| *** bhavikdbavishi has joined #zuul | 02:40 | |
| *** bhavikdbavishi1 has joined #zuul | 02:43 | |
| *** bhavikdbavishi has quit IRC | 02:44 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:44 | |
| *** irclogbot_2 has joined #zuul | 02:48 | |
| mnaser | ianw: did you end up breaking through with that dib in container bug? | 03:00 |
| ianw | mnaser: so i think dib is ok in containers; the one real issue was that it was inspecting the host init system, which failed in a container without an init system | 03:43 |
| ianw | mnaser: so i have had nodepool-builder+dib build and boot images in the functional test. there's a lot in flight though about how we generate those images -- so at the moment it's no so much if we can, but *how* we go about building the containers | 03:44 |
| ianw | 697614 697936 add "sibling" installs to podman/docker builds, so we can build nodepool+dib+openstacksdk all into one container in the gate | 03:45 |
| ianw | corvus: ^ reviews of those most welcome, it's been updated around the recently merged podman work | 03:46 |
| mnaser | ianw: awesome. I’ll have a look at those tomorrow when I get sometime | 03:46 |
| *** bhavikdbavishi has quit IRC | 03:47 | |
| ianw | thanks, with those merged, i expect i can get the nodepool functional job using containers (theoretically, assuming no other issues pop up in practice :) | 03:48 |
| *** jamesmcarthur has joined #zuul | 04:11 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** bhavikdbavishi has joined #zuul | 04:34 | |
| *** bhavikdbavishi1 has joined #zuul | 04:36 | |
| *** bhavikdbavishi has quit IRC | 04:38 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:38 | |
| *** jamesmcarthur has joined #zuul | 04:42 | |
| *** jamesmcarthur has quit IRC | 04:48 | |
| *** jamesmcarthur has joined #zuul | 05:15 | |
| *** jamesmcarthur has quit IRC | 05:20 | |
| *** jamesmcarthur has joined #zuul | 05:25 | |
| *** jamesmcarthur has quit IRC | 05:30 | |
| *** raukadah is now known as chkumar|ruck | 06:04 | |
| *** jamesmcarthur has joined #zuul | 06:26 | |
| *** jamesmcarthur has quit IRC | 06:31 | |
| *** jamesmcarthur has joined #zuul | 07:05 | |
| *** AJaeger has quit IRC | 07:09 | |
| *** jamesmcarthur has quit IRC | 07:11 | |
| *** AJaeger has joined #zuul | 07:13 | |
| *** igordc has joined #zuul | 08:01 | |
| *** jamesmcarthur has joined #zuul | 08:07 | |
| *** sshnaidm|off is now known as sshnaidm | 08:11 | |
| *** jamesmcarthur has quit IRC | 08:11 | |
| *** jangutter has joined #zuul | 08:11 | |
| *** themroc has joined #zuul | 08:13 | |
| *** jangutter has quit IRC | 08:16 | |
| *** jangutter has joined #zuul | 08:16 | |
| *** avass has joined #zuul | 08:17 | |
| *** tosky has joined #zuul | 08:25 | |
| *** jcapitao|afk has joined #zuul | 08:30 | |
| *** igordc has quit IRC | 08:36 | |
| *** jamesmcarthur has joined #zuul | 08:46 | |
| *** saneax has joined #zuul | 08:51 | |
| *** jamesmcarthur has quit IRC | 08:51 | |
| *** avass has quit IRC | 09:04 | |
| *** jcapitao|afk is now known as jcapitao | 09:07 | |
| *** yolanda has joined #zuul | 09:09 | |
| *** hashar has joined #zuul | 09:29 | |
| *** jamesmcarthur has joined #zuul | 09:47 | |
| *** jamesmcarthur has quit IRC | 09:52 | |
| *** ssbarnea has quit IRC | 10:16 | |
| *** jamesmcarthur has joined #zuul | 10:28 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: enqueue: make trigger deprecated https://review.opendev.org/695446 | 10:30 |
| *** saneax has quit IRC | 10:30 | |
| *** saneax has joined #zuul | 10:30 | |
| *** jamesmcarthur has quit IRC | 10:33 | |
| *** mhu has joined #zuul | 10:58 | |
| *** avass has joined #zuul | 11:00 | |
| *** ssbarnea has joined #zuul | 11:06 | |
| *** themroc has quit IRC | 11:26 | |
| *** jamesmcarthur has joined #zuul | 11:29 | |
| *** jamesmcarthur has quit IRC | 11:33 | |
| *** jcapitao is now known as jcapitao|lunch | 11:36 | |
| avass | how does nodepool handle if a static node and a cloud provider node has the same label. Does it check if the static node is available before launching another node or could that be a bit random? | 11:43 |
| *** pcaruana has joined #zuul | 11:46 | |
| *** hashar has quit IRC | 11:56 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Aws cloud-image is referred to from pool labels section https://review.opendev.org/697998 | 11:56 |
| *** jamesmcarthur has joined #zuul | 12:06 | |
| *** jamesmcarthur has quit IRC | 12:11 | |
| sugaar | Hi, I was wondering why for the demo is it used a custom Dockerfile to launch nodepool (https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples/node-Dockerfile) instead of using the official one from Dockerhub https://hub.docker.com/r/zuul/nodepool-launcher | 12:30 |
| sugaar | I am having a problem with the nodepool driver, even if I changed the demo to use the kubernetes driver and even the modified etc/nodepool/nodepool.yaml file is being loaded, still the static driver is running. Does anyone has a clue about why is happening that? I tried deleting the nodepool image from my host thinking that maybe instead of being | 12:33 |
| sugaar | created anew one the old one was getting launch, but it didn't work | 12:33 |
| *** hashar has joined #zuul | 12:37 | |
| avass | sugaar: the node-Dockerfile is an example workernode. nodepool is called 'launcher' in the docker-compose file: https://opendev.org/zuul/zuul/src/branch/master/doc/source/admin/examples/docker-compose.yaml#L99 | 12:40 |
| *** bhavikdbavishi has quit IRC | 12:53 | |
| *** themroc has joined #zuul | 12:56 | |
| Shrews | avass: it would be random. there is no priority among providers | 12:56 |
| *** rlandy has joined #zuul | 12:59 | |
| *** jamesmcarthur has joined #zuul | 13:00 | |
| avass | shrews: too bad. was hoping we could use aws for overcapacity somehow since we still have those static nodes. | 13:03 |
| Shrews | you are not the first to desire such a feature | 13:07 |
| *** jcapitao|lunch is now known as jcapitao | 13:08 | |
| tobiash | Shrews: do you have any objection about adding nodepool image metadata as env variables during image build? | 13:12 |
| tobiash | this way we would have a way to add this metadata into the image via a special element | 13:13 |
| tobiash | the use case is that we would like to print some metadata (image name and number) within the jobs | 13:13 |
| tobiash | I thought about adding something like DIB_NODEPOOL_IMAGE_NAME and DIB_NODEPOOL_IMAGE_BUILD | 13:14 |
| *** jamesmcarthur has quit IRC | 13:15 | |
| *** jamesmcarthur has joined #zuul | 13:15 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure: remove connectors burden and simplify code https://review.opendev.org/696134 | 13:19 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure: remove connectors burden and simplify code https://review.opendev.org/696134 | 13:20 |
| *** zbr has quit IRC | 13:27 | |
| *** zbr has joined #zuul | 13:32 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: enqueue: make trigger deprecated https://review.opendev.org/695446 | 13:33 |
| *** jamesmcarthur has quit IRC | 13:34 | |
| Shrews | tobiash: not particularly, although image name is already in the metadata. is DIB_NODEPOOL_IMAGE_BUILD the build ID? because only nodepool has that info at build time | 13:34 |
| Shrews | oh, i guess the builder would be the one providing that info via env var | 13:35 |
| *** Goneri has joined #zuul | 13:35 | |
| tobiash | yes, that was my idea | 13:35 |
| Shrews | tobiash: i won't object, but it is a community decision, not mine alone. :) | 13:36 |
| tobiash | Shrews: cool :) | 13:37 |
| Shrews | tobiash: i'm not clear on how that info is passed to the launcher though, since it's the one setting metadata | 13:37 |
| tobiash | Shrews: the idea is that nodepool supplies that var to dib and then a dib element can put this information e.g. into /etc/image-info | 13:38 |
| tobiash | then e.g. the base playbook can read out this file if it exists | 13:38 |
| tobiash | the launcher and zuul then don't have to know this data | 13:39 |
| tobiash | another idea could be to put this into the nodepool node metadata in zk (I just checked, only the label is there currently, not the image or the build id) | 13:40 |
| Shrews | i don't like the idea of storing any number of environment variables used for debugging purposes in zk | 13:43 |
| Shrews | i'd rather restrict zk data to just what is necessary for nodepool/zuul to operate | 13:43 |
| *** jamesmcarthur has joined #zuul | 13:45 | |
| sugaar | thanks avass I didn't noticed the image underneath | 13:52 |
| avass | sugaar: no problem! | 13:53 |
| sugaar | Is the nodepool-launcher a generic image? or is it predefined to use a driver by default? Because I keep configuring etc/nodepool.yaml to use kubernetes driver but it always uses static driver | 13:53 |
| avass | sugaar: what does your nodepool.yaml look like? | 13:54 |
| *** Petar_T has joined #zuul | 13:58 | |
| yoctozepto | pabelanger, tobiash: just wrote to the ML, but thought I might as well try you here, it seems https://review.opendev.org/678273 is deemed highly controversial in its simplicity :-) | 14:05 |
| tobiash | yoctozepto: you already have my review ;) | 14:12 |
| Petar_T | Hi all! | 14:13 |
| * Petar_T waves | 14:13 | |
| Petar_T | I was wondering if one of you could help me with a (hopefully simple) question I have.I am fairly new to Zuul and I've been tasked to set up a few projects with pipelines in my company. The infrastructure and a few projects were already set up by someone else but they have now left.I have something I'd like done across multiple projects so I am | 14:13 |
| Petar_T | writing the config for it in a sort of "common zuul jobs" repo. There are a few examples of common jobs in there already, but I have spotted that they all have multiple items listed under "job.required-projects" config item - one for each project we'd like the job run to run on, as well as the "common zuul jobs" repo so we have access to the actual | 14:13 |
| Petar_T | playbooks.Is this the idiomatic way of doing this? What if you had a large amount of projects you'd like to run a small job on? Is there a way to get Zuul to not clone repos the job isn't running on? E.g. if I had projects "Foo" and "Bar" which ran something from "common-jobs" then when Foo is running its pipeline, it wouldn't need to clone | 14:13 |
| Petar_T | "Bar".One way that pops into mind is having a variable in the "job.required-projects" list that is set at project level, and is merged with hardcoded required projects but I was wondering if there's a more elegant way of doing it.Thanks | 14:13 |
| Petar_T | I was wondering if one of you could help me with a (hopefully simple) question I have.I am fairly new to Zuul and I've been tasked to set up a few projects with pipelines in my company. The infrastructure and a few projects were already set up by someone else but they have now left.I have something I'd like done across multiple projects so I am | 14:13 |
| Petar_T | writing the config for it in a sort of "common zuul jobs" repo. There are a few examples of common jobs in there already, but I have spotted that they all have multiple items listed under "job.required-projects" config item - one for each project we'd like the job run to run on, as well as the "common zuul jobs" repo so we have access to the actual | 14:13 |
| yoctozepto | tobiash: yes, for which I thank you :-) yet it did not get much heat since then | 14:13 |
| Petar_T | playbooks.Is this the idiomatic way of doing this? What if you had a large amount of projects you'd like to run a small job on? Is there a way to get Zuul to not clone repos the job isn't running on? E.g. if I had projects "Foo" and "Bar" which ran something from "common-jobs" then when Foo is running its pipeline, it wouldn't need to clone | 14:13 |
| Petar_T | "Bar".One way that pops into mind is having a variable in the "job.required-projects" list that is set at project level, and is merged with hardcoded required projects but I was wondering if there's a more elegant way of doing it.Thanks | 14:13 |
| Petar_T | (oops sorry for the duplicate) | 14:14 |
| tobiash | yoctozepto: most people here were traveling/recovering from travel the last few weeks afaik | 14:14 |
| *** jamesmcarthur has quit IRC | 14:15 | |
| sugaar | avass https://paste.gnome.org/pdhrgfoy7 | 14:15 |
| tobiash | Petar_T: you don't need required-projects for running a job in a different repo, you only need to list repos that the job needs at least in the workspace while running the job | 14:16 |
| tobiash | Petar_T: the repos containing playbooks and roles will be automatically there | 14:16 |
| yoctozepto | tobiash: ack, hence /me providing the heat instead via ML and IRC | 14:16 |
| *** jamesmcarthur has joined #zuul | 14:20 | |
| tobiash | Shrews: I'd have another topic about nodepool-builders, since you fixed image deletion (thanks for that!) nodepool doesn't leak images anymore but our cloud occationally has the problem that it leaves zombie volumes of instances behind that block the deletion of images. | 14:21 |
| tobiash | with many builder (we run 10 atm) and a growing number of non-deletable images the builders hammer glance with delete requests that all get rejected | 14:22 |
| tobiash | I think we need coordination using zk and some backoff mechanism in the image cleanup | 14:23 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Add back-off mechanism for image deletions https://review.opendev.org/698023 | 14:27 |
| tobiash | Shrews: what do you think about something like this? ^ | 14:27 |
| Petar_T | tobiash: Let me give you a bit more detail. The job in question is a linter. Its job config and playbooks live in the "common-jobs" repo. I have Foo and Bar, two C/C++ projects which need linting independently. If I have each of them under "required-projects" then when Foo's pipeline is running, cloning Bar would be a waste of effort. Obviously | 14:27 |
| Petar_T | this wasted effort is multiplied when there's even more projects that need linting. | 14:27 |
| Shrews | tobiash: yeah that’s the cinder issue we struggle with as well. Not much nodepool can do with those except eventually give up since it requires manual intervention | 14:27 |
| tobiash | Shrews: yes, manual intervention will required anyways, but I think we should at least using a back-off to not hammer the cloud | 14:28 |
| tobiash | tests are missing yet though | 14:28 |
| Shrews | tobiash: will look in a bit | 14:28 |
| tobiash | Petar_T: you don't need any of foo or bar in required projects | 14:30 |
| tobiash | Petar_T: the project triggering the job is always part of the job | 14:31 |
| *** mhu has quit IRC | 14:32 | |
| *** jamesmcarthur has quit IRC | 14:33 | |
| Petar_T | tobiash: Right I see! Existing config for other jobs have mislead me then. So would I need the "common-jobs" repo? Or since the job is defined in that repo then it knows to clone that too? | 14:36 |
| tobiash | Petar_T: correct since the job is defined in common-jobs you don't need to list it as well | 14:36 |
| tobiash | Petar_T: you would need required-projects only if the job needs to have access to a repo baz regardless against which repo it would run | 14:37 |
| Petar_T | tobiash: Great, thanks. Much clearer now. I might have to correct the existing jobs at some point in that case :) | 14:38 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Add back-off mechanism for image deletions https://review.opendev.org/698023 | 14:44 |
| *** jamesmcarthur has joined #zuul | 14:46 | |
| *** jamesmcarthur has joined #zuul | 14:46 | |
| *** mhu has joined #zuul | 14:57 | |
| avass | sugaar: went on a fika break :). That doesn't have a static driver pool so I'm gonna guess that there's a configuration problem which maybe causes nodepool to keep an older config? | 15:01 |
| avass | line 13: should be labels and not nodes right? | 15:03 |
| sugaar | indeed | 15:03 |
| avass | are the labels defined in the label section? | 15:05 |
| avass | sugaar: https://paste.gnome.org/pbehhrlxv | 15:06 |
| Shrews | tobiash: i think something like that would be ok (though there are issues with that change as-is). i think we might also want to consider a max delete attempts or something and just give up entirely on it, otherwise we just accumulate un-deletable uploads forever. | 15:07 |
| *** sgw has joined #zuul | 15:09 | |
| *** jamesmcarthur_ has joined #zuul | 15:09 | |
| sugaar | even thought I changed from node to label I achieve the same. Those labels are defined here: https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[kubernetes].pools.labels.type | 15:11 |
| tobiash | Shrews: makes sense | 15:12 |
| sugaar | avass I just understood what you meant by defined | 15:12 |
| avass | sugaar: :) | 15:12 |
| *** jamesmcarthur has quit IRC | 15:13 | |
| Shrews | tobiash: maybe a max timeframe instead, like 1 month or something, to give an admin time to manually intervene, then nodepool would be able to cleanup after that. i'm not sure of the best way to deal with that, tbh. It would be nice if nodepool could email an admin saying "hey, i give up on deleting this thing. it's up to you now." :) | 15:16 |
| *** chkumar|ruck is now known as raukadah | 15:19 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Keys must be defined for host-key-checking: false https://review.opendev.org/698029 | 15:19 |
| tobiash | Shrews: yes, a max timeframe sounds more reasonable than attempts when we use a back-off | 15:21 |
| tobiash | we also could regularly generate a summary of non-deletable images into the logs | 15:21 |
| Shrews | tobiash: yeah, but that idea depends on someone regularly checking the logs before you rotate them away | 15:22 |
| Shrews | i never look at our logs unless there's an issue, tbh, or a new feature we're rolling out | 15:23 |
| tobiash | ok, then be it email | 15:23 |
| Shrews | email is wishful thinking and hand-wavy, ala mordred :) | 15:24 |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Keys must be defined for host-key-checking: false https://review.opendev.org/698029 | 15:25 |
| tobiash | well, I think email is the next best thing in that regard unless someone wants to write a jira connector :) | 15:25 |
| * Shrews does NOT volunteer lol | 15:26 | |
| avass | ^any tip on where to start if I want to add a test case for those kinds of fixes? :) | 15:26 |
| Shrews | avass: likely in nodepool/tests/unit/test_driver_aws.py | 15:28 |
| *** sgw has quit IRC | 15:28 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Keys must be defined for host-key-checking: false https://review.opendev.org/698029 | 15:42 |
| avass | shrews: something like that I guess | 15:42 |
| Shrews | avass: i'm not terribly familiar with the aws portion. we should ask SpamapS or tristanC to have a look when they get a chance | 15:44 |
| avass | Shrews: Sure. Gonna make sure it fails if I remove the fix later aswell :) | 15:45 |
| Shrews | ++ | 15:47 |
| *** johanssone has quit IRC | 15:48 | |
| *** johanssone has joined #zuul | 15:49 | |
| *** saneax has quit IRC | 15:53 | |
| *** jcapitao is now known as jcapitao|afk | 16:04 | |
| *** themroc has quit IRC | 16:10 | |
| pabelanger | morning, I incorrectly posted this in #openstack-infra | 16:22 |
| pabelanger | https://review.opendev.org/#/q/status:open+project:zuul/zuul+branch:master+topic:multi-ansible-wip | 16:22 |
| pabelanger | could we review and approve support for ansible 2.9 this week in zuul? I'd like to start to use it in zuul.ansible.com | 16:23 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure: remove connectors burden and simplify code https://review.opendev.org/696134 | 16:25 |
| mordred | pabelanger: don't kill me - but I kind of think that stack should be inverted - add 2.9 as a patch (very uncontroversial), then switch default to 2.8 - not super controversial, but still is a user-facing impact we should think about - then remove 2.5 - since even though it's EOL upstream, we've had community concerns about removing ansible support from zuul and that would be a conversation too | 16:25 |
| mordred | pabelanger: so - you know - if they were in the opposite order, I think we could land "add 2.9" pretty quickly and easily without blocking it on the other two conversations | 16:25 |
| SpamapS | avass: Shrews is probably right, that test file has some faked aws stuff that should make it possible to recreate the scenario you see as breaking. | 16:26 |
| pabelanger | okay, I can invert. But I thought we already had the discuss to drop ansible 2.5 support, that's why I rebased the stack that way to start | 16:26 |
| mordred | pabelanger: oh did we? | 16:27 |
| mordred | pabelanger: I'm sorry - if we have, then I'm less worried about it | 16:27 |
| pabelanger | I _think_ so, but maybe we never officially announced anything | 16:27 |
| mordred | clarkb, corvus: ^^ | 16:27 |
| pabelanger | but yah, no issues updating changes if that is what is needed | 16:28 |
| pabelanger | however, from the zuuls I know of, I don't think any of them use 2.5 jobs | 16:29 |
| tristanC | but what if a zuul relies on 2.5? | 16:31 |
| pabelanger | tristanC: I am not sure I understand | 16:32 |
| avass | SpamapS: Yep i noticed that after going down the rabbit hole | 16:33 |
| mordred | pabelanger: if someone is running a zuul and they have jobs written that rely on 2.5 | 16:33 |
| pabelanger | right, of the zuuls I know of, I don't think any are doing that. | 16:34 |
| pabelanger | however, it is possible that there are zuuls using 2.5 | 16:34 |
| pabelanger | https://zuul-ci.org/docs/zuul/developer/specs/multiple-ansible-versions.html has listed 2.5 as deprecated for a long time | 16:35 |
| pabelanger | but doesn't list a process of actually removing ansible version | 16:35 |
| clarkb | it would probably be friendly to users to have a 2.5 removal release | 16:36 |
| clarkb | separate of other changes | 16:36 |
| clarkb | inverting the stack may make that easier? but I dont think it is necessary | 16:37 |
| corvus | if someone wants to propose a policy for how long after ansible EOLs a release we keep it in zuul, i think adding that to https://zuul-ci.org/docs/zuul/developer/specs/multiple-ansible-versions.html#job-configuration would be great | 16:37 |
| pabelanger | I am okay with us remove 2.5 before landing 2.9 | 16:37 |
| corvus | basically, come up with when we think we should mark something as deprecated, when we should remove it, write a docs patch, merge it, and we can stick to it | 16:38 |
| pabelanger | (ansible 2.6 is eol now too) | 16:38 |
| pabelanger | +1 | 16:38 |
| SpamapS | *crikey* | 16:39 |
| SpamapS | To be honest.. since 2.5-ish, I haven't seen a lot of breakage when I upgraded Ansible. | 16:40 |
| *** jamesmcarthur has joined #zuul | 16:40 | |
| SpamapS | It was bad, 2.0->2.4 .. lots of aggressive changes.. but feels like it's pretty steady-eddie now. | 16:40 |
| clarkb | there has been a bunch if deprecated and not replaced functionality in ansible | 16:40 |
| corvus | SpamapS: yeah, and they've even backed down from one of their more aggressive changes recently | 16:40 |
| clarkb | mostly that results in lots of logging so farbut not breakage | 16:41 |
| pabelanger | SpamapS: 2.10 will have some pain, IMO. With the move to collections | 16:41 |
| corvus | the removal of with_first_found has been stayed pending improvements in loop | 16:41 |
| pabelanger | but agree, last few years have been nice | 16:41 |
| *** jamesmcarthur_ has quit IRC | 16:41 | |
| *** jamesmcarthur_ has joined #zuul | 16:41 | |
| SpamapS | Heh, perhaps part of my non-troubles is that I avoid loop as much as possible... if it's not just looping on a list.. I write a python module. ;) | 16:42 |
| pabelanger | I'll work on the doc change this week for ansible removal | 16:43 |
| corvus | pabelanger: thanks | 16:43 |
| *** jcapitao|afk is now known as jcapitao | 16:43 | |
| *** hashar has quit IRC | 16:44 | |
| corvus | clarkb, pabelanger: how about a 3.12.0 release to clear out the current stuff, then merge the 2.5 removal and release that as 3.13, then merge the default change and addition of 2.9 and release as 3.14? | 16:44 |
| corvus | or do we want to explore mordred's idea of inverting the stack? | 16:44 |
| pabelanger | I am okay with that | 16:44 |
| *** jamesmcarthur has quit IRC | 16:45 | |
| clarkb | corvus: that plan wfm | 16:45 |
| pabelanger | any 2.5 user could not move pass 3.12.0 until they made the switch to at least 2.6. | 16:45 |
| pabelanger | also, unrelated, I've been working on a buildset-galaxy server, in zuul.a.c. I've confirmed the idea, from zuuls POC, works as expected :) | 16:46 |
| pabelanger | https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/ansible_52/52/2ed1e5fafc215852f746d854ca03c22c839483db/check/galaxy-quick-start-child/7f08f85/job-output.html#l623 | 16:46 |
| pabelanger | ansible-galaxy CLI pulling from 2 different galaxy servers | 16:46 |
| corvus | pabelanger: awesome! | 16:47 |
| pabelanger | that is parent job with galaxy server and child pulling from it :) | 16:47 |
| pabelanger | I still have work to create general role, but plan to push up into zuul-jobs at some point | 16:47 |
| corvus | pabelanger: thank you :) | 16:48 |
| corvus | it looks like we'll restart opendev zuul soon, so i'll make the 3.12 release after that | 16:57 |
| mhu | corvus, you think these changes could make the cut for 3.12? https://review.opendev.org/#/q/status:open+project:zuul/zuul+branch:master+topic:zuul_admin_web+label:%22Code-Review%252B2%22 | 17:00 |
| corvus | mhu: it looks like we're changing max_validity_time to max_token_age with no backwards-compat handling? | 17:10 |
| *** dtroyer has joined #zuul | 17:19 | |
| corvus | mhu: we might want to support both spellings for a while | 17:23 |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Keys must be defined for host-key-checking: false https://review.opendev.org/698029 | 17:23 |
| mhu | corvus, right, I'll have a look | 17:23 |
| avass | Gonna continue working on that from home. Looks like it's gonna take a bit longer than i planned :) | 17:24 |
| mordred | pabelanger: cool! is that intended to work sort of like the speculative container stuff? | 17:24 |
| corvus | mordred, tristanC: speaking of speculative containers: https://review.opendev.org/696939 is ready for review | 17:27 |
| pabelanger | mordred: Yup, I hope so. I believe there will be some changes needed on ansible-galaxy side, today there is no options to install pre-release versions (semver) | 17:28 |
| pabelanger | but, since users are expected to install collections from galaxy, and not git, it only make sense to have an ephemeral galaxy in testing, like we did for docker images | 17:29 |
| pabelanger | also mean, we get to provide feedback to galaxy team now too :) | 17:29 |
| pabelanger | and, is a good story for cross project testing | 17:30 |
| *** jamesmcarthur_ has quit IRC | 17:30 | |
| *** pcaruana has quit IRC | 17:31 | |
| *** jamesmcarthur has joined #zuul | 17:31 | |
| mordred | corvus: I'm curious - the openshift role is doing docker things - any reason to not just use install-docker role? | 17:31 |
| mordred | corvus: I'm guessing because that's what it was already doing and that refactor is out of scope? | 17:32 |
| tobiash | pabelanger: there is the possibility that some of our jobs might still be on 2.5 ( not sure about that right now). May I get some additional time to check that before removing 2.5? | 17:32 |
| corvus | mordred: yeah that's the gist. | 17:34 |
| pabelanger | tobiash: what are your thoughts on n+6 months after a version of ansible is EOL'd for removal from zuul? | 17:34 |
| mordred | corvus: cool. patch looks awesome | 17:34 |
| pabelanger | (in 2.5 case, we need to figure out some timeline as n+6 months as passed, in this case) | 17:34 |
| tobiash | pabelanger: that sounds reasonable | 17:34 |
| corvus | pabelanger, tobiash: how about we invert the stack then, that way pabelanger can keep moving on 2.9, and tobiash can have a bit more time to check on 2.5? | 17:35 |
| tobiash | Sounds good, I don't see a reason why adding 2.9 should depend on removing 2.5 | 17:35 |
| corvus | clarkb: ^ | 17:36 |
| *** jamesmcarthur has quit IRC | 17:36 | |
| tobiash | Actually even nike 2.8 the default is independent | 17:36 |
| *** hashar has joined #zuul | 17:36 | |
| tobiash | What do you think about defaulting straight away to 2.9? | 17:36 |
| tobiash | Is that too risky? | 17:36 |
| pabelanger | too risky, IMO | 17:36 |
| tobiash | k | 17:37 |
| pabelanger | I think 2.8 is right | 17:37 |
| pabelanger | that is pretty stable | 17:37 |
| corvus | so new plan would be: release current as 3.12, merge 2.9 plus default 2.8, release that as 3.13, merge 2.5 removal, release that as 3.14? | 17:37 |
| clarkb | corvus: plan seems reasonable to me | 17:38 |
| tobiash | ++ | 17:39 |
| corvus | cool, after we release 3.12, i'll send an email to zuul-announce with the plan to give folks a heads up | 17:40 |
| *** avass has quit IRC | 17:40 | |
| pabelanger | okay, I'll need some time to rebase bit, but plan is fine | 17:40 |
| corvus | i'm assuming we'll try to do all of that this week, unless we find we need to keep 2.5 longer | 17:41 |
| tobiash | I guess I can tell tomorrow if we need it longer | 17:41 |
| *** igordc has joined #zuul | 17:49 | |
| *** sgw has joined #zuul | 17:53 | |
| tristanC | corvus: that plan works for me too, though it doesn't seems like a burden to keep old ansible versions, and perhaps we should consider doing that in the futur to avoid forcing the user to change their job. | 17:53 |
| fungi | there will come a time when an older version of ansible doesn't work with whatever newer version of python you've got on the executors | 17:55 |
| fungi | so folks will be forced to drop old versions from their environments at some point | 17:55 |
| fungi | but also, testing that zuul can manage every version of ansible dating back to when it gained multi-ansible support will eventually become untenable (if it hasn't already) | 17:56 |
| corvus | tristanC: i'm pretty uncomfortable with zuul supporting eol versions of ansible. i can see the point that it's not strictly necessary (at least as long as we can physically continue to install them). but i also think it's a confusing message to send to users. what do we do if there is some security vulnerability in an old ansible? | 17:56 |
| tristanC | fungi: corvus: i didn't meant to keep them forever, but then it seems like we should be dropping 2.6 too. | 17:58 |
| fungi | we probably should | 17:58 |
| fungi | we've been talking about dropping 2.5 for quite a while | 17:58 |
| fungi | and it still hasn't happened for various reasons | 17:59 |
| mordred | I thnk largely because we hadn't gotten all the way to writing up an actual strategy for dropping yet | 17:59 |
| pabelanger | yup, we just talked about that doc now | 17:59 |
| mordred | yup | 18:00 |
| fungi | the previous idea was that we were marking 2.5 as depracated when we introduced 2.7, and then removing it when we added 2.8 and marking 2.6 as deprecated, to be removed when we added 2.9 | 18:00 |
| pabelanger | as example, 6 months after ansible EOLs a version, we also remove it from zuul | 18:00 |
| pabelanger | if not 6months, what are people okay with | 18:00 |
| fungi | i think the old plan is fine... basically leaves any zuul release with three versions of ansible: a deprecated version, an existing stable version, and a new/experimental version | 18:01 |
| pabelanger | tristanC: ansible 2.5 has been deprecated in zuul for a whlie now, maybe 9-12 months. If users haven't migrated now, how do we encourage zuul operators to do that? | 18:01 |
| corvus | that helps keep the testing matrix reasonable | 18:02 |
| fungi | when you upgrade to a version of zuul which has deprecated the version of ansible you're using, that should be the signal to spend time updating your default version and/or jobs to use one of the two newer versions it supports | 18:02 |
| *** hashar has quit IRC | 18:02 | |
| clarkb | pabelanger: I think that points to my concern with this. As a CI tool do we want to force users to go through frequent churn of updating playbooks in addition to zuul itself? | 18:02 |
| clarkb | typically what users want is jobs that continue to work once they are stable | 18:03 |
| corvus | clarkb: well, we've chosen ansible. this comes with the territory i think. | 18:03 |
| fungi | we've seen ansible stabilize over time, hopefully that trend continues | 18:04 |
| clarkb | corvus: yes, except that there is little harm on the zuul side to continue supporting 2.5? | 18:04 |
| tristanC | i think the plan is fine, but this might be inconvenient for users if a new supported version doesn't work with their jobs | 18:04 |
| clarkb | fungi: pabelanger just pointed out 2.10 is likely to undo that trned | 18:04 |
| corvus | clarkb: see the harms fungi and i laid out aboveL | 18:04 |
| fungi | clarkb: yeah, at least those major shifts in behavior may be farther apart now than previously | 18:04 |
| *** jcapitao has quit IRC | 18:04 | |
| clarkb | yeah I think the 6 month proposal or the trailing removal based on upstream releases is a reasonable compromise | 18:05 |
| tobiash | ok, we at least default to 2.7 in our base job, now I need to check if any job overwrites this with 2.5 | 18:05 |
| clarkb | corvus: ^ | 18:05 |
| pabelanger | yah, I have some concerns with 2.10 personally. That will need a bit of work on our side to support collections | 18:05 |
| tristanC | it seems like 2.5 -> 2.8 mostly adds warning, but what about the point where roles needs adjustment for collections | 18:05 |
| tobiash | I guess I have to crawl the api | 18:05 |
| clarkb | corvus: but we should also be aware that some users are likely to be unhappy with needing to update every 6 months especially if ansible updates become more difficult again | 18:06 |
| tristanC | clarkb: that's my concern yes | 18:07 |
| pabelanger | tristanC: roles and collections are separate. So, if a job uses a role, it _should_ still be fine in 2.10 | 18:08 |
| corvus | clarkb: yes. i think our options are: 1) continue to support all ansible versions indefinitely. i do not think we have the resources for this, so we have to take it off the table. 2) amplify our users concerns so that ansible is aware of this and they minimize disruptive changes. happily, zuul users and ansible users are in the same boat -- we're not asking for anything weird. 3) decide that ansible is | 18:08 |
| corvus | too unreliable for zuul and replace it. i don't think we've experienced remotely enough pain to consider this an option at this point (and i hope we do not). | 18:08 |
| fungi | i also think that deciding not to upgrade a component of zuul regularly should be an indication that upgrading zuul regularly isn't right for you either | 18:10 |
| corvus | so if we're all on the same page that #1 and #3 are not things we should consider right now, then yes, i think we have some room to talk about how to minimize disruption for zuul users (in terms of when we add/remove supported ansible versions). but also i think that just getting that process decided on and communicated will be a big help. if we say "hey, we're adding a new version of ansible, it may have | 18:10 |
| corvus | some things you need to change, you should work on that over the next year before we drop the oldest" then that's a big help. | 18:10 |
| fungi | zuul 3.9.0 will continue to support ansible 2.5.0 forever, after all | 18:10 |
| corvus | fungi: that is a useful perspective | 18:11 |
| *** hashar has joined #zuul | 18:11 | |
| pabelanger | so, if we way 2.5 is supported for ever, I would be okay with that, but in zuul.a.c, i don't want job to use it. So, if we could expose a way for zuul-manage to only install version you want, I think that makes me happy. | 18:12 |
| fungi | if zuul 3.14.0 drops support for ansible 2.5.0 and you need that, don't upgrade to zuul 3.14.0 | 18:12 |
| fungi | (you've already decided to continue relying on an ansible version which gets no security support, so it's hard to argue you're losing security support by not continuing to upgrade zuul) | 18:12 |
| pabelanger | s/we way/we say/ | 18:12 |
| corvus | exactly | 18:13 |
| pabelanger | fungi: ++ | 18:13 |
| clarkb | ya I think the only bit that makes this special to me is that it can force you to update your jobs | 18:14 |
| corvus | i think one thing that we can derive from clarkb's point is that having the widest possible window can help reduce churn. so, if it comes down to it, not limiting ourselves arbitrarily to 3 versions will help people by potentially allowing larger jumps (or a longer time to make changes). | 18:15 |
| clarkb | updating jobs is an end user facing thing and not just an operational thing | 18:15 |
| fungi | and i agree that's a downside to choosing what is basically a turing-complete automation language as a job definition language, but it's also a strong feature | 18:15 |
| corvus | (ie, we're about to more or less accidentally facilitate people performing 2.5 -> 2.9 upgrades) | 18:15 |
| clarkb | corvus: ++ if you can go from say 2.5 to 2.10 (I dunno how feasible that actually is) then you do it once intsead of 4 times | 18:15 |
| tristanC | fungi: well if you know you need 2.5.0, then you likely know what to do to fix your job and avoid being broken by a new release of zuul. My concern is for users who are not familiar with Ansible... In anycase, I agree with corvus choice, #1 and #3 are not good either. | 18:15 |
| fungi | users can shield themselves from some of the impact of that by avoiding using as many of ansible's more complex features (as SpamapS noted with his avoidance of complex loop constructs) | 18:16 |
| fungi | with flexibility in the job definition language comes complexity, and with complexity comes churn | 18:17 |
| clarkb | fungi: yup | 18:17 |
| fungi | in opendev we took a similar route with jenkins, eschewing most of the plugin ecosystem and encoding more complex behaviors into jobs as shell blobs which served similar purposes | 18:20 |
| pabelanger | could somebody help with the following traceback we are seeing in zuul-scheduler, we just added a new github.com repo | 18:22 |
| pabelanger | http://paste.openstack.org/show/787340/ | 18:22 |
| pabelanger | something odd about the repo, is https://github.com/ansible/workshops/tree/devel has both devel and master branch | 18:22 |
| corvus | that's a big repo | 18:23 |
| *** pcaruana has joined #zuul | 18:23 | |
| fungi | yeah, that git error usually comes about when there are ambiguous ref names (like a branch and a tag with the same name and the git command wasn't given sufficient context to differentiate them) | 18:24 |
| corvus | pabelanger: look up a few lines -- zuul deleted the devel branch and apparently did not add it back? | 18:24 |
| pabelanger | yes, I see that | 18:25 |
| corvus | pabelanger: can you track that event back to the zuul-merger which processed it originally? | 18:25 |
| corvus | pabelanger: that's responsible for deciding what branches should exist on the executor (that tb is from the executor, not the scheduler -- i assume that was just a typo) | 18:26 |
| fungi | i'm not immediately seeing any obvious conflicts between branch and directory/file and tag names | 18:26 |
| pabelanger | (looking) | 18:27 |
| corvus | pabelanger: do you have the exclude-unprotected-branches setting enabled? and if so, is devel unprotected? | 18:28 |
| pabelanger | ah, you know what | 18:29 |
| pabelanger | I think that is it | 18:29 |
| pabelanger | I can see only master is protected, let me add back and recheck | 18:29 |
| corvus | pabelanger: you may need either some kind of event to the devel branch or a full reconfiguration for zuul to notice the change | 18:30 |
| pabelanger | ack | 18:31 |
| pabelanger | thanks! | 18:31 |
| pabelanger | let me test this and confirm | 18:31 |
| pabelanger | corvus: thanks! that was it, only master had branch protection | 18:34 |
| * fungi files that bit of github integration away for future reference | 18:35 | |
| pabelanger | yah, for some reason, before I thought zuul wouldn't start the job | 18:35 |
| pabelanger | this time it did | 18:35 |
| pabelanger | but yes, better docs on myself helps here | 18:36 |
| pabelanger | sorry for the noise | 18:36 |
| *** igordc has quit IRC | 18:37 | |
| corvus | well, also, if zuul said "I've been asked to check out a branch which was excluded due to exclude_unprotected_branches" that wouldn't be the worst error message in the world | 18:40 |
| tobiash | I just checked, and we don't have ansible 2.5 in use by any job, only 2.7 and 2.8 | 18:42 |
| corvus | tobiash: that took much less time than you estimated :) | 18:43 |
| tobiash | corvus: I wrote a script that crawled the zuul api :) | 18:43 |
| tobiash | in less than 50 lines of python code | 18:44 |
| tristanC | tobiash: you might be interested by https://review.opendev.org/#/c/633667/4 that can display such crawl result in the web interface | 18:48 |
| tobiash | tristanC: interesting | 18:49 |
| tristanC | the web interface doesn't group by ansible-version, it currently only does per secret, semaphore, label/nodeset, project, ... but it should be relatively easy to add a view per ansible-version too | 18:50 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Add quick and dirty api crawler for ansible versions https://review.opendev.org/698062 | 18:51 |
| tobiash | for reference, that's the quick and dirty crawler ^ | 18:52 |
| tristanC | tobiash: it seems like your crawler can miss ansible-job attribute set in project pipeline configs | 18:53 |
| tobiash | tristanC: oops that's correct | 18:54 |
| tristanC | tobiash: here is another crawler we wrote to auto-patch nodepool labels (it goes over every project pipeline config): https://softwarefactory-project.io/cgit/software-factory/sf-ops/tree/scripts/cleanup-nodepool-images.py | 18:54 |
| tristanC | oops, that actually was using the /config endpoint | 18:55 |
| tristanC | anyway, the function is the same, to check every job you have to: for each project, for each project config, for each pipeline, for each job, for each job config | 18:57 |
| tobiash | ok, also no project pipeline using ansible 2.5 | 19:13 |
| *** igordc has joined #zuul | 19:14 | |
| fungi | being able to filter jobs by ansible version in the dashboard could be neat, if that logic is embeddable into the config api | 19:14 |
| fungi | tobiash: you might need to recheck 698062, we restarted opendev's zuul scheduler and that was the only change with jobs queued in the zuul tenant | 19:16 |
| tobiash | fungi: thanks, that wasn't expected to succeed anyway | 19:17 |
| *** gmann is now known as gmann_afk | 19:17 | |
| corvus | Shrews: should we add a release note to nodepool and make a release with the sdk version bump? | 19:22 |
| corvus | (right now, there's nothing new since 3.9.0: https://zuul-ci.org/docs/nodepool/releasenotes.html ) | 19:22 |
| Shrews | i guess it would be odd to release with something in release notes. i know tobiash was asking for a new release with the sdk microversion fix during the ptg | 19:24 |
| Shrews | sorry it's taken so long for that, tobiash. fell off the radar | 19:25 |
| Shrews | s/with something/without something | 19:25 |
| tobiash | Shrews: no worries, it fell off my radar as well | 19:25 |
| Shrews | doh | 19:25 |
| corvus | yeah, if we want to do a release with just an sdk version bump, let's add a note that just says something like "updated openstack sdk dependency due to ..." | 19:25 |
| Shrews | corvus: ++ | 19:26 |
| clarkb | "updated openstack sdk dependency to pick up OVH profile updates which may be come necessary in the new year as OVH modifies their catalog" | 19:26 |
| tobiash | corvus, Shrews: shall we also add a release note for the post upload hook? | 19:27 |
| fungi | well, also it more generally added support for a bunch more ovh regions | 19:27 |
| tobiash | that's new functionality | 19:27 |
| Shrews | but fyi, we are not yet running that version of the launcher in production | 19:27 |
| clarkb | Shrews: I'm happy to do that restart once we have the zuul upgrade settled in | 19:28 |
| clarkb | probably can do that after lunch today? | 19:28 |
| Shrews | sounds good | 19:28 |
| corvus | tobiash: ++ | 19:28 |
| corvus | that seems noteworthy | 19:28 |
| tobiash | k, I'll prepare a note | 19:29 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Add missing release note about post-upload-hook https://review.opendev.org/698072 | 19:35 |
| Shrews | tobiash: can we add the sdk stuff to that note as well? | 19:36 |
| tobiash | Shrews: sure, do you have a text in mind? I'm not sure about the correct abstraction level translation from https://review.opendev.org/690876 | 19:37 |
| corvus | maybe clarkb's wording from 19:26 | 19:38 |
| tobiash | corvus: you mean the ovh profiles? | 19:39 |
| corvus | yes | 19:40 |
| corvus | though i guess there was also the microversion thing? i don't know anything about that... | 19:40 |
| tobiash | the fix was independent of ovh, nodepool was broken with any cloud with newer openstacksdk | 19:41 |
| Shrews | tobiash: maybe "updated openstack sdk dependency to handle newer nova microversions" ??? i believe you experienced the bug firsthand, so feel free to add more detailed context there if you find it helpful | 19:43 |
| tobiash | maybe 'Fixed incompatibility issue with openstacksdk <version to be added> and greater.' | 19:43 |
| *** hashar has joined #zuul | 19:44 | |
| tobiash | as I understood the nova microversion thing is an implementation detail behind openstacksdk which wasn't expected to have a user visible impact | 19:44 |
| tobiash | so for a release note it might be enough to state a compatibility issue or regression with newer openstacksdk has been fixed | 19:45 |
| clarkb | oh there was another reason to update sdk then | 19:45 |
| clarkb | fwiw ovh users need to update to latest sdk due to cloud side changes | 19:45 |
| Shrews | wait, so, we didn't actually update sdk requirement | 19:46 |
| Shrews | we just added a fix for a bug around referencing flavors when using a newer sdk | 19:47 |
| tobiash | clarkb: nodepool didn't pull in a new sdk version, it just got incompatible with the recent one which has been fixed by 690876 | 19:47 |
| Shrews | i'm not sure that needs a release note | 19:47 |
| Shrews | (sorry, been bouncing between things) | 19:47 |
| clarkb | ok. I think that we should consider forcing the newer version given OVH's planned changes | 19:47 |
| clarkb | but I don't know if anyone else uses ovh with nodepool | 19:47 |
| Shrews | well that would be a NEW change, so that can include a release note with the change | 19:47 |
| clarkb | ++ | 19:48 |
| Shrews | with that in mind, i'm going to +2 the change tobiash put up | 19:48 |
| corvus | if we're wanting to release nodepool with 690876 merged because nodepool 3.9.0 is uninstallable, that's probably relnote worthy | 19:48 |
| corvus | well, not uninstallable | 19:48 |
| corvus | but does-not-work-with-openstack-if-freshly-installed | 19:49 |
| tobiash | yepp, that was exactly the reason why I asked for a release back then (and later forgot about that) | 19:50 |
| clarkb | is I1f7b592265ac612ea6ca1b2f977e1507c6251da3 the fix we are talking about? | 19:50 |
| corvus | yes that is "the microversion thing" | 19:50 |
| tobiash | clarkb: yes | 19:50 |
| Shrews | corvus: that's a good point on a different perspective | 19:50 |
| clarkb | in that case I think we should restart opendev's launchers on a commit that incldues that (current HEAD). Concurrently merge the relnote update. THen if opendev is happy make a release against the relnote update | 19:51 |
| corvus | clarkb: ++ | 19:51 |
| corvus | Shrews: thanks, the approach i'm taking to see if there are relnote is: "why does {clarkb, tobiash} want a release? and might anyone else benefit from that knowledge?" | 19:52 |
| clarkb | and we should make sure we have sdk 0.39.0 installed when we do that restart | 19:52 |
| clarkb | to ensure that latest works | 19:52 |
| corvus | er, "if there are relnote gaps" | 19:52 |
| corvus | that wouldn't have been sufficiently ironic without omitting the word 'gap' | 19:53 |
| mordred | ha | 19:53 |
| tobiash | mordred: do you know what version of openstacksdk required that fix? | 19:57 |
| tobiash | mordred: I guess 0.37.0? https://docs.openstack.org/releasenotes/openstacksdk/unreleased.html#upgrade-notes | 19:59 |
| mordred | tobiash: yes, that's right | 20:00 |
| tobiash | so a release note might be: "Fixed compatibility issue with openstacksdk 0.37.0 and above."? | 20:01 |
| tobiash | following corvus' approach I'd consider this as an important info as 3.9.0 is not working when freshly installed | 20:02 |
| Shrews | i think that's pretty clear for the current state | 20:03 |
| ianw | corvus / modred: could i ask for your eyes on https://review.opendev.org/697936 & https://review.opendev.org/697614 ; it adds sibling copy to podman/generic builds (936) but of immediate interest it fixes the docker sibling copy (614) | 20:03 |
| ianw | i will need this to get the nodepool+containers functional test working | 20:04 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Add missing release notes https://review.opendev.org/698072 | 20:06 |
| Shrews | tobiash: lgtm. corvus? | 20:08 |
| clarkb | I'm restarting our launchers at nodepool==3.9.1.dev11 # git sha e391572 and they seem happy (and happy against vexxhost and ovh which should confirm that latest sdk is working) | 20:12 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Sort autoholds by request ID https://review.opendev.org/698077 | 20:16 |
| Shrews | the unsorted chaos was starting to annoy me | 20:17 |
| mordred | corvus: https://review.opendev.org/#/c/611191/ - I grok what is wrong here, but this might be an opportunity to report a different error | 20:58 |
| corvus | mordred: what is wrong? (that is an unhandled error, so likely unanticipated) | 21:00 |
| mordred | corvus: job definition referenced a secret name that didn't exist | 21:01 |
| mordred | corvus: (because of pebkac - the change also adds a secret that didn't get used) | 21:01 |
| corvus | yeah, seems like that should be handled before it gets that far | 21:02 |
| *** sgw has quit IRC | 21:02 | |
| mordred | yah. | 21:02 |
| *** hashar has quit IRC | 21:02 | |
| ianw | https://github.com/bgeesaman/registry-search-order <- interesting lesson on container registry search order and squatters | 21:09 |
| mordred | ianw: that's yet another reason why I think we should use nothing but fully qualified container names in our stuff :) | 21:11 |
| ianw | mordred: yeah, would that mean adding the hostname added by the registry roles when pulling speculative images? | 21:13 |
| mordred | ianw: nope! the specualtive image system supports multi-registry | 21:13 |
| mordred | ianw: this is one of the reasons corvus wrote zuul-registry | 21:13 |
| corvus | ianw: wow, thanks. i had been ambivalent about the search order thing (but out of a habit of conservatism, remove it when i see it). i am no longer | 21:16 |
| *** jamesmcarthur has joined #zuul | 21:22 | |
| mordred | corvus: ++ | 21:26 |
| openstackgerrit | Merged zuul/zuul master: Sort autoholds by request ID https://review.opendev.org/698077 | 21:30 |
| *** pcaruana has quit IRC | 21:31 | |
| *** jamesmcarthur has quit IRC | 21:48 | |
| *** gmann_afk is now known as gmann | 21:48 | |
| clarkb | I've rechecked the nodepool release notes change. Appears to have failed installing gcc? | 21:51 |
| clarkb | both zuul and nodepool have been stable for opendev since I restarted them. Porbably safe to make those releases once the release notes change merges | 21:52 |
| *** Goneri has quit IRC | 21:56 | |
| Shrews | oh fudge. scheduler needs to be restarted to get the autohold sort thing. oh well, it can wait | 22:30 |
| openstackgerrit | Merged zuul/nodepool master: Add missing release notes https://review.opendev.org/698072 | 22:34 |
| corvus | zuul 3.12.0 57aa3a06af28c99e310c53cc6b88d64012029d98 | 23:22 |
| clarkb | neat | 23:23 |
| corvus | nodepool 3.10.0 805d2b4d180e2a3382e718e036ad4c8c521ecfc8 | 23:23 |
| corvus | well, it was more of a question | 23:23 |
| corvus | clarkb, Shrews: do those look right? | 23:24 |
| corvus | fungi: ^ | 23:24 |
| clarkb | oh /me double checks | 23:24 |
| clarkb | corvus: zuul lgtm | 23:25 |
| clarkb | nodepool lgtm too | 23:26 |
| corvus | i'll push those up as soon as i figure out how to run gpg | 23:26 |
| corvus | since that is now hard | 23:26 |
| clarkb | corvus: I do GNUPGHOME=/path/to/mounted/normally/offline/device git tag -s $tag then a ps -elf | grep gpg-agent and I kill the most recent agent | 23:27 |
| clarkb | for some reason my desktop starts one at boot time but that isn't the one that gets used at git tag time so I can't just pkill | 23:27 |
| corvus | thanks... i had to kill the agent since it was attached to emacs for pinentry... ugh. i hate everything about this now. | 23:29 |
| corvus | this has to be some kind of insider plot to sabotoge pki | 23:29 |
| corvus | commit 57aa3a06af28c99e310c53cc6b88d64012029d98 (tag: 3.12.0) | 23:30 |
| corvus | okay, that's what git show 3.12.0 shows me, so i think that's right for zuul | 23:30 |
| corvus | commit 805d2b4d180e2a3382e718e036ad4c8c521ecfc8 (HEAD -> master, tag: 3.10.0, origin/master, origin/HEAD, refs/changes/72/698072/2) | 23:31 |
| corvus | and that's for nodepool | 23:31 |
| corvus | both pushed | 23:32 |
| openstackgerrit | Merged zuul/zuul-jobs master: build-container-image: support sibling copy https://review.opendev.org/697936 | 23:42 |
| *** rlandy has quit IRC | 23:43 | |
| openstackgerrit | Merged zuul/zuul-jobs master: build-docker-image: fix up siblings copy https://review.opendev.org/697614 | 23:49 |
| fungi | corvus: i do this to get gpg to stop trying to spawn graphical pinentry crap, if that's what you need... env DISPLAY="" GPG_TTY=$(tty) git tag -s 3.12.0 | 23:51 |
| corvus | fungi: ah thanks, that's the other half of the equation :) | 23:52 |
| ianw | corvus: hrm, i noticed we just restarted zuul and now my change has a failing buildset registry job | 23:53 |
| fungi | yep, bits 'n' pieces, bits 'n' pieces | 23:53 |
| ianw | see gate currently @ https://zuul.opendev.org/t/zuul/status | 23:53 |
| ianw | no link to logs, although the warnings might be printed now? | 23:53 |
| corvus | ianw: it looks like the parent change still doesn't have a successful image build | 23:54 |
| corvus | 697393 is parent? | 23:54 |
| corvus | its parent is building now, so a recheck of 697393 and 693464 should do it | 23:54 |
| corvus | or rather, its parent has succesfully built | 23:54 |
| clarkb | unrelated to ianw's thing. I have confirmed that the zuul.attempts data in the job inventory seems to be working as expected in production | 23:55 |
| corvus | in other news, zuul and nodepool are both on pypi now; i'll send out announcements | 23:55 |
| fungi | oh!!! | 23:55 |
| fungi | yay releases! | 23:56 |
| *** tosky has quit IRC | 23:56 | |
| ianw | corvus: but the result on 697393 was from several days ago; it checks the last run? (sorry, clearly still not quite ontop of this depends behaviour) | 23:57 |
| corvus | ianw: yep, last or current | 23:57 |
| ianw | ok, thanks | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!