| ykarel | thanks fungi clarkb | 03:51 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:13 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:18 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:19 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:36 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:37 |
| opendevreview | Clark Boylan proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 04:42 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:42 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:46 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 04:47 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 04:55 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 04:56 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 04:59 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 04:59 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 05:32 |
| *** liuxie is now known as liushy | 06:06 | |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 07:08 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 07:08 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 07:08 |
| opendevreview | Dr. Jens Harbott proposed opendev/zuul-providers master: Add debian-trixie build https://review.opendev.org/c/opendev/zuul-providers/+/951471 | 07:15 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 07:31 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 07:31 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 07:31 |
| mnasiadka | clarkb: I took the liberty to update your patch, the whole stack up to RL10 looks green now | 08:50 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 09:12 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 09:12 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 09:12 |
| noonedeadpunk | hey folks! It seems that for some reason we got debian-bullseye jobs quite broken now: https://zuul.opendev.org/t/openstack/build/0bf15350542c440899581cd1b90c73cc | 09:58 |
| noonedeadpunk | I wonder if that could be due to merge of https://review.opendev.org/c/zuul/zuul-jobs/+/954280 | 09:59 |
| noonedeadpunk | as we were relying on backports being absent there: https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/jobs.yaml#L42 | 09:59 |
| noonedeadpunk | I think we need a hold here to figure out on how to workaround difference in "plain" image with CI image | 10:01 |
| noonedeadpunk | and bookworm for the same change passes nicely: https://zuul.opendev.org/t/openstack/build/5539b78bfa4d4c19a13cf37666f6bc22 | 10:08 |
| jrosser | noonedeadpunk: i dont think 954280 is merged yet | 10:48 |
| noonedeadpunk | oh, I was sure I saw it as merged.... but it's obviously not | 10:53 |
| noonedeadpunk | my bad | 10:53 |
| noonedeadpunk | but it could be that we'd still need some hold... as etc folder is not gathered for some reason, so apt update failure is not clear at all | 11:00 |
| fungi | noonedeadpunk: no, backports have ~always been enabled in our images (going back at least 7-8 years as far as i was able to trace), but when we switched to building images with zuul jobs recently the default package mirror baked into the images switched to not being reachable from most clouds. normally that isn't a problem because jobs swap the package mirror hostname out to the one | 13:08 |
| fungi | for the region where the nodes are booted, but we had a useless toggle in that role that allowed jobs to skip updating the mirror hostname for the backports suite sources, leading to that error | 13:08 |
| fungi | in the prior nodepool-built images, that problem was overlooked because apt updates were hitting a reachable mirror address for the backports package indices even if they were in a different cloud provider on the other side of the planet | 13:10 |
| noonedeadpunk | yeah, I somehow saw patch being merged while it's not, so we're having an unrelated issue with bullseye right now | 13:32 |
| fungi | noonedeadpunk: have a link to the failing build? | 13:43 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/0bf15350542c440899581cd1b90c73cc | 13:49 |
| noonedeadpunk | sure, posted it in a first message | 13:49 |
| noonedeadpunk | `Failed to update apt cache: unknown reason` | 13:49 |
| fungi | ah, hard to know for sure if it doesn't log the stdout from the underlying commands it's running | 13:50 |
| fungi | so it certainly could be the same problem | 13:51 |
| noonedeadpunk | bookworm does pass at the same time | 13:53 |
| fungi | did it run in rax-dfw? | 13:53 |
| fungi | if it did, it would be able to reach the baked-in mirror address | 13:54 |
| opendevreview | Elod Illes proposed openstack/project-config master: [release-tools] Improve tagging logs https://review.opendev.org/c/openstack/project-config/+/954340 | 13:54 |
| fungi | clarkb: i actually thought of a question for https://review.opendev.org/954280 (see comments) | 13:59 |
| noonedeadpunk | yeah, so https://review.opendev.org/c/openstack/openstack-ansible/+/954339 seems to work indeed | 14:40 |
| noonedeadpunk | so it indeed might be related | 14:40 |
| fungi | worth noting that option never did what the documentation implied, it wasn't actually disabling the backports source, it was just skipping updating it to use the local mirror | 14:46 |
| clarkb | mnasiadka: thanks for frickler's suggestion maybe we do that in a followup since the stack is now green? I left that in a comment on my change | 14:46 |
| clarkb | fungi: I've responded to your question on 954280 | 14:48 |
| mnasiadka | clarkb: can post a followup on top | 14:55 |
| clarkb | mnasiadka: before you do I just posted a note on the EL 10 changes | 14:57 |
| clarkb | mnasiadka: if we decide to udpate those changes then maybe we should update the parent too. Otherwise it can all go in followups I think | 14:57 |
| clarkb | I can go either way now | 14:57 |
| mnasiadka | clarkb: nested virt rl10 labels would probably be only for naming compatibility from what I understand - because rax classic is the only one that doesn’t support it hardware wise? | 15:07 |
| clarkb | mnasiadka: correct. I think the question is whether or not we feel that explicit naming convention is useful information to convey | 15:07 |
| clarkb | its possible that we think it is redundant and we don't need to update anything | 15:07 |
| clarkb | corvus: ^ you may have an opinion as yuo've been driving much of the bootstrapping work in niz and deciding what needs to stay and what doesn't | 15:08 |
| corvus | i think i lean toward keeping it. because .... technically it's orthogonal, it just happens that the only cloud that doesn't support nested virt is the only cloud that doesn't have recent enough cpu flags to run el10 | 15:11 |
| corvus | but i do not feel strongly about that :) | 15:12 |
| mnasiadka | let me update it, I have a feeling we might omit questions from people using nested-virt labels why is it missing for rl10 ;) | 15:12 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 15:13 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 15:13 |
| mnasiadka | corvus, clarkb: I see centos 9 stream only has 8GB nested virt label - should I stick to that? | 15:15 |
| mnasiadka | or 8 and 16? | 15:15 |
| clarkb | mnasiadka: ya I think the nested virt labels are sticking to the legacy format for now at least so only the 8GB is fine I think | 15:15 |
| mnasiadka | ah, there's only 8G nested-virt flavor | 15:16 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 15:16 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 15:16 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 15:18 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 15:18 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 15:20 |
| mnasiadka | clarkb: I think that should be it ^^ | 15:20 |
| clarkb | ack I'll rereview after the board meeting | 15:20 |
| corvus | i installed python3-virtualenv on zk01 in order to create a personal venv to run zk-shell | 15:50 |
| corvus | ...which is borked: ModuleNotFoundError: No module named 'distutils' | 15:51 |
| clarkb | corvus: that is because python3.12 no longer includes setuptools by default (which vendors distutils) | 15:51 |
| clarkb | corvus: I think you can fix that by installing setuptools in the virtualenv | 15:51 |
| corvus | that helps, thanks | 15:52 |
| fungi | yeah, for pbr we recently started listing setuptools explicitly as a runtime requirement (install-requires) because you can't assume it's present any longer | 15:55 |
| fungi | anything directly relying on setuptools or things that got moved into it from stdlib (distutils, pkg_resources, et cetera) should explicitly declare a setuptools dependency these days | 15:56 |
| clarkb | mnasiadka: there is a bug in the package normalization change update | 15:58 |
| clarkb | mnasiadka: you can see it in CI job results for the rocky 10 change | 15:58 |
| clarkb | but I also left a comment | 15:58 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 15:59 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 15:59 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 16:00 |
| mnasiadka | ugh, missed that when applying frickler’s version | 16:00 |
| corvus | the bug fixed by https://review.opendev.org/954358 has a request stuck.. i will try to manually unstick it. | 16:03 |
| corvus | i fixed it by dequeing the associated change | 16:05 |
| corvus | looking for more problems, i see a node that zuul thinks has been building for 5 days... the cloud also thinks that. so zuul is right, but missing a timeout. | 16:13 |
| corvus | https://review.opendev.org/954360 should take care of that | 16:33 |
| stephenfin | clarkb: fungi: doesn't have to be a today thing, but if we're able to keep the pbr review train moving along slowly I'd appreciate it https://review.opendev.org/c/openstack/pbr/+/954041 | 16:34 |
| fungi | absolutely! thanks for hacking on it | 16:35 |
| stephenfin | fwiw, I am working on adding a PEP-517 backend and parsing of a tool.pbr section from pyproject.toml, since that seems to be the preferred way to extend setuptools going forward | 16:35 |
| clarkb | I'll put it on the list for reviews this afternoon after the morning block of meetings | 16:35 |
| corvus | looks like two requests hit the ready-node-assignment race. i dequeued the change for the second one as well. | 16:37 |
| corvus | i'm going to leave the request for the should-be-timed-out node because that should be self-correcting once https://review.opendev.org/954360 merges and we restart with that. | 16:38 |
| stephenfin | the only snag I've hit so far is that the finalize_distribution_options hookpoint that e.g. setuptools-scm uses is called *before* setuptools starts pulling stuff from pyproject.toml, meaning we need to implement some degree of parsing this ourselves and can't just rely on setuptools populating distribution for us :( | 16:38 |
| stephenfin | It should probably have been called initialize_distribution_options... | 16:38 |
| mnasiadka | hmm, I think the rocky9 image is still from 27th May | 17:07 |
| mnasiadka | corvus: what's the usual workflow? I see that periodic-image-build pipeline did go through at 2AM - but there was no promote job being run after that? | 17:08 |
| corvus | mnasiadka: here's the rocky 9 image status: https://zuul.opendev.org/t/opendev/image/rockylinux-9 | 17:12 |
| mnasiadka | ah, I didn't scroll down | 17:12 |
| corvus | ah yeah, should be sorted by time, recent at bottom | 17:13 |
| corvus | oh and we only need the promote job for gate; the dedicated image-build pipelines create the uploads directly when the buildset is finished | 17:17 |
| mnasiadka | nice | 17:18 |
| mnasiadka | https://review.opendev.org/c/opendev/zuul-providers/+/954299 should be good to go finally, the stacked on top centos and rocky patches are now finishing arm64 jobs (converting images from what I see) | 17:19 |
| opendevreview | Merged opendev/zuul-providers master: Add debian-trixie build https://review.opendev.org/c/opendev/zuul-providers/+/951471 | 17:23 |
| corvus | mnasiadka: ^ oof, conflict just merged | 17:25 |
| mnasiadka | clarkb, corvus: Although it still seems that Kayobe is missing the cirros image - https://zuul.opendev.org/t/openstack/build/fd2ad8dfcd4b458f9e70bc619a8cfb6f/log/primary/ansible/seed-deploy#13531 - any chance it used the old image? | 17:25 |
| mnasiadka | corvus: oh boy | 17:25 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 17:25 |
| fungi | sorry! | 17:26 |
| fungi | (for approving the conflict i mean) | 17:26 |
| corvus | mnasiadka: https://review.opendev.org/954299 is wip | 17:27 |
| corvus | does it automatically do that if you rebase ttw? | 17:27 |
| mnasiadka | yes | 17:28 |
| mnasiadka | just updating it - sometimes it's faster for me to use the web interface | 17:28 |
| clarkb | I guess it does that beacuse all of the conflict markers are put into the files in the rebase patchset | 17:28 |
| corvus | https://zuul.opendev.org/t/openstack/build/fd2ad8dfcd4b458f9e70bc619a8cfb6f/log/zuul-info/zuul-info.primary.txt#6 looks like it's the old image | 17:28 |
| fungi | having not really used gerrit's remote rebase often, i didn't realize they'd changed that | 17:29 |
| corvus | oh heh yeah the conflict markers are in there | 17:29 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 17:29 |
| fungi | neat, so maybe only happens if it's a nontrivial rebase? | 17:29 |
| clarkb | ya I think it must consider that a staging ground for fixingthem via the web ui | 17:29 |
| fungi | it used to just refuse to rebase in those circumstances, so i guess i'll consider that a sort of feature | 17:29 |
| mnasiadka | but now clarkb needs to mark it as active since I'm not the owner :) | 17:30 |
| clarkb | on it | 17:30 |
| clarkb | done | 17:30 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 17:31 |
| corvus | mnasiadka: yeah, that job ran a few hours before the upload of the new image started. i have a pending update to the images page that will show the completion time for the image upload; but for now, we can see that it even predates the start of the upload. | 17:31 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 17:31 |
| corvus | fungi: https://review.opendev.org/954299 could use a +3 from you | 17:32 |
| priteau | Regarding the rocky9 image, I think something must have happened on June 30 or July 1. Before this time, we were using images built on the same day, but now jobs always use an image from end of may | 17:32 |
| priteau | Did something change in the image selection logic? | 17:32 |
| mnasiadka | nodepool to zuul-launcher switch? | 17:32 |
| clarkb | mnasiadka: priteau yes that was the change | 17:33 |
| priteau | This? https://review.opendev.org/c/opendev/zuul-providers/+/952721 | 17:33 |
| clarkb | since there was the bug in the zuul-launcher image builds that didn't include cirros image caching on images other than ubuntu | 17:33 |
| mnasiadka | priteau: yup, this | 17:33 |
| corvus | new rocky 9 images are uploading now | 17:34 |
| priteau | indeed, but they don't appear to be used by our jobs. Do we need to change nodeset? | 17:35 |
| corvus | no i mean they are literally being uploaded to the cloud providers, but that takes like an hour | 17:35 |
| priteau | OK, I will try again later | 17:35 |
| fungi | being uploaded *now* | 17:35 |
| priteau | So it is expected that we switched to a rocky9 image built on May 27, following this 952721 change? | 17:36 |
| mnasiadka | yes, because there were some other problems that caused no newer uploads of these images - but that was fixed sort of yesterday I think - see https://opendev.org/opendev/zuul-providers/commit/1ed3cf6514d863ccca74f5c5d1e265486eed749c | 17:37 |
| priteau | I had missed that this zuul-launcher migration was happening, thanks | 17:38 |
| mnasiadka | and https://opendev.org/opendev/zuul-providers/commit/d492a86f2fbdb16b2db495015b330dcf41accfab | 17:38 |
| fungi | priteau: we tried really hard to make it as seamless as possible, there wasn't supposed to be any user-facing impact from the transition, but ran into several bugs after starting to switch the openstack tenant that we didn't hit with the other zuul tenants unfortunately | 17:41 |
| fungi | or at least that we didn't spot when doing them first | 17:41 |
| fungi | we actually rolled back the switch for the openstack tenant once, this is the second attempt | 17:42 |
| clarkb | the changes allow us to test changes to the image builds before we publish them. It also allows you to have nodesets with nodes from different resource provider types. It should be a nice benefit over time but we're helping find all the corner cases before everyone is expected to run zuul this way | 17:44 |
| corvus | aiui, the may 27 image is still acceptable; it's just missing a cached file. if it wasn't, we could have deleted it at any time in accordance with our normal practice. | 17:45 |
| clarkb | corvus: correct | 17:45 |
| clarkb | we intended to cache the file (which is now corrected) but also the job doesn't properly fall back to pulling the file if not present in the cache | 17:46 |
| clarkb | and we may proceed wtih removing that file from the cache due to its age at some point in the future | 17:46 |
| mnasiadka | priteau: I assume it would be wise to check for existence of the file and download it if it's not judging by what clarkb is saying | 17:47 |
| fungi | yes, that's what devstack does, for example | 17:49 |
| fungi | because if you run it locally then there's a good chance you won't have a copy of that file cached initially, but subseuent runs can take advantage of not having to re-download it then | 17:49 |
| clarkb | and it allows us to have infrequently run stable jobs continue working as we cleanup old resources on the images | 17:50 |
| fungi | that was the original reason devstack did that, and we supplemented that behavior by simply pre-warming the cache in our test node images | 17:50 |
| corvus | mnasiadka: i got my wires crossed on https://zuul.opendev.org/t/openstack/build/fd2ad8dfcd4b458f9e70bc619a8cfb6f/log/zuul-info/zuul-info.primary.txt -- that build started 3 minutes after the rocky9 image from the nightly update finished uploading; but the node was launched for that build 5 minutes before it finished uploading. | 17:56 |
| corvus | mnasiadka: so, yeah, missed it by '' <-- that much :) | 17:57 |
| mnasiadka | nice | 17:57 |
| mnasiadka | corvus: but yes, some timestamps in zuul images tab would be helpful for end users like me ;-) | 17:57 |
| mnasiadka | (timestamps when these images got imported on a cloud) | 17:58 |
| corvus | yep, i'm expecting https://review.opendev.org/954063 to get reviewed tomorrow... | 17:59 |
| corvus | but if you want, you can visit https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_980/zuul/980496d1b44d4243aebcbf129ac57014/npm/html/ and use that change now before it merges | 18:00 |
| corvus | i confirmed that does show the upload finished time... the only extra piece of information i have from the logs is when the node creation started. but, with only a 3 minutes difference, it's a pretty good guess it started with the old image. | 18:01 |
| mnasiadka | that's surely enough | 18:01 |
| corvus | oh, hrm.... so about those nested-virt labels... apparently the labels we used with nodepool are "nested-virt-ubuntu-noble" not "ubuntu-noble-nested-virt" | 18:14 |
| clarkb | oops | 18:15 |
| corvus | i guess i will need to switch those around for the legacy labels at least... but do we want to switch the order for the new labels? | 18:15 |
| clarkb | I do think the new order (distro-attribute) makes more sense to me | 18:16 |
| clarkb | since distro is the primary thing people care about when considering a platform. then its the details | 18:16 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 18:21 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Correct nested-virt nodepool labels https://review.opendev.org/c/opendev/zuul-providers/+/954368 | 18:21 |
| corvus | oh no sorry about that | 18:21 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 18:22 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 18:23 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 18:23 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Correct nested-virt nodepool labels https://review.opendev.org/c/opendev/zuul-providers/+/954368 | 18:25 |
| corvus | okay that's all fixed. i stacked the nested-virt fix on the end of that stack since it conflicted. | 18:26 |
| opendevreview | James E. Blair proposed opendev/zuul-providers master: Correct nested-virt nodepool labels https://review.opendev.org/c/opendev/zuul-providers/+/954368 | 18:28 |
| priteau | corvus: our jobs are working now, thanks! | 18:32 |
| fungi | awesome | 18:32 |
| noonedeadpunk | fungi: I have a question about this one: https://opendev.org/openstack/openstack-zuul-jobs/commit/70fa0bd649fe39ee36130bbfeeefc08325fcfc6c | 19:27 |
| noonedeadpunk | or well... | 19:27 |
| fungi | what's the question? | 19:27 |
| noonedeadpunk | I found answer :) | 19:27 |
| noonedeadpunk | but it;s basically https://zuul.opendev.org/t/openstack/build/8c144f3894cf41a9a3db8323ac6199f7 | 19:27 |
| noonedeadpunk | and the job on bionic for some reason | 19:27 |
| noonedeadpunk | so it has very old git | 19:28 |
| fungi | oh! | 19:28 |
| noonedeadpunk | and quite some are affected https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-update&pipeline=periodic&skip=0 | 19:28 |
| fungi | yes, i didn't anticipate that we still had any jobs running old enough git that they don't support the --trailer cli option | 19:28 |
| fungi | yeah, i suspect only the translation jobs are still running on bionic, because of zanata only working with very old java | 19:29 |
| noonedeadpunk | yeah https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L1181 | 19:29 |
| noonedeadpunk | but I heard smth of ubuntu support of openjdk 8 on some later releases... not sure though | 19:30 |
| fungi | i can try to put together a patch to check the git version or platform and then skip that option | 19:30 |
| noonedeadpunk | I have openjdk-8-jre-headless on noble fwiw | 19:32 |
| noonedeadpunk | https://paste.openstack.org/show/bEg0RFbMGoQcIc0HYJYu/ | 19:32 |
| fungi | ooh | 19:33 |
| noonedeadpunk | which is exactly what we need? https://opendev.org/openstack/openstack-zuul-jobs/src/commit/70fa0bd649fe39ee36130bbfeeefc08325fcfc6c/roles/prepare-zanata-client/vars/Debian.yaml#L1 | 19:33 |
| noonedeadpunk | so, like | 19:33 |
| noonedeadpunk | should I try just updating nodeset? | 19:33 |
| clarkb | that seems promising. And ya updating the nodeset is where I would start | 19:33 |
| fungi | absolutely | 19:34 |
| noonedeadpunk | it's just that I guess we will see result only on periodic | 19:34 |
| noonedeadpunk | and if it would fly | 19:34 |
| fungi | that's the best possible outcome short of finishing the weblate migration | 19:34 |
| noonedeadpunk | but then I wonder if we're really stick with Zanata on bionic as server... | 19:34 |
| noonedeadpunk | anyway, let's start from client | 19:34 |
| fungi | yeah, deploying a new zanata server is possible, but would be a lot of work that we may just avoid for a little longer until weblate is the thing | 19:35 |
| clarkb | swapping the zanata os doesn't necessarily get you the runtime framework (wildfly/jboss) nor does it update zanata itself | 19:36 |
| clarkb | I don't personally think the effort involved there is worth the minor improvement | 19:36 |
| noonedeadpunk | ugh, there're still some jobs with Xenial even... | 19:38 |
| noonedeadpunk | ok, I needed to rebase :) | 19:38 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/project-config master: Switch translate proposal jobs to noble https://review.opendev.org/c/openstack/project-config/+/954370 | 19:41 |
| fungi | too bad that won't get tested speculatively | 19:42 |
| noonedeadpunk | yeah | 19:42 |
| noonedeadpunk | as the job is post-review | 19:42 |
| noonedeadpunk | (as require secrets) | 19:43 |
| fungi | clarkb: i'm going to pop out for a quick dinner before i finish up quarterly planning paperwork, but will be around in an hour-ish to talk about cla removal from the gerrit image if that works for you | 19:43 |
| clarkb | fungi: yup it should. I can dig up lunch in that rough same block of time | 19:43 |
| fungi | perfect | 19:44 |
| clarkb | fungi: I've rechecked https://review.opendev.org/c/opendev/system-config/+/882900 as it is a bit older and want to amke sure it is still working today. That change moves gerrit images to quay.io | 20:46 |
| clarkb | fungi: I think at the very least we could land that change and restart to cover the quay.io move and the zuul status tab updates. But also I think cleaning up the cla resources in the gerrit image may be able to be consolidated into that restart as well since all three items are largely independent of one another the chance that one of them impacts another is low | 20:47 |
| fungi | clarkb: makes sense | 20:51 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/gerrit/run.yaml#L37-L56 is the code that we need to clean up to stop including the cla files in gerrit | 20:51 |
| clarkb | I guess my main questions are do we need to remove cla configuration from gerrit on the config side first (I think so otherwise we'll be pointing at files that don't exist but maybe this is ok) and if so what is the timeline we're thinking about for that | 20:52 |
| fungi | we can't really take the clas out of gerrit until we remove enforcement in remaining acls, and i want to give readers of service-announce a bit of time to react and submit changes to add dco enforcement or whatever, thinking like a week lead time | 20:52 |
| clarkb | fungi: do we know which projects haven't updated yet? I'm guessing its stuff in x/? | 20:52 |
| fungi | will know in a moment... | 20:52 |
| fungi | (stand by) | 20:53 |
| corvus | [hold music plays] | 20:53 |
| fungi | 265 acl files in total | 20:55 |
| fungi | 210 are in the x namespace | 20:55 |
| fungi | other namespaces include: cfn, opendev, openinfra, performa, recordsansible, sardonic, windmill | 20:57 |
| clarkb | windmill is effectively eol. I wonder what opendev repos are affected. I suspect we can just change openinfra | 20:58 |
| clarkb | re cfn I responded to them about the "corrupt" git repo and never heard back... | 20:58 |
| clarkb | anyway this list seems mostly reasonable | 20:58 |
| clarkb | I suspect that gerrit will force us to unconfigure all the enforce cla rules before removing the cla from config too but I'm not sure of that | 20:59 |
| fungi | change on the way | 20:59 |
| fungi | just composing the commit message now | 20:59 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Remove CLA enforcement from all projects and lock https://review.opendev.org/c/openstack/project-config/+/954374 | 21:04 |
| fungi | clarkb: ^ | 21:04 |
| fungi | i'll do the actual config removal and cleanup as a child of that | 21:06 |
| clarkb | fungi: for the openinfra/ projects should they be switched to dco? | 21:11 |
| clarkb | I think for everything else removal makes sense given the commit message | 21:11 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop installing and configuring CLAs https://review.opendev.org/c/opendev/system-config/+/954376 | 21:13 |
| clarkb | and I guess https://review.opendev.org/c/opendev/system-config/+/954376/1/doc/source/gerrit.rst implies a manual step between the two changes (not a big deal just calling that out) | 21:14 |
| fungi | clarkb: i'll propose a separate change to add dco enforcement to the openinfra namespace projects after i confer with foundation staff | 21:14 |
| fungi | yeah | 21:14 |
| clarkb | ack that can be a child of the removal and merging them together shoudl avoid any unexpected behaviors | 21:15 |
| fungi | also just noticed i need to add a depends-on to that one | 21:15 |
| * mordred would vote for things to just require GPG signed commits, but knows he's lost that battle in the general developer ecosystem mindspace | 21:15 | |
| fungi | different repo | 21:15 |
| clarkb | the only other consideration I have is that I'm out most of next week so maybe we want to try and land the gerrit image affecting changes on the 21st? | 21:15 |
| fungi | mordred: we can just create our own theme park. with blackjack et cetera | 21:16 |
| clarkb | we'll definitely want to do a gerrit restart after the changes land so want to do that when we think we'll be able to restart the service too | 21:16 |
| fungi | yeah | 21:16 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop installing and configuring CLAs https://review.opendev.org/c/opendev/system-config/+/954376 | 21:16 |
| fungi | depends-on to the mass acl change added, and set to wip for now | 21:17 |
| mordred | fungi: fungi: https://bojackhorseman.fandom.com/wiki/Disneyland like this? | 21:17 |
| opendevreview | Ian Y. Choi proposed openstack/project-config master: Move to use prepare-weblate-client job https://review.opendev.org/c/openstack/project-config/+/954377 | 21:17 |
| fungi | mordred: i was going for a futurama reference, but absolutely sure | 21:18 |
| clarkb | fungi: we could also aim for the 18th and if it doesn't happen then do the 21st | 21:18 |
| clarkb | maybe thats best | 21:18 |
| clarkb | (I'm just worried I'll be busy catching back up again on the 18th but maybe it will be fine) | 21:18 |
| mordred | fungi: Hooray ... question mark? | 21:18 |
| mordred | Sorry - we can all blame mtreinish for my Bojack addiction | 21:19 |
| clarkb | I think I have a new season of futurama Istill need to watch | 21:20 |
| clarkb | maybe I'll watch that alongside the new king of the hill when it comes out | 21:20 |
| clarkb | to summarize we'll land https://review.opendev.org/c/opendev/system-config/+/882900 and https://review.opendev.org/c/opendev/system-config/+/954376 and their parent change(s) on the 18th or shortly before with a plan to restart gerrit on the 18th to pick up the new zuul status stuff, quay.io image hosting, and cla file removals. IF the 18th is too busy we'll defer to the 21st | 21:22 |
| clarkb | in the maen time fungi will announce the CLA removal which gives ~10 days of notice | 21:22 |
| clarkb | fungi: ^ is that the plan? | 21:22 |
| fungi | wfm, yep | 21:23 |
| mordred | clarkb: have y'all watched the Great North yet? if not, highly recommend adding it to your adult cartoon rotation | 21:23 |
| fungi | i'll try to get something out to service-announce today or tomorrow | 21:23 |
| clarkb | fungi: perfect, thanks! | 21:24 |
| * clarkb scribbles some notes for the 18th | 21:24 | |
| opendevreview | Merged opendev/zuul-providers master: Normalize infra-package-needs packages on current releases https://review.opendev.org/c/opendev/zuul-providers/+/954299 | 21:32 |
| opendevreview | Merged opendev/zuul-providers master: Add CentOS Stream 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/953460 | 21:32 |
| opendevreview | Merged opendev/zuul-providers master: Add Rocky Linux 10 builds https://review.opendev.org/c/opendev/zuul-providers/+/954265 | 21:32 |
| corvus | clarkb: fungi can you +3 https://review.opendev.org/954368 ? (nested virt fix as discussed earlier) | 21:33 |
| fungi | looking | 21:37 |
| clarkb | +2 from me | 21:39 |
| fungi | and approved | 21:40 |
| opendevreview | Merged opendev/zuul-providers master: Correct nested-virt nodepool labels https://review.opendev.org/c/opendev/zuul-providers/+/954368 | 21:40 |
| opendevreview | Merged openstack/project-config master: Switch translate proposal jobs to noble https://review.opendev.org/c/openstack/project-config/+/954370 | 22:46 |
| fungi | noonedeadpunk: ^ i'll try to check the daily runs when i wake up, but i expect you'll be around far earlier | 22:48 |
| noonedeadpunk | yeah, we have some translations pending | 22:52 |
| noonedeadpunk | so we either see a patch incoming or not :) | 22:52 |
| fungi | and if not, i'll rectify it tomorrow | 22:53 |
| noonedeadpunk | will check on it as well if not so no worrie | 22:54 |
| clarkb | stephenfin: I properly reviewed the first half or so of the pbr stack. I think in general things are fine but I noticed some small bugs here and there which I noted. Then as I ran out of steam in the later half of the stack I tried to leave higher level observations. I'm not opposed to any of the deprecations you are adding but as noted I think we may want to decide early (as part of | 23:00 |
| clarkb | this work) what the removal timeline is for that if at all and then try to make that clear within the code | 23:00 |
| clarkb | stephenfin: PBR has at times been a honey pot for people making easy changes to "clean up" things not undersatnding its support timeframes are different than openstack's proper because old openstack still exists | 23:00 |
| fungi | also part of the timeframe planning needs to include release checkpoints | 23:01 |
| clarkb | stephenfin: anyway I'd like to capture a minimal "this is what deprecation means for pbr" if we can. I think from my perspecitve we decprecate things to encourage peopel to change to the modern canonical options but we don't necessarily remove them if they may have been in use and we can continue to support the old packages | 23:01 |
| fungi | how many releases are we going to split this work across? which ones need to be major version bumps? | 23:02 |
| clarkb | stephenfin: the reason I have that position is the instant we stop accepting summary instead of description for packages we'll break every single package that used the old version | 23:02 |
| clarkb | fungi: major versions bumps aren't useful here due to how pbr works | 23:02 |
| clarkb | the old packages on old python with old pip and setuptools are going to get the latest pbr version when you try to manipulate those sdists | 23:02 |
| fungi | i think it's still a useful means of communicating backward-incompatible changes, even if it can't be used to delay their effects | 23:03 |
| clarkb | well my point is that we shouldn't release them in the first place if they are backward incompatible :) | 23:03 |
| fungi | also not entirely true wrt old versions of python, if we're careful about setting supported python versions, but it would depend on the specifics | 23:04 |
| fungi | we've dropped support for some pbr features already in recent years | 23:04 |
| clarkb | I think what I would suggest if we want to ultimately remove that code is that we make a PBR2 or PBRng package that removes that stuff and then packages can convert to it | 23:04 |
| clarkb | then we leave old pbr alone | 23:04 |
| fungi | and those are backward-incompatible changes | 23:04 |
| fungi | or were | 23:05 |
| clarkb | fungi: but only because we couldn't support them anymorewithout copying large chunks of setuptools | 23:05 |
| clarkb | I'm talking about summary ~= description | 23:05 |
| clarkb | breaking that is silly imo | 23:05 |
| clarkb | its trivial to support forever | 23:05 |
| fungi | well, yes that's the sort of thing we could just carry a forever alias on | 23:05 |
| clarkb | and almost certainly packages will break if we stop | 23:05 |
| fungi | you're clearly further down (or up?) the stack of changes than i am | 23:06 |
| fungi | so i'm speaking in generalities | 23:06 |
| clarkb | ya I am too fwiw. I think each deprecation and potential removal will need its own consdieration. What I mostly want to do is firmly assert that pbr doesn't use openstack deprecation process | 23:06 |
| clarkb | which most people who push packages to pbr to delete deprecated things assume | 23:07 |
| fungi | i don't yet know what the pending removal of scriptwriter and eventually pkg_resources is going to mean for features in pbr, at least not yet | 23:07 |
| clarkb | I think stephenfin has fixed all but one pkg_resources usage in that stack | 23:07 |
| clarkb | so hopefully that one is largely a non issue | 23:07 |
| fungi | oh wow, that's sooner thani anticipated | 23:07 |
| clarkb | and if we ignore wsgiscripts then pip apparently writes cli scripts without the costly entrypoint scanning and that is largely a non issue for everyone but openstack (which relies on wsgiscript) | 23:08 |
| clarkb | anyway I think the main thing is we recognize that deprecation doesnt' start a 12 month clock. Some things may be deprecated and exist forever. Others might be cleaned up. We have to use careful judgement | 23:08 |
| clarkb | which is different than deprecating something in nova | 23:09 |
| fungi | agreed, there is no absolute correlation between "deprecated" and "will be removed at some point" | 23:13 |
| fungi | deprecated just means there's a better/newer/preferred alternative | 23:13 |
| fungi | not necessarily pressure to stop doing things the old way if they can be kept working | 23:14 |
| clarkb | yup and I do think taking stock of them from our side and deciding if a PBRng makes sense is another useful side effect | 23:14 |
| clarkb | its possible we reach a point where that makes sense and one signal for that could be the amount of deprecated code we're needing to maintain | 23:15 |
| clarkb | my fear with doing that is that og PBR is likely to break due to future setuptools changes so now we're fixingtwo code bases instead of one | 23:16 |
| clarkb | and I'm not sure that is simpler overall | 23:16 |
| clarkb | but it may still be worthwhile in the future particularly when packages have greater control over the setuptools version they use (via pyproject.toml for example0 | 23:16 |
| clarkb | I'll also try to properly review the second half of that stack tomorrow | 23:26 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!