| *** mrunge_ is now known as mrunge | 00:56 | |
| dtantsur | Hey folks, is openstack-tox-py313 permafailing already known? It seems to be an infra / mirror / node issue | 09:42 |
|---|---|---|
| dtantsur | https://zuul.opendev.org/t/openstack/build/3fee0c50eee744aeacd40e38703358ff | 09:43 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [WIP] Make fips setup compatible to 10-stream https://review.opendev.org/c/zuul/zuul-jobs/+/961208 | 09:53 |
| frickler | dtantsur: it is neither, just the switch from running py313 jobs on noble to trixie in order to be able to use native python. https://review.opendev.org/c/openstack/ironic/+/973818 should fix it | 13:12 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [WIP] Make fips setup compatible to 10-stream https://review.opendev.org/c/zuul/zuul-jobs/+/961208 | 13:13 |
| dtantsur | frickler: ah, thanks, I didn't realize the patch is fixing the CI.. | 13:18 |
| janders | thank you frickler and dtantsur | 13:35 |
| *** rcruise1 is now known as rcruise | 13:37 | |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Fix missing packages for doc build in translation update https://review.opendev.org/c/openstack/project-config/+/973947 | 14:45 |
| janders | frickler I rechecked a patch once the fix you linked merged, it seems my CI run is still having issues: https://zuul.opendev.org/t/openstack/build/b5ee53e4c67a4cb385adbbab4ae0536a | 15:01 |
| janders | I wanted to check if there's another issue? | 15:01 |
| janders | https://zuul.opendev.org/t/openstack/build/b5ee53e4c67a4cb385adbbab4ae0536a/log/job-output.txt#394 makes me wonder | 15:01 |
| janders | context: https://review.opendev.org/c/openstack/ironic/+/966344 | 15:02 |
| frickler | ah, that's the arm64 job still running on noble. either we need to switch it to trixie, too, or it needs fixed job settings. | 15:25 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Validate crc32 checksum for S3 image upload https://review.opendev.org/c/zuul/zuul-jobs/+/973689 | 15:25 |
| clarkb | fungi: do we wait for openstack ansible to ack https://review.opendev.org/c/openstack/project-config/+/973363 and https://review.opendev.org/c/openstack/project-config/+/973387 before approving? | 15:45 |
| clarkb | I seem to recall we may need to manually unretire projects? I can't remember if manage-projects is capable of resetting the read only flag if we set it | 15:46 |
| clarkb | oh wait noonedeadpunk proposed the changes so thats the ack | 15:48 |
| clarkb | I didn't approve them in case the unretire steps require us to do something more. I just don't remember and its early | 15:48 |
| fungi | i thought i had left a comment about not needing to do any manual steps because the openstack/retired acl only limits permissions and doesn't switch the project to read-only state | 15:49 |
| clarkb | fungi: oh you did on the freezer stack but not the watcher stack and I was looking at watcher first | 15:50 |
| clarkb | and agreed if read only isn't set then we should be good. I'll approve the first two I guess | 15:50 |
| fungi | aha, i missed the watcher changes, thanks! | 15:52 |
| opendevreview | Merged openstack/project-config master: Fix missing packages for doc build in translation update https://review.opendev.org/c/openstack/project-config/+/973947 | 15:54 |
| opendevreview | Merged openstack/project-config master: Revive os_freezer role for OSA https://review.opendev.org/c/openstack/project-config/+/973363 | 15:55 |
| opendevreview | Merged openstack/project-config master: Revive os_watcher role for OSA https://review.opendev.org/c/openstack/project-config/+/973387 | 15:55 |
| clarkb | infra-root any objections to me approving https://review.opendev.org/c/opendev/system-config/+/970173 and https://review.opendev.org/c/opendev/system-config/+/970180 ? This will add gerrit 3.12 images and testing to our system. It shouldn't updated gerrit 3.11 which we use in production but accidentially updating 3.11 and thus wanting to do a gerrit restart is probably the main risk | 16:44 |
| fungi | i'm in favor of going ahead, yes | 16:47 |
| opendevreview | James E. Blair proposed openstack/project-config master: Remove #opendev from gerritbot https://review.opendev.org/c/openstack/project-config/+/973963 | 16:48 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch #opendev logging to Matrix https://review.opendev.org/c/opendev/system-config/+/973964 | 16:48 |
| corvus | i don't remember seeing those changes, so i went ahead and staged them for later today ^ | 16:49 |
| fungi | related to ^ we should also merge https://review.opendev.org/c/opendev/system-config/+/972510 today | 16:50 |
| clarkb | fungi: I'll review all three changes now then we can approve them as we switch over? | 16:50 |
| fungi | sgtm | 16:51 |
| fungi | the documentation one is probably a little looser on how exactly we need to time deploying | 16:51 |
| corvus | clarkb: re https://review.opendev.org/970173 does that change all gerrit images to trixie? so that would be a change to 3.11 if we were to build it? (so, if we decided we needed to do a 3.11 update, we would end up on trixie after re-enabling that?) | 16:52 |
| clarkb | corvus: we changed gerrit to trixie last week friday and the change that did so failed to update the zuul job level requires which I try to clean up there | 16:53 |
| clarkb | corvus: so yes it would be a change but that change should already be in place as part of the java 21 update. I just failed to fully capture it in the zuul config | 16:54 |
| corvus | oh got it | 16:54 |
| fungi | sometimes, zuul also tests the past ;) | 16:55 |
| corvus | something about disabling 3.11 rubs me the wrong way -- like, why are we afraid of building those images? | 16:56 |
| clarkb | corvus: https://opendev.org/opendev/system-config/src/branch/master/docker/gerrit/base/Dockerfile#L18 I double checked too and the Dockerfile has already moved | 16:56 |
| clarkb | corvus: the main concern I have is we're likely to build a new version of gerrit (whth new bugfixes from upstream) and I don't want that to hang around as a potential problem if Gerrit restarts on the new image for some reason | 16:57 |
| clarkb | corvus: I think the risk is low and maybe what we should be doing going forward is building gerrit from tags instead of the stable branch? | 16:57 |
| clarkb | basically I don't awnt the pressure of I need to restart gerrit in production just to add gerrit 3.12 images | 16:57 |
| clarkb | another approach would be to reenable the jobs in an indpendent change so that we aren't waiting for the next expected image update to reenable them? | 16:58 |
| clarkb | looking at https://gerrit.googlesource.com/gerrit/+log/refs/heads/stable-3.11 if we rebuild the image right now we shouldn't pull in any additional commits compared to what is running in production | 17:00 |
| corvus | yeah, if we don't want the image to change unless we do something explicit, we should probably switch to tags. if we want to keep up with images better, then we should maybe restart gerrit more often (weekend? :) but mainly, if our workflow is "comment something out otherwise critical error may happen" we should probably address it. :) | 17:01 |
| corvus | i think either of your suggestions would improve things, depending on which way you want to go | 17:02 |
| clarkb | I think my main concern is problems like the 3.11.6 release getting withdrawn | 17:03 |
| clarkb | it isn't clear why that is the case and maybe it was release building specific rather than a codebase problem which we're doing for ourselves | 17:04 |
| clarkb | but considering they have pulled releases as recently as a bugfix for the major release we're on I try not to blindly upgrade gerrit | 17:04 |
| clarkb | maybe we wait for 3.11.8 and then switch to tags on all the builds? | 17:05 |
| clarkb | corvus: does ^ seem reasonable to you if I split the nodejs and job reenable changes into two so we can land the job reenable changes with the gerrit 3.12 updates? | 17:07 |
| clarkb | mordred wrote the original jobs and may have thoughts on why stable was chosen (and to be fair to stable gerrit branches we have had pretty reliable success with them) | 17:07 |
| corvus | clarkb: yeah, getting those comments out asap is good, then later we can figure out how to avoid putting them in. :) | 17:08 |
| clarkb | ack I'll rework that last chagne in the stack momentarily | 17:09 |
| corvus | maybe we can see what we need to to do move the build stamp to a zuul variable? | 17:09 |
| clarkb | corvus: iirc the main issue there is triggering the promotion jobs. Simply updating the jobs themselves didn't work in the past (but maybe that is different now with gerrit event reordering/) | 17:09 |
| clarkb | corvus: when we updated the image build jobs say to only add gerrit 3.12 images then check and gate worked fine but we didn't promote the image after merging | 17:10 |
| clarkb | the fix was to update the Dockerfile and match that file in the files list on the jobs including the promote job | 17:10 |
| fungi | i'll note that there were past cases where gerrit pulled a release because they tagged the wrong commit, or because they accidentally pulled in commits from another branch without noticing | 17:12 |
| corvus | i think the promote issue will remain, because it'll be running after the reconfig, so there won't be a speculative state change to trigger it. but maybe we can take a page from the zuul-providers repo and run all the promote jobs after a dispatch job that decides which should actually be run based on what ran in gate. i think that would work. | 17:12 |
| corvus | btw dispatch jobs are easier now with init jobs, we can probably end up removing zuul config by doing that :) | 17:13 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update node to 24 for Gerrit builds https://review.opendev.org/c/opendev/system-config/+/970451 | 17:15 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Reenable Gerrit 3.11 image builds https://review.opendev.org/c/opendev/system-config/+/973970 | 17:15 |
| clarkb | corvus: cool I tried to capture that thought in the commit message in the split change | 17:15 |
| clarkb | another approach would be to use version specific files to trigger off of | 17:16 |
| clarkb | since I think part of the issue here is the common backend for building multiple images. Maybe we just need some distinct data to manage this | 17:16 |
| corvus | could work, but i think the dispatch job is a good model here | 17:18 |
| corvus | i'd want to try that first since it lets us keep a lot of stuff in common, if we want. | 17:18 |
| corvus | (we can always shard more with the dispatch job if we want, it's just not required) | 17:18 |
| clarkb | sounds good | 17:20 |
| clarkb | I'm about to be in the meeting tunnel until lunch then have antoher meeting afterwards. I'm thinking maybe I wait on approving those until after the matrix switchover changes have landed | 17:21 |
| clarkb | but I'll try to get to them togday | 17:21 |
| mnasiadka | corvus: is the testing approach in https://review.opendev.org/c/zuul/zuul-jobs/+/973423 fine - or do you have some better recommendation? | 17:40 |
| corvus | mnasiadka: iiuc, we're basically adding mirror-info to the test playbook in zuul-jobs even though it's not defined in there. i think it's okay to do that since we see opendev as a good test case, so relying on its role to do some additional testing seems fine (basically, it's representing a consumer of our api). if opendev removes that role later, we can drop it from the testing in zuul-jobs with no problem. i think it might be worth a note | 17:46 |
| corvus | in the test-playbook mentioning why we're doing that. | 17:46 |
| corvus | alternately, moving mirror-info into zuul-jobs might be a good idea. or even just updating configure-mirrors to do what mirror-info does? i haven't looked at that. but those might produce a cleaner system. | 17:46 |
| mnasiadka | In the longer run maybe yes, but I’m unsure all env variables in mirror-info are really used anywhere | 17:47 |
| clarkb | corvus: the current opendev mirror-info role has some old nodepool compatibility stuff in it | 17:47 |
| clarkb | (that is what the env vars are) | 17:47 |
| corvus | oh yeah that shouldn't go in zuul-jobs then | 17:48 |
| clarkb | it is possible that opendev could drop that at this point. I know that tripleo was a big consumer of it. I'm not sure if there aer others though | 17:48 |
| corvus | probably that should be deleted? :) | 17:48 |
| clarkb | https://codesearch.opendev.org/?q=NODEPOOL_&i=nope&literal=nope&files=&excludeFiles=&repos= this shows use of it in puppet-openstack, dib, ironic, etc | 17:51 |
| clarkb | but yes maybe we should put an expiration date on that finally | 17:51 |
| mnasiadka | I can just propose patches to respective repos that use nodepool vars to switch to what is available now (and remove the vars from mirror-info in the end) | 18:00 |
| clarkb | I think we can approve https://review.opendev.org/c/openstack/project-config/+/973963 now to remove gerritbot from irc here | 19:40 |
| fungi | doing so | 19:41 |
| fungi | standing by to stop meetbot once our meeting wraps up | 19:41 |
| clarkb | then approve https://review.opendev.org/c/opendev/system-config/+/973964 to do the logging switch with someone ready to shutdown limnoria | 19:42 |
| clarkb | sounds like you're on it thanks | 19:42 |
| clarkb | then we can update docs afterwards | 19:42 |
| clarkb | fungi: I think we can probably also approve https://review.opendev.org/c/opendev/system-config/+/973964 now as check jobs took 20 something minutes | 19:45 |
| fungi | i approved it about 5 minutes ago | 19:46 |
| clarkb | oh heh | 19:46 |
| opendevreview | Merged openstack/project-config master: Remove #opendev from gerritbot https://review.opendev.org/c/openstack/project-config/+/973963 | 19:52 |
| clarkb | the change to update channel logging is about 10 minutes away from merging then a few more minutes after that to deploy | 19:54 |
| clarkb | fungi: infra-prod-service-eavesdrop runs during hourly deploys so the shutdown of limnoria may need to happen before the hourly job depending on when things merge as well | 19:55 |
| clarkb | oh looks like the latest estimate is ~16 minutes to merge | 19:55 |
| clarkb | I just sent the email | 20:01 |
| clarkb | hourlies are running now so I don't think we need to worry about conflict there | 20:03 |
| fungi | i'm logged into eavesdrop02 with `sudo docker-compose -f /etc/ircbot-docker/docker-compose.yaml down` queued in my terminal | 20:04 |
| fungi | once 973964 merges i'll run that | 20:05 |
| fungi | of course, we won't see gerritbot announce 973964 in here because the change to remove it from the channel has already deployed | 20:05 |
| clarkb | perfect | 20:05 |
| clarkb | yup but it should announcein the matrix room | 20:05 |
| fungi | right, which we won't log ;) | 20:05 |
| fungi | but those of us present in both channels will still see it | 20:06 |
| clarkb | I half wonder if limnoria will automatically start if not running since we don't auto restart | 20:07 |
| clarkb | so may also need to up -d after deployment is done | 20:07 |
| fungi | yeah, i planned to do that too if necessary | 20:08 |
| fungi | here we go | 20:09 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 973689: Validate crc32 checksum for S3 image upload https://review.opendev.org/c/zuul/zuul-jobs/+/973689 | 15:25 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Takashi Kajinami: [openstack/project-config] 973947: Fix missing packages for doc build in translation update https://review.opendev.org/c/openstack/project-config/+/973947 | 15:54 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 973363: Revive os_freezer role for OSA https://review.opendev.org/c/openstack/project-config/+/973363 | 15:55 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 973387: Revive os_watcher role for OSA https://review.opendev.org/c/openstack/project-config/+/973387 | 15:55 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 16:48 | |
| - [openstack/project-config] 973963: Remove #opendev from gerritbot https://review.opendev.org/c/openstack/project-config/+/973963 | ||
| - [opendev/system-config] 973964: Switch #opendev logging to Matrix https://review.opendev.org/c/opendev/system-config/+/973964 | ||
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 970451: Update node to 24 for Gerrit builds https://review.opendev.org/c/opendev/system-config/+/970451 | 17:15 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 973970: Reenable Gerrit 3.11 image builds https://review.opendev.org/c/opendev/system-config/+/973970 | 17:15 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [openstack/project-config] 973963: Remove #opendev from gerritbot https://review.opendev.org/c/openstack/project-config/+/973963 | 19:52 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 973964: Switch #opendev logging to Matrix https://review.opendev.org/c/opendev/system-config/+/973964 | 20:09 | |
| @fungicide:matrix.org | we have a logbot! | 20:28 |
| @fungicide:matrix.org | let's move further discussion here now | 20:28 |
| @clarkb:matrix.org | https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2026-01-20.log shows the last two messagse in it | 20:29 |
| @fungicide:matrix.org | yep, just got there myself | 20:29 |
| @clarkb:matrix.org | I'm going to eat lunch now, but I think this is working as expected | 20:29 |
| @fungicide:matrix.org | okay, so we should update the topic in the old irc channel | 20:29 |
| @fungicide:matrix.org | how about something like... | 20:29 |
| @clarkb:matrix.org | ++ | 20:29 |
| @fungicide:matrix.org | "OpenDev discussion has moved to https://matrix.to/#opendev:opendev.org" | 20:30 |
| @fungicide:matrix.org | or do we need something more? | 20:30 |
| @clarkb:matrix.org | That looks fine to me | 20:31 |
| @clarkb:matrix.org | Simple and to the point | 20:31 |
| @fungicide:matrix.org | and done | 20:31 |
| @fungicide:matrix.org | i'm self-approving https://review.opendev.org/972510 now | 20:32 |
| @clarkb:matrix.org | thanks! | 20:36 |
| @clarkb:matrix.org | hrm https://review.opendev.org/c/opendev/system-config/+/973970 is failing on the over pruning of the registry now. In theory if I land the two parent changes that will sort itself out. | 21:04 |
| @clarkb:matrix.org | I have a meeting in about 25 minutes, but I guess I'll approve the two parent changes and see what happens? | 21:05 |
| @clarkb:matrix.org | The first two are approved. We have a bit of time to overrule that if we think there is a problem | 21:06 |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 974005: WIP: Update description for Freezer project https://review.opendev.org/c/openstack/project-config/+/974005 | 21:14 | |
| -@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 974005: WIP: Update description for Freezer project https://review.opendev.org/c/openstack/project-config/+/974005 | 21:20 | |
| -@gerrit:opendev.org- Ivan Anfimov marked as active: [openstack/project-config] 974005: WIP: Update description for Freezer project https://review.opendev.org/c/openstack/project-config/+/974005 | 21:31 | |
| -@gerrit:opendev.org- Ivan Anfimov proposed: | 21:31 | |
| - [openstack/project-config] 974005: Update description for Freezer project https://review.opendev.org/c/openstack/project-config/+/974005 | ||
| - [openstack/project-config] 974005: Update description for Freezer projects https://review.opendev.org/c/openstack/project-config/+/974005 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 972510: Update documentation for Matrix move https://review.opendev.org/c/opendev/system-config/+/972510 | 22:10 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 970173: Add Gerrit 3.12 image builds and testing https://review.opendev.org/c/opendev/system-config/+/970173 | 22:10 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 970180: Test Gerrit 3.11 to 3.12 Upgrades https://review.opendev.org/c/opendev/system-config/+/970180 | 22:10 | |
| @fungicide:matrix.org | hey there Doug Goldstein! | 22:12 |
| @fungicide:matrix.org | are you testing thunderbird as a client then? | 22:12 |
| @cardoe:matrix.org | No. I just did the javascript in firefox | 22:12 |
| @clarkb:matrix.org | fungi: note that 972510 will deploy a new gitea build which we should check afterwards | 22:12 |
| @clarkb:matrix.org | the gitea backend I hit has updated to list the matrix room | 22:18 |
| -@gerrit:opendev.org- Zuul merged on behalf of Damian Fajfer: [zuul/zuul-jobs] 972051: Cleanup upload-pypi role https://review.opendev.org/c/zuul/zuul-jobs/+/972051 | 22:48 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Damian Fajfer: [zuul/zuul-jobs] 972244: Ensure twine is installed before using it https://review.opendev.org/c/zuul/zuul-jobs/+/972244 | 22:52 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!