Wednesday, 2026-11-04

@clarkb:matrix.orgok I have rechecked https://review.opendev.org/c/opendev/system-config/+/945143 and put a hold in place in order to check the current results of the lodgeit python3.14 work00:10
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 1013612: Add support for named build contexts to container jobs https://review.opendev.org/c/zuul/zuul-jobs/+/101361200:26
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 1013975: Collapse gerrit image build https://review.opendev.org/c/opendev/system-config/+/101397500:48
-@gerrit:opendev.org- Amy Marrich proposed: [opendev/irc-meetings] 1014003: Change diversity WG start time https://review.opendev.org/c/opendev/irc-meetings/+/101400300:49
-@gerrit:opendev.org- Zuul merged on behalf of Amy Marrich: [opendev/irc-meetings] 1014003: Change diversity WG start time https://review.opendev.org/c/opendev/irc-meetings/+/101400305:05
@harbott.osism.tech:regio.chatI didn't find the exact cause yet, but maybe the new ansible is handling the cpu_count check differently? this failure did happen on a node where ansible says there is only one cpu, so the end result - retry - is what we want, but the `'cpu_count' is undefined` part is still irritating to me https://zuul.opendev.org/t/openstack/build/cc08502a210842a1bbe498032640203306:59
-@gerrit:opendev.org- Stephen Finucane proposed: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/100249412:25
@stephenfin:matrix.orgClark I lost your +2 on that. Would it be possible to re-add it when you've a chance?12:51
@stephenfin:matrix.orgerr, sorry, that was meant for fungi. I need an initial review from you, Clark, when you've some time12:52
@jim:acmegating.comJens Harbott: i don't think we've switched any jobs to ansible 13 yet14:10
@fungicide:matrix.orgstephenfin: was the rebase because of a merge conflict somewhere?14:20
@fungicide:matrix.organyway, +2 reapplied14:20
@stephenfin:matrix.orgYes. corvus' change to add `length` filters everywhere to keep Ansible happy14:20
@stephenfin:matrix.orgThanks for the review, as always 🙏14:21
@fungicide:matrix.orgah, yep i see now, i had to switch to the gerrit webui to filter out the changes in uninvolved files due to the rebase14:23
@clarkb:matrix.orgI'll take a look once properly at a computer. fungi: if you're able to review more hashtag:opendev-container-update changes I should be able to monitor any that get approved15:29
-@gerrit:opendev.org- Zuul merged on behalf of Antoine Musso: [zuul/zuul-jobs] 1010485: [fetch-zuul-cloner] Add missing copyrights https://review.opendev.org/c/zuul/zuul-jobs/+/101048515:35
@fungicide:matrix.orgClark: still catching up, busy morning (again), but looks like these get deployed by the hourly jobs then after the image updates promote?15:35
@fungicide:matrix.orgthat's what i was unclear on yesterday after approving the ptgbot change to try it out15:35
@fungicide:matrix.orgcorvus: does `2026-11-04 12:49:25,309 DEBUG zuul.layout: [e: ab05b77f62394e5d87fc930fba04bd89] Variant <Job neutron-tempest-plugin-openvswitch-2025-1 explicit: None implied: {MatchAny:{ImpliedBranchMatcher:master}} source: openstack/neutron-tempest-plugin/zuul.d/2025_1_jobs.yaml@master#1> matched <Change 0x7cf2f11d65d0 openstack/neutron 1014080,2>` mean that 1014080,2 will run neutron-tempest-plugin-openvswitch-2025-1? becuase it didn't seem to run it. trying to work out a complex job addition case related to an unmerged change in a depends-on15:40
@fungicide:matrix.orgmaybe the depended-on change with the job modifications needs to merge first? this is further complicated by jobs being defined in one single-branch project used in a non-default branch of another multi-branch project15:42
@clarkb:matrix.orgfungi: sort of. Because these image updates happen outside of system config we don't directly trigger their related deployments when they merge. For images that are managed by jobs in the hourly queue they will update hourly. For images not managed by hourly jobs they will update daily. We can also manually pull and restart containers. Matrix/irc bot related stuff happens hourly as the eavesdrop job runs hourly15:51
@fungicide:matrix.orgokay, but some periodic job whether it's hourly or daily will pick them up though, point being we don't need to make no-op comment changes in compose files to force them to redeploy or anything15:53
@jim:acmegating.comfungi: re 1014080 -- i uploaded a new ps with debug enabled that should hopefully provide more information on why jobs weren't run15:53
@fungicide:matrix.orgah thanks, i wasn't sure if that would provide information on what wasn't selected15:54
@clarkb:matrix.org> <@fungicide:matrix.org> okay, but some periodic job whether it's hourly or daily will pick them up though, point being we don't need to make no-op comment changes in compose files to force them to redeploy or anything15:55
Correct. The daily job run is our backstop here and ensures we don't miss things for longer than ~24 hours
@fungicide:matrix.orgcool, i didn't know how many of these we wanted to merge at a time between deploys15:57
@clarkb:matrix.orgOne thing I haven't done yet is check the lodgeit held node after updating the python3.14 change for it15:59
@clarkb:matrix.orgBut there should be a held node now and it can be checked. The others I'm less worried about as they didn't need quite as much surgery15:59
@clarkb:matrix.orghttps://104.130.26.29/show/bzUf9NIEeT1WljiL20F7/ <- the held paste node with a test paste I just added to it.16:37
@fungicide:matrix.orglgtm16:39
@fungicide:matrix.orgi'm still going through the werkzeug update for it16:40
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1014134: Upgrade etherpad to 3.3.14 https://review.opendev.org/c/opendev/system-config/+/101413416:44
@clarkb:matrix.orgI suspect ^ fixes the disconnect issue that was observed during the PTG16:44
@clarkb:matrix.orgdetails in the commit message16:44
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 840972: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/84097216:45
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritlib] 1013079: Update testing to modern python and gerrit versions https://review.opendev.org/c/opendev/gerritlib/+/101307916:51
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/lodgeit] 982311: Uncap Werkzeug for python 3.14 compatibility https://review.opendev.org/c/opendev/lodgeit/+/98231116:59
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritbot] 1013080: Update Gerritbot testing and docker image to python 3.14 https://review.opendev.org/c/opendev/gerritbot/+/101308017:00
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/statusbot] 1013088: Update statusbot container image to python3.14 https://review.opendev.org/c/opendev/statusbot/+/101308817:04
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/lodgeit] 982307: Update lodgeit to python3.14 https://review.opendev.org/c/opendev/lodgeit/+/98230717:07
@clarkb:matrix.orgfungi: lodgeit does not run hourly so doing that one manually probably makes sense. I can do that once the image promotes17:07
@clarkb:matrix.orgThen I'll test the held etherpad that should be held shortly and can approve the etherpad upgrade ifthat looks good17:08
@fungicide:matrix.orgcool17:08
@fungicide:matrix.orglodgeit promote completed btw17:10
@clarkb:matrix.orgcool I'll get on that momentarily17:11
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/grafyaml] 1013089: Update to python3.14 testing and container image https://review.opendev.org/c/opendev/grafyaml/+/101308917:13
-@gerrit:opendev.org- Zuul merged on behalf of Stephen Finucane: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/100249417:14
@clarkb:matrix.orglodgeit has been restarted on the new image and pastes still work for me: https://paste.opendev.org/show/bXkoZlysfNBBt3fO6UYM/17:15
@clarkb:matrix.orgfungi: I'm not worried about grafyaml because we test that pretty extensively in CI it should be fine to run daily per usual17:15
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/jeepyb] 1013085: Update gerritlib integration testing to match gerritlib https://review.opendev.org/c/opendev/jeepyb/+/101308517:26
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1014140: Rebuild base container images https://review.opendev.org/c/opendev/system-config/+/101414017:30
@clarkb:matrix.orgheld etherpad is at 23.253.164.81 and I'm testing in the clarkb-test pad now17:38
@clarkb:matrix.organd it lgtm. I'll approve the etherpad update shortly but will give it a few minutes if anyone else wants to check the held node17:41
@fungicide:matrix.orgheld etherpad node lgtm too, approving17:54
@clarkb:matrix.orgThanks!17:59
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritlib] 1013087: Drop older gerrit testing jobs https://review.opendev.org/c/opendev/gerritlib/+/101308718:12
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1014134: Upgrade etherpad to 3.3.14 https://review.opendev.org/c/opendev/system-config/+/101413418:32
@fungicide:matrix.orgcorvus: so digging deeper into that debug output, https://review.opendev.org/c/openstack/neutron/+/1014080 is on branch `unmaintained/2025.1` and it looks like the reason neutron-tempest-plugin-openvswitch-2025-1 wasn't selected is that its grandparent (neutron-tempest-plugin-base-nested-switch) didn't have any matching variants, however...18:39
@fungicide:matrix.orgeven though zuul reported `Variant <Job neutron-tempest-plugin-base-nested-switch explicit: {MatchAny:{BranchMatcher:^(unmaintained|stable/(victoria|wallaby|xena|yoga|zed|2023|2024)).$}} implied: None source: openstack/neutron-tempest-plugin/zuul.d/base-nested-switch.yaml@master#32> did not match` that was supposed to be a negated match, and zuul lists it as negated in the job page too? https://zuul.opendev.org/t/openstack/job/neutron-tempest-plugin-base-nested-switch18:40
@fungicide:matrix.orgoh, i bet that was supposed to be `(unmaintained|stable)/...`18:41
@fungicide:matrix.orgyeah, i'm almost positive that was it18:44
@clarkb:matrix.orgetherpad upgraded and reports it is healthy. https://etherpad.opendev.org/p/opendev-weekly-meeting-agenda still loads for me so I think that deployed properly18:54
@jim:acmegating.comfungi: ++ lmk if you run into any more issues18:56
@fungicide:matrix.orgcorvus: of course, and thanks!18:59
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1014140: Rebuild base container images https://review.opendev.org/c/opendev/system-config/+/101414019:38
@harbott.osism.tech:regio.chatI'm almost eod, but it seems like we have a very high number of retry results right now20:36
@harbott.osism.tech:regio.chatmirror failures https://zuul.opendev.org/t/openstack/build/1d6851634ca8407aa2acddb54f1e0f03 and possibly broken git cache? https://zuul.opendev.org/t/openstack/build/74bd3f0dbae94383b4b6ae70a0b672e1/console20:37
@fungicide:matrix.organd in different regions20:40
@fungicide:matrix.org(but both raxflex)20:40
@clarkb:matrix.orgthe broken git cache we theorized is the corrupted image upload problem returning again20:41
@clarkb:matrix.organd the solution there is to disable the cloud and work with the cloud to figure out why that is happening cc Dan With 20:41
@fungicide:matrix.orgstill seeing some retries from "ansible setup timeout" too20:42
@fungicide:matrix.orghttps://zuul.opendev.org/t/openstack/build/c9542a4ba081477ea5525afc75e6ea0920:42
@clarkb:matrix.orgwhcih I theorized had to do with network setup that eventually fixes itself20:44
@clarkb:matrix.orgthe timeout for that playbook is 60 seoncds but it looks like we need about 120 or so to reliably get a connection working after the initial connection happens20:44
@clarkb:matrix.orgits super weird and I was able to replicate via connections from my home network so pretty sure it is a cloud side problem20:45
@clarkb:matrix.orgwow there is already an etherpad 3.3.15. But this appears to have been an issue with github release packaging going above file size limits. We build it ourselves so aren't affected. I think we can ignore it for now20:50
@clarkb:matrix.orgdfw3 is where we have the network connection issues? and sjc3 is where we think we get corrupted image uploads?21:20
@clarkb:matrix.orgI'm trying to put a more concrete message together and want to make sure I get my regions straight21:21
@clarkb:matrix.orgya I think those are both correct looking at logs and notes21:21
@fungicide:matrix.orgi believe so on both counts, yes21:51

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!