| @clarkb:matrix.org | ok I have rechecked https://review.opendev.org/c/opendev/system-config/+/945143 and put a hold in place in order to check the current results of the lodgeit python3.14 work | 00:10 |
|---|---|---|
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 1013612: Add support for named build contexts to container jobs https://review.opendev.org/c/zuul/zuul-jobs/+/1013612 | 00:26 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/system-config] 1013975: Collapse gerrit image build https://review.opendev.org/c/opendev/system-config/+/1013975 | 00:48 | |
| -@gerrit:opendev.org- Amy Marrich proposed: [opendev/irc-meetings] 1014003: Change diversity WG start time https://review.opendev.org/c/opendev/irc-meetings/+/1014003 | 00:49 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Amy Marrich: [opendev/irc-meetings] 1014003: Change diversity WG start time https://review.opendev.org/c/opendev/irc-meetings/+/1014003 | 05:05 | |
| @harbott.osism.tech:regio.chat | I didn't find the exact cause yet, but maybe the new ansible is handling the cpu_count check differently? this failure did happen on a node where ansible says there is only one cpu, so the end result - retry - is what we want, but the `'cpu_count' is undefined` part is still irritating to me https://zuul.opendev.org/t/openstack/build/cc08502a210842a1bbe4980326402033 | 06:59 |
| -@gerrit:opendev.org- Stephen Finucane proposed: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/1002494 | 12:25 | |
| @stephenfin:matrix.org | Clark I lost your +2 on that. Would it be possible to re-add it when you've a chance? | 12:51 |
| @stephenfin:matrix.org | err, sorry, that was meant for fungi. I need an initial review from you, Clark, when you've some time | 12:52 |
| @jim:acmegating.com | Jens Harbott: i don't think we've switched any jobs to ansible 13 yet | 14:10 |
| @fungicide:matrix.org | stephenfin: was the rebase because of a merge conflict somewhere? | 14:20 |
| @fungicide:matrix.org | anyway, +2 reapplied | 14:20 |
| @stephenfin:matrix.org | Yes. corvus' change to add `length` filters everywhere to keep Ansible happy | 14:20 |
| @stephenfin:matrix.org | Thanks for the review, as always 🙏 | 14:21 |
| @fungicide:matrix.org | ah, yep i see now, i had to switch to the gerrit webui to filter out the changes in uninvolved files due to the rebase | 14:23 |
| @clarkb:matrix.org | I'll take a look once properly at a computer. fungi: if you're able to review more hashtag:opendev-container-update changes I should be able to monitor any that get approved | 15:29 |
| -@gerrit:opendev.org- Zuul merged on behalf of Antoine Musso: [zuul/zuul-jobs] 1010485: [fetch-zuul-cloner] Add missing copyrights https://review.opendev.org/c/zuul/zuul-jobs/+/1010485 | 15:35 | |
| @fungicide:matrix.org | Clark: still catching up, busy morning (again), but looks like these get deployed by the hourly jobs then after the image updates promote? | 15:35 |
| @fungicide:matrix.org | that's what i was unclear on yesterday after approving the ptgbot change to try it out | 15:35 |
| @fungicide:matrix.org | corvus: does `2026-11-04 12:49:25,309 DEBUG zuul.layout: [e: ab05b77f62394e5d87fc930fba04bd89] Variant <Job neutron-tempest-plugin-openvswitch-2025-1 explicit: None implied: {MatchAny:{ImpliedBranchMatcher:master}} source: openstack/neutron-tempest-plugin/zuul.d/2025_1_jobs.yaml@master#1> matched <Change 0x7cf2f11d65d0 openstack/neutron 1014080,2>` mean that 1014080,2 will run neutron-tempest-plugin-openvswitch-2025-1? becuase it didn't seem to run it. trying to work out a complex job addition case related to an unmerged change in a depends-on | 15:40 |
| @fungicide:matrix.org | maybe the depended-on change with the job modifications needs to merge first? this is further complicated by jobs being defined in one single-branch project used in a non-default branch of another multi-branch project | 15:42 |
| @clarkb:matrix.org | fungi: sort of. Because these image updates happen outside of system config we don't directly trigger their related deployments when they merge. For images that are managed by jobs in the hourly queue they will update hourly. For images not managed by hourly jobs they will update daily. We can also manually pull and restart containers. Matrix/irc bot related stuff happens hourly as the eavesdrop job runs hourly | 15:51 |
| @fungicide:matrix.org | okay, but some periodic job whether it's hourly or daily will pick them up though, point being we don't need to make no-op comment changes in compose files to force them to redeploy or anything | 15:53 |
| @jim:acmegating.com | fungi: re 1014080 -- i uploaded a new ps with debug enabled that should hopefully provide more information on why jobs weren't run | 15:53 |
| @fungicide:matrix.org | ah thanks, i wasn't sure if that would provide information on what wasn't selected | 15:54 |
| @clarkb:matrix.org | > <@fungicide:matrix.org> okay, but some periodic job whether it's hourly or daily will pick them up though, point being we don't need to make no-op comment changes in compose files to force them to redeploy or anything | 15:55 |
| Correct. The daily job run is our backstop here and ensures we don't miss things for longer than ~24 hours | ||
| @fungicide:matrix.org | cool, i didn't know how many of these we wanted to merge at a time between deploys | 15:57 |
| @clarkb:matrix.org | One thing I haven't done yet is check the lodgeit held node after updating the python3.14 change for it | 15:59 |
| @clarkb:matrix.org | But there should be a held node now and it can be checked. The others I'm less worried about as they didn't need quite as much surgery | 15:59 |
| @clarkb:matrix.org | https://104.130.26.29/show/bzUf9NIEeT1WljiL20F7/ <- the held paste node with a test paste I just added to it. | 16:37 |
| @fungicide:matrix.org | lgtm | 16:39 |
| @fungicide:matrix.org | i'm still going through the werkzeug update for it | 16:40 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1014134: Upgrade etherpad to 3.3.14 https://review.opendev.org/c/opendev/system-config/+/1014134 | 16:44 | |
| @clarkb:matrix.org | I suspect ^ fixes the disconnect issue that was observed during the PTG | 16:44 |
| @clarkb:matrix.org | details in the commit message | 16:44 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 840972: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 16:45 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritlib] 1013079: Update testing to modern python and gerrit versions https://review.opendev.org/c/opendev/gerritlib/+/1013079 | 16:51 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/lodgeit] 982311: Uncap Werkzeug for python 3.14 compatibility https://review.opendev.org/c/opendev/lodgeit/+/982311 | 16:59 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritbot] 1013080: Update Gerritbot testing and docker image to python 3.14 https://review.opendev.org/c/opendev/gerritbot/+/1013080 | 17:00 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/statusbot] 1013088: Update statusbot container image to python3.14 https://review.opendev.org/c/opendev/statusbot/+/1013088 | 17:04 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/lodgeit] 982307: Update lodgeit to python3.14 https://review.opendev.org/c/opendev/lodgeit/+/982307 | 17:07 | |
| @clarkb:matrix.org | fungi: lodgeit does not run hourly so doing that one manually probably makes sense. I can do that once the image promotes | 17:07 |
| @clarkb:matrix.org | Then I'll test the held etherpad that should be held shortly and can approve the etherpad upgrade ifthat looks good | 17:08 |
| @fungicide:matrix.org | cool | 17:08 |
| @fungicide:matrix.org | lodgeit promote completed btw | 17:10 |
| @clarkb:matrix.org | cool I'll get on that momentarily | 17:11 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/grafyaml] 1013089: Update to python3.14 testing and container image https://review.opendev.org/c/opendev/grafyaml/+/1013089 | 17:13 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Stephen Finucane: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/1002494 | 17:14 | |
| @clarkb:matrix.org | lodgeit has been restarted on the new image and pastes still work for me: https://paste.opendev.org/show/bXkoZlysfNBBt3fO6UYM/ | 17:15 |
| @clarkb:matrix.org | fungi: I'm not worried about grafyaml because we test that pretty extensively in CI it should be fine to run daily per usual | 17:15 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/jeepyb] 1013085: Update gerritlib integration testing to match gerritlib https://review.opendev.org/c/opendev/jeepyb/+/1013085 | 17:26 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1014140: Rebuild base container images https://review.opendev.org/c/opendev/system-config/+/1014140 | 17:30 | |
| @clarkb:matrix.org | held etherpad is at 23.253.164.81 and I'm testing in the clarkb-test pad now | 17:38 |
| @clarkb:matrix.org | and it lgtm. I'll approve the etherpad update shortly but will give it a few minutes if anyone else wants to check the held node | 17:41 |
| @fungicide:matrix.org | held etherpad node lgtm too, approving | 17:54 |
| @clarkb:matrix.org | Thanks! | 17:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritlib] 1013087: Drop older gerrit testing jobs https://review.opendev.org/c/opendev/gerritlib/+/1013087 | 18:12 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1014134: Upgrade etherpad to 3.3.14 https://review.opendev.org/c/opendev/system-config/+/1014134 | 18:32 | |
| @fungicide:matrix.org | corvus: so digging deeper into that debug output, https://review.opendev.org/c/openstack/neutron/+/1014080 is on branch `unmaintained/2025.1` and it looks like the reason neutron-tempest-plugin-openvswitch-2025-1 wasn't selected is that its grandparent (neutron-tempest-plugin-base-nested-switch) didn't have any matching variants, however... | 18:39 |
| @fungicide:matrix.org | even though zuul reported `Variant <Job neutron-tempest-plugin-base-nested-switch explicit: {MatchAny:{BranchMatcher:^(unmaintained|stable/(victoria|wallaby|xena|yoga|zed|2023|2024)).$}} implied: None source: openstack/neutron-tempest-plugin/zuul.d/base-nested-switch.yaml@master#32> did not match` that was supposed to be a negated match, and zuul lists it as negated in the job page too? https://zuul.opendev.org/t/openstack/job/neutron-tempest-plugin-base-nested-switch | 18:40 |
| @fungicide:matrix.org | oh, i bet that was supposed to be `(unmaintained|stable)/...` | 18:41 |
| @fungicide:matrix.org | yeah, i'm almost positive that was it | 18:44 |
| @clarkb:matrix.org | etherpad upgraded and reports it is healthy. https://etherpad.opendev.org/p/opendev-weekly-meeting-agenda still loads for me so I think that deployed properly | 18:54 |
| @jim:acmegating.com | fungi: ++ lmk if you run into any more issues | 18:56 |
| @fungicide:matrix.org | corvus: of course, and thanks! | 18:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1014140: Rebuild base container images https://review.opendev.org/c/opendev/system-config/+/1014140 | 19:38 | |
| @harbott.osism.tech:regio.chat | I'm almost eod, but it seems like we have a very high number of retry results right now | 20:36 |
| @harbott.osism.tech:regio.chat | mirror failures https://zuul.opendev.org/t/openstack/build/1d6851634ca8407aa2acddb54f1e0f03 and possibly broken git cache? https://zuul.opendev.org/t/openstack/build/74bd3f0dbae94383b4b6ae70a0b672e1/console | 20:37 |
| @fungicide:matrix.org | and in different regions | 20:40 |
| @fungicide:matrix.org | (but both raxflex) | 20:40 |
| @clarkb:matrix.org | the broken git cache we theorized is the corrupted image upload problem returning again | 20:41 |
| @clarkb:matrix.org | and the solution there is to disable the cloud and work with the cloud to figure out why that is happening cc Dan With | 20:41 |
| @fungicide:matrix.org | still seeing some retries from "ansible setup timeout" too | 20:42 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/openstack/build/c9542a4ba081477ea5525afc75e6ea09 | 20:42 |
| @clarkb:matrix.org | whcih I theorized had to do with network setup that eventually fixes itself | 20:44 |
| @clarkb:matrix.org | the timeout for that playbook is 60 seoncds but it looks like we need about 120 or so to reliably get a connection working after the initial connection happens | 20:44 |
| @clarkb:matrix.org | its super weird and I was able to replicate via connections from my home network so pretty sure it is a cloud side problem | 20:45 |
| @clarkb:matrix.org | wow there is already an etherpad 3.3.15. But this appears to have been an issue with github release packaging going above file size limits. We build it ourselves so aren't affected. I think we can ignore it for now | 20:50 |
| @clarkb:matrix.org | dfw3 is where we have the network connection issues? and sjc3 is where we think we get corrupted image uploads? | 21:20 |
| @clarkb:matrix.org | I'm trying to put a more concrete message together and want to make sure I get my regions straight | 21:21 |
| @clarkb:matrix.org | ya I think those are both correct looking at logs and notes | 21:21 |
| @fungicide:matrix.org | i believe so on both counts, yes | 21:51 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!