| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/irc-meetings] 1002385: Remove extra # from matrix meeting locations https://review.opendev.org/c/opendev/irc-meetings/+/1002385 | 10:19 | |
| -@gerrit:opendev.org- Stephen Finucane proposed wip: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/1002494 | 12:22 | |
| -@gerrit:opendev.org- Stephen Finucane marked as active: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/1002494 | 13:21 | |
| -@gerrit:opendev.org- Stephen Finucane proposed: [zuul/zuul-jobs] 1002494: Support doc dependency-groups https://review.opendev.org/c/zuul/zuul-jobs/+/1002494 | 13:58 | |
| @fungicide:matrix.org | heading to lunch, back in an hour | 15:23 |
|---|---|---|
| @mordred:waterwanders.com | Yeah- I think putting that first would be nice | 15:25 |
| @mordred:waterwanders.com | Followup from the other day. I have merged the hashtag addition to the gerrit ui, as well as an update to bulk change edit to add "Restore" since there was already an "Abandon" button. | 15:55 |
| @mordred:waterwanders.com | and apparenty I have submit access now :) | 15:55 |
| @mordred:waterwanders.com | those are both new features so I don't think we can get either in to a point release, but they should both be in 3.15, so by next gerrit upgrade | 16:00 |
| @mordred:waterwanders.com | if there are any other UI annoyances people have, let me know, I've somewhat figured out how to do UI dev :) | 16:01 |
| @clarkb:matrix.org | mordred: thank you for digging into that and ya unless we backport the changes we won't get them for a bit. I think that is ok, but if we do want to backport doing that upstream is best if possible as we've tried to avoid carrying a downstream fork (and have been successful for some time) | 16:03 |
| @mordred:waterwanders.com | yeah. I'm betting upstream wouldn't accept it as an backport. however, if someone REALLY wants the new ui features, there's a chrome plugin that lets you run a local copy of the UI code with a production gerrit instance, I'm happy to point someone in that direction if it would make life helpful | 16:06 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/irc-meetings] 1002668: Add OpenDev Office Hour Meeting https://review.opendev.org/c/opendev/irc-meetings/+/1002668 | 16:46 | |
| @clarkb:matrix.org | infra-root ^ I would like to start that next week. I think there is some flexibility in timing if this doesn't work for everyone who is interested. But I also think we can adjust that as we go if the current proposal doesn't work | 16:46 |
| @clarkb:matrix.org | I don't want to duplicate the existing meeting just a couple hours later, so I'm thinking this can be a less formal more generic office hour time to bring things up including recapping meeting topics for those itnerested. I also figure we can break out to meetpad if necessary to dig into specific topics but the primary location will be matrix | 16:47 |
| @clarkb:matrix.org | Anil Belur: ^ fyi | 16:47 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Anil Shashikumar Belur: | 16:50 | |
| - [opendev/system-config] 1002425: Default the python base images to python3.14 https://review.opendev.org/c/opendev/system-config/+/1002425 | ||
| - [opendev/system-config] 1002420: Build hound on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002420 | ||
| - [opendev/system-config] 1002421: Build haproxy and zk statsd containers on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002421 | ||
| - [opendev/system-config] 1002422: Use asyncio.run() to start the matrix-eavesdrop bot https://review.opendev.org/c/opendev/system-config/+/1002422 | ||
| - [opendev/system-config] 1002423: Build irc and matrix bots on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002423 | ||
| - [opendev/system-config] 1002424: Build gerrit-base on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002424 | ||
| @clarkb:matrix.org | Anil Belur: also I decided to go ahead and reorder things since mordred agreed with the idea that rebuilding the base images is less overall churn and a good idea | 16:50 |
| @clarkb:matrix.org | infra-root ^ I'm around today if we want to start landing those changes. I do haev a family lunch midday but otherwise I can monitor if people want to check off on some of this stuff | 16:51 |
| @clarkb:matrix.org | there is also the prometheus collector update and the gitea container restarts on config file update change | 16:52 |
| @jim:acmegating.com | first 2 approved | 16:56 |
| @jim:acmegating.com | +2 on all | 16:56 |
| @clarkb:matrix.org | thanks I should be able to check on things after those land then continue approving down the stack | 17:02 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/base-jobs] 1002670: Set all three bridge hostkeys in the setup-keys playbook https://review.opendev.org/c/opendev/base-jobs/+/1002670 | 17:18 | |
| @jim:acmegating.com | i went ahead and approved that ^ | 17:45 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/base-jobs] 1002670: Set all three bridge hostkeys in the setup-keys playbook https://review.opendev.org/c/opendev/base-jobs/+/1002670 | 17:45 | |
| @clarkb:matrix.org | corvus: the hourly jobs are the most likely to exercise that first | 17:46 |
| @mordred:waterwanders.com | corvus: if you have a sec: https://review.opendev.org/c/opendev/project-config/+/1002429 - clark already +2'd it | 17:49 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/project-config] 1002429: Re-enqueue vouched changes on trivial rebases https://review.opendev.org/c/opendev/project-config/+/1002429 | 17:52 | |
| @clarkb:matrix.org | hourly jobs are starting now and I'm watching | 18:01 |
| @clarkb:matrix.org | seems to be working and I can see that those tasks ran. Will check for the warnings after the buildset completes | 18:09 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/openstack/buildset/e3dab73f9f784be9aba80d2111aae556 success and if you compare to https://zuul.opendev.org/t/openstack/buildset/e04a1bae89e14b41a5c0b66cd1f768e5 the warnings have gone away | 18:12 |
| @clarkb:matrix.org | corvus: so I think that confirms the issue is openssh adding additional keys after verifying with the supplied key | 18:12 |
| @jim:acmegating.com | Clark: great, thanks! i'm glad we understand that now :) | 18:29 |
| @clarkb:matrix.org | those first two container image update changes are still gating. I'm going to pop out for lunch shortly. I don't think they are very risky so this shoudl eb fine. I'll check back in after lunch and continue to approve things if all is well | 18:48 |
| -@gerrit:opendev.org- Zuul merged on behalf of Anil Shashikumar Belur: | 19:24 | |
| - [opendev/system-config] 1002425: Default the python base images to python3.14 https://review.opendev.org/c/opendev/system-config/+/1002425 | ||
| - [opendev/system-config] 1002420: Build hound on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002420 | ||
| @clarkb:matrix.org | codesearch seems to work and the python version is 3.14. I will approve the next change now | 20:44 |
| -@gerrit:opendev.org- Zuul merged on behalf of Anil Shashikumar Belur: | 22:27 | |
| - [opendev/system-config] 1002421: Build haproxy and zk statsd containers on python3.14 https://review.opendev.org/c/opendev/system-config/+/1002421 | ||
| - [opendev/system-config] 1002422: Use asyncio.run() to start the matrix-eavesdrop bot https://review.opendev.org/c/opendev/system-config/+/1002422 | ||
| @clarkb:matrix.org | the statsd sidecar containers have restarted on zuul-lb, gitea-lb and zk01. Once zk02 and 03 are done I'll wait a couple of minutes then check the data is making its way through to grafana | 22:31 |
| @clarkb:matrix.org | ok all are updated | 22:32 |
| @clarkb:matrix.org | https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&from=now-5m&to=now&timezone=utc https://grafana.opendev.org/d/39b50608de/zuul-load-balancer?orgId=1&from=now-5m&to=now&timezone=utc and https://grafana.opendev.org/d/1f6dfd6769/opendev-load-balancer?orgId=1&from=now-5m&to=now&timezone=utc all have data so I think that worked | 22:37 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/1002423/ is next on the list which updates irc bots. Its late enough in the day now that I'm probably going to defer that one until tomorrow morning. Lokos like there aren't any friday meetings after like 1500 UTC or so | 22:39 |
| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 1001740: Add dib-init-image tool for creating empty disk images https://review.opendev.org/c/openstack/diskimage-builder/+/1001740 | 22:44 | |
| @mordred:waterwanders.com | Clark: apparently we've having some issues with ubuntu-resolute in rax-dfw | 23:06 |
| @clarkb:matrix.org | is that in the graphs I linked? | 23:09 |
| @mordred:waterwanders.com | don't think so? this is from a separate investigation | 23:12 |
| @mordred:waterwanders.com | rax-DFW is serving a stale June 11 ubuntu-resolute image. It happens to have a kernel issue which causes the db on my integration test nodes to get SIGKILLd every 29 seconds or so - which only happens on rax-dfw | 23:12 |
| @mordred:waterwanders.com | and image uploads since then have not been successful | 23:13 |
| @mordred:waterwanders.com | I'm not sure *why* 7.0.0-22-generic breaks and 7.0.0-30-generic doesn't, but feels like one of those layer-cake problems | 23:15 |
| @clarkb:matrix.org | looks like uploads have been failing to ORD and IAD as well | 23:16 |
| @mordred:waterwanders.com | yah. but they started failing more recently, so happen to have a newer kernel | 23:17 |
| @clarkb:matrix.org | The uploads are timing out | 23:21 |
| @clarkb:matrix.org | `2026-08-27 00:41:53,355 ERROR zuul.Launcher: openstack.exceptions.SDKException: Image creation failed: Request to https://dfw.images.api..... timed out` | 23:22 |
| @clarkb:matrix.org | corvus: looks like it timed out in under 10 minutes | 23:23 |
| @clarkb:matrix.org | `2026-08-27 00:33:26,448 INFO zuul.Launcher: [upload: 213187eea3bb4ff18ea5e4850a35da4c] [iba: d33f112b47a648f1912fd236709cdad4] Starting upload` upload starts here then times out at `2026-08-27 00:41:53,355 ERROR zuul.Launcher: [upload: 213187eea3bb4ff18ea5e4850a35da4c] [iba: d33f112b47a648f1912fd236709cdad4] Error in endpoint upload job` Is this possibly related to the new timeouts that were put in place? | 23:23 |
| @mordred:waterwanders.com | that's rax classic too, so it's that just straight put to the glance API iirc? | 23:29 |
| @clarkb:matrix.org | no thats upload to swift then do the undocumented glance task api task to import from swift | 23:33 |
| @mordred:waterwanders.com | OH RIGHT my favorite | 23:42 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!