| clarkb | meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Sep 2 19:00:33 2025 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/AYKNDGLH46IV3N5NI2BBSVMYMI6W4MQP/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | There is a matrix.org outage right now which I think may be impacting the irc bridge to oftc as well as any matrix accounts hosted by matrix.org (like mine) | 19:01 |
| clarkb | https://status.matrix.org/ is tracking the issue if you want to know when things return to normal | 19:01 |
| clarkb | Also fungi is out today so this meeting may just be me running through the agenda | 19:01 |
| clarkb | but feel free to jump in on any topics if there is anything to share and you are following along | 19:02 |
| clarkb | #topic Gerrit 3.11 Upgrade Planning | 19:02 |
| clarkb | I don't have any updates on this item. I've been distracted by other things lately | 19:02 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/957555 Gerrit image updates for bugfix releases | 19:03 |
| clarkb | this change could still use reviews though and I'll and it amongst the other container update changes when its convenient to restart gerrit | 19:04 |
| clarkb | #topic Upgrading old servers | 19:04 |
| clarkb | As mentioned last week fungi managed to update most of the openafs cluster to jammy and I expect when he gets back that we will continue that effort all the way to noble | 19:04 |
| fungi | i'll pick the afs/kerberos upgrades back up toward the end of this week once i'm home | 19:05 |
| clarkb | this is major progress and getting off of old ubuntu releases and onto more modern stuff | 19:05 |
| clarkb | fungi: thanks! | 19:05 |
| clarkb | then the major remaining nodes on the list are backup servers and the graphite server | 19:05 |
| clarkb | Help continues to be very much appreciated if anyone else is able to dig into this | 19:06 |
| clarkb | #topic Matrix for OpenDev comms | 19:06 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing | 19:06 |
| corvus_ | timely | 19:07 |
| clarkb | followup reviews on the spec are where we're sort of treading water | 19:07 |
| corvus_ | the bridge appears to be broken right now | 19:07 |
| clarkb | and yes the matrix.org outage may provide new thoughts/ideas on this spec if you want to wait and see how that gets resolved | 19:07 |
| clarkb | corvus_: yes I think anything hosted by matrix.org (including my account and the oftc irc bridge) is not working with matrix right now | 19:07 |
| clarkb | https://status.matrix.org/ is tracking the outage | 19:07 |
| clarkb | reviews still very much welcome and I understand if we want to wait and see some further triage/resolution on the current issue before doing so | 19:08 |
| fungi | yeah, my weechat is logging repeated connection refusal errors from the matrix.org servers | 19:08 |
| corvus_ | i only noticed because of this meeting | 19:09 |
| clarkb | yes my browser was getting 429s from cloudflare. I suspect they've configured limits quite low to ease traffic on the backend while they fix it | 19:09 |
| clarkb | #topic Pre PTG Planning | 19:10 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document | 19:10 |
| clarkb | Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 | 19:10 |
| clarkb | This will replace our team meeting on October 7 | 19:10 |
| clarkb | please add discussion topics to the agenda on that etherpad | 19:10 |
| clarkb | and I'll see you there in just over a month | 19:10 |
| clarkb | #topic Loss of upstream Debian bullseye-backports mirror | 19:11 |
| clarkb | Zuul-jobs will no longer enable debian backports by default on September 9 | 19:11 |
| clarkb | #link https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/thread/NZ54HYFHIYW3OILYYIQ72L7WAVNSODMR/ | 19:11 |
| clarkb | Once zuul-jobs' default is updated then we'll be able to delete the debian bullseye backports repo from our mirror and drop our workaround | 19:11 |
| clarkb | just waiting for sufficient time to pass since this was announced on the zuul announce list | 19:12 |
| clarkb | #topic Etherpad 2.5.0 Upgrade | 19:13 |
| clarkb | #link https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md | 19:13 |
| corvus_ | regrading matrix: btw the opendev ems instance is working (gerritbot msgs are going through) and eavesdrop is logging | 19:13 |
| clarkb | ack | 19:13 |
| clarkb | etherpad claims the 2.5.0 release fixes our problems. I still think the root page's css is weird but the js errors related to 2.4.2 did go away | 19:14 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/956593/ | 19:14 |
| clarkb | 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. | 19:14 |
| clarkb | if you want to test you can punch that ip into /etc/hosts and check out the root page locally as well as the clarkb-test pad (or any other pad you create) | 19:14 |
| clarkb | again I don't think this is urgent. Mostly looking for feedback on whether we think this is workable so that we don't fall behind or continue to pester them to fix things better | 19:15 |
| clarkb | #topic Moving OpenDev's python-base/python-builder/uwsig-base Images to Quay | 19:16 |
| clarkb | last week corvus_ suggested that the way to not wait forever on merging this change is to prep changes to update all the child images as reminders that those need updating at some point then proceeding with moving the base image publication location | 19:16 |
| clarkb | I did propose those changes and it caught a problem! | 19:16 |
| clarkb | Turns out to use speculative images via the buildset registry when building with docker we always need to build with a custom buildx builder | 19:17 |
| clarkb | earlier this year I had changed image building to use podman by default in system-config. I don't remember the details btu I suspect I ran into this problem and just didn't track it down fully and this was the out. The problem with thati s multiarch builds | 19:17 |
| clarkb | its actually probably more ideal for us to keep using docker to build images and switch to the custom buildx builder for all image builds so that single arch and multiarch builds use the same toolchain (podman doesn't support multiarch in our jobs yet but the underlying tool does) | 19:18 |
| clarkb | #link https://review.opendev.org/c/zuul/zuul-jobs/+/958783 Always build docker images with custom buildx builder | 19:18 |
| clarkb | this change updates zuul-jobs to do that for everyone as it plays nice with speculative image builds | 19:18 |
| clarkb | so I think the rough plan here for moving base images to quay is land that zuul-jobs change, then move base images to quay, then update the child images to both build with docker and pull base images from quay | 19:19 |
| clarkb | that zuul-jobs chnage has a child followup change that adds testing to zuul-jobs to cover all of this too so we should be good moving foward and any regressions should be caught early | 19:20 |
| clarkb | #topic Adding Debian Trixie Base Python Container Images | 19:20 |
| clarkb | Then once base images move to quay we can also add trixie based python container images | 19:20 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/958480 | 19:20 |
| clarkb | with plans to worry about python3.13 after trixie is in place | 19:21 |
| clarkb | just to keep the total number of images we're juggling to a reasonable number | 19:21 |
| clarkb | #topic Dropping Ubuntu Bionic Test Nodes | 19:22 |
| clarkb | After last week's meeting I think I convinced myself we don't need to do any major announcements for Bionic cleanups yet. Mostly because the release is long EOL at this point | 19:22 |
| clarkb | #link https://review.opendev.org/q/hashtag:%22drop-bionic%22+status:open | 19:22 |
| clarkb | I did write a few changes to continue to remove opendev's dependence on bionic under that hashtag. I Think all of those changes are likely quick reviews and easy approvals | 19:23 |
| clarkb | and dropping releases like bionic reduces the total storage used in openafs which should make things like upgrading openafs servers easier | 19:24 |
| clarkb | (though I'm not sure we'll get this done before fungi completes the upgrades. I'm just trying to justify the effort generally) | 19:24 |
| clarkb | #topic Temporary Shutdown of raxflex sjc3 for provider maintenance window | 19:25 |
| clarkb | last week rackspcae notified us via email that the cinder volume backing the rax flex sjc3 mirror would undergo maintenance tomorrow at 10:30am to 12:30pm cnetral time | 19:26 |
| clarkb | #link https://review.opendev.org/c/opendev/zuul-providers/+/959200 | 19:26 |
| clarkb | this change disables this region in zuul launcher so that I can safely shutdown the mirror while they do that work. My plan is to approve that change after lunch today and then manually shutdown the mirror before EOD | 19:26 |
| clarkb | that should be plenty of time for running jobs to complete | 19:26 |
| clarkb | then tomorrow after the maintenance window completes I can start the mirror back up again and revert 959200 | 19:27 |
| clarkb | #topic Fixing Zuul's Trixie Image Builds | 19:28 |
| clarkb | This item wasn't on the agenda (my bad) but it was pointed out that our Trixie images are still actually debian testing | 19:28 |
| clarkb | #link https://review.opendev.org/c/opendev/zuul-providers/+/958561 Build actual Trixie now that it is released | 19:28 |
| clarkb | 958561 will fix that but depends on a DIB update | 19:29 |
| clarkb | did anyone else want to review the DIB update? I'm thinking I may approve that one today with mnasiadka's review as the sole +2 in order to not let this problem fester for too long | 19:29 |
| clarkb | #topic Open Discussion | 19:29 |
| clarkb | And with that we have reached the end of the agenda. Anything else? | 19:30 |
| clarkb | I know I kinda speedran through that but with fungi out, corvus impacted by matrix bridging issues, and frickler and tonyb not typically attending I figured I should just get through it | 19:31 |
| clarkb | I'll leave the floor open until 19:35 UTC then call it a meeting if nothing comes up | 19:31 |
| clarkb | as always feel free to continue any discssion on the mailing list or in #opendev | 19:31 |
| fungi | finishing the afs/kerberos upgrades shouldn't take long, btw, it's fairly mechanical now and hopefully i can have the rw volume migration back to the noble afs01.dfw going by the weekend | 19:32 |
| corvus_ | fyi there's a zuul-scheduler memory leak, but i think i have a fix | 19:32 |
| corvus_ | we'll probably need to restart the schedulers tomorrow whether or not it lands | 19:33 |
| clarkb | corvus_: oh right that came up on Friday and over the weekend | 19:33 |
| fungi | that was the root cause for the connection issues last week? | 19:33 |
| corvus_ | #link https://review.opendev.org/959228 fix zuul-scheduler memory leak | 19:33 |
| corvus_ | yeah i think so | 19:33 |
| corvus_ | i mean, this is all well-informed supposition, not hard proof | 19:33 |
| tonyb | I've been following along just didn't have thoughts | 19:34 |
| corvus_ | but i think we're at "fix obvious things first" and if stuff is still broken, dig deeper. | 19:34 |
| clarkb | corvus_: sounds good | 19:34 |
| clarkb | I'll make a note now for tomorrow to restart schedulers | 19:34 |
| clarkb | and we're at the time I noted we'd end. Thank you everyone! | 19:35 |
| clarkb | We should be back here at the same time and location next week | 19:35 |
| clarkb | see you then | 19:35 |
| corvus_ | thanks clarkb ! | 19:35 |
| clarkb | #endmeeting | 19:35 |
| opendevmeet | Meeting ended Tue Sep 2 19:35:44 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:35 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-09-02-19.00.html | 19:35 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-09-02-19.00.txt | 19:35 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-09-02-19.00.log.html | 19:35 |
| clarkb | corvus_: the zuul fix lgtm. I suppose if simon reviews it before my day starts tomorrow that may be ready for manual restarts tomorrow? | 19:36 |
| corvus_ | clarkb: yep, though i feel that for this particular change, simon would probably be fine if we just merged it and got a jump on validating it as a fix. :) | 19:40 |
| clarkb | corvus_: that also works for me if you want to approve it now | 19:48 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!