19:00:17 <clarkb> #startmeeting infra
19:00:17 <opendevmeet> Meeting started Tue Aug 26 19:00:17 2025 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:17 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:17 <opendevmeet> The meeting name has been set to 'infra'
19:00:25 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/NNABJTQSALCFXBGPP4SNCGGN4TGLYB6E/ Our Agenda
19:00:30 <clarkb> #topic Announcements
19:00:40 <clarkb> I didn't have anything to announce. Did anyone else?
19:01:57 <clarkb> #topic Gerrit 3.11 Upgrade Planning
19:02:15 <clarkb> This is the perpetually stalled out task... Its not urgent and other things keep fighting for my attention
19:03:02 <clarkb> and unfortunately there are nwe gerrit bug fix releases we need to update to again. I think I'm aiming to get that done when fungi gets back so not a rush
19:03:18 <clarkb> but if you have time to review the gerrit bugfix update change that is always appreciated
19:03:38 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/957555
19:03:47 <clarkb> I'm happy to approve that when I've got time to pull the image and restart gerrit
19:03:59 <clarkb> #topic Upgrading old servers
19:04:48 <clarkb> fungi picked up the kerberos and openafs server upgrades
19:05:08 <clarkb> all but one server is running jammy now. Plan is to upgrade the last server to jammy then pretty quickly start the jammy -> noble upgrades
19:05:33 <clarkb> thank you fungi for getting that started. Other than the slowness of waiting for afs RW volume moves I think this has gone reasonably well
19:05:59 <corvus> ++
19:06:04 <clarkb> One thing to note is that the RW volume moves do seem to consume additional disk space on the fileservers. We should keep this in mind as we add potential new content to afs and also look at deleting things (its good to delete stuff if we don't need ti anymore)
19:06:39 <clarkb> then the other remaining servers on the todo list are the graphite server and the backup servers
19:07:48 <clarkb> with graphite we'll want to preserve/copy/move the existing data volume so that we don't lose data. For the backup servers I think what we've done in the past is boot new servers and attach new volumes to the new servers. Do new backups there. Then we attach the old backup volumes to the new servers as an archive of the old backups allowing us to delete the old servers
19:08:05 <clarkb> I think that approach with the backup servers continues to make sense so is probably the one we'll use here
19:08:16 <clarkb> any questions/concerns/comments with server upgrades/replacements?
19:09:24 <clarkb> #topic Matrix for OpenDev comms
19:09:31 <clarkb> #link https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing
19:09:39 <clarkb> looks like we have some community feedback in favor of this change
19:09:47 <clarkb> still needs additional reviews from infra-root
19:09:57 <clarkb> if you have time to look at the latest patchset your input is appreciated
19:10:09 <clarkb> but I think we can keep most of the discussion of the change itself in gerrit code review. See you there
19:11:03 <clarkb> #topic Pre PTG Planning
19:11:19 <clarkb> #link https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document
19:11:26 <clarkb> Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700
19:11:35 <clarkb> the Tuesday block there will replace our weekly team meeting
19:11:53 <clarkb> please feel free to add agenda items to that etherpad
19:12:07 <clarkb> I'm sure I'm not the only one with ideas
19:12:47 <clarkb> #topic Service Coordinator Election Planning
19:13:34 <clarkb> This nomination period ended and as far as I can tell I was the only nominee.
19:14:05 <clarkb> This means I'm it by default. I wanted to call that out here so that anyone can point out missed nominations or object or otherwise provide input on the election
19:14:23 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/WNUYDT47NMYC3SC5QA44OG4PWK5ENQEF/ clarkbs nomination
19:14:40 <clarkb> but if there is no input then I think we can consider this official and drop this item from next week's agenda
19:15:39 <clarkb> #topic Loss of upstream Debian bullseye-backports mirror
19:15:52 <clarkb> fungi is out this week and I'm not sure where he got to on this effort
19:16:12 <clarkb> I think the idea was to notify zuul users that zuul-jobs would update to match upstream
19:16:24 <clarkb> I don't see that notification to the zuul list
19:16:45 <clarkb> corvus if you get a chance maybe you can double check it isn't queued up for zuul-announce? and if not then I can probably go ahead and draft something
19:16:51 <clarkb> that way we can get the clock ticking on the cleanup
19:17:58 <clarkb> We claened up xenial and arm64 bionic mirror content as part of the openafs server volume moves (we needed more disk space to do the moves) and I think cleaning up debian bullseye backports is another good cleanup we can add to that pile
19:18:25 <clarkb> #topic Etherpad 2.5.0 Upgrade
19:18:29 <corvus> Will do
19:18:45 <clarkb> After filing an issue for the etherpad 2.4.2 release breaking the no-skin skin (that we use) upstream "fixed" the no-skin skin and made this 2.5.0 release
19:18:51 <clarkb> #link https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md
19:18:57 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/956593/
19:19:26 <clarkb> Using that change and its child I held a node with IP address 104.130.127.119 for testing. The / page still has slightly odd css rendering but it is better than before
19:19:37 <clarkb> and the error that occurred in pads when additional clients join seems to have gone away
19:20:00 <clarkb> if you want to see for yourself you can add that IP address into your local /etc/hosts for etherpad.opendev.org. I created a clarkb-test pad too fi you want to look at some content
19:20:18 <clarkb> all this to say its not perfect but is better than before and we might consider how bad we think that css is and/or maybe patch the css ourselves
19:20:40 <clarkb> I think this is probably workable if we needed to upgrade. But as before I don't think anything in the cahngelog makes this urgent for us. Mostly just a desire to not fall too far behind
19:21:12 <clarkb> #topic Moving OpenDev's python-base/python-builder/uwsig-base Images to Quay
19:21:19 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/957277
19:21:42 <clarkb> all of our python based containers are now running on noble so we can switch things to quay without losing speculative image testing
19:22:14 <clarkb> However, this change updates the parent images then we should plan to rebuild and redeploy all of our services that depend on them afterawrds. Which probably makes this a task best for when we don't have people on vacation
19:22:31 <clarkb> the change itself is straightforward but ensuring we've updating all the things to catch up is less so
19:22:46 <clarkb> but if we feel the risk is worth it here let me know and I can work through that list after this change lands
19:23:34 <clarkb> #topic Adding Debian Trixie Base Python Container Images
19:23:53 <clarkb> this is related to the previous change in that it adds more base python images. I think we can land these two changes in either order then update the other to accomodate
19:23:54 <corvus> clarkb: sorry
19:24:09 <corvus> clarkb: why do we need to deploy stuff right after we switch?
19:24:29 <corvus> like, why can't we just switch and when redeploys happen they happen?
19:24:37 <clarkb> corvus: not necessarily immediately after but I don't want to forget that the upstream image source has changed and then end up building images with the old source down the line
19:24:56 <clarkb> corvus: the main concern is that we'll forget to update the images and accounting problems will not be caught until much later
19:25:09 <clarkb> corvus: since we have to modfiy the actual Dockerfile for each downstream consumer to fetch from quay.io
19:25:52 <clarkb> so it isn't strictly necessary but I want to avoid forgetting and having problems later
19:25:54 <corvus> we could stage all of those changes then merge as convenient
19:26:07 <clarkb> thats true having open changes for them all would help remind us
19:26:15 <clarkb> maybe that is the approach we should take. Compromise
19:26:34 <corvus> we can avoid approving the move until those are all written
19:26:45 <clarkb> ++ I can start on writing those this afternoon too so shouldn't take long
19:27:03 <clarkb> then maybe we base the new trixie image chagne atop the move to quay so that we only ever publish trixie images to quay and skip docker hub?
19:27:07 <corvus> cool... sounds like that would address your concerns while avoiding blocking on vacation
19:27:28 <corvus> that trixie thing would be great
19:27:36 <corvus> would help ferret out any things we missed
19:27:38 <clarkb> I also don't both to add python3.13 images yet since I think the lateral move from bookworm -> trixie then updating python makes more sense to me
19:27:50 <clarkb> *I also didn't bother to add python3.13
19:28:33 <clarkb> ok so plan is put trixie update on top of quay move. Then also push up changes to consume base images from quay so we don't forget to land those. Then we can proceedwith updates to the base images
19:28:43 <corvus> ++
19:29:12 <clarkb> #topic Dropping Ubuntu Bionic Test Nodes
19:29:25 <clarkb> Zuul is going to drop support for Ansible 9 soon
19:29:35 <clarkb> when that happens we won't be able to run jobs on Bionic and older with Ansible 11
19:30:18 <clarkb> I'd like to start the process of removing bionic test nodes in parallel to the zuul changes so that we're not lagging behind. This will also allow us to delete bionic mirror content from the mirror freeing up more space in afs
19:30:37 <clarkb> so far we've dropped jobs that run on bionic in system-config and I'ev got a change to do this in DIB too
19:30:46 <clarkb> if you see jobs elsewhere please start removing them
19:31:05 <clarkb> then for our broader user base do you think we should announce bionic removal to service-announce?
19:31:54 <clarkb> I suspect that openstack's old branches are the biggest users of the platform so we could even just reach out to openstack
19:32:27 <clarkb> open to ideas here. I don't want to be super rigid about this. The release is EOL and we've never promised to host things past their EOL
19:32:49 <clarkb> I'm probably good with just doing cleanup as part of EOL removal but if anyone feels strongly I can make an announcement
19:33:00 <clarkb> #topic Open Discussion
19:33:14 <clarkb> it was pointed out today that our debian trixie images are actually debian testing due to how we were building the images
19:33:27 <clarkb> #link https://review.opendev.org/c/opendev/zuul-providers/+/958561 Fix debian trixie image builds
19:33:41 <clarkb> this change fixes that but depends on an update to DIB which depends on the DIB removal of bionic testing
19:34:33 <clarkb> I'm comfortable self approving the bionic testing removal from dib. But someone who understands debian package repo setups may want to look at the other update in dib
19:34:50 <clarkb> I think we have reasonable test coverage here too so I think its good as is
19:35:41 <clarkb> Monday is a US holiday. I expect that I won't be around much that day. But plan to host a meeting Tuesday. I may just be late getting the agenda out
19:35:47 <clarkb> Anything else?
19:37:20 <clarkb> Sounds like that may be everything
19:37:26 <clarkb> we'll be back here same time and location next week
19:37:34 <clarkb> Thank you for all the help running OpenDev
19:37:44 <clarkb> #endmeeting