| clarkb | Almost meeting time. I'm operating under the assumption that this meeting will be lightly attended. Thats fine, but if you're around and have somethign to say don't hesitate to jump in otherwise I'll probably move fairly quickyl | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Aug 26 19:00:17 2025 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/NNABJTQSALCFXBGPP4SNCGGN4TGLYB6E/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | I didn't have anything to announce. Did anyone else? | 19:00 |
| clarkb | #topic Gerrit 3.11 Upgrade Planning | 19:01 |
| clarkb | This is the perpetually stalled out task... Its not urgent and other things keep fighting for my attention | 19:02 |
| clarkb | and unfortunately there are nwe gerrit bug fix releases we need to update to again. I think I'm aiming to get that done when fungi gets back so not a rush | 19:03 |
| clarkb | but if you have time to review the gerrit bugfix update change that is always appreciated | 19:03 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/957555 | 19:03 |
| clarkb | I'm happy to approve that when I've got time to pull the image and restart gerrit | 19:03 |
| clarkb | #topic Upgrading old servers | 19:03 |
| clarkb | fungi picked up the kerberos and openafs server upgrades | 19:04 |
| clarkb | all but one server is running jammy now. Plan is to upgrade the last server to jammy then pretty quickly start the jammy -> noble upgrades | 19:05 |
| clarkb | thank you fungi for getting that started. Other than the slowness of waiting for afs RW volume moves I think this has gone reasonably well | 19:05 |
| corvus | ++ | 19:05 |
| clarkb | One thing to note is that the RW volume moves do seem to consume additional disk space on the fileservers. We should keep this in mind as we add potential new content to afs and also look at deleting things (its good to delete stuff if we don't need ti anymore) | 19:06 |
| clarkb | then the other remaining servers on the todo list are the graphite server and the backup servers | 19:06 |
| clarkb | with graphite we'll want to preserve/copy/move the existing data volume so that we don't lose data. For the backup servers I think what we've done in the past is boot new servers and attach new volumes to the new servers. Do new backups there. Then we attach the old backup volumes to the new servers as an archive of the old backups allowing us to delete the old servers | 19:07 |
| clarkb | I think that approach with the backup servers continues to make sense so is probably the one we'll use here | 19:08 |
| clarkb | any questions/concerns/comments with server upgrades/replacements? | 19:08 |
| clarkb | #topic Matrix for OpenDev comms | 19:09 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing | 19:09 |
| clarkb | looks like we have some community feedback in favor of this change | 19:09 |
| clarkb | still needs additional reviews from infra-root | 19:09 |
| clarkb | if you have time to look at the latest patchset your input is appreciated | 19:09 |
| clarkb | but I think we can keep most of the discussion of the change itself in gerrit code review. See you there | 19:10 |
| clarkb | #topic Pre PTG Planning | 19:11 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document | 19:11 |
| clarkb | Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 | 19:11 |
| clarkb | the Tuesday block there will replace our weekly team meeting | 19:11 |
| clarkb | please feel free to add agenda items to that etherpad | 19:11 |
| clarkb | I'm sure I'm not the only one with ideas | 19:12 |
| clarkb | #topic Service Coordinator Election Planning | 19:12 |
| clarkb | This nomination period ended and as far as I can tell I was the only nominee. | 19:13 |
| clarkb | This means I'm it by default. I wanted to call that out here so that anyone can point out missed nominations or object or otherwise provide input on the election | 19:14 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/WNUYDT47NMYC3SC5QA44OG4PWK5ENQEF/ clarkbs nomination | 19:14 |
| clarkb | but if there is no input then I think we can consider this official and drop this item from next week's agenda | 19:14 |
| clarkb | #topic Loss of upstream Debian bullseye-backports mirror | 19:15 |
| clarkb | fungi is out this week and I'm not sure where he got to on this effort | 19:15 |
| clarkb | I think the idea was to notify zuul users that zuul-jobs would update to match upstream | 19:16 |
| clarkb | I don't see that notification to the zuul list | 19:16 |
| clarkb | corvus if you get a chance maybe you can double check it isn't queued up for zuul-announce? and if not then I can probably go ahead and draft something | 19:16 |
| clarkb | that way we can get the clock ticking on the cleanup | 19:16 |
| clarkb | We claened up xenial and arm64 bionic mirror content as part of the openafs server volume moves (we needed more disk space to do the moves) and I think cleaning up debian bullseye backports is another good cleanup we can add to that pile | 19:17 |
| clarkb | #topic Etherpad 2.5.0 Upgrade | 19:18 |
| corvus | Will do | 19:18 |
| clarkb | After filing an issue for the etherpad 2.4.2 release breaking the no-skin skin (that we use) upstream "fixed" the no-skin skin and made this 2.5.0 release | 19:18 |
| clarkb | #link https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md | 19:18 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/956593/ | 19:18 |
| clarkb | Using that change and its child I held a node with IP address 104.130.127.119 for testing. The / page still has slightly odd css rendering but it is better than before | 19:19 |
| clarkb | and the error that occurred in pads when additional clients join seems to have gone away | 19:19 |
| clarkb | if you want to see for yourself you can add that IP address into your local /etc/hosts for etherpad.opendev.org. I created a clarkb-test pad too fi you want to look at some content | 19:20 |
| clarkb | all this to say its not perfect but is better than before and we might consider how bad we think that css is and/or maybe patch the css ourselves | 19:20 |
| clarkb | I think this is probably workable if we needed to upgrade. But as before I don't think anything in the cahngelog makes this urgent for us. Mostly just a desire to not fall too far behind | 19:20 |
| clarkb | #topic Moving OpenDev's python-base/python-builder/uwsig-base Images to Quay | 19:21 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/957277 | 19:21 |
| clarkb | all of our python based containers are now running on noble so we can switch things to quay without losing speculative image testing | 19:21 |
| clarkb | However, this change updates the parent images then we should plan to rebuild and redeploy all of our services that depend on them afterawrds. Which probably makes this a task best for when we don't have people on vacation | 19:22 |
| clarkb | the change itself is straightforward but ensuring we've updating all the things to catch up is less so | 19:22 |
| clarkb | but if we feel the risk is worth it here let me know and I can work through that list after this change lands | 19:22 |
| clarkb | #topic Adding Debian Trixie Base Python Container Images | 19:23 |
| clarkb | this is related to the previous change in that it adds more base python images. I think we can land these two changes in either order then update the other to accomodate | 19:23 |
| corvus | clarkb: sorry | 19:23 |
| corvus | clarkb: why do we need to deploy stuff right after we switch? | 19:24 |
| corvus | like, why can't we just switch and when redeploys happen they happen? | 19:24 |
| clarkb | corvus: not necessarily immediately after but I don't want to forget that the upstream image source has changed and then end up building images with the old source down the line | 19:24 |
| clarkb | corvus: the main concern is that we'll forget to update the images and accounting problems will not be caught until much later | 19:24 |
| clarkb | corvus: since we have to modfiy the actual Dockerfile for each downstream consumer to fetch from quay.io | 19:25 |
| clarkb | so it isn't strictly necessary but I want to avoid forgetting and having problems later | 19:25 |
| corvus | we could stage all of those changes then merge as convenient | 19:25 |
| clarkb | thats true having open changes for them all would help remind us | 19:26 |
| clarkb | maybe that is the approach we should take. Compromise | 19:26 |
| corvus | we can avoid approving the move until those are all written | 19:26 |
| clarkb | ++ I can start on writing those this afternoon too so shouldn't take long | 19:26 |
| clarkb | then maybe we base the new trixie image chagne atop the move to quay so that we only ever publish trixie images to quay and skip docker hub? | 19:27 |
| corvus | cool... sounds like that would address your concerns while avoiding blocking on vacation | 19:27 |
| corvus | that trixie thing would be great | 19:27 |
| corvus | would help ferret out any things we missed | 19:27 |
| clarkb | I also don't both to add python3.13 images yet since I think the lateral move from bookworm -> trixie then updating python makes more sense to me | 19:27 |
| clarkb | *I also didn't bother to add python3.13 | 19:27 |
| clarkb | ok so plan is put trixie update on top of quay move. Then also push up changes to consume base images from quay so we don't forget to land those. Then we can proceedwith updates to the base images | 19:28 |
| corvus | ++ | 19:28 |
| clarkb | #topic Dropping Ubuntu Bionic Test Nodes | 19:29 |
| clarkb | Zuul is going to drop support for Ansible 9 soon | 19:29 |
| clarkb | when that happens we won't be able to run jobs on Bionic and older with Ansible 11 | 19:29 |
| clarkb | I'd like to start the process of removing bionic test nodes in parallel to the zuul changes so that we're not lagging behind. This will also allow us to delete bionic mirror content from the mirror freeing up more space in afs | 19:30 |
| clarkb | so far we've dropped jobs that run on bionic in system-config and I'ev got a change to do this in DIB too | 19:30 |
| clarkb | if you see jobs elsewhere please start removing them | 19:30 |
| clarkb | then for our broader user base do you think we should announce bionic removal to service-announce? | 19:31 |
| clarkb | I suspect that openstack's old branches are the biggest users of the platform so we could even just reach out to openstack | 19:31 |
| clarkb | open to ideas here. I don't want to be super rigid about this. The release is EOL and we've never promised to host things past their EOL | 19:32 |
| clarkb | I'm probably good with just doing cleanup as part of EOL removal but if anyone feels strongly I can make an announcement | 19:32 |
| clarkb | #topic Open Discussion | 19:33 |
| clarkb | it was pointed out today that our debian trixie images are actually debian testing due to how we were building the images | 19:33 |
| clarkb | #link https://review.opendev.org/c/opendev/zuul-providers/+/958561 Fix debian trixie image builds | 19:33 |
| clarkb | this change fixes that but depends on an update to DIB which depends on the DIB removal of bionic testing | 19:33 |
| clarkb | I'm comfortable self approving the bionic testing removal from dib. But someone who understands debian package repo setups may want to look at the other update in dib | 19:34 |
| clarkb | I think we have reasonable test coverage here too so I think its good as is | 19:34 |
| clarkb | Monday is a US holiday. I expect that I won't be around much that day. But plan to host a meeting Tuesday. I may just be late getting the agenda out | 19:35 |
| clarkb | Anything else? | 19:35 |
| clarkb | Sounds like that may be everything | 19:37 |
| clarkb | we'll be back here same time and location next week | 19:37 |
| clarkb | Thank you for all the help running OpenDev | 19:37 |
| clarkb | #endmeeting | 19:37 |
| opendevmeet | Meeting ended Tue Aug 26 19:37:44 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:37 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-26-19.00.html | 19:37 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-26-19.00.txt | 19:37 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-26-19.00.log.html | 19:37 |
| corvus | thanks clarkb ! | 19:37 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!