19:00:02 <fungi> #startmeeting infra
19:00:02 <opendevmeet> Meeting started Tue Jun 17 19:00:02 2025 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:02 <opendevmeet> The meeting name has been set to 'infra'
19:01:30 <fungi> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/SCQGMDGA25UKFCKPKBO2RFTJRETB5XFT/ Our Agenda
19:01:35 <fungi> sorry, took me a sec
19:01:43 <fungi> #topic Announcements
19:05:48 <fungi> #link https://cfp.openstack.org/app/2025summit/all-plans/36/presentations/new/summary The OpenInfra Summit Europe 2025 call for Forum session proposals is open until July 8
19:06:10 <fungi> i clearly should have prepared announcements in advance, that url was a lot more work to find than i anticipated
19:06:37 <fungi> anyway, if anyone wants to propose an opendev-related forum session, now's the time to figure that out
19:07:02 <fungi> and remember, the gerrit folks will be there too so this is an opportunity for some cross-project work there as well
19:07:17 <fungi> any other announcements?
19:08:03 <fungi> #topic Zuul-launcher image builds (corvus 20240910)
19:08:23 <fungi> i know there's a stack of changes pending for label switches in nodesets
19:08:28 <corvus> couple of zuul-launcher things:
19:08:34 <corvus> 1) i have a change up to move the openstack tenant (and others) to use zuul-launcher nodes https://review.opendev.org/952712
19:08:38 <corvus> this would affect some, but not all jobs
19:08:43 <corvus> 2) i have a series of changes to move even more nodesets over to zuul-launcher for the tenants that use it.  that series *ends* at https://review.opendev.org/952726
19:09:00 <corvus> we could merge the change to switch openstack over, then merge the changes in the series to use progressively more zuul-launcher nodes as we gain confidence
19:09:01 <corvus> are we ready for that?
19:09:07 <corvus> frickler noted that we can't autohold zuul-launcher nodes yet.  i can probably get that implemented this week so it's in saturday's restart.  should we wait on that or proceed without it in order to ramp up our load testing?
19:10:02 <frickler> I wouldn't see that as blocker if the expected timeframe is that short
19:10:23 <fungi> yeah, i think if we tell people we can't autohold their job for now it's not the end of the world
19:11:41 <corvus_> i think my homeserver just decided to do something else...
19:11:43 <fungi> i'm on board with approving 952712 after the meeting today, or first thing tomorrow depending on what others are comfortable with
19:11:50 <corvus_> yeah, i think that's reasonable.  and i think it's worth getting more data now
19:11:53 <corvus_> 12:11
19:11:58 <corvus_> so i'll plan on switching over the openstack tenant
19:12:05 <corvus_> at the end of the series that moves over nodesets, you can see there are only two images we're missing in zuul-launcher: ubuntu-bionic and ubuntu-focal
19:12:10 <corvus_> should we send out a "last call" for folks to port them?  or does someone here want to do it?  i don't expect it to be very hard.
19:12:48 <fungi> i think we (opendev) are still using focal nodes for at least some tests right?
19:13:06 <fungi> things that haven't been moved to jammy/noble yet
19:13:49 <fungi> i'm hesitant to commit to finding time to do the focal label migration myself at this time though
19:14:16 <fungi> bionic i'm tempted to just let openstack folks take care of or we drop it
19:14:36 <corvus_> ok.  i'm sure one of us opendev folks will end up doing focal if no one else does.
19:14:56 <corvus_> it's looking highly likely that bionic may just not get done unless there's a motivated openstack-focused person
19:15:00 <fungi> or we take it as a sign to replace the lingering focal servers and move testing to later ubuntu ;)
19:15:25 <fungi> which would be better for us overall, but maybe more work in the near term
19:15:57 <corvus_> let's let it sit out there for a while, and when shutting off nodepool is in sight, send out one last notice.  sound good?
19:15:58 <fungi> i think when i looked, all the openstack releases still under maintenance were tested on jammy and newer
19:16:25 <fungi> so even openstack may just say people who care about unmaintained branches should either step up asap or drop testing
19:18:14 <fungi> yeah, sounds good to me. any other feedback on this topic?
19:18:42 <frickler> how bad would it be to keep those labels running on nodepool for some time?
19:19:03 <fungi> it would mean continuing to maintain nodepool and run the servers for it
19:19:53 <corvus_> i think the effort to maintain nodepool for one label that no one uses is not worth doing
19:20:19 <fungi> i'm with you on that
19:20:35 <corvus> so i'll plan on switching over the openstack tenant
19:20:38 <fungi> i think the effort to maintain anything no one uses (or at least cares enough about to help with) is not worth it
19:20:46 <frickler> can we gather data how often a label is getting used?
19:20:54 <corvus_> (hey the homeserver is back, sorry about the delayed/repeat message ^)
19:21:27 <frickler> I'm just arguing that we don't need to switch things off. if they break somehow, that's another story
19:21:30 <corvus_> it's probably like 15 minutes of work for someone to port over the image, if it's important.
19:22:12 <fungi> but also a good opportunity to stop spending cycles and wasting space/bandwidth on an image that isn't being used, if it's not being used that is
19:22:13 <corvus_> i very much want to switch it off when we get there.  i'd like to recover the resources, and start to see what zuul looks like without nodepool.  note that they don't cooperate on things like quota...
19:23:35 <fungi> so i guess to repeat/restate frickler's earlier question, what's the best way to figure out how much or often a given label is requested?
19:23:46 <corvus_> should show up in the launcher logs
19:23:46 <fungi> (or how recently)
19:24:08 <fungi> we have ~2 weeks retention on those? or am i remembering wrong?
19:25:16 <corvus> maybe 10 days it looks like?
19:25:33 <fungi> i concur
19:25:35 <fungi> just checked
19:25:52 <fungi> so we could say a label not requested in 10 days is probably not that important
19:26:06 <fungi> and someone could also port it later if that turns out to be an incorrect assumption
19:27:06 <fungi> anything else on this topic?
19:27:14 <corvus> nope
19:27:20 <corvus> thanks!
19:27:28 <fungi> #topic Gerrit shutdown problems (clarkb 20250527)
19:27:40 <fungi> i don't think there's anything new on this, we can revisit next week
19:27:49 <fungi> #topic Gerrit 3.11 Upgrade Planning (clarkb 20250401)
19:28:52 <fungi> #link https://www.gerritcodereview.com/3.11.html Please check this for any concerns with the way we use Gerrit
19:29:03 <fungi> 104.130.253.194 is a held Gerrit 3.11 node for testing purposes
19:29:17 <fungi> #link https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io
19:29:42 <fungi> and its parent change
19:30:00 <fungi> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
19:30:23 <fungi> no changes since last week, but all good reminders as we should move forward on this soon
19:30:47 <fungi> #topic Upgrading Old Servers (clarkb 20230627)
19:31:00 <fungi> #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
19:31:11 <fungi> #link https://etherpad.opendev.org/p/opendev-focal-server-upgrades
19:31:35 <fungi> those are somewhat relevant to the earlier zuul-launcher discussion as well
19:32:16 <fungi> i haven't seen tonyb so unsure what the wiki (and cacti) situation is, presumably the same as last week
19:32:44 <fungi> similarly i haven't found time to work out what the communication schedule should be (if any) for decommissioning refstack
19:33:31 <fungi> we're already halfway through the hour so i'm going to skip a few older topics for now...
19:33:36 <fungi> #topic Enabling hashtags globally (corvus 20250523)
19:33:39 <fungi> this happened
19:33:46 <fungi> i haven't heard any complaints
19:33:56 <corvus> i think we can drop from agenda now?
19:34:00 <fungi> likely
19:34:21 <fungi> should be resolved, i just wanted to double-check there's been no fallout that escaped my attention
19:35:42 <fungi> #topic Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527)
19:36:13 <fungi> i don't think there's anything new on this one, but as there are a bunch of moving parts and multiple people involved i want to make sure
19:37:17 <fungi> sounds like no
19:37:29 <fungi> #topic projects.yaml normalization (frickler 20250617)
19:37:31 <fungi> a (possibly intentional) regression in ruamel.yaml has started adding spaces to the end of wrapped strings, akin to format=flowed E-mail wrapping
19:37:46 <fungi> #link https://sourceforge.net/p/ruamel-yaml/tickets/546/ Trailing whitespaces in some generated YAML outputs
19:38:06 <fungi> #link https://review.opendev.org/952006 this is impacting our normalization job output
19:38:53 <corvus> i really hope that's not intentional :)
19:39:06 <fungi> yeah, it seems odd to me
19:39:17 <corvus> ruamel does allow for a lot of control, i wonder if we could code around it
19:39:31 <fungi> as i mentioned in one of the comments, the only place i've ever seen that is in format=flowed e-mail
19:40:03 <corvus> (which is a mitigation for the quirks of a 40 year old standard)
19:40:09 <fungi> the lack of maintainer feedback on the linked ticket means i don't know whether it's worth hacking around something they may just fix in the next release
19:40:14 <corvus> (not something anyone should be doing this century)
19:40:56 <fungi> and yeah, wrapping is explicit in yaml, no need for inference like in e-mail message bodies
19:41:22 <fungi> probably just missed chomping the whitespace at the wrap point in some update
19:41:49 <corvus> my thoughts: 1) this is not a good format, we shouldn't adapt to it; 2) pinning for a couple months to see if things shake out seems okay; 3) if it doesn't get reverted, we should code around it... somehow.
19:42:08 <fungi> so anyway, the choices as currently presented to us are to temporarily pin/skip versions of that library, accept the new output, or keep ignoring it and hope something changes
19:42:22 <fungi> i agree with your take, corvus
19:42:35 <frickler> +1
19:43:18 <frickler> the second point I wanted to mention: why does the bot keep updating the change when there is no delta to the previous PS?
19:43:22 <fungi> also worth keeping an eye on (and linking in an inline comment for the pin/skip) the upstream bug report
19:44:10 <fungi> frickler: without having dug into it, i'm guessing it either doesn't attept to check whether the diff is equivalent to the existing change already in gerrit, or the attempt it makes is flawed in some way
19:44:41 <fungi> someone needs to figure out which that is if we don't want to continue getting new identical revisions daily
19:45:32 <frickler> I think I saw a check, but I don't have the ref handy
19:46:27 <fungi> #link https://review.opendev.org/952315 Block broken ruamel.yaml versions
19:46:52 <fungi> that's worth calling out as the (currently abandoned) option we seem to be favoring
19:47:06 <fungi> perhaps worth restoring and updating if necessary
19:47:11 <corvus> ++
19:47:23 <frickler> restored now
19:47:48 <fungi> thanks
19:47:56 <frickler> will check whether the denylist is still current tomorrow
19:48:03 <fungi> thanks!
19:48:15 <fungi> i think that's still current, fwiw
19:49:01 <fungi> it has 2x+2 already, feel free to self-approve if you confirm there aren't newer releases yet to worry about
19:50:04 <fungi> okay, i don't know that this is worth spending more time on since those of us present seem to have reached a quick agreement
19:50:20 <fungi> #topic Open discussion
19:50:45 <fungi> we have 10 minutes left, anyone have updates on the older topics i skipped or anything else they want to bring up?
19:54:24 <fungi> seems like no. i return the balance of 5 minutes, don't spend it all in one place!
19:54:34 <fungi> #endmeeting