| fungi | meeting here in 30 minutes (19:00 utc) | 18:30 |
|---|---|---|
| fungi | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Jun 17 19:00:02 2025 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| fungi | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/SCQGMDGA25UKFCKPKBO2RFTJRETB5XFT/ Our Agenda | 19:01 |
| fungi | sorry, took me a sec | 19:01 |
| fungi | #topic Announcements | 19:01 |
| fungi | #link https://cfp.openstack.org/app/2025summit/all-plans/36/presentations/new/summary The OpenInfra Summit Europe 2025 call for Forum session proposals is open until July 8 | 19:05 |
| fungi | i clearly should have prepared announcements in advance, that url was a lot more work to find than i anticipated | 19:06 |
| fungi | anyway, if anyone wants to propose an opendev-related forum session, now's the time to figure that out | 19:06 |
| fungi | and remember, the gerrit folks will be there too so this is an opportunity for some cross-project work there as well | 19:07 |
| fungi | any other announcements? | 19:07 |
| fungi | #topic Zuul-launcher image builds (corvus 20240910) | 19:08 |
| fungi | i know there's a stack of changes pending for label switches in nodesets | 19:08 |
| corvus | couple of zuul-launcher things: | 19:08 |
| corvus | 1) i have a change up to move the openstack tenant (and others) to use zuul-launcher nodes https://review.opendev.org/952712 | 19:08 |
| corvus | this would affect some, but not all jobs | 19:08 |
| corvus | 2) i have a series of changes to move even more nodesets over to zuul-launcher for the tenants that use it. that series *ends* at https://review.opendev.org/952726 | 19:08 |
| corvus | we could merge the change to switch openstack over, then merge the changes in the series to use progressively more zuul-launcher nodes as we gain confidence | 19:09 |
| corvus | are we ready for that? | 19:09 |
| corvus | frickler noted that we can't autohold zuul-launcher nodes yet. i can probably get that implemented this week so it's in saturday's restart. should we wait on that or proceed without it in order to ramp up our load testing? | 19:09 |
| frickler | I wouldn't see that as blocker if the expected timeframe is that short | 19:10 |
| fungi | yeah, i think if we tell people we can't autohold their job for now it's not the end of the world | 19:10 |
| corvus_ | i think my homeserver just decided to do something else... | 19:11 |
| fungi | i'm on board with approving 952712 after the meeting today, or first thing tomorrow depending on what others are comfortable with | 19:11 |
| corvus_ | yeah, i think that's reasonable. and i think it's worth getting more data now | 19:11 |
| corvus_ | 12:11 | 19:11 |
| corvus_ | so i'll plan on switching over the openstack tenant | 19:11 |
| corvus_ | at the end of the series that moves over nodesets, you can see there are only two images we're missing in zuul-launcher: ubuntu-bionic and ubuntu-focal | 19:12 |
| corvus_ | should we send out a "last call" for folks to port them? or does someone here want to do it? i don't expect it to be very hard. | 19:12 |
| fungi | i think we (opendev) are still using focal nodes for at least some tests right? | 19:12 |
| fungi | things that haven't been moved to jammy/noble yet | 19:13 |
| fungi | i'm hesitant to commit to finding time to do the focal label migration myself at this time though | 19:13 |
| fungi | bionic i'm tempted to just let openstack folks take care of or we drop it | 19:14 |
| corvus_ | ok. i'm sure one of us opendev folks will end up doing focal if no one else does. | 19:14 |
| corvus_ | it's looking highly likely that bionic may just not get done unless there's a motivated openstack-focused person | 19:14 |
| fungi | or we take it as a sign to replace the lingering focal servers and move testing to later ubuntu ;) | 19:15 |
| fungi | which would be better for us overall, but maybe more work in the near term | 19:15 |
| corvus_ | let's let it sit out there for a while, and when shutting off nodepool is in sight, send out one last notice. sound good? | 19:15 |
| fungi | i think when i looked, all the openstack releases still under maintenance were tested on jammy and newer | 19:15 |
| fungi | so even openstack may just say people who care about unmaintained branches should either step up asap or drop testing | 19:16 |
| fungi | yeah, sounds good to me. any other feedback on this topic? | 19:18 |
| frickler | how bad would it be to keep those labels running on nodepool for some time? | 19:18 |
| fungi | it would mean continuing to maintain nodepool and run the servers for it | 19:19 |
| corvus_ | i think the effort to maintain nodepool for one label that no one uses is not worth doing | 19:19 |
| fungi | i'm with you on that | 19:20 |
| corvus | so i'll plan on switching over the openstack tenant | 19:20 |
| fungi | i think the effort to maintain anything no one uses (or at least cares enough about to help with) is not worth it | 19:20 |
| frickler | can we gather data how often a label is getting used? | 19:20 |
| corvus_ | (hey the homeserver is back, sorry about the delayed/repeat message ^) | 19:20 |
| frickler | I'm just arguing that we don't need to switch things off. if they break somehow, that's another story | 19:21 |
| corvus_ | it's probably like 15 minutes of work for someone to port over the image, if it's important. | 19:21 |
| fungi | but also a good opportunity to stop spending cycles and wasting space/bandwidth on an image that isn't being used, if it's not being used that is | 19:22 |
| corvus_ | i very much want to switch it off when we get there. i'd like to recover the resources, and start to see what zuul looks like without nodepool. note that they don't cooperate on things like quota... | 19:22 |
| fungi | so i guess to repeat/restate frickler's earlier question, what's the best way to figure out how much or often a given label is requested? | 19:23 |
| corvus_ | should show up in the launcher logs | 19:23 |
| fungi | (or how recently) | 19:23 |
| fungi | we have ~2 weeks retention on those? or am i remembering wrong? | 19:24 |
| corvus | maybe 10 days it looks like? | 19:25 |
| fungi | i concur | 19:25 |
| fungi | just checked | 19:25 |
| fungi | so we could say a label not requested in 10 days is probably not that important | 19:25 |
| fungi | and someone could also port it later if that turns out to be an incorrect assumption | 19:26 |
| fungi | anything else on this topic? | 19:27 |
| corvus | nope | 19:27 |
| corvus | thanks! | 19:27 |
| fungi | #topic Gerrit shutdown problems (clarkb 20250527) | 19:27 |
| fungi | i don't think there's anything new on this, we can revisit next week | 19:27 |
| fungi | #topic Gerrit 3.11 Upgrade Planning (clarkb 20250401) | 19:27 |
| fungi | #link https://www.gerritcodereview.com/3.11.html Please check this for any concerns with the way we use Gerrit | 19:28 |
| fungi | 104.130.253.194 is a held Gerrit 3.11 node for testing purposes | 19:29 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io | 19:29 |
| fungi | and its parent change | 19:29 |
| fungi | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade | 19:30 |
| fungi | no changes since last week, but all good reminders as we should move forward on this soon | 19:30 |
| fungi | #topic Upgrading Old Servers (clarkb 20230627) | 19:30 |
| fungi | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades | 19:31 |
| fungi | #link https://etherpad.opendev.org/p/opendev-focal-server-upgrades | 19:31 |
| fungi | those are somewhat relevant to the earlier zuul-launcher discussion as well | 19:31 |
| fungi | i haven't seen tonyb so unsure what the wiki (and cacti) situation is, presumably the same as last week | 19:32 |
| fungi | similarly i haven't found time to work out what the communication schedule should be (if any) for decommissioning refstack | 19:32 |
| fungi | we're already halfway through the hour so i'm going to skip a few older topics for now... | 19:33 |
| fungi | #topic Enabling hashtags globally (corvus 20250523) | 19:33 |
| fungi | this happened | 19:33 |
| fungi | i haven't heard any complaints | 19:33 |
| corvus | i think we can drop from agenda now? | 19:33 |
| fungi | likely | 19:34 |
| fungi | should be resolved, i just wanted to double-check there's been no fallout that escaped my attention | 19:34 |
| fungi | #topic Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) | 19:35 |
| fungi | i don't think there's anything new on this one, but as there are a bunch of moving parts and multiple people involved i want to make sure | 19:36 |
| fungi | sounds like no | 19:37 |
| fungi | #topic projects.yaml normalization (frickler 20250617) | 19:37 |
| fungi | a (possibly intentional) regression in ruamel.yaml has started adding spaces to the end of wrapped strings, akin to format=flowed E-mail wrapping | 19:37 |
| fungi | #link https://sourceforge.net/p/ruamel-yaml/tickets/546/ Trailing whitespaces in some generated YAML outputs | 19:37 |
| fungi | #link https://review.opendev.org/952006 this is impacting our normalization job output | 19:38 |
| corvus | i really hope that's not intentional :) | 19:38 |
| fungi | yeah, it seems odd to me | 19:39 |
| corvus | ruamel does allow for a lot of control, i wonder if we could code around it | 19:39 |
| fungi | as i mentioned in one of the comments, the only place i've ever seen that is in format=flowed e-mail | 19:39 |
| corvus | (which is a mitigation for the quirks of a 40 year old standard) | 19:40 |
| fungi | the lack of maintainer feedback on the linked ticket means i don't know whether it's worth hacking around something they may just fix in the next release | 19:40 |
| corvus | (not something anyone should be doing this century) | 19:40 |
| fungi | and yeah, wrapping is explicit in yaml, no need for inference like in e-mail message bodies | 19:40 |
| fungi | probably just missed chomping the whitespace at the wrap point in some update | 19:41 |
| corvus | my thoughts: 1) this is not a good format, we shouldn't adapt to it; 2) pinning for a couple months to see if things shake out seems okay; 3) if it doesn't get reverted, we should code around it... somehow. | 19:41 |
| fungi | so anyway, the choices as currently presented to us are to temporarily pin/skip versions of that library, accept the new output, or keep ignoring it and hope something changes | 19:42 |
| fungi | i agree with your take, corvus | 19:42 |
| frickler | +1 | 19:42 |
| frickler | the second point I wanted to mention: why does the bot keep updating the change when there is no delta to the previous PS? | 19:43 |
| fungi | also worth keeping an eye on (and linking in an inline comment for the pin/skip) the upstream bug report | 19:43 |
| fungi | frickler: without having dug into it, i'm guessing it either doesn't attept to check whether the diff is equivalent to the existing change already in gerrit, or the attempt it makes is flawed in some way | 19:44 |
| fungi | someone needs to figure out which that is if we don't want to continue getting new identical revisions daily | 19:44 |
| frickler | I think I saw a check, but I don't have the ref handy | 19:45 |
| fungi | #link https://review.opendev.org/952315 Block broken ruamel.yaml versions | 19:46 |
| fungi | that's worth calling out as the (currently abandoned) option we seem to be favoring | 19:46 |
| fungi | perhaps worth restoring and updating if necessary | 19:47 |
| corvus | ++ | 19:47 |
| frickler | restored now | 19:47 |
| fungi | thanks | 19:47 |
| frickler | will check whether the denylist is still current tomorrow | 19:47 |
| fungi | thanks! | 19:48 |
| fungi | i think that's still current, fwiw | 19:48 |
| fungi | it has 2x+2 already, feel free to self-approve if you confirm there aren't newer releases yet to worry about | 19:49 |
| fungi | okay, i don't know that this is worth spending more time on since those of us present seem to have reached a quick agreement | 19:50 |
| fungi | #topic Open discussion | 19:50 |
| fungi | we have 10 minutes left, anyone have updates on the older topics i skipped or anything else they want to bring up? | 19:50 |
| fungi | seems like no. i return the balance of 5 minutes, don't spend it all in one place! | 19:54 |
| fungi | #endmeeting | 19:54 |
| opendevmeet | Meeting ended Tue Jun 17 19:54:34 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:54 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-17-19.00.html | 19:54 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-17-19.00.txt | 19:54 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-17-19.00.log.html | 19:54 |
| corvus | thanks! | 19:54 |
| -opendevstatus- NOTICE: Zuul jobs reporting POST_FAILURE were due to an incident with one of our cloud providers; this provider has been temporarily disabled and changes can be rechecked | 22:36 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!