| clarkb | just about meeting time. We'll get started in a minute or two | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Jun 24 19:00:19 2025 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/YJ4OIEZ4ARUKB5DRBWTRKT3U6BNBCFME/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | Thank you fungi for running the last couple of meetings. I had some family stuff pop up and wasn't able to consistently be around | 19:00 |
| clarkb | but I'm slowly catching back up now and finding my routine again | 19:01 |
| fungi | welcome back! | 19:01 |
| clarkb | the open infra summit's forum session proposals are open right now | 19:01 |
| clarkb | I don't know that we need any opendev forums sessions but figured I'd mention that | 19:02 |
| clarkb | anything else to announce? | 19:02 |
| clarkb | alright lets jump into the agenda then | 19:04 |
| clarkb | #topic Zuul-launcher transition | 19:04 |
| clarkb | I think that auto hold stuff has rolled out or is rolling out | 19:04 |
| clarkb | which means we're largely ready to start migrating more jobs over to zuul launcher from nodepool. Early changes to make that happened merged just before the meeting | 19:04 |
| corvus | yep... | 19:05 |
| corvus | and i think we want to revert it | 19:05 |
| corvus | rax-flex-sjc3 is very sad | 19:05 |
| corvus | and in a new way that z-l isn't handling well | 19:05 |
| fungi | deploy was at 18:50 utc, for the record | 19:05 |
| corvus | so i'm going to propose a revert, manually merge it, then add some exception handling. | 19:05 |
| fungi | thanks for spotting that quickly | 19:06 |
| clarkb | ack | 19:06 |
| corvus | but yeah, we're really close :) | 19:06 |
| corvus | and after we merge this change: https://review.opendev.org/952721 | 19:06 |
| clarkb | fwiw looking at the nodepool grafana graphs (not the zl grafana graphs) it looks similar to what we saw yseterday which was the cloud having some issues that needed to be resolved on the cloud side to get successful boots | 19:06 |
| clarkb | but having zuul launcher handle that gracefully is a good idea (nodepool mostly does) | 19:06 |
| corvus | we will only have ubuntu-focal and ubuntu-xenial images left in nodepool | 19:07 |
| corvus | i forget -- where did we land on whether those were used, and what do we want to do before turning off nodepool? | 19:07 |
| clarkb | for xenial one of the biggest users is/was opendev. But I did some cleanup of that on the opendev side and I think we can rip the bandaid off for opendev | 19:08 |
| clarkb | focal is more difficult | 19:08 |
| clarkb | or did you mean bionic? | 19:08 |
| clarkb | in either case I Think we have openstack stable stuff and translation stuff relying on either bionic or focal (maybe both) | 19:08 |
| corvus | oh sorry i meant bionic | 19:09 |
| corvus | https://review.opendev.org/952726 is the change i meant to link | 19:09 |
| clarkb | ya I think the translation sync stuff runs on bionic due to java versions or something along those lines | 19:09 |
| corvus | no wait | 19:09 |
| corvus | it's bionic and focal that are left | 19:09 |
| corvus | and focal? | 19:10 |
| fungi | possible we could do something with a tarred-up bionic chroot or container image, i suppose | 19:10 |
| clarkb | got it. I see my comment on 952726 now too. To summarize I think we can rip the bandaid off for xenial and live with the fallout. We may need to check bionic/focal things more carefully. fungi not sure if you know how many if any stable branches for openstack rely on either of those. | 19:10 |
| clarkb | fungi: yes I think one option for the small number of things that need the old tooling would be to use a container to run the tools | 19:10 |
| corvus | fungi: that would almost certainly be *harder* than porting over the build, no? | 19:11 |
| fungi | porting it over to what? | 19:11 |
| corvus | don't we have working bionic and focal dib builds with nodepool-launcher? | 19:11 |
| corvus | er nodepool-builder | 19:11 |
| clarkb | corvus: yes | 19:11 |
| clarkb | so yes another option available to us is to add those images to zuul launcher | 19:11 |
| fungi | i mean using a bionic chroot with zanata tools installed on a newer (jammy or noble) test node | 19:12 |
| corvus | yeah, which is probably like 5m of work | 19:12 |
| fungi | if the goal is to drop the bionic nodes | 19:12 |
| clarkb | I think that is a good goal (and why I think we should proceed wtih xenial that way). But I recognize there is some improtant functionality relying on it so maybe comprimise is better there | 19:13 |
| clarkb | if its only 5m of work that seems like a reasonable thing to do | 19:13 |
| clarkb | anything else on this topic? | 19:14 |
| corvus | yeah, so if opendev is happy to continue hosting bionic and focal images, then we just need someone to copy over the config to zuul-providers | 19:14 |
| fungi | right now some openstack projects are still testing on their unmaintained/victoria branches as the oldest not yet eol. victoria's official test runtime included ubuntu 20.04 (focal), so bionic should not be needed if it weren't for the translation jobs requiring older java, i think | 19:14 |
| clarkb | fungi: ack that is good info | 19:15 |
| clarkb | that tells me we should probably go ahead and add focal images. Then bionic is a matter of deciding about translation jobs but probably easy enough to do when we do focal | 19:15 |
| fungi | in theory the bionic requirement for those jobs goes away once openstack's i18n team finishes migrating to weblate | 19:15 |
| tonyb | I'm planning on helping the i18n team with that in July. | 19:16 |
| corvus | i think we need an action item about the images | 19:16 |
| corvus | from my pov, we sent out an email a few months ago and got some help, and now all but those 2 are done | 19:16 |
| clarkb | do we have a volunteer to add focal and xenial images to zuul launcher? | 19:16 |
| corvus | focal and bionic, right? :) | 19:17 |
| clarkb | yes right sorry | 19:17 |
| corvus | i don't think anyone has volunteered | 19:17 |
| clarkb | mnasiadka isn't in here but did do a big part of that move for openstack. I can ask mnasiadka if they would be willing to do focal and bionic too | 19:17 |
| corvus | ok, then if not, maybe send a "last call" email | 19:17 |
| clarkb | ++ | 19:17 |
| clarkb | #action clarkb to ask mnasiadka about adding focal and bionic image builds to zuul launcher | 19:18 |
| corvus | thanks... i'm happy to send the last-call email if needed | 19:18 |
| corvus | i think that's all the decision making i need right now | 19:18 |
| clarkb | lets continue on with our agenda since we're already 1/3 of the way through our time | 19:19 |
| clarkb | #topic Gerrit Shutdown Process and 3.11 Upgrade | 19:19 |
| fungi | for further data points, openstack unmaintained/zed is the most recent branch officially relying on focal nodes (it's been unmaintained for over a year now), and i think the openstack position on it's going to be that if opendev wants to pull the rug out from under those jobs because nobody who claims to care about those branches volunteered to make this work, then that's how | 19:19 |
| fungi | it goes | 19:19 |
| clarkb | I'm going to combine the next two topics into one as I'm largely driving them and don't have any real updates | 19:19 |
| clarkb | I am hoping that I will finally have the ability to update the gerrit command line string to remove the h2 cache cleanup timeout change and restart gerrit this week | 19:20 |
| clarkb | and use that process to test manual sighup signalling and possibly a followup to do a docker compose driven sigint as well | 19:20 |
| clarkb | then assuming I get that done I'm hoping Gerrit upgrade planning and testing can start up again early july | 19:21 |
| clarkb | no real updates other than tyhis is still on my radar and I plan to look at it again as I catch up | 19:21 |
| clarkb | #topic Upgrading old servers | 19:21 |
| clarkb | corvus replaced zuul mergers and zuul launchers with new noble nodes | 19:21 |
| corvus | planning to do schedulers and executors later this week/weekend | 19:22 |
| clarkb | that seems to have largely gone well. The one hiccup was reducing the size of zuul launchers which reduced their epehermal disk sizes which a fair chunk is needed for to shuffle image data around | 19:22 |
| clarkb | this has since been corrected with some cinder volumes | 19:22 |
| corvus | also, the restart script needed a change because "docker-compose ps -q" behavior changed | 19:22 |
| corvus | i think that command is only used for the zuul servers; does that sound right? | 19:23 |
| clarkb | corvus: yes I think that is the only situation where we use that | 19:23 |
| clarkb | I'm also like 95% of the way through replacing mirror-update02 with a noble mirror-update03 | 19:23 |
| clarkb | I thoguht I was going to delete 02 before this meeting then discovered a problem with debian reprepro config that I wanted to fix first. The fix is in place and one of the two mirror locations is happy again. The other is waiting for the next run in about 45 minutes | 19:24 |
| clarkb | once that is successful I'll proceed with cleaning up the old mirror-update02 node | 19:24 |
| clarkb | corvus: worht noting that mirror-update03 is using openafs to read and write data on noble which is a good indicator for executors | 19:24 |
| clarkb | previously we were doing read only operations on mirrors I think | 19:25 |
| tonyb | I don't know if the centos team have done it yet but they have a ticket to fix the IP situation | 19:25 |
| clarkb | tonyb: yup that seems to be working now too | 19:25 |
| fungi | tonyb: yeah, that's been resolved since around 13:00z | 19:25 |
| tonyb | \o/ | 19:26 |
| clarkb | my hope is to start on the zookeeper cluster tomorrow aswell | 19:27 |
| clarkb | currently we have zk04, zk05, and zk06. Do you want zk07-09 or zk01-03 as the new servers? | 19:27 |
| clarkb | changing things in place is maybe more complicated with zk since it uses the digits suffix as the cluster member id or whatever it is called | 19:27 |
| clarkb | I think new servers should have new ids as a result | 19:28 |
| clarkb | I'm guessing 01-03 is preferred. But let me know if not and I can do something else | 19:28 |
| corvus | i don't feel strongly; slight pref for 1-3 | 19:28 |
| fungi | sounds good to me as well | 19:29 |
| clarkb | other things to note: don't forget to use the --config-drive flag to launch node when booting our noble image in rax classic (it is required there but not in other clouds) | 19:29 |
| clarkb | and fungi was there any refstack update? | 19:29 |
| fungi | nope! | 19:29 |
| clarkb | that was all I had on this topic. Anything else before I go to the next one? | 19:29 |
| clarkb | #topic OFTC Matrix bridge no longer supporting new users | 19:31 |
| clarkb | the last time I was around to discuss this I volunteered to write a spec to explicitly call out what we're trying to achieve with a move to matrix for opendev comms and a plan for testing that through doing it | 19:31 |
| clarkb | I have not done that. But doing so is one of the things I intend to do as I catch up | 19:31 |
| clarkb | is there anything new to consider from the last couple of weeks before I do that? | 19:32 |
| corvus | no news i'm aware of | 19:33 |
| fungi | not really. there were discussions in the kubernetes community about moving off slack, but they ruled out matrix basically straight away and most people who were involved in the conversation seemed to want to use discord, predictably | 19:33 |
| clarkb | ack I'll proceed with the plan from a couple of weeks ago as soon as I dig through enough of my backlog | 19:34 |
| clarkb | fungi: ya they seem to have immediately dismissed matrix as being incapable of handling their load/demand/users | 19:34 |
| clarkb | mattermost and zulip people got invovled in the discussion as well but they seem to have been dismissed too | 19:34 |
| fungi | it could be useful to dig into the size/performance concerns they raised about matrix, though for opendev's purposes we're unlikely to get anywhere near where it would start to be impacted | 19:35 |
| clarkb | ya I don't think that is a concern for us | 19:35 |
| clarkb | if they can quantify that somehow (rather than just asserting it as fact) that could be useful info generally | 19:35 |
| clarkb | #topic Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool | 19:36 |
| clarkb | glean is done | 19:36 |
| clarkb | #link https://review.opendev.org/c/openstack/diskimage-builder/+/949942 DIB functional testing without Nodepool | 19:36 |
| clarkb | this change and its children to have dib stop relying on nodepool for testing image builds is the next step. This will allow dib greater control over devstack and its nova cpu configuration so that centos 10 can be booted with all of the cpu flags it requires | 19:37 |
| clarkb | once those are in we should be able to land support for centos 10 and rocky 10 | 19:37 |
| clarkb | #link https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream | 19:37 |
| clarkb | and at this point all the related changes are ready for review. I intend on reviewing those today | 19:37 |
| clarkb | tonyb: any other concerns or things to call out? | 19:38 |
| tonyb | The series adds devstack builds along side of the nodepool ones, but the removal is more complex due to job users in other repos so I'll rework that today so we can drop the nodepool testing | 19:38 |
| tonyb | assuming that sounds fair to others | 19:38 |
| clarkb | do we define those jobs in dib? I thought we were just consuming them | 19:39 |
| clarkb | but yes we need to update glean as well to stay in sync | 19:39 |
| tonyb | we define and use jobs like dib-nodepool-funcational-openstack-$distro ; which are uses in openstacksdk | 19:40 |
| clarkb | gotcha | 19:40 |
| tonyb | I'm less worried about glean | 19:40 |
| tonyb | openstacksdk is a little harder as it has several open branches ;P | 19:41 |
| fungi | ah, yeah i guess when we pulled shade out of nodepool we wanted to test that it didn't break, and then when shade was merged into openstacksdk we kept that testing | 19:41 |
| clarkb | I wonder if the motivation from the openstacksdk side is ensuring they don't break nodepool or if they want to test the sorts of operations nodepool does | 19:41 |
| fungi | the former, i'm almost certain | 19:41 |
| clarkb | in that case cleanup is probably most appropriate at this point? | 19:42 |
| clarkb | since nodepool is going away and I don't know that zuul wants to be that tightly coupled to openstacksdk | 19:42 |
| fungi | i believe so | 19:42 |
| clarkb | but maybe there is some middle ground I'm not considering | 19:42 |
| tonyb | I'll double check with them and proceed as appropriate | 19:42 |
| clarkb | sounds good thanks | 19:42 |
| clarkb | #topic projects.yaml normalization | 19:42 |
| tonyb | I have the patches out there to switch from nodepool to devstack but cleanup would be better | 19:43 |
| corvus | yeah, i don't think that's necessary anymore | 19:43 |
| clarkb | #link https://sourceforge.net/p/ruamel-yaml/tickets/546/ | 19:43 |
| corvus | (from a zuul perspective) | 19:43 |
| clarkb | #link https://review.opendev.org/952006 | 19:43 |
| clarkb | corvus: ack thanks for confirming | 19:43 |
| clarkb | sorry I'm moving ahead since we're now 3/4 of the way through our time | 19:43 |
| clarkb | and I'm not caught up on this one so want to make sure we talk about it | 19:43 |
| corvus | ++ | 19:44 |
| clarkb | basically it seems there is some bug in ruamel and maybe we worked around it? | 19:44 |
| fungi | i believe we excluded the recent releases | 19:44 |
| clarkb | 952006 says that there is some other thing that may fix things but then unfortunately links to itself rather than the actual chagne that I think fixed things | 19:44 |
| corvus | confused by that last comment, suspect link is wrong | 19:44 |
| clarkb | corvus: yes that. This is where I got lost trying to followup on this | 19:44 |
| corvus | possibly https://review.opendev.org/c/openstack/project-config/+/952315 | 19:45 |
| clarkb | ok so basically we're not needing to do a big normalization pass beacuse we're using an old library verison which produces output consistent with what we current have | 19:45 |
| corvus | yes, also, the output of the new version is wrong | 19:46 |
| clarkb | I guess we can rollforward for now as is. Monitor the upstream bug (sourceforge!) and see if upstream is going to fix it | 19:46 |
| corvus | (so the choices we feel acceptable are (1) pin and wait/hope for them to fix; (2) do some more wrapping of ruamel output to work around it) | 19:47 |
| corvus | we chose 1... then 2 if that fails | 19:47 |
| corvus | the option of accepting the output was rejected | 19:47 |
| clarkb | got it. And ya I agree that new output is not how I would format things | 19:47 |
| corvus | the bot hasn't updated that change.. so maybe that means the pin worked? | 19:47 |
| clarkb | I suspect so. It runs daily iirc | 19:48 |
| fungi | i believe that's the reason, yes | 19:48 |
| corvus | i think contents of repo == ideal means no update is needed | 19:48 |
| corvus | we should probably abandon that change now | 19:48 |
| clarkb | ++ | 19:49 |
| corvus | done | 19:49 |
| clarkb | ok now i feel caught up | 19:49 |
| fungi | though as an unrelated bug, it became apparent that existing change == generated diff doesn't prevent it from pushing a new patchset | 19:49 |
| clarkb | fungi: we rely on gerrit to reject it? | 19:49 |
| fungi | no, commit dates are different each time so it doesn't get rejected | 19:50 |
| clarkb | gotcha so we get a new patchset each day as long as there is a delta to master | 19:50 |
| corvus | yep | 19:50 |
| fungi | fixing that is probably a trivial line or two in the script, but nobody's had time to work out the patch | 19:51 |
| clarkb | good to know. Anything else on this item? | 19:51 |
| fungi | i don't think so | 19:52 |
| clarkb | #topic Working through our TODO list | 19:52 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-january-2025-meetup | 19:52 |
| clarkb | Before my schedule went upside downI said I should formaize this better | 19:52 |
| clarkb | which is still my intention but until then here is a friendly reminder that this list exists in that etherpad | 19:52 |
| clarkb | #topic Pre PTG Planning | 19:52 |
| clarkb | which is related to the last thing on the agenda: Planning pre ptg dates | 19:53 |
| clarkb | OpenStack's next release is happening October 1, The summit is October 17-19, and the actual ptg is October 27-31 | 19:53 |
| clarkb | considering all of that I thought that October 6-10 might work for picking 2-3 days to try and have a PTG if we don't feel that October is already full of stuff | 19:53 |
| fungi | wfm | 19:54 |
| clarkb | I think it was helpful to do the january thing at the beginning of the year and doing a quick checkup before the ptg again would be useful | 19:54 |
| fungi | i've got a talk at all things open just prior to the summit, but the proposed pre-ptg dates are early enough not to conflict | 19:55 |
| clarkb | I'll probably pencil in those dates on my own calendar but please say something if that doesn't work for some reason and we can look at late september intsead potentially | 19:55 |
| tonyb | Works for me. There's a solid chance I'll be in the US at that time which possibly simplifes the time-of-day selection | 19:55 |
| clarkb | I suspect we'd do 2 or three days with blocks of a couple of hours | 19:55 |
| clarkb | very similar to what we did in january | 19:56 |
| clarkb | #topic Open Discussion | 19:56 |
| clarkb | Anything else? | 19:56 |
| fungi | i don't think i had anything | 19:56 |
| clarkb | thanks again everyone for helping out when I wasn't able to be around much. Really appreciate it | 19:57 |
| clarkb | and thank you for helping keep opendev up and running | 19:57 |
| corvus | good to have you back :) | 19:57 |
| fungi | yes, i don't mind pitching in, it's my pleasure | 19:58 |
| clarkb | sounds like that may be everything. I expect to be back here same time and location next week | 19:58 |
| clarkb | #endmeeting | 19:58 |
| opendevmeet | Meeting ended Tue Jun 24 19:58:59 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-24-19.00.html | 19:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-24-19.00.txt | 19:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-24-19.00.log.html | 19:58 |
| fungi | thnks clarkb! | 19:59 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!