19:01:27 <fungi> #startmeeting infra
19:01:27 <opendevmeet> Meeting started Tue Jun 10 19:01:27 2025 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:27 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:27 <opendevmeet> The meeting name has been set to 'infra'
19:02:47 <fungi> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HZS4FSFDLLNS4ESMUXONFWI65DRUA5IC/ Our Agenda
19:04:21 <fungi> #topic Announcements
19:04:48 <fungi> the openinfra summit europe 2025 cfp closes at the end of this week
19:04:54 <clarkb> My availability will be spotty for the next week or so due to some family stuff
19:05:11 <clarkb> thank you fungi for volunteering to chair the meeting today. Its a big help
19:05:17 <fungi> of course!
19:05:42 <fungi> also ptg signup has started, we should think about whether we (opendev collaboratory) wants to sign up to have any sessions
19:06:26 <fungi> anything else need announcing?
19:06:56 <clarkb> I didn't have anything else
19:07:29 <tonyb> I think we've had good value from the preptg model
19:07:55 <clarkb> maybe we should instead plan for something in september then?
19:08:02 <fungi> cool. fair warning my prior meeting is running over so apologies for my slowness chairing
19:08:34 <clarkb> I should be able to put together another agenda and proposals for timing if we want to do late september / very early october pre ptg option
19:08:51 <fungi> i don't think we have any pending action items from last meeting or proposed spec reviews, so i'll jump into the main topics
19:09:03 <fungi> #topic Gerrit shutdown problems (clarkb 20250527)
19:09:36 <clarkb> no updates here. Sorry
19:09:54 <fungi> no, no worries, i was just looking at where we were on this
19:09:55 <clarkb> still need to go through the motions of stopping gerrit and applying the new config for disabling extra h2 cache compaction
19:10:18 <fungi> is there a proposed change to disable that?
19:10:42 <clarkb> ya the change is up and has enough reviews. It just needs to be approved and then someone needs to restart gerrit using manual sighup to pick it up
19:10:48 <fungi> #link https://review.opendev.org/c/opendev/system-config/+/950595 Revert "Set h2.maxCompactTime to 15 seconds"
19:10:58 <fungi> looks like it's that one linked in the agenda
19:11:04 <clarkb> yup that is the one
19:11:56 <fungi> looks like i already reviewed it and then conveniently forgot, sorry
19:12:22 <fungi> #topic Gerrit 3.11 Upgrade Planning (clarkb 20250401)
19:12:57 <fungi> #link https://www.gerritcodereview.com/3.11.html Gerrit 3.11 release notes
19:12:58 <clarkb> the biggest thing here is getting more clarity on the gerrit shutdown process and then going through the relase notes and testing thigns on a held node where we have concerns
19:13:39 <clarkb> once we're sufficiently happy with the release notes and our ability to stop and start gerrit we can schedule the downtime and do the upgrade. I also like to test the downgrade process and ensure I know how to make it work but that can happen after we've scheduled the downtime as they have typically been straightforward
19:13:40 <fungi> agenda says we have a held test node at 104.130.253.194 for manual inspection as well
19:14:30 <fungi> #link https://review.opendev.org/c/opendev/system-config/+/882900 Migrate gerrit images to quay.io
19:14:32 <fungi> Change metadata
19:14:42 <fungi> that seems related too
19:14:54 <clarkb> yes though I think I've realized its more of a nice to have and less of ap riority?
19:15:09 <clarkb> we're definitely hitting fewer docker issues these days with the chagnes we've made.
19:15:23 <clarkb> but we should get to that eventually just to avoid docker hub as much as we can
19:15:45 <fungi> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
19:16:22 <fungi> #topic Upgrading Old Servers (clarkb 20230627)
19:16:24 <clarkb> unfortunately there have been a few things pullig me in different directions the last couple of weeks and I just haven't had time to focus on all the various gerrit things
19:16:26 <fungi> #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
19:16:38 <fungi> #link https://etherpad.opendev.org/p/opendev-focal-server-upgrades
19:16:50 <fungi> #link https://etherpad.opendev.org/p/opendev-server-replacement-sprint
19:17:10 <clarkb> on this topic I got testing sorted out for replacing mirror-update and zookeeper nodes with new noble nodes
19:17:15 <fungi> #link https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
19:17:47 <clarkb> and then ran into the same stuff that distracted me from gerrit things. I think the mirror-update replacement should be relatively straightforward if someone else wants to pick that up. Just need to make sure the existing server is idle or even off before we deploy the new one to avoid conflicts in afs updates
19:17:50 <tonyb> nothing new from me on the wiki update
19:18:22 <fungi> i guess let's make sure we prioritize reviews and manual testing for these things, as they impact our ability to provide continued maintenance for the affected services
19:19:13 <fungi> #topic Working through our TODO list (clarkb 20250107)
19:19:18 <fungi> #link https://etherpad.opendev.org/p/opendev-january-2025-meetup
19:19:38 <fungi> these are the priorities we agreed on at our last pre-ptg discussion
19:19:39 <clarkb> it occurs to me that as part of planning a pre ptg I should probably put that list somewhere a bit more standalone
19:19:57 <fungi> maybe an excuse to blow out the old infra specs too
19:20:01 <clarkb> ++
19:20:19 <clarkb> I think we can mark one of the items off that list now though (the gerrit hashtags update)
19:20:35 <fungi> yes!
19:21:10 <fungi> though that's also got a topic on the agenda, coming up
19:21:10 <clarkb> done. Feel free to make edist to the list to either add items or strike off ones that are done
19:21:38 <fungi> moving on in the interest of time...
19:21:48 <fungi> #topic OFTC Matrix bridge no longer supporting new users (clarkb 20250520)
19:22:01 <fungi> #link https://github.com/matrix-org/matrix-appservice-irc/issues/1851
19:22:54 <fungi> we've debated this previously, talked about moving from irc to matrix for opendev collaboratory/sysadmin discussions at least
19:23:12 <fungi> no recent changes that i'm aware, but keep it in mind
19:23:17 <clarkb> since the discussion in the meeting last week there have been strong opinions against the move. Primarily form jayf and frickler
19:24:25 <corvus> that's unfortunate
19:24:40 <clarkb> I personally still think it is worthwhile for opendev to try and do something different here in the interest of supporting users. But I'm not sure if this is a disagree and commit situation or a try to keep both systems running or a full send and see how we do situation now
19:25:32 <clarkb> I think what we've seen is that people who are really comfortable on IRC really like IRC and change (particularly needing to maintain multiple clients since IRC isn't going away and also changing workflows is difficult)
19:25:54 <fungi> sure, for now it's probably a question of whether we move to provide an(other) example
19:25:55 <clarkb> which is in direct opposition to trying to support people who are not comfortable on IRC who are using a tool like matrix to make IRC easier for them
19:25:57 <corvus> i feel like we're going to miss an opportunity to promote positive change for the community here
19:26:17 <fungi> one applicable term is "survivor bias"
19:26:23 <clarkb> and the reason I Think this move is still worthwhile is that the move was always about trying to make it easier for the people who struggle with IRC
19:27:14 <tonyb> we found the article about running a bridge, is that viable for us?
19:27:27 <fungi> the surprising objection was that matrix/element has an image of doing closed/proprietary development locking users into their platform or protocol
19:27:59 <fungi> and that it's distributed or federated only in principle, but centralized in practice
19:28:00 <tonyb> can we switch to matrix is officially the home for opendev but host a best effort bridge to IRC?
19:28:07 <corvus> fungi: that seems entirely backwards to me.  and the outcome will be the status quo of people on irc and people in slack.
19:28:58 <fungi> i agree, i'm dubious of the claims that the matrix foundation is trying to create an exclusive business model around it
19:29:16 <clarkb> tonyb: I think it is technically feasible given the infromation we have today (basically the tool exists and is minimally maintained via a fork). The biggest questions I have are what the ongoing maintenance is like as well as user moderation
19:29:32 <fungi> i personally have usability concerns, but they don't prevent me from using the platfor or protocol
19:30:08 <corvus> there are many public homeservers (not to mention private ones) which would seem to dispute that, so i don't really understand that argument
19:30:32 <JayF> My primary concern, to be clear, wasn't "we shouldn't swap from IRC", but more that Matrix doesn't represent a significant enough change to be worth the churn.
19:30:54 <fungi> right, it's on my to do list to set up a personal homeserver, but in the meantime i have a matrix.org account which is entirely usable
19:31:29 <corvus> running a public irc/matrix bridge properly would be a significant amount of work, mostly in the realm of policy (interacting with irc ops) and moderation (dealing with objectionable content or unruly users)
19:31:40 <corvus> technically, i don't think it's hard
19:32:06 <fungi> JayF: is it also a case of users who already need to continue to maintain a presence on irc for other reasons? i mean, i'm in that situation, though i have my own workarounds
19:32:56 <tonyb> corvus: ah thanks that helps  clarify the work involved
19:32:58 <JayF> I mean, that's absolutely true for me, but my concern is plainly stated as: it's easier to onboard new folks to irccloud than it is to matrix, at least so far
19:33:12 <JayF> but that also has a significant amount of reflection on *my ability to teach* as much as anything else
19:33:15 <corvus> i think it's near-zero-delta for advanced users, minor positive delta for new irc users who don't have a bouncer, major positive delta for people who would use slack otherwise.  to me, that nets out to positive delta all around.
19:33:39 <corvus> JayF: i can't see how it could be easier to on-board people to matrix.  it's literally just a username and password
19:33:55 <fungi> JayF: and is that because of needing to help them set up plumbing through the oftc bridge too, or would it apply equally to matrix-only rooms?
19:34:33 <JayF> corvus: as I stated at the time (and I don't think was communicated here when I was initally pinged in), my experience is entirely around the IRC bridge and might not be reflective
19:35:19 <fungi> yeah, it's probably worth separating "helping people use matrix" from "helping people use matrix as an irc client"
19:35:22 <corvus> JayF: okay, that is different. but that's also the status quo.  i think the discussion is more around going matrix-native, which is much simpler.
19:36:47 <fungi> what i think we're talking about here, for opendev collaboratory operations at least, is something like a native #opendev:opendev.org matrix room rather than bridging to an irc channel
19:36:54 <clarkb> as far as paths through the impasse go do we think we'd be comfortable doing a trial with the intention of sticking matrix? Basically keep the irc channel up and use it to redirect to matrix. If there are major problems we can fall back to IRC? I don't think this addresses frickler's concerns which are far more personal workflow and client related but this should give us
19:36:56 <clarkb> reasonable input on whether or not it is better for our users
19:37:00 <JayF> I think irccloud is close enough to an ideal solution for me and the folks I'm responsible for that any change for *us* would be net negative; but that's like .001% of folks :)
19:37:56 <clarkb> Unfortunately, we're always going to have differences of opinion and preference. I personally would prefer all IRC all the time. But I also recognize that this isn't ideal for many and I'm willing to compromise to make their experience better (or at least I suspect that will be the end result)
19:38:25 <clarkb> if we can at least minimally agree that that objective is worthwhile and that the change is likely to result in a benefit for those individuals then I think we should proceed
19:38:47 <fungi> i'll be clear that i have other community obligations outside of opendev/openinfra/openstack that require my presence on oftc irc anyway (particularly debian/spi), so i'm perfecty happy to operate as a meat bridge where necessary
19:39:47 <corvus> i agree with the objective, and doing a trial sounds good.  i would add that i think avoiding fragmentation is a worthwhile goal
19:40:13 <tonyb> what corvus said
19:41:16 <fungi> thankfully, it seems like none of the current situation necessitates an urgent move, but getting a more concrete proposal together might be warranted
19:41:50 <clarkb> ok I think we can probably plan to proceed then. Originally I had intended on getting things rolling this week but that seems very unlikely now. Given that yes we can write up a spec and explain what we are doing and why and then proceed from there?
19:42:09 <corvus> ++
19:42:10 <fungi> i'm happy to handle doing the room creation steps if we arrive at an agreement on something
19:42:20 <clarkb> I don't think it needs to sit in review for a long time since we've discussed it a bit but just formalize the plan and expectations. Then if we see a month later the expectations haven't been met we can reevaluate
19:42:36 <tonyb> ++
19:42:40 <fungi> right, its not irreversible
19:42:58 <tonyb> time permitting of course clarkb
19:43:03 <clarkb> ok I can take an action to write up a spec I just don't know when I can get that done by. I may be able to do it later this week or early next but it is difficult to commit to anything right now
19:43:42 <fungi> doesn't seem like there's a pressing timeline
19:44:06 <tonyb> yeah if it happens in June that's awesome
19:44:17 <clarkb> #action clarkb write up spec to trail matrix for opendev comms
19:44:25 <clarkb> this should serve as a reminder when I inevitably forget
19:44:32 <fungi> thanks!
19:44:50 <fungi> #topic Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527)
19:45:11 <fungi> i've seen some changes flying around for this
19:45:32 <clarkb> the glean change landed so we need a glean release to make that easily reconsumable
19:45:44 <tonyb> I've made some progress with the sub+devstack change but the built image isn't networking so I'm debugging that.
19:46:04 <fungi> the glean and dib changes linked in the agenda seem to already be merged
19:46:07 <clarkb> tonyb: don't forget to use config drive (glean requires it)
19:46:12 <fungi> seems like we need a dib release
19:46:29 <fungi> and a glean release?
19:46:37 <clarkb> dib isn't done yet: https://review.opendev.org/c/openstack/diskimage-builder/+/934045
19:46:46 <fungi> oh, clarkb beat me to it
19:46:55 <clarkb> but it should be safe to release glean ahead of the dib release
19:47:00 <clarkb> so we can go ahead and do that now
19:47:01 <tonyb> clarkb: yup, I am and it (glean) runs
19:47:13 <fungi> ah, yeah, the dib change is still open. reviews appreciated
19:47:26 <clarkb> tonyb: ack
19:47:45 <clarkb> mnasiadka also had a question about the centos 10 upstream image partition labels and whether dib should just workaround their buggyness
19:47:55 <clarkb> I think the issue here is that you cannot lable / as /boot they are different
19:48:11 <tonyb> yeah it's on my list for today to form an opinion on that
19:48:16 <clarkb> and while the partition may be bootable and otherwise workable dib and probably other tools rely on accurate partition labels to inspect and modify images
19:49:04 <clarkb> we would have to create a workaround that basically said if there is no / but there is a /boot and that is bootable then treat that as /boot. Which may work forever or we may run into a problem with some other partition scheme
19:49:18 <clarkb> * then treat /boot as /
19:49:27 <tonyb> I suspect that while accurate that's not going to be motivation for change in CentOS/rhel teams
19:49:48 <clarkb> ideally they would explain why they are mislabeling their partitions
19:49:57 <clarkb> rather than us try to justify why it is wrong can they justfiy why it is correct?
19:50:13 <clarkb> because I'd like to avoid carrying hacky workarounds for stuff like this for the lifetime of rhel10 which is like a decade
19:50:15 <tonyb> let's see!
19:51:00 <fungi> referring to it as "mislabeling" is a judgement (though one that i personally happen to share)
19:51:37 <clarkb> my primary concern here is that I don't want to have special cases for everyone's upstream images
19:51:57 <clarkb> there is a common spec with very a detailed labeling system that has worked for quite some time.
19:52:08 <tonyb> I don't think that CentOS(10) has a 10 year lifespan
19:52:09 <clarkb> ideally we just keep using that
19:52:19 <clarkb> tonyb: no but rhel does and presumably they do the same thing in the rhel images
19:52:39 <clarkb> if they don't then that might be an easy way to justify why centos needs to change
19:52:52 <fungi> centos 9 had a 9-year lifespan and i assume centos 11 will have an 11-year lifespan ;)
19:53:13 <clarkb> anyway we don't need to get too distracted on this particular issue. We can build centos 10 stream iamges from scratch without an upstream image base
19:53:19 <tonyb> true it'll be the same for rhel
19:53:51 <tonyb> what's the advantage to using the image over build from scratch?
19:53:51 <clarkb> so I think the focus should be on making that work for now. And the next steps there are a glean release, then fixing up tonyb's dib testing framework change so that centos 10 stream support using centos-minimal can land
19:54:09 <clarkb> tonyb: I think it is largely a preference thing. You get a bit more batteries included when you start with the upstream image
19:54:12 <tonyb> ..... actually I table that question it isn't important for the meeting
19:54:28 <fungi> i like the middle ground we started at with bootstrapping from container images
19:55:00 <fungi> #topic Open discussion
19:55:24 <fungi> we have 5 minutes remaining in the hour, but can keep talking about centos if that's everyone's passion
19:56:02 <corvus> we could talk about zuul-launcher
19:56:11 <fungi> let's!
19:56:17 <tonyb> that sounds like more fun
19:56:36 <fungi> how are things going with z-l? seems fine so far
19:57:16 <corvus> the changes that i think we need for operational control (seeing and modifying nodes and requests) are either merged or in review
19:57:35 <fungi> looks like i accidentally skipped over the z-l topic in the agenda, but i see it now, apologies for that
19:57:39 <corvus> enough have merged that i would be comfortable switching larger tenants now; the unmerged ones only add the zuul-client commond line
19:57:42 <corvus> but
19:58:06 <corvus> i noticed some node allocation algorithm problems that i think would cause us problems, so i have changes up for that
19:58:37 <corvus> in summary, i think the current set of open "hashtag:niz" changes should probably all merge before we move openstack
19:59:00 <clarkb> but once that is done we should be in good shape to move tenants?
19:59:02 <corvus> but aside from those, i don't see any blockers
19:59:06 <corvus> yep
19:59:12 <fungi> sounds good, i'll try to revisit any of those i haven't reviewed yet
19:59:40 <corvus> thanks!
20:00:50 <fungi> and we're past the top of the hour. thanks everyone! feel free to -> #opendev with any followup discussion
20:00:52 <fungi> #endmeeting