| fungi | reminder that we're meeting in 10 minutes! | 18:49 |
|---|---|---|
| fungi | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jun 10 19:01:27 2025 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| fungi | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HZS4FSFDLLNS4ESMUXONFWI65DRUA5IC/ Our Agenda | 19:02 |
| fungi | #topic Announcements | 19:04 |
| fungi | the openinfra summit europe 2025 cfp closes at the end of this week | 19:04 |
| clarkb | My availability will be spotty for the next week or so due to some family stuff | 19:04 |
| clarkb | thank you fungi for volunteering to chair the meeting today. Its a big help | 19:05 |
| fungi | of course! | 19:05 |
| fungi | also ptg signup has started, we should think about whether we (opendev collaboratory) wants to sign up to have any sessions | 19:05 |
| fungi | anything else need announcing? | 19:06 |
| clarkb | I didn't have anything else | 19:06 |
| tonyb | I think we've had good value from the preptg model | 19:07 |
| clarkb | maybe we should instead plan for something in september then? | 19:07 |
| fungi | cool. fair warning my prior meeting is running over so apologies for my slowness chairing | 19:08 |
| clarkb | I should be able to put together another agenda and proposals for timing if we want to do late september / very early october pre ptg option | 19:08 |
| fungi | i don't think we have any pending action items from last meeting or proposed spec reviews, so i'll jump into the main topics | 19:08 |
| fungi | #topic Gerrit shutdown problems (clarkb 20250527) | 19:09 |
| clarkb | no updates here. Sorry | 19:09 |
| fungi | no, no worries, i was just looking at where we were on this | 19:09 |
| clarkb | still need to go through the motions of stopping gerrit and applying the new config for disabling extra h2 cache compaction | 19:09 |
| fungi | is there a proposed change to disable that? | 19:10 |
| clarkb | ya the change is up and has enough reviews. It just needs to be approved and then someone needs to restart gerrit using manual sighup to pick it up | 19:10 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/950595 Revert "Set h2.maxCompactTime to 15 seconds" | 19:10 |
| fungi | looks like it's that one linked in the agenda | 19:10 |
| clarkb | yup that is the one | 19:11 |
| fungi | looks like i already reviewed it and then conveniently forgot, sorry | 19:11 |
| fungi | #topic Gerrit 3.11 Upgrade Planning (clarkb 20250401) | 19:12 |
| fungi | #link https://www.gerritcodereview.com/3.11.html Gerrit 3.11 release notes | 19:12 |
| clarkb | the biggest thing here is getting more clarity on the gerrit shutdown process and then going through the relase notes and testing thigns on a held node where we have concerns | 19:12 |
| clarkb | once we're sufficiently happy with the release notes and our ability to stop and start gerrit we can schedule the downtime and do the upgrade. I also like to test the downgrade process and ensure I know how to make it work but that can happen after we've scheduled the downtime as they have typically been straightforward | 19:13 |
| fungi | agenda says we have a held test node at 104.130.253.194 for manual inspection as well | 19:13 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/882900 Migrate gerrit images to quay.io | 19:14 |
| fungi | Change metadata | 19:14 |
| fungi | that seems related too | 19:14 |
| clarkb | yes though I think I've realized its more of a nice to have and less of ap riority? | 19:14 |
| clarkb | we're definitely hitting fewer docker issues these days with the chagnes we've made. | 19:15 |
| clarkb | but we should get to that eventually just to avoid docker hub as much as we can | 19:15 |
| fungi | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade | 19:15 |
| fungi | #topic Upgrading Old Servers (clarkb 20230627) | 19:16 |
| clarkb | unfortunately there have been a few things pullig me in different directions the last couple of weeks and I just haven't had time to focus on all the various gerrit things | 19:16 |
| fungi | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades | 19:16 |
| fungi | #link https://etherpad.opendev.org/p/opendev-focal-server-upgrades | 19:16 |
| fungi | #link https://etherpad.opendev.org/p/opendev-server-replacement-sprint | 19:16 |
| clarkb | on this topic I got testing sorted out for replacing mirror-update and zookeeper nodes with new noble nodes | 19:17 |
| fungi | #link https://etherpad.opendev.org/p/opendev-mediawiki-upgrade | 19:17 |
| clarkb | and then ran into the same stuff that distracted me from gerrit things. I think the mirror-update replacement should be relatively straightforward if someone else wants to pick that up. Just need to make sure the existing server is idle or even off before we deploy the new one to avoid conflicts in afs updates | 19:17 |
| tonyb | nothing new from me on the wiki update | 19:17 |
| fungi | i guess let's make sure we prioritize reviews and manual testing for these things, as they impact our ability to provide continued maintenance for the affected services | 19:18 |
| fungi | #topic Working through our TODO list (clarkb 20250107) | 19:19 |
| fungi | #link https://etherpad.opendev.org/p/opendev-january-2025-meetup | 19:19 |
| fungi | these are the priorities we agreed on at our last pre-ptg discussion | 19:19 |
| clarkb | it occurs to me that as part of planning a pre ptg I should probably put that list somewhere a bit more standalone | 19:19 |
| fungi | maybe an excuse to blow out the old infra specs too | 19:19 |
| clarkb | ++ | 19:20 |
| clarkb | I think we can mark one of the items off that list now though (the gerrit hashtags update) | 19:20 |
| fungi | yes! | 19:20 |
| fungi | though that's also got a topic on the agenda, coming up | 19:21 |
| clarkb | done. Feel free to make edist to the list to either add items or strike off ones that are done | 19:21 |
| fungi | moving on in the interest of time... | 19:21 |
| fungi | #topic OFTC Matrix bridge no longer supporting new users (clarkb 20250520) | 19:21 |
| fungi | #link https://github.com/matrix-org/matrix-appservice-irc/issues/1851 | 19:22 |
| fungi | we've debated this previously, talked about moving from irc to matrix for opendev collaboratory/sysadmin discussions at least | 19:22 |
| fungi | no recent changes that i'm aware, but keep it in mind | 19:23 |
| clarkb | since the discussion in the meeting last week there have been strong opinions against the move. Primarily form jayf and frickler | 19:23 |
| corvus | that's unfortunate | 19:24 |
| clarkb | I personally still think it is worthwhile for opendev to try and do something different here in the interest of supporting users. But I'm not sure if this is a disagree and commit situation or a try to keep both systems running or a full send and see how we do situation now | 19:24 |
| clarkb | I think what we've seen is that people who are really comfortable on IRC really like IRC and change (particularly needing to maintain multiple clients since IRC isn't going away and also changing workflows is difficult) | 19:25 |
| fungi | sure, for now it's probably a question of whether we move to provide an(other) example | 19:25 |
| clarkb | which is in direct opposition to trying to support people who are not comfortable on IRC who are using a tool like matrix to make IRC easier for them | 19:25 |
| corvus | i feel like we're going to miss an opportunity to promote positive change for the community here | 19:25 |
| fungi | one applicable term is "survivor bias" | 19:26 |
| clarkb | and the reason I Think this move is still worthwhile is that the move was always about trying to make it easier for the people who struggle with IRC | 19:26 |
| tonyb | we found the article about running a bridge, is that viable for us? | 19:27 |
| fungi | the surprising objection was that matrix/element has an image of doing closed/proprietary development locking users into their platform or protocol | 19:27 |
| fungi | and that it's distributed or federated only in principle, but centralized in practice | 19:27 |
| tonyb | can we switch to matrix is officially the home for opendev but host a best effort bridge to IRC? | 19:28 |
| corvus | fungi: that seems entirely backwards to me. and the outcome will be the status quo of people on irc and people in slack. | 19:28 |
| fungi | i agree, i'm dubious of the claims that the matrix foundation is trying to create an exclusive business model around it | 19:28 |
| clarkb | tonyb: I think it is technically feasible given the infromation we have today (basically the tool exists and is minimally maintained via a fork). The biggest questions I have are what the ongoing maintenance is like as well as user moderation | 19:29 |
| fungi | i personally have usability concerns, but they don't prevent me from using the platfor or protocol | 19:29 |
| corvus | there are many public homeservers (not to mention private ones) which would seem to dispute that, so i don't really understand that argument | 19:30 |
| JayF | My primary concern, to be clear, wasn't "we shouldn't swap from IRC", but more that Matrix doesn't represent a significant enough change to be worth the churn. | 19:30 |
| fungi | right, it's on my to do list to set up a personal homeserver, but in the meantime i have a matrix.org account which is entirely usable | 19:30 |
| corvus | running a public irc/matrix bridge properly would be a significant amount of work, mostly in the realm of policy (interacting with irc ops) and moderation (dealing with objectionable content or unruly users) | 19:31 |
| corvus | technically, i don't think it's hard | 19:31 |
| fungi | JayF: is it also a case of users who already need to continue to maintain a presence on irc for other reasons? i mean, i'm in that situation, though i have my own workarounds | 19:32 |
| tonyb | corvus: ah thanks that helps clarify the work involved | 19:32 |
| JayF | I mean, that's absolutely true for me, but my concern is plainly stated as: it's easier to onboard new folks to irccloud than it is to matrix, at least so far | 19:32 |
| JayF | but that also has a significant amount of reflection on *my ability to teach* as much as anything else | 19:33 |
| corvus | i think it's near-zero-delta for advanced users, minor positive delta for new irc users who don't have a bouncer, major positive delta for people who would use slack otherwise. to me, that nets out to positive delta all around. | 19:33 |
| corvus | JayF: i can't see how it could be easier to on-board people to matrix. it's literally just a username and password | 19:33 |
| fungi | JayF: and is that because of needing to help them set up plumbing through the oftc bridge too, or would it apply equally to matrix-only rooms? | 19:33 |
| JayF | corvus: as I stated at the time (and I don't think was communicated here when I was initally pinged in), my experience is entirely around the IRC bridge and might not be reflective | 19:34 |
| fungi | yeah, it's probably worth separating "helping people use matrix" from "helping people use matrix as an irc client" | 19:35 |
| corvus | JayF: okay, that is different. but that's also the status quo. i think the discussion is more around going matrix-native, which is much simpler. | 19:35 |
| fungi | what i think we're talking about here, for opendev collaboratory operations at least, is something like a native #opendev:opendev.org matrix room rather than bridging to an irc channel | 19:36 |
| clarkb | as far as paths through the impasse go do we think we'd be comfortable doing a trial with the intention of sticking matrix? Basically keep the irc channel up and use it to redirect to matrix. If there are major problems we can fall back to IRC? I don't think this addresses frickler's concerns which are far more personal workflow and client related but this should give us | 19:36 |
| clarkb | reasonable input on whether or not it is better for our users | 19:36 |
| JayF | I think irccloud is close enough to an ideal solution for me and the folks I'm responsible for that any change for *us* would be net negative; but that's like .001% of folks :) | 19:37 |
| clarkb | Unfortunately, we're always going to have differences of opinion and preference. I personally would prefer all IRC all the time. But I also recognize that this isn't ideal for many and I'm willing to compromise to make their experience better (or at least I suspect that will be the end result) | 19:37 |
| clarkb | if we can at least minimally agree that that objective is worthwhile and that the change is likely to result in a benefit for those individuals then I think we should proceed | 19:38 |
| fungi | i'll be clear that i have other community obligations outside of opendev/openinfra/openstack that require my presence on oftc irc anyway (particularly debian/spi), so i'm perfecty happy to operate as a meat bridge where necessary | 19:38 |
| corvus | i agree with the objective, and doing a trial sounds good. i would add that i think avoiding fragmentation is a worthwhile goal | 19:39 |
| tonyb | what corvus said | 19:40 |
| fungi | thankfully, it seems like none of the current situation necessitates an urgent move, but getting a more concrete proposal together might be warranted | 19:41 |
| clarkb | ok I think we can probably plan to proceed then. Originally I had intended on getting things rolling this week but that seems very unlikely now. Given that yes we can write up a spec and explain what we are doing and why and then proceed from there? | 19:41 |
| corvus | ++ | 19:42 |
| fungi | i'm happy to handle doing the room creation steps if we arrive at an agreement on something | 19:42 |
| clarkb | I don't think it needs to sit in review for a long time since we've discussed it a bit but just formalize the plan and expectations. Then if we see a month later the expectations haven't been met we can reevaluate | 19:42 |
| tonyb | ++ | 19:42 |
| fungi | right, its not irreversible | 19:42 |
| tonyb | time permitting of course clarkb | 19:42 |
| clarkb | ok I can take an action to write up a spec I just don't know when I can get that done by. I may be able to do it later this week or early next but it is difficult to commit to anything right now | 19:43 |
| fungi | doesn't seem like there's a pressing timeline | 19:43 |
| tonyb | yeah if it happens in June that's awesome | 19:44 |
| clarkb | #action clarkb write up spec to trail matrix for opendev comms | 19:44 |
| clarkb | this should serve as a reminder when I inevitably forget | 19:44 |
| fungi | thanks! | 19:44 |
| fungi | #topic Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) | 19:44 |
| fungi | i've seen some changes flying around for this | 19:45 |
| clarkb | the glean change landed so we need a glean release to make that easily reconsumable | 19:45 |
| tonyb | I've made some progress with the sub+devstack change but the built image isn't networking so I'm debugging that. | 19:45 |
| fungi | the glean and dib changes linked in the agenda seem to already be merged | 19:46 |
| clarkb | tonyb: don't forget to use config drive (glean requires it) | 19:46 |
| fungi | seems like we need a dib release | 19:46 |
| fungi | and a glean release? | 19:46 |
| clarkb | dib isn't done yet: https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 19:46 |
| fungi | oh, clarkb beat me to it | 19:46 |
| clarkb | but it should be safe to release glean ahead of the dib release | 19:46 |
| clarkb | so we can go ahead and do that now | 19:47 |
| tonyb | clarkb: yup, I am and it (glean) runs | 19:47 |
| fungi | ah, yeah, the dib change is still open. reviews appreciated | 19:47 |
| clarkb | tonyb: ack | 19:47 |
| clarkb | mnasiadka also had a question about the centos 10 upstream image partition labels and whether dib should just workaround their buggyness | 19:47 |
| clarkb | I think the issue here is that you cannot lable / as /boot they are different | 19:47 |
| tonyb | yeah it's on my list for today to form an opinion on that | 19:48 |
| clarkb | and while the partition may be bootable and otherwise workable dib and probably other tools rely on accurate partition labels to inspect and modify images | 19:48 |
| clarkb | we would have to create a workaround that basically said if there is no / but there is a /boot and that is bootable then treat that as /boot. Which may work forever or we may run into a problem with some other partition scheme | 19:49 |
| clarkb | * then treat /boot as / | 19:49 |
| tonyb | I suspect that while accurate that's not going to be motivation for change in CentOS/rhel teams | 19:49 |
| clarkb | ideally they would explain why they are mislabeling their partitions | 19:49 |
| clarkb | rather than us try to justify why it is wrong can they justfiy why it is correct? | 19:49 |
| clarkb | because I'd like to avoid carrying hacky workarounds for stuff like this for the lifetime of rhel10 which is like a decade | 19:50 |
| tonyb | let's see! | 19:50 |
| fungi | referring to it as "mislabeling" is a judgement (though one that i personally happen to share) | 19:51 |
| clarkb | my primary concern here is that I don't want to have special cases for everyone's upstream images | 19:51 |
| clarkb | there is a common spec with very a detailed labeling system that has worked for quite some time. | 19:51 |
| tonyb | I don't think that CentOS(10) has a 10 year lifespan | 19:52 |
| clarkb | ideally we just keep using that | 19:52 |
| clarkb | tonyb: no but rhel does and presumably they do the same thing in the rhel images | 19:52 |
| clarkb | if they don't then that might be an easy way to justify why centos needs to change | 19:52 |
| fungi | centos 9 had a 9-year lifespan and i assume centos 11 will have an 11-year lifespan ;) | 19:52 |
| clarkb | anyway we don't need to get too distracted on this particular issue. We can build centos 10 stream iamges from scratch without an upstream image base | 19:53 |
| tonyb | true it'll be the same for rhel | 19:53 |
| tonyb | what's the advantage to using the image over build from scratch? | 19:53 |
| clarkb | so I think the focus should be on making that work for now. And the next steps there are a glean release, then fixing up tonyb's dib testing framework change so that centos 10 stream support using centos-minimal can land | 19:53 |
| clarkb | tonyb: I think it is largely a preference thing. You get a bit more batteries included when you start with the upstream image | 19:54 |
| tonyb | ..... actually I table that question it isn't important for the meeting | 19:54 |
| fungi | i like the middle ground we started at with bootstrapping from container images | 19:54 |
| fungi | #topic Open discussion | 19:55 |
| fungi | we have 5 minutes remaining in the hour, but can keep talking about centos if that's everyone's passion | 19:55 |
| corvus | we could talk about zuul-launcher | 19:56 |
| fungi | let's! | 19:56 |
| tonyb | that sounds like more fun | 19:56 |
| fungi | how are things going with z-l? seems fine so far | 19:56 |
| corvus | the changes that i think we need for operational control (seeing and modifying nodes and requests) are either merged or in review | 19:57 |
| fungi | looks like i accidentally skipped over the z-l topic in the agenda, but i see it now, apologies for that | 19:57 |
| corvus | enough have merged that i would be comfortable switching larger tenants now; the unmerged ones only add the zuul-client commond line | 19:57 |
| corvus | but | 19:57 |
| corvus | i noticed some node allocation algorithm problems that i think would cause us problems, so i have changes up for that | 19:58 |
| corvus | in summary, i think the current set of open "hashtag:niz" changes should probably all merge before we move openstack | 19:58 |
| clarkb | but once that is done we should be in good shape to move tenants? | 19:59 |
| corvus | but aside from those, i don't see any blockers | 19:59 |
| corvus | yep | 19:59 |
| fungi | sounds good, i'll try to revisit any of those i haven't reviewed yet | 19:59 |
| corvus | thanks! | 19:59 |
| fungi | and we're past the top of the hour. thanks everyone! feel free to -> #opendev with any followup discussion | 20:00 |
| fungi | #endmeeting | 20:00 |
| opendevmeet | Meeting ended Tue Jun 10 20:00:52 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-10-19.01.html | 20:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-10-19.01.txt | 20:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-06-10-19.01.log.html | 20:00 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!