19:00:06 <clarkb> #startmeeting infra
19:00:06 <opendevmeet> Meeting started Tue Aug  5 19:00:06 2025 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:06 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:06 <opendevmeet> The meeting name has been set to 'infra'
19:00:08 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/GPTGNBBSJONHY3AZX64L5HECQ2YZYXB7/ Our Agenda
19:00:14 <clarkb> #topic Announcements
19:00:19 <clarkb> I didn't have anything to announce. Did anyone else?
19:00:33 <clarkb> I guess I can note the service coordinator nomination period is open now
19:00:38 <clarkb> we'll discuss that in more depth later in the meeting
19:02:42 <fungi> i didn't, no
19:03:03 <clarkb> #topic Gerrit 3.11 Upgrade Planning
19:03:23 <clarkb> Nothing really new here. I had hoped to start testing upgrade things with our latest image builds last week but then got distracted by other things
19:03:37 <clarkb> it is still relatively high on my todo list though so I'm hoping to have updates soon
19:03:47 <clarkb> #link https://www.gerritcodereview.com/3.11.html Gerrit 3.11 Changelog
19:03:55 <clarkb> #link https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 Held Nodes A
19:04:03 <clarkb> #link https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 Held Nodes B
19:04:12 <clarkb> and these are useful links if anyone else wants to look into it a bit more too
19:04:27 <clarkb> any questions/concerns/comments around Gerrit 3.11 upgrades before we move on?
19:06:08 <clarkb> #topic Upgrading old servers
19:06:12 <clarkb> There are updates here
19:06:27 <clarkb> fungi replaced eavesdrop01 with eavesdrop02 updating it to Noble
19:06:53 <clarkb> there were some minor hiccups during the replacement but overall things went smoothly
19:07:13 <clarkb> fungi: is the old server still running? And if so when do you think we should consider cleaning it up?
19:07:45 <fungi> still up at the moment
19:07:56 <fungi> but yes needs cleaning up along with dns
19:08:11 <fungi> i was going to do it this week
19:08:24 <clarkb> sounds good. Any thing else to note on the eavesdrop server replacement?
19:08:51 <fungi> other than me forgetting to add the acme cname initially, it went fine
19:09:10 <clarkb> yup the actual data recording outage ended up being pretty minimal as planned
19:09:10 <fungi> the cinder volume name still references eavesdrop01 of course
19:09:18 <clarkb> the ssl cert problem only affected reading the logs back out again
19:09:34 <clarkb> I wonder if you can rename a volume
19:09:56 <fungi> we can do a seamless replacement later if we want, using pvmove
19:10:03 <clarkb> good point
19:10:08 <fungi> but it's mainly cosmetic
19:10:38 <clarkb> fungi also announced the plan to shutdown refstack and retire its git repos
19:10:45 <clarkb> #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/WNI4PE2TZ3G52C3U5FT2YNVRUAJB3CMO/
19:10:54 <corvus> google suggests "openstack volume set --name" can rename but that should be verified
19:10:54 <clarkb> I don't think any of those changes have actually happend yet but announcing that they will happen is a good start
19:11:01 <fungi> yeah, side note, turns out refstack crashed weeks ago and nobody noticed
19:11:13 <corvus> (maybe worth a try before doing pvmove)
19:11:19 <fungi> so it's already down
19:11:53 <clarkb> ha confirming that shutting it down is the right course of action
19:12:14 <clarkb> fungi: is there anything you need to continue to make progress on the refstack cleanup?
19:12:22 <fungi> nah
19:12:54 <clarkb> thank you for pushing these along. It helps a lot when we're all chipping away at the big list of server upgrades
19:12:58 <fungi> i just wanted to give the announcement thread a few days before i stop/snapshot/delete the server
19:13:28 <clarkb> anything else on the larger server upgrades topic?
19:14:17 <clarkb> #topic Matrix for OpenDev comms
19:14:24 <clarkb> #link https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing
19:14:37 <clarkb> I've gotten a couple more reviews now. I'll plan to make some updates and push a new patchset this week in response to those
19:15:11 <clarkb> thank you for that. I haven't really seen any feedback that diverges from what we've already talked about. But some good ideas on tightening up the plan
19:15:52 <clarkb> we can keep the discussion largely in Gerrit so that we don't have diverging threads. I just want to keep calling attention to the spec here while it is getting iterated on
19:16:00 <clarkb> #topic Working through our TODO list
19:16:05 <clarkb> #link https://etherpad.opendev.org/p/opendev-running-todo-list
19:16:20 <clarkb> we have a new dedicated etherpad that is also linked to from the front page of our specs site
19:16:31 <clarkb> https://docs.opendev.org/opendev/infra-specs/latest/#todo-list
19:16:41 <corvus> oh matrix
19:17:07 <corvus> ianw suggested looking into mjonlr
19:17:10 <clarkb> Idea is to capture things on our radar that we either haven't been able to get to yet or are slowly making progress on and don't want to forget
19:17:16 <clarkb> corvus: ya I was going to look at the options there then update the spec
19:17:38 <clarkb> I thought we were already subscribed to some sort of global moderation system but maybe this is an extra layer on top of that?
19:17:42 <corvus> ok... cool... yeah, that might change the work item stream.
19:17:56 <corvus> i think maybe they just provide access to it and we have to start using it
19:18:08 <clarkb> got it. I'll dig in and see what I find
19:18:12 <corvus> sounds good
19:18:16 <corvus> thanks!
19:18:30 <clarkb> on the todo list front feel free to update that document with other things you're aware of that aren't already captured
19:18:38 <clarkb> or mark things done if I've missed things getting completed
19:19:06 <clarkb> the idea is to have an easy system to acpture stuff at a very high level so that we don't completely forget but also give people a thread to pull on if/when they are ready to start the next thing (which may lead to writing a spec etc)
19:19:40 <clarkb> and related to that is pre PTG planning
19:19:42 <clarkb> #topic Pre PTG Planning
19:19:45 <clarkb> #link https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document
19:19:59 <clarkb> I've added some real content ideas there and everyone should feel free to add more
19:20:24 <clarkb> One thing I wanted feedback on as we get closer is how much time do we think we can dedicate to this in october and what time of day are we thinking?
19:20:51 <clarkb> I was thinking may be 2 hours a day tuesday through thursday and if we run out of discussion topics we can cancel the later blocks of time
19:21:35 <clarkb> tonyb: will you be in an au timezone or somewhere else? Wondering if we should try to have an early and late session or maybe something around 1900 UTC which may "work" (for some value of work) for everyone?
19:22:10 <clarkb> we are about 2 months away now so plenty of time to figure that out, but I don't want to forget that we need a more concrete set of times so am starting that discussion now
19:22:23 <fungi> should be doable for me
19:23:19 <clarkb> if I don't hear and specific feedback I'll probably just pick some times in a couple of weeks then see if anyone complains :)
19:23:36 <clarkb> so ya get back to me on what times you expect to work for you and I'll see what I can do schedule wise
19:23:43 <clarkb> #topic Service Coordinator Election Planning
19:23:49 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/YXRD23ZWJGDPZ3WESBNZNEYO7NBCXFT4/
19:23:59 <clarkb> Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. Is the summary
19:24:04 <clarkb> this has all been made official on the mailing list now
19:25:06 <clarkb> I am more than happy for someone else to step into the role and can act as backup and contineu to support whoever it may be
19:25:39 <clarkb> let me know if you have any questions about what is involved
19:26:20 <clarkb> #topic Loss of upstream Debian bullseye-backports mirror
19:26:30 <clarkb> #link https://review.opendev.org/956497 temporarily ignore undefinedtarget
19:26:38 <clarkb> fungi: want to catch us up on this one?
19:27:23 <fungi> yeah...
19:27:58 <fungi> so, the backports suite in debian for bullseye was eol for a while, finally removed from the upstream mirrors a couple weeks ago
19:28:20 <fungi> this broke our reprepro mirroring of course, so there was a change to remove it from the config
19:28:51 <fungi> unfortunately, because it still persists in the reprepro database, it's treated as an error causing the script to abort
19:29:15 <fungi> in the short term we can work around it by telling reprepro to ignore that error and get the mirror back up to date
19:29:19 <clarkb> fungi: so we've already removed it from the config?
19:29:27 <fungi> yes, in git
19:29:41 <fungi> the script change you linked above refers to the earlier config change
19:29:51 <clarkb> so I guess we need to run the steps we have documented for cleaning up targets like that? Similar to what we would do after the changes to cleanup xenial land?
19:30:27 <clarkb> in the case of xenial we probably won't get an error because ubuntu isn't removing that content from the mirrors I guess. But I suspect the cleanup fixup steps are similar/the same?
19:30:27 <fungi> we could instead just purge the bullseye-backports suite, but that's going to cause jobs which are using bullseye with the configure-mirror role to start breaking unless they override its default
19:31:20 <clarkb> got it. Do we know if any actually depend on that?
19:31:38 <clarkb> just thinking that bullseye is old enough and backports are uncommon enough that maybe it is a non issue?
19:31:44 <fungi> i highly doubt they're installing packages from it
19:31:53 <clarkb> and if it is an issue maybe that is an indication the jobs should be updated
19:32:06 <fungi> or the default
19:32:42 <fungi> #link https://opendev.org/zuul/zuul-jobs/src/commit/06098ab/roles/configure-mirrors/defaults/main.yaml#L14
19:32:52 <clarkb> I'm ok with that workaround change and I've +2'd it. But I suspect we can probably just proceed with clearing out backports as the longer term solution
19:33:22 <fungi> as we've discussed before, that role enables the backports repo in all jobs that use it, unless they explicitly set `configure_mirrors_extra_repos: False` as an override
19:33:24 <clarkb> fungi: I think we can set a site vars override for that
19:33:41 <clarkb> then you don't have to do it on a per job basis
19:33:46 <fungi> one possibility that could be less intrusive is to make the default version-dependent
19:33:55 <clarkb> or we could set it in the base jobs that configure mirrors I guess which is only a small number of locations
19:34:18 <clarkb> fungi: well I'm thinking that we should set it to false because debian doesn't keep the packages up for long enough
19:34:37 <clarkb> rather than make it version dependent just force things to opt in and then deal with the fallout when the packlages go away in those locations
19:34:50 <fungi> yeah, i just wonder if (and this is more a topic for zuul discussion venues) that role's default should be rethought
19:35:24 <corvus> there's a completely different way of configuring mirrors already specced out
19:35:28 <corvus> but no one is working on it
19:35:52 <corvus> https://zuul-ci.org/docs/zuul-jobs/latest/mirror.html
19:36:06 <clarkb> ya that appraoch is far more specific and less convention based. Definitely a good thing to pick up if someone has time
19:37:29 <clarkb> so for next steps I think we can land the workaround to start syncing packages again, then concurrently flip the var value in opendev one way or another, then finally cleanup the old packages db and revert the workaround
19:37:43 <clarkb> if that seems reaosnable i think fungi can go ahead and +A the workaround
19:38:37 <fungi> yeah, can do
19:39:05 <fungi> this had enough moving parts i didn't want to go making too many changes before we had a chance to talk through the options
19:39:48 <clarkb> #topic Adding Ansible 11 to Zuul
19:40:07 <clarkb> Zuul is growing support for Ansible 11 (this may already be deployed in opendev as of our friday restarts?) and losing support for ansible 8
19:40:19 <corvus> yep is deployed
19:40:34 <corvus> #link ansible 11 test https://review.opendev.org/956490
19:40:40 <corvus> that initial sanity check came back green
19:41:01 <corvus> so do we want to go ahead and set the tenant defaults for opendev and zuul to 11?
19:41:11 <clarkb> I'm open to try that and see what if anything breaks
19:41:48 <clarkb> have we run into any problems at all yet?
19:41:54 <clarkb> (just wondering if there are known gotchas)
19:42:23 <corvus> nothing yet; i don't expect the changes to impact us (or probably almost anyone)
19:42:29 <corvus> but nobody expects, etc...
19:42:51 <clarkb> then ya I think the next step is to toggle over a tenant or two and exercise it
19:43:20 <corvus> cool, i'll propose a change
19:43:41 <clarkb> #topic Gitea 1.24.4 Upgrade
19:43:46 <clarkb> #link https://github.com/go-gitea/gitea/blob/v1.24.4/CHANGELOG.md
19:43:54 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/956591
19:44:19 <clarkb> this appears to be a fairly straight update for us. I suggested earlier today that we land fungi's opendev.org main page update first then land this to upgrade the service itself
19:44:39 <fungi> yeah, i'll take a look after the meeting
19:44:42 <clarkb> the main page update has enough reviews now but my upgrade change could use some reviews
19:44:45 <clarkb> thanks!
19:45:03 <clarkb> so ya not a major item but wanted to bring it up as I think we can get it done and rolled out fairly quickly
19:45:10 <clarkb> #topic Etherpad 2.4.2 Upgrade
19:45:17 <clarkb> This change however is probably a bit more involved
19:45:21 <clarkb> #link https://github.com/ether/etherpad-lite/blob/v2.4.2/CHANGELOG.md
19:45:33 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/956593
19:45:46 <fungi> i love the bug fix "fix various bugs"
19:45:50 <clarkb> in particular they've added dark mode support that I guess toggles based on your browser settings
19:46:04 <fungi> (that was about the gitea upgrade)
19:46:05 <clarkb> and they added a home button to the toolbar list which I'm not entirely sure what it does
19:46:36 <clarkb> so anyway I've held a node looks like it is 158.69.64.117
19:46:49 <clarkb> we'll want to edit local /etc/hosts to point etherpad.opendev.org to that IP and test out these new UI updates
19:47:52 <clarkb> reviews welcome as well as any feedback on the ui updates
19:48:05 <clarkb> #topic Open Discussion
19:48:19 <clarkb> I did want to call out I've got a couple changes up to clear out xenial mirror content and bionic arm64 content
19:48:43 <clarkb> but I'm thinking now the priority is probably in fixing the bullseye situation so that we've got a stable and happy reprepro before we start trying to modify the ubuntu content?
19:50:16 <clarkb> Anything else?
19:52:11 <clarkb> sounds like no. Thank you for your time and help everyone! Let me know about timing for pre ptg schedule blocks and consider running for service coordinator
19:52:17 <clarkb> we'll be back here next week at the asme time and location
19:52:23 <clarkb> #endmeeting