19:00:09 <clarkb> #startmeeting infra
19:00:09 <opendevmeet> Meeting started Tue Apr 23 19:00:09 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:09 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:09 <opendevmeet> The meeting name has been set to 'infra'
19:00:16 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/WBD5BLMI6ZPKUB6FOSZ65P7P5RR7CHFS/ Our Agenda
19:00:31 <clarkb> #topic Announcements
19:00:46 <clarkb> I have no announcements. Did I miss anything announcement worthy?
19:01:24 <fungi> i don't think so
19:01:43 <clarkb> #topic Upgrading Old Servers
19:02:13 <clarkb> haven't seen anything on this subject over the last week. I did have a mini panic when I thought Focal was EOL this month but it has another 12 months before that happens
19:02:57 <fungi> it's been brought up that wiki.o.o is still very far behind and its ua/esm subscription expired a while ago
19:03:18 <fungi> i'm trying to work out where that discussion with canonical stalled
19:03:39 <clarkb> thanks. I think the best case short term there is to renew the esm sub if possible and then contineu to look at a replacement
19:03:58 <fungi> but rebuilding it on a newer system, if anyone has time, would address both that and the impending ssl cert expiration for it
19:04:25 <clarkb> ya though I'm happy to deal with the cert in a few weeks (which is on the agenda for later)
19:05:47 <clarkb> #topic MariaDB Upgrades
19:06:10 <clarkb> Etherpad and mailman have new shiny upgraded mariadbs now. Gerrit and Gitea are the remaining services that need similar treatment
19:06:29 <clarkb> After thinking about it a bit due to the ease of upgrades for everything else I'm kidna thinking we may get away with just sending it for gitea
19:07:05 <clarkb> CI should check general compatibility with newer mariadb and if that checks out then the upgrade process itself has been pretty reliable so far and we can probably expect it to continue to be reliable
19:07:36 <clarkb> I'll push up a change for gitea and if anyone is concerned about that we can post those concerns in review
19:07:57 <clarkb> On the gerrit side of the house we will have to upgrade the db manually since the current ansible is very hands off with container management (and we want it to be)
19:08:18 <clarkb> But I do think we can just pick a time for gerrit and get it done since gerrit's db isn't very critical and the actual downtime should be quite short
19:08:55 <fungi> i agree
19:09:18 <clarkb> in that case I guess I should also push up a change to reflect the update that will ultimately be somewhat manual in gerrit as well
19:09:55 <clarkb> #topic AFS Mirror Cleanups
19:10:24 <clarkb> I haven't made any new progress on Xenial cleanup. I think the scope of it is daunting enough that I'm happy to keep kicking it down the road if I can :) but I should just dive in and see what I learn
19:11:26 <clarkb> #topic Building Ubuntu Noble Nodes
19:11:42 <clarkb> https://review.opendev.org/c/opendev/glean/+/915907 merged which adds python3.12 support to glean
19:11:56 <fungi> official release day for noble is thursday this week, right?
19:11:57 <clarkb> I believe that our image builds use glean from releases by default though
19:12:05 <frickler> fungi: yes
19:12:23 <clarkb> This means we should plan to make a glean release. I can do that probably on Thursday ish as well
19:12:39 <clarkb> once we have a release we then need to monitor our next image builds but we have decent coverage of glean in CI so not too worried
19:12:55 <clarkb> #link https://review.opendev.org/c/zuul/nodepool/+/916053 Update Nodepool debootstrap to support Noble
19:13:08 <clarkb> separately we need to update Nodepool images to add support to debootstrap for noble
19:13:25 <clarkb> corvus: ^ not sure if you have any input on that, but I think it does have the votes it needs so maybe we just go ahead and approve it now
19:14:06 <corvus> sgtm; i'll add a general +2 (i haven't reviewed the details)
19:14:12 <clarkb> ack thanks
19:14:35 <clarkb> Good news is there is solid progress here. Thank you frickler for testing it out and finding where the initial issues are
19:14:41 <frickler> 3rd patch needed will be https://review.opendev.org/c/openstack/project-config/+/916050
19:16:02 <clarkb> once we've got these three items sorted out we should be able to start building images and/or adding mirroring
19:16:30 <fungi> looks like those changes are approved now
19:16:54 <fungi> so just need the glean tag (assuming things merge/promote successfully)
19:17:09 <clarkb> yup and I've got that on my todo list now
19:17:18 <fungi> awesome, thanks
19:17:44 <clarkb> I'm not committing to doing it immediately because I'm not sure what i need to do to get my gpg key out of cold storage and usable again
19:17:56 <frickler> there is also a testing patch in dib fwiw https://review.opendev.org/c/openstack/diskimage-builder/+/915915
19:18:25 <frickler> I'll update that once the glean release is done
19:18:28 <clarkb> sounds good
19:18:39 <clarkb> #topic Etherpad 2.0.x Upgrade
19:19:26 <clarkb> I can't remember the state of this during our last meeting, but I got the unreleased state of Etherpad to work with new api auth methods and added a bunch of testing to ensure that our admin tasks don't regress with the new setup
19:19:57 <clarkb> Everything seems to be working which means that as soon as etherpad makes a release we should be read to upgrade. I don't want to upgrade until a release is done just in case anything else changes
19:20:29 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/914119 Etherpad updates if interested
19:20:40 <clarkb> that chagne has all the new testing and docs and so on if you are interested in taking an early look at it
19:20:51 <clarkb> #topic Add reference to the project-team-guide in the fail ci msg
19:21:00 <clarkb> #link https://review.opendev.org/914189
19:21:07 <clarkb> fungi: you added this topic want to fill us in?
19:21:21 <fungi> this was more of a concern with blurring lines between opendev and openstack
19:21:31 <fungi> see my comments on the linked review
19:22:15 <clarkb> gotcha, I think I'm ok with adding that to the openstack tenant and having those of us still in the openstack tenant that probably shouldn't deal with it
19:22:15 <fungi> basically, are we (collectively) okay having failure results in the openstack zuul tenant linking to openstack-specific guidance/documentation?
19:22:39 <clarkb> and if anyone has questions or concerns we can point them in the direction of using a different tenant
19:23:04 <clarkb> (just to be clear that I don't think we should add all the possible debug doc links to the openstack tenant, more that we're in the openstack tenant so openstack things are probably ok)
19:23:16 <fungi> normally i'd say yes, but the openstack tenant was our original/default tenant and is still used by a lot of non-openstack projects as a result
19:23:25 <frickler> I agree that this may be a good opportunity to progress with tenant separation if needed
19:23:57 <clarkb> its also something the vast majority of people are unlikely to read unfornately
19:24:04 <corvus> i agree with clarkb and frickler
19:24:10 <clarkb> I know I click the zuul summary button myself :)
19:24:15 <fungi> i'm okay approving it, but wanted to make sure there weren't major objections
19:24:24 <frickler> the openstack guidelines don't sound too unreasonable for other projects I'd say
19:24:37 <fungi> right, they're pretty general and not bad
19:24:59 <fungi> i'm just wary of setting new precedent without further consultation
19:25:04 <clarkb> I'll +2 it after the meeting to make my stance clear in review (and to record it better)
19:25:07 <frickler> thx for mentioning the patch, though, I had missed it somehow
19:25:12 <fungi> sounds good, thanks
19:25:41 <clarkb> #topic Gerrit 3.9 Upgrade Planning
19:26:04 <clarkb> We're in a position now where I think we could upgrade Gerrit tomorrow if we wanted to
19:26:22 <clarkb> Gerrit 3.9 reindexing should work, our images are updated to pull that fix in, etc
19:26:30 <clarkb> #link https://www.gerritcodereview.com/3.9.html Release Notes
19:26:54 <clarkb> there are some intentional breaking changes though as noted in the release notes. None of them really impact us from what I can tell.
19:27:06 <clarkb> There is also a very straightforward downgrade path back to 3.8 if necessary
19:27:47 <fungi> i'm up for a tomorrow upgrade ;)
19:27:51 <clarkb> All that to say our next steps here should be to write up a document to track things like downgrade testing and breaking change review. Then we can schedule a day for the upgrade and get it done. I'll volunteer to get that document started and we can coordinate from there
19:28:02 <clarkb> to be clear I don't think it will happen tomorrow :) just that the major known issues have been addressed :)
19:28:10 <fungi> yeah, got it
19:28:53 <clarkb> I also want us to keep in mind that we should start thinking about a gerrit server upgrade. Historically we've tried to give people some notice of these in order to update their firewall rules. THough honestly with the number of old clients attempting to authenticate and failing it may be a good idea to make it a surprise :)
19:28:56 <fungi> also my availability for the first half of next month is pretty limited, so if i'm going to help it will either be next week or several weeks out
19:29:05 <clarkb> several weeks out is probably fine
19:29:15 <clarkb> since there is prep work to get done and plenty of other stuff to do as well
19:30:12 <clarkb> Talking out loud here I think our longer term planning should be something like upgrade to 3.9 in Mayish, upgrade the underlying server to noble during the summer sometime, then upgrade to 3.10 in November/December/January time frame
19:30:46 <clarkb> One other thing to note that doesn't really affect us since we've already done the work is JGit just proposed that they will drop java 11 support
19:30:56 <clarkb> We're already on java 17 so that is fine for us
19:31:05 <frickler> would we consider moving away from vexxhost with the server? like we still don't have feedback about the unplanned shutdown yet? also recurring network issues
19:31:56 <clarkb> Considering the alternative options I'm not sure there is a clear cut winner in hosting. However, that is a good question and somethign we should keep in mind as we plan for server replacement
19:32:20 <clarkb> vexxhost in particular lets us run a large server which the jvm running gerrit seems to appreciate
19:33:24 <frickler> I would be seeing ovh as possible option, maybe we can the the flavor question with them if there is interest in that
19:33:27 <clarkb> Let's bring that back up after we've upgraded to 3.9 and start planning the server replacement. Ideally we'd be able to work with vexxhost to address the concerns
19:33:44 <frickler> s/the/address/
19:35:08 <clarkb> #topic Wiki Cert Renewal
19:35:24 <clarkb> In the past I've renewed this cert when there has been a week or less of validity
19:35:56 <clarkb> the current cert expires May 18, 2024. Which means renewing it sometime after May 11
19:36:12 <clarkb> I'm happy to do that, but didn't want others worried it was ignored as we get the daily warning email
19:36:26 <fungi> ftr i'll be out of town, though it's a straight-up file replacement and apache2 restart
19:37:02 <frickler> just noticing that iirc I still have no login for that server, so could not help right away if needed
19:37:39 <clarkb> frickler: thats a good call out. I suspect tonyb doesn't either. I can probably manually address that too when dealing with the cert
19:38:01 <tonyb> I do. fwiw
19:38:11 <clarkb> ah ok
19:38:12 <fungi> i'm happy to add/update ssh keys in a few minutes
19:38:16 <clarkb> fungi: thanks!
19:38:34 <fungi> i can just copy/merge from a properly config-managed server, i expect
19:38:40 <tonyb> I'm wondering if we can build a new server in that opendev domain to get LE so this becomes less of a problem
19:38:49 <frickler> note there's a review with a new key up
19:38:58 <tonyb> ..... as part of the server upgrades away from bionic
19:39:05 <fungi> tonyb: the domain isn't the problem, just need ansible working
19:39:20 <tonyb> ahh okay.  nevermind
19:39:31 <fungi> but yes, if we can get it onto a newer ubuntu version, easily solved
19:39:33 <tonyb> we'll get that "soon"
19:39:50 <clarkb> In any case I wanted to make sure that others know the deadline here is known and that I intend on addressing it manually
19:40:01 <frickler> +1
19:40:13 <clarkb> Normally I would ask for volunteers but it involves paying money and filign an expense report and I don't expect others to do that here
19:40:46 <clarkb> side note: The cost of a cert and the cost of a beer have converged and in the wrong direction
19:41:00 <fungi> fingers crossed this is the last year you expense a cert on domaincheap
19:41:20 <clarkb> #topic Open Discussion
19:41:21 <fungi> so you can focus on expensing beer
19:41:24 <clarkb> Anything else?
19:41:38 <tonyb> if we can get a new machine before may 18 donee need to do the expense dance?
19:42:11 <fungi> as mentioned above, i'll be travelling may 2-13 and not available to help with things
19:42:32 <frickler> regarding the ceph reef mirror change, do we want to/can we move all the repos onto a single volume, so we don't need a new one for each release?
19:42:48 <clarkb> tonyb: probably not, but I wouldn't worry about it
19:42:50 <fungi> i could do the cert replacement on may 14 but that's cutting it close
19:42:51 <tonyb> fwiw I'll be in the US from Sunday 28/April to mid June (maybe the 17th)
19:43:07 <frickler> also I'll be away this Fri and Wed-Fri next week
19:43:12 <clarkb> tonyb: basically renewing the cert is cheap enough we shouldn't consider it a hard deadlien thing
19:43:35 <tonyb> clarkb: okay.
19:44:00 <fungi> oh, in other news, i've started in on a dns record audit for openstack.org as mentioned in last week's meeting
19:44:04 <clarkb> I hope you all manage to enjoy your time off
19:44:48 <clarkb> fungi: manage to delete any records yet?
19:45:09 <fungi> it's complicated by the domain being shared jurisdiction with foundation staff and webdev contractors, so all three parties err on the side of caution in not deleting records we've collectively forgotten why we added
19:45:35 <fungi> i see quite a few i'm pretty sure can be deleted, but i'll float a list first
19:46:33 <clarkb> sounds good, thank you for looking at that
19:46:50 <clarkb> I'll give it a few more minutes if anyone else has somethign to share otherwise we can probably end about 10 minutes early today
19:46:54 <fungi> ideally, we'll clean up what we know can go, then provide the other two parties with a list of records we don't have any interest in and let them separately filter the remainder
19:48:08 <fungi> current record count is close to 300, so it's nontrivial to untangle
19:48:36 <tonyb> wow that's much more than I would have guessed.
19:49:15 <fungi> there are still records in there pointing to sites that contained openstack project policy board (tc predecessor) election results
19:49:46 <fungi> easily irrelevant for over a decade
19:50:08 <tonyb> fair enough
19:50:15 <clarkb> sounds like that was everything. Thank you for your time today and help running OpenDev. We'll be back here same time and location next week.
19:50:19 <clarkb> #endmeeting