| clarkb | almost meeting time | 18:57 |
|---|---|---|
| fungi | indeed | 18:58 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Apr 23 19:00:09 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/WBD5BLMI6ZPKUB6FOSZ65P7P5RR7CHFS/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | I have no announcements. Did I miss anything announcement worthy? | 19:00 |
| fungi | i don't think so | 19:01 |
| clarkb | #topic Upgrading Old Servers | 19:01 |
| clarkb | haven't seen anything on this subject over the last week. I did have a mini panic when I thought Focal was EOL this month but it has another 12 months before that happens | 19:02 |
| fungi | it's been brought up that wiki.o.o is still very far behind and its ua/esm subscription expired a while ago | 19:02 |
| fungi | i'm trying to work out where that discussion with canonical stalled | 19:03 |
| clarkb | thanks. I think the best case short term there is to renew the esm sub if possible and then contineu to look at a replacement | 19:03 |
| fungi | but rebuilding it on a newer system, if anyone has time, would address both that and the impending ssl cert expiration for it | 19:03 |
| clarkb | ya though I'm happy to deal with the cert in a few weeks (which is on the agenda for later) | 19:04 |
| clarkb | #topic MariaDB Upgrades | 19:05 |
| clarkb | Etherpad and mailman have new shiny upgraded mariadbs now. Gerrit and Gitea are the remaining services that need similar treatment | 19:06 |
| clarkb | After thinking about it a bit due to the ease of upgrades for everything else I'm kidna thinking we may get away with just sending it for gitea | 19:06 |
| clarkb | CI should check general compatibility with newer mariadb and if that checks out then the upgrade process itself has been pretty reliable so far and we can probably expect it to continue to be reliable | 19:07 |
| clarkb | I'll push up a change for gitea and if anyone is concerned about that we can post those concerns in review | 19:07 |
| clarkb | On the gerrit side of the house we will have to upgrade the db manually since the current ansible is very hands off with container management (and we want it to be) | 19:07 |
| clarkb | But I do think we can just pick a time for gerrit and get it done since gerrit's db isn't very critical and the actual downtime should be quite short | 19:08 |
| fungi | i agree | 19:08 |
| clarkb | in that case I guess I should also push up a change to reflect the update that will ultimately be somewhat manual in gerrit as well | 19:09 |
| clarkb | #topic AFS Mirror Cleanups | 19:09 |
| clarkb | I haven't made any new progress on Xenial cleanup. I think the scope of it is daunting enough that I'm happy to keep kicking it down the road if I can :) but I should just dive in and see what I learn | 19:10 |
| clarkb | #topic Building Ubuntu Noble Nodes | 19:11 |
| clarkb | https://review.opendev.org/c/opendev/glean/+/915907 merged which adds python3.12 support to glean | 19:11 |
| fungi | official release day for noble is thursday this week, right? | 19:11 |
| clarkb | I believe that our image builds use glean from releases by default though | 19:11 |
| frickler | fungi: yes | 19:12 |
| clarkb | This means we should plan to make a glean release. I can do that probably on Thursday ish as well | 19:12 |
| clarkb | once we have a release we then need to monitor our next image builds but we have decent coverage of glean in CI so not too worried | 19:12 |
| clarkb | #link https://review.opendev.org/c/zuul/nodepool/+/916053 Update Nodepool debootstrap to support Noble | 19:12 |
| clarkb | separately we need to update Nodepool images to add support to debootstrap for noble | 19:13 |
| clarkb | corvus: ^ not sure if you have any input on that, but I think it does have the votes it needs so maybe we just go ahead and approve it now | 19:13 |
| corvus | sgtm; i'll add a general +2 (i haven't reviewed the details) | 19:14 |
| clarkb | ack thanks | 19:14 |
| clarkb | Good news is there is solid progress here. Thank you frickler for testing it out and finding where the initial issues are | 19:14 |
| frickler | 3rd patch needed will be https://review.opendev.org/c/openstack/project-config/+/916050 | 19:14 |
| clarkb | once we've got these three items sorted out we should be able to start building images and/or adding mirroring | 19:16 |
| fungi | looks like those changes are approved now | 19:16 |
| fungi | so just need the glean tag (assuming things merge/promote successfully) | 19:16 |
| clarkb | yup and I've got that on my todo list now | 19:17 |
| fungi | awesome, thanks | 19:17 |
| clarkb | I'm not committing to doing it immediately because I'm not sure what i need to do to get my gpg key out of cold storage and usable again | 19:17 |
| frickler | there is also a testing patch in dib fwiw https://review.opendev.org/c/openstack/diskimage-builder/+/915915 | 19:17 |
| frickler | I'll update that once the glean release is done | 19:18 |
| clarkb | sounds good | 19:18 |
| clarkb | #topic Etherpad 2.0.x Upgrade | 19:18 |
| clarkb | I can't remember the state of this during our last meeting, but I got the unreleased state of Etherpad to work with new api auth methods and added a bunch of testing to ensure that our admin tasks don't regress with the new setup | 19:19 |
| clarkb | Everything seems to be working which means that as soon as etherpad makes a release we should be read to upgrade. I don't want to upgrade until a release is done just in case anything else changes | 19:19 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/914119 Etherpad updates if interested | 19:20 |
| clarkb | that chagne has all the new testing and docs and so on if you are interested in taking an early look at it | 19:20 |
| clarkb | #topic Add reference to the project-team-guide in the fail ci msg | 19:20 |
| clarkb | #link https://review.opendev.org/914189 | 19:21 |
| clarkb | fungi: you added this topic want to fill us in? | 19:21 |
| fungi | this was more of a concern with blurring lines between opendev and openstack | 19:21 |
| fungi | see my comments on the linked review | 19:21 |
| clarkb | gotcha, I think I'm ok with adding that to the openstack tenant and having those of us still in the openstack tenant that probably shouldn't deal with it | 19:22 |
| fungi | basically, are we (collectively) okay having failure results in the openstack zuul tenant linking to openstack-specific guidance/documentation? | 19:22 |
| clarkb | and if anyone has questions or concerns we can point them in the direction of using a different tenant | 19:22 |
| clarkb | (just to be clear that I don't think we should add all the possible debug doc links to the openstack tenant, more that we're in the openstack tenant so openstack things are probably ok) | 19:23 |
| fungi | normally i'd say yes, but the openstack tenant was our original/default tenant and is still used by a lot of non-openstack projects as a result | 19:23 |
| frickler | I agree that this may be a good opportunity to progress with tenant separation if needed | 19:23 |
| clarkb | its also something the vast majority of people are unlikely to read unfornately | 19:23 |
| corvus | i agree with clarkb and frickler | 19:24 |
| clarkb | I know I click the zuul summary button myself :) | 19:24 |
| fungi | i'm okay approving it, but wanted to make sure there weren't major objections | 19:24 |
| frickler | the openstack guidelines don't sound too unreasonable for other projects I'd say | 19:24 |
| fungi | right, they're pretty general and not bad | 19:24 |
| fungi | i'm just wary of setting new precedent without further consultation | 19:24 |
| clarkb | I'll +2 it after the meeting to make my stance clear in review (and to record it better) | 19:25 |
| frickler | thx for mentioning the patch, though, I had missed it somehow | 19:25 |
| fungi | sounds good, thanks | 19:25 |
| clarkb | #topic Gerrit 3.9 Upgrade Planning | 19:25 |
| clarkb | We're in a position now where I think we could upgrade Gerrit tomorrow if we wanted to | 19:26 |
| clarkb | Gerrit 3.9 reindexing should work, our images are updated to pull that fix in, etc | 19:26 |
| clarkb | #link https://www.gerritcodereview.com/3.9.html Release Notes | 19:26 |
| clarkb | there are some intentional breaking changes though as noted in the release notes. None of them really impact us from what I can tell. | 19:26 |
| clarkb | There is also a very straightforward downgrade path back to 3.8 if necessary | 19:27 |
| fungi | i'm up for a tomorrow upgrade ;) | 19:27 |
| clarkb | All that to say our next steps here should be to write up a document to track things like downgrade testing and breaking change review. Then we can schedule a day for the upgrade and get it done. I'll volunteer to get that document started and we can coordinate from there | 19:27 |
| clarkb | to be clear I don't think it will happen tomorrow :) just that the major known issues have been addressed :) | 19:28 |
| fungi | yeah, got it | 19:28 |
| clarkb | I also want us to keep in mind that we should start thinking about a gerrit server upgrade. Historically we've tried to give people some notice of these in order to update their firewall rules. THough honestly with the number of old clients attempting to authenticate and failing it may be a good idea to make it a surprise :) | 19:28 |
| fungi | also my availability for the first half of next month is pretty limited, so if i'm going to help it will either be next week or several weeks out | 19:28 |
| clarkb | several weeks out is probably fine | 19:29 |
| clarkb | since there is prep work to get done and plenty of other stuff to do as well | 19:29 |
| clarkb | Talking out loud here I think our longer term planning should be something like upgrade to 3.9 in Mayish, upgrade the underlying server to noble during the summer sometime, then upgrade to 3.10 in November/December/January time frame | 19:30 |
| clarkb | One other thing to note that doesn't really affect us since we've already done the work is JGit just proposed that they will drop java 11 support | 19:30 |
| clarkb | We're already on java 17 so that is fine for us | 19:30 |
| frickler | would we consider moving away from vexxhost with the server? like we still don't have feedback about the unplanned shutdown yet? also recurring network issues | 19:31 |
| clarkb | Considering the alternative options I'm not sure there is a clear cut winner in hosting. However, that is a good question and somethign we should keep in mind as we plan for server replacement | 19:31 |
| clarkb | vexxhost in particular lets us run a large server which the jvm running gerrit seems to appreciate | 19:32 |
| frickler | I would be seeing ovh as possible option, maybe we can the the flavor question with them if there is interest in that | 19:33 |
| clarkb | Let's bring that back up after we've upgraded to 3.9 and start planning the server replacement. Ideally we'd be able to work with vexxhost to address the concerns | 19:33 |
| frickler | s/the/address/ | 19:33 |
| clarkb | #topic Wiki Cert Renewal | 19:35 |
| clarkb | In the past I've renewed this cert when there has been a week or less of validity | 19:35 |
| clarkb | the current cert expires May 18, 2024. Which means renewing it sometime after May 11 | 19:35 |
| clarkb | I'm happy to do that, but didn't want others worried it was ignored as we get the daily warning email | 19:36 |
| fungi | ftr i'll be out of town, though it's a straight-up file replacement and apache2 restart | 19:36 |
| frickler | just noticing that iirc I still have no login for that server, so could not help right away if needed | 19:37 |
| clarkb | frickler: thats a good call out. I suspect tonyb doesn't either. I can probably manually address that too when dealing with the cert | 19:37 |
| tonyb | I do. fwiw | 19:38 |
| clarkb | ah ok | 19:38 |
| fungi | i'm happy to add/update ssh keys in a few minutes | 19:38 |
| clarkb | fungi: thanks! | 19:38 |
| fungi | i can just copy/merge from a properly config-managed server, i expect | 19:38 |
| tonyb | I'm wondering if we can build a new server in that opendev domain to get LE so this becomes less of a problem | 19:38 |
| frickler | note there's a review with a new key up | 19:38 |
| tonyb | ..... as part of the server upgrades away from bionic | 19:38 |
| fungi | tonyb: the domain isn't the problem, just need ansible working | 19:39 |
| tonyb | ahh okay. nevermind | 19:39 |
| fungi | but yes, if we can get it onto a newer ubuntu version, easily solved | 19:39 |
| tonyb | we'll get that "soon" | 19:39 |
| clarkb | In any case I wanted to make sure that others know the deadline here is known and that I intend on addressing it manually | 19:39 |
| frickler | +1 | 19:40 |
| clarkb | Normally I would ask for volunteers but it involves paying money and filign an expense report and I don't expect others to do that here | 19:40 |
| clarkb | side note: The cost of a cert and the cost of a beer have converged and in the wrong direction | 19:40 |
| fungi | fingers crossed this is the last year you expense a cert on domaincheap | 19:41 |
| clarkb | #topic Open Discussion | 19:41 |
| fungi | so you can focus on expensing beer | 19:41 |
| clarkb | Anything else? | 19:41 |
| tonyb | if we can get a new machine before may 18 donee need to do the expense dance? | 19:41 |
| fungi | as mentioned above, i'll be travelling may 2-13 and not available to help with things | 19:42 |
| frickler | regarding the ceph reef mirror change, do we want to/can we move all the repos onto a single volume, so we don't need a new one for each release? | 19:42 |
| clarkb | tonyb: probably not, but I wouldn't worry about it | 19:42 |
| fungi | i could do the cert replacement on may 14 but that's cutting it close | 19:42 |
| tonyb | fwiw I'll be in the US from Sunday 28/April to mid June (maybe the 17th) | 19:42 |
| frickler | also I'll be away this Fri and Wed-Fri next week | 19:43 |
| clarkb | tonyb: basically renewing the cert is cheap enough we shouldn't consider it a hard deadlien thing | 19:43 |
| tonyb | clarkb: okay. | 19:43 |
| fungi | oh, in other news, i've started in on a dns record audit for openstack.org as mentioned in last week's meeting | 19:44 |
| clarkb | I hope you all manage to enjoy your time off | 19:44 |
| clarkb | fungi: manage to delete any records yet? | 19:44 |
| fungi | it's complicated by the domain being shared jurisdiction with foundation staff and webdev contractors, so all three parties err on the side of caution in not deleting records we've collectively forgotten why we added | 19:45 |
| fungi | i see quite a few i'm pretty sure can be deleted, but i'll float a list first | 19:45 |
| clarkb | sounds good, thank you for looking at that | 19:46 |
| clarkb | I'll give it a few more minutes if anyone else has somethign to share otherwise we can probably end about 10 minutes early today | 19:46 |
| fungi | ideally, we'll clean up what we know can go, then provide the other two parties with a list of records we don't have any interest in and let them separately filter the remainder | 19:46 |
| fungi | current record count is close to 300, so it's nontrivial to untangle | 19:48 |
| tonyb | wow that's much more than I would have guessed. | 19:48 |
| fungi | there are still records in there pointing to sites that contained openstack project policy board (tc predecessor) election results | 19:49 |
| fungi | easily irrelevant for over a decade | 19:49 |
| tonyb | fair enough | 19:50 |
| clarkb | sounds like that was everything. Thank you for your time today and help running OpenDev. We'll be back here same time and location next week. | 19:50 |
| clarkb | #endmeeting | 19:50 |
| opendevmeet | Meeting ended Tue Apr 23 19:50:19 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:50 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-04-23-19.00.html | 19:50 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-04-23-19.00.txt | 19:50 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-04-23-19.00.log.html | 19:50 |
| fungi | thanks clarkb! | 19:50 |
| clarkb | and now I can go find some food. I think I have leftover garlic noodles | 19:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!