19:01:09 <clarkb> #startmeeting infra
19:01:09 <opendevmeet> Meeting started Tue Jul 20 19:01:09 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:09 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:09 <opendevmeet> The meeting name has been set to 'infra'
19:01:12 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-July/000269.html Our Agenda
19:01:33 <clarkb> #topic Announcements
19:01:38 <clarkb> I didn't have any announcements
19:01:53 <clarkb> #topic Actions from last meeting
19:02:02 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-07-13-19.01.txt minutes from last meeting
19:02:11 <clarkb> #action someone write spec to replace Cacti with Prometheus
19:02:26 <diablo_rojo_phone> O/
19:02:29 <ianw> o/
19:02:31 <clarkb> As far as I know that action hasn't been done. But good news is I think we're getting throug some of the stuff that had been distracting us and so that might get done soon :)
19:02:45 <clarkb> #topic Specs Approval
19:02:53 <clarkb> #link https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver
19:03:14 <clarkb> Last week we said we'd put this up for approval this week. Big part of that was people weren't around last week that might want to review the change
19:03:36 * mordred does dance
19:03:43 <clarkb> fungi: ianw ^ if you can give that a read before Thursday EOD clarkb time that would be great
19:03:53 <fungi> yeah
19:03:57 <mordred> clarkb: you're your own timezone now
19:04:06 <clarkb> tristanC has a change up to run a matrix gerritbot too which might be interesting to others
19:04:51 <clarkb> #topic Topics
19:04:57 <clarkb> #topic Gerrit Server Upgrade
19:05:08 <clarkb> This happened. Thank you ianw for doing the bulk of the work on this one.
19:05:53 <clarkb> We've had a few minor hiccups with things like java sql conncetor libraries and getting ssh host keys sorted last minute and making the change to reflect this in config management pass actually pass
19:06:01 <clarkb> all told those are small issues and this went really well.
19:06:30 <clarkb> I wanted to call out for followups we now have the ability to do some gerrit cache and memory usage tuning that we didn't really have room for before
19:06:36 <fungi> the end result is we now have a bigger gerrit with a local db for the reviewed file checkmarks?
19:06:43 <clarkb> Maybe we let the server run for a week or two then look at data?
19:06:45 <clarkb> fungi: yup
19:07:23 <clarkb> The othe rthing that ianw has mentioned is we should all check our homedirs and the gerrit2 homedir for files we want to live on the new server too
19:07:47 <clarkb> I've already moved the ones I know I wanted to move, but still need to go look at the rest to see if any of it is worth carrying over
19:08:12 <clarkb> #link https://etherpad.opendev.org/p/gerrit-upgrade-2021 Look here for changes to review in followup to the server move
19:08:29 <clarkb> the etherpad also has a few changes that ianw has pushed in followup. I think I've reviewed them all, and other reviews would be nice too
19:08:50 <clarkb> ianw: anything else to say about this?
19:09:07 <clarkb> Do we want to keep it on the agenda for next week or can we consider this done?
19:09:25 <ianw> nope, it seems like everything is going well.  i'll wait for frickler_pto to become frickler and then i think we can retire the old server
19:09:42 <ianw> oh and i think i have some log archives i can move across too
19:09:44 <fungi> how is the mariadb connector fix applied to the image build? sed -i?
19:10:05 <ianw> fungi: currently we reverted to the mysql connector, which is still compatible
19:10:06 <clarkb> fungi: we landed it via normal gerrit changes after manually editing the config file
19:10:30 <clarkb> we have both libs present in the image to support the old and new server db setups so we just had to change the connection line to specify the other lib
19:10:30 <fungi> oh, got it, so all in configuration
19:10:57 <clarkb> yup, once mariadb connector works properly we can update the config as well as update our image to remove the old mysql lib
19:10:58 <fungi> we're not actually running with the mariadb connector patched
19:11:05 <clarkb> correct
19:11:08 <fungi> that makes sense, thanks
19:11:09 <ianw> yep; no rush to get back.  i was just using the mariadb connector because, well, we were connecting to mariadb :)
19:11:29 <fungi> i wonder why they added a special one for that
19:12:15 <ianw> just the way the subclasses work i think
19:12:31 <ianw> basically string matching the db uri and then choosing classes
19:12:50 <ianw> i do wonder why they added it with at least two bugs that made it not work at all, and nobody noticed, however
19:13:31 <fungi> just seems odd that with as little as gerrit uses an rdbms for now, they'd incur the maintenance overhead of slight variations of a generic sql connector
19:13:42 <fungi> historical baggage i suppose
19:13:51 <fungi> anyway, not necessary to dig into now
19:13:52 <clarkb> it may have been added back in 2.16 or prior for the actual db when that existed
19:13:54 <clarkb> ya
19:14:15 <clarkb> #topic Gerrit Account Cleanup
19:14:33 <clarkb> I've been thinking about scheduling for this and the project renaming that we said we'd all do "week after the upgrade"
19:15:16 <clarkb> In my head it makes sense to do the external id pruning Say on Monday. That is 3 weeks after I disabled the related accounts. Then do the project renames later in the week, say Friday.
19:15:28 <fungi> that sounds reasonable to me
19:15:36 <clarkb> The reason for this is if we need to do account surgery after the external id cleanup happens due to the cleanups we can take advantage of the outage to do it
19:15:58 <clarkb> we did surgery on two accounts during the move for unrelated reasons and it went well but does require a downtime
19:16:15 <clarkb> I don't expect we'll end up doing that as part of the project rename downtime, but it is nice to have the option
19:16:37 <clarkb> Ok I'll plan to run the external id cleanup script on Monday then.
19:16:37 <fungi> yep
19:17:10 <clarkb> #topic Gerrit Project Renames
19:17:35 <clarkb> As just mentioned I'm suggesting Friday July 30 for this and probably early that day. Say 15:00UTC ?
19:17:58 <clarkb> that gives us time to do more testing if we can find time to do that testing but also the buffer to account cleanups which we can use to fix accounts if necessary
19:18:00 <fungi> i'm around and happy to do it then
19:18:14 <clarkb> ok lets do it
19:18:30 <clarkb> I can send out email about that as well
19:19:29 <clarkb> #topic Gitea01 backup issues
19:19:38 <clarkb> I don't expect anything new has come up about this but wanted to double check
19:19:53 <ianw> no i haven't looked into it sorry
19:20:01 <clarkb> no worries you were busy :)
19:20:02 <ianw> i haven't heard any more on the ipv6 issues
19:20:38 <ianw> i could take it out of the lb and reboot it i guess
19:20:55 <fungi> and does look like it's still going on
19:20:58 <ianw> the switch it off and on again method probably has a high likelyhood of fixing things
19:21:16 <clarkb> hah ok
19:21:26 <fungi> yeah, neutron likes to get a kick in the interface every now and then
19:22:01 <ianw> we can sync back after that happens one way or the other
19:22:09 <clarkb> sounds good
19:22:13 <clarkb> #topic Gitea 1.14.4 upgrade
19:22:41 <clarkb> I started working on this when all the other gerrit and zuul work was happening and didn't want to impact those as they seemed more important
19:23:11 <clarkb> but now that is largely behind us so I'd like to go ahead and land this upgrade when I have time to monitor it. Currently I think I have time tomorrow morning any objections to landing the change to upgrade gitea then?
19:23:20 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/800274 gitea upgrade
19:23:38 <clarkb> Its a fairly large gitea upgrade as far as the change delta goes, but I did hold a node and it looked good
19:23:50 <clarkb> and our testing caught one issue with a changing api so that seems to be working well for us
19:24:46 <fungi> i'm good with it, finally reviewed and checked it out today
19:24:49 <clarkb> Sounds like no objections. If things are quiet when I start my day tomorrow I'll approve it
19:25:06 <ianw> ++
19:25:14 <clarkb> #topic High AFS utilization
19:25:15 <fungi> i'll be more around tomorrow anyway, more than today
19:25:20 <clarkb> fungi: cool
19:25:47 <clarkb> Yesterday it was pointed out that the opensuse mirror was sad. It had a stale lock which I removed and then manually did updates for after and is happy now
19:26:00 <clarkb> In the process of looking into that I noticed htat our afs vicepa utilization is high
19:26:08 <clarkb> we have less than 10% remaining disk space
19:26:34 <clarkb> The debian volume looks like it may need to grow as well as we are right up against quota there
19:26:38 <fungi> were we able to reclaim much with the centos cleanup?
19:26:49 <clarkb> fungi: yup, but then we backfilled with openeuler mirroring :)
19:26:59 <fungi> we can push to try and drop debian-stretch images
19:27:20 <ianw> yeah, although bullseye is fully frozen now, so we can probably drop some debian soon
19:27:24 <clarkb> ya I was going to ask about that particularly since debian is near its quota
19:27:27 <ianw> or yeah, just push to drop it now
19:27:42 <clarkb> I think that would be a good next step if we think it isn't too disruptive to users for some reason
19:27:55 <clarkb> The other one that is large and I'm not sure still used is mirror.yum-puppetlabs
19:28:00 <fungi> but yes, stretch is already "oldstable" (the release before the current stable) and is about to become "oldoldstable"
19:28:48 <clarkb> I think tripleo and/or puppet-openstack may be the users of the yum puppetlabs mirror. I'm wondering if they continue to use it at all and if so if they only use a small subset
19:28:52 <fungi> we should probably ask the puppet-openstack team if they're still using that, and if so whether they're using all or just a small part
19:29:00 <clarkb> currently that mirror seems to have packages for old fedora and sles and so on
19:29:05 <clarkb> fungi: ++ exactly
19:29:29 <clarkb> I suspect we can trim that done one way or another.
19:29:46 <clarkb> I can send email to openstack-discuss asking puppet-openstack and tripleo to weigh in on it
19:29:56 <clarkb> fungi: ianw: any idea what the first step for debian trimming would be?
19:30:21 <fungi> probably check for zuul configs referencing debian-stretch, though searching stable branches will be hard
19:30:36 <fungi> probably better would be to start a discussion proposing we drop it, and then see if we get pushback
19:31:04 <clarkb> fungi: that seems reasonable. Is that something you might want to send to service-discuss?
19:31:10 <ianw> yeah, i can have a look at it
19:31:18 <clarkb> ianw: that would be great thanks
19:31:30 <ianw> i think both discussion, but also directly proposing changes might help
19:31:35 <fungi> stretch has been obsolete for ~2 years since buster came out, and most projects aren't supporting their branches past that anyway
19:31:59 <ianw> anything we can do to get dib's testing more under control is good too :)
19:32:07 <clarkb> The last note I put related to this on the agenda is I've noticed discussion about adding Alma linux to our node list.
19:32:09 <fungi> i can bring up dropping stretch on the openstack-discuss ml
19:32:54 <clarkb> I think starting with clenaup is the right option but we should probably be prepared to add a third 1TB device to our vicepa setups if the linux distros keep multiplying
19:33:15 <clarkb> There is probably another disucssion there about whether we can delete centos entirely if we do alma or similar but I think it is a bit early for that
19:33:51 <ianw> i mean we can build images before putting the mirror in
19:33:54 <fungi> aha, alma linux is in the same vein as rocky linux and (non-stream) centos. i knew the name sounded familiaer
19:34:21 <clarkb> ianw: yup that is true, though I suspect ne wdistros like that may want us to offloda for them as much as possible
19:34:31 <clarkb> fungi: yes, alma seems to be the one iwth all the momentum right now
19:34:35 <fungi> yes, mirrors are more of a general speedup, and matter more for our more heavily-used distros
19:34:58 <fungi> so we might just consider mirrors to be out of scope for less-used images we offer
19:35:53 <fungi> if a particular label only sees use in a handful of builds a week, the overhead in maintaining a mirror for the software is clearly overkill
19:36:34 <fungi> taking that into account, we could always just consider dropping some of our mirrors of significantly under-utilized stuff even if we don't drop the images
19:36:38 <clarkb> fungi: yup, that is why I also mentioned the possibility of mv centos alma && sed -i -e s/centos/alma/ *zuul.yaml :)
19:37:08 <ianw> (this was a bit of my concern with openeuler as it seems like that has a long path to being used)
19:37:12 <clarkb> But all of that currnetly depends on imgaes and people being interested in the work and so
19:38:05 <clarkb> ianw: thats a good point. I guess we can take a high level view once the afs pruning happens and we know where we stand
19:38:18 <clarkb> then start thinking a bit more critically about the various distros and whether we need full mirrors for all of them
19:38:44 <clarkb> Not something we need to solve today, I just saw the demand seems to be creeping up and wanted to call it out
19:39:24 <clarkb> #topic PTG Participation
19:39:38 <clarkb> I discovered yesterday that the deadline to sign up for PTG participation is tomorrow
19:40:03 <clarkb> The PTG is running October 18-22, 2021. If you are curious yes this sign up deadline is earlier than in the past
19:40:18 <fungi> it feels sudden, but i guess that's because i've been on vacation
19:40:34 <clarkb> The office hours we did last time around seemed to work well for 2 hours in the EU friendly timezone.
19:40:40 <clarkb> s/timezone/time block/
19:40:54 <clarkb> I'm happy to sign up for similar this time around as a result.
19:41:01 <clarkb> Does anyone think we should have more time than that?
19:41:19 <clarkb> Or perhaps think we should skip entirely?
19:42:24 <fungi> what we did last time worked for me
19:43:09 <clarkb> fungi: ok, I'm proposing we trim that back to a single 2 hour block during the EU afternoon NA morning since that is where we got participation last time
19:43:19 <clarkb> ianw: ^ will you feel excluded if we do that ?
19:43:39 <ianw> no
19:44:23 <clarkb> alright I'll sign us up for that. If anyone wants mor etime let me know and I'll adjust to try and include that
19:44:49 <clarkb> #topic Open Discussion
19:45:08 <clarkb> That was all I had. Except I remember thinking of something I should bring up during the meeting earlier today and I've completely forgotten what it was
19:45:27 <clarkb> Anything else?
19:45:42 <fungi> i got nuthin'
19:46:49 <ianw> nothing more from me
19:47:31 <clarkb> Alright then, thank you everyone for your time and now you can go find $meal :)
19:47:40 <fungi> thanks clarkb!
19:47:54 <clarkb> As always feel free to find us in #opendev or at service-discuss@lists.opendev.org and we can discuss anything there that was missed here
19:47:57 <clarkb> #endmeeting