19:01:14 <clarkb> #startmeeting infra
19:01:14 <opendevmeet> Meeting started Tue May 17 19:01:14 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:14 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:14 <opendevmeet> The meeting name has been set to 'infra'
19:01:21 <ianw_> o/
19:01:48 <clarkb> #link https://lists.opendev.org/pipermail/service-discuss/2022-May/000336.html
19:01:53 <clarkb> Our Agenda
19:01:58 <clarkb> #topic Announcements
19:02:27 <clarkb> A friendly reminder that teh Summit happens June 7-9. I expect some of us will be traveling for that and distracted that week and probably on either end of it as well
19:03:05 <clarkb> Also I'll be out tomorrow
19:03:28 <clarkb> #topic Topics
19:03:32 <clarkb> Time to dive right in
19:03:40 <clarkb> #topic Improving CD Throughput
19:04:14 <clarkb> I don't have much new on this topic. But did want to call out something ianw_ noticed yesterday. If our base job fails then we don't periodicallyrun manage-projects
19:04:58 <clarkb> We should probably monitor https://zuul.opendev.org/t/openstack/builds?project=opendev%2Fsystem-config&pipeline=periodic&skip=0 and check that it is running through its list of jobs
19:05:25 <frickler> maybe add it as a regular topic to check that?
19:05:33 <frickler> works fine in the qa meetings
19:05:34 <clarkb> frickler: to the meeting? We can do that
19:05:42 <frickler> yes
19:06:13 <clarkb> sure, I can add that and we can give it a try
19:06:59 <clarkb> #topic Container Maintenance
19:07:16 <clarkb> ianw_: mentioned wanting to look at mariadb upgrades when doing the Gerrit 3.5 upgrade.
19:07:29 <clarkb> ianw_: have you had a chance to look into that yet? Looks like you were working on holding a test node
19:08:15 <ianw_> not yet sorry, but yeah, working on it with that held node, and also testing the downgrade there
19:08:26 <ianw_> (meant to do it yesterday but got distracted by other things)
19:08:38 <clarkb> no problem. Just wanted to check if there was new info on that as it was something I identified under the broad container maintenance header
19:08:50 <clarkb> Will be interesting to see what we learn about it
19:09:07 <clarkb> #topic Support for Jammy Jellyfish
19:09:30 <clarkb> we're still waiting on wheel mirrors for Jammy before we consider this done.
19:09:48 <clarkb> I did want to mention that I think the openafs cleanups that jammy triggered are largely compelte with exception of xenial removal
19:10:10 <clarkb> however, even without xenial removal we have freed up significant disk space and managed to reset quotas to better reflect current volume consumption
19:10:16 <ianw_> i can get onto that soon, but what was the latest on reprepro?  it's sort-of-working-but-with-errors-that-we-can-seem-to-ignore-for-now?
19:10:26 <frickler> yes
19:11:25 <clarkb> ianw_: for the PPA stuff do we have jammy openafs packages built now and managed by changes to the new repos?
19:12:06 <ianw> yes we do, there is a test ...
19:12:25 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/841525
19:12:29 <ianw> be good to get that in
19:12:40 <clarkb> ++ I'll review that
19:13:21 <clarkb> Sounds like we mostly just need to push changes to add wheel mirroring and review them etc. No major issues at this point?
19:13:39 <clarkb> new afs volumes will need to be created too. Note I set all the wheel volume quotas to 20GB today
19:14:48 <clarkb> Are there any other Jammy spin up concerns at this point?
19:15:42 <ianw> i still need to fully look into the phased deb bits from the dib side, on the todo
19:18:14 <clarkb> Sounds like that may be it
19:18:19 <clarkb> #topic Gerrit 3.5 Upgrade
19:18:23 <ianw> (oh and tangentially to jammy openafs, i also fixed an issue with the new bionic openafs packages that were built relying on the backports repo being enabled.  i've disbled that in the ppa and rebuilt them)
19:18:39 <ianw> thanks to frickler for noticing that
19:20:15 <fungi> that was a great find
19:21:25 <ianw> i put the gerrit 3.5 upgrade on the agenda
19:21:35 <clarkb> ya sorry being distracted
19:21:47 <ianw> i have started a checklist at
19:21:50 <ianw> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.5
19:22:24 <clarkb> I left some notes on there which you seem to have noticed
19:23:19 <clarkb> the main thing that jumped out to me is enabling the conflicts checking. The mergeability checking showed us that people do rely on and use those features so we shoudl toggle them on before upgrading
19:23:50 <ianw> yeah, i can send changes to turn that on explicitly
19:24:04 <clarkb> ++ thanks
19:24:30 <ianw> the other one was that there's a new ssh "set-account" ability to remove external id's
19:24:54 <ianw> might make it slightly easier for duplicate accounts
19:25:16 <clarkb> ya I think the migration automatically adds that perm to admins? But then we can possibly take advantage of it to clean up the user db further
19:26:02 <clarkb> learning what we can about the downgrade and possibility of a mariadb upgrade at the same time are the other major things. Basically we seem to have the known things under control and now its a matter of investigating the unknowns a bit
19:26:18 <ianw> and out of interest i did dump the username: id's to see which ones conflicted in case-insensitive; that's an interesting list
19:26:42 <clarkb> yup, a few of them are people getting a second account and needing to set a non conflicting usename
19:26:51 <fungi> i ran into one for a third-party ci system last week
19:26:53 <clarkb> I noticed patterns like ^ when doing the db cleanups
19:27:44 <ianw> assuming i get those other bits marked off, do we want to think about a date?
19:28:08 <clarkb> Yes, I made a note on the agenda to talk about scheduling
19:28:24 <clarkb> https://releases.openstack.org/zed/schedule.html is the openstack release schedule which is a big thing to consider when planning this
19:28:35 <clarkb> that makes note of the summit too which is nice
19:29:03 <ianw> the options seem to be pre summit or post -- pre meaning we have it done and deployed for that, post being i guess slightly safer
19:29:05 <clarkb> personally I feel like I have enough distractions pre summit to make doing it between now and then difficult. But maybe soon after the summit? That gives us plenty of time to announce it too
19:30:19 <clarkb> With previous releases I've done enough java'ing that I'd worry about doing an upgrade and not having some time to do that if necessary again
19:31:12 <ianw> fair enough, i do like the idea that we go into the summit with it fully updated just as a point about the way we deploy, but i see the argument
19:31:12 <fungi> sounds good to me
19:31:37 <fungi> after i mean. just worried about summit prep and maintenance
19:31:44 <fungi> not that i'm super against before
19:31:54 <clarkb> ya I think if I didn't have a bunch of commitments for the summit I need to prepare for and then attend while there I'd be ok before
19:32:10 <clarkb> but I do have those things and I need to be sure I get around to them :)
19:32:52 <clarkb> The week after a summit tends to be pretty quiet too (not sure if that will hold during a pandemic) so could be a good time for it from a user perspective
19:32:56 <ianw> something like the 12th?
19:33:39 <clarkb> I return the 11th. The 12th would work for me assuming I can power through jetlag
19:34:06 <clarkb> as far as conflicts go thats a fairly easy one to deal with.
19:34:15 <clarkb> fungi: frickler ^ thoughts?
19:34:46 <frickler> I won't summit so I'm pretty flexible about it
19:34:50 <fungi> i'm not going to be around i don't think... checking
19:35:12 <ianw> hrm 13th june is a public holiday here actually, so i am likely as not to be afk from that weekend
19:35:27 <fungi> yeah, i'll be on my way home from boston on the 12th
19:35:44 <clarkb> what about the 19th then? gives us a week to recover from the event then do the upgrade
19:35:54 <fungi> i can absolutely do the 19th
19:35:58 <clarkb> or depending on location enjoy a public holiday :)
19:36:33 <clarkb> and then we can also give everyone a months notice if we announce it in the next day or two
19:37:32 <ianw> ++ i'll be happy to drive and do it very early on my monday, and hopefully !australian people can just be standyby in case of issues
19:37:56 <clarkb> sounds like a plan
19:38:17 <clarkb> and to be clear thats australian 20th, but late 19th UTC ?
19:38:48 <clarkb> ianw: and is that something you want to send email for or should I work on that bit?
19:38:57 <ianw> say 20:00 UTC if that's ok.  that's 6am which is fine
19:39:05 <clarkb> wfm
19:39:10 <ianw> i can send the announcement today
19:39:29 <clarkb> June 19 starting at 20:00 UTC. Shouldn't take more than a few minutes so blocking out an hour is probably plenty?
19:39:42 <clarkb> (I really do appreciate that gerrit upgrades are quick now)
19:39:52 <ianw> ++ (famous last words, haha)
19:40:54 <clarkb> Anything else on the subject of Gerrit upgrades?
19:41:04 <ianw> but yeah, we really have a ton of ci/cd that gives us great confidence!
19:41:07 <clarkb> ++
19:41:15 <ianw> nope, sounds good, thanks!
19:41:24 <fungi> wfm!
19:41:30 <clarkb> #topic Open Discussion
19:41:55 <clarkb> Just to reiterate I do plan to get a new annual cert for wiki.o.o next week (one of my pre summit todos that interferes with pre summit gerrit upgrades)
19:42:04 <clarkb> frickler: ^ fyi since you called that out
19:42:09 <corvus> i have thing
19:42:30 <clarkb> corvus: go for it
19:43:36 <corvus> i'm a little behind, but shortly i'm going to send an email reminder to zuul-discuss about the deprecation of the 'queue' attribute on project-pipeline stanzas
19:43:51 <corvus> we (opendev) should run this script to find instances of it: https://opendev.org/zuul/zuul/src/branch/master/tools/deprecated-queues.py
19:44:11 <corvus> or at least our tenants should :)
19:44:26 <corvus> so my question is, how would we like to handle that?
19:44:29 <clarkb> I don't mind running it and sending email to our various users. I have done similar in the past for zuul config errors
19:44:57 <corvus> cool, should be easy to run, and you can do a whole tenant at a time
19:45:07 <corvus> istr there were many instances in the openstack tenant
19:45:10 <fungi> and we have only a handful of tenants
19:45:18 <corvus> there's like 1 or 2 in zuul; i'll take care of that
19:45:19 <clarkb> corvus: any ballpark on when it will become an error to have in your config?
19:45:50 <corvus> clarkb: v7 i think; but no idea when that is.  probably > 1 mo.
19:46:13 <clarkb> ok that works. Mostly so that I can warn them with info on when it will convert to an error in the emails
19:46:25 <corvus> (should have been v6, but i think v6 was too fast/urgent for that)
19:46:45 <corvus> note that "become an error" in the case of a project-pipeline definition is sort of equivalent to "stop running many/all jobs"
19:46:52 <clarkb> ah right
19:47:11 <clarkb> I'll also call this out in the TC meeting thursday (that is likely to be before I send email about it) so that they are aware
19:47:23 <clarkb> and ya I spot checked with codesearch last week and there are a number with those configs in place
19:48:16 <clarkb> I'll likely get to running it and sending emails on thursady just not before the first thing in the mroning meeting
19:48:20 <corvus> that's all from me
19:49:00 <clarkb> I'll give ti a couple more minutes for any other topics and if not we can end 10 minutes early
19:51:05 <clarkb> Seems like that was it. Thank you everyone!
19:51:19 <clarkb> As always feel free to reach out in #opendev or on the mailing list if something comes up before the next meeting
19:51:24 <clarkb> #endmeeting