19:01:14 #startmeeting infra 19:01:14 Meeting started Tue May 17 19:01:14 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:14 The meeting name has been set to 'infra' 19:01:21 o/ 19:01:48 #link https://lists.opendev.org/pipermail/service-discuss/2022-May/000336.html 19:01:53 Our Agenda 19:01:58 #topic Announcements 19:02:27 A friendly reminder that teh Summit happens June 7-9. I expect some of us will be traveling for that and distracted that week and probably on either end of it as well 19:03:05 Also I'll be out tomorrow 19:03:28 #topic Topics 19:03:32 Time to dive right in 19:03:40 #topic Improving CD Throughput 19:04:14 I don't have much new on this topic. But did want to call out something ianw_ noticed yesterday. If our base job fails then we don't periodicallyrun manage-projects 19:04:58 We should probably monitor https://zuul.opendev.org/t/openstack/builds?project=opendev%2Fsystem-config&pipeline=periodic&skip=0 and check that it is running through its list of jobs 19:05:25 maybe add it as a regular topic to check that? 19:05:33 works fine in the qa meetings 19:05:34 frickler: to the meeting? We can do that 19:05:42 yes 19:06:13 sure, I can add that and we can give it a try 19:06:59 #topic Container Maintenance 19:07:16 ianw_: mentioned wanting to look at mariadb upgrades when doing the Gerrit 3.5 upgrade. 19:07:29 ianw_: have you had a chance to look into that yet? Looks like you were working on holding a test node 19:08:15 not yet sorry, but yeah, working on it with that held node, and also testing the downgrade there 19:08:26 (meant to do it yesterday but got distracted by other things) 19:08:38 no problem. Just wanted to check if there was new info on that as it was something I identified under the broad container maintenance header 19:08:50 Will be interesting to see what we learn about it 19:09:07 #topic Support for Jammy Jellyfish 19:09:30 we're still waiting on wheel mirrors for Jammy before we consider this done. 19:09:48 I did want to mention that I think the openafs cleanups that jammy triggered are largely compelte with exception of xenial removal 19:10:10 however, even without xenial removal we have freed up significant disk space and managed to reset quotas to better reflect current volume consumption 19:10:16 i can get onto that soon, but what was the latest on reprepro? it's sort-of-working-but-with-errors-that-we-can-seem-to-ignore-for-now? 19:10:26 yes 19:11:25 ianw_: for the PPA stuff do we have jammy openafs packages built now and managed by changes to the new repos? 19:12:06 yes we do, there is a test ... 19:12:25 #link https://review.opendev.org/c/opendev/system-config/+/841525 19:12:29 be good to get that in 19:12:40 ++ I'll review that 19:13:21 Sounds like we mostly just need to push changes to add wheel mirroring and review them etc. No major issues at this point? 19:13:39 new afs volumes will need to be created too. Note I set all the wheel volume quotas to 20GB today 19:14:48 Are there any other Jammy spin up concerns at this point? 19:15:42 i still need to fully look into the phased deb bits from the dib side, on the todo 19:18:14 Sounds like that may be it 19:18:19 #topic Gerrit 3.5 Upgrade 19:18:23 (oh and tangentially to jammy openafs, i also fixed an issue with the new bionic openafs packages that were built relying on the backports repo being enabled. i've disbled that in the ppa and rebuilt them) 19:18:39 thanks to frickler for noticing that 19:20:15 that was a great find 19:21:25 i put the gerrit 3.5 upgrade on the agenda 19:21:35 ya sorry being distracted 19:21:47 i have started a checklist at 19:21:50 #link https://etherpad.opendev.org/p/gerrit-upgrade-3.5 19:22:24 I left some notes on there which you seem to have noticed 19:23:19 the main thing that jumped out to me is enabling the conflicts checking. The mergeability checking showed us that people do rely on and use those features so we shoudl toggle them on before upgrading 19:23:50 yeah, i can send changes to turn that on explicitly 19:24:04 ++ thanks 19:24:30 the other one was that there's a new ssh "set-account" ability to remove external id's 19:24:54 might make it slightly easier for duplicate accounts 19:25:16 ya I think the migration automatically adds that perm to admins? But then we can possibly take advantage of it to clean up the user db further 19:26:02 learning what we can about the downgrade and possibility of a mariadb upgrade at the same time are the other major things. Basically we seem to have the known things under control and now its a matter of investigating the unknowns a bit 19:26:18 and out of interest i did dump the username: id's to see which ones conflicted in case-insensitive; that's an interesting list 19:26:42 yup, a few of them are people getting a second account and needing to set a non conflicting usename 19:26:51 i ran into one for a third-party ci system last week 19:26:53 I noticed patterns like ^ when doing the db cleanups 19:27:44 assuming i get those other bits marked off, do we want to think about a date? 19:28:08 Yes, I made a note on the agenda to talk about scheduling 19:28:24 https://releases.openstack.org/zed/schedule.html is the openstack release schedule which is a big thing to consider when planning this 19:28:35 that makes note of the summit too which is nice 19:29:03 the options seem to be pre summit or post -- pre meaning we have it done and deployed for that, post being i guess slightly safer 19:29:05 personally I feel like I have enough distractions pre summit to make doing it between now and then difficult. But maybe soon after the summit? That gives us plenty of time to announce it too 19:30:19 With previous releases I've done enough java'ing that I'd worry about doing an upgrade and not having some time to do that if necessary again 19:31:12 fair enough, i do like the idea that we go into the summit with it fully updated just as a point about the way we deploy, but i see the argument 19:31:12 sounds good to me 19:31:37 after i mean. just worried about summit prep and maintenance 19:31:44 not that i'm super against before 19:31:54 ya I think if I didn't have a bunch of commitments for the summit I need to prepare for and then attend while there I'd be ok before 19:32:10 but I do have those things and I need to be sure I get around to them :) 19:32:52 The week after a summit tends to be pretty quiet too (not sure if that will hold during a pandemic) so could be a good time for it from a user perspective 19:32:56 something like the 12th? 19:33:39 I return the 11th. The 12th would work for me assuming I can power through jetlag 19:34:06 as far as conflicts go thats a fairly easy one to deal with. 19:34:15 fungi: frickler ^ thoughts? 19:34:46 I won't summit so I'm pretty flexible about it 19:34:50 i'm not going to be around i don't think... checking 19:35:12 hrm 13th june is a public holiday here actually, so i am likely as not to be afk from that weekend 19:35:27 yeah, i'll be on my way home from boston on the 12th 19:35:44 what about the 19th then? gives us a week to recover from the event then do the upgrade 19:35:54 i can absolutely do the 19th 19:35:58 or depending on location enjoy a public holiday :) 19:36:33 and then we can also give everyone a months notice if we announce it in the next day or two 19:37:32 ++ i'll be happy to drive and do it very early on my monday, and hopefully !australian people can just be standyby in case of issues 19:37:56 sounds like a plan 19:38:17 and to be clear thats australian 20th, but late 19th UTC ? 19:38:48 ianw: and is that something you want to send email for or should I work on that bit? 19:38:57 say 20:00 UTC if that's ok. that's 6am which is fine 19:39:05 wfm 19:39:10 i can send the announcement today 19:39:29 June 19 starting at 20:00 UTC. Shouldn't take more than a few minutes so blocking out an hour is probably plenty? 19:39:42 (I really do appreciate that gerrit upgrades are quick now) 19:39:52 ++ (famous last words, haha) 19:40:54 Anything else on the subject of Gerrit upgrades? 19:41:04 but yeah, we really have a ton of ci/cd that gives us great confidence! 19:41:07 ++ 19:41:15 nope, sounds good, thanks! 19:41:24 wfm! 19:41:30 #topic Open Discussion 19:41:55 Just to reiterate I do plan to get a new annual cert for wiki.o.o next week (one of my pre summit todos that interferes with pre summit gerrit upgrades) 19:42:04 frickler: ^ fyi since you called that out 19:42:09 i have thing 19:42:30 corvus: go for it 19:43:36 i'm a little behind, but shortly i'm going to send an email reminder to zuul-discuss about the deprecation of the 'queue' attribute on project-pipeline stanzas 19:43:51 we (opendev) should run this script to find instances of it: https://opendev.org/zuul/zuul/src/branch/master/tools/deprecated-queues.py 19:44:11 or at least our tenants should :) 19:44:26 so my question is, how would we like to handle that? 19:44:29 I don't mind running it and sending email to our various users. I have done similar in the past for zuul config errors 19:44:57 cool, should be easy to run, and you can do a whole tenant at a time 19:45:07 istr there were many instances in the openstack tenant 19:45:10 and we have only a handful of tenants 19:45:18 there's like 1 or 2 in zuul; i'll take care of that 19:45:19 corvus: any ballpark on when it will become an error to have in your config? 19:45:50 clarkb: v7 i think; but no idea when that is. probably > 1 mo. 19:46:13 ok that works. Mostly so that I can warn them with info on when it will convert to an error in the emails 19:46:25 (should have been v6, but i think v6 was too fast/urgent for that) 19:46:45 note that "become an error" in the case of a project-pipeline definition is sort of equivalent to "stop running many/all jobs" 19:46:52 ah right 19:47:11 I'll also call this out in the TC meeting thursday (that is likely to be before I send email about it) so that they are aware 19:47:23 and ya I spot checked with codesearch last week and there are a number with those configs in place 19:48:16 I'll likely get to running it and sending emails on thursady just not before the first thing in the mroning meeting 19:48:20 that's all from me 19:49:00 I'll give ti a couple more minutes for any other topics and if not we can end 10 minutes early 19:51:05 Seems like that was it. Thank you everyone! 19:51:19 As always feel free to reach out in #opendev or on the mailing list if something comes up before the next meeting 19:51:24 #endmeeting