19:01:19 #startmeeting infra 19:01:19 Meeting started Tue Jan 25 19:01:19 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:19 The meeting name has been set to 'infra' 19:01:25 #link https://lists.opendev.org/pipermail/service-discuss/2022-January/000315.html Our Agenda 19:01:48 #topic Announcements 19:01:53 OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ 19:02:16 I'm going to propose something related to opendev once I can sort out what I want to talk about 19:02:23 if you've got something interesting to share definitely go for it 19:02:32 Service coordinator nominations run January 25, 2022 - February 8, 2022 That is now :) 19:02:56 It has been 6 months since the last service coordinator election. I'm more than happy for someone else to take it one and run these meetings :) 19:03:03 If you are interested but need more info feel free to reach out 19:03:14 You have a couple of weeks to make up your mind 19:03:24 #topic Actions from last meeting 19:03:30 #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-01-18-19.01.txt minutes from last meeting 19:03:43 None recorded. We can dive right into the agenda then 19:03:48 #topic Topics 19:03:52 #topic Gerrit 3.4 upgrade 19:04:05 This is a last minute meeting chair addition to the agenda. 19:04:17 Yesterday we renamed a couple of projects and upgraded Gerrit to 3.4 from 3.3 19:04:30 Overall Gerrit seems to be working, but we did discovered that acls for pushing tags don't work correctly. 19:04:58 fungi has a workaround update in progress which should get project releases moving again. I diagnosed the issue and filed a bug upstream as well as pushed a fix to upstream gerrit 19:05:09 #link https://bugs.chromium.org/p/gerrit/issues/detail?id=15616 Gerrit 3.4 tag pushing acl bug 19:05:24 #link https://gerrit-review.googlesource.com/c/gerrit/+/328839 Gerrit 3.4 tag pushing acl fix 19:05:54 yeah, workaround is deploying now and i'll send an announcement to service-announce (already drafted) once that's done 19:06:01 I expect we'll be able to move past this one relatively quickly once we get fungi's workaround in place. We can then remove those updates once upstream lands a/the fix 19:06:24 the removal is also already proposed, just wip 19:06:28 Please let us know if you notice any other odd behavior. At least one user noticed that gerrit doesn't check mergeablity anymore (this was documented as a breaking change in the release notes) 19:06:45 it may need some rebasing/updating before it can merge, depending on how long we go without a proper fix applied 19:06:55 good point 19:07:13 the litner script change included in it will tell us if that's the case 19:07:22 Also a huge thank you to everyone who helped with the renames and upgrade and all the testing and prep and work that goes into making this happen relatively smoothly 19:07:36 It is nice that we don't end up stuck on gerrit releases for half a decade because upgrading is almost impossibl3 19:07:36 so it's safe to approve if in doubt, and we can update it if zuul kicks it back 19:08:12 we can also audit the repo after it merges 19:09:16 #topic Improving OpenDev's CD throughput 19:09:40 ianw isn't able to make it today, but I think he largely needs reviews on this. unfortunately due to all the other happenings (like project renames and gerrit upgrades) I haven't had time to get to this yet. 19:09:46 #link https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets 19:09:50 #link https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog 19:09:55 please take a look and leave your thoughts when you have time 19:10:01 #topic Container maintenance 19:10:21 Similarly I haven't had time to dive into the dedicated user chagne stuff. jentoio we should just set up an hour to go over this and then I'll commit to it :) 19:10:33 #link https://etherpad.opendev.org/p/opendev-container-maintenance 19:10:41 I did start to classify how difficult/painful I thought each of these might be 19:10:47 that info is in the etherpad 19:11:12 sounds good, will take a look at it 19:11:41 #topic Nodepool Image Cleanup 19:11:57 In the email announcements I sent about this I basically said end of January we'd remove centos 8 19:12:09 seems a number of projects are working through the switch to stream (or removing the jobs entirely) 19:12:22 I think we are still on track to remove centos 8 from nodepool and our mirrors starting next week as a result 19:12:52 One thing to keep in mind is that the whole ping doesn't work situation is still a thing aiui. So some people converting to stream may run into problems. But there isn't much we can do about that. CentOS 8 Stream is actually broken 19:13:15 broken for anyone who want to run ping as a normal user at least 19:13:20 right that 19:14:46 #topic Spring cleaning for old reviews 19:14:58 frickler: brought this up and I asked people to think a bit about it last week 19:15:29 For me personally I think we can go ahead and do a time based abandoning. I would start at 2 years rather than one though just looking at my changes some of them are older than a yaer and still potentially worth having in my list 19:15:36 a component of this may also be spring cleaning for git repos we could be retiring entirely 19:15:51 I think a big part of that is our switch to ansible + docker really picked up steam in that time frame 19:16:09 fungi: ya I noticed a number of our old puppet module repos are getting the tag workaround and many of them could just be retired I expect 19:17:08 i do have quite a few reviews waiting since 2014, but a bunch are obsolete because of exactly that 19:17:42 maybe we take two axis then. First is determine which repos can have all changes abandoned and then retire the repos. Then for everything else do an auto abandon at ~2 years? 19:17:49 frickler: does something like that seem reasonable? 19:18:04 looks like the oldest thing i have not obsoleted by puppet droppage is a stack to rework return codes in bindep from 2017 19:19:03 but some stuff >2yo like https://review.opendev.org/489711 probably warrants a second look 19:19:38 probably got done or obsoleted more recently, but i'd want to be sure 19:19:44 sorry, was distracted, catching up 19:19:53 fungi: ++ 19:21:24 so for comparison, devstack is in a similar situation. I started going through old reviews there manually, but it is very time consuming and I haven't gotten very far yet 19:21:52 otoh I did find a small number of patches that were worth reviving 19:22:34 yeah, i'm okay with forging ahead, i'll just make a very close inspection of the abandomnents gerrit notifies me about and hopefully find time to look into whether they need to be restored 19:22:44 right we can always restore later. 19:23:03 And I think abandoning things that should be retired next will prune quite a bit potentially 19:24:56 If that sounds good I can probably help with the actual tasking end of next week? I've got a bunch of stuff I need to get done this week like CFP things and the gerrit stuff and prep for centos 8 removal and so on 19:25:05 But once that tapers off I should have some time to help 19:26:11 Anything else on this topic? 19:26:22 I don't think that I would have much time for anything beyond doing things scripted 19:26:46 but if we agree to not do this yet, I'm fine with it, too 19:26:58 frickler: ok, maybe I can help produce a list of repos to retire and you can script up abandoning all changes for those repos 19:27:12 ack 19:27:20 and we can give everyone a week or two to look over old changes if they like and make them not old then do a 2 year old abandonment 19:27:47 I can help with the digging aspect if you do the scripting bits. I think that should work well 19:27:51 probably retiring things and then abandoning older changes in whatever's left is the order we want to go in 19:28:10 fungi: ya I guess that would prevent new changes 19:28:16 since the repos go RO 19:28:33 but ya I can start on that aspect late next week I expect 19:29:02 #topic Updating Grafana 19:29:09 #link https://review.opendev.org/c/opendev/grafyaml/+/825990 needed to fix api use against newer grafana 19:29:22 well, also it makes it easier for those of us with abandoned changes to know which ones were abandoned because the repos were retired 19:29:22 ianw pointed out this fix to grafyaml is necessary to accomodate api changes in grafana before we upgrade 19:29:40 fungi: ++ 19:29:52 If we can get the grafyaml change reviewed that would be good then we can look at updating grafana 19:29:58 #link https://review.opendev.org/c/opendev/system-config/+/825410 update to latest grafana 19:31:13 That was basically all I had on the agenda. 19:31:16 #topic Open Discussion 19:31:21 Is there anything else to call out? 19:32:12 the tagging workaround deployed successfully so i'm sending the announcement about it to the ml now 19:32:56 I'm going to send mail to the gerrit mailing list after the meeting showing what we found in case anyone else is running into this. Then lunch and then testing my fix 19:33:05 thanks 19:35:32 sounds like that may be everything 19:35:36 Thank you everyone! 19:35:40 We'll see you here next week 19:35:46 thanks clarkb! 19:36:10 #endmeeting