19:01:19 <clarkb> #startmeeting infra
19:01:19 <opendevmeet> Meeting started Tue Jan 25 19:01:19 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:19 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:19 <opendevmeet> The meeting name has been set to 'infra'
19:01:25 <clarkb> #link https://lists.opendev.org/pipermail/service-discuss/2022-January/000315.html Our Agenda
19:01:48 <clarkb> #topic Announcements
19:01:53 <clarkb> OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/
19:02:16 <clarkb> I'm going to propose something related to opendev once I can sort out what I want to talk about
19:02:23 <clarkb> if you've got something interesting to share definitely go for it
19:02:32 <clarkb> Service coordinator nominations run January 25, 2022 - February 8, 2022 That is now :)
19:02:56 <clarkb> It has been 6 months since the last service coordinator election. I'm more than happy for someone else to take it one and run these meetings :)
19:03:03 <clarkb> If you are interested but need more info feel free to reach out
19:03:14 <clarkb> You have a couple of weeks to make up your mind
19:03:24 <clarkb> #topic Actions from last meeting
19:03:30 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-01-18-19.01.txt minutes from last meeting
19:03:43 <clarkb> None recorded. We can dive right into the agenda then
19:03:48 <clarkb> #topic Topics
19:03:52 <clarkb> #topic Gerrit 3.4 upgrade
19:04:05 <clarkb> This is a last minute meeting chair addition to the agenda.
19:04:17 <clarkb> Yesterday we renamed a couple of projects and upgraded Gerrit to 3.4 from 3.3
19:04:30 <clarkb> Overall Gerrit seems to be working, but we did discovered that acls for pushing tags don't work correctly.
19:04:58 <clarkb> fungi has a workaround update in progress which should get project releases moving again. I diagnosed the issue and filed a bug upstream as well as pushed a fix to upstream gerrit
19:05:09 <clarkb> #link https://bugs.chromium.org/p/gerrit/issues/detail?id=15616 Gerrit 3.4 tag pushing acl bug
19:05:24 <clarkb> #link https://gerrit-review.googlesource.com/c/gerrit/+/328839 Gerrit 3.4 tag pushing acl fix
19:05:54 <fungi> yeah, workaround is deploying now and i'll send an announcement to service-announce (already drafted) once that's done
19:06:01 <clarkb> I expect we'll be able to move past this one relatively quickly once we get fungi's workaround in place. We can then remove those updates once upstream lands a/the fix
19:06:24 <fungi> the removal is also already proposed, just wip
19:06:28 <clarkb> Please let us know if you notice any other odd behavior. At least one user noticed that gerrit doesn't check mergeablity anymore (this was documented as a breaking change in the release notes)
19:06:45 <fungi> it may need some rebasing/updating before it can merge, depending on how long we go without a proper fix applied
19:06:55 <clarkb> good point
19:07:13 <fungi> the litner script change included in it will tell us if that's the case
19:07:22 <clarkb> Also a huge thank you to everyone who helped with the renames and upgrade and all the testing and prep and work that goes into making this happen relatively smoothly
19:07:36 <clarkb> It is nice that we don't end up stuck on gerrit releases for half a decade because upgrading is almost impossibl3
19:07:36 <fungi> so it's safe to approve if in doubt, and we can update it if zuul kicks it back
19:08:12 <clarkb> we can also audit the repo after it merges
19:09:16 <clarkb> #topic Improving OpenDev's CD throughput
19:09:40 <clarkb> ianw isn't able to make it today, but I think he largely needs reviews on this. unfortunately due to all the other happenings (like project renames and gerrit upgrades) I haven't had time to get to this yet.
19:09:46 <clarkb> #link https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets
19:09:50 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog
19:09:55 <clarkb> please take a look and leave your thoughts when you have time
19:10:01 <clarkb> #topic Container maintenance
19:10:21 <clarkb> Similarly I haven't had time to dive into the dedicated user chagne stuff. jentoio we should just set up an hour to go over this and then I'll commit to it :)
19:10:33 <clarkb> #link https://etherpad.opendev.org/p/opendev-container-maintenance
19:10:41 <clarkb> I did start to classify how difficult/painful I thought each of these might be
19:10:47 <clarkb> that info is in the etherpad
19:11:12 <jentoio> sounds good, will take a look at it
19:11:41 <clarkb> #topic Nodepool Image Cleanup
19:11:57 <clarkb> In the email announcements I sent about this I basically said end of January we'd remove centos 8
19:12:09 <clarkb> seems a number of projects are working through the switch to stream (or removing the jobs entirely)
19:12:22 <clarkb> I think we are still on track to remove centos 8 from nodepool and our mirrors starting next week as a result
19:12:52 <clarkb> One thing to keep in mind is that the whole ping doesn't work situation is still a thing aiui. So some people converting to stream may run into problems. But there isn't much we can do about that. CentOS 8 Stream is actually broken
19:13:15 <fungi> broken for anyone who want to run ping as a normal user at least
19:13:20 <clarkb> right that
19:14:46 <clarkb> #topic Spring cleaning for old reviews
19:14:58 <clarkb> frickler: brought this up and I asked people to think a bit about it last week
19:15:29 <clarkb> For me personally I think we can go ahead and do a time based abandoning. I would start at 2 years rather than one though just looking at my changes some of them are older than a yaer and still potentially worth having in my list
19:15:36 <fungi> a component of this may also be spring cleaning for git repos we could be retiring entirely
19:15:51 <clarkb> I think a big part of that is our switch to ansible + docker really picked up steam in that time frame
19:16:09 <clarkb> fungi: ya I noticed a number of our old puppet module repos are getting the tag workaround and many of them could just be retired I expect
19:17:08 <fungi> i do have quite a few reviews waiting since 2014, but a bunch are obsolete because of exactly that
19:17:42 <clarkb> maybe we take two axis then. First is determine which repos can have all changes abandoned and then retire the repos. Then for everything else do an auto abandon at ~2 years?
19:17:49 <clarkb> frickler: does something like that seem reasonable?
19:18:04 <fungi> looks like the oldest thing i have not obsoleted by puppet droppage is a stack to rework return codes in bindep from 2017
19:19:03 <fungi> but some stuff >2yo like https://review.opendev.org/489711 probably warrants a second look
19:19:38 <fungi> probably got done or obsoleted more recently, but i'd want to be sure
19:19:44 <frickler> sorry, was distracted, catching up
19:19:53 <clarkb> fungi: ++
19:21:24 <frickler> so for comparison, devstack is in a similar situation. I started going through old reviews there manually, but it is very time consuming and I haven't gotten very far yet
19:21:52 <frickler> otoh I did find a small number of patches that were worth reviving
19:22:34 <fungi> yeah, i'm okay with forging ahead, i'll just make a very close inspection of the abandomnents gerrit notifies me about and hopefully find time to look into whether they need to be restored
19:22:44 <clarkb> right we can always restore later.
19:23:03 <clarkb> And I think abandoning things that should be retired next will prune quite a bit potentially
19:24:56 <clarkb> If that sounds good I can probably help with the actual tasking end of next week? I've got a bunch of stuff I need to get done this week like CFP things and the gerrit stuff and prep for centos 8 removal and so on
19:25:05 <clarkb> But once that tapers off I should have some time to help
19:26:11 <clarkb> Anything else on this topic?
19:26:22 <frickler> I don't think that I would have much time for anything beyond doing things scripted
19:26:46 <frickler> but if we agree to not do this yet, I'm fine with it, too
19:26:58 <clarkb> frickler: ok, maybe I can help produce a list of repos to retire and you can script up abandoning all changes for those repos
19:27:12 <frickler> ack
19:27:20 <clarkb> and we can give everyone a week or two to look over old changes if they like and make them not old then do a 2 year old abandonment
19:27:47 <clarkb> I can help with the digging aspect if you do the scripting bits. I think that should work well
19:27:51 <fungi> probably retiring things and then abandoning older changes in whatever's left is the order we want to go in
19:28:10 <clarkb> fungi: ya I guess that would prevent new changes
19:28:16 <clarkb> since the repos go RO
19:28:33 <clarkb> but ya I can start on that aspect late next week I expect
19:29:02 <clarkb> #topic Updating Grafana
19:29:09 <clarkb> #link https://review.opendev.org/c/opendev/grafyaml/+/825990 needed to fix api use against newer grafana
19:29:22 <fungi> well, also it makes it easier for those of us with abandoned changes to know which ones were abandoned because the repos were retired
19:29:22 <clarkb> ianw pointed out this fix to grafyaml is necessary to accomodate api changes in grafana before we upgrade
19:29:40 <clarkb> fungi: ++
19:29:52 <clarkb> If we can get the grafyaml change reviewed that would be good then we can look at updating grafana
19:29:58 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/825410 update to latest grafana
19:31:13 <clarkb> That was basically all I had on the agenda.
19:31:16 <clarkb> #topic Open Discussion
19:31:21 <clarkb> Is there anything else to call out?
19:32:12 <fungi> the tagging workaround deployed successfully so i'm sending the announcement about it to the ml now
19:32:56 <clarkb> I'm going to send mail to the gerrit mailing list after the meeting showing what we found in case anyone else is running into this. Then lunch and then testing my fix
19:33:05 <fungi> thanks
19:35:32 <clarkb> sounds like that may be everything
19:35:36 <clarkb> Thank you everyone!
19:35:40 <clarkb> We'll see you here next week
19:35:46 <fungi> thanks clarkb!
19:36:10 <clarkb> #endmeeting