19:01:09 <clarkb> #startmeeting infra
19:01:09 <opendevmeet> Meeting started Tue Apr 12 19:01:09 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:09 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:09 <opendevmeet> The meeting name has been set to 'infra'
19:01:21 <clarkb> #link https://lists.opendev.org/pipermail/service-discuss/2022-April/000331.html Our Agenda
19:01:34 <clarkb> #topic Announcements
19:02:09 <clarkb> No announcements. We've got the PTG behind us and a couple of months to the summit. I don't think there are any other major events that are likely to impact us until the summit
19:03:02 <clarkb> #topic Actions from last meeting
19:03:10 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-04-05-19.01.txt minutes from last meeting
19:03:32 <clarkb> no actions recorded last meeting. I did think about dropping this part of hte meeting but then ianw took an action (now complete) so I didn't at the time. I think we can keep it for now
19:03:35 <clarkb> #topic Topics
19:03:39 <clarkb> Let's dive right in
19:03:50 <clarkb> #topic Improving OpenDev CD Throughput
19:04:40 <clarkb> The zuul changes that remove the patched ansible stuff have begun to arrive in gerrit. I have not yet had a chance to review them but we should probably try to stay on top of that
19:05:40 <clarkb> As far as meeting goes to discuss our use of Zuul for CD and the potential impacts ^ has I can do tomorrow around 2000 or 2100 UTC  but then the rest of the week is looking less good for me. Could do Monday next week? I know frickler is on vacation already and not sure if that bumps into ianw's holidays?
19:06:23 <fungi> it's probably not urgent if we want to push it off a bit
19:06:30 <ianw> umm that is easter weekend here, unlikely to be around
19:06:53 <clarkb> ya I know. Not sure when you'll be back :)
19:07:01 <fungi> (i mean, like, push it to closer to the end of april)
19:07:06 <clarkb> fungi: ah ya that works for me too
19:08:00 <clarkb> ya ok why don't we worry about finding time after the holiday then
19:08:18 <ianw> ++
19:08:38 <clarkb> and I will try to review those changes and keep on top of the incremental updates that I expect opendev to start seeing in the near future (since we deploy from ~master)
19:09:58 <fungi> from my perspective, the ansible restrictions in zuul aren't really adding any effective security layer, so when they end up in our deployment doesn't have a lot of bearing on when or if we make changes to our cd approach
19:10:44 <clarkb> good point
19:11:01 <fungi> the ansible restrictions were always wishful thinking from a security perspective
19:11:15 <fungi> them going away doesn't change that
19:12:22 <clarkb> ya I think where we ended up is that the change prompts a discussion but not necessarily an urgent reworking of anything
19:12:34 <clarkb> good to make sure we're still comfortable with things periodically
19:13:25 <clarkb> Seems like that may be it for this topic. We can pick this up again afte rthe holiday (which ya'll should enjoy)
19:13:29 <clarkb> #topic Container Maintenance
19:13:49 <clarkb> Don't see any changes to review on this yet. jentoio feel free to reach out with questions. Though I totally understand if you are busy
19:14:00 <jentoio> working on grafana should have something to review soon
19:14:07 <clarkb> that is great to hear. THank you!
19:14:41 <clarkb> #topic Spring Cleaning Reviews
19:14:59 <clarkb> #link https://review.opendev.org/q/project:opendev/system-config+status:open+topic:system-config-cleanup Resurrected changes
19:15:15 <clarkb> These changes have been getting reviews. Thank you everyone
19:15:53 <clarkb> ianw dug into some of the gerrit config related changes and pointed out that we probably don't need to incrase our open file ulimits. I suspect that our more aggressive git packing is to blame for this
19:16:09 <clarkb> Should we go ahead and abandon https://review.opendev.org/c/opendev/system-config/+/445616 given that?
19:17:10 <ianw> i think so, i think if we start to see it at this point we should debug again, as it seems like a lot has changed since then
19:17:14 <clarkb> ++
19:17:20 <clarkb> I'll go ahead and abandon it after the meetingthen
19:17:40 <clarkb> And the other thing ianw noticed was that while we do have git diff timeouts and there is an old change to increase that timeout it seems that only stackalytics generates those errors. So we may be able to work with stackalytics and avoid landing a change like this too
19:18:05 <ianw> ++ i will send an email today about that to relevant parties
19:19:01 <clarkb> https://review.opendev.org/c/opendev/system-config/+/529426 https://review.opendev.org/c/opendev/system-config/+/689858 are a couple of super easy ones if you have time fungi
19:19:09 <clarkb> there are others that could use review too but ar elikely a bit more involved
19:19:11 <fungi> sure!
19:19:37 <clarkb> thanks again everyone for whittling away at this. We aren't down to the ~10 or so changes that is probably ideal, but it is progress :)
19:20:18 <ianw> thanks for driving it too!
19:20:57 <clarkb> #topic Scheduling Project Renames
19:21:36 <clarkb> so just today my friday (and a bit of thursday) schedule has changed due to kids being out of school and wife needing to do things. This means I'm definitely not going to be able to drive this but can possibly be a second set of hands on friday still
19:21:56 <clarkb> I'm happy to reschedule, but also sounded like fungi didn't htink it was a problem for him to drive it and I'll do my best to support
19:22:16 <fungi> yeah, i can take care of it. we've still only got the one rename to do, right?
19:22:38 <clarkb> yup I've only seen the one
19:23:12 <clarkb> Ok sounds like we're still on track then. Definitely say something if we need to reschedule and I can send email
19:23:27 <fungi> sure thing
19:23:31 <clarkb> #topic Time for a git-review release
19:24:02 <clarkb> I think we're || that close to being ready for a git-review release? And in particular you were asking for clarifation on whether or not we are comfortable dropping support for older python3 and I guess older git ~1.8 now?
19:24:38 <clarkb> Where I've landed on that is I think it is ok for older platforms to use older git-review. They can also manually git push gerrit HEAD:refs/for/$branch as well (which is how I work with upstream gerrit)
19:24:54 <fungi> yeah, the python 3.5 support removal merged independently of the change which needed it
19:25:15 <clarkb> Put another way I'm happy to hav ea release note that says python3.6 or newer is necessary as well as anothe release note saying git 2.10.x or whatever the versions is is required
19:25:21 <fungi> so i've pushed up a revert in case we decide to defer or adjust the other change resulting in us not needing to drop 3.5 after all
19:25:56 <clarkb> ianw: ^ you indicated a concern for centos 7 and its git version (~1.8) do you know of active users on that platform that wouldn't be able to pin git-review? Or was that just a general concern?
19:26:17 <fungi> yeah, the other change still under discussion includes such a release note about git minimum version
19:26:34 <ianw> it was just a general concern about a platform with 3.6 and a lower git ootb
19:27:06 <clarkb> Got it, I think if the relase notes (and maybe the manpage/docs) indicate a minimum git version we're ok there
19:27:07 <ianw> i did propose https://review.opendev.org/c/opendev/git-review/+/837451 to enforce the minimum version with a sane message
19:27:13 <clarkb> ah ok that works too
19:27:22 <fungi> to be clear, the only reason i pushed the python 3.5 removal change up was because the otehr change needed newer git than we have on xenial, so if we went forward there it would force us to drop the xenial job, removing our python 3.5 testing
19:27:56 <clarkb> fungi: yup and I think we're trying to remove xenial testing anyway so this is all a step in the right direction
19:28:04 <fungi> if we're going to the trouble of testing whether there's git 2.10 present, we could simply use that to determine whether to add that command-line flag anyway
19:28:10 <clarkb> doesn't sound like anyone is really opposed to the is direction we just need to review the changes and land them?
19:28:12 <ianw> yeah, i think we have to move on.  centos 7 is right on the edge of things i'd expect maybe someone still uses
19:28:35 <clarkb> fungi: ya but then we aren't testing really old git either...
19:28:48 <fungi> but if the goal is to support centos 7, we need jobs which test it
19:29:03 <fungi> right
19:29:25 <clarkb> thats the main reason why I'm ok with moving deps up and telling users to pin on older platforms
19:29:37 <ianw> that's fair enough.  i would just like it to fail with "your git is too old" rather than raise an exception in the middle with the system() call passing invalid flags, though
19:29:45 <clarkb> ianw: yup that wfm
19:30:19 <ianw> we already do a git version check anyway, so it's mostly just pulling it out into common code
19:30:21 <fungi> okay, so seems like we have consensus to merge the signed commits workaround, require git>=2.10, drop python 3.5 support, and tag a 2.3.0 (or do people think dropping a python version means we need to make it 3.0.0?)
19:30:35 <clarkb> fungi: 2.3.0 is probably fine here
19:31:03 <fungi> given that, i'll drop the revert, and we can finish up merging the rest
19:31:03 <ianw> yeah, i could go either way
19:31:06 <clarkb> this is a command line tool not a library where these changes might have transitive affects
19:31:19 <clarkb> if it were a lib 3.0 would probably be appropriate
19:31:44 <ianw> or you can take the linux approach of "this feels like a 3-type-thing" :)
19:32:22 <ianw> not sure git-review will ever have a "big" change, it seems like it does what it does
19:32:51 <fungi> well, we made 2.0.0 when we dropped python 2.7 support
19:32:53 <clarkb> in general I'm in favor of ensuring these tools are easier to use for active up to date users. And I think simplifying testing and bumping those minimums helps us do that without a bunch of effort
19:33:05 <ianw> ++
19:33:29 <fungi> yeah, totally
19:34:56 <clarkb> alright sounds like we have a plan then. I'll look to review the followup with helpful error message shortly
19:35:03 <clarkb> #topic Time for a Bindep release
19:35:21 <clarkb> Bindep has similar received some updates from modern users. In this case a change adding Alma Linux support
19:35:35 <clarkb> but before that can land we need a change to address our test suite's trouble with the latest distro 1.7.0 release
19:36:14 <clarkb> I've just realized we should add a release note for the ALma linux support. I'll get that pushed up after the meeting too
19:36:30 <clarkb> then if we can land the changes (fedora mirror trouble currently hitting us) I think we can push a 2.11.0 release?
19:37:06 <clarkb> does that sound reasonable?
19:37:39 <clarkb> oh wait there is a release note in the change already. Even better :)
19:38:31 <fungi> yeah
19:38:49 <fungi> looks like we haven't merged any changes since 2.10.2 so these would be the only ones to consider for the release
19:39:01 <ianw> seems ok to me.  this seems the second time the mirror has been out of sync, i'll try having a look, i know it's been discussed today extensively
19:39:16 <fungi> and i agree, alma linux support and changing a dep minimum count for a minor rev
19:39:47 <clarkb> cool I'll try to stay on top of that and ensure it gets done
19:39:48 <fungi> ianw: looks like we pulled a lot of changes from it around 05-06z today and then updates after that were empty
19:40:00 <fungi> so i think the mirror we're pulling from is stuck mid-update
19:41:40 <ianw> :/
19:41:58 <clarkb> ya if we can find a more up to date upstream we could swap them out
19:42:44 <fungi> though there might be a lengthy resync due to slight differences on the new mirror
19:43:15 <fungi> (variations in timestamp resolution, perms, et cetera)
19:44:55 <clarkb> right
19:44:58 <clarkb> Anything else on this topic?
19:45:04 <fungi> not from me
19:45:14 <clarkb> #topic Removing our ELK systems
19:45:26 <clarkb> This is a topic that I'm ninja adding to the agenda
19:46:03 <clarkb> I think dpawlik's efforts to use opensearch for openstack's needs are ocming along well and recently fungi had to restart the logstash geard anyway since it had a massive queue and had fallen over
19:46:14 <clarkb> additionally we told openstack we could keep things running until theend of yoga which just happened
19:46:30 <clarkb> All that to say any objections to start unwinding these services and return resources back to the cloud provider?
19:46:44 <clarkb> I think the first step is to remove the gearman log job submissions from our base jobs
19:46:55 <fungi> agreed
19:47:02 <clarkb> then if no panic occurs we can shutdown the gearman works, the elasticsearches, and the kibana etc
19:47:23 <fungi> i was tempted to do that during the release since it was clearly not actually serving any purpose and only slowing the jobs down
19:47:26 <corvus> there may be a grafana dashboard or some graphs to clean up after that too
19:47:41 <clarkb> corvus: oh right good thing to note
19:48:07 <clarkb> the zuul dashboard tracks a job queue. I'll go ahead and propos ea clneap of that as one of the first things since its impact is small
19:48:27 <clarkb> fungi: I should be able to push some changes up for that this afternoon
19:49:06 <fungi> i'll be able to review after dinner, thanks
19:50:23 <clarkb> cool, doesn't sound like there is any concern proceeding with the plan we put in place about a year ago. I just wanted to start making progress on it since the deadline we gave has now passed
19:51:57 <clarkb> #topic Open Discussion
19:51:59 <clarkb> Anything else?
19:52:12 <fungi> one thing which came up earlier today so i didn't have time to get added to the agenda
19:52:34 <fungi> opendev/irc-meetings could use a broader pool of core reviewers
19:53:11 <fungi> changes for it are relatively time-sensitive, but trivial, usually fine with only a single core reviewer approving
19:53:38 <fungi> right now its core team is our infra core set plus ttx
19:53:55 <fungi> and he's indicated to me that he probably can't give it the attention he has in the past
19:54:07 <clarkb> I'd be willing to add whoever volunteers if they've had some history with code review on opendev
19:54:10 <fungi> figured it's an opportunity to do something like we do with project-config
19:54:14 <clarkb> ++
19:54:39 <fungi> maybe even lower-risk, so could be a great place for newer folks to get familiar with the approval workflow
19:55:52 <fungi> but also i wonder if we have a lot of smaller projects like that which might benefit from a blanket core review group to ease management/membership. i haven't really put much thought into that though
19:56:22 <fungi> getting more volunteer reviewers is the first step either way
19:56:38 <fungi> so probably best not to prematurely optimize for them
19:56:48 <clarkb> yup I think finding interesting people is the hard part :)
19:56:59 <fungi> well, interested people anyway
19:57:01 <clarkb> but if we can find them then I agree that is a low risk project to add them to and get them involved
19:57:03 <clarkb> oh heh
19:57:11 <clarkb> well I'm definitely not interesting :)
19:57:13 <fungi> we have plenty of interesting people (but more never hurts!)
19:58:08 <fungi> anyway, sounds like there's not a concern with adding some new reviewers to irc-meetings if we get volunteers
19:58:32 <clarkb> not from me
19:58:45 <fungi> projects like that and project-config are where i see an opportunity to direct community volunteers to help with some of the review burden
19:59:03 <clarkb> ++
19:59:19 <fungi> if projects feel like their changes to those aren't merging quickly enough, there's a pretty obvious answer ;)
19:59:55 <ianw> oh i had a quick one to consider after fixing our status tweeting
19:59:58 <ianw> #link https://review.opendev.org/c/opendev/statusbot/+/837082
20:00:21 <ianw> will also log #log messages.  just to increase engagement with what we're doing.  just a thought
20:00:42 <clarkb> oh we both +2'd at the same time :)
20:00:49 <clarkb> I'm going tocall the meeting here. Thank you everyone
20:01:00 <fungi> thanks clarkb!
20:01:00 <clarkb> enjoy the long weekend if it is one for you :)
20:01:06 <clarkb> #endmeeting