19:01:09 #startmeeting infra 19:01:09 Meeting started Tue Apr 12 19:01:09 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:09 The meeting name has been set to 'infra' 19:01:21 #link https://lists.opendev.org/pipermail/service-discuss/2022-April/000331.html Our Agenda 19:01:34 #topic Announcements 19:02:09 No announcements. We've got the PTG behind us and a couple of months to the summit. I don't think there are any other major events that are likely to impact us until the summit 19:03:02 #topic Actions from last meeting 19:03:10 #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-04-05-19.01.txt minutes from last meeting 19:03:32 no actions recorded last meeting. I did think about dropping this part of hte meeting but then ianw took an action (now complete) so I didn't at the time. I think we can keep it for now 19:03:35 #topic Topics 19:03:39 Let's dive right in 19:03:50 #topic Improving OpenDev CD Throughput 19:04:40 The zuul changes that remove the patched ansible stuff have begun to arrive in gerrit. I have not yet had a chance to review them but we should probably try to stay on top of that 19:05:40 As far as meeting goes to discuss our use of Zuul for CD and the potential impacts ^ has I can do tomorrow around 2000 or 2100 UTC but then the rest of the week is looking less good for me. Could do Monday next week? I know frickler is on vacation already and not sure if that bumps into ianw's holidays? 19:06:23 it's probably not urgent if we want to push it off a bit 19:06:30 umm that is easter weekend here, unlikely to be around 19:06:53 ya I know. Not sure when you'll be back :) 19:07:01 (i mean, like, push it to closer to the end of april) 19:07:06 fungi: ah ya that works for me too 19:08:00 ya ok why don't we worry about finding time after the holiday then 19:08:18 ++ 19:08:38 and I will try to review those changes and keep on top of the incremental updates that I expect opendev to start seeing in the near future (since we deploy from ~master) 19:09:58 from my perspective, the ansible restrictions in zuul aren't really adding any effective security layer, so when they end up in our deployment doesn't have a lot of bearing on when or if we make changes to our cd approach 19:10:44 good point 19:11:01 the ansible restrictions were always wishful thinking from a security perspective 19:11:15 them going away doesn't change that 19:12:22 ya I think where we ended up is that the change prompts a discussion but not necessarily an urgent reworking of anything 19:12:34 good to make sure we're still comfortable with things periodically 19:13:25 Seems like that may be it for this topic. We can pick this up again afte rthe holiday (which ya'll should enjoy) 19:13:29 #topic Container Maintenance 19:13:49 Don't see any changes to review on this yet. jentoio feel free to reach out with questions. Though I totally understand if you are busy 19:14:00 working on grafana should have something to review soon 19:14:07 that is great to hear. THank you! 19:14:41 #topic Spring Cleaning Reviews 19:14:59 #link https://review.opendev.org/q/project:opendev/system-config+status:open+topic:system-config-cleanup Resurrected changes 19:15:15 These changes have been getting reviews. Thank you everyone 19:15:53 ianw dug into some of the gerrit config related changes and pointed out that we probably don't need to incrase our open file ulimits. I suspect that our more aggressive git packing is to blame for this 19:16:09 Should we go ahead and abandon https://review.opendev.org/c/opendev/system-config/+/445616 given that? 19:17:10 i think so, i think if we start to see it at this point we should debug again, as it seems like a lot has changed since then 19:17:14 ++ 19:17:20 I'll go ahead and abandon it after the meetingthen 19:17:40 And the other thing ianw noticed was that while we do have git diff timeouts and there is an old change to increase that timeout it seems that only stackalytics generates those errors. So we may be able to work with stackalytics and avoid landing a change like this too 19:18:05 ++ i will send an email today about that to relevant parties 19:19:01 https://review.opendev.org/c/opendev/system-config/+/529426 https://review.opendev.org/c/opendev/system-config/+/689858 are a couple of super easy ones if you have time fungi 19:19:09 there are others that could use review too but ar elikely a bit more involved 19:19:11 sure! 19:19:37 thanks again everyone for whittling away at this. We aren't down to the ~10 or so changes that is probably ideal, but it is progress :) 19:20:18 thanks for driving it too! 19:20:57 #topic Scheduling Project Renames 19:21:36 so just today my friday (and a bit of thursday) schedule has changed due to kids being out of school and wife needing to do things. This means I'm definitely not going to be able to drive this but can possibly be a second set of hands on friday still 19:21:56 I'm happy to reschedule, but also sounded like fungi didn't htink it was a problem for him to drive it and I'll do my best to support 19:22:16 yeah, i can take care of it. we've still only got the one rename to do, right? 19:22:38 yup I've only seen the one 19:23:12 Ok sounds like we're still on track then. Definitely say something if we need to reschedule and I can send email 19:23:27 sure thing 19:23:31 #topic Time for a git-review release 19:24:02 I think we're || that close to being ready for a git-review release? And in particular you were asking for clarifation on whether or not we are comfortable dropping support for older python3 and I guess older git ~1.8 now? 19:24:38 Where I've landed on that is I think it is ok for older platforms to use older git-review. They can also manually git push gerrit HEAD:refs/for/$branch as well (which is how I work with upstream gerrit) 19:24:54 yeah, the python 3.5 support removal merged independently of the change which needed it 19:25:15 Put another way I'm happy to hav ea release note that says python3.6 or newer is necessary as well as anothe release note saying git 2.10.x or whatever the versions is is required 19:25:21 so i've pushed up a revert in case we decide to defer or adjust the other change resulting in us not needing to drop 3.5 after all 19:25:56 ianw: ^ you indicated a concern for centos 7 and its git version (~1.8) do you know of active users on that platform that wouldn't be able to pin git-review? Or was that just a general concern? 19:26:17 yeah, the other change still under discussion includes such a release note about git minimum version 19:26:34 it was just a general concern about a platform with 3.6 and a lower git ootb 19:27:06 Got it, I think if the relase notes (and maybe the manpage/docs) indicate a minimum git version we're ok there 19:27:07 i did propose https://review.opendev.org/c/opendev/git-review/+/837451 to enforce the minimum version with a sane message 19:27:13 ah ok that works too 19:27:22 to be clear, the only reason i pushed the python 3.5 removal change up was because the otehr change needed newer git than we have on xenial, so if we went forward there it would force us to drop the xenial job, removing our python 3.5 testing 19:27:56 fungi: yup and I think we're trying to remove xenial testing anyway so this is all a step in the right direction 19:28:04 if we're going to the trouble of testing whether there's git 2.10 present, we could simply use that to determine whether to add that command-line flag anyway 19:28:10 doesn't sound like anyone is really opposed to the is direction we just need to review the changes and land them? 19:28:12 yeah, i think we have to move on. centos 7 is right on the edge of things i'd expect maybe someone still uses 19:28:35 fungi: ya but then we aren't testing really old git either... 19:28:48 but if the goal is to support centos 7, we need jobs which test it 19:29:03 right 19:29:25 thats the main reason why I'm ok with moving deps up and telling users to pin on older platforms 19:29:37 that's fair enough. i would just like it to fail with "your git is too old" rather than raise an exception in the middle with the system() call passing invalid flags, though 19:29:45 ianw: yup that wfm 19:30:19 we already do a git version check anyway, so it's mostly just pulling it out into common code 19:30:21 okay, so seems like we have consensus to merge the signed commits workaround, require git>=2.10, drop python 3.5 support, and tag a 2.3.0 (or do people think dropping a python version means we need to make it 3.0.0?) 19:30:35 fungi: 2.3.0 is probably fine here 19:31:03 given that, i'll drop the revert, and we can finish up merging the rest 19:31:03 yeah, i could go either way 19:31:06 this is a command line tool not a library where these changes might have transitive affects 19:31:19 if it were a lib 3.0 would probably be appropriate 19:31:44 or you can take the linux approach of "this feels like a 3-type-thing" :) 19:32:22 not sure git-review will ever have a "big" change, it seems like it does what it does 19:32:51 well, we made 2.0.0 when we dropped python 2.7 support 19:32:53 in general I'm in favor of ensuring these tools are easier to use for active up to date users. And I think simplifying testing and bumping those minimums helps us do that without a bunch of effort 19:33:05 ++ 19:33:29 yeah, totally 19:34:56 alright sounds like we have a plan then. I'll look to review the followup with helpful error message shortly 19:35:03 #topic Time for a Bindep release 19:35:21 Bindep has similar received some updates from modern users. In this case a change adding Alma Linux support 19:35:35 but before that can land we need a change to address our test suite's trouble with the latest distro 1.7.0 release 19:36:14 I've just realized we should add a release note for the ALma linux support. I'll get that pushed up after the meeting too 19:36:30 then if we can land the changes (fedora mirror trouble currently hitting us) I think we can push a 2.11.0 release? 19:37:06 does that sound reasonable? 19:37:39 oh wait there is a release note in the change already. Even better :) 19:38:31 yeah 19:38:49 looks like we haven't merged any changes since 2.10.2 so these would be the only ones to consider for the release 19:39:01 seems ok to me. this seems the second time the mirror has been out of sync, i'll try having a look, i know it's been discussed today extensively 19:39:16 and i agree, alma linux support and changing a dep minimum count for a minor rev 19:39:47 cool I'll try to stay on top of that and ensure it gets done 19:39:48 ianw: looks like we pulled a lot of changes from it around 05-06z today and then updates after that were empty 19:40:00 so i think the mirror we're pulling from is stuck mid-update 19:41:40 :/ 19:41:58 ya if we can find a more up to date upstream we could swap them out 19:42:44 though there might be a lengthy resync due to slight differences on the new mirror 19:43:15 (variations in timestamp resolution, perms, et cetera) 19:44:55 right 19:44:58 Anything else on this topic? 19:45:04 not from me 19:45:14 #topic Removing our ELK systems 19:45:26 This is a topic that I'm ninja adding to the agenda 19:46:03 I think dpawlik's efforts to use opensearch for openstack's needs are ocming along well and recently fungi had to restart the logstash geard anyway since it had a massive queue and had fallen over 19:46:14 additionally we told openstack we could keep things running until theend of yoga which just happened 19:46:30 All that to say any objections to start unwinding these services and return resources back to the cloud provider? 19:46:44 I think the first step is to remove the gearman log job submissions from our base jobs 19:46:55 agreed 19:47:02 then if no panic occurs we can shutdown the gearman works, the elasticsearches, and the kibana etc 19:47:23 i was tempted to do that during the release since it was clearly not actually serving any purpose and only slowing the jobs down 19:47:26 there may be a grafana dashboard or some graphs to clean up after that too 19:47:41 corvus: oh right good thing to note 19:48:07 the zuul dashboard tracks a job queue. I'll go ahead and propos ea clneap of that as one of the first things since its impact is small 19:48:27 fungi: I should be able to push some changes up for that this afternoon 19:49:06 i'll be able to review after dinner, thanks 19:50:23 cool, doesn't sound like there is any concern proceeding with the plan we put in place about a year ago. I just wanted to start making progress on it since the deadline we gave has now passed 19:51:57 #topic Open Discussion 19:51:59 Anything else? 19:52:12 one thing which came up earlier today so i didn't have time to get added to the agenda 19:52:34 opendev/irc-meetings could use a broader pool of core reviewers 19:53:11 changes for it are relatively time-sensitive, but trivial, usually fine with only a single core reviewer approving 19:53:38 right now its core team is our infra core set plus ttx 19:53:55 and he's indicated to me that he probably can't give it the attention he has in the past 19:54:07 I'd be willing to add whoever volunteers if they've had some history with code review on opendev 19:54:10 figured it's an opportunity to do something like we do with project-config 19:54:14 ++ 19:54:39 maybe even lower-risk, so could be a great place for newer folks to get familiar with the approval workflow 19:55:52 but also i wonder if we have a lot of smaller projects like that which might benefit from a blanket core review group to ease management/membership. i haven't really put much thought into that though 19:56:22 getting more volunteer reviewers is the first step either way 19:56:38 so probably best not to prematurely optimize for them 19:56:48 yup I think finding interesting people is the hard part :) 19:56:59 well, interested people anyway 19:57:01 but if we can find them then I agree that is a low risk project to add them to and get them involved 19:57:03 oh heh 19:57:11 well I'm definitely not interesting :) 19:57:13 we have plenty of interesting people (but more never hurts!) 19:58:08 anyway, sounds like there's not a concern with adding some new reviewers to irc-meetings if we get volunteers 19:58:32 not from me 19:58:45 projects like that and project-config are where i see an opportunity to direct community volunteers to help with some of the review burden 19:59:03 ++ 19:59:19 if projects feel like their changes to those aren't merging quickly enough, there's a pretty obvious answer ;) 19:59:55 oh i had a quick one to consider after fixing our status tweeting 19:59:58 #link https://review.opendev.org/c/opendev/statusbot/+/837082 20:00:21 will also log #log messages. just to increase engagement with what we're doing. just a thought 20:00:42 oh we both +2'd at the same time :) 20:00:49 I'm going tocall the meeting here. Thank you everyone 20:01:00 thanks clarkb! 20:01:00 enjoy the long weekend if it is one for you :) 20:01:06 #endmeeting