| clarkb | Just about meeting time. We'll get started in a ocuple of minutes | 18:59 |
|---|---|---|
| ianw | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Apr 12 19:01:09 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/pipermail/service-discuss/2022-April/000331.html Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | No announcements. We've got the PTG behind us and a couple of months to the summit. I don't think there are any other major events that are likely to impact us until the summit | 19:02 |
| clarkb | #topic Actions from last meeting | 19:03 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-04-05-19.01.txt minutes from last meeting | 19:03 |
| clarkb | no actions recorded last meeting. I did think about dropping this part of hte meeting but then ianw took an action (now complete) so I didn't at the time. I think we can keep it for now | 19:03 |
| clarkb | #topic Topics | 19:03 |
| clarkb | Let's dive right in | 19:03 |
| clarkb | #topic Improving OpenDev CD Throughput | 19:03 |
| clarkb | The zuul changes that remove the patched ansible stuff have begun to arrive in gerrit. I have not yet had a chance to review them but we should probably try to stay on top of that | 19:04 |
| clarkb | As far as meeting goes to discuss our use of Zuul for CD and the potential impacts ^ has I can do tomorrow around 2000 or 2100 UTC but then the rest of the week is looking less good for me. Could do Monday next week? I know frickler is on vacation already and not sure if that bumps into ianw's holidays? | 19:05 |
| fungi | it's probably not urgent if we want to push it off a bit | 19:06 |
| ianw | umm that is easter weekend here, unlikely to be around | 19:06 |
| clarkb | ya I know. Not sure when you'll be back :) | 19:06 |
| fungi | (i mean, like, push it to closer to the end of april) | 19:07 |
| clarkb | fungi: ah ya that works for me too | 19:07 |
| clarkb | ya ok why don't we worry about finding time after the holiday then | 19:08 |
| ianw | ++ | 19:08 |
| clarkb | and I will try to review those changes and keep on top of the incremental updates that I expect opendev to start seeing in the near future (since we deploy from ~master) | 19:08 |
| fungi | from my perspective, the ansible restrictions in zuul aren't really adding any effective security layer, so when they end up in our deployment doesn't have a lot of bearing on when or if we make changes to our cd approach | 19:09 |
| clarkb | good point | 19:10 |
| fungi | the ansible restrictions were always wishful thinking from a security perspective | 19:11 |
| fungi | them going away doesn't change that | 19:11 |
| clarkb | ya I think where we ended up is that the change prompts a discussion but not necessarily an urgent reworking of anything | 19:12 |
| clarkb | good to make sure we're still comfortable with things periodically | 19:12 |
| clarkb | Seems like that may be it for this topic. We can pick this up again afte rthe holiday (which ya'll should enjoy) | 19:13 |
| clarkb | #topic Container Maintenance | 19:13 |
| clarkb | Don't see any changes to review on this yet. jentoio feel free to reach out with questions. Though I totally understand if you are busy | 19:13 |
| jentoio | working on grafana should have something to review soon | 19:14 |
| clarkb | that is great to hear. THank you! | 19:14 |
| clarkb | #topic Spring Cleaning Reviews | 19:14 |
| clarkb | #link https://review.opendev.org/q/project:opendev/system-config+status:open+topic:system-config-cleanup Resurrected changes | 19:14 |
| clarkb | These changes have been getting reviews. Thank you everyone | 19:15 |
| clarkb | ianw dug into some of the gerrit config related changes and pointed out that we probably don't need to incrase our open file ulimits. I suspect that our more aggressive git packing is to blame for this | 19:15 |
| clarkb | Should we go ahead and abandon https://review.opendev.org/c/opendev/system-config/+/445616 given that? | 19:16 |
| ianw | i think so, i think if we start to see it at this point we should debug again, as it seems like a lot has changed since then | 19:17 |
| clarkb | ++ | 19:17 |
| clarkb | I'll go ahead and abandon it after the meetingthen | 19:17 |
| clarkb | And the other thing ianw noticed was that while we do have git diff timeouts and there is an old change to increase that timeout it seems that only stackalytics generates those errors. So we may be able to work with stackalytics and avoid landing a change like this too | 19:17 |
| ianw | ++ i will send an email today about that to relevant parties | 19:18 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/529426 https://review.opendev.org/c/opendev/system-config/+/689858 are a couple of super easy ones if you have time fungi | 19:19 |
| clarkb | there are others that could use review too but ar elikely a bit more involved | 19:19 |
| fungi | sure! | 19:19 |
| clarkb | thanks again everyone for whittling away at this. We aren't down to the ~10 or so changes that is probably ideal, but it is progress :) | 19:19 |
| ianw | thanks for driving it too! | 19:20 |
| clarkb | #topic Scheduling Project Renames | 19:20 |
| clarkb | so just today my friday (and a bit of thursday) schedule has changed due to kids being out of school and wife needing to do things. This means I'm definitely not going to be able to drive this but can possibly be a second set of hands on friday still | 19:21 |
| clarkb | I'm happy to reschedule, but also sounded like fungi didn't htink it was a problem for him to drive it and I'll do my best to support | 19:21 |
| fungi | yeah, i can take care of it. we've still only got the one rename to do, right? | 19:22 |
| clarkb | yup I've only seen the one | 19:22 |
| clarkb | Ok sounds like we're still on track then. Definitely say something if we need to reschedule and I can send email | 19:23 |
| fungi | sure thing | 19:23 |
| clarkb | #topic Time for a git-review release | 19:23 |
| clarkb | I think we're || that close to being ready for a git-review release? And in particular you were asking for clarifation on whether or not we are comfortable dropping support for older python3 and I guess older git ~1.8 now? | 19:24 |
| clarkb | Where I've landed on that is I think it is ok for older platforms to use older git-review. They can also manually git push gerrit HEAD:refs/for/$branch as well (which is how I work with upstream gerrit) | 19:24 |
| fungi | yeah, the python 3.5 support removal merged independently of the change which needed it | 19:24 |
| clarkb | Put another way I'm happy to hav ea release note that says python3.6 or newer is necessary as well as anothe release note saying git 2.10.x or whatever the versions is is required | 19:25 |
| fungi | so i've pushed up a revert in case we decide to defer or adjust the other change resulting in us not needing to drop 3.5 after all | 19:25 |
| clarkb | ianw: ^ you indicated a concern for centos 7 and its git version (~1.8) do you know of active users on that platform that wouldn't be able to pin git-review? Or was that just a general concern? | 19:25 |
| fungi | yeah, the other change still under discussion includes such a release note about git minimum version | 19:26 |
| ianw | it was just a general concern about a platform with 3.6 and a lower git ootb | 19:26 |
| clarkb | Got it, I think if the relase notes (and maybe the manpage/docs) indicate a minimum git version we're ok there | 19:27 |
| ianw | i did propose https://review.opendev.org/c/opendev/git-review/+/837451 to enforce the minimum version with a sane message | 19:27 |
| clarkb | ah ok that works too | 19:27 |
| fungi | to be clear, the only reason i pushed the python 3.5 removal change up was because the otehr change needed newer git than we have on xenial, so if we went forward there it would force us to drop the xenial job, removing our python 3.5 testing | 19:27 |
| clarkb | fungi: yup and I think we're trying to remove xenial testing anyway so this is all a step in the right direction | 19:27 |
| fungi | if we're going to the trouble of testing whether there's git 2.10 present, we could simply use that to determine whether to add that command-line flag anyway | 19:28 |
| clarkb | doesn't sound like anyone is really opposed to the is direction we just need to review the changes and land them? | 19:28 |
| ianw | yeah, i think we have to move on. centos 7 is right on the edge of things i'd expect maybe someone still uses | 19:28 |
| clarkb | fungi: ya but then we aren't testing really old git either... | 19:28 |
| fungi | but if the goal is to support centos 7, we need jobs which test it | 19:28 |
| fungi | right | 19:29 |
| clarkb | thats the main reason why I'm ok with moving deps up and telling users to pin on older platforms | 19:29 |
| ianw | that's fair enough. i would just like it to fail with "your git is too old" rather than raise an exception in the middle with the system() call passing invalid flags, though | 19:29 |
| clarkb | ianw: yup that wfm | 19:29 |
| ianw | we already do a git version check anyway, so it's mostly just pulling it out into common code | 19:30 |
| fungi | okay, so seems like we have consensus to merge the signed commits workaround, require git>=2.10, drop python 3.5 support, and tag a 2.3.0 (or do people think dropping a python version means we need to make it 3.0.0?) | 19:30 |
| clarkb | fungi: 2.3.0 is probably fine here | 19:30 |
| fungi | given that, i'll drop the revert, and we can finish up merging the rest | 19:31 |
| ianw | yeah, i could go either way | 19:31 |
| clarkb | this is a command line tool not a library where these changes might have transitive affects | 19:31 |
| clarkb | if it were a lib 3.0 would probably be appropriate | 19:31 |
| ianw | or you can take the linux approach of "this feels like a 3-type-thing" :) | 19:31 |
| ianw | not sure git-review will ever have a "big" change, it seems like it does what it does | 19:32 |
| fungi | well, we made 2.0.0 when we dropped python 2.7 support | 19:32 |
| clarkb | in general I'm in favor of ensuring these tools are easier to use for active up to date users. And I think simplifying testing and bumping those minimums helps us do that without a bunch of effort | 19:32 |
| ianw | ++ | 19:33 |
| fungi | yeah, totally | 19:33 |
| clarkb | alright sounds like we have a plan then. I'll look to review the followup with helpful error message shortly | 19:34 |
| clarkb | #topic Time for a Bindep release | 19:35 |
| clarkb | Bindep has similar received some updates from modern users. In this case a change adding Alma Linux support | 19:35 |
| clarkb | but before that can land we need a change to address our test suite's trouble with the latest distro 1.7.0 release | 19:35 |
| clarkb | I've just realized we should add a release note for the ALma linux support. I'll get that pushed up after the meeting too | 19:36 |
| clarkb | then if we can land the changes (fedora mirror trouble currently hitting us) I think we can push a 2.11.0 release? | 19:36 |
| clarkb | does that sound reasonable? | 19:37 |
| clarkb | oh wait there is a release note in the change already. Even better :) | 19:37 |
| fungi | yeah | 19:38 |
| fungi | looks like we haven't merged any changes since 2.10.2 so these would be the only ones to consider for the release | 19:38 |
| ianw | seems ok to me. this seems the second time the mirror has been out of sync, i'll try having a look, i know it's been discussed today extensively | 19:39 |
| fungi | and i agree, alma linux support and changing a dep minimum count for a minor rev | 19:39 |
| clarkb | cool I'll try to stay on top of that and ensure it gets done | 19:39 |
| fungi | ianw: looks like we pulled a lot of changes from it around 05-06z today and then updates after that were empty | 19:39 |
| fungi | so i think the mirror we're pulling from is stuck mid-update | 19:40 |
| ianw | :/ | 19:41 |
| clarkb | ya if we can find a more up to date upstream we could swap them out | 19:41 |
| fungi | though there might be a lengthy resync due to slight differences on the new mirror | 19:42 |
| fungi | (variations in timestamp resolution, perms, et cetera) | 19:43 |
| clarkb | right | 19:44 |
| clarkb | Anything else on this topic? | 19:44 |
| fungi | not from me | 19:45 |
| clarkb | #topic Removing our ELK systems | 19:45 |
| clarkb | This is a topic that I'm ninja adding to the agenda | 19:45 |
| clarkb | I think dpawlik's efforts to use opensearch for openstack's needs are ocming along well and recently fungi had to restart the logstash geard anyway since it had a massive queue and had fallen over | 19:46 |
| clarkb | additionally we told openstack we could keep things running until theend of yoga which just happened | 19:46 |
| clarkb | All that to say any objections to start unwinding these services and return resources back to the cloud provider? | 19:46 |
| clarkb | I think the first step is to remove the gearman log job submissions from our base jobs | 19:46 |
| fungi | agreed | 19:46 |
| clarkb | then if no panic occurs we can shutdown the gearman works, the elasticsearches, and the kibana etc | 19:47 |
| fungi | i was tempted to do that during the release since it was clearly not actually serving any purpose and only slowing the jobs down | 19:47 |
| corvus | there may be a grafana dashboard or some graphs to clean up after that too | 19:47 |
| clarkb | corvus: oh right good thing to note | 19:47 |
| clarkb | the zuul dashboard tracks a job queue. I'll go ahead and propos ea clneap of that as one of the first things since its impact is small | 19:48 |
| clarkb | fungi: I should be able to push some changes up for that this afternoon | 19:48 |
| fungi | i'll be able to review after dinner, thanks | 19:49 |
| clarkb | cool, doesn't sound like there is any concern proceeding with the plan we put in place about a year ago. I just wanted to start making progress on it since the deadline we gave has now passed | 19:50 |
| clarkb | #topic Open Discussion | 19:51 |
| clarkb | Anything else? | 19:51 |
| fungi | one thing which came up earlier today so i didn't have time to get added to the agenda | 19:52 |
| fungi | opendev/irc-meetings could use a broader pool of core reviewers | 19:52 |
| fungi | changes for it are relatively time-sensitive, but trivial, usually fine with only a single core reviewer approving | 19:53 |
| fungi | right now its core team is our infra core set plus ttx | 19:53 |
| fungi | and he's indicated to me that he probably can't give it the attention he has in the past | 19:53 |
| clarkb | I'd be willing to add whoever volunteers if they've had some history with code review on opendev | 19:54 |
| fungi | figured it's an opportunity to do something like we do with project-config | 19:54 |
| clarkb | ++ | 19:54 |
| fungi | maybe even lower-risk, so could be a great place for newer folks to get familiar with the approval workflow | 19:54 |
| fungi | but also i wonder if we have a lot of smaller projects like that which might benefit from a blanket core review group to ease management/membership. i haven't really put much thought into that though | 19:55 |
| fungi | getting more volunteer reviewers is the first step either way | 19:56 |
| fungi | so probably best not to prematurely optimize for them | 19:56 |
| clarkb | yup I think finding interesting people is the hard part :) | 19:56 |
| fungi | well, interested people anyway | 19:56 |
| clarkb | but if we can find them then I agree that is a low risk project to add them to and get them involved | 19:57 |
| clarkb | oh heh | 19:57 |
| clarkb | well I'm definitely not interesting :) | 19:57 |
| fungi | we have plenty of interesting people (but more never hurts!) | 19:57 |
| fungi | anyway, sounds like there's not a concern with adding some new reviewers to irc-meetings if we get volunteers | 19:58 |
| clarkb | not from me | 19:58 |
| fungi | projects like that and project-config are where i see an opportunity to direct community volunteers to help with some of the review burden | 19:58 |
| clarkb | ++ | 19:59 |
| fungi | if projects feel like their changes to those aren't merging quickly enough, there's a pretty obvious answer ;) | 19:59 |
| ianw | oh i had a quick one to consider after fixing our status tweeting | 19:59 |
| ianw | #link https://review.opendev.org/c/opendev/statusbot/+/837082 | 19:59 |
| ianw | will also log #log messages. just to increase engagement with what we're doing. just a thought | 20:00 |
| clarkb | oh we both +2'd at the same time :) | 20:00 |
| clarkb | I'm going tocall the meeting here. Thank you everyone | 20:00 |
| fungi | thanks clarkb! | 20:01 |
| clarkb | enjoy the long weekend if it is one for you :) | 20:01 |
| clarkb | #endmeeting | 20:01 |
| opendevmeet | Meeting ended Tue Apr 12 20:01:06 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2022/infra.2022-04-12-19.01.html | 20:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2022/infra.2022-04-12-19.01.txt | 20:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2022/infra.2022-04-12-19.01.log.html | 20:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!