19:01:15 <clarkb> #startmeeting infra
19:01:15 <opendevmeet> Meeting started Tue Mar 15 19:01:15 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:15 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:15 <opendevmeet> The meeting name has been set to 'infra'
19:01:24 <clarkb> #link https://lists.opendev.org/pipermail/service-discuss/2022-March/000325.html Our Agenda
19:01:31 <clarkb> #topic Announcements
19:02:01 <clarkb> OpenStack's Yoga release is nearing completion. I think there are ~2 more weeks until that is out the door. Calling it out so that we all know to apply an appropriate amount of care to changes around here
19:02:15 <clarkb> fungi: ^ anything else worth mentioning around the release?
19:04:37 <fungi> not really, no
19:05:03 <clarkb> I guess also worth mentioning that next week is spring break for us, but I plan to be around a bit. Expect to have a meeting etc
19:05:24 <fungi> oh, also lest we forget, our inap environment goes away in a couple weeks
19:05:34 <fungi> in theory anyway
19:05:51 <clarkb> thank you for the reminder
19:06:07 <fungi> and there's ovh maintenance today
19:06:29 <fungi> gra1 starting a few minutes ago and going until 01:00
19:06:39 <fungi> api impact, so we probably can't boot servers there
19:07:06 <clarkb> nodepool should handle that gracefully enough
19:07:07 <fungi> the announcement didn't seem to mention if it would also impact swift, but if we start seeing post_failure from log uploads...
19:08:12 <clarkb> ++
19:09:09 <clarkb> #topic Actions from last meeting
19:09:12 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-03-01-19.01.txt minutes from last meeting
19:09:16 <clarkb> I didn't see any actions recorded
19:09:25 <clarkb> #topic Topics
19:09:30 <clarkb> #topic Improving OpenDev's CD throughput
19:09:46 <clarkb> ianw: are all the jobs producing encrypted log files now?
19:10:13 <ianw> yes, they are
19:11:02 <clarkb> neat I really should make a note to add my key to the list
19:11:23 <ianw> it's now covered in https://docs.opendev.org/opendev/system-config/latest/open-infrastructure.html
19:11:45 <clarkb> Is there anything else to make note of on this topic? I think things kind of stalled out on the spec review (I'm partially to blame there as I have yet to review it)
19:11:56 <ianw> yeah i hope to get back to it
19:12:29 <ianw> it's tangential in a way, but i wanted to restore the dashboard of job tracking we used to have
19:13:05 <ianw> https://review.opendev.org/c/openstack/project-config/+/833213 proposes this, but we've had discussions in #opendev over the use of exported grafana graphs
19:13:22 <ianw> anyway
19:13:24 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/832880
19:13:49 <clarkb> ah websockets of course :) I guess they are useful for responsive javascript
19:13:50 <ianw> is an actual service fix for grafana, after i noticed constant errors in the web console when playing with graphs
19:16:29 <clarkb> sounds like that may be it
19:16:35 <clarkb> #topic Container maintenance
19:16:59 <clarkb> jentoio made a successful update to run the zuul insecure ci registry as a dedicate duser
19:17:33 <clarkb> Testing turned out to be super valuable there. A huge thank you to everyone that has made efforts to improve our functional testing
19:18:00 <clarkb> jentoio: I think now we can just pick off a service or two at a time and continue to try and make progress. Hound/codesearch or one of the irc bots may be a good next option?
19:19:06 <clarkb> Anyway I'd like to continue making progress in this space and will try to help jentoio if jentoio is still interested. Thanks again for the help
19:19:17 <clarkb> #topic Spring cleaning for old reviews
19:19:24 <clarkb> #link https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire.
19:19:44 <clarkb> These repos have largely been retired. The last step is to pull them out of zuul and retire them via gerrit acls
19:19:48 <clarkb> #link https://review.opendev.org/c/openstack/project-config/+/831837 Finalize repo retirements
19:20:19 <clarkb> I've also started looking at old system-config changes and have been abandoning those that appear safe to do so
19:20:39 <clarkb> https://review.opendev.org/q/project:opendev/system-config+status:abandoned and https://review.opendev.org/q/project:opendev/system-config+status:open,225 are helpful links if you want to see what I've been doing
19:21:10 <clarkb> I've been leaving changes behind that we need to take a second look at or that just need a bit more evaluation. I figure doing a quicker bulk pass is more helpful than going one by one in series
19:22:49 <clarkb> There is actually a few things I've seen that probably make sense to forward port too. That might be easier with new chagnes? I dunno
19:23:01 <clarkb> I called out one in #opendev around gerrit open file limits
19:23:10 <clarkb> https://review.opendev.org/c/opendev/system-config/+/326441/1/modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb is another
19:23:35 <clarkb> On the whole though some good progress and hopefully we'll have much easier to process lists of changes in the future
19:24:37 <ianw> was 326441 about building the inventory, that we don't do any more?
19:25:06 <ianw> previously we'd scan everything, but now we have the static inventory files?
19:25:07 <clarkb> ianw: it may be and ya it may not be helpful anymore
19:25:25 <clarkb> but changes like that and evaluating how to forward port if still valuable is what I mean
19:25:34 <ianw> sure :)
19:25:38 <clarkb> however, ya I think you may be right that our static inventory negates the need for that
19:25:46 <clarkb> ianw: feel free to abandon it with a message along those lines :)
19:26:57 <clarkb> #topic Gitea 1.16
19:27:08 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.4
19:27:32 <clarkb> I think we're about as ready as we are going to be for updating gitea. A number of us have looked at the changes previously. Note that this is a new point release since the last time people looked though
19:27:52 <clarkb> fungi: you already acked the change, if ianw is comfortable with it maybe we can land that today
19:29:25 <ianw> i can take a poke at that held node soon
19:29:33 <fungi> yeah, should be all ready
19:29:34 <ianw> i must have looked at .3 before?
19:29:41 <clarkb> ianw: I think .2
19:29:50 <ianw> ok
19:29:56 <clarkb> .3 fixed the diffing issues but I'm not sure many people looked at it
19:30:07 <clarkb> thanks!
19:30:16 <clarkb> #topic Backup Server Pruning
19:30:37 <clarkb> fungi: you mentioned that you were going to take a look at this yesterday. Wanted to call it out in case this needed help or additional attention
19:31:01 <fungi> yeah, hope to get to it today
19:31:23 <fungi> it's pretty easy, but if someone else wants to take a go at it, that's cool too
19:32:28 <ianw> i may have a few hours to just keep an eye on it, will let you know
19:32:35 <clarkb> thanks. I feel like my plate it pretty full with gitea and system-config cleanups and so on.
19:33:42 <clarkb> #topic Rocky Linux
19:33:50 <clarkb> This was removed from the agenda but one small item came up since I did that
19:33:55 <clarkb> #link https://review.opendev.org/c/opendev/base-jobs/+/828436 Add a rocky linux nodeset
19:34:01 <clarkb> basically adding a nodeset for the distro is the last thing
19:35:07 <ianw> lgtm
19:36:09 <clarkb> #topic Open Discussion
19:36:13 <clarkb> That was it for the agenda
19:36:16 <clarkb> is there anything else?
19:36:31 <fungi> i didn't have anything
19:37:23 <ianw> i have a few that could use review
19:37:33 <clarkb> One thing that has started bouncing around in my head is taht it might be a good idea to move all our python3.7 container images to python3.9 and then add a python3.10 set of images
19:37:40 <ianw> #link https://review.opendev.org/q/topic:zuul-registry-focal
19:37:42 <clarkb> but that seems like something to do after the openstack release likely
19:38:03 <ianw> that stack has some future fixes (moving to focal) but also some features (adding mitmproxying and logs to zuul-registry tests)
19:38:22 <clarkb> ah yup I think I've reviewed some of those changes. I'll try to catch up
19:38:35 <ianw> #link https://review.opendev.org/c/zuul/nodepool/+/831895
19:39:00 <ianw> is one from the rocky linux work; an attempt to better ensure glean is working as we hope during testing
19:39:25 <ianw> #link https://review.opendev.org/q/topic:http-options
19:39:50 <ianw> this zuul stack fell out of our prior work last week getting better status checking on haproxy
19:40:23 <ianw> turns out by default it likes to send a OPTIONS / request, which got me poking as to why zuul was behaving differently to gitea
19:40:39 <clarkb> does that add an OPTIONS route handler?
19:42:01 <ianw> it already has a OPTIONS handler for the auth end-points.  that stack does a couple of things, mostly makes the existing functions only respond to GET / requests (405'ing other requests)
19:42:29 <ianw> the final step it does wrap it all up into having every function wrapped by the OPTIONS handler, as suggested in reviews
19:42:57 <clarkb> cool I'll have to take a look
19:43:35 <ianw> (it's not really necessary on all end-points, but also not against any rules, and probably makes sense to be consistent in an API)
19:44:04 <ianw> #link https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/828217
19:44:16 <ianw> is another quickish one to use venv instead of virtualenv in the wheel builds
19:44:41 <ianw> i hope to very soon clean up the old centos8 wheel volumes, and probably put in 9-stream builds for completeness
19:45:41 <ianw> oh, and
19:45:43 <ianw> #link https://review.opendev.org/c/zuul/zuul-jobs/+/829853
19:45:59 <clarkb> I have opened all these links in my browser and will attempt to get to them after some lunch :)
19:46:02 <ianw> is the final one from comments on the log encryption.  i'm open to suggestions on ways to do that better
19:46:29 <ianw> addresses the concerns raised by fungi for importing keys that expire
19:46:50 <ianw> thanks!
19:47:29 <clarkb> and thank you for continuing to push stuff along
19:47:42 <clarkb> last call, otherwise I think we can all go seek out $meal :)
19:47:55 <fungi> thanks clarkb!
19:48:14 <ianw> ++
19:48:43 <clarkb> #endmeeting