19:01:15 #startmeeting infra 19:01:15 Meeting started Tue Mar 15 19:01:15 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:15 The meeting name has been set to 'infra' 19:01:24 #link https://lists.opendev.org/pipermail/service-discuss/2022-March/000325.html Our Agenda 19:01:31 #topic Announcements 19:02:01 OpenStack's Yoga release is nearing completion. I think there are ~2 more weeks until that is out the door. Calling it out so that we all know to apply an appropriate amount of care to changes around here 19:02:15 fungi: ^ anything else worth mentioning around the release? 19:04:37 not really, no 19:05:03 I guess also worth mentioning that next week is spring break for us, but I plan to be around a bit. Expect to have a meeting etc 19:05:24 oh, also lest we forget, our inap environment goes away in a couple weeks 19:05:34 in theory anyway 19:05:51 thank you for the reminder 19:06:07 and there's ovh maintenance today 19:06:29 gra1 starting a few minutes ago and going until 01:00 19:06:39 api impact, so we probably can't boot servers there 19:07:06 nodepool should handle that gracefully enough 19:07:07 the announcement didn't seem to mention if it would also impact swift, but if we start seeing post_failure from log uploads... 19:08:12 ++ 19:09:09 #topic Actions from last meeting 19:09:12 #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-03-01-19.01.txt minutes from last meeting 19:09:16 I didn't see any actions recorded 19:09:25 #topic Topics 19:09:30 #topic Improving OpenDev's CD throughput 19:09:46 ianw: are all the jobs producing encrypted log files now? 19:10:13 yes, they are 19:11:02 neat I really should make a note to add my key to the list 19:11:23 it's now covered in https://docs.opendev.org/opendev/system-config/latest/open-infrastructure.html 19:11:45 Is there anything else to make note of on this topic? I think things kind of stalled out on the spec review (I'm partially to blame there as I have yet to review it) 19:11:56 yeah i hope to get back to it 19:12:29 it's tangential in a way, but i wanted to restore the dashboard of job tracking we used to have 19:13:05 https://review.opendev.org/c/openstack/project-config/+/833213 proposes this, but we've had discussions in #opendev over the use of exported grafana graphs 19:13:22 anyway 19:13:24 #link https://review.opendev.org/c/opendev/system-config/+/832880 19:13:49 ah websockets of course :) I guess they are useful for responsive javascript 19:13:50 is an actual service fix for grafana, after i noticed constant errors in the web console when playing with graphs 19:16:29 sounds like that may be it 19:16:35 #topic Container maintenance 19:16:59 jentoio made a successful update to run the zuul insecure ci registry as a dedicate duser 19:17:33 Testing turned out to be super valuable there. A huge thank you to everyone that has made efforts to improve our functional testing 19:18:00 jentoio: I think now we can just pick off a service or two at a time and continue to try and make progress. Hound/codesearch or one of the irc bots may be a good next option? 19:19:06 Anyway I'd like to continue making progress in this space and will try to help jentoio if jentoio is still interested. Thanks again for the help 19:19:17 #topic Spring cleaning for old reviews 19:19:24 #link https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. 19:19:44 These repos have largely been retired. The last step is to pull them out of zuul and retire them via gerrit acls 19:19:48 #link https://review.opendev.org/c/openstack/project-config/+/831837 Finalize repo retirements 19:20:19 I've also started looking at old system-config changes and have been abandoning those that appear safe to do so 19:20:39 https://review.opendev.org/q/project:opendev/system-config+status:abandoned and https://review.opendev.org/q/project:opendev/system-config+status:open,225 are helpful links if you want to see what I've been doing 19:21:10 I've been leaving changes behind that we need to take a second look at or that just need a bit more evaluation. I figure doing a quicker bulk pass is more helpful than going one by one in series 19:22:49 There is actually a few things I've seen that probably make sense to forward port too. That might be easier with new chagnes? I dunno 19:23:01 I called out one in #opendev around gerrit open file limits 19:23:10 https://review.opendev.org/c/opendev/system-config/+/326441/1/modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb is another 19:23:35 On the whole though some good progress and hopefully we'll have much easier to process lists of changes in the future 19:24:37 was 326441 about building the inventory, that we don't do any more? 19:25:06 previously we'd scan everything, but now we have the static inventory files? 19:25:07 ianw: it may be and ya it may not be helpful anymore 19:25:25 but changes like that and evaluating how to forward port if still valuable is what I mean 19:25:34 sure :) 19:25:38 however, ya I think you may be right that our static inventory negates the need for that 19:25:46 ianw: feel free to abandon it with a message along those lines :) 19:26:57 #topic Gitea 1.16 19:27:08 #link https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.4 19:27:32 I think we're about as ready as we are going to be for updating gitea. A number of us have looked at the changes previously. Note that this is a new point release since the last time people looked though 19:27:52 fungi: you already acked the change, if ianw is comfortable with it maybe we can land that today 19:29:25 i can take a poke at that held node soon 19:29:33 yeah, should be all ready 19:29:34 i must have looked at .3 before? 19:29:41 ianw: I think .2 19:29:50 ok 19:29:56 .3 fixed the diffing issues but I'm not sure many people looked at it 19:30:07 thanks! 19:30:16 #topic Backup Server Pruning 19:30:37 fungi: you mentioned that you were going to take a look at this yesterday. Wanted to call it out in case this needed help or additional attention 19:31:01 yeah, hope to get to it today 19:31:23 it's pretty easy, but if someone else wants to take a go at it, that's cool too 19:32:28 i may have a few hours to just keep an eye on it, will let you know 19:32:35 thanks. I feel like my plate it pretty full with gitea and system-config cleanups and so on. 19:33:42 #topic Rocky Linux 19:33:50 This was removed from the agenda but one small item came up since I did that 19:33:55 #link https://review.opendev.org/c/opendev/base-jobs/+/828436 Add a rocky linux nodeset 19:34:01 basically adding a nodeset for the distro is the last thing 19:35:07 lgtm 19:36:09 #topic Open Discussion 19:36:13 That was it for the agenda 19:36:16 is there anything else? 19:36:31 i didn't have anything 19:37:23 i have a few that could use review 19:37:33 One thing that has started bouncing around in my head is taht it might be a good idea to move all our python3.7 container images to python3.9 and then add a python3.10 set of images 19:37:40 #link https://review.opendev.org/q/topic:zuul-registry-focal 19:37:42 but that seems like something to do after the openstack release likely 19:38:03 that stack has some future fixes (moving to focal) but also some features (adding mitmproxying and logs to zuul-registry tests) 19:38:22 ah yup I think I've reviewed some of those changes. I'll try to catch up 19:38:35 #link https://review.opendev.org/c/zuul/nodepool/+/831895 19:39:00 is one from the rocky linux work; an attempt to better ensure glean is working as we hope during testing 19:39:25 #link https://review.opendev.org/q/topic:http-options 19:39:50 this zuul stack fell out of our prior work last week getting better status checking on haproxy 19:40:23 turns out by default it likes to send a OPTIONS / request, which got me poking as to why zuul was behaving differently to gitea 19:40:39 does that add an OPTIONS route handler? 19:42:01 it already has a OPTIONS handler for the auth end-points. that stack does a couple of things, mostly makes the existing functions only respond to GET / requests (405'ing other requests) 19:42:29 the final step it does wrap it all up into having every function wrapped by the OPTIONS handler, as suggested in reviews 19:42:57 cool I'll have to take a look 19:43:35 (it's not really necessary on all end-points, but also not against any rules, and probably makes sense to be consistent in an API) 19:44:04 #link https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/828217 19:44:16 is another quickish one to use venv instead of virtualenv in the wheel builds 19:44:41 i hope to very soon clean up the old centos8 wheel volumes, and probably put in 9-stream builds for completeness 19:45:41 oh, and 19:45:43 #link https://review.opendev.org/c/zuul/zuul-jobs/+/829853 19:45:59 I have opened all these links in my browser and will attempt to get to them after some lunch :) 19:46:02 is the final one from comments on the log encryption. i'm open to suggestions on ways to do that better 19:46:29 addresses the concerns raised by fungi for importing keys that expire 19:46:50 thanks! 19:47:29 and thank you for continuing to push stuff along 19:47:42 last call, otherwise I think we can all go seek out $meal :) 19:47:55 thanks clarkb! 19:48:14 ++ 19:48:43 #endmeeting