19:01:08 #startmeeting infra 19:01:08 Meeting started Tue Jan 18 19:01:08 2022 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:08 The meeting name has been set to 'infra' 19:01:23 I expect this meeting to be another quicker one as we're all still sort of rolling into the new year 19:01:27 o/ 19:01:33 #link https://lists.opendev.org/pipermail/service-discuss/2022-January/000314.html Our Agenda 19:01:39 But we do have an agenda to run trhough 19:01:47 ianw: good morning! 19:01:49 #topic Announcements 19:01:54 Service coordinator nominations run January 25, 2022 - February 8, 2022 (that starts next week) 19:02:17 Another reminder that service coordinator nominations will begin soon 19:02:20 OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ 19:02:40 the OpenInfra Summit CFP is accepting papers and if you'd like to help program the event you can nominate yourself to be on the programming committee 19:02:55 there are a number of tracks including a CI/CD track that we might find interesting 19:04:34 Deadlines for both are fast approaching (I think early february for CFP and end of the week ish for programming committee?0 19:04:54 #topic Actions from last meeting 19:04:59 #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-01-11-19.01.txt minutes from last meeting 19:05:46 Last meeting was pretty easy going. No actions recorded 19:05:50 #topic Topics 19:05:57 #topic Improving OpenDev's CD throughput 19:06:07 I have on my todo list an item that says I should review this spec but I haven't yet 19:06:14 #link https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets 19:06:47 infra-root ^ would be good to look at that in the near future. I'll try to prioritize it myself once I've flushed a bit more of the backlog (getting really close) 19:07:25 ianw: anything else to add on this topic? I suspect you are in backlog catch up mode too and this may not be near the top 19:07:57 yeah nothing more on this one yet 19:08:44 #topic Container maintenance 19:08:49 #link https://etherpad.opendev.org/p/opendev-container-maintenance 19:09:05 Good news: all buster based images should be updated to bullseye now. Thank you to everyone who helped me push that along 19:09:38 contrats! 19:09:42 I've got a change that came out of that to stop running a limnoria fork at https://review.opendev.org/c/opendev/system-config/+/821331 which I updated today to consume from a tag instead of tip of master 19:09:45 and thanks for all the work on it 19:09:55 I think that might want to be landed on a friday when there are fewer meetings 19:10:31 ++ thanks for pushing that through! 19:11:31 jentoio I'll be reaching out sometime soon to start looking at the dedicated user work with our containers as that is the next thing on the container todo list 19:12:25 sounds good, thanks 19:12:46 i'd love to find the time to work on deploying the storyboard containers we're already building 19:12:56 There are also a few other large todos on that etherpad including zookeeper upgrades, mariadb upgrades, and intermediate registry pruning if anyone gets bored and wants to take a look :) 19:13:01 also we'll have container deployment work involved in the upcoming mailman3 effort 19:13:26 fungi: yup. Re storyboard converting it to a newer mariadb may be a good idea as a first step on the mariadb upgrades planning 19:14:03 oh, in the dockerfile? 19:14:04 I think the zk upgrade may be as simple as shifting the version ahead and letting ansible do its thing as newer zk is supposedly compat with older 19:14:08 fungi: in the docker compose file 19:14:20 er, that's what i meant, of course ;) 19:14:26 for zk we should probably test that assumption a bit before going ahead with it though 19:15:05 Anyway that was all I had. Happy to help others that dig into this and I plan to pull tasks off the list as I'm able myself 19:15:12 #topic Nodepool image cleanups 19:15:24 Tumbleweed is gone. Haven't heard any complains 19:15:48 CentOS 8 removal has begun. I pushed up a bunch of changes to infra related repos as well as openstack requirements and zuul-jobs to remove the use of centos8 19:16:00 We should no longer be updating the wheel mirror for centos 8 as well 19:16:39 I sent email to service-discuss and openstack-discuss warning people and some specific projects of our plan to remove centos 8 19:17:04 Those emails indicated we would remove the label from nodepool/zuul entirely around the end of the month 19:17:38 Overall it seems people have been receptive and are converting jobs to centos 8 stream or removing centos entirely. Which is basically what we were looking for so yay 19:18:18 However, `ping` is currently broken for non root users on centos 8 stream. Something to be aware of if conversion happen and don't work. That distro needs to revert its most recent iputils package update or update systemd to systemd-239-55 to correct this 19:18:44 A couple of projects like devstack and tripleo have worked around this by adding sysctl to mimic the system-239-55 update 19:19:47 ianw: for fedora 34 frickler indicated that neutron is using that platform for testing. Do you know what is required to move them ahead to fedora 35? 19:20:01 fedora 34 is the other distro in nodepool that would be good to remove in the near future as that one doesn't boot reliably 19:20:37 yeah, i think we have it booting successfully in only two providers? 19:20:41 i think we need to go through a few bootloader things and cut a dib release 19:21:12 ianw: to remove fedora 34 or to fix it? 19:22:42 sorry, to fully fix fedora 35 booting and keep 9-stream happy, etc. 19:23:25 just context switching back in, sorry 19:23:30 Gotcha general rhel system improvements then. But no reason to keep fedora 34 around if we can sort out neutron? and even then fedora has a short shelf life so neutron may need to accept it is going away soon either way 19:23:56 so f35 wasn't booting, https://review.opendev.org/c/openstack/diskimage-builder/+/818851 fixed that and i think is released now 19:24:44 no, it's just merged 19:25:15 Oh, sorry to butt in, just a quick note about openSUSE. SUSE dropped OpenStack in 2019: https://www.zdnet.com/article/suse-drops-openstacks/ 19:25:16 That's why the mirrors were broken, apparently. I have a change semi-ready to address it 19:25:52 it caused a regression in 9-stream "centos" element -- fixed with https://review.opendev.org/c/openstack/diskimage-builder/+/821772 19:26:37 there's a couple of other changes in the queue related now 19:27:00 so i think we go through those and try to find a coherent point to make a 3.17.0 release. that should free up the fedora work 19:27:09 ianw: sounds good, thanks 19:27:42 #topic Scheduling Gerrit Project Renames 19:28:06 last week we penciled in January 24 for this as that seemed to fit the openstack release cycle well and shouldn't interfere with zuul much either 19:28:25 fungi: I think you were thinking later in the day UTC time. Like 22:00 UTC? 19:28:33 i plan to be on hand all day, so am happy to help as long as it's while i'm still awake 19:29:08 very early like in the 00:00-03:00 UTC range would also work for me 19:29:17 should we perhaps schedule the 3.4 upgrade at the same time? 19:30:06 (i need to finish the donwgrade/revert section, but i think it's good to go) 19:30:07 I'm happy to try the 3.4 upgrade around then too. But is ~6 days enough advancewarning for a gerrit upgrade? 19:30:28 I guess the delta here is much smaller. The biggest gap is probably zuul working with gerrit 19:30:32 it seems tight, but i'm okay with the idea 19:30:51 If we do it that way I think we do the renames first 19:30:59 that way if we rever the 3.4 upgrade we'll have completed the renames 19:31:05 ++ 19:31:14 also we should plan enough time in the window to test and roll back 19:31:20 fungi: ++ 19:31:26 ianw: when is a good time to start all that for you? 19:31:32 is 22:00 UTC too early? 19:32:48 no, that is fine and probably is better for EOD in US time 19:33:21 (than earlier) 19:33:42 ok I can put together an email today announcing that we'll plan to rename projects starting at 22:00 UTC January 24 as well as upgrade gerrit to 3.4 with an expected end time around 00:00 UTC (I think 2 hours should be enough, an hour for each action) 19:34:01 #link https://etherpad.opendev.org/p/gerrit-upgrade-3.4 19:34:11 infra-root ^ does that proposed plan seem reasonable? 19:34:21 wfm 19:34:27 sounds excellent, thanks 19:34:42 alright I think we can go to the next topic then 19:34:52 #topic Spring Cleaning for Old Reviews 19:35:18 frickler: put this on the agenda and poinst out that repos like system-config have >300 open reviews most of which are more than a year old and have merge conflicts 19:35:27 should we (auto) abandon these? 19:36:37 My initial thought is that maybe we should all try to spend an hour going through the backlogs and manually prune but that is a lot of effort. Maybe we can do a better breakdown and see how many are puppet era and abandon those at least (since we aren't really puppeting anymore) 19:37:08 things from a more modern ansible+docker era are likely worth double checking for relevance before abandoning 19:38:34 i'm not opposed 19:39:29 i also see old things as "ideas that may be kind of valid but never quite floated to the top at the time they were proposed, but may one day" and it doesn't bother me so much having them around 19:39:55 off the top of my head anything >3 years seems very safe to abandon. Anything >2 is probably fine too. But I see at least one change in my backlog from september 2020 that is probably worth another look at 19:40:29 ianw: yup that is my hesitation whcih is why I'm ok with clearing stuff that we are reasonably sure is related to the old puppet way of doing things 19:40:38 since they are very unlikely to apply any longer 19:41:43 I guess think on this a bit and we can propose a plan next week? I do think that having cleaner gerrit dashboards would be nice for getting myself organized 19:41:57 agree 19:42:27 #topic Open Discussion 19:42:39 i guess if i see a notification from gerrit about anything i've proposed getting abandoned and i think it's still worthwhile, i'll revisit and restore it 19:42:54 fungi: that seems reasonable too. We can always restore later 19:43:14 My Gerrit 3.3.9 update change just landed. ianw re your autohold may be worth rerunning anyway to ensure you're on the latest images for the revert testing 19:43:26 ianw: I do wonder if the autohold hit a scale otu scheduler bug. Might need to go look at zuul debug logs 19:43:43 But I bring this up because I'd like to do a much smaller gerrit restart today. Probably after the kids get back from school this afternoon and things are quiet 19:44:06 also i'm getting ready to turn the jitsi-meet servers back on once the meeting wraps (new stable images without any log4j in them were finally tagged a few hours ago) 19:44:17 \o/ 19:44:53 I'll give this a few more minutes before calling it a meeting. 19:45:13 a comment on opendev docs, specifically opendev/system-confg 19:45:25 jentoio: go for it 19:46:03 as I read them I'm finging lots of outdated info. For example, major systems, logstash 19:46:36 I guess its fine to update them as I find outdated info, is my question 19:46:58 for logstash, probably better to not spend time updating those docs 19:47:03 is it worth working on 19:47:03 jentoio: yup pushing changes to update out of date info is greatly appreciated. Even if you just push a note warning that indicates the info is out of date 19:47:15 we've got an active effort underway which should result in us removing all the logstash docs anyway 19:47:33 logstash is a special case beacuse we are in the process of removing it entirely. But in general pointing things out with a warning blob or updating out of date info to be correct would be great 19:47:39 though sure, a warning banner on the logstash docs can't hurt 19:47:42 lots of lionks with http not http(s) 19:48:22 we've added https to a lot of our services/sites recently, so yes any of them that work via https would be great to switch in hyperlinks 19:48:34 okay, i'm hearing that pushing updates to updated docs is fine 19:48:48 ++ 19:48:53 except logstash 19:48:55 ianw's design for broad automation of lets encrypt has allowed us to be far less choosy about what gets a cert 19:50:55 Sounds like that may be all. Thank you everyone. I'll give you about 10 minutes back for $meal :) 19:50:59 I myself need some lunch 19:51:02 #endmeeting