19:01:08 <clarkb> #startmeeting infra
19:01:08 <opendevmeet> Meeting started Tue Jan 18 19:01:08 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:08 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <opendevmeet> The meeting name has been set to 'infra'
19:01:23 <clarkb> I expect this meeting to be another quicker one as we're all still sort of rolling into the new year
19:01:27 <ianw> o/
19:01:33 <clarkb> #link https://lists.opendev.org/pipermail/service-discuss/2022-January/000314.html Our Agenda
19:01:39 <clarkb> But we do have an agenda to run trhough
19:01:47 <clarkb> ianw: good morning!
19:01:49 <clarkb> #topic Announcements
19:01:54 <clarkb> Service coordinator nominations run January 25, 2022 - February 8, 2022 (that starts next week)
19:02:17 <clarkb> Another reminder that service coordinator nominations will begin soon
19:02:20 <clarkb> OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/
19:02:40 <clarkb> the OpenInfra Summit CFP is accepting papers and if you'd like to help program the event you can nominate yourself to be on the programming committee
19:02:55 <clarkb> there are a number of tracks including a CI/CD track that we might find interesting
19:04:34 <clarkb> Deadlines for both are fast approaching (I think early february for CFP and end of the week ish for programming committee?0
19:04:54 <clarkb> #topic Actions from last meeting
19:04:59 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2022/infra.2022-01-11-19.01.txt minutes from last meeting
19:05:46 <clarkb> Last meeting was pretty easy going. No actions recorded
19:05:50 <clarkb> #topic Topics
19:05:57 <clarkb> #topic Improving OpenDev's CD throughput
19:06:07 <clarkb> I have on my todo list an item that says I should review this spec but I haven't yet
19:06:14 <clarkb> #link https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets
19:06:47 <clarkb> infra-root ^ would be good to look at that in the near future. I'll try to prioritize it myself once I've flushed a bit more of the backlog (getting really close)
19:07:25 <clarkb> ianw: anything else to add on this topic? I suspect you are in backlog catch up mode too and this may not be near the top
19:07:57 <ianw> yeah nothing more on this one yet
19:08:44 <clarkb> #topic Container maintenance
19:08:49 <clarkb> #link https://etherpad.opendev.org/p/opendev-container-maintenance
19:09:05 <clarkb> Good news: all buster based images should be updated to bullseye now. Thank you to everyone who helped me push that along
19:09:38 <fungi> contrats!
19:09:42 <clarkb> I've got a change that came out of that to stop running a limnoria fork at https://review.opendev.org/c/opendev/system-config/+/821331 which I updated today to consume from a tag instead of tip of master
19:09:45 <fungi> and thanks for all the work on it
19:09:55 <clarkb> I think that might want to be landed on a friday when there are fewer meetings
19:10:31 <ianw> ++ thanks for pushing that through!
19:11:31 <clarkb> jentoio I'll be reaching out sometime soon to start looking at the dedicated user work with our containers as that is the next thing on the container todo list
19:12:25 <jentoio> sounds good, thanks
19:12:46 <fungi> i'd love to find the time to work on deploying the storyboard containers we're already building
19:12:56 <clarkb> There are also a few other large todos on that etherpad including zookeeper upgrades, mariadb upgrades, and intermediate registry pruning if anyone gets bored and wants to take a look :)
19:13:01 <fungi> also we'll have container deployment work involved in the upcoming mailman3 effort
19:13:26 <clarkb> fungi: yup. Re storyboard converting it to a newer mariadb may be a good idea as a first step on the mariadb upgrades planning
19:14:03 <fungi> oh, in the dockerfile?
19:14:04 <clarkb> I think the zk upgrade may be as simple as shifting the version ahead and letting ansible do its thing as newer zk is supposedly compat with older
19:14:08 <clarkb> fungi: in the docker compose file
19:14:20 <fungi> er, that's what i meant, of course ;)
19:14:26 <clarkb> for zk we should probably test that assumption a bit before going ahead with it though
19:15:05 <clarkb> Anyway that was all I had. Happy to help others that dig into this and I plan to pull tasks off the list as I'm able myself
19:15:12 <clarkb> #topic Nodepool image cleanups
19:15:24 <clarkb> Tumbleweed is gone. Haven't heard any complains
19:15:48 <clarkb> CentOS 8 removal has begun. I pushed up a bunch of changes to infra related repos as well as openstack requirements and zuul-jobs to remove the use of centos8
19:16:00 <clarkb> We should no longer be updating the wheel mirror for centos 8 as well
19:16:39 <clarkb> I sent email to service-discuss and openstack-discuss warning people and some specific projects of our plan to remove centos 8
19:17:04 <clarkb> Those emails indicated we would remove the label from nodepool/zuul entirely around the end of the month
19:17:38 <clarkb> Overall it seems people have been receptive and are converting jobs to centos 8 stream or removing centos entirely. Which is basically what we were looking for so yay
19:18:18 <clarkb> However, `ping` is currently broken for non root users on centos 8 stream. Something to be aware of if conversion happen and don't work. That distro needs to revert its most recent iputils package update or update systemd to systemd-239-55 to correct this
19:18:44 <clarkb> A couple of projects like devstack and tripleo have worked around this by adding sysctl to mimic the system-239-55 update
19:19:47 <clarkb> ianw: for fedora 34 frickler indicated that neutron is using that platform for testing. Do you know what is required to move them ahead to fedora 35?
19:20:01 <clarkb> fedora 34 is the other distro in nodepool that would be good to remove in the near future as that one doesn't boot reliably
19:20:37 <fungi> yeah, i think we have it booting successfully in only two providers?
19:20:41 <ianw> i think we need to go through a few bootloader things and cut a dib release
19:21:12 <clarkb> ianw: to remove fedora 34 or to fix it?
19:22:42 <ianw> sorry, to fully fix fedora 35 booting and keep 9-stream happy, etc.
19:23:25 <ianw> just context switching back in, sorry
19:23:30 <clarkb> Gotcha general rhel system improvements then. But no reason to keep fedora 34 around if we can sort out neutron? and even then fedora has a short shelf life so neutron may need to accept it is going away soon either way
19:23:56 <ianw> so f35 wasn't booting, https://review.opendev.org/c/openstack/diskimage-builder/+/818851 fixed that and i think is released now
19:24:44 <ianw> no, it's just merged
19:25:15 <ecsantos[m]> Oh, sorry to butt in, just a quick note about openSUSE. SUSE dropped OpenStack in 2019: https://www.zdnet.com/article/suse-drops-openstacks/
19:25:16 <ecsantos[m]> That's why the mirrors were broken, apparently. I have a change semi-ready to address it
19:25:52 <ianw> it caused a regression in 9-stream "centos" element -- fixed with https://review.opendev.org/c/openstack/diskimage-builder/+/821772
19:26:37 <ianw> there's a couple of other changes in the queue related now
19:27:00 <ianw> so i think we go through those and try to find a coherent point to make a 3.17.0 release.  that should free up the fedora work
19:27:09 <clarkb> ianw: sounds good, thanks
19:27:42 <clarkb> #topic Scheduling Gerrit Project Renames
19:28:06 <clarkb> last week we penciled in January 24 for this as that seemed to fit the openstack release cycle well and shouldn't interfere with zuul much either
19:28:25 <clarkb> fungi: I think you were thinking later in the day UTC time. Like 22:00 UTC?
19:28:33 <fungi> i plan to be on hand all day, so am happy to help as long as it's while i'm still awake
19:29:08 <fungi> very early like in the 00:00-03:00 UTC range would also work for me
19:29:17 <ianw> should we perhaps schedule the 3.4 upgrade at the same time?
19:30:06 <ianw> (i need to finish the donwgrade/revert section, but i think it's good to go)
19:30:07 <clarkb> I'm happy to try the 3.4 upgrade around then too. But is ~6 days enough advancewarning for a gerrit upgrade?
19:30:28 <clarkb> I guess the delta here is much smaller. The biggest gap is probably zuul working with gerrit
19:30:32 <fungi> it seems tight, but i'm okay with the idea
19:30:51 <clarkb> If we do it that way I think we do the renames first
19:30:59 <clarkb> that way if we rever the 3.4 upgrade we'll have completed the renames
19:31:05 <ianw> ++
19:31:14 <fungi> also we should plan enough time in the window to test and roll back
19:31:20 <clarkb> fungi: ++
19:31:26 <clarkb> ianw: when is a good time to start all that for you?
19:31:32 <clarkb> is 22:00 UTC too early?
19:32:48 <ianw> no, that is fine and probably is better for EOD in US time
19:33:21 <ianw> (than earlier)
19:33:42 <clarkb> ok I can put together an email today announcing that we'll plan to rename projects starting at 22:00 UTC January 24 as well as upgrade gerrit to 3.4 with an expected end time around 00:00 UTC (I think 2 hours should be enough, an hour for each action)
19:34:01 <ianw> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.4
19:34:11 <clarkb> infra-root ^ does that proposed plan seem reasonable?
19:34:21 <fungi> wfm
19:34:27 <ianw> sounds excellent, thanks
19:34:42 <clarkb> alright I think we can go to the next topic then
19:34:52 <clarkb> #topic Spring Cleaning for Old Reviews
19:35:18 <clarkb> frickler: put this on the agenda and poinst out that repos like system-config have >300 open reviews most of which are more than a year old and have merge conflicts
19:35:27 <clarkb> should we (auto) abandon these?
19:36:37 <clarkb> My initial thought is that maybe we should all try to spend an hour going through the backlogs and manually prune but that is a lot of effort. Maybe we can do a better breakdown and see how many are puppet era and abandon those at least (since we aren't really puppeting anymore)
19:37:08 <clarkb> things from a more modern ansible+docker era are likely worth double checking for relevance before abandoning
19:38:34 <ianw> i'm not opposed
19:39:29 <ianw> i also see old things as "ideas that may be kind of valid but never quite floated to the top at the time they were proposed, but may one day" and it doesn't bother me so much having them around
19:39:55 <clarkb> off the top of my head anything >3 years seems very safe to abandon. Anything >2 is probably fine too. But I see at least one change in my backlog from september 2020 that is probably worth another look at
19:40:29 <clarkb> ianw: yup that is my hesitation whcih is why I'm ok with clearing stuff that we are reasonably sure is related to the old puppet way of doing things
19:40:38 <clarkb> since they are very unlikely to apply any longer
19:41:43 <clarkb> I guess think on this a bit and we can propose a plan next week? I do think that having cleaner gerrit dashboards would be nice for getting myself organized
19:41:57 <ianw> agree
19:42:27 <clarkb> #topic Open Discussion
19:42:39 <fungi> i guess if i see a notification from gerrit about anything i've proposed getting abandoned and i think it's still worthwhile, i'll revisit and restore it
19:42:54 <clarkb> fungi: that seems reasonable too. We can always restore later
19:43:14 <clarkb> My Gerrit 3.3.9 update change just landed. ianw re your autohold may be worth rerunning anyway to ensure you're on the latest images for the revert testing
19:43:26 <clarkb> ianw: I do wonder if the autohold hit a scale otu scheduler bug. Might need to go look at zuul debug logs
19:43:43 <clarkb> But I bring this up because I'd like to do a much smaller gerrit restart today. Probably after the kids get back from school this afternoon and things are quiet
19:44:06 <fungi> also i'm getting ready to turn the jitsi-meet servers back on once the meeting wraps (new stable images without any log4j in them were finally tagged a few hours ago)
19:44:17 <clarkb> \o/
19:44:53 <clarkb> I'll give this a few more minutes before calling it a meeting.
19:45:13 <jentoio> a comment on opendev docs, specifically opendev/system-confg
19:45:25 <clarkb> jentoio: go for it
19:46:03 <jentoio> as I read them I'm finging lots of outdated info. For example, major systems, logstash
19:46:36 <jentoio> I guess its fine to update them as I find outdated info, is my question
19:46:58 <fungi> for logstash, probably better to not spend time updating those docs
19:47:03 <jentoio> is it worth working on
19:47:03 <clarkb> jentoio: yup pushing changes to update out of date info is greatly appreciated. Even if you just push a note warning that indicates the info is out of date
19:47:15 <fungi> we've got an active effort underway which should result in us removing all the logstash docs anyway
19:47:33 <clarkb> logstash is a special case beacuse we are in the process of removing it entirely. But in general pointing things out with a warning blob or updating out of date info to be correct would be great
19:47:39 <fungi> though sure, a warning banner on the logstash docs can't hurt
19:47:42 <jentoio> lots of lionks with http not http(s)
19:48:22 <fungi> we've added https to a lot of our services/sites recently, so yes any of them that work via https would be great to switch in hyperlinks
19:48:34 <jentoio> okay, i'm hearing that pushing updates to updated docs is fine
19:48:48 <clarkb> ++
19:48:53 <jentoio> except logstash
19:48:55 <fungi> ianw's design for broad automation of lets encrypt has allowed us to be far less choosy about what gets a cert
19:50:55 <clarkb> Sounds like that may be all. Thank you everyone. I'll give you about 10 minutes back for $meal :)
19:50:59 <clarkb> I myself need some lunch
19:51:02 <clarkb> #endmeeting