19:01:09 <clarkb> #startmeeting infra
19:01:10 <opendevmeet> Meeting started Tue Jan 10 19:01:09 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:10 <opendevmeet> The meeting name has been set to 'infra'
19:01:12 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/5WW6CBUKQUWVNW7C5RRLXW7PWUXDPY75/ Our Agenda
19:01:49 <clarkb> #topic Announcements
19:02:03 <clarkb> The Open Infra Foundation individual board member election is happening right now
19:02:20 <clarkb> if you are a member of the foundation you should've received a ballot via email.
19:03:03 <clarkb> Please vote if you are able to.
19:03:17 <clarkb> #topic Bastion Host Updates
19:03:43 <clarkb> ianw: I know prior to holidays you were querying people to see if anything else needed migrating from the old bastion before it was shutdown.
19:04:10 <clarkb> Looks like the old bastion is still up, but maybe we can shut it down soon? I didn't see anyone indicate anything needed to be moved at last poll
19:04:21 <ianw> yeah i think i got a response from everyone active
19:04:55 <ianw> i can shut it down, and we can consider removal after a little
19:05:00 <clarkb> sounds good
19:05:00 <ianw> more time
19:05:07 <clarkb> Anything else bastion related?
19:05:30 <ianw> i don't think so ... the backup bits i think might need a rebase now
19:06:07 <clarkb> #topic Mailman 3
19:06:34 <clarkb> I think we may still need to restart services to pick up the site owner update. ALso frickler discovered that local root alias is not working anyway.
19:06:54 <clarkb> do we want to use a different site owner value or fix the root@lists01 alias in exim? I assume we prefer fixing the alias?
19:07:01 <fungi> and i have some more related changes up for review
19:07:27 <fungi> including an upgrade to the latest mm3 versions from last week
19:07:31 <clarkb> these changes are related to fixing the vhosting of lists themselves?
19:07:33 <fungi> yes
19:08:14 <clarkb> https://review.opendev.org/c/opendev/system-config/+/867986/ and children? Looks like they are all workinprogress right now
19:08:35 <clarkb> https://review.opendev.org/q/topic:mailman3+status:open is a better link. Finds a couple extra changes
19:09:01 <clarkb> fungi: what do you think the plan there should be?
19:09:17 <fungi> just a sec, need to page a lot of this back into my head ;)
19:09:55 <fungi> i think i still need to review the failures on 867987
19:10:40 <fungi> i have a feeling it's probably just going to need test adjustments, but don't think i've confirmed that yet
19:11:04 <fungi> the upgrade is up for discussion. the change passes at least
19:11:25 <clarkb> fungi: was that generated to be in sync with how the upstream docker images are handling the version upgrade too?
19:11:26 <fungi> and the upgrade notes for the new versions don't indicate any extra steps needed
19:11:39 <clarkb> my main concern is with the dep pinning as that seems to be a major headache for everyone doing mm3
19:11:55 <clarkb> but once you have a good install it sounds like they handle upgrades pretty transparently
19:11:56 <fungi> yes, much of that was ported from the docker image repo
19:12:14 <clarkb> awesome. In that case I think we can land that either before or after fixing the other issues, Probably your preference really
19:12:50 <fungi> i think i'd like to dig into fixing the domain separation first
19:13:18 <fungi> after we get that ironed out though, and upgrade mailman to the latest releases, i should be ready to start planning for the next site migrations
19:13:31 <clarkb> works for me. Maybe ping us when you're in a state where extra eyeballs could be helpful? Whether just code review or helping with debugging on held nodes
19:13:38 <fungi> yep
19:13:58 <fungi> migrating lists.katacontainers.io should probably be prioritized since it frees up an entire server
19:14:25 <clarkb> makes sense
19:14:28 <fungi> but also i've been fighting oom kills of random processes on the old lists.openstack.org server so the more we move off of it the better
19:14:48 <fungi> (i just moments ago killed and restarted all relevant processes on that server yet again)
19:15:04 <clarkb> Anything else on this topic?
19:15:14 <corvus> fungi: i'm assuming you shut down/disabled the migrated processes already?
19:15:21 <fungi> yes
19:15:30 <corvus> figured :)
19:15:48 <fungi> yeah, if i hadn't, that would be an easy way to relieve some pressure on it
19:15:52 <fungi> too bad
19:17:00 <clarkb> #topic Quo vadis Storyboard
19:17:36 <clarkb> I swear I sent that followup email I said I would send but I'm not finding it now...
19:19:49 <clarkb> In any case there hasn't been much movement on this topic. As mentioned before I think we need to make a proposal (probably towards turning it off in a gracefaul manner if possible) based on the feedback we've gotten
19:20:11 <clarkb> I feel like I've got enough distractions this week that I won't get to that this week whcih means we can let it simmer for a bit longer :/
19:20:24 <clarkb> Still happy to receive any and all additional feedback others may have though
19:21:10 <clarkb> #topic Gerrit 3.6
19:21:17 <clarkb> The upgrade happened and we haven't reverted
19:21:37 <clarkb> ianw sent email to the repo discuss list to bring up the label listing behavior that is new in 3.6 (I have't seent an responses yet)
19:22:07 <clarkb> I think we are at a point where we should remove 3.5 image builds, add 3.7 image builds, and update our upgrade testing to perform a 3.6 to 3.7 upgrade
19:22:20 <ianw> ++ i don't see us going back to 3.5
19:22:34 <fungi> sgtm
19:22:40 <clarkb> ianw: I'm happy to help with the wrangling of those changes if you'd like.
19:23:08 <ianw> sure :)
19:23:35 <ianw> i also started on
19:23:46 <ianw> #link https://review.opendev.org/c/openstack/project-config/+/867931
19:24:06 <ianw> to update copyConditions in our configs
19:24:25 <clarkb> this is one of potentially several similar types of changes we'll need to make prior to upgrading to 3.7
19:24:47 <ianw> we have some in our All-Projects which I think will need manual intervention, but i'd like to put some of it into the gate testing first
19:25:05 <clarkb> ++ there is less blast radius outside of all-projects
19:25:11 <ianw> some work started on that @
19:25:13 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/868054
19:26:28 <clarkb> anything else gerrit upgrade related before we jump to the next thing?
19:26:48 <ianw> nope, more to discuss in the future as we work on submit requirements etc :)
19:27:06 <clarkb> #topic Nox
19:27:18 <clarkb> This is the evolution of the Tox v4 topic we had previously
19:27:38 <clarkb> The way the tox v4 situation has panned out has led to us needing to deal with new issues with every tox v4 release
19:27:57 <clarkb> Eventually some of us got tired of that and started looking at alternatives: in particular nox.
19:28:25 <clarkb> Nox is a tox alternative that uses a simple python file to specify its targets (it calls them sessions) and relies on standard tools and processes for installing things
19:28:49 <clarkb> This to me is the main feature with nox that has me wanting to switch. Tox has reimplemented libraries for certain peps and it led to weird behavior
19:29:01 <clarkb> Sticking to standard tools ensures we don't need todebug oddness like that
19:29:16 <clarkb> Long story short Zuul is already largely moved to nox and I think OpenDev should do the same
19:29:22 <clarkb> #link https://review.opendev.org/c/opendev/bindep/+/868004 An example conversion
19:29:33 <clarkb> I've got this change up for bindep that serves as an example too
19:29:57 <clarkb> All of the zuul-jobs tooling for artifact retrieval and siblings support should be there now too
19:30:28 <fungi> the only counterargument against nox i've heard so far is that some folks prefer the declarative nature of tox.ini files over turing-complete python scripts for configuration, but tox.ini is pretty far down the road to being a programming language of its own at this point too
19:30:32 <clarkb> And as a user I've been relly happy using nox. I don't think I would suggest the move if I wasn't. The syntax for command is a little different but its not so different and it is quick to pick up
19:30:56 <clarkb> fungi: ya and I would argue that this is tox's greatest flw right now because they keep changing the semantic meaning of that declarativeconfig
19:31:00 <clarkb> making it effectively useless
19:32:16 <fungi> i'm good with switching bindep over to see how it goes (and zuul/nodepool are already basically there)
19:32:30 <clarkb> Anyway, I intend on pushing changes to move active opendev repos to nox. But I don't want to get too far down that path until we've got general consensus this isn't a problem
19:33:03 <clarkb> So please do take a look at the bindep change, fetch that change locally and run nox yourselves and see what you think. THen let me know :)
19:33:42 <fungi> convincing large and complex projects like openstack to switch away from tox is probably not a windmill i feel like tilting at (especially since they're stuck using tox on older branches for years to come), but i think for opendev's projects it makes sense and shouldn't be too big of a lift
19:34:15 <corvus> if it were me, i'd pin those older branches to tox v3 then nox on master
19:34:43 <fungi> yeah, they're pinning the older branches anyway, as it turns out
19:34:53 <fungi> but stuff like release tooling will have to support both
19:35:21 <clarkb> depending on where I get with zuul and opedev I might take a look at the release tooling, but I agree its a much stronger preexisting force you run up against there
19:35:33 <corvus> i'm very happy with nox and i like the idea of us using it for opendev projects
19:36:18 <clarkb> woot my first couple of bits of feedback :)
19:36:24 <clarkb> I think we can move on to the next thing
19:36:31 <clarkb> #topic Service Coordinator Election
19:36:41 <clarkb> It has been about 5 months since we last elected our service coordinator
19:37:12 <clarkb> 6 months after our previous nomination period is January 31, 2023 - February 14, 2023
19:37:35 <clarkb> I'd like to propose we open that two week period for nominations for the next service coordinator and take it from there
19:38:18 <clarkb> If there are no objections to this timeframe I'll send email to service-discuss making it official soon
19:38:50 <clarkb> as I've said before I'm happy for someone else to do the work too :)
19:39:38 <clarkb> #topic Linaro Cloud Move and Cleanup
19:39:54 <clarkb> ianw: want to fill us in on the recent updates for the linaro cloud move
19:40:50 <ianw> yep sure, we have credentials
19:41:18 <ianw> there are changes to enable the cloud for CI @
19:41:20 <ianw> #link https://review.opendev.org/q/topic:linaro-2022
19:42:08 <ianw> one issue is that it doesn't have much disk.  so much so that our 800gb volume for the builder /opt would leave nothing else
19:42:27 <ianw> so i asked osuosl if they could give us compute quota to run a nodepool builder there, which they kindly provided
19:43:16 <ianw> i have the host up, storage attached, etc, and some changes out there to add it to system-config & dns
19:43:44 <ianw> so i think i'd like to add that, and get it working and building things
19:43:53 <ianw> then disable nb03
19:44:04 <ianw> then cut testing over to new cloud
19:44:06 <fungi> sounds great
19:44:11 <ianw> then disable old cloud
19:44:16 <clarkb> oh I just approved the first two chagnes that shift over to the new cloud
19:44:21 <clarkb> should I WIP them?
19:44:38 <clarkb> I guess its only the second that is a problem. I'll WIP it
19:44:56 <ianw> it's ok, just we will want to monitor to make sure it is actually working
19:45:12 <clarkb> heh ok switched back to +A
19:45:48 <ianw> it can all really happen in any order, depending on our attention span to monitor the different bits :)
19:45:50 <clarkb> thank you for picking this up. I think it will make ed happy and in theory these newer test nodes should be very quick
19:46:46 <clarkb> Anything else related to the cloud moves?
19:46:50 <ianw> yep, good to try and work with the providers to keep everyone happy :)
19:47:33 <ianw> no more on that, will keep working at it
19:47:53 <clarkb> #topic Upgrading Bionic Servers
19:47:58 <clarkb> #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
19:48:09 <clarkb> This is going to take on a bit more urgency due to bionic EOL in april
19:48:28 <clarkb> I'm personally hoping to be able to really start picking things off of that list starting next week.
19:48:54 <clarkb> If anyone else ends up helping out please stick a note in the etherpad so that we don't step on each other. Also, help is appreciated :)
19:50:06 <clarkb> #topic Open Discussion
19:50:17 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/866781 Retire Mordred as infra root
19:50:45 <ianw> one thing frickler pointed out was the increasing storage numbers
19:50:46 <clarkb> Can other infra-root review this change? mordred pushed it himself so no concerns there. Unfortunately I think the time has come for this little bit of cleanup
19:50:55 <ianw> #link https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6M&to=now
19:51:28 <frickler> yes, not critical yet, but worth watching
19:51:28 <clarkb> looks like ubuntu and ubuntu.ports have grown a bit
19:51:30 <clarkb> I wonder why
19:51:40 <clarkb> since they should be pretty fixed after the jammy release last april
19:51:49 <ianw> which, apropos the prior topic, one win we could have there is purging everything xenial related
19:52:11 <clarkb> the open euler mirror also doubled in size
19:52:23 <clarkb> so ya we probably needto make another pruning pass
19:52:33 <corvus> i'd like to land the nodepool openstack statemachine change soon.  there is a non-zero chance of real-world performance problems.
19:52:40 <corvus> once we get all the nodepool tox/nox/k8s changes merged, how about i tag a 8.0.1 release, then if there is a problem we can revert opendev to it quickly?
19:52:50 <clarkb> corvus: ++ to tagging first
19:53:01 <ianw> clarkb: yeah, i thought we got a new openeuler release but purged the old one
19:53:55 <clarkb> debian security almost doubled too
19:54:10 <clarkb> definitely a good idea to dig into that. Thank you for calling it out ianw and frickler
19:54:39 <corvus> cool, i'll make sure the tag happens and will keep folks updated.  let me know if there's any other concerns, preparation, or safeguards we'd like for opendev.  i figured i would manually upgrade launchers slowly to keep an eye on things.
19:55:05 <clarkb> corvus: you'll need to put them in the emergency file to do them manually as our hourly jobs are really good at updating them otherwise
19:55:19 <clarkb> (remember this is on bridge01.opendev.org now)
19:55:40 <corvus> clarkb: ack.  and i've already retrained to use bridge.opendev.org  :)
19:56:00 <corvus> and thanks, i forgot that could happen hourly and not nightly
19:56:24 <corvus> so: tag, emergency file, manual upgrade.
19:56:24 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/848796/3
19:56:48 <ianw> ^ is the openeuler mirror drop -- i knew i'd seen something about it.  we should probably do that at least
19:57:00 <clarkb> ++ I'll review it now
19:58:11 <corvus> (incidentally, tagging 8.0.1 will be a nice double check of any release job nox updates)
19:58:26 <corvus> if we have to do an 8.0.2 it's no big deal
19:58:53 <fungi> numbers are free
19:58:59 <fungi> we won't run out
19:59:09 <clarkb> And we are just about at time
19:59:39 <clarkb> thank you everyone. Apologies for not communicating better last week. I ened up sick and wasn't in a place to computer. I'm still sort of on the tail end of that too but much more with it now :)
19:59:53 <clarkb> We'll be back next week. Happy new year!
19:59:57 <fungi> we didn't have anything worth covering last week anyway
20:00:00 <fungi> thanks clarkb!
20:00:01 <clarkb> #endmeeting