19:01:06 <clarkb> #startmeeting infra
19:01:06 <opendevmeet> Meeting started Tue Jan 31 19:01:06 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:06 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:06 <opendevmeet> The meeting name has been set to 'infra'
19:01:07 <ianw> o/
19:01:19 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/R7TJT7TMQICXIIWNGACKHQN4HB3ORQWF/ Our Agenda
19:01:23 <clarkb> #topic Announcements
19:01:35 <clarkb> The service coordinator nomination period opened up today
19:01:41 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/32BIEDDOWDUITX26NSNUSUB6GJYFHWWP/
19:01:59 <clarkb> It runs for ~2 weeks. I will send a followup email to ^ to remind everyone that today is the day
19:03:11 <clarkb> That was all I had for announcements. Let's dive in
19:03:15 <clarkb> #topic Bastion Host Updates
19:03:30 <clarkb> I've been terrible and not reviewed the backup script change :(
19:03:37 <clarkb> I believe that is still outstanding
19:04:00 <clarkb> In better news the old bridge did get shutdown I believe. Thank you ianw for contining to move this along
19:04:34 <ianw> yep, maybe give it another week and we can "server delete" it
19:04:43 <clarkb> ++
19:05:07 <clarkb> anything else bridge related? as far as I can tell we've moved onto the new bridge at this point and are happy with it
19:07:16 <clarkb> #topic Mailman 3
19:07:20 <ianw> yep, i should get back to some of the parallel running work, remerge and re-evaluate
19:07:25 <ianw> (sorry, that's it)
19:07:32 <clarkb> sounds good
19:07:48 <clarkb> fungi: I've seen you pushing on mailman 3 items and seem to have made progress? Want to catch us up?
19:08:36 <fungi> i managed to work out the configuration fiddling dance necessary to bootstrap the server into a steady configuration state that supports correct per-domain filtering of lists
19:09:11 <fungi> however, there are outstanding steps we need to perform to complete addition of the sites and domain mappings
19:09:48 <clarkb> This is the stuff we were just discussing in #opendev that may involve db migrations?
19:09:49 <fungi> the greatest hurdle is that each list domain needs to be associated with a distinct django "site" and creating new django sites with automation is... nontrivial
19:10:28 <clarkb> just talking out loud here: we add domains infrequently enough that maybe "root approves new domain then uses django web ui to create the site" is fine?
19:10:37 <fungi> apparently it's assumed you'll "just" visit the django admin webui and click the "add site" button which runs database migrations in the background to create and populate new tables and create new configs
19:10:39 <clarkb> but maybe its too soon to give up
19:11:21 <fungi> in order to do that from the django-admin cli you need to make new migrations for each site and run them
19:11:50 <fungi> i'm sure for someone well-versed in django-based applications this is easy, but i'm struggling to wrap my head around it still
19:12:48 <fungi> part of the problem with doing that part manually is order of operations. if we want a new mailman domain to be automatically associated with a site, we need to create the site before creating the domain before adding mailing lists...
19:13:00 <clarkb> right considering that the value we get from mm3 is pretty high I'm ok punting on automating this specific bit if it gets us moving. But I'll let you decide when you've spent enough time trying to automate it first
19:13:23 <clarkb> oh :(
19:13:29 <fungi> we have a lot of automation that would need to be blocked to prevent it running before we do the manual step
19:13:58 <fungi> or else we do additional manual steps to move the domain to the new site after creating it
19:14:40 <clarkb> ya considering that I guess we should try to figure out the migrations process if possible
19:14:51 <clarkb> otherwise we'd have to break up new domains into two changes and manually create sites between?
19:14:58 <clarkb> doable, but annoying/complicated too
19:15:16 <fungi> it's probably quite similar to some of the commands ansible is already running in playbooks/roles/mailman3/tasks/main.yaml
19:15:38 <fungi> django does have a cli to "makemigrations" and so on
19:15:58 <fungi> so it seems like a lot of this is automatable if i can wrap my head around the terminology
19:16:58 <fungi> #link https://docs.djangoproject.com/en/4.1/topics/migrations/
19:17:44 <ianw> is https://docs.djangoproject.com/en/4.1/ref/contrib/sites/#enabling-the-sites-framework the same thing?
19:17:45 <clarkb> having the held nodes definitely helped me when I did the initial bootstrapping
19:17:48 <clarkb> ianw: yes
19:19:25 <clarkb> fungi: the other thing I wanted to mention is that the services did get restarted which should've fixed our site owner email addr. But we still have a broken root alias for receiging mail at at that address? I guess fixing one thing at a time
19:20:45 <fungi> oh, right, i think i tracked that down to a missing bit in the host vars
19:21:07 <fungi> we failed to include the alias stuff that we normally put on our other servers
19:21:38 <clarkb> that would do it
19:21:44 <clarkb> fungi: is there a change for that yet?
19:22:20 <fungi> no, it sounded like frickler was going to push a fix, i thought, but i can do it i just need to remind myself what specifically needs adding
19:22:43 <clarkb> mostly wanted to ensure I wasn't holding things up by missing a change to review
19:22:48 <clarkb> thanks! anything else on this topic?
19:24:28 <fungi> mmm, actually it looks like the necessary bit is already there in lists01.opendev.org.yaml
19:24:40 <fungi> nope, nothing else for now
19:24:43 <clarkb> #topic Git updates on docker images
19:25:13 <clarkb> Upstream debian has patched git which means we don't need our manually patched packages anymore. In fact due to version differences apt complains about this in some situations too.
19:25:16 <clarkb> #link https://review.opendev.org/q/topic:flip-to-distro-git+status:open
19:25:36 <clarkb> I pushed changes to our images to unwind that and flip back to distro packges.
19:26:15 <clarkb> At this point gitea and gerrit are outstanding. ianw took care of zuul and nodepool too so by the weekend hopefully we'll have put all of this in the rear view mirror
19:26:33 <clarkb> Mostly a heads up that those changes need reviews. The gitea update should auto deploy and the gerritone will need a manual gerrit restart
19:26:47 <ianw> thanks for the work making the packages etc!
19:26:51 <clarkb> I did check if the base python images have updated git yet and they have not (did this a few hours ago)
19:27:07 <clarkb> but I think once the base python images do update we should update our opendev base python images
19:27:28 <clarkb> I'll try to check on that periodically and push a base image update change once that is in place
19:27:54 <clarkb> #topic Gerrit updates
19:28:05 <clarkb> I have a handful of changes out there for various Gerrit updates
19:28:10 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/872238 Fixup Gerrit Submit Requirements usage
19:28:26 <clarkb> This one is straightfowrard update to our gerrit testing and use of submit requirements as part of the 3.6 -> 3.7 transition prep
19:29:07 <clarkb> My emails to the repo discuss list didn't get any responders and I ended up digging into the source code and think I decided that pushing MaxWithBlock explicitly is not allowed, but if you don't set a function value then it is still used via the default :/
19:29:58 <clarkb> Anyway that addresses this by explicitly setting Noop to ensure we're exercising submit requirements (and the screenshots seem to show this is working too)
19:30:05 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/870114 Add Gerrit 3.6 -> 3.7 Upgrade test job
19:30:36 <clarkb> this adds the upgrade job. ianw I did respond to your comments and basically punted on trying to fully automate the upgrade process. I think doing that isn't a bad idea but maybe out of scope for this change (basically good improvement for later)
19:31:35 <ianw> ++ i'll go back over those two
19:31:40 <clarkb> #link https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs
19:31:48 <clarkb> this is ianw's which i asked we not merge until after we announced it
19:32:02 <clarkb> just beacuse potential is there for behavior changes. But I think it is ready to go when we are ready to announce it and do it
19:32:15 <clarkb> That leaves the two "scary" changes
19:32:21 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/870874 Convert Gerrit to run on our base python image
19:32:44 <clarkb> This swaps out the docker openjdk image for debian bullseye openjdk packages on top of our python base image
19:33:28 <clarkb> also I've just realized this change needs a new ps to explicitly install git to ensure that it is up to date
19:34:01 <clarkb> In theory this is largely a noop for us other than that the python version goes from debian package to version specific compile for the image and the openjdk is switched to the distro openjdk
19:34:23 <clarkb> I'll push a new ps to this after the meeting to address the git install
19:34:42 <clarkb> Careful review is always appreciated (our CI helps a lot too!)
19:34:53 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17
19:35:23 <clarkb> and this is the last one. I think this one deserves the most careful consideration since java version changes can be a big deal. In particular I've had to implement a workaround for a bug despite gerrit saying java 17 is fully compatible (it isn't)
19:35:52 <clarkb> we should think about whether or not we can live with that workaround (both because it is clunky as we have to execute java a specific way anytime we invoke gerrit and because there may be security/runtime implications?)
19:36:03 <clarkb> This email hasn't gotten any movement upstream either unfortunately.
19:36:28 <clarkb> reviews much appreciated.
19:36:38 <clarkb> #topic Linaro Cloud Migration
19:36:52 <clarkb> We are running jobs on the new linaro cloud (thank you ianw and kevinz!)
19:37:16 <clarkb> we should be off of the old cloud with our nodepool configs and we told Ed at equinix the underlying systems could go away at the end of this month (today)
19:37:36 <clarkb> one thing I noticed is that we're still checking the ssl cert for the old cloud's api endpoint. ianw  should we update that to be the new cloud's endpoint?
19:38:12 <ianw> ummm, yes.  i have a change out that might fix that
19:38:31 <ianw> i think it's in
19:38:33 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/871672
19:38:39 <ianw> which removes all of the cloud bits
19:40:07 <clarkb> excellent
19:40:58 <clarkb> The other bit you wanted to discuss was how involved we wanted to be in hte management of this cloud
19:41:19 <clarkb> I selfishly think the less the better :) but I have no objections to us having that access if it is useful (and it already has been)
19:42:07 <ianw> yeah i'm not sure we want to run base.yaml against it ... but we could
19:42:49 <ianw> if we're going to have access, we should probably at least have bridge deploying our keys?
19:44:26 <fungi> we did that for some cloud previously (not infracloud, more recent). which one was it? inmotion? older?
19:45:44 <clarkb> inmotion I did manually
19:46:08 <clarkb> but a good idea for both I guess
19:46:41 <ianw> just not sure if we add to inventory if the base setup will take it over in weird ways, it might do something that kolla doesn't like
19:47:03 <ianw> but i can check it out if we're ok with basically including it
19:47:30 <clarkb> ianw: maybe we can add a new group that is basically a subset of base
19:47:48 <clarkb> I would worry about the firewall rule changes in particular. THose are likely to break openstack
19:48:00 <clarkb> so ya we should limit it to users if we can
19:48:09 <ianw> yeah, ok i'll take a look
19:48:37 <clarkb> sound sgood thanks
19:49:39 <clarkb> #topic SqlAlchemy 2.0
19:50:01 <clarkb> last week zzzeek released sqlalachemy 2.0. Unfortunately this version of sqlalchemy is not backward compatible with 1.x
19:50:14 <clarkb> the good news is that there are good docs about it and zzzeek has been responsive to questions I've asked about this.
19:50:42 <clarkb> I know that Zuul and lodgeit are affected. Zuul I've been working on fixing. Lodgeit I don't know that anything has been done yet but we should probably push a sqlalchemy pin change for lodgeit
19:51:16 <clarkb> I wanted to bring this up so people are aware of the issues and also to double check there aren't any other services relying on sqlalchemy
19:53:26 <clarkb> I guess thats it for sqlalchemy using projects?
19:53:33 <clarkb> #topic Server Upgrades
19:53:37 <ianw> oh wow, yes lodgeit would be a pin
19:53:47 <clarkb> Maybe no surprise but I haven't made progress here. Too many things
19:53:59 <clarkb> #topic Quo vadis Storyboard
19:54:01 <clarkb> same story here.
19:54:18 <clarkb> Sorry for rushing along but we are almost at time and I wanted to open things up to ensure we aren't overlooking anything
19:54:23 <clarkb> #topic Open Discussion
19:54:49 <ianw> lodgeit is so far behind on everything, this is another thing :/
19:55:16 <ianw> i guess there's an element of "if it ain't broke" ... it takes some input and saves it, so maybe it can just live in it's container forever more
19:55:18 <fungi> i fixed the missing root alias for lists01, no gerrit change needed. apparently it's set as a private hostvar on bridge, and we missed copying it from the old listservers. that's done now, so the next periodic run should update /etc/aliases accordingly
19:55:34 <clarkb> fungi: aha
19:55:50 <clarkb> fungi: what is the var name? I'm curious to grep for it and see where it is used
19:55:57 <fungi> listadmins
19:57:12 <fungi> the listservs define their own custom aliases lists and have a generator in there to create a root: alias, but was defaulting to empty on the new server and so the /etc/aliases template was skipping it since there was no value for the "root" key
19:57:34 <clarkb> gotcha
19:57:45 <fungi> also i just pushed a new revision of the donor logos addition addressing your layout comment
19:57:59 <fungi> #link https://review.opendev.org/869091 Feature our cloud donors on opendev.org
19:58:08 <clarkb> cool I'll try to review the linaro cloud cleanup change and the donors change after lunch.
19:58:57 <ianw> fungi: speaking of both, i don't think i saw linaro on that list?
19:59:29 <fungi> we should add them. they're not currently on the sets of logos on openstack.org or openinfra.dev either
19:59:34 <ianw> a linaro logo anyway.  although i'm not sure between equinix/linaro who is actually in charge :)
19:59:45 <fungi> i copied the current sets but left iweb out
19:59:54 <fungi> (it needs to be cleaned up in the old locations too)
20:00:15 <clarkb> I think we've not done that because the total numebr of nodes is small. ut consdering the effort to make this happen I think that is fine
20:00:23 <clarkb> and we are at time.
20:00:26 <clarkb> THank you everyone
20:00:30 <fungi> thanks clarkb!
20:00:33 <clarkb> We'll be back here same time and place next week
20:00:37 <clarkb> #endmeeting