| clarkb | meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jan 31 19:01:06 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| ianw | o/ | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/R7TJT7TMQICXIIWNGACKHQN4HB3ORQWF/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | The service coordinator nomination period opened up today | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/32BIEDDOWDUITX26NSNUSUB6GJYFHWWP/ | 19:01 |
| clarkb | It runs for ~2 weeks. I will send a followup email to ^ to remind everyone that today is the day | 19:01 |
| clarkb | That was all I had for announcements. Let's dive in | 19:03 |
| clarkb | #topic Bastion Host Updates | 19:03 |
| clarkb | I've been terrible and not reviewed the backup script change :( | 19:03 |
| clarkb | I believe that is still outstanding | 19:03 |
| clarkb | In better news the old bridge did get shutdown I believe. Thank you ianw for contining to move this along | 19:04 |
| ianw | yep, maybe give it another week and we can "server delete" it | 19:04 |
| clarkb | ++ | 19:04 |
| clarkb | anything else bridge related? as far as I can tell we've moved onto the new bridge at this point and are happy with it | 19:05 |
| clarkb | #topic Mailman 3 | 19:07 |
| ianw | yep, i should get back to some of the parallel running work, remerge and re-evaluate | 19:07 |
| ianw | (sorry, that's it) | 19:07 |
| clarkb | sounds good | 19:07 |
| clarkb | fungi: I've seen you pushing on mailman 3 items and seem to have made progress? Want to catch us up? | 19:07 |
| fungi | i managed to work out the configuration fiddling dance necessary to bootstrap the server into a steady configuration state that supports correct per-domain filtering of lists | 19:08 |
| fungi | however, there are outstanding steps we need to perform to complete addition of the sites and domain mappings | 19:09 |
| clarkb | This is the stuff we were just discussing in #opendev that may involve db migrations? | 19:09 |
| fungi | the greatest hurdle is that each list domain needs to be associated with a distinct django "site" and creating new django sites with automation is... nontrivial | 19:09 |
| clarkb | just talking out loud here: we add domains infrequently enough that maybe "root approves new domain then uses django web ui to create the site" is fine? | 19:10 |
| fungi | apparently it's assumed you'll "just" visit the django admin webui and click the "add site" button which runs database migrations in the background to create and populate new tables and create new configs | 19:10 |
| clarkb | but maybe its too soon to give up | 19:10 |
| fungi | in order to do that from the django-admin cli you need to make new migrations for each site and run them | 19:11 |
| fungi | i'm sure for someone well-versed in django-based applications this is easy, but i'm struggling to wrap my head around it still | 19:11 |
| fungi | part of the problem with doing that part manually is order of operations. if we want a new mailman domain to be automatically associated with a site, we need to create the site before creating the domain before adding mailing lists... | 19:12 |
| clarkb | right considering that the value we get from mm3 is pretty high I'm ok punting on automating this specific bit if it gets us moving. But I'll let you decide when you've spent enough time trying to automate it first | 19:13 |
| clarkb | oh :( | 19:13 |
| fungi | we have a lot of automation that would need to be blocked to prevent it running before we do the manual step | 19:13 |
| fungi | or else we do additional manual steps to move the domain to the new site after creating it | 19:13 |
| clarkb | ya considering that I guess we should try to figure out the migrations process if possible | 19:14 |
| clarkb | otherwise we'd have to break up new domains into two changes and manually create sites between? | 19:14 |
| clarkb | doable, but annoying/complicated too | 19:14 |
| fungi | it's probably quite similar to some of the commands ansible is already running in playbooks/roles/mailman3/tasks/main.yaml | 19:15 |
| fungi | django does have a cli to "makemigrations" and so on | 19:15 |
| fungi | so it seems like a lot of this is automatable if i can wrap my head around the terminology | 19:15 |
| fungi | #link https://docs.djangoproject.com/en/4.1/topics/migrations/ | 19:16 |
| ianw | is https://docs.djangoproject.com/en/4.1/ref/contrib/sites/#enabling-the-sites-framework the same thing? | 19:17 |
| clarkb | having the held nodes definitely helped me when I did the initial bootstrapping | 19:17 |
| clarkb | ianw: yes | 19:17 |
| clarkb | fungi: the other thing I wanted to mention is that the services did get restarted which should've fixed our site owner email addr. But we still have a broken root alias for receiging mail at at that address? I guess fixing one thing at a time | 19:19 |
| fungi | oh, right, i think i tracked that down to a missing bit in the host vars | 19:20 |
| fungi | we failed to include the alias stuff that we normally put on our other servers | 19:21 |
| clarkb | that would do it | 19:21 |
| clarkb | fungi: is there a change for that yet? | 19:21 |
| fungi | no, it sounded like frickler was going to push a fix, i thought, but i can do it i just need to remind myself what specifically needs adding | 19:22 |
| clarkb | mostly wanted to ensure I wasn't holding things up by missing a change to review | 19:22 |
| clarkb | thanks! anything else on this topic? | 19:22 |
| fungi | mmm, actually it looks like the necessary bit is already there in lists01.opendev.org.yaml | 19:24 |
| fungi | nope, nothing else for now | 19:24 |
| clarkb | #topic Git updates on docker images | 19:24 |
| clarkb | Upstream debian has patched git which means we don't need our manually patched packages anymore. In fact due to version differences apt complains about this in some situations too. | 19:25 |
| clarkb | #link https://review.opendev.org/q/topic:flip-to-distro-git+status:open | 19:25 |
| clarkb | I pushed changes to our images to unwind that and flip back to distro packges. | 19:25 |
| clarkb | At this point gitea and gerrit are outstanding. ianw took care of zuul and nodepool too so by the weekend hopefully we'll have put all of this in the rear view mirror | 19:26 |
| clarkb | Mostly a heads up that those changes need reviews. The gitea update should auto deploy and the gerritone will need a manual gerrit restart | 19:26 |
| ianw | thanks for the work making the packages etc! | 19:26 |
| clarkb | I did check if the base python images have updated git yet and they have not (did this a few hours ago) | 19:26 |
| clarkb | but I think once the base python images do update we should update our opendev base python images | 19:27 |
| clarkb | I'll try to check on that periodically and push a base image update change once that is in place | 19:27 |
| clarkb | #topic Gerrit updates | 19:27 |
| clarkb | I have a handful of changes out there for various Gerrit updates | 19:28 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/872238 Fixup Gerrit Submit Requirements usage | 19:28 |
| clarkb | This one is straightfowrard update to our gerrit testing and use of submit requirements as part of the 3.6 -> 3.7 transition prep | 19:28 |
| clarkb | My emails to the repo discuss list didn't get any responders and I ended up digging into the source code and think I decided that pushing MaxWithBlock explicitly is not allowed, but if you don't set a function value then it is still used via the default :/ | 19:29 |
| clarkb | Anyway that addresses this by explicitly setting Noop to ensure we're exercising submit requirements (and the screenshots seem to show this is working too) | 19:29 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/870114 Add Gerrit 3.6 -> 3.7 Upgrade test job | 19:30 |
| clarkb | this adds the upgrade job. ianw I did respond to your comments and basically punted on trying to fully automate the upgrade process. I think doing that isn't a bad idea but maybe out of scope for this change (basically good improvement for later) | 19:30 |
| ianw | ++ i'll go back over those two | 19:31 |
| clarkb | #link https://review.opendev.org/c/openstack/project-config/+/867931 Cleaning up deprecated copy conditions in project ACLs | 19:31 |
| clarkb | this is ianw's which i asked we not merge until after we announced it | 19:31 |
| clarkb | just beacuse potential is there for behavior changes. But I think it is ready to go when we are ready to announce it and do it | 19:32 |
| clarkb | That leaves the two "scary" changes | 19:32 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/870874 Convert Gerrit to run on our base python image | 19:32 |
| clarkb | This swaps out the docker openjdk image for debian bullseye openjdk packages on top of our python base image | 19:32 |
| clarkb | also I've just realized this change needs a new ps to explicitly install git to ensure that it is up to date | 19:33 |
| clarkb | In theory this is largely a noop for us other than that the python version goes from debian package to version specific compile for the image and the openjdk is switched to the distro openjdk | 19:34 |
| clarkb | I'll push a new ps to this after the meeting to address the git install | 19:34 |
| clarkb | Careful review is always appreciated (our CI helps a lot too!) | 19:34 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/870877 Run Gerrit under Java 17 | 19:34 |
| clarkb | and this is the last one. I think this one deserves the most careful consideration since java version changes can be a big deal. In particular I've had to implement a workaround for a bug despite gerrit saying java 17 is fully compatible (it isn't) | 19:35 |
| clarkb | we should think about whether or not we can live with that workaround (both because it is clunky as we have to execute java a specific way anytime we invoke gerrit and because there may be security/runtime implications?) | 19:35 |
| clarkb | This email hasn't gotten any movement upstream either unfortunately. | 19:36 |
| clarkb | reviews much appreciated. | 19:36 |
| clarkb | #topic Linaro Cloud Migration | 19:36 |
| clarkb | We are running jobs on the new linaro cloud (thank you ianw and kevinz!) | 19:36 |
| clarkb | we should be off of the old cloud with our nodepool configs and we told Ed at equinix the underlying systems could go away at the end of this month (today) | 19:37 |
| clarkb | one thing I noticed is that we're still checking the ssl cert for the old cloud's api endpoint. ianw should we update that to be the new cloud's endpoint? | 19:37 |
| ianw | ummm, yes. i have a change out that might fix that | 19:38 |
| ianw | i think it's in | 19:38 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/871672 | 19:38 |
| ianw | which removes all of the cloud bits | 19:38 |
| clarkb | excellent | 19:40 |
| clarkb | The other bit you wanted to discuss was how involved we wanted to be in hte management of this cloud | 19:40 |
| clarkb | I selfishly think the less the better :) but I have no objections to us having that access if it is useful (and it already has been) | 19:41 |
| ianw | yeah i'm not sure we want to run base.yaml against it ... but we could | 19:42 |
| ianw | if we're going to have access, we should probably at least have bridge deploying our keys? | 19:42 |
| fungi | we did that for some cloud previously (not infracloud, more recent). which one was it? inmotion? older? | 19:44 |
| clarkb | inmotion I did manually | 19:45 |
| clarkb | but a good idea for both I guess | 19:46 |
| ianw | just not sure if we add to inventory if the base setup will take it over in weird ways, it might do something that kolla doesn't like | 19:46 |
| ianw | but i can check it out if we're ok with basically including it | 19:47 |
| clarkb | ianw: maybe we can add a new group that is basically a subset of base | 19:47 |
| clarkb | I would worry about the firewall rule changes in particular. THose are likely to break openstack | 19:47 |
| clarkb | so ya we should limit it to users if we can | 19:48 |
| ianw | yeah, ok i'll take a look | 19:48 |
| clarkb | sound sgood thanks | 19:48 |
| clarkb | #topic SqlAlchemy 2.0 | 19:49 |
| clarkb | last week zzzeek released sqlalachemy 2.0. Unfortunately this version of sqlalchemy is not backward compatible with 1.x | 19:50 |
| clarkb | the good news is that there are good docs about it and zzzeek has been responsive to questions I've asked about this. | 19:50 |
| clarkb | I know that Zuul and lodgeit are affected. Zuul I've been working on fixing. Lodgeit I don't know that anything has been done yet but we should probably push a sqlalchemy pin change for lodgeit | 19:50 |
| clarkb | I wanted to bring this up so people are aware of the issues and also to double check there aren't any other services relying on sqlalchemy | 19:51 |
| clarkb | I guess thats it for sqlalchemy using projects? | 19:53 |
| clarkb | #topic Server Upgrades | 19:53 |
| ianw | oh wow, yes lodgeit would be a pin | 19:53 |
| clarkb | Maybe no surprise but I haven't made progress here. Too many things | 19:53 |
| clarkb | #topic Quo vadis Storyboard | 19:53 |
| clarkb | same story here. | 19:54 |
| clarkb | Sorry for rushing along but we are almost at time and I wanted to open things up to ensure we aren't overlooking anything | 19:54 |
| clarkb | #topic Open Discussion | 19:54 |
| ianw | lodgeit is so far behind on everything, this is another thing :/ | 19:54 |
| ianw | i guess there's an element of "if it ain't broke" ... it takes some input and saves it, so maybe it can just live in it's container forever more | 19:55 |
| fungi | i fixed the missing root alias for lists01, no gerrit change needed. apparently it's set as a private hostvar on bridge, and we missed copying it from the old listservers. that's done now, so the next periodic run should update /etc/aliases accordingly | 19:55 |
| clarkb | fungi: aha | 19:55 |
| clarkb | fungi: what is the var name? I'm curious to grep for it and see where it is used | 19:55 |
| fungi | listadmins | 19:55 |
| fungi | the listservs define their own custom aliases lists and have a generator in there to create a root: alias, but was defaulting to empty on the new server and so the /etc/aliases template was skipping it since there was no value for the "root" key | 19:57 |
| clarkb | gotcha | 19:57 |
| fungi | also i just pushed a new revision of the donor logos addition addressing your layout comment | 19:57 |
| fungi | #link https://review.opendev.org/869091 Feature our cloud donors on opendev.org | 19:57 |
| clarkb | cool I'll try to review the linaro cloud cleanup change and the donors change after lunch. | 19:58 |
| ianw | fungi: speaking of both, i don't think i saw linaro on that list? | 19:58 |
| fungi | we should add them. they're not currently on the sets of logos on openstack.org or openinfra.dev either | 19:59 |
| ianw | a linaro logo anyway. although i'm not sure between equinix/linaro who is actually in charge :) | 19:59 |
| fungi | i copied the current sets but left iweb out | 19:59 |
| fungi | (it needs to be cleaned up in the old locations too) | 19:59 |
| clarkb | I think we've not done that because the total numebr of nodes is small. ut consdering the effort to make this happen I think that is fine | 20:00 |
| clarkb | and we are at time. | 20:00 |
| clarkb | THank you everyone | 20:00 |
| fungi | thanks clarkb! | 20:00 |
| clarkb | We'll be back here same time and place next week | 20:00 |
| clarkb | #endmeeting | 20:00 |
| opendevmeet | Meeting ended Tue Jan 31 20:00:37 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-31-19.01.html | 20:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-31-19.01.txt | 20:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-01-31-19.01.log.html | 20:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!