19:01:22 <clarkb> #startmeeting infra
19:01:22 <opendevmeet> Meeting started Tue Dec 13 19:01:22 2022 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:22 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:22 <opendevmeet> The meeting name has been set to 'infra'
19:01:44 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/6DVJVHEPMASFL4PQWIQZFUT3H6T5BXP5/ Our Agenda
19:01:56 <clarkb> #topic Announcements
19:02:36 <clarkb> I didn't have anything to announce other than what ended up in our agenda
19:02:40 <clarkb> anything else to make note of?
19:03:06 <clarkb> this meeting does conflict with a world cup semifinal game
19:03:34 <fungi> will you be able to announce the winner at the end of the meeting?
19:03:54 <clarkb> no the game is at least 90 minutes long and we only have 60 minutes here
19:04:05 <clarkb> ok lets dive right in then
19:04:11 <clarkb> #topic Bastion Host Updates
19:04:39 <clarkb> #link https://review.opendev.org/q/topic:prod-bastion-group this is the lower priority stack for parallel runs. Ideally landed whe nwe can observe carefully
19:04:54 <clarkb> #link https://review.opendev.org/q/topic:bridge-backups tooling to backup content in encrypted tarball
19:05:02 <clarkb> #link https://review.opendev.org/c/opendev/zone-opendev.org/+/867540 bridge CNAME
19:05:17 <clarkb> and this last change came out of a reminder to corvus today not to run the zuul restart playbook from old bridge
19:05:38 <clarkb> I think we are really close to being able to shutdown old bridge but suspected that the backup tooling might want to be used on old bridge as a final wrapup?
19:06:07 <clarkb> I'll try to find time to revie wthose but anytime gpg and encryption is involved I like to take my time and do my best to actually understand it
19:06:41 <clarkb> ianw any other thoughts on what our next steps here should be?
19:07:01 <ianw> sure -- yes i can probably shutdown the old bridge at this point, it's just been waiting on finalising the other changes
19:07:15 <ianw> if infra-roots would like to confirm that they have everything they want from their ~ we can shut it down
19:07:51 <clarkb> I think I did look at that and then left notes on the etherpad about things like the dns stuff and as far as I know all of that has been addressed. But I'll double check
19:08:00 <clarkb> and I guess we can shutdown before deleting too
19:08:18 <corvus> i don't need anything from ~corvus; "delete at will" from me
19:08:23 <ianw> yep no need to delete straight away, but also probably good to not have it lying around once we've moved on
19:08:55 <clarkb> ++
19:09:21 * clarkb makes a note to take a look (probably after the afternoon school run)
19:09:27 <clarkb> anything else?
19:09:53 <fungi> i've cleaned out my homedir on old bridge
19:10:31 <ianw> no that's it, thanks
19:10:52 <clarkb> #topic Mailman 3
19:11:11 <clarkb> I think all of the oustanding change srelated to the mm3 migration have landed at this point?
19:11:33 <fungi> yes
19:11:33 <clarkb> I was curious if we know if the site_owner update has been applied. I believe that would require a container restart and I'm not sure if that happened
19:11:44 <fungi> we haven't restarted the container yet, i don't thinkl
19:11:53 <clarkb> fungi: ^ you might know or be able to check? I think you look in /etc/mailman/mailman.conf in the core container
19:12:17 <fungi> also i noticed that the top-left corner of https://lists.zuul-ci.org/archives/ says lists.opendev.org, not sure yet where to configure that
19:12:19 <clarkb> ok I think that is worth following up on and restarting if it hasn't happened yet just to ensure we don't get bitten by that later if something is wrong
19:12:32 <clarkb> fungi: the thing I noticed about that is the link itself has the correct target
19:12:43 <fungi> right, it's just the name apparently
19:12:47 <clarkb> fungi: for this reason I suspect this is a bug in mailman itself where they apply the value to the target but not the rendered name
19:12:59 <clarkb> might just need to stick the same string in both locations of the template or something
19:13:51 <fungi> apparently also the page <title> says "Available lists - lists.opendev.org"
19:14:34 <fungi> but as far as that hyperlink goes, it's relative which i guess is why it doesn't go to the wrong site
19:14:38 <fungi> looks like this:
19:14:40 <fungi> <a class="navbar-brand" href="/archives/">lists.opendev.org</a>
19:14:42 <clarkb> ah
19:15:07 <clarkb> probably need to dig up the source for that and take it from there. It might be missing config or it may be buggy vhost handling
19:15:10 <fungi> so i expect there's a site-specific name we have to give hyperkitty somewhere
19:15:23 <fungi> i didn't notice postorius doing it
19:15:49 <clarkb> I will say as a user I find this much easier to use
19:15:59 <clarkb> on the whole I think it is a big improvement even with the small issues like that
19:16:06 <corvus> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/UAHC7L2M3KCKQXRQY3QCZZEQFGK4L4PK/
19:16:52 <fungi> oh, though i see that the page <title> for e.g. https://lists.zuul-ci.org/mailman3/lists/ says "List Index - lists.opendev.org"
19:17:02 <fungi> so i guess postorius needs it somehow too
19:17:12 <clarkb> sound slik ewe need to look at the django site admin
19:17:20 <fungi> seems likely
19:17:25 <clarkb> (that should work, we interacted with it on the test node)
19:17:38 <corvus> postorius also uses site_name for the title  https://gitlab.com/mailman/postorius/blob/master/src/postorius/templates/postorius/base.html#L10
19:17:50 <corvus> so at least, that's a thread to pull on :)
19:17:55 <clarkb> ++ good find
19:18:19 <clarkb> fungi: there should be a django admin user set as part of the ansible vars and you can use that to login via the admin url iirc
19:19:11 <fungi> yeah, wondering if this is something we should be setting via an api on creation or just accept that we'll have to manually edit it for each site
19:19:25 <clarkb> unfortunately the docs aren't so great for details like this
19:19:39 <clarkb> but we can usually work backward fro mthe expected process and figure it out
19:20:31 <clarkb> anything else?
19:20:39 <corvus> (or is it something we can set with proxy headers?)
19:22:03 <clarkb> We can dig into this more after the meeting. There are a few other topics I want to cover so we'll continue and can swing back to this later if necessary
19:22:13 <clarkb> #topic Quo vadis Storyboard
19:22:26 <clarkb> #link https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html
19:22:29 <clarkb> no new movement on this
19:22:43 <corvus> status quo vadis
19:23:20 <clarkb> I've got a number of writing exercises on my todo list and am thinking maybe I/we should just start drafting a plan in etherpad since feedback on the list isn't happening?
19:23:34 <clarkb> (that might actually mean drafting a couple of possible proposals and then choosing one)
19:24:36 <clarkb> there are no good options so we should maybe write them down in enough dtdail to make the least painful decision
19:25:53 <clarkb> no one is screaming, I'll take that as the next step here and pin th etodo to my list
19:26:57 <clarkb> #topic Gerrit 3.6
19:27:12 <clarkb> ianw upgraded Gerrit from 3.5.4 to 3.6.3 yesterday
19:27:43 <ianw> well i pushed the buttons but everyone, mostly zuul, helped get it there :)
19:27:44 <clarkb> seems to have gone well. The only issues I've seen so far are mostly around the UI changing
19:28:14 <clarkb> The main thing tripping people up seems to be how Gerrit renders votes
19:28:35 <clarkb> in particular it summarizes votes and doesn't put names next to them in the requirements section (it does put them in the reviewers section so you have to look elsewhere)
19:28:54 <clarkb> it also seems to only show required labels for merging in the change listing pages
19:29:13 <clarkb> this means review-priority and similar type sof labels that openstack use don't show up in the list summaries
19:29:30 <corvus> because they're not submit-requirements, yeah?
19:29:41 <clarkb> corvus: yes they are purely informational/trigger requirements
19:29:51 <clarkb> that seems to be the criteria for including them in the UI listings
19:30:22 <clarkb> it also looks like change owners might be able to delete positive votes on their changes. This doesn't seem harmful but may create confusion at some point
19:30:40 <corvus> is something like a submit requirement that accepts any or no value a possible workaround?
19:30:51 <ianw> the summaries is maybe worth raising?  although the answer might be "make a dashboard, it works better anyway"?
19:31:09 <clarkb> corvus: yes that was one suggestion from fungi. I think we have options
19:31:20 <clarkb> ianw: the problem with dashborad is they don't show up thre either so it can be confusing
19:31:31 <clarkb> so you need a dashboard with lists for each possible vote value
19:31:52 <clarkb> I don't know how to make a submit requirement that accepts any value but if we can express that that may be the best option
19:31:57 <ianw> yeah, i meant a separate section like "changes with backport-candidate" or whatever.  but yeah, it's not mixed in
19:32:28 <corvus> looks like a non-submit-requirement label is called a "trigger vote"?
19:32:36 <clarkb> corvus: yes these are trigger votes currently
19:32:42 <ianw> i wonder with that approach though, will we ever see anything on the overview page that isn't a grey, crossed circle?
19:33:23 <ianw> because our submit requirements are really not the same as the idea of "things that need to be done before you can push the merge button", if that makes sense
19:33:47 <ianw> because we never push the merge button, zuul does.  so we're "ready to submit" when we have a +2 vote and +1 verified
19:34:04 <ianw> the +w is our "merge button"
19:34:18 <clarkb> ya  ithink a followup thing might be to have a conversation upstrema about making this directly configurable somehow? like an annotation in the label definition?
19:34:39 <clarkb> then we can more directly express how our labels are used without assumptions based on upstrea mgerrit workflows
19:35:41 <ianw> does anyone happen to have a screenshot of the old layout, perchance?
19:35:49 <corvus> if the suggestion is that we would like to avoid having Verified+2 be a submit requirement, i don't agree with that
19:35:50 <clarkb> I don't, but our CI will take them iirc
19:36:02 <clarkb> corvus: no i think it is about how we render the requirements
19:36:18 <clarkb> you would still need a +2 but we can treat a +1 as "ready for zuul"
19:36:23 <corvus> okay.  i may not be understanding where ianw was going with the merge button thing then.
19:36:27 <fungi> but we might like being able to avoid having "review-priority" as a submit requirement
19:37:09 <clarkb> corvus: the way they render submit requirements today is built around trying to show you what oyu need to click the submit button. But in our system we would ideally render it to show what is necessary for zuul to gate it
19:37:10 <ianw> coruvs: i'm just saying that i think the idea of the new column is to show you "things that are ready to merge".  but it will never really show anything but a grey circle for us
19:37:39 <clarkb> ianw: and that specifically workflow
19:37:47 <corvus> fungi: yes.  my only suggestion there was maybe a submit requirement can have a "satisfied by no vote" to work around the fact that it's not displayed there.  that's all.  no idea if that's possible, just thought it worth looking into.
19:37:49 <clarkb> because its usually set to 0 until set to +1 to trigger gating
19:37:58 <corvus> ianw: ah you're talking about the status column
19:38:10 <corvus> ianw: that is interesting and unfortunate
19:39:49 <ianw> perhaps to move it on, i can come up with a draft email, or maybe two really -- one about the status column -- one about the other flags -- and we can revise it between us and send to gerrit for feedback?
19:39:58 <clarkb> ++
19:40:08 <fungi> sounds great
19:40:42 <ianw> ok, will do something up in an etherpad and let people know
19:40:48 <corvus> (we could, conceivably, lift the verified+2 requirement and rely entirely on zuul, but i think it's better to express the requirements in gerrit itself)
19:40:57 <corvus> ++
19:41:14 <clarkb> thanks. And thank you to everyon ewho made the upgrade happen and happen so smoothly. Now we need to look at 3.7 which might be more complicated :?
19:41:24 <clarkb> and ya lets move on we have 19 minutes left and a couple more things I'd like to cover
19:41:30 <clarkb> #topic Tox v4
19:42:00 <clarkb> Last week tox released 4.0.0 and that very quickly broke all the things. This version is a rewrite and isn't backward compatible
19:42:17 <clarkb> to stop the bleeding we pinned the tox install to <4 in zuul-job' ensure-tox
19:42:24 <clarkb> but the plan is to remove that cap next week
19:42:55 <clarkb> What this means is we should be prepared to update our tox.ini files to be compatible with v4 and v3.
19:42:57 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/867269 system-config tox v4 compat
19:43:09 <clarkb> A change for openstack/project-config has already landed
19:43:38 <clarkb> Most of the issues seem to be related to whitelist_externals becoming allowlist_externals and unlisted commands becoming errors instead of warnings
19:43:51 <clarkb> neutron is running into a new issue where the project isn't installed to the venv for some reason that will need to be sorted out
19:44:10 <clarkb> so far though with zuul-jobs, project-config, and system-config it hasn't been too bad
19:44:12 <fungi> seems to be ignoring the usedevelop option
19:46:03 <clarkb> so ya reviews and changes welcome to help get ahead of that. But I'm sure we'll find problems once we uncap too
19:46:16 <clarkb> #topic Winding down iweb
19:46:48 <clarkb> Earlier this year we were told the iweb cloud would be going away. We thought it might initially go away mid year but they kept it running until the end of the year
19:47:06 <clarkb> we are at the end of the year now and need to shut it down. Ideally in a graceful way for nodepool to avoid manual cleanup
19:47:14 <clarkb> link https://review.opendev.org/q/topic:wind-down-iweb
19:47:28 <clarkb> I've got a stack of changes that should do the bulk of this work. We will need to manually delete the mirror node too
19:47:52 <clarkb> I'm happy for those to land anytime between now and about this time next week. After that my ability to pay attention will be decreased
19:48:09 <clarkb> Once we've done the cleanup I'll send them an email saying we've moved out
19:48:28 <ianw> ++
19:48:50 <ianw> hopefully avoid end of lease cleaning fees :)
19:48:55 <clarkb> so ya reviews welcome and I'll do my best to get that cleared out gracefully
19:49:24 <clarkb> Due to time constraints I'm going to skip the vexxhost rescue instance and bionic upgrades topics. There isn't much new for them as far as I know
19:49:30 <clarkb> #topic End of year Holiday party
19:50:00 <clarkb> corvus: suggested last week that our December 20 meeting coul dbe treated more like a holiday party. I like this idea. We've spent a ton of time this year running opendev we can take an hour to have some fun :)
19:50:26 <clarkb> One idea I've got is we can play games on board game arena if people are interested. You do need to create a free account but a paid account can host games that include free tier players
19:50:38 <clarkb> I'm happy to upgrade to a paid subscription for this and do the hosting
19:50:43 <corvus> ☃️
19:50:45 <clarkb> But I'm also open to other ideas
19:51:06 <clarkb> https://en.boardgamearena.com/ sorry board game erna
19:51:07 <corvus> i'm in
19:51:29 <clarkb> we can use meetpad for voice and play ticket to ride or carcassone etc
19:51:43 <ianw> ok, wow i'd never heard of that site!  i'm surprised they have licensed so many games
19:52:16 <fungi> it's quite an amazing collection of games they've implemented, yes
19:52:28 <fungi> excellent for when you want to remotely play board or card games with friends
19:52:29 <clarkb> the ui can be clunky at times but a lot of the games are actually really well done
19:52:46 <ianw> it looks like my games cupboard online, haha
19:52:53 <fungi> i was thinking of upgrading my account to paid as well
19:54:06 <clarkb> I haven't started on the annual report content  Ineed to write for 2022 yet, but it was a busy year just off the top of my head. So ya thank you everyone and lets have som efun next week :)
19:54:21 <ianw> ++
19:54:40 <corvus> clarkb: thanks!
19:54:46 <clarkb> I'll send out an agenda email but give details on where to sign up and what meetpad room we'll use
19:55:01 <clarkb> #topic Open Discussion
19:55:12 <clarkb> Anything else before our hour is up?
19:56:42 <clarkb> I'll probably be floating around after the 25th but attempting to focus on writing those annual report things
19:56:52 <clarkb> I expect things will be quiet as they usually are
19:57:24 <clarkb> I hope everyone else is able to enjoy some down time too
19:57:44 <clarkb> It might snow here next week too. I'll have to get the sleds down
20:00:03 <clarkb> sounds like that is it. Thank you everyone!
20:00:06 <clarkb> #endmeeting