19:00:55 <clarkb> #startmeeting infra
19:00:55 <opendevmeet> Meeting started Tue Aug 20 19:00:55 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:55 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:55 <opendevmeet> The meeting name has been set to 'infra'
19:01:02 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/XHQIBDONZ6RMOINLKWHDFGKVN5ADB3N3/ Our Agenda
19:01:07 <clarkb> #topic Announcements
19:01:45 <clarkb> The openinfra summit in korea is just under 2 weeks away
19:02:03 <clarkb> I'm planning to attend so will miss the September 3 meeting. Do plan to chair the August 27 meeting though
19:03:23 <clarkb> #topic Upgrading Old Servers
19:03:52 <clarkb> Looking at Gerrit it appears tonyb has been pushing noble server replacements along
19:03:58 <clarkb> apologies that these changes got lost in the shuffle
19:04:02 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/925447/2
19:04:21 <tonyb> Slowly.  I have some updates to make to https://review.opendev.org/q/topic:noble-mirror+is:open which will be the first focal upgrade
19:04:45 <clarkb> #link https://review.opendev.org/q/topic:noble-mirror+is:open
19:05:14 <tonyb> Also I've finally finished testing the recaptcha and bulk delete extensions on the held wiki node
19:05:27 <fungi> frickler had some comments on 925438 or else i'd just insta-approve it
19:05:43 <clarkb> ya I +2'd those changes but didn't approve since there were comment sfrom frickler
19:05:48 <tonyb> I plan to respin the mediawiki role with a bunch of questions.
19:05:57 <clarkb> tonyb: I take it there weren't any major concerns that popped up in the wiki testing?
19:06:05 <clarkb> at least for those particular items?
19:06:18 <fungi> my cursory testing was good with the wiki held node
19:06:27 <tonyb> Yeah I don't know how I missed that's what the pause was on the noble updates
19:07:02 <fungi> i didn't find time to make edits with a separate account and test partrolling them, but i'd also be fine just "testing in production" for that bit since i'm the only one currently doing it anyway
19:07:28 <tonyb> clarkb, fungi: Nope once I creted a valid recatchpa token (it's a personal one) recaptcha worked as expected and once that was done the there was stuff to bulk delete
19:07:59 <clarkb> that is reassuring that things are working as you get to them
19:08:03 <tonyb> I'd like another review of https://etherpad.opendev.org/p/opendev-wiki-announce and then we can float some timelines
19:08:30 <clarkb> #link https://etherpad.opendev.org/p/opendev-wiki-announce Draft announcement for wiki server replacement
19:08:31 <tonyb> Yeah the only think that doesn't "just work" is the skin which was also expected
19:09:00 <fungi> and also purely cosmetic
19:09:23 <tonyb> We can think about how to handle the database update/migration inline on the review once they're updated
19:09:49 <clarkb> sounds good. I'll take a look at that announcement draft today. I know i've looked at it before but there appear to be new edits
19:10:16 <tonyb> Yup I incorporated your feedback and added some extra details
19:10:52 <clarkb> anything else?
19:12:27 <clarkb> #topic AFS Mirror Cleanups
19:12:57 <clarkb> This item is the first one to fall off my list as I get distracted by other things. Unfortanately, that means I haven't made any real progress on this since the removal of projects from the zuul tenant
19:13:20 <clarkb> I think the next step here is going to be continuing to do project removals from the zuul tenant config and also cleaning up jobs in projects that are still active
19:13:30 <clarkb> but I don't have a sense yet for what is left to do in the current state of things
19:13:47 <clarkb> That said there is some interest in mirroring rocky packages so it would be good to get xenial cleared out
19:14:08 <clarkb> However, it would be good to determine if we need to mirror rocky packages before proceeding with that (this came up in the TC meeting about an hour ago)
19:14:56 <clarkb> #topic Testing Rax's New Cloud Offering
19:15:17 <clarkb> Progress is slowly being made here. I'm hoping to have a quick chat with some folks on thursday to get some base details
19:15:44 <clarkb> So while I don't have anything concrete yet I expect to have more info this week
19:16:24 <tonyb> That'd be good.  It's all very "handwavy"
19:16:52 <fungi> yeah, basically hoping cloudnull can clarify it all for us
19:17:04 <clarkb> #topic Etherpad 2.2.2 Upgrade
19:17:57 <clarkb> Recently Etherpad made a 2.2.2 release. There are actually tags for releases between 2.2.2 and the release we are running but not official release objects on github
19:18:27 <clarkb> Testing quickly showed that this doesn't just work for us like some previous updates. In particular they have completely rewored how js code is imported/loaded into the browser and this appears to break our ep_headings plugin
19:19:11 <clarkb> There is an alternative plugin called ep_headings2 that does similar things, but an ancient github issue indicates ep_headings2 is not compatible with ep_headings markup. It isn't clear to me yet if this incompatibility is fatal to etherpad or if pads will just require some manual reformatting
19:19:41 <clarkb> I think we can live with manual reformatting as people can do that for pads that are active and that they care about. But if this is fatal for the service or just pads using ep_headings we will need to do more work to fix things
19:20:14 <fungi> the data is fully encapsulated in mysql right? so in theory we can dump/source production into a held node and see what happens?
19:20:16 <clarkb> Anyway that all requires some testing. I think we can hold a node on the version we are running today. Make a pad with all the variou headings content, then update to 2.2.2 and see what breaks if anything
19:20:32 <clarkb> yes an alternative to ^ is to restore the prod db into the current 2.2.2 held node
19:20:51 <clarkb> I just haven't prioritized this testing yet as I've been largely focused on getting prepared for the summit
19:21:01 <clarkb> if anyone else wants to poke at this that would be appreciated
19:21:08 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/926078 WIP Change implementing the upgrade
19:21:18 <fungi> i'll see how my tomorrow shapes up
19:21:21 <clarkb> that change is WIP until we know we won't break things with an upgrade
19:21:38 * fungi is still catching up from emergency travel madness
19:22:01 <fungi> seems like it would be straightforward to test though so i'll try to give it a shot
19:22:05 <clarkb> thanks
19:22:36 <clarkb> also worth noting we did confirm that ep_headings2 works with 2.2.2. The main question is what compatibilty looks like coming from ep_headings
19:23:09 <clarkb> I suspect in an absolute worst case we might end up needing to export and reimport all the pads in some sort of automated fashion. That will almost certainly lose all the headings formatting but should preserve the content
19:24:09 <tonyb> and the etherpad team aren't interested in the fact ep_headings is broken?
19:24:22 <fungi> it's "just a plugin"
19:24:29 <tonyb> Ah
19:24:32 <clarkb> ya ist an ancient plugin that was archived
19:24:39 <tonyb> that's a little sad
19:24:50 <tonyb> but understandable
19:25:12 <fungi> i think the idea that there are people running decade+ old etherpad instances and not auto-expiring pads is surprising to them
19:25:46 <clarkb> and more generally the new maintainer is far more interested in modernizing things than keeping compatibility with really old stuff
19:25:54 <fungi> the project itself has had several leadership turnovers and renames in that span of time, after all
19:25:56 <clarkb> for example the new auth system. Though they did eventually add the old auth system abck into etherpad
19:27:15 <clarkb> #topic Service Coorindator Election
19:27:42 <clarkb> Today is the last day for the nomination period. I haven't seen any nominations. Assuming I haven't missed one does that mean I'm it again?
19:28:09 <frickler> congratulations :)
19:28:24 <clarkb> I can make it official after the meeting if no one else wants it I guess
19:29:18 <fungi> amd you have my condolences
19:29:23 <fungi> s/amd/and/
19:29:49 <fungi> congratudolences
19:29:56 <tonyb> clarkb: Thank you for your bravery
19:30:13 <clarkb> the crowd is making their will clear
19:30:22 <clarkb> #topic Updating the Default Zuul Nodeset
19:30:37 <clarkb> last week I announced we would set ubuntu-noble as the default nodeset tomorrow
19:30:47 <clarkb> #link https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/DWF57QP75BC6GBIG7RV6PRLBDOVANET3/
19:30:54 <fungi> and there was much rejoicing
19:31:06 <clarkb> #link https://review.opendev.org/c/opendev/base-jobs/+/926360
19:31:12 <clarkb> plan is still to merge that tomorrow as announced
19:31:29 <clarkb> I intend on doing that first thing in the morning so I have plenty of time to be around to help debug if necessary
19:31:40 <clarkb> do ya'll think that is early enough in the day or would you prefer to merge it before my day starts?
19:31:53 <clarkb> I can drop the -W if that is the case
19:32:37 <frickler> I'm busy in my morning, so fine with waiting for you
19:32:52 <fungi> seems fine. i can approve it earlier if you want though
19:33:06 <clarkb> no I think that is enough time. I just wanted to give the option if we felt strongly about it
19:33:21 <clarkb> #topic openstack.org DNS Hosting Moved to CloudFlare
19:33:31 <fungi> it happened
19:33:32 <clarkb> as of very recently this manuever has been completed
19:33:40 <fungi> a few hours ago now
19:33:55 <clarkb> please be on the lookout for unexpected behavior that can be attributed to DNS
19:34:08 <tonyb> noted
19:34:16 <clarkb> fungi is able to edit the zone directly and while I can't yet do things myself I can also message other people who can
19:34:19 <fungi> thanks frickler for spotting the problem with the proxied openstack.org redirect to www.openstack.org
19:34:41 <clarkb> I've got on my todo list to create an account so that I can make edits in addition to fungi. In general though we don't make a ton of edits so I don't expect issues
19:34:54 <fungi> more generally, we should avoid making changes to that domain for a few days anyway while authoritative dns settles out
19:35:51 <fungi> also it looks like we can probably adapt our raxdns backup to use the cloudflare api, we'll just need an api key created and (obviously) code changes to enable it
19:36:20 <clarkb> I wonder if cloudflare has direct export apis
19:36:30 <clarkb> the raxdns backup has to go record by record and construct the file itself iirc
19:36:39 <frickler> so are all non-opendev websites now proxied by cloudflare?
19:36:46 <fungi> they definitely have a rest api because i perused the docs for it
19:36:52 <fungi> frickler: no, only openstack.org
19:37:28 <clarkb> (and it was before this change. I think some of the motivation for the change was to streamline that existing integration)
19:37:46 <fungi> basically, the foundation relies on cloudflare for cdn services with www.openstack.org but cloudflare makes that less expensive if you host the domain with them, while rackspace charges the foundation for dns hosting
19:38:16 <frickler> I think the good thing about it is that it is reachable via IPv6 now
19:38:19 <tonyb> yeah it looks like you can basically get a bind zone file via the API
19:38:22 <fungi> so moving the hosting for that specific domain was a cost-cutting measure
19:38:58 <frickler> openstack.org did point to a rax IP before, but I didn't check www.
19:39:22 <clarkb> frickler: oh ya I think only one of them pointed to cloudflare before
19:39:34 <clarkb> and the other redirected to the cdn or something.
19:39:41 <fungi> actually openstack.org was going to a vexxhost ip address i think, but regardless it was just hosting a redirect to the cloudflare cdn for www.o.o
19:39:42 <frickler> didn't occur to me that anyone is actually still using www.* these days
19:40:57 <fungi> welcome to the wonders of the world wide web
19:41:11 <frickler> I'd also still prefer for us to have higher TTLs on our CNAMEs again (like 1h), but also not mission critical I guess
19:41:48 <clarkb> I think that is a generally good idea. At least historically we saw dns resolution failures at a higher rate with low ttls from hosts behind NAT
19:41:54 <fungi> yeah, as i pointed out, an http patch call can adjust the ttl per record so we could do that to the (currently) 104 records which refer to opendev services
19:42:06 <clarkb> all of those udp packets can easily get lost in the NAT state tables I guess
19:42:34 <fungi> but also we try not to depend on the openstack.org domain where we can, so it should eventually just be for openstack project content (afs/kerberos is the biggest hold-out at the moment)
19:43:16 <frickler> well I don't think we will change docs.openstack.org, which I'd consider the most important one
19:43:50 <fungi> docs.openstack.org is openstack-specific content hosted from static.opendev.org via a cname in dns
19:44:29 <fungi> so basically the sort of thing i said the domain should eventually consist of once we're done moving opendev services out
19:45:29 <clarkb> #topic OpenMetal Cloud Cert Refresh
19:45:43 <fungi> with my openstack tact sig chair hat on i do think the dns and hosting for docs.openstack.org is important, but opendev's own operations don't depend on it
19:45:56 <clarkb> ya and we've got a path towards addressing that
19:46:00 <clarkb> might just take a moment
19:46:11 <frickler> the openmetal issue looks like it could be fixed
19:46:24 <clarkb> yup the background here is LE emailed us warning us that the cert was going to expire soon
19:46:24 <fungi> we'll know tomorrow-ish
19:46:27 <frickler> I checked logs earlier and they no longer showed the error
19:46:36 <fungi> oh good
19:46:39 <clarkb> frickler looked at the kolla logs and saw that the config appeared to have been updated to use the wrong email account
19:46:41 <frickler> instead there was a ratelimit error from LE
19:46:57 <clarkb> we passed this info along to openmetal and they made some changes which hopefully will resolve things
19:47:05 <frickler> but hopefulle it will be all resolved after the next cron trigger
19:47:17 <fungi> stupid rate limits. doesn't everyone know that time and space are relative anyway?
19:47:30 <clarkb> fungi: did you change to add it to certcheck land?
19:47:37 <fungi> not yet
19:47:54 <fungi> #link https://review.opendev.org/926488 "Track our OpenMetal environment HTTPS cert expiry"
19:48:12 <clarkb> maybe we should go ahead and +W that since its a minor change with minimal impact if we got it wrong
19:48:42 <fungi> looks like frickler just did. thanks!
19:48:43 <frickler> +3
19:48:49 <frickler> good idea to check port 5000
19:49:09 <fungi> agreed, it hadn't dawned on me that they could diverge in the future
19:49:10 <clarkb> cool, that should help us easily confirm things are issuing properly in the next day or so
19:49:34 <clarkb> #topic Open Discussion
19:49:40 <clarkb> Anything else with our last ~10 minutes?
19:50:07 <tonyb> not from me.
19:50:22 * fungi hungers
19:51:28 <frickler> there was some discussion about the "opendev team" name in https://review.opendev.org/c/openstack/contributor-guide/+/926511
19:51:45 <frickler> and we do reference that multiple times in our own docs
19:52:08 <fungi> oh, yeah, i think what's there now was the result of a quick stream edit
19:52:09 <frickler> so if someone feels strongly to not use that term, an update would be in order IMO
19:53:03 <fungi> i'll try to take a look tomorrow and adjust the terminology to match what we settled on when we established our new identity
19:53:04 <clarkb> https://codesearch.opendev.org/?q=OpenDev%20Team&i=nope&literal=nope&files=&excludeFiles=&repos= shows a couple of small instances
19:53:19 <clarkb> might be good to link to others if you see them
19:53:44 <frickler> I also failed to find a good reference for "all (openstack/opendev related) IRC channels"
19:53:56 <frickler> do we intentionally not have that?
19:54:11 <clarkb> frickler: openstack/project-config/accessbot/channels.yaml is probably the most reliable
19:54:15 <fungi> we don't run any irc servers, so technically speaking, no
19:54:39 <frickler> there were some refs in system-config
19:55:10 <fungi> we have channels where we maintain access controls, channels where we provide logging and meeting services, channels where we emit code review event data, channels where we supply announcements...
19:55:10 <tonyb> there was a master list on the wiki but that's certainly wrong.  apart from that you could trall meetings.opendev.org/irclogs
19:55:15 <frickler> https://docs.opendev.org/opendev/system-config/latest/project.html and https://docs.opendev.org/opendev/system-config/latest/irc.html
19:55:39 <clarkb> frickler: thanks
19:55:57 <fungi> but to be clear, we haven't attempted to offer an index of channels indicating what's on topic for each of them
19:56:31 <clarkb> and I think thats more the purvue of the projects themselves if they choose to have 100 channels
19:56:33 <frickler> yes, I resorted to referring to individual project's contributor docs
19:56:43 <clarkb> (I'm much more a fan of collapsing things down into more central channels)
19:56:46 <fungi> i agree that's the most correct guidance
19:57:02 <fungi> referring to project documentation i mean
20:00:18 <clarkb> and we are at time. Thank you everyone! we'll be back here next week at the same time and location.
20:00:22 <clarkb> #endmeeting