19:00:55 #startmeeting infra 19:00:55 Meeting started Tue Aug 20 19:00:55 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:55 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:55 The meeting name has been set to 'infra' 19:01:02 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/XHQIBDONZ6RMOINLKWHDFGKVN5ADB3N3/ Our Agenda 19:01:07 #topic Announcements 19:01:45 The openinfra summit in korea is just under 2 weeks away 19:02:03 I'm planning to attend so will miss the September 3 meeting. Do plan to chair the August 27 meeting though 19:03:23 #topic Upgrading Old Servers 19:03:52 Looking at Gerrit it appears tonyb has been pushing noble server replacements along 19:03:58 apologies that these changes got lost in the shuffle 19:04:02 #link https://review.opendev.org/c/opendev/system-config/+/925447/2 19:04:21 Slowly. I have some updates to make to https://review.opendev.org/q/topic:noble-mirror+is:open which will be the first focal upgrade 19:04:45 #link https://review.opendev.org/q/topic:noble-mirror+is:open 19:05:14 Also I've finally finished testing the recaptcha and bulk delete extensions on the held wiki node 19:05:27 frickler had some comments on 925438 or else i'd just insta-approve it 19:05:43 ya I +2'd those changes but didn't approve since there were comment sfrom frickler 19:05:48 I plan to respin the mediawiki role with a bunch of questions. 19:05:57 tonyb: I take it there weren't any major concerns that popped up in the wiki testing? 19:06:05 at least for those particular items? 19:06:18 my cursory testing was good with the wiki held node 19:06:27 Yeah I don't know how I missed that's what the pause was on the noble updates 19:07:02 i didn't find time to make edits with a separate account and test partrolling them, but i'd also be fine just "testing in production" for that bit since i'm the only one currently doing it anyway 19:07:28 clarkb, fungi: Nope once I creted a valid recatchpa token (it's a personal one) recaptcha worked as expected and once that was done the there was stuff to bulk delete 19:07:59 that is reassuring that things are working as you get to them 19:08:03 I'd like another review of https://etherpad.opendev.org/p/opendev-wiki-announce and then we can float some timelines 19:08:30 #link https://etherpad.opendev.org/p/opendev-wiki-announce Draft announcement for wiki server replacement 19:08:31 Yeah the only think that doesn't "just work" is the skin which was also expected 19:09:00 and also purely cosmetic 19:09:23 We can think about how to handle the database update/migration inline on the review once they're updated 19:09:49 sounds good. I'll take a look at that announcement draft today. I know i've looked at it before but there appear to be new edits 19:10:16 Yup I incorporated your feedback and added some extra details 19:10:52 anything else? 19:12:27 #topic AFS Mirror Cleanups 19:12:57 This item is the first one to fall off my list as I get distracted by other things. Unfortanately, that means I haven't made any real progress on this since the removal of projects from the zuul tenant 19:13:20 I think the next step here is going to be continuing to do project removals from the zuul tenant config and also cleaning up jobs in projects that are still active 19:13:30 but I don't have a sense yet for what is left to do in the current state of things 19:13:47 That said there is some interest in mirroring rocky packages so it would be good to get xenial cleared out 19:14:08 However, it would be good to determine if we need to mirror rocky packages before proceeding with that (this came up in the TC meeting about an hour ago) 19:14:56 #topic Testing Rax's New Cloud Offering 19:15:17 Progress is slowly being made here. I'm hoping to have a quick chat with some folks on thursday to get some base details 19:15:44 So while I don't have anything concrete yet I expect to have more info this week 19:16:24 That'd be good. It's all very "handwavy" 19:16:52 yeah, basically hoping cloudnull can clarify it all for us 19:17:04 #topic Etherpad 2.2.2 Upgrade 19:17:57 Recently Etherpad made a 2.2.2 release. There are actually tags for releases between 2.2.2 and the release we are running but not official release objects on github 19:18:27 Testing quickly showed that this doesn't just work for us like some previous updates. In particular they have completely rewored how js code is imported/loaded into the browser and this appears to break our ep_headings plugin 19:19:11 There is an alternative plugin called ep_headings2 that does similar things, but an ancient github issue indicates ep_headings2 is not compatible with ep_headings markup. It isn't clear to me yet if this incompatibility is fatal to etherpad or if pads will just require some manual reformatting 19:19:41 I think we can live with manual reformatting as people can do that for pads that are active and that they care about. But if this is fatal for the service or just pads using ep_headings we will need to do more work to fix things 19:20:14 the data is fully encapsulated in mysql right? so in theory we can dump/source production into a held node and see what happens? 19:20:16 Anyway that all requires some testing. I think we can hold a node on the version we are running today. Make a pad with all the variou headings content, then update to 2.2.2 and see what breaks if anything 19:20:32 yes an alternative to ^ is to restore the prod db into the current 2.2.2 held node 19:20:51 I just haven't prioritized this testing yet as I've been largely focused on getting prepared for the summit 19:21:01 if anyone else wants to poke at this that would be appreciated 19:21:08 #link https://review.opendev.org/c/opendev/system-config/+/926078 WIP Change implementing the upgrade 19:21:18 i'll see how my tomorrow shapes up 19:21:21 that change is WIP until we know we won't break things with an upgrade 19:21:38 * fungi is still catching up from emergency travel madness 19:22:01 seems like it would be straightforward to test though so i'll try to give it a shot 19:22:05 thanks 19:22:36 also worth noting we did confirm that ep_headings2 works with 2.2.2. The main question is what compatibilty looks like coming from ep_headings 19:23:09 I suspect in an absolute worst case we might end up needing to export and reimport all the pads in some sort of automated fashion. That will almost certainly lose all the headings formatting but should preserve the content 19:24:09 and the etherpad team aren't interested in the fact ep_headings is broken? 19:24:22 it's "just a plugin" 19:24:29 Ah 19:24:32 ya ist an ancient plugin that was archived 19:24:39 that's a little sad 19:24:50 but understandable 19:25:12 i think the idea that there are people running decade+ old etherpad instances and not auto-expiring pads is surprising to them 19:25:46 and more generally the new maintainer is far more interested in modernizing things than keeping compatibility with really old stuff 19:25:54 the project itself has had several leadership turnovers and renames in that span of time, after all 19:25:56 for example the new auth system. Though they did eventually add the old auth system abck into etherpad 19:27:15 #topic Service Coorindator Election 19:27:42 Today is the last day for the nomination period. I haven't seen any nominations. Assuming I haven't missed one does that mean I'm it again? 19:28:09 congratulations :) 19:28:24 I can make it official after the meeting if no one else wants it I guess 19:29:18 amd you have my condolences 19:29:23 s/amd/and/ 19:29:49 congratudolences 19:29:56 clarkb: Thank you for your bravery 19:30:13 the crowd is making their will clear 19:30:22 #topic Updating the Default Zuul Nodeset 19:30:37 last week I announced we would set ubuntu-noble as the default nodeset tomorrow 19:30:47 #link https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/DWF57QP75BC6GBIG7RV6PRLBDOVANET3/ 19:30:54 and there was much rejoicing 19:31:06 #link https://review.opendev.org/c/opendev/base-jobs/+/926360 19:31:12 plan is still to merge that tomorrow as announced 19:31:29 I intend on doing that first thing in the morning so I have plenty of time to be around to help debug if necessary 19:31:40 do ya'll think that is early enough in the day or would you prefer to merge it before my day starts? 19:31:53 I can drop the -W if that is the case 19:32:37 I'm busy in my morning, so fine with waiting for you 19:32:52 seems fine. i can approve it earlier if you want though 19:33:06 no I think that is enough time. I just wanted to give the option if we felt strongly about it 19:33:21 #topic openstack.org DNS Hosting Moved to CloudFlare 19:33:31 it happened 19:33:32 as of very recently this manuever has been completed 19:33:40 a few hours ago now 19:33:55 please be on the lookout for unexpected behavior that can be attributed to DNS 19:34:08 noted 19:34:16 fungi is able to edit the zone directly and while I can't yet do things myself I can also message other people who can 19:34:19 thanks frickler for spotting the problem with the proxied openstack.org redirect to www.openstack.org 19:34:41 I've got on my todo list to create an account so that I can make edits in addition to fungi. In general though we don't make a ton of edits so I don't expect issues 19:34:54 more generally, we should avoid making changes to that domain for a few days anyway while authoritative dns settles out 19:35:51 also it looks like we can probably adapt our raxdns backup to use the cloudflare api, we'll just need an api key created and (obviously) code changes to enable it 19:36:20 I wonder if cloudflare has direct export apis 19:36:30 the raxdns backup has to go record by record and construct the file itself iirc 19:36:39 so are all non-opendev websites now proxied by cloudflare? 19:36:46 they definitely have a rest api because i perused the docs for it 19:36:52 frickler: no, only openstack.org 19:37:28 (and it was before this change. I think some of the motivation for the change was to streamline that existing integration) 19:37:46 basically, the foundation relies on cloudflare for cdn services with www.openstack.org but cloudflare makes that less expensive if you host the domain with them, while rackspace charges the foundation for dns hosting 19:38:16 I think the good thing about it is that it is reachable via IPv6 now 19:38:19 yeah it looks like you can basically get a bind zone file via the API 19:38:22 so moving the hosting for that specific domain was a cost-cutting measure 19:38:58 openstack.org did point to a rax IP before, but I didn't check www. 19:39:22 frickler: oh ya I think only one of them pointed to cloudflare before 19:39:34 and the other redirected to the cdn or something. 19:39:41 actually openstack.org was going to a vexxhost ip address i think, but regardless it was just hosting a redirect to the cloudflare cdn for www.o.o 19:39:42 didn't occur to me that anyone is actually still using www.* these days 19:40:57 welcome to the wonders of the world wide web 19:41:11 I'd also still prefer for us to have higher TTLs on our CNAMEs again (like 1h), but also not mission critical I guess 19:41:48 I think that is a generally good idea. At least historically we saw dns resolution failures at a higher rate with low ttls from hosts behind NAT 19:41:54 yeah, as i pointed out, an http patch call can adjust the ttl per record so we could do that to the (currently) 104 records which refer to opendev services 19:42:06 all of those udp packets can easily get lost in the NAT state tables I guess 19:42:34 but also we try not to depend on the openstack.org domain where we can, so it should eventually just be for openstack project content (afs/kerberos is the biggest hold-out at the moment) 19:43:16 well I don't think we will change docs.openstack.org, which I'd consider the most important one 19:43:50 docs.openstack.org is openstack-specific content hosted from static.opendev.org via a cname in dns 19:44:29 so basically the sort of thing i said the domain should eventually consist of once we're done moving opendev services out 19:45:29 #topic OpenMetal Cloud Cert Refresh 19:45:43 with my openstack tact sig chair hat on i do think the dns and hosting for docs.openstack.org is important, but opendev's own operations don't depend on it 19:45:56 ya and we've got a path towards addressing that 19:46:00 might just take a moment 19:46:11 the openmetal issue looks like it could be fixed 19:46:24 yup the background here is LE emailed us warning us that the cert was going to expire soon 19:46:24 we'll know tomorrow-ish 19:46:27 I checked logs earlier and they no longer showed the error 19:46:36 oh good 19:46:39 frickler looked at the kolla logs and saw that the config appeared to have been updated to use the wrong email account 19:46:41 instead there was a ratelimit error from LE 19:46:57 we passed this info along to openmetal and they made some changes which hopefully will resolve things 19:47:05 but hopefulle it will be all resolved after the next cron trigger 19:47:17 stupid rate limits. doesn't everyone know that time and space are relative anyway? 19:47:30 fungi: did you change to add it to certcheck land? 19:47:37 not yet 19:47:54 #link https://review.opendev.org/926488 "Track our OpenMetal environment HTTPS cert expiry" 19:48:12 maybe we should go ahead and +W that since its a minor change with minimal impact if we got it wrong 19:48:42 looks like frickler just did. thanks! 19:48:43 +3 19:48:49 good idea to check port 5000 19:49:09 agreed, it hadn't dawned on me that they could diverge in the future 19:49:10 cool, that should help us easily confirm things are issuing properly in the next day or so 19:49:34 #topic Open Discussion 19:49:40 Anything else with our last ~10 minutes? 19:50:07 not from me. 19:50:22 * fungi hungers 19:51:28 there was some discussion about the "opendev team" name in https://review.opendev.org/c/openstack/contributor-guide/+/926511 19:51:45 and we do reference that multiple times in our own docs 19:52:08 oh, yeah, i think what's there now was the result of a quick stream edit 19:52:09 so if someone feels strongly to not use that term, an update would be in order IMO 19:53:03 i'll try to take a look tomorrow and adjust the terminology to match what we settled on when we established our new identity 19:53:04 https://codesearch.opendev.org/?q=OpenDev%20Team&i=nope&literal=nope&files=&excludeFiles=&repos= shows a couple of small instances 19:53:19 might be good to link to others if you see them 19:53:44 I also failed to find a good reference for "all (openstack/opendev related) IRC channels" 19:53:56 do we intentionally not have that? 19:54:11 frickler: openstack/project-config/accessbot/channels.yaml is probably the most reliable 19:54:15 we don't run any irc servers, so technically speaking, no 19:54:39 there were some refs in system-config 19:55:10 we have channels where we maintain access controls, channels where we provide logging and meeting services, channels where we emit code review event data, channels where we supply announcements... 19:55:10 there was a master list on the wiki but that's certainly wrong. apart from that you could trall meetings.opendev.org/irclogs 19:55:15 https://docs.opendev.org/opendev/system-config/latest/project.html and https://docs.opendev.org/opendev/system-config/latest/irc.html 19:55:39 frickler: thanks 19:55:57 but to be clear, we haven't attempted to offer an index of channels indicating what's on topic for each of them 19:56:31 and I think thats more the purvue of the projects themselves if they choose to have 100 channels 19:56:33 yes, I resorted to referring to individual project's contributor docs 19:56:43 (I'm much more a fan of collapsing things down into more central channels) 19:56:46 i agree that's the most correct guidance 19:57:02 referring to project documentation i mean 20:00:18 and we are at time. Thank you everyone! we'll be back here next week at the same time and location. 20:00:22 #endmeeting