Friday, 2023-12-15

fricklerseems to be the year of ML transitions
SvenKieskefrickler: first I thought you're writing about machine learning :D08:34
SvenKieskeI didn't know about though08:36
fungii miss majordomo. i was on so many majordomo based mailing lists back in the day13:51
fungithen again, i miss usenet even more13:52
fungiinfra-root: we just got notified by ems that our matrix homeserver hosting plan is being discontinued and we either need to upgrade to (and pre-pay for a year of) business grade service which will be 10x what the foundation has been paying so far. it might be time to think about migrating our domain to a homeserver we run ourselves, but i'll stick an item on the meeting agenda. we have until14:55
fungi2024-02-07 to work something out14:55
fungiour next weekly meeting isn't until a month before that deadline, so it might be worth mulling over our options in the meantime15:00
fricklerwell I'm pretty sure I don't have any interest in running a homeserver. so maybe a) check with the foundation whether the increase in budget would be feasible (what amount are we talking about?) b) check with the consuming communities whether some contribution might be possible?15:06
fricklerand maybe c) compare the pricing for other hosted homeserver offerings15:06
fungiwe've been paying us$10/mo for our homeserver, the minimum possible in february will be us$1200/yr15:07
fungiand yeah, i'm looking around to see who else does small-scale matrix homeserver hosting with custom domains15:08
fungi is cheaper than that, though still a lot more than we have been paying so far15:13
fungi looks closer to what we've been paying, though still a little more15:16
fungi is several times the new ems minimum15:17
fungi is in between but includes a lot of other services we're unlikely to use15:19
fungibut yeah, i'm also talking to people at the foundation who control the budget to find out how much of a problem this represents15:21
Clark[m]That is unfortunate, but I'm sure we can sort something out.15:51
fungiyeah, my main annoyance is that they announced it with <2 months warning while many people are not around to notice until there's little time left to do something about it15:52
Clark[m]infra-root: I:m here today but having a bit of a slow start (I got home much later than anticipated due to flight delays). I think should probably be the priority out of my backlog so that backups can be more reliable. Unless they started working on ipv6 while I wasn't checking email15:52
* fungi checks again15:52
Clark[m]And then followup with so that we can upgrade gitea maybe first thing next week (I think there may be a new point release I'll check that)15:53
fungiDate: Fri, 15 Dec 2023 05:59:36 +000015:53
fungiSubject: ACTION REQUIRED: Backup failed on gitea0915:53
fungiseems it's still broken15:53
Clark[m]Seems so. In that case I think we should start with fixing it. Please double check the ssh config file contents in that change15:56
fungiyep, i did15:56
fungiconfirmed it's relevant for the version of openssh-client on the gitea servers even15:57
Clark[m]We can run a manual backup once that applies just to confirm it resolves the issue as expected16:00
Clark[m]Re EMS. The timing is made extra bad by holidays and the year in advance pre payment...16:01
fungithe cynic in me wonders whether that was a calculated move on their part16:02
Clark[m]infra-root we are hitting in system-config jobs that run docker/docker-compose. In theory production will hit these issues too if/when we upgrade python docker lib? I think short term we should cap the lib and then maybe in the new yaer ramp up plans to switch to go docker-compose16:45
opendevreviewMerged opendev/system-config master: Force borg backups to run over ipv4
Clark[m]infra-root I'm working on eating breakfast if anyone else wants to push a change for that it would be appreciated. otherwise I'll see what I can do in a bit16:46
mnasiadkaClark[m]: fyi - builds are also failing with docker-py==7.0.0 if you use registry on a port - - so capping/pinning is what everybody is doing right now ;-)16:52
Clark[m]Hahaha I mean wow ok16:53
Clark[m]That is good to know because we run registries on nonstandard ports too iirc16:53
opendevreviewClark Boylan proposed opendev/system-config master: Pin py docker when installing docker-compose
opendevreviewClark Boylan proposed opendev/system-config master: Set both replication gitea ssh keys to the same value
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.21.2
opendevreviewClark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node
clarkbok that rebases the gitea stack on top of the docker fix as extra testing it will fix things. I also update the gitea upgrade to 1.21.2 (from 1.21.1) and have recycled the autohold17:46
clarkbI'm going to manually run the vexxhost backup on gitea09 now as the ssh config appears to have updated17:46
clarkbFri Dec 15 17:47:35 UTC 2023 Backup finished successfully17:48
clarkbheld gitea 1.21.2 lives here:
clarkbat first glance this is working. I think I'll aim for a monday upgrade then19:06
clarkbassuming reviewers are willing to rereview... sorry about all those updates. I know those changes aren't very small. There were some minor template updates too unfortunately19:06
fungithanks for going to the trouble of piecing all that together19:07
clarkbgithub emailed one of our accounts to say 2fa is coming but that we already meet the requirement19:27
clarkbso thats good19:27
tonybclarkb: Nice.22:55
tonybIf y'all are "keen" I can do the gitea update on my Monday (your Sunday)  which might be nice from a downtime POV22:56
fungii'm cool with that23:09
opendevreviewMerged opendev/system-config master: Pin py docker when installing docker-compose
opendevreviewMerged opendev/system-config master: Set both replication gitea ssh keys to the same value
clarkbtonyb: ya I think that works. The main thing is reviewing the change and making sure I addressed the changelog items properly and cross checking against the held node23:20
clarkbarg 902842 didn't trigger the infra-prod-service-gitea job23:21
clarkboh no things are still queued23:22
clarkbsorry I saw it only triggered promote jobs and looked for deploy but didn't see them beacuse there is nothing in zuul on a december friday afternoon and the column shifted23:23
fungiyeah only promotes are finished so far23:23

Generated by 2.17.3 by Marius Gedminas - find it at!