| frickler | seems to be the year of ML transitions https://lwn.net/ml/linux-kernel/20231212-unselfish-real-myna-67e444@lemur/ | 06:24 |
|---|---|---|
| SvenKieske | frickler: first I thought you're writing about machine learning :D | 08:34 |
| SvenKieske | I didn't know about http://mlmmj.org/ though | 08:36 |
| fungi | i miss majordomo. i was on so many majordomo based mailing lists back in the day | 13:51 |
| fungi | then again, i miss usenet even more | 13:52 |
| fungi | infra-root: we just got notified by ems that our matrix homeserver hosting plan is being discontinued and we either need to upgrade to (and pre-pay for a year of) business grade service which will be 10x what the foundation has been paying so far. it might be time to think about migrating our domain to a homeserver we run ourselves, but i'll stick an item on the meeting agenda. we have until | 14:55 |
| fungi | 2024-02-07 to work something out | 14:55 |
| fungi | our next weekly meeting isn't until a month before that deadline, so it might be worth mulling over our options in the meantime | 15:00 |
| frickler | well I'm pretty sure I don't have any interest in running a homeserver. so maybe a) check with the foundation whether the increase in budget would be feasible (what amount are we talking about?) b) check with the consuming communities whether some contribution might be possible? | 15:06 |
| frickler | and maybe c) compare the pricing for other hosted homeserver offerings | 15:06 |
| fungi | we've been paying us$10/mo for our homeserver, the minimum possible in february will be us$1200/yr | 15:07 |
| fungi | and yeah, i'm looking around to see who else does small-scale matrix homeserver hosting with custom domains | 15:08 |
| fungi | https://ungleich.ch/u/products/hosted-matrix-chat/ is cheaper than that, though still a lot more than we have been paying so far | 15:13 |
| fungi | https://etke.cc/order/ looks closer to what we've been paying, though still a little more | 15:16 |
| fungi | https://ossrox.org/ is several times the new ems minimum | 15:17 |
| fungi | https://www.federated.computer/ is in between but includes a lot of other services we're unlikely to use | 15:19 |
| fungi | but yeah, i'm also talking to people at the foundation who control the budget to find out how much of a problem this represents | 15:21 |
| Clark[m] | That is unfortunate, but I'm sure we can sort something out. | 15:51 |
| fungi | yeah, my main annoyance is that they announced it with <2 months warning while many people are not around to notice until there's little time left to do something about it | 15:52 |
| Clark[m] | infra-root: I:m here today but having a bit of a slow start (I got home much later than anticipated due to flight delays). I think https://review.opendev.org/c/opendev/system-config/+/903356 should probably be the priority out of my backlog so that backups can be more reliable. Unless they started working on ipv6 while I wasn't checking email | 15:52 |
| * fungi checks again | 15:52 | |
| Clark[m] | And then followup with https://review.opendev.org/c/opendev/system-config/+/902842 so that we can upgrade gitea maybe first thing next week (I think there may be a new point release I'll check that) | 15:53 |
| fungi | Date: Fri, 15 Dec 2023 05:59:36 +0000 | 15:53 |
| fungi | Subject: ACTION REQUIRED: Backup failed on gitea09 | 15:53 |
| fungi | seems it's still broken | 15:53 |
| Clark[m] | Seems so. In that case I think we should start with fixing it. Please double check the ssh config file contents in that change | 15:56 |
| fungi | yep, i did | 15:56 |
| fungi | confirmed it's relevant for the version of openssh-client on the gitea servers even | 15:57 |
| Clark[m] | We can run a manual backup once that applies just to confirm it resolves the issue as expected | 16:00 |
| Clark[m] | Re EMS. The timing is made extra bad by holidays and the year in advance pre payment... | 16:01 |
| fungi | the cynic in me wonders whether that was a calculated move on their part | 16:02 |
| Clark[m] | infra-root we are hitting https://stackoverflow.com/questions/77641240/getting-docker-compose-typeerror-kwargs-from-env-got-an-unexpected-keyword-ar in system-config jobs that run docker/docker-compose. In theory production will hit these issues too if/when we upgrade python docker lib? I think short term we should cap the lib and then maybe in the new yaer ramp up plans to switch to go docker-compose | 16:45 |
| opendevreview | Merged opendev/system-config master: Force borg backups to run over ipv4 https://review.opendev.org/c/opendev/system-config/+/903356 | 16:45 |
| Clark[m] | infra-root I'm working on eating breakfast if anyone else wants to push a change for that it would be appreciated. otherwise I'll see what I can do in a bit | 16:46 |
| mnasiadka | Clark[m]: fyi - builds are also failing with docker-py==7.0.0 if you use registry on a port - https://github.com/docker/docker-py/issues/3195 - so capping/pinning is what everybody is doing right now ;-) | 16:52 |
| Clark[m] | Hahaha I mean wow ok | 16:53 |
| Clark[m] | That is good to know because we run registries on nonstandard ports too iirc | 16:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pin py docker when installing docker-compose https://review.opendev.org/c/opendev/system-config/+/903776 | 17:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Set both replication gitea ssh keys to the same value https://review.opendev.org/c/opendev/system-config/+/902842 | 17:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to 1.21.2 https://review.opendev.org/c/opendev/system-config/+/897679 | 17:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 17:44 |
| clarkb | ok that rebases the gitea stack on top of the docker fix as extra testing it will fix things. I also update the gitea upgrade to 1.21.2 (from 1.21.1) and have recycled the autohold | 17:46 |
| clarkb | I'm going to manually run the vexxhost backup on gitea09 now as the ssh config appears to have updated | 17:46 |
| clarkb | Fri Dec 15 17:47:35 UTC 2023 Backup finished successfully | 17:48 |
| fungi | yay! | 18:15 |
| clarkb | held gitea 1.21.2 lives here: https://104.130.158.155:3081/opendev/system-config | 19:05 |
| clarkb | at first glance this is working. I think I'll aim for a monday upgrade then | 19:06 |
| clarkb | assuming reviewers are willing to rereview... sorry about all those updates. I know those changes aren't very small. There were some minor template updates too unfortunately | 19:06 |
| fungi | thanks for going to the trouble of piecing all that together | 19:07 |
| clarkb | github emailed one of our accounts to say 2fa is coming but that we already meet the requirement | 19:27 |
| clarkb | so thats good | 19:27 |
| fungi | yep | 20:07 |
| tonyb | clarkb: Nice. | 22:55 |
| tonyb | If y'all are "keen" I can do the gitea update on my Monday (your Sunday) which might be nice from a downtime POV | 22:56 |
| fungi | i'm cool with that | 23:09 |
| opendevreview | Merged opendev/system-config master: Pin py docker when installing docker-compose https://review.opendev.org/c/opendev/system-config/+/903776 | 23:12 |
| opendevreview | Merged opendev/system-config master: Set both replication gitea ssh keys to the same value https://review.opendev.org/c/opendev/system-config/+/902842 | 23:12 |
| clarkb | tonyb: ya I think that works. The main thing is reviewing the change and making sure I addressed the changelog items properly and cross checking against the held node | 23:20 |
| clarkb | arg 902842 didn't trigger the infra-prod-service-gitea job | 23:21 |
| clarkb | oh no things are still queued | 23:22 |
| clarkb | sorry I saw it only triggered promote jobs and looked for deploy but didn't see them beacuse there is nothing in zuul on a december friday afternoon and the column shifted | 23:23 |
| fungi | yeah only promotes are finished so far | 23:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!