| @clarkb:matrix.org | 1007977 seems to work (it passed testing at least) maybe you can review that mnasiadka | 00:12 |
|---|---|---|
| @harbott.osism.tech:regio.chat | ubuntu mirror updates are stuck once again, I can look only later today, got an appointment now | 05:48 |
| -@gerrit:opendev.org- Goutham Pacha Ravi proposed: [openstack/project-config] 1008004: Add #openstack-contrib-exp IRC channel https://review.opendev.org/c/openstack/project-config/+/1008004 | 06:03 | |
| @mnasiadka:matrix.org | Jens Harbott: I'm looking at the reprepro stuck for ubuntu right now | 08:52 |
| @mnasiadka:matrix.org | Ah, wrong checksums, oh boy | 08:52 |
| @mnasiadka:matrix.org | Ok, retry seems to work | 08:56 |
| @mnasiadka:matrix.org | vos release is running | 10:02 |
| @harbott.osism.tech:regio.chat | yay. openstack release is running, too. just ftr ;) | 10:02 |
| @mnasiadka:matrix.org | Ok, re-running the Ubuntu sync so we're sure everything is fine | 10:09 |
| @mnasiadka:matrix.org | seems it is, it's running Contents exports | 10:10 |
| @fungicide:matrix.org | thanks! | 10:16 |
| @mnasiadka:matrix.org | Well, dropping the contents for Ubuntu would make the sync way faster - I'll have a look into cleaning up Contents of Debian stuff and raising a patch to skipping Contents generation in Ubuntu | 11:21 |
| @fungicide:matrix.org | oh right, that was on my list of things to eventually follow up on if you don't beat me to it | 11:24 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org marked as active: [openstack/project-config] 1007492: Revert "Temporarily remove release docs semaphores" https://review.opendev.org/c/openstack/project-config/+/1007492 | 11:37 | |
| @mnasiadka:matrix.org | Ok, vos release for the second run of ubuntu reprepro done, so we're good. | 12:10 |
| @mnasiadka:matrix.org | Cleaned Contents-{amd64,arm64}.gz in debian mirror directories (those didn't get updated since Sep 12, so that lines out with the change we rolled out) | 12:14 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 1008055: reprepro: Disable Contents generation for remaining repos https://review.opendev.org/c/opendev/system-config/+/1008055 | 12:20 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [openstack/project-config] 1007492: Revert "Temporarily remove release docs semaphores" https://review.opendev.org/c/openstack/project-config/+/1007492 | 12:42 | |
| @fungicide:matrix.org | apache worker slots are all busy in "sending reply" (W) state on lists01 at the moment | 12:46 |
| @fungicide:matrix.org | mostly busy serving back anubis honeypot mazes to crawlers | 12:48 |
| @fungicide:matrix.org | though a fair bit of traffic getting through to scrape the list archives as well | 12:50 |
| @fungicide:matrix.org | enough that mod`_proxy is eventually reporting `Error reading from remote server` | 12:52 |
| @fungicide:matrix.org | lots of hits from semrush and sogou crawlers getting through, in particular | 12:53 |
| @fungicide:matrix.org | i wonder if workers are too busy with the frontend vhost to process requests for the backend vhost, since the configuration is apache->anubis->apache->mailman | 12:56 |
| @fungicide:matrix.org | i wonder if we can refactor it to be just apache->anubis->mailman like we did with gitea | 12:58 |
| @fungicide:matrix.org | skimming the second vhost, i don't immediately see anything that couldn't move into the first | 13:00 |
| @fungicide:matrix.org | though on closer inspection, that would mean serving static content (including the old pipermail archives) without going through anubis, which may make the situation worse | 13:08 |
| @fungicide:matrix.org | so maybe we just need to limit how many concurrent requests the frontend vhost allows to something less than the total in order to leave capacity for the backend vhost to process things that make it through (or bypass) anubis | 13:09 |
| @fungicide:matrix.org | i'm trying to remember where we did something similar recently | 13:10 |
| @fungicide:matrix.org | aha, https://review.opendev.org/c/opendev/system-config/+/995978 a few months ago | 13:10 |
| @fungicide:matrix.org | oh, wait that went the other way from what i was thinking, it's limiting the number of connections that can be passed through `mod_proxy` to the backend, not actually reserving capacity for the backend. though still maybe a similar approach would help if we increase the max connections so the frontend can't starve them as easily | 13:16 |
| @fungicide:matrix.org | anyway, the storm on lists01 has subsided for the moment | 13:16 |
| @fungicide:matrix.org | okay, i need to grab a quick shower now that openstack release activity is winding down, so i may be a few minutes late joining the room for the first opendevent session today | 14:38 |
| @clarkb:matrix.org | Looks like the openstack release went well. Congrats! | 14:51 |
| @clarkb:matrix.org | I'm in meetpad and happy to use the next couple of hours to work through $things once everyone recovers from the reelase rush | 15:04 |
| @fungicide:matrix.org | omw now | 15:06 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008091: Sync gitea container accounts with upstream setup https://review.opendev.org/c/opendev/system-config/+/1008091 | 16:08 | |
| @fungicide:matrix.org | in large part thanks to the hard work of opendev collaborators, openstack's 2026.2 release completed today was one of the quickest and smoothest in history. thanks everyone! | 16:32 |
| @mnasiadka:matrix.org | ok, on my way to the preptg | 16:53 |
| @fungicide:matrix.org | d'oh! i just dropped off | 16:55 |
| @fungicide:matrix.org | we had scheduled 15:00-17:00 utc but it ended up being just me and Clark | 16:56 |
| @jim:acmegating.com | i popped in for about 5 minutes after you left :) | 16:58 |
| @fungicide:matrix.org | hah | 16:59 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008091: Sync gitea container accounts with upstream setup https://review.opendev.org/c/opendev/system-config/+/1008091 | 17:12 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008102: Update Gitea to version 28 https://review.opendev.org/c/opendev/system-config/+/1008102 | 17:30 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1008091: Sync gitea container accounts with upstream setup https://review.opendev.org/c/opendev/system-config/+/1008091 | 18:54 | |
| @fungicide:matrix.org | infra-prod-service-gitea is running in deploy | 18:57 |
| @fungicide:matrix.org | it's already done gitea09 and on to 10 | 18:58 |
| @fungicide:matrix.org | and done | 19:09 |
| @fungicide:matrix.org | things lgtm so far | 19:09 |
| @fungicide:matrix.org | i guess what we mainly need is a replication test | 19:09 |
| @fungicide:matrix.org | https://opendev.org/starlingx/root/commit/bc77d19f8f296b481078ce9aac23b0e699f4139d was pushed at 19:08:01 | 19:12 |
| @fungicide:matrix.org | the ansible run completed several minutes earlier at 19:05:15 | 19:14 |
| @fungicide:matrix.org | so replication is still chugging along | 19:14 |
| @clarkb:matrix.org | fungi: so all looks well? | 19:39 |
| @fungicide:matrix.org | yes, peachy | 19:50 |
| @fungicide:matrix.org | i think we can focus on gitea upgrade next | 19:50 |
| @clarkb:matrix.org | Looks like it passed testing so the next step is holding a node. I can work on that after I eat something | 20:02 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 848181: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 20:16 | |
| @clarkb:matrix.org | ok autohold is in place | 20:17 |
| @clarkb:matrix.org | held gitea 28 node is at 200.225.47.40 (that is actually the haproxy) | 21:15 |
| @clarkb:matrix.org | if you overload /etc/hosts to point opendev.org at that IP you can test the gitea 28 deployment there. It seems to be working for me. I can browse thinsg and clone the system-config repo out of it | 21:15 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 1008144: Disable openmetal https://review.opendev.org/c/opendev/zuul-providers/+/1008144 | 21:24 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zuul-providers] 1008144: Disable openmetal https://review.opendev.org/c/opendev/zuul-providers/+/1008144 | 21:30 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zuul-providers] 1008144: Disable openmetal https://review.opendev.org/c/opendev/zuul-providers/+/1008144 | 21:36 | |
| @clarkb:matrix.org | I'm going to hop onto the meetpad for the pre ptg in a couple of minutes. I know Anil mentioned having a conflict for today (but not tomorrow). I'm not sure if anyone else will join | 21:56 |
| @clarkb:matrix.org | if I'm the only one on there for the first 10-15 minutes I will probably call it and drop from the call. If you show up later and we are not there let us know here that you're trying to join the call and I should be able to rejoin | 21:56 |
| @fungicide:matrix.org | i may only stick around for a little while, been going for over 12 hours so far today | 21:56 |
| @clarkb:matrix.org | yup you should get some rest. It was a long productive day | 21:56 |
| @fungicide:matrix.org | i pushed the big red openstack release button at 10:00 utc on the dot | 21:57 |
| @fungicide:matrix.org | akin to unlatching the ball bearing at the top of the rube-goldberg machine that is openstack's release automation | 21:59 |
| @fungicide:matrix.org | and this time the boot did successfully kick the watering can | 21:59 |
| @clarkb:matrix.org | on the first attempt | 21:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Steve Baker: [openstack/diskimage-builder] 983814: Refactor 03-reset-bls-entries into an element https://review.opendev.org/c/openstack/diskimage-builder/+/983814 | 23:29 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!