| *** gmaan is now known as gmaan_pto | 00:20 | |
| noonedeadpunk | fwiw, seems that ubuntu mirrors are broken since yesterday on this path specifically: https://archive.ubuntu.com/ubuntu/pool/main/l/linux-firmware/ | 06:17 |
|---|---|---|
| noonedeadpunk | everything else, like https://archive.ubuntu.com/ubuntu/pool/main/l/linux-base/ works nicely | 06:18 |
| fungi | noonedeadpunk: it's a known problem, incident with canonical's infrastructure | 13:16 |
| fungi | noonedeadpunk: https://askubuntu.com/questions/1555546 has some details | 13:17 |
| fungi | our own ubuntu mirrors, other than being temporarily stale, remain unaffected | 13:17 |
| fungi | we saw similar disruption in jobs pulling packages from a ppa or snaps from snapcraft too | 13:18 |
| noonedeadpunk | aha, cool, thanks | 13:24 |
| noonedeadpunk | I actually believe I saw some timeouts related to this in our jobs last week... but anyway, good it's resolved now | 13:24 |
| fungi | noonedeadpunk: the outage "incident" ended within a short time, but the impact from that outage (broken mirror syncs, backlogged publishing queues) persist still. i'm trying to upgrade one of opendev's servers and getting a http/500 error for a package even now | 14:01 |
| fungi | "Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux-firmware/linux-firmware_20240318.git3b128b60-0ubuntu2.17_amd64.deb 500 Internal Server Error" | 14:02 |
| noonedeadpunk | yeah, linux-firmware is close to the only thing being broken | 19:28 |
| noonedeadpunk | which also makes it a bit surprising for me how this happened at the first place | 19:29 |
| noonedeadpunk | unless some kind of weird .htaccess was placed by mistake | 19:29 |
| fungi | i think it was just a very large package that got uploaded/built around the time of the outage and non-atomic publication steps raced the incident | 19:30 |
| fungi | so it was in the middle of being rolled out when everything fell over | 19:30 |
| noonedeadpunk | jsut `curl https://archive.ubuntu.com/ubuntu/pool/main/l/linux-firmware/` was failing with 500 and quite instantly | 19:31 |
| noonedeadpunk | even if you don't attempt to download the package - just list this folder | 19:31 |
| fungi | if you try to fetch it enough times you'll end up eventually getting a "working" download session so slow that it will take hours to complete | 19:31 |
| noonedeadpunk | eh, maybe depends on the region. I had 500 consistently where I am | 19:32 |
| noonedeadpunk | but also it depends on DNS RR I guess | 19:32 |
| noonedeadpunk | as domain resolves to bunch of ips | 19:32 |
| noonedeadpunk | anyway | 19:32 |
| fungi | i'm watching one such upgrade on one of opendev's servers now. tried 10-20x and now it's finally going instead of erroring but says the upgrade to noble is estimated to take 22+ hours | 19:32 |
| noonedeadpunk | heh | 19:33 |
| fungi | good thing it's saturday! | 19:33 |
| fungi | i'm also thankful for redundant servers and transparent failover protocols ;) | 19:34 |
| noonedeadpunk | I found another "mirrror" which used in the meanwhile, but for non-prod systems, as it's signed with own gpg | 19:34 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!