| *** dmellado7 is now known as dmellado | 11:19 | |
| mhu | clarkb: big "Aliens" vibe here https://tenor.com/byPfR.gif | 12:24 |
|---|---|---|
| clarkb | I still only see the one occurrence from fungi's testing so they didn't come last night :) | 15:50 |
| clarkb | fungi: thoughts on proceeding with https://review.opendev.org/c/opendev/system-config/+/971469 today to upgrade gitea to 1.25.3? You're the only one to review it so far. Its a bugfix update so I think it should be fine | 15:51 |
| fungi | seems good to me, though i'm disappearing for the next hour to grab lunch | 15:56 |
| clarkb | it will take about that long to gate anyway | 15:56 |
| clarkb | I can approve it in ~10 minutes if there are no objections | 15:57 |
| fungi | cool | 15:57 |
| clarkb | it is approved | 16:10 |
| clarkb | slittle1_: it looks like you've managed to merge all but one of those branch merge changes. And the one that hasn't merged yet appears to be due to someone removing an approval (and it was reapproved recently so should also merge) | 16:12 |
| slittle1_ | yes, all good now. Thank | 16:15 |
| clarkb | Looking at zuul status for the gitea upgrade I notice a number of post failures and the first one I check doesn't have job logs. This implies a swift backend is failing. | 16:19 |
| clarkb | ok the gitea change just failed on the same problem :/ I'm looking into executor logs now | 16:21 |
| clarkb | that particular instance failed to upload to rax-iad on ze06. Now to see if it is only rax iad or all of rax | 16:23 |
| clarkb | second one I have checked is also rax iad and also ze06 | 16:25 |
| clarkb | the third is rax dfw on ze09. So no rax iad specific or ze06 specific | 16:27 |
| clarkb | https://zuul.opendev.org/t/openstack/build/7b1ec02b9414459081e7c2694fab0853 https://zuul.opendev.org/t/openstack/build/6d83ec480caa4dfd8f30bb1c6e23c9cb https://zuul.opendev.org/t/openstack/build/d5dcf4c627ab4a83aa49a26f6c8105ae in that order | 16:27 |
| clarkb | I'm going to check a couple more but I suspect we need to disable swift for rax entirely given both iad and dfw show up | 16:27 |
| clarkb | yes https://zuul.opendev.org/t/openstack/build/b4aca19c424c4e7aad420d13fe1ae13f is a case against rax ord so that is all three | 16:29 |
| clarkb | (that one was also on ze06 which is curious to me) | 16:29 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Disable job log uploads to rax swift https://review.opendev.org/c/opendev/base-jobs/+/972755 | 16:34 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Revert "Disable job log uploads to rax swift" https://review.opendev.org/c/opendev/base-jobs/+/972756 | 16:34 |
| clarkb | I've self approved 972755. It sets up base-test to only use rax swift for log uploads so we can test if things look better that way | 16:36 |
| clarkb | corvus: the zuul status grafana graph shows we have a really high number of available nodes relative to in use nodes. 480:160 currently | 16:43 |
| clarkb | I noticed because I was surprised 972755 is taking so long to gate. The zuul ui nodes tab doesn't seem to show available nodes so it is hard to gauge what the distribution is there but the in-use numbers seem to match up somewhat with the grafana graph | 16:45 |
| corvus | available=ready | 16:45 |
| clarkb | aha in that case there are only ~53 ready nodes in the openstack tenant | 16:45 |
| clarkb | so maybe we're leaking them somewhere? | 16:45 |
| corvus | not sure why that label was changed | 16:45 |
| clarkb | ya looking at other tenants I'm not finding the other 400 ready/available nodes | 16:46 |
| corvus | i scraped the api and got 113 | 16:47 |
| clarkb | maybe the graph is broken (counting in-use + ready as available or something like that?) | 16:48 |
| corvus | maybe? or multi counting tenants? | 16:48 |
| corvus | - target: alias(stats.gauges.zuul.nodes.state.ready, 'Available') | 16:48 |
| corvus | i dunno, that looks a lot like pointing the finger at zuul's metrics :) | 16:49 |
| clarkb | the jobs launched per hour graph implies to me that we were running at basically full steam ahead the two hours prior to this one (that is a lagging indicator) | 16:52 |
| clarkb | so ya I'm thinking this is less of a problem with zuul itself and more of a graph/metrics issue unless I find more info pointing fingers at an underlying problem | 16:52 |
| clarkb | but that would also explain why it takes a while toget nodes assigned if we are full steam ahead | 16:53 |
| clarkb | the log upload fix has its last job running now so I expect that to land shortly | 17:09 |
| clarkb | and it failed on a linter thing in my change. Let me fix that and try again | 17:16 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Disable job log uploads to rax swift https://review.opendev.org/c/opendev/base-jobs/+/972755 | 17:19 |
| clarkb | I'll update the revert once this actually merges | 17:20 |
| fungi | okay, back | 17:23 |
| opendevreview | Merged opendev/base-jobs master: Disable job log uploads to rax swift https://review.opendev.org/c/opendev/base-jobs/+/972755 | 17:36 |
| clarkb | ok updating revert now | 17:36 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Revert "Disable job log uploads to rax swift" https://review.opendev.org/c/opendev/base-jobs/+/972756 | 17:37 |
| clarkb | of course by the time I get the workaround landed the cloud seems happier. I'll test with base-test in a bit and see if things look happy again and then can land the revert if so | 17:41 |
| fungi | just as a quick follow-up, my test client from yesterday was able to get 200/ok responses from docs.opendev.org again today, then i requested the honeypot url and it's back to 403/forbidden so the expiration seems to work as expected | 18:37 |
| clarkb | thank you for testing that | 18:42 |
| clarkb | it was the main thing that isn't covered by ci due to the 24 hour delay | 18:42 |
| fungi | before i forget, paste.debian.net got an overhaul recently, this e-mail mentions things we might think about for our paste service: https://lists.debian.org/debian-devel-announce/2026/01/msg00006.html | 20:55 |
| clarkb | https://lib.rs/crates/iocaine is iocaine for anyone looking | 21:04 |
| fungi | written by another long-time debian developer | 21:07 |
| fungi | though looks like he stopped being a dd ~7 years ago now | 21:08 |
| clarkb | #status notice Zuul will be shutdown for maintenance work. See https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/WBBLBI6ZS6FA6Q5ZMH4C2MWPL3WG3H24/ for more details. | 23:41 |
| opendevstatus | clarkb: sending notice | 23:41 |
| -opendevstatus- NOTICE: Zuul will be shutdown for maintenance work. See https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/WBBLBI6ZS6FA6Q5ZMH4C2MWPL3WG3H24/ for more details. | 23:42 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
