| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 11:08 | |
| @fungicide:matrix.org | another jitsi-meet behavior observed this morning: you can only use the recording feature if you're a moderator in the room. sounds like that might be a recent change, possible it's also configurable | 13:08 |
|---|---|---|
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 15:03 | |
| @mnasiadka:matrix.org | Clark: I've checked zp03, zuul-preview is running there, I'm not really familiar with zuul-preview - but I assume it's responsible for docs previews? Anyway - the promote patch is up and waiting - https://review.opendev.org/c/opendev/zone-opendev.org/+/985938 | 15:46 |
| @jim:acmegating.com | mnasiadka: it's actually used for more technically difficult previews than docs builds; those typically get served directly from the object storage system. zp is used for preview builds where we need to proxy data and rewrite urls. | 15:50 |
| @jim:acmegating.com | it isn't required very often | 15:51 |
| @mnasiadka:matrix.org | corvus: Sure, I don't use that in my downstream Zuul installation and the docs are not super clear on this component as well :) | 15:51 |
| @jim:acmegating.com | yep -- i mostly wanted to clarify it's not used for, like, openstack docs builds | 15:52 |
| @mnasiadka:matrix.org | Thanks :) | 15:56 |
| @dpanech:matrix.org | Hi, this Zuul job TIMED_OUT for us, could somebody help troubleshoot it? https://zuul.opendev.org/t/openstack/build/cb2ba310ce5c4f3f8345428533f89857/logs | 16:01 |
| @clarkb:matrix.org | davlet: looks like it took more than a half an hour to copy the expanded docs archive from the build job artifact: https://zuul.opendev.org/t/openstack/build/5384ae428a5d496db8812d1a580d15e2/artifacts into afs then it timed out | 16:04 |
| @clarkb:matrix.org | that tarball is 13.4MB large. Not sure how big it is if expanded (as I dont' really want to expand it locally in case its crazy) | 16:04 |
| @clarkb:matrix.org | but maybe double checking that makes sense? | 16:04 |
| @fungicide:matrix.org | looks like `TASK [upload-afs-roots : Synchronize files to AFS]` too 30 minutes, so it could mean we have something slowing down writes to afs at the moment | 16:04 |
| @clarkb:matrix.org | ya either the data is much largr than anticipated or the writes are slower than expected fro some reason | 16:06 |
| @clarkb:matrix.org | checking the source tarball content is probably the easiest place to start | 16:06 |
| @clarkb:matrix.org | might also explain why they hit the disk quota if its writing gigabytes out of that tarball? | 16:06 |
| @clarkb:matrix.org | (this is all a bit handwavy) | 16:06 |
| @fungicide:matrix.org | according to https://zuul.opendev.org/t/openstack/builds?job_name=promote-stx-tox-docs-site&project=starlingx/docs it started timing out with a run at 2026-04-16 13:34:23 | 16:07 |
| @fungicide:matrix.org | or rather 2026-04-16 13:34:30 | 16:07 |
| @fungicide:matrix.org | successful runs took roughly 1-3 minutes to complete | 16:07 |
| @fungicide:matrix.org | i'll note that there was a successful run that completed today at 15:04:02 utc (approximately an hour ago) | 16:08 |
| @fungicide:matrix.org | davlet: is the site content currently wrong? or did the successful run an hour ago catch things up? | 16:14 |
| @clarkb:matrix.org | fungi: if its specific jobs failing the maybe we have afs throughput issues between specific providers? | 16:15 |
| @clarkb:matrix.org | fungi: as the PTG winds down https://review.opendev.org/c/opendev/system-config/+/985620 and child from mnasiadka are the zp zuul-preview server dns updates. Are there any other changes people know about that should get review on a Friday afternoon/evening? | 17:40 |
| @mnasiadka:matrix.org | This is actually merged, the DNS switch needs reviews (if we want to switch it on Friday - maybe it's better to do it next week) - https://review.opendev.org/c/opendev/zone-opendev.org/+/985938 | 17:43 |
| @fungicide:matrix.org | i think it's fine. that service doesn't get a lot of use | 17:46 |
| @clarkb:matrix.org | oh I linked the wrong one. Thanks for fixing it for me. I think I reviewed the correct changes arleady just posted here incorrectly | 17:49 |
| @dpanech:matrix.org | fungi: yes it's caught up now . Is something wrong with that AFS thing ? | 17:51 |
| @clarkb:matrix.org | it is unexpected that you'd have a timeout there if the size of the file content hasn't grown significantly. It is possible there is an underlying network issue or smething that impacted that (its a network filesystem). If the problem persists we should dig in more. I still think checking the total size of that tarball after extraction is a good idea to sanity check that | 17:53 |
| @dpanech:matrix.org | clark: understood, I'll pass this along, thanks | 17:58 |
| @fungicide:matrix.org | i haven't looked, but i'd expect promote timeouts for other projects besides starlingx in that case | 18:02 |
| @clarkb:matrix.org | fungi: ya that is why I wonder about that tarball | 18:04 |
| @clarkb:matrix.org | but If the tarball is a tarbomb then I don't want to open it locally :) | 18:04 |
| @fungicide:matrix.org | browsing https://zuul.opendev.org/t/openstack/builds?job_name=promote-openstack-tox-docs&skip=0&limit=100 over similar timeframes i don't see any timeout results | 18:13 |
| @fungicide:matrix.org | not even a small increase in build times for those | 18:15 |
| @fungicide:matrix.org | certainly not the order of magnitude or greater increase starlingx's job was showing | 18:15 |
| @mnasiadka:matrix.org | Ok, I think I've had enough this week - I'll continue with zp02 removal next week and try to move forward with mediawiki role testing | 18:16 |
| @fungicide:matrix.org | the docs-html.tar.gz i downloaded only had 2249 files and was 45mib uncompressed | 18:17 |
| @fungicide:matrix.org | thanks mnasiadka! i'm pretty well beat myself | 18:17 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 18:17 | |
| @fungicide:matrix.org | i compared the docs-html.tar.gz between a timed_out build and a success build, the file counts and decompressed sizes didn't differ by more than statistically insignificant amounts | 18:21 |
| @fungicide:matrix.org | i wonder if something could have been temporarily blocking writes to that specific volume? | 18:22 |
| @clarkb:matrix.org | fungi: ok so not a tarbomb which ya I think has us going back to could ti be an afs issue somewhere in the stack? | 18:25 |
| @clarkb:matrix.org | mnasiadka: enjoy your weekend | 18:25 |
| @clarkb:matrix.org | as a reminder I'ev traveling Sunday to Friday for in person meetings so we won't have our team meeting Tuesday | 18:26 |
| @fungicide:matrix.org | popping out for a bit, be back in an hour or two | 18:34 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!