| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Add temporary noop job https://review.opendev.org/c/opendev/zuul-jobs/+/932256 | 16:25 |
|---|---|---|
| fungi | that's a fun bug | 16:27 |
| corvus | yeah, i'm pretty sure we can reproduce that in tests, but it's certainly weird. | 16:28 |
| opendevreview | Takashi Kajinami proposed openstack/diskimage-builder master: tox: Drop envdir https://review.opendev.org/c/openstack/diskimage-builder/+/932266 | 16:32 |
| corvus | hrm what's really weird is that is having the same problem... maybe due to the empty nodeset | 16:32 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Add temporary noop job https://review.opendev.org/c/opendev/zuul-jobs/+/932256 | 16:33 |
| corvus | yep that works better. so something about repo-state plus no nodes | 16:34 |
| opendevreview | Merged opendev/zuul-jobs master: Add temporary noop job https://review.opendev.org/c/opendev/zuul-jobs/+/932256 | 16:34 |
| gouthamr | o/ is it me; or is the login for wiki.openstack.org not working for anyone else? I'm able to use UbuntuOne login; but, the "return_to" link isn't populated properly; and the login doesn't persist | 17:13 |
| corvus | gouthamr: i see the same behavior | 17:19 |
| fungi | yeah, not working for me either | 17:31 |
| fungi | maybe finishing the replacement is becoming more urgent | 17:31 |
| opendevreview | Merged opendev/zuul-jobs master: Enable the debian-bullseye image build job https://review.opendev.org/c/opendev/zuul-jobs/+/932119 | 17:35 |
| fungi | load average on the wiki server is hovering around/above 20, pretty much all apache but backups also kicked off about 45 minutes ago. i've added a bunch more ai crawler user agent strings to the filter in its apache config, but can also try rebooting the server once the current backup completes (last reboot was 1177 days ago and there's a kernel update waiting) | 18:28 |
| fungi | load average is has climbed higher, approaching 50 now, but that may be related to me asking apache to gracefully reload its config a few minutes ago so workers might be cycling out and driving the cpu usage up | 18:30 |
| fungi | backup finished, load average is down under 5 too, rebooting the server | 19:03 |
| fungi | the rootfs is 300+gb and hasn't been fsckd in a while, so will probably be refusing connections for some time | 19:08 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Temporarily indicate all images are raw https://review.opendev.org/c/opendev/zuul-jobs/+/932269 | 19:08 |
| fungi | i'll check on it when it comes back up | 19:08 |
| corvus | zuul correctly recognized the image was built, but did not upload it because it think it's the wrong format. i'm proposing we lie about the format for a bit until we finish the implementation of multiple format handling in zuul ^ | 19:10 |
| fungi | from the glance end that will probably fail on newer/patched openstack because of the https://security.openstack.org/ossa/OSSA-2024-002.html fixes | 19:14 |
| fungi | basically now if you tell glance you're uploading a raw image, then it really has to be raw and not qcow2 or vhd or whatever | 19:14 |
| fungi | though it might get as far as appearing in glance and then nova will refuse to boot it, i'm not entirely sure | 19:15 |
| corvus | oh yeah, i think you're right; that does end up in the upload call, so it probably won't work. i'll have to do the slightly bigger fix in zuul. | 19:16 |
| corvus | (it would get as far as the upload, which is still useful information, but better to try for the next step too) | 19:17 |
| fungi | the actual behavior change might have been in the OSSA-2024-001 fixes for that matter. but essentially openstack discovered that it's not safe to rely on qemu's image auto-detection features so now checks images ahead to make sure they're what they claim to be before passing them to any qemu tools | 19:18 |
| fungi | and all the openstack services that deal with images keep track of what kind of image it's supposed to be and then tell qemu to only use the specific driver for that image type | 19:19 |
| fungi | wiki.o.o is up again, load average is around 3 at the moment | 19:20 |
| fungi | content is loading | 19:21 |
| fungi | i'm able to log in and stay logged into mediawiki again now | 19:22 |
| fungi | gouthamr: see if things are any better for you | 19:23 |
| gouthamr | \o/ works for me now fungi.. thank you very much for fixing this up! :) i really didn't expect you looking over the weekend.. | 19:25 |
| fungi | gouthamr: no worries, i was taking a break from hallowe'en decorating and oid-na packing | 19:27 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!