| rojaeswaran | Can I please get a review for this PR? https://review.opendev.org/c/openstack/diskimage-builder/+/999989 | 17:03 |
|---|---|---|
| rojaeswaran | I think the gated jobs, dib-devstack-functional-openstack-ubuntu* are failing since 07/30/26 due to network issues, any ETA on when they will fixed? | 17:08 |
| clarkb | its not a network issue its an http 502 error from the glance image service: https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/job-output.txt#21258 I think it maps to https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/controller/logs/screen-g-api.txt#1329 fungi are you aware of glance problems like that? | 17:10 |
| fungi | no, other than my opendev/zuul-providers change that can't merge because at least some glance endpoints time out or error when we're uploading to them | 17:11 |
| clarkb | this is a devstack operated glance within the test job. We build an image, upload to that glance, then boot the image and check it | 17:12 |
| fungi | but this is glance in devstack not in public clouds, so no idea if it's similar | 17:12 |
| clarkb | it looks liek glance is saying I have failed because I couldn't delete the staged image content because it is already gone | 17:12 |
| clarkb | which is a great error mode | 17:12 |
| fungi | sounds like a race condition or parallel test collision | 17:12 |
| clarkb | there is only one image upload here and one test (its dib specific build one image and boot it not tempest etc) but ya maybe glance has background cleanups that can interfere or maybe it has to do with the double // in the path | 17:13 |
| clarkb | I wonder if that makes the file path lookup break | 17:13 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/controller/logs/screen-g-api.txt#185 there is also this | 17:14 |
| clarkb | I'm beginning to suspect that glance made a change requiring this path to exist and didn't update devstack | 17:14 |
| clarkb | I'll ask the openstack qa team | 17:14 |
| fungi | that sounds plausible | 17:16 |
| clarkb | `Resource image_size_total is over limit of 2000 due to current usage 0 and delta 4759` ok I think this may be the actual issue | 17:41 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/controller/logs/screen-g-api.txt#1322 | 17:42 |
| clarkb | why glance returns an http 502 for invalid user provided content I have no idea | 17:42 |
| clarkb | should be a 413 | 17:42 |
| clarkb | so something likely changed to more than double the size of the images? | 17:43 |
| clarkb | or they reduces the size of the default limit I guess | 17:43 |
| clarkb | no it was 1000 prior to ~January 2025 and increased to 2000 at that point | 17:44 |
| clarkb | rojaeswaran: is it possible that this change is making much bigger images | 17:45 |
| clarkb | no other unrelated changes to debootstrap appear to hit the same problem | 17:46 |
| clarkb | so the issue is some global image generation issue with dib itself or something change in the openstack clouds. Maybe those limits weren't enforced on first upload or something | 17:47 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Increase devstack Glance total image size limits https://review.opendev.org/c/openstack/diskimage-builder/+/1001395 | 17:52 |
| clarkb | maybe that will help? | 17:52 |
| fungi | i guess we don't know what caused the images to suddenly double in size? | 18:00 |
| clarkb | well I'm not convinced anything has changed that but I'm not sure we have the logs to know for certain? | 18:01 |
| clarkb | I think it is just as likely that glance was not enforcing this limit and suddenly started to | 18:02 |
| clarkb | but also maybe qemu-img or something updated. But all distro families appear affected so not debootstrap related I don't think | 18:05 |
| clarkb | ok I think that fixes it but dib-functests are failing on a rocky 9 package issue: https://zuul.opendev.org/t/openstack/build/7a5f4949e00845049437cdb165eaaec7/log/logs/rocky-container_rocky-9-build-succeeds.FAIL.log#2052-2058 NeilHanlon any chance you understand what may be happening there? | 18:19 |
| clarkb | following up on the rocky build failure: From a failed image build it says `baseos: using metadata from Thu Aug 13 10:42:05 2026.` but from a successful build we have `baseos: using metadata from Tue Aug 18 05:59:43 2026` so I think this is a mirror sync issue. I will recheck the change as a result | 21:39 |
| clarkb | rojaeswaran isn't here but I reviewed that change and I suggseted one of two paths for dealing with all of the newly proposed variables | 21:57 |
| clarkb | if anyone else has ideas on how to properly pass extra args into mmdebstrap or debootstrap while maintaing some backward compaitiblity your review is also appreciated | 21:57 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!