| opendevreview | Pranali Deore proposed openstack/tempest master: Add Tempest tests for Glance metadata injection plugin https://review.opendev.org/c/openstack/tempest/+/1001287 | 06:13 |
|---|---|---|
| opendevreview | Roberto Alfieri proposed openstack/tempest master: DNM: Test s2i speculative build pipeline https://review.opendev.org/c/openstack/tempest/+/1001292 | 09:38 |
| opendevreview | Pranali Deore proposed openstack/tempest master: Add Tempest tests for Glance metadata injection plugin https://review.opendev.org/c/openstack/tempest/+/1001287 | 10:30 |
| clarkb | Hello diskimage-builder jobs are failing due to http 502 errors from glance when we upload an image. The test jobs essentially build an image with dib, upload that image to a local devstack install in the test node, then boot a node off of that image to check that things are functional | 17:15 |
| clarkb | But the upload step hits a 502 here: https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/job-output.txt#21258 digging into the glance api logs I see https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/controller/logs/screen-g-api.txt#1329 which seems to be glance complaining that a file couldn't be deleted because it | 17:16 |
| clarkb | doesn't exist. But if I look elsewhere in the log glance is complaining that the staging dir doesn't exist at all: https://zuul.opendev.org/t/openstack/build/5cc9865391ab43479331549dfee4b11f/log/controller/logs/screen-g-api.txt#159 | 17:16 |
| clarkb | is this a known issue with glance and devstack? Maybe we need to configure a different staging directory or have devstack create that one? | 17:17 |
| clarkb | separately does anyone know why glance does not record an error in its log when it reports an error to the user? | 17:21 |
| clarkb | I suspect there may be a few bugs here :/ | 17:21 |
| clarkb | looking in lib/glance we only seem to configure the staging dir if we set multiple stores otherwise that code path isn't traversed. Additionally, we don't seem to use the top level variable for the staging dir path we just set the same path as that var's default | 17:31 |
| clarkb | looks like tempest jobs create the same warnings but they don't fail for whatever reason (maybe the image upload is succssful and tempest isn't checking the http status code?) | 17:40 |
| clarkb | oh! `Resource image_size_total is over limit of 2000 due to current usage 0 and delta 4759` I suspect this is the real source of the failure then the thing i was looking at was a red herring (one which we should probably clean up) | 17:41 |
| clarkb | I have pushed https://review.opendev.org/c/openstack/diskimage-builder/+/1001395 to see if that fixes things. Do we knwo if these limits only recently started being enforced? I can't yet find evidence that our images have more than doubled in size recently (though that is another possibility) | 17:54 |
| frickler | the limit was doubled in https://review.opendev.org/c/openstack/devstack/+/938656 because of jobs failing, so I don't think it is a new one | 18:32 |
| opendevreview | Merged openstack/devstack-plugin-lustre master: Add Lustre functional test job https://review.opendev.org/c/openstack/devstack-plugin-lustre/+/989623 | 18:46 |
| opendevreview | Merged openstack/devstack-plugin-lustre master: Fix quota enforcement and tempest config https://review.opendev.org/c/openstack/devstack-plugin-lustre/+/996909 | 18:46 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!