| iurygregory | hi all, seems like https://review.opendev.org/c/openstack/tempest/+/831018 broke almost all ironic jobs .-. | 00:50 |
|---|---|---|
| iurygregory | we are hitting the non-existent-image image in our jobs | 00:50 |
| iurygregory | fyi TheJulia JayF | 00:50 |
| iurygregory | tks fungi for the help finding the patch | 00:51 |
| iurygregory | example of the failure https://zuul.opendev.org/t/openstack/build/faa331a6f85c44b2832ab9c4bf16d258/log/job-output.txt#2785-2787 | 00:51 |
| fungi | i'm not convinced it's the cause, but the time it merged and the fact that it alters the default image name in the devstack-tempest job makes it seem like a smoking gun | 00:52 |
| fungi | i overlooked it initially because the commit title didn't seem like something that would alter job definitions | 00:53 |
| fungi | lpiwowar: ^ heads up since that's your change, you might also be able to speak to the expectation for jobs inheriting that | 00:55 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/e7d4da19f6e34547ad92ed46565f2aa6/log/job-output.txt#2820-2823 previous run of the same job | 00:55 |
| iurygregory | I'm going to try to place a workaround for ironic, probably changing how we use the DEFAULT_IMAGE_NAME for IRONIC_IMAGE_NAME =) | 00:58 |
| TheJulia | eh, that looks pretty definitive a break for us | 01:02 |
| TheJulia | since we suddenly just asserted "non-existent-image" by default for any path | 01:02 |
| TheJulia | which is exactl ywhere things detonate for us now | 01:02 |
| TheJulia | We explictly reset the value in our plugin if the image is a wholedisk image | 01:03 |
| TheJulia | iurygregory: devstack/common_settings | 01:04 |
| iurygregory | yeah | 01:05 |
| TheJulia | since the variable is defined, we inherit it as an override | 01:05 |
| TheJulia | I'd swap the order around, tbh | 01:05 |
| iurygregory | doing this | 01:05 |
| TheJulia | that way we stay in the desired cirros path | 01:05 |
| TheJulia | k | 01:05 |
| iurygregory | pushing the patch | 01:05 |
| fungi | related, be aware that devstack's default cirros image version in master changed to 0.6.2, so once this is addressed in ironic's jobs you might also separately want to think about ways to follow the default image version lest that override grow stale | 01:06 |
| fungi | opendev occasionally goes through and cleans up old image versions from our on-node caches once they've no longer been default in commonly tested branches of devstack | 01:07 |
| fungi | not that we have plans to remove 0.6.1 any time soon | 01:08 |
| iurygregory | fungi, ack | 01:13 |
| TheJulia | we did for the most part, but had to force override it afaik because of changes which broke us | 01:19 |
| TheJulia | At least, that seems to be what I recall as to why we had to go and assert it directly | 01:19 |
| TheJulia | on the other hand, the change to tempest has broken all of our stable branches by asserting a default name | 01:19 |
| TheJulia | I'm honestly not exactly thrilled by this | 01:19 |
| gmann | iurygregory: hi, we have the stupid logic to configure two images in image and image_ref and that is based on DEFAULT_IMAGE_NAME and that is reason we configured DEFAULT_IMAGE_NAME with some non exit name | 01:39 |
| gmann | iurygregory: let me push the actual fix not to reply on DEFAULT_IMAGE_NAME for and not to set DEFAULT_IMAGE_NAME to non-existent-image | 01:40 |
| iurygregory | gmann, tks! | 01:40 |
| iurygregory | for now we pushed a possible fix in ironic | 01:40 |
| iurygregory | so no need to hurry | 01:40 |
| gmann | I did not know ironic does not directly use the images set by the devstack instead use the same var which devstack use to build/uoload the image | 01:44 |
| gmann | may be you want to remove that logic to avoid such break and we can handle those changes/testing in single place | 01:44 |
| iurygregory | makes sense, we probably have a reason to do this, I will talk with the ironic folks to see how we should handle, for now the workaround I pushed seems to work =) | 01:53 |
| iurygregory | https://zuul.opendev.org/t/openstack/status#886790 | 01:53 |
| iurygregory | <happy dance> | 01:53 |
| TheJulia | gmann: we have a few different scenarios we setup, one of which we need to actually edit the payload contents. a lot of it could likely be refactored though, ut we have always wanted to keep things minimal given the overall resource impact for OS images | 01:59 |
| gmann | TheJulia: ok but all those you can configure via devstack var also instead of new var here https://review.opendev.org/c/openstack/ironic/+/886790/1/devstack/common_settings#11 | 02:02 |
| opendevreview | Ghanshyam proposed openstack/devstack master: Set two different image in tempest irespective of DEFAULT_IMAGE_NAME https://review.opendev.org/c/openstack/devstack/+/886795 | 02:18 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Remove setting of DEFAULT_IMAGE_NAME to non exit image https://review.opendev.org/c/openstack/tempest/+/886796 | 02:21 |
| gmann | fixing the devstack logic for that and remove the setting of DEFAULT_IMAGE_NAME to non exist image | 02:23 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Add ironic-tempest-bios-ipmi-direct-tinyipa job in tempest gate https://review.opendev.org/c/openstack/tempest/+/886797 | 02:33 |
| gmann | iurygregory: TheJulia: ^^ we have it in devstack gate but not in tempest so adding it there to avoid such breaking change in future | 02:33 |
| TheJulia | gmann: much appreciated | 02:34 |
| iurygregory | tks! | 02:34 |
| TheJulia | gmann: w/r/t the var, I'd have to check, just not in a place to focus on that or the code until tomorrow | 02:36 |
| gmann | TheJulia: ack | 03:14 |
| opendevreview | Ghanshyam proposed openstack/devstack master: Set two different image in tempest irespective of DEFAULT_IMAGE_NAME https://review.opendev.org/c/openstack/devstack/+/886795 | 03:40 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Remove setting of DEFAULT_IMAGE_NAME to non exit image https://review.opendev.org/c/openstack/tempest/+/886796 | 03:40 |
| gthiemonge | FYI https://review.opendev.org/c/openstack/tempest/+/831018 also broke the octavia jobs.. reading the backlog now ;-) | 06:48 |
| iurygregory | oh we were not alone :D | 11:26 |
| opendevreview | Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: [WIP][DNM] Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 14:51 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Add ironic-tempest-bios-ipmi-direct-tinyipa job in tempest gate https://review.opendev.org/c/openstack/tempest/+/886797 | 16:28 |
| gmann | gthiemonge: I am fixing that in this and devstack change you already commented can be debugged more https://review.opendev.org/c/openstack/tempest/+/886796 | 16:31 |
| gmann | kopecmartin: can you check this as it block a few projects gate https://review.opendev.org/c/openstack/tempest/+/886796 | 16:31 |
| gmann | i mean configuring default image name break them and ^^ fix it | 16:32 |
| gmann | I rechecked the failure as they are not related to this | 16:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!