| dansmith | gmann: I need help. again. | 18:57 |
|---|---|---|
| dansmith | I'm still debugging issues with this test: https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_extend.py#L102 | 18:58 |
| dansmith | while trying to figure out what is going on, I've been trying to add some shell commands against the guest, | 18:59 |
| dansmith | and I'm realizing that I don't think wait_until=SSHABLE is actually being honored here because validateable is not True | 18:59 |
| dansmith | setting it to true and passing self.get_test_validation_resources() is failing to setup fip stuff | 19:00 |
| dansmith | Details: {'type': 'ExternalGatewayForFloatingIPNotFound', 'message': 'External network 9e3110b3-853c-4559-bfc2-b48e3004ab10 is not reachable from subnet 239480b1-0e3e-41f2-acfa-e9e720b8c781. Therefore, cannot associate Port 1aece2ec-9659-46d7-b251-cf69c3fd7054 with a Floating IP.', 'detail': ''} | 19:00 |
| dansmith | I've got the set_network_resources() bit to create all the networking stuff, but still no go | 19:01 |
| gmann | dansmith: hi | 20:13 |
| gmann | checking | 20:13 |
| gmann | i see, it is using volume base create_server | 20:16 |
| gmann | but network resource should be good here as it is set to True https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_extend.py#L66 | 20:16 |
| dansmith | gmann: is that different from doing set_network_resources() with all the true flags? | 20:35 |
| gmann | dansmith: no, same thing. re you trying with router, dhcp also true along with network and subnet in set_network_resources ? | 20:36 |
| gmann | that is missing in volume.base.py compare to compute.base.py | 20:36 |
| dansmith | oh jeez, you know what? re-looking at this with fresh eyes, | 20:37 |
| dansmith | I put it on the top class, which is not the class the test is in | 20:37 |
| dansmith | VolumesExtendTest vs BasevolumesExtendAttachedTest | 20:37 |
| gmann | ohk | 20:37 |
| dansmith | that said, the one I'm using already has create_default_network=True | 20:38 |
| dansmith | oh, that's what you said | 20:38 |
| gmann | yeah BaseVolumesExtendAttachedTest has that flag true | 20:38 |
| dansmith | but are you saying it doesn't inherit from whatever honors that? | 20:38 |
| dansmith | or you think it should be working as it is? | 20:39 |
| gmann | but as you mentioned, validation resource and validable flag missing there | 20:39 |
| dansmith | right, but if I pass those, I get the above floating ip error | 20:39 |
| gmann | dansmith: I think we can set router and dhcp also to true here https://github.com/openstack/tempest/blob/cb008e2167826ad0950b4fd694f344c639413c91/tempest/api/volume/base.py#L52 | 20:39 |
| gmann | like this https://github.com/openstack/tempest/blob/master/tempest/api/compute/base.py#L67 | 20:39 |
| dansmith | aha, okay I see | 20:40 |
| dansmith | so I was getting *some* of the default networking and not all, | 20:40 |
| gmann | yeah | 20:40 |
| dansmith | and not overriding all in the right test ;P | 20:40 |
| dansmith | okay now I'm getting actual auth failures, so muuch closer, thanks | 20:46 |
| dansmith | ah, which is actually leftover debug | 20:47 |
| opendevreview | Dan Smith proposed openstack/tempest master: Allow wait_until=SSHABLE in scenario manager https://review.opendev.org/c/openstack/tempest/+/881675 | 20:49 |
| opendevreview | Dan Smith proposed openstack/tempest master: Actually make volume extend tests validate https://review.opendev.org/c/openstack/tempest/+/881925 | 20:49 |
| gmann | +1, sorry for missing it in original change, I should have checked that. | 20:51 |
| gmann | dansmith: and this is nova change you are testing it right? https://review.opendev.org/c/openstack/nova/+/881585 | 20:51 |
| dansmith | gmann: not your fault, my fault | 20:51 |
| dansmith | gmann: yep | 20:51 |
| gmann | k | 20:51 |
| dansmith | gmann: last two runs of that were just exactly this failure, so it's possible we'll see a clean run on jammy/cephadm after this .. if this is really the problem making the test fail | 20:52 |
| dansmith | the same test passes locally on all the new ceph stuff, which gives me hope | 20:52 |
| gmann | k | 20:53 |
| dansmith | gibi has been saying that some of the places we wait_until=SSHABLE do not appear to actually be waiting, | 20:53 |
| dansmith | and the ceph job had validation *disabled* so it definitely wasn't | 20:53 |
| dansmith | but could be some other examples similar to this one so I'll circle back on those too | 20:53 |
| gmann | +1. thanks | 20:54 |
| dansmith | I would have assumed that we'd fail if not, | 20:55 |
| dansmith | so maybe also good to make wait_until=SSHABLE fail if no validatable=True is passed | 20:55 |
| dansmith | and only pass over it if the conf is not set | 20:55 |
| gmann | I think we can remove validatable flag itself and work with wait_until, conf, and validation resource itself. that will be clean if conf set not to validate then skip all validation else do validation as per wait_until ask | 20:59 |
| gmann | dansmith: on the SSHABLE thing. I am try to make it default in API tests, https://review.opendev.org/c/openstack/tempest/+/842240 | 21:00 |
| dansmith | gmann: I think we still need to fail if we don't get validation_resources right? | 21:00 |
| gmann | got distracted to other things and could not finish it | 21:01 |
| dansmith | because create_server is not on the test class, so we can't call get_test_validation_resources() | 21:01 |
| dansmith | gmann: nice | 21:01 |
| gmann | dansmith: yes, without validatable flag then wait_until=SSHABLE must require validation_resource | 21:01 |
| dansmith | ack yeah okay, I will work on that | 21:01 |
| dansmith | then I can run with that and just let things fail that might be missing it now | 21:02 |
| gmann | dansmith: only trick is, as common.py is stable interface for plugins so we need to keep arg in create_test_server() https://github.com/openstack/tempest/blob/master/tempest/common/compute.py#L154 | 21:03 |
| dansmith | gmann: ack, that's fine.. it's easy to check | 21:03 |
| opendevreview | Dan Smith proposed openstack/tempest master: WIP: Fail if validation flags do not agree https://review.opendev.org/c/openstack/tempest/+/881926 | 21:04 |
| dansmith | will see if any tests are already calling for SSHABLE without validation_resources or validateable ^ and will go from there | 21:05 |
| dansmith | gmann: found three offenders, all volume tests: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cb1/881926/1/check/tempest-full-xena/cb1bea2/testr_results.html | 22:15 |
| dansmith | so these are things we thought were waiting for sshable to fix detach issues, but were not | 22:16 |
| dansmith | er, two offenders I guess | 22:19 |
| opendevreview | Dan Smith proposed openstack/tempest master: Fail create if validation flags do not agree https://review.opendev.org/c/openstack/tempest/+/881926 | 22:28 |
| dansmith | zomg, I never thought I'd see it: - Failed: 0 | 22:54 |
| dansmith | gouthamr: nova job finally passed ^ .. I will recheck a couple more times and then will get the various patches lined up so they will merge in the right order | 22:55 |
| gouthamr | dansmith++ awesome! | 22:56 |
| dansmith | gouthamr: this includes several changes to volumes tests that I hope will also yield more stability on the other jobs as well.. several places we thought we were waiting for the instance to be sshable, but weren't | 22:58 |
| gouthamr | dansmith: nice; i'm seeing https://review.opendev.org/q/topic:sshable-volume-tests now.. sounds like a lot of hair pulling debugging to track these down! :) | 23:01 |
| dansmith | yeah ... | 23:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!