| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Run the native CephFS job with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/881519 | 02:35 |
|---|---|---|
| opendevreview | Lukas Piwowarski proposed openstack/tempest master: [WIP] Catching broken tests in tempest-full-test-account-* jobs https://review.opendev.org/c/openstack/tempest/+/880630 | 07:09 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 12:16 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 14:57 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 15:04 |
| opendevreview | Luigi Dino Tamagnone proposed openstack/tempest master: Avoiding the clean of not tempest created resources https://review.opendev.org/c/openstack/tempest/+/879923 | 15:13 |
| *** haleyb_ is now known as haleyb | 15:14 | |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Revert "Temporary pin the ceph jobs nodeset to Focal" https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | 15:45 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 15:45 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 16:01 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: [DNM][CI] Add CEPHADM_DEPLOY flag to py3 tests https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/834223 | 16:05 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Revert "Temporary pin the ceph jobs nodeset to Focal" https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | 16:11 |
| opendevreview | Dan Smith proposed openstack/tempest master: Always validate in scenario create_server() https://review.opendev.org/c/openstack/tempest/+/881675 | 16:12 |
| opendevreview | ribaudr proposed openstack/tempest master: [WIP] Test tempest for Nova and Manila https://review.opendev.org/c/openstack/tempest/+/838743 | 16:13 |
| dansmith | gmann: I probably need to pass validation_resources to create_server() as well right? | 17:14 |
| gmann | dansmith: we do not need that as method itself creating and wait_until=SSHABLE means we need to create validation _resources either in common place or test pass | 17:17 |
| gmann | dansmith: 1 comment i missed earlier https://review.opendev.org/c/openstack/tempest/+/881675/2/tempest/scenario/manager.py#323 | 17:17 |
| dansmith | gmann: okay sorry so get_test_validation_resources() will set up and cache those and somehow create_test_server() will see them and I don't need to pass them? | 17:18 |
| gmann | dansmith: you are passing those in kwargs to create_test_server() right? https://review.opendev.org/c/openstack/tempest/+/881675/1..2/tempest/scenario/manager.py#325 | 17:19 |
| dansmith | oh hehe, right | 17:20 |
| dansmith | I did that yesterday and this morning I'm like "wtf, I don't even pass them" | 17:20 |
| gmann | :) | 17:20 |
| dansmith | gmann: even with what I have up there, I would think it should work right? | 17:35 |
| dansmith | https://zuul.opendev.org/t/openstack/build/b4cb70dc8b5e4e239233072462fead31/log/controller/logs/grenade.sh_log.txt#1986 | 17:35 |
| dansmith | I assume that's a failure to setup validation_resources? | 17:35 |
| dansmith | the other failure looks different, but perhaps due to lack of network setup? https://zuul.opendev.org/t/openstack/build/b4cb70dc8b5e4e239233072462fead31/log/controller/logs/grenade.sh_log.txt#489 | 17:36 |
| dansmith | perhaps I should just take the approach of trying to make sure all the volume tests wait for ssh'able, but..it just feels like whack-a-mole | 17:37 |
| gmann | dansmith: yeah, if we setup network we should have proper network for tenant | 17:37 |
| gmann | ssh failure, need to check, this is with class level validation resource right | 17:39 |
| dansmith | ah, that test class has set_network_resources() | 17:39 |
| dansmith | gmann: oh, is that why? | 17:39 |
| gmann | dansmith: ohk, yes. so we need to pass wait_until=ACTIVE from thre | 17:39 |
| gmann | I think we have few class doing that ipv6 and sec grp test ? | 17:39 |
| dansmith | acj | 17:40 |
| dansmith | okay so I might as well push up another PS because I assume the regular non-grenade jobs will fail this way too | 17:41 |
| gmann | yeah, it fail on tempest-full also | 17:41 |
| dansmith | okay.. I see tempest-slow passed one scenario test with volumes, so hopefully that's good news | 17:41 |
| opendevreview | Dan Smith proposed openstack/tempest master: Always validate in scenario create_server() https://review.opendev.org/c/openstack/tempest/+/881675 | 17:42 |
| dansmith | gmann: ^ | 17:42 |
| gmann | child class set_network_resources take preference | 17:42 |
| dansmith | I hope this is going to simultaneously improve stability for regular tests and enable the jammy ceph job to work | 17:42 |
| dansmith | feeling a little hopeless at the moment :/ | 17:43 |
| gmann | dansmith: there are few more test not setting network resource and does own validation , we need to pass wait_until=SSHABLE from there too | 17:50 |
| gmann | 1. https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_v6.py#L63 | 17:50 |
| gmann | 2. https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_advanced_server_ops.py#L61 | 17:50 |
| gmann | 3. https://github.com/openstack/tempest/blob/master/tempest/scenario/test_security_groups_basic_ops.py#L160 | 17:50 |
| gmann | sorry, *pass wait_until=ACTIVE | 17:51 |
| dansmith | gmann: you mean wait_until=ACTIVE? | 17:51 |
| gmann | yeah | 17:52 |
| dansmith | I'm trying to get my devstack locally back to working so I can probe for more | 17:52 |
| gmann | k | 17:52 |
| dansmith | gmann: okay that one where we need to pass wait_for=active worked, but still getting this: | 18:27 |
| dansmith | https://zuul.opendev.org/t/openstack/build/7238bf8b049c4e71b0a2a8c9a085ae8f/log/controller/logs/grenade.sh_log.txt#1500 | 18:27 |
| dansmith | that looks like not having a keypair configured properly, I assume in validation_resources | 18:28 |
| dansmith | that one already does verify ssh itself, so maybe I should wait_for=ACTIVE that one as well? | 18:28 |
| gmann | yeah, in that case we do not need to double SSH | 18:29 |
| dansmith | ah, it creates its own keypair | 18:29 |
| dansmith | so I assume that's why | 18:29 |
| dansmith | yeah | 18:29 |
| dansmith | okay | 18:29 |
| dansmith | I missed understanding that earlier | 18:29 |
| gmann | ohk, that is why. | 18:29 |
| gmann | validation resource does create keypair but there was mismatch in what server got and test trying | 18:30 |
| dansmith | yeah | 18:30 |
| opendevreview | Dan Smith proposed openstack/tempest master: Default to SSHABLE in scenario create_server() https://review.opendev.org/c/openstack/tempest/+/881675 | 18:31 |
| opendevreview | Dan Smith proposed openstack/tempest master: Default to SSHABLE in scenario create_server() https://review.opendev.org/c/openstack/tempest/+/881675 | 19:12 |
| dansmith | clarkb: has zuul started automatically restarting jobs when a depends-on is updated while the jobs are running? | 19:17 |
| clarkb | dansmith: I don't think so I would expect it to continue to kick them out | 19:49 |
| dansmith | clarkb: yeah, that's the normal behavior but it surely seems like it's restarting them | 20:03 |
| opendevreview | Dan Smith proposed openstack/tempest master: Default to SSHABLE in scenario create_server() https://review.opendev.org/c/openstack/tempest/+/881675 | 20:08 |
| dansmith | clarkb: so that ^ just restarted the jobs on the patch that depends on it, change 881585 | 20:09 |
| dansmith | no recheck required on my end | 20:09 |
| clarkb | that must be new then. I don't recall reviewing a change to make that happen | 20:16 |
| dansmith | yeah, twice now I've pushed up a patch and then gone to the dependent and refreshed in a loop until I see the zuul -1 abort, only to never see it and realize the jobs got restarted | 21:13 |
| dansmith | gmann: https://4950815d31f217aabeea-33bb14cb0ac75615a3427d1ffce62a66.ssl.cf1.rackcdn.com/881675/6/check/tempest-multinode-full-py3/9d0a185/testr_results.html | 21:26 |
| dansmith | gmann: the shelve one sets its own keypair as well, so I assume that's why we can't verify it, but it also doesn't do its own sshable check.. | 21:27 |
| dansmith | and the first one seems to be failing on secgroup, but that seems like it's unrelated, unless it's part of set_network stuff changing things? | 21:27 |
| gmann | dansmith: checking | 21:30 |
| gmann | dansmith: it does ssh to server in create_timestamp method so we can skip SSH in create_server https://github.com/openstack/tempest/blob/e4ec47a063ab03ae3d5d621556377717186668c2/tempest/scenario/test_shelve_instance.py#L88 | 21:31 |
| gmann | create_timestamp is ssh to server and write file | 21:31 |
| gmann | sec group seems not related to me too let's check again. | 21:32 |
| dansmith | gmann: ahh, makes sense.. I figured it must be because it creates the keypair, I just wasn't thinking timestamp | 21:33 |
| dansmith | okay yeah I'm hoping secgroup is just a fluke.. i'll see if the nova job hits that too | 21:33 |
| gmann | k | 21:34 |
| dansmith | gmann: there's another one I just found, in cinder encrypted volumes test, which definitely needs wait-for-ssh | 21:38 |
| dansmith | we create with keypair, but I'm not sure I see us actually use it, which is weird | 21:38 |
| dansmith | https://github.com/openstack/tempest/blob/e4ec47a063ab03ae3d5d621556377717186668c2/tempest/scenario/test_encrypted_cinder_volumes.py#L48 | 21:38 |
| dansmith | is there some easy way I can just create a validation_resources with that keypair in it or something? | 21:39 |
| dansmith | might be able to just remove this one, but not sure | 21:39 |
| gmann | in this case test one is used for server - https://github.com/openstack/tempest/blob/e64f4c2b07e048ea77d08a29b7afbd17448abc32/tempest/common/compute.py#L224 but validation one for ssh - https://github.com/openstack/tempest/blob/e64f4c2b07e048ea77d08a29b7afbd17448abc32/tempest/common/compute.py#L140 | 21:41 |
| gmann | we should fix that or just override the created kaypair in validation_resource['keypair'] | 21:41 |
| dansmith | oh we pass the name to create_server so it gets created that way in nova, but why? | 21:42 |
| gmann | dansmith: or yeah, remove one from test and let create_server() handle that will be easy | 21:42 |
| dansmith | we don't use it again right? | 21:42 |
| gmann | yeah | 21:42 |
| dansmith | maybe just copy/pasta? | 21:43 |
| gmann | may be | 21:43 |
| gmann | making create_server() handle all will be consistent | 21:44 |
| dansmith | yeah | 21:45 |
| dansmith | the nova job has failed three things, that shelve test from the tempest run and two cinder tests for this encrypted case | 21:45 |
| dansmith | hoping we're getting close | 21:46 |
| gmann | yeah | 21:46 |
| dansmith | although it's definitely taking a while | 21:46 |
| dansmith | oh actually that timer is wrong | 21:47 |
| dansmith | clarkb: when it restarts the jobs it doesn't reset the time-in-queue (not surprisingly) so this looks like it's been going for 3.5 hours :) | 21:47 |
| dansmith | ugh, cinder tempest plugin has some tests I need to fix | 21:49 |
| gmann | dansmith: oh, thinking of plugins. there are might be cases in many other plugins like neutron tempest plugin etc | 21:54 |
| dansmith | gmann: UGH | 21:54 |
| gmann | dansmith: I think we can keep arg default as ACTIVE and explicitly change tempest required scenario tests to pass SSHABLE ? | 21:54 |
| dansmith | well, the thing is, | 21:56 |
| dansmith | with this ceph thing it seems like it affects even non-volume-related tests because our instance storage is all based on ceph | 21:57 |
| dansmith | although I admit I've lost track of the range of failures through the steps I've made here | 21:57 |
| dansmith | man that really ruined my afternoon :) | 21:58 |
| dansmith | okay so.. I guess I need to revert this to default to ACTIVE and then let it run again and see what fails and decorate those as SSHABLE and see | 21:59 |
| gmann | dansmith: or another way is to check in create_server() if network resources are created and then go for self.get_tenant_network() and sshable things ? and plugins will not fail and can pass ACTIVE explicitly? | 22:01 |
| dansmith | gmann: yeah I thought about that, but we also have these keypair problems and it seems like maybe it's getting too magical there | 22:02 |
| dansmith | like also hard to tell what the validations are that it's doing for you that way | 22:02 |
| dansmith | let me just switch to opt-in and we'll see | 22:02 |
| gmann | yeah | 22:03 |
| gmann | ok | 22:03 |
| dansmith | actually the problem scenario was in the cinder test plugin anyway | 22:07 |
| dansmith | so I should probably start there after I roll back this default | 22:07 |
| dansmith | gmann: okay what do we do about this: https://github.com/openstack/tempest/blob/e64f4c2b07e048ea77d08a29b7afbd17448abc32/tempest/scenario/manager.py#L1201 | 22:12 |
| dansmith | tests can optionally pass a keypair in, so we can't just wait_until sshable there | 22:13 |
| dansmith | should I get_test_validation_resources() and then just .update(keypair=keypair) ? | 22:13 |
| gmann | dansmith: I think that will work instead of fixing many test not pass the keypair which we can do later | 22:14 |
| dansmith | I mean I will do the update if the test passes keypair= to create_server() | 22:15 |
| dansmith | oh wait | 22:16 |
| dansmith | validation_resources[keypair] is a boolean not a keypair? | 22:16 |
| dansmith | hmm, no it must be | 22:19 |
| gmann | yeah it should be storing actual keypair in validation_resources['keypair'] | 22:19 |
| gmann | the validation fixture take keypair as boolean to create keypair | 22:19 |
| dansmith | oh I see the fixture okay | 22:23 |
| gmann | ideally we should be able to pass validation resources bool in get_test_validation_resources() so that test can override those over configuration one. and validation fixture can create only requested resources | 22:24 |
| gmann | but anyways that is something we should optimize later. | 22:24 |
| gmann | currently, it create all the configured one for all tests. we can make it test based request also | 22:25 |
| gmann | but for this change, let's override it in validation_resources dict itself | 22:26 |
| opendevreview | Dan Smith proposed openstack/tempest master: Default to SSHABLE in scenario create_server() https://review.opendev.org/c/openstack/tempest/+/881675 | 22:27 |
| dansmith | ah crap, need to update the commit message (again) | 22:27 |
| gmann | dansmith: wait | 22:29 |
| * dansmith just ctrl-c'd git review | 22:29 | |
| dansmith | waiting :) | 22:29 |
| gmann | you need to copy whole keypair dict including private_key not just name https://review.opendev.org/c/openstack/tempest/+/881675/7/tempest/scenario/manager.py#328 | 22:29 |
| dansmith | we don't get the whole dict, just the name, but compute.create_test_server doesn't seem to need more than the name? | 22:30 |
| gmann | because that is what used in ssh logic on common.compute.py | 22:30 |
| gmann | yeah, that is issue | 22:30 |
| dansmith | gmann: I was going based on this: https://github.com/openstack/tempest/blob/e64f4c2b07e048ea77d08a29b7afbd17448abc32/tempest/common/compute.py#L226 | 22:31 |
| gmann | dansmith: this only pass it to nova to inject in server but later when tempest do ssh it use private_key https://github.com/openstack/tempest/blob/e64f4c2b07e048ea77d08a29b7afbd17448abc32/tempest/common/compute.py#L140 | 22:31 |
| dansmith | ah, validation_resources needs it for the ssh part you mean | 22:32 |
| gmann | yeah | 22:32 |
| dansmith | okay, that's going to be harder | 22:32 |
| dansmith | I guess I need to make it take keypair or key_name and then fix the tests to pass all of keypair | 22:32 |
| gmann | that need some plugins also to change which is main issue here. I am thinking in common.compute.py we can get kaypair and override in validation_resources in else part https://github.com/openstack/tempest/blob/e64f4c2b07e048ea77d08a29b7afbd17448abc32/tempest/common/compute.py#L226 | 22:33 |
| gmann | get keypair from nova using name | 22:33 |
| dansmith | well, I was thinking of making it either or, and I can fix these tests and the tests in the cinder test plugin (at least) to pass the whole keypair | 22:33 |
| dansmith | and then I can strip out the name for the actual create | 22:34 |
| gmann | dansmith: ohk that will work as those are only one you are passing SSHABle now | 22:34 |
| gmann | got it. +1 | 22:34 |
| gmann | this ssh thing is meshed up in tempest, i should clean it up sometime | 22:35 |
| dansmith | in all your free time? :) | 22:37 |
| gmann | yeah :) | 22:37 |
| opendevreview | Dan Smith proposed openstack/tempest master: Allow wait_until=SSHABLE in scenario manager https://review.opendev.org/c/openstack/tempest/+/881675 | 22:40 |
| dansmith | gmann: is tempest pinned for xena? | 23:32 |
| dansmith | ah, no I'm failing on zed too | 23:33 |
| opendevreview | Dan Smith proposed openstack/tempest master: Allow wait_until=SSHABLE in scenario manager https://review.opendev.org/c/openstack/tempest/+/881675 | 23:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!