Thursday, 2017-04-27

*** slaweq has joined #openstack-shade00:32
openstackgerritRosario Di Somma proposed openstack-infra/shade master: Use REST API for volume type_access and volume create  https://review.openstack.org/46035700:36
rods^^ since I changed a test I'm keeping this ps small :)00:37
*** slaweq has quit IRC00:37
*** gouthamr has joined #openstack-shade00:46
*** gouthamr has quit IRC01:42
*** gouthamr has joined #openstack-shade02:51
*** yolanda has quit IRC03:56
*** gouthamr has quit IRC04:01
*** yolanda has joined #openstack-shade04:04
*** yolanda has quit IRC04:10
*** gkadam has joined #openstack-shade04:12
*** slaweq has joined #openstack-shade05:20
*** slaweq has quit IRC05:26
*** slaweq has joined #openstack-shade06:27
*** slaweq has quit IRC06:32
*** cdent has joined #openstack-shade09:43
*** yfried has joined #openstack-shade10:47
*** cdent has quit IRC11:30
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in router commands  https://review.openstack.org/46051811:51
mordredrods: the gate robots did not like that :)12:06
rodsyea :)12:06
rodslooking into it12:06
mordredrods: ah! so - I think I see it - there is a weirdness with testtools.ExpectedException (that I do not like very much)12:13
mordredit matches exceptions _exactly_ - instead of matching them like python except: does12:13
mordredso the new code is throwing a more specific exception (OpenStackCloudURINotFound) which is a subclass of the more general exception those tests are written to check for (OpenStackCloudException)12:14
mordredand also the message is different now - so I think this is another case in which you get to change the test at the same time you change the code12:15
mordredyou get all the fun with this one12:15
rodsmordred lucky me :)12:16
Shrewsi think ExepectedException matching the _exact_ exception is exactly what you want12:17
Shrewsin a test scenario, you should KNOW which exact exception is being thrown12:17
rodsmordred I'm running a devstack instance to run the functional tests against it, I was just running unit tests locally :)12:18
Shrewsslaweq_: are you manually adding reviewers to your reviews? for example, https://review.openstack.org/46051812:28
Shrewsslaweq_: if so, that's not necessary12:28
*** slaweq has joined #openstack-shade12:32
mordredShrews: yah - I guess you're right12:36
Shrewsmordred: was there ever any doubt?  :-P12:36
slaweq_Shrews: yes, I was adding it manually12:37
slaweq_Should I stop doing it?12:37
*** slaweq has quit IRC12:37
Shrewsslaweq_: yeah. the folks that are interested in shade will already have things in place to get notified. adding _everyone_ just causes spam for those that don't actually review12:38
slaweq_Shrews: ok, so sorry for that12:38
* Shrews is very glad to see slaweq_ and rods getting involved in shade12:38
slaweq_I will not do it anymore12:38
Shrewsslaweq_: no worries  :)12:38
*** gkadam has quit IRC12:41
mordredShrews: me too!12:44
*** Guest94155 has quit IRC13:10
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in router commands  https://review.openstack.org/46051813:54
*** gouthamr has joined #openstack-shade14:31
*** slaweq has joined #openstack-shade14:33
*** slaweq has quit IRC14:38
rodsI might be missing something, I'm looking at this test here http://logs.openstack.org/57/460357/1/check/gate-shade-dsvm-functional-legacy/199378b/console.html#_2017-04-27_00_58_42_079351, looks like it is getting a 404 from cinder and is correctly raising a OpenStackCloudURINotFound here https://github.com/openstack-infra/shade/blob/master/shade/exc.py#L10815:36
rodsshouldn't OpenStackCloudURINotFound be caught here OpenStackCloudException and reraised as OpenStackCloudException with the correct message here https://github.com/openstack-infra/shade/blob/master/shade/operatorcloud.py#L2042 as the test is expecting https://github.com/openstack-infra/shade/blob/master/shade/tests/functional/test_volume_type.py#L10715:37
rodsmordred ^^15:38
rodssorry for all the links pasting :)15:38
rods*be caught here https://github.com/openstack-infra/shade/blob/master/shade/_utils.py#L46215:39
mordredrods: it should be directly re-raised here: https://github.com/openstack-infra/shade/blob/master/shade/_utils.py#L45615:45
mordredrods: since OpenStackCloudURINotFound is a subclass of OpenStackCloudException15:45
mordredso the original REST GET call will generate OpenStackCloudURINotFound which we will no longer wrap (since we made it ourselves)15:46
mordredhrm - that said- ... I wonder if we should alter the message attribute of the exception in https://github.com/openstack-infra/shade/blob/master/shade/_utils.py#L456 before the raise as a way to give nicer error messages but still not mess up the tracebacks15:47
mordredwe currently wrap exceptions thrown by the clients, which is pretty bad form since it makes the tracebacks harder to follow - but we also didn't want shade's users to need to catch exceptions from python-*client since we've always known we wanted to get rid of them15:48
mordredso one of the nice things about the move to REST is that we can get out of the game of intercepting and wrapping exceptions, since we can throw the right exception in the first place15:49
mordredbut, because the call is something like "adapter.get('/foo')" - the error message that throws is a little more ugly - I didn't have a _great_ idea of what to do about that yet15:50
rodsmordred yah I see, is there any doc about how to configure shade to run functional tests against a running devstack? I'd like to play a with it a little to get more familiar with the code15:52
*** yfried has quit IRC15:58
Shrewsrods: You should only need a clouds.yaml with entries for 'devstack' and 'devstack-admin' (devstack outputs one somewhere...) and just run 'tox -e functional'15:59
rodsShrews ok, thx16:00
Shrewsyou can place the clouds.yaml in your ~/.config/openstack/ directory16:00
mordredyah - devstack should already make the clouds.yaml in /etc/openstack on the devstack machine16:00
rodscool, thx16:00
*** yolanda has joined #openstack-shade16:09
*** slaweq has joined #openstack-shade16:10
*** slaweq has quit IRC16:12
*** gouthamr has quit IRC16:31
*** gouthamr has joined #openstack-shade16:40
*** slaweq has joined #openstack-shade17:13
*** slaweq has quit IRC17:18
*** slaweq has joined #openstack-shade17:47
slaweqmordred: hello, can You have a look at result of tests in https://review.openstack.org/#/c/460518/ ?17:48
slaweqmordred: gate-shade-dsvm-functional-legacy are failing but I'm not sure how it could be connected to this patch17:48
slaweqIMHO it's not connected but maybe I'm wrong :)17:49
mordredslaweq: looking18:01
*** yfried has joined #openstack-shade18:02
*** Matias has joined #openstack-shade18:05
mordredslaweq: ah- I believe I know what that is, and no, it's not related to your patch18:05
mordredslaweq: lemme go poke a few folks18:05
slaweqmordred: thx, that's good information for me because I had no idea how I could fix it :P18:06
*** yfried has quit IRC18:22
*** slaweq has quit IRC18:26
*** yfried has joined #openstack-shade18:27
*** yfried has quit IRC18:36
*** dhellmann has quit IRC18:38
*** dhellmann has joined #openstack-shade18:39
*** openstackgerrit has quit IRC18:48
*** slaweq has joined #openstack-shade18:50
*** slaweq has quit IRC19:00
*** cdent has joined #openstack-shade19:01
*** slaweq has joined #openstack-shade19:25
*** yfried has joined #openstack-shade20:06
*** cdent has quit IRC20:19
*** yfried has quit IRC20:41
*** ianw has quit IRC22:13
*** ianw has joined #openstack-shade22:18
*** purplerbot has quit IRC22:19
mordredslaweq, rods, Shrews: I had an all-afternoon meeting so haven't been able to dig in to the gate failure22:39
mordredit's related toa recent devstack change, so has nothing to do with shade, but it'll likely be tomorrow before we can get it fixed22:39
slaweqmordred: ok, thx for info22:40
slaweqI will just wait22:40
rodsmordred np, I'm able to reproduce the functional tests failure locally and working on fix them22:41
clarkbmordred: oh thanks for reminding me thats what i was going to look at more closedly after lunch. That and sdagues thing22:42
mordredclarkb: oh sweet - if you find the culprit that would be ossum22:42
Shrewsmordred: this was the functional-legacy failure? devstack should NOT change in those22:42
Shrewsunless something was backported22:43
Shrewsseems unlikely for devstack22:43
clarkbShrews: thats what I said22:43
clarkbhence my interest (and confusion)22:44
mordredyah. but - the keystone there seems to clearly be doing suburl and not port-based22:44
mordredso - yes - it seems something is wrong in a way that it should not be22:44
Shrewsi like things being wrong in ways that they should be wrong22:44
mordredclarkb: we have BRANCH_OVERRIDE set in the legacy job - is there any thing else we're maybe doing wrong to tell devstack to give us newton?22:45
clarkbmordred: maybe that isn't working properly?22:47
clarkbor maybe things got backported22:47
mordredclarkb: I don't see any devstack backport patches22:48
clarkbya I don't either22:49
mordredand the devstack-gate log looks like it ran setup_project with stable/newton22:49
mordredhttp://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-04-27_16_14_55_84022:49
clarkba595076 is what we chceked out22:49
clarkbwhich looks right22:50
mordredclarkb: oh - actually- the suburl is a red-herring- keystone has been deploying suburl for a while22:51
clarkbbecause of the wsgi switch22:51
mordredyah22:51
clarkbits odd to me that we would do it the way we did it though where http and port 5000 are apparently both valid22:51
mordredbut these: 2017-04-27 16:18:46.167 | KEYSTONE_SERVICE_URI=http://198.72.124.194/identity22:52
mordred2017-04-27 16:18:46.167 | KEYSTONE_SERVICE_URI_V3=http://198.72.124.194/identity/v322:52
mordredare not the endpoints that wound up in clouds.yaml22:52
clarkbhttp://198.72.124.194/identity_v2_admin is in clouds.yaml for all accounts22:53
mordredclarkb: yah - so - that's stupid and a bug - but I think it's not changed in a while22:58
Shrewsmordred: the version of keystoneauth being used is the latest: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_21374823:00
Shrewsthat doesn't seem right23:00
mordredShrews: yah - we don't cap libraries23:00
clarkbcould it be that keystoneauth broke talking to old keystone?23:01
clarkb(I'd expect more screaming if it did though)23:01
mordredyah - that would break everything23:01
Shrewsthat's what i'm suspecting23:01
mordredclarkb: where do we log devstack gate hook scripts?23:02
clarkbmordred: in console.html iirc23:02
mordredyes. thanks23:02
Shrewsmordred: i assumed k/a had a stable/newton branch, similar to the other libs. i guess not23:03
mordredclarkb, Shrews: it's making the connections to the service23:05
clarkbok not firewall which makes sense because its localhost to loachost23:05
clarkbthe magnum issue was it was from nested vm to localhost23:05
clarkband that tripped iptables23:05
mordredlike - it does version discovery properly here: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_21928523:06
*** slaweq has quit IRC23:06
*** slaweq has joined #openstack-shade23:07
clarkbya it seems like its 404ing on the actual listings of users23:09
*** slaweq has quit IRC23:11
mordredit 404s on creating a service here: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_22740423:11
clarkbmordred: and if I grep for the request id in the keystone logs no match23:11
mordredI found 2 entries forthat23:12
mordredhttp://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/logs/apache/keystone.txt.gz#_2017-04-27_16_40_57_09123:12
mordredhttp://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/logs/apache/keystone.txt.gz#_2017-04-27_16_40_57_13123:12
mordredbut they're not much help23:12
clarkbya and no errors in that log23:13
mordredI also see: 198.72.124.194 - - [27/Apr/2017:16:40:57 +0000] "POST /identity/v2.0/OS-KSADM/services HTTP/1.1" 404 381 "-" "python-keystoneclient"23:14
mordredin the access.log23:14
clarkbmordred: what are the two strings after the request id in the keystone log?23:21
clarkbtokens/23:22
mordredclarkb: I do not know23:22
clarkbalso why is the auth uri the admin endpoint23:25
mordredbecause the clouds.yaml writing is dumb23:26
clarkbwell its done that way all over though not just clouds.yaml23:26
clarkbanyways reading teh devstack log it appears that everything is using v323:26
clarkband shade is using v2, so perhaps v2 isn't being allowed here for some reason?23:27
clarkbmordred: can you try forcing v3 and see what happens ? at the very least it might generate better errors we can trace in the logs23:27
mordredclarkb: https://review.openstack.org/460775 this is the weird thing with the clouds.yaml23:28
mordredclarkb: I can - although my main question then will be "what changed since yesterday"23:29
clarkbtempest stopped testing v2 looks like23:29
clarkbso it could've regressed without us noticing maybe23:29
mordredhttp://logs.openstack.org/27/460127/2/gate/gate-shade-dsvm-functional-legacy/da647ab/23:31
mordredis a run from yesterday that worked23:31
mordredclarkb: the only thing that's different from a pip install perspective is nova :)23:32
mordredclarkb: and there is a point-release difference of mysql23:34
mordredin terms of apt packages23:34
clarkbwhat about keystone configs?23:34
mordredwe dont seem to save tox dirs on these do we ...23:37
clarkbfor tempest? no23:37
*** slaweq has joined #openstack-shade23:38
mordredwell, I should add that to shade functional tests since we run them from tox23:38
mordredclarkb: I just move the dir into $WORKSPACE/logs and it gets picked up, right?23:39
clarkbI think only if the files have a recognized .txt/.log suffix? I'd have to look23:40
mordredbleh23:40
clarkbmordred: what I would do is grab reproduce.sh and then interact with it "locally"23:41
clarkbor just hold a node23:41
clarkbbceause this si really weird looking at configs I don't see diffs23:41
clarkbalso try v323:42
mordredos-client-config changed23:42
mordrednow - I have ZERO clue how that would result in this particular behavior23:42
*** slaweq has quit IRC23:42
clarkbdoesn't it make the keysteonauth objects?23:43
clarkband I bet its not tested against v223:43
mordredit does - and I'm sure it's the culprit23:43
mordredbut I don't understand how making the objects could have changed such that this changed ...23:43
mordredbut I now have something to investigate23:44
clarkbmordred: 01ff292e078206e487751228be4a7062ba0c6048 perhaps23:47
*** openstackgerrit has joined #openstack-shade23:47
openstackgerritMonty Taylor proposed openstack-infra/shade master: Add some more debugging to the post_test_hook  https://review.openstack.org/46077923:47
openstackgerritMonty Taylor proposed openstack-infra/shade master: DNM block os-client-config 1.27.0  https://review.openstack.org/46078023:47
mordredclarkb: yup. totally willing to bet23:48
mordredjamielennox: hey - so - remember this: https://review.openstack.org/#/c/450259/ ?23:49
clarkband old keystone + keystoneauth rely on some side effecting behavior there is my guess23:49
jamielennoxmordred: hmm23:49
mordredjamielennox: we're tracking down shade gate break that started yesterday and currently have a hunch that on old keystonev2 something about that is still needed23:50
mordredjamielennox: (os-client-config is the only thing different between a failing and passing test run)23:50
mordredjamielennox: however, http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/ is one of the fails if you wanna look at the rest interactions23:51
jamielennoxmordred: the only reason i can see that failing is if the url in the catalog is bad23:51
Shrewsbut we have occ jobs that cross test with shade, and they seem to have passed: https://review.openstack.org/#/c/450259/23:51
mordredShrews: we use venv for our functional tets- that did not install os-client-config change into our tox venv :(23:51
clarkbit looks like get_endpoint defaults interface to None and not public (at least in the current checkout I have of keystoneauth23:52
mordredShrews: (I'm just now realizing this hole in how we have that setup)23:52
mordredjamielennox: it winds up trying to POST to http://198.72.124.194/identity/v2.0/OS-KSADM/services23:53
mordredjamielennox: for which it gets a 40423:53
mordredhttp://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_22589423:53
Shrewsmordred: hmm, big hole23:53
mordredShrews: yah23:53
jamielennoxmordred: ok, i can see the problem, but not the why23:53
mordredShrews: I'll fix that next :)23:53
mordredjamielennox: sweet! what's the problem?23:54
jamielennoxso looking here: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_43_18_42027823:54
jamielennoxlooks like the auth_url is: GET http://198.72.124.194/identity_v2_admin23:54
jamielennoxMaking authentication request to http://198.72.124.194/identity_v2_admin/v2.0/tokens23:55
mordredyah - I just made a patch: https://review.openstack.org/460775 to change how that's getting written out23:55
mordredyou think that might be the issue?23:55
clarkbit wasn't an issue before the occ release23:56
clarkbsomething to consider if other users are possibly impacted23:56
mordredwell - it may be a combo thing23:56
jamielennoxbut then the catalog hasn't got the right admin interface?23:56
mordredthe previous occ had a behavior that masked a broken config23:56
jamielennoxcause the next step is REQ: curl -g -i -X GET http://198.72.124.194/identity23:56
jamielennoxwhich is the non-admin23:57
jamielennoxthat OS-KSADM is only available on the keystone admin interface - which is not what you're hitting23:57
mordredjamielennox: I wish that was showing the catalog ...23:58
jamielennoxmordred: was deemed to spammy23:58
mordredbut yah - you're right - it's super strange that it's going from http://198.72.124.194/identity_v2_admin to http://198.72.124.194/identity23:58
mordredwell - I can push up a patch that will print the catalog in that test :)23:59
jamielennoxmordred: does it set interface='admin'?23:59
jamielennoxcause it's defaulting to public23:59
clarkbthe occ patch appears to have stopped setting interface entirely23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!