*** TheJulia has quit IRC | 02:26 | |
*** gouthamr has quit IRC | 02:27 | |
*** TheJulia has joined #openstack-shade | 02:29 | |
*** gkadam has joined #openstack-shade | 03:15 | |
*** gkadam is now known as gkadam-brb | 03:15 | |
*** gkadam-brb is now known as gkadam | 03:46 | |
*** jamielennox is now known as jamielennox|away | 05:12 | |
*** jamielennox|away is now known as jamielennox | 05:17 | |
*** yfried has joined #openstack-shade | 06:21 | |
yfried | hi, any core members around? | 06:42 |
---|---|---|
*** ioggstream has joined #openstack-shade | 07:22 | |
*** ioggstream has quit IRC | 07:32 | |
*** jamielennox is now known as jamielennox|away | 07:34 | |
*** ioggstream has joined #openstack-shade | 07:56 | |
*** jamielennox|away is now known as jamielennox | 08:28 | |
*** openstackgerrit has quit IRC | 08:33 | |
*** openstackgerrit has joined #openstack-shade | 08:51 | |
openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add ability to pass in user_agent https://review.openstack.org/452550 | 08:51 |
mordred | yfried: sup? | 09:01 |
yfried | mordred: hi | 09:08 |
yfried | mordred: how are you online so early? | 09:08 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Pass in app_name information to keystoneauth https://review.openstack.org/456268 | 09:11 |
mordred | yfried: sometimes one wakes up early in the morning for no good reason | 09:11 |
yfried | mordred: good for me :) | 09:11 |
yfried | mordred: we have a problem with shade dep | 09:11 |
yfried | mordred: I wonder if you could somehow help me | 09:12 |
yfried | mordred: https://github.com/openstack-infra/shade/commit/15d64f2af5d9f45ba91bc7f4f5f48402ab8c339b | 09:12 |
yfried | mordred: we were hit by this | 09:12 |
yfried | mordred: we are looking for a reliable way to depend on shade, but it keeps breaking us | 09:13 |
yfried | mordred: you guys ususually won't see it in shade or in ansible, but we are hit when argparse is evaluating dependencies | 09:13 |
yfried | pkg_resources.ContextualVersionConflict: (pbr 2.1.0 (/tmp/ir-venv-Bt1WP4k/lib/python2.7/site-packages), Requirement.parse('pbr!=2.1.0,>=2.0.0'), set(['openstacksdk'])) | 09:14 |
yfried | mordred: is there a way to make shade dep imprevious to this kind of stuff? | 09:14 |
mordred | yfried: hrm - well - lemme look real quick | 09:15 |
mordred | yfried: (we just started following global-requirements in shade so it's a new process to us) | 09:16 |
yfried | mordred: I try to be patient, but my users aren't so accomodating | 09:17 |
mordred | yfried: oh - are you depending on shade master? | 09:17 |
yfried | mordred: no | 09:17 |
yfried | mordred: I'm pining shade (currently 1.19.0) | 09:17 |
yfried | mordred: and testing every release before bumping it | 09:18 |
yfried | mordred: I'm doing the same with ansible | 09:18 |
yfried | mordred: but we (and you also, though you might not see this) are being hit by clients bump becuase shade have uncapped deps | 09:19 |
mordred | I mean - I ask because that patch just landed a few hours ago | 09:19 |
yfried | python-novaclient>=7.1.0 # Apache-2.0 | 09:19 |
mordred | yah. well, _that_ I'm working on fixing as quickly as humanly possible | 09:19 |
yfried | mordred: yes, and we had to do the same for our req.txt untill we have a shade release that fixes it | 09:19 |
yfried | for example, if you have a novaclient release that breaks, we are breaking top | 09:20 |
yfried | too | 09:20 |
mordred | yes. this is why we're removing the deps on novaclient | 09:20 |
yfried | mordred: novaclient is one example :) | 09:20 |
mordred | but I'm confused - are you saying the issue is that you need https://github.com/openstack-infra/shade/commit/15d64f2af5d9f45ba91bc7f4f5f48402ab8c339b in a release? | 09:20 |
mordred | yfried: I know - but the client libs are the source of almost all of these problems for shade | 09:21 |
mordred | so the solution is removing their deps :) | 09:21 |
yfried | mordred: we got hits by other clients, as well as oslo | 09:21 |
yfried | mordred: I'm saying that we had the same issue you did and had push the same fix | 09:21 |
mordred | dhellmann: ^^ btw, when you wake up, let's dig in to the above | 09:22 |
yfried | mordred: how can you remove clients dep from shade | 09:22 |
yfried | ? | 09:22 |
mordred | yfried: we are switching to making direct rest calls | 09:22 |
mordred | shade has already dropped glanceclient, heatclient, magnumclient, etc. morgan is almost done removing keystoneclient, I'm working on novaclient, we have someone working on cinderclient ... hoping to be done soon | 09:23 |
yfried | mordred: like tempest? really? | 09:23 |
mordred | yes. the python client libraries are awful | 09:23 |
yfried | mordred: what about cinder? | 09:23 |
mordred | all of them | 09:23 |
mordred | we will literally not use any of them soon | 09:23 |
yfried | sorry, I meant Neutron | 09:23 |
mordred | yup. that's coming soon | 09:24 |
yfried | mordred: why not join hands with tempest on this? | 09:24 |
mordred | because tempest and shade have opposite needs | 09:24 |
yfried | mordred: I'm asking about Neutron specifically, as I know it's a risky one | 09:24 |
mordred | neutron is actually the easiest one because neutronclient has almost no logic in it - it's one of the most direct passthrough libs | 09:25 |
yfried | mordred: for now, can you plz cap the clients? | 09:25 |
yfried | mordred: to the currently working version? | 09:25 |
mordred | no, I cannot. it is literally forbidden by openstack policy | 09:25 |
mordred | and there is nothing I can do about that | 09:25 |
mordred | dhellmann: ^^ | 09:26 |
yfried | mordred: ok | 09:26 |
mordred | but I can make a release that contains that requirements fix if you like | 09:26 |
yfried | mordred: would be appreciated | 09:26 |
yfried | mordred: might I suggest a periodic gate that tests for this breakage? | 09:26 |
yfried | mordred: we have one running every 1hr on our latest and stable branches | 09:27 |
yfried | mordred: sending emails in case of breakage so we can fix it before it hits users | 09:27 |
mordred | yah - I'll try to get that working today | 09:27 |
yfried | mordred: I would like to subscribe to such shade early warning system | 09:28 |
mordred | yfried: can you give me a quick example of something that causes the error you saw? I just installed latest shade from pip then imported it and it worked fine - so I want to make sure I'm testing what's breaking you | 09:29 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Remove dead ImageSnapshotCreate task https://review.openstack.org/457543 | 09:32 |
yfried | from pkg_resources import load_entry_point | 09:33 |
yfried | we have a CLI entry point | 09:36 |
yfried | it fails on load_entry_point('infrared', 'console_scripts', 'infrared')() | 09:36 |
yfried | http://paste.openstack.org/show/606985/ | 09:37 |
yfried | mordred: ^ | 09:37 |
mordred | yfried: thanks. I have made a script locally that also breaks | 09:40 |
yfried | mordred: let me review when you are ready. plz | 09:41 |
mordred | yfried: this: http://paste.openstack.org/show/606987/ shows the error -although it also shows an error on a conflict with requests too | 09:47 |
mordred | yfried: I'll work with dhellmann when he's up to figure out a better way to catch this a little more broadly (the issue is that openstacksdk released with a cap but other things don't have matching releases with the same cap) | 09:47 |
mordred | of course, the 'funny' part is that shade doesn't even use openstacksdk - so the fact that openstacksdk made a release and it broke us is just super sadmaking | 09:48 |
mordred | (and will be fixed by removing more depends - this time ironicclient, which depends on python-openstackclient which depends on openstacksdk - so maybe I should put ironicclient higher on the list to remove) | 09:49 |
openstackgerrit | Merged openstack-infra/shade master: Upgrade list volumes tests to use requests-mock https://review.openstack.org/456736 | 09:50 |
mordred | yfried: sorry it's taking so long to remove the depends and that it's causing you issues - I very much am ready to be done fixing this problem :) | 09:51 |
yfried | mordred: that's good to know | 09:54 |
yfried | mordred: do you have a storyboard I can track about this? | 09:54 |
yfried | mordred: also, I have several ppl who might like to contribute to this effort (not making any promises) | 09:55 |
yfried | mordred: in the past, tempest had the same effort so we helped a little. | 09:55 |
yfried | mordred: If you have a task list ppl can pull from I might be able to lighten your load a little | 09:56 |
mordred | yfried: awesome. we do have a storyboard for it: https://storyboard.openstack.org/#!/worklist/195 - but I need to break it out into more tasks (that list isn't super useful right now is it?) | 10:01 |
mordred | yfried: so I'll do that and then let you know about it | 10:01 |
yfried | mordred: tnx | 10:01 |
yfried | mordred: I'd like to publish this so I'll wait for you to break it down and let my ppl know they can help | 10:02 |
yfried | mordred: storyboard always fails on the first try. I have to log out and log back in :( | 10:02 |
yfried | 400: GET /api/v1/users/preferences: Invalid input for field/attribute user_id. Value: 'preferences'. unable to convert to int | 10:03 |
mordred | yfried: yah. I wish I knew why that was the case :( | 10:03 |
mordred | and awesome - thanks!!! | 10:03 |
mordred | (I'm going to work this morning on getting nova security group transitioned, because novaclient v8 doesn't have security group support anymore and I'd love to get that fix into this morning's release | 10:04 |
mordred | then I'll do storyboard breakdown - then likely try to get ironicclient done so that we can drop a bunch of depends | 10:04 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Remove dead ImageSnapshotCreate task https://review.openstack.org/457543 | 10:05 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Pass in app_name information to keystoneauth https://review.openstack.org/456268 | 10:05 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: WIP Transition nova security groups to REST https://review.openstack.org/457548 | 10:05 |
yfried | mordred: could you plz explain why tempest rest clients can't work for you? | 10:06 |
mordred | 2 reasons | 10:08 |
mordred | first - tempest clients are designed to test that openstack clouds work properly - which they should be - it's very important | 10:08 |
mordred | but shade clients need to handle it for the user when the clouds _don't_ work properly and hide it | 10:08 |
mordred | so they have opposite desires in life | 10:09 |
mordred | for instance, in tempest-lib's create_flavor is this line: | 10:09 |
mordred | self.validate_response(schema.create_get_flavor_details, resp, body) | 10:09 |
mordred | that's _awesome_ - but it's not the thing we want/need in shade - we have to be very permissive and work around people doing evil things | 10:09 |
mordred | the second is much smaller, which is one of the main things we've learned with the client libs is that a wrapper library doesn't add much value for us- making the rest calls just using keystoneauth is easier than using another library | 10:10 |
mordred | oh - also - tempestlib is not built on top of keystoneauth | 10:13 |
mordred | it does its http/auth itself - which is bad for shade/ansible - we need to be able to use all of the keystoneauth auth plugins that exist - again, I think tempest made the right choice in _not_ using that so that they can be an independent validation that things work, but for us we need to consume keystoneauth | 10:14 |
*** ioggstream has quit IRC | 11:13 | |
*** rcarrillocruz has quit IRC | 11:34 | |
*** yolanda has quit IRC | 11:44 | |
*** yolanda has joined #openstack-shade | 11:44 | |
*** ioggstream has joined #openstack-shade | 11:48 | |
*** yolanda has quit IRC | 11:49 | |
*** yolanda has joined #openstack-shade | 11:50 | |
*** yolanda_ has joined #openstack-shade | 11:56 | |
*** yolanda has quit IRC | 11:58 | |
*** yolanda_ has quit IRC | 12:19 | |
*** yolanda_ has joined #openstack-shade | 12:19 | |
*** gouthamr has joined #openstack-shade | 12:31 | |
*** yolanda_ has quit IRC | 12:31 | |
*** yolanda_ has joined #openstack-shade | 12:31 | |
*** yolanda_ has quit IRC | 12:43 | |
*** yolanda_ has joined #openstack-shade | 12:44 | |
*** gkadam has quit IRC | 12:44 | |
*** Matias has joined #openstack-shade | 12:53 | |
*** yolanda_ has quit IRC | 13:04 | |
*** yolanda_ has joined #openstack-shade | 13:05 | |
*** yolanda_ is now known as yolanda | 13:05 | |
rods | mordred I can switch from cinder to ironic if it has a higher priority | 14:11 |
mordred | rods: sure! ironic definitely is responsible for pulling in a pile of dependencies that we don't actually use | 14:13 |
mordred | rods: but honestly removing both are important, so whichever one sounds more exciting :) | 14:14 |
rods | ok, I'll stick with cinder to avoid context switching :) | 14:17 |
mordred | cool. I'll rock on the ironicclient stuff after I finish this lovely nova-network security group support (I do so much love ensuring features almost nobody is probably using work) | 14:21 |
rods | eheh :) | 14:21 |
openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Use REST for cinder list volumes https://review.openstack.org/457665 | 14:30 |
mordred | woot! | 14:31 |
rods | ok, gonna switch on same paid work and get back on this later in the evening :) | 14:32 |
rods | mordred thx for helping me to get up and running | 14:33 |
rods | *some | 14:33 |
mordred | rods: thanks for the help! | 14:34 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Transition nova security group tests to REST https://review.openstack.org/457548 | 15:18 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Remove dead ImageSnapshotCreate task https://review.openstack.org/457543 | 15:18 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Pass in app_name information to keystoneauth https://review.openstack.org/456268 | 15:18 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Replace nova security groups with REST https://review.openstack.org/457690 | 15:18 |
mordred | Shrews: if you have a sec ^^ I'd love to get those landed and get a release cut | 15:18 |
mordred | Shrews: there's a dependency issue yfried is running in to due to openstacksdk having released wiht a pbr pin but we haven't yet done so as well | 15:19 |
clarkb | mordred: you shouldn't pin pbr | 15:20 |
mordred | clarkb: I agree | 15:20 |
mordred | there was an exclusion that went out in global-requirements updates | 15:20 |
Shrews | mordred: did you fix the py35 fails in that iteration? | 15:32 |
mordred | Shrews: I did | 15:32 |
Shrews | cool. i'll look again | 15:32 |
mordred | Shrews: oh - I'm going to push up a follow up on that - I covered over an issue and it can be fixed better | 15:41 |
Shrews | mordred: addressing Jens' comment? | 15:42 |
mordred | Shrews: yup | 15:43 |
slaweq | ih | 15:48 |
Shrews | slaweq: hello. welcome | 15:56 |
slaweq | Shrews: hello | 15:56 |
slaweq | sorry for my last message, it was typed by mistake :) | 15:57 |
*** rcarrillocruz has joined #openstack-shade | 16:05 | |
*** rcarrillocruz has quit IRC | 16:07 | |
*** rcarrillocruz has joined #openstack-shade | 16:07 | |
*** yfried has quit IRC | 16:12 | |
*** rcarrillocruz has quit IRC | 16:18 | |
*** rcarrillocruz has joined #openstack-shade | 16:18 | |
mordred | Shrews: you're going to LOVE the next two patches | 16:26 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually fix the app_name protection https://review.openstack.org/457717 | 16:33 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Futureproof keystone unit tests against new occ https://review.openstack.org/457718 | 16:33 |
mordred | Shrews: please enjoy the wonder that is https://review.openstack.org/457718 | 16:33 |
mordred | Shrews: I accidentally stumbled on that while testing the app_name patch | 16:34 |
samueldmq | morning shade | 16:37 |
mordred | morning samueldmq | 16:38 |
mordred | samueldmq: enjoy https://review.openstack.org/457718 ... I think you'll "like" it | 16:38 |
samueldmq | mordred: o/ I am taking a look at restification ... | 16:38 |
* samueldmq looks | 16:38 | |
mordred | samueldmq: it's the result of http://git.openstack.org/cgit/openstack/os-client-config/commit/?id=01ff292e078206e487751228be4a7062ba0c6048 fwiw | 16:38 |
Shrews | mordred: that's special | 16:41 |
mordred | Shrews: right? | 16:41 |
mordred | Shrews: I mean, luckily _all_ of that is getting better - but yay | 16:41 |
samueldmq | mordred: I got what your patch does | 16:43 |
samueldmq | mordred: just did not what changed in the specific bits on how ksc is constructed | 16:43 |
mordred | samueldmq: we _used_ to pass "kwargs['interface'] = plugin.AUTH_INTERFACE" to ksc | 16:45 |
mordred | because there was a time sometime in the past when not doing that confused ksc | 16:45 |
mordred | (plugin.AUTH_INTERFACE being a special singleton) | 16:45 |
samueldmq | mordred: yes I see mock_session.get_endpoint.assert_called_with(interface=ksa_plugin.AUTH_INTERFACE) in the occ change | 16:46 |
mordred | but now that we actually pass the interface value like normal - it somehow changed how ksc did discovery :) | 16:46 |
mordred | samueldmq: now - _why_ that happens I have no idea :) | 16:47 |
samueldmq | wow , I have no idea either :-) | 16:47 |
samueldmq | mordred: commented on https://review.openstack.org/#/c/457718 | 16:48 |
mordred | btw - I have learned when doing restification of nova things, that running the unittests under py35 is nicer because novaclient is stricter on data in 3.5 than in 2.7 | 16:48 |
samueldmq | nice | 16:49 |
samueldmq | mordred: btw how did you figure that out ? the keystone unit tests thing ... ? | 16:49 |
samueldmq | mordred: is there a non-voting test job for shade in occ? | 16:49 |
mordred | samueldmq: here's the http trace for a test that uses that v2 codepath: http://paste.openstack.org/show/607032/ | 16:51 |
mordred | samueldmq: I think the first comment you have is fine because of the way we construct the catalog and the client - I agree with the second one :) | 16:51 |
mordred | samueldmq: and we do - but only for devstack/functional tests ... we don't have anything that runs occ-master for shade unittests (which _most_ of the time should not matter) | 16:52 |
samueldmq | mordred: OK so if it's a v2 client it will append /v2.0 to the url anyways (as per your paste) | 16:53 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually fix the app_name protection https://review.openstack.org/457717 | 16:54 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Futureproof keystone unit tests against new occ https://review.openstack.org/457718 | 16:54 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Transition nova security group tests to REST https://review.openstack.org/457548 | 16:54 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Replace nova security groups with REST https://review.openstack.org/457690 | 16:54 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Change versioned_endpoint to endpoint_uri https://review.openstack.org/457732 | 16:54 |
mordred | Shrews: fixed the python3 test failure in https://review.openstack.org/457548 - and learned that novaclient is stricter in python3 - which is awesome | 16:55 |
samueldmq | mordred: FYI you just rebased https://review.openstack.org/#/c/457718 rather than updating it :) | 16:56 |
samueldmq | mordred: what's the motivation for restification? | 16:56 |
samueldmq | so that I make sure I fully understand what's behind the effort before doing it for keystone | 16:57 |
samueldmq | as I see, we are moving from tasks (which use the clients) to rest calls | 16:57 |
samueldmq | hmmm in order to assert the APIs are backward compatible instead of the python clients??? | 16:58 |
mordred | samueldmq: yes - I fixed it in the followup change | 16:58 |
mordred | samueldmq: well - yes - we have a different worldview on api compat from the clients - so, for instance, novaclient v8 just removed support for nova-network | 16:59 |
mordred | this is a good thing, of course, it's a part of getting rid of nova-network | 16:59 |
mordred | but in shade, we still need to suppor that | 16:59 |
samueldmq | mordred: no I think you forgot to add the change, https://review.openstack.org/#/c/457718/1..2 is empty :-) | 16:59 |
mordred | samueldmq: yah - I added it in a followup patch | 16:59 |
mordred | https://review.openstack.org/#/c/457732/1 | 16:59 |
samueldmq | mordred: ++ | 17:00 |
samueldmq | mordred: so you keep supporting nova-network in shade | 17:01 |
samueldmq | via the rest calls | 17:01 |
mordred | samueldmq: the other reason is that when shade depends on python clients, it makes it harder for deployers and users to use it, because they may not be running the most recent version of openstack, but shade would depend on the most recent version of client libs | 17:01 |
mordred | so it made it hard for people to install/package | 17:01 |
mordred | samueldmq: that's right | 17:01 |
mordred | and using requests_mock we can even continue to test it, at least from a regression perspective, even when we stop having devstack that can deploy it | 17:02 |
samueldmq | mordred: makes sense, and from now on (when we migrate all to rest) they'll just need to have the serviecs running | 17:02 |
mordred | yup | 17:02 |
mordred | exactly | 17:02 |
Shrews | mordred: i'm confused as to why 457543 fails functional-libs-nv | 17:04 |
samueldmq | mordred: cool, I am taking notes, I may need to write details like this later in a thesis or something ;) | 17:04 |
*** ioggstream has quit IRC | 17:06 | |
mordred | Shrews: looks like we just go lucky and had a devstack timeout | 17:07 |
mordred | Shrews: the next patch in the series passed functional-libs-nv in its run | 17:08 |
Shrews | yeah | 17:09 |
*** rods has quit IRC | 17:15 | |
*** yfried has joined #openstack-shade | 17:30 | |
openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Move user list to REST https://review.openstack.org/457745 | 17:43 |
samueldmq | mordred: ^ I think I am doing the right thing. I also left a couple of thoughts as comments... | 17:43 |
*** yfried has quit IRC | 17:46 | |
*** ioggstream has joined #openstack-shade | 18:21 | |
*** rods has joined #openstack-shade | 18:32 | |
mordred | samueldmq: looks great - left some responses | 18:41 |
*** cdent has joined #openstack-shade | 18:58 | |
samueldmq | mordred: shouldn't it work with the unversioned endpoint ? | 19:01 |
samueldmq | mordred: hmm it's not using the ksc/ksa version discovery thing, correct ? and doing direct HTTP calls instead | 19:01 |
samueldmq | is that correct? | 19:01 |
*** ioggstream has quit IRC | 19:01 | |
mordred | I mean, it should be using ksa but then doing direct http calls | 19:02 |
mordred | it's possible it's not doing _enough_ discovery via ksa | 19:02 |
mordred | or that we need to do some different discovery | 19:02 |
samueldmq | mordred: how does that work for the auth_url now ? | 19:02 |
samueldmq | mordred: does it discover properly ? I think so in that case | 19:02 |
samueldmq | but not for keystone API (not auth) calls | 19:03 |
mordred | well, auth_url works - that comes from config (we assume you must have an auth_url to start) | 19:03 |
mordred | then we do get the catalog, which we have fixtures for in tree | 19:03 |
mordred | and I know _something_ does keystone version discovery after the catalog (since we have those calls to GET https://identity.example.com/) | 19:03 |
mordred | BUT - self._identity_client may not be doing the keystone version discovery stuff it needs to yet | 19:04 |
mordred | so that might be step one | 19:04 |
samueldmq | mordred: so that could be done by calling something in ksa/ksc? not sure | 19:04 |
samueldmq | ksa is auth, and auth_url version discovery works properly already | 19:04 |
mordred | samueldmq: yah - not sure either. I have code in shade to do it for glance | 19:04 |
samueldmq | not sure it needs to know version discovery for the APIs | 19:05 |
mordred | right. I agree - that starts to get weird | 19:05 |
samueldmq | mordred: I will do that in shade and add jamielennox to the review, he will -3 me if that's wrong | 19:05 |
samueldmq | :-) | 19:05 |
mordred | I think doing it for now in a private method in shade is fine - and probably once we get one or two more done we may find that it's a pattern we _do_ want os-client-config to know how to do | 19:05 |
mordred | "os_client_config.please_get_me_the_endpoint_from_version_discovery()" :) | 19:06 |
mordred | samueldmq: and yes he will :) | 19:06 |
samueldmq | mordred: agreed, I will see what I can do there with the URL available + self.cloud_config.get_api_version('identity') | 19:07 |
mordred | patch bomb incoming | 19:13 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually fix the app_name protection https://review.openstack.org/457717 | 19:14 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Change versioned_endpoint to endpoint_uri https://review.openstack.org/457732 | 19:14 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Futureproof keystone unit tests against new occ https://review.openstack.org/457718 | 19:14 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Transition nova security group tests to REST https://review.openstack.org/457548 | 19:14 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Replace nova security groups with REST https://review.openstack.org/457690 | 19:14 |
openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Remove extra unneeded API calls https://review.openstack.org/457775 | 19:14 |
mordred | Shrews: sorry - had to fix a broken test earlier in the stack - turns out it's really nice that we have both functional and unit tests ... keeps us honest | 19:14 |
*** ioggstream has joined #openstack-shade | 19:41 | |
morgan | mordred: damn i need to rebase my rebase now | 20:04 |
morgan | mordred: something changed and broke it again :P | 20:04 |
* morgan goes back to working on that. | 20:04 | |
mordred | morgan: sorry bout that | 20:04 |
mordred | morgan: oh, also, samueldmq just started looking at switching list_users - y'all might be able to keystone-language at each other :) | 20:05 |
morgan | mordred: nah, really i just took too long | 20:05 |
samueldmq | mordred: morgan o/ | 20:06 |
samueldmq | morgan: I just started ... https://review.openstack.org/#/c/457745/ | 20:06 |
morgan | samueldmq: cool | 20:08 |
mordred | morgan: if you have a sec, jamielennox has blessed this already: https://review.openstack.org/#/c/452550/ | 20:13 |
morgan | looking | 20:13 |
mordred | Shrews: stack ending on https://review.openstack.org/#/c/457775/1 finally passes all the tests (sorry for the false starts earlier) | 20:17 |
morgan | +2 | 20:17 |
mordred | morgan: woot! thanks! | 20:17 |
morgan | witll +A as long as no one complains in the next few das | 20:17 |
morgan | ys | 20:17 |
morgan | or feel free to self +A. | 20:17 |
morgan | but LGTM | 20:17 |
mordred | morgan: maybe I'll even browbeat Shrews into approving https://review.openstack.org/#/c/452550/ :) | 20:18 |
morgan | hehe | 20:18 |
* mordred hands Shrews a box of brows | 20:18 | |
openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: _discover_latest_version is private and not used https://review.openstack.org/457787 | 20:21 |
samueldmq | mordred: ^ are there intentions on using this function in the future ? | 20:22 |
mordred | samueldmq: I imagine it was just there while starting to think about that topic | 20:23 |
Shrews | enough review browbeating, plz | 20:23 |
* samueldmq nods | 20:24 | |
openstackgerrit | Merged openstack-infra/shade master: Pass in app_name information to keystoneauth https://review.openstack.org/456268 | 20:29 |
openstackgerrit | Merged openstack-infra/shade master: Remove dead ImageSnapshotCreate task https://review.openstack.org/457543 | 20:29 |
mordred | Shrews: that's all the browbeating I have for today | 20:29 |
mordred | samueldmq: and thank you! | 20:29 |
samueldmq | np | 20:30 |
samueldmq | mordred and Shrews the only core-reviewers in shade? | 20:31 |
samueldmq | I was trying to look up at LP but looks like shade is not using LP | 20:32 |
*** openstackgerrit has quit IRC | 20:33 | |
mordred | nope. we use the storyboard. infra are all cores on it too, as well as SpamapS - but it's been a while since SpamapS has done anything | 20:33 |
mordred | so - you know, super happy to get people in a core-able place - we're a bit strict on it since it's a super-heavily used library in infra, so brekaing it can break the entire gate - but other than that we'd love to have more folks with the codebase in their head | 20:36 |
*** yfried has joined #openstack-shade | 20:41 | |
samueldmq | mordred: nice | 20:47 |
samueldmq | mordred: when using the ksclient, where does the version come from? | 20:54 |
samueldmq | I am trying to figure out if will need to support /v2.0 URLs at all for shade (the the APIs not auth) | 20:55 |
mordred | samueldmq: yes - we need to still support the v2.0 urls | 20:55 |
mordred | samueldmq: for now you can assume that the cloud_config.get_api_version('identity') is correct | 20:56 |
samueldmq | mordred: what's the format of that ? | 20:56 |
samueldmq | mordred: v3 / v2.0 ? 3 / 2 ? | 20:56 |
mordred | 3 / 2 | 20:56 |
*** openstackgerrit has joined #openstack-shade | 20:56 | |
openstackgerrit | Merged openstack-infra/shade master: Transition nova security group tests to REST https://review.openstack.org/457548 | 20:56 |
mordred | samueldmq: sorry, I lied: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/defaults.json | 20:57 |
mordred | seems to be 2.0 or 3 | 20:57 |
samueldmq | mordred: kk: if URL is unversioned: url = url + ('/v3' if cloud_config.get_api_version('identity') == '3' else '/v2.0') | 20:58 |
samueldmq | mordred: I'm assuming it's okay to assume v2.0 if it's not 3 (2.0 or unspecified?) | 20:58 |
samueldmq | anyways, impl specific, you'll look at the review anyways | 20:59 |
mordred | it will always have a value - so it is safe to assume it has one | 20:59 |
samueldmq | nice thanks | 20:59 |
mordred | so we should never just append strings to the url - we want to consume the catalog and version discovery as much as possible | 20:59 |
mordred | samueldmq: and then, as we get better at version discovery, shade shoudl know what's the latest version it knows how to deal with, and if the user hasn't specified something different, we should try to use the latest version of the api we understand | 21:00 |
mordred | that's future though ... for right now, there will always be a "requested" version from shade's pov | 21:00 |
mordred | just mostly babbling about philosophy | 21:00 |
openstackgerrit | Merged openstack-infra/shade master: Replace nova security groups with REST https://review.openstack.org/457690 | 21:04 |
openstackgerrit | Merged openstack-infra/shade master: Actually fix the app_name protection https://review.openstack.org/457717 | 21:04 |
samueldmq | mordred: ok, but for now we can just append the identity_api_version correct ? | 21:05 |
*** slaweq has quit IRC | 21:05 | |
samueldmq | mordred: I mean, there is no /v3.6 for example, so looking at the catalog is not useful I think | 21:06 |
samueldmq | right now | 21:06 |
openstackgerrit | Merged openstack-infra/shade master: Futureproof keystone unit tests against new occ https://review.openstack.org/457718 | 21:06 |
openstackgerrit | Merged openstack-infra/shade master: Change versioned_endpoint to endpoint_uri https://review.openstack.org/457732 | 21:06 |
mordred | right - but ... so we should use whatever the catalog returns to us, no? | 21:06 |
mordred | lemme go get a few examples - it might make it easier to reason about | 21:06 |
samueldmq | sure | 21:07 |
*** cdent has quit IRC | 21:07 | |
mordred | samueldmq: ok. this is actually really good - it's super confusing so it's very clear I need to write a document | 21:14 |
samueldmq | mordred: hehe kk | 21:16 |
*** yfried has quit IRC | 21:36 | |
openstackgerrit | Merged openstack/os-client-config master: Add ability to pass in user_agent https://review.openstack.org/452550 | 21:41 |
*** gouthamr has quit IRC | 21:50 | |
openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Move user list to REST and build API URL https://review.openstack.org/457745 | 22:02 |
mordred | samueldmq, morgan: https://etherpad.openstack.org/p/shade-identity-versions | 22:06 |
mordred | samueldmq, morgan: that is half narrative and half pseudo-code | 22:06 |
morgan | hehe | 22:06 |
mordred | but I hope should explain (in bad words) what should happen - does it make sense? | 22:06 |
samueldmq | mordred: I am looking, I also left another idea in that patch | 22:06 |
mordred | samueldmq: yah - so, the version in the catalog isn't necessarily latest available- and there are both versioned and unversioned endpoints in the catalog out in the wild :) | 22:08 |
pabelanger | ohai! | 22:09 |
pabelanger | this is likely for mordred, but if anybody else knows.... | 22:09 |
mordred | pabelanger: uhoh | 22:09 |
pabelanger | nodepool seems to be aggressive with NeutronFloatingipList against tripleo-test-cloud-rh1: eg Manager tripleo-test-cloud-rh1 ran task NeutronFloatingIPList in 0.0846412181854s | 22:10 |
pabelanger | curious what I could do to maybe see what is going on | 22:10 |
*** ianw_pto is now known as ianw | 22:12 | |
mordred | pabelanger: yah - it'll totally be doing floating ip list for each server | 22:12 |
mordred | pabelanger: however, we've got a patch in trunk that should reduce that a little bit | 22:12 |
mordred | so that we'll only ask for the floating ip when we're done with the server, and not on every poll | 22:13 |
pabelanger | ah, okay | 22:13 |
pabelanger | cool, then I'll hold off looking at it until we use latest shade | 22:13 |
mordred | pabelanger: I think tl;dr - we're very close to making a release - probably first thing in the morning and it shoudl fix that problem | 22:13 |
mordred | pabelanger: ++ | 22:13 |
mordred | samueldmq: you may hate me by the end of that document ... | 22:14 |
samueldmq | mordred: :-) | 22:14 |
mordred | samueldmq: but I _think_ I got all of the edge conditions down - sorry it's not the best structured | 22:16 |
samueldmq | mordred: no problem, I've got the overall idea, need to re-read it though | 22:16 |
samueldmq | mordred: is there any relationship between client.endpoint_override and identity_api_version today? | 22:17 |
mordred | samueldmq: yah - it's ... both complicated and disorganized | 22:17 |
mordred | samueldmq: no, not this instant - at the moment client is just mounted on whatever is in the catalog | 22:17 |
mordred | well - let me rephrase | 22:17 |
mordred | for keystone rest client- there is on relationship | 22:18 |
mordred | for keystoneclient, identity_api_version gets passed to the keystoneclient constructor | 22:18 |
mordred | so there _is_ a relationship in released shade - but not in the _identity_client right now (which is one of the issues) | 22:18 |
samueldmq | mordred: since we're dropping ksc, couldn't we drop that info too ? | 22:19 |
samueldmq | and just discover what's available from catalog... | 22:19 |
samueldmq | I mean, the less we get the user to specify the better | 22:20 |
mordred | no - it's important info for the user to pass to us - sometimes clouds that people are using are broken in some way, so they need to be able to tell us "I don't care what the cloud says it can do, please do this other thing" | 22:20 |
samueldmq | if we can figure that out | 22:20 |
mordred | if we don't let the user tell us the cloud is wrong, the user can be broken with no recourse. BUT - we should also not require the user to tell us anything :) | 22:20 |
samueldmq | mordred: like even if v3 isn't in the catalog, use v3 anyways | 22:21 |
samueldmq | ? | 22:21 |
mordred | yah | 22:21 |
mordred | for instance, vexxhost right now has v2 in the catalog, but they have v3 and it works, so I can set identity_api_version to 3 to make use of it | 22:21 |
samueldmq | ok so the whole problem is the combinations between user explicitly specifying the version | 22:21 |
mordred | but if I don't set anything, I should get v2 like the catalog tells me | 22:21 |
mordred | yup | 22:22 |
samueldmq | and what comes in the URL of the catalog | 22:22 |
mordred | yup | 22:22 |
samueldmq | and does the URL in auth matter? | 22:22 |
mordred | AND - the fact that some clouds put versioned and some clouds put unversioned in the catalog :) | 22:22 |
mordred | not for identity, no | 22:22 |
mordred | the auth_url should not matter for this at all | 22:22 |
samueldmq | ok so we completely ignore auth_url | 22:22 |
samueldmq | ++ | 22:22 |
mordred | yup | 22:22 |
samueldmq | cool I think we are in the same page now | 22:22 |
mordred | excellent | 22:23 |
samueldmq | oh I just CR -1 my own patch instead of W-1 | 22:23 |
mordred | cool | 22:23 |
samueldmq | mordred: I will update a patch by its own for the discovery thing and base the user list one on it | 22:24 |
mordred | also - welcome to the fun part of shade - solving tons of conflicting and overlapping problems! :) | 22:24 |
samueldmq | so we can make sure it works | 22:24 |
mordred | samueldmq: sounds like a great plan to me | 22:24 |
clarkb | and doing it without making new problems | 22:24 |
mordred | clarkb: yup | 22:24 |
samueldmq | mordred: yay that's nice. I am looking forward to be doing the same for other projects too | 22:24 |
samueldmq | I think this is a good way to have a good understanding of openstack overall | 22:24 |
mordred | clarkb: btw - if you feel like reading that etherpad - I believe I've got everything | 22:24 |
mordred | samueldmq: ++ | 22:24 |
samueldmq | rather than just keystone. | 22:24 |
mordred | samueldmq: you will definitely wind up learning _way_ too many things | 22:25 |
samueldmq | clarkb: o/ | 22:25 |
samueldmq | mordred: nice, I am looking forward to | 22:25 |
clarkb | the shade-identity-versions one? | 22:25 |
mordred | clarkb: but it's possible you'll spot a logical flaw: https://etherpad.openstack.org/p/shade-identity-versions (it's also poorly organized) | 22:25 |
clarkb | samueldmq: ohai | 22:25 |
mordred | yah | 22:25 |
mordred | clarkb: I correct myself at the end - so don't be too bothered when it doesn't at first honor identity_api_version | 22:26 |
clarkb | do you think it should be stronger about rejecting things if the specified version is not useable? currently its a warning and tries to use latest if I read it right | 22:27 |
mordred | I think I could go either way on that... | 22:28 |
clarkb | I don't think one version of keystone is less secure than another right? | 22:29 |
clarkb | if there is a case where ^ could be true then I think you'd want to fail hard but in this case its not about http vs https or passwd vs cert, its purely api version | 22:29 |
mordred | I do not believe so, no - and also auth and CRUD are different | 22:29 |
mordred | so identity_api_version does not affect which version you auth with | 22:30 |
clarkb | wait what | 22:30 |
mordred | they're not related at all | 22:30 |
mordred | auth version is that's controlled by auth_type and/or what auth parameters you have | 22:30 |
mordred | so you can totally have auth_type: v3password and identity_api_version: 2 | 22:31 |
mordred | or vice versa | 22:31 |
mordred | and both totally work | 22:31 |
clarkb | that seems less that ideal particularly if the goal is to deprecate and remove v2 eventually | 22:31 |
mordred | (I just verified on vexxhost that both combos work fine, fwiw) | 22:31 |
clarkb | basically it seems like unnecesary footgun | 22:32 |
mordred | clarkb: yah - I agree. although from shade's pov my expectations are that we'll never be able to deprecate/remove v2 | 22:32 |
clarkb | it works today until v2 goes away then you can only auth and you get all confused | 22:32 |
clarkb | mordred: right I mean from cloud side. Say I am vexxhost user in that setup wti hshade and my config is that way. Then a year from now vexxhost upgrades and I stop working | 22:33 |
clarkb | except I clearly see keystone is working for auth and then I go drinking :) | 22:33 |
mordred | right - I do not think we should let shade break in that case | 22:33 |
mordred | which is why I lean more towards "if you explicitly ask for v2 identity crud but it's not there, give you v3 crud and a warning" | 22:34 |
mordred | since we're going to be hiding the difference between _using_ those anyway | 22:34 |
mordred | so list_users should still work in both cases | 22:34 |
mordred | but if you thought you were overriding what we'd discover, then we should at least let you know | 22:34 |
clarkb | ya | 22:35 |
mordred | clarkb, samueldmq: tomorrow I will try to write up a pure narrative version of that etherpad generalized for all of the services | 22:37 |
mordred | so it can be a document of "here's how versions work in shade" | 22:37 |
samueldmq | mordred: nice, what should be part of shade's official docs | 22:38 |
samueldmq | that* | 22:38 |
mordred | yup | 22:38 |
mordred | I will write it as a doc patch | 22:39 |
samueldmq | ++ | 22:39 |
samueldmq | mordred: how does a shade user see those warnings ? | 22:39 |
samueldmq | they're just put on stdout as any other warnings correct? | 22:39 |
mordred | samueldmq: in _discover_image_endpoint, we do this: | 22:40 |
mordred | if warning_msg: | 22:40 |
mordred | self.log.debug(warning_msg) | 22:40 |
mordred | warnings.warn(warning_msg) | 22:40 |
mordred | so we debug log it _and_ put it in to warnings.warn | 22:40 |
samueldmq | warnings.warn will go to stdout eventually correct | 22:40 |
samueldmq | I just wanted to check we clearly communicate that to the user | 22:41 |
mordred | yes. so warnings.warn is for the normal command line case - but self.log.debug is so that people running inside of service (like nodepool) can get the messages too | 22:41 |
mordred | ++ | 22:41 |
mordred | we should likely also document what log things we log to | 22:41 |
mordred | there are a few specifically named ones that are there to allow users to turn them on or off | 22:41 |
mordred | liek, "shade.request_ids" - we log request_ids to that :) | 22:42 |
samueldmq | mordred: I was reading up your convo with clarkb and I saw you discussing about the user provided version not being usable | 22:42 |
mordred | yah | 22:42 |
samueldmq | how do you determine it's not usable? user giving identity_api_version=3 might not be usable | 22:42 |
samueldmq | if v3 doesn't exist at all in the cloud | 22:42 |
mordred | right. that's exactly it | 22:42 |
samueldmq | and shade could still interpret that as "I will do what the user says as the cloud might be broken" | 22:43 |
samueldmq | does it try an API call? does it understand a 404 to define it's not usable ? | 22:43 |
mordred | if the user gives identity_api_version = 3, but it doesn't exist in the cloud via discovery, we should grab 2 and use it instead and log a warning | 22:43 |
mordred | because shade's job ultimately is to work around broken clouds :) | 22:43 |
mordred | but we try to hide the difference between versions for the most part | 22:44 |
samueldmq | mordred: what is involved in "via discovery"? | 22:44 |
samueldmq | mordred: looking up the catalog or trying an HTTP call? | 22:44 |
mordred | samueldmq: the / discovery document | 22:44 |
samueldmq | mordred: so looking up the catalog? | 22:44 |
mordred | nope, it's the root endpoint for the service | 22:44 |
mordred | if the catalog has https://example.com/v3 | 22:44 |
samueldmq | ah, the / (root) URL. ok | 22:45 |
mordred | andyou do GET https://example.com/v3 - you'll get back a {'version': { }} with some stuff | 22:45 |
mordred | yah | 22:45 |
samueldmq | and then you understand that exists | 22:45 |
mordred | yah | 22:45 |
samueldmq | if you do /v3 and gets 404 the user config is invalid | 22:45 |
mordred | well - you'd never get to getting /v3 just from the user config | 22:46 |
samueldmq | so the service catalog is completely ignored/does not relate at all to this? | 22:46 |
mordred | if you GET whatever is in the catalog, and the id is v2.0 and the user requested 3 | 22:46 |
mordred | then you know you have a versioned endpoint but it's for the wrong version | 22:46 |
mordred | so you can remove an element from the end of the URL, then try again - and you should ge ta {'versions': 'values': []} document instead | 22:47 |
mordred | then you can look for a version that matches what the user requested | 22:47 |
samueldmq | always giving priority for the version the user asked for | 22:47 |
mordred | and if you still can't find one that matches, you can warn the user and then pick the best one available | 22:47 |
mordred | yah | 22:47 |
samueldmq | does this relate at all to the service catalog stored in keystone ? | 22:48 |
mordred | it does - in that we always _first_ try the entry in teh service catalog | 22:48 |
mordred | because if the user didn't give us a version at all, we default to using what's in the catalog | 22:48 |
mordred | or also what's in the catalog might be unversioned and we have to make some choices | 22:49 |
mordred | but what's in the catalog is always step one- unless the user has _also_ configured identity_endpoint_override | 22:49 |
* samueldmq nods | 22:49 | |
mordred | in which case we should fail if that url is bad - since the user was trying to be very specific and there is no way we can guess what else he meant | 22:49 |
samueldmq | if identity_endpoint_override is invalid ? | 22:49 |
samueldmq | then we set to none and do discovery from 0 ? | 22:49 |
samueldmq | and warn? | 22:49 |
mordred | no - I think then we fail | 22:50 |
samueldmq | makes sense | 22:50 |
mordred | we shoudl always treat endpoint_override as authoritative | 22:50 |
samueldmq | yes, and that's what we set at the end of the discovery | 22:50 |
mordred | also - it's a good mechanism for power users to tell us how to skip doing any discovery calls if htey want to optimize things | 22:50 |
mordred | exactly | 22:50 |
samueldmq | if it's set by the user, let's just use it | 22:50 |
samueldmq | or error. | 22:51 |
mordred | ++ | 22:51 |
mordred | yup. that's right | 22:51 |
samueldmq | ok, I think I know enough to at least review your doc patch tomorrow :-) | 22:51 |
mordred | woot! | 22:51 |
samueldmq | I'll mull it a bit more, and don't be surprised if I come with more questions :-) | 22:51 |
mordred | hopefully that will be a review that is mostly "yes, that is exactly what I understood" | 22:51 |
samueldmq | hehe | 22:52 |
mordred | oh - yes- it's a rather complex topic | 22:52 |
mordred | although, to be fair, it's a complex topic for all of openstack's users and many do not understand it - so if we can understand it fully, then we hopefully remove the pain from them :) | 22:52 |
samueldmq | ++ | 22:53 |
samueldmq | let's make sure we write awesome docs, so they won't be afraid anymore | 22:53 |
samueldmq | know I understand that there is so much work to be done here in shade | 22:54 |
samueldmq | now* ( I'm tired) | 22:54 |
mordred | I'm tired too :) | 22:54 |
samueldmq | mordred: _identity_client.get_endpoint() already comes from the service catalog ? | 23:00 |
mordred | samueldmq: yes | 23:01 |
mordred | samueldmq: that's essentially keystoneauth.adapter.Adapter.get_endpoint | 23:01 |
samueldmq | mordred: ++ | 23:05 |
samueldmq | mordred: I wrote a pseudocode at the end of that etherpad | 23:05 |
samueldmq | mordred: I am reading up your description again to see if it matches | 23:05 |
openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Move user list to REST and build API URL https://review.openstack.org/457745 | 23:11 |
samueldmq | mordred: yes, I believe my pseudocode matches yours, we're in the same page | 23:24 |
samueldmq | what I did was to put all those questions/answers in a single code so I could process it better :-) | 23:25 |
*** gouthamr has joined #openstack-shade | 23:33 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!