18:01:29 #startmeeting keystone 18:01:29 Meeting started Tue Jul 29 18:01:29 2014 UTC and is due to finish in 60 minutes. The chair is dolphm. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:30 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:33 * morganfainberg quickly adds something 18:01:33 The meeting name has been set to 'keystone' 18:01:37 #topic Entertainment by stevemar 18:01:40 stevemar: o/ 18:01:40 dolphm: is that a song? 18:01:43 o/ 18:01:55 by popular demand... (╯°□°)╯ ┻━┻ 18:02:01 w00t! 18:02:18 #action stevemar encore! 18:02:20 oh jeez 18:02:25 ┬──┬ ノ( ゜-) 18:02:25 I'm here 18:02:29 any tables left? 18:02:41 ┬─┬ ノ( ゜-゜ノ) 18:02:50 yay! 18:02:53 #topic open discussion 18:03:03 dolphm, tokens? 18:03:04 i can juggle…but hard to show on IRC 18:03:12 morganfainberg: tokens? 18:03:19 https://review.openstack.org/#/c/109881/ -- JSON Home change for API 18:03:23 the non-persistent kind 18:03:38 everybody dance? (>'-')> <('-'<) ^(' - ')^ <('-'<) (>'-')> 18:03:42 and I rewrote the server to support it -- https://review.openstack.org/#/c/103983/ 18:03:55 i started tracking blueprints against juno-3 this morning: https://launchpad.net/keystone/+milestone/juno-3 18:04:18 dolphm, the whole uuid vs pki thing. 18:04:30 that excludes hierarchical multitenancy, which i think should go into a feature branch for juno 18:04:41 morganfainberg: ah 18:04:57 bknudson, i groaned so hard when I saw all the code you needed to re-write for JSON home 18:05:14 every router :( 18:05:20 stevemar: I should split it up... 18:05:31 bknudson, ++ if you don't mind. 18:05:33 but I needed some way for the router to have some state. 18:05:40 bknudson: yeah 18:05:43 bknudson, it looks straight forward, just a lot of code 18:05:44 bknudson, nah, it's not that, I'm just used to seeing the routers look the same 18:05:50 stevemar, I have the same feeling, seem like its scream out for a decorator or something 18:06:28 ok. I'll work on splitting it up into reviewable chunks 18:06:45 dolphm: (as an aside, I am sure my name continues to be taken in vain/ referred to as a PITA on the hierachical MT spec…since I will keep -1ing it until we get a spec that actually explains how we solve a use case) 18:06:46 bknudson, thank you! :) 18:07:15 henrynash, sounds like a good reason to be a PITA 18:07:18 henrynash, i'm supporting your -1's fwiw. we shouldn't be adding code for the sake of adding code. 18:07:21 henrynash: ++ 18:07:21 henrynash, yeah, there are known unknowns with that spec 18:07:24 henrynash: ++ 18:08:00 dolphm, i agree, this sounds like something that will need to be a feature branch for juno, possibly merged in Kilo 18:08:30 stevemar: and thanks to your federation K2K we have a good example of how to relate a proposed change to solving a use case 18:09:39 henrynash, speaking of k2k.. i really need to take a deep dive into pysaml2 and figure out how to make the darn assertion 18:09:45 morganfainberg: i need to figure out how to make a feature branch for it, get the code proposed there, and target the bp to 'next' 18:09:47 kilo reminds me of scarface 18:09:50 which will reflect the merge 18:09:56 is kilo the official name? 18:10:00 dolphm, yep 18:10:02 i think it is 18:10:03 I snort a kilo! 18:10:14 stevemar: ah, yes, it did kinda of asume we could work out how to do that :-) 18:10:19 woo! two of us independently came up with that name :D 18:10:36 is there a town in France called Kilo? 18:10:41 it had to be kilo…. 18:10:44 near to france the "kilogram" (the offical measurement of what a kilo is) is stored 18:10:57 interesting 18:10:59 the actual physical reference 18:11:05 yep 18:11:08 bknudson: there’s certainly a lump of metal that’s kind imprortant that the French have... 18:11:12 which also defines a liter of water 18:11:25 1 kilogram of pure water = 1 liter, i think 18:11:41 dolphm, interesting 18:11:43 dolphm, i think you are right 18:11:53 if i recall high school science class... 18:12:05 hmm, experiemnt: so how “un-pure: is coffee….waht happens if I drink a kilo of it.... 18:12:12 oh, and 1ml of water cubed has 1cm sides, so that's how a centimeter is defined 18:12:31 sounds like dolphm is a fan of the metric system 18:12:35 stevemar: ++ 18:12:41 ice water? 18:12:52 metric system lesson in Keystone Meeting today, hosted by dolphm 18:13:00 \o/ 18:13:13 so, i guess we have a real topic, but this is also worthy of a spec that doesn't exist yet 18:13:18 #topic PKI or UUID as the default provider 18:13:24 #link https://etherpad.openstack.org/p/pki-vs-uuid 18:13:26 L will probably be Liter. 18:13:36 this is a comparison of the state/history of PKI & UUID ^ 18:13:37 who do we break? we have to break someone 18:13:47 not as glamourous as cosmos hosted by NDT or sagan. 18:13:55 which looks like a pretty good argument to change the default back to UUID 18:13:59 there's deployments using both PKI and UUID 18:14:17 some have tried using PKI and they wound up using UUID 18:14:24 the original reason for changing the default to PKI was to catch issues with PKI earlier by exposing a wider audience to it via devstack (early in the grizzly cycle, IIRC) 18:14:37 bknudson: that's the most common scenario i've heard 18:15:03 as much as it pains me to say it, we could focus on making uuid tokens way better 18:15:06 lots of people try PKI, find problems, and give up. it should work the other way around - adventurous deployments should be experimenting with the unknown quantity 18:15:10 I just saw a bug roll through recently pertaining to PKI tokens 18:15:11 rather than forcing people to PKI 18:15:12 morganfainberg: ++ 18:15:14 I think there was talk of a nightly test run that could have PKI tokens. 18:15:29 bknudson, i'd setup a tempest run that only ran for keystone that does PKI 18:15:33 bknudson: that would work; where was that discussion? 18:15:37 bknudson, it's the *way* forward. 18:15:51 if we don’t use PKI, do we have a solution for production-at -scale? 18:16:05 I'm pretty sure there's already some kind of occasional run that posts failures to some -qa list 18:16:07 henrynash, i have some ideas on how to make uuid scale. 18:16:31 morganfainberg: would you want to add that to the etherpad? 18:16:53 bknudson, from what i gathered on infra, we can have tempest runs that are exclusive to keystone. they would test a specific scenario 18:16:57 what other problems beside size? 18:17:04 but they wouldn't be triggered by say nova patches. 18:17:11 Do you want me to explain? 18:17:15 morganfainberg: i'd like to see them, i wasn't around for the initial part of PKI tokens but my understanding was that they just ended up impractical 18:17:17 bknudson, pki vs uuid is a good approach 18:17:35 gyee: Horizon doesn’t work (well not today) 18:17:45 bknudson, erm good use-case for a specific tempest test 18:17:50 guee: although that may be relate dto size... 18:17:53 morganfainberg: we'd probably want keystone & middleware 18:17:56 henrynash, right, most browser cookies max out at 4k 18:18:06 I wouldn't expect a nova change to break PKI tokens, so that works for me. 18:18:10 bknudson, it would be worth doing for keystone, middleware, and client 18:18:12 & keystoneclient 18:18:18 bknudson, ++++ 18:18:23 Horizion was working under the assumption that the Horizon server would have no memcached 18:18:28 henrynash: horizon works afaik, but i'm sure something breaks beyond a certain size 18:18:52 dolphm, horizon doesn't store PKI token, they store the hash of it 18:18:53 They want to stick the token in a cookie 18:19:03 hardcoded to be md5 hash, BTW 18:19:06 gyee: thereby negating the advantage of PKI? 18:19:14 dolphm, exactly 18:19:19 dolphm: I think that’s probably true…it matches what we found…..and today IceHouse Horizon didn’t use ?nocatalog 18:20:06 #link http://git.openstack.org/cgit/openstack/django_openstack_auth/tree/openstack_auth/user.py#n71 18:20:21 henrynash: ?notcatalog hasn't been useful yet anyway; and that's not horizon's fault\ 18:20:47 but don't we have the same problem with SAML assertions? 18:20:51 Yes 18:20:52 do you know if there is a limitation for url token in v2 when using pki? 18:20:52 they are just as big 18:20:53 dophm: ‘cause you can’t get it the catalog when you DO want it? 18:21:15 If Horizon has memcached available it becomes moot 18:21:22 and the same is true with any service. 18:21:29 I'm fine with defaulting to UUID tokens, and when we figure out how to make PKI tokens work in general then we can change the default to PKI. 18:21:32 dolphm: (prior to you recent change) 18:21:52 In all cases there is 1) how do I get the token data up front and then 2) how do I optimized network traffic 18:21:58 I think we have a pretty good idea of what the problems are. 18:21:59 bknudson: ++ 18:22:11 UUID tokens pushed all of the burden onto online calls to keystone 18:22:44 Right now, we need to fix Horizon 18:22:46 i'm kinda impressed at the amount of edits happening to the etherpad 18:23:05 henrynash: right, there's no alternative to get a catalog until now (and even now, nothing is using that alternative) 18:23:53 I like the idea of injecting the token into memcache 18:23:53 ayoung: right now, keystone's default configuration isn't useful 18:23:53 I broken it 18:24:31 memcache? we are talking about client-side 18:24:33 dolphm, Horizon is doing what I explained to them they could do based on the requirements from 2 years ago 18:24:43 fixing DJango-Openstack-Auth is simpler 18:25:01 Keystone should not be broadcasting tokens 18:25:05 ayoung: that's just one issue on a long list of issues that need to be resolved for PKI to be a viable option for most deployments 18:25:07 please don't go down that path 18:25:09 cookie is client-side thingy, are we talking about memcache on the client-side? I am lost 18:25:23 dolphm, no, that is the primary. Beyond that, it is all about optimizing network traffic 18:25:32 the UUID vs PKI is a false dichotomy 18:25:43 ayoung: i don't care if you call it the "primary" or not - it's a significant issue on a list of significant issues 18:26:16 are there non-significant issue on the list of significant issues? :D 18:26:22 lol 18:26:27 gyee: user experience, arguably? 18:26:37 gyee: it's not breaking, but it's frustrating for a lot of people 18:26:58 Once a PKIZ token goes from one service to another, they can use a hash of the token to refer to the token. THat is an optimization. 18:27:20 It reduces network traffic even further 18:27:25 ayoung, I still lost, hash of PKIZ token is no good 18:27:37 they need the whole shebang 18:27:40 gyee, its the BP that Dolph decided to -2 18:27:47 ayoung: one of many necessary optimizations that have not landed yet. we're what, 2 years into PKI and still having problems with it? 18:28:02 UUID is the stable choice 18:28:35 stable choice != right choice ? 18:28:41 which means it's the most reasonable default choice until PKI catches up, and can with it's architectural advantages 18:28:46 but I kinda agree with dolphm, we are in half-ass land right now 18:28:53 can win with* 18:29:24 gyee, https://review.openstack.org/#/c/109295/ 18:29:25 is Horizon the only issue we need to deal with? 18:29:40 gyee, swift complained about request size 18:29:46 gyee: i think that everything that goes to apache is a problem 18:29:48 gyee, and deployers who hate that the keystone token is 2-10k request size overhead 18:29:53 there are a couple of services heading down that route 18:30:28 the size of the token increases with the size of the cloud 18:30:31 so it's not scalable 18:30:34 i'm hearing people want AWS-style auth. it's a short key. 18:30:34 ceratinly from an IBM point of view, we’d use PKI(Z) if we could get it to work reliably across the core projects….. 18:30:41 bknudson: ++ 18:30:44 bknudson: for now 18:30:53 morganfainberg, AWS is signature access 18:30:54 bknudson: that seems like the easiest issue to address to me 18:30:59 gyee, sortof 18:31:00 dolphm: y, I agree it can be fixed 18:31:11 gyee, dpeneds on the interface 18:31:12 actually PKI(Z) with “catalog on demand" 18:31:23 but if it's not going to be fixed in J then the default in J should be UUID 18:31:26 gyee: morganfainberg: we kind of already have that with oauth1, we just don't have middleware that can validate your request (oauth1_token?) 18:31:32 bknudson: ++ 18:31:55 but does catalog on demand or even id only catalogs end up with the same amount of keystone traffic as simply uuid tokens? 18:32:00 henrynash, quick calculation shows that with PKIZ and catalog on demand, we will still avg a 2k token size 18:32:14 stupid question: what does Cern use? 18:32:23 marekd|away, ^ 18:32:25 henrynash, or within 1.5-2k range for the heavier tokens (e.g federation) 18:32:28 morganfainberq: which is big, but not terrible 18:32:54 henrynash, the complaint is for each request you're doing 2k, extra... which adds up 18:33:15 do we know if the other projects are OK with PKI? or just haven't played around with it yet? 18:33:25 jamielennox: if you mean having a separate request i would guess not since you only need to get it once instead of passing it around 18:33:26 stevemar: other services? 18:33:27 stevemar, the projects are mostly agnostic 18:33:44 stevemar: swift hates them, but i think they're the only ones with a strong opinion either way 18:33:45 stevemar, middleware makes it less of an issue, swift *dislikes* pki 18:33:52 dolphm, yes services, we mention swift and horizon dislike pki 18:34:04 stevemar, horizon has a technical issue with pki 18:34:07 stevemar, not that they dislike it 18:34:19 morganfainberg: ++ horizon is fixable, afaict 18:34:20 morganfainnerg: true 18:34:26 unless we run into another issue there *shrug* 18:34:26 so why not address those 2? 18:34:44 I'm working on it 18:35:00 stevemar, how do you address swift's concern? even 2k on a 50k response is a big overhead. 18:35:09 morganfainberg, that is what my BP was for 18:35:12 stevemar: the point is that this has been a two year history of issues, and we likely have more issues down the road that we're not aware of yet 18:35:25 use PKI to deliver the token, and then use a cookie to refer to it on additioanl requiests 18:35:32 ayoung, your BP doesn't really solve that concern, it masks it sometimes sortof 18:35:39 it is really the same as UUID tokane, just no need to persist them 18:35:46 No it solve it 18:35:57 the token data needs to get to swift somehow 18:35:58 ayoung: if i'm only ever making one request to swift per token, you're not solving anything 18:36:03 with a UUID token, swift needs to fetch it 18:36:04 dolphm, ++ 18:36:18 dolphm, you are correct, and there is no net difference between PKI and UUID 18:36:25 ayoung, but that is internal to the cloud not *external* user-facing 18:36:25 ayoung: ?! 18:36:28 it is only a problem when multiple requests are made 18:36:30 ayoung, whcih is the complaint. 18:36:32 afaict 18:36:54 only 18:37:14 notmyname, ping 18:37:20 ayoung, how does it work on the client side though? you still require memcache running on the client which is not realistic 18:37:25 notmyname, you around? have a question regarding swift if you are. 18:37:45 browser javascripts making direct Keystone calls 18:38:04 gyee, justy like every other web app out there. After the PKIZ token coes across the wire, the service returns a cookie. Instead of sending the PKIZ token across the wire again, resend only to cookie 18:38:15 difference here is that the cookie is assigned by the remote service, not by Keystone 18:38:41 gyee, its really the same thing that Horizon is doing with Form Auth. 18:38:43 but you still need the real token to make Keystone calls 18:39:11 * morganfainberg looks for other swift types 18:39:15 gyee, say this is horizon to Nova. After the first call, Nova has the PKIZ token in its memcache 18:39:20 i think this is a question we should *ask* them directly 18:39:31 dolphm, who else is swift besides notmyname ? 18:39:38 creiht: ^ 18:39:39 morganfainberg, I did. THey asked me, and we had this conversation at the last summit 18:40:07 ayoung, and i want to have it clear what they're complaints are where we all see it 18:40:09 ayoung: i don't mind the idea but if we did that i would want to do it properly, have a POST route where you exchanged the PKI token for a UUID (like SAML) rather than some hybrid cookie solution 18:40:26 jamielennox, why reinvent HTTP? 18:40:33 morganfainberg: they must not use this meeting channel, none of them are here lol 18:40:40 THe rest of the world uses secure cookies for this. We should be using them, too 18:41:14 ayoung: if we are having the problem with header size, why not fix both problems 18:41:29 ayoung, in that case, dolphm is correct, cookies are essentially UUID tokens 18:41:45 since you are storing them into memcache anyway 18:41:52 gyee, with one significant difference: UUID as implemented needs to be in a Keystone database. Always 18:42:21 THe way we were heade with OPKI tokens was that even hte keystone server should be able to disappear for short times and still all of the tokens would work 18:42:29 you could also combine shared secrets with UUID, to get some of the advantages of PKI with almost none of the complexity. do some light crypto validation on the token to verify encryption with a shared secret, before bothering to call back to keystone 18:42:46 gyee, memcached local to Horizon is different from making a REST call to Keystone 18:43:08 dolphm, you would have the same size isssues 18:43:09 we should make keystone faster than memcached 18:43:26 bknudson: rewrite in C? 18:43:36 ayoung: i'm not talking about adding signature overhead 18:43:36 heh 18:43:38 dolphm, I was inthe middle of a conversation with one of our X509 guys right before this. The smallest we could get for overhead woult be about 700 bytes 18:43:38 jamielennox, i hear earlang is fast. 18:43:48 ayoung: that's waaaay bigger than what i'm thinking 18:43:56 ayoung: i'm talking < 64 chars 18:44:35 what do you mean by "light crypto validation on the token" then? 18:44:48 either way this puts a requirement on middleware to have access to a persistent (not memcached) token storage though? 18:44:58 ayoung: symmetric 18:45:16 jamielennox: memcached is still fine 18:45:20 jamielennox, that was a requirement before, thought 18:45:21 though 18:45:26 jamielennox: unless you're talking about morgan's thing? 18:45:43 jamielennox: which is just some shared kvs 18:45:44 just not explicit, but it turns out that people didn't want to go to Keystone for every uuid lookup, just the first 18:45:55 ayoung's thing, if the services start emitting tokens then they need a persistent store of them 18:46:39 which i thought you were talking about when you were talkling symetric but maybe you were refering to general UUID tokens 18:46:46 swift type folks must be busy / elsewhere it's really quiet in their channel 18:48:05 keystone would be a lot faster validating tokens if it didn't generate the catalog 18:48:16 its it fair to say that if we solved the catalog problem for the services, we are fine with PKI tokens? 18:48:22 bknudson, ++ the catalog is *really* bad performance wise 18:48:43 gyee, not entirely, i think folks don't like the setup part 18:48:45 i.e., if it was just an indexed lookup in the token table and return 200 or 404 18:48:53 honestly, i think as long as we support uuid tokens, people will tend to use it instead 18:48:59 question: I know very system dependant….but how long do people kind of see a token call toaking that has to generate a catalog? 18:49:18 gyee: you still have setup complexity and user experience burden, both of which we will never overcome 18:49:21 bknudson, i am working to get us there, a lot of the non-persistent token stuff will help make it easier. 18:49:52 bknudson, so we could do something like an IMS check on a token or some such. 18:50:02 Catalog is there only because the old validate call required the service catalog. I'm pretty sure nothing uses it 18:50:04 morganfainberg: IMS? 18:50:17 dolphm, if-modified-since, or something equally light 18:50:19 well, barbican will help us out right? 18:50:34 dolphm, "is token X valid since TIME" 18:50:44 (10 minutes) 18:50:49 ayoung: would that make it subject to removal then? 18:51:04 hrybacki, I think it means we can just yank the Catalog\ 18:51:08 but, but , key provisioning/management can be automated 18:51:12 #action dolphm to file a wishlist bug on the change in default provider 18:51:20 ^ and submit a patch 18:51:30 ayoung: i don't think that's right, nova -> glance for a user token must know the URL for glance from the validated user token 18:51:35 gyee, i don't think that will *really* convince people to use PKI. but just a hunch 18:51:41 lol ++ 18:52:03 this complex thing can be made easier with another complex thing! yay 18:52:10 with 10 mins to go: I’ll suggest a couple o f things for people to look at once we’re done here 18:52:24 quick relief approval: https://review.openstack.org/#/c/109290/ 18:52:26 henrynash: go for it, i don't think there's much left to add to the above ^ 18:52:29 #topic open discussion 18:52:45 a new guy form IBM with his first patch 18:52:49 also 18:53:13 ah cool 18:53:14 feelf ree to comment on my/adam’s spec on endpoint policy: https://review.openstack.org/#/c/99842/ 18:53:46 henrynash: ooh, if that's approved, what's the likelihood that we'll see an implementation land in juno? 18:54:03 FYI: Bugs opened against Keystone this week: http://paste.openstack.org/show/88936/ 18:54:04 henrynash: it's in the juno/ directory, but still worth an ask 18:54:10 i think gordc's patch for moving audit over is worth checking out too: https://review.openstack.org/#/c/102958/6 18:54:10 keystone implemtation for sure….how much we get into other projects is less clear 18:54:19 "landed" including middleware changes? 18:54:33 our middleware, yes 18:54:47 I’m commited to both of those 18:55:01 and horizon? :) 18:55:14 afaik, horizon is busy parsing policy files 18:55:57 dolphm, you already +2'ed this before, want to do it again: https://review.openstack.org/#/c/108839/ 18:56:13 gyee: as I said, we’ll see how much we can get into the otehrs 18:56:44 i'd love people to have a look at https://review.openstack.org/#/c/106659/ 18:56:51 i'm trying to rip out httpretty - too many problems 18:56:53 stevemar: done 18:57:03 jamielennox, who maintains request-mock? 18:57:03 jamielennox: holy wow 18:57:12 but to do it is going to be a fairly big changeover patch and it's going to get in the way of everyone 18:57:32 gyee: me, and in the process of putting it on stackforge 18:57:38 jamielennox: is that ready to merge right now? 18:57:39 no more httpretty screw ups 18:57:53 hrybacki, ^^ 18:57:55 maybe we could get request-mock into openstack? 18:58:08 bknudson: https://review.openstack.org/#/c/106659/5/test-requirements.txt ? 18:58:16 bknudson, it sounds like openstack-dev if anything. 18:58:19 bknudson: i've got it passed requirements 18:58:26 bknudson: https://github.com/openstack/requirements/blob/master/global-requirements.txt#L113 18:58:37 (2 min) 18:58:42 dolphm: i'll have to check i assume something has landed since it last passed 18:58:42 as long as it's apl 18:58:55 dolphm: but that's kind of my point, i need to get it uploaded then merged really quickly 18:59:05 jamielennox: i'd be happy to review that, but not more than once :P 18:59:09 ayoung: heh 18:59:28 jamielennox, would it be possible to convert some tests to request-mock before others? 18:59:41 jamielennox, as in... split it up so rebase hell doesn't kill you on the massive changeset? 18:59:48 jamielennox: https://review.openstack.org/#/c/106659/5/keystoneclient/tests/auth/test_identity_common.py switches from using a constant variable to a magic string... seems like a step backwards 19:00:12 e.g., httpretty.GET to 'GET' 19:00:17 time! 19:00:17 #endmeeting