17:59:56 #startmeeting keystone 17:59:57 Meeting started Tue Jan 29 17:59:56 2013 UTC. The chair is heckj. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:59:58 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:00 The meeting name has been set to 'keystone' 18:00:05 OLA! 18:00:09 hi there 18:00:13 So - what do we have on the plate for today? 18:00:18 #link http://wiki.openstack.org/Meetings/KeystoneMeeting 18:00:20 I know some topics around SQL & migrations 18:00:31 might as well do it i in order 18:00:42 #topic high priority bugs or immediate issues? 18:00:50 Anything burning to the ground? 18:01:00 Hi 18:01:17 heckj we need to come to an agreement on the authn API 18:01:24 time is running short for G3 18:01:25 gyee, its on the agenda 18:01:37 we can bump it though 18:01:46 3rd from top - happy to pull it up now 18:01:57 want to hit that first? I don't think we have any critical bugs etc... 18:02:00 What's the outstanding quandry on the authn API? 18:02:30 #link https://etherpad.openstack.org/grizzly-keystone-v3api 18:02:34 1) token ID no longer in token data 18:02:43 dolphm have some doubts on that 18:03:20 my argument is that the token data is signed. THe signature is turned into the identifier. Sticking that into the data breaks the signature 18:03:37 so, while a validate response should return the ID, it should not be in the signed data itself. 18:04:16 2) have an explicit "scope" layer to convey the scoping information, suggested by jsavak 18:04:17 ayoung: seems sensible to me 18:04:20 I am fine with that 18:04:34 a much larger issue is whether to unify the API for all attributes or not 18:04:53 3) renaming "mechanisms" to "methods", suggested by dolphm 18:04:56 I am fine with it too 18:05:06 gyee, link to 2) above? 18:05:09 gyee: you mean "scope: i.e. domain or project…or mrore than that? 18:05:30 https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md 18:05:36 henrynash, I would also think service catalog 18:05:37 henrynash, yes 18:05:58 4) service catalog formatting 18:06:18 we need an agreement on what the service catalog's going to look like 18:06:41 maybe jaypipes have some strong opinion on this one 18:06:49 gyee: was service catalog format substantially changing? Not sure where that one came in 18:07:02 gyee, on the scope thing, I think we need to clarify that there are values used to request the token, and then values used to specify the scope of access for the token. 18:07:42 heckj, from dolphm ... 18:07:46 Perhaps another change, but we've discussed indexing the service catalog by service type, region and interface, rather than providing it as a list clients have to iterate through: 18:07:46 catalog['identity']['north']['public']['url'] 18:08:07 as the userID might be from one domain, but requesting access to a different domain, and we would need to track both pieces of information. Just make it clear which is which. 18:08:28 gyee, I'm going to nix that 18:08:37 ayoung, sounds good 18:08:38 dolphm around today? don't want to run roughshod over him without having him around to defend himself 18:08:49 :) 18:08:49 that would be part of specifying the service catalog in the token request, which is a fine addition 18:08:59 but the signed data should be the set of endpoints 18:09:27 do we have any detail or writeup on what scoping in the token data looks like - representations, etc? 18:09:30 should we defer this later, till dolphm shows up? 18:09:34 I get the potential value 18:09:49 If we can get people to agree on a wider definition of "geographic entity" we will be happy to sign that, too, but it isn't going to be in V3 out the gate. 18:10:03 item 1 I think I'd like dolph here to voice his opinion 18:10:17 item 2 I'm trying to undetstand the specifics 18:10:20 item 3 I'm fine with 18:10:29 and it looks like we perhaps just nixed item 4 18:10:51 heckj, to be clear, I think item 4 can be added later without breaking backwards compat 18:10:52 k, lemme clean it up and submit another patch later today 18:10:53 are there two active proposals for what the token representation looks like? Current and something modified with scope? 18:11:02 ayoung: yep, agreed 18:11:03 heckj, not current 18:11:10 current is a mess 18:11:22 propsed is at the bottom of the etherpad I linked to above 18:11:34 line 158 ist 18:11:35 ish 18:11:40 gyee: the token structure is "multi-project/domain scoped" ready….are we all agreed we should make it so (even though we have not agreed on multi projects/domains yet) 18:11:43 ayoung: so your asserting is something new needs to be made, and there's a single proposal. Is anyone arguing with it, or is it just pending "yeah, looks OK" from the core? 18:12:02 ayoung: did you get any responses from other PTLs or project folks looking at the token defn from email yesterday? 18:12:20 henrynash, correct, design with future in mind :) 18:12:34 heckj, no responses 18:12:41 arg 18:12:49 Okay, I'll go explicit kick some folks with that data 18:12:50 heckj, but Joe Savak posted on the review 18:13:22 heckj, we need to freeze the v3 spec soon so other service have time to integrity 18:13:23 hi! : ) 18:13:24 heck, gyee: how much more complicated will the policy engine parsing need to be because we have made it multi-project/domain? 18:13:34 integrate 18:13:46 heckj, I would state that we probably need a "georgraphic location blueprint" in Hoodvana 18:14:24 gyee: totally agree - I was viewing these changes more as needed exceptions to an otherwise locked down spec, not reopening for a free-for-all 18:14:45 joesavak thanks for responding on https://review.openstack.org/#/c/20524/ and we were discussing the need for others to kick in input there, too 18:14:55 ayoung: yep, region has been ill defined far too long in terms of implication and rational use. Needs to be codified, esp. as some of the large implementations are using it actively 18:15:00 np - thanks ayoung (just got back to my desk) 18:15:34 joesavak, I taake it your response was "really close but some nits" no? 18:15:47 henrynash, I am fine with removing multi* scoping 18:15:57 just at a quick read, I have a few quesitons on the data - what do various specific keys mean, that sort of thing. 18:16:10 i think we need to make it clearer what is meant by domain_id and domain_name when authN and domain_id and domain_name when scoping 18:16:13 gyee: I'm just asking the question…you may be right that we should include it…. 18:16:26 gyee nailed it in his reply - just need to add it to the md 18:16:42 kewl 18:17:06 alrighty then 18:17:20 so we are leaving service catalog as is for now? 18:17:26 gyee, yes 18:17:31 w00t! 18:17:46 what's in the etherpad, and what's in https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md are somewhat different. Is the review the active discussion for lock down? 18:17:47 until we have geographic location blueprint. We are not ad-hocking that one 18:17:58 ayoung: ++ 18:18:07 heckj, etherpad is active discussion. 18:18:19 heckj, I'll sync up etherpad and identity-api 18:18:22 that should drive the review, and they should be in agreement 18:18:26 gyee, +1 18:18:35 Okay - sounds good, thanks gyee 18:18:39 On to SQL? 18:19:24 #topic SQL ick 18:19:38 OK, so We want to have good testing 18:19:44 and we are a data driven app 18:19:57 hencergoqed we need to do good Database testing. 18:20:09 And...might I include LDAP in there.... 18:20:31 yes, include LDAP 18:20:32 so, one of my efforts has been to clear the way to run our unit tests against a live DB, and henrynash is doing the same 18:20:38 and...might I add 18:20:50 topol is getting devstack wired up for LDAP support for the same reason 18:21:08 so...on the SQL front, we are seeing a couple issues 18:21:15 1. SQLite is drainbed 18:21:30 ppl use sqlite in production?!!! 18:21:34 it won't let us alter things we really nead sqlite 18:21:37 gyee, no 18:21:39 no 18:21:42 a thousand times no 18:21:50 so, of course someone is doing it somewhere 18:21:54 but they are wrong 18:22:08 but...the sqlupgrade scripts get tested against sqlite 18:22:18 cuz...how else are you going to test them? 18:22:40 mongo? 18:22:50 wait, that's not sql 18:22:52 carry on 18:22:52 gyee, bite your tongue...er fingers? 18:23:05 worth a side bar here - I know of two back-end implementations (private) against KV stores or DHT's as well - not just SQL 18:23:10 gyee, I suspect henrynash and comapny might add DB2 support at some point 18:23:19 and someone is going to ask about mssql and Oracle 18:23:24 ayoung: yep, already underway 18:23:26 but we'll burn those bridges when we get there 18:23:46 (not be me, however) 18:23:47 bottom line, we need a ver y low common denominator 18:24:18 so...we are going to have to hack the scripts to work with sqlite and with all the others...or do some nastiness for sql lite and different nastiness for the others 18:24:21 ayoung: presumably that's not sqlite 18:24:36 sorry - I'll wait 18:24:40 dfor now, I am going to suggest we are going to assume we can't alter tables 18:24:50 and do the copy thing, and resetup constraints, and the like 18:25:15 if that gets too nasty, we can do the table renames for postgres and the other grown ups, and do something different for sqlite 18:25:33 ayoung: so I have worked through many of the issues, like, you….and I think once I add the change you suggested), we'll have a good upgrade/downgrade that works for sqlite, mysql etc…..manages the foreign key constraints etc. 18:25:35 I've just worked through my first unified one of this for tenant_to_project 18:25:50 henrynash, very good 18:26:08 henrynash, are you doing this for the existing tests? 18:26:16 013 and below? 18:26:36 ayoung: yes….I think they all run now (assuming you fix works for closing sessions) 18:26:43 So, the second issue is the hang 18:27:00 I think it is safe to say each of the scripts need to run stand alone 18:27:11 thus, any that open a session need to do 18:27:22 session.commit() and session.close() at the end 18:27:30 ayoung: to tie up the first issue - let's add a segment in the README/HACKING or docs that explicitly states that we should assume we can't alter tables for the migration pieces to codify that for now 18:27:37 and, I think that this means that, for example, dropping tables should be done like this: 18:27:52 session.execute("drop table user_tenant_membership") 18:28:25 heckj, how about one we work out the issues, we will do a comprehensive overhall of the hacking for SQL upgrade/downgrade? There are a few best practicies there 18:28:33 but lets do the woprk to figure them all out first? 18:28:45 ayoung: agreed 18:29:01 ayoung: sounds good 18:29:16 heckj, please open a bug for that, and assign to henrynash 18:29:18 * ayoung ducks 18:29:30 ayoung: :-) 18:29:40 bug or blueprint? Happy to do either, seems a bit more like a blueprint, but whatever 18:29:50 henrynash: pref? 18:29:55 heckj, bug is sufficient 18:30:05 kk - will do 18:30:10 "HACKING does not reflect actual best pracices on upgrades." 18:30:16 ayoung, there will be no sql-injections right? :) 18:30:19 ayoung: do you think we need to work though all the other the regular (i.e. non upgrade) tests….I think we may have some issue with foreign keys not pointing to anything... 18:30:40 henrynash, quite likely 18:30:41 #action heckj to open "HACKING does not reflect actual best pracices on upgrades." bug for henrynash 18:31:01 heckj: bug is fine…and yes, assign to me 18:31:24 henrynash, the thing is, once the upgrade runs, we should have a working database, so it makes sense to use the upgrade to get the database in setUp 18:31:39 young absolutely 18:31:46 We'll keep beating on them until they run 18:32:02 ayoung: what was the second issue there? 18:32:15 ayoung: since domain_id as FKs all other the place, makes sense for me to fix it all 18:32:21 henrynash, I think those were the bug issues with SQL. topol might find comparable thing with the LDAP tests once we get there, but that is a tale for a different day 18:32:32 heckj, the hang 18:32:40 ah - kk 18:32:45 heckj, I was seeing the hang happen on table renames 18:32:52 and I think that is also a session commit issue 18:33:10 ayoung: saw that too 18:33:28 thus, I think the short of it is that all of the migrate work is best done using the session commands directly unless we can trace down that the sqlalchmey later commits 18:33:42 sounds like that's reasonable wrapped up then 18:33:51 yeah, everything but the coding 18:33:57 not ideal, but yeah 18:33:57 which leads me into 18:34:10 o_O 18:34:14 Project to Tenant change over 18:34:18 #topic Project to Tenant change over 18:34:27 hopefully that other way round... 18:34:34 heh 18:34:36 yeah 18:34:43 make that tenant to project 18:34:49 heh 18:34:57 summary of issues? 18:35:07 Or jsut a shit ton of work to do related to that? 18:35:07 OK, so I have been trying to do this in reasonable sized patches 18:35:16 but it is kindof arbitraty where I split them 18:35:28 gyee acked the first, and it is merging now 18:35:33 ayoung: fwiw - I didn't think there'd be any way to do this easily 18:35:37 https://review.openstack.org/#/c/20516/ 18:35:47 the second is also just internal code changes 18:36:00 https://review.openstack.org/#/c/20524/ 18:36:14 I'll need to address those issues, but they are not insurmountable 18:36:31 er https://review.openstack.org/#/c/20534/ 18:36:51 20524 is not relevant to this discussion, sorry 18:36:57 #link https://review.openstack.org/#/c/20534/ 18:37:14 #action all - review please: https://review.openstack.org/#/c/20534/ 18:37:15 Brent has done a decent first pass, and flushed out a few issues 18:37:35 to complete the task, however is going to require something like this 18:38:50 * ayoung having trouble with gist...1 sec 18:39:27 http://www.fpaste.org/wXdI/ 18:39:34 Don't link that as it is ephemeral 18:39:40 kk 18:40:00 But bascially converting tenant to project on the way in, and project->tenant on the way out 18:40:35 Yeah, something equiv for the whole docs tree will probably be useful 18:40:42 (sorry, docs project) 18:40:50 I can't post as a WIP yet, as gerrit will convert my other patches to WIP 18:40:57 ayoung, I wonder if its easier to do the conversion at the middleware and leave the internal along 18:41:06 alone 18:41:17 clients doesn't care about the internals 18:41:23 gyee: maybe short term, but that's just inviting more confusion as we go down the road 18:41:32 gyee, that is sort of the idea 18:41:40 but we'll do the conversion on the controllers 18:41:47 it should only be for V2 calls 18:42:05 I'm good with that 18:42:15 (for V2 translation calls) 18:42:16 Yeah, V3 should be project only 18:42:40 yeah 18:42:58 we can phase out the internal data structure over time 18:43:04 I'm going to get it working, and then I'll see if I can split the patch up into a tenant-project portion and a membership->roles portion 18:43:11 word 18:43:31 so likely to be 2 more patches with significant code impact, befor I can resubmit trusts 18:43:41 needless to say, this is what I am working on for the rest of the release 18:43:45 I'll keep an eye for reviewing to keep it moving along 18:43:46 that and code reviews, of course 18:43:55 ayoung: there is other filtering needed too (for instance I filter out domain_id from entities depending on whether it is a v2 or v3 call….its in my WIP) 18:44:15 trusts is mostly done except for delete... 18:44:44 henrynash, I'll post my WIP to git hub. 1 sec 18:45:42 #link https://github.com/admiyo/keystone/tree/replace-tenant-user-membership 18:46:19 I'm not going to be looking for comments there, but the code is available 18:46:26 ayoung: nice, thanks 18:46:46 ayoung: +1 18:47:05 https://github.com/admiyo/keystone/commit/39fa4839feac4b720295e857109f63962b7343ac#L6L1 shows the sql hack I am currently doing 18:47:26 mind if we switch over to talking a bit about current blueprints and status? 18:47:33 (want to get updates prepped for the release mtg today) 18:47:34 heckj, the conch is yours 18:48:13 I'm seeing the following blueprints all at significant risk of not getting completed in grizzly-3 given work remaining: 18:48:14 delegation of keystone servers. Not started. Deferred to H 18:48:19 Token trusts 18:48:25 Role assignment to a domain should be more flexible 18:48:28 token trusts should make it 18:48:30 Pluggable Identity Authentication Handlers 18:48:40 working on it 18:48:41 Multi-factor-authentication 18:48:41 Additional metadata for endpoint templating 18:48:51 MFA is a maybe 18:48:55 but not sure 18:48:58 Multi factor is built on the Auth API 18:49:01 I know these are all getting active work - the question is will they really get done in the next 3-4 weeks? 18:49:05 right 18:49:28 I can wired up google authenticator 18:49:31 #link https://blueprints.launchpad.net/keystone 18:49:32 if I have time 18:50:17 the authn proposal should lay the plumbing for MFA 18:50:33 so I'm thinking defer MFA to "H" 18:50:48 based on gyee finishing up plugg'd authn in the next couple of weeks 18:50:50 heckj, sort of 18:50:55 k 18:51:03 heckj, MFA was always "we will lay the groundwork" 18:51:16 so I think full impl will require comminuty contrib anyway 18:51:18 heckj: role assignments to a domain is done really, is part of domain scoping 18:51:31 henrynash: yeah, sorry - saw that after the fact 18:51:34 the groundwork for MFA should be in place with the V3 auth api, though 18:52:09 ayoung, yes, the current authn API should cover that 18:52:11 so for the next 3-4 meetings, let's have a progress update on all open blueprints to make sure we keep track and can adjust release appropriately. That cool with y'all? 18:52:20 domain scoping: almost all complete, expect for linking to v3 auth 18:52:24 it has support for both plugin and multi-roundtrip auths 18:52:44 domain name spaces: fine once we get domain scoping & v3 auth 18:52:46 heckj, sounds good 18:52:50 adding-idps-to-service-catalog is finished coding 18:52:54 heckj: +1 18:53:44 cool - updating the BPs - if folks run into things that assert otherwise, please holler 18:53:52 trusts will be ready to go. It gates on project->tenant, but that will fall 18:54:53 gyee: plgguable authn is reporting on blocked - doesn't sound accurate 18:55:04 heckj, no, that is right 18:55:15 blocked on V3 auth api... 18:55:28 right 18:55:48 https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md 18:55:59 once we get a +4 on it, it will be unblocked 18:56:02 kk 18:56:23 I'll pushing another patch today 18:56:32 hopefully we can freeze that one 18:56:40 Okay - BP's updated, thanks guys 18:57:00 #topic open discussion 18:57:07 for all 3 minutes left 18:57:23 is jaypipes around? 18:57:40 I would like to get his opinion on the endpoint versioning 18:57:41 gyee: yes, sorry, been on phone.. trying to read back. 18:57:46 is devstack keystone v3 capable yet? 18:58:03 and what the service catalog should look like 18:58:23 henrynash: not as yet. authn sort of critical in that flow.. dtroyer has been helping move that forward tho 18:58:37 heckj: ok, thx 18:59:41 jaypipes, https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md 18:59:45 line 1137 19:00:12 yes? 19:00:17 dolphm's comment 19:00:31 can you chime in on that? 19:00:54 sure, one sec, offf phone now for a second and reading.. 19:01:08 gotta wrap this meeting - continue on #dev? 19:01:10 current plan is to go with what we have 19:01:12 #endmeeting keystone