17:59:56 <heckj> #startmeeting keystone
17:59:57 <openstack> Meeting started Tue Jan 29 17:59:56 2013 UTC.  The chair is heckj. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:00 <openstack> The meeting name has been set to 'keystone'
18:00:05 <heckj> OLA!
18:00:09 <henrynash> hi there
18:00:13 <heckj> So - what do we have on the plate for today?
18:00:18 <ayoung> #link http://wiki.openstack.org/Meetings/KeystoneMeeting
18:00:20 <heckj> I know some topics around SQL & migrations
18:00:31 <heckj> might as well do it i in order
18:00:42 <heckj> #topic high priority bugs or immediate issues?
18:00:50 <heckj> Anything burning to the ground?
18:01:00 <topol> Hi
18:01:17 <gyee> heckj we need to come to an agreement on the authn API
18:01:24 <gyee> time is running short for G3
18:01:25 <ayoung> gyee, its on the agenda
18:01:37 <ayoung> we can bump it though
18:01:46 <heckj> 3rd from top - happy to pull it up now
18:01:57 <ayoung> want to hit that first?  I don't think we have any critical bugs etc...
18:02:00 <heckj> What's the outstanding quandry on the authn API?
18:02:30 <ayoung> #link https://etherpad.openstack.org/grizzly-keystone-v3api
18:02:34 <gyee> 1) token ID no longer in token data
18:02:43 <gyee> dolphm have some doubts on that
18:03:20 <ayoung> my argument is that the token data is signed.  THe signature is turned into the identifier.  Sticking that into the data breaks the signature
18:03:37 <ayoung> so, while a validate response should return the ID,  it should not be in the signed data itself.
18:04:16 <gyee> 2) have an explicit "scope" layer to convey the scoping information, suggested by jsavak
18:04:17 <henrynash> ayoung: seems sensible to me
18:04:20 <gyee> I am fine with that
18:04:34 <dwchadwick> a much larger issue is whether to unify the API for all attributes or not
18:04:53 <gyee> 3) renaming "mechanisms" to "methods", suggested by dolphm
18:04:56 <gyee> I am fine with it too
18:05:06 <ayoung> gyee, link to 2) above?
18:05:09 <henrynash> gyee: you mean "scope: i.e. domain or project…or mrore than that?
18:05:30 <gyee> https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md
18:05:36 <ayoung> henrynash, I would also think service catalog
18:05:37 <gyee> henrynash, yes
18:05:58 <gyee> 4) service catalog formatting
18:06:18 <gyee> we need an agreement on what the service catalog's going to look like
18:06:41 <gyee> maybe jaypipes have some strong opinion on this one
18:06:49 <heckj> gyee: was service catalog format substantially changing? Not sure where that one came in
18:07:02 <ayoung> gyee, on the scope thing, I think we need to clarify that there are values used to request the token, and then values used to specify the scope of access for the token.
18:07:42 <gyee> heckj, from dolphm ...
18:07:46 <gyee> Perhaps another change, but we've discussed indexing the service catalog by service type, region and interface, rather than providing it as a list clients have to iterate through:
18:07:46 <gyee> catalog['identity']['north']['public']['url']
18:08:07 <ayoung> as the userID might be from one domain, but requesting access to a different domain, and we would need to track both pieces of information.  Just make it clear which is which.
18:08:28 <ayoung> gyee, I'm going to nix that
18:08:37 <gyee> ayoung, sounds good
18:08:38 <heckj> dolphm around today? don't want to run roughshod over him without having him around to defend himself
18:08:49 <gyee> :)
18:08:49 <ayoung> that would be part of specifying the service catalog in the token request, which is a fine addition
18:08:59 <ayoung> but the signed data should be the set of endpoints
18:09:27 <heckj> do we have any detail or writeup on what scoping in the token data looks like - representations, etc?
18:09:30 <gyee> should we defer this later, till dolphm shows up?
18:09:34 <heckj> I get the potential value
18:09:49 <ayoung> If we can get people to agree on a wider definition of "geographic entity" we will be happy to sign that, too, but it isn't going to be in V3 out the gate.
18:10:03 <heckj> item 1 I think I'd like dolph here to voice his opinion
18:10:17 <heckj> item 2 I'm trying to undetstand the specifics
18:10:20 <heckj> item 3 I'm fine with
18:10:29 <heckj> and it looks like we perhaps just nixed item 4
18:10:51 <ayoung> heckj, to be clear, I think item 4 can be added later without breaking backwards compat
18:10:52 <gyee> k, lemme clean it up and submit another patch later today
18:10:53 <heckj> are there two active proposals for what the token representation looks like? Current and something modified with scope?
18:11:02 <heckj> ayoung: yep, agreed
18:11:03 <ayoung> heckj, not current
18:11:10 <ayoung> current is a mess
18:11:22 <ayoung> propsed is at the bottom of the etherpad I linked to above
18:11:34 <ayoung> line 158 ist
18:11:35 <ayoung> ish
18:11:40 <henrynash> gyee: the token structure is "multi-project/domain scoped" ready….are we all agreed we should make it so (even though we have not agreed on multi projects/domains yet)
18:11:43 <heckj> ayoung: so your asserting is something new needs to be made, and there's a single proposal. Is anyone arguing with it, or is it just pending "yeah, looks OK" from the core?
18:12:02 <heckj> ayoung: did you get any responses from other PTLs or project folks looking at the token defn from email yesterday?
18:12:20 <gyee> henrynash, correct, design with future in mind :)
18:12:34 <ayoung> heckj, no responses
18:12:41 <heckj> arg
18:12:49 <heckj> Okay, I'll go explicit kick some folks with that data
18:12:50 <ayoung> heckj, but Joe Savak posted on the review
18:13:22 <gyee> heckj, we need to freeze the v3 spec soon so other service have time to integrity
18:13:23 <joesavak> hi! : )
18:13:24 <henrynash> heck, gyee: how much more complicated will the policy engine parsing need to be because we have made it multi-project/domain?
18:13:34 <gyee> integrate
18:13:46 <ayoung> heckj, I would state that we probably need a "georgraphic location blueprint" in Hoodvana
18:14:24 <heckj> gyee: totally agree - I was viewing these changes more as needed exceptions to an otherwise locked down spec, not reopening for a free-for-all
18:14:45 <ayoung> joesavak thanks for responding on  https://review.openstack.org/#/c/20524/  and we were discussing the need for others to kick in input there, too
18:14:55 <heckj> ayoung: yep, region has been ill defined far too long in terms of implication and rational use. Needs to be codified, esp. as some of the large implementations are using it actively
18:15:00 <joesavak> np - thanks ayoung (just got back to my desk)
18:15:34 <ayoung> joesavak, I taake it your response was  "really close but some nits" no?
18:15:47 <gyee> henrynash, I am fine with removing multi* scoping
18:15:57 <heckj> just at a quick read, I have a few quesitons on the data - what do various specific keys mean, that sort of thing.
18:16:10 <joesavak> i think we need to make it clearer what is meant by domain_id and domain_name when authN and domain_id and domain_name when scoping
18:16:13 <henrynash> gyee: I'm just asking the question…you may be right that we should include it….
18:16:26 <joesavak> gyee nailed it in his reply - just need to add it to the md
18:16:42 <ayoung> kewl
18:17:06 <gyee> alrighty then
18:17:20 <gyee> so we are leaving service catalog as is for now?
18:17:26 <ayoung> gyee, yes
18:17:31 <gyee> w00t!
18:17:46 <heckj> what's in the etherpad, and what's in https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md are somewhat different. Is the review the active discussion for lock down?
18:17:47 <ayoung> until we have geographic location blueprint.  We are not ad-hocking that one
18:17:58 <heckj> ayoung: ++
18:18:07 <ayoung> heckj, etherpad is active discussion.
18:18:19 <gyee> heckj, I'll sync up etherpad and identity-api
18:18:22 <ayoung> that should drive the review, and they should be in agreement
18:18:26 <ayoung> gyee, +1
18:18:35 <heckj> Okay - sounds good, thanks gyee
18:18:39 <ayoung> On to SQL?
18:19:24 <heckj> #topic SQL ick
18:19:38 <ayoung> OK,  so We want to have good testing
18:19:44 <ayoung> and we are a data driven app
18:19:57 <ayoung> hencergoqed we need to do good Database testing.
18:20:09 <ayoung> And...might I include LDAP in there....
18:20:31 <topol> yes, include LDAP
18:20:32 <ayoung> so, one of my efforts has been to clear the way to run our unit tests against a live DB, and henrynash is doing the same
18:20:38 <ayoung> and...might I add
18:20:50 <ayoung> topol is getting devstack wired up for LDAP support for the same reason
18:21:08 <ayoung> so...on the SQL front,  we are seeing a couple issues
18:21:15 <ayoung> 1. SQLite is drainbed
18:21:30 <gyee> ppl use sqlite in production?!!!
18:21:34 <ayoung> it won't let us alter things we really nead sqlite
18:21:37 <ayoung> gyee, no
18:21:39 <ayoung> no
18:21:42 <ayoung> a thousand times no
18:21:50 <ayoung> so, of course someone is doing it somewhere
18:21:54 <ayoung> but they are wrong
18:22:08 <ayoung> but...the sqlupgrade scripts get tested against sqlite
18:22:18 <ayoung> cuz...how else are you going to test them?
18:22:40 <gyee> mongo?
18:22:50 <gyee> wait, that's not sql
18:22:52 <gyee> carry on
18:22:52 <ayoung> gyee, bite your tongue...er fingers?
18:23:05 <heckj> worth a side bar here - I know of two back-end implementations (private) against KV stores or DHT's as well - not just SQL
18:23:10 <ayoung> gyee, I suspect henrynash and comapny might add DB2 support at some point
18:23:19 <ayoung> and someone is going to ask about mssql and Oracle
18:23:24 <henrynash> ayoung: yep, already underway
18:23:26 <ayoung> but we'll burn those bridges when we get there
18:23:46 <henrynash> (not be me, however)
18:23:47 <ayoung> bottom line, we need a ver y low common denominator
18:24:18 <ayoung> so...we are going to have to hack the scripts to work with sqlite and with all the others...or do some nastiness for sql lite and different nastiness for the others
18:24:21 <heckj> ayoung: presumably that's not sqlite
18:24:36 <heckj> sorry - I'll wait
18:24:40 <ayoung> dfor now, I am going to suggest we are going to assume we can't alter tables
18:24:50 <ayoung> and do the copy thing, and resetup constraints, and the like
18:25:15 <ayoung> if that gets too nasty, we can do the table renames for postgres and the other grown ups, and do something different for sqlite
18:25:33 <henrynash> ayoung: so I have worked through many of the issues, like, you….and I think once I add the change you suggested), we'll have a good upgrade/downgrade that works for sqlite, mysql etc…..manages the foreign key constraints etc.
18:25:35 <ayoung> I've just worked through my first unified one of this for tenant_to_project
18:25:50 <ayoung> henrynash, very good
18:26:08 <ayoung> henrynash, are you doing this for the existing tests?
18:26:16 <ayoung> 013 and below?
18:26:36 <henrynash> ayoung: yes….I think they all run now (assuming you fix works for closing sessions)
18:26:43 <ayoung> So, the second issue is the hang
18:27:00 <ayoung> I think it is safe to say each of the scripts need to run stand alone
18:27:11 <ayoung> thus, any that open a session need to do
18:27:22 <ayoung> session.commit()  and session.close() at the end
18:27:30 <heckj> ayoung: to tie up the first issue - let's add a segment in the README/HACKING or docs that explicitly states that we should assume we can't alter tables for the migration pieces to codify that for now
18:27:37 <ayoung> and, I think that this means that, for example, dropping tables should be done like this:
18:27:52 <ayoung> session.execute("drop table user_tenant_membership")
18:28:25 <ayoung> heckj, how about one we work out the issues, we will do a comprehensive overhall of the hacking for SQL upgrade/downgrade?  There are a few best practicies there
18:28:33 <ayoung> but lets do the woprk to figure them all out first?
18:28:45 <henrynash> ayoung: agreed
18:29:01 <heckj> ayoung: sounds good
18:29:16 <ayoung> heckj, please open a bug for that, and assign to henrynash
18:29:18 * ayoung ducks
18:29:30 <henrynash> ayoung: :-)
18:29:40 <heckj> bug or blueprint? Happy to do either, seems a bit more like a blueprint, but whatever
18:29:50 <heckj> henrynash: pref?
18:29:55 <ayoung> heckj, bug is sufficient
18:30:05 <heckj> kk - will do
18:30:10 <ayoung> "HACKING does not reflect actual best pracices on upgrades."
18:30:16 <gyee> ayoung, there will be no sql-injections right? :)
18:30:19 <henrynash> ayoung: do you think we need to work though all the other the regular (i.e. non upgrade) tests….I think we may have some issue with foreign keys not pointing to anything...
18:30:40 <ayoung> henrynash, quite likely
18:30:41 <heckj> #action heckj to open "HACKING does not reflect actual best pracices on upgrades." bug for henrynash
18:31:01 <henrynash> heckj: bug is fine…and yes, assign to me
18:31:24 <ayoung> henrynash, the thing is, once the upgrade runs, we should have a working database, so it makes sense to use the upgrade to get the database in setUp
18:31:39 <henrynash> young absolutely
18:31:46 <ayoung> We'll keep beating on them until they run
18:32:02 <heckj> ayoung: what was the second issue there?
18:32:15 <henrynash> ayoung: since domain_id as FKs all other the place, makes sense for me to fix it all
18:32:21 <ayoung> henrynash, I think those were the bug issues with SQL.  topol might find comparable thing with the LDAP tests once we get there, but that is a tale for a different day
18:32:32 <ayoung> heckj, the hang
18:32:40 <heckj> ah - kk
18:32:45 <ayoung> heckj, I was seeing the hang happen on table renames
18:32:52 <ayoung> and I think that is also a session commit issue
18:33:10 <henrynash> ayoung: saw that too
18:33:28 <ayoung> thus, I think the short of it is that all of the migrate work is best done using the session commands directly unless we can trace down that the sqlalchmey later commits
18:33:42 <heckj> sounds like that's reasonable wrapped up then
18:33:51 <ayoung> yeah, everything but the coding
18:33:57 <heckj> not ideal, but yeah
18:33:57 <ayoung> which leads me into
18:34:10 <heckj> o_O
18:34:14 <ayoung> Project to Tenant change over
18:34:18 <heckj> #topic Project to Tenant change over
18:34:27 <henrynash> hopefully that other way round...
18:34:34 <ayoung> heh
18:34:36 <ayoung> yeah
18:34:43 <ayoung> make that tenant to project
18:34:49 <heckj> heh
18:34:57 <heckj> summary of issues?
18:35:07 <heckj> Or jsut a shit ton of work to do related to that?
18:35:07 <ayoung> OK, so I have been trying to do this in reasonable sized patches
18:35:16 <ayoung> but it is kindof arbitraty where I split them
18:35:28 <ayoung> gyee acked the first, and it is merging now
18:35:33 <heckj> ayoung: fwiw - I didn't think there'd be any way to do this easily
18:35:37 <ayoung> https://review.openstack.org/#/c/20516/
18:35:47 <ayoung> the second is also just internal code changes
18:36:00 <ayoung> https://review.openstack.org/#/c/20524/
18:36:14 <ayoung> I'll need to address those issues, but they are not insurmountable
18:36:31 <ayoung> er https://review.openstack.org/#/c/20534/
18:36:51 <ayoung> 20524 is not relevant to this discussion, sorry
18:36:57 <ayoung> #link  https://review.openstack.org/#/c/20534/
18:37:14 <heckj> #action all - review please: https://review.openstack.org/#/c/20534/
18:37:15 <ayoung> Brent has done a decent first pass, and flushed out a few issues
18:37:35 <ayoung> to complete the task, however is going to require something like this
18:38:50 * ayoung having trouble with gist...1 sec
18:39:27 <ayoung> http://www.fpaste.org/wXdI/
18:39:34 <ayoung> Don't link that as it is ephemeral
18:39:40 <heckj> kk
18:40:00 <ayoung> But bascially converting tenant to project on the way in, and project->tenant  on the way out
18:40:35 <heckj> Yeah, something equiv for the whole docs tree will probably be useful
18:40:42 <heckj> (sorry, docs project)
18:40:50 <ayoung> I can't post as a WIP yet, as gerrit will convert my other patches to WIP
18:40:57 <gyee> ayoung, I wonder if its easier to do the conversion at the middleware and leave the internal along
18:41:06 <gyee> alone
18:41:17 <gyee> clients doesn't care about the internals
18:41:23 <heckj> gyee: maybe short term, but that's just inviting more confusion as we go down the road
18:41:32 <ayoung> gyee, that is sort of the idea
18:41:40 <ayoung> but we'll do the conversion on the controllers
18:41:47 <ayoung> it should only be for V2 calls
18:42:05 <heckj> I'm good with that
18:42:15 <heckj> (for V2 translation calls)
18:42:16 <ayoung> Yeah, V3 should be project only
18:42:40 <gyee> yeah
18:42:58 <gyee> we can phase out the internal data structure over time
18:43:04 <ayoung> I'm going to get it working, and then I'll see if I can split the patch up into a tenant-project portion and a membership->roles portion
18:43:11 <heckj> word
18:43:31 <ayoung> so likely to be 2 more patches with significant code impact, befor I can resubmit trusts
18:43:41 <ayoung> needless to say, this is what I am working on for the rest of the release
18:43:45 <heckj> I'll keep an eye for reviewing to keep it moving along
18:43:46 <ayoung> that and code reviews, of course
18:43:55 <henrynash> ayoung: there is other filtering needed too (for instance I filter out domain_id from entities depending on whether it is a v2 or v3 call….its in my WIP)
18:44:15 <ayoung> trusts is mostly done except for delete...
18:44:44 <ayoung> henrynash, I'll post my WIP to git hub.  1 sec
18:45:42 <ayoung> #link https://github.com/admiyo/keystone/tree/replace-tenant-user-membership
18:46:19 <ayoung> I'm not going to be looking for comments there, but the code is available
18:46:26 <heckj> ayoung: nice, thanks
18:46:46 <henrynash> ayoung: +1
18:47:05 <ayoung> https://github.com/admiyo/keystone/commit/39fa4839feac4b720295e857109f63962b7343ac#L6L1  shows the sql hack I am currently doing
18:47:26 <heckj> mind if we switch over to talking a bit about current blueprints and status?
18:47:33 <heckj> (want to get updates prepped for the release mtg today)
18:47:34 <ayoung> heckj, the conch is yours
18:48:13 <heckj> I'm seeing the following blueprints all at significant risk of not getting completed in grizzly-3 given work remaining:
18:48:14 <ayoung> delegation of keystone servers.   Not started.  Deferred to H
18:48:19 <heckj> Token trusts
18:48:25 <heckj> Role assignment to a domain should be more flexible
18:48:28 <ayoung> token trusts should make it
18:48:30 <heckj> Pluggable Identity Authentication Handlers
18:48:40 <gyee> working on it
18:48:41 <heckj> Multi-factor-authentication
18:48:41 <heckj> Additional metadata for endpoint templating
18:48:51 <gyee> MFA is a maybe
18:48:55 <gyee> but not sure
18:48:58 <ayoung> Multi factor is built on the Auth API
18:49:01 <heckj> I know these are all getting active work - the question is will they really get done in the next 3-4 weeks?
18:49:05 <gyee> right
18:49:28 <gyee> I can wired up google authenticator
18:49:31 <ayoung> #link https://blueprints.launchpad.net/keystone
18:49:32 <gyee> if I have time
18:50:17 <gyee> the authn proposal should lay the plumbing for MFA
18:50:33 <heckj> so I'm thinking defer MFA to "H"
18:50:48 <heckj> based on gyee finishing up plugg'd authn in the next couple of weeks
18:50:50 <ayoung> heckj, sort of
18:50:55 <gyee> k
18:51:03 <ayoung> heckj, MFA was always "we will lay the groundwork"
18:51:16 <ayoung> so I think full impl will require comminuty contrib anyway
18:51:18 <henrynash> heckj: role assignments to a domain is done really, is part of domain scoping
18:51:31 <heckj> henrynash: yeah, sorry - saw that after the fact
18:51:34 <ayoung> the groundwork for MFA should be in place with the V3 auth api, though
18:52:09 <gyee> ayoung, yes, the current authn API should cover that
18:52:11 <heckj> so for the next 3-4 meetings, let's have a progress update on all open blueprints to make sure we keep track and can adjust release appropriately. That cool with y'all?
18:52:20 <henrynash> domain scoping: almost all complete, expect for linking to v3 auth
18:52:24 <gyee> it has support for both plugin and multi-roundtrip auths
18:52:44 <henrynash> domain name spaces: fine once we get domain scoping & v3 auth
18:52:46 <gyee> heckj, sounds good
18:52:50 <dwchadwick> adding-idps-to-service-catalog is finished coding
18:52:54 <henrynash> heckj: +1
18:53:44 <heckj> cool - updating the BPs - if folks run into things that assert otherwise, please holler
18:53:52 <ayoung> trusts will be ready to go.  It gates on project->tenant, but that will fall
18:54:53 <heckj> gyee: plgguable authn is reporting on blocked - doesn't sound accurate
18:55:04 <ayoung> heckj, no, that is right
18:55:15 <ayoung> blocked on V3 auth api...
18:55:28 <gyee> right
18:55:48 <gyee> https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md
18:55:59 <gyee> once we get a +4 on it, it will be unblocked
18:56:02 <heckj> kk
18:56:23 <gyee> I'll pushing another patch today
18:56:32 <gyee> hopefully we can freeze that one
18:56:40 <heckj> Okay - BP's updated, thanks guys
18:57:00 <heckj> #topic open discussion
18:57:07 <heckj> for all 3 minutes left
18:57:23 <gyee> is jaypipes around?
18:57:40 <gyee> I would like to get his opinion on the endpoint versioning
18:57:41 <jaypipes> gyee: yes, sorry, been on phone.. trying to read back.
18:57:46 <henrynash> is devstack keystone v3 capable yet?
18:58:03 <gyee> and what the service catalog should look like
18:58:23 <heckj> henrynash: not as yet. authn sort of critical in that flow.. dtroyer has been helping move that forward tho
18:58:37 <henrynash> heckj: ok, thx
18:59:41 <gyee> jaypipes, https://review.openstack.org/#/c/20524/3/openstack-identity-api/src/markdown/identity-api-v3.md
18:59:45 <gyee> line 1137
19:00:12 <jaypipes> yes?
19:00:17 <gyee> dolphm's comment
19:00:31 <gyee> can you chime in on that?
19:00:54 <jaypipes> sure, one sec, offf phone now for a second and reading..
19:01:08 <heckj> gotta wrap this meeting - continue on #dev?
19:01:10 <gyee> current plan is to go with what we have
19:01:12 <heckj> #endmeeting keystone