16:00:11 <cmurphy> #startmeeting keystone
16:00:12 <openstack> Meeting started Tue Oct  2 16:00:11 2018 UTC and is due to finish in 60 minutes.  The chair is cmurphy. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:16 <openstack> The meeting name has been set to 'keystone'
16:00:23 <gagehugo> o/
16:00:26 <jgrassler> o/
16:00:26 <cmurphy> #link https://etherpad.openstack.org/p/keystone-weekly-meeting agenda
16:00:46 <cmurphy> lbragstad is starting his 18-year indentured servitute so i am standing in
16:01:06 <cmurphy> please add things to the agenda
16:01:17 <knikolla> o/
16:02:50 <cmurphy> #topic Outreachy project submissions
16:02:58 <cmurphy> #link https://etherpad.openstack.org/p/keystone-outreachy-proposals keystone outreachy proposals
16:03:20 <cmurphy> I started an etherpad to draft outreachy project submissions, it's a little more structured than the brainstorming etherpad
16:03:39 <hrybacki> o/
16:03:59 <cmurphy> there's one draft there, feel free to give me feedback on it
16:04:09 <cmurphy> and placeholders for kmalloc's ideas
16:04:24 <cmurphy> and if you plan to propose other projects feel free to add them there
16:04:55 <cmurphy> any thoughts on that topic?
16:05:47 <ayoung> What is the time frame for submissions?
16:06:07 <cmurphy> ayoung: deadline is Oct 16 I think
16:06:29 <cmurphy> but they start accepting them and letting applicants apply for them before that iirc so it's good to get them in asap
16:06:33 <knikolla> i like the tasks in there
16:07:37 <cmurphy> ayoung: interesting, would be good to have a spec for that idea before submitting it as a project i think
16:07:39 <gagehugo> heh "REALLY motivated"
16:08:20 <ayoung> cmurphy, I'll work on it
16:08:27 <cmurphy> okay thanks ayoung
16:08:46 <ayoung> we've discussed it a bunch, so I think we should be able to come up with something clear fairly quickly on the keystone side
16:09:11 <ayoung> client might take more work, but that would probably be follow on
16:10:20 <cmurphy> okay, let's move on, we can come back to this in open discussion if we like
16:10:33 <cmurphy> #topic Application credentials fine grained access control spec
16:10:39 <cmurphy> #link http://specs.openstack.org/openstack/keystone-specs/specs/keystone/stein/capabilities-app-creds.html spec
16:10:55 <jgrassler> Yeah, that one...
16:11:01 <cmurphy> so jgrassler has let me know that he has a bit too much on his plate to make this feasible for him to accomplish
16:11:13 <ayoung> Outreachy!
16:11:31 <cmurphy> :) i worry this is a little too involved for an outreachy project
16:11:38 <ayoung> Heh.  You Think?
16:11:41 <jgrassler> I wouldn't have phrased it quite so diplomatically ("I fear I have to flake out on this"), but yes, that's the gist of it.
16:11:52 <cmurphy> i wanted to ask if anyone was clamoring for a new project or would want to split up the work with me
16:12:03 <ayoung> cmurphy, I'll help
16:12:06 <jgrassler> ayoung: yeah...the thought had crossed my mind, too during the previous agenda item :-)
16:12:20 <cmurphy> ayoung: awesome, thanks
16:12:22 <ayoung> the fine grained stuff was kindof something I wanted
16:12:38 <cmurphy> also i wanted to thanks jgrassler for getting the spec in shape, that was no small effort
16:13:08 <ayoung> cmurphy, should I update this spec to support it: https://review.openstack.org/#/c/456974/
16:13:14 <ayoung> Role check on body key
16:13:26 <jgrassler> Thanks :-) I'd meant to write the code at some stage, though...
16:13:32 <ayoung> that was originally for the policy-in-rbac,
16:13:41 <cmurphy> ayoung: i'd rather we stick to the spec we agreed to on this iteration
16:14:11 <ayoung> cmurphy, agreed.  I meant as a mitigation for things that spec couldn't cover, I could rewrite this spec to be "here's how we do that in the future"
16:14:42 <ayoung> things like the actions api in nova that might not map perfectly to URL+Verb along
16:14:43 <ayoung> alone
16:15:35 <ayoung> Or are we confident we can get away without the body checks.  I'm ok deferring it
16:16:25 <cmurphy> i think this iteration doesn't drastically need it because we're still doing the policy checks as well which cover the different policies for different bodies
16:16:44 <ayoung> ++
16:17:06 <cmurphy> anyways, thanks jgrassler and ayoung
16:17:09 <ayoung> I'll keep it in mind.  Anyway, I can tackley the SQL and API changes
16:17:19 <cmurphy> ++
16:17:27 <cmurphy> #topic Shared-Nothing Keystone for Multisite
16:17:34 <cmurphy> ayoung: i'm guessing this is you
16:17:34 <ayoung> Is App Creds Flaskified yet?
16:17:40 <cmurphy> not yet
16:18:00 <ayoung> OK, so SHared nothing is another way to say "different databases"
16:18:19 <ayoung> i.e.  we put a keystone server at a region, and it handles only requests for that reqion
16:18:52 <ayoung> and I have a couple reviews outstanding in support of this approach.  I'd really appreciate getting any battles about them out of the way soonish
16:19:23 <ayoung> I have a spec, too, that I am resurrecting, based on the Federated ID changes, that should help multis-site
16:19:51 <ayoung> basically, the Fed Query APIs let  a user ask "what would Keystone do about this REMOTE_USER/REMOTE_GROUPS set of variables
16:20:15 <ayoung> it will support pre-population of values, and it will support using the Keystone data from an Application as well
16:20:32 <ayoung> i.e. Identity as a Service?
16:20:38 <ayoung> or something like that
16:20:53 <ayoung> So, please look at them, and tell me what you don't like
16:20:57 <cmurphy> ayoung: we have pushed back really hard on allowing user-set IDs and I don't fully understand the reasoning this is needed now, why was the predictable ID not sufficient?
16:21:03 <cmurphy> i think kmalloc had strong objections too
16:21:24 <ayoung> cmurphy, user-set IDs for Domain is to give us a top-level division between regions
16:21:35 <ayoung> it is expected that only the domain_id will be user-set,
16:21:46 <ayoung> not the userids, or project ids
16:21:58 <ayoung> so, I am kindof sticking with kmalloc 's reasoning on the majority
16:22:16 <ayoung> domains are a little different, in that the domain_id is used to generate the user-id in LDAP and, with the other path, Federation
16:23:16 <ayoung> it also lets us say "this domain is at this regions, that domain is at that region" in a central server, and have the rest of the data be local to the regional keystone servers
16:23:34 <ayoung> kmalloc, has since given his approval, at least in IRC, to this approach
16:24:30 <ayoung> IdPs are also (iirc) allowed to have user-set Identifiers, which will let us have a unified approach to identity across the multi-sites
16:24:47 <ayoung> cmurphy, any real objection, or just wondering "why now?"
16:24:55 <cmurphy> ayoung: mostly wondering why
16:25:15 <ayoung> cmurphy, cool.
16:25:31 <cmurphy> ayoung: i will read the spec and ask you more questions
16:25:35 <ayoung> ++
16:26:15 <ayoung> cmurphy, we have at least 2 customers that are tackling this problem and I would like to set them up for success
16:26:42 <ayoung> I'm willing to listen to any and all wisdom on how to handle multi-site when you can't scale galera to all of the nodes
16:27:20 <ayoung> according to what I've heard, galera will cover ~ 9 nodes, which, if you HA at a site with 3 msql instances, lets you get up to a 3 site cluster
16:27:37 <ayoung> but, if galera goes down, you've lost the the whole cluster
16:27:58 <ayoung> and that is a non-rare occurrence
16:28:31 <ayoung> So the thought experiement is:  what if we have a hub-and-spoke model, and try to minimize the amount of data to keep in sync between them
16:28:53 <ayoung> at the hub, we would have keystone and the minimal service catalog with the regional keystone only
16:29:09 <ayoung> hub would also have a domain-to-region mapping...
16:29:18 <ayoung> and I don;'t have a good answer on how to do that today
16:30:52 <cmurphy> i think i'm missing why you need a domain to region mapping, one domain per region doesn't fit most use cases
16:30:52 <ayoung> Anyway, that is the goal, and I am looking for input on how to acheive it, or smart alternatives that are not just "completely separate openstack deployments"
16:31:10 <ayoung> cmurphy, so, the domain-to-region does not need to be one-to-one
16:31:15 <cmurphy> ah
16:31:27 <ayoung> instead, it is a way of saying "if this domain is linked to this region, this regional keystone owns the data in it"
16:32:18 <cmurphy> so kind of like a catalog for your regions
16:32:19 <ayoung> I think the Federated domain, where the users are mapped, would still be owned by the central
16:32:25 <ayoung> cmurphy, exactly, yes
16:32:53 <ayoung> cmurphy, I would only put the identity servers into the central catalog, though
16:33:12 <ayoung> if you want to see what is really supported there (nova, sahara, whatever) go to that Keystone server and look at the catalog
16:35:59 <ayoung> So...I'm done
16:36:03 <cmurphy> anyone have questions for ayoung ?
16:36:25 <cmurphy> #topic open discussion
16:36:55 <cmurphy> fyi i won't be at office hours
16:37:12 <kmalloc> O/
16:37:42 <cmurphy> kmalloc we were talking about ayoung's explicit_domain_id plan
16:38:12 <kmalloc> Reading backlog
16:38:15 <kmalloc> Gimme a sec
16:39:21 <kmalloc> i fall back to the oath model
16:39:40 <ayoung> Oaf?
16:39:57 <kmalloc> and predictable ids, meaning it's not "supplied" but it is something we predictably generate
16:40:01 <kmalloc> domains being the separate case
16:40:13 <kmalloc> meaning we need to allow some level of autoprovisioning or mechanism to supply a known domain id.
16:40:26 <kmalloc> oath - yahoo folks doing the federation
16:40:47 <kmalloc> the domain-to-region mapping is the key that way
16:41:45 <kmalloc> i don't see an issue with ayoung's approach architecturally within keystone
16:42:27 <kmalloc> and i am willing to cede a minimal level of keystone-auto-id-owning to make the model work
16:42:38 <ayoung> #link http://superuser.openstack.org/articles/massive-hyperscale-insfrastructure-openstack/
16:42:40 <kmalloc> it is a real concern in the direction everything is taking
16:42:46 <cmurphy> cool, i'll look closer at it soon
16:43:21 <ayoung> looking for design docs
16:43:33 <kmalloc> i am also ok if a local keystone generates a local "id" that is combined with a specific local domain to generate the ultimate user-id.
16:44:02 <kmalloc> and the "generated local-id" then is transmitted to the remote keystones, meaning the id, pending the domain-id being in sync, would be consistent between deployments
16:44:34 <ayoung> so...sync is a problem
16:44:40 <kmalloc> that is assuming a single keystone is the sole owner of a user resource though... so no a->b->a->c->a
16:44:45 <ayoung> right now, what we could do is have a listener
16:44:48 <kmalloc> just a->c, a->b....
16:45:01 <ayoung> that would try and talk to a remote server, but if that server is down, we need to hold on to the changes
16:45:15 <kmalloc> i am ok with a "domains must be made and configured when a federated source of id is configured"
16:45:17 <ayoung> or, have some way to "requery all since X" when the remote site comes back up
16:45:31 <kmalloc> there is work to do to trust a source of id, one step may be making a domain.
16:45:46 <ayoung> so, I am really trying to avoid data sync.  THus, limiting it to domains, and the rest be "provisioned on demand with predicatble IDS"
16:46:10 <ayoung> keeping Federated configuration (IDP etc) in sync is a low risk
16:46:24 <ayoung> users and project assignements are much more likely to be dropped
16:46:31 <kmalloc> the goal is to avoid needing to sync everything. i don't want to sync domains, i am inclined to say operationally, make domain X, and configure IDP as trusted and linked to domain x
16:46:42 <kmalloc> it is part of the setup.
16:47:44 <ayoung> so, the other part is "if we need info from Region X to work with Region Y, use K2K"
16:48:03 <ayoung> and now we have some modicum of a framework to keep the assignements we need in sync
16:48:21 <kmalloc> that is largely what we've been driving at. or "central IDP" and locally assignments are derived from IDP info (on demand) or concretely set.
16:48:23 <ayoung> like, DomainXY maps to Domain Y
16:48:42 <kmalloc> the central idp may be keystone, may be something else with advanced mapping in the region-local keystones
16:49:30 <ayoung> I'm thinking specifically of knikolla 's use cases where the Cinder Server is owned by one Keystone, and the Nova server by another
16:49:51 <kmalloc> nothing says you still can't have a central source of ID
16:50:03 <kmalloc> which in knikolla's case, right now is a keycloak.
16:50:32 <kmalloc> the proxy/k2k bits for service federation is separate from strict id federation
16:50:42 <kmalloc> don't try to over engineer a single solution
16:50:45 <ayoung> cmurphy, does this make sense?
16:51:12 <kmalloc> it is likely they are related (lean on similar tech) but are setup with some separation.
16:51:17 <kmalloc> since the concerns are slightly different
16:51:27 <kmalloc> autoprovision, consistent/predictable ids is part of it
16:51:36 <cmurphy> ayoung: so far yes, though I'm still not 100% on why plain k2k isn't sufficient
16:51:52 <kmalloc> but you have to look at user->service interaction and service->service interactions
16:52:05 <kmalloc> so, k2k can fill s->s (on behalf of the user)
16:52:30 <kmalloc> but typically users will go idp->[auth through keystone]->Nova (or similar) and not have to bounce through k2k
16:53:17 <ayoung> One thing we can't do today is tell a user "you can only use this region's servers"
16:53:46 <kmalloc> realistically, we're talking about changing k2k to more of "it isn't specifically a keystone you're talking to, it is any form of ID"
16:53:51 <ayoung> however, if the regional keystone is the only thing that can supply tokens for those servers, you now have an auth-point for limiting access
16:53:57 <ayoung> can't do that solely with K2k
16:54:21 <ayoung> but K2K would be useful for then saying "OK, now allow user U2 to use this nova as well"
16:54:31 <cmurphy> ayoung: we could give them a mapping that only maps to the domains in one region
16:54:38 <cmurphy> but i guess you're saying the user can't discover that
16:54:40 <kmalloc> cmurphy: correct.
16:55:54 <ayoung> Service catalog is not used to enforce, either
16:56:03 <kmalloc> and should not be used to enforce*
16:56:12 <ayoung> so a user can use an endpoint not in their catalog if the token is still valid
16:56:54 <ayoung> So,  say we have 3 tiers (Gold, Silvern Bronze)
16:57:17 <cmurphy> 3 minutes left
16:57:20 <ayoung> do it as 3 regions, each with their own keystone, each with a distinct service catalog
16:57:43 <kmalloc> the enforcement should not be predicated on the SC, the SC may communicate what is really enforced (I worry about shipping the SC around as auth[z] data)
16:57:54 <ayoung> the only time you need to do K2K is to set up an arraingement that cuts horizontally, and that should be owned by one of the regions
16:58:50 <kmalloc> all of this is enhancements to how keystone does federation and how sources of IDentity are handled.
17:00:03 <cmurphy> times up
17:00:06 <kmalloc> the end goal is: source of identity -> keystone, and keystone is locally independant/not synchronized.
17:00:09 <cmurphy> #endmeeting