17:00:02 <lbragstad> #startmeeting keystone-office-hours
17:00:06 <openstack> Meeting started Tue Apr 17 17:00:02 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:09 <openstack> The meeting name has been set to 'keystone_office_hours'
17:01:41 <cmurphy> o/
17:02:03 <knikolla> \o
17:04:15 <openstackgerrit> Russell Tweed proposed openstack/keystone master: Add prerequisite package note to Keystone install guide  https://review.openstack.org/552568
17:05:11 <openstackgerrit> Lance Bragstad proposed openstack/keystone master: Remove the sample .conf file  https://review.openstack.org/521249
17:37:43 <cmurphy> so did we decide how much we care about the jwt token being encrypted or not?
17:38:44 <knikolla> i'd rather them not be for interop reasons, but we might have that configurable
17:39:07 <knikolla> also the library that supported jwe, was gpl lincensed if i'm not wrong.
17:39:41 <lbragstad> what if we provided a jws token provider and a jwe token provider later?
17:40:05 <knikolla> and we could mark it as experimental
17:40:33 <lbragstad> yeah - and i think we would have to be opinionated about it given the vulnerabilities of jws
17:41:03 <knikolla> opinionated is fine i think.
17:41:23 <lbragstad> e.g. we might only support a specific signing implementation
17:41:48 <lbragstad> which could limit the interop use case
17:43:46 <knikolla> probably not, as most clients will support at least the popular signing algos.
17:44:02 <knikolla> unless they're strongly opinionated.
17:47:42 <lbragstad> i assume we would be targetting something like HS256
17:47:48 <lbragstad> initially
17:48:44 <knikolla> that raises the question of symmetric vs asymmetric
17:49:55 <lbragstad> if we do asymmetric, it would be easier for other service to validate tokens offline in middleware if they aren't encrypted
17:50:01 <lbragstad> services*
17:51:05 <lbragstad> if that's something we want to support
17:51:29 <knikolla> depending on if we're putting the roles in the token
17:51:51 <lbragstad> ayoung had a specification to include a maximum of one role in the token payload
17:53:24 <knikolla> would be hard to have that AND default (and more granular) roles happen at the same time.
17:53:44 <lbragstad> how so?
17:54:32 <knikolla> maybe too big of a step for the community to do at once?
17:54:35 <cmurphy> wouldn't the token need the catalog too? and then we're back to the problem with pki
17:55:01 <knikolla> i don't mind keystonemiddleware making a validate call, in general
17:55:11 <knikolla> because roles and service catalog are openstack concepts
17:55:18 <knikolla> for outside services which don't care about that
17:55:33 <knikolla> an asymmetrically signed token they can validate with enough user information may be enough for interop
17:56:00 <lbragstad> does outside services mean other openstack services or something else?
17:56:08 <knikolla> non-opensatck
17:56:16 <lbragstad> ok
17:56:59 <lbragstad> yeah - it's tough where to draw the line between what makes sense to include and not include in the token
17:57:22 <lbragstad> it really depends on what level of validation the service is going to do
17:57:35 <lbragstad> i'm not sure if i clearly understand what that is yet =/
17:58:21 <lbragstad> do they just want to know if the token came from keystone? do they want to know if contents are valid? do they do something special with expiration? do they want the roles?
17:58:41 <lbragstad> do they want the service catalog?
17:59:00 <knikolla> i don't think they want the service catalog for one.
17:59:12 <knikolla> because if they want that, they know about openstack. so they can make a token validate call and get it.
18:00:03 <knikolla> As in, if you want the service catalog -> you know this token came from keystone -> you know how to make a call to get the service catalog if it's not in the token
18:02:59 <lbragstad> so - maybe the question is, what do services want out of the token used to make the request?
18:03:18 <lbragstad> do they want to be able to have all the bit necessary to do policy enforcement, for example?
18:04:23 <knikolla> that will be abstracted by keystonemiddleware anyway.
18:04:52 <knikolla> scope/role information will be available to the services regardless.
18:04:55 <cmurphy> the service needs to validate that it is a valid token issued by keystone and needs to check that the project and user and the role assignments in the token are still valid at that point in time, which could have changed in between the token being issued and the token being used
18:05:46 <lbragstad> that's true
18:06:20 <knikolla> my gut feeling is to only include standard user information (username, email, etc) and the current project the token is scoped to.
18:06:27 <knikolla> have everything else be fetched on a validate call.
18:08:24 <knikolla> (and of course the usual issued at, expires at, issuer, etc)
18:11:11 <lbragstad> so - aside from the possibility of making it easier to validate tokens at the service
18:11:39 <cmurphy> yeah, i guess the only difference is right now the service is forced to ask keystone to decrypt the token to get that information, with an unencrypted token a badly behaving service could decide not to validate, but that's not really our problem i guess
18:12:15 <lbragstad> there is the possibility to make the key repository bits easier to manage between the keystone nodes of a deployment
18:12:30 <cmurphy> the only other concern i had was that right now if you have an expired/revoked token you can't get any information out of it even if you try to validate it, but with an unencrypted token that information is just in plain sight
18:13:34 <knikolla> i don't think there's anything particularly sensitive in there.
18:14:13 <lbragstad> i guess if i'm an operator and i think that stuff is sensitive, then i'd opt for fernet
18:14:58 <knikolla> dims: we're discussing jwt in case you want to join the discussion
18:15:03 <lbragstad> or maybe my deployment has to fill a check box that says that information can't be exposed
18:16:00 <knikolla> fernet would be a viable solution for that. and maybe in the future we implement jwe to replace fernet.
18:16:07 * gagehugo sneaks back in
18:16:31 <lbragstad> or just a comparable alternative
18:17:20 <lbragstad> i still believe we're primarily pursuing jwt (jws/jwe) to offer some alternative in the worst case scenario that something goes wrong with fernet
18:17:57 <lbragstad> if we implement jwe, then replace or remove fernet, we're pretty close to square one in that we don't have a viable backup option
18:18:14 <knikolla> for jws i'm also looking at the federation with external systems case
18:18:37 <knikolla> a signed jwt with user information is similar to a saml assertion
18:19:00 <lbragstad> regarding the asymmetric vs symmetric bits - do you have a inclination at which approach would make that story easier for you?
18:19:01 <cmurphy> saml assertions can be encrypted too
18:19:30 <knikolla> cmurphy: yes, but the key of the intended audience.
18:19:44 <knikolla> with jwe we're targeting keystone to be the issuer and audience.
18:20:30 <cmurphy> that's true
18:20:38 <knikolla> lbragstad: i think asymmetric. since external systems can't know the symmetric key.
18:20:54 <lbragstad> #link https://stackoverflow.com/questions/8276233/is-it-recommended-to-sign-and-encrypt-saml-and-use-ssl
18:21:03 <lbragstad> yeah
18:21:31 <lbragstad> there is another requirement i've heard before where a deployment spans large geographic regions
18:21:48 <lbragstad> region A and region B are part of the same cloud
18:22:14 <lbragstad> but there are data restrictions in region B that prevent users in region A from using it (or data crossing the border)
18:22:31 <lbragstad> which is problematic when syncing the key repository
18:23:07 <cmurphy> we'd have to sync keys no matter what, it's just a question of syncing public keys or private keys
18:23:23 <lbragstad> right - i agree there
18:24:04 <lbragstad> publishing a public key would be "less scary" than syncing private keys is the justification i've heard
18:24:37 <knikolla> you want the private key is an fewer places as you can generally.
18:24:56 <cmurphy> if you publish a public key you'd have to establish some way of trusting it
18:25:05 <cmurphy> i guess maybe just http and tls
18:25:19 <knikolla> or ansible/puppet/etc.
18:25:25 <lbragstad> if we did asymmetric with jws - each keystone node would only have it's own private key
18:25:53 <lbragstad> which would be more consistent with the intended usage of a private key file
18:26:11 <lbragstad> and the public key would be shared or distribute (however that happens)
18:26:21 <knikolla> potentially that could solve the data restriction you mentioned above.
18:27:46 <lbragstad> yeah - possibly
18:27:51 <knikolla> as in you could have multiple keystones in the region and use the same token
18:28:07 <knikolla> i remember this is one of the use cases of the oranje (i think) people.
18:28:17 <lbragstad> yeah
18:28:55 <lbragstad> they have datacenters in europe - and some of the new legislation affects stuff like that (also on the db replication layer)
18:29:12 <cmurphy> that wouldn't solve their issue, there wouldn't be synchronized projects in each region so the tokens wouldn't be valid in all regions
18:29:15 <gagehugo> yeah that's their usecase
18:30:42 <lbragstad> i thought they were going to work around that with a custom backend to handle orchestrating projects with the same ID across regions?
18:31:04 <lbragstad> i thought we discussed something like that with them in office hours
18:31:09 <knikolla> cmurphy: not necessarily. if keystonemiddleware is smart enough, it can check the issuer of the token, and see that it is region A, so make the validate call to region A keystone rather than region B.
18:32:31 <cmurphy> lbragstad: yes that was the workaround, but i think that was actually bad advice because we decided the resource driver is sql only
18:33:46 <lbragstad> bah
18:33:46 <cmurphy> knikolla: so then the token has to be validated across datacenters? that seems like a similar data restriction problem and also not very performant
18:34:09 <lbragstad> https://github.com/openstack/keystone/blob/master/keystone/conf/resource.py#L23-L27
18:34:47 <cmurphy> which is why i'd like to undo all those foreign keys and fix that
18:35:05 <lbragstad> so - iirc we're back to making k2k more performant
18:35:09 <lbragstad> or what cmurphy said
18:35:14 <cmurphy> there are probably a lot of people still on newton who are going to be surprised by that
18:36:33 <knikolla> a jwt token can be equivalent to k2k and probably more performant. since you wouldn't need an additional call to have keystone generate one.
18:36:58 <knikolla> aka make k2k use jwt.
18:37:50 <knikolla> where a jwt token is able to be validated offline and contains all the information that is in a saml assertion.
18:39:10 <cmurphy> that's kind of the opposite of what you said before "my gut feeling is to only include standard user information (username, email, etc) and the current project the token is scoped to."
18:39:26 <cmurphy> a saml assertion has a lot of crap
18:39:57 <knikolla> boilerplate crap.
18:40:11 <knikolla> it can be boiled down to username, email, domain and project.
18:41:29 <knikolla> so it's not really the opposite, it's almost the same information i was arguing for.
18:42:22 <cmurphy> okay that's fair :)
18:44:38 <lbragstad> ok - approaching the question from a different angle
18:45:28 <lbragstad> does it make sense to implement jws with symmetric signing if we already have a token provider that uses symmetric encryption?
18:46:59 <knikolla> i can't really think of a use case where that would give us something over fernet.
18:47:22 <knikolla> you need the private key to validate the token.
18:47:28 <knikolla> if you have the private key you can sign the token.
18:47:43 <knikolla> same is valid for fernet (+encryption)
18:49:19 <lbragstad> jwe would be nice to have as a compliment to fernet, but it doesn't look like we're going to get that in the near future
18:49:27 <knikolla> in other words: if you need the private key to verify it, potentially it doesn't matter if it is encrypted or not.
18:50:23 <lbragstad> sure
18:52:36 <lbragstad> i kinda like the approach paseto takes
18:53:05 <lbragstad> you have a thing and you can configure it to give you encrypted tokens or signed tokens
18:53:18 <lbragstad> which ever you chose depends on your deployment
18:56:25 <lbragstad> cmurphy: i guess you take on the resource backend might have an impact on https://review.openstack.org/#/c/559676/
18:56:31 <lbragstad> your take*
18:56:55 <knikolla> arguably by sticking with a secure algorithm for jwt, and not processing anything else we get pretty close to paseto.
18:57:32 <lbragstad> essentially - yeah
18:57:53 <knikolla> as we release new versions we might deprecate validating that algorithm and starting to use another one, and then remove it entirely (giving us the versioning)
18:58:16 <cmurphy> lbragstad: yeah i have mixed feelings on it
18:59:06 <lbragstad> cmurphy: i see what you mean (and i remember dstanek_ talking about stuff like that a long time ago)
18:59:32 <lbragstad> it feels like we're constantly stuck in the middle
19:00:01 <lbragstad> we want to be flexible in what we back to, but because of that we can't use the default thing we back to in ways it was meant to be used
19:01:31 <lbragstad> knikolla: yeah - we will have to validate the algorithm in the token header within keystone for sure
19:02:00 <lbragstad> since that was one of the main issues in the JWT library vulnerability
19:02:13 <lbragstad> er - JWT vulnerability across libraries*
19:04:25 <knikolla> lbragstad: do you think we have enough information from this discussion to start shaping up the spec?
19:05:36 <lbragstad> i think so - but i wouldn't mind recapping
19:05:50 <lbragstad> based on the discussion
19:06:05 <lbragstad> do we see value in a signing implementation of a token provider?
19:06:32 <lbragstad> i would say yes - only because it's not the only option available and people can always opt into using fernet if they want added security
19:07:39 <lbragstad> s/security/security through obscurity of encryption/
19:13:33 <lbragstad> knikolla: cmurphy what other bigger points would you want included in the spec if any?
19:14:49 <cmurphy> lbragstad: I'd like to see huawei's use cases mentioned
19:15:05 <lbragstad> which ones?
19:15:24 <lbragstad> bah - that was a dumb question, you probably want to see all of them
19:15:34 <cmurphy> lbragstad: i don't know, what are your use cases? why is huawei pushing for it?
19:15:37 <knikolla> Federation use cases. Enough information to be useful for external systems. Aka keystone as an idp with jwt.
19:15:45 <cmurphy> we had it in the backlog for some vague "it would be nice" reasons, if we're committing to it this cycle it would be good to justify why it's more important now
19:16:23 <lbragstad> the main reason we want it is for the asymmetric signing bit
19:17:02 <lbragstad> huawei used PKI for a long time, and they wanted something that would eventually behave similarly
19:17:03 <gagehugo> keystone as an idp with jwt is something I know was brought up before
19:17:39 <gagehugo> I think kubernetes intergration is something too
19:18:07 <lbragstad> there was also hesitation internally about sharing symmetric keys across large deployments
19:23:56 <lbragstad> even though there are other hurdles there
19:24:05 <lbragstad> (db replication and whatnot)
19:31:56 <lbragstad> so - i think those are the two huawei usecases
19:32:06 <lbragstad> but wxy might be able to give more insight
19:35:55 <cmurphy> I think solidifying those use cases would inform the technical decisions we need to make
19:36:52 <lbragstad> i can add them
19:36:54 <cmurphy> if huawei is banking on this being a PKI replacement and that's not something we can offer then we need to be realistic about that and prioritize accordingly
19:37:38 <cmurphy> or if having a PKI replacement is a top design goal then we might need to rethink some things
19:38:15 <lbragstad> sure - that makes sense, i don't think it would be a PKI replacement
19:39:11 <lbragstad> i guess my interpretation of it is that it would allow deployments using PKI an alternative if they are willing to roll their own token formatter to include bits we've taken a hard stance against
19:39:57 <lbragstad> while providing some level of security in the sense that we have something to backup fernet
20:18:22 <kmalloc> lbragstad: if we support JWT, at least if fernet has an OOPSE, we have an out
20:18:30 <kmalloc> JWE long term goal when we van
20:18:31 <kmalloc> can*
20:19:55 <kmalloc> right now, a PKI replacement is not realistic, short of JWE
20:21:23 <lbragstad> yeah - i'm not saying we should reimplement PKI with JWT
20:22:03 <kmalloc> and i would LOVE to see JWE
20:22:14 <kmalloc> but... i'm realistic
20:22:27 <kmalloc> i would like us to be on JWT in general for reasons of $STANDARDS$
20:22:39 <lbragstad> but... theoretically, if a deployment was using PKI for a long time, they could extend the jws token provider to get similar functionality without having to maintain the pki token provider out of tree
20:22:45 <kmalloc> yeah
20:22:50 <kmalloc> that is true
20:23:16 <kmalloc> also... if your employer wants to dedicate resources to PKI-like implementation on the JWT base [it's a lot of leg work]
20:23:20 <kmalloc> i'm happy to have that.
20:23:36 <kmalloc> but I want it to be clearly part of the standards and it would need ot be (pref) in a lib we can consume
20:24:03 <kmalloc> (one of the main-stream libs)
20:25:12 <kmalloc> i also agree with cmurphy, clear use-cases documented = win
20:25:29 <lbragstad> that's what i'm working on now
20:25:58 <kmalloc> cool
20:26:22 <lbragstad> fwiw - if we did the asymmetric route - we'd be targetting the RS256 algorithm
20:30:39 <kmalloc> yeah
20:30:57 <kmalloc> i would expect as much
20:31:06 <kmalloc> fwiw, it should be easy to implement both rS256 and HS256
20:31:13 <kmalloc> (relatively)
20:58:32 <openstackgerrit> Doug Hellmann proposed openstack/keystoneauth master: add lower-constraints job  https://review.openstack.org/555625
20:58:33 <openstackgerrit> Doug Hellmann proposed openstack/keystoneauth master: fix pep8 errors caused by pycodestyle>=2.4.0  https://review.openstack.org/562052
21:11:37 <openstackgerrit> Lance Bragstad proposed openstack/keystone-specs master: Repropose JWT specification for Rocky  https://review.openstack.org/541903
21:16:47 <lbragstad> kmalloc: cmurphy knikolla gagehugo wxy updated with usecases
21:16:50 <lbragstad> ^
21:17:52 <lbragstad> i guess the next step is determining if they are valid and adding missing ones
21:18:06 <lbragstad> i'll rework the specifications technical details after that
21:21:00 <lbragstad> #endmeeting