17:01:30 <lbragstad> #startmeeting keystone-office-hours
17:01:31 <openstack> Meeting started Tue Jun 26 17:01:30 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:35 <openstack> The meeting name has been set to 'keystone_office_hours'
17:01:48 <kmalloc> if anyone has questions on the RBACEnforcer, i know it's super dense.
17:02:00 <kmalloc> I can speak to it and a lot of the quirks in our policy code.
17:03:35 <lbragstad> awesome
17:03:44 <lbragstad> i'm going to grab lunch quick and i'll be right back
17:03:56 <lbragstad> knikolla: and i were going to try and tag team a few bugs today
17:04:01 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1658641
17:04:01 <openstack> Launchpad bug 1658641 in OpenStack Identity (keystone) "Moving/disabling LDAP users break Keystone queries depending on role ID" [Medium,In progress] - Assigned to Kristi Nikolla (knikolla)
17:04:10 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1757022
17:04:10 <openstack> Launchpad bug 1757022 in OpenStack Identity (keystone) ""keystone-manage mapping_purge" ignores --type option" [Undecided,In progress] - Assigned to Dai Hanada (dai-hanada)
17:04:18 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1775207
17:04:18 <openstack> Launchpad bug 1775207 in OpenStack Identity (keystone) "Fetching all mappings may become too slow" [Undecided,In progress] - Assigned to Pavlo Shchelokovskyy (pshchelo)
17:05:26 * knikolla going for lunch
17:05:29 <kmalloc> lbragstad: i'm going to try and get the "move an API" patch up today.
17:05:45 <kmalloc> so its easier to see how the flask stuff actually shakes out.
17:05:48 <gagehugo> o/
17:06:07 <lbragstad> that'd help
17:23:17 <kmalloc> lbragstad: yeah, so working on the scaffolding update patches now and then api move will be soon
18:01:24 <pas-ha> lbragstad: hi, re bug 1775207, I noticed you've put an 'office-hours' tag on it - wdym and is my attention required/expected?
18:01:24 <openstack> bug 1775207 in OpenStack Identity (keystone) "Fetching all mappings may become too slow" [Undecided,In progress] https://launchpad.net/bugs/1775207 - Assigned to Pavlo Shchelokovskyy (pshchelo)
18:01:56 <lbragstad> pas-ha: we use the office-hours tag as a way to focus on a specific set of bugs or reviews
18:02:17 <pas-ha> oh, ok, just saw it mentioned in the scrollback :)
18:02:26 <lbragstad> we had a user come through the channel yesterday and we noticed a few reviews related to keystone-manage that could use some attention
18:02:40 <lbragstad> i added the tag to it so that we could hopefully get some eyes on it
18:09:07 <knikolla> lbragstad: i have a meeting now, but will join you in 1 hr or so for the ldap stuff
18:09:27 <lbragstad> sounds good - cleaning up one of the patches now, should be ready for review by then
19:15:47 <openstackgerrit> Lance Bragstad proposed openstack/keystone master: Fix keystone-manage mapping_purge with --type option  https://review.openstack.org/554397
19:16:09 <lbragstad> knikolla:  ^ those could be a bit more dry - but they're functional
20:03:14 <openstackgerrit> Morgan Fainberg proposed openstack/keystone master: Add support for enforce_call to set value on flask.g  https://review.openstack.org/578189
20:03:14 <openstackgerrit> Morgan Fainberg proposed openstack/keystone master: Update Scaffolding (flask) for json home documents  https://review.openstack.org/578190
20:04:04 <openstackgerrit> Morgan Fainberg proposed openstack/keystone master: Update Scaffolding (flask) for json home documents  https://review.openstack.org/578190
20:10:34 <knikolla> lbragstad: looking
20:28:13 <knikolla> lbragstad: looks good to me. +2
20:29:28 <lbragstad> knikolla: cool - thanks
20:29:39 <lbragstad> i'm a little worried about the duplication
20:30:06 <lbragstad> but i'm open to refactoring it if we can find a better way
20:30:28 <knikolla> i generally like tests to be verbose.
20:31:29 <knikolla> duplication in that case should be fine as it makes it pretty clear what the test is doing.
20:31:36 <lbragstad> that's fair
20:31:37 <knikolla> but that's just my opinion :)
20:36:02 <aning_> Hi lbragstad, I use your fernet-inspector to inspect a fernent-token, the result is this:
20:36:05 <aning_> fernet-inspector -k /opt/cgcs/keystone/fernet-keys gAAAAABbMpejHDDFLNkopYu5_PrFMKo16qidKmOXe5NvctVmja1FxqNBglzJcpma5CqiWG9L7YIVHuXlL29KotzdeHdA50IThiPhzKGREGhpVtKHFoRkGHRRHNK9VRpKSQpj7eTaKBDrRDc61NJ46H1Hh2VARmj1kv3andlwZ9ztHUYvipv86Ng
20:36:05 <aning_> [2, [True, '\xd3]\xb3{\x1c{B\xed\x8e\x9b\xe8\xc1`\x81M`'], 2, [True, '\xe6\x99u\xe0\xf4\xbdI-\x8b\x9bF%J\xbd\\X'], 1530045875.0, ['NP0\xfe\x08TC\xa4\x83\xc2\xc5\xdb\xe4;\x88;']]
20:37:19 <aning_> the Audit id from base64.urlsafe_b64encode('NP0\xfe\x08TC\xa4\x83\xc2\xc5\xdb\xe4;\x88;') is
20:37:31 <aning_> 'TlAw_ghUQ6SDwsXb5DuIOw=='
20:37:57 <aning_> And the UUID from  uuid.UUID(bytes='\xd3]\xb3{\x1c{B\xed\x8e\x9b\xe8\xc1`\x81M`').hex is
20:38:11 <aning_> 'd35db37b1c7b42ed8e9be8c160814d60'
20:38:58 <aning_> [True, '\xe6\x99u\xe0\xf4\xbdI-\x8b\x9bF%J\xbd\\X'] in the middel after the second number 2, what is it?
20:39:27 <aning_> and what's Audit id?
20:40:32 <aning_> where are user id and project id hidden in the decoded data?
20:40:39 <lbragstad> https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/token_formatters.py#n452
20:40:58 <lbragstad> this is going to go into the implementation details a bit
20:41:14 <lbragstad> but keystone users different payload classes to pack up the payload before encrypting it
20:41:24 <lbragstad> which keeps the two things separate
20:41:36 <lbragstad> (building of the payload from the thing that actually does the encryption) ]
20:41:47 <lbragstad> each payload has a version
20:42:05 <lbragstad> which is the first thing in the list when you decrypt a token
20:42:50 <lbragstad> so - in your example, you're dealing with a ProjectScopedPayload because the first element of the list is an integer of 2
20:43:56 <lbragstad> the ProjectScopedPayload returns a tuple which gets used here - https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/token_formatters.py#n158
20:44:14 <lbragstad> notice that the version is coming from the payload classes that was used to build the payload
20:45:18 <lbragstad> the second integer is a compressed representation of the authentication methods associated with that token
20:45:21 <lbragstad> https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/token_formatters.py#n464
20:45:36 <knikolla> lbragstad: my brain is fried for now. i'll head home and then work on https://review.openstack.org/#/c/487579/ later tonight
20:45:44 <lbragstad> we do this instead of passing method: ['password', 'token']
20:45:47 <lbragstad> knikolla: sounds good
20:46:39 <lbragstad> aning_: because using methods: ['password', 'token'] in a token payload bloats it significantly, so we convert the configured authentication methods to a unique integer that can be reinflated at validation time
20:47:24 <lbragstad> see https://git.openstack.org/cgit/openstack/keystone/tree/keystone/auth/plugins/core.py#n46
20:47:33 <lbragstad> and https://git.openstack.org/cgit/openstack/keystone/tree/keystone/auth/plugins/core.py#n63
21:12:03 <aning_> Sorry I was pulled away for while ... these are very valuable information.
21:12:29 <aning_> but jus from a high level, I saw three hex strings
21:12:53 <aning_> The first one is UUID, the last one is Audit ID, what's the middle one?
21:15:08 <aning_> If I guess, it should be password
21:16:21 <lbragstad> this is the payload
21:16:23 <lbragstad> https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/token_formatters.py#n469
21:16:29 <lbragstad> or the format of the payload
21:16:37 <aning_> or token depends on the integer before it, since that integer is the auth method.
21:17:00 <lbragstad> so version = 2
21:17:12 <lbragstad> b_user_id is [True, '\xd3]\xb3{\x1c{B\xed\x8e\x9b\xe8\xc1`\x81M`']
21:17:20 <lbragstad> 2 is the methods
21:17:22 <aning_> right, version = 2 in my example.
21:17:35 <lbragstad> b_project_id is [True, '\xe6\x99u\xe0\xf4\xbdI-\x8b\x9bF%J\xbd\\X']
21:17:49 <lbragstad> expires_at_int is 1530045875.0
21:18:02 <lbragstad> and b_audit_ids is ['NP0\xfe\x08TC\xa4\x83\xc2\xc5\xdb\xe4;\x88;']
21:18:18 <aning_> Great
21:18:58 <aning_> so audit id contains credentials?
21:19:26 <lbragstad> nope - audit ids are a specific property of a token
21:19:31 <aning_> probably not, since there is no need for credentials in token ...
21:19:35 <lbragstad> right
21:19:52 <aning_> Ok got it
21:19:58 <lbragstad> an audit id is generated whenever you create a token
21:20:13 <lbragstad> we call them audit ids because they help us track which tokens are related
21:20:19 <lbragstad> so - for example
21:20:39 <lbragstad> if you authenticate for a token using your username and password you'll get back a token
21:20:49 <lbragstad> which will have an audit id
21:20:58 <lbragstad> if you use that token to reauthenticate for a new token
21:21:17 <lbragstad> your new token will contain a list of audit ids, one of which will be the audit id of the first token you authenticated for with your password
21:22:18 <lbragstad> since tokens are non-persistent, audit ids help us when a user wants to "delete" a specific token
21:22:52 <lbragstad> we can persist the audit id of the deleted token, and flag it as invalid if we ever attempt to validate a token with that audit (decrypted from the token payload)
21:23:55 <aning_> ok
21:25:20 <lbragstad> that's a lot of details about the internal guts of keystone token system... hopefully it makes sense
21:26:49 <aning_> Yes, it all makes sense ... wouldn't get them anywhere else. Fantastic!
21:28:01 <aning_> Rather complicated, need time to dig and digest.
21:28:17 <kmalloc> lbragstad: i'm trying to avoid a massive rebase/reset the stack https://review.openstack.org/#/c/577586/
21:28:19 <kmalloc> thats all
21:28:25 <aning_> Thanks a lot
21:28:43 <openstackgerrit> Morgan Fainberg proposed openstack/keystone master: Update Scaffolding (flask) for json home documents  https://review.openstack.org/578190
21:29:16 <lbragstad> aha
21:29:21 <kmalloc> this stack is a bit unweildy as is.
21:29:28 <kmalloc> just because it is a LOT of moving parts.
21:29:31 <lbragstad> yeah
21:29:42 <lbragstad> aning_: no problem
21:29:54 <kmalloc> and keeping my brain in one place at a given time has been hard, touches a lot of really overly complex parts.
21:30:29 <lbragstad> kmalloc: do we need this bit though? https://review.openstack.org/#/c/577586/1/requirements.txt
21:31:05 <lbragstad> shouldn't we be able to get away with just Flask>=1.0.2
21:32:10 <kmalloc> well, we need to adhere to what is in reqirements
21:32:16 <kmalloc> i suck and forgot to remove that part :P
21:32:43 <kmalloc> https://github.com/openstack/requirements/blob/master/global-requirements.txt#L62
21:32:47 <kmalloc> *oops*
21:33:00 <kmalloc> i dunno if the checker will get cranky or not with removing that
21:33:54 <kmalloc> i know this stack is getting deep =/
21:34:05 <kmalloc> and it's not super easy to follow because of what it touches to begin with
21:34:54 <kmalloc> but fwiw, the "dummy API" will be stood up in https://review.openstack.org/#/c/578190/ [the full end-to-end test]
21:35:00 <lbragstad> ok
21:35:05 <kmalloc> now that I have json_home scaffolding in place.
21:35:39 <kmalloc> fwiw, my brain is fried as hell working on these now =/ testing the RBACEnforcer took 3 days to write the tests.
21:35:56 <lbragstad> yeah...
21:36:07 <lbragstad> the good thing is that most of the stack leading up to that looks good
21:36:11 <lbragstad> at least IMO
21:37:03 <lbragstad> getting those through the gate will give us time to parse the RBACEnforcer change
21:39:03 <kmalloc> the NITs on the 404/418 one, do you want me to fix and rebase or as a side-addendum patch
21:39:16 <lbragstad> i'm not sure i have a solution for it...
21:39:36 <lbragstad> i'm not sure what the fix would be, it was just a concern
21:39:42 <kmalloc> i meant the other nits
21:39:57 <kmalloc> the 418 bit, i can pick another status_code [any]
21:40:14 <kmalloc> i also added the expressive comment to explain this is a testing-only-thing and what it means
21:40:34 <kmalloc> right below your review-comment (the code-comment is expressive that is)
21:40:43 <lbragstad> ahh
21:40:58 <lbragstad> that one is pretty late in the chain
21:41:11 <lbragstad> if you rebase it's only going to affect 4 patches, right?
21:42:21 <kmalloc> yeh, the enforcer patch and the newest ones on top of it
21:42:36 <kmalloc> i am hesitant to rebase the enforcer if people are actively reviewing...
21:42:53 <lbragstad> oh - sure
21:42:54 <kmalloc> but i also realize that is unlikely with the current preceeding patches not fully reviewed
21:43:17 <lbragstad> i'm just about to wrap up my review of the RBACEnforcer patch
21:43:28 <kmalloc> cool.
21:43:56 <kmalloc> i'll add an addendum patch to the 418 one to address the nits and we can swap out the expected_status bit to a different code if we want at anytime
21:44:14 <kmalloc> it's 2 lines to swap to someting else... 4 if you count the comment and the error msg
21:48:04 <openstackgerrit> Morgan Fainberg proposed openstack/keystone master: Address minor comments to 404 error detection  https://review.openstack.org/578216
21:59:57 <lbragstad> #endmeeting