18:01:14 #startmeeting keystone 18:01:15 Meeting started Tue Jul 26 18:01:14 2016 UTC and is due to finish in 60 minutes. The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:16 its like the resistance killed the ping 18:01:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:17 \o 18:01:19 The meeting name has been set to 'keystone' 18:01:30 topol: spies 4 life 18:01:36 +++ 18:01:38 agenda https://etherpad.openstack.org/p/keystone-weekly-meeting 18:01:51 o/ 18:01:59 anything to discuss since everyone scooted early from the Midcycle? 18:02:13 thanks everyone for coming to the midcycle! 18:02:28 thanks cburgess for helping host! 18:02:38 and notmorgan for organizing :) 18:02:39 Sure no problem 18:02:43 +++ Thanks cburgess 18:02:54 anyone care if I clear authorship colors from the Agenda? Hard to read as is 18:03:08 cburgess, thanks so much! 18:03:13 ayoung: you can change a setting in etherpad so you don't see colors 18:03:30 dolphm already posted a summary of the retrospective: http://dolphm.com/retrospective-on-openstack-midcycles/ 18:03:43 i'll be posting a summary on big topics 18:03:51 good job dolphm 18:03:59 but if you could not attend, then check out https://etherpad.openstack.org/p/keystone-newton-midcycle 18:04:04 an excellent midcycle 18:04:04 ayoung under the settings wheel select/deselect authorship colors. 18:04:04 its got pretty decent notes 18:04:09 hoping other projects find it useful, and we learn something in return 18:04:22 hopefully we'll remember to look at it for the next midcycle. 18:04:26 maybe we need a checklist. 18:04:38 bknudson: definitely 18:04:39 nice summary 18:04:50 bknudson: i thought about structuring it that way - it's not far off 18:05:03 especially for organizers 18:05:07 * rodrigods has a lot to catch up 18:05:08 s/bullet points/check boxes/ 18:05:35 first topic coming up 18:05:39 i found it'd be too tempting to lose the "why's" if i went straight to a checklist 18:05:40 #topic nasty bugs that need triaging and fixing before we release 18:05:55 hi all 18:05:56 dolphm the justifications are very helpful 18:06:00 o/ 18:06:04 dolphm topol agreed 18:06:15 bug 1606426, bug 1604479, bug 1602407, and bug 1600393 18:06:15 bug 1606426 in OpenStack Identity (keystone) "Upgrading to Mitaka casues significant slow down on user-list " [Critical,New] https://launchpad.net/bugs/1606426 - Assigned to Ron De Rose (ronald-de-rose) 18:06:16 bug 1604479 in OpenStack Identity (keystone) "tenantId/default_project_id missing on Keystone service user in Mitaka" [Critical,New] https://launchpad.net/bugs/1604479 - Assigned to Kam Nasim (knasim-wrs) 18:06:17 bug 1602407 in OpenStack Identity (keystone) "MySQL tests significantly slower than PostgreSQL tests" [Undecided,New] https://launchpad.net/bugs/1602407 18:06:18 bug 1600393 in OpenStack Identity (keystone) "AttributeError: 'list' object has no attribute 'items'" [High,New] https://launchpad.net/bugs/1600393 18:06:33 i am investigating 1606426 18:06:34 i did a preliminary review of open bugs and those were nasty ones 18:06:42 those all sound pretty bad 18:06:43 thanks breton 18:06:59 1602407 i don't think is a keystone issue 18:07:04 any one else want to jump on one (if you're not busy with reviews/features)? 18:07:13 crinkle: yeah, thanks for looking into that one 18:07:14 the mysql vs. postgres sounds like an interesting one, but is it keystone specific? 18:07:29 dstanek: see crinkle's comment ^ :) 18:07:30 dstanek: i don't believe so, i saw the same thing running the nova tests 18:07:30 "MySQL tests significantly slower than PostgreSQL tests" so we are going to swtich to Postgres? 18:07:42 crinkle: should it be marked as incomplete or invalid? 18:07:57 MySQL is a bug? 18:08:04 it's possible that the live tests are doing something wrong in the setup 18:08:06 dolphm: maybe incomplete for now? i'd like to see if clarkb has any other comments on it 18:08:07 crinkle: could it be a misuse of certain features in our code or is it truly the db? 18:08:14 like not re-using the database or something 18:08:29 i wouldn't mark it as invalid until we know why it's a problem 18:08:34 dolphm: probably yes but i'm not sure which project it should be filed under 18:08:56 dstanek: i think it's the db connector engine 18:09:03 bknudson: good point 18:09:06 crinkle: oslo.db and nova would be good starts 18:09:15 since we know nova is also affected 18:09:37 anyone know Kam from bug 1604479 ? 18:09:37 bug 1604479 in OpenStack Identity (keystone) "tenantId/default_project_id missing on Keystone service user in Mitaka" [Critical,New] https://launchpad.net/bugs/1604479 - Assigned to Kam Nasim (knasim-wrs) 18:09:40 crinkle: can you document in the bug how you came to that conclusion so i can duplicate? 18:09:47 dstanek: i did 18:09:57 crinkle: cool, thanks. i'll take a look today then 18:10:29 stevemar: so I conversed with him a bit on that 18:10:54 henrynash, is that bug reproducible? 18:10:56 stevemar: I think the issues is that the services users weren't created with default projects 18:11:53 crinkle, great investigation, btw 18:12:02 rodrigods: ty 18:12:02 stevemar, gyee: and (perhaps) with how we now map v2 from v3, there is a comment saying don;t but tennatID attribute in the entity if the value of devault projet is None 18:12:43 henrynash, we don't touch the default_project_id, even with local user migration 18:12:50 rderose: have you looked at that bug? 18:13:05 is it possible that the "user" table is conflicting with something in Postgresql? 18:13:09 henrynash: if the service wasn't created with a default project that's fine, but if he had data there then it should still work and have been migrated 18:13:20 gyee: I don't think there is anything wrong with our migration 18:13:29 ayoung: we've always had a user table 18:13:29 that bug report looks suspiciously like postgres is executing a view when doing select * from user; 18:13:39 I am very confused by the bug description, especially comment #8 18:14:02 dolphm, I know, and I think that user is a reserved word in postgres. 18:14:04 all, we don't have to diagnose them all *here* :) 18:14:08 dolphm: I did briefly, but as henrynash says, I think the issue was that the services weren't created with a default project 18:14:11 it was more of a call to action :) 18:14:12 ayoung: yeah, the final query makes no sense in the context of a keystone db 18:14:26 default_project_id remained in the user table; wasn't moved 18:14:28 rderose: they should still have the attribute in the API though 18:14:59 dolphm: true, I was going to look at it further, but already assigned to Kam, who said he had a solution 18:15:08 i can get jamie to diagnose the last one since it's probably middleware or auth related :P 18:15:10 dolphm: was waiting on him 18:15:25 gyee, stevemar: I'll drive resolution on https://bugs.launchpad.net/keystone/+bug/1604479 18:15:25 Launchpad bug 1604479 in OpenStack Identity (keystone) "tenantId/default_project_id missing on Keystone service user in Mitaka" [Critical,New] - Assigned to Kam Nasim (knasim-wrs) 18:15:25 ++ 18:15:29 rderose: kam seems afk, don't wait too long if it's a critical bug 18:15:38 rderose ^ thanks henrynash 18:15:47 rderose: you got a lot on your plate already :P 18:15:56 stevemar :) you keep saying that 18:16:02 henrynash, go get'em tiger! 18:16:03 and you keep adding to it 18:16:14 rderose: glutton for punishment! 18:16:21 / join #plugdj 18:16:35 dstanek is a DJ in his spare time 18:16:35 lol, sorry! 18:16:37 dstanek is bored. 18:16:41 bknudson: ++ 18:16:59 stevemar: call me what you want, but at least I'm not a cheater! 18:17:07 lol 18:17:08 shhh 18:17:11 lol 18:17:12 #topic keep plugging away with code reviews for new features 18:17:20 #link https://launchpad.net/keystone/+milestone/newton-3 18:17:21 rderose: are you sure? 18:17:30 dstanek: not really 18:17:36 most of the BPs are in good shape 18:17:58 anyone want to deprecate something in newton before i mark the BP as implemented? 18:18:03 when is n3 happening ? just to state it here in the meeting 18:18:12 stevemar: ^ 18:18:25 R-5 18:18:25 Aug 29-02 R-5 newton-3 milestone 18:18:35 Aug 29 - Sept 02 18:18:37 pff, still a month to go 18:18:50 we also need to add henrynash's migrate complete stuff 18:18:53 stevemar: lbragstad thanks! 18:18:57 breton: it'll fly by :) 18:18:58 stevemar: nothing specific that i want to deprecate, but i'd be more than happy to review deprecations if we have them 18:19:16 dstanek: all existing ones are merged 18:19:22 i'll mark it as complete 18:19:34 deprecate the mongo dogpile backend if we haven't already 18:19:43 so if you're looking for stuff to review, those are always available 18:19:50 gyee: that'll be an oslo change, bug them 18:20:07 stevemar, sure 18:20:25 #topic one last review for migrate complete 18:20:33 i think this is ready: https://review.openstack.org/#/c/337680/ 18:20:35 gyee: i think ours can be deleted now 18:20:47 dstanek, yes 18:20:54 stevemar, I just posted a comment on the spec 18:21:16 dstanek: it's just an entry point, check the existing warning, i dont remember when it was changed 18:21:22 xek: what was the comment? 18:21:32 nooo -1 18:21:35 stevemar: deprecated in M and to be removed in +1 18:21:56 dstanek: there ya go, reference removed-as-of-newton 18:22:04 stevemar, I think there is a problem, where some instances can read outdated data in the scenario outlined in the spec 18:22:19 henrynash: xek has some comments 18:22:54 xex: ok, I'll look 18:23:25 xek: yes, that is a good point you brought up 18:23:28 adding a step in between will fix it, I hope it doesn't get too complicated 18:23:52 xek: would you consider your comment a stopper or can we start implementation now (i think we can based on my understand of it) 18:24:45 that should be a minor update in the spec 18:24:52 samueldmq, ++ 18:24:56 and just adpt it in the implementation 18:24:58 adopt 18:25:16 I think we can start implementing it, the spec is somewhat high level, the devils are always in the details 18:25:29 :) 18:25:46 xek: always! 18:25:49 okay, unless someone has something against the idea now, i'll approve the next revision 18:25:49 basically that means that release n will read and write from both columns and only in release n+1 can we actually just use the new column? 18:25:52 but this is conceptual, I believe the spec could be easily updated 18:26:07 henrynash: ^ just fixing a sentence/paragraph? :) 18:26:32 having to keep columns around for an entire release isn't going to work for us since we're releasing from master. 18:26:34 stevemar: want us to merge it this week right ? 18:26:37 dstanek, it can still be done in one release, but there would be to changes of the configuration / data compatibility mode 18:26:38 * ayoung has to check out soon (double booked on meetings) but will leave client up and will field any questions directed at /me in #openstack-keystone 18:26:42 stevemar, xek: and of course we don't support on-the-fly migration in N anyway, but it woul dbe good to get the sequence correct 18:26:46 *two changes 18:27:12 xek: that means that all keystones would need to get the new configuration at the exact same time right? 18:27:13 xek henrynash thanks 18:27:46 bknudson: I am absolutely goingto make sure this lets someone run close to master and still do rolling upgrades 18:27:48 dstanek, if there is that additional step of writing to both columns and reading from the new one, then no 18:27:50 dstanek: good point. update config/restart service consistently 18:27:53 samueldmq: not so sure it's just a sentence change. i at least want to look at the spec again with this new information 18:28:34 dstanek: okay. agree it's worth it to take a look at it again as a whole to make sure it looks correct 18:28:50 who is going to make the update? 18:29:09 dstanek: henrynash volunteered 18:29:21 stevemar: yep, down to me 18:29:34 dstanek: give it a once over when it's updated 18:29:34 coolio... henrynash can you kick me when you get it done so i can have a peek 18:29:55 may i recommend a poke rather than a kick 18:30:02 dstanek: absolutely 18:30:03 hah 18:30:20 #topic MFA = password + TOTP 18:30:25 this is a fun topic... 18:30:26 (poke with toe extended) 18:30:39 gyee: you're up 18:30:43 I like the MFA patch, an elegant solution 18:30:56 problem is how do we make it backward compatible 18:31:09 we have two options as commented in the review 18:31:15 it's opt-in, right? 18:31:30 bknudson, opt-in can be done in two different ways 18:31:38 1) do not set a totp credential 18:31:40 bknudson: sort of? 18:31:54 2) explicitly configure the passwordtotp plugin 18:32:17 I like 2) for a number of reasons 18:32:48 1) we save an extra roundtrip to the backend for credential lookup 18:32:54 gyee: i think it should be it's own incase the authenticator gets broken or some nonsense 18:32:54 configuring passwordtotp means that using methods: ["password"] does password:totp? 18:33:03 i also like #2, but i would make it MFAPlugin so you can configure it to use any other plugins, not just password and totp 18:33:27 or do I have to do methods: ["passwordtotp"] 18:33:37 bknudson, just "password" 18:33:42 so we don't have to change the clients 18:33:53 no need for another plugin on the client side 18:34:02 but then every user has to do totp, right? 18:34:03 not changing the clients would make adoption easier 18:34:06 like service users? 18:34:07 bknudson: nope 18:34:10 well - don't the clients have to append the totp? 18:34:31 lbragstad: yes 18:34:33 bknudson: there will be a check to see if the authenticated user *has* any TOTP credentials 18:34:39 so there is a client change required 18:34:43 bknudson, no, it is teeing off on the credential lookup 18:34:55 bknudson: if no credentials, then regular password 18:34:57 oh, it still does credential lookup 18:35:01 bknudson: if you have a totp secret it would force you to use it 18:35:04 only users have a totp credential will be participate in totp 18:35:09 if it has credentials, then it'll do password then passcode (totp) 18:35:26 the credential lookup needs to be optional otherwise that's going to slow down token issue. 18:35:41 bknudson, right, that's why I suggested option #2 18:35:44 thats another aspect 18:35:51 make the plugin explicit 18:35:56 gyee: okay, i think we need to go with #2 18:36:04 it'll be more "opt-in" like that 18:36:10 stevemar, alllrighty then 18:36:13 we'll need to create a new auth plugin :( 18:36:34 stevemar, no, even #2 does not require client-side changes 18:36:37 is that a bad thing? 18:36:47 gyee: how so? 18:36:51 just keystone.conf changes 18:36:59 lbragstad: would be nice to be able to do it out of box 18:37:00 password = PasswordTOTP 18:37:14 gyee: that'll make it for the whole cloud 18:37:16 is there going to be a keystoneauth plugin, also? 18:37:18 gyee the clients need to know how to append the totp to the password 18:37:31 stevemar, no, still toggle on the totp credential 18:37:43 i thought in order for people to use the new feature, they'd need a new client that supports totp 18:38:02 lbragstad: check the patch, you'll see why 18:38:13 lbragstad, no, on the client side, just + 18:38:36 bknudson, no client-side changes needed 18:38:41 oh - so the user just passes that *as* the password and the plugin doesn't concatenate them... 18:38:52 right 18:38:55 gotit 18:39:07 the magic happens at the server side 18:39:14 so with this in play users will have to do something different than they do now and we should document that 18:39:28 the password keystoneauth plugin is going to be useless, it will try to refresh the token with the old totp value. 18:39:28 dstanek, we should document the behavior 18:39:45 dstanek: only if they have totp credentials 18:39:51 bknudson yeah - that would be a problem 18:39:55 bknudson, yes, token refresh will be a bit problematic 18:40:04 eek 18:40:06 is there a plan for how keystoneauth will reup tokens using this? 18:40:14 but for MFA, we never have a good solution for token refresh anyway 18:40:17 token refresh will have to be user initiated... 18:40:19 bknudson great catch! 18:40:26 stevemar: agreed. as a user though it may be confusing? 18:40:44 if the auth plugin had the key it could generate a new totp value. 18:40:51 well - token refresh and automation in general gets harder with more than one factor 18:40:57 remember, totp is not really intended for non-interactive users 18:41:31 this is all ocata work anyway 18:41:46 glad we're having the discussion early though 18:41:53 stevemar, we can't sneak it in? :( 18:41:56 how will this affect other libraries like horizon auth lib ? 18:41:59 gyee: noooo! 18:42:09 gyee, what could go wrong??? 18:42:18 nothing could go wrong 18:42:23 gyee: not a chance :P 18:42:32 death or glory gyee in the house 18:42:37 it works, guaranteed 18:42:45 gyee: unless you've proven that you've thought of all the edge cases 18:42:52 (famous last words)? 18:42:56 you can deploy the plugin internallly and tell us how it goes. 18:42:56 hehe 18:42:57 if I had a dime everytime I heard that... 18:43:00 I only use the 80/20 rule 18:43:03 when it comes to design 18:43:17 gyee: just upgrade weekly like bknudson does 18:43:25 ++ 18:43:37 gyee: next topic? 18:43:44 that's all I have 18:43:50 80% this should work, 20% close enough 18:43:50 we didn't end on anything here 18:44:04 smh 18:44:07 #topic should PCI-DSS lockout include LDAP 18:44:10 rderose: ^ 18:44:11 I thought we are going with #2 18:44:18 The lockout feature locks out a user after x number of failed auth attempts. Currently, it's specific to the SQL backend driver for identity. However, we've had a request to include LDAP as well. 18:44:25 The idea being that a user is locked out of keystone, but not locked out of all corporate systems. 18:44:35 LDAP services typically do have a lockout policy, but doesn't allow you to only lockout a specific application. 18:44:40 rderose: selfishly, this is not my problem :P 18:44:48 exactly ;) 18:44:51 rderose: how hard is it to make it an option? 18:44:52 I'm on the side of not adding LDAP, as the LDAP lockout is expected behavior, but want to hear your thoughts. 18:44:55 I can see some wanting and some not 18:44:59 gyee: stevemar: yes, #2 with docs on how users must change, token refresh issues and other corner cases 18:45:02 shaleh: not hard 18:45:12 dstanek, amen, brother! 18:45:14 it was stakeholder requested by AT&T 18:45:29 rderose: I have worked with real hard cases that insist on locking early and often :-) 18:45:29 so its causing real pain 18:45:31 dstanek: gyee theres a spec out there for the mfa stuff, comment on the spec 18:45:36 oh, I thought this was moving it into the manager and enforcing there. 18:45:48 stevemar, yes will do 18:45:50 I would be fine if this was configurable in the ldap backend. 18:46:13 I think an option, disabled by default is the right path. 18:46:18 bknudson: elaborate? 18:46:32 topol: so they wanted a lockout shorter in openstack that didn't lockout a user from the corporate system? 18:46:34 I thought it was going to be a global config option so enforced on every backend 18:46:35 lockout is done at shadow user right? 18:46:36 I believe the issues is through OpenStack/Keystone folks could repeatedly lock out LDAP. 18:46:37 if it"s in the manager, maybe have a list of backends it apply ? 18:46:42 but if it was a config per backend I'd be fine with that. 18:46:44 is that the whole point for shadow users, consistency? 18:46:49 dstanek yes! 18:46:52 bknudson meaning that if you were to deploy an ldap backend you could opt into using the lockout provided by ldap, or choose a more strict lockout through keystone config 18:47:04 topol: isn't that why corporate locks exist? 18:47:16 keystone lockout ok, LDAP lockout bit corp headach 18:47:54 must have been happening too much initiated from OpenStack 18:47:54 it's no different then the user using the wrong password (or getting attacked) on their email...i'm surprised they care 18:47:57 lockout is a *keystone* policy 18:48:00 lbragstad: yes, I agree with that. 18:48:07 the ldap server may already have a lockout configured 18:48:35 lbragstad bknudson: in that case, why not just enforce lockout for SQL and LDAP 18:48:36 we can go back and ask Tan from AT&T to explain the diff 18:48:41 topol: so our internal messaging system relies on our LDAP, if i log into that thing 10x i'll be locked out of my corporate credentials too. 18:48:45 right - so the only case that would make sense would be to set keystone's lockout to a smaller value than the corp system 18:48:46 topol: we don't change it there 18:48:54 otherwise its going to be confusing user experience 18:48:55 topol: did they publish their usecase? 18:49:08 lbragstad: the LDAP corpaorte lockout will happen on not depending on the config of LDAP, we can't opt in/out of that....we could only augment with a keystone if we chose to allow you to opt in to that 18:49:10 dstanek. dont think so. just discussed at midcycle 18:49:16 lamt: did the login happen with the CLI or horizon? 18:49:24 henrynash right 18:49:25 CLI 18:49:39 maybe this is an issue with branding or lack of docs about what credentials to use? 18:50:00 the use case we have, not published, is that using keystone, we can send incorrect credential to LDAP, so the LDAP lockout policy kicks in 18:50:11 and locks the users out of the corporate LDAP 18:50:12 but ... if i'm logging into something with "stevemar@ca.ibm.com" << i *know* i better use my corporate password 18:50:23 lamt: do you'd have a more stick keystone lockout policy? 18:50:41 henrynash lbragstad: all PCI is opt-in, but don't think we need a separate config for SQL and LDAP 18:50:58 this is outside of PCI 18:50:58 for instance, ldap locks after 5 tries and keystone after 4... 18:51:09 being more strict or not doesn't make much sense... if you allow 10 for keystone and 11 for ldap there might be 5 invalid ldap attempts already from another app. 18:51:43 bknudson: yep, it only helps you in the case where there are no pending bad logins 18:52:16 it is true that there may be bad pending logins from other application 18:52:35 this feels a little like the PCI implementation from SQL is bleeding into LDAP 18:52:50 lbragstad: thing is, it's independent of pci 18:53:03 for PCI though, LDAP has a lockout policy, whereas SQL doesn't currently 18:53:04 account lockouts are pci, right? 18:53:05 if i setup an ldap now, i can toast my corporate credentials 18:53:06 if your ldap server already has lockout policy, don't know why you want one in keystone too 18:53:07 btw, do we show useful messages in keystone side if there is a ldap-side lockout ? 18:53:24 lbragstad: right 18:53:27 stevemar: ...but is it? we only implemented this is get SQL update for PCI compliance 18:53:38 PCI-DSS 8.1.6: Limit repeated access attempts by locking out the user 18:53:39 ID after not more than 6 attempts. 18:53:40 samueldmq: probably not 18:53:47 samueldmq: ++ good question 18:54:11 samueldmq: at best we can show the message we got and i have no idea what that would be 18:54:19 honor the ldap policy and expose the proper error messages IMO 18:54:30 PCI is application-specific, not backend-specific right? 18:54:31 browne: ++ 18:54:38 question is whether we support pci in keystone regardless backend vs backend specific (sql) 18:54:42 browne: ++ agreed 18:54:45 what was our primary decision? 18:54:47 there's no question about honoring the ldap policy. there's no way to override it. 18:54:55 samueldmq: we stated that its SQL only 18:55:01 dstanek: ++ 18:55:17 the main reason for the PCI work was so that keystone would have a somewhat PCI compliant backend out of the box 18:55:18 samueldmq: we support being PCI compliant, but the organization still have work to do. nothing is compliant out of the box 18:55:26 stevemar: got it. who has the usecase? perhaps we should hear more why it is needed. and if it really is 18:55:27 although I think we might have had problems with honoring ldap policy in the past when cn=admin was used. 18:55:41 bknudson: well, i can imagine wonky states where keystone user is locked but the ldap user backing it was already unlocked 18:55:44 lbragstad: right 18:55:50 dstanek: ++ 18:56:02 browne: right 18:56:14 how do you unlock the locked ldap user? 18:56:20 Hey sorry in a physical meeting too. the issue was raised that keystone could be used to query ldap and lockout users 18:56:27 unless there is a clear usecase i think we should move on.... 18:56:31 also, we'd have to store ldap user data in sql. 18:56:37 bknudson doughnut bribe admin 18:56:37 bknudson: making a phone call to IT and complaining 18:56:39 dstanek: ++ 18:56:45 bknudson you'd have to contact someone on the corp ldap team, or something like that 18:56:56 jaugustine_: yes, that's correct 18:57:00 jaugustine_: so keytone lockout users in ldap (and maybe affecting other apps ) ? 18:57:07 I mean when the ldap user is locked in keystone. 18:57:09 if we went down this route, the ldap user could be locked out of keystone, and their corp credentials; then the user would have to contact the LDAP admin to be unlocked; then the openstack admin to be enabled again 18:57:17 bknudson, go to the self-service portal and answer a few personal questions 18:57:22 bknudson, I'll send you a link 18:57:26 what's my favorite color? 18:57:26 gyee: lol 18:57:31 or sea animal? 18:57:39 turtle!! 18:57:44 stevemar: it's back because you can be locked out of either or both 18:57:47