17:59:53 #startmeeting keystone 17:59:54 Meeting started Tue Aug 2 17:59:53 2016 UTC and is due to finish in 60 minutes. The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:59:56 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:59:57 ping ajayaa, amakarov, ayoung, breton, browne, crinkle, claudiub, davechen, david8hu, dolphm, dstanek, edmondsw, gagehugo, gyee, henrynash, hogepodge, htruta, jamielennox, jaugustine, joesavak, jorge_munoz, knikolla, lbragstad, MaxPC, morgan, nkinder, notmorgan, raildo, rodrigods, rderose, roxanaghe, samleon, samueldmq, shaleh, stevemar, tsymanczyk, topol, vivekd, wanghong, xek 17:59:58 hi 17:59:58 The meeting name has been set to 'keystone' 17:59:59 o/ 18:00:01 o/ 18:00:02 \o 18:00:03 o/ 18:00:03 o/ 18:00:03 o/ 18:00:05 hello 18:00:08 o\ 18:00:13 \\o// 18:00:13 o/ 18:00:22 dolphm, rderose and henrynash gotta be different 18:00:24 o/ 18:00:31 stevemar, hey, cascade deletion is useful! 18:00:32 :) 18:00:33 just kidding 18:00:33 o/ 18:00:39 we're missing gyee and samuel today 18:00:49 stevemar: someone has to wave back 18:00:49 (practicing my yoga) 18:00:55 stevemar: we can't all just wave in the same direction 18:00:57 henrynash: lol 18:01:12 henrynash: kudo to you if you can still get into \\o// 18:01:23 agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting 18:01:29 zero chance...I'm the entertainment for the class 18:01:41 hehe 18:01:52 o/ 18:01:55 ////o\\\\ 18:01:58 wait a minute for the rest of the folks to trickle in 18:02:04 \o 18:02:28 breton: those spiders really get me 18:02:29 Oyez Oyez 18:02:33 * stevemar avoids topol because he's gonna ask about our openstackSV presentation and if i'm done the slides... 18:02:41 o/ 18:02:54 stevemar!!!!! Im comin for ya 18:02:55 we got critical mass, lets go! 18:03:03 #topic release status 18:03:18 we've got a few weeks to go til newton-3! http://releases.openstack.org/newton/schedule.html 18:03:22 #link http://releases.openstack.org/newton/schedule.html 18:03:41 we've got a bunch of bugs and blueprints, see https://launchpad.net/keystone/+milestone/newton-3 18:03:52 good news is almost everything has patches up 18:04:04 so we're in decent shape 18:04:33 and almost all patches associated with those blueprints and bugs have been seeing good reviews 18:04:42 so thanks everyone for that 18:04:53 +2 18:05:06 just gotta keep it up for another 4 weeks til newton-3 and we're smooth sailing til barcelona (tapas on me) 18:05:26 * topol wish our OpenStackSV prez was in as good shape :-) 18:05:37 OK that was good 18:05:41 :) 18:05:58 dngit we need the OSC client support for DSRs 18:06:19 so if you're looking for things to review, pick a bug or BP from https://launchpad.net/keystone/+milestone/newton-3 and look for the gerrit link 18:06:27 ayoung: henrynash is working on that 18:06:35 i'm done my spiel 18:06:45 ayoung: DSR patch for osc is in final review 18:06:54 thanks again for the reviews and triaging everyone! 18:06:57 #topic fix cache invalidation 18:07:00 #link https://bugs.launchpad.net/keystone/+bug/1607553 18:07:00 Launchpad bug 1607553 in OpenStack Identity (keystone) "Revocation event caching is broken across processes" [High,New] 18:07:03 breton, lbragstad, dstanek ^ 18:07:19 hey 18:07:24 i and lbragstad ran into the same issue independently: https://bugs.launchpad.net/keystone/+bug/1607553 18:07:25 ok - I spent a good portion of friday trying to figure out why we can't switch to fernet 18:07:28 this is a fun one 18:07:38 so just looking at the review comments "this is a hack" and "this doesn't work" 18:07:43 patch for it doesn't fully work, but because of a technical issue 18:07:46 not sure what the point is of discussing it. 18:07:58 the bug is a blocker for making fernet the default 18:08:01 bknudson: slow down turbo 18:08:13 the patch we are talking about is https://review.openstack.org/#/c/327885/ . 18:08:24 I think we need to figure out what our best option is for moving forward so that we can invalidate cache regions across processes 18:08:28 the problem with it is that it *will* break with the new release of dogpile.cache 18:08:40 because it relies on internal interfaces of dogpile.cache 18:08:45 so -2 18:08:57 but using internal interfaces looks like the only way in stable dogpile.cache 18:09:01 breton: i think that my version won't once it's working 18:09:17 bknudson: OTOH it's the only solution for current stable/* branches 18:09:18 the fix for the bug also needs to be backported to stable/mitaka 18:09:20 does dogpile.cache not accept code submissions 18:09:47 bknudson they do - amakarov added support for region invalidation across processes 18:09:52 bknudson: the problem is (and according to dolphm we can push back) the policy for bumping version numbers in stable branches 18:10:01 #link https://review.openstack.org/#/c/349704/ this is dstanek's approach to fixing the problem 18:10:09 dstanek: have you and lbragstad tested it against fernet patches? 18:10:10 i see amakarov's commit here: https://bitbucket.org/zzzeek/dogpile.cache/commits/all 18:10:17 i think this is a solved problem there and ideally that's what i'd like to use 18:10:28 lbragstad: but I needed to ask zzzeek for permission - it's not automatic 18:10:49 breton: i get similar results. less test failures, but i was still getting some 18:11:19 dstanek: is it backportable to stable/mitaka? 18:11:30 breton: should be, but i didn't try it 18:11:40 stevemar: yes, and with this update it's possible to solve the issue in oslo 18:11:43 but i'd rather push forward 18:12:16 dogpile is currently capped at 0.5.7 in stable/mitaka 18:12:37 stevemar: exactly 18:12:49 stevemar: can we merge one fix for stable/mitaka and completely different to master? 18:12:54 amakarov: what would the new version of dogpile be released as? 18:13:04 http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt?h=stable/mitaka#n41 18:13:10 breton: you ... can, it's not ideal, but it's possible 18:13:14 according to this dogpile.cache is not capped. 18:13:46 bknudson: https://github.com/openstack/requirements/blob/stable/mitaka/upper-constraints.txt#L118 18:13:49 I suggest using breton's patch and cap dogpile.cache to the version prior to my patch 18:14:04 that's not a cap that's just what is used for testing. 18:14:36 the one after 0.5.7 is 0.6.0 18:14:39 if the min version is updated then that means every deployer/packager needs to upgrade. 18:14:43 amakarov: i'd want to immediately follow up with the right way to do it in master 18:14:54 dstanek: me too 18:14:57 bknudson: yes, though if we miss dogpile.cache release with my patch with breton's on merged, we'll be in trouble ) 18:14:58 bknudson: ? http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt?h=stable/mitaka#n118 18:15:01 dstanek: but we need to do something with mitaka 18:15:08 dstanek breton but are we sure the propose path actually fixes the race conditions? 18:15:23 upper-constraints is what the community uses for testing. It's not a cap. 18:15:23 lbragstad: lets leave technical issue aside 18:15:36 dstanek: right now breton follows it ) 18:15:39 breton: right, that's why i said we should immediately fix master 18:15:54 lbragstad: i thing we can figure it out in reviews/#openstack-keystone 18:15:54 lbragstad: i don't think this fixes the race at all 18:15:55 sounds like a consensus 18:16:13 bknudson: but that's what'll be laid down when the dsvm tests run 18:16:21 packagers are not likely going to repackage dogpile 0.6.0 just for us (potentially breaking other consumers), especially if it contains a breaking API change 18:16:33 unspoken alternative is to recomment revocation caching be turned off 18:16:38 dolphm: good point! 18:16:43 dolphm: it doesn't contain breaking api change 18:16:47 dstanek: so i might have to take back the idea to bump the requirement 18:16:54 dolphm: the patch relies on private stuff :( 18:16:56 dstanek at least until it works properly 18:17:02 dolphm: i don't know that it contains a breaking api change 18:17:18 oh, only we would break if we upgraded, then? 18:17:27 right - it breaks because we rely on internal interfaces of dogpile 18:17:30 dolphm: out hacks are monkey patching protected variables 18:17:31 dolphm: yes 18:17:40 can dogpile.cache do a fix release 0.5.8 ? 18:17:41 s/out/our/ 18:17:43 probably not. 18:17:44 why would dogpile require a 0.6.0 release instead of a 0.5.8 release, then? because this is perceived as a feature? 18:18:01 amakarov ^ ? 18:18:08 breton: the 0.6.0 release had an api change, they moved things around IIRC 18:18:20 stevemar: oh, ok. 18:18:26 zzz... is not here 18:18:33 stevemar: unrelated to this fix? 18:18:34 stevemar: really?j 18:18:49 dolphm: don't remember exactly if the patch has a bug specified ( 18:19:06 ok, since dstanek has a patch, here is what i suggest 18:19:24 1. Try his patch out, it doesn't rely on protected variables 18:19:39 2. if it works, merge, backport, code using amakarov's new feature 18:19:39 #link https://review.openstack.org/#/c/349704/ 18:19:45 dolphm: i thought so? https://bitbucket.org/zzzeek/dogpile.cache/commits/761dc0a9e4c08f9af4b732ced3604e72d74f09af 18:19:52 #link https://bitbucket.org/zzzeek/dogpile.cache/issues/38/invalidate-entire-cache 18:20:02 3. if not, figure out at the next meeting 18:20:03 oops... it's a bugfix 18:20:28 what do you think? 18:20:32 breton: i still need to figure out if the transients are my fault or not 18:20:35 breton that path will require us to bump the version of dogpile for the backport - right? 18:20:42 that's the case for both patches i guess 18:20:46 lbragstad: dstanek's? No. 18:21:02 breton: lbragstad: no version bump needed for mine 18:21:11 breton: #2 "code using amakarov's new feature" << whats that mean? 18:21:18 stevemar: ++ 18:21:27 consume it in master, maybe? 18:21:33 stevemar: ^ > i see amakarov's commit here 18:21:46 stevemar: the one in dogpile.cache 18:22:10 stevemar: https://gerrit.sqlalchemy.org/#/c/108/ 18:22:10 #link https://bitbucket.org/zzzeek/dogpile.cache/commits/d521db7eb4eab4ad2bacf5f1f5fd89bd3e43a0fa 18:22:20 breton: hmm, his dogpile commit, has it been released yet? 18:22:37 stevemar: it is not release yet, but i guess it will soon be 18:22:48 stevemar: and when it is, "code using amakarov's new feature". 18:22:56 ohhhh okay 18:23:04 when you backport to mitaka you can have the code check for the right version of dogpile.cache before using the new features. 18:23:16 i get it 18:23:21 breton: sounds like a plan 18:23:28 bknudson: dstanek's patch won't need any version checking 18:23:43 nice work on chasing this down 18:24:04 ok, so concensus is breton, lbragstad and i will work on making sure my patch works and we'll get that merged and backported 18:24:06 ? 18:24:07 merge dstaneks patch in master and mitaka, then using amakarov's new feature when it's released in master 18:24:09 #action breton, dstanek, lbragstad: check if patch by dstanek works not worse than https://review.openstack.org/#/c/327885/ 18:24:22 ++ sounds good 18:24:24 #action breton, dstanek, lbragstad: check if patch by dstanek works not worse than https://review.openstack.org/#/c/327885/ 18:24:31 sweet 18:24:37 i was expecting some bot feedback *shrugs* 18:24:42 great 18:24:54 thanks breton, lbragstad and dstanek 18:25:00 thanks for the time 18:25:06 np 18:25:12 #topic Validating trust-scoped tokens with v2.0 API 18:25:16 breton you're up again 18:25:20 a trust-scoped token can be obtained via v2.0 API 18:25:30 why cannot it be validated against v2.0 API? 18:25:58 maybe it was just overlooked? is this by design ayoung? 18:26:08 I believe that was something that changed when we consolidated the fernet token provider into the common provider 18:26:09 good point: trust doesn't scope to domains 18:26:31 ./test_auth.py heavily relies on getting trusts via v2.0 18:26:54 breaking that is still a backwards non-compat change until v2.0 goes away 18:27:06 I worked on getting a patch up for supporting trust auth/validate for v2.0 18:27:26 #link https://review.openstack.org/#/c/278693/ 18:27:29 lbragstad: for fernet or for all types of token? 18:27:39 breton specifically for fernet 18:27:44 is this a fernet only problem? 18:27:48 jamielennox: no 18:27:55 jamielennox: initially i found it for uuid 18:27:57 i'm sure there is trust code in v2 18:27:57 jamielennox nope 18:29:31 so this should be working with any token provider 18:29:49 breton: file a bug and let's get it fixed 18:30:07 breton we can probably fix it with https://review.openstack.org/#/c/278693/ (once it's revived) 18:30:41 https://review.openstack.org/#/c/278693/ says you can validate trust with v2 18:30:41 ok 18:30:59 uuid 18:31:31 breton: file a bug for it, currently https://review.openstack.org/#/c/278693/ is missing one :\ 18:31:40 bknudson: pki 18:32:00 pki what? 18:32:08 https://review.openstack.org/#/c/278693/ says that uuid works 18:32:39 bknudson: ah damn, it does eh 18:32:48 but breton found the bug with uuid 18:33:30 ok, i will file a bug 18:33:31 there's a contradiction here. 18:33:45 breton: yeah, let's start there, we need more info on this one 18:34:03 i guess the only good part is that no one has noticed this is broken 18:34:23 next topic 18:34:28 #topic Rolling upgrades 18:34:32 henrynash: you're up 18:34:41 ok 18:34:49 i'm just about ready to +A this spec, but if anyone else wants to chime in, now is the time 18:34:59 so that was first thing.... 18:35:17 any other comments? (dtsanek, thanks for yours) 18:35:34 spec is here: https://review.openstack.org/#/c/337680/ 18:35:54 3 patches are up for implementation, starting at: https://review.openstack.org/#/c/349703/2 18:35:58 dolphm: ayoung lbragstad ? 18:36:12 and it would land for newton 18:36:51 stevemar I haven't weighed in on it yet 18:36:59 some of the implementation is still WIP, but it IS functional, and I am closing out some corner cases, adding more testing and then add docs 18:37:09 henrynash: the only think i didn't quite get was the steps that appeared to be duplicated 18:37:45 stevemar: i was pretty happy with our discussion at the midcycle, only had a few nits on the spec itself that i don't care that much about 18:37:51 dstanek: so I added a comment on that...it is to prevent problems once we have on-the-fly migrations 18:38:12 dtsanek: but happy to followup with clarifications 18:38:33 okay i'll give lbragstad til EOD then punt it through 18:38:52 henrynash: keep workin on the code ;) 18:39:05 stevemar I might not get around to it today - but I trust the judgement of others who have weighed in on it 18:39:05 stevemar: yes sir 18:39:08 henrynash: you even got a +1 from xek 18:39:16 lbragstad: rgr that 18:39:19 stevemar: yep, gonna bank that one 18:39:37 henrynash: turns out mulling this over for a year makes it safe 18:39:43 next topic 18:39:52 stevemar; shockin' that 18:40:08 henrynash: so in step #3 we have all nodes upgraded and only reading/writing from old column. after that you only need to start reading from the new column 18:40:41 #topic MFA 18:40:45 adriant: o/ 18:40:50 sorry for the wait :) 18:40:52 Hey 18:40:56 np :) 18:41:00 so 18:41:05 henrynash: i thought that set #5 would actually stop using the old column altogether 18:41:09 spec: https://review.openstack.org/#/c/345113/ and patch: https://review.openstack.org/#/c/343422/ 18:41:13 I'd like to get MFA working in OpenStack, and easiest way without breaking anything is password+passcode concatenation. 18:41:36 Optional MFA plugin, works like password auth 18:41:44 adriant: does it matter to you if this lands in N or O? 18:41:47 just also checked TOTP if present. 18:42:01 stevemar: N would be better, but can wait 18:42:11 *nods* 18:42:13 stevemar: I can backport for our own purposes easily enough 18:42:14 dstanek: (whct you need to prevent is one node that thinks its Ok to write to just the new column and one thinking it only has to read from teh old column...and since I decided against a spooky entaglement at a distance lock, we need a send phase of reboots, I think) 18:42:29 adriant: so you guys are running something like this already correct? 18:42:33 My main worry is the v2 api 18:42:37 stevemar: Not yet 18:42:57 stevemar: but the goal is to be up and running with MFA in the next 3-6 months 18:43:04 rgr that 18:43:06 i have most of the same concerns i've had in the past once we get in to which projects require what level of 2fa, what happens when they don't ahve a totp key etc 18:43:39 jaminelennox: I've kept per project MFA out of this spec and patch 18:43:56 if no TOTP key, it should act just like password auth 18:43:59 and do federation, but i feel like i've beat that drum a fair bit now 18:44:01 jamielennox: as it's a requirement that would need too much work across all of keystone to make viable 18:44:02 and no per-project support 18:44:21 yep, ok 18:44:37 if a project needs MFA, it is up to the project to enforce all users have it active 18:44:51 most other concerns are negated by the fact we already support a totp of some form 18:45:01 jamielennox: it'll enable MFA support for SQL users, which as we've found out, a lot of people use -- and they won't be moving all those users to federation any time soon 18:45:24 Should also work for LDAP users. 18:46:22 I've suggested in the docs that disabling V2 is part of the suggested method to enable MFA 18:46:38 adriant: can you elaborate why it won't work with v2? 18:46:38 as I'm not sure touching the V2 auth code for this is a good idea 18:47:04 v2 doesn't appear to be pluggable, and unless edited, does not do the TOTP check. 18:47:04 we don't require tempest testing for features? 18:47:51 bknudson, no :( 18:48:00 scary. 18:48:08 stevemar: are there ways to edit the v2 auth without changing the default? 18:48:40 My worry is, a user with a TOTP cred on v3 will need password+passcode 18:48:45 on v2, just password 18:49:03 so v2 is a way around MFA unless v2 also has some MFA changes 18:49:04 adriant: don't deploy v2 18:49:09 adriant: disable v2? 18:49:33 dstanek: dolphm right, that's what adriant recommended 18:49:45 v2 auth won't be going away for a long time 18:50:27 I think if the v2 auth limitation is clearly documented then it's good enough. 18:50:27 why doesn't v2 use the password plugin? 18:50:35 not the v3 one 18:50:57 bknudson: as in, just hardcode it to use the native password plugin? 18:51:00 bknudson: It looks to be it's own separate auth plugin 18:51:18 right, just like v3 auth hardcodes the "external" plugin. 18:51:31 https://github.com/openstack/keystone/blob/9d54eb33c1d74ff39c947af6ff984ef2e0bf4be4/keystone/token/controllers.py#L63 18:51:43 this seems like a rabbit hole. why does v3 hardcode the external plugin? 18:52:25 we could create a v2_password plugin 18:52:59 v3 uses the "external" plugin if there's EXTERNAL_USER or whatever it is. 18:53:03 lol, don't look at auth! 18:53:19 it makes you sad 18:53:28 stevemar: yep, I've looked at how i could edit that v2 auth controller to support MFA, but I didn't want to edit older defaults. 18:53:42 adriant: we could just add credential lookup to the v2 auth code 18:54:09 stevemar: maybe, but it would need to be based on if the v3 auth is using the the "password_totp" plugin 18:54:10 so core team, i'm putting you on the spot here, thoughts on this? y'all are being unusually quiet... 18:54:15 found it: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/auth/controllers.py#n504 18:54:33 my opinion - we need to clean up existing keystone before adding new features. 18:54:39 lbragstad: dstanek bknudson dolphm jamielennox everyone, yay or nay? 18:54:40 otherwise we're just adding technical debt on top of debt. 18:55:11 bknudson, this is something that has being raised a couple of times 18:55:18 and using crappy existing code as a reason to not support v2 is a poor excuse. 18:55:40 bknudson: in my grand scheme, PCI and MFA were the last "features" i wanted to add, then start paying off technical debt 18:55:53 so i'm a bit biased here 18:56:36 stevemar, bknudson, can we track somewhere stuff that is simply not good enough and need to be revisited? i guess the next topic falls into that 18:56:39 stevemar: so what is the proposal...to fix v2? 18:56:45 I'm actually fine with the feature since it's optional. 18:57:07 dstanek: whether or not to add MFA as its designed in the currently spec 18:57:18 although we really need to start requiring tempest testing for features. 18:57:29 i agree with bknudson, i'm concerned about the layers on layers of stuff and how you do things like message that a 2fa cred is required 18:57:38 the whole pipeline through there is a disaster 18:58:03 but honestly the scariest parts of this have already been merged so at this point whatever 18:58:05 adding totp auth support to v2 is not an API change, right? 18:58:11 jamielennox: the alternative is we wait until the S release to include new features :P 18:58:36 adding totp to v2 is as much an api change as adding it to v3. 18:58:46 ++ 18:58:52 dolphm: the problem isn't adding totp auth, it's adding password+totp auth 18:59:04 to the existing password auth 18:59:18 bah, move to -keystone 18:59:23 #endmeeting