18:00:03 <lbragstad> #startmeeting keystone
18:00:04 <openstack> Meeting started Tue Aug  8 18:00:03 2017 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:07 <openstack> The meeting name has been set to 'keystone'
18:00:07 <lbragstad> ping ayoung, breton, cmurphy, dstanek, edmondsw, gagehugo, henrynash, hrybacki, knikolla, lamt, lbragstad, lwanderley, notmorgan, rderose, rodrigods, samueldmq, spilla, aselius, dpar
18:00:17 <lbragstad> #link https://etherpad.openstack.org/p/keystone-weekly-meeting
18:00:19 <rodrigods> o/
18:00:19 <lbragstad> agenda ^
18:00:19 <cmurphy> o/
18:00:21 <gagehugo> o/
18:00:21 <raildo> o/
18:00:22 <samueldmq> hi o/
18:01:01 <knikolla> o/
18:01:35 <lbragstad> #topic announcements
18:01:44 <lbragstad> #info rc-1 is targeted for this week
18:02:10 <lbragstad> #info planning for the PTG should be underway
18:02:37 <lbragstad> i'm going to start going through the etherpad and organizing the items into like categories
18:02:45 <lbragstad> I plan to start doing that as soon as we can cut rc1
18:02:52 <samueldmq> lbragstad: are there any critical/high priority things for rc1?
18:03:00 <samueldmq> i.e bugs
18:03:21 <lbragstad> we do have one release blocker
18:03:33 <lbragstad> https://bugs.launchpad.net/keystone/+bug/1705081
18:03:33 <openstack> Launchpad bug 1705081 in OpenStack Identity (keystone) "DELETE project API is failing in forbidden(403) error message" [High,In progress] - Assigned to Lance Bragstad (lbragstad)
18:03:35 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1705081
18:03:49 <lbragstad> and the fix for that should be gating
18:03:57 <lbragstad> #link https://review.openstack.org/#/c/491546/
18:04:23 <lbragstad> #topic rc1 homestretch
18:04:39 <lbragstad> since we were just talking about it
18:04:41 <lbragstad> #link https://goo.gl/ygTNJF
18:05:15 <samueldmq> lbragstad: what does that filter say?
18:05:17 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1705081 has a patch in review
18:05:17 <openstack> Launchpad bug 1705081 in OpenStack Identity (keystone) "DELETE project API is failing in forbidden(403) error message" [High,In progress] - Assigned to Lance Bragstad (lbragstad)
18:05:43 <lbragstad> samueldmq: that's a list of all bugs that are open and targeted to RC1
18:05:58 <lbragstad> so - it's a list of things we need to complete before we can release a candidate for pike
18:06:07 <samueldmq> ok seems a pretty decent amount of job for the office hours today?
18:06:09 <samueldmq> :)
18:06:24 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1674676 is a doc fix - per dstanek's comment
18:06:24 <openstack> Launchpad bug 1674676 in OpenStack Identity (keystone) "The URL listed against the details of identity resources returns 404 Not Found error" [Medium,Triaged]
18:06:30 <lbragstad> ^ that'd be a good one to get gating today
18:06:50 <lbragstad> anyone feel like picking that up today?
18:06:59 <lbragstad> or submitting a doc patch to close that one?
18:08:16 <cmurphy> *crickets*
18:08:20 <gagehugo> Is that about those relationship links?
18:08:23 <lbragstad> :)
18:08:27 <lbragstad> yes
18:08:28 <cmurphy> i've never really understood those relationship links
18:08:34 <gagehugo> ^
18:08:39 <lbragstad> that's exactly why we need a docs patch
18:08:58 <lbragstad> they are not suppose to resolve - so the point of the patch would be to document why they don't resolve
18:09:20 <lbragstad> essentially consolidating https://bugs.launchpad.net/keystone/+bug/1674676/comments/3
18:09:20 <openstack> Launchpad bug 1674676 in OpenStack Identity (keystone) "The URL listed against the details of identity resources returns 404 Not Found error" [Medium,Triaged]
18:09:20 <sjain_> me too, I tried working on the relationship links, but I'm not able to understand those
18:09:48 <samueldmq> ahaa sjain_ ^
18:09:59 <gagehugo> oh
18:10:07 <samueldmq> oops I'm late. yes that's explaining what those relationship URLs are for. we had that in our TODO
18:11:00 <gagehugo> So we want to change the docs to described what those links really are then
18:11:21 <lbragstad> gagehugo: yeah - according to dstanek, a blanket statement to the api-ref would suffice
18:11:38 <lbragstad> just explaining what they are and why they're in the documentation
18:11:45 <gagehugo> lbragstad I can tackle that then I think, shouldn't be too bad if that's just the case
18:11:47 <samueldmq> that should be in the api-docs
18:11:52 <samueldmq> since that's where those links appear
18:12:03 <lbragstad> #action gagehugo to pickup https://bugs.launchpad.net/keystone/+bug/1674676
18:12:03 <openstack> Launchpad bug 1674676 in OpenStack Identity (keystone) "The URL listed against the details of identity resources returns 404 Not Found error" [Medium,Triaged]
18:12:04 <gagehugo> Put them wherever relationship links are imo
18:12:05 <lbragstad> thanks gagehugo
18:12:24 <lbragstad> next
18:12:26 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1689644
18:12:27 <openstack> Launchpad bug 1689644 in OpenStack Identity (keystone) "Keystone does not report microversion headers" [Medium,In progress] - Assigned to Rohan Arora (ra271w)
18:12:32 <lbragstad> that ^ one has a fix in review
18:12:46 <lbragstad> #link https://review.openstack.org/#/c/468189/
18:13:07 <lbragstad> and it's moving along - it'd be nice to have that gating no later than tomorrow afternoon
18:13:15 <cmurphy> so keystone doesn't report a microversion header - but we also don't have microversions
18:13:25 <gagehugo> lbragstad I'll let those working on that know
18:13:26 <cmurphy> so i was kind of confused about the point of that one
18:13:30 <samueldmq> lbragstad: would be nice to have mordred looking at that review too
18:13:39 <gagehugo> rarora ^
18:13:44 <samueldmq> and cmurphy who has been in sync with some microversion stuff
18:13:46 <knikolla> we support one microversion at a time
18:14:01 <lbragstad> cmurphy: right - so the implementation needs to be able to take that into account
18:14:05 <edmondsw> calling that a microversion is not really accurate
18:14:33 <knikolla> edmondsw: totally agree.
18:15:24 <gagehugo> macroversion?
18:15:41 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1689644/comments/2 sums up the need pretty well
18:15:41 <openstack> Launchpad bug 1689644 in OpenStack Identity (keystone) "Keystone does not report microversion headers" [Medium,In progress] - Assigned to Rohan Arora (ra271w)
18:16:15 <cmurphy> well that comment is saying we should support microversions, that's not what this patch does
18:17:05 <knikolla> cmurphy: not really. that comment is saying we should report a microversion. so the microversion selection code in the client doesn't need a special case for keystone.
18:17:19 <lbragstad> the patch does the bare minimum to allow clients to apply code evenly across openstack services without committing to microversions
18:18:08 <lbragstad> i'm not sure the bug suggests an opinion on microversions one way or another
18:19:00 <edmondsw> lbragstad that comment you referenced says "by not supporting microversions" not "by not returning microversion headers"
18:20:19 <cmurphy> right
18:20:45 <lbragstad> so do we want to pull this from rc1 queue until we discuss the possibility of microversions further?
18:21:48 <samueldmq> I think that is appropriate. We don't support microversions as of today
18:22:07 <lbragstad> because i don't expect to have a productive conversation on microversions until the ptg
18:23:26 <edmondsw> I'm leery of starting to return a header that seems to indicate keystone supports microversions, when we don't
18:23:37 <edmondsw> and wondering if that will bite us down the road
18:23:38 <lbragstad> that's fair
18:23:40 <cmurphy> +1
18:24:36 <lbragstad> the tough part is that the microversion ship has mostly sailed
18:24:45 <lbragstad> we're one of the few projects that haven't adopted it
18:24:55 <lbragstad> but that's a discussion for the PTG
18:25:02 <edmondsw> yep
18:25:05 <lbragstad> pulling it from the queue
18:25:55 <lbragstad> cc morgan ^
18:27:16 <morgan> i'm never going to be a fan nor support microversions
18:27:18 <morgan> i wont block it
18:27:37 <rarora> should I update the patch set's commit message or anything like that?
18:27:53 <morgan> and keystone should NEVER emit a header indicating microversion support unless it does
18:28:23 <morgan> that is a hard -2 on additional data. you can support microversions and indicate it, but you cannot emit data indicating support for it if there is no support
18:28:52 <cmurphy> yep that's the concern
18:29:18 <edmondsw> morgan I'd agree... and then you'll want to drop a -2 on https://review.openstack.org/#/c/468189/
18:29:32 <lbragstad> rarora: placed a procedural -2 on the patch for now so that we are aware of the status
18:29:50 <rarora> lbragstad: ok sounds good
18:30:04 <lbragstad> alright - moving on
18:30:09 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1694525
18:30:09 <openstack> Launchpad bug 1694525 in OpenStack Identity (keystone) "keystone reports 404 User Not Found during grenade tests" [Medium,Triaged]
18:30:34 <lbragstad> clarkb: ^ originally brought that to our attention a while ago and i targeted it to RC1 since it seemed to be affecting the gate quite a bit at the time
18:30:45 <morgan> done
18:31:06 <morgan> edmondsw: commented saying i'd remove the -2 when we support microversions
18:31:12 <lbragstad> the logs from the occurrence have since expired and the logstash query is kind of useless since it returns a lot of false positives
18:31:19 <edmondsw> morgan +!
18:31:21 <clarkb> ya I wasn't sure if it was a regression in upgrades or not as well
18:31:21 <morgan> edmondsw: i wont +2/+1 microversion support but def. wont block implementation
18:31:33 <morgan> (-1/-2 on that basis)
18:31:45 <morgan> clarkb: it is a regression from previous releases as far as we know
18:32:36 <cmurphy> well "user not found" isn't necessarily an indication of a problem
18:32:43 <cmurphy> the way osc is designed it just does that
18:33:05 <lbragstad> it could also be a test asserting something about the state of a user
18:33:27 <clarkb> well in this case it was failing to boot instances because the user being used went away in the upgrade iirc
18:33:41 <clarkb> I'd have to go reread things to remember properly though
18:34:18 <morgan> lbragstad: likely
18:34:50 <lbragstad> clarkb: do you want to do that in -keystone after the meeting?
18:34:59 <lbragstad> we have office hours and it might be a good time to investigate
18:35:08 <lbragstad> if your day is busy though, i understand
18:36:45 <clarkb> ya I'm currently unleaking nova instances in one of our clouds
18:36:56 <clarkb> it turns out nova can do that apparently so not sure if otday is a good day
18:37:10 <lbragstad> clarkb: no worries - ping me if you end up wanting to go through it
18:37:27 <lbragstad> otherwise i can try and find another occurrance of it
18:37:42 <lbragstad> alright - next
18:37:51 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1700852
18:37:51 <openstack> Launchpad bug 1700852 in OpenStack Identity (keystone) "Slow listing projects for user with many role assignments" [Medium,In progress] - Assigned to Lance Bragstad (lbragstad)
18:38:01 <lbragstad> ^ that's a performance issue with effective role assignments
18:38:16 <cmurphy> it seemed like you were waiting for feedback from the bug reporter on that one?
18:38:31 <cmurphy> or do you want to un-wip it and get it in?
18:38:50 <lbragstad> cmurphy: i think the patch i proposed is failing tests
18:38:58 <lbragstad> which i need to investigate
18:39:15 <lbragstad> so - it boils down to this
18:39:29 <lbragstad> the role assignment api for listing effective role assignments is terribly slow
18:39:38 <lbragstad> it's also super dense python logic
18:40:10 <lbragstad> from the perspective of someone who hasn't spent a lot of time in the assignment API - it's pretty confusing
18:40:50 <lbragstad> i suspect the problem is that we're doing some processing in python that is causing performance to degrade - specifically for effective assignemtn
18:41:01 <lbragstad> my knee-jerk reaction was to implement caching for it
18:41:32 <lbragstad> as a way to stop the bleeding until we can get more time to unwind and understand the role assignment API
18:41:56 <lbragstad> and look for ways to improve performance by using/implementing better python
18:42:29 <lbragstad> does anyone object to that approach? or should this wait until queens where we can do that investigation
18:42:42 <samueldmq> so..
18:42:45 <lbragstad> I won't have time to unwind the entire role assignment api by the time rc1 rolls around
18:43:07 <samueldmq> if we go for caching, we need to be really really careful. if we fail to invalidate in a single case (tehre are many)
18:43:08 <cmurphy> caching sounds like a good start but i'm not sure it can get much better than that, it's basically a O(N^M) op because it has to map every user and every project
18:43:21 <samueldmq> we'll be leaking roles to tokens, which is not fun
18:43:30 <lbragstad> samueldmq: that's the other concerns
18:43:32 <lbragstad> cmurphy: right
18:43:45 <lbragstad> cmurphy: it's a lot of processing in python
18:43:59 <lbragstad> with no real good way to offload it
18:44:07 <samueldmq> lbragstad: yes it is. I've worked a lot in that code in the past
18:44:20 <samueldmq> and that was per design, we decided to go and expand users into groups
18:44:35 <samueldmq> and then we have project hierarchies, and then inherited role assingments
18:45:00 <samueldmq> all that logic gets processed in those methods for role assignments, since all those mechanisms are for assigning roles anyways
18:45:23 <lbragstad> yeah
18:45:29 <samueldmq> question is, how bad is it?
18:45:32 * samueldmq looks at the bug
18:45:38 <lbragstad> the performance numbers are in the bug
18:46:00 <lbragstad> i'll see if i can get the patch passing tests and propose it
18:46:22 <lbragstad> if we get too close to rc1 we'll cut the release with out it and backport the fix when we have it
18:46:43 <lbragstad> (we backported a bunch of caching fixes in the mitaka timeframe for the same reasons when we dealt with fernet)
18:47:04 <lbragstad> #action lbragstad to get https://review.openstack.org/#/c/487143/ passing tests
18:47:28 <lbragstad> the next two seems related and have been causing issues in the gate
18:47:36 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1702211
18:47:36 <openstack> Launchpad bug 1702211 in OpenStack Identity (keystone) "test_password_history_not_enforced_in_admin_reset failed in tempest test" [Medium,Confirmed]
18:47:43 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1703917
18:47:43 <openstack> Launchpad bug 1703917 in OpenStack Identity (keystone) "Sometimes test_update_user_password fails with Unauthorized" [Medium,Triaged]
18:48:02 <lbragstad> those mostly need investigation
18:48:19 <knikolla> Spent a day looking at them and came up short.
18:48:23 <morgan> failed in a tempest test?
18:48:36 <lbragstad> same here
18:48:42 <samueldmq> when I see "sometimes" in a bug report, I feel the pain
18:48:52 <lbragstad> seems like a race condition but i couldn't understand why
18:48:53 <samueldmq> lbragstad: remember when we were trying to debug some fernet/tokens stuff
18:49:03 <lbragstad> yeah - those are like that
18:49:24 <cmurphy> i've reproduced the admin_reset one by running it over and over but for some reason not the regular one
18:50:07 <morgan> cmurphy: sounds about par for the course, there was a bug i found acting similar and it required a sleep to repro... the one that you can repro should be the focus
18:50:23 <morgan> the other one *shrug* best we can do is keep an eye on it if we can repro it
18:50:31 <lbragstad> yeah
18:50:42 <cmurphy> morgan: the sleep was because of the fernet weirdness, but i've seen it at least once with uuid
18:50:45 <morgan> can't repro it*
18:50:55 <morgan> cmurphy: same think in uuid at times.
18:51:07 <lbragstad> just dealing with a much smaller window
18:51:09 <morgan> cmurphy: sometimes ... things are racy and pain.
18:51:14 <cmurphy> heh
18:51:54 <lbragstad> ok - if we get to RC1 without understanding what's going on we'll cut it without fixes and backport them if we can
18:52:03 <lbragstad> next
18:52:07 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1705072
18:52:07 <openstack> Launchpad bug 1705072 in OpenStack Identity (keystone) "clearing default project_id from users using wrong driver implementation" [Medium,Triaged]
18:52:28 <lbragstad> last i heard praskre was working on a fix for this
18:52:31 <lbragstad> but i haven't seen anything
18:52:40 <samueldmq> I had a fix for it. 1 line change
18:52:55 <samueldmq> make the LDAP driver do a "pass"
18:53:08 <lbragstad> samueldmq: this is a different bug i think
18:53:17 <samueldmq> hm
18:53:18 <lbragstad> this is if you have multiple domains and different identity backends for each
18:53:33 <lbragstad> the default project id is only cleared for the default domain backend
18:53:38 <morgan> it should be "if_readonly: pass"
18:53:44 <morgan> not just "if ldap"
18:53:51 <morgan> ldap driver explicitly says read-only
18:53:54 <lbragstad> the logic to unset default project ids should be applied to all identity backends
18:53:56 <morgan> so, focus on that.
18:54:05 <lbragstad> morgan: that's being handled in a separate patch
18:54:18 <lbragstad> this is specifically referencing multi-domain support
18:54:25 <samueldmq> ++
18:54:31 <morgan> h,.
18:54:32 <morgan> hm.
18:54:36 <samueldmq> just do a for known backend, as opposed to the default one
18:54:59 <lbragstad> if a project is deleted the default project id is only cleared for the default domain backend
18:55:02 <morgan> you'd have to iterate
18:55:07 <lbragstad> right
18:55:13 <morgan> in as many words, default_project_id is a dumb construct
18:55:21 <samueldmq> iterate and check if it's read_only or not
18:55:21 <lbragstad> find all available drivers and call unset_default_project_id() on each
18:55:31 <morgan> eh.
18:55:34 <morgan> yeah
18:55:39 <samueldmq> or just call them and let the driver decide if there is something to be done
18:55:49 <samueldmq> that'd be better maybe. the driver knows the backend.
18:56:04 <samueldmq> anyways, looks like we agree in a solution
18:56:25 <lbragstad> anyone interested in proposing a fix and a test?
18:56:38 <lbragstad> i haven't seen anything from praskre yet
18:57:29 <lbragstad> i can take a stab at it
18:57:54 <lbragstad> #action lbragstad to propose a fix for https://bugs.launchpad.net/keystone/+bug/1705072
18:57:54 <openstack> Launchpad bug 1705072 in OpenStack Identity (keystone) "clearing default project_id from users using wrong driver implementation" [Medium,Triaged]
18:58:11 <lbragstad> last one before the meeting is done
18:58:15 <lbragstad> #link https://bugs.launchpad.net/keystone/+bug/1673157
18:58:15 <openstack> Launchpad bug 1673157 in OpenStack Identity (keystone) "type: local must be set in order to get domain parsed when mapping federated users" [Low,In progress] - Assigned to Colleen Murphy (krinkle)
18:58:21 <lbragstad> that one ^ has a fix in review
18:58:25 <lbragstad> just needs another +2
18:58:30 <lbragstad> it's a documentation fix
18:58:35 <lbragstad> (thanks cmurphy)
18:59:11 <lbragstad> #link https://review.openstack.org/#/c/491478/
18:59:22 * mordred wves - has opened review from earlier
18:59:37 <lbragstad> alright - we're out of time but we can continue this in office hours
18:59:49 <lbragstad> mordred: we're headed to -keystone for office hours
18:59:57 <lbragstad> #endmeeting