17:01:33 <lbragstad> #startmeeting keystone-office-hours
17:01:34 <openstack> Meeting started Tue Mar 27 17:01:33 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:37 <openstack> The meeting name has been set to 'keystone_office_hours'
17:01:45 <hrybacki> bridge is open btw: https://redhat.bluejeans.com/8559013623/
17:02:18 * lbragstad grabs water quick
17:02:29 <lbragstad> i'll be on in about 3 minutes
17:02:34 * hrybacki visits a water closet
17:03:58 <cmurphy> :/ guys i love your faces but i can't have a multi-hour video conference every week, that starts to feel a lot like work
17:05:18 <gagehugo> when you have them everyday, why not have another one?
17:05:20 <gagehugo> :(
17:05:40 <hrybacki> video calls are the only way I can stay on topic these days -_- my current role is wrecking me
17:08:43 <cmurphy> i'm starting an etherpad for the help wanted list https://etherpad.openstack.org/p/keystone-help-wanted-list
17:08:56 <cmurphy> if you need my input on the call just ping me
17:10:27 <hrybacki> adding that to the OO etherpad cmurphy
17:10:51 <kmalloc> cmurphy: can i entice you with doggo cam again? (soon to be puppy cam too)
17:11:02 <kmalloc> (2 more weeks and puppy arrives)
17:11:26 <cmurphy> kmalloc: hmmmmmmmm maybe
17:11:44 <kmalloc> cmurphy: ooh sec.
17:12:56 <cmurphy> puppies aside a video call isn't easily reconsumeable even if it's recorded, so it's harder for say wxy to come back to it and figure out what happened
17:17:45 <lbragstad> true
17:18:27 <kmalloc> https://usercontent.irccloud-cdn.com/file/ss1edaf7/OMG%20PUPPY
17:18:39 <kmalloc> For your puppy consumption needs.
17:19:18 <hrybacki> The video call was so that someone can drive a specific conversation e.g. look at this review with me. Like, someone outside of the core team for example could ask for more immediate input and get feedback if they are having troubles with just comments on reviews
17:19:55 <cmurphy> they can do that on irc
17:20:10 <cmurphy> and we can jump on a call if irc isn't cutting it for a given discussion
17:20:32 <hrybacki> I guess I was shooting for some consistency
17:21:31 <hrybacki> but it is what the team wants :)
17:21:48 <gagehugo> yeah I like the video call for focused conversation (ie reviews,specs)
17:22:09 <gagehugo> but otherwise irc can probably cover most of what we need
17:22:17 <cmurphy> please go ahead with it, it's just evening here for me so i'm going to relax and stuff and am available if needed specifically
17:31:37 <lbragstad> jgrassler: updated https://review.openstack.org/#/c/396331/20 to summarize the meeting
17:34:14 <jgrassler> lbragstad: Thanks!
17:34:23 <lbragstad> jgrassler: no problem
17:34:39 <jgrassler> lbragstad: I'll update the spec tomorrow morning (gotta dash now)
17:34:57 <lbragstad> jgrassler: yeah - no worries
17:45:23 <lbragstad> hrybacki: reviewed https://review.openstack.org/#/c/523973/
17:45:35 <hrybacki> lbragstad: ack, thank you! looking now
17:45:47 <lbragstad> looks good, just a few suggestions, but I think we can probably take this to some of the other projects and get feedback
17:46:29 <hrybacki> lbragstad: ack. Need folks to ask a few hard questions so we can flesh it out in a meaningful way rather than just speculating imo
18:00:44 <kmalloc> lbragstad: the mysql and pgsql errors are different
18:00:54 <kmalloc> one is an issue, loosk like with the dataset
18:00:57 <kmalloc> the other is a deadlock
18:01:03 <lbragstad> huh
18:01:28 <lbragstad> for mysql, are you specifically referencing the issue brought up this morning by tmcm?
18:01:37 <kmalloc> yep
18:01:54 <kmalloc> that looks to be an issue with the FK constraint cannot be made, there is bad data
18:02:05 <kmalloc> like project.id reference in a column that doesn't exist in the project table
18:02:07 <kmalloc> (example)
18:02:17 <lbragstad> kmalloc: yeah - we found out that the domain being referenced didn't actually exist
18:02:23 <kmalloc> yep
18:02:43 <kmalloc> the hard part here is... we don't really test pgsql
18:02:51 <kmalloc> this might be a pgsql issue
18:03:04 <lbragstad> there was some discussion about postgres support, but i don't know where that ended up
18:03:25 <kmalloc> well, let me check the gate, but... i think we aren't testing pgsql meaningfully
18:03:30 <openstackgerrit> Lance Bragstad proposed openstack/keystone master: Log warning when using token_flush  https://review.openstack.org/556889
18:03:48 <lbragstad> i thought i remember various TC members being involved there
18:04:03 <lbragstad> iirc - it was a long discussion and i never kept up with it
18:04:32 <gyee> lbragstad: https://bugs.launchpad.net/keystone/+bug/1758460
18:04:34 <openstack> Launchpad bug 1758460 in OpenStack Identity (keystone) "UUID (or any persistent) token providers unable to validate federation token" [Undecided,New]
18:04:53 <gyee> tell me with a straight face, how much do we care about UUID provider at this point, even in stable/pike :-)
18:05:08 <kmalloc> gyee: not
18:05:14 <kmalloc> gyee: it was deleted in Rocky
18:05:18 <gyee> heh
18:05:32 <gyee> gotta ask
18:05:33 <kmalloc> if it is a major bug, we can fix as long as P isn't EOL
18:05:44 <kmalloc> and if the bug is in Q, we can address it
18:05:46 <gyee> P is near EOL
18:05:49 <lbragstad> gyee: i already pulled that plug
18:06:06 <cmurphy> lbragstad: it got a resolution https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.html but it was not really a real decision
18:06:07 <lbragstad> and started rewriting all the interfaces in that part of keystone
18:06:11 <kmalloc> but frankly, i wouldn't care about the P bug unless it's critical, and if the bug is in Q too
18:06:19 <kmalloc> since technically we support uuid in both P and Q
18:06:33 <gyee> that bug's been there since P
18:06:43 <gyee> only for UUID provider though
18:06:51 <kmalloc> as a stable core, i'd merge a fix for Q and backport to P
18:07:16 <kmalloc> but i wouldn't write the code myself.
18:07:19 <kmalloc> if that helps you out
18:08:04 <gyee> that's enough info for me to convey back to the decision makers :-)
18:08:06 <gyee> thanks guys
18:08:20 <kmalloc> gyee: and i say that as the only keystone-stable-core member :P
18:08:44 <kmalloc> gyee: my recommendation to the decision makers is "fernet"
18:09:14 <gyee> kmalloc, agree
18:09:23 <kmalloc> gyee: if you are writing the fix, propose straight to Q but comment that it cannot be merged to master because uuid has been removed.
18:09:53 <gyee> kmalloc, nah, I'll push for fernet, no point of touching UUID
18:09:54 <kmalloc> and once it's good (inc. a test) port to P
18:09:58 <kmalloc> i figured
18:10:02 <kmalloc> but just in case you have to ;)
18:10:13 <kmalloc> and then ping me directly so we can push it through (if you end up needing it)
18:10:32 <cmurphy> gyee: with my internal hat on, our other product already does fernet so you could copy that implementation :)
18:11:17 <gyee> cmurphy, most of the stuff are already there, just the rotation bit needs work
18:12:15 <kmalloc> lbragstad: i'm also going to reference the TC resolution regarding PGSQL there.
18:12:30 <kmalloc> lbragstad: it's going to take a bit more work to know what and why PG is having the issues.
18:12:43 <kmalloc> lbragstad: i'm guessing it is an issue with load on the DB and a table lock.
18:12:52 <kmalloc> lbragstad: possibly a delete operation
18:19:12 <lbragstad> gyee: in case you're interested https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:token-provider-refactor
18:19:28 <lbragstad> i'm not sure if you are maintaining an out-of-tree token provider anywhere
18:20:24 <lbragstad> kmalloc: you mean reference it in the bug?
18:20:30 <gyee> lbragstad, no out-of-tree, I got pulled in to reverify the federation stuff recently
18:20:38 <lbragstad> gyee: oh
18:20:39 <gyee> but I haven't touch that stuff in awhile
18:20:44 <kmalloc> lbragstad: yeah
18:20:51 <lbragstad> kmalloc: ok
18:21:25 <kmalloc> lbragstad: just commented on the code
18:21:47 <kmalloc> lbragstad: i think we might need zzzeek to help us here.
18:22:07 <hrybacki> lbragstad: where was that community tag stuff doc'd again?
18:22:29 <hrybacki> lbragstad: disregard
18:22:34 <lbragstad> here? https://governance.openstack.org/tc/reference/tags/index.html
18:22:48 <lbragstad> kmalloc: thanks for digging
18:22:55 <hrybacki> aye
18:24:28 <hrybacki> lbragstad: update pushed
18:24:33 * hrybacki fetches lunch
18:26:41 <lbragstad> sweet
18:27:54 <kmalloc> lbragstad: i downgraded the bug to medium
18:28:00 <kmalloc> it's PGonly and it's contract
18:28:27 <lbragstad> kmalloc: ack - so we're waiting on feedback then?
18:28:30 <kmalloc> zzzeek: if you could help out some, trying to chase down potential deadlocks in a contract phase https://bugs.launchpad.net/keystone/+bug/1755906
18:28:30 <openstack> Launchpad bug 1755906 in OpenStack Identity (keystone) "Occasional deadlock during db_sync --contract during Newton to Pike live upgrade" [Medium,Confirmed]
18:28:58 <kmalloc> zzzeek: i just don't see it, and it's happening in PG but not MySQL AFAICT
18:35:58 <zzzeek> kmalloc: we're spuporting postgresql again?
18:36:26 <kmalloc> zzzeek: no, just a best effort, feel free to say "not my problem/can't help"
18:36:36 <zzzeek> kmalloc: postgresql is very locky at the DDL level
18:36:49 <kmalloc> just said I would ask -- mostly to be sure we aren't doing something dumb that could bite us in MySQL as well
18:36:59 <kmalloc> zzzeek: yeah PG is very locky for integrity reasons(tm)
18:37:54 <zzzeek> kmalloc: so, what did you have in mind here?
18:38:17 <kmalloc> if you could look at the migration and give a "yeah no clear issues that would impact, this is an edge case"
18:38:19 <kmalloc> that is good for me
18:38:37 <zzzeek> kmalloc: are the deadlocks against the normal app server running SELECT statements?
18:38:55 <kmalloc> the deadlock is happening in a contract phase (new FK constraint) while selects are happening
18:39:05 <openstackgerrit> Gage Hugo proposed openstack/keystone master: Make tags filter match subset rather than exact  https://review.openstack.org/553108
18:39:12 <gagehugo> lbragstad added a releasenote
18:39:32 <kmalloc> this is the no-downtime(limited downtime) upgrade thing
18:40:21 <kmalloc> zzzeek: this is the migrtation in question https://github.com/openstack/keystone/blob/master/keystone/common/sql/contract_repo/versions/014_contract_add_domain_id_to_user_table.py
18:41:41 <zzzeek> kmalloc: this is adding a foreign key in the contract, huh
18:42:06 <kmalloc> because we can't add the forign key in expand... lets just say the no-downtime thing has been headaches
18:42:20 <kmalloc> and pivoting the whole table is also... very detrimental
18:42:49 <zzzeek> kmalloc: i dont see a quick win on this thered' ahve to be some hey make sure the app server isn't running while the migration happens thing, e.g. more locks
18:43:00 <kmalloc> zzzeek: and for all i know the triggers is causing issues
18:43:03 <zzzeek> kmalloc: online schema migrations for PG seems like a non-starter given the status of PG
18:43:08 <kmalloc> good to know
18:43:15 <kmalloc> i'll respond with that and reference this convo
18:43:15 <zzzeek> kmalloc: yo're doing the triggers w/ PG as well?
18:43:20 <kmalloc> sigh
18:43:32 <kmalloc> against all my protests
18:43:47 <zzzeek> kmalloc: oh I was arguing in *favor* of the triggers :)
18:43:56 <kmalloc> i greatly dislike them.
18:44:12 <kmalloc> they're so very hard to debug.
18:44:19 <kmalloc> esp. when the app can do all the logic.
18:44:22 <zzzeek> kmalloc: im not saying this bug cant be fixed but it would require thinking, looking, coding, and testing and i dont know we have resources for that for PG
18:44:28 <kmalloc> yeah thats fine
18:44:44 <kmalloc> i'll reference this as well confirming my statement, it's just not something we have resources for
18:44:48 <zzzeek> yes
18:44:53 <kmalloc> but we will happily accept external help
18:45:00 <kmalloc> and if they have a fix, we'll evaluate it
18:45:06 <kmalloc> and include it if we can.
18:49:28 <kmalloc> lbragstad: ^, marked the bug as incomplete
18:49:34 <lbragstad> kmalloc: zzzeek thanks
18:49:38 <kmalloc> lbragstad: if they can supply help, we'll accept it
18:49:41 <lbragstad> gagehugo: works for me locally
18:50:19 <kmalloc> lbragstad: otherwise, we should update our documentation, live-schema changes only supported/tested under MySQL, PGSQL is recommended that live-schema changes not be performed (downtime-only)
18:50:28 <kmalloc> lbragstad: acutally,w e sould update docs for that anyway
18:50:33 <lbragstad> right
18:52:56 <lbragstad> kmalloc: when should we remove keystone-manage token_flush?
18:52:58 <lbragstad> Solar?
18:53:08 <kmalloc> lbragstad: sure.
19:04:59 <openstackgerrit> Lance Bragstad proposed openstack/keystone master: Log warning when using token_flush  https://review.openstack.org/556889
19:14:57 <hrybacki> lbragstad: cmurphy perhaps we could conduct an audit of our API vs scope levels
19:16:26 <hrybacki> for example, I assumed user related actions would be domain-scoped as opposed to system-scoped. BUT if user/group actions are system-scoped that would resolve our earlier issue lbragstad
19:20:25 <openstackgerrit> Lance Bragstad proposed openstack/keystone master: Removal of deprecated direct driver loading  https://review.openstack.org/350815
19:23:40 <lbragstad> hrybacki: i think some of that would audit would have been taken care of when we implement scope_types
19:24:18 <hrybacki> lbragstad: ack. I'm just thinking doing part of it now would make sure we have a sane, consistent messgae in the spec
19:24:27 <hrybacki> e.g. are user operations domain or system level discussion
19:24:52 <lbragstad> sure
19:26:12 <lbragstad> we could put a disclaimer for that specific case in the spec
20:31:11 <gagehugo> is Solar the official name?
20:32:36 <cmurphy> not yet
20:34:29 <lbragstad> oh - shoot, i should probably wip my review then
20:37:38 <gagehugo> I liked stein
21:18:48 <lbragstad> knikolla: are you happy with response here https://review.openstack.org/#/c/556022/ ?
21:23:45 <adriant> lbragstad: want me to update the auth receipt spec and move that down?
21:23:57 <adriant> or would you prefer doing it as another patch?
21:24:22 <lbragstad> adriant: oh - we can do that in another patch set... we talked about that in today's meeting and i said i was going to propose a follow on
21:24:30 <lbragstad> but i haven't gotten to it yet
21:24:35 <adriant> cool :)
21:24:59 <adriant> sorry I wasn't able to attend
21:25:29 <cmurphy> i think kmalloc was also hoping for a final look from ayoung
21:26:19 <lbragstad> adriant: no worries - it's not the best time for APAC
21:26:55 <adriant> As I start making progress on the implementation WIP I'll come by for office hours and ask silly questions!
21:28:51 <openstackgerrit> Lance Bragstad proposed openstack/keystone-specs master: Log queens specifications with previous releases  https://review.openstack.org/557060
21:35:44 <openstackgerrit> Merged openstack/keystone master: Make tags filter match subset rather than exact  https://review.openstack.org/553108
21:45:56 <kmalloc> cmurphy: only because ayound was commenting on it
21:46:00 <kmalloc> cmurphy: but i'm fine with it +A now
21:46:06 <kmalloc> lbragstad, adriant: ^
21:46:35 <openstackgerrit> Merged openstack/keystone master: Fix integer -> method conversion for python3  https://review.openstack.org/555339
21:51:09 <knikolla> lbragstad: pushed it :)
21:52:28 <openstackgerrit> Merged openstack/keystone-specs master: Add spec for MFA auth receipts  https://review.openstack.org/553670
21:53:56 <adriant> woo!
21:54:11 <adriant> now I have to actually implement it. Which shouldn't be so bad