Thursday, 2019-04-11

*** rcernin has joined #openstack-keystone00:22
*** mchlumsky_ has joined #openstack-keystone00:30
*** mchlumsky has quit IRC00:32
*** kukacz_ has quit IRC00:32
*** timburke has quit IRC00:32
*** kukacz has joined #openstack-keystone00:33
*** timburke has joined #openstack-keystone00:34
*** rcernin has quit IRC00:59
*** sapd1 has quit IRC00:59
*** sapd1 has joined #openstack-keystone00:59
*** rcernin has joined #openstack-keystone01:09
*** gyee has quit IRC01:10
*** sapd1 has quit IRC01:13
*** sapd1 has joined #openstack-keystone01:14
*** jamesmcarthur has joined #openstack-keystone01:34
*** lbragstad has quit IRC01:37
*** jamesmcarthur has quit IRC01:48
*** jamesmcarthur has joined #openstack-keystone01:49
*** jamesmcarthur has quit IRC02:25
*** markvoelker has joined #openstack-keystone02:31
*** erus has joined #openstack-keystone02:38
*** erus has quit IRC02:51
*** markvoelker has quit IRC03:05
*** sapd1 has quit IRC03:25
*** jamesmcarthur has joined #openstack-keystone03:25
*** sapd1 has joined #openstack-keystone03:26
*** jamesmcarthur has quit IRC03:41
*** whoami-rajat has joined #openstack-keystone03:54
*** jamesmcarthur has joined #openstack-keystone04:01
*** markvoelker has joined #openstack-keystone04:02
*** sapd1 has quit IRC04:07
*** sapd1 has joined #openstack-keystone04:08
*** jamesmcarthur has quit IRC04:19
*** sapd1 has quit IRC04:28
*** sapd1 has joined #openstack-keystone04:29
*** markvoelker has quit IRC04:36
*** sapd1 has quit IRC04:37
*** sapd1 has joined #openstack-keystone04:44
*** sapd1 has quit IRC05:11
*** sapd1 has joined #openstack-keystone05:25
*** sapd1 has quit IRC05:29
*** sapd1 has joined #openstack-keystone05:29
*** markvoelker has joined #openstack-keystone05:33
*** sapd1 has quit IRC05:36
*** sapd1 has joined #openstack-keystone05:37
*** sapd1 has quit IRC05:46
*** sapd1 has joined #openstack-keystone05:47
*** awalende has joined #openstack-keystone05:48
*** awalende has quit IRC05:53
*** awalende has joined #openstack-keystone05:54
*** sapd1 has quit IRC05:57
*** sapd1 has joined #openstack-keystone05:58
*** markvoelker has quit IRC06:06
*** sapd1 has quit IRC06:07
*** whoami-rajat has quit IRC06:13
*** awalende has quit IRC06:24
*** whoami-rajat has joined #openstack-keystone06:25
*** awalende has joined #openstack-keystone06:33
*** phasespace has quit IRC06:40
*** obre has quit IRC06:52
*** obre has joined #openstack-keystone06:52
*** sapd1 has joined #openstack-keystone06:52
*** markvoelker has joined #openstack-keystone07:03
*** awalende has quit IRC07:13
*** mvkr has quit IRC07:27
*** markvoelker has quit IRC07:36
*** phasespace has joined #openstack-keystone07:56
openstackgerritRico Lin proposed openstack/keystoneauth master: Update auth plugin name list in document
*** markvoelker has joined #openstack-keystone08:34
*** tkajinam has quit IRC08:58
*** markvoelker has quit IRC09:07
*** mvkr has joined #openstack-keystone09:30
*** markvoelker has joined #openstack-keystone10:04
*** mvkr has quit IRC10:34
*** markvoelker has quit IRC10:36
*** mvkr has joined #openstack-keystone10:48
*** mvkr has quit IRC11:04
*** mvkr has joined #openstack-keystone11:05
*** dave-mccowan has joined #openstack-keystone11:38
*** yan0s has joined #openstack-keystone11:42
*** mvkr has quit IRC11:44
*** mvkr has joined #openstack-keystone11:56
*** jamesmcarthur has joined #openstack-keystone12:13
*** starborn has joined #openstack-keystone12:20
*** jamesmcarthur has quit IRC12:34
*** lbragstad has joined #openstack-keystone12:36
*** ChanServ sets mode: +o lbragstad12:36
*** raildo has joined #openstack-keystone12:39
*** jamesmcarthur has joined #openstack-keystone12:47
*** jamesmcarthur has quit IRC13:04
*** ybunker has joined #openstack-keystone13:05
*** jmlowe has quit IRC13:08
*** rcernin has quit IRC13:18
*** pcaruana has quit IRC13:20
*** jmlowe has joined #openstack-keystone13:37
*** phasespace has quit IRC13:37
*** pcaruana has joined #openstack-keystone13:42
openstackgerritJens Harbott (frickler) proposed openstack/keystonemiddleware master: DNM: Test running without identity admin endpoint
*** erus has joined #openstack-keystone14:02
*** awalende has joined #openstack-keystone14:22
openstackgerritJens Harbott (frickler) proposed openstack/keystonemiddleware master: DNM: Test running without identity admin endpoint
*** awalende has quit IRC14:26
ayoungPredictable role ids: If we do  we can have a later database migration that makes use of it, right?14:48
ayoungI was thinking something like:  take all the existing roles, give them a new name like -migrating-Member.  Then recreate the roles with the new id, convert all the role assignments to use the new IDs, and finally drop the old roles.  Make sense?14:50
*** dklyle has joined #openstack-keystone14:51
cmurphyayoung: what if some external tool is depending on the original role IDs?14:53
ayoungIt will break14:54
ayoungWe need to be able to do that, though.  Perhaps at a major release14:54
ayoungIf we can't, OpenStack can't scale.14:55
cmurphywhat if we added a keystone-manage command so people could opt-in to breaking their role IDs?14:55
ayoungYes, that would work14:55
ayoungThat makes more sense, as it would be on their cadence, not ours14:56
ayoungOK, I can spec that out.  Could we do the same thing for Project IDs?14:56
ayoungI think in order to do that, we'd need project names to be immutable.14:56
ayoungsame with roles, actually.  I see lbragstad has a spec on that14:57
cmurphydo you mean ?14:58
*** gyee has joined #openstack-keystone15:12
*** mvkr has quit IRC15:27
kmallocwhy would you need to change project_ids?15:38
kmallocoh you mean use the actual id generator bit15:39
kmallocayoung: the immutable-ness was a flag (resource option) not an inherent property iirc15:39
kmallocso roles, etc could be marked as immutable15:40
kmallocsome issues have come up with "oops i deleted the wrong, very important thing"15:40
cmurphyhe wants to migrate old projects to the newly-generated IDs so that the deployment could be expanded15:40
kmallocI think that will never be a good idea, even as an opt in15:41
kmallocnotably, it will break some stuff really badly.15:41
kmallocand part of that is the hassle/issues of uuid as PK for some of the columns/tables15:42
*** pcaruana has quit IRC15:42
kmallocnow... if we had a DB layer that could use the old ID and the new id (projects, roles are... something different)15:44
kmallocfor compat, and we re-structure for the new-id to be canonical. I could see that working.15:44
kmalloce.g. legacy_id we carried in the past for some thing15:45
cmurphythat might work15:45
kmallocthis however would never be removable.15:45
kmallocas long as the new IDs can be distinguished, e.g. more bytes, it is simple to say "oh ID is clearly not 32bytes hex? great, use non-legacy"15:46
kmalloci would also move to an autoinc PK on projects/anything else getting this treatment, and isolate the public ID like we do on the more modern additions to keystone15:46
kmallocif you're already adding deep data changing migrations15:46
kmallocayoung: ^ cc15:47
*** jamesmcarthur has joined #openstack-keystone15:49
*** jamesmcarthur_ has joined #openstack-keystone15:50
*** jamesmcarthur has quit IRC15:50
*** yan0s has quit IRC15:58
*** ybunker has quit IRC16:01
*** ybunker has joined #openstack-keystone16:02
*** ybunker has quit IRC16:05
*** ybunker has joined #openstack-keystone16:05
ayoungcmurphy, sorry, pullled away.  Yeah, that is the spec.16:28
*** erus has quit IRC16:28
*** erus has joined #openstack-keystone16:29
ayoungkmalloc, processing what you are saying.  I think we are converging on a solution....ok let me lay out what I am looking for16:29
ayoungin the hub and spoke model, I want to be able to do something at the hub and see it at the spoke.  ROle assignment is the big one16:29
ayoungright now, the only way we have to do that is Federated mapping, and I am toying with the idea of doing something based on that, but the current impl is too monolithic for my tastes.  PLus, LDAP is still king for multisite16:30
kmallocand since roles are almost 100% referenced by name in policy/etc i see zero issue with an opt-in "fix role IDs"16:30
ayoungI'd rather have a single mechanism for both16:30
kmallocproject_ids and other resources not so much16:30
ayoungkmalloc, yeah, I'm with you on that16:30
kmallocthe ids for those must remain forever unless the project is deleted.16:31
ayoungto do a role assignment you need userid, projectid, roleid16:31
kmallocbecause project ids / domain ids, user ids are used in the auth flow, user facing16:31
ayoungagain, it could be opt in16:31
kmallocit's not something you can "look up"16:31
kmallocrole ids you can, once you've authed16:31
ayoungThe norm is OS_USER_DOMAIN_NAME nad OS_PROJECT_DOMAIN_NAME plus username and project name16:32
kmallocnot in all cases.16:32
ayoungI know16:32
ayoungI said norm16:32
kmallocID is used frequently16:32
kmalloci'd argue id is used as much as name16:32
ayoungso...something like this16:32
ayoung1.  specify immutable project names16:32
kmallocthat is an API break. need microversions16:32
ayoung2.  run keystone-manange --normalize-project-ids16:33
ayoungnot if it is a config option16:33
ayoungit can be opt in16:33
kmallocgod, no.16:33
ayoungthis is only for multisite16:33
kmallocdon't do that.16:33
ayoungyou always react this way16:33
kmalloccreate a legacy_id.16:33
kmalloccarry either/or lookup16:33 me through that16:33
kmalloclegacy id, either column or new mapping table16:34
ayoungback up16:34
ayoungimmutable is kindof necessary16:34
kmalloclegacy id is the old ID before predictible ids.16:34
ayoungotherwise we have no way to do predictability16:34
ayoungI think I'm ok with a duplicate ID approach, but suspect it won't really be necessary.  What is wrong with making immutable IDs a config option16:35
kmallocchanging the whole functionality of the system and making it so you have no idea without trying to utilize the API how it's configured makes openstack suck as a user16:35
kmallocthis is a huge problem with openstack in general16:36
kmallocdepending on config etc the API acts massively different16:36
kmallocdeployment to deployment16:36
kmallocit's poor design.16:37
kmallocit also is not really testable without adding a distinct testing suite.16:38
kmallocit would not pass tempest.16:38
ayoungI won't argue on poor desing, but termie  has long since left the project16:38
ayoungI can't spell, even when I spell it right16:39
kmallocthese permutations on deployment make for consistent testing/not-so-edge-case testing really tough16:39
*** dave-mccowan has quit IRC16:39
kmallocand likely to break things in weird ways16:39
ayoungI'd argue that project renames are done like 0% of thetime16:40
kmallocit's, from my discussions, about 5-10%16:40
kmallocbecause the name is used to encode purpose16:40
kmallocor other relevant things.16:40
ayoungok, most resources are tagged by proejct ID, so changing project IDs really is not an option anyway16:40
ayoungroles and assignments, sure16:41
kmallocand that ^16:41
ayounghow would we roll in a multisite approach?16:41
kmallocso, unless you're maintaining a new column and a legacy column it doesn't work.16:41
kmallocreally, the best option is centralizing project information in a tool like athenz16:41
kmallocand auto-provision16:41
kmallocwith a side-band cleanup.16:41
ayoungI'm not pushing for an external tool16:42
kmallocand policy off "project-update" outside of the provision/external work-flow tool16:42
ayoungthat just punts on the problem, not solves it16:42
kmalloceither replicate the database or don't16:42
kmallocin the don't case, provision on demand16:42
ayoungso, we had a customer propose async db replication16:43
ayoungI think we could get away with that if TripleO supported an option to do so16:43
ayounglike "here is the sql load script for the Keystone database"16:43
kmallocyou should chat with eandersson, there are some serious issues with it in how keystone is built16:43
kmallocit's not really doable above a small number of sites because we have autoinc PKs in the shadow table (if ldap/fed is used)16:44
kmallocand we didn't cleanly address async repl16:44
ayoungbut there are still day 2 ops that we'd want to make sure worked with local calls.  Like, Heat etc need to create app creds.  Are those going to only be local?16:44
kmallocso maybe we need to back away from auto-inc PKs?16:44
kmallocit's terrible for performance and mem utilization16:44
kmallocit is a bad choice to mix exposed/API IDs and internal record keeping IDs.16:45
ayoungand with predictable IDs there is less reason to have them16:45
kmallocexcept that it will bite us in large environments.16:45
ayoungI'm really only concerned about large environs.  Go on16:46
kmallocit's an issue with how (assuming mysql/mariadb) works with the PK and indexing on the uuid.16:46
kmallocints are more efficient/performant on these fronts16:46
ayounghow large of an int do we need?16:46
kmallocit's fundamentally the bytes in the column16:47
kmalloca hugeint is less than a varchar25516:47
ayoungso,  we do either 32 or 64 char fields for ids.  A uuid is 32 chars.  That fits in hugeint, if converted, right?16:48
kmallocand it is unlikely we will need uint64 space16:48
kmalloc64char does not fit16:49
kmallocit is likely we need a uint32 at most16:49
kmalloc~4bn project records. somehow i don't think we'll get there.16:49
ayoungthat is, what, 64 hex chars16:49
kmallocwithout hitting a LOT more issues16:49
kmallocanyway. it's mechanisms for how the rdbms works.16:50
ayoung80 bits, right?16:50
ayoungso a 128 bit number should store any of our current ids in a compact format.16:51
kmallocmysql: bigint is 8 bytes16:52
kmalloca uuid hex (32) compresses to ~16bytes in bin format in python16:52
*** jamesmcarthur has joined #openstack-keystone16:52
kmallocand bigint is int(64), so -2^63 - 2^63-1 or 2^64-1 [unsigned]16:53
ayoungso it won't work16:53
kmallocuuids have a LOT of extra content.16:53
kmalloccompared to ints.16:53
kmallocnow, if we decide it is super important to be non-auto-inc and PK specified (generated as uuid or etc in the app), we take the hit in mem and performance16:54
ayoungI'll have to convince myself of the math, but I'll accept it for now to move things along16:54
kmallocand we explicitly outline it.16:54
kmalloc>>> sys.getsizeof(uuid.uuid4().bytes)16:55
kmalloc>>> sys.getsizeof(uuid.uuid4().hex)16:55
*** erus has quit IRC16:55
kmallocthat is including overhead.16:55
kmalloc>>> len(uuid.uuid4().bytes)16:55
*** jamesmcarthur_ has quit IRC16:56
kmalloc>>> len(uuid.uuid4().hex)16:56
*** erus has joined #openstack-keystone16:56
kmallocso, issue 1) predictable ID, not so predictable because we can't change them.16:56
ayoungand we do sha256 in a few places now which is double that16:57
kmalloc(e.g. can't make them conforming with manage, it'll break nova and everything else)16:57
kmallocyeah we do.16:57
kmalloc2) int vs uuid PK performance in MySQL/MariaDB16:57
kmalloc2 is "make the choice and outline why the choice is made"16:57
kmallocbut we have things we need to fix / migrate away from due to choice made when we weren't dealing with async replication16:58
ayoungPerformance is less of an issue if we get scale out right16:58
kmallocissue 1 is much harder to work around.16:58
kmallocthe performance hit is DB operation related16:58
kmallocso it's not so much "many keystones" solve it16:58
kmallocit's going to be how much crunch the DB can handle16:58
ayoungCan we add an immutable field to a project object now?16:59
kmallocwe can. i argue it should be a resource-option and be an option for ALL resources16:59
ayoungI'd be OK with it for only stuff moving forward16:59
ayoungOK, I like that idea16:59
ayoungwe do it for users, roles, and projects16:59
kmalloci envision it being like chattr +i16:59
kmallocroles, projects, users, domains, federation protocols, etc17:00
ayoungwhat would the default be?17:00
kmallocdefault would be "off" unless we add a template set of options for a given resource17:00
kmallocand that is for api compatibility17:00
ayoungnot federation protocols, and not mapping.  We want to be able to manipulate those, I think17:00
kmallocnah, it's an opt in17:00
kmallocyou can always chattr -i a resource17:01
kmallocit's the same deal here, un-set immutable (if you are allowed)17:01
kmallocand change it17:01
kmalloci also see resource options as having their own policy entries17:01
ayoungmapping is too monolithic as is17:01
kmalloca "generic" for the resource and an override.17:01
ayoungok, so for a multisite, we would want all new resources to be immutable17:02
kmallocso you can say an system-user can always set immutable (for example) [or unset], but an owner can set other flags.17:02
ayoungi.e.  only sync down from the center17:02
kmallocin policy, since resource_option:immutable is system(writer) only17:02
ayoungok,  you've obviously thought this through.  Can you write up your thoughts?17:03
kmalloci think i wrote a spec already....17:03
kmalloclet me check17:03
kmallocor adriant did17:03
kmalloc is the resource-options-for-all spec17:03
kmalloc that is the core of it17:04
*** erus has quit IRC17:04
kmallocand there is a spec missing that is "add policy for resource-options"17:05
*** erus has joined #openstack-keystone17:05
ayoungOh, he's targetting something else with that.  You just weant to piggyback...interensting17:05
*** pcaruana has joined #openstack-keystone17:05
kmallocthe concept is super useful outside of your usecase17:06
kmallocand it lines up really nicely to solve the problem17:06
kmallocthis is exactly one of the cases i saw immutable as solving17:06
kmallocand i was wrong, cmurphy was the person writing up the immutable resource-options17:08
kmallocadriant had another use case where resource-option(s) were relevant17:08
ayoungShe had immutable roles....are there others?17:08
kmalloci figured if we add immutable to roles we should add it to most resources17:09
kmallocit prevents accidental updates/deletions.17:09
kmalloconce we have the template for it to work in one place, it's super easy to add it elsewhere17:09
ayoungNeed a way to make it the default for things, tho17:12
ayoungIs that on a per user basis?  Something that is inherited?17:12
cmurphyin the spec it's the roles that are created by bootstrap that become immutable by default17:15
ayoungcmurphy, are you OK with morphing  into a general immutable spec, or would you prefer to keep the role specific stuff there and creata new one for all resources17:15
kmallocthe way i'd roll it is keep that role-immutable and then propose an additional spec to replicate that to other resources17:16
cmurphyayoung: i should fix the commit message but it already is generic, it just has a special case for roles since the idea is to change the default behavior for them17:17
*** erus has quit IRC17:17
*** erus has joined #openstack-keystone17:18
ayoungCool.  But those really are two distinct things17:23
ayoungBut I am 100% on board17:23
*** gyee has quit IRC17:29
*** jamesmcarthur has quit IRC17:35
*** jamesmcarthur has joined #openstack-keystone17:47
*** jamesmcarthur has quit IRC17:54
*** jamesmcarthur has joined #openstack-keystone17:56
*** jamesmcarthur has quit IRC17:58
*** jamesmcarthur has joined #openstack-keystone17:59
*** jamesmcarthur has quit IRC18:01
*** jamesmcarthur has joined #openstack-keystone18:02
*** gmann is now known as gmann_afk18:15
*** ybunker has quit IRC18:23
*** jamesmcarthur has quit IRC18:26
*** jamesmcarthur has joined #openstack-keystone18:28
*** jmlowe has quit IRC18:32
*** pcaruana has quit IRC19:02
openstackgerritMerged openstack/keystoneauth master: Update auth plugin name list in document
*** erus has quit IRC19:09
*** erus has joined #openstack-keystone19:10
*** jmlowe has joined #openstack-keystone19:15
*** gmann_afk is now known as gmann19:27
*** zaneb has quit IRC19:53
*** erus has quit IRC19:53
*** erus has joined #openstack-keystone19:53
*** starborn has quit IRC20:04
*** jamesmcarthur has quit IRC20:10
openstackgerritMerged openstack/keystonemiddleware master: Fix string format error
*** whoami-rajat has quit IRC21:03
*** gyee has joined #openstack-keystone21:49
*** jamesmcarthur has joined #openstack-keystone22:01
*** jamesmcarthur has quit IRC22:14
*** rcernin has joined #openstack-keystone22:26
*** tkajinam has joined #openstack-keystone23:00
*** kukacz has quit IRC23:21
*** prometheanfire has quit IRC23:21
*** bbobrov has quit IRC23:21
*** irclogbot_1 has quit IRC23:24
*** kukacz has joined #openstack-keystone23:27
*** prometheanfire has joined #openstack-keystone23:27
*** bbobrov has joined #openstack-keystone23:27
*** raildo has quit IRC23:29

Generated by 2.15.3 by Marius Gedminas - find it at!