15:05:22 <d34dh0r53> #startmeeting keystone
15:05:22 <opendevmeet> Meeting started Wed Oct 16 15:05:22 2024 UTC and is due to finish in 60 minutes.  The chair is d34dh0r53. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:05:22 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:05:22 <opendevmeet> The meeting name has been set to 'keystone'
15:05:52 <d34dh0r53> Reminder: This meeting takes place under the OpenInfra Foundation Code of Conduct
15:05:58 <d34dh0r53> #link https://openinfra.dev/legal/code-of-conduct
15:06:03 <d34dh0r53> #topic roll call
15:06:11 <d34dh0r53> admiyo, bbobrov, crisloma, d34dh0r53, dpar, dstanek, hrybacki, lbragstad, lwanderley, kmalloc, rodrigods, samueldmq, ruan_he, wxy, sonuk, vishakha, Ajay, rafaelwe, xek, gmann, zaitcev, reqa, dmendiza[m], dmendiza, mharley, jph, gtema, cardoe
15:06:13 <d34dh0r53> o/
15:07:25 <mhen> o/
15:08:22 <d34dh0r53> #topic review past meeting work items
15:08:34 <d34dh0r53> #link https://meetings.opendev.org/meetings/keystone/2024/keystone.2024-10-09-15.05.html
15:08:50 <d34dh0r53> no action items from last week
15:08:51 <d34dh0r53> #topic liaison updates
15:09:59 <d34dh0r53> nothing from VMT or releases
15:10:41 <cardoe> \o
15:11:02 <d34dh0r53> #topic specification OAuth 2.0 (hiromu)
15:11:06 <d34dh0r53> #link https://review.opendev.org/q/topic:bp%252Foauth2-client-credentials-ext
15:11:15 <d34dh0r53> #link https://review.opendev.org/q/topic:bp%252Fenhance-oauth2-interoperability
15:11:23 <d34dh0r53> External OAuth 2.0 Specification
15:11:33 <d34dh0r53> #link https://review.opendev.org/c/openstack/keystone-specs/+/861554 (merged)
15:11:37 <d34dh0r53> OAuth 2.0 Implementation
15:11:40 <d34dh0r53> #link https://review.opendev.org/q/topic:bp%252Fsupport-oauth2-mtls
15:11:44 <d34dh0r53> OAuth 2.0 Documentation
15:11:50 <d34dh0r53> #link https://review.opendev.org/c/openstack/keystone/+/838108 (merged)
15:11:55 <d34dh0r53> #link https://review.opendev.org/c/openstack/keystoneauth/+/838104 (merged)
15:12:05 <d34dh0r53> no updates from me on any of this
15:12:13 <d34dh0r53> #topic specification Secure RBAC (dmendiza[m])
15:12:16 <d34dh0r53> #link https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#z-release-timeline_
15:12:21 <d34dh0r53> 2024.1 Release Timeline
15:12:24 <d34dh0r53> Update oslo.policy in keystone to enforce_new_defaults=True
15:12:27 <d34dh0r53> Update oslo.policy in keystone to enforce_scope=True
15:12:50 <d34dh0r53> not sure if dmendiza is around, he messaged me that he might be away-ish this morning
15:15:42 <d34dh0r53> ok, moving on
15:15:48 <d34dh0r53> #topic specification OpenAPI support (gtema)
15:15:53 <d34dh0r53> #link https://review.opendev.org/q/topic:%22openapi%22+project:openstack/keystone
15:15:54 <d34dh0r53> https://review.opendev.org/c/openstack/keystone/+/925020 could now also land to ease api-ref work
15:15:57 <d34dh0r53> gtema: api-ref work started
15:16:11 <d34dh0r53> I think I also saw that gtema (Artem Goncharov) is on PTO
15:17:38 <d34dh0r53> looks like it, moving on
15:17:45 <d34dh0r53> #topic specification domain manager (mhen)
15:17:48 <d34dh0r53> #link https://review.opendev.org/q/topic:%22domain-manager%22
15:17:50 <d34dh0r53> tempest core lib patch has been merged, only keystone-tempest-plugin left
15:17:54 <d34dh0r53> created a patchset for documentation: https://review.opendev.org/c/openstack/keystone/+/928135
15:18:47 <mhen> yes, only tempest plugin and documentation patchsets are left
15:19:24 <mhen> got a lot of upvotes by now - is there anything missing still?
15:19:40 <d34dh0r53> ok, I'll bug dmendiza and Grzegorz Grasza to push those through
15:19:46 <mhen> nice, thanks
15:19:51 <d34dh0r53> mhen: I think we're just waiting on final reviews
15:19:57 <d34dh0r53> ack, YW
15:19:57 <mhen> ok
15:20:15 <d34dh0r53> next up
15:20:25 <d34dh0r53> #topic specification Type annotations (stephenfin)
15:20:28 <d34dh0r53> #link https://review.opendev.org/q/project:openstack/keystoneauth+topic:typing
15:20:31 <d34dh0r53> This is just pending reviews now. I will push the remaining patches as soon as a sufficient quantity of the current ones land.
15:20:45 <d34dh0r53> many reviews are owed
15:22:36 <admiyo> Sorry I missed the secure RBAC topic...I was distracted by discussing it over in #openstack-nova
15:23:17 <d34dh0r53> ahh, no worries admiyo
15:23:42 <admiyo> So...while I am not 100% certain, I think the solution in place means we should close bug 96869
15:23:49 <Vivek-vnetpops> Dear community.
15:24:17 <admiyo> It is the longest open bug in the tracker, and it is addrtessed, I think, but the secure RBAC approach.  I don't feel 100% certain there.
15:24:23 <admiyo> maybe like, 55%
15:24:54 <d34dh0r53> nooooo that bug is legendary
15:25:05 <admiyo> I think that the solution is kinda wrong
15:25:39 <admiyo> and I am pretty sure that nova is self contradictory by ignoring the project Id for the operation, but requiring a project scoped token
15:25:59 <d34dh0r53> #link https://bugs.launchpad.net/keystone/+bug/968696
15:25:59 <admiyo> and they are certainly wrong for marking it 'wishlist" and "won't fix"
15:26:04 <admiyo> BUT....
15:26:07 <d34dh0r53> that's the bug being referred to
15:26:14 <admiyo> Aye.  sorry for missing the lin
15:26:16 <admiyo> k
15:27:19 <admiyo> I think that ...if and only iff ADMIN is reserved  for operations that should never be allowed to be granted to managers of projects, then it is OK to say that they have addressed it
15:27:28 <d34dh0r53> dmendiza: is really the one to talk to about this
15:28:07 <admiyo> as I recall, there were cases where reources got orphaned and you had to be aadmin to clean them up, which means that a project MANAGER would  not have been able to clean up after a mistake, and thuse projects were not self maintaining
15:28:59 <admiyo> It is a PTL issue.  It is an openstack wide issue, and it limits adoption.  They are pushing the fix back on keystone.
15:29:44 <admiyo> And I don't know that Keystone alone can fix it, and I would argue that the fix is very fragile
15:29:46 <cardoe> So with Secure RBAC being a top level priority, I think this is something we can work to clean up.
15:30:13 <opendevreview> Dmitriy Rabotyagov proposed openstack/keystone master: Re-initiate OSA rolling upgrade tests  https://review.opendev.org/c/openstack/keystone/+/932512
15:30:19 <cardoe> I'd like to work with some folks on defining some more user personas which I think would be helpful.
15:30:42 <admiyo> So if ANY service in openstack requires ADMIN for any user-level operation, and they assign it, they have opened up a security hole in Nova
15:30:43 <cardoe> We don't have to craft roles for each one but we can say "this persona should not ever have admin role" or some such.
15:32:15 <admiyo> So, from a keystone perspective, we can be dilligent and avoid things like we used  to do:
15:32:17 <cardoe> I think that might help downstream or operators by understanding those personas. Cause I feel like those leaks happen from downstream / operators trying to make something work how they think and then nova accepting a band aid.
15:32:24 <admiyo> you needed admin to add a user to a project
15:32:33 <admiyo> which means projects were not self managing
15:32:54 <admiyo> but that has to be true of every service in the catalog
15:33:12 <admiyo> and, yeah, just default policy.  People can always break things via custom policy
15:33:36 <admiyo> Are nested projects still a thing?
15:34:18 <admiyo> That actually helped with a lot of the operators concerns
15:36:06 <cardoe> They are documented and the Red Hat docs talk about them.
15:36:14 <admiyo> Anyway,I had a long write up of this from when I was assigned the bug:  https://adam.younglogic.com/2017/05/fixing-bug-96869/
15:37:19 <admiyo> The issue was that operators wanted to be able to use a single, project scoped token to do operations across the whole tree.  It was, and is, a mess
15:37:54 <d34dh0r53> Let me talk with dmendiza about this, there may be a case for finally closing that bug with the domain manager role
15:37:55 <admiyo> Because they don't want to have to authenticate to each project
15:38:26 <admiyo> No...it still does'nt handle it.  Apologies, pulling stuff from long term memory....
15:39:17 <d34dh0r53> What is lacking in that role?
15:39:20 <admiyo> the token is scoped to a project but they want to do a global operation, and there is no way for Nova to know that a given project is nested under another project
15:39:30 <admiyo> it is fine for Keystone, not for the other services
15:39:57 <mhen> I guess the problem is that domain-ownership awareness is not really there in other services for their resources
15:40:02 <admiyo> I think the "is admin project" hack is still needed.
15:40:13 <admiyo> mhen, yes
15:40:40 <mhen> it would require a lot more work for some services to properly adapt the domain manager persona then
15:40:50 <admiyo> so...IF we create a majik ADMIN project, and if we add a flag to the token saying "this is an admin project" and if Nova checks for that as well as ADMIN then things would work
15:41:10 <admiyo> domain tokens never go to other services
15:41:15 <admiyo> domain were a mistake.
15:41:25 <admiyo> as was calling things projects instead of tennants
15:41:38 <admiyo> sins of the past that will be visted on our descendants....
15:41:40 <admiyo> but anyway
15:42:23 <admiyo> if a token is sent to nova with "is_admin_project=yes" and they chose to ignore it, that is on them
15:43:01 <admiyo> Its not a hard check to implement, and it can be done in keystone middleware if we really want, but that is a whole nother approach that I abandoned
15:44:48 <admiyo> That way, if, say HEAT requires Admin for something, and that slips through, Nova could protect itself by also required is_admin_project
15:45:19 <admiyo> anyway, I have said my piece, and will stop beat d34dh0r53
15:45:37 <d34dh0r53> lol
15:46:31 <d34dh0r53> thanks for bringing it up, I mistakenly thought the domain manager work helped to fix this but I think there is still work to do
15:46:36 <admiyo> also...we never addressed how  to limit things that required acccess to two proejcts
15:47:02 <admiyo> I think there was an issue  where deleting a project in keystone orphaned all the resources everywhere.
15:47:08 <admiyo> And...I think I had a fix for that
15:47:22 <admiyo> but I don';t thik that was RBAC
15:47:48 <mhen> https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/project-cleanup.html
15:47:49 <mhen> :)
15:47:49 <admiyo> AH yes
15:47:52 <admiyo> https://review.opendev.org/c/openstack/keystone/+/664746
15:48:30 <admiyo> It was KINDA an RBAC issue  in that you needed to get a system admin to do all your clean up.  This avoids that case.
15:49:09 * admiyo really done now
15:52:00 <admiyo> meeting over?
15:52:03 <d34dh0r53> Thanks admiyo
15:52:37 <d34dh0r53> #topic specification Include bad password details in audit messages (stanislav-z)
15:52:40 <d34dh0r53> # link https://review.opendev.org/c/openstack/keystone-specs/+/915482
15:52:46 <d34dh0r53> # link https://review.opendev.org/c/openstack/keystone/+/932423
15:52:59 <d34dh0r53> s///
15:53:19 <d34dh0r53> s///
15:53:56 <stanislav-z> Hello! I'm looking for any feedback on both. And in the meantime was going to work on adding tests for the implementation
15:54:46 <admiyo> @stanislav-z, is the idea that an admin can compare the hash to the hash in the database and say "yep they don't match?"
15:57:01 <stanislav-z> there is no relation to the database hash. the idea is that when there is a bruteforce attack the hash will be changing frequently. while if e.g. some end-user automation got stuck with the old password the hash won't be changing. so by checking the hash, it may be possible to perform a more educated guess if user is experiencing an attack or just has some broken automation
15:57:34 <stanislav-z> by checking how hash is changing*
15:57:44 <admiyo> Ah. THat makes more sense.  But any change in the users passwords attempt end up as differnt hashes, and look like an attack?
15:58:14 <admiyo> password  pa$$word  p@$$word p@$$w0rd
15:58:50 <stanislav-z> only if something/someone would be continuing to submit the old (wrong) password. no hashes will be showing up for successful authentication events
15:59:32 <admiyo> but is a user submits password  pa$$word  p@$$word p@$$w0rd  and they all get hashed, it is safe to put the first 5 chars of  the hash in the log?
16:00:42 <d34dh0r53> According to the spec the N characters isn't logged, it's sent in a message, is that correct?
16:01:21 <stanislav-z> with the default hashing function from possible WIP the hash size would be 88 characters. providing just 5 (default; configurable) should be safe - and it will only appear in event notifications
16:02:21 <stanislav-z> that's correct. but N - is the number of character that would passed to the event (not how much would be truncated from the hash)
16:02:37 <admiyo> same thing, though.  Written down somewhere not in ephemeral memory and the database.  Thus widening exposure to the hash.  If a user types the same thing 3 times, the partial hash will show up.  The the person notified will have access to that partial hash.  Which could itslef be used in a brute force  attack on the password.
16:03:09 <admiyo> "If the feature is enabled, it will expose N (N=5 by default) characters of hash
16:03:09 <admiyo> 91
16:03:09 <admiyo> of bad password. These symbols will not be communicated with the response, but
16:03:09 <admiyo> 92
16:03:09 <admiyo> will be sent via messages."
16:03:15 <admiyo> I think that this is OK
16:03:35 <admiyo> if you have access  to event notification, you have access to  the user database  and the main password hash.
16:03:40 <admiyo> I mean, likely.
16:03:53 <d34dh0r53> yeah, likely
16:04:26 <admiyo> if someone only had access to the event notification, it WOULD expand the attack window, but that has to be configured by Keystone admin, and so the end effect is the same
16:04:29 <d34dh0r53> That's my concern is that a potentially good password hash will be written somewhere
16:04:45 <admiyo> I think it is a valid concern
16:05:32 <admiyo> If you know that sha1 is used, and you brute force, you will end up with a few false positives, but it would still be enough info for you to crack simple passwords
16:06:15 <admiyo> I wonder if a better approach would  be to keep this information internal.
16:06:29 <stanislav-z> it also offers to provide a "personalization salt" in config - e.g. if event consumer doesn't have access to that config entry, it would be much more difficult to recover the password from partial hash
16:07:02 <stanislav-z> (without knowing that salt)
16:07:13 <admiyo> But the only thing  they  are then finding out is "it is the same as previous" or different.  COuldn't keystone keep track of that?
16:07:52 <d34dh0r53> Internally we could keep a counter of identical hash attempts and alert based on a threshold
16:08:36 <admiyo> So, I would not kill this effort outright, but I would suggest that the spec include a larger discussion about how this would be used, the security needed to protect it, and alternatives  and modifications to the approach?
16:08:51 <d34dh0r53> indeed
16:09:06 <stanislav-z> sound good, thanks
16:09:17 <d34dh0r53> Thank you Stanislav Zaprudskiy !
16:09:42 <d34dh0r53> #topic open discussion
16:09:55 <d34dh0r53> nothing on the agenda, does anyone have anything?
16:11:07 <d34dh0r53> we're over time, and I don't see any new bugs so I'll end it here
16:11:16 <d34dh0r53> #endmeeting