| opendevreview | OpenStack Proposal Bot proposed openstack/keystone master: Imported Translations from Zanata https://review.opendev.org/c/openstack/keystone/+/930663 | 04:08 |
|---|---|---|
| stanislav-z | Hello 👋. A while ago a spec was proposed to "Include bad password details in audit messages" - https://review.opendev.org/c/openstack/keystone-specs/+/915482. I was working on it recently, including spec update and possible implementation for it https://review.opendev.org/c/openstack/keystone/+/932423 - would you please have a look at both and provide your feedback/reviews here or in the changes? I'm also open for suggestion | 10:58 |
| stanislav-z | for how to move it forward. Thanks! | 10:58 |
| bbobrov | cardoe: re your question about inherited roles: yes, they do work like that. Inherited role assignments on a domain provide assignments on all its project | 12:07 |
| gtema | Dave Wilde (d34dh0r53): I'm on PTO this will not participate this week | 13:14 |
| admiyo | I am here, for once... | 14:03 |
| admiyo | bbobrov, is that new? I have to refresh on how it works, but I thought domain roles and project roles were separate, and you needed to have an assignment on domain-as-a-project for that to work. | 14:06 |
| cardoe | There's an inherited API endpoint. | 14:06 |
| admiyo | https://adam.younglogic.com/2016/11/keystone-domains-are-projects/ | 14:06 |
| cardoe | Which I can confirm does work like bbobrov mentioned. | 14:06 |
| * admiyo needs to update that article | 14:07 | |
| cardoe | https://docs.openstack.org/api-ref/identity/v3/#assign-role-to-group-on-projects-owned-by-a-domain | 14:07 |
| admiyo | Cool. THat is a nice extension. Well done | 14:08 |
| admiyo | Are we still in meeting mode? | 14:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/keystone master: Re-initiate OSA rolling upgrade tests https://review.opendev.org/c/openstack/keystone/+/932512 | 14:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/keystone master: [DNM] Ensure that upgrade test fails https://review.opendev.org/c/openstack/keystone/+/932513 | 14:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/keystone master: [DNM] Ensure that upgrade test fails https://review.opendev.org/c/openstack/keystone/+/932513 | 14:16 |
| d34dh0r53 | #startmeeting keystone | 15:05 |
| opendevmeet | Meeting started Wed Oct 16 15:05:22 2024 UTC and is due to finish in 60 minutes. The chair is d34dh0r53. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:05 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:05 |
| opendevmeet | The meeting name has been set to 'keystone' | 15:05 |
| d34dh0r53 | Reminder: This meeting takes place under the OpenInfra Foundation Code of Conduct | 15:05 |
| d34dh0r53 | #link https://openinfra.dev/legal/code-of-conduct | 15:05 |
| d34dh0r53 | #topic roll call | 15:06 |
| d34dh0r53 | admiyo, bbobrov, crisloma, d34dh0r53, dpar, dstanek, hrybacki, lbragstad, lwanderley, kmalloc, rodrigods, samueldmq, ruan_he, wxy, sonuk, vishakha, Ajay, rafaelwe, xek, gmann, zaitcev, reqa, dmendiza[m], dmendiza, mharley, jph, gtema, cardoe | 15:06 |
| d34dh0r53 | o/ | 15:06 |
| mhen | o/ | 15:07 |
| d34dh0r53 | #topic review past meeting work items | 15:08 |
| d34dh0r53 | #link https://meetings.opendev.org/meetings/keystone/2024/keystone.2024-10-09-15.05.html | 15:08 |
| d34dh0r53 | no action items from last week | 15:08 |
| d34dh0r53 | #topic liaison updates | 15:08 |
| d34dh0r53 | nothing from VMT or releases | 15:09 |
| cardoe | \o | 15:10 |
| d34dh0r53 | #topic specification OAuth 2.0 (hiromu) | 15:11 |
| d34dh0r53 | #link https://review.opendev.org/q/topic:bp%252Foauth2-client-credentials-ext | 15:11 |
| d34dh0r53 | #link https://review.opendev.org/q/topic:bp%252Fenhance-oauth2-interoperability | 15:11 |
| d34dh0r53 | External OAuth 2.0 Specification | 15:11 |
| d34dh0r53 | #link https://review.opendev.org/c/openstack/keystone-specs/+/861554 (merged) | 15:11 |
| d34dh0r53 | OAuth 2.0 Implementation | 15:11 |
| d34dh0r53 | #link https://review.opendev.org/q/topic:bp%252Fsupport-oauth2-mtls | 15:11 |
| d34dh0r53 | OAuth 2.0 Documentation | 15:11 |
| d34dh0r53 | #link https://review.opendev.org/c/openstack/keystone/+/838108 (merged) | 15:11 |
| d34dh0r53 | #link https://review.opendev.org/c/openstack/keystoneauth/+/838104 (merged) | 15:11 |
| d34dh0r53 | no updates from me on any of this | 15:12 |
| d34dh0r53 | #topic specification Secure RBAC (dmendiza[m]) | 15:12 |
| d34dh0r53 | #link https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#z-release-timeline_ | 15:12 |
| d34dh0r53 | 2024.1 Release Timeline | 15:12 |
| d34dh0r53 | Update oslo.policy in keystone to enforce_new_defaults=True | 15:12 |
| d34dh0r53 | Update oslo.policy in keystone to enforce_scope=True | 15:12 |
| d34dh0r53 | not sure if dmendiza is around, he messaged me that he might be away-ish this morning | 15:12 |
| d34dh0r53 | ok, moving on | 15:15 |
| d34dh0r53 | #topic specification OpenAPI support (gtema) | 15:15 |
| d34dh0r53 | #link https://review.opendev.org/q/topic:%22openapi%22+project:openstack/keystone | 15:15 |
| d34dh0r53 | https://review.opendev.org/c/openstack/keystone/+/925020 could now also land to ease api-ref work | 15:15 |
| d34dh0r53 | gtema: api-ref work started | 15:15 |
| d34dh0r53 | I think I also saw that gtema (Artem Goncharov) is on PTO | 15:16 |
| d34dh0r53 | looks like it, moving on | 15:17 |
| d34dh0r53 | #topic specification domain manager (mhen) | 15:17 |
| d34dh0r53 | #link https://review.opendev.org/q/topic:%22domain-manager%22 | 15:17 |
| d34dh0r53 | tempest core lib patch has been merged, only keystone-tempest-plugin left | 15:17 |
| d34dh0r53 | created a patchset for documentation: https://review.opendev.org/c/openstack/keystone/+/928135 | 15:17 |
| mhen | yes, only tempest plugin and documentation patchsets are left | 15:18 |
| mhen | got a lot of upvotes by now - is there anything missing still? | 15:19 |
| d34dh0r53 | ok, I'll bug dmendiza and Grzegorz Grasza to push those through | 15:19 |
| mhen | nice, thanks | 15:19 |
| d34dh0r53 | mhen: I think we're just waiting on final reviews | 15:19 |
| d34dh0r53 | ack, YW | 15:19 |
| mhen | ok | 15:19 |
| d34dh0r53 | next up | 15:20 |
| d34dh0r53 | #topic specification Type annotations (stephenfin) | 15:20 |
| d34dh0r53 | #link https://review.opendev.org/q/project:openstack/keystoneauth+topic:typing | 15:20 |
| d34dh0r53 | This is just pending reviews now. I will push the remaining patches as soon as a sufficient quantity of the current ones land. | 15:20 |
| d34dh0r53 | many reviews are owed | 15:20 |
| admiyo | Sorry I missed the secure RBAC topic...I was distracted by discussing it over in #openstack-nova | 15:22 |
| d34dh0r53 | ahh, no worries admiyo | 15:23 |
| admiyo | So...while I am not 100% certain, I think the solution in place means we should close bug 96869 | 15:23 |
| Vivek-vnetpops | Dear community. | 15:23 |
| admiyo | It is the longest open bug in the tracker, and it is addrtessed, I think, but the secure RBAC approach. I don't feel 100% certain there. | 15:24 |
| admiyo | maybe like, 55% | 15:24 |
| d34dh0r53 | nooooo that bug is legendary | 15:24 |
| admiyo | I think that the solution is kinda wrong | 15:25 |
| admiyo | and I am pretty sure that nova is self contradictory by ignoring the project Id for the operation, but requiring a project scoped token | 15:25 |
| d34dh0r53 | #link https://bugs.launchpad.net/keystone/+bug/968696 | 15:25 |
| admiyo | and they are certainly wrong for marking it 'wishlist" and "won't fix" | 15:25 |
| admiyo | BUT.... | 15:26 |
| d34dh0r53 | that's the bug being referred to | 15:26 |
| admiyo | Aye. sorry for missing the lin | 15:26 |
| admiyo | k | 15:26 |
| admiyo | I think that ...if and only iff ADMIN is reserved for operations that should never be allowed to be granted to managers of projects, then it is OK to say that they have addressed it | 15:27 |
| d34dh0r53 | dmendiza: is really the one to talk to about this | 15:27 |
| admiyo | as I recall, there were cases where reources got orphaned and you had to be aadmin to clean them up, which means that a project MANAGER would not have been able to clean up after a mistake, and thuse projects were not self maintaining | 15:28 |
| admiyo | It is a PTL issue. It is an openstack wide issue, and it limits adoption. They are pushing the fix back on keystone. | 15:28 |
| admiyo | And I don't know that Keystone alone can fix it, and I would argue that the fix is very fragile | 15:29 |
| cardoe | So with Secure RBAC being a top level priority, I think this is something we can work to clean up. | 15:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/keystone master: Re-initiate OSA rolling upgrade tests https://review.opendev.org/c/openstack/keystone/+/932512 | 15:30 |
| cardoe | I'd like to work with some folks on defining some more user personas which I think would be helpful. | 15:30 |
| admiyo | So if ANY service in openstack requires ADMIN for any user-level operation, and they assign it, they have opened up a security hole in Nova | 15:30 |
| cardoe | We don't have to craft roles for each one but we can say "this persona should not ever have admin role" or some such. | 15:30 |
| admiyo | So, from a keystone perspective, we can be dilligent and avoid things like we used to do: | 15:32 |
| cardoe | I think that might help downstream or operators by understanding those personas. Cause I feel like those leaks happen from downstream / operators trying to make something work how they think and then nova accepting a band aid. | 15:32 |
| admiyo | you needed admin to add a user to a project | 15:32 |
| admiyo | which means projects were not self managing | 15:32 |
| admiyo | but that has to be true of every service in the catalog | 15:32 |
| admiyo | and, yeah, just default policy. People can always break things via custom policy | 15:33 |
| admiyo | Are nested projects still a thing? | 15:33 |
| admiyo | That actually helped with a lot of the operators concerns | 15:34 |
| cardoe | They are documented and the Red Hat docs talk about them. | 15:36 |
| admiyo | Anyway,I had a long write up of this from when I was assigned the bug: https://adam.younglogic.com/2017/05/fixing-bug-96869/ | 15:36 |
| admiyo | The issue was that operators wanted to be able to use a single, project scoped token to do operations across the whole tree. It was, and is, a mess | 15:37 |
| d34dh0r53 | Let me talk with dmendiza about this, there may be a case for finally closing that bug with the domain manager role | 15:37 |
| admiyo | Because they don't want to have to authenticate to each project | 15:37 |
| admiyo | No...it still does'nt handle it. Apologies, pulling stuff from long term memory.... | 15:38 |
| d34dh0r53 | What is lacking in that role? | 15:39 |
| admiyo | the token is scoped to a project but they want to do a global operation, and there is no way for Nova to know that a given project is nested under another project | 15:39 |
| admiyo | it is fine for Keystone, not for the other services | 15:39 |
| mhen | I guess the problem is that domain-ownership awareness is not really there in other services for their resources | 15:39 |
| admiyo | I think the "is admin project" hack is still needed. | 15:40 |
| admiyo | mhen, yes | 15:40 |
| mhen | it would require a lot more work for some services to properly adapt the domain manager persona then | 15:40 |
| admiyo | so...IF we create a majik ADMIN project, and if we add a flag to the token saying "this is an admin project" and if Nova checks for that as well as ADMIN then things would work | 15:40 |
| admiyo | domain tokens never go to other services | 15:41 |
| admiyo | domain were a mistake. | 15:41 |
| admiyo | as was calling things projects instead of tennants | 15:41 |
| admiyo | sins of the past that will be visted on our descendants.... | 15:41 |
| admiyo | but anyway | 15:41 |
| admiyo | if a token is sent to nova with "is_admin_project=yes" and they chose to ignore it, that is on them | 15:42 |
| admiyo | Its not a hard check to implement, and it can be done in keystone middleware if we really want, but that is a whole nother approach that I abandoned | 15:43 |
| admiyo | That way, if, say HEAT requires Admin for something, and that slips through, Nova could protect itself by also required is_admin_project | 15:44 |
| admiyo | anyway, I have said my piece, and will stop beat d34dh0r53 | 15:45 |
| d34dh0r53 | lol | 15:45 |
| d34dh0r53 | thanks for bringing it up, I mistakenly thought the domain manager work helped to fix this but I think there is still work to do | 15:46 |
| admiyo | also...we never addressed how to limit things that required acccess to two proejcts | 15:46 |
| admiyo | I think there was an issue where deleting a project in keystone orphaned all the resources everywhere. | 15:47 |
| admiyo | And...I think I had a fix for that | 15:47 |
| admiyo | but I don';t thik that was RBAC | 15:47 |
| mhen | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/project-cleanup.html | 15:47 |
| mhen | :) | 15:47 |
| admiyo | AH yes | 15:47 |
| admiyo | https://review.opendev.org/c/openstack/keystone/+/664746 | 15:47 |
| admiyo | It was KINDA an RBAC issue in that you needed to get a system admin to do all your clean up. This avoids that case. | 15:48 |
| * admiyo really done now | 15:49 | |
| admiyo | meeting over? | 15:52 |
| d34dh0r53 | Thanks admiyo | 15:52 |
| d34dh0r53 | #topic specification Include bad password details in audit messages (stanislav-z) | 15:52 |
| d34dh0r53 | # link https://review.opendev.org/c/openstack/keystone-specs/+/915482 | 15:52 |
| d34dh0r53 | # link https://review.opendev.org/c/openstack/keystone/+/932423 | 15:52 |
| d34dh0r53 | s/// | 15:52 |
| d34dh0r53 | s/// | 15:53 |
| stanislav-z | Hello! I'm looking for any feedback on both. And in the meantime was going to work on adding tests for the implementation | 15:53 |
| admiyo | @stanislav-z, is the idea that an admin can compare the hash to the hash in the database and say "yep they don't match?" | 15:54 |
| stanislav-z | there is no relation to the database hash. the idea is that when there is a bruteforce attack the hash will be changing frequently. while if e.g. some end-user automation got stuck with the old password the hash won't be changing. so by checking the hash, it may be possible to perform a more educated guess if user is experiencing an attack or just has some broken automation | 15:57 |
| stanislav-z | by checking how hash is changing* | 15:57 |
| admiyo | Ah. THat makes more sense. But any change in the users passwords attempt end up as differnt hashes, and look like an attack? | 15:57 |
| admiyo | password pa$$word p@$$word p@$$w0rd | 15:58 |
| stanislav-z | only if something/someone would be continuing to submit the old (wrong) password. no hashes will be showing up for successful authentication events | 15:58 |
| admiyo | but is a user submits password pa$$word p@$$word p@$$w0rd and they all get hashed, it is safe to put the first 5 chars of the hash in the log? | 15:59 |
| d34dh0r53 | According to the spec the N characters isn't logged, it's sent in a message, is that correct? | 16:00 |
| stanislav-z | with the default hashing function from possible WIP the hash size would be 88 characters. providing just 5 (default; configurable) should be safe - and it will only appear in event notifications | 16:01 |
| stanislav-z | that's correct. but N - is the number of character that would passed to the event (not how much would be truncated from the hash) | 16:02 |
| admiyo | same thing, though. Written down somewhere not in ephemeral memory and the database. Thus widening exposure to the hash. If a user types the same thing 3 times, the partial hash will show up. The the person notified will have access to that partial hash. Which could itslef be used in a brute force attack on the password. | 16:02 |
| admiyo | "If the feature is enabled, it will expose N (N=5 by default) characters of hash | 16:03 |
| admiyo | 91 | 16:03 |
| admiyo | of bad password. These symbols will not be communicated with the response, but | 16:03 |
| admiyo | 92 | 16:03 |
| admiyo | will be sent via messages." | 16:03 |
| admiyo | I think that this is OK | 16:03 |
| admiyo | if you have access to event notification, you have access to the user database and the main password hash. | 16:03 |
| admiyo | I mean, likely. | 16:03 |
| d34dh0r53 | yeah, likely | 16:03 |
| admiyo | if someone only had access to the event notification, it WOULD expand the attack window, but that has to be configured by Keystone admin, and so the end effect is the same | 16:04 |
| d34dh0r53 | That's my concern is that a potentially good password hash will be written somewhere | 16:04 |
| admiyo | I think it is a valid concern | 16:04 |
| admiyo | If you know that sha1 is used, and you brute force, you will end up with a few false positives, but it would still be enough info for you to crack simple passwords | 16:05 |
| admiyo | I wonder if a better approach would be to keep this information internal. | 16:06 |
| stanislav-z | it also offers to provide a "personalization salt" in config - e.g. if event consumer doesn't have access to that config entry, it would be much more difficult to recover the password from partial hash | 16:06 |
| stanislav-z | (without knowing that salt) | 16:07 |
| admiyo | But the only thing they are then finding out is "it is the same as previous" or different. COuldn't keystone keep track of that? | 16:07 |
| d34dh0r53 | Internally we could keep a counter of identical hash attempts and alert based on a threshold | 16:07 |
| admiyo | So, I would not kill this effort outright, but I would suggest that the spec include a larger discussion about how this would be used, the security needed to protect it, and alternatives and modifications to the approach? | 16:08 |
| d34dh0r53 | indeed | 16:08 |
| stanislav-z | sound good, thanks | 16:09 |
| d34dh0r53 | Thank you Stanislav Zaprudskiy ! | 16:09 |
| d34dh0r53 | #topic open discussion | 16:09 |
| d34dh0r53 | nothing on the agenda, does anyone have anything? | 16:09 |
| d34dh0r53 | we're over time, and I don't see any new bugs so I'll end it here | 16:11 |
| d34dh0r53 | #endmeeting | 16:11 |
| opendevmeet | Meeting ended Wed Oct 16 16:11:16 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:11 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/keystone/2024/keystone.2024-10-16-15.05.html | 16:11 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/keystone/2024/keystone.2024-10-16-15.05.txt | 16:11 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/keystone/2024/keystone.2024-10-16-15.05.log.html | 16:11 |
| d34dh0r53 | Thanks everyone! | 16:11 |
| admiyo | d34dh0r53, it just occured to me that the mistake with 968696 was asking the operators. Of course they said no. It broke their workflows. The right people to ask were the people that refused to adopt openstack for security reasons.... | 16:13 |
| d34dh0r53 | admiyo: yeah, that's a really good point | 16:13 |
| admiyo | I was tempted to submit a talk called "OpenStack is Doomed and Its ALL MY FAULT." | 16:15 |
| admiyo | https://review.opendev.org/c/openstack/keystone-specs/+/391624 Holy cow! That merged, I guess I better get on implementing it....heh | 16:26 |
| admiyo | 9 years later | 16:26 |
| d34dh0r53 | hah | 16:33 |
| cardoe | gtema: I updated that OIDC documentation as you asked. | 16:34 |
| opendevreview | Oria Weng proposed openstack/keystone master: Add JSON schema to `limits` https://review.opendev.org/c/openstack/keystone/+/931528 | 19:27 |
| *** gmann is now known as gmann_afk | 20:58 | |
| *** gmann_afk is now known as gmann_ | 21:41 | |
| *** gmann_ is now known as gmann | 21:41 | |
| admiyo | d34dh0r53, so, I just had a long chat with sean-k-mooney[m] and I think I get where this is going. So the rule for Keystone grants needs to change. I just generated the policy for it, and here is what it expands to | 21:58 |
| admiyo | #"identity:create_grant": "(rule:admin_required) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domai | 21:59 |
| admiyo | n.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" | 21:59 |
| admiyo | the problem is that initial or | 21:59 |
| admiyo | well, actually, the problem is that the role required is admin, but that the policy rule is only going to get as far as that first check | 22:00 |
| admiyo | if I want to make it possible for the admiyo user to assign roles on the fubar project, I need to grant admiyo the admin role. If instead it was a more limited role (project_manager) then everything would work as expected | 22:01 |
| admiyo | but the way that rule is written, nothing will ever get caught by any of the rules after the initial (rule:admin_required) | 22:01 |
| admiyo | So, if you were to change the default policy today, you would break no one | 22:02 |
| admiyo | because the only people able to use that already have the admin role | 22:02 |
| admiyo | but...moving forward, they would be able to make more localized role assignments | 22:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!