15:02:02 <d34dh0r53> #startmeeting keystone
15:02:02 <opendevmeet> Meeting started Wed Dec  3 15:02:02 2025 UTC and is due to finish in 60 minutes.  The chair is d34dh0r53. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:02 <opendevmeet> The meeting name has been set to 'keystone'
15:02:16 <d34dh0r53> Reminder: This meeting takes place under the OpenInfra Foundation Code of Conduct
15:02:25 <d34dh0r53> #link https://openinfra.dev/legal/code-of-conduct
15:02:35 <d34dh0r53> #topic roll call
15:02:59 <d34dh0r53> admiyo, bbobrov, crisloma, d34dh0r53, dpar, dstanek, hrybacki, lbragstad, lwanderley, kmalloc, rodrigods, samueldmq, ruan_he, wxy, sonuk, vishakha, Ajay, rafaelwe, xek, gmann, zaitcev, reqa, dmendiza[m], dmendiza, mharley, jph, gtema, cardoe, deydra
15:03:26 <cardoe> o/ (been a while since I've been able to listen in)
15:03:59 <gtema> O/
15:05:05 <moutazchaara[m]> O/
15:05:06 <cardoe> be curious to ask some questions of you folks during the free time
15:11:20 <moutazchaara[m]> gtema: Hi, [you were right about the token invalidation process as it should be part of the group changes.](https://review.opendev.org/c/openstack/keystone/+/967048/comments/abbdbb21_811d0a29)... (full message at <https://matrix.org/oftc/media/v1/media/download/AUNTxPSWNk-4tZUPaDAx94By_GFKtEcvc6Wi056FxTcCD3MrF-Vvm0ZTtwB8phqVs1jQ1jBhnkMAeQ827lzIUUlCebLN2eCwAG1hdHJpeC5vcmcvYlNqUUVDaVJYd2lvWlF5ZWVlYndvblJv>)
15:12:24 <gtema> I have seen that. I am this week on the PTO and not really having enough time to review the stuff. Your explanation looks like I was expecting, so it should be correct. Maybe we will have some code review comments, but functionally it should be correct
15:13:58 <moutazchaara[m]> gtema: np, ok so it make sense. i only found this out while doing manual testing. These scenarios are very hard to cover it with unit tests. Will try to add more coverage in the mean time...
15:14:50 <gtema> I know, I am not going to insist on the unittests
15:15:28 <d34dh0r53> sorry, got pulled away for a minute
15:16:09 <gtema> all good Dave
15:16:13 <d34dh0r53> #topic review past meeting work items
15:16:31 <d34dh0r53> #link https://meetings.opendev.org/meetings/keystone/2025/keystone.2025-11-26-15.07.html
15:16:38 <d34dh0r53> no action items from the last meeting
15:16:45 <d34dh0r53> #topic liaison updates
15:16:49 <d34dh0r53> nothing from me
15:17:00 <gtema> nothing here as well
15:17:30 <d34dh0r53> #topic specification OAuth 2.0 (hiromu)
15:17:34 <d34dh0r53> #link https://review.opendev.org/q/topic:bp%252Foauth2-client-credentials-ext
15:17:37 <d34dh0r53> #link https://review.opendev.org/q/topic:bp%252Fenhance-oauth2-interoperability
15:17:51 <d34dh0r53> no updates from me on this one
15:17:52 <d34dh0r53> #topic specification Secure RBAC (dmendiza)
15:17:53 <d34dh0r53> #link https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#z-release-timeline_
15:17:55 <d34dh0r53> 2026.1 Release Timeline
15:18:01 <d34dh0r53> Update oslo.policy in keystone to enforce_new_defaults=True
15:18:02 <d34dh0r53> Update oslo.policy in keystone to enforce_scope=True
15:18:33 <d34dh0r53> I didn't do my bespoke dmendiza ping
15:18:51 <gtema> oh lol
15:22:29 <cardoe> So that OAuth 2.0 parts just seem tests?
15:24:03 <gtema> yes, should be
15:26:10 <opendevreview> Takashi Kajinami proposed openstack/oslo.limit stable/2025.2: fixture: Mock out services, regions  https://review.opendev.org/c/openstack/oslo.limit/+/969406
15:26:22 <opendevreview> Takashi Kajinami proposed openstack/oslo.limit stable/2025.2: Handle missing endpoint, region, service  https://review.opendev.org/c/openstack/oslo.limit/+/969407
15:27:25 <cardoe> okay if I ask questions?
15:27:32 <d34dh0r53> Yeah, the OAuth 2.0 is tests that are blocked waiting on tacker
15:27:39 <d34dh0r53> cardoe: please do
15:28:29 <cardoe> So I previously mentioned I'm working with OIDC and with the keystone-ng work I just wanted to see if my approach to things was lining up with a use case.
15:29:00 <gtema> cool
15:29:45 <cardoe> So we've got folks that'll be assigned into different groups on the identity system and they'll get a groups claim for those groups.
15:30:14 <cardoe> I also want to have a good flow for device auth (for the CLI) but I also need machine authentication and that's one place where things are a little gross.
15:30:53 <gtema> the rust cli (osc) does support the cli flow
15:31:09 <cardoe> link?
15:31:18 <gtema> the machine access is to be solved with service accounts, but in the meanwhile you can use jwt auth
15:31:31 <gtema> https://github.com/gtema/openstack/
15:32:09 <cardoe> I've attempted to use v3oidcdeviceauthz but I use mnaser's websso plugin that I previously suggested we merge in.
15:32:24 <gtema> I also started updating docs for that - i.e. https://openstack-experimental.github.io/keystone/federation/keycloak.html
15:32:42 <gtema> the rust cli also support the websso plugin
15:33:01 <cardoe> We'll be getting websso into gophercloud soon as well.
15:33:57 <cardoe> So you think I should make a static account and then map the OIDC to it with client credentials?
15:34:00 <gtema> from my pov this is a no-go. I mean gophercloud itself is ok, but you can't have this in the TF or anything without a human attached
15:34:29 <gtema> which credentials do you mean?
15:34:31 <cardoe> other terraform providers have an option for web client auth
15:34:36 <cardoe> OIDC client credentials
15:35:05 <cardoe> As far as keycloak, I'd really hope we test against something other than keycloak. There's a number of keycloak-isms that I've tried to battle in keystone today.
15:35:20 <gtema> hmm, okay
15:35:25 <mnaser> we (historically) have told people to use app creds if they want to use terraform or whatnot
15:35:30 <gtema> wrt keycloak - my docs describe the okta as well
15:36:14 <gtema> and I am willing to add that for others (like Entra, etc), just need to find whether others offer dev-based accounts
15:36:16 <cardoe> Yeah making a service account and using app creds is what we do today.
15:36:41 <gtema> mnaser - that is also my "perception" right now
15:36:59 <cardoe> OIDC users can use app creds as well but they stop working once the expiration hits
15:37:10 <cardoe> gtema: dex is probably an easy test as well.
15:37:13 <gtema> cardoe - under service account I really mean service account - not a regular user with just the jwt login
15:37:28 <gtema> yeah, right, will look at dex soon
15:37:47 <cardoe> gtema: explain the service account... in keystone or you're saying in keycloak/etc?
15:38:15 <gtema> maybe even tomorrow - I am adding the "enabled" flag to idp and mapping in the v4 now and will check how complex it would be to add dex functests
15:38:46 <cardoe> So the last item is really the projects and how to map it all out.
15:38:54 <mnaser> we have built this gross use case once for a customer lol - https://vexxhost.github.io/atmosphere/user/auth.html#using-external-token-from-identity-provider
15:39:13 <cardoe> There's a want so that all the project_ids are identical across the different regions.
15:39:31 <gtema> cardoe - I was briefly describing this during the summit and also during the ptg. Conceptually this is a new type of account that can only auth using the jwt and has the logic similar to appcreds (binds to scopes with roles) and certain additional protection
15:39:43 <cardoe> So I've got a global keystone that's really just my official project store and group mapping.
15:39:58 <cardoe> gtema: ah okay. yeah I missed that session cause I was triple booked :/
15:40:35 <gtema> wrt project_id - just yesterday I was writing here that we can extend project creation api with specifying the id upon creation in v4
15:41:38 <cardoe> then regional keystones and have a custom auth plugin that's really just grabbing the projects so that they're in scope during the mapping time to creaet them
15:42:10 <cardoe> Could we not allow project_id in v3 for a specific permission?
15:42:23 <cardoe> I think domain_id can be set on a domain creation for a permission?
15:43:01 <gtema> i don't think it is possible now
15:43:58 <gtema> with global and regional keystones you also do not have the same user_id
15:44:23 <gtema> well, the unique_id is same, but i.e. for nova/neutron it is still different
15:44:40 <cardoe> So dealing with OpenStack Helm
15:45:01 <cardoe> I've tried to get the PTL to split up the service accounts... he's done nova, cinder, and neutron but a bit different than my goal.
15:45:25 <cardoe> My goal was each part of the service used its own... like nova-conductor vs nova-api
15:46:20 <cardoe> Since I've got a keystone in each region, the different users aren't really an issue... but for OIDC I'm using the OIDC sub for the user_id in mapping so its consistent
15:46:46 <gtema> should we maybe try to get a sync call to discuss on that? it is not a tiny thing for typing
15:47:19 <gtema> I also want to understand how people are thinking to approach the multiregional keystone - I have too many ? in my head
15:48:30 <cardoe> sure thing. I'm happy to help.
15:48:57 <cardoe> I just want an easy setup that follows along patterns that are in the goals of the project.
15:49:03 <cardoe> Don't wanna go off on my own thing
15:49:14 <gtema> I was even thinking into something similar to SCIM just for provisioning domains/projects with one central keystone managing that and those pushing this info to the regions (or pull)
15:49:44 <gtema> you are right in time - I am gathering the functional requirements to address this properly with ng
15:51:17 <gtema> sorry Dave Wilde (d34dh0r53) let's go back to the agenda to finish it quickly
15:51:29 <gtema> and we can continue chatting afterwards
15:53:11 <d34dh0r53> ack, I was just about to jump in
15:53:17 <d34dh0r53> next topic
15:53:21 <d34dh0r53> #topic specification Secuirty Compliance Testing (dmendiza)
15:53:24 <d34dh0r53> #link https://review.opendev.org/c/openstack/devstack/+/957969
15:53:51 <d34dh0r53> I don't think dmendiza is around
15:53:56 <d34dh0r53> #topic specification OpenAPI support (gtema)
15:54:01 <d34dh0r53> #link https://review.opendev.org/q/topic:%22openapi%22+project:openstack/keystone
15:54:10 <gtema> nothing on that
15:55:35 <d34dh0r53> #topic open discussion
15:56:11 <stanislav-z> do you review new bugs later? or now is the time?
15:56:11 <cardoe> mnaser: just looked at your link... that's exactly what we had to do..
15:56:32 <d34dh0r53> we're about to do that Stanislav Zaprudskiy
15:56:49 <d34dh0r53> #topic bug review
15:56:54 <d34dh0r53> #link https://bugs.launchpad.net/keystone/?orderby=-id&start=0
15:57:05 <d34dh0r53> looks like we have a couple of new bugs for Keystone
15:57:14 <d34dh0r53> first up
15:57:17 <d34dh0r53> #link https://bugs.launchpad.net/keystone/+bug/2131658
15:57:28 <gtema> I have responded in them just an hour ago
15:57:38 <d34dh0r53> ack, reading that now
15:59:14 <moutazchaara[m]> d34dh0r53: Seen that response thanks, and i think it can be included in the patch that i'm on.
15:59:39 <d34dh0r53> ack, thanks moutaz.chaara
15:59:41 <d34dh0r53> next up
15:59:54 <d34dh0r53> #link https://bugs.launchpad.net/keystone/+bug/2133706
16:00:17 <gtema> we touched that during the ptg iirc
16:00:37 <gtema> it was a coincidence and I do not know how we could resolve that now
16:00:48 <gtema> the thing is gone now
16:01:06 <bbobrov> i have an idea
16:01:13 <bbobrov> we could revert the change for 2025.1 only
16:01:47 <bbobrov> actually leave the module deleted in 2025.2 and 2026.1
16:01:56 <gtema> there was a proposal to backport the migration procedure to 2024.2
16:02:30 <bbobrov> 2024.2 is a SLURP release - optional
16:02:50 <gtema> and 2024.1 is now gone - great
16:02:52 <bbobrov> the official openstack messaging is that people can skip and upgrade directly from 2024.1 to 2025.1 without any downsides
16:04:16 <d34dh0r53> revert from 2025.1 seems reasonable, do you see downsides gtema ?
16:04:29 <bbobrov> also, 2025.1 is not supposed to run on python3.13 - https://governance.openstack.org/tc/reference/runtimes/2025.1.html
16:04:36 <bbobrov> only on 3.12
16:04:54 <gtema> hmm. okay
16:05:03 <bbobrov> 3.13 is also not mandatory for 2025.2 - https://governance.openstack.org/tc/reference/runtimes/2025.2.html
16:06:21 <gtema> https://review.opendev.org/c/openstack/keystone/+/959279 is the one I mentioned above
16:06:38 <gtema> it was actually into the 2024.1
16:07:35 <d34dh0r53> could we backport that into 2025.1?
16:08:06 <gtema> we were both unsure in the approach anyway ;-)
16:08:22 <gtema> but yes, it can
16:08:43 <gtema> it just need certain runtime for everything to be "migrated"
16:10:31 <bbobrov> is it really a backport? The patch doesn't seem to be present in the current master. It seems to be a direct commit to a stable branch
16:11:06 <gtema> right - we were discussing back those days exactly that we commit it directly back to the branch
16:11:15 <gtema> it SHOULD NOT exist in master
16:11:23 <bbobrov> so both restoration and migration patches need to be applied to 2025.1, and nothing needs to be done with the rest of releases.
16:11:49 <bbobrov> (except maybe a removal note moved to 2025.2)
16:12:38 <gtema> the main point of concern was and stays - this hashing method was marked deprecated 10 years ago. The fact that those are still there means nobody rotated password - this is just purely wrong
16:13:40 <d34dh0r53> indeed
16:14:38 <d34dh0r53> I'm good with either approach, but I think applying all of the patches to 2025.1 is the best option for long term maintainability
16:15:30 <gtema> from the maintainability we should not do anything and mark that before upgrading to 2025.1 affected passwords must be rotated
16:16:02 <bbobrov> keystone deleted a supported hashing mechanism without any warning.
16:16:06 <gtema> that is the safest and most reasonable approach fmpov
16:16:32 <gtema> bbobrov - people were warned 10 years ago that the hashing method is doomed
16:16:53 <bbobrov> gtema: could you point me to the deprecation notice about it in keystone?
16:16:54 <gtema> I know what you mean, still
16:17:35 <gtema> it was not such a deprecation how you would expect this, I wasn't with OpenStack back those days. Will try to grep through logs
16:19:48 <bbobrov> it seems that the notice is OSSN-0081
16:20:02 <gtema> see in https://docs.openstack.org/releasenotes/keystone/stein.html  "The deprecated option crypt_strength is removed now. It was only useful for sha512_crypt password hashes which has been superseded by more secure hashing implementations."
16:21:51 <gtema> but your hint to the OSSN is great
16:22:18 <gtema> anyway - I admit it was a mistake to land deprecation note and dropping in a single release though
16:22:39 <gtema> just a bad timing
16:25:36 <d34dh0r53> poor timing
16:26:06 <gtema> we could backport the deprecation note to 2024.2
16:26:23 <bbobrov> 2024.2 is a slurp release, it is usually skipped
16:26:31 <gtema> ah damn
16:27:19 <d34dh0r53> We could do as bbobrov suggested and just add a release note ensuring that folks rotate their passwords
16:27:50 <gtema> maybe add this as a pre-upgrade note to 2025.1
16:28:38 <bbobrov> that's the problem - there are many people who "own" different users, and running after them convincing them to rotate is time-consuming.
16:29:12 <gtema> from what we have seen this case usually affects service users and not the regular ones
16:29:36 <gtema> in the security compliance we can now also enforce password rotation ;-)
16:30:43 <bbobrov> stanislav-z: it is unclear from the bugreport - does keystone crash with a ValueError or is there a more graceful error handling?
16:31:46 <gtema> I think users are not able to login anymore, nothing else
16:31:49 <stanislav-z> it doesn't crash
16:35:28 <bbobrov> i see a huge traceback in the logs, but don't know what the users got - 401 or 500
16:35:57 <gtema> should be 401 - tracebacks are known
16:38:37 <gtema> we are heavily overtime now, let's come to the end pls
16:44:14 <gtema> Dave Wilde (d34dh0r53): ??
16:45:10 <bbobrov> maybe he decided to end the meeting asynchronously and is already asleep :D
16:45:34 <gtema> he has a lunch time by now
16:45:55 <gtema> I think I can't even end the meeting since he started it
16:46:03 <gtema> but lemme try
16:46:06 <gtema> #endmeeting