15:00:34 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:34 <opendevmeet> Meeting started Tue Aug 15 15:00:34 2023 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:34 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:39 <noonedeadpunk> #topic rollcall
15:00:41 <noonedeadpunk> o/
15:01:51 <jrosser> o/ hello
15:03:15 <noonedeadpunk> #topic office hours
15:03:24 <noonedeadpunk> We have couple of things going on
15:03:47 <noonedeadpunk> 1. Heat and telemetry are blocked, we need to land https://review.opendev.org/c/openstack/openstack-ansible/+/888517 to fix them
15:04:34 <noonedeadpunk> 2. Fix for keystone regarding passwords that are longer 54 symbols has landed to master. Backports are proposed but not merged yet
15:04:46 <noonedeadpunk> This blocks our stable bracnhes upgrade jobs
15:05:08 <noonedeadpunk> I'm inlcined not to merge anything to our repos and just wait for keystone fixes
15:05:54 <NeilHanlon> o/
15:06:23 <noonedeadpunk> Regarding _member_ role that's not invalid - I've proposed patch to mark the role as implied. And series of patches to stop reffering to _member_
15:06:34 <noonedeadpunk> #link https://review.opendev.org/q/topic:osa%252Fmember_role
15:07:02 <jrosser> this is a big surprise at upgrade
15:07:07 <noonedeadpunk> Interestingly, rocky failed a lot there. Either on tempest or with timeouts
15:07:31 <noonedeadpunk> Yeah... Keystone has brought plenty of surprises this time I would say
15:07:39 <jrosser> do we need a releasenote about the member changes, or should the patches take care of it automatically?
15:07:46 <noonedeadpunk> Not saying about invalid tokens
15:08:11 <noonedeadpunk> That is very good question
15:08:33 <noonedeadpunk> I was thinking about release note, but I failed to find when Keystone has marked _memeber_ as deprecated
15:08:49 <noonedeadpunk> So I got slightly confused on what to say in a release note
15:08:55 <jrosser> wheres the implied role patch?
15:09:52 <noonedeadpunk> um.... don't tell me I haven't pushed it and just did `git reset origin/master --hard`
15:10:29 * jrosser not seeing it under the topic link
15:10:49 <noonedeadpunk> ah
15:10:50 <noonedeadpunk> https://review.opendev.org/c/openstack/openstack-ansible/+/891473
15:11:24 <jrosser> aaaah ok so this is addressed at upgrade
15:11:35 <jrosser> we never skip 2023.1? :)
15:12:01 <noonedeadpunk> According to officially supported upgrade process - we should not
15:12:05 <jrosser> maybe *that* is actually where the releasenote needs to go
15:15:58 <noonedeadpunk> ok, what we wanna mention there?
15:16:16 <noonedeadpunk> refference the keystone bug that application credentials are still borked?
15:16:27 <noonedeadpunk> (with _member_ role)?
15:16:58 <jrosser> maybe saying that this is the release which OSA transitions from _member_ to member
15:17:16 <jrosser> handled automatically in upgrade scripts blah blah
15:17:39 <jrosser> or rather, existing deployments get an implied role added for backward compatibility
15:17:40 <noonedeadpunk> also, I'd say that ones who want skip slurp releases during upgrades - doing that on their own risk and will have plenty of other hacks. And should execute all upgrade steps regardless
15:18:25 <noonedeadpunk> yeah, last one sounds good
15:18:57 <noonedeadpunk> Will add reno then
15:19:26 <noonedeadpunk> And I guess we'd need to backport https://review.opendev.org/q/topic:osa%252Fmember_role to 2023.1 as well
15:19:35 <jrosser> like you say about waiting for keytone to merge password length fixes....
15:19:51 <jrosser> we have also not taken 2023.1 upgrade past lab tests because of all the keystone things
15:20:01 <jrosser> also waiting for proper upstream fixes to merge
15:20:22 <noonedeadpunk> yeah, I was going to look into flushing memcached after keystone upgrade as well
15:20:39 <noonedeadpunk> not sure when exactly to run this though
15:20:41 <noonedeadpunk> and where to add
15:21:12 <noonedeadpunk> sounds like adding a variable and running either in keystone playbook or in the role itself is by far only ways
15:21:14 <jrosser> though that might be not needed once the password length thing is addressed?
15:21:37 <noonedeadpunk> I think these are 2 independent regressions
15:21:41 <noonedeadpunk> caused by different patches
15:22:08 <noonedeadpunk> And I'm not sure how to fix this one on keystone side
15:24:02 <noonedeadpunk> #link https://bugs.launchpad.net/keystone/+bug/2029134
15:24:39 <noonedeadpunk> I asked patch submitter to have a look into that, but not sure if they did...
15:25:07 <noonedeadpunk> But if that's trivial to workaround - probably we should do that then...
15:25:22 <noonedeadpunk> or well - possible to workaround at very least.
15:26:51 <jrosser> i did look at that with andrewbonney and it looks like something that needs patching in keystone
15:27:22 <noonedeadpunk> but issue goes after cache timeout, from what I've read?
15:27:28 <jrosser> yes thats right
15:27:36 <noonedeadpunk> so, if flush cache...?
15:27:41 <jrosser> ah so this is where memcached flush
15:27:42 <jrosser> uyes
15:28:48 <noonedeadpunk> I haven't tested though, as that sounds not super trivial to reproduce/witness and ensure that flushing cache is not a co-incidence
15:29:13 <noonedeadpunk> But was going to try it out later today
15:29:27 <jrosser> you need a deployed cloud with services running
15:29:37 <jrosser> i suspect with no monitoring you wont see anything
15:30:06 <noonedeadpunk> Also openstack_resources role looks very close to get it's initial state. Tempest is passing now while using the role. Octavia was almost passing - keypair was owned by a wrong user.
15:30:28 <noonedeadpunk> Yeah, and monitoring should be also "proper" one
15:30:53 * noonedeadpunk not even sure that their production monitoring will catch it either
15:31:30 <noonedeadpunk> But I guess, that if I run tempest test, then upgrade keystone, running tempest again should fail?
15:31:53 <noonedeadpunk> as it's interaction between services that is affected?
15:32:00 <noonedeadpunk> like nova can't query neutron or placement
15:33:03 <jrosser> from our notes, as soon as you upgrade keystone then everything else fails to auth
15:33:58 <jrosser> because it expects oauth2_thumbprint in the tokens, and it's missing
15:34:29 <noonedeadpunk> but not CLI? As cli does not cache tokens?
15:34:35 <noonedeadpunk> (I assume)
15:35:03 <jrosser> i assume not
15:35:06 <noonedeadpunk> I guess anything that uses keystone_authtoken
15:35:26 <jrosser> but iirc our alerting (haproxy?) all went bananas at that point
15:35:44 <jrosser> andrew is back tomorrow and might be able to say exactly what it did
15:36:10 <noonedeadpunk> hm... maybe patch to keystone is more trivial then I thought
15:36:32 <jrosser> right - it just needs to not try to parse that field if it's absent
15:36:55 <jrosser> or have some non-failing accessor method to get() it
15:38:13 <noonedeadpunk> Yeah, I will try to patch that actually as well
15:38:24 <noonedeadpunk> As it's failing here https://opendev.org/openstack/keystone/src/commit/f6a0cce4409232d8ade69b7773dbabcf4c53ec0f/keystone/common/render_token.py#L145-L148
15:39:05 <jrosser> thats it
15:39:06 <noonedeadpunk> Not sure if that's the only place that needs adjustment
15:39:21 <noonedeadpunk> as such assumptions are everywhere in code kind of
15:39:41 <noonedeadpunk> So it could be just first place
15:40:05 <noonedeadpunk> anyway, will see :)
15:43:13 * noonedeadpunk hopes to get keystone fixes to land before September
15:44:00 <jrosser> i think there is at least a regular keystone meeting now
15:44:28 <noonedeadpunk> yup, was on previous one
15:47:13 <opendevreview> James Denton proposed openstack/openstack-ansible-os_nova master: Allow Glance region to be set via variable  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/891515
16:01:53 <noonedeadpunk> #endmeeting