15:00:48 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:48 <opendevmeet> Meeting started Tue Dec 20 15:00:48 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:48 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:48 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:53 <noonedeadpunk> #topic rollcall
15:00:58 <noonedeadpunk> o/
15:01:09 <jamesdenton> o/
15:01:17 <damiandabrowski> hi!
15:01:33 <mgariepy> hey o/
15:02:58 <noonedeadpunk> #topic office hours
15:03:20 <noonedeadpunk> So, congrats and thanks for all contributors who put an effort for Zed release
15:03:46 <noonedeadpunk> I wanted to count contributors but failed in a way time-wise for that :(
15:03:52 <jamesdenton> \o/
15:03:58 <damiandabrowski> \o/
15:04:16 <jamesdenton> big thanks to the heavy hitters who keep it all together
15:05:19 <noonedeadpunk> We obviously have issue with neutron :D
15:05:38 <noonedeadpunk> The thing that hits us on Zed with updated SHA for neutron, also hits for master
15:05:55 <noonedeadpunk> So we need to find the reason for https://review.opendev.org/c/openstack/openstack-ansible/+/867954
15:06:00 <jamesdenton> is there a known upstream bug by chance?
15:06:19 <jamesdenton> i can try to look into that
15:06:50 <noonedeadpunk> I reached neutron folks but they were not helpful and were saying that we try to do weird things liek create network multiple times. But likely I just need to narrow down to specific commit that brings this regression...
15:07:07 <noonedeadpunk> The thing is that only metal deploy does fail
15:07:12 <noonedeadpunk> and LXC is good
15:07:27 <jamesdenton> it's the lock wait timeout issue, right?
15:07:29 <noonedeadpunk> so it's smth when running neutron-server outside of the container
15:07:35 <noonedeadpunk> well... I guess
15:07:44 <noonedeadpunk> at least that's what I see in neutron logs.
15:08:14 <noonedeadpunk> I'd assume it can be realted to the sqlalchemy changes...
15:08:14 <jamesdenton> ok, i will try one locally today and see what's upo
15:08:50 <noonedeadpunk> As there was a workaround during branching/initial release that afterwards should be replaced with proper fix
15:09:21 <noonedeadpunk> I won't have time for that until end of the week for sure
15:11:46 <noonedeadpunk> Also I'm not really sure what we are testing until 867954 lands. So I'd say it's better to sort it out
15:12:11 <noonedeadpunk> Once it's done I'm thinking to check ansible-core 2.14 for 2023.1
15:13:49 <noonedeadpunk> I'm not sure what to do with openstack collection though
15:14:08 <noonedeadpunk> Also in terms of Zed. They were planning release of 2.0 early 2023
15:14:28 <noonedeadpunk> And now we have SHA in the middle of the master that obviously has some bugs and things in progress
15:15:29 <NeilHanlon> o/, here & late (as usual)
15:15:52 <noonedeadpunk> But I guess it would be good to backport usage of 2.0 after it get's tagged to Z
15:16:54 <noonedeadpunk> other then that we should add upgrade jobs from Yoga which means updating our upgrade script in some way
15:17:11 <noonedeadpunk> that will allow selection of destination?
15:17:18 <noonedeadpunk> or well, ask for the input?
15:18:41 <noonedeadpunk> I wonder if it should be smth like dialog or just super simple
15:18:43 <jamesdenton> meaning, Y->Z or Y->A?
15:18:47 <noonedeadpunk> Yup
15:18:55 <noonedeadpunk> or well
15:19:34 <noonedeadpunk> yes, options are correct, but I've jsut realized that for upgrade script we just checkout to destination and then run it
15:20:21 <noonedeadpunk> so it sounds, like we should detect just where we are based on the current git state and verify if it's valid or not....
15:20:23 <damiandabrowski> +1 for backporting openstack ansible collection later if 1.x works fine with Zed
15:20:32 <damiandabrowski> but as I can see, it does not officially support Zed :/
15:20:35 <damiandabrowski> "1.x.x releases of Ansible OpenStack collection are compatible with OpenStack SDK 0.x.x prior to 0.99.0 only (OpenStack Yoga and earlier)."
15:21:23 <noonedeadpunk> We use >1 but <2.0. So we use SHA from master, and they will create 2.0 from master as well, but in fairly later commit
15:21:34 <noonedeadpunk> 2.0 simply doesn't exist atm
15:21:58 <damiandabrowski> ah okok
15:22:03 <damiandabrowski> fine for me
15:22:26 <noonedeadpunk> so we're using even not alpha but some "work in progress" version that is stable enough according to our CI
15:23:45 <noonedeadpunk> the more I think about upgrsade jobs the more I feel complexity
15:25:30 <noonedeadpunk> Oh. Also we discussed yestarday that for PKI role it would be great to allow using *_pipe modules from crypto. Which means that module can accept cert or private key from variable rather from file. And variable can be set to any lookup plugin eventually.
15:25:52 <noonedeadpunk> That would enable us to store certs/private keys in vault/sops/encrypted with ansible-vault
15:26:58 <noonedeadpunk> The only thing is how to save/create/generate certs for the first time. as while reading them is relatively easy, creating is not in fact
15:28:10 <noonedeadpunk> I think question I have for everyone is - how do you save/store/manage certs? Hashi vault?
15:28:38 <noonedeadpunk> trying to narrow down options that we want to implement outside of file as it is today
15:31:43 <damiandabrowski> ouh, these *_pipe modules look promising
15:39:57 <jrosser> our deployments just have their certs using the pki role files on the deploy node today
15:40:07 <jrosser> but any other certs we have in hashi vault
15:40:46 <jrosser> i think that we should refactor the PKI role a bit as it stands to make a `file` backend and switch to the *_pipe modules
15:40:50 <noonedeadpunk> oh, btw, interesting question - how does hashi lookup module perform?
15:41:05 <jrosser> that should put in place all the structure needed to make the backend pluggable
15:41:34 <jrosser> we make it be in pre_tasks so it only calls it once, using lookup rather than the native module, but that is just history really
15:42:30 <noonedeadpunk> so while I do understand how to store in Vault, but I don't really understand how to store using sops for example, or ansible-vault
15:42:58 <jrosser> ansible vault is difficult - i did not find an example for using ansible to create a new ansible-vault-encrypted var
15:44:04 <noonedeadpunk> sops should support that though....
15:46:00 <noonedeadpunk> well, I guess creating such var/file can be matter of command, given that you've defined password file in ENV var
15:46:02 <jrosser> yeah community.sops.sops_encrypt
15:46:23 <noonedeadpunk> but it's tricky/nasty indeed
15:46:39 <jrosser> so what i was thinking is we should make tasks/standalone/{{backend}}_read.yml / _write.yml
15:47:09 <jrosser> and in the middle of including those is some 'generate' function that is conditional on if the read worked or not
15:48:09 <jrosser> abstract away how/where the private key comes from, it just needs to end up in some well known var and a status fact be set
15:48:47 <noonedeadpunk> and feed result of that to pipe... Yeah, that can work, though will be super complex
15:49:10 <noonedeadpunk> I wonder if we want to resurrect our idea with roles testing before messing up with that?
15:49:16 <noonedeadpunk> or well, your idea :)
15:49:21 <jrosser> that would be a great thing to do
15:49:56 <noonedeadpunk> that way we can try to ensure that we won't break current deplooyments very badly
15:50:09 <noonedeadpunk> as workflow sounds quite complex
15:50:39 <noonedeadpunk> btw we also might want to get our vault role in for tests of integration for that backend
15:51:09 <noonedeadpunk> (or use some 3rd party at worst)
15:51:53 <noonedeadpunk> well.. this has now quite solid list of pre-requisitives
15:53:03 <noonedeadpunk> but that's for good I think
15:56:08 <noonedeadpunk> I do like the idea
15:57:01 <damiandabrowski> btw. as I mentioned today, we have broken gating for xena, this patch aims to fix it: https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/868107
15:57:33 <damiandabrowski> yesterday*
15:57:57 <noonedeadpunk> damiandabrowski: well, you can't do that I think
15:58:30 <noonedeadpunk> max supported erlang is 24.2 for rabbit 3.9.8
15:58:34 <noonedeadpunk> https://www.rabbitmq.com/which-erlang.html
16:00:03 <damiandabrowski> awwww, you're right :|
16:00:13 <noonedeadpunk> well, time is over, so will wrap this up
16:00:14 <damiandabrowski> so i have to bump rabbitmq version as well
16:00:25 <noonedeadpunk> Also want to remind, that next 2 meetings are cancelled
16:01:02 <noonedeadpunk> So see you all on next meeting in 2023 :)
16:01:17 <noonedeadpunk> #endmeeting