15:03:56 <noonedeadpunk> #startmeeting openstack_ansible_meeting 15:03:56 <opendevmeet> Meeting started Tue Mar 17 15:03:56 2026 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:56 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:03:56 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting' 15:04:00 <noonedeadpunk> #topic rollcall 15:04:10 <noonedeadpunk> o/ 15:04:14 <noonedeadpunk> hey there 15:04:33 <chubinidzedr_> Hello 15:04:59 <jrosser> o/ hello 15:06:06 <noonedeadpunk> #topic office hours 15:06:30 <noonedeadpunk> so I am still failing to fill in etherpad with topics we're tracking right now :( 15:06:44 <noonedeadpunk> (which should be also etherpad for ptg basically) 15:07:01 <noonedeadpunk> I do have that in my backlog to process, but it's not done yet 15:07:23 <noonedeadpunk> There is quite a good progress around magnum/capi refactoring 15:07:56 <noonedeadpunk> while I posted couple of comments, I intend to test out patches this week 15:08:02 <jrosser> something to decide is if we want to turn more of it into roles, when we do that 15:08:13 <jrosser> as i'm not sure andrew has the cycles to work much more on it for a while 15:08:25 <noonedeadpunk> yeah. openstack.osa.k8s looks quite overwhelming at this point 15:08:43 <noonedeadpunk> but I potentially can pick this up... 15:10:47 <noonedeadpunk> and work further on azimuth driver support 15:10:56 <noonedeadpunk> especially given that existing patches work, and I think they dfo 15:10:57 <noonedeadpunk> *do 15:11:03 <noonedeadpunk> or we can actually merge them, and continue improvements in a follow-ups 15:11:39 <opendevreview> Damian Dąbrowski proposed openstack/openstack-ansible-os_glance master: Add hashi_vault pki backend support https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/949428 15:11:57 <opendevreview> Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Add hashi_vault pki backend support https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/949418 15:12:57 <jrosser> refactoring can always follow if we have time 15:13:08 <noonedeadpunk> and it might be also easier to test/play with them this way 15:13:27 <jrosser> depends if we want to leave it unmerged for a release and finalise, or if its more important to make a start 15:14:11 <jrosser> anyway regardless, main point is we kind of run out of resource here to work in it much more, so if you can look also it would be great 15:14:23 <noonedeadpunk> I'd love to try to finalize before the release :) 15:14:41 <jrosser> of course :) 15:15:26 <noonedeadpunk> I have some env where I'd need to run magnum, so potentially gonna test in prod :D 15:16:12 <noonedeadpunk> but ok, gotcha, great progress anyway 15:17:10 <noonedeadpunk> we're also a bit stuck with tagging config_template 15:17:15 <noonedeadpunk> #link https://review.opendev.org/c/openstack/releases/+/977916 15:17:36 <noonedeadpunk> and by latest comment I'm afraid that we won't end up in chicken-egg thing 15:18:04 <noonedeadpunk> as currently AIO is pretty much broken 15:18:24 <noonedeadpunk> the only thing we can possibly do, is to set version to master for https://opendev.org/openstack/openstack-ansible/src/branch/master/ansible-collection-requirements.yml#L10 15:19:29 <noonedeadpunk> also hashi_vault patches are slowly landing, which is very nice to see 15:19:47 <jrosser> that does look like human error with config_template 15:20:08 <jrosser> yes, i did look at the vault patches - there were a lot not passing a couple of days ago 15:20:29 <noonedeadpunk> I think main reason were groups renaming 15:20:41 <noonedeadpunk> and this leads me to known issues we have right now 15:21:12 <noonedeadpunk> 1. Octavia is broken, and it seems due to u-c not respected in our CI. I did not look into that so far. 15:21:32 <noonedeadpunk> 2. Skyline deployment in multiple LXC is also broken, in case yarn build is used 15:22:41 <noonedeadpunk> and then vault patches should highlight some more issues with core 2.19 I believe 15:23:03 <jrosser> we can test that in AIO with affinity=3 for skyline? 15:23:26 <jrosser> oh and also unrelated i don't know what to do with all the bindep patches 15:23:58 <noonedeadpunk> yeah, we should be able to. Need to figure out where exactly affinity should be applied - to repo, to skyline, to both... 15:24:12 <noonedeadpunk> as it's python_venv_build which is failing with invalid constraints 15:24:24 <noonedeadpunk> I think there was a patch with discussion around the issue 15:24:52 <noonedeadpunk> #link https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/972974 15:25:24 <noonedeadpunk> regarding bindep patches - I think we need to fix synchronize script, as it's obviously malfunctioning 15:25:36 <noonedeadpunk> and then let the script do propose patches 15:26:33 <noonedeadpunk> I think this is what does not work: https://opendev.org/openstack/openstack-ansible/src/branch/master/sync/gen-projects-list.sh#L73 but also - I'm not really sure about how this supposed to work even... 15:27:49 <jrosser> there was also a patch for octavia not working with ssl zookeeper 15:27:59 <jrosser> that feels like some other broken thing as well 15:28:13 <noonedeadpunk> yeah, but that is broken since 2023.1 afaik 15:28:21 <jrosser> ahhha right :) 15:28:28 <noonedeadpunk> and it's more about issue inside of the octavia's taskmanager 15:28:33 <jrosser> did we open a bug with octavia for that? 15:28:40 <noonedeadpunk> they don't use tooz or smth like that 15:29:12 <noonedeadpunk> I can recall talking with jonher about it years back, but I can't find the bug report now 15:30:02 <noonedeadpunk> we probably should... 15:30:27 <noonedeadpunk> at the same time etcd was working there... 15:34:44 <chubinidzedr_> I'll try to turn off ssl from octavia side, although I thought it was disabled by default 15:37:26 <noonedeadpunk> It could be we enabled tls globally, but I also failed to find where 15:37:51 <noonedeadpunk> though I'd really assume that disabling tls server-wide is not best thing to do... 15:38:08 <noonedeadpunk> better just to disable coordination for octavia I guess 15:38:24 <noonedeadpunk> but also submitting bug report to the team :) 15:40:06 <noonedeadpunk> fungi: will summon you as an expert:) do you have any idea how this was supposed to work before? https://opendev.org/openstack/openstack-ansible/src/branch/master/sync/gen-projects-list.sh#L73 15:40:21 <noonedeadpunk> as I am really confused how/why ssh key would be loaded for auth 15:40:50 <noonedeadpunk> or wait... it's running in post-review pipeline... 15:41:05 <noonedeadpunk> so it's potentially matter of just adding a new ssh key to the user? 15:41:23 <fungi> what job runs that script? 15:41:53 <fungi> i'm wondering if it's using a separate copy of the secret for that ssh key 15:42:04 <fungi> and never got replaced when the key was rotated 15:42:51 <noonedeadpunk> I moved the script from openstack-ansible-tests repo 15:43:00 <noonedeadpunk> so yeah, ssh key changed potentially 15:44:17 <noonedeadpunk> so, it's initiated somewhere here: https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/pre.yaml#L24-L32 15:44:26 <noonedeadpunk> But I am not 100% sure about the compelte flow 15:47:22 <noonedeadpunk> https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/propose_update.sh#L57-L63 15:48:13 <noonedeadpunk> fungi: so here it is: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L2594-L2597 15:48:32 <opendevreview> Merged openstack/ansible-hardening master: Rename reserved variable check_mode to check_skip https://review.opendev.org/c/openstack/ansible-hardening/+/980809 15:50:26 <fungi> okay, so it's running the propose-updates job which is defined in openstack/project-config and should be reusing the same ssh key as all the other propose-updates uses 15:50:51 <noonedeadpunk> so yeah, I guess zuul key should be taken for https://zuul.opendev.org/api/tenant/openstack/project-ssh-key/openstack/openstack-ansible.pub 15:51:13 <noonedeadpunk> but it's defined for specific project? 15:51:22 <noonedeadpunk> so ssh key should be taken from that project? 15:52:13 <fungi> no, the job itself is defined here: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L756-L766 15:52:28 <noonedeadpunk> I think this what made it fail https://opendev.org/openstack/project-config/commit/ff9aab67ffcd1d86cb196d0d4509bcee26ab51d2 15:52:30 <fungi> you're just running it and setting a job var to indicate the preconfigured "target" you want updated 15:52:53 <noonedeadpunk> as I've moved job from openstack/openstack-ansible-tests to openstack/openstack-ansible 15:53:01 <fungi> so propose-updates is using the same `proposal_ssh_key` secret for every variation of how it's run 15:53:56 <noonedeadpunk> fungi: but it's running not in project-config scope: https://zuul.opendev.org/t/openstack/build/0b8d13270da544fcae33a6dca27b0d13 15:54:12 <fungi> right, it's about where the job is defined, not what project it runs on 15:54:21 <noonedeadpunk> job is defined there, but it's assigned to the project 15:54:32 <noonedeadpunk> so I'd assume SSH keys should be not for project-config as well 15:55:07 <fungi> it's added to a project-pipeline but that doesn't prevent it from using the secret, which is tied to the project the job is defined in rather than where it's instantiated 15:55:43 <fungi> the build you linked seems to have succeeded 15:55:55 <fungi> what specific problem behavior are you observing with it? 15:56:01 <noonedeadpunk> yeah, but the command to gather project list is not 15:56:03 <noonedeadpunk> https://zuul.opendev.org/t/openstack/build/0b8d13270da544fcae33a6dca27b0d13/log/job-output.txt#638 15:56:30 <noonedeadpunk> oops wrong pointer 15:57:11 <noonedeadpunk> https://zuul.opendev.org/t/openstack/build/0b8d13270da544fcae33a6dca27b0d13/log/job-output.txt#602 15:57:40 <noonedeadpunk> so yeah, it's running the script, but output is not enough to see what exactly goes wrong 15:58:18 <fungi> it changes to /home/zuul/scripts and then runs ./sync/gen-projects-list.sh 15:58:59 <noonedeadpunk> `ssh -p 29418 noonedeadpunk@review.opendev.org gerrit ls-projects --prefix openstack/openstack-ansible-` for me populates tons of projects... 15:59:12 <noonedeadpunk> but script returns https://zuul.opendev.org/t/openstack/build/0b8d13270da544fcae33a6dca27b0d13/log/job-output.txt#603-608 15:59:30 <noonedeadpunk> only what is explicitly defined 16:00:00 <noonedeadpunk> huh 16:00:05 <noonedeadpunk> I think I see the issue now 16:01:12 <fungi> the copy-proposal-common-scripts role seems to be responsible for putting that script into place 16:01:23 <opendevreview> Damian Dąbrowski proposed openstack/ansible-role-zookeeper master: Add hashi_vault pki backend support https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/949422 16:01:27 <noonedeadpunk> I think it's grep messing things up 16:02:47 <noonedeadpunk> this doesn't work with grep nicely: https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/sync_openstack_ansible_common_files.sh#L30-L34 16:07:12 <fungi> i don't see grep being used there 16:07:16 <noonedeadpunk> ok, thanks a lot! you really helped to untangled that 16:07:25 <noonedeadpunk> grep is here https://opendev.org/openstack/openstack-ansible/src/branch/master/sync/gen-projects-list.sh#L74 16:07:34 <fungi> ah okay 16:07:50 <noonedeadpunk> and as `openstack-ansible` is in the excluded, and we try to find only that pattern - it breaks 16:08:01 <noonedeadpunk> and it worked before, as it was `openstack-ansible-tests` 16:21:21 <noonedeadpunk> #endmeeting