| birbilakos | Hi team, I need some help with this: I recently had to recreate the rabbitmq cluster of my OSA 2023.2 installation. The steps I followed were: | 04:51 |
|---|---|---|
| birbilakos | openstack-ansible lxc-containers-destroy.yml -e 'container_group=rabbitmq_all' | 04:51 |
| birbilakos | openstack-ansible lxc-containers-create.yml -e 'container_group=rabbitmq_all' | 04:51 |
| birbilakos | sudo openstack-ansible rabbitmq-install.yml -e "rabbitmq_upgrade=true" | 04:51 |
| birbilakos | the new containers are created and rabbitmq seems to be running in all of them, however, i don't see any users or vhosts being configured. In addition, every openstack service fails to auth with the new rabbitmq cluster with these type of messages: (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. | 04:53 |
| birbilakos | Any ideas how to recreate the users and vhosts too? | 04:53 |
| birbilakos | jrosser: any ideas on the above? | 07:05 |
| noonedeadpunk | hey | 08:05 |
| noonedeadpunk | you pretty much need to run `openstack-ansible setup-openstack --tags common-mq` to create vhosts/users | 08:06 |
| noonedeadpunk | good morning | 08:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Move provider_networks module into os_neutron https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/658130 | 08:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: The memcached module seems very unmaintained and it looks like none of our roles depend on it https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/850016 | 08:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server unmaintained/2023.1: Bump Erlang version to cover CVE-2025-32433 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/957202 | 08:33 |
| jrosser | o/ morning | 08:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Switch amphora url to noble https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/958526 | 10:11 |
| noonedeadpunk | seems we have smth weird going on with octavia role now.... | 10:38 |
| noonedeadpunk | upgrade to 2025.1 is kinda a disaster tbh | 11:33 |
| noonedeadpunk | birbilakos: hey! | 11:33 |
| noonedeadpunk | have you seen a reply regarding rabbitmq? | 11:33 |
| noonedeadpunk | > [10:06] <noonedeadpunk> you pretty much need to run `openstack-ansible setup-openstack --tags common-mq` to create vhosts/users | 11:34 |
| noonedeadpunk | I can't recall such a bad upgrade in a long time | 11:35 |
| noonedeadpunk | folks, seems like Octavia already got broken with some of PKI changes which landed recently | 11:51 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/00c3ff122d6c45828e34a0f7a9fb836e | 11:51 |
| noonedeadpunk | as indeed /etc/openstack_deploy/pki/roots/OctaviaClientRoot/certs/OctaviaClientRoot-chain.crt is not there | 11:52 |
| noonedeadpunk | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_00c/openstack/00c3ff122d6c45828e34a0f7a9fb836e/logs/etc/host/openstack_deploy/pki/roots/OctaviaClientRoot/certs/ | 11:52 |
| noonedeadpunk | damiandabrowski: can you maybe check on this one? | 11:52 |
| damiandabrowski | sure thing! | 11:53 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server unmaintained/2023.1: Bump Erlang version to cover CVE-2025-32433 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/957202 | 13:46 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari stable/2025.1: Add masakari user to libvirt Group https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/958373 | 13:50 |
| jrosser | birbilakos: did you see the replies to your question? | 14:02 |
| opendevreview | Merged openstack/openstack-ansible master: Drop centos-10-stream distro job https://review.opendev.org/c/openstack/openstack-ansible/+/957859 | 14:14 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Aug 26 15:00:21 2025 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| damiandabrowski | hi! | 15:01 |
| noonedeadpunk | #topic office hours | 15:05 |
| noonedeadpunk | So as there was no feedback about moving playbooks to ops vs plugins - I marked my patch for moving it to plugins as ready for review | 15:08 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/956949 | 15:08 |
| noonedeadpunk | and also made another patch to move haproxy-endpoint-manage from ops repo to plugins as a follow-up | 15:09 |
| noonedeadpunk | rest in ops repo seem a bit of opinionated still | 15:10 |
| noonedeadpunk | and I'm not sure about them at all | 15:10 |
| noonedeadpunk | on topic of EL10 support - there was no progress so far in terms of systemd-networkd and epel | 15:10 |
| noonedeadpunk | so I decided to decouple CentOS 10 Stream from Rocky 10 patches | 15:10 |
| noonedeadpunk | thus we can vote and backport them separately | 15:11 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible/+/958170 | 15:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Ensure no CQ mirroring policies applied https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/958428 | 15:12 |
| damiandabrowski | ack | 15:12 |
| noonedeadpunk | This one is actually created quite some headache during rabbitmq upgfrade for us ^ | 15:12 |
| noonedeadpunk | I think we should have it backported before tagging epoxy as 31.1.0 | 15:13 |
| noonedeadpunk | PKI role progress | 15:13 |
| noonedeadpunk | I believe that Octavia got pretty much broken with merging cert installation by name | 15:14 |
| noonedeadpunk | I was not digging deep, but it seems that role tries to install chain which is not created for root | 15:15 |
| damiandabrowski | yeah, I already found a culprit. Will push a fix really soon, definitely today. | 15:15 |
| noonedeadpunk | these are great news then!@ | 15:15 |
| jrosser | o/ hello | 15:16 |
| noonedeadpunk | o/ | 15:18 |
| noonedeadpunk | and some next changes to pki role seems to be missing one more vote\ | 15:19 |
| jrosser | damiandabrowski: needs to see these patches i think and rebase some https://review.opendev.org/c/openstack/ansible-role-pki/+/957848 | 15:21 |
| noonedeadpunk | https://review.opendev.org/q/project:openstack/ansible-role-pki+status:open+label:verified+label:Code-Review%3D2 | 15:21 |
| damiandabrowski | ahh, there's another chain. Sorry, I didn't see it | 15:22 |
| damiandabrowski | I'll have a look tomorrow | 15:22 |
| noonedeadpunk | I think we need to start coming up with etherpad of things for the release | 15:23 |
| noonedeadpunk | let's maybe use this link | 15:24 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/oct2025-ptg-os-ansible | 15:24 |
| noonedeadpunk | hopefully it will match with the meetpad.... | 15:24 |
| noonedeadpunk | what things are we have as ongoing.... | 15:26 |
| noonedeadpunk | jrosser: I guess we wanna finalize Debian 13? | 15:26 |
| jrosser | oh goodness i completely forgot about that :/ | 15:27 |
| jrosser | yes we do | 15:27 |
| jrosser | afaik we were OK locally but not in CI | 15:27 |
| noonedeadpunk | python 3.13 got way closer I guess... | 15:28 |
| noonedeadpunk | but I can't recall what was missing from CI at this point... | 15:28 |
| noonedeadpunk | I'm guessing usual things, like rabbitmq/mariadb | 15:29 |
| noonedeadpunk | We also need to fix gather_subset | 15:29 |
| noonedeadpunk | as with switch to 2.18 it's just silently ignored now | 15:29 |
| noonedeadpunk | I haven't yet started looking into improvements to haproxy :( | 15:30 |
| jrosser | i thought i had got a lot of stuff sorted for trixie, but it was a while ago | 15:30 |
| noonedeadpunk | but it's also not a blocker at all | 15:30 |
| noonedeadpunk | eh | 15:32 |
| noonedeadpunk | you didn't use a topic for them, did you:? | 15:32 |
| noonedeadpunk | found https://review.opendev.org/c/openstack/openstack-ansible/+/954616 | 15:32 |
| jrosser | it could be that i got it working in a VM but not more than that | 15:33 |
| jrosser | i'll rebase 954616 and see where it is today | 15:33 |
| noonedeadpunk | sounds good | 15:35 |
| noonedeadpunk | anything else what comes to mind which we might wana target? | 15:35 |
| * noonedeadpunk checking previos ptg notes https://etherpad.opendev.org/p/apr2025-ptg-os-ansible | 15:36 | |
| noonedeadpunk | we mentioned PKI refactoring | 15:36 |
| noonedeadpunk | in terms of not storing certs on deploy host | 15:37 |
| noonedeadpunk | but I think it's worth doing that only after dust with vault will settle | 15:37 |
| noonedeadpunk | Migration from OVS/LXB to OVN is still a black box for me | 15:38 |
| noonedeadpunk | there were couple of really great articles, specifically from CERN, for LXB migration | 15:38 |
| noonedeadpunk | but I did not take time to dig deep there | 15:39 |
| noonedeadpunk | And I think we still have a really problematic bug with upgrade order for OVN | 15:39 |
| noonedeadpunk | as ovn-controller should be upgraded before sb/nb dbs | 15:39 |
| noonedeadpunk | while we are running upgrade same way as setup, where ovn-controller is targeted later on | 15:40 |
| opendevreview | Damian Dąbrowski proposed openstack/ansible-role-pki master: Fix creation of certs signed by selfsigned issuers https://review.opendev.org/c/openstack/ansible-role-pki/+/958550 | 15:40 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add debian trixie job definitions https://review.opendev.org/c/openstack/openstack-ansible/+/954616 | 15:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_octavia master: [DNM] Check if 958550 fixes octavia CI jobs https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/958551 | 15:43 |
| noonedeadpunk | damiandabrowski: hm, do we also need something for create_ca? | 15:43 |
| noonedeadpunk | as failure happens on root isntallation I'd guess? | 15:43 |
| jrosser | as we now have it i think that the tests should probably cover these cases | 15:44 |
| jrosser | sooo much opportunity to break * here | 15:44 |
| damiandabrowski | no no, failure happens on certificate creation, not the installation | 15:44 |
| damiandabrowski | "Create certificate ca bundle for octavia_client" task | 15:44 |
| noonedeadpunk | I'm talking about https://zuul.opendev.org/t/openstack/build/db82f298d73144fc95e90d86c1b21ff9 | 15:44 |
| noonedeadpunk | ah, ok, yes, makes sense then | 15:45 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_octavia master: [DNM] Check if 958550 fixes octavia CI jobs https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/958551 | 15:46 |
| noonedeadpunk | I just a bit confused I guess... | 15:48 |
| noonedeadpunk | yeah, and then we really don't know if ca was having intermediate or not | 15:49 |
| noonedeadpunk | just trying to think if there could be more neat way rather then stat | 15:50 |
| noonedeadpunk | as maybe instead we should be producing bundle for CA anyway? | 15:51 |
| noonedeadpunk | when we generate root? | 15:51 |
| damiandabrowski | yeah, that would be an alternative approach but I was a bit afraid of fixing already existing environments | 15:51 |
| noonedeadpunk | Well, root creation is first step for upgrade anyway? | 15:52 |
| noonedeadpunk | I mean - if the file does not exist - it will be created | 15:52 |
| damiandabrowski | yeah... | 15:53 |
| noonedeadpunk | so upgrade should be fine, I'd guess | 15:53 |
| noonedeadpunk | unless we override existing chains with some random stuff | 15:53 |
| noonedeadpunk | but we should not do that anyway :D | 15:53 |
| noonedeadpunk | as that would be somehow in line with other approaches we selected to always produce $things | 15:54 |
| jrosser | hrrm i am not sure 958550 will be vary obvious what is happening in the future | 15:56 |
| damiandabrowski | ack, I can create an alternative patch that would always trigger generation of *-chain.crt | 15:57 |
| noonedeadpunk | sounds good, thanks! | 15:58 |
| noonedeadpunk | #endmeeting | 15:58 |
| opendevmeet | Meeting ended Tue Aug 26 15:58:22 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-08-26-15.00.html | 15:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-08-26-15.00.txt | 15:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2025/openstack_ansible_meeting.2025-08-26-15.00.log.html | 15:58 |
| jrosser | `scripts/scripts-library.sh: line 107: VERSION_ID: unbound variable` | 16:05 |
| noonedeadpunk | this is smth we're fetching from /etc/os-release | 16:13 |
| noonedeadpunk | I don't have debina13 image handy :( | 16:13 |
| jrosser | let me check | 16:14 |
| noonedeadpunk | for ubuntu it looks like https://paste.openstack.org/show/bejzliK5qySm7yzfYMrV/ | 16:16 |
| jrosser | https://paste.opendev.org/show/bMA4kMTi9HR8ROAqrKFB/ | 16:18 |
| jrosser | ^ thats from an AIO thats ~40 days uptime | 16:19 |
| noonedeadpunk | well :( | 16:21 |
| noonedeadpunk | Totally could be CI image then :( | 16:22 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add debian trixie job definitions https://review.opendev.org/c/openstack/openstack-ansible/+/954616 | 16:24 |
| jrosser | oh VERSION_CODENAME=forky | 17:04 |
| jrosser | that’s unexpected | 17:04 |
| noonedeadpunk | it'[s not for sure | 17:28 |
| jrosser | see #opendev | 17:33 |
| jrosser | and indeed /etc/os-release was missing VERSION_ID for whatever actually is the debian-trixie CI image | 17:34 |
| noonedeadpunk | ah, I see, thanks! | 17:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible unmaintained/2023.1: Use roles from top of unmaintained branch https://review.opendev.org/c/openstack/openstack-ansible/+/958563 | 17:53 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Add mariadb-client-utils to packages for removal https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/956778 | 18:09 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari master: Remove outdated code https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/954867 | 18:41 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!