| senan | is it correct to deploy keepalived, haproxy and proxysql on separate nodes (ha01, ha02), and not on controller nodes? | 06:30 |
|---|---|---|
| fprzewozn | senan: it's completely fine, all of my production clusters are configured this way | 07:35 |
| fprzewozn | database and message queue got own nodes as well | 07:35 |
| mnasiadka | fprzewozn: merging your patches | 08:00 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Implement multi-stage builds for OpenStack services and auxiliary tools https://review.opendev.org/c/openstack/kolla/+/969968 | 08:24 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Implement multi-stage builds for OpenStack services and auxiliary tools https://review.opendev.org/c/openstack/kolla/+/969968 | 08:30 |
| senan_ | fprzewozn: in the past such setup worked for me too, but in the current implementation deployment fails with "deployment fails TASK [service-check : mariadb | Get container facts] FAILED - RETRYING: [controller01.os]: Wait for MariaDB service to be ready through VIP (6 retries left)" | 09:03 |
| senan_ | i.e mariadb is not accessible via VIP | 09:04 |
| opendevreview | Merged openstack/kolla-ansible master: Add support for libvirt swtpm (vTPM) configuration. https://review.opendev.org/c/openstack/kolla-ansible/+/946340 | 09:08 |
| opendevreview | Merged openstack/kolla-ansible master: Add RabbitMQ management interface SSL https://review.opendev.org/c/openstack/kolla-ansible/+/964292 | 09:09 |
| opendevreview | Merged openstack/kolla-ansible master: Fixed kolla_externally_managed_cert behaviour https://review.opendev.org/c/openstack/kolla-ansible/+/968017 | 09:09 |
| fprzewozn | mnasiadka: thanks! | 09:12 |
| fprzewozn | senan_: proxysql should expose mariadb on VIP | 09:13 |
| fprzewozn | check `mysql_variables` in /etc/kolla/proxysql/proxysql.yaml | 09:14 |
| fprzewozn | it must be some misconfig in your env | 09:14 |
| senan_ | fprzewozn: you mean I need additional configuration? | 09:15 |
| senan_ | fprzewozn: my setup is specific one in sense that VIP (172.31.255.254) is not from the same subnet as network_interface IPs on controllers (10.10.11.2, 10.10.12.2, 10.10.13.2) | 09:18 |
| senan_ | prechecks fails because of that, although there's a "# FIXME(yoctozepto): this req seems arbitrary, they need not be, just routable is fine" in the precheck tasks for kolla_internal_vip_address | 09:20 |
| senan_ | and 172.31.255.0/24 is actually routable from all nodes | 09:24 |
| fprzewozn | are you 100% sure you don't block somehow those connections? you could check manually with netcat and curl on port 3306 | 09:28 |
| fprzewozn | and if it's fine check if proxysql is binding correctly | 09:29 |
| fprzewozn | and if that's also fine check if proxysql can connect to database | 09:29 |
| senan_ | should I perform all those checks from ha nodes? | 09:30 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Add OCI standard labels to container images https://review.opendev.org/c/openstack/kolla/+/971681 | 09:33 |
| fprzewozn | senan_: start with connection from failed node to VIP (even using mysql client) | 09:37 |
| Vii | On a controller that failed prechecks, connecting to the VIP via "mysql -h 172.31.255.254 -u kolla -p" or "nc -vz 172.31.255.254 3306" | 09:45 |
| Vii | Then, on the HA node, check if ProxySQL actually binds to the VIP: ss -tlnp | grep 3306 | 09:45 |
| Vii | Check the ProxySQL logs for errors when connecting to MariaDB backends. | 09:45 |
| Vii | If the VIP is on a different subnet, try adding static routing or temporarily using the VIP on the same subnet as the controllers to see if the problem goes away. | 09:46 |
| senan | cool, thanks, let me deploy again with separate ha nodes to trigger errors and debug | 09:50 |
| tafkamax | opendevreview: Should this be back ported to 2025.2? | 09:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Revert^2 "kolla-toolbox: Bump ansible-core to 2.20" https://review.opendev.org/c/openstack/kolla/+/971693 | 10:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.19/2.20 https://review.opendev.org/c/openstack/kolla-ansible/+/971606 | 10:42 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Implement multi-stage builds for OpenStack services and auxiliary tools https://review.opendev.org/c/openstack/kolla/+/969968 | 10:47 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Implement multi-stage builds for OpenStack services and auxiliary tools https://review.opendev.org/c/openstack/kolla/+/969968 | 10:48 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add support for custom OVMF firmware in nova-libvirt https://review.opendev.org/c/openstack/kolla-ansible/+/971696 | 11:16 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add support for custom OVMF firmware in nova-libvirt https://review.opendev.org/c/openstack/kolla-ansible/+/971696 | 11:18 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add support for custom OVMF firmware in nova-libvirt https://review.opendev.org/c/openstack/kolla-ansible/+/971696 | 12:41 |
| *** fprzewozn3 is now known as fprzewozny | 13:52 | |
| opendevreview | Verification of a change to openstack/kayobe stable/2025.2 failed: Use opendevmirror/ubuntu:24.04 image for molecule https://review.opendev.org/c/openstack/kayobe/+/971528 | 14:19 |
| *** jhorstmann is now known as Guest34278 | 14:42 | |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Switch to pypi version of j2lint and fix linting errors https://review.opendev.org/c/openstack/kolla-ansible/+/963980 | 16:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.19/2.20 https://review.opendev.org/c/openstack/kolla-ansible/+/971606 | 16:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.19/2.20 https://review.opendev.org/c/openstack/kolla-ansible/+/971606 | 16:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.19/2.20 https://review.opendev.org/c/openstack/kolla-ansible/+/971606 | 16:01 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Add additional SPICE configuration options. https://review.opendev.org/c/openstack/kolla-ansible/+/967800 | 19:19 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Use a routable IP for qemu SPICE consoles. https://review.opendev.org/c/openstack/kolla-ansible/+/967801 | 19:19 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Allow requiring secure channels with SPICE. https://review.opendev.org/c/openstack/kolla-ansible/+/967802 | 19:19 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Simplify cron jobs for log rotate. https://review.opendev.org/c/openstack/kolla-ansible/+/969138 | 19:20 |
| opendevreview | Merged openstack/kayobe stable/2025.2: Use opendevmirror/ubuntu:24.04 image for molecule https://review.opendev.org/c/openstack/kayobe/+/971528 | 19:25 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!