| opendevreview | Merged openstack/kolla-ansible master: CI: Temporarily disable keepalived track script https://review.opendev.org/c/openstack/kolla-ansible/+/992284 | 00:17 |
|---|---|---|
| opendevreview | Merged openstack/kolla-ansible master: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992020 | 00:17 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine https://review.opendev.org/c/openstack/kolla/+/982275 | 05:58 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine https://review.opendev.org/c/openstack/kolla/+/982275 | 06:19 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine https://review.opendev.org/c/openstack/kolla/+/982275 | 06:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Revert "CI: Temporarily disable keepalived track script" https://review.opendev.org/c/openstack/kolla-ansible/+/992393 | 06:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2026.1: keepalived: Install mariadb client package https://review.opendev.org/c/openstack/kolla/+/992394 | 06:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2025.2: keepalived: Install mariadb client package https://review.opendev.org/c/openstack/kolla/+/992395 | 06:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/2025.1: keepalived: Install mariadb client package https://review.opendev.org/c/openstack/kolla/+/992396 | 06:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2026.1: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992397 | 06:41 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine https://review.opendev.org/c/openstack/kolla/+/982275 | 06:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2026.1: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992397 | 07:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.2: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992404 | 07:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.2: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992404 | 07:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.1: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992405 | 07:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.1: proxysql: fix keepalived healthcheck to use MySQL protocol https://review.opendev.org/c/openstack/kolla-ansible/+/992405 | 07:33 |
| opendevreview | Sylvère K proposed openstack/kayobe stable/2026.1: Adds seed IP to default no-proxy list https://review.opendev.org/c/openstack/kayobe/+/992037 | 08:35 |
| kevko | I have a question about CI ... I have a patch https://review.opendev.org/c/openstack/kolla-ansible/+/989850 ... Debian-upgrade-trixie failing Ubuntu-noble-upgrade passing ... it's because .. trixie config.json | 08:40 |
| kevko | https://170f09f23e206988e4d5-231091edc31f40b3d022f1a19bae7260.ssl.cf5.rackcdn.com/openstack/279ff02eef174746bbc8767eab8b0632/primary/logs/kolla_configs/kolla-toolbox/config.json , ubuntu | 08:40 |
| kevko | https://5cbe6c29f69ab5039a71-fb3a89cd73503eeaa3ff317ecc3301da.ssl.cf2.rackcdn.com/openstack/b9de3f2b7ce7430d8d1d9ef68e6c0d22/primary/logs/kolla_configs/kolla-toolbox/config.json are different ...why ? for me it looks like debian-trixie-upgrade don't checkout patch version of kolla-ansible code ...so it means upgrade jobs are somehow broken ? | 08:40 |
| kevko | or - i don't understand ..it's very weird | 08:45 |
| kevko | hmm ..okay ..maybe because deploy didn't complete :/ .. | 08:52 |
| opendevreview | Merged openstack/kolla stable/2025.2: keepalived: Install mariadb client package https://review.opendev.org/c/openstack/kolla/+/992395 | 09:13 |
| *** jhorstmann is now known as Guest11033 | 09:32 | |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: haproxy: add support for disabling basic auth per service https://review.opendev.org/c/openstack/kolla-ansible/+/958861 | 09:47 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Centralize service log directory management https://review.opendev.org/c/openstack/kolla-ansible/+/989850 | 10:14 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix kolla-toolbox initialization startup race https://review.opendev.org/c/openstack/kolla-ansible/+/992420 | 10:14 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: CI: add Rocky 10 aarch64 jobs https://review.opendev.org/c/openstack/kayobe/+/992288 | 10:22 |
| opendevreview | Merged openstack/kolla-ansible master: kolla-toolbox: Retry ansible --version check https://review.opendev.org/c/openstack/kolla-ansible/+/985324 | 11:30 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Centralize service log directory management https://review.opendev.org/c/openstack/kolla-ansible/+/989850 | 11:34 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Centralize service log directory management https://review.opendev.org/c/openstack/kolla-ansible/+/989850 | 12:37 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix missing register in toolbox initialization handler https://review.opendev.org/c/openstack/kolla-ansible/+/992443 | 12:37 |
| kevko | mnasiadka, bbezak quick fix previously bad commit | 12:39 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/992443 | 12:39 |
| Sylvr | Helloo #openstack-kolla | 12:57 |
| dcapone2004 | I am having an issue using the most recent dev release of KA where after runnng a reconfigure, the internal VIP does not seem to be coming up. I have been exploring logs, checked the keepalived container, haproxy check passes, proxysql fails. docker logs of proxysql show no errors. I can connect to all mariadb containers by using the host ips from the command line, but not via the VIP which is the step KA fails | 13:07 |
| Sylvr | Is it possible to use kayobe configuration dump to get all the variables for a given host or am I encountering a bug ? | 13:07 |
| dcapone2004 | the keepalived containers does show that the bridge interfaces are NOT UP and NOT running, but since the KA process has not gotten to the Neutron steps, I have assumed this to be normal | 13:18 |
| dcapone2004 | likewise haproxy logs show a bunch of connection refused to all openstack services because KA cannot get to those deployment steps | 13:19 |
| PrzemekK | dcapone2004 Most common is keepalived_virtual_router_id - try keepalived_virtual_router_id: "52" or smth like that | 13:50 |
| dcapone2004 | PrzemekK: I did see something like that on google. Do you know what the specific cause is that makes it stop working? | 13:53 |
| PrzemekK | I have no idea . Crucial are logs from HAproxy keepalive . Maybe try move vip to another controller ? | 13:56 |
| dcapone2004 | PrzemekK, the root of the issue seems to be that none of the bridge interfaces are up. Openstack cluster is completely stopped. This is a new cluster, testing latest KA that is technically still RC status. I could easily destroy and start again, but trying to troubleshoot for educational purposes if this is encountered in the future and also to possible help report a bug | 14:04 |
| dcapone2004 | if this is caused by a bug | 14:04 |
| PrzemekK | bridge interface i see problems . It was common 10 years ago | 14:06 |
| kevko | dcapone2004 : which distro ? | 14:25 |
| dcapone2004 | new virtual router id did not resolve. | 14:33 |
| dcapone2004 | kevko: ubuntu 24.04 | 14:33 |
| dcapone2004 | i just rebooted each node 1 at a time and I think the bridge interfaces came back, about to try to deploy again now | 14:34 |
| frickler | hmm, couple of CI failures in setup_gate.sh, I wonder whether that's indicating some infra issue, e.g. https://zuul.opendev.org/t/openstack/build/2698ef8fa6cd47fbbf89850db0ea0c16 | 15:19 |
| dcapone2004 | kevko: no go still after reboot. bridge interfaces appear with ip -a addr, but I am guessing something is not configured correctly. I do not see any netplan files in /etc/netplan for the bridges, how does kolla configure them? | 15:34 |
| kevko | frickler: https://quay.io/repository/openstack.kolla/debian webbrowser requires login | 15:35 |
| Vii | https://quay.io/organization/openstack.kolla | 15:37 |
| Vii | no image debian | 15:37 |
| kevko | Vii: i rhought it can be debian ... | 15:39 |
| kevko | Vii: i am not sure ...but maybe it's because of openstack library | 15:39 |
| Vii | docker pull quay.io/openstack.kolla/keystone:2026.1-debian-trixie | 15:39 |
| kevko | Vii: or, better said ..somewhere I saw in CI that kolla_toolbox which is calling openstack can't be run ..because of version missing (some python3 incompatibility) | 15:40 |
| kevko | Vii: ip a on all controllers... can u see the VIP ? | 15:40 |
| kevko | anywhere ? | 15:40 |
| Vii | I have no problem with VIP :) someone else asked | 15:42 |
| kevko | aaa, sorry .. Vii what is your problem | 15:42 |
| kevko | dcapone2004: openvswitch image i think has a logic ... and k-a is ensuring bridges are created ... | 15:43 |
| Vii | I have no problem :) I just noticed that you wrote https://quay.io/repository/openstack.kolla/debian and I don't know such a path, only https://quay.io/organization/openstack.kolla | 15:43 |
| Vii | nwm :) | 15:43 |
| kevko | michalarbet@pixla:~/ultimum/git/upstream/kolla-ansible$ docker pull quay.io/openstack.kolla/debian:bookworm | 15:44 |
| kevko | Error response from daemon: unauthorized: access to the requested resource is not authorized | 15:44 |
| dcapone2004 | kevko: No IP configuration is present on ANY of the controllers for the bridge interfaces, nor is the VIP assigned to any of the bond1 (specified interface for api_interface in globals.yml) interfaces of any of the controllers | 15:45 |
| Vii | you have this error (requires auth) because there is no such image in quay | 15:45 |
| kevko | yep | 15:46 |
| Vii | docker pull quay.io/openstack.kolla/keystone:2026.1-debian-trixie. <- works | 15:46 |
| Vii | docker pull quay.io/openstack.kolla/keystone:2027.1-debian-trixie | 15:46 |
| Vii | Error response from daemon: manifest for quay.io/openstack.kolla/keystone:2027.1-debian-trixie not found: manifest unknown: manifest unknown | 15:46 |
| dcapone2004 | kevko: server ips on assigned on all controllers on correct subnet to bond1, all are reachable via ping from each via hostname, and I could connect to mariadb directly on the individual IPs on each controller, just no VIP | 15:46 |
| Vii | never mind :) I'm running away bb | 15:47 |
| kevko | dcapone2004: try this, on all servers where keepalived/haproxy/proxysql is | 15:48 |
| kevko | dcapone2004: docker exec -t keepalived bash -c '/check_alive.sh' | 15:48 |
| dcapone2004 | yeah I ran that before and got the same I am getting now, haproxy suceeds, proxysql fails | 15:49 |
| mnasiadka | There are patches being merged that address this | 15:49 |
| mnasiadka | See https://review.opendev.org/c/openstack/kolla-ansible/+/992397 | 15:50 |
| mnasiadka | dcapone2004: ^^ | 15:50 |
| mnasiadka | Weekly meeting tomorrow is cancelled - I’ve just sent a mail to the mailing list | 15:51 |
| dcapone2004 | mnasiadka: Is that URL private or is that server down for maintenance? URL will not load for me for some reason. However, you confirmed the general question which is that it seemingly was some type of issue (hopefully I will be able to read the full report and confirm things match soon) | 15:53 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Deploy MariaDB before the load balancer. https://review.opendev.org/c/openstack/kolla-ansible/+/992478 | 15:53 |
| mnasiadka | Gerrit seems to be slow today | 15:53 |
| dcapone2004 | ok, i will keep trying throughout the day to read it. Is there a specific way I will be able to see when the patch is merged into the stable/2026.1 branch? | 16:00 |
| mnasiadka | dcapone2004: the OpenDev admins are looking into Gerrit slowness, so just check the URL I pasted in some time | 16:06 |
| mnasiadka | frickler: I think https://opendev.org/openstack/openstacksdk/commit/3ae89d61b4f49b74a0f933c95e2df57a87816316 has broken openstack.cloud modules... | 16:18 |
| mnasiadka | Well, not this | 16:20 |
| mnasiadka | But we’re lacking openstack.version :D | 16:20 |
| mnasiadka | Task failed: Module failed: module 'openstack' has no attribute 'version' in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c7/openstack/2c76e73c09f947b98b3588f806644fee/primary/logs/ansible/deploy | 16:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: kolla-toolbox: Pin openstacksdk to 4.14.0 https://review.opendev.org/c/openstack/kolla/+/992483 | 16:27 |
| dcapone2004 | mnasiadka: exactly the issue I seem to be experiencing....is there a way to bypass that check by any chance or do I need to wait for the code merge? | 16:30 |
| kevko | yeah, that's what i've seen in logs ... 4.14.0 helped ..tested locally | 16:35 |
| opendevreview | Merged openstack/kolla-ansible master: Fix missing register in toolbox initialization handler https://review.opendev.org/c/openstack/kolla-ansible/+/992443 | 16:48 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Add YAML support to kolla_set_configs engine https://review.opendev.org/c/openstack/kolla/+/982275 | 16:49 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Deploy MariaDB before configuring loadbalancer https://review.opendev.org/c/openstack/kolla-ansible/+/992478 | 17:00 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Deploy MariaDB before configuring loadbalancer https://review.opendev.org/c/openstack/kolla-ansible/+/992478 | 17:00 |
| frickler | mnasiadka: see also https://bugs.launchpad.net/python-openstackclient/+bug/2155226 | 17:36 |
| frickler | looks like a-o-c needs a similar fix | 17:36 |
| mikal | mnasiadka: I tried the -wider COPR packages for https://review.opendev.org/c/openstack/kolla/+/986283, but it didn't work out. I'd say for now we should just merge the patch as is with the small known issue that USB sound doesn't work on Rocky (but there are alternative sound options). | 19:37 |
| mikal | mnasiadka: https://review.opendev.org/c/openstack/ansible-collections-openstack/+/992526 is an attempt to resolve the `openstack.version` error in ansible-collections-openstack so that it can eventually be unpinned. | 20:18 |
| opendevreview | Michal Arbet proposed openstack/kolla master: kolla-toolbox: Pin openstacksdk to 4.14.0 https://review.opendev.org/c/openstack/kolla/+/992483 | 21:08 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Deploy MariaDB before configuring loadbalancer https://review.opendev.org/c/openstack/kolla-ansible/+/992478 | 21:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: kolla-toolbox: Pin openstacksdk to 4.14.0 https://review.opendev.org/c/openstack/kolla/+/992483 | 22:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: kolla-toolbox: Pin openstacksdk to 4.14.0 https://review.opendev.org/c/openstack/kolla/+/992483 | 22:28 |
| mnasiadka | mikal: I took the liberty to update the a-c-o patch | 22:42 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Remove external_net_name variable https://review.opendev.org/c/openstack/kayobe/+/992570 | 22:51 |
| mikal | mnasiadka: no worries, sorry it was failing I had to rush out the door to get to work on time. | 22:57 |
| mnasiadka | mikal: no worries, it’s not that trivial because we need to workaround pbr .devX versions ;) | 22:59 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!