| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-rabbitmq_server stable/2025.1: Pin setuptools https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/976322 | 00:23 |
|---|---|---|
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-rabbitmq_server stable/2025.1: Pin setuptools https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/976322 | 00:23 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Migrate k8s cluster from osa ops into main tree https://review.opendev.org/c/openstack/openstack-ansible/+/975277 | 08:21 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_magnum master: Migrate capi driver installation from osa-ops https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/976261 | 08:22 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible stable/2025.1: wip https://review.opendev.org/c/openstack/openstack-ansible/+/976340 | 08:54 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible stable/2025.1: wip https://review.opendev.org/c/openstack/openstack-ansible/+/976340 | 08:55 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_magnum master: Migrate capi driver installation from osa-ops https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/976261 | 09:23 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_magnum master: Migrate capi driver installation from osa-ops https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/976261 | 09:43 |
| kleini | I am trying to distro upgrade my primary infrastructure host. Services do not recognize that host being down and don't connect to RabbitMQs on other infrastructure hosts. Can I support this process somehow? | 10:56 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible stable/2025.1: wip https://review.opendev.org/c/openstack/openstack-ansible/+/976340 | 12:22 |
| mgariepy | that's weird @kleini usualy there is an issue with memcached for this but not rabbit. | 13:28 |
| kleini | my rabbitmq cluster became inconsistent like last time, I upgraded an infra node | 13:28 |
| kleini | solution was to run rabbitmq_server role with rabbitmq_upgrade=true to have all queues again on all nodes | 13:29 |
| kleini | I missed that again | 13:29 |
| mgariepy | usually the queues are replicated accross all the nodes.. | 13:30 |
| mgariepy | what version are you running? | 13:30 |
| mgariepy | or are you upgrading from-to | 13:30 |
| mgariepy | usualy this is a good base for me : https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/distribution-upgrades.html | 13:32 |
| mgariepy | what version are you doing ? | 13:33 |
| kleini | I am doing distro upgrades on 31.1.3 | 13:34 |
| kleini | https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/tasks/rabbitmq_upgrade_check.yml#L143-L179 <- this needs to run to have all queues again spread across all nodes | 13:34 |
| kleini | one of the errors was then this one: https://paste.opendev.org/show/b8J0y8WDoRfW3pedvkIK/ | 13:36 |
| kleini | so, some reply_something queues of nova are broken | 13:37 |
| kleini | I had to delete the from disc and then in rabbitmq with rabbitmqctl delete_queue --vhost nova reply_something | 13:37 |
| kleini | and now it seems to be working again | 13:38 |
| kleini | just checking rabbitmqctl cluster_status is totally not sufficient | 13:39 |
| mgariepy | hm good to know | 13:54 |
| noonedeadpunk | yes, you should check rabbitmq-diagnostics check_if_node_is_quorum_critical actually | 13:56 |
| noonedeadpunk | for each node separately | 13:56 |
| noonedeadpunk | though streams can be misleading about issues | 13:56 |
| noonedeadpunk | I faced just 2 days ago, when you re-setup OS, streams migt report as critical, if only one member is absent/unreachable | 13:57 |
| noonedeadpunk | even if they have a proper quorum | 13:57 |
| mgariepy | having the patch before the upgrade would prevent that i guess | 14:00 |
| mgariepy | some stuff are quite hard to get to via CI testing. | 14:00 |
| kleini | Horizon only seems to use primary memcached, making Horizon unavailable if primary node is reinstalled. I had to remove first memcached entry in local_settings.py | 14:11 |
| mgariepy | keystone was also very slow (waiting for timeout in each resquet) on primary memcached was down. | 14:17 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:16 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:22 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:24 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:27 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: Added new configuration option: horizon_keystone_prefer_domain_token https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:28 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: Added new configuration option horizon_keystone_prefer_domain_token https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:30 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_horizon master: Added new configuration option horizon_keystone_prefer_domain_token https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/976516 | 17:30 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_nova master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/976520 | 17:34 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_nova master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/976520 | 17:45 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_nova master: wip https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/976520 | 17:56 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_nova master: Added new configuration option nova_live_migration_parallel_connections https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/976520 | 17:57 |
| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-os_nova master: Added new configuration option nova_live_migration_parallel_connections https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/976520 | 17:58 |
| opendevreview | Dmitriy Chubinidze proposed openstack/openstack-ansible-os_nova master: Added new configuration option nova_live_migration_parallel_connections https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/976520 | 22:06 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!