| opendevreview | Merged openstack/openstack-ansible-galera_server master: Add distro infra jobs https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/914691 | 02:35 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Implement variables to address oslo.messaging improvements https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/919069 | 02:56 |
| semantic | Hello, everyone! So I've been testing this https://github.com/openstack/openstack-ansible/commit/d4530e242db7c45c10729123be8d7a8fbab38296 and it seems to me, that even with it I still have the problem of services stuck in 'waiting for message' state. I slowly take infra nodes down one at a time, then turn them on back again and eventually nova-compute service on some host, or neutron-ovs-agent stuck. I got similar | 06:49 |
| semantic | behaviour when try to use quorum queues... Maybe someone could suggest any ideas? Rabbit cluster itself reports as healthy and no network partitions when any of the nodes is down. | 06:49 |
| noonedeadpunk | hey | 07:28 |
| noonedeadpunk | semantic: well, for caracal we're landing a bunch of improvements for quorum queues behaviour (which become available only on caracal) | 07:30 |
| noonedeadpunk | but for the mentioned commit to be respected, you'd need to run pretty much all roles, as the policy is applied per vhost | 07:31 |
| noonedeadpunk | so question is - how you was testing that? | 07:31 |
| jrosser_ | semantic: for https://github.com/openstack/openstack-ansible/commit/d4530e242db7c45c10729123be8d7a8fbab38296 specifically we could ask andrewbonney - he did a lot of work on our rabbitmq related to that | 07:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: [doc] Rename extending-osa page https://review.opendev.org/c/openstack/openstack-ansible/+/915078 | 07:49 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: [doc] Add information about hook playbooks to the 'extending-osa' docs https://review.opendev.org/c/openstack/openstack-ansible/+/919555 | 08:00 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Define lock directory for oslo_concurrency https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/919061 | 08:57 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Implement variables to address oslo.messaging improvements https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/917999 | 08:57 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi master: Ensure Gnocchi is connected to MySQL coordination with TLS https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/918036 | 09:01 |
| noonedeadpunk | I think worth adding logic there to add zookeeper for coordination as well ^ | 09:19 |
| opendevreview | Merged openstack/openstack-ansible-os_skyline master: Add designate and masakari to service mapping https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/919523 | 09:25 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Bump SHAs for 2023.1 https://review.opendev.org/c/openstack/openstack-ansible/+/919066 | 09:39 |
| opendevreview | Merged openstack/openstack-ansible-os_skyline master: Reflect keystone service variables in config https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/918160 | 09:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Switch nginx with Apache https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/919529 | 10:12 |
| semanticspaces | Well, technically i basically do a new cluster installation over empty hosts and then run tests. And I'm little confused with amqp_durable_queues = true, osa does not include this in config, while kolla does. Though it does not seem to change anything in my case. | 10:15 |
| noonedeadpunk | semanticspaces: so improvements I've talked about are these: https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/919059/2/templates/neutron.conf.j2 | 10:20 |
| noonedeadpunk | not everything is merged yet | 10:20 |
| noonedeadpunk | semanticspaces: also amqp_durable_queues is noop when quorum queues are used | 10:21 |
| noonedeadpunk | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L169-L171 | 10:21 |
| jrosser_ | semanticspaces: what version of openstack-ansible are you testing? | 10:23 |
| semanticspaces | 28.2.0 | 10:24 |
| noonedeadpunk | semanticspaces: if you're playing with setups anyway... can you check out current master?:) | 10:29 |
| noonedeadpunk | as it seems - all core services already do have these quorum improvements merged | 10:29 |
| noonedeadpunk | (except telemetry, magnum, ironic, manila and zun) | 10:29 |
| semanticspaces | i can check master, but we only install nova,glance,neutron,placement,ceilometer,horizon...if i do this what settings should i configure? oslomsg_rabbit_quorum_queues: True only? | 10:33 |
| noonedeadpunk | yeah | 10:39 |
| noonedeadpunk | all rest should be implied from that | 10:39 |
| noonedeadpunk | ok, so changes for ceilometer were not merged yet | 10:40 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/918107 | 10:40 |
| jrosser_ | we have this all setup via SCENARIO env var for all-in-ones too? | 10:40 |
| noonedeadpunk | but the rest are | 10:40 |
| noonedeadpunk | we do, yes | 10:40 |
| noonedeadpunk | I think `quorum` is a keyword | 10:41 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L362-L364 | 10:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: [doc] Expand documentation on OVN useful commands https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/913588 | 10:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Change example to contain domain name instead of UUID https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/919563 | 11:41 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Add service policies defenition https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/917947 | 12:26 |
| spotz[m] | noonedeadpunk: You mind if I fix some grammar vs comments on that doc? | 13:39 |
| noonedeadpunk | spotz[m]: I never mind that | 13:55 |
| spotz[m] | Ok patch up in a few | 14:01 |
| opendevreview | Amy Marrich proposed openstack/openstack-ansible-os_neutron master: [doc] Expand documentation on OVN useful commands https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/913588 | 14:03 |
| noonedeadpunk | thanks so much | 14:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Fix rootwrap files idempotency https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/909104 | 14:26 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/919153 | 14:44 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue May 14 15:00:17 2024 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic office hours | 15:00 |
| noonedeadpunk | well, it should have been rollcall... but whatever :D | 15:00 |
| damiandabrowski | hi! | 15:01 |
| noonedeadpunk | huge chunk of https://review.opendev.org/q/topic:%22osa/messaging_improvements%22 has been merged | 15:02 |
| noonedeadpunk | thanks jrosser_ and andrewbonney for taking time on reviewing that! | 15:02 |
| jrosser_ | o/ hello | 15:02 |
| jrosser_ | no worries | 15:02 |
| noonedeadpunk | some roles are broken, so I was planning to look deeper during the week about reasons | 15:02 |
| jrosser_ | there are some awkward bits left i think | 15:02 |
| jrosser_ | but tbh this is not terrible, becasue these patches for messaging give us a good health check before we release | 15:03 |
| noonedeadpunk | yeah | 15:03 |
| noonedeadpunk | I was expecting more worse situation kinda | 15:03 |
| noonedeadpunk | I actually tried to add tempest tests for trove, but it expects for datastores to be created outside of tempest runtime | 15:04 |
| noonedeadpunk | which is non-trivial due to missing ansible modules for that | 15:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add information about hook playbooks to the 'extending-osa' docs https://review.opendev.org/c/openstack/openstack-ansible/+/919555 | 15:06 |
| noonedeadpunk | we also have quite some failures for upgrade jobs, which are mainly intermittent, but highly annoying | 15:06 |
| jrosser_ | seems to be some amount of tls / rocky / upgrade failures | 15:07 |
| jrosser_ | i did look in some logs and it was pretty hard to find anything specific | 15:07 |
| noonedeadpunk | yeah, also haven't found anything too obvious | 15:07 |
| noonedeadpunk | rather then potentially some nodes just being slow or smth... | 15:08 |
| NeilHanlon | o/ hiya | 15:08 |
| NeilHanlon | i am going to check on repos for rocky today; make sure there aren't bad entries still being served. i.e, mirrors which are not serving 9.4 | 15:08 |
| jrosser_ | tbh i have not seen repo specific errors for ~ a week at least | 15:09 |
| noonedeadpunk | actually ,another part we've recently realized being annoying - is that volume types are created regardless if operator want to manage them through osa | 15:09 |
| jrosser_ | just somehow our rocky CI feels less stable than other things | 15:09 |
| noonedeadpunk | or well, naming of default one is really opinionated | 15:09 |
| NeilHanlon | jrosser_: ack; well, i'll still do that, but will check into CI generally and see if I can find anything common | 15:09 |
| noonedeadpunk | to be completely frank - I was spotting issues with Ubuntu as well | 15:10 |
| jrosser_ | yep, i also saw what looked like sqla troubles too | 15:10 |
| noonedeadpunk | out of capi topic - one patch left: https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/916647 | 15:12 |
| noonedeadpunk | and actually - magnum feels tough on passing ci lately :( | 15:13 |
| jrosser_ | yes, that is not super critical, but it's needed to make a bunch of jobs to test all the k8s versions | 15:13 |
| jrosser_ | yes it does | 15:13 |
| noonedeadpunk | ie - there're some capi intermittent issues | 15:13 |
| jrosser_ | and i think there is brokenness in vexxhost nodepool too | 15:13 |
| noonedeadpunk | on top of upgrade failures | 15:13 |
| jrosser_ | ^ mnaser | 15:14 |
| jrosser_ | yeah i agree that the capi job has failed for unspecific reasons | 15:14 |
| jrosser_ | though surprisingly we already collect a ton of logs from the control plane which should make it possible to see whats happening | 15:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Move insecure param to keystone_auth section https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/905110 | 15:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Implement support for octavia-ovn-provider driver https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 15:16 |
| noonedeadpunk | octavia-ovn-provider - I still failed to work closer on testing this path :( | 15:16 |
| jrosser_ | hmm yes another NODE_FAILURE - thats because we need the 32G flavor for capi tests | 15:17 |
| jrosser_ | and something is broken with those | 15:17 |
| noonedeadpunk | or well, there's probably something validly broken with ovn driver in octavia in our aio | 15:18 |
| noonedeadpunk | or well... a different set of tempest tests should run for ovn: https://96de7bed307fb2a6a065-7f155e7c59383dfa4a196ee8803910a9.ssl.cf1.rackcdn.com/868462/16/check/openstack-ansible-deploy-aio_lxc_ovnprovider-ubuntu-jammy/de203dd/logs/openstack/aio1-utility-container-3ffa4338/utility/stestr_results.html | 15:19 |
| noonedeadpunk | as it's getting just `Got NotImplemented error` | 15:19 |
| noonedeadpunk | which is fair.... | 15:19 |
| noonedeadpunk | will try to talk to octavia folks to see how we can test that | 15:19 |
| noonedeadpunk | and if they do have any tempest for that at all.. | 15:20 |
| jrosser_ | maybe thats just not valid to let octavia tempest go against the ovn provider | 15:20 |
| noonedeadpunk | yeah, but then how to test it... | 15:20 |
| jrosser_ | there might be some clue in how neutron set up to test that | 15:20 |
| jrosser_ | (i assume this happens.....) | 15:20 |
| jrosser_ | perhaps some already existing skip list we can use for example | 15:21 |
| mgariepy | isn't i only that the flavor in octavia is not needed for ovn ? | 15:26 |
| mgariepy | https://docs.openstack.org/ovn-octavia-provider/latest/admin/driver.html | 15:26 |
| mgariepy | * knows nothing about octavia tho. haha | 15:26 |
| noonedeadpunk | well, ovn also is only l4 balancing... and only source_ip_port algo iirc | 15:27 |
| mgariepy | Details: b'{"faultcode": "Server", "faultstring": "Provider \'ovn\' does not support a requested action: This provider does not support validating flavors.", "debuginfo": null}' | 15:28 |
| noonedeadpunk | yeah, which is fair... | 15:28 |
| noonedeadpunk | ok, will check on that a bit later | 15:28 |
| * noonedeadpunk looking through https://etherpad.opendev.org/p/osa-dalmatian-ptg | 15:29 | |
| noonedeadpunk | ovn-bgp-agent merged... | 15:29 |
| noonedeadpunk | inactive projects are removed... | 15:29 |
| noonedeadpunk | EOM branches... | 15:29 |
| noonedeadpunk | EOM branches are pita... | 15:30 |
| noonedeadpunk | We outdue to create Zed EOM | 15:30 |
| noonedeadpunk | *overdue | 15:30 |
| jrosser_ | we also need to choose if we work in the unmaintained branches at all | 15:30 |
| noonedeadpunk | yeah | 15:30 |
| jrosser_ | victoria is nearly OK, and i was planning to work back toward the maintained ones | 15:31 |
| noonedeadpunk | I guess this depends... but for work on them, they should be revived first and I failed to follow on that | 15:31 |
| johnsom | jrosser_ The Octavia tempest plugin will run against the OVN provider, however most of the tests will "skip" as the OVN provider doesn't support many of the features of Octavia. There are gate jobs that run with it. | 15:31 |
| jrosser_ | but just not managed to make progress | 15:31 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Add variable to globally control notifications enablement https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/917948 | 15:31 |
| noonedeadpunk | johnsom: well, we kinda try to run just octavia_tempest_plugin.tests.scenario.v2.test_load_balancer | 15:32 |
| jrosser_ | and this happens https://96de7bed307fb2a6a065-7f155e7c59383dfa4a196ee8803910a9.ssl.cf1.rackcdn.com/868462/16/check/openstack-ansible-deploy-aio_lxc_ovnprovider-ubuntu-jammy/de203dd/logs/openstack/aio1-utility-container-3ffa4338/utility/stestr_results.html | 15:32 |
| noonedeadpunk | which fails... but I guess I need to check what you're running in gates | 15:32 |
| noonedeadpunk | also, looking at test list... I wonder if we kinda want to always run `tempest.scenario.test_server_basic_ops.TestServerBasicOps`? | 15:33 |
| johnsom | I know what it is, one second | 15:33 |
| jrosser_ | i have to go to another meeting - but andrewbonney was interested in feedback on https://review.opendev.org/q/topic:%22osa/rmq-migrate%22 | 15:33 |
| noonedeadpunk | oh, yes | 15:33 |
| jrosser_ | particularly fixing up the tags, and whats best approach | 15:33 |
| johnsom | jrosser_ https://github.com/openstack/octavia-tempest-plugin/blob/master/zuul.d/jobs.yaml#L1100 | 15:34 |
| jrosser_ | i did discuss it with him and we could not see the point in tags like 'nova' - and the behaviour of them is just very wierd right now | 15:34 |
| jrosser_ | johnsom: ahha! | 15:34 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Add variable to globally control notifications enablement https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/917226 | 15:35 |
| johnsom | We are strict with the main test jobs | 15:35 |
| noonedeadpunk | ok, that explains it | 15:35 |
| noonedeadpunk | I wonder if anything will run though | 15:35 |
| johnsom | Mostly just a few TCP and UDP tests work with OVN | 15:36 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Implement variables to address oslo.messaging improvements https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/917997 | 15:36 |
| noonedeadpunk | so we have just that right now: https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables_octavia.yml.j2#L14 | 15:38 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/919171 | 15:38 |
| noonedeadpunk | as we pretty much want just bare minimal thing to see if it's basically operational | 15:39 |
| noonedeadpunk | we don't want to take your job by re-running all tests available :D | 15:39 |
| mgariepy | i guess that we need to set: not_implemented_is_error: False | 15:39 |
| johnsom | Yep, understandable. Yeah, I think all you need is to set that tempest variable | 15:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not fail on NotImplemented tests for OVN Octavia https://review.opendev.org/c/openstack/openstack-ansible/+/919599 | 15:40 |
| noonedeadpunk | let's see:) | 15:40 |
| johnsom | Here is a list of scenario tests run with OVN: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_833/917076/1/check/neutron-ovn-provider-v2-scenario/83362d1/testr_results.html | 15:40 |
| noonedeadpunk | thanks! | 15:40 |
| noonedeadpunk | ok, so octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest is part of that, amazing | 15:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Implement support for octavia-ovn-provider driver https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 15:41 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Implement variables to address oslo.messaging improvements https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/917949 | 15:48 |
| noonedeadpunk | so, looking at ptg doc, we're pretty much done? | 15:53 |
| noonedeadpunk | except renaming of groups.... | 15:53 |
| noonedeadpunk | but I think, we will do that for the next release.... | 15:54 |
| noonedeadpunk | and potentially - if we wanna replace nginx with apache for SKyline? https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/919529 | 15:54 |
| NeilHanlon | I'm pretty okay with it. would rather have haproxy + one other thing, not three | 15:55 |
| noonedeadpunk | #endmeeting | 16:11 |
| opendevmeet | Meeting ended Tue May 14 16:11:54 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:11 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-05-14-15.00.html | 16:11 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-05-14-15.00.txt | 16:11 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-05-14-15.00.log.html | 16:11 |
| noonedeadpunk | NeilHanlon: yeah, the only place where nginx is left - repo container | 16:12 |
| noonedeadpunk | worth to be replaced as well I guess... | 16:12 |
| NeilHanlon | just in time to move everything to Caddyserver! :P | 16:14 |
| noonedeadpunk | hahaha | 16:14 |
| opendevreview | Merged openstack/openstack-ansible-os_glance master: Add qos_prefetch_count to variables https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/919087 | 16:35 |
| noonedeadpunk | regarding https://review.opendev.org/q/topic:%22osa/rmq-migrate%22 - it looks quite fair to me | 16:41 |
| noonedeadpunk | andrewbonney: ^ | 16:41 |
| andrewbonney | Thanks. I'll sort out the full set of patches in the next day or two | 16:41 |
| noonedeadpunk | but, I think for things like nova - you might need some extra tasks | 16:41 |
| noonedeadpunk | like to detect virt type | 16:41 |
| noonedeadpunk | ah, but it has `always` | 16:42 |
| noonedeadpunk | so yeah, not needed :) | 16:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: DNM https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/919604 | 16:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/898914 | 16:46 |
| opendevreview | Merged openstack/openstack-ansible-os_manila master: reno: Update master for unmaintained/zed https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/919173 | 17:06 |
| noonedeadpunk | ok so manial fails on ceph-ansible with `nfs-ganesha : Depends: liburcu6 but it is not installable` | 19:08 |
| noonedeadpunk | as it's liburcu8 on jammy | 19:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!