| opendevreview | liuyulong proposed openstack/neutron-tempest-plugin master: Re-enable the neutron-tempest-plugin-dvr-multinode-scenario case https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/936318 | 01:12 |
|---|---|---|
| opendevreview | liuyulong proposed openstack/neutron-tempest-plugin master: Re-enable the neutron-tempest-plugin-dvr-multinode-scenario case https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/936318 | 01:17 |
| ricolin | Hi team, just like to raise some attention on the OVN DB sync patches https://review.opendev.org/c/openstack/ovn-octavia-provider/+/925747 would be greate to get some review on it. Thanks in advance!! | 04:10 |
| ralonsoh | ykarel, https://5cd68356f521473305f8-cd725815d30cf2ab8520aad695ae6c76.ssl.cf2.rackcdn.com/935280/4/gate/neutron-ovn-tempest-ipv6-only-ovs-release/1b2bc99/controller/logs/screen-neutron-api.txt | 07:06 |
| ralonsoh | another ocurrence of the network creation issue | 07:06 |
| ralonsoh | I'm investigating it right now | 07:06 |
| ykarel | ralonsoh, so with my test results with single api worker i didn't hit single failure in 300 runs(2 job failed but those unrelated) | 07:16 |
| ykarel | while with higher workers it reproduces more constantly | 07:16 |
| ykarel | 1 worker https://review.opendev.org/c/openstack/neutron/+/936147 | 07:17 |
| ykarel | 5 worker https://review.opendev.org/c/openstack/neutron/+/936272 | 07:17 |
| ykarel | so it looks issue arising with some conflict when workers >=2 | 07:19 |
| ralonsoh | right, I think this could be a concurrency issue | 07:35 |
| ralonsoh | I'm checking the problem with the Neutron DB update | 07:35 |
| ralonsoh | there is an issue with the std attributes update | 07:36 |
| ralonsoh | ykarel, inhttps://review.opendev.org/c/openstack/neutron/+/936272 there are also issues related to the PG deletion | 07:38 |
| ykarel | yes failed that way many times | 07:39 |
| ralonsoh | I think this issue is covered in https://review.opendev.org/c/openstack/neutron/+/934409 | 07:39 |
| ralonsoh | ^ this patch is waiting for a new ovsdbapp release | 07:39 |
| ykarel | but strange is why we hitting it constantly with uwsgi, so likely need to fix root cause | 07:40 |
| ykarel | with concurrency | 07:40 |
| ralonsoh | when we have problems in an execution, this is always present | 07:41 |
| ralonsoh | Nov 27 14:41:34.787587 np0039183722 devstack@neutron-api.service[59381]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ab5be635-3125-485a-84c3-d921faae9c81'". All 0 nodes were found offline. This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpt | 07:41 |
| ralonsoh | y: Hash Ring returned empty when hashing "b'ab5be635-3125-485a-84c3-d921faae9c81'". All 0 nodes were found offline. This should never happen in a normal situation, please check the status of your cluster | 07:41 |
| ykarel | yes ^ caused as hash ring tasks running each worker | 07:42 |
| ykarel | one it runs for next worker, it clear for previous one and creates again | 07:42 |
| ykarel | in past it used to add node for each worker, but now it will be always 1 irrespective of number of workers running | 07:42 |
| ralonsoh | sorry how it was before? | 07:43 |
| ralonsoh | I'm checking this nwo | 07:43 |
| ralonsoh | now* | 07:43 |
| ykarel | ralonsoh, https://github.com/openstack/neutron/blob/0ac3fbdf86909e22ea0659146b05eac258626f4b/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L315-L318 | 07:45 |
| ralonsoh | but that should work the same, lockutils.synchronized should work interprocess | 07:46 |
| ykarel | no idea why it's not working as expected | 07:54 |
| ralonsoh | ykarel, ok I think I have something | 07:54 |
| ykarel | great \o/ | 07:55 |
| ralonsoh | the line "Hash Ring probing thread has started" should be only one in the logs | 07:55 |
| ralonsoh | because this section of the code must be executed only once per node (regardless of the workers) | 07:55 |
| ralonsoh | and we have many in our jobs... | 07:55 |
| ykarel | yes correct | 07:55 |
| ralonsoh | I think this is because the 'self._hash_ring_probe_event.set()" is set at the end | 07:55 |
| ralonsoh | must be set just at the beginning | 07:56 |
| ralonsoh | or use a context | 07:56 |
| ralonsoh | I'll try that | 07:56 |
| ykarel | mmm but why we didn't hit that earlier with no uwsgi | 07:56 |
| ralonsoh | because how we spawn the workers in eventlet, that is one by one | 07:56 |
| ykarel | and as there is lock synchronized why other worker using it? | 07:56 |
| ralonsoh | I think wsgi spawns all workers at the same time | 07:56 |
| opendevreview | Eduardo Olivares proposed openstack/ovn-bgp-agent master: Fix text from CrLrpChassisChangeWithdrawEvent docstring https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936427 | 08:03 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Improve initial hash ring setup https://review.opendev.org/c/openstack/neutron/+/936428 | 08:16 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 4 workers https://review.opendev.org/c/openstack/neutron/+/936429 | 08:18 |
| opendevreview | Merged openstack/ovn-bgp-agent stable/2024.2: Support ovn-bgp agent with DVR disabled https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936397 | 08:18 |
| opendevreview | Merged openstack/ovn-bgp-agent stable/2024.1: Support ovn-bgp agent with DVR disabled https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936398 | 08:18 |
| opendevreview | Amir Nikpour proposed openstack/ovsdbapp master: Execute run methods in parallel https://review.opendev.org/c/openstack/ovsdbapp/+/936327 | 08:20 |
| opendevreview | Merged openstack/ovn-bgp-agent stable/2023.2: Support ovn-bgp agent with DVR disabled https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936399 | 08:22 |
| ralonsoh | bcafarel, hello! good morning. If you have 1 min: https://review.opendev.org/q/Ie6210f7dab4d54d734255d3ac2271cac99590f46 | 08:43 |
| ralonsoh | thanks! | 08:43 |
| ralonsoh | and https://review.opendev.org/q/I3ac6364200f5124d760587612d3a9de55830f2b1, if possible | 08:43 |
| ralonsoh | slaweq, hello! please check https://review.opendev.org/c/openstack/releases/+/936345 | 08:44 |
| ralonsoh | thanks! | 08:44 |
| ralonsoh | ykarel, https://zuul.opendev.org/t/openstack/status?change=936428 it looks much better (with 4 workers). But we still have one failed execution | 08:50 |
| ralonsoh | I'm trying to find a better way to sync the workers for this task | 08:50 |
| ykarel | ralonsoh, ack thx | 09:03 |
| opendevreview | Amir Nikpour proposed openstack/ovsdbapp master: Execute run methods in parallel https://review.opendev.org/c/openstack/ovsdbapp/+/936327 | 09:20 |
| opendevreview | Amir Nikpour proposed openstack/ovsdbapp master: Execute run methods in parallel https://review.opendev.org/c/openstack/ovsdbapp/+/936327 | 10:13 |
| opendevreview | Merged openstack/neutron-lib master: Add resource and collection name in uplink-status-propagation-updatable https://review.opendev.org/c/openstack/neutron-lib/+/936234 | 10:40 |
| opendevreview | Merged openstack/neutron master: Drop duplicate hacking check https://review.opendev.org/c/openstack/neutron/+/936320 | 11:18 |
| opendevreview | Merged openstack/neutron stable/2024.2: [OVN] Add a creation wait event for the PG drop tests https://review.opendev.org/c/openstack/neutron/+/936011 | 11:18 |
| opendevreview | Amir Nikpour proposed openstack/ovsdbapp master: Execute run methods in parallel https://review.opendev.org/c/openstack/ovsdbapp/+/936327 | 11:20 |
| sean-k-mooney | ralonsoh: if your ok with this form a neutron perspective https://review.opendev.org/c/openstack/os-vif/+/923036 im sure bence would be happy to see that merged. | 11:29 |
| ralonsoh | let me check | 11:29 |
| sean-k-mooney | no rush just an fyi | 11:30 |
| opendevreview | Merged openstack/neutron master: Migrate from tenant_id to project_id in ``test_network.py`` https://review.opendev.org/c/openstack/neutron/+/935280 | 11:43 |
| ralonsoh | zzzeek, hello! I have a tricky question for you (although maybe you are in PTO). I need to sync some workers around a single operation in the DB | 11:57 |
| ralonsoh | scenario: when the Neutron API starts, it spawns several workers (as any other API) | 11:58 |
| ralonsoh | so there is a table that registers each worker. We have a signal handler to remove this register when the worker is stopped | 11:59 |
| ralonsoh | but, of course, that could fail | 11:59 |
| ralonsoh | so the goal is that only one single worker, at the beginning, should delete this table and register itself | 11:59 |
| ralonsoh | the other workers should register themselves BUT should not wipe out the table | 11:59 |
| ralonsoh | do happen to know how to use the sql engine transactionality to implement this? | 12:00 |
| opendevreview | yatin proposed openstack/neutron master: [WIP] Global venv for func tests https://review.opendev.org/c/openstack/neutron/+/935847 | 13:25 |
| opendevreview | yatin proposed openstack/neutron master: [WIP] Global venv for func tests https://review.opendev.org/c/openstack/neutron/+/935847 | 13:27 |
| frickler | ralonsoh: reading the above, I wonder how that should work if the api is started on multiple servers in parallel. or maybe not even in parallel. how should a new controller decide whether registrations are stale or belong to a running other controller? | 14:44 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add octavia_client with openstacksdk https://review.opendev.org/c/openstack/ovn-octavia-provider/+/930692 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add LB sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/925324 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Listener sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931250 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Pool sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931266 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Member sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931267 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Health monitor sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931288 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add sync floating IP support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 | 14:55 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add octavia_client with openstacksdk https://review.opendev.org/c/openstack/ovn-octavia-provider/+/930692 | 15:19 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add LB sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/925324 | 15:19 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Listener sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931250 | 15:19 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Pool sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931266 | 15:19 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Member sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931267 | 15:19 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add Health monitor sync logic https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931288 | 15:19 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Add sync floating IP support https://review.opendev.org/c/openstack/ovn-octavia-provider/+/929039 | 15:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ovn-bgp-agent master: Ensure that ARP/NDP is enabled for vlan devices https://review.opendev.org/c/openstack/ovn-bgp-agent/+/935801 | 16:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/neutron master: [doc] Update plugins needed for auto-allocation https://review.opendev.org/c/openstack/neutron/+/936643 | 16:26 |
| opendevreview | yatin proposed openstack/neutron master: DNM - Test tempest jobs in Ubuntu Noble (24.04) https://review.opendev.org/c/openstack/neutron/+/932920 | 17:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ovn-bgp-agent stable/2024.2: Handle trimming of vlan interface namings https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936651 | 17:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ovn-bgp-agent stable/2024.1: Handle trimming of vlan interface namings https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936652 | 17:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ovn-bgp-agent stable/2023.2: Handle trimming of vlan interface namings https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936653 | 17:54 |
| opendevreview | Ghanshyam proposed openstack/neutron-tempest-plugin master: Update gate jobs as per the 2025.1 cycle testing runtime https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/933277 | 23:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!