| opendevreview | Merged openstack/neutron-lib master: vpnaas: add support for AES CTR https://review.opendev.org/c/openstack/neutron-lib/+/903971 | 03:12 |
|---|---|---|
| opendevreview | Gaudenz Steinlin proposed openstack/neutron master: Fixup conntrackd support https://review.opendev.org/c/openstack/neutron/+/938800 | 06:08 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Reduce the OVN hash ring touch interval https://review.opendev.org/c/openstack/neutron/+/937351 | 06:38 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == [OVN] ``PortBindingUpdateUpEvent`` https://review.opendev.org/c/openstack/neutron/+/939345 | 08:14 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with WSGI - BUG 2094736 https://review.opendev.org/c/openstack/neutron/+/939347 | 08:27 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Reduce the OVN hash ring touch interval https://review.opendev.org/c/openstack/neutron/+/937351 | 09:45 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Do not fail if the ACL does not exists in the deletion https://review.opendev.org/c/openstack/neutron/+/934409 | 09:46 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-dynamic-routing master: Use context reader to wrap several DB operations https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/939428 | 10:04 |
| ralonsoh | ykarel, hello! would you consider reviewing https://review.opendev.org/c/openstack/neutron/+/934409 again? | 10:13 |
| ralonsoh | btw, I'm checking the issue wiht n-d-r (https://bugs.launchpad.net/neutron/+bug/2085421) and I don't find anything relevant | 10:13 |
| ralonsoh | I don't know why the port deletion transaction is locked | 10:14 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [FT] Reduce arping interval in ``test_send_ipv4_addr_adv_notif`` https://review.opendev.org/c/openstack/neutron/+/939435 | 11:45 |
| ralonsoh | slaweq, ykarel ^^ trivial fix | 12:11 |
| ralonsoh | or anyone else, thanks in advance! | 12:12 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread https://review.opendev.org/c/openstack/neutron/+/937843 | 12:27 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: async_process: remove usage of eventlet for AsyncProcess https://review.opendev.org/c/openstack/neutron/+/939348 | 12:27 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: reimplement signals handling https://review.opendev.org/c/openstack/neutron/+/939321 | 12:27 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent https://review.opendev.org/c/openstack/neutron/+/937765 | 12:27 |
| sahid | ralonsoh: o/ regarding AsyncProcess, fi you have a moment to have a look int it https://review.opendev.org/c/openstack/neutron/+/939348 | 12:30 |
| sahid | and btw about the signal part that we have discussed, i have started implementation here: https://review.opendev.org/c/openstack/neutron/+/939321 | 12:31 |
| sahid | the ovs agent does not look to use olso service with the handler thing that you mentioned, but I may have missed something | 12:32 |
| sahid | :-) | 12:32 |
| sahid | i'm trying to see how I can carticulate the change with some tests | 12:32 |
| ralonsoh | sure, I need to finish other stuff before | 12:51 |
| opendevreview | Vasyl Saienko proposed openstack/neutron master: Increase ovs operation timeout for functional tests https://review.opendev.org/c/openstack/neutron/+/939439 | 13:00 |
| ykarel | ralonsoh, ack for both | 13:18 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [FT] Reduce arping interval in ``test_send_ipv4_addr_adv_notif`` https://review.opendev.org/c/openstack/neutron/+/939435 | 13:40 |
| opendevreview | Merged openstack/neutron stable/2024.2: ovn: Disable activation-strategy=rarp for DPDK ports https://review.opendev.org/c/openstack/neutron/+/939235 | 13:56 |
| opendevreview | Bodo Petermann proposed openstack/neutron-vpnaas master: Add support for additional auth, encryption, PFS choices https://review.opendev.org/c/openstack/neutron-vpnaas/+/898830 | 14:09 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Check existence of GW port before trying to delete it https://review.opendev.org/c/openstack/neutron/+/939451 | 14:09 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Check existence of GW port before trying to delete it https://review.opendev.org/c/openstack/neutron/+/939451 | 14:15 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with WSGI - BUG 2094736 https://review.opendev.org/c/openstack/neutron/+/939347 | 14:16 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == [OVN] ``PortBindingUpdateUpEvent`` https://review.opendev.org/c/openstack/neutron/+/939345 | 14:35 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Check existence of GW port before trying to delete it https://review.opendev.org/c/openstack/neutron/+/939451 | 14:35 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with WSGI - BUG 2094736 https://review.opendev.org/c/openstack/neutron/+/939347 | 14:35 |
| opendevreview | Vasyl Saienko proposed openstack/neutron master: Increase ovs operation timeout for functional tests https://review.opendev.org/c/openstack/neutron/+/939439 | 14:55 |
| ralonsoh | ykarel, ihrachys please check https://review.opendev.org/c/openstack/neutron/+/939345. This is WIP, and is in the middle of a chain of patches. In this order: | 14:57 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/937351/8 | 14:57 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/939345/3 | 14:57 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/939451/3 | 14:57 |
| ralonsoh | on top of this, I've pushed a testing patch | 14:58 |
| ralonsoh | ykarel, btw, the CI is still failing due to the PG deletion. So please reconsider https://review.opendev.org/c/openstack/neutron/+/934409/11 | 14:59 |
| ykarel | ralonsoh, can you share the failures without involving 934409 i just want to ensure ovn db don't leave stale data and hide the issue | 15:30 |
| ykarel | can do share over the patch, sorry i need to drop for now | 15:30 |
| ralonsoh | ykarel, for sure, and thanks! | 15:31 |
| ihrachys | ralonsoh: tl;dr for the WIP is use pb instead of lsp (unless it's router port because these don't have pb updated or smth?) how does this scratch with the assumption that it's hash ring members losing events depending on the hash calculation (so it can either be a pb update or lsp update - but they will lose some of these nevertheless)? | 15:35 |
| ihrachys | (the fact Up handlers are not symmetric with Down handlers is also a concern, but probably just left out of scope for now because it's WIP I assume) | 15:36 |
| ihrachys | wonder if BOTH events should be handled? (this doesn't address the lost events root cause - whatever it is - but at least raises the chance that at least one will be processed? just spitballing here) | 15:36 |
| ihrachys | did a test patch confirm the series is helpful? | 15:37 |
| ralonsoh | ihrachys, router ports are active when created, without any PB | 15:51 |
| ralonsoh | ihrachys, because we had problems only with up=true, I only implemented the UP event | 15:51 |
| ralonsoh | ihrachys, the test patch is still on-going, I've added more patches to the mail chain | 15:52 |
| ralonsoh | the test patch detected an issue with ACLs, the one I commented with Yatin | 15:52 |
| ralonsoh | patch related: https://review.opendev.org/c/openstack/neutron/+/934409 | 15:52 |
| ralonsoh | (there are multiple open fronts...) | 15:53 |
| ihrachys | the going theory I think is that we have problems not with lsps but with lost events in general. if so, switching from lsp to pb shouldn't do a thing. so I wonder. | 15:57 |
| ihrachys | and we know that this issue is wsgi env specific | 15:57 |
| ihrachys | the expectation is that the maintenance thread touches hash ring nodes once in a while. but it doesn't. | 15:58 |
| ihrachys | do we know that at least the eventloop is aware of the thread should be scheduled but just busy with something else for the whole minute without letting the control away? is there a way to confirm this? | 15:59 |
| ihrachys | (an alternative could be e.g. that the periodic is not even in consideration for some reason.) | 15:59 |
| ralonsoh | ihrachys, at least we can try increasing the timeout and reducing the refresh time | 15:59 |
| ralonsoh | (the patch proposed) | 15:59 |
| ralonsoh | that should aleviate the issue of the hash ring offline nodes | 16:00 |
| ihrachys | I've seen one failure where there were no logs in neutron-server for whole 2 minutes, then hash ring members went offline (there were no ovsdb-monitor event handler messages - so these did not touch nodes either) and events got lost. | 16:00 |
| ihrachys | can we maybe log every time someone touches a node, with info about who did? | 16:00 |
| ralonsoh | ihrachys, in what method exactly? | 16:00 |
| ihrachys | timeouts - yes I agree that patch should not make it worse | 16:00 |
| ralonsoh | the one updating the hash ring registers in the DB? | 16:01 |
| ihrachys | ralonsoh: in both maintenance periodic and ovsdb-monitor notify handler | 16:01 |
| ihrachys | (I think these are the only two who call to touch_node) | 16:01 |
| ralonsoh | we can directly add this message in the touch_node method | 16:01 |
| ralonsoh | or maybe you want to add who is calling it | 16:02 |
| ihrachys | yeah would be nice to trace who did. we can have separate log messages in all the callers for this. | 16:03 |
| ihrachys | there are not many anyway | 16:03 |
| ralonsoh | ihrachys, I'll do it tomorrow morning, I need to go leave in 2 minutes | 16:04 |
| ralonsoh | actually I'll do it this afternoon/night | 16:04 |
| ihrachys | I'll send a patch in a sec | 16:04 |
| ralonsoh | I'll review it later | 16:04 |
| ihrachys | great. ciao, see you later. | 16:04 |
| opendevreview | Ihar Hrachyshka proposed openstack/neutron master: ovn: Log every time a hash ring node is touched https://review.opendev.org/c/openstack/neutron/+/939469 | 16:12 |
| opendevreview | Vasyl Saienko proposed openstack/neutron master: Increase ovs operation timeout for functional tests https://review.opendev.org/c/openstack/neutron/+/939439 | 16:13 |
| opendevreview | Ihar Hrachyshka proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with WSGI https://review.opendev.org/c/openstack/neutron/+/939360 | 16:14 |
| opendevreview | Merged openstack/neutron stable/2024.1: ovn: Disable activation-strategy=rarp for DPDK ports https://review.opendev.org/c/openstack/neutron/+/939236 | 17:33 |
| haleyb | ihrachys: do you want to merge that ovn hash ring debug change? or is more for debugging? | 17:56 |
| ihrachys | I think it's fine to merge it but obv it won't solve anything. assuming it doesn't blow up the size of the log files too much? let me check what it produced. | 18:02 |
| * haleyb thinks you mentioned the log files yesterday, can't remember his name... | 18:05 | |
| haleyb | s/you/who | 18:05 |
| ihrachys | it increased size of logs by 1% in a tempest run | 18:06 |
| ihrachys | (size as measured in locs, not bytes) | 18:08 |
| ihrachys | that said I think it's helpful; I am looking at some logs and already have some theories I think but I will have to think about these for a bit longer. | 18:18 |
| ihrachys | definitely futurist periodics doesn't get triggered in promised frequency | 18:21 |
| ihrachys | but I don't think the thread is busy with anything while it's stuck | 18:22 |
| opendevreview | Merged openstack/neutron master: Install DVR src to MAC flow for AAPs https://review.opendev.org/c/openstack/neutron/+/938657 | 21:04 |
| opendevreview | Ihar Hrachyshka proposed openstack/neutron stable/2024.2: functional: Handle ovsdb monitor returning inserts in different checks https://review.opendev.org/c/openstack/neutron/+/939495 | 21:25 |
| opendevreview | Ihar Hrachyshka proposed openstack/neutron stable/2024.1: functional: Handle ovsdb monitor returning inserts in different checks https://review.opendev.org/c/openstack/neutron/+/939496 | 21:25 |
| opendevreview | Ihar Hrachyshka proposed openstack/neutron stable/2023.2: functional: Handle ovsdb monitor returning inserts in different checks https://review.opendev.org/c/openstack/neutron/+/939497 | 21:26 |
| opendevreview | Merged openstack/neutron master: ovn: Log every time a hash ring node is touched https://review.opendev.org/c/openstack/neutron/+/939469 | 21:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!