| opendevreview | Mohammed Naser proposed openstack/neutron master: Skip trunk sub-ports in "update_virtual_port_host" https://review.opendev.org/c/openstack/neutron/+/896590 | 01:33 |
|---|---|---|
| opendevreview | Natz cheng proposed openstack/os-vif master: cleanup py27 support https://review.opendev.org/c/openstack/os-vif/+/896598 | 01:57 |
| *** ralonsoh_afk is now known as ralonsoh | 06:21 | |
| slaweq | ralonsoh hi, please check mnaser's patch https://review.opendev.org/c/openstack/neutron/+/896590 and his findings related to the bug https://bugs.launchpad.net/neutron/+bug/2024160 which he talked about yesterday evening our time | 06:43 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [OVN] Add baremetal support without Neutron DHCP agent for IPv6 https://review.opendev.org/c/openstack/neutron/+/890683 | 07:05 |
| slaweq | ralonsoh and please also check my last comment in https://review.opendev.org/c/openstack/neutron/+/890683 - thx in advance | 07:05 |
| ralonsoh | sure | 07:08 |
| ralonsoh | slaweq, https://review.opendev.org/c/openstack/neutron/+/890683/comments/2c953a3b_97614283 | 07:34 |
| ralonsoh | I think that makes sense (but please check it) | 07:34 |
| opendevreview | Merged openstack/neutron master: contributor docs: Architectural overview for metadata https://review.opendev.org/c/openstack/neutron/+/894911 | 08:06 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/2023.1: [OVN] Fix rate and burst for stateless security groups https://review.opendev.org/c/openstack/neutron/+/896614 | 08:19 |
| slaweq | ralonsoh thx, I will address Your comment and add this upgrade check to this patch | 08:23 |
| ralonsoh | slaweq, thanks a lot! | 08:44 |
| ralonsoh | please, check https://review.opendev.org/c/openstack/neutron/+/893447 if you have time | 08:44 |
| opendevreview | Stefan Hoffmann proposed openstack/neutron stable/2023.1: Reduce lock contention on subnets https://review.opendev.org/c/openstack/neutron/+/893082 | 09:14 |
| opendevreview | Stefan Hoffmann proposed openstack/neutron stable/zed: Reduce lock contention on subnets https://review.opendev.org/c/openstack/neutron/+/893084 | 09:15 |
| opendevreview | Stefan Hoffmann proposed openstack/neutron stable/yoga: Reduce lock contention on subnets https://review.opendev.org/c/openstack/neutron/+/889238 | 09:17 |
| opendevreview | Merged openstack/networking-bagpipe master: Update master for stable/2023.2 https://review.opendev.org/c/openstack/networking-bagpipe/+/895083 | 09:43 |
| MaxLamprecht[m] | ralonsoh: hi, I checked your mysql snipped on https://bugs.launchpad.net/neutron/+bug/2037107 and actually it proves the subnet X networkrbacs problem to me. Maybe you can have a look at that topic and my new comments again. thx | 10:09 |
| ralonsoh | MaxLamprecht[m], how is that? | 10:14 |
| ralonsoh | why this query proves the problem? | 10:14 |
| MaxLamprecht[m] | because the group by happens only in a subquery(not sure why sqlalcehmy is behaving like that). Afterwads we can see again a left outer join to networkrbacs in the query which is not needed (IMO) | 10:18 |
| ralonsoh | MaxLamprecht[m], exactly, what this subquery is retrieving? | 10:19 |
| MaxLamprecht[m] | the 2nd outer join happens due to the orm/ovo model with the lazy='joined" option | 10:19 |
| MaxLamprecht[m] | but the subquery it retrieving the network.id | 10:21 |
| ralonsoh | exactly and that subquery is returning just one ID | 10:21 |
| ralonsoh | this is why the group-by clause | 10:21 |
| ralonsoh | that is explained in the series of patches I pushed | 10:21 |
| MaxLamprecht[m] | yeah and all the networkrbacs and subnets have the same network_id -> thats why the combinatorial blow-up happens for subnets X networkrbacs | 10:23 |
| ralonsoh | again, the subquery is returning one network ID | 10:24 |
| ralonsoh | then the outer query is retrieving the subnets from this network ID | 10:25 |
| ralonsoh | the inner query is optimized with the group-by clause and returns one value | 10:25 |
| ralonsoh | the outer query selects the subnets from this single ID | 10:25 |
| ralonsoh | subnets don't have RBACs, the query filters first the network using the RBAC registers | 10:26 |
| ralonsoh | then retrieves the subnets from the filtered network IDs | 10:26 |
| MaxLamprecht[m] | ralonsoh: exactly, but why do we have than the left outer join to networkrbacs in the outer query | 10:27 |
| ralonsoh | you can't execute the inner query without adding the networkrbacs to the query | 10:28 |
| ralonsoh | but your assumption that this is combining both is wrong | 10:29 |
| ralonsoh | did you performed these queries manually? | 10:29 |
| MaxLamprecht[m] | but in the inner query there is also a left outer join to networkrbacs | 10:29 |
| ralonsoh | did you execute this query manually? | 10:30 |
| ralonsoh | did you check how many results return? | 10:30 |
| MaxLamprecht[m] | 16588 rows in set (0.06 sec) | 10:30 |
| ralonsoh | please post the query | 10:30 |
| MaxLamprecht[m] | I used the non admin query from your comment #3 (link [3]) | 10:31 |
| MaxLamprecht[m] | just replaced the project_id | 10:32 |
| ralonsoh | MaxLamprecht[m], no, I don't have this result | 10:35 |
| ralonsoh | so the reproducer script is not correct and something is missing | 10:35 |
| ralonsoh | please check the networkrbac table and what other registers have | 10:36 |
| MaxLamprecht[m] | but you see the 2 left outer joins to networkrbacs in this query? | 10:36 |
| ralonsoh | I do, yes | 10:36 |
| MaxLamprecht[m] | I will try to reproduce it in a new devstack | 10:36 |
| ykarel | bhaley, lajoskatona ralonsoh, ovs/linuxbridge job fails even with os-vif reverted patch so it's something else | 10:40 |
| ralonsoh | ykarel, when the issue started? | 10:40 |
| ykarel | ralonsoh, 22nd evening | 10:41 |
| lajoskatona | ykarel: the traceback was the same as I remember, but possible that we have to make the lb job non-voting? | 10:46 |
| lajoskatona | ralonsoh: see the opensearch link in the comment: https://bugs.launchpad.net/neutron/+bug/2037239/comments/2 | 10:46 |
| ykarel | lajoskatona, ovs jobs also impacted | 10:46 |
| lajoskatona | ykarel: I mean with os-vif fix/revert | 10:47 |
| ralonsoh | there is nothing relevant merged recently in the Neutron code | 10:47 |
| ralonsoh | are we using a different OVS version? | 10:47 |
| ykarel | lajoskatona, yes with os-vif revert issue still reproduces with both ovs/lb jobs | 10:47 |
| lajoskatona | ralonsoh: that would not effect linuxbridge jobs | 10:48 |
| ykarel | last night run had ovs jobs passing, but i rechecked today and it reproduces even in ovs jobs so os-vif patch is not related | 10:48 |
| MaxLamprecht[m] | <MaxLamprecht[m]> "but you see the 2 left outer..." <- The 1st join joins (**networks** X networkrbacs). The 2nd join joins (**subnets** X networkrbacs). | 10:52 |
| MaxLamprecht[m] | The group by/rbac filtering happens only for (**network** X networkrbacs) but not for the 2nd join. So due to that 2nd join we have the combinatorial blow-up between **subnets** and networkrbacs in one network. | 10:52 |
| ralonsoh | MaxLamprecht[m], ok, now I can reproduce it. I'll check how to introduce a new group-by in the subnet query | 11:07 |
| ralonsoh | ykarel, the GW (HA) port of the L3 agent receives an empty host | 11:12 |
| ralonsoh | Sep 22 21:26:41.021000 np0035327482 neutron-server[57688]: DEBUG neutron.db.l3_hamode_db [None req-c532b479-c945-41c8-81ef-e6fb0530d8e2 None None] HA router e8e3b263-ff89-4e6d-b1c0-44abd2403282 gateway port 6d7bf419-238a-4af4-8f78-9c459e82e22c binding host: , status: DOWN {{(pid=57688) _get_gateway_port_host /opt/stack/neutron/neutron/db/l3_hamode_db.py:784}} | 11:12 |
| ralonsoh | from https://89ad67c64b91ca5d8e4c-59f0897e687e7dfcd443cecd2cd3f6cf.ssl.cf1.rackcdn.com/896299/1/check/neutron-tempest-plugin-openvswitch/61cc665/job-output.txt | 11:13 |
| ralonsoh | well, the server logs | 11:13 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/2023.1: [OVN] Fix rate and burst for stateless security groups https://review.opendev.org/c/openstack/neutron/+/893460 | 11:16 |
| MaxLamprecht[m] | <ralonsoh> "Max Lamprecht, ok, now I can..." <- ahh nice, thought already that i´m doing something heavily wrong with my setup :D... (full message at <https://matrix.org/_matrix/media/v3/download/matrix.org/pccEHcGrurFBgriAJquxbjUD>) | 11:20 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 11:21 |
| Continuity | Hey all, ran across something over the last week or so with openvswitch on zed. Deployed using kolla ansible, with ubuntu containers. When making use of hardware offloading, we have been hitting this bug https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2018500 | 11:39 |
| Continuity | util(handler27)|EMER|../include/openvswitch/ofpbuf.h:194: assertion offset + size <= b->size failed in ofpbuf_at_assert() shown in the log | 11:40 |
| Continuity | causes OVS to crash and restart causing a slight drop in connectivity. | 11:40 |
| Continuity | just thought I would mention it if anyone else has/is seeing the issue. | 11:40 |
| Continuity | Its fixed in yoga, and antollope, I have requested a backport fix to zed | 11:40 |
| Continuity | for reference. https://bugs.launchpad.net/cloud-archive/+bug/2037532 | 11:41 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: nb: add 'nexthop' argument to 'lr_route_del' https://review.opendev.org/c/openstack/ovsdbapp/+/896645 | 11:52 |
| opendevreview | Reason li proposed openstack/neutron master: Parameter filters may be None, which cannot be called with ** https://review.opendev.org/c/openstack/neutron/+/896649 | 12:23 |
| opendevreview | Felipe Figueroa Vergara proposed openstack/tap-as-a-service master: Enables external network mirroring https://review.opendev.org/c/openstack/tap-as-a-service/+/896515 | 12:46 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/885111 | 13:08 |
| opendevreview | Mohammed Naser proposed openstack/neutron master: [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent https://review.opendev.org/c/openstack/neutron/+/896590 | 13:11 |
| mnaser | ralonsoh: thank you for your review, your fix is a lot cleaner lol | 13:12 |
| ralonsoh | checking now | 13:12 |
| mnaser | but hey i'm improving my ovn chops | 13:12 |
| ralonsoh | mnaser, do you feel you can implement a FT to avoid future regressions? | 13:13 |
| ralonsoh | if not, I can push a new PS with this FT | 13:13 |
| mnaser | ralonsoh: i tried to play around with that (i saw a `test_virtual_port_host_update`) but i think its a (little) outside of it | 13:14 |
| mnaser | i would appreciate if you could | 13:15 |
| ralonsoh | ok | 13:15 |
| mnaser | thank you! | 13:16 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == Add policy enforcer for "tags" service plugin https://review.opendev.org/c/openstack/neutron/+/896509 | 13:19 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent https://review.opendev.org/c/openstack/neutron/+/896590 | 13:48 |
| ralonsoh | slaweq, hi! if you have 1 min: https://review.opendev.org/c/openstack/neutron/+/893447 | 13:56 |
| ralonsoh | thanks! | 13:56 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.2: Revert "[OVN][Trunk] Add port binding info on subport when parent is bound" https://review.opendev.org/c/openstack/neutron/+/895899 | 14:38 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.1: Revert "[OVN][Trunk] Add port binding info on subport when parent is bound" https://review.opendev.org/c/openstack/neutron/+/894791 | 14:38 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/zed: Revert "[OVN][Trunk] Add port binding info on subport when parent is bound" https://review.opendev.org/c/openstack/neutron/+/894792 | 14:39 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/yoga: Revert "[OVN][Trunk] Add port binding info on subport when parent is bound" https://review.opendev.org/c/openstack/neutron/+/894793 | 14:39 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/xena: Revert "[OVN][Trunk] Add port binding info on subport when parent is bound" https://review.opendev.org/c/openstack/neutron/+/894794 | 14:39 |
| opendevreview | Lajos Katona proposed openstack/networking-bagpipe master: DNM: test master https://review.opendev.org/c/openstack/networking-bagpipe/+/896677 | 14:44 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: nb: add 'nexthop' argument to 'lr_route_del' https://review.opendev.org/c/openstack/ovsdbapp/+/896645 | 14:55 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: venv: unit tests don't work for 'hardware_vtep' https://review.opendev.org/c/openstack/ovsdbapp/+/896682 | 15:36 |
| ralonsoh | ykarel, hi, did you find something? | 15:42 |
| ralonsoh | the only difference I see between a working job and a failing one is that the ovs agent (I know it is failing too in ML2/LB) is not binding the GW port | 15:43 |
| ralonsoh | and thus the port is not set to up and the agent is not declared as active | 15:44 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/yoga: Check multiple address of a LRP plugged to LS https://review.opendev.org/c/openstack/ovn-octavia-provider/+/896269 | 15:44 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: Introduce ovn_nb_global config section https://review.opendev.org/c/openstack/neutron/+/896545 | 16:29 |
| mnaser | ralonsoh: seems like these original test has failed, i marked where it failed, i _suspect_ the old test was maybe not correct | 16:30 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: nb: allow to add route even if learned route exists https://review.opendev.org/c/openstack/ovsdbapp/+/896686 | 16:30 |
| ralonsoh | mnaser, I tested it manually and wasn't failing | 16:31 |
| mnaser | the new test or the old one | 16:31 |
| ralonsoh | both | 16:31 |
| mnaser | ah ok | 16:31 |
| mnaser | ovn version dependent ? | 16:31 |
| mnaser | well, we know the behaviour does depend on the ovn version actually | 16:31 |
| mnaser | seems like that job used v23.09.0 | 16:33 |
| ralonsoh | that patch was backported up to ovn-nbctl 22.03 | 16:34 |
| ralonsoh | but what version in particular? | 16:34 |
| haleyb | https://review.opendev.org/c/openstack/neutron/+/895849 just changed functional job to v23.09.0 | 16:35 |
| ralonsoh | I'm running with the jammy package | 16:35 |
| mnaser | i think jammy with uca is different than jammy without uca | 16:35 |
| ralonsoh | mnaser, ok, with this check and the new version, this check does not apply | 16:36 |
| ralonsoh | because the port_binding is deleted but the port is not yet virtual | 16:36 |
| ralonsoh | it is in the next event call | 16:36 |
| mnaser | yeah thats what i saw, you can see in the issue i put some logs of the events from neutron debug | 16:37 |
| mnaser | thats kinda why i resorted to my solution cause i couldnt think of a way of detecting a "deleted because being recreated" | 16:37 |
| ralonsoh | I'll reinstall my dev env with v23.09.0 | 16:37 |
| mnaser | yeah in my case I encountered this in 23.03.0 | 16:39 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Match LSP_TYPE_VIRTUAL in PortBindingUpdateVirtualPortsEvent https://review.opendev.org/c/openstack/neutron/+/896590 | 16:39 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == Add policy enforcer for "tags" service plugin https://review.opendev.org/c/openstack/neutron/+/896509 | 16:56 |
| opendevreview | Merged openstack/ovn-octavia-provider stable/zed: Cover the use case of a member non existing https://review.opendev.org/c/openstack/ovn-octavia-provider/+/894926 | 17:02 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: venv: unit tests don't work for 'hardware_vtep' https://review.opendev.org/c/openstack/ovsdbapp/+/896682 | 17:02 |
| *** ralonsoh is now known as ralonsoh_ooo | 17:02 | |
| ykarel | ralonsoh, no haven't got the root cause for it | 17:04 |
| ykarel | just Error while reading HA state for <router id> Error while reading HA state for [Errno 2] No such file or directory: '/opt/stack/data/neutron/ha_confs/<router id>/state' | 17:05 |
| ykarel | Updating server with state unknown for router <router id> | 17:05 |
| ykarel | but may be that's just the side effect of gw port not bound ? | 17:06 |
| *** ykarel is now known as ykarel|away | 17:16 | |
| haleyb | ykarel|away: that error might be a red herring, if keepalived has never been started for a router there should be no files. The errors about missing pid files are just as annoying and printed multiple times :-/ | 17:38 |
| mnaser | ralonsoh_ooo: wooo it passed this time around (well the functional test at least) | 17:49 |
| mnaser | https://review.opendev.org/c/openstack/neutron/+/896590 this is ready to review | 19:15 |
| *** JayF is now known as Guest1505 | 20:10 | |
| *** JasonF is now known as jayf | 20:10 | |
| *** jayf is now known as JayF | 20:11 | |
| *** jlibosva is now known as Guest1506 | 20:11 | |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: functional: Enable test_fip_connection_for_address_scope test https://review.opendev.org/c/openstack/neutron/+/896728 | 20:54 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: venv: unit tests don't work for 'hardware_vtep' https://review.opendev.org/c/openstack/ovsdbapp/+/896682 | 20:57 |
| *** kopecmartin is now known as kopecmartin|off | 20:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!