| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: Temporary turn off l3_ha in ovs/lb master/bobcat jobs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897233 | 04:44 |
|---|---|---|
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: Use RE2 compatible regex for branches https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/896188 | 04:48 |
| opendevreview | yatin proposed openstack/neutron master: [DNM] Test oslo.log pin https://review.opendev.org/c/openstack/neutron/+/896504 | 07:28 |
| opendevreview | Michel Nederlof proposed openstack/neutron stable/2023.1: Update lsp host id and remove mac binding when virtual parent changes https://review.opendev.org/c/openstack/neutron/+/896882 | 07:38 |
| ykarel | thanks ralonsoh for including more jobs in keepalived bash switch, but i see still some failures in those jobs | 07:46 |
| ralonsoh | ykarel, the ip monitor is not the cause here | 07:54 |
| ralonsoh | but yes, we still see "...is not active on any of the L3 agents" | 07:54 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: Add job definitions for 2023.2 (Bobcat) branch https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/895516 | 08:00 |
| ykarel | ralonsoh, ack yes i just saw those "not active on any", not checked the cause | 08:01 |
| opendevreview | Michel Nederlof proposed openstack/neutron master: [OVN] Update lsp host id when virtual parent changes https://review.opendev.org/c/openstack/neutron/+/896883 | 08:12 |
| opendevreview | Michel Nederlof proposed openstack/neutron master: [OVN] Update lsp host id when virtual parent changes https://review.opendev.org/c/openstack/neutron/+/896883 | 08:13 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVS][QoS] Add QoS support for Trunk service, OVS driver https://review.opendev.org/c/openstack/neutron/+/839523 | 08:20 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP - Add ``OVNGatewayHAChassisGroup`` scheduler class https://review.opendev.org/c/openstack/neutron/+/872033 | 08:36 |
| opendevreview | Jakub Skunda proposed openstack/neutron-tempest-plugin master: Remove test duplications between tempest and n-t-p plugin https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/896410 | 08:54 |
| ralonsoh | slaweq, hello! If you have less than 1 min: https://review.opendev.org/c/openstack/neutron/+/896926 | 09:05 |
| ralonsoh | thanks! | 09:06 |
| opendevreview | Michel Nederlof proposed openstack/neutron master: [OVN] Delete entries for virtual ip from Mac_Binding table https://review.opendev.org/c/openstack/neutron/+/896884 | 09:08 |
| opendevreview | Michel Nederlof proposed openstack/neutron master: [OVN] Delete entries for virtual ip from Mac_Binding table https://review.opendev.org/c/openstack/neutron/+/896884 | 09:12 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [S-RBAC] Add service role in neutron policy https://review.opendev.org/c/openstack/neutron/+/886724 | 09:16 |
| slaweq | ralonsoh done, but You can also review https://review.opendev.org/c/openstack/neutron/+/886724 as I rebased it and addressed Your comments | 09:21 |
| ralonsoh | for sure, give me 1 minute | 09:22 |
| slaweq | lajoskatona ykarel haleyb please also check ^^ when You will have some time, thx in advance | 09:22 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: Turn off host-passthrough in scenario jobs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897308 | 09:35 |
| ykarel | slaweq, ack | 09:35 |
| lajoskatona | slaweq: on my list | 09:36 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.2: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/897019 | 09:48 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2023.1: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/897020 | 09:48 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/zed: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/897041 | 09:48 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/yoga: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/897044 | 09:50 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/xena: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/897043 | 09:50 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/wallaby: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/897045 | 09:51 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/2023.1: [OVN] Fix rate and burst for stateless security groups https://review.opendev.org/c/openstack/neutron/+/893460 | 10:26 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron stable/2023.1: Use safer methods to get security groups on security group logging https://review.opendev.org/c/openstack/neutron/+/896614 | 10:26 |
| opendevreview | Felix Huettner proposed openstack/neutron master: ovn-l3: reschedule priorities on new chassis https://review.opendev.org/c/openstack/neutron/+/893655 | 11:06 |
| opendevreview | Felix Huettner proposed openstack/neutron master: ovn-l3 router scheduler: reproduce scheduling issue https://review.opendev.org/c/openstack/neutron/+/893656 | 11:06 |
| opendevreview | Felix Huettner proposed openstack/neutron master: ovn-l3: implement caching for Scheduler https://review.opendev.org/c/openstack/neutron/+/893657 | 11:06 |
| opendevreview | Felix Huettner proposed openstack/neutron master: ovn-l3: try to keep routers at current chassis https://review.opendev.org/c/openstack/neutron/+/893658 | 11:06 |
| opendevreview | Felix Huettner proposed openstack/neutron master: ovn-l3: value the higher priorities when scheduling https://review.opendev.org/c/openstack/neutron/+/893659 | 11:06 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: Turn off host-passthrough in scenario jobs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897308 | 11:11 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Create a policy rule to control if a rule belongs to the default SG https://review.opendev.org/c/openstack/neutron/+/896222 | 11:16 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Create a policy rule to control if a rule belongs to the default SG https://review.opendev.org/c/openstack/neutron/+/896222 | 11:17 |
| ralonsoh | slaweq, ^^ I think this is what you were looking for | 11:17 |
| ralonsoh | no changes in the API but a new rule (and test overriding the default policies!! <-- cool) | 11:18 |
| slaweq | ralonsoh++ looks good for me, thx a lot | 12:10 |
| opendevreview | Justin Lamp proposed openstack/ovn-bgp-agent master: [EVPN] Add dynamic ASN lookup and fix route deletion https://review.opendev.org/c/openstack/ovn-bgp-agent/+/897329 | 12:37 |
| ralonsoh | ykarel, shouldn't we try https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897308 on top of master? I mean, without disabling l3_ha | 12:52 |
| ralonsoh | nevermind, this is not related to the problem with HA | 12:53 |
| ykarel | ralonsoh, yes right that is different | 12:54 |
| ykarel | ralonsoh, can you check https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/896188 too , it avoids those zuul warnings | 12:55 |
| ralonsoh | approved | 12:56 |
| ykarel | ralonsoh, thx | 12:56 |
| ykarel | ralonsoh, lajoskatona i think i got the root cause of that router not active issue | 12:56 |
| ralonsoh | no way! | 12:56 |
| ykarel | it's oslo.log==5.3.0 | 12:57 |
| ralonsoh | really?? | 12:57 |
| ykarel | specifically commit https://github.com/openstack/oslo.log/commit/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17 | 12:57 |
| ykarel | i did tested 5.2.0 in https://review.opendev.org/c/openstack/neutron/+/896504 and all 10 runs passed, before that it was failing consistantly | 12:58 |
| slaweq | ykarel wow, that's great news | 12:58 |
| ykarel | and locally also couldn't reproduce the issue with 5.2.0, have a script running since last night and no failures with that version | 12:58 |
| ralonsoh | but that was avoidable with https://review.opendev.org/c/openstack/oslo.log/+/864252 | 12:58 |
| ykarel | ralonsoh, yes but seems some conditions is missing and thus issue still happens | 12:59 |
| ralonsoh | I think we need to find where in our code are calling log.setup() | 13:00 |
| ralonsoh | and use fix_eventlet=False | 13:00 |
| ralonsoh | we found this issue 1 year ago, more or less | 13:00 |
| ralonsoh | and that prevented the version bump | 13:00 |
| ralonsoh | but eventually they did | 13:00 |
| ralonsoh | pfffff | 13:00 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM - WIP - Fix log.setup --> fix_eventlet=False https://review.opendev.org/c/openstack/neutron/+/897332 | 13:04 |
| ralonsoh | ykarel, ^ | 13:05 |
| ykarel | ++ i was doing same :) | 13:05 |
| ykarel | let's see how it goes | 13:06 |
| ralonsoh | haleyb, ^ | 13:09 |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron master: [OVN] Enhanced external port scheduling https://review.opendev.org/c/openstack/neutron/+/894767 | 13:09 |
| lajoskatona | ykarel: cool | 13:10 |
| lajoskatona | ralonsoh: 🤞 | 13:13 |
| ralonsoh | hehehe | 13:13 |
| haleyb | ralonsoh: it will be interesting if that's it, and thanks ykarel for finding that! | 13:28 |
| bcafarel | logging would definitely not have been my first suspect in HA issue :) | 13:32 |
| ralonsoh | the problem is the ip_monitor | 13:43 |
| ralonsoh | I found this issue 1 year ago when trying to bump the oslo.log version. But I recommended not to make this change of the eventlet enabled by default | 13:44 |
| ralonsoh | and my suggestion was not attended and the version was bumped months later | 13:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Replace "tenant_id" with "project_id" in IPAM engine https://review.opendev.org/c/openstack/neutron/+/877533 | 13:45 |
| jlibosva | haleyb: slaweq if you can have another look https://review.opendev.org/c/openstack/neutron/+/890339 pretty please :) You already had a look before. | 13:50 |
| opendevreview | Michel Nederlof proposed openstack/neutron master: [OVN] Update lsp host id when virtual parent changes https://review.opendev.org/c/openstack/neutron/+/896883 | 14:03 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add policy enforcer for "tags" service plugin https://review.opendev.org/c/openstack/neutron/+/896509 | 14:51 |
| ralonsoh | ykarel, lajoskatona haleyb it seems that https://review.opendev.org/c/openstack/neutron/+/897332 is working fine (https://zuul.opendev.org/t/openstack/status#897332). I'll wait until we have all the CI results and then I'll push an updated patch. Then we'll revert the n-t-p patch that sets l3_ha=False | 14:52 |
| lajoskatona | great job ykarel & ralonsoh :-) | 14:52 |
| ykarel | ralonsoh, so once results are confirmed, before reverting let's merge the fix in both master and stable/2023.2 | 14:53 |
| ralonsoh | right | 14:53 |
| ykarel | btw i noticed a failure(router not active) even on 2023.1 with oslo.log==5.0.2, checking it but seems it's some other issue | 14:54 |
| ykarel | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ea5/895516/3/check/neutron-tempest-plugin-openvswitch-iptables_hybrid-2023-1/ea59b52/testr_results.html | 14:55 |
| ralonsoh | I hope this is another issue | 14:57 |
| * ykarel afk | 14:58 | |
| opendevreview | Merged openstack/neutron-tempest-plugin master: Temporary turn off l3_ha in ovs/lb master/bobcat jobs https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/897233 | 15:06 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: nb: add 'nexthop' argument to 'lr_route_del' https://review.opendev.org/c/openstack/ovsdbapp/+/896645 | 15:09 |
| opendevreview | Anton Vazhnetsov proposed openstack/ovsdbapp master: nb: provide 'route_table' in lr-route-{add,del,list} https://review.opendev.org/c/openstack/ovsdbapp/+/896793 | 15:19 |
| opendevreview | Merged openstack/neutron-tempest-plugin master: Use RE2 compatible regex for branches https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/896188 | 15:26 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: Set segmentation ID outside the ranges assigned to project network https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/894503 | 15:29 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVS][QoS] Add QoS support for Trunk service, OVS driver https://review.opendev.org/c/openstack/neutron/+/839523 | 15:31 |
| haleyb | ralonsoh: so should we file a bug against oslo.log or we just live with fix_eventlet=False ? Seems there's still a bug | 15:42 |
| ralonsoh | that was implemented to fix another issue in cinder | 15:43 |
| ralonsoh | but, IMO (and as I commented in the olso channel) | 15:43 |
| ralonsoh | this change should have been conditional and not enforced by default | 15:43 |
| ralonsoh | this is why I pushed https://review.opendev.org/c/openstack/oslo.log/+/864252 | 15:43 |
| ralonsoh | we can enforce "fix_eventlet=False" | 15:44 |
| ralonsoh | but next time I will request not to merge these kind of changes changing the default behaviour | 15:44 |
| haleyb | ralonsoh: ack, thanks, and i see the revert was only done in 2023.1 which is why we don't see it there | 15:45 |
| MaxLamprecht[m] | ralonsoh: I added 2 new comments to https://bugs.launchpad.net/neutron/+bug/2037107 regarding the group_by clause. Would be nice if you can have a look at it :) | 15:51 |
| ralonsoh | I did 1 second ago | 15:51 |
| ralonsoh | https://bugs.launchpad.net/neutron/+bug/2037107/comments/10 | 15:52 |
| MaxLamprecht[m] | But the problem there is that it has no "low cardinality". If you check my paste in #8 (https://paste.opendev.org/show/bCCkZ01zquynsUm6lUSw/). There is a left outer join to networkrbacs again after the inner query. | 15:56 |
| MaxLamprecht[m] | And because it joins the network.id to networkrbacs.object_id in the outer query, the group_by(in the inner query) doesn´t help for the results(produced by the outer query). | 15:59 |
| opendevreview | Corey Bryant proposed openstack/networking-bagpipe master: Drop b1 from min versions in requirements.txt https://review.opendev.org/c/openstack/networking-bagpipe/+/897344 | 16:00 |
| opendevreview | Corey Bryant proposed openstack/networking-bagpipe master: Drop b1 from min versions in requirements.txt https://review.opendev.org/c/openstack/networking-bagpipe/+/876421 | 16:01 |
| ralonsoh | MaxLamprecht[m], please check the results. Execute the inner query with and without the group by clause | 16:02 |
| MaxLamprecht[m] | I did. Results are the same | 16:03 |
| ralonsoh | MaxLamprecht[m], maybe in your env, where you have multiple projects and one resource (network) | 16:05 |
| ralonsoh | but not when you have multiple projects and multiple resources (networks), and the corresponding rbacs | 16:06 |
| opendevreview | Fernando Royo proposed openstack/neutron master: Add constant to identify OVN LB HM ports https://review.opendev.org/c/openstack/neutron/+/897345 | 16:06 |
| ralonsoh | and I have a customer happy because of this fix | 16:06 |
| ralonsoh | that went from 90 seconds to less than 100ms in this query | 16:06 |
| MaxLamprecht[m] | Also like I wrote in #9, the Sqlalchemy docs say that we cannot do that.... | 16:07 |
| ralonsoh | so, the next issue is to fix the cardinality of the outer query | 16:07 |
| MaxLamprecht[m] | We have also multiple networks in our env ^^ | 16:07 |
| ralonsoh | this is why I changed the loading method | 16:08 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Change device_owner used for OVN LB HM port https://review.opendev.org/c/openstack/ovn-octavia-provider/+/897348 | 16:14 |
| MaxLamprecht[m] | I don´t get it. The goal from the group by is to lower/group the returned results from the networkrbacs table. But since we are joining it again in the outer query, the result is the same. Is the customer really using that fix? | 16:16 |
| ralonsoh | MaxLamprecht[m], again, in your case | 16:17 |
| ralonsoh | there are other configurations | 16:17 |
| ralonsoh | this is the script used by our internal QA team to check this bug | 16:19 |
| ralonsoh | https://paste.opendev.org/show/bsvcKiGnZ4hqzBw0BQ7b/ | 16:19 |
| ralonsoh | use with and without the n-lib patch | 16:19 |
| ralonsoh | "Is the customer really using that fix?" please, avoid this kind of comments | 16:20 |
| MaxLamprecht[m] | thx - I will try that tomorrow | 16:21 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Fix the ``log.setup`` method call with "fix_eventlet=False" https://review.opendev.org/c/openstack/neutron/+/897332 | 16:26 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Add policy enforcer for "tags" service plugin https://review.opendev.org/c/openstack/neutron/+/896509 | 16:35 |
| opendevreview | Miro Tomaska proposed openstack/neutron master: OVN Metadata handle process execeptions https://review.opendev.org/c/openstack/neutron/+/890986 | 18:47 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: functional: Enable test_fip_connection_for_address_scope test https://review.opendev.org/c/openstack/neutron/+/896728 | 18:58 |
| opendevreview | Merged openstack/neutron master: [OVN] Add the default condition check in ``PortBindingChassisEvent`` https://review.opendev.org/c/openstack/neutron/+/896926 | 21:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!