| *** vindict_ is now known as vindict | 01:08 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == [OVN] The external networks GW chassis must the same as the GW LRP https://review.opendev.org/c/openstack/neutron/+/962155 | 07:14 |
|---|---|---|
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2024.1: [stable] Pin rally to 4.1.0 https://review.opendev.org/c/openstack/neutron/+/962157 | 07:38 |
| tafkamax | Hi I have a question regarding networking in openstack. We have deployed a 2024.2 release cluster and we are seeing that with floating ip we are getting an issue where the ping responds even if the associated VM is in poweroff state. | 07:38 |
| tafkamax | Why is this happening? | 07:38 |
| ralonsoh | tafkamax, from the documentation | 07:40 |
| ralonsoh | In OVN (Open Virtual Network), the logical router is responsible for responding to a ping (ICMP echo request) directed to one of its IP addresses. This is a built-in feature of OVN's architecture designed to handle these requests efficiently at the logical level. | 07:40 |
| ralonsoh | When a virtual machine or container sends a ping to an OVN logical router's IP address, the packet doesn't need to be forwarded to a physical router or a dedicated network service VM. Instead, OVN's control plane proactively installs flows in the Open vSwitch (OVS) integration bridge on the hypervisor where the sending VM resides. These flows are specifically designed to recognize and respond to such ICMP requests | 07:40 |
| ralonsoh | locally. | 07:40 |
| tafkamax | Thanks | 07:46 |
| tafkamax | Could you please link me the docs reference? | 07:48 |
| tafkamax | I can't seem to find it when searching myself :S | 07:48 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron stable/2024.1: [stable] Pin rally=4.1.0, rally-openstack=3.0.0 https://review.opendev.org/c/openstack/neutron/+/962157 | 07:55 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [eventlet-removal] Don't use eventlet in the unit/functional tests https://review.opendev.org/c/openstack/neutron/+/952258 | 08:41 |
| ralonsoh | hello folks, any review is welcome: https://review.opendev.org/c/openstack/neutron-lib/+/961612 | 08:41 |
| zigo | Hi there! I've just validated Flamingo on my CI, and all worked, but I found out that Neutron is using WAAAYYY more memory than it used to. I found my 2 network nodes doing OOM-kill, being completely dead. On my CI, my network nodes used to run with 8 GB of RAM, I had to increase it (I've set 16 GB). | 08:43 |
| zigo | Is this something known by the team? | 08:43 |
| ralonsoh | zigo, with ML2/OVS, right? | 08:46 |
| zigo | Yeah. | 08:46 |
| ralonsoh | I saw this in our CI too | 08:47 |
| ralonsoh | but only in one patch (maybe because is triggering this job) | 08:47 |
| ralonsoh | this could be related to the eventlet removal | 08:47 |
| zigo | My thoughts as well. | 08:47 |
| ralonsoh | but for L3/OVS/DHCP agents, the eventlet removal was finished time ago | 08:47 |
| ralonsoh | so this should have been happening months ago | 08:48 |
| ralonsoh | zigo, I saw this issue here: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest-plugin-openvswitch&project=openstack/neutron | 08:49 |
| ralonsoh | but only in one patch | 08:50 |
| zigo | What I saw in my network node, was so many neutron-keepalived-state-change processes, but maybe that's a consequence, not a root cause. | 08:52 |
| ralonsoh | yeah, this is a recurrent problem with this process | 08:53 |
| ralonsoh | we need one process per HA router and each one consumes more than 100MB | 08:53 |
| ralonsoh | I started, 2 years ago (never finished), to move this code to a bash script | 08:53 |
| zigo | What does that process do exactly? | 08:53 |
| ralonsoh | nothing: check the keepalived status (main, backup) and write a file | 08:54 |
| ralonsoh | but because we import neutron, oslo, etc, the memory usage is massive | 08:54 |
| * zigo reads the code | 08:55 | |
| ralonsoh | zigo, https://review.opendev.org/c/openstack/neutron/+/836140 | 08:55 |
| ralonsoh | that goes from 100MB to 7MB | 08:55 |
| ralonsoh | So according to https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest-plugin-openvswitch&project=openstack%2Fneutron&change=952258&skip=0 | 08:56 |
| ralonsoh | ^^ just this job for this specific change | 08:56 |
| ralonsoh | that started on Sep 22 | 08:56 |
| ralonsoh | sorry Sep 23, the Sep 22 job passed fine | 08:57 |
| ralonsoh | I see nothing in requirements nor Neutron (relevant for this) | 08:58 |
| zigo | Oh, so keepalived_state_change is the thing that monitors active/standby thing for HA routers? | 09:00 |
| ralonsoh | zigo, yes, this is how Neutron knows the status of each HA instance | 09:01 |
| ralonsoh | I know this is trivial and it could be done with a bash script (^^ the upper link) | 09:01 |
| zigo | Yeah, indeed ! :P | 09:01 |
| ralonsoh | that could aleviate a lot the memory requirements for OVS envs with many HA routers (1/10 size) | 09:02 |
| ralonsoh | but, to be honest, now I can't spend time on this | 09:02 |
| ralonsoh | in any case, I'll add this to the PTG topics, for sure | 09:02 |
| ralonsoh | lajoskatona, hello! Do you happen to know if we have a PTG etherpad? | 09:02 |
| ralonsoh | ahh yes | 09:03 |
| ralonsoh | https://etherpad.opendev.org/p/oct2025-ptg-neutron | 09:03 |
| zigo | Just checked one of our network node, we have 673 instances of neutron-keepalived-state-change running, so that's 65 GB of RAM ! | 09:04 |
| zigo | With your rewrite, that'd be only 4... | 09:04 |
| ralonsoh | exactly (and 4GB just to monitor is still insane...) | 09:04 |
| ralonsoh | we can propose: | 09:04 |
| zigo | Your proposal is also: having a single daemon per L3 agent, monitoring *ALL* HA routers? | 09:07 |
| zigo | Sounds good to me. :P | 09:07 |
| ralonsoh | I made these 2 proposals: migrate to bash or, with the same code, monitor several instances | 09:08 |
| ralonsoh | the double shot is to migrate to bash (1 monitor per HA router) and then use this single bash script for multiple instances | 09:08 |
| zigo | Could it be both? ie: a bash script that would monitor several HA routers? | 09:08 |
| ralonsoh | yes, but step by step | 09:08 |
| zigo | :P | 09:08 |
| zigo | Would be awesome. | 09:09 |
| ralonsoh | the migration could fix many problems initially | 09:09 |
| zigo | Thanks for all of these details. | 09:09 |
| zigo | ralonsoh: Did you compare dash to bash? Writting in pure sh isn't hard, I could help, and it should also make the footprint a way nicer. | 09:11 |
| ralonsoh | zigo, I did: https://review.opendev.org/c/openstack/neutron/+/836140 | 09:12 |
| ralonsoh | ^^ that was almost finished, but I didn't have more time | 09:12 |
| zigo | That's full of bashism ! :P | 09:12 |
| zigo | "function" must be removed, double-test (ie: [[ ) must be removed too. | 09:13 |
| zigo | Plus there's no arrays in dash. | 09:13 |
| ralonsoh | well, that is supposed to be a bash script | 09:15 |
| zigo | Dash would be 1/3rd of the memory footprint. | 09:16 |
| ralonsoh | I think the migration from python to bash is more than enough | 09:16 |
| zigo | Probably ! :) | 09:16 |
| ralonsoh | we can continue with this script I proposed, I think | 09:16 |
| *** dmellado7 is now known as dmellado | 09:23 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Remove the check for external ports support https://review.opendev.org/c/openstack/neutron/+/961810 | 10:32 |
| opendevreview | Martin Kalcok proposed openstack/neutron master: ovn_db_sync: Improve coexistence support. https://review.opendev.org/c/openstack/neutron/+/958679 | 11:04 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Optimize _get_current_operating_statuses to reduce octavia_lib calls https://review.opendev.org/c/openstack/ovn-octavia-provider/+/962179 | 11:49 |
| opendevreview | Merged openstack/neutron stable/2024.1: [stable] Pin rally=4.1.0, rally-openstack=3.0.0 https://review.opendev.org/c/openstack/neutron/+/962157 | 11:56 |
| *** ykarel_ is now known as ykarel | 12:35 | |
| opendevreview | yatin proposed openstack/neutron master: [DNM] check fips-mode-setup https://review.opendev.org/c/openstack/neutron/+/961206 | 13:11 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Optimize _get_current_operating_statuses to reduce octavia_lib calls https://review.opendev.org/c/openstack/ovn-octavia-provider/+/962179 | 13:43 |
| opendevreview | Eduardo Olivares proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 13:47 |
| opendevreview | Eduardo Olivares proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 13:51 |
| opendevreview | Eduardo Olivares proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 13:53 |
| opendevreview | Eduardo Olivares proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 13:54 |
| *** masayukig_ is now known as masayukig | 14:20 | |
| *** zseguin_ is now known as zseguin | 14:26 | |
| *** mnasiadka_ is now known as mnasiadka | 14:26 | |
| *** TheJulia_ is now known as TheJulia | 14:26 | |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Always exclude the local chassis creating ``HA_Chassis_Group`` https://review.opendev.org/c/openstack/neutron/+/962191 | 14:33 |
| *** masayukig_ is now known as masayukig | 14:39 | |
| *** johnsom_ is now known as johnsom | 14:39 | |
| *** mnaser_ is now known as mnaser | 14:39 | |
| *** haleyb_ is now known as haleyb | 14:39 | |
| *** haleyb_ is now known as haleyb | 14:42 | |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [S-RBAC] Fix tests for port RBAC policies https://review.opendev.org/c/openstack/neutron/+/962192 | 14:50 |
| ralonsoh | haleyb, hi! no rush: https://review.opendev.org/c/openstack/neutron-lib/+/961612. Just if you have a couple of mins | 14:54 |
| *** haleyb_ is now known as haleyb | 15:05 | |
| *** cardoe_ is now known as cardoe | 15:32 | |
| *** gthiemon1e is now known as gthiemonge | 16:01 | |
| opendevreview | Eduardo Olivares proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 16:04 |
| opendevreview | Eduardo Olivares proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 16:06 |
| *** iurygregory_ is now known as iurygregory | 23:41 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!