Tuesday, 2024-12-17

opendevreviewArnaud Morin proposed openstack/neutron master: Fix: handle initial router state for any state  https://review.opendev.org/c/openstack/neutron/+/92940600:48
opendevreviewMerged openstack/neutron-tempest-plugin master: Deprecate options to enable/disable plugin tests  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93604500:50
*** gryf is now known as Guest334005:36
*** Guest3340 is now known as gryf05:49
opendevreviewMerged openstack/neutron master: [functional][CI] split tests in more runs  https://review.opendev.org/c/openstack/neutron/+/93775906:35
opendevreviewDong Ma proposed openstack/ovn-bgp-agent master: WIP: Add the support of create kubernetes resource  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/93745706:52
opendevreviewDong Ma proposed openstack/ovn-bgp-agent master: WIP: Add the support of create kubernetes resource  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/93745707:08
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: Turn off wsgi in linuxbridge stable branch jobs  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93784207:31
ralonsohykarel, hello! I think we should also add 2024.2 in https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93784208:12
ykarelralonsoh, i saw 2024.2 was passing so didn't touched that08:12
ykarelbut can include that too08:12
ralonsohthen perfect08:12
ralonsohno no08:12
ralonsohif 2024.2 is working, that's fine08:12
ralonsoh+208:12
ykarelok thx08:13
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent  https://review.opendev.org/c/openstack/neutron/+/93776508:15
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread  https://review.opendev.org/c/openstack/neutron/+/93784308:15
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN] Set always the GW LRP "gateway_mtu" option  https://review.opendev.org/c/openstack/neutron/+/93702608:58
opendevreviewChris Buggy proposed openstack/ovn-octavia-provider master: Add Tobiko to gate jobs  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/93634109:05
opendevreviewRodolfo Alonso proposed openstack/neutron stable/2024.2: [OVN] Update the LRP gateway_mtu when the external network changes  https://review.opendev.org/c/openstack/neutron/+/93703610:19
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Member batch actions to increase performance  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/93676510:23
opendevreviewRodolfo Alonso proposed openstack/neutron master: Read the environment variable EVENTLET_MONKEYPATCH  https://review.opendev.org/c/openstack/neutron/+/93753510:25
ralonsohfolks, I need some reviews on https://review.opendev.org/c/openstack/neutron/+/93683811:13
ralonsohthis improves the OVN hash ring handling in ML2/OVN with WSGI11:14
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Member batch actions to increase performance  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/93676511:16
yosefHi, something interesting, (I am on neutron yoga), it seems setting inactivity probe interval to 60 seconds does not change it on some sections of code and Neutron connection to SB db closes by default 5 seconds inactivity probe, is this fixed on master?11:22
ykarelralonsoh, when you get chance can you check https://review.opendev.org/c/openstack/neutron/+/93775811:23
ralonsohsure11:30
ralonsohyosef, yoga is in unmaintained mode, did you try with a newer version?11:31
ralonsohykarel, but https://review.opendev.org/c/openstack/neutron/+/811751 was actually solving a real issue, while keeping https://bugs.launchpad.net/neutron/+bug/1837635 functionality11:32
yosefrolonsoh: cluster is highly under load, can't upgrade, and hard to replicate on stage. i have used `ovsdb_probe_interval` and it is defaulted to 60000, but it seems it is not applied on many connections to ovn southbound11:34
ralonsohykarel, when the tests are failing? In the initial state?11:36
ralonsohyosef, where it is missing?11:36
ralonsohykarel, so this is failing, for example, when one of the processes is killed11:40
ralonsohright?11:40
ykarelralonsoh, no failing on initial transition as multiple agent get's router active state11:41
ralonsohso when it is failing?11:41
ykareli added details in my last two comments in https://bugs.launchpad.net/neutron/+bug/208360911:42
yosefralonsoh: did not tried to find it on the neutron source code, but I had many `no response to inactivity probe after 5 seconds` from neutron workers and it made me to switch neutron to southbound relays, I guess this config is not being applied in the code! I like to read neutron codes, but it is very confusing, cant find where ovsdb client is created for workers11:42
ralonsohykarel, but what I see from https://bugs.launchpad.net/neutron/+bug/2083609/comments/20 is that both routers get MASTER at the same time11:46
ralonsohbut it wasn't the first state11:46
ykarelralonsoh, yes right11:46
ralonsohAm I missing something there?11:46
ykarelralonsoh, correct, so both got master state and due to https://review.opendev.org/c/openstack/neutron/+/811751 delay was skipped and resulted into 'active' ha state on both agents as it transitioned to 'backup' before11:47
ykareldelay was skipped for 1st primary with that patch11:47
ykarelwhile delay should be there for each primary as multiple agents can get primary state at same time11:48
ykarel^ what we seeing in fullstack tests11:48
ralonsohnow I get it...11:49
ykarelbut now looking into https://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha_router.py#L188-L189, 'backup' will always be the initial state11:49
ralonsohwith the eventlet.sleep(self.conf.ha_vrrp_advert_int) we can wait for a vrrp update11:49
ykarelthen do we really need that initial_state_change method so the revert11:49
ralonsohand actually define what instance is primary11:49
ralonsoh+211:49
ralonsohslaweq, ^^ please check this patch https://review.opendev.org/c/openstack/neutron/+/93775811:49
ralonsohmaybe we can have again FT problems, but that could be fixed in a better way11:50
ralonsohbut indeed this patch was introducing issues to the delay state implementation11:51
ykarelralonsoh, also there was another similar fix https://review.opendev.org/c/openstack/neutron/+/929406 open for some time, but based on above reasoning backup will always be initial state that method doesn't look necessary11:51
ykarelwill do comment there11:52
ykarelamorin, fyi ^11:54
ralonsohyosef, when you are saying that Neutron doesn't apply the inactivity probe, are you referring to the NB/SB connection registers?11:56
slaweqralonsoh: approved11:57
ralonsohthanks!11:57
ralonsohykarel, I think we should backport it11:57
ralonsohand maybe the fullstack tests too, if possible11:57
ykarelralonsoh, yes will do backport that patch once master one is merged11:58
ykarelfor fullstack yes can also do backport those, i recall there were some follow up fixes for those, so have to be checked11:59
ykarelslaweq, please also check https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93784212:00
yosef_ralonsoh: Yes, Neutron doesn't apply inactivity probe on SB connection, on Yoga, which section of code is responsible for that?12:08
ralonsohyosef_, no, Neutron is not responsible of this. This should be done by the installer12:09
yosef_omm what is installer?12:09
ralonsohwhat ever deploys openstack and the backends in the system: devstack, kolla, tripleo12:11
yosef_Ok I am the operator, configuration on neutron, does not affect its inactivity probe, `ovsdb_probe_interval` is only option and is not taking effect12:14
opendevreviewRodolfo Alonso proposed openstack/neutron stable/2024.2: [OVN] Update the LRP gateway_mtu when the external network changes  https://review.opendev.org/c/openstack/neutron/+/93703612:19
opendevreviewRodolfo Alonso proposed openstack/neutron master: [WIP] Metadata proxy server without eventlet  https://review.opendev.org/c/openstack/neutron/+/93754512:22
*** ykarel_ is now known as ykarel12:28
ykarelyosef_, as ralonsoh mentioned neutron doesn't set that since https://review.opendev.org/q/Ia1dc8072ecec1c019dd02039dadd78d544dbd84312:29
ralonsohykarel, correct12:30
ykarelyou can check ovn-sbctl list connection . and set with ovn-sbctl set connection . inactivity_probe=6000012:30
ykareland if you using some installer, installer may have some option to set for you12:31
ykarellike tripleo have OVNOvsdbProbeInterval12:31
yosef_Thank you for your explanation, but inactivity probe in ovsdb is two way, meaning if it is set on the ovsdb southbound and not set from client to southbound it has no effect and will be droped on 5 seconds, i have that configuration you mentioned on sb12:45
ykarelwhat exact error you seeing and where?12:47
amorinhey13:08
amorinmaybe easier to answer you here ykarel. The initial state is set by calling initial_state_change()13:11
ralonsohyosef_, the IDL client applies this configuration when calling Ml2OvnIdlBase13:11
ralonsohyou can see the code there13:11
amorinwith my patch, we call this method no matter the current state, which is what we want I believe13:11
amorin(talking about this: https://review.opendev.org/c/openstack/neutron/+/929406)13:12
ykarelamorin, initial_state_change was just checking if the request was initial request or not and based on it add a delay on 2nd primary transition before your or reverted patch13:39
ykarelbackup(no sleep) -> primary(no sleep) -> backup(no sleep) -> primary(sleep) -> backup(no sleep)13:40
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2)  https://review.opendev.org/c/openstack/neutron/+/93771513:40
amorinykarel, ack, I dont remember the details, I am affraid about this bug: https://bugs.launchpad.net/neutron/+bug/208323713:55
amorinI hope the revert is fixing it13:55
opendevreviewRodolfo Alonso proposed openstack/neutron master: [WIP] Metadata proxy server without eventlet  https://review.opendev.org/c/openstack/neutron/+/93754513:58
haleyb#startmeeting networking14:00
opendevmeetMeeting started Tue Dec 17 14:00:49 2024 UTC and is due to finish in 60 minutes.  The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'networking'14:00
haleybPing list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slaweq, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki, haleyb, ralonsoh14:00
ralonsohhello14:00
obondarevhi14:00
s3rj1khi all14:01
cbuggyHello o/14:01
rubasovo/14:02
bcafarelo/14:02
haleybok we can get started14:02
haleyb#announcements14:02
haleyb#link https://releases.openstack.org/epoxy/schedule.html14:02
haleybWe are currently in week R-1514:02
haleybEpoxy-2 milestone R-12 (week of Jan 6th)14:02
haleybReminder: If you have a topic for the drivers meeting on Friday, please add it to the wiki @ https://wiki.openstack.org/wiki/Meetings/NeutronDrivers14:03
haleybBut there is not another scheduled drivers meeting for a while14:04
haleybI hope everyone saw my email on meetings for rest of year (i know ykarel did :)14:05
haleybNext CI meeting is Jan 6th14:05
haleybNext Neutron meeting is Jan 7th (mlavalle chair)14:05
haleybNext Drivers meeting Jan 10th14:06
haleybI did have a note in that meeting about my own availability14:06
haleybI will be offline from December 19th to January 8th, checking-in occasionally to make sure there are no fires14:07
haleybthat is my plan ^^ but i probably will check email every other day until Jan 1st14:07
haleybThen you will only find me if you are sitting at the beach next to me sharing a cocktail14:08
slaweqhi, sorry for being late14:09
haleybI hope everyone will take some time off to enjoy their family/friends too, Neutron will still be here :)14:09
* haleyb feels like a Dad today14:10
ihrachys:D14:10
haleyblast announcement i have is my weekly reminder14:11
haleybLet's continue to use the priorities dashboard for patches in the "ready to merge" state (weekly reminder)14:11
ihrachysthank you Grandpa for the holiday wisdom14:11
slaweq:D14:12
haleybhey! no grandkids yet, don't rush things14:12
haleyband get off my lawn!14:13
haleybagain, bad Dad jokes is all i have14:13
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ml2/ovs: improve log message for ports without VLAN tags/net-id during OVS restore  https://review.opendev.org/c/openstack/neutron/+/92990814:14
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ml2/ovs: change log level from DEBUG to WARNING for port deletion during binding  https://review.opendev.org/c/openstack/neutron/+/92990914:14
haleybany other announcements?14:14
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ml2/ovs: change log level from DEBUG to INFO for port deletion during binding  https://review.opendev.org/c/openstack/neutron/+/92990914:14
haleyblet's move on14:14
haleyb#topic bugs14:15
haleybihrachys was the deputy last week14:15
haleyb#link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/QNUVMQ4WRKHVAQIAKIRLQZMX5FBMMT42/14:15
haleybthere were a couple related to gate issues that have owners and fixes14:16
haleybthis revert should help with the l3-ha issues14:16
haleyb#link https://review.opendev.org/c/openstack/neutron/+/93775814:17
opendevreviewMerged openstack/neutron master: Revert "[HA] Do not add initial state change delay in HA router"  https://review.opendev.org/c/openstack/neutron/+/93775814:17
opendevreviewMerged openstack/neutron-tempest-plugin master: Turn off wsgi in linuxbridge stable branch jobs  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93784214:17
haleybbut as amorin asked there, will that just re-introduce the original bug?14:17
ralonsohthe bug this patch was solving was in the FTs only14:17
ihrachys"FTs"?14:18
ralonsohfunctional tests14:18
ralonsohhttps://bugs.launchpad.net/neutron/+bug/194551214:18
ralonsohso I think we are ok with this revert14:18
ralonsohif we find again the FTs errors, we can address them in a better way14:18
amorinwhat I saw in our downstream openstack was the following: https://bugs.launchpad.net/neutron/+bug/208323714:19
amorinI was wondering if the revert will fix it?14:19
ralonsohyes it should14:19
ykarelamorin, yes the revert should help in the scenario mentioned in the bug14:19
amorinperfect, thanks :)14:19
haleybok, so https://review.opendev.org/c/openstack/neutron/+/929406 shouldn't be required any more14:20
amorinI believe so, I can abandone and mark this as related to the revert14:20
amorinsame on the launchpad bug, agree?14:20
ralonsohyes14:20
haleybgreat14:21
haleybnext bug14:21
opendevreviewStanislav Dmitriev proposed openstack/neutron master: HA VRRP health check parameters  https://review.opendev.org/c/openstack/neutron/+/93271614:21
haleyb#link https://bugs.launchpad.net/neutron/+bug/209171114:21
haleybSG list failure on stable branches14:22
ralonsohI saw it once only14:22
ralonsohand in postgre CI14:22
ralonsohif we don't see it again, we can close it14:22
haleybralonsoh: ack14:23
haleybnext bug14:24
haleyb#link https://bugs.launchpad.net/neutron/+bug/209121114:24
haleybnetwork leakage for flat network14:24
haleybas ihrachys noted, it looks like a legit bug, and there are some good notes there14:25
haleybi will watch it but do not have time to pick it up14:26
ihrachysshould mac-aging-time update with "other" traffic as the reporter suggests?14:26
ralonsohI don't know if Saeed is only using the explicitly_egress_direct=True option14:27
ralonsohthis is preventing from using the MAC table14:28
ralonsohbecause the packet is not using the normal action14:28
ralonsohthat was a reported issue14:28
haleybralonsoh: that is set in his notes14:28
ralonsohI know14:28
ralonsohbut I don't know if he tested with and without this flag14:28
ralonsohthere: https://bugs.launchpad.net/neutron/+bug/1884708 (also in the notes)14:29
haleybralonsoh: when you say it was a reported issue - with neutron or OVS? ah14:29
ralonsohit is supposed to be fixed but maybe this patch was not considering FLAT networks14:30
ralonsohI'll comment that in the bug today14:31
haleybthanks ralonsoh 14:31
haleybthat was all for the bugs14:32
haleybi did have one question regarding the l3-ha revert (which just merged), guess that needs backports14:33
haleyb#link https://review.opendev.org/c/openstack/neutron/+/93775814:33
ykarelyes pushing those14:33
ykarelfixing conflicts :)14:33
haleybykarel: thanks14:34
haleybare there any other gate issues we should focus on?14:34
ralonsohthe issues with wsgi migration (and the related fixes)14:34
ykarel++ ^ the other main ones14:35
ralonsohthat usually affect loaded envs (with more API workers)14:35
ralonsohstill working on why the Neutron API misses the events...14:35
haleybok, just want to cut-down the number of rechecks if possible14:36
haleybok, if no other bugs we can move on14:37
haleyb#topic community goals14:37
haleybi see there was a little movement on neutronclient deprecation ?14:38
opendevreviewyatin proposed openstack/neutron stable/2024.2: Revert "[HA] Do not add initial state change delay in HA router"  https://review.opendev.org/c/openstack/neutron/+/93787814:38
ralonsohnot really14:38
haleyb#link https://review.opendev.org/q/topic:%22bug/1999774%2214:38
haleybwell at least one +2 on the routers one14:38
opendevreviewyatin proposed openstack/neutron stable/2024.1: Revert "[HA] Do not add initial state change delay in HA router"  https://review.opendev.org/c/openstack/neutron/+/93787914:39
haleybi don't see lajos here today to discuss further14:39
opendevreviewyatin proposed openstack/neutron stable/2023.2: Revert "[HA] Do not add initial state change delay in HA router"  https://review.opendev.org/c/openstack/neutron/+/93788014:39
haleybother community goal is eventlet14:39
haleyb#link https://bugs.launchpad.net/neutron/+bugs?field.tag=eventlet-deprecation14:39
ralonsohI have this14:40
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/93754514:40
ralonsohI have a working metadata server (for OVN metadata agent and OVN agent) that works without eventlet14:40
ralonsohthe issue (and I'll send a mail for this)14:40
ralonsohthis implementation only adds the threading option, that means the embbeded server that runs in the same process14:41
ralonsohso the option metadata_workers is not not valid14:41
ralonsohI implemented this option only (for now) because it was easier and not dependant on, for example, oslo.services14:42
ralonsohto implement a multiprocess metadata server we should maybe use a wsgi server or use oslo.services worker to spawn several processes14:42
ralonsohso this will be a limitation, for sure14:42
ralonsohthis implementation uses the same strategy: haproxy with unix socket14:43
ralonsohand the metadata server listening to this socket14:43
ralonsohI'm now trying to export this to the L3 agent metadata14:44
ralonsohthat's all14:44
ralonsoh(btw, we don't have any CI testing metadata in DHCP)14:44
haleybso when oslo.service work progresses will there be something we can consume?14:44
ralonsohwe'll need to implement something (I don't know what) to replace the multiprocess metadata14:45
ralonsohnow is using eventlet.wsgi.server14:45
ralonsohI replaced all this logic14:45
ralonsohand most probably we'll be able to use the current logic to spawn processes using the new server14:45
ralonsohbut still unknown if that will work14:46
haleybyes, it's a large patch, thanks for all the work on it14:46
haleybi did see one other change to use threading in the ovs agent i think14:46
haleyb#link https://review.opendev.org/c/openstack/neutron/+/93776514:47
haleybi have not reviewed yet just watching14:47
haleybi need to prepare for another meeting so will move on14:48
haleyb#topic on-demand14:48
haleybany other items to discuss?14:48
haleyboh, completely forgot about bug deputy...14:48
ralonsohI am this week14:49
haleybralonsoh is covering this week, jlibosva next week, then obondarev 14:49
haleybit should be very light with holidays i hope14:49
haleyband if some have to wait until january it will be Ok14:49
haleybok, any other topics?14:50
haleybhave a great week and enjoy the holidays, will see you in January14:51
haleyb#endmeeting14:51
opendevmeetMeeting ended Tue Dec 17 14:51:05 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:51
opendevmeetMinutes:        https://meetings.opendev.org/meetings/networking/2024/networking.2024-12-17-14.00.html14:51
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/networking/2024/networking.2024-12-17-14.00.txt14:51
opendevmeetLog:            https://meetings.opendev.org/meetings/networking/2024/networking.2024-12-17-14.00.log.html14:51
ralonsohyou too, bye14:51
ykarelo/14:51
slaweqo/14:51
s3rj1kthanks, all have a good holiday14:51
opendevreviewSlawek Kaplonski proposed openstack/neutron master: Add limit of tags for every resource  https://review.opendev.org/c/openstack/neutron/+/93788715:15
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN] Set always the GW LRP "gateway_mtu" option  https://review.opendev.org/c/openstack/neutron/+/93702615:22
opendevreviewMerged openstack/neutron master: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests (2)  https://review.opendev.org/c/openstack/neutron/+/93771516:09
yosefralonsoh: I have read all the ovn ml2 mech driver codes on neutron, and as you said, inactivity probe is really set on all connections to ovsdb, my problem was something else, i have added an extra remote on southbound which overrided the connections table remote! thanks for your guide19:20
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: common: fix wait_until_true to support native thread  https://review.opendev.org/c/openstack/neutron/+/93784319:58
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: ovs: remove the usage of eventlet in the OVS agent  https://review.opendev.org/c/openstack/neutron/+/93776519:58
opendevreviewSahid Orentino Ferdjaoui proposed openstack/neutron master: common: add is_monked_patched as helper to eventlet_utils  https://review.opendev.org/c/openstack/neutron/+/93790819:58
-opendevstatus- NOTICE: Gerrit will be restarted to pick up a small configuration update. You may notice a short Gerrit outage.21:01
mnaserDoes anyone know if there's a way to use OVN alongside a DHCP relay for multi-segment networks?  I can't find any reference on it at all21:22
mnaserI know I can switch to neutron-dhcp-agent to get this feature, but it would be nice to avoid it..21:22
-opendevstatus- NOTICE: You may have noticed the Gerrit restart was a bit bumpy. We have identified an issue with Gerrit caches that we'd like to address which we think will make this better. This requires one more restart22:13

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!