Friday, 2021-07-09

opendevreviewMerged openstack/ovn-octavia-provider master: setup.cfg: Replace dashes with underscores  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/78867500:36
opendevreviewMerged openstack/ovn-octavia-provider master: Add a Kuryr Kubernetes co-gating job  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/76022501:10
opendevreviewMerged openstack/ovn-octavia-provider master: Change minversion of tox to 3.18.0  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/79490201:40
opendevreviewLajos Katona proposed openstack/neutron master: WIP: Privsep with timeout for some privsep call  https://review.opendev.org/c/openstack/neutron/+/79499408:12
*** mgoddard- is now known as mgoddard08:40
lajoskatonaralonsoh: Hi, could You please check my pyroute2 related change for bagpipe: https://review.opendev.org/c/openstack/networking-bagpipe/+/800062 ?08:54
ralonsohlajoskatona, sure08:54
lajoskatonaralonsoh: I am not sure if it is really needed, but pep8/pylint started to fail and I ralized that IPDB is deprecated from 1st July, and things like that08:55
opendevreviewzhangyanxian proposed openstack/neutron master: [OVN] Update the local.conf for devstack  https://review.opendev.org/c/openstack/neutron/+/80017610:01
opendevreviewSzymon Wróblewski proposed openstack/neutron master: Update AFTER callbacks of L3 DB FloatingIP  https://review.opendev.org/c/openstack/neutron/+/79800910:34
*** luis5tb is now known as ltomasbo10:36
opendevreviewSzymon Wróblewski proposed openstack/neutron master: Update AFTER callbacks of L3 DB FloatingIP  https://review.opendev.org/c/openstack/neutron/+/79800910:48
opendevreviewLajos Katona proposed openstack/networking-bagpipe master: Follow pyroute2 changes  https://review.opendev.org/c/openstack/networking-bagpipe/+/80006211:30
opendevreviewLajos Katona proposed openstack/neutron master: WIP: Privsep with timeout for some privsep call  https://review.opendev.org/c/openstack/neutron/+/79499411:46
opendevreviewMerged openstack/neutron master: Use IP_VERSION_{4,6} constants in ovn_client module  https://review.opendev.org/c/openstack/neutron/+/79965012:42
simondodsleyHi - looking for help with an issue on devstack where it appears the OVN routing is not getting set correctly and instances cannot ping the external world. They can successfully ping the compute host, but nothing beyond that, including the compute hosts gateway. This is causing major issues for our 3rd party Zuul CI for manila.13:12
simondodsleygouthamr: ^13:13
*** liuyulong__ is now known as liuyulong_13:40
mlavalle#startmeeting neutron_drivers14:00
opendevmeetMeeting started Fri Jul  9 14:00:12 2021 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'neutron_drivers'14:00
ralonsohhi14:00
fnordahlo/14:00
mlavalleGood morning / afternoon / evening14:00
haleybhi14:00
amotokihi14:00
rubasovo/14:01
lajoskatonaHi14:01
manubhi14:02
dmitriiso/14:02
obondarevhi14:03
mlavallelet's see if yamamoto or njohnston show up. We need one of them to discuss the first RFE ion out agenda. Let's geve them a few minutes14:03
mlavallehttps://review.opendev.org/admin/groups/5b063c96511f090638652067cf0939da1cb6efa7,members14:04
ralonsohnate is on PTO14:04
mlavallelet's wait a few minutes. If we don't get any of them, we can discuss the other two RFEs in our agenda since we have quorum for those14:05
mlavallefnordahl, dmitriis: are you here to discuss a RFE? I don't want to have you waiting until the end14:08
dmitriisyes14:08
fnordahlyes14:08
mlavalleok, I see fnordahl filed https://bugs.launchpad.net/neutron/+bug/193215414:09
mlavalleso let's start there14:09
mlavalle#topic RFEs14:09
mlavallehttps://bugs.launchpad.net/neutron/+bug/1932154 is up for discussion now14:10
ralonsohI have one question about the spec (about the architecture)14:11
dmitriissure14:11
ralonsohin the schema, the "SmartNIC DPU Board" is hosting the OVN processes (controller, vswtichd, etc)14:11
ralonsohbut any Neutron process?14:11
opendevreviewLajos Katona proposed openstack/networking-bagpipe master: Follow pyroute2 changes  https://review.opendev.org/c/openstack/networking-bagpipe/+/80006214:12
dmitriisralonsoh: We aim to avoid it in the OVN case. At least the goal is to not introduce any Neutron agents14:12
dmitriisat the moment there is a discussion upstream around representor plugging at the SmartNIC DPU side14:12
ralonsohok so is running the backend, not the orchestrator processes14:12
ralonsohperfect then14:12
dmitriisyes, extra comms between Neutron and every SmartNIC DPU wouldn't be good14:13
ralonsohso far, I'm ok with the RFE (and the spec although I need to review it again)14:13
dmitriiswe are actively discussing the OVN bits upstream since a lot depends on them14:14
amotokiif it is only related to the backend, why do you need to use "binding:profile" (i.e. API visible field)? Can't you pass information from neutron (driver) to the backend?14:14
amotokiI understand you need to pass information on host to the backend (ovn).14:15
ralonsoh(I think this is for Nova)14:15
fnordahlas I understand it this is part of the current Nova -> Neutron communication which we continue to use here14:15
dmitriisthe board serial number is needed to get an OVN chassis hostname14:16
dmitriisin case of port deletion that info needs to be present in the port as well14:16
dmitriisotherwise we don't know which SmartNIC DPU host should handle representor unplugging14:16
dmitriis(if that makes sense)14:18
fnordahlThe OVN mechanism driver would most likely need it throughout the lifetime of the port, to know what requested-chassis to put on the LSP14:19
amotokiI see. no neutron agent like sriov agent, so you need to pass host information to OVN via the  ovn mech driver running in the API server.14:19
fnordahlyes14:19
mlavalleand that is why you have a converstion upstream with ovn, to be able to coordinate that, correct?14:20
dmitriisyes, we need an entity that would do the job similar to os-vif14:21
mlavallehow is that conversation going?14:21
fnordahlOur conversation with OVN is to get code to look up representor ports running alongside / as part of the ovn-controller and extending the CMS API to specify how to find it.14:21
mlavalledo you think you will get the necessary support implemented?14:21
fnordahlOur target is to get it into OVN 21.09, it has been a conversation since May, and recently we have made progress with a few compromises from both sides suggested. We don't have a +2/-2 as of now, but I feel confident we will be able to move forward.14:22
* mlavalle shudders at the thought of having to get approvals from ovn upstream, nova and worst of all, neutron ;-)14:23
fnordahladd libvirt to the mix14:23
fnordahl;)14:23
mlavalleLOL14:23
dmitriismlavalle: yes, that's a tricky feature to coordinate14:24
dmitriisthe libvirt part is just about retrieving PCIe VPD and exposing it14:24
mlavalleI'm fine with this RFE. I propose that we continue the detailed conversation in the spec14:24
ralonsohright14:24
dmitriiswe had a conversation about it already in the ML and it was approved14:24
amotokiwhich ML?14:25
fnordahl^ I think dmitriis is talking about libvirt?14:25
dmitriishttps://listman.redhat.com/archives/libvir-list/2021-May/msg00873.html14:25
dmitriislet me find a reply as well14:25
amotokithanks14:25
dmitriishttps://listman.redhat.com/archives/libvir-list/2021-June/msg00037.html14:26
amotokithere are several components involved. where can we get the whole picture on this? I think it is an important thing for people to understand how it works.14:26
dmitriisamotoki: the Nova spec contains a lot of information https://review.opendev.org/c/openstack/nova-specs/+/787458. It originally contained more Neutron and OVN pieces but we removed them during the review to keep it Nova-related14:27
mlavalleamotoki: I agree with you. Should we strive to get that whole picture in the spec phase?14:28
fnordahlThe nova spec still has schematics (ascii) of the whole thing if I don't misremember dmitriis?14:28
fnordahlhttps://review.opendev.org/c/openstack/nova-specs/+/787458/9/specs/xena/approved/integration-with-off-path-network-backends.rst line 53214:28
dmitriisfnordahl: yes there's a bit more information in it regarding certain pieces14:29
fnordahlamotoki: mlavalle: would that schema help?14:29
mlavalleI might ask for more in the spec, but at first glance it looks like a good start to me14:31
amotokiI cannot read through it right now, but it is okay as it looks like it covers the main picture. if we find something missing, we can cover it in the neutron spec (or add smth to the nova spec if needed)14:31
amotokiwe can look thru the nova spec when reviewing the neutron spec.14:31
fnordahlack14:31
dmitriismlavalle, amotoki: ack14:32
amotokiI am feeling okay with this rfe.14:32
mlavalleyeah, we will have to read also at least the nova spec14:33
mlavallehaleyb: what are your thoughts?14:34
haleybi'm +1 on it, just trying to read through the spec too14:34
haleybcomplicated with all the dependencies14:34
mlavalleok, I think we have the votes necessary yo approve this RFE today. Thanks fnordahl and dmitriis for this proposal. We'll see you in gerrit in the spec conversation14:35
amotokifnordahl, dmitriis: the subject of the rfe says Off-path SmartNIC Port Binding, but can we add "with OVN"?14:35
dmitriissure14:36
dmitriisdone14:36
fnordahlthank you all for taking the time to review and discuss it with us14:36
amotokidmitriis: thanks. it highlights the scope more :)14:36
dmitriisYes, thanks a lot for considering it. The input is much appreciated as there are many pieces to this effort.14:37
mlavallegood luck herding all these cats!14:37
dmitriis👍14:38
* mlavalle shudders again14:38
fnordahl:) I might just add professional cat herder to my CV after this14:38
mlavalleok, next up for discussion is https://bugs.launchpad.net/neutron/+bug/193322214:38
ralonsohI have a couple of question (that I'll add to the LP bug)14:40
ralonsoh1) shouldn't be called metadata aggregation? this is what he is proposing14:40
ralonsoh2) what is the resource (memory) saving we gain?14:40
ralonsohI saw that an haproxy is using around 2MB14:40
ralonsohwe can have tens of then in one host without any problem14:41
ralonsohthat's all from my side14:41
mlavallehe also raises reliability concerns in his rfe14:42
obondarevRPC chattiness is another concern as I see14:43
obondarevif not the main14:43
ralonsohbecause if I'm not wrong, what he is proposing is NOT having a single metadata agent in one single place14:44
ralonsohbut one agent with one haproxy per host, right?14:45
mlavallecorrect.... there will be an ovs bridge in each compute with flows properly set up14:45
ralonsoh(from c#3)14:45
ralonsohso I don't see too much benefit on this (at least in memory or CPU usage)14:45
haleybis there a spec?  i guess i would like to see a diagram better describing what this META_IP range is and how the mapping is going to work14:46
ralonsohno14:46
obondarevso there is meta agent and meta proxy now, the RFE is going to replace meta agent only IIUC14:47
mlavallesame understanding14:47
lajoskatonaI think this proposal is in line with the distributed dhcp ( https://bugs.launchpad.net/neutron/+bug/1900934 )14:47
lajoskatonato have only as minimum agents as possible and that is ovs-agent14:48
mlavalleyeah, it is really an overall effort on yulong's part to distribute a lot of the functions14:48
ralonsohok this is what I though initially but I didn't read that in the RFE14:49
mlavalleit is part of a broader picture14:50
amotokimy understanding is as yours. it tries to replace the metadata agent to per-host agent-like feature. I am not sure who plays a role of the metadata agent i.e. who laucnhes local haproxy.14:50
amotokibut perhaps it can be covered by a spec.14:51
liuyulong_Hi guys,14:51
ralonsohhi, we are discussing https://bugs.launchpad.net/neutron/+bug/193322214:51
liuyulong_amotoki, it is ovs-agent14:51
liuyulong_The datapath is VM -> br-int -> br-meta -> tap-meta -> host haproxy.14:52
mlavalleand all that path is in the same host14:52
liuyulong_Yep14:52
ralonsohso you want OVS agent to control the metadata proxy (I din't read that in the RFE)14:52
obondarevso may it result in more time needed to handle each new port?14:53
liuyulong_ralonsoh, "So, I'd like to request for implementing an agent extension for Neutron14:53
liuyulong_ openvswitch agent to make the metadata datapath distributed."14:53
obondarevI mean overall14:53
obondarevcurrently the load is kind of shared between metadata/dhcp/l3 agents and ovs agent14:53
liuyulong_obondarev, No, at least for our cloud.14:53
obondarevif ovs agent becomes responsible for all this.. 14:54
liuyulong_We implement this about 2 years ago, like distribtued DHCP extension for ovs-agent.14:54
obondarevso did you measure port provisioning time?14:55
liuyulong_comparing to metadata agent, this is more reliable and fast.14:55
mlavalleif you did, it would be useful data in the rfe / spec14:55
liuyulong_The origin metadata work procedure is:14:56
liuyulong_VM -> router namespace -> haproxy -> metadata agent14:56
mlavallewith node hops in the middle14:57
liuyulong_#link https://review.opendev.org/c/openstack/neutron/+/63387114:57
liuyulong_This is the fix of race conditon between VM booting and L3 agent processing router ( creating "router namespace + haproxy" ).14:58
ralonsohone question: that feature will be independent to the L3 deployment (legacy, DVR, HA) now you don't need the router14:59
ralonsohright?14:59
liuyulong_"router namespace -> haproxy -> metadata agent" any of these point down will have effect on all hosts for VMs.14:59
ralonsohbut with your feature that won't be needed15:00
ralonsohright?15:00
liuyulong_Yes15:00
ralonsohperfect15:00
mlavalleok, we are at the top of the hour. We will start next meeting with this RFE, which will take place in two weeks15:00
mlavalleHave a nice weekend15:00
ralonsohyou too15:00
liuyulong_OK15:00
amotokithis proposal replaces existing metadata-agent along with l3-agent and dhcp-agent with a per-host metadata-agent as part of ovs-agent.15:00
mlavalle#endmeeting15:00
opendevmeetMeeting ended Fri Jul  9 15:00:57 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.2021-07-09-14.00.html15:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.2021-07-09-14.00.txt15:00
opendevmeetLog:            https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.2021-07-09-14.00.log.html15:00
lajoskatonaBye15:01
rubasovo/15:01
mlavalleo/15:01
liuyulong_amotoki, partitially right, no metadata-agent anymore.15:01
amotokiliuyulong_: yes. so perhaps i need to say per-host metadata-agent FEATURE as part of ovs-agent.15:02
liuyulong_And metatadata data (packets) will not be sent to ovs-agent.15:02
liuyulong_Packets are sent to host haproxy.15:02
ralonsohok, now I see the picture much better15:03
amotokidon't meatadata packet from VM go thru ovs-agent?15:03
liuyulong_Host haproxy is host only, only one.15:03
liuyulong_Allow me quote from the LP bug:15:04
liuyulong_For VMs which try to access metadata via 169.254.169.254:80 will be send to a new ovs-bridge, called br-meta. In this br-meta, the VM's fixed_IP + MAC (source) will be translated to a local META_IP + MAC, while the (dest) 169.254.169.254 is translated to local META_gateway IP which resides in tap-meta. Then send packets to a tap-meta device which resides in br-meta. A host only haproxy which listen the port 80 via 15:04
liuyulong_the device tap-meta and then trasimit the request to nova-metadata-api.15:04
ralonsohliuyulong_, one qq: what if you have IP+MAC duplicated from different networks?15:04
amotokiliuyulong_: so how can metadata from different networks handled?15:04
amotokiah, ralonsoh asked the same question15:05
amotokidifferent networks can have same IP/MAC15:05
liuyulong_ VM fixed_IP + MAC pair will be translated to host only META_IP + MAC15:05
liuyulong_No matter the VM coming from.15:05
amotokisame pair of (fixed_IP, MAC) from different networks work? how are they distinguished?15:06
liuyulong_No overlaping in " META_IP + MAC" in one host.15:06
ralonsohwhy? you can have several networks in one host15:06
ralonsohand duplicated MACs15:06
liuyulong_META_IP can be something like this "100.100.0.0/16" which META_MAC range can be "fe:ee:00:00:00:00".15:07
amotoki(IP1,MAC1) on net1 and (IP1,MAC1 on net2 can be on a same host. how are they distinguished?15:07
liuyulong_VM1 from network1 has IP 192.168.1.1 and MAC aa:aa:aa:aa:aa:aa. While VM2 has same MAC + IP from network2.15:07
liuyulong_VM1's IP + MAC will be translated to, for instance, 100.100.1.100 + fe:ee:00:00:11:1115:08
liuyulong_while VM2 is to 100.100.1.101 + fe:ee:00:00:11:1215:08
amotokiso does the mapping key include netowrk infomration too?15:09
liuyulong_Am i clear?15:09
ralonsohnot really, sorry15:09
amotokiyou said "(IP, MAC)" wil be mapped to (MATA_IP, MAC), but (net, ip, mac) is mapped to (meta_ip, mac). perhaps you assume vlan infomration in ovs implicitlly15:10
amotoki(or port info in ovs)15:10
liuyulong_amotoki, local vlan (match and check) will be applied in flows.15:10
ralonsohyes, you have vlan isolation in OVS, but when this request reaches haproxy, how do you filter by VLAN?15:11
ralonsohanyway, maybe we can wait for the RFE for this15:11
liuyulong_VM -> br-int -> br-meta -> tap-meta -> host haproxy.15:12
liuyulong_I will expand this...15:12
amotokiperhaps br-meta applies this mapping from (local-vlan, ip, mac) to (meta-ip, mac) and send it to tap-meta.15:13
amotokithis can be covered in a comment in the rfe.15:13
liuyulong_VM ->  (192.168.1.1 + aa:aa:aa:aa:aa:aa) to 169.254.169.254:80 -> br-meta -> match local vlan + (192.168.1.1 + aa:aa:aa:aa:aa:aa) ->  100.100.1.100 + fe:ee:00:00:11:11 15:13
liuyulong_ 169.254.169.254:80 ->  100.100.0.1:80 which resides in br-meta15:14
liuyulong_169.254.169.254:80 ->  100.100.0.1:80 which resides in tap-meta15:15
liuyulong_strip vlan and send to tap-meta15:15
liuyulong_Then packet goes back.15:15
amotokii think DNAT happens when going thru br-meta (not in tap-meta)15:16
liuyulong_Before send back, ARP will do first,  tap-meta(100.100.0.1) will try to find  100.100.1.100's MAC15:16
liuyulong_so, the ARP responder will be added to br-meta to answer this.15:16
ralonsohyes, now I understand it: the DNAT address will depend on the network15:16
amotokiralonsoh: do you mean DNAT'ed address assigned to local haproxy is per-network?15:18
liuyulong_Back packet is firstly from haproxy is   TCP:[from 100.100.1.1:80 to  100.100.1.100:TCP_SRC_PORT + fe:ee:00:00:11:11] 15:18
ralonsohamotoki, I think so15:18
amotokiralonsoh: it is different from what liuyulong_ said.15:18
ralonsohhahaha anyway, today is not my best day understanding concepts...15:19
amotokiralonsoh: he said same (IP, MAC) can be covered into (100.100.1.101 + fe:ee:00:00:11:11) for net1 and (100.100.1.101 + fe:ee:00:00:11:12) for net2.15:19
liuyulong_This packet goes to br-meta, then match " 100.100.1.100 + fe:ee:00:00:11:11", with action change dest to "192.168.1.1 + aa:aa:aa:aa:aa:aa"15:19
liuyulong_and mod_vlan with local_vlan of this port15:20
amotokiralonsoh: hahaha. anyways i agree this is complicated enought when we try to explain it only by words..... 15:20
liuyulong_While source IP address is changing from 100.100.0.1 to 169.254.169.25415:20
liuyulong_then go to br-int, finally hit a direct (dest mac is VM's port) flow to VM.15:21
opendevreviewLajos Katona proposed openstack/networking-bagpipe master: Follow pyroute2 changes  https://review.opendev.org/c/openstack/networking-bagpipe/+/80006215:24
liuyulong_ralonsoh, https://review.opendev.org/c/openstack/neutron/+/799159, hi, would have time to take a look of this again?15:25
ralonsohliuyulong_, sure15:25
opendevreviewMerged openstack/neutron master: Check router routes connectivity when GW port is updated  https://review.opendev.org/c/openstack/neutron/+/78992915:59
opendevreviewElvira García Ruiz proposed openstack/neutron master: [OVN] Change ControllerAgent type dinamically  https://review.opendev.org/c/openstack/neutron/+/80027816:17
opendevreviewMerged openstack/ovn-octavia-provider master: Improve enabled_provider_drivers default in devstack  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/78294117:06
*** melwitt is now known as Guest32217:31
*** melwitt_ is now known as melwitt17:57
*** melwitt is now known as jgwentworth17:58
opendevreviewMichael Johnson proposed openstack/neutron-tempest-plugin master: Update service client access test_dns_integration  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/80029118:24
opendevreviewMichael Johnson proposed openstack/neutron-tempest-plugin master: Update service client access test_dns_integration  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/80029119:26
opendevreviewBrian Haley proposed openstack/ovn-octavia-provider master: Add Health Monitor support  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/71325322:39

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!