Thursday, 2021-04-15

*** acidfu has joined #openvswitch00:14
*** dcbw has quit IRC00:34
*** fdangelo has quit IRC01:24
*** Anticimex has quit IRC01:56
*** Anticimex has joined #openvswitch01:59
*** rcernin has quit IRC02:10
*** rcernin has joined #openvswitch02:36
*** fdangelo has joined #openvswitch02:37
*** acidfu has quit IRC03:03
*** rcernin has quit IRC03:34
*** rcernin has joined #openvswitch03:44
*** rcernin has quit IRC03:55
*** rcernin has joined #openvswitch03:55
*** rcernin has quit IRC04:13
*** fdangelo has quit IRC04:15
*** osmanlicilegi has left #openvswitch04:18
*** ygk_12345 has joined #openvswitch04:41
*** donhw has quit IRC04:47
*** donhw has joined #openvswitch04:49
*** rcernin has joined #openvswitch04:55
*** rcernin has quit IRC05:10
*** donhw has quit IRC05:43
*** ygk_12345 has quit IRC05:43
*** donhw has joined #openvswitch05:43
*** rcernin has joined #openvswitch05:49
*** rcernin has joined #openvswitch05:50
*** slaweq has quit IRC05:55
*** slaweq_ has joined #openvswitch05:55
*** slaweq_ is now known as slaweq05:55
*** rcernin has quit IRC06:10
*** ralonsoh has joined #openvswitch06:10
*** osmanlicilegi has joined #openvswitch06:16
*** slaweq_ has joined #openvswitch06:21
*** slaweq_ has quit IRC06:27
*** eelco has joined #openvswitch06:27
*** rcernin has joined #openvswitch06:28
*** slaweq_ has joined #openvswitch06:29
*** slaweq_ has quit IRC06:29
*** rcernin has quit IRC06:29
*** slaweq_ has joined #openvswitch06:30
*** rcernin has joined #openvswitch06:30
*** slaweq has quit IRC06:36
*** slaweq_ is now known as slaweq06:36
*** dholler has joined #openvswitch06:39
*** amorenoz has joined #openvswitch06:40
*** ygk_12345 has joined #openvswitch06:46
*** ygk_12345 has quit IRC06:52
*** ygk_12345 has joined #openvswitch06:53
*** ygk_12345 has joined #openvswitch06:53
*** ygk_12345 has quit IRC06:56
*** jaicaa has quit IRC07:10
*** jaicaa has joined #openvswitch07:13
*** mdgray has joined #openvswitch07:17
*** mdgray has quit IRC07:23
*** mdgray has joined #openvswitch07:27
*** ygk_12345 has joined #openvswitch07:33
*** ygk_12345 has quit IRC07:34
*** ygk_12345 has joined #openvswitch07:35
*** elvira has joined #openvswitch07:46
*** ygk_12345 has quit IRC07:51
*** ygk_12345 has joined #openvswitch07:56
*** ygk has joined #openvswitch08:00
*** ygk has left #openvswitch08:04
*** ygk_12345 has left #openvswitch08:05
*** ygk_12345_ has joined #openvswitch08:08
*** rcernin has quit IRC08:37
*** slaweq has quit IRC08:51
*** slaweq has joined #openvswitch08:51
*** istokes has joined #openvswitch08:54
*** psahoo has joined #openvswitch10:12
*** fdangelo has joined #openvswitch10:54
*** deadalnix has joined #openvswitch11:18
*** psahoo_ has joined #openvswitch11:28
*** psahoo has quit IRC11:31
*** yamamoto has quit IRC11:51
*** rfolco has joined #openvswitch11:56
*** yamamoto has joined #openvswitch11:59
*** yamamoto has quit IRC12:04
*** bostondriver has joined #openvswitch12:04
*** yamamoto has joined #openvswitch12:46
*** yamamoto has quit IRC12:57
*** acidfu has joined #openvswitch13:17
*** istokes has quit IRC13:32
*** istokes has joined #openvswitch13:35
*** ygk_1234555 has joined #openvswitch13:42
*** fdangelo has quit IRC13:58
*** fdangelo has joined #openvswitch14:04
*** links has joined #openvswitch14:29
*** thaller_ has joined #openvswitch14:31
*** thaller has quit IRC14:34
*** eelco has quit IRC14:55
*** ygk_1234555 has quit IRC15:17
*** ygk_1234581 has joined #openvswitch15:26
*** ygk_1234581 has quit IRC15:26
*** ygk_1234584 has joined #openvswitch15:27
*** ygk_1234584 has quit IRC15:32
*** psahoo_ has quit IRC15:53
*** psahoo has joined #openvswitch15:57
*** elvira has quit IRC16:16
*** psahoo has quit IRC16:18
*** fdangelo has quit IRC16:25
*** fdangelo has joined #openvswitch16:25
*** psahoo has joined #openvswitch16:32
*** links has quit IRC16:40
*** psahoo has quit IRC16:52
*** yamamoto has joined #openvswitch16:53
*** deadalnix has quit IRC16:54
*** eelco has joined #openvswitch17:00
*** deadalnix has joined #openvswitch17:07
*** mdgray has quit IRC17:08
*** ralonsoh has quit IRC17:12
*** yamamoto has quit IRC17:18
*** KpuCko has quit IRC17:19
mmichelsonHi everyone, sorry for being late on the meeting17:21
mmichelsonGot caught up in other things17:21
mmichelson#startmeeting17:21
openstackmmichelson: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee'17:21
_lore_hi all17:21
mmichelson#startmeeting ovn_community_development_meeting17:21
openstackMeeting started Thu Apr 15 17:21:22 2021 UTC and is due to finish in 60 minutes.  The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot.17:21
imaximetshi17:21
mmichelsonuh, openstack?17:21
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:21
openstackThe meeting name has been set to 'ovn_community_development_meeting'17:21
mmichelsonah there we go17:21
*** zhouhan has joined #openvswitch17:22
*** zhouhan has quit IRC17:25
*** zhouhan_hzhou8 has joined #openvswitch17:25
mmichelsonOK, so, I finally have a new patch series up for the floating IP issue that I started working on a long time ago17:26
mmichelsonIt's gone from a 2 patch series to a 5 patch series in an attempt to both fix the issue and to make things more efficient by no longer requiring ARPs to be sent in cases where they shouldn't be necessary17:26
mmichelsonWe also released new versions of OVN 20.03 and 20.06 for Ubuntu's purposes.17:27
numansHello17:27
mmichelsonI was hoping blp would be here today so I could ask about the bug report I sent for ovn-northd-ddlog, but I guess I'll just need to bump the email I sent.17:27
mmichelsonUm...I think that's it for me.17:28
numansI can go real fast17:28
numansI've submitted a couple of patches for review related to conntrack improvement and on usage of ct.inv.17:29
*** tbachman has joined #openvswitch17:29
numanszhouhan_hzhou8, ^ appreciate if you can take a look.17:29
numansI also addressed zhouhan_hzhou8's comments and submitted another version of physical flow split patch.17:29
numansI did some code reviews.17:29
numansThat's it from me.17:29
imaximetsI have a small update too.17:30
imaximetsI finished and posted v2 of stream record/replay functionality with integration to ovsdb-server.17:31
imaximets#link https://patchwork.ozlabs.org/project/openvswitch/list/?series=23883017:31
imaximetsOnce this accepted to OVS, we will be able to integrate into ovn daemons, i.e. northd or ovn-controller.  for debugging and performance testing purposes.17:32
imaximetsDumitru and zhouhan_hzhou8 reviewed bundles support for ofctrl.  So, I guess, the patch is good to go now. :)17:33
imaximetsthat's it from my side.17:33
_lore_can I go next? quite fast17:33
_lore_this week I worked on skip_force_snat patch, thx mark for the review17:34
_lore_then I started working on 2 items and I would like to have your opinion:17:34
_lore_1- I noticed whenever we wake ovn-controller main thread from pinctrl we run all the handlers even if just one will do some goodput17:35
_lore_does it worth to just run the related handler in pinctrl_run()?17:36
_lore_any opinions?17:37
mmichelson_lore_, so you want to add incremental processing for pinctrl, essentially?17:38
_lore_mmichelson: nope17:39
imaximets_lore_, I'm not sure about the idea (I just do not know that code), but I'm also not sure how thread-safe ovn-controller is.  Is it?17:39
_lore_there is a mutex between ovn-controller main thread and pinctrl_thread17:39
_lore_imaximets: ..^17:39
imaximets_lore_, ack.17:40
_lore_mmichelson: what I mean is having a mask to run just the handlers in pinctl_run() that has been set in pinctrl_thread()17:40
_lore_it is just an idea, I need to review the code to see if it is feasible17:41
*** zhouhan has joined #openvswitch17:41
mmichelson_lore_, If it's causing a noticeable problem, then sure I'd say to go ahead and pursue that as a possible option. I'm not sure how much of a problem this actually poses right now though.17:42
_lore_mmichelson: I do not have any report, I just noticed lookig at the code17:42
numansI agree with mmichelson.17:43
* zhouhan finally got passwd back on a new computer17:43
_lore_numans: mmichelson: ok, I will see if it easy to implement17:43
numanszhouhan, cool17:43
_lore_2- we have an issue related to garp that can make ovn-northd clocks at 100% cpu17:44
zhouhan_lore_ I don't remember exactly but shouldn't pinctrl run just one handler based on the message type?17:44
_lore_zhouhan: what I mean is not pinctrl_thread17:44
_lore_I mean if we can optmize main thread17:45
_lore_but it is just an idea17:45
_lore_I am not sure17:45
_lore_related to 2, I was wondering if we can implement something specific to arp reply to limit the rate of wakes or we can just use a meter on the action17:45
_lore_what do you think?17:46
zhouhan_lore_ oh, sorry, just re-read your message and I got your point now. Yes, you are right, but the most costly part is in I-P and since there is no input change, the I-P engine will not do any compute.17:46
_lore_zhouhan: maybe there is no difference, I spotted this just looking at the code17:46
zhouhan_lore_ If there are handlers too costly to be run on each main thread wakeup, then we should put it in I-P eng17:46
_lore_ack17:46
_lore_any opinion about point 2?17:47
mmichelson_lore_, can you be more specific about why the garps are causing ovn-northd to run at 100% cpu?17:48
mmichelsonis it because mac_bindings are being updated too often?17:48
_lore_according to the bz keepalived keeps moving the vip from one place to another  and send garps to it17:49
_lore_yes17:49
_lore_mac_bindings is updated very opten by the garp17:49
_lore_this will end-up in a sb update17:49
zhouhanbut mac_bindings are not handled by northd, right?17:49
zhouhanok, so northd just got wake up, and there is no I-P there in the C version :)17:50
_lore_yes17:50
_lore_correct17:50
zhouhanOh, wait, northd shouldn't even monitor mac_binding17:50
mmichelsonzhouhan, northd monitors mac_bindings and deletes them if the logical port has been deleted.17:51
mmichelsonzhouhan, see cleanup_mac_bindings()17:51
zhouhanoh, ok, I recall this been added sometime ago17:51
_lore_this is the related bz: https://bugzilla.redhat.com/show_bug.cgi?id=194791317:51
openstackbugzilla.redhat.com bug 1947913 in ovn2.13 "[OVN][RFE] Add protection mechanism against gARPs / flapping ports" [High,New] - Assigned to lorenzo.bianconi17:51
mmichelsonWhat's happening here is that the MAC_Bindings are being updated, but since no ports are being changed, it means that northd is doing useless work.17:51
_lore_yes17:52
zhouhanDoes the ddlog northd help?17:52
mmichelsonSo a possible workaround that dceara discussed at one point was making ovn-northd stop prematurely if the only change was a MAC_Binding. But that's kind of a poor man's I-P17:52
mmichelsonzhouhan, presumably, it would.  But that's not being used in production17:53
_lore_the bz even say ovs-db is quite loaded so maybe better to filter them in ovn-controller?17:53
mmichelsonBut the flip side to this is that the garps themselves should probably be metered. And I think that's the part you're bringing up here _lore_17:53
_lore_mmichelson: correct17:53
_lore_my question is:17:53
_lore_is it better a meter for the action (need to check the code) or to implement something in c for mac_binding?17:54
zhouhanI remember dumitru started some work 1 - 2 years ago for control plane ratelimiting in general17:57
mmichelson_lore_, I think probably both are good ideas, personally.17:57
_lore_ok, I will come up with something17:58
zhouhanbut can't remember how did it go. In general, there was a dilemma for ratelimiting. Either it could block real request or it could require huge amount of meters causing performance problem.17:59
imaximetsmmichelson, _lore_:  about rate limiting, don't we need to just ban certain garps that causes problems instead of limiting all of them?  We will loose some valid binding events due to limiting and that may be bad.17:59
_lore_imaximets: yes, this is what I would like to do in c18:00
_lore_in pinctrl code18:00
_lore_zhouhan: maybe it is better to just ratelimit a given IP18:00
_lore_not all the IPs18:00
zhouhan_lore_ hmm, but how do you know which IP to meter? Through config?18:01
_lore_in the GARP we have the src mac and IP18:01
_lore_right?18:01
_lore_or in the arp reply in general18:01
_lore_I will try to come up with a PoC18:02
mmichelsonBut how do you know how to program which IPs to meter on? Don't you have to know that before the garps arrive?18:02
*** eelco has quit IRC18:02
_lore_and then we can continue the discussion on the ml18:02
zhouhanI mean, we can't predict what IPs could appear in GARPs, i.e. which GARPs to apply rateliimiting18:02
zhouhan(same as what mmichelson said :)18:04
_lore_I mean when we receive the first request we can have a map for them18:05
_lore_and so keep track of the incoming requests18:05
_lore_it is just an idea18:05
mmichelson_lore_, if you can put together a PoC, then I agree that we can continue the discussion on the mailing list18:06
_lore_ack thx18:06
_lore_:)18:06
zhouhan_lore_: so when there are lots of new GARP comes with different IPs, that map can increase which would require huge amount of meters, right?18:07
zhouhanyep, maybe a POC is great. Some tradeoffs could be made18:07
_lore_I will not use meters18:07
_lore_I mean, I will implement the logic of a meter, but in pinctrl thread18:08
_lore_and the map will have a max size18:08
_lore_we the map is full we will discard the new packets since we are under attack18:08
_lore_right?18:08
zhouhan_lore_: ok, but discarding new packets basically is blocking "healthy" ones (as a result of DDoS)18:09
_lore_zhouhan: yes, but if the maps is full it means somthing not right is happening, so it is better just to discard packets and not run at 100% cpu right18:10
_lore_?18:10
_lore_it is a tradeoff18:10
*** dholler has quit IRC18:10
_lore_let me review the code better and we can continue discussing about it18:11
zhouhanyes, I believe some tradeoff has to be made, to solve the dilemma18:11
zhouhansorry I have to run for another meeting. ttyl18:11
mmichelson_lore_, but if the map gets full, and we stop processing new garps, then wouldn't that still result in a DoS since we're not processing garps any longer?18:11
_lore_zhouhan: this is a perfect world :D18:11
mmichelsonI'll wait for a PoC since it's hard to know exactly how this will work.18:12
_lore_mmichelson: it is better to not processing just arp but the system continue working, right?18:12
*** zhouhan_hzhou8 has quit IRC18:12
*** zhouhan has quit IRC18:12
mmichelson_lore_, Ok I see what you're saying18:12
_lore_I will keep you updated :)18:13
_lore_that's all from me18:13
_lore_thx18:13
mmichelsonAnybody else?18:14
mmichelsonOK I guess that's it for today18:15
mmichelson#endmeeting18:15
openstackMeeting ended Thu Apr 15 18:15:23 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:15
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-04-15-17.21.html18:15
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-04-15-17.21.txt18:15
openstackLog:            http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-04-15-17.21.log.html18:15
_lore_bye all18:15
*** KpuCko has joined #openvswitch18:45
*** donhw has quit IRC18:57
*** donhw has joined #openvswitch18:58
*** dcbw has joined #openvswitch18:59
*** istokes has quit IRC19:56
*** dcbw has quit IRC21:36
*** fdangelo has quit IRC21:39
*** bostondriver has quit IRC21:43
*** yamamoto has joined #openvswitch22:31
*** yamamoto has quit IRC22:35
*** yamamoto has joined #openvswitch22:36
*** rcernin has joined #openvswitch22:51
bdonnahue2hey folks can anyone help me setup a vxlan tunnel on a proxmox host?23:16
bdonnahue2cant seem to get the configs right23:16
*** KpuCko has quit IRC23:23

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!