Thursday, 2020-11-12

*** fdangelo has quit IRC00:35
*** johnkeates has joined #openvswitch02:03
*** johnkeates is now known as oneplane02:10
*** rcernin has quit IRC02:18
*** oneplane has quit IRC02:25
*** rcernin has joined #openvswitch02:50
*** anilvenkata has joined #openvswitch04:52
*** armax has quit IRC05:25
*** rcernin_ has joined #openvswitch05:59
*** rcernin has quit IRC06:01
*** anilvenkata has quit IRC06:33
*** anilvenkata has joined #openvswitch06:34
*** dholler has joined #openvswitch07:17
*** ralonsoh has joined #openvswitch07:23
*** rcernin_ has quit IRC07:26
*** eelco has joined #openvswitch07:27
*** slaweq has joined #openvswitch07:29
*** rcernin_ has joined #openvswitch08:10
*** mdgray has joined #openvswitch08:16
*** rcernin_ has quit IRC08:22
*** rcernin_ has joined #openvswitch08:49
*** elvira has joined #openvswitch08:54
*** elvira has joined #openvswitch08:56
*** rcernin_ has quit IRC08:57
*** anilvenkata_ has joined #openvswitch08:59
*** anilvenkata has quit IRC09:02
*** zhouhan has quit IRC09:02
*** zhouhan has joined #openvswitch09:08
*** zhouhan has quit IRC09:09
*** zhouhan has joined #openvswitch09:10
*** anilvenkata___ has joined #openvswitch09:16
*** anilvenkata_ has quit IRC09:18
*** moldorcoder7 has quit IRC09:19
*** moldorcoder7 has joined #openvswitch09:24
*** acidfu_ has joined #openvswitch09:28
*** acidfoo has quit IRC09:31
*** darkemon has quit IRC09:52
*** darkemon has joined #openvswitch09:56
*** jaicaa has quit IRC09:57
*** jaicaa has joined #openvswitch10:00
*** tbachman has quit IRC11:58
*** fdangelo has joined #openvswitch12:09
*** anilvenkata_ has joined #openvswitch12:14
*** anilvenkata___ has quit IRC12:17
*** anilvenkata_ has quit IRC12:48
*** teardown has quit IRC12:52
*** teardown has joined #openvswitch12:54
*** tbachman has joined #openvswitch13:41
*** bostondriver has joined #openvswitch13:54
*** spatel has joined #openvswitch14:38
*** armax has joined #openvswitch15:11
*** aconole has quit IRC15:59
*** eelco has quit IRC16:25
*** aconole has joined #openvswitch16:32
*** thaller has quit IRC16:49
*** thaller has joined #openvswitch16:49
*** thaller has quit IRC16:57
*** dceara has joined #openvswitch16:58
*** thaller has joined #openvswitch17:04
*** elvira has quit IRC17:39
*** spatel has quit IRC18:11
*** mmichelson_ has quit IRC18:15
*** blp has joined #openvswitch18:15
*** mmichelson has joined #openvswitch18:15
mmichelsonI'm going to start the meeting18:15
mmichelson#startmeeting ovn_community_development_discussion18:15
openstackMeeting started Thu Nov 12 18:15:50 2020 UTC and is due to finish in 60 minutes.  The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot.18:15
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.18:15
openstackThe meeting name has been set to 'ovn_community_development_discussion'18:15
flaviofhi everybody!18:16
numans_Hi18:16
*** numans_ is now known as numans18:16
_lore_hi all18:16
mdgrayHi18:16
mmichelsonI guess I can go ahead and get things going18:16
mmichelsonFirst off, I sent an email last week announcing that tomorrow (13 November) is the soft freeze for OVN 20.12. So if you have any patches you want included in 20.12, please do your best to get it submitted to the mailing list by tomorrow.18:17
mmichelsonAs far as work is concerned, I got my SNAT CT zone selection patch submitted yesterday. I submitted a v2 today that addresses 0-day robot's problems.18:18
mmichelsonI'll be doing reviews today and tomorrow to try to reduce the review load after soft freeze18:18
mmichelsonAnd that's all from me, I suppose.18:18
flaviofmmichelson: you fixed the robot or did the robot complained about something you had to fix?18:19
mmichelsonflaviof, the robot complained about something I had to fix.18:19
blphi18:19
mmichelsonflaviof, there were some minor checkpatch problems, but there was also a complaint about losing const-ness in a function call I was making18:19
mmichelsonIt wasn't reported by gcc locally, so that was strange. But it's fixed in v218:19
flaviofmmichelson: ack. ++18:20
blpMust be different compiler versions.18:20
mmichelsonblp, that's what I assume18:21
blpOK, I'll go next then.18:22
flaviofmmichelson blp are we planning to have ovn-nrthd-ddlog in ovn 20.12?18:22
mdgraymmichelson: I had a similar issue a while back. Passed in travis as well - compiler versions for sure.18:22
blpI am continuing to iterate on the ddlog series. I'd love to have it in 20.12. I'd assume that it would be OK after the soft freeze because it's been cooking for some time, but I'd accept a contrary ruling of course.18:22
numansblp, If I may, there are a couple of issues I found with my ddlog testing.18:24
mmichelsonblp, FYI we did some scale testing with DDLog. We encountered some issues after a while. We think it has something to do with probe intervals. I'll let numans expand on it18:24
blpnumans: Please explain18:24
numans1. There is compilation issue with the travis CI. If you could please take a look. Looks like make dist has few issues. (not a big issue :)) - https://travis-ci.com/github/numansiddique/ovn/jobs/43477831418:25
numans2. Here are the results with ddlog with 500 fake nodes - https://imgur.com/JJpsMhv   vs no ddlog - https://imgur.com/gmbhfoa18:26
numansblp, In the logs I see that ovn-northd-ddlog is sending probe interval every 1 second, even if I disable probe interval  - ovn-nbctl set NB_Global . options:northd_probe_interval=018:26
numansAnd I think as scale increases, ovn-northd-ddlog is dropping the connection to sb db due to 1 second probe interval.18:26
numansI think its happening because of this - https://github.com/blp/ovs-reviews/blob/ddlog9/northd/ovn-northd-ddlog.c#L89318:26
numansI'm planning to hack the probe interval thing and run again to rule out if the issue is because of constant reconnections to the sb db.18:28
blpAh. OK. Thanks for those reports. I'll fix them for v6.18:28
blpProbably simple to fix.18:29
numansblp, In the northd-ddlog logs I observed this - 2020-11-10T13:51:32.186Z|01351|timeval|WARN|Unreasonably long 335427ms poll interval (214ms user, 1ms system)18:29
numansblp, not sure if it is related to the probe interval.18:30
numansblp, welcome.18:30
flaviofnumans: nice graphs! Is the probe interval on the non-ddlog version test set to 0?18:30
numansflaviof, I think with ddlog, the north_probe_interval configured on the db is not taking effect.18:31
blpWow, that's a really long main loop. The difference between elapsed and user+system is suspicious though.18:31
numansblp, Also with really huge SB DB, we are hitting this issue - https://github.com/ovn-org/ovn/commit/1e59feea933610b28fd4442243162ce35595cfee with ddlog too18:32
numansblp, again related to probe interval.18:32
numansYou may want to consider the same for ddlog too.18:32
numansother than the ddlog testing, I worked on my hairpin patches18:33
numansand submitted v4.18:33
numansI also did some code reviews.18:33
numansThat's it from me.18:33
blpnumans: Thanks for pointing that issue out. I'll look at fixing it too.18:33
numansblp, thanks.18:34
numansIf some one wants to go next.18:35
*** dholler has quit IRC18:35
_lore_can I go next?18:35
_lore_pretty fast18:36
numanssure18:36
mmichelsongo for it18:36
_lore_this week I posted a native implementation for bfd in ovn18:36
_lore_(actually today)18:36
_lore_it is not complete yet, but I wanted to post it to collect feedbacks18:36
_lore_we decided to implement it in ovn since the ovs one seems not so suitable to our needs, we want to enable it on ovn entities, like logical router ports18:37
blpMakes sense.18:37
_lore_moreover we would enstablish a connection between a logical router port and multiple external entities18:38
_lore_not belonging to ovn "domain"18:38
blpDoes it make sense to try to share code between the implementations?18:38
blp(I don't have a guess at that.)18:38
_lore_I looked at ovs implementation18:38
_lore_and it relies on ovs stuff (netdev, interface, ecc.)18:38
_lore_so it is not so easy18:38
_lore_maybe we can do as follow-up series18:39
_lore_I implemented basic functionalities18:40
_lore_tx/rx, state machine18:40
blpI think that any code sharing would have to be factoring out a common library and using it both places. I don't have any idea whether that would be productive.18:40
_lore_yes, I agree18:41
_lore_when the code is more or less ready let's see how much we can share18:41
_lore_agree?18:41
blpyes18:42
_lore_ok, cool18:42
_lore_so I will continue to work on my implementation18:42
_lore_that's all from my side, thx18:42
flaviof_lore_ native in ovn meaning the bfd packets are generated/handled by the ovn-controller process?18:42
_lore_yes18:43
_lore_flaviof: correct18:43
flaviof_lore_ cool beans.18:43
blpHow big is this likely going to need to scale? Hundreds or thousands of packets/sec might use a lot of CPU%.18:43
_lore_blp: I do not think so18:43
_lore_but even in ovs packets are sent in userspace18:43
_lore_as upcall18:44
*** ralonsoh has quit IRC18:44
blpYes, of course. This takes one more hop across IPC, which could be an additional bottleneck.18:44
_lore_I agree18:45
_lore_let's sync with osp/osh folks on it18:46
_lore_I think the main use-case is lb health-check18:46
_lore_for ecmp routing18:47
blpOK. Some network virtualization systems end up running BFD (etc.) in a full mesh across all chassis. That is expensive.18:47
blpIf it's just for a limited number of connections, it will scale better.18:48
_lore_I do not have any exact number now18:48
_lore_I will sync with osh folks and get back on it18:48
blpWhat is osh?18:48
_lore_open-shift18:49
_lore_:)18:49
flaviofg-osh! ;)18:49
blpAh. I did not know the abbrev.18:50
blpAnd osp?18:50
imaximetsblp, OpenStack Platform, IIUC.18:50
numansopenstack18:50
_lore_yes18:50
blpThanks.18:50
_lore_sorry for the confusion :(18:51
blpAnyone else?18:51
blpI have my homework list, I think.18:51
imaximetsI have a quick update.18:51
imaximetsThere is a recurring issue with the size of sb db.18:51
imaximetsSo, this week I tried to prepare PoC patches to combine lflows.18:52
imaximetsThere are lots of lflows that differs only in logical datapath column, so they could be combined to save space.18:52
imaximetsby introducing Logical_Datapath_Group table or something similar.18:53
imaximetsnorthd changes was kind of straightforward, but I found it really hard to make changes in ovn-controller.18:54
blpOh, interesting. Or the logical_datapath column could be made a set: empty -> all datapaths, otherwise all the listed datapaths.18:54
imaximetsblp, yep.18:54
blpovn-controller makes me sad right now.18:54
blpIt's complicated.18:54
imaximetsthe issue with ovn-controller is that it works directly with database without any abstraction, so it's hard to modify schema in general.18:55
imaximetsthe second big issue is that almost all parts of lflow processing depends on fact that lflow to datapath is 1:1 relation.18:55
blpeventually it might make sense to apply ddlog to ovn-controller, but it does not solve any problems immediately18:56
imaximetsdceara said that he could help reducing dependencies from the logical datapath, at least we could eliminate dependency in match expression parsing code.18:56
imaximetsSo, I'll wait for him to do that before continuing my efforts.18:56
blpI guess another option would be to add a match on datapath to the expression logic.18:57
blpAnd then allow the logical_datapath field to be empty.18:58
blpA match on a port would have a prerequisite on the logical datapath that contains the port, I guess.18:58
imaximetsbut will this reduce number of lflows in sb db?18:59
numansblp, there are many lflows like match(1), action=next; which imaximets wants to reduce is what I understand.18:59
numansthis is just one example. there could be many18:59
imaximetsnumans, there are lots of different flows. :)19:00
blpSure it would. That kind of flow would naturally work on any logical datapath. No prerequisites would be generated, since it doesn't match on a port.19:00
zhouhanblp: I am thinking the other way around. I think putting everything in match condition as a string makes it hard for optimization in ovn-controller because we can't get any metadata before parsing the logical flow19:00
blpOh, i guess the other issue is that it's hard to just get the flows a chassis needs. Currently that's pretty easy.19:00
zhouhanblp: yes19:01
numansthat's a good point.19:01
numansblp, in the case of large scale deployments we have noticed that disabling conditional monitoring helps.19:01
imaximetsthis will hit performance of ovn-controller.19:01
blpnumans: Is that because of CPU% on the dbserver?19:02
numansotherwise ovsdb-server takes lot of cpu cycles to handle conditional monitoring19:02
numansblp, yes19:02
zhouhanblp: even today, because port exists only in match, we lose some chance to use that to filter out the lflows that doesn't need to be parsed by the chassis19:02
blpOK.19:02
*** aconole has quit IRC19:03
imaximetsAnyway ovn-controller needs some refactoring and untangling.19:03
*** spatel has joined #openvswitch19:03
mmichelsonthat it does19:04
imaximetsOnce this done I'll be able to continue with lflow combining.19:04
imaximetsIn any way of implementation it should reduce size of sb db by factor of number of logical switches in some cases.19:05
imaximetsThat's it from my side.19:05
mmichelsonAnybody else?19:06
flaviofmay I?19:06
* flaviof thinking yes.... ;)19:06
flaviofAs mentioned in the ovs-dev ML, I'm wrapping up the "fair meters" changes, now using a bool column in the NB Meters table. TY blp and dceara for all the support!19:06
flaviof#link https://mail.openvswitch.org/pipermail/ovs-dev/2020-November/377277.html the current plan on implementing fair meters19:07
flaviof#link https://patchwork.ozlabs.org/project/ovn/patch/20201107213913.GC2753733@ovn.org/ blp generously explaining fair meters changes for ovn-northd-ddlog19:07
flaviofI should be done soon, but want to wait for ddlog's merges first.19:07
flaviofBesides ACL logs, I see only  one other potential user of Meters table in NB:19:07
flaviof#link https://github.com/ovn-org/ovn-kubernetes/blob/061d543f363d48d2a4f0fc1df95fea2c6129e3cf/go-controller/pkg/ovn/ovn.go#L435  controller_event19:07
blpflaviof: Do you want any support for me at the moment on the fair meters changes?19:07
flaviofThere is nothing in northd that deal with that, so this is looking easier than I thought. I hope I'm not wrong. ;)19:07
flaviofThat is all from me.19:07
flaviofblp maybe. But let me try it first. I'll let you know!19:08
numansblp, one question on the ddlog, I notice that northd-ddlog is using update method instead of update2/update3. is that intentional ? Just curious.19:09
numansi mean the jsonrpc updates from the ovsdb-server.19:09
blpnumans: someone else wrote that code and I haven't revisited it. I'll take a look.19:09
numansblp, ack. thanks.19:09
blpI already got in the coreutils doc update that Dumitru pointed out was needed. https://lists.gnu.org/archive/html/coreutils/2020-11/msg00014.html19:11
imaximetsblp, side question, there is a concern that ddlog doesn't use ovsdb-idl, but re-implements it.  Maybe it's better to use ovsdb-idl instead?  That might be easier to maintain.19:11
blpThe reason that ddlog doesn't use the idl is that it needs the deltas, which the idl hides. We could extend the idl to expose the deltas; it's a valid option.19:12
blpIf you like, I can explore that path.19:12
numansblp, that could be useful to ovn-controller as well.19:13
imaximetsit would be great.  At least to check.19:13
blpI'll explore.19:13
zhouhanblp: for deltas do you mean something different from change-tracking?19:13
imaximetsblp, thanks.19:13
zhouhanblp: or improvements on the IDL change-tracking?19:14
blpChanges and deltas are the same. I don't know whether the idl change-tracking code exposes all the information I need. That will be one of the things I check.19:14
numansI think idl change tracking doesn't expose what column changed.19:14
zhouhanblp: ok. IDL change-tracking currently doesn't provide old data, if a row is modified.19:14
zhouhannumans: yes, that's another one19:15
numanspython ovs idl provides a callback for any updates. and the callback function can figure out what columns changed.19:16
blpOld data is essential, because ddlog needs to be passed the row that was deleted and the row that was added.19:16
blpMaybe the API can be adapted. I will see.19:16
zhouhanone more thing, For set/map columns, it would be better to get just what's changed, because there can be huge data in a column, such as address-set, static-route, etc.19:17
* numans says bye.19:17
blpDDlog isn't prepared to consume anything other than full row updates at the moment.19:17
zhouhanblp: ok, I guess that might be needed19:18
blpIf there's a bottleneck, then there are various ways we could deal with it.19:18
blpFor example, we could break a table down into multiple relations.19:18
blpI don't think that's a problem yet though.19:18
zhouhanfor scenarios like: add a member in a group for policy enforcement - we don't want all the related flows got recomputed19:19
zhouhanif the group is huge, e.g. thousands of members19:19
zhouhanor, in a deployment we could have a single router, with ten thousands of routes. If any of the routes change, it is just one row change in the DB, but the row contains all the routes.19:21
blpIt's likely that in that particular situation what's changing is a northbound group membership. That would normally just translate to a single southbound group membership change. If it actually affected thousands of flows, that's a problem in the flow table design.19:21
mmichelsonzhouhan, that sounds like it could be solved by blp's idea of breaking into multiple relations19:21
blpIf there is indeed a problem here, then we will come up with a way to systematically break up the tables into multiple relations.19:22
zhouhanmmichelson: oh, yes, if that's discussed already19:22
mmichelsonActually, no it wouldn't. It just would shift where the burden is some.19:22
mmichelsonHmm19:22
blpIf a change to nb will cause a large change to sb, then there's no way to avoid computation at least linear in the change to the sb.19:23
blpOtherwise, it can probably be avoided.19:23
blpI have a meeting with my boss in 5 minutes.19:23
blpWell, probably more like 15, he's usually late.19:24
zhouhanIf we wont to solve this by schema change, there will be many places to change ...19:24
blpThe "breaking into multiple relations" appraoch I am talking about would not require any change to the schemas, only to ovn-northd-ddlog.19:24
*** dcbw has joined #openvswitch19:24
zhouhanblp: oh, I misunderstood then19:25
blp(schemas? schemata?)19:25
mmichelsonyeah I misunderstood as well19:25
blpBut I don't want to start down that path until we know there is a problem19:25
blpand it might make more sense to look at ovn-controller first19:25
zhouhanblp: if the "breaking into multiple relations" is in ddlog, then still we will need to solve the problem of observing what's really changed in the huge column19:27
flaviofzhouhan I thought you were referring to columns that have a long list of values. Like ports in a port_group? Ins't that something what could be turned into a separate table? Forgive me if I'm not understanding it right.19:27
blpThat's not a problem, it's just a matter of writing code.19:28
blpwe're good at writing code19:28
flaviofblp++19:28
zhouhanblp: sure :)19:28
zhouhanit's the change-tracking enhancement I was talking about19:28
blpflaviof: That's what I was talking about.19:29
zhouhanflaviof: yes that's another approach, but not what blp mentioned19:29
zhouhan:D19:29
flaviofack. TY19:29
blpOK folks, I need to run. Talk to you next week!19:29
*** blp has quit IRC19:29
zhouhanthanks, bye19:29
flaviofbye all!19:29
imaximetsbye19:29
mmichelsonbye!19:29
mmichelson#endmeeting19:30
openstackMeeting ended Thu Nov 12 19:30:02 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:30
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-11-12-18.15.html19:30
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-11-12-18.15.txt19:30
openstackLog:            http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-11-12-18.15.log.html19:30
zhouhanflaviof: I thought the schema could be changed so that the big columns can be put in a separate table. blp was talking about not changing the schema but breaking into separate tables in ddlog.19:31
zhouhanflaviof: I am not sure if you were saying tables in schema or tables in ddlog19:32
flaviofzhouhan: ack. Sounds like an approach that makes sense once ovn-controller goes ddlog way. Good stuff!19:32
flaviofI was saying tables in the schema. But schema changes are not pleasant.19:33
zhouhanflaviof: agree :)19:35
*** mdgray has quit IRC20:02
*** rcernin has joined #openvswitch20:32
*** rcernin has quit IRC20:35
*** spatel has quit IRC20:36
*** aconole has joined #openvswitch20:39
*** aconole has quit IRC20:51
*** aconole has joined #openvswitch21:19
*** slaweq has quit IRC21:30
*** slaweq has joined #openvswitch21:33
*** dyusupov has joined #openvswitch21:34
*** dyusupov has quit IRC21:37
*** dceara has quit IRC21:38
*** slaweq has quit IRC21:41
*** rcernin has joined #openvswitch21:47
*** rcernin has quit IRC22:01
*** zhouhan_ has joined #openvswitch22:14
*** rcernin has joined #openvswitch22:14
*** zhouhan has quit IRC22:17
*** zhouhan has joined #openvswitch22:29
*** zhouhan_ has quit IRC22:29
*** acidfoo_ has joined #openvswitch22:29
*** dcbw has quit IRC22:30
*** acidfu_ has quit IRC22:32
*** spatel has joined #openvswitch22:36
*** spatel has quit IRC22:41
*** moldorcoder7 has quit IRC23:00
*** moldorcoder7 has joined #openvswitch23:00
*** bostondriver has quit IRC23:01
*** moldorcoder7 has quit IRC23:08
*** moldorcoder7 has joined #openvswitch23:09
*** moldorcoder7 has quit IRC23:10
*** moldorcoder7 has joined #openvswitch23:10
*** tryauuum has quit IRC23:11
*** spinningmonkey has quit IRC23:24
*** spinningmonkey has joined #openvswitch23:25

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!