Thursday, 2020-10-22

*** ari98 has joined #openvswitch01:18
*** thaller has quit IRC02:01
*** thaller has joined #openvswitch02:02
*** dholler has quit IRC02:25
*** acidfu has quit IRC02:30
*** larsks has quit IRC02:36
*** dholler has joined #openvswitch02:38
*** larsks has joined #openvswitch02:50
*** rcernin has quit IRC02:52
*** rcernin has joined #openvswitch02:57
*** larsks has quit IRC02:59
*** larsks has joined #openvswitch03:02
*** larsks has quit IRC03:05
*** larsks has joined #openvswitch03:08
*** larsks has quit IRC03:09
*** larsks has joined #openvswitch03:12
*** links has joined #openvswitch03:37
*** kobis1 has joined #openvswitch04:47
*** kobis1 has quit IRC04:47
*** blahdodo has quit IRC06:08
*** tbachman has quit IRC06:10
*** blahdodo has joined #openvswitch06:12
*** tbachman has joined #openvswitch06:15
*** slaweq has joined #openvswitch06:21
*** slaweq has quit IRC06:44
*** slaweq has joined #openvswitch06:46
*** ralonsoh has joined #openvswitch07:05
*** jraju__ has joined #openvswitch07:22
*** links has quit IRC07:22
*** dceara has joined #openvswitch07:37
*** rcernin has quit IRC07:42
*** imaximets has quit IRC07:48
*** imaximets has joined #openvswitch07:48
*** rebrec has quit IRC07:49
*** rebrec has joined #openvswitch07:50
*** elvira has joined #openvswitch07:57
*** rcernin has joined #openvswitch08:01
*** rcernin has quit IRC08:12
*** jraju__ has quit IRC08:26
*** zhouhan_ has quit IRC08:32
*** zhouhan has joined #openvswitch08:33
*** labelette has joined #openvswitch08:39
*** labelette has quit IRC08:41
*** labelette has joined #openvswitch08:42
*** labelette has quit IRC08:43
*** labelette has joined #openvswitch08:43
*** labelette has quit IRC08:45
*** labelette has joined #openvswitch08:45
*** labelette has quit IRC08:46
*** labelette has joined #openvswitch08:47
*** labelette is now known as blt08:48
*** blt has quit IRC08:54
*** lablt has joined #openvswitch08:55
*** jaicaa has quit IRC09:22
*** jaicaa has joined #openvswitch09:23
*** jraju__ has joined #openvswitch10:18
*** jraju__ has quit IRC10:19
*** cpaelzer has quit IRC10:56
*** acidfu has joined #openvswitch10:57
*** tbachman has quit IRC11:04
*** cpaelzer__ has joined #openvswitch11:51
*** thaller has quit IRC12:02
*** thaller has joined #openvswitch12:02
*** bostondriver has joined #openvswitch12:10
*** tbachman has joined #openvswitch12:39
*** donhw has quit IRC13:27
*** donhw has joined #openvswitch13:28
*** thaller has quit IRC13:52
*** thaller has joined #openvswitch13:57
*** thaller has quit IRC13:59
*** thaller has joined #openvswitch14:00
*** Kamilion has quit IRC14:07
*** dcbw has joined #openvswitch14:27
*** Kamilion has joined #openvswitch14:27
*** tbachman_ has joined #openvswitch14:33
*** tbachman has quit IRC14:35
*** tbachman_ is now known as tbachman14:35
*** acidfu has quit IRC14:36
*** acidfu has joined #openvswitch14:41
*** acidfu has quit IRC14:46
*** tbachman_ has joined #openvswitch15:10
*** tbachman has quit IRC15:11
*** tbachman_ is now known as tbachman15:11
*** |subz3r0| has quit IRC15:12
*** |subz3r0| has joined #openvswitch15:14
*** |subz3r0| has joined #openvswitch15:14
*** acidfu has joined #openvswitch15:19
*** gregwork has joined #openvswitch15:26
*** |subz3r0| has quit IRC15:42
*** |subz3r0| has joined #openvswitch15:43
*** zhouhan_ has joined #openvswitch16:03
*** zhouhan has quit IRC16:07
gregworkis pvlan in ovs a thing ?16:10
*** donhw has quit IRC16:16
*** donhw has joined #openvswitch16:19
*** ralonsoh has quit IRC17:08
*** _mdgray_ has joined #openvswitch17:08
*** mmichelson_ has quit IRC17:14
*** mmichelson has joined #openvswitch17:14
mmichelsonHi everyone17:15
mmichelsonI'm going to go ahead and start the meeting17:15
dcearaHi17:16
mmichelson#startmeeting ovn_community_development_discussion17:16
openstackMeeting started Thu Oct 22 17:16:35 2020 UTC and is due to finish in 60 minutes.  The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot.17:16
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:16
openstackThe meeting name has been set to 'ovn_community_development_discussion'17:16
numansHello17:17
mmichelsonAs a quick note at the beginning, we're coming up on that awkward time of year where the clocks start getting set back. Europe sets their clocks back an hour this weekend. North America waits one more week before doing it. And I'm not sure about the rest of the world.17:17
*** blp has joined #openvswitch17:18
mmichelsonThe email that goes out should take that into account when announcing the meeting for next week.17:18
blpI made it!17:18
mmichelsonblp, hi!17:18
numansblp, hI17:18
imaximetshi17:18
numanssorry my caps were on :)17:18
mmichelsonAnyway, with time change stuff out of the way, we can get this thing started.17:19
*** _mdgray_ has quit IRC17:19
mmichelsonBiggest things to report are that I'm giving Anton's newest patches a review and I'm reworking the unit test patch series based on Ilya's feedback17:19
_lore_hi all17:19
*** mdgray has joined #openvswitch17:19
mmichelsonAnd that's all I have to report. I also plan to review Numan's loopback optimization patches sometime real soon17:19
numansI can go real quick next17:22
numansI worked on the load balancer hairpin lflow improvement patches and submitted for review yesterday17:22
numansthanks mmichelson for planning to take a look17:22
numansI did some code reviews today.17:23
numansone of the OVS patch broke the OVN compilation. I submitted a patch for that - https://patchwork.ozlabs.org/project/ovn/patch/20201022155339.300989-1-numans@ovn.org/17:23
blpI have a brief report when there's an opportunity.17:24
numansI think we need to discuss about this on how to fix it in OVN.17:24
numansThat's it from myside.17:24
numansblp, go ahead.17:24
imaximetsnumans, IMHO, we need to support building OVS with all supported versions of OVS. at least starting from 2.13 if possible.17:24
blpTwo things.17:25
mmichelsonimaximets, agreed17:25
blpFirst, we got 20 submissions for ovscon (not 17 as I originally thought). I think we're on track to announce the accepted talks by the end of the month.17:25
blpSecond, I'm starting to push ddlog patches toward OVN. It's a lot of patches, so I'm doing them in batches. I sent out the first batch last night. They should be easy to review because they are only documentation improvements.17:26
blpThe documentation improvements relate to the current state of OVN. They are not ddlog specific.17:26
*** donhw has quit IRC17:27
blpI'd appreciate it if people could take a look at them and pass along their feedback. Once those are in, I'll prepare and send the next batch, which will actually include some code changes.17:27
mmichelsonblp, numan acked the doc patches and I pushed them to master earlier today17:28
*** donhw has joined #openvswitch17:28
mmichelsonblp, so we at least got that part done :)17:28
blpoh! That was fast.17:28
blpI'll prepare the second batch today then. Honestly I was expecting it to take longer.17:28
blpBut it's fantastic that it was quick.17:28
blpThat's all from me.17:28
mmichelsonDoc patches are pretty easy to review :)17:28
blpWell, if they're correct, anyhow. I hope that was reviewed :-)17:29
blpSome of these were me figuring things out from code and commit messages, and who knows whether I misunderstood something.17:29
mmichelsonblp, it all looked correct to me17:29
blpyay!17:29
imaximetsblp, now it's documented and we'll change the code to correspond. :D17:30
numansblp, shall we hold on merging any patches to OVN mainly the ones which change ovn-northd ? until your ddlog patches are reviewed and merged ?17:31
blpnumans: If you can afford to do that, it would be great!17:32
numansI'm fine with it17:32
blpI've been struggling to keep up.17:32
numansothers, does that sounds reasonable ?17:32
numansblp, I understand. with many changes, you have to rework again. And that could be frustrating.17:33
*** zhouhan_ has quit IRC17:34
blpSomeone else wants to speak?17:34
*** zhouhan has joined #openvswitch17:34
imaximetsI have a small update.17:35
zhouhannumans: maybe holding on new features, but allowing bug fixes.17:35
blpzhouhan: +1 on that17:35
zhouhanimaximets: sorry, please go ahead17:35
imaximetsI'm swamped into memory consumption issues of ovsdb-server.  Technically, one thing that bothers the most is that ovsdb-server doesn't free memory back to system if we're cleaning up the database.17:36
blpimaximets: You mean ovsdb-server calls free() but that doesn't return memory to the kernel?17:37
imaximetsIt seems like reusing most of this memory later if we're performing same tests, ubt not always.17:37
blp(That's a very old C FAQ.)17:37
imaximetsblp, no, it seems like not calling free() on some memory.17:37
blpimaximets: Oh! That's more of an issue.17:37
imaximetsbut I'm not sure right now.17:37
imaximetsblp, right now I have an instance of ovsdb-server that consumes 13GB with empty database.17:38
blpI think ovsdb-server has some features for memory statistics.17:38
blpAt least, ovs-vswitchd does, I don't recall whether we put the same thing into ovsdb-server.17:38
blp13GB!17:38
blpwow17:38
imaximetsblp, these are not enough.  I'm extending them.17:38
blpgreat!17:38
imaximetsblp, I suppose, these are raft or ovsdb transaction logs.17:39
blpProbably. The raft transaction log stays in memory until it gets snapshotted. You could force a snapshot and see what happens.17:39
imaximetsblp, I did.  Does not help.  So, I'm digging different places and trying to get more statistics right now.17:40
blpThe C heap is so opaque.17:41
imaximetsthe test is to create one port group and add few thousands  of acls one by one.  This generates huge trafic since each transaction is a few hundreds of kilobytes.17:41
imaximetsinvestigating.17:42
zhouhanimaximets: does 13GB reflect the peak usage (data or raft log) in your test? If you start new tests with the empty DB does it increase before the data/log size get large enough?17:42
imaximetszhouhan, memory consumption continuously grows while adding new acls to port group and stays at the peak forever.  Even after removin all acls and the port group.  and after db compaction.17:43
zhouhanimaximets: yes, but the 13GB doesn't grow until you have enough new data that exceeds the previous peak, right?17:44
imaximetszhouhan, it depends.  I'm not sure.  In most cases it doesn't grow if I'm starting the test again.  But there are cases where it grows higher.17:45
zhouhanok. thanks! I am trying to understand is it something like memory leak which is critical for production, or just something to be optimized, but it seems not quite clear yet. Will see what you find later.17:46
imaximetsI'll continue my investigation and will send something on a mail list, likely.17:46
zhouhanthanks imaximets17:47
imaximetsAnd I'll be on PTO next week. :)17:47
imaximetsThat's it form me.17:47
zhouhanimaximets: no worries, we still have enough memory17:47
zhouhan(I've nothing to report except some reviews)17:48
numanszhouhan, sounds good to me accept just bug fixes.17:49
*** blp has left #openvswitch17:49
*** blp has joined #openvswitch17:49
blpAnyone else?17:50
imaximetsmaybe one more thing17:50
imaximetsThe issue that OVN depends on non-exported sources of OVS that leads to OVN build failures.17:51
imaximetsi.e. non-public OVS parts.17:51
blpYes, that's nasty.17:51
blpThere are multiple possible solutions. All of them require work.17:52
imaximetsI just think tht we need a policy for this kind of things, to not discuss each time17:52
numansright now OVN compilation is broken after this commit - 91fc374a9c5a("Eliminate use of term "slave" in bond, LACP, and bundle contexts.") in OVS17:53
imaximetsmmichelson seems to agree that we need to support OVN build with all stable OVS versions starting from 2.13, if possible.17:53
numansmmichelson During the split we did discuss about this right ?17:54
numansthere are two things, one is run time compatibility and other compilation.17:54
numansI'm not sure if we want to support compilation of OVN with older versions of OVS as it could be hard to maintain that17:55
imaximetsBut this is hard to handle, because OVN releases are more frequent.17:56
numansI think if some one is compiling OVN from master, he/she can pick up the latest OVS.17:56
numansjust for compilation.17:56
blpOVN shouldn't use non-public bits of OVS. Either OVS should export the bits it needs, or OVN should copy-paste the bits it uses, I think.17:56
mmichelsonnumans, imaximets Thinking about it again, the most important thing is for OVN to be able to run against older 2.13 and newer versions of OVS. When it comes to compilation, it's not nearly as important to be compatible with older versions17:56
numansmmichelson, that's what I think too.17:57
imaximetsmmichelson, but building with unstable OVS sources doesn't seem like a good idea.17:57
imaximetsor we need to say that OVN builds with latest stable release and not support building with master.17:58
imaximets* latest stabel OVS release17:58
mmichelsonimaximets, But then we run into the issue that may happen where a change we make in OVN requires an OVS change as well. I guess we can go with Anton's method for handling that right now.17:59
mmichelsonAt least that will work for new functionality.17:59
blpOK, I'm switching to a different meeting now. Thanks so much for the reviews, especially! I'll post another series later today.18:00
mmichelsonIf you're not sure what I mean, I can link Anton's paralellization patch. Just a sec.18:00
numansI think for compilation we should not complicate things or add #ifdef code which checks the ovs version and compile conditionally.18:00
imaximetsmmichelson, numans: is conditional compilation that bad?18:00
numansblp, Bye18:00
mmichelsonSee the bottom of: https://patchwork.ozlabs.org/project/ovn/patch/20201014162714.5978-1-anton.ivanov@cambridgegreys.com/18:00
*** blp has left #openvswitch18:00
numansimaximets, I think in a long run it would complicate the code.18:01
zhouhanIn fact it seems even runtime compatibility is tricky. In redhat it may be fine but for ubuntu since deb package uses dynamic .so libs, would this be concerned?18:01
mmichelsonYes that can be a problem.18:02
numansI think if some one wants to compile OVN from sources, he/she can definitely pull up latest OVS sources too.18:03
numansfor production anyway, a user is expected to consume OVN from distributions.18:03
numansand stable versions of OVN.18:03
mmichelsonCorrect. We just need to ensure that we don't break debian packages.18:04
imaximetsnumans, how can I find out which version of OVS is compatible with specific OVN commit?18:04
numansIn the case of fedora which I maintain, if compilation breaks due to OVS, I usually sync to the latest OVS or backport such patches.18:05
zhouhanNow that we release OVN more frequently than OVS, does it mean the mid-cycle release of OVN is regarded unstable (because it may depend on unreleased OVS?)18:05
numansimaximets, we discussed that during split. During a release, OVN would mention the commit of OVS.18:05
imaximetsnumans, so how can I bisect issues if my OVN bisect will always require different versions of OVS and I do not even know which exactly?18:06
numansimaximets, you mean for production deployments ?18:06
imaximetsnumans, doesn't matter.18:06
numansimaximets, when you compile OVN, you need to provide the OVS sources right ?18:07
numansas a developer if you're compiling OVN, you would probably know which OVS version of code base is used for compilation.18:07
zhouhanimaximets: I guess now bisect means both OVN and OVS (it is not a bisect anymore). That's the case when I debugged the northd performance regression caused by an OVSDB IDL change.18:08
imaximetsnumans, right.  Assuming I have a bug and I'm truing to identify the commit that introduced it.  I'm using git bisect, but I can not automate it and I need to find out correct OVS commit to build with each time.18:08
imaximetszhouhan, at least you were able to use OVS master with all your OVN versions higher that 2.12.18:09
numansimaximets,  I don't think we will have a straightforward answer until we either use a openvswitch lib or copy the ovs bits to OVN18:09
mmichelsonWe've been doing the separate OVN builds for nearly a year now. I think the war stories you have imaximets can factor into how we manage the project going forward18:09
numansas suggested by Ben18:09
mmichelsonIt may be that we need to pin our builds to a specific version of OVS, in order to stabilize things and make debugging easier.18:10
mmichelsonnumans, which OVS bits would we copy to OVN?18:10
numansimaximets, in the case of RHEL and fedora we would be knowing which OVS version/tar ball is used.18:10
numansmmichelson, this is what Ben said a while back - <blp> OVN shouldn't use non-public bits of OVS. Either OVS should export the bits it needs, or OVN should copy-paste the bits it uses, I think.18:11
mmichelsonnumans, OK, it probably makes sense to make the "non-public" pieces public then. In my view, there's no much reason why hmap is public but smap is not.18:11
mmichelsonAs an example18:11
mmichelsonAssuming that "public" refers to headers in include/ and "non-public" refers to headers in lib/18:12
numansmmichelson, we also use lib/packets.h too.18:13
numansall that needs to be moved to include/ of ovs.18:13
imaximetsnumans, mmichelson: making things public will require versioning anyway and OVN will have to detect version of OVS public libraries.18:13
imaximetsand stick to specific versions or support multiple with conditional compilation.18:15
zhouhanOr, if OVN depends on a new feature of OVS which is not released, we could create a branch for the stable OVS release and backport the new features to the stable OVS branch. This way we can avoid using unstable OVS as much as possible.18:15
mmichelsonimaximets, I agree that making things public still requires versioning. You have to crawl before you can walk or run, though. So step one would be to make the library we need to consume. Step 2 would be to version this API.18:16
numansThe big question is how we want to handle the compilation issue now ? I submitted a patch here - https://patchwork.ozlabs.org/project/ovn/patch/20201022155339.300989-1-numans@ovn.org/18:16
numansif we accept the submitted patch, then we release ovn 20.12 we need to consume OVS from unstable branch for "compilation".18:17
mmichelsonnumans, the current attitude with OVN is to do exactly what you did. Compile against the latest OVS, even if it's unstable.18:17
mmichelsonnumans, but going forward, we probably need to come up with a more structured way of doing things for our own sanity18:17
imaximetsnumans, mmichelson, zhouhan: what about git submodule?  Was this option discussed during the split?18:17
imaximetsI mean submodules are fixed to specific commit hash.18:18
*** elvira has quit IRC18:18
zhouhanimaximets: that was discussed, but I remember it was dropped because of other drawbacks18:19
zhouhanmaybe mmichelson numans remember more details18:20
numansAnd the agreement at that time was to compile OVN master from OVS master and when OVN is released we document the OVS commit to use to.18:20
numansI think until now we didn't face a compilation issue so far in which released OVN depends on OVS master.18:21
imaximetssubmodules are not pretty, but it's less evil if we're sticking to OVS master for compilation anyway.  At least we will know against which version we're building exactly.18:21
numansor unreleased OVS18:21
zhouhanimaximets: Would OVS stable branch with backporting work? It solves the compile problem (and also the debian runtime problem, probably), but has lower risk than using OVS master18:22
mmichelsonyeah we went the submodule way (actually I think we used subtree, but it's very similar) as a test, but ended up not going that way because it took us away from the long-term goal of being able to just install an ovs-devel package on the system and compile against that.18:22
numansI think its better if we work towards having a ovs-devel package and then OVN using it for compilation.18:24
numansI got to sign off now.18:25
numansBye18:25
mmichelsonOK18:25
mmichelsonI think we can continue this discussion over email. For the record, I like zhouhan's idea of having a clone of a stable OVS branch that we can backport fixes/necessary features to.18:26
mmichelsonIncluding that branch as a submodule or otherwise requiring it for compilation would be the next step18:26
numansmmichelson, may be we can  also think of cloning ovs repo in the ovn-org git and use it for compilation.18:27
numansJust a thought.18:27
mmichelsonnumans, yeah that's a good place for the backport to live18:27
zhouhanyes18:27
zhouhanWe can continue discussing in ML18:27
numanssounds good.18:28
numansBye18:28
zhouhanI need to drop, too. Bye folks18:28
mmichelsonWe've gone pretty long today for this meeting, so I'm going to end it.18:28
mmichelson#endmeeting18:28
openstackMeeting ended Thu Oct 22 18:28:14 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:28
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-10-22-17.16.html18:28
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-10-22-17.16.txt18:28
openstackLog:            http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-10-22-17.16.log.html18:28
imaximetsOK.  Bye.18:28
mmichelsonOK, now for the moment of truth. Does my python script that sends the email meeting know about the Europe time change this weekend...18:28
mmichelsonSuccess!18:28
mmichelsonWriting a time library has got to be a giant PITA18:29
*** mdgray has quit IRC18:47
*** dceara has quit IRC19:22
*** cpaelzer__ has quit IRC20:07
*** cpaelzer__ has joined #openvswitch20:08
*** slaweq has quit IRC20:26
*** dceara has joined #openvswitch20:57
*** slaweq has joined #openvswitch21:08
*** bostondriver has quit IRC21:21
*** dcbw has quit IRC21:23
*** mutin-s has quit IRC21:25
*** mutin-s has joined #openvswitch21:26
*** mutin-s has quit IRC21:26
*** slaweq has quit IRC21:27
*** dceara has quit IRC21:39
*** dceara has joined #openvswitch22:00
*** rcernin has joined #openvswitch22:31
*** rcernin has quit IRC22:49
*** rcernin has joined #openvswitch22:53
*** dceara has quit IRC23:01
*** tbachman has quit IRC23:40
*** tbachman has joined #openvswitch23:41
*** tbachman_ has joined #openvswitch23:44
*** tbachman has quit IRC23:47
*** tbachman_ is now known as tbachman23:47

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!