Thursday, 2021-03-11

*** crabbey has joined #openvswitch00:14
*** fab23 has joined #openvswitch01:17
*** fab23_ has quit IRC01:17
*** dmellado has quit IRC01:25
*** dmellado has joined #openvswitch01:52
*** rcernin has quit IRC02:01
*** rcernin has joined #openvswitch02:05
*** rcernin has quit IRC02:39
*** rcernin has joined #openvswitch02:42
*** rcernin has quit IRC02:51
*** rcernin has joined #openvswitch03:05
*** rcernin has quit IRC03:12
*** acidfu has joined #openvswitch03:22
*** rcernin has joined #openvswitch03:23
*** acidfoo_ has quit IRC03:23
*** rcernin has quit IRC03:24
*** rcernin has joined #openvswitch03:24
*** acidfu has quit IRC03:35
*** fdangelo_ has joined #openvswitch04:36
*** fdangelo has quit IRC04:39
*** gizmoguy has quit IRC04:40
*** anilvenkata has joined #openvswitch05:35
*** gizmoguy has joined #openvswitch05:49
*** gizmoguy has quit IRC06:14
*** gizmoguy has joined #openvswitch06:15
*** thaller has joined #openvswitch06:27
*** ralonsoh has joined #openvswitch06:51
*** dholler has joined #openvswitch06:52
*** eelco has joined #openvswitch07:00
*** slaweq has joined #openvswitch07:13
*** jaicaa has quit IRC07:33
*** jaicaa has joined #openvswitch07:34
*** rcernin has quit IRC07:40
*** rcernin has joined #openvswitch07:47
*** links has joined #openvswitch07:55
*** gizmoguy has quit IRC08:01
*** gizmoguy has joined #openvswitch08:02
*** rcernin has quit IRC08:06
*** rcernin has joined #openvswitch08:17
*** mdgray has joined #openvswitch08:19
*** elvira has joined #openvswitch08:35
*** dholler has quit IRC08:44
*** dholler has joined #openvswitch08:57
*** rcernin has quit IRC09:01
*** rcernin has joined #openvswitch09:28
*** zhouhan has joined #openvswitch09:28
*** blahdodo has quit IRC09:47
*** rcernin has quit IRC09:47
*** blahdodo has joined #openvswitch09:49
*** istokes has joined #openvswitch09:54
*** rcernin has joined #openvswitch10:04
*** rcernin has quit IRC10:20
*** jangutter_ has quit IRC10:41
*** jangutter has joined #openvswitch10:42
*** rcernin has joined #openvswitch11:07
*** jangutter has quit IRC11:07
*** jangutter has joined #openvswitch11:07
*** jangutter has quit IRC11:42
*** jangutter has joined #openvswitch11:42
*** rcernin has quit IRC11:58
*** jangutter_ has joined #openvswitch11:59
*** jangutter has quit IRC12:02
*** rcernin has joined #openvswitch12:23
*** rcernin has quit IRC12:29
*** acidfu has joined #openvswitch12:37
*** rcernin has joined #openvswitch12:41
*** rcernin has quit IRC12:46
*** rcernin has joined #openvswitch12:52
*** rcernin has quit IRC12:57
*** rcernin has joined #openvswitch12:58
*** openstack has joined #openvswitch13:17
*** bostondriver has joined #openvswitch13:29
*** jangutter has joined #openvswitch13:45
*** jangutter_ has quit IRC13:48
*** thaller has quit IRC13:56
*** thaller has joined #openvswitch13:56
*** fdangelo_ has joined #openvswitch14:29
*** jangutter_ has joined #openvswitch14:29
*** fdangelo__ has quit IRC14:32
*** jangutter has quit IRC14:32
*** Limech has joined #openvswitch14:37
*** Limech has quit IRC14:37
*** rcernin has joined #openvswitch14:59
*** rcernin has quit IRC15:04
*** teardown_ has joined #openvswitch15:06
*** teardown has quit IRC15:06
*** jangutter has joined #openvswitch15:52
*** jangutter_ has quit IRC15:54
*** bern- is now known as bern115:57
*** jangutter_ has joined #openvswitch16:37
*** jangutter has quit IRC16:40
*** dceara has joined #openvswitch16:59
*** rcernin has joined #openvswitch17:00
*** fnordahl has joined #openvswitch17:11
*** rcernin has quit IRC17:16
*** eelco has quit IRC17:23
*** jangutter has joined #openvswitch17:26
*** jangutter_ has quit IRC17:30
*** elvira has quit IRC17:34
*** istokes has quit IRC17:49
*** jangutter_ has joined #openvswitch17:53
*** jangutter has quit IRC17:57
*** karthikc has joined #openvswitch18:02
*** blp has joined #openvswitch18:07
blpHello all.18:07
blpAlmost time for the meeting.18:07
mmichelsonHi blp18:11
mmichelsonblp, your patch series with ddlog improvements is a good way for me to learn about it more.18:11
blpI have a report I'm looking forward to giving when the time comes.18:11
*** dhathri has joined #openvswitch18:11
blpmmichelson: I'm glad to hear that!18:11
blpmmichelson: I plan to write up some additional ddlog documentation/advice sometime soon.18:12
blpmmichelson: Also, there's ddlog IDE integration that should be available soon as a "language server". I believe that it can integrate with multiple editors and IDEs, including vim and Emacs, but I haven't personally tried it yet.18:12
mmichelsonCool, I've used the vim syntax highlighting18:13
blpIs there some vim syntax highlighting suitable for ddlog?18:14
mmichelsonYeah it's pointed to by the README in the ddlog repo18:14
mmichelsonI guess it's about time to start18:15
mmichelson#startmeeting ovn_community_development_meeting18:15
openstackMeeting started Thu Mar 11 18:15:23 2021 UTC and is due to finish in 60 minutes.  The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot.18:15
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.18:15
openstackThe meeting name has been set to 'ovn_community_development_meeting'18:15
blpAh. I guess I never noticed that because I don't use vim much.18:15
mmichelsonBLASPHEMY18:15
mmichelsonAnyways, hi everyone!18:15
blp(I can use it effectively, it's just not my preference.)18:16
blphi everyone!18:16
mmichelsonI was hoping to start the meeting off by getting an update about what (if any) OVS patches we might be waiting on before we can release 21.03.018:16
_lore_hi all18:16
dcearahi18:16
mmichelsonAre there any patches we're waiting on before we can release? If not, then I'd like to release tomorrow.18:17
*** ihrachys has quit IRC18:17
dcearammichelson: There's the IDL bug I'm trying to fix (v2 here: https://patchwork.ozlabs.org/project/openvswitch/list/?series=231872&state=*)18:18
dcearammichelson: I have v3 almost ready, I've been struggling a bit today with an OVN test that seems to be failing more often with my IDL changes.  I'm still not sure if it's a flaky test or not.18:19
blpdceara: Which test?18:19
*** ihrachys has joined #openvswitch18:19
dcearablp: 139: ovn -- 4 HV, 1 LS, 1 LR, packet test with HA distributed router gateway port -- ovn-northd FAILED (ovs-macros.at:25318:19
blpThat one doesn't ring a bell for me, so maybe I haven't seen a lot of problems with it.18:20
blpI've spent a lot of time trying to figure out whether some tests are flaky, so some of them are familiar.18:20
blpUgh I wish we didn't need end-to-end tests so badly, they're so hard to make reliable.18:20
dcearablp: I think I saw this test failing in the past but with my change it fails more often.  So I'd like to make sure it's not another IDL bug that I'm introducing/uncovering.18:21
mmichelsonUnfortunately, I can't with any confidence say that dceara's issue is a flaky test. It's also possible due to weirdness in some OVN component that things process in a slightly different order sometimes and cause a problem.18:21
dcearammichelson: That's a possibility too.18:22
* mmichelson has flashbacks to a CT zone issue from the past18:22
numanshello18:22
blpFor ovn-northd-ddlog, I often compare the southbound flow table dump against ovn-northd.18:22
numansmmichelson, I'd say we should go ahead and release tomorrow.18:22
numansWe can probably updte the ovs submodule commit once the patch is merged.18:23
blpFor your change, you might be able to compare dumps of sbflows or something else, with and without your change.18:23
* numans oops. sorry for jumping in.18:23
dcearablp: Yes, will do.18:23
blpMany of the tests now dump southbound flows to a file 'sbflows' to make this easier.18:23
dcearablp: The main problem though is that the test fails in github CI.  On my machine it mostly passes.18:24
mmichelsonWhat I found in the CT zone issue I referred to was that sometimes I'd have multiple ovn-nbctl commands that would get handled by ovn-northd as one operation, and other times they'd get handled as two separate transactions. Then this would result in ovn-controller either processing it all as one change vs. as two separate changes. And depending on which happened, we'd have different behavior. And yes, if you guessed it was an18:24
mmichelsonincremental processing issue, you would be correct.18:24
blpdceara: That makes it harder.18:25
dceara:)18:25
blpdceara: I often see tests fail when I run them with high parallelism, e.g. TESTSUITEFLAGS=-j12 on a 6-core laptop.18:25
blpdceara: If you don't already try that, it's worth trying.18:26
dcearablp: Same here, i've been trying with -j20 on a dev server I'm using.  Not so much luck.18:26
blp(At some point I might switch to developing on my 3990x box and then I'll use -j128.)18:26
*** imaximets_ is now known as imaximets18:26
imaximetsmmichelson, if we're going to release tomorrow than we should move submodule now to ac09cbfcb70a ("ovsdb-cs: Fix use-after-free for the request id.") before the release.18:26
mmichelsonimaximets, ack18:28
mmichelsonAnyways, aside from release talk, I didn't have anything else I wanted to share. I can update the ovs submodule and put that up for review after this meeting18:29
blpIs there more to discuss on this topic or shall we move on?18:29
mmichelsonAnd after that, I can post the release commits as well18:29
mmichelsonSo taht should allow for us to release tomorrow.18:29
mmichelsonblp, I think that's it, if you'd like to go next.18:30
numans+118:30
blpCool.18:30
imaximetsmmichelson, I'll send a patch for submodule shortly.18:30
blpI posted a series of improvements to ovn-northd-ddlog last week.18:30
blpNuman pointed out some issues that I should have noticed, but had not.18:30
blpI have posted a number of small fixes that can apply separately from that series.18:30
blpAnd I will also post a revision of the series itself soon.18:30
blpBut, more excitingly, Leonid spent some time optimizing the ddlog code18:31
blpand he gave me a tarball of the changes last night18:31
blpwhich I will look at and probably post (maybe add to my series?).18:31
blpThey are pretty small changes and he says they obtain more than 3x performance improvement18:31
blpover the improvement I had gained in my series.18:31
mmichelsonThat's impressive!18:32
blpHe also says that with them the cost of each step in the benchmark that Numan provided seems to go to O(1) rather than increasing with each iteration.18:32
numansthat's cool.18:32
blpOnce we get that in, I would really appreciate it if people would throw more benchmark challenges at me.18:32
blpI think that I understand the optimization principles that Leonid used.18:32
blpI'm going to apply them myself18:32
numansup for the challenge :)18:33
blpand I'm going to try to write them up for others to understand as well.18:33
blpnumans: awesome18:33
mmichelsonblp, just so I'm clear, are these changes from Leonid to DDLog the language or to ovn-northd-ddlog?18:33
blpThey are mainly to ovn-north-ddlog.18:33
mmichelsonGot it18:34
blpLeonid did add a small feature to the ovsdb2ddlog program we use for generating .dl files.18:34
blpwhich is here if you want to look at it: https://github.com/vmware/differential-datalog/pull/93418:34
blpI can also report that we've hired a couple of people to work on ddlog at VMware.18:35
blpwhich is a good indication that it will be maintained.18:35
blpand enhanced18:35
blpI am done with my report but I'm happy to answer more questions if anyone has them.18:36
blp(I'm hoping that one of the new hires will write a ddlog formatter like "indent" for C or "rustfmt" for Rust.)18:36
numansblp, I had reported few memory leaks.18:37
numansI'm not sure if you've addressed them in your series.18:37
mmichelsonNo questions from me. I'm really happy to see the improvements being made to DDLog18:37
blpI posted patches to fix those, I think.18:37
numanscool then.18:37
numansI haven't looked into the patches.18:37
numansI remember dceara mentioned that in his testing northd-ddlog took huge ram like 75gb or so. I may be wrong, dceara can update if he is still here.18:38
blpnumans: Yes, the leak fixes are specifically:18:38
blphttps://mail.openvswitch.org/pipermail/ovs-dev/2021-March/381117.html18:38
blphttps://mail.openvswitch.org/pipermail/ovs-dev/2021-March/381119.html18:38
blpnumans: They should be easy to review if you have a few minutes to look at them.18:38
numanssure. I'll take a look. thanks.18:38
blpnumans: northd-ddlog does take more RAM. I think that the optimization pathces (and leak fixes) should help.18:39
numansok. The patches looks straightforward.18:39
dcearablp: It's very nice to see the ddlog related activity! I did give ovn-northd-ddlog a try on one of the large NB databases extracted from one of our scale tests.  I hope the 75gb was due to the memleaks you fixed :)18:39
blp75 GB is a lot.18:40
dcearablp: I also had a small bug report that I didn't get a chance to report on the mailing list yet: it seems that ovn-northd-ddlog clears NB_Global.options:use_logical_dp_groups after the first run, effectively disabling the feature.18:40
blpdceara: I'll look at that. Should be an easy fix.18:41
dcearablp: Yep, looked relatively straightforward indeed.18:41
zhouhanblp: Do you know what's the expected memory consumption without leak? How many times more than regular northd?18:42
blpzhouhan: I don't have an estimate for that yet. We can target memory use like we target speed, by dumping a memory profile and looking for excessive consumption then making ddlog code adjustments.18:43
zhouhanblp: ok, just want to have a rough idea about. I remember last time (1 - 2 years ago) when I was testing it, it was about 10x of regular northd18:44
blpIn other software we've built with ddlog, I think it was more like 2x or 3x after we did a little work to bring it down.18:45
zhouhanblp: 2x - 3x is much better now :)18:45
blpIt is more or less unavoidable that an incremental version of anything would take more memory than a nonincremental version.18:45
blpBut if it uses so much memory that it's unusable, obviously that's not a good tradeoff.18:45
zhouhanblp: this is reasonable. Just want to know the worse case. On central node I think 2x - 3x memory is not an issue at all. I am only thinking about the future of ovn-controller using ddlog :)18:46
blpThat is a good point. Memory is much more critical for ovn-controller.18:46
blpI used to joke that NVP (back at Nicira circa 2011) required infinite memory.18:47
blpAnd that its "nlog" language should be called "n exponential".18:47
mmichelsone^nlog18:48
blp(Optimization helped.)18:48
numansI've observed ovn-controller taking up like 9gb on a scaled env and vswitchd takes around ~2.5gb.18:48
blpHoly crap that's a lot already.18:48
dcearanumans: With the lflow cache enabled?18:48
numansdceara, yes.18:48
blpI'm embarrassed, how did my baby become a monster?18:48
zhouhansounds like a proof that it is doing I-P :D18:48
*** ralonsoh has quit IRC18:48
mmichelsonDoes anyone else want to share?18:49
imaximetsmmichelson, blp: https://patchwork.ozlabs.org/project/ovn/patch/20210311183719.2517358-1-i.maximets@ovn.org/18:49
mmichelsonimaximets, that should be an easy ack18:50
numansI can go real quick. I was busy almost this week working on a couple of crashes seen in ovn-controller18:50
numansAll thanks to me and the runtime data I-P handling :)18:50
numansI submitted the patch for review - https://patchwork.ozlabs.org/project/ovn/patch/20210311124757.2997057-1-numans@ovn.org/18:51
numanszhouhan, I couldn't reply to your questions on the ct.inv drop patch. I'll get to that next week. thanks for the comments and review so far.18:51
zhouhannumans: np. I also reviewed your I-P refactor, will review the ofctrl refactor RFC next couple of days.18:52
numansI'm also inclined to submit v2 making it as a config option so that regular users are not missed out on this, in case there are scenarios where invalid pkts needs to be dropped.18:52
numanszhouhan, thanks.18:53
blpimaximets: Thank you Ilya. I should have sent that earlier. acked.18:53
numansthat's it from me.18:54
karthikchello, can I go next? Had a question regarding upgrade18:54
numanssure.18:55
karthikchttps://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050988.html18:55
karthikcIs this upgrade supported? I seem to be hitting a backward compatibility issue18:56
karthikcwe are basically trying to move from v2.11 to ovn-20.09. If the controller is upgraded first, the chassis don't show up18:57
numanskarthikc, I saw that email. Sorry I couldn't reply.18:57
mmichelsonkarthikc, Someone can correct me if I'm wrong, but I think the upgrade order assumes that you also will update the central components too. It sounds like you've upgraded ovn-controller but not ovn-northd18:57
*** dholler has quit IRC18:58
numansthat's correct.18:58
blpmmichelson: The upgrade doc says to upgrade ovn-controller first.18:58
karthikcthat's correct. ovn-northd is yet to be upgraded. But there is a transient state where chassis don't show up18:58
numanskarthikc, is your concern that during this time, the traffic is broken ?18:58
karthikcyes18:58
numanskarthikc, recently we added a feature to pin ovn-controller to a specific version of ovn-northd18:59
numansif ovn-controller sees this mismatch, it does nothing until ovn-northd is also upgraded.18:59
numanskarthikc, I'll reply about that in the ML.18:59
numansI'm not sure if that commit is 20.09 or 20.1218:59
karthikcneat, that will be helpful, thanks18:59
mmichelsonI think it was 20.1219:00
mmichelsonBut I might be wrong19:00
numansmmichelson, you may be right.19:00
numanskarthikc, may be you can consider uprading to 20.1219:00
karthikcok19:00
dhathrihi, can i go real quick next?19:01
mmichelsonThe other option is to use the --restart option when restarting ovn-controller so that ovn-controller doesn't "clean up" when you stop it. That way when it starts back up, it should be in the same configuration as when it was running previously. You at least shouldn't have interrupted traffic if you do that. But you also won't be able to make changes.19:01
mmichelsondhathri, go for it19:01
dhathrijust wanted to request for review on the multiple gateway router support patch (https://mail.openvswitch.org/pipermail/ovs-dev/2021-March/380979.html)19:02
numanslooks like we need to be careful when we add new columns which ovn-controlller needs to update.19:02
numansdhathri, sure. Meanwhile could you also plan for another version with ddlog support ?19:02
numansdhathri, I'll take a look next week.19:03
dhathrisure, thanks.. ddlog will take more time since there is a learning curve. Wanted to get the c patch reviewed while i am working on the ddlog changes19:03
blpI'm in another meeting now, so I'll talk to everyone next week.19:04
mmichelsonbye blp19:04
*** blp has quit IRC19:04
mmichelsonDoes anyone else have a report?19:05
mmichelsonI'll take the lack of response to mean "no"19:06
mmichelsonBye everyone19:06
zhouhan bye!19:06
dcearaBye19:06
karthikcbye19:06
imaximetsbye19:06
mmichelson#endmeeting19:06
openstackMeeting ended Thu Mar 11 19:06:39 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)19:06
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-03-11-18.15.html19:06
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-03-11-18.15.txt19:06
openstackLog:            http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-03-11-18.15.log.html19:06
*** dhathri has quit IRC19:07
*** karthikc has left #openvswitch19:08
*** tbachman has quit IRC19:26
*** tbachman has joined #openvswitch19:30
*** links has quit IRC19:44
*** dalvarez has quit IRC19:47
*** dalvarez has joined #openvswitch19:47
*** rcernin has joined #openvswitch20:00
*** mdgray has quit IRC20:06
*** anilvenkata has quit IRC20:24
*** acidfoo has joined #openvswitch20:33
*** acidfu has quit IRC20:34
*** slaweq has quit IRC20:56
*** rcernin has quit IRC21:04
*** rcernin has joined #openvswitch21:35
*** dceara has quit IRC21:48
*** rcernin has quit IRC21:51
*** rcernin has joined #openvswitch21:54
*** acidfoo has quit IRC21:56
*** rcernin has quit IRC22:00
*** rcernin has joined #openvswitch22:00
*** bostondriver has quit IRC22:09
*** circ-user-qLPh6 has joined #openvswitch22:48
*** zhouhan has quit IRC23:04
*** mmichelson has quit IRC23:17
*** mmichelson has joined #openvswitch23:18

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!