*** crabbey has joined #openvswitch | 00:14 | |
*** fab23 has joined #openvswitch | 01:17 | |
*** fab23_ has quit IRC | 01:17 | |
*** dmellado has quit IRC | 01:25 | |
*** dmellado has joined #openvswitch | 01:52 | |
*** rcernin has quit IRC | 02:01 | |
*** rcernin has joined #openvswitch | 02:05 | |
*** rcernin has quit IRC | 02:39 | |
*** rcernin has joined #openvswitch | 02:42 | |
*** rcernin has quit IRC | 02:51 | |
*** rcernin has joined #openvswitch | 03:05 | |
*** rcernin has quit IRC | 03:12 | |
*** acidfu has joined #openvswitch | 03:22 | |
*** rcernin has joined #openvswitch | 03:23 | |
*** acidfoo_ has quit IRC | 03:23 | |
*** rcernin has quit IRC | 03:24 | |
*** rcernin has joined #openvswitch | 03:24 | |
*** acidfu has quit IRC | 03:35 | |
*** fdangelo_ has joined #openvswitch | 04:36 | |
*** fdangelo has quit IRC | 04:39 | |
*** gizmoguy has quit IRC | 04:40 | |
*** anilvenkata has joined #openvswitch | 05:35 | |
*** gizmoguy has joined #openvswitch | 05:49 | |
*** gizmoguy has quit IRC | 06:14 | |
*** gizmoguy has joined #openvswitch | 06:15 | |
*** thaller has joined #openvswitch | 06:27 | |
*** ralonsoh has joined #openvswitch | 06:51 | |
*** dholler has joined #openvswitch | 06:52 | |
*** eelco has joined #openvswitch | 07:00 | |
*** slaweq has joined #openvswitch | 07:13 | |
*** jaicaa has quit IRC | 07:33 | |
*** jaicaa has joined #openvswitch | 07:34 | |
*** rcernin has quit IRC | 07:40 | |
*** rcernin has joined #openvswitch | 07:47 | |
*** links has joined #openvswitch | 07:55 | |
*** gizmoguy has quit IRC | 08:01 | |
*** gizmoguy has joined #openvswitch | 08:02 | |
*** rcernin has quit IRC | 08:06 | |
*** rcernin has joined #openvswitch | 08:17 | |
*** mdgray has joined #openvswitch | 08:19 | |
*** elvira has joined #openvswitch | 08:35 | |
*** dholler has quit IRC | 08:44 | |
*** dholler has joined #openvswitch | 08:57 | |
*** rcernin has quit IRC | 09:01 | |
*** rcernin has joined #openvswitch | 09:28 | |
*** zhouhan has joined #openvswitch | 09:28 | |
*** blahdodo has quit IRC | 09:47 | |
*** rcernin has quit IRC | 09:47 | |
*** blahdodo has joined #openvswitch | 09:49 | |
*** istokes has joined #openvswitch | 09:54 | |
*** rcernin has joined #openvswitch | 10:04 | |
*** rcernin has quit IRC | 10:20 | |
*** jangutter_ has quit IRC | 10:41 | |
*** jangutter has joined #openvswitch | 10:42 | |
*** rcernin has joined #openvswitch | 11:07 | |
*** jangutter has quit IRC | 11:07 | |
*** jangutter has joined #openvswitch | 11:07 | |
*** jangutter has quit IRC | 11:42 | |
*** jangutter has joined #openvswitch | 11:42 | |
*** rcernin has quit IRC | 11:58 | |
*** jangutter_ has joined #openvswitch | 11:59 | |
*** jangutter has quit IRC | 12:02 | |
*** rcernin has joined #openvswitch | 12:23 | |
*** rcernin has quit IRC | 12:29 | |
*** acidfu has joined #openvswitch | 12:37 | |
*** rcernin has joined #openvswitch | 12:41 | |
*** rcernin has quit IRC | 12:46 | |
*** rcernin has joined #openvswitch | 12:52 | |
*** rcernin has quit IRC | 12:57 | |
*** rcernin has joined #openvswitch | 12:58 | |
*** openstack has joined #openvswitch | 13:17 | |
*** bostondriver has joined #openvswitch | 13:29 | |
*** jangutter has joined #openvswitch | 13:45 | |
*** jangutter_ has quit IRC | 13:48 | |
*** thaller has quit IRC | 13:56 | |
*** thaller has joined #openvswitch | 13:56 | |
*** fdangelo_ has joined #openvswitch | 14:29 | |
*** jangutter_ has joined #openvswitch | 14:29 | |
*** fdangelo__ has quit IRC | 14:32 | |
*** jangutter has quit IRC | 14:32 | |
*** Limech has joined #openvswitch | 14:37 | |
*** Limech has quit IRC | 14:37 | |
*** rcernin has joined #openvswitch | 14:59 | |
*** rcernin has quit IRC | 15:04 | |
*** teardown_ has joined #openvswitch | 15:06 | |
*** teardown has quit IRC | 15:06 | |
*** jangutter has joined #openvswitch | 15:52 | |
*** jangutter_ has quit IRC | 15:54 | |
*** bern- is now known as bern1 | 15:57 | |
*** jangutter_ has joined #openvswitch | 16:37 | |
*** jangutter has quit IRC | 16:40 | |
*** dceara has joined #openvswitch | 16:59 | |
*** rcernin has joined #openvswitch | 17:00 | |
*** fnordahl has joined #openvswitch | 17:11 | |
*** rcernin has quit IRC | 17:16 | |
*** eelco has quit IRC | 17:23 | |
*** jangutter has joined #openvswitch | 17:26 | |
*** jangutter_ has quit IRC | 17:30 | |
*** elvira has quit IRC | 17:34 | |
*** istokes has quit IRC | 17:49 | |
*** jangutter_ has joined #openvswitch | 17:53 | |
*** jangutter has quit IRC | 17:57 | |
*** karthikc has joined #openvswitch | 18:02 | |
*** blp has joined #openvswitch | 18:07 | |
blp | Hello all. | 18:07 |
---|---|---|
blp | Almost time for the meeting. | 18:07 |
mmichelson | Hi blp | 18:11 |
mmichelson | blp, your patch series with ddlog improvements is a good way for me to learn about it more. | 18:11 |
blp | I have a report I'm looking forward to giving when the time comes. | 18:11 |
*** dhathri has joined #openvswitch | 18:11 | |
blp | mmichelson: I'm glad to hear that! | 18:11 |
blp | mmichelson: I plan to write up some additional ddlog documentation/advice sometime soon. | 18:12 |
blp | mmichelson: Also, there's ddlog IDE integration that should be available soon as a "language server". I believe that it can integrate with multiple editors and IDEs, including vim and Emacs, but I haven't personally tried it yet. | 18:12 |
mmichelson | Cool, I've used the vim syntax highlighting | 18:13 |
blp | Is there some vim syntax highlighting suitable for ddlog? | 18:14 |
mmichelson | Yeah it's pointed to by the README in the ddlog repo | 18:14 |
mmichelson | I guess it's about time to start | 18:15 |
mmichelson | #startmeeting ovn_community_development_meeting | 18:15 |
openstack | Meeting started Thu Mar 11 18:15:23 2021 UTC and is due to finish in 60 minutes. The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:15 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:15 |
openstack | The meeting name has been set to 'ovn_community_development_meeting' | 18:15 |
blp | Ah. I guess I never noticed that because I don't use vim much. | 18:15 |
mmichelson | BLASPHEMY | 18:15 |
mmichelson | Anyways, hi everyone! | 18:15 |
blp | (I can use it effectively, it's just not my preference.) | 18:16 |
blp | hi everyone! | 18:16 |
mmichelson | I was hoping to start the meeting off by getting an update about what (if any) OVS patches we might be waiting on before we can release 21.03.0 | 18:16 |
_lore_ | hi all | 18:16 |
dceara | hi | 18:16 |
mmichelson | Are there any patches we're waiting on before we can release? If not, then I'd like to release tomorrow. | 18:17 |
*** ihrachys has quit IRC | 18:17 | |
dceara | mmichelson: There's the IDL bug I'm trying to fix (v2 here: https://patchwork.ozlabs.org/project/openvswitch/list/?series=231872&state=*) | 18:18 |
dceara | mmichelson: I have v3 almost ready, I've been struggling a bit today with an OVN test that seems to be failing more often with my IDL changes. I'm still not sure if it's a flaky test or not. | 18:19 |
blp | dceara: Which test? | 18:19 |
*** ihrachys has joined #openvswitch | 18:19 | |
dceara | blp: 139: ovn -- 4 HV, 1 LS, 1 LR, packet test with HA distributed router gateway port -- ovn-northd FAILED (ovs-macros.at:253 | 18:19 |
blp | That one doesn't ring a bell for me, so maybe I haven't seen a lot of problems with it. | 18:20 |
blp | I've spent a lot of time trying to figure out whether some tests are flaky, so some of them are familiar. | 18:20 |
blp | Ugh I wish we didn't need end-to-end tests so badly, they're so hard to make reliable. | 18:20 |
dceara | blp: I think I saw this test failing in the past but with my change it fails more often. So I'd like to make sure it's not another IDL bug that I'm introducing/uncovering. | 18:21 |
mmichelson | Unfortunately, I can't with any confidence say that dceara's issue is a flaky test. It's also possible due to weirdness in some OVN component that things process in a slightly different order sometimes and cause a problem. | 18:21 |
dceara | mmichelson: That's a possibility too. | 18:22 |
* mmichelson has flashbacks to a CT zone issue from the past | 18:22 | |
numans | hello | 18:22 |
blp | For ovn-northd-ddlog, I often compare the southbound flow table dump against ovn-northd. | 18:22 |
numans | mmichelson, I'd say we should go ahead and release tomorrow. | 18:22 |
numans | We can probably updte the ovs submodule commit once the patch is merged. | 18:23 |
blp | For your change, you might be able to compare dumps of sbflows or something else, with and without your change. | 18:23 |
* numans oops. sorry for jumping in. | 18:23 | |
dceara | blp: Yes, will do. | 18:23 |
blp | Many of the tests now dump southbound flows to a file 'sbflows' to make this easier. | 18:23 |
dceara | blp: The main problem though is that the test fails in github CI. On my machine it mostly passes. | 18:24 |
mmichelson | What I found in the CT zone issue I referred to was that sometimes I'd have multiple ovn-nbctl commands that would get handled by ovn-northd as one operation, and other times they'd get handled as two separate transactions. Then this would result in ovn-controller either processing it all as one change vs. as two separate changes. And depending on which happened, we'd have different behavior. And yes, if you guessed it was an | 18:24 |
mmichelson | incremental processing issue, you would be correct. | 18:24 |
blp | dceara: That makes it harder. | 18:25 |
dceara | :) | 18:25 |
blp | dceara: I often see tests fail when I run them with high parallelism, e.g. TESTSUITEFLAGS=-j12 on a 6-core laptop. | 18:25 |
blp | dceara: If you don't already try that, it's worth trying. | 18:26 |
dceara | blp: Same here, i've been trying with -j20 on a dev server I'm using. Not so much luck. | 18:26 |
blp | (At some point I might switch to developing on my 3990x box and then I'll use -j128.) | 18:26 |
*** imaximets_ is now known as imaximets | 18:26 | |
imaximets | mmichelson, if we're going to release tomorrow than we should move submodule now to ac09cbfcb70a ("ovsdb-cs: Fix use-after-free for the request id.") before the release. | 18:26 |
mmichelson | imaximets, ack | 18:28 |
mmichelson | Anyways, aside from release talk, I didn't have anything else I wanted to share. I can update the ovs submodule and put that up for review after this meeting | 18:29 |
blp | Is there more to discuss on this topic or shall we move on? | 18:29 |
mmichelson | And after that, I can post the release commits as well | 18:29 |
mmichelson | So taht should allow for us to release tomorrow. | 18:29 |
mmichelson | blp, I think that's it, if you'd like to go next. | 18:30 |
numans | +1 | 18:30 |
blp | Cool. | 18:30 |
imaximets | mmichelson, I'll send a patch for submodule shortly. | 18:30 |
blp | I posted a series of improvements to ovn-northd-ddlog last week. | 18:30 |
blp | Numan pointed out some issues that I should have noticed, but had not. | 18:30 |
blp | I have posted a number of small fixes that can apply separately from that series. | 18:30 |
blp | And I will also post a revision of the series itself soon. | 18:30 |
blp | But, more excitingly, Leonid spent some time optimizing the ddlog code | 18:31 |
blp | and he gave me a tarball of the changes last night | 18:31 |
blp | which I will look at and probably post (maybe add to my series?). | 18:31 |
blp | They are pretty small changes and he says they obtain more than 3x performance improvement | 18:31 |
blp | over the improvement I had gained in my series. | 18:31 |
mmichelson | That's impressive! | 18:32 |
blp | He also says that with them the cost of each step in the benchmark that Numan provided seems to go to O(1) rather than increasing with each iteration. | 18:32 |
numans | that's cool. | 18:32 |
blp | Once we get that in, I would really appreciate it if people would throw more benchmark challenges at me. | 18:32 |
blp | I think that I understand the optimization principles that Leonid used. | 18:32 |
blp | I'm going to apply them myself | 18:32 |
numans | up for the challenge :) | 18:33 |
blp | and I'm going to try to write them up for others to understand as well. | 18:33 |
blp | numans: awesome | 18:33 |
mmichelson | blp, just so I'm clear, are these changes from Leonid to DDLog the language or to ovn-northd-ddlog? | 18:33 |
blp | They are mainly to ovn-north-ddlog. | 18:33 |
mmichelson | Got it | 18:34 |
blp | Leonid did add a small feature to the ovsdb2ddlog program we use for generating .dl files. | 18:34 |
blp | which is here if you want to look at it: https://github.com/vmware/differential-datalog/pull/934 | 18:34 |
blp | I can also report that we've hired a couple of people to work on ddlog at VMware. | 18:35 |
blp | which is a good indication that it will be maintained. | 18:35 |
blp | and enhanced | 18:35 |
blp | I am done with my report but I'm happy to answer more questions if anyone has them. | 18:36 |
blp | (I'm hoping that one of the new hires will write a ddlog formatter like "indent" for C or "rustfmt" for Rust.) | 18:36 |
numans | blp, I had reported few memory leaks. | 18:37 |
numans | I'm not sure if you've addressed them in your series. | 18:37 |
mmichelson | No questions from me. I'm really happy to see the improvements being made to DDLog | 18:37 |
blp | I posted patches to fix those, I think. | 18:37 |
numans | cool then. | 18:37 |
numans | I haven't looked into the patches. | 18:37 |
numans | I remember dceara mentioned that in his testing northd-ddlog took huge ram like 75gb or so. I may be wrong, dceara can update if he is still here. | 18:38 |
blp | numans: Yes, the leak fixes are specifically: | 18:38 |
blp | https://mail.openvswitch.org/pipermail/ovs-dev/2021-March/381117.html | 18:38 |
blp | https://mail.openvswitch.org/pipermail/ovs-dev/2021-March/381119.html | 18:38 |
blp | numans: They should be easy to review if you have a few minutes to look at them. | 18:38 |
numans | sure. I'll take a look. thanks. | 18:38 |
blp | numans: northd-ddlog does take more RAM. I think that the optimization pathces (and leak fixes) should help. | 18:39 |
numans | ok. The patches looks straightforward. | 18:39 |
dceara | blp: It's very nice to see the ddlog related activity! I did give ovn-northd-ddlog a try on one of the large NB databases extracted from one of our scale tests. I hope the 75gb was due to the memleaks you fixed :) | 18:39 |
blp | 75 GB is a lot. | 18:40 |
dceara | blp: I also had a small bug report that I didn't get a chance to report on the mailing list yet: it seems that ovn-northd-ddlog clears NB_Global.options:use_logical_dp_groups after the first run, effectively disabling the feature. | 18:40 |
blp | dceara: I'll look at that. Should be an easy fix. | 18:41 |
dceara | blp: Yep, looked relatively straightforward indeed. | 18:41 |
zhouhan | blp: Do you know what's the expected memory consumption without leak? How many times more than regular northd? | 18:42 |
blp | zhouhan: I don't have an estimate for that yet. We can target memory use like we target speed, by dumping a memory profile and looking for excessive consumption then making ddlog code adjustments. | 18:43 |
zhouhan | blp: ok, just want to have a rough idea about. I remember last time (1 - 2 years ago) when I was testing it, it was about 10x of regular northd | 18:44 |
blp | In other software we've built with ddlog, I think it was more like 2x or 3x after we did a little work to bring it down. | 18:45 |
zhouhan | blp: 2x - 3x is much better now :) | 18:45 |
blp | It is more or less unavoidable that an incremental version of anything would take more memory than a nonincremental version. | 18:45 |
blp | But if it uses so much memory that it's unusable, obviously that's not a good tradeoff. | 18:45 |
zhouhan | blp: this is reasonable. Just want to know the worse case. On central node I think 2x - 3x memory is not an issue at all. I am only thinking about the future of ovn-controller using ddlog :) | 18:46 |
blp | That is a good point. Memory is much more critical for ovn-controller. | 18:46 |
blp | I used to joke that NVP (back at Nicira circa 2011) required infinite memory. | 18:47 |
blp | And that its "nlog" language should be called "n exponential". | 18:47 |
mmichelson | e^nlog | 18:48 |
blp | (Optimization helped.) | 18:48 |
numans | I've observed ovn-controller taking up like 9gb on a scaled env and vswitchd takes around ~2.5gb. | 18:48 |
blp | Holy crap that's a lot already. | 18:48 |
dceara | numans: With the lflow cache enabled? | 18:48 |
numans | dceara, yes. | 18:48 |
blp | I'm embarrassed, how did my baby become a monster? | 18:48 |
zhouhan | sounds like a proof that it is doing I-P :D | 18:48 |
*** ralonsoh has quit IRC | 18:48 | |
mmichelson | Does anyone else want to share? | 18:49 |
imaximets | mmichelson, blp: https://patchwork.ozlabs.org/project/ovn/patch/20210311183719.2517358-1-i.maximets@ovn.org/ | 18:49 |
mmichelson | imaximets, that should be an easy ack | 18:50 |
numans | I can go real quick. I was busy almost this week working on a couple of crashes seen in ovn-controller | 18:50 |
numans | All thanks to me and the runtime data I-P handling :) | 18:50 |
numans | I submitted the patch for review - https://patchwork.ozlabs.org/project/ovn/patch/20210311124757.2997057-1-numans@ovn.org/ | 18:51 |
numans | zhouhan, I couldn't reply to your questions on the ct.inv drop patch. I'll get to that next week. thanks for the comments and review so far. | 18:51 |
zhouhan | numans: np. I also reviewed your I-P refactor, will review the ofctrl refactor RFC next couple of days. | 18:52 |
numans | I'm also inclined to submit v2 making it as a config option so that regular users are not missed out on this, in case there are scenarios where invalid pkts needs to be dropped. | 18:52 |
numans | zhouhan, thanks. | 18:53 |
blp | imaximets: Thank you Ilya. I should have sent that earlier. acked. | 18:53 |
numans | that's it from me. | 18:54 |
karthikc | hello, can I go next? Had a question regarding upgrade | 18:54 |
numans | sure. | 18:55 |
karthikc | https://mail.openvswitch.org/pipermail/ovs-discuss/2021-February/050988.html | 18:55 |
karthikc | Is this upgrade supported? I seem to be hitting a backward compatibility issue | 18:56 |
karthikc | we are basically trying to move from v2.11 to ovn-20.09. If the controller is upgraded first, the chassis don't show up | 18:57 |
numans | karthikc, I saw that email. Sorry I couldn't reply. | 18:57 |
mmichelson | karthikc, Someone can correct me if I'm wrong, but I think the upgrade order assumes that you also will update the central components too. It sounds like you've upgraded ovn-controller but not ovn-northd | 18:57 |
*** dholler has quit IRC | 18:58 | |
numans | that's correct. | 18:58 |
blp | mmichelson: The upgrade doc says to upgrade ovn-controller first. | 18:58 |
karthikc | that's correct. ovn-northd is yet to be upgraded. But there is a transient state where chassis don't show up | 18:58 |
numans | karthikc, is your concern that during this time, the traffic is broken ? | 18:58 |
karthikc | yes | 18:58 |
numans | karthikc, recently we added a feature to pin ovn-controller to a specific version of ovn-northd | 18:59 |
numans | if ovn-controller sees this mismatch, it does nothing until ovn-northd is also upgraded. | 18:59 |
numans | karthikc, I'll reply about that in the ML. | 18:59 |
numans | I'm not sure if that commit is 20.09 or 20.12 | 18:59 |
karthikc | neat, that will be helpful, thanks | 18:59 |
mmichelson | I think it was 20.12 | 19:00 |
mmichelson | But I might be wrong | 19:00 |
numans | mmichelson, you may be right. | 19:00 |
numans | karthikc, may be you can consider uprading to 20.12 | 19:00 |
karthikc | ok | 19:00 |
dhathri | hi, can i go real quick next? | 19:01 |
mmichelson | The other option is to use the --restart option when restarting ovn-controller so that ovn-controller doesn't "clean up" when you stop it. That way when it starts back up, it should be in the same configuration as when it was running previously. You at least shouldn't have interrupted traffic if you do that. But you also won't be able to make changes. | 19:01 |
mmichelson | dhathri, go for it | 19:01 |
dhathri | just wanted to request for review on the multiple gateway router support patch (https://mail.openvswitch.org/pipermail/ovs-dev/2021-March/380979.html) | 19:02 |
numans | looks like we need to be careful when we add new columns which ovn-controlller needs to update. | 19:02 |
numans | dhathri, sure. Meanwhile could you also plan for another version with ddlog support ? | 19:02 |
numans | dhathri, I'll take a look next week. | 19:03 |
dhathri | sure, thanks.. ddlog will take more time since there is a learning curve. Wanted to get the c patch reviewed while i am working on the ddlog changes | 19:03 |
blp | I'm in another meeting now, so I'll talk to everyone next week. | 19:04 |
mmichelson | bye blp | 19:04 |
*** blp has quit IRC | 19:04 | |
mmichelson | Does anyone else have a report? | 19:05 |
mmichelson | I'll take the lack of response to mean "no" | 19:06 |
mmichelson | Bye everyone | 19:06 |
zhouhan | bye! | 19:06 |
dceara | Bye | 19:06 |
karthikc | bye | 19:06 |
imaximets | bye | 19:06 |
mmichelson | #endmeeting | 19:06 |
openstack | Meeting ended Thu Mar 11 19:06:39 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:06 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-03-11-18.15.html | 19:06 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-03-11-18.15.txt | 19:06 |
openstack | Log: http://eavesdrop.openstack.org/meetings/ovn_community_development_meeting/2021/ovn_community_development_meeting.2021-03-11-18.15.log.html | 19:06 |
*** dhathri has quit IRC | 19:07 | |
*** karthikc has left #openvswitch | 19:08 | |
*** tbachman has quit IRC | 19:26 | |
*** tbachman has joined #openvswitch | 19:30 | |
*** links has quit IRC | 19:44 | |
*** dalvarez has quit IRC | 19:47 | |
*** dalvarez has joined #openvswitch | 19:47 | |
*** rcernin has joined #openvswitch | 20:00 | |
*** mdgray has quit IRC | 20:06 | |
*** anilvenkata has quit IRC | 20:24 | |
*** acidfoo has joined #openvswitch | 20:33 | |
*** acidfu has quit IRC | 20:34 | |
*** slaweq has quit IRC | 20:56 | |
*** rcernin has quit IRC | 21:04 | |
*** rcernin has joined #openvswitch | 21:35 | |
*** dceara has quit IRC | 21:48 | |
*** rcernin has quit IRC | 21:51 | |
*** rcernin has joined #openvswitch | 21:54 | |
*** acidfoo has quit IRC | 21:56 | |
*** rcernin has quit IRC | 22:00 | |
*** rcernin has joined #openvswitch | 22:00 | |
*** bostondriver has quit IRC | 22:09 | |
*** circ-user-qLPh6 has joined #openvswitch | 22:48 | |
*** zhouhan has quit IRC | 23:04 | |
*** mmichelson has quit IRC | 23:17 | |
*** mmichelson has joined #openvswitch | 23:18 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!