18:15:21 <mmichelson> #startmeeting ovn_community_development_discussion
18:15:22 <openstack> Meeting started Thu Dec 17 18:15:21 2020 UTC and is due to finish in 60 minutes.  The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:15:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:15:25 <openstack> The meeting name has been set to 'ovn_community_development_discussion'
18:15:55 <mmichelson> Hi everyone
18:16:08 <blp> hi!
18:16:10 <mmichelson> Tomorrow is the proposed release date for 20.12.0
18:16:12 <dceara> Hi
18:16:25 <mmichelson> So I have sent out the commits just before this meeting started
18:16:37 <mmichelson> That way we can get it released early-ish (for the western hemisphere) tomorrow
18:17:20 <mmichelson> In other news, I reworked my unit test patches so that rather than trying to introduce a framework that requires instrumenting source files, it just is a small refactor of some IPAM code and addition of tests using ovstest
18:17:22 <blp> mmichelson: +1
18:17:56 <mmichelson> blp, I'm glad you're here today :) The release of 20.12.0 means we should be in position to merge DDLog to master whenever you think you're ready.
18:18:05 <blp> mmichelson: That is good news.
18:18:06 <mmichelson> I understand you're currently working on a new version of your patches
18:18:28 <blp> I see that Dumitru passed along some more reviews.
18:18:31 <mmichelson> Would you want to get the latest version posted or would you rather us merge what currently exists?
18:18:35 <blp> I think that my job is then:
18:18:43 <blp> - Rebase and deal with whatever changes have happened.
18:18:50 <blp> - Address Dumitru's feedback.
18:19:13 <blp> If the changes are relatively small based on those two things, then I'd be inclined to merge it.
18:19:22 <blp> If they need more work, then I'd be inclined to post a v10.
18:19:29 <blp> What do you think?
18:19:54 <blp> And post-merge there are at least two jobs left:
18:19:59 <blp> - Keep up-to-date with changes.
18:20:13 <blp> - Run benchmarks and iterate on making it faster and scale better.
18:20:26 <dceara> blp: The last patch of the series depends on the IDL split changes (which need a rebase).  I'm not sure if there are more changes needed in the IDL patches, but with the current version quite a few OVN tests were failing.
18:20:27 <blp> I think it was Numan who passed along a useful benchmark script.
18:20:37 <blp> Oh, crap.
18:20:41 <mmichelson> blp, for the rebase step, I assume you're not going to aim for feature parity, right?
18:20:56 <blp> mmichelson: I've kept up with feature parity so far.
18:21:16 <mmichelson> blp, ah, ok. I just know that some of the changes (such as the datapath groups change) could be kind of big.
18:21:41 <blp> I'll rebase the IDL split changes then.
18:22:00 <blp> Last time I did that I did not see any test failures, but maybe some were introduced aftewrad.
18:22:16 <blp> It's hard to keep rebasing a big series against a constantly shifting base.
18:22:21 <mmichelson> yes it is
18:22:41 <mmichelson> that's why I was wondering if we should just merge what we have and then work directly in master from this point?
18:23:00 <mmichelson> That way, devs introducing new features to northd can introduce to C and DDLog simultaneously
18:23:15 <blp> If the IDL split changes can get into OVS, then we can do that.
18:23:16 <mmichelson> And we only have to chase that gap that currently exists from between your last posting and now
18:23:29 <blp> Without the IDL split changes, ovn+ddlog won't compile.
18:23:36 <mmichelson> got it
18:23:38 <blp> I agree with the philosophy otherwise though.
18:23:44 <imaximets> blp, I'm going to review idl-split patch-set tomorrow.
18:23:53 <blp> imaximets: Thanks! That will enable the rest.
18:24:06 <blp> imaximets: I'll try to get the rebase of it done today, then.
18:24:17 <zhouhan> blp: for the post-merge task "keep up-to-date with changes", is it assumed that the developer who submit the change should keep the DDlog part up-to-date?
18:24:41 <blp> zhouhan: I would hope so, but I am also available to help, either to eduate developers or to help doing the porting.
18:24:55 <zhouhan> blp: sounds good!
18:25:19 <imaximets> blp, ok.  One quick comment: can we re-name reconnect_try_receive to reconnect_receive_attempt ?  Current name sounds like we're requesting it to receive something.
18:25:38 <blp> imaximets: OK.
18:25:47 <imaximets> blp, thanks!
18:27:01 <blp> imaximets: I'll fold that into my rebase then.
18:27:17 <mmichelson> anyway, I kind of transitioned the conversation to ddlog, but to be clear I also was done talking about the things I had worked on.
18:27:34 <blp> I don't think I have anything else.
18:28:08 <blp> 9
18:28:12 <blp> oops wrong window
18:28:32 <imaximets> I have a topic to discuss.
18:29:27 <imaximets> last time I talked about this patch: https://patchwork.ozlabs.org/project/openvswitch/patch/20201211205447.3874314-1-i.maximets@ovn.org/  for ovsdb-server.
18:30:04 <blp> imaximets: yes
18:30:10 <imaximets> basically, it changes the format of file transactions that stored in database file and in raft logs to contain diffs of columns.
18:31:16 <imaximets> recent tests shows decrease by 90% of memory consumption for northbound database for ovn-k8s tests, 40% for southbound datbase and overall 40% reduction in disk IO.
18:31:22 <blp> wow!
18:31:33 <imaximets> blp, zhouhan: it'll be great if you can review it.
18:31:47 <zhouhan> imaximets: ok!
18:32:06 <imaximets> main problem, I guess, is the upgrade process, since ovd ovsdb-server will not be able to read new database format.
18:32:10 <blp> I'll put it on my to-do list.
18:32:40 <imaximets> that is also a problem if we'll have different versions of ovsdb-server within same raft cluster.
18:32:44 <imaximets> blp, thanks!
18:33:11 <blp> Why is that a problem for upgrades? I would expect that to be a problem for downgrades, which are still worth considering but not in the same way.
18:34:20 <imaximets> It's because if raft leader will send new append request to old follower that will be a problem.
18:34:29 <blp> Oh, OK, I see, we're changing not just the on-disk format but the format that gets sent around the raft cluster.
18:34:40 <imaximets> yep
18:35:17 <imaximets> I also worked on adding memory reports to ovn processes today: https://patchwork.ozlabs.org/project/ovn/patch/20201217181427.639377-1-i.maximets@ovn.org/
18:35:35 <imaximets> that's it from my side.
18:35:54 <blp> Maybe this format parameter should be part of the database definition, so that the cluster has to agree on what format to use from the start.
18:36:19 <blp> Then, the format would only be upgraded once all the cluster members can handle it.
18:37:11 <imaximets> blp, problem is that we must be sure that new old server will not be connected to a cluster once format changed.
18:37:33 <imaximets> blp, and also old ovsdb-server will not understand that something is wrong.
18:38:02 <panda> can ovsdb raft cluster size change during normal operations ?
18:38:15 <blp> imaximets: The format upgrade would have to happen after the software upgrade was finished.
18:38:40 <blp> panda: OVSDB supports adding and removing raft cluster members, yes.
18:39:17 <blp> panda: And, of course, cluster members can fail and come back at runtime. I don't know whether you consider failures to be normal operations.
18:39:20 <imaximets> blp, I added a cmdline option to disable new format.  After upgrade user will be able to enable it via unixctl or be re-starting without the option.
18:39:38 <imaximets> *by
18:40:35 <imaximets> but I'm not sure if that is a best way to handle this.
18:40:37 <blp> imaximets: That's kind of an admin-intensive way to deal with it. The admin will have to make sure to use the right command-line options during upgrade, and then turn them off afterward.
18:41:13 <blp> imaximets: It might be harder to automate than telling people to upgrade the database format once they have finished their server upgrade.
18:41:17 <blp> imaximets: I'm not an admin though.
18:43:39 <imaximets> blp, I can't think of a correct automatic way that will work in all cases.  But I'd like to discuss ideas on a mail-list.
18:43:46 <blp> imaximets: OK.
18:44:19 <mmichelson> cool, y'all. Anybody else?
18:45:36 <dceara> I just wanted to mention that since yesterday OVN CI runs with AddressSanitizer enabled for clang (memleak detection too) on GitHub.  Thanks imaximets for the suggestion and review!
18:45:45 <dceara> That's it on my side, thanks!
18:47:53 <blp> AddressSanitizer is a big help. Glad to see it in use for CI.
18:48:27 <blp> Anyone else?
18:49:54 <mmichelson> I guess that's it
18:50:09 <mmichelson> Thanks everyone!
18:50:21 <mmichelson> #endmeeting