17:16:35 <mmichelson> #startmeeting ovn-community-development-discussion
17:16:36 <openstack> Meeting started Thu Jun 11 17:16:35 2020 UTC and is due to finish in 60 minutes.  The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:16:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:16:40 <openstack> The meeting name has been set to 'ovn_community_development_discussion'
17:16:52 <panda> o/
17:17:25 <blp> Hi!
17:17:56 <numans> Hi
17:18:00 <dceara> Hi
17:18:04 <mmichelson> hi everyone!
17:18:12 <flaviof> hi all
17:18:14 <blp> Do we need an #ovn channel or is everyone happy with using #openvswitch?
17:18:32 <mmichelson> blp, funny thing is I'm actually in a #ovn channel on freenode already
17:18:41 <blp> Oh. Should the meeting be there?
17:18:43 <zhouhan> hello
17:18:45 <mmichelson> It's not very heavily populated
17:18:57 <numans> The meeting bot is configured for #openvswitch I think.
17:19:03 <mmichelson> blp, I'm ambivalent on the matter. We're not interrupting conversations in here really
17:19:06 <blp> OK.
17:19:12 <blp> No problem.
17:19:25 <blp> Anyway, there's a discussion of OVN mailing lists going on in ovs-discuss.
17:19:42 <blp> https://mail.openvswitch.org/pipermail/ovs-discuss/2020-June/050221.html
17:20:01 <blp> Did you #startmeeting?
17:20:10 <blp> I have not.
17:20:13 <numans> I think mmichelson did
17:20:18 <mmichelson> Yes I did
17:20:18 <dceara> blp, just a quick question about "closed" lists. The archives stay open and available to everyone regardless if they subscribed or not, right?
17:20:33 <blp> Yes. By "closed" I mean only "closed to posting by non-subscribers".
17:20:36 <blp> Not "private".
17:20:46 <dceara> blp, ack, thanks for confirming
17:21:32 <blp> Numerous people have mentioned that and it took me by surprise because it never even occurred to me that we'd have private lists.
17:22:05 <blp> I guess there's the "committers" and "security" lists but they're basically zero traffic.
17:24:17 <mmichelson> OK, so I guess since blp dropped that's a signal that someone else is free to give their update.
17:24:28 <blp> Weird, Pidgin hung and I had to restart it.
17:24:53 <mmichelson> blp, funny I had just said that since you had dropped someone else could give their update. But I'll re-yield the floor to you for now
17:25:47 <blp> OK I'm just not going to try to update the topic. It breaks Pidgin.
17:25:56 <blp> Anyway, I was done anyway so I'll listen now.
17:28:17 <mmichelson> OK cool
17:28:22 <mmichelson> I'll go real quick I suppose
17:28:44 <mmichelson> I released OVN 20.06.0 earlier this week, and I've got an ACK to release 20.03.1. I'll release it before the end of the day
17:29:09 <mmichelson> I sent an email to ovs-announce but I think I'm not actually subscribed, so my email went to moderation.
17:29:43 <mmichelson> And aside from that, it's been a lot of catch-up on existing patches of mine that are out there and need updating. Also did some reviews of patches from Han and Anton
17:29:51 <mmichelson> That's about all from me.
17:30:11 <flaviof> May I go next? I actually have more of a question on v20.06.0.
17:30:52 <numans> mmichelson, I didn't see any email on ovs-announce
17:31:03 <numans> and I don't see it in the archives too.
17:31:20 <numans> flaviof, sure.
17:31:39 <numans> mmichelson, can you please double check on that.
17:31:56 <numans> And thanks for releasing 20.06
17:32:08 <flaviof> ++
17:32:14 <flaviof> Not too long ago, we changed OVN to use OF1.5 so it can handle dp_hash selection for load_balancer.
17:32:27 <flaviof> #link https://github.com/ovn-org/ovn/commit/6ec0b82038052866533f12823fe410308b3e457a [commit] controller: Use OpenFlow version 1.5
17:32:27 <flaviof> #link https://mail.openvswitch.org/pipermail/ovs-discuss/2020-April/049896.html [discuss] OVN Load balancing algorithm
17:32:37 <flaviof> So I think that we will have ovn-controller using OF1.5, starting on v20.06.0.
17:32:47 <numans> mmichelson, oops. I didn't read your full message.
17:33:06 <numans> flaviof, right.
17:33:10 <flaviof> In the upgrade path from v20.03.0 to v20.06.0 br-int may be already created and using OF1.3...
17:33:20 <flaviof> Do you foresee special handling needed?  If so, would the proper thing to do is to have both OF1.3 and OF1.5 as supported versions?
17:33:29 <flaviof> I'm assuming that the original code in the controller would only enable OF1.3. Please correct me if I'm wrong.
17:34:42 <flaviof> We can take this to discuss ML, of course, but I just wanted a general idea if any of you have already covered this.
17:34:50 <flaviof> And that is all I have.
17:34:55 <mmichelson> flaviof, So a couple of things: First, I think that if ovn-controller is stopped and restarted, it will renegotiate the OF version with OVS.
17:35:42 <mmichelson> Second, I'm pretty sure the documented update path involves restarting ovs-vswitchd, rather than updating OVN on top of a running OVS instance. I may be wrong about that, though.
17:36:17 <mmichelson> Someone please correct me if I'm wrong on either of those two points
17:36:25 <numans> flaviof, I know that devstack sets or was setting the protocol for the br-int in ovs db.
17:36:32 <numans> if it doesn't set, I think it should work fine.
17:36:41 <numans> or it should also set the protocol OF1.5
17:37:11 <numans> mmichelson, I think your first is correct. For the 2nd, I don't know.
17:37:29 <numans> flaviof, I didn't find any issues in my testing. But I'll test it out again to be sure.
17:37:42 <flaviof> Ack. My question is on what happens if br-int exists and it is configured to use OF1.3... What would we expect to happen?
17:38:13 <numans> flaviof, let me test out that scenario and come back to you on that.
17:38:24 <flaviof> numans++ TY
17:38:41 <numans> I can go real quick next.
17:39:00 <numans> if flaviof is done.
17:39:00 <zhouhan> mmichelson: I didn't expect the point 2 either.
17:39:17 * flaviof is done. except for saying TY to mmichelson too ;)
17:39:53 <numans> I think for OVN, there is no need for CMS to set the protocol of the integration brdige
17:40:11 <numans> As ovn-controller never reads that and try to use that as OF flow version.
17:40:42 <numans> I've been working on I-P patches this week addressing review comments.
17:41:15 <numans> Yesterday I applied first 3 patches of the series to master and branch-20.06 since I had 2 acks and a few rounds of reviews.
17:41:29 <numans> Soon after dceara found 2 issues and fixed them. Thanks dceara for fixing and for the thorough reviews this week.
17:41:46 <numans> I'm not sure if zhouhan had additional comments.
17:41:55 <numans> zhouhan, If so, please let me know and I'll address them.
17:41:56 <dceara> numans, no problem, sorry for not spotting the issues earlier though
17:42:26 <numans> dceara, no worries. I've never tested that scenario. I will test that now. And you added a test  case.
17:42:30 <numans> that's great.
17:42:31 <zhouhan> numans: I will take a look for v12
17:42:39 <numans> I submitted v12 of the series. I think there are in decent stage now.
17:42:41 <numans> zhouhan, thanks.
17:43:08 <blp> The one thing that *might* turn up about OF1.5 is that it's much less tested than OF1.3.
17:43:30 <blp> Perhaps we will turn up some bugs or points of  nonconformance. I hope not.
17:43:39 <numans> Hope not :)
17:43:44 <flaviof> +1
17:43:48 <numans> I also worked on ovs idl patch which tries to return a valid txn object if it can. Thanks zhouhan for the reviews so far. I've addressed your comments
17:43:54 <numans> and submitted v3.
17:44:00 <numans> Request to take a look - https://patchwork.ozlabs.org/project/openvswitch/patch/20200605083029.129633-1-numans@ovn.org/
17:44:18 <numans> blp, not sure if you got some time for this patch. but if you can, that would be great.
17:44:27 <blp> numans: It's on my to-do list now.
17:44:33 <numans> blp++
17:44:34 <numans> thanks.
17:44:52 <numans> I have a couple of smaller patches for review. Request to take a look
17:44:55 <numans> https://patchwork.ozlabs.org/project/openvswitch/patch/20200601092742.550334-1-numans@ovn.org/
17:44:55 <numans> https://patchwork.ozlabs.org/project/openvswitch/patch/20200430081000.751498-1-numans@ovn.org/
17:44:58 <numans> And that
17:45:00 <numans> that
17:45:01 <zhouhan> numans: sure, thx!
17:45:05 <numans> that's it from me.
17:45:19 <numans> thanks.
17:45:25 <zhouhan> I can go next
17:45:46 <zhouhan> Thanks imaximets for reviewing the dp_hash recirc fix
17:45:57 <blp> OK, I added "es" to my "Review Numan's patch" to-do item.
17:46:28 <zhouhan> imaximets: I replied your email. blp: please take a look as well, as requested by imaximets, too :)
17:47:05 <blp> zhouhan: I missed the context. What should I look at?
17:47:52 <zhouhan> imaximets: we can discuss offline, but the major point I wanted make is that the problem is caused only when there is a slowpath action required by the flow, so in that case I think always computing dp_hash in slowpath for the flow wouldn't have consistency issue.
17:48:34 <zhouhan> blp: #link https://patchwork.ozlabs.org/project/openvswitch/patch/1589527067-91901-1-git-send-email-hzhou@ovn.org/
17:48:53 <zhouhan> blp: this is dp_hash issue we have discussed in a previous meeting.
17:49:38 <zhouhan> I also sent a RFC series for avoiding the flow explosion problem reported by ovn-k8s project.
17:49:45 <imaximets> zhouhan, ok. I understand. Still looks a bit dangerous for me, need a fresh look from blp. :)
17:50:04 <zhouhan> Thanks mmichelson and numans for reviewing it. Sorry for missing some documentation in the RFC.
17:50:24 <zhouhan> imaximets: ok, thx
17:50:27 <numans> zhouhan, that's fine. I think you wanted Girish to test them first.
17:50:50 <zhouhan> numans: yeah, I wanted to send out as early as possible so that they can try.
17:51:04 <zhouhan> that's it from me
17:51:12 <blp> zhouhan: OK, will do.
17:51:28 <zhouhan> blp: thx
17:52:07 <mmichelson> OK anyone else want to give an update?
17:52:36 <blp> imaximets: Thanks so much for coming up with a simpler RCU fix.
17:52:39 <dceara> a oneliner from me this week: I started a thread on ovs-discuss about the current state of packet tracing in OVS/OVN. Feedback and suggestions are much appreciated: https://mail.openvswitch.org/pipermail/ovs-discuss/2020-June/050205.html
17:52:45 <dceara> Thanks!
17:53:28 <numans> dceara++ for working on this.
17:53:35 <imaximets> blp: no problem. Thanks for review.
17:53:43 <zhouhan> dceara: I saw it, that's wonderful
17:54:01 <aginwala> thanks dceara ++ from my side too :)
17:54:26 <blp> dceara: Is it worth discussing this here or would you prefer to keep it in email?
17:54:59 <blp> It's a big email so sometimes I find it exhausting to try to reply to every point at once, yet I feel bad if I only reply to one point in a long series.
17:55:17 <blp> And so sometimes that results in me just ignoring things I should help with.
17:55:26 <dceara> blp, i'm ok either way (ML or here)
17:55:26 <blp> A discussion, however, could be helpful, if you're up for it here.
17:55:33 <dceara> blp, sure
17:56:15 <blp> Last point first: it should be possible to make the output format better.
17:56:40 <blp> IIRC, ovn-trace has multiple output formats because it has a decent internal data structure that can be serialized multiple ways.
17:56:57 <blp> ofproto/trace used to be very primitive in that respect, but I think I redid it a few years ago to use a similar internal data structure.
17:57:16 <blp> So it might be possible to even just mostly copy code from ovn-trace into ofproto/trace to get a json output format.
17:57:36 <dceara> blp, yes, I looked into it too a bit and it should be relatively straightforward indeed
17:58:28 * numans also thinks that we can add --json option to most of the ovn (and ovs) utilities to output. That way ovn-kubernetes, ovn-scale-test etc can easily parse the output.
17:59:49 <blp> So, the other thing that occurs to me is that ovn-trace already knows how to talk to OVS instances, via the --ovs option.
18:00:08 <blp> It might be better to teach it to ofproto/trace than to create a new utility to do that.
18:00:37 <mmichelson> numans, all DB operations have json as an option. I actually used it when adding support for unidling in ovn-kubernetes
18:01:05 * numans is not happy with his ignorance.
18:01:29 <numans> mmichelson, thanks. I haven't noticed that.
18:01:30 <mmichelson> The json schema is...interesting, to say the least :)
18:01:41 <blp> ovn-trace, after all, already does a number of the operations listed in the description of ovn-global-trace.
18:02:29 <blp> Currently it doesn't know how to SSH or to connect to ovs-appctl, but it could learn; any other implementation would have to do that too.
18:02:39 <dceara> blp, ok, that's possible too. I guess we'd drop the need for ovn-detrace then (for this use case at least).
18:03:53 <blp> dceara: yes
18:04:13 <blp> I think that the overall idea is a good one, it puts together a lot of the stuff that is now in bits and pieces.
18:04:26 <dceara> blp, I agree that a single ovn-trace tool would be ideal.
18:04:54 <blp> I think the shakiest part is probably the SSH interface. I guess popen("ssh...") would be the way to do that.
18:06:01 <flaviof> blp: dceara: so would would think that using ssh is better than using geneve? I think it makes sense so data and ctrl are completely separate, right?
18:06:19 <numans> I agree with single ovn-trace. But if ovn-global-trace is written in python then probably it will be easier to do ssh and other stuff
18:06:53 <dceara> blp, would it be acceptable to have the connection to the node be externally setup somehow (e.g., socat a unix connection)?
18:06:59 <blp> flaviof: We're talking about running ofproto/trace. That has to happen over SSH because OVS only listens on a unix domain socket for that kind of request.
18:07:13 <flaviof> blp: ack!
18:08:16 <blp> dceara: I don't want to prescribe anything, I was just assuming SSH was the best way.
18:08:46 <imaximets> blp: theoretically we could allow appctl over ssl.
18:08:50 * imaximets hides
18:08:58 <dceara> imaximets, :)
18:10:00 <blp> imaximets: I think you can do that today if you specify --unixctl to a pssl: socket, but I think we only support a single such socket so you'd lose the unix domain socket support if you did that.
18:11:43 <blp> It looks like the discussion here is slowing down, please @ me if you want me.
18:11:46 <zhouhan> dceara: you mentioned recirculation. I think that's also important for the debugging tool to be useful. It includes not only conntrack, but also dp_hash :)
18:12:33 <zhouhan> not sure if there is a better idea other than passing an hint
18:12:41 <imaximets> blp: it should be possible with a little magic to specify --unixctl pssl:<>,unix:<>.  But anyway, I don't think it's a way good way for production deployment.
18:13:08 <blp> imaximets: +1
18:13:40 <dceara> blp, popen is fine I guess, we probably need to be a bit flexible and allow the protocol to be controlled by the user though. The node running OVS might not be running an ssh server.
18:14:43 <dceara> zhouhan, good point, we should find a solution for dp_hash too. However with dp_hash I think hints are ok. The chances that you choose a "wrong" bucket are quite small. I would expect all buckets to produce the same behavior in a sane deployment.
18:15:09 * numans waves and says goodbye to everyone.
18:15:30 <panda> numans: bye, and thanks for the reviews
18:15:40 <zhouhan> dceara: but if we are talking about the global tracing, a wrong bucket would lead to tracing on a wrong remote (at least in the case of ECMP)
18:16:10 * zhouhan waves bye to numan
18:16:21 <dceara> numans, bye!
18:16:57 <dceara> zhouhan, great point I didn't think of that. Do you have any other suggestions except hints?
18:16:59 <blp> dceara: I think that an option or environment variable to specify the program to invoke would be suitable.
18:17:10 <dceara> blp, ack
18:17:30 <zhouhan> dceara: not yet, just brainstorming the problems :)
18:17:45 <dceara> zhouhan, ack :)
18:18:02 <blp> dceara: although for awful programs I sometimes end up doing PATH=/path/to/my/special/binary:$PATH to override their program choices, I don't like it ;-)
18:18:03 <dceara> blp, what about the conntrack related part?
18:19:21 <blp> dceara: Chasing the link, is the main point here that your patch hasn't been applied? Or is there something deeper?
18:19:22 <dceara> blp, I have the impression that providing hints for conntrack (and dp_hash) would make the tool quite user "unfriendly"
18:19:38 <dceara> blp, that patch addresses a small part of the problem
18:20:15 <dceara> blp, that's why I didn't push for it too much. It only improves the case when we hit a ct(nat(src/dst=...))
18:20:23 <blp> dceara: conntrack might be hard, but maybe it depends on what assumptions we can make in the common case.
18:20:48 <blp> dceara: In the common case, are we tracking down a problem where a packet is getting dropped? If so, then conntrack is a big ??? since it might be the thing doing the drop.
18:21:35 <dceara> blp, maybe adding support to conntrack for "what would happen to a packet like this?" is an alternative
18:22:47 <dceara> blp, To answer your question: I think yes, most of the times we'll be tracking down why a packet is dropped.
18:23:21 <dceara> blp, But it might be that we need to track why packets are forwarded to the wrong port (for example).
18:23:34 <blp> So, ovn-trace and ofproto/trace mostly play "what-if" games. This is fine as far as it goes.
18:23:40 <blp> We might need something else though.
18:24:03 <blp> We might need something where we can inject a real packet and then attempt to actually track where it really goes in the system.
18:24:42 <blp> NVP/NSX offer this feature to admins. I believe it gets used regularly.
18:25:02 <mmichelson> That would be a super-useful thing, I think.
18:25:08 <blp> Or, alternatively, a feature where we can "put a trace" on packets that match a particular specification.
18:25:22 <blp> "All my UDP to a.b.c.d is getting dropped, what's happening?"
18:25:31 <blp> "Where's my DNS going?"
18:25:32 <blp> etc.
18:25:53 <blp> There's already a way to log ACL matches, I think, this would be more advanced.
18:26:07 <dceara> blp, If the packet hits a datapath flow it might be quite hard to determine the oflow that created it and subsequently the OVN records that inserted it.
18:26:57 <blp> All the information to trace that back is there.
18:27:30 <dceara> blp, hmm, OK, I had the impression that we have a many-to-one relationship from oflow to dp-flow
18:27:42 <dceara> blp, I'll look more into that.
18:28:01 <blp> It's not always efficient to look it up.
18:28:59 <blp> It might be worth starting from the viewpoint, "What's the ideal debugging tool that I want to use in deployments?" and then see how close it can be approximated.
18:29:21 <dceara> blp, ack
18:30:06 <blp> I've always assumed we'd need a way to track real packets for debugging but I was curious how far we could take the ovn-trace model. Pretty far, it seems.
18:30:53 <dceara> blp, thanks for all the pointers. I'll take a step back and have a look at this from the other perspective too. It might turn up to be more useful and easier to implement in the end :)
18:32:41 <zhouhan> blp: we definitely need to track real packets, usually as a start in the debugging process. With OVN ECMP used, it is even not easy to tell which remote did the packet go. And ovs-dpctl dump-flows can't tell that in many cases.
18:33:28 <zhouhan> tcpdump used to help much, but with tunnel (especially, STT), this step is even troublesome
18:34:14 <zhouhan> (geneve has better support by linux, so it is more straightforward)
18:34:40 <zhouhan> dceara: It sounds like a big project, but I hope it can be started small and progesses made incrementally :)
18:36:01 <mmichelson> I think this is a good time to end the meeting. If people still want to talk, feel free to continue here or through e-mail
18:36:04 <mmichelson> #endmeeting