| *** zhouhan_ has quit IRC | 00:39 | |
| *** zhouhan has joined #openvswitch | 00:41 | |
| *** anilvenkata has joined #openvswitch | 00:55 | |
| *** yamamoto has joined #openvswitch | 01:00 | |
| *** tbachman has quit IRC | 01:14 | |
| *** tbachman has joined #openvswitch | 01:15 | |
| *** ihrachys has quit IRC | 01:55 | |
| *** dholler has quit IRC | 02:10 | |
| *** acidfu has quit IRC | 02:24 | |
| *** dholler has joined #openvswitch | 02:24 | |
| *** acidfu has joined #openvswitch | 02:32 | |
| *** acidfu has quit IRC | 02:34 | |
| *** acidfu has joined #openvswitch | 02:36 | |
| *** acidfu has quit IRC | 02:50 | |
| *** acidfu has joined #openvswitch | 02:52 | |
| *** rcernin has quit IRC | 03:08 | |
| *** zhouhan_ has joined #openvswitch | 03:48 | |
| *** zhouhan has quit IRC | 03:49 | |
| *** zhouhan has joined #openvswitch | 03:50 | |
| *** ihrachys has joined #openvswitch | 03:51 | |
| *** zhouhan_ has quit IRC | 03:54 | |
| *** rcernin has joined #openvswitch | 03:55 | |
| *** rcernin has quit IRC | 03:55 | |
| *** rcernin has joined #openvswitch | 03:55 | |
| *** ihrachys has quit IRC | 03:56 | |
| *** links has joined #openvswitch | 04:50 | |
| *** acidfu has quit IRC | 04:50 | |
| *** acidfu has joined #openvswitch | 04:52 | |
| *** links has quit IRC | 05:12 | |
| *** acidfu has quit IRC | 05:20 | |
| *** links has joined #openvswitch | 05:21 | |
| *** acidfu has joined #openvswitch | 05:28 | |
| *** acidfu has quit IRC | 05:35 | |
| *** acidfu has joined #openvswitch | 05:35 | |
| *** acidfu has quit IRC | 05:50 | |
| *** acidfu has joined #openvswitch | 05:52 | |
| *** eelco has joined #openvswitch | 05:58 | |
| *** acidfu has quit IRC | 06:21 | |
| *** zhouhan_ has joined #openvswitch | 06:25 | |
| *** zhouhan has quit IRC | 06:25 | |
| *** blahdodo has quit IRC | 06:25 | |
| *** acidfu has joined #openvswitch | 06:29 | |
| *** blahdodo has joined #openvswitch | 06:29 | |
| *** acidfu has quit IRC | 06:51 | |
| *** acidfu has joined #openvswitch | 06:52 | |
| *** blahdodo has quit IRC | 06:54 | |
| *** blahdodo has joined #openvswitch | 06:55 | |
| *** blahdodo_ has joined #openvswitch | 06:58 | |
| *** blahdodo has quit IRC | 06:59 | |
| *** blahdodo_ has quit IRC | 07:02 | |
| *** blahdodo has joined #openvswitch | 07:03 | |
| *** dceara has quit IRC | 07:14 | |
| *** acidfu has quit IRC | 07:22 | |
| *** acidfu has joined #openvswitch | 07:26 | |
| *** dceara has joined #openvswitch | 07:43 | |
| *** factor has quit IRC | 07:46 | |
| *** factor has joined #openvswitch | 07:47 | |
| *** zhouhan_ has quit IRC | 07:55 | |
| *** zhouhan has joined #openvswitch | 07:56 | |
| *** rcernin has quit IRC | 08:00 | |
| *** yamamoto has quit IRC | 08:06 | |
| *** darkemon has quit IRC | 08:11 | |
| *** jaicaa has quit IRC | 08:13 | |
| *** darkemon has joined #openvswitch | 08:13 | |
| *** slaweq has quit IRC | 08:15 | |
| *** jaicaa has joined #openvswitch | 08:16 | |
| *** slaweq has joined #openvswitch | 08:20 | |
| *** yamamoto has joined #openvswitch | 08:23 | |
| *** acidfu has quit IRC | 08:25 | |
| *** yamamoto has quit IRC | 08:26 | |
| *** itandops has joined #openvswitch | 08:26 | |
| *** FH_thecat has quit IRC | 08:29 | |
| *** itandops has quit IRC | 08:29 | |
| *** FH_thecat has joined #openvswitch | 08:30 | |
| *** acidfu has joined #openvswitch | 08:33 | |
| *** yamamoto has joined #openvswitch | 08:36 | |
| *** yamamoto has quit IRC | 08:42 | |
| *** itandops has joined #openvswitch | 08:51 | |
| *** rcernin has joined #openvswitch | 08:56 | |
| *** mmirecki has joined #openvswitch | 09:05 | |
| *** factor has quit IRC | 09:16 | |
| *** factor has joined #openvswitch | 09:16 | |
| *** rcernin has quit IRC | 09:17 | |
| *** slaweq has quit IRC | 09:19 | |
| *** acidfu has quit IRC | 09:22 | |
| *** acidfu has joined #openvswitch | 09:35 | |
| *** acidfu has quit IRC | 09:36 | |
| *** ktraynor has quit IRC | 09:47 | |
| *** mmirecki has quit IRC | 09:49 | |
| *** acidfu has joined #openvswitch | 09:50 | |
| *** ktraynor has joined #openvswitch | 09:50 | |
| *** slaweq has joined #openvswitch | 09:58 | |
| *** yamamoto has joined #openvswitch | 09:59 | |
| *** yamamoto has quit IRC | 10:01 | |
| *** acidfu has quit IRC | 10:07 | |
| *** acidfu has joined #openvswitch | 10:11 | |
| *** inovas has joined #openvswitch | 10:16 | |
| *** slaweq has quit IRC | 10:20 | |
| *** acidfu has quit IRC | 10:23 | |
| *** acidfu has joined #openvswitch | 10:30 | |
| *** yamamoto has joined #openvswitch | 10:32 | |
| *** jraju__ has joined #openvswitch | 10:35 | |
| *** links has quit IRC | 10:36 | |
| *** acidfu has quit IRC | 10:37 | |
| *** yamamoto has quit IRC | 10:38 | |
| *** acidfu has joined #openvswitch | 10:50 | |
| *** acidfu has quit IRC | 10:51 | |
| *** acidfu has joined #openvswitch | 10:52 | |
| *** acidfoo has joined #openvswitch | 10:56 | |
| *** acidfu has quit IRC | 10:59 | |
| *** acidfoo has quit IRC | 11:07 | |
| *** acidfoo has joined #openvswitch | 11:11 | |
| *** acidfoo has quit IRC | 11:24 | |
| *** slaweq has joined #openvswitch | 11:24 | |
| *** acidfoo has joined #openvswitch | 11:31 | |
| *** yamamoto has joined #openvswitch | 11:35 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** acidfoo has quit IRC | 12:08 | |
| *** acidfoo has joined #openvswitch | 12:12 | |
| *** slaweq has quit IRC | 12:13 | |
| *** acidfoo has quit IRC | 12:23 | |
| *** yamamoto has joined #openvswitch | 12:28 | |
| *** edsousa_ has quit IRC | 12:32 | |
| *** edwarnicke has quit IRC | 12:32 | |
| *** edwarnicke has joined #openvswitch | 12:33 | |
| *** edsousa_ has joined #openvswitch | 12:33 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** yamamoto has joined #openvswitch | 12:34 | |
| *** itandops has quit IRC | 12:42 | |
| *** slaweq has joined #openvswitch | 12:43 | |
| *** slaweq has quit IRC | 12:47 | |
| *** bostondriver has joined #openvswitch | 12:48 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** yamamoto has joined #openvswitch | 12:50 | |
| *** bostondriver1 has joined #openvswitch | 12:50 | |
| *** bostondriver1 has quit IRC | 12:50 | |
| *** bostondriver1 has joined #openvswitch | 12:50 | |
| *** bostondriver has quit IRC | 12:52 | |
| *** itandops has joined #openvswitch | 12:58 | |
| *** acidfoo has joined #openvswitch | 12:58 | |
| *** yamamoto has quit IRC | 13:00 | |
| *** inovas has quit IRC | 13:05 | |
| *** acidfoo has quit IRC | 13:08 | |
| *** acidfoo has joined #openvswitch | 13:13 | |
| *** dholler has quit IRC | 13:19 | |
| *** troulouliou_div2 has quit IRC | 13:21 | |
| *** acidfoo has quit IRC | 13:23 | |
| *** troulouliou_div2 has joined #openvswitch | 13:24 | |
| *** troulouliou_div2 has quit IRC | 13:31 | |
| *** troulouliou_div2 has joined #openvswitch | 13:31 | |
| *** dholler has joined #openvswitch | 13:32 | |
| *** yamamoto has joined #openvswitch | 13:36 | |
| *** acidfoo has joined #openvswitch | 13:36 | |
| *** troulouliou_div2 has quit IRC | 13:39 | |
| *** acidfoo has quit IRC | 13:41 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** acidfoo has joined #openvswitch | 13:51 | |
| *** ihrachys has joined #openvswitch | 13:53 | |
| *** yamamoto has joined #openvswitch | 14:00 | |
| *** troulouliou_div2 has joined #openvswitch | 14:03 | |
| *** yamamoto has quit IRC | 14:05 | |
| *** thaller has quit IRC | 14:06 | |
| *** thaller has joined #openvswitch | 14:06 | |
| *** vkuramshin57 has joined #openvswitch | 14:08 | |
| *** vkuramshin57 has quit IRC | 14:19 | |
| *** dcbw has joined #openvswitch | 14:40 | |
| *** yamamoto has joined #openvswitch | 14:43 | |
| *** yamamoto has quit IRC | 14:48 | |
| *** troulouliou_div2 has quit IRC | 14:52 | |
| *** slaweq has joined #openvswitch | 14:52 | |
| *** factor has quit IRC | 14:52 | |
| *** jraju__ has quit IRC | 14:57 | |
| *** troulouliou_div2 has joined #openvswitch | 15:00 | |
| *** troulouliou_div2 has quit IRC | 15:00 | |
| *** troulouliou_div2 has joined #openvswitch | 15:00 | |
| *** apus has quit IRC | 15:13 | |
| *** itandops has quit IRC | 15:14 | |
| *** itandops has joined #openvswitch | 15:15 | |
| *** slaweq has quit IRC | 15:20 | |
| *** panda has quit IRC | 15:51 | |
| *** eelco has quit IRC | 15:53 | |
| *** panda has joined #openvswitch | 15:53 | |
| *** eelco has joined #openvswitch | 15:54 | |
| *** eelco has quit IRC | 15:57 | |
| *** dholler has quit IRC | 16:01 | |
| *** eelco has joined #openvswitch | 16:01 | |
| *** aconstan has quit IRC | 16:09 | |
| *** inovas has joined #openvswitch | 16:22 | |
| *** inovas has quit IRC | 16:23 | |
| *** eelco has quit IRC | 16:34 | |
| *** ktraynor has quit IRC | 17:06 | |
| *** aginwala has joined #openvswitch | 17:16 | |
| mmichelson | I'm going to start the meeting | 17:16 |
|---|---|---|
| mmichelson | #startmeeting ovn-community-development-discussion | 17:16 |
| openstack | Meeting started Thu Jun 11 17:16:35 2020 UTC and is due to finish in 60 minutes. The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:16 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:16 |
| openstack | The meeting name has been set to 'ovn_community_development_discussion' | 17:16 |
| panda | o/ | 17:16 |
| *** blp has joined #openvswitch | 17:17 | |
| blp | Hi! | 17:17 |
| numans | Hi | 17:17 |
| dceara | Hi | 17:18 |
| mmichelson | hi everyone! | 17:18 |
| flaviof | hi all | 17:18 |
| blp | Do we need an #ovn channel or is everyone happy with using #openvswitch? | 17:18 |
| mmichelson | blp, funny thing is I'm actually in a #ovn channel on freenode already | 17:18 |
| blp | Oh. Should the meeting be there? | 17:18 |
| zhouhan | hello | 17:18 |
| mmichelson | It's not very heavily populated | 17:18 |
| numans | The meeting bot is configured for #openvswitch I think. | 17:18 |
| mmichelson | blp, I'm ambivalent on the matter. We're not interrupting conversations in here really | 17:19 |
| blp | OK. | 17:19 |
| blp | No problem. | 17:19 |
| blp | Anyway, there's a discussion of OVN mailing lists going on in ovs-discuss. | 17:19 |
| blp | https://mail.openvswitch.org/pipermail/ovs-discuss/2020-June/050221.html | 17:19 |
| blp | Did you #startmeeting? | 17:20 |
| blp | I have not. | 17:20 |
| numans | I think mmichelson did | 17:20 |
| mmichelson | Yes I did | 17:20 |
| dceara | blp, just a quick question about "closed" lists. The archives stay open and available to everyone regardless if they subscribed or not, right? | 17:20 |
| blp | Yes. By "closed" I mean only "closed to posting by non-subscribers". | 17:20 |
| blp | Not "private". | 17:20 |
| dceara | blp, ack, thanks for confirming | 17:20 |
| blp | Numerous people have mentioned that and it took me by surprise because it never even occurred to me that we'd have private lists. | 17:21 |
| blp | I guess there's the "committers" and "security" lists but they're basically zero traffic. | 17:22 |
| *** yamamoto has joined #openvswitch | 17:23 | |
| *** ChanServ sets mode: +o blp | 17:23 | |
| *** blp has quit IRC | 17:23 | |
| mmichelson | OK, so I guess since blp dropped that's a signal that someone else is free to give their update. | 17:24 |
| *** blp has joined #openvswitch | 17:24 | |
| blp | Weird, Pidgin hung and I had to restart it. | 17:24 |
| *** ChanServ sets mode: +o blp | 17:24 | |
| mmichelson | blp, funny I had just said that since you had dropped someone else could give their update. But I'll re-yield the floor to you for now | 17:24 |
| *** blp has quit IRC | 17:25 | |
| *** blp has joined #openvswitch | 17:25 | |
| blp | OK I'm just not going to try to update the topic. It breaks Pidgin. | 17:25 |
| blp | Anyway, I was done anyway so I'll listen now. | 17:25 |
| *** itandops has left #openvswitch | 17:27 | |
| mmichelson | OK cool | 17:28 |
| mmichelson | I'll go real quick I suppose | 17:28 |
| *** yamamoto has quit IRC | 17:28 | |
| mmichelson | I released OVN 20.06.0 earlier this week, and I've got an ACK to release 20.03.1. I'll release it before the end of the day | 17:28 |
| mmichelson | I sent an email to ovs-announce but I think I'm not actually subscribed, so my email went to moderation. | 17:29 |
| mmichelson | And aside from that, it's been a lot of catch-up on existing patches of mine that are out there and need updating. Also did some reviews of patches from Han and Anton | 17:29 |
| mmichelson | That's about all from me. | 17:29 |
| flaviof | May I go next? I actually have more of a question on v20.06.0. | 17:30 |
| numans | mmichelson, I didn't see any email on ovs-announce | 17:30 |
| numans | and I don't see it in the archives too. | 17:31 |
| numans | flaviof, sure. | 17:31 |
| numans | mmichelson, can you please double check on that. | 17:31 |
| numans | And thanks for releasing 20.06 | 17:31 |
| flaviof | ++ | 17:32 |
| flaviof | Not too long ago, we changed OVN to use OF1.5 so it can handle dp_hash selection for load_balancer. | 17:32 |
| flaviof | #link https://github.com/ovn-org/ovn/commit/6ec0b82038052866533f12823fe410308b3e457a [commit] controller: Use OpenFlow version 1.5 | 17:32 |
| flaviof | #link https://mail.openvswitch.org/pipermail/ovs-discuss/2020-April/049896.html [discuss] OVN Load balancing algorithm | 17:32 |
| flaviof | So I think that we will have ovn-controller using OF1.5, starting on v20.06.0. | 17:32 |
| numans | mmichelson, oops. I didn't read your full message. | 17:32 |
| numans | flaviof, right. | 17:33 |
| flaviof | In the upgrade path from v20.03.0 to v20.06.0 br-int may be already created and using OF1.3... | 17:33 |
| flaviof | Do you foresee special handling needed? If so, would the proper thing to do is to have both OF1.3 and OF1.5 as supported versions? | 17:33 |
| flaviof | I'm assuming that the original code in the controller would only enable OF1.3. Please correct me if I'm wrong. | 17:33 |
| flaviof | We can take this to discuss ML, of course, but I just wanted a general idea if any of you have already covered this. | 17:34 |
| flaviof | And that is all I have. | 17:34 |
| mmichelson | flaviof, So a couple of things: First, I think that if ovn-controller is stopped and restarted, it will renegotiate the OF version with OVS. | 17:34 |
| mmichelson | Second, I'm pretty sure the documented update path involves restarting ovs-vswitchd, rather than updating OVN on top of a running OVS instance. I may be wrong about that, though. | 17:35 |
| mmichelson | Someone please correct me if I'm wrong on either of those two points | 17:36 |
| numans | flaviof, I know that devstack sets or was setting the protocol for the br-int in ovs db. | 17:36 |
| numans | if it doesn't set, I think it should work fine. | 17:36 |
| numans | or it should also set the protocol OF1.5 | 17:36 |
| numans | mmichelson, I think your first is correct. For the 2nd, I don't know. | 17:37 |
| numans | flaviof, I didn't find any issues in my testing. But I'll test it out again to be sure. | 17:37 |
| flaviof | Ack. My question is on what happens if br-int exists and it is configured to use OF1.3... What would we expect to happen? | 17:37 |
| numans | flaviof, let me test out that scenario and come back to you on that. | 17:38 |
| flaviof | numans++ TY | 17:38 |
| numans | I can go real quick next. | 17:38 |
| numans | if flaviof is done. | 17:39 |
| zhouhan | mmichelson: I didn't expect the point 2 either. | 17:39 |
| * flaviof is done. except for saying TY to mmichelson too ;) | 17:39 | |
| numans | I think for OVN, there is no need for CMS to set the protocol of the integration brdige | 17:39 |
| numans | As ovn-controller never reads that and try to use that as OF flow version. | 17:40 |
| numans | I've been working on I-P patches this week addressing review comments. | 17:40 |
| numans | Yesterday I applied first 3 patches of the series to master and branch-20.06 since I had 2 acks and a few rounds of reviews. | 17:41 |
| numans | Soon after dceara found 2 issues and fixed them. Thanks dceara for fixing and for the thorough reviews this week. | 17:41 |
| numans | I'm not sure if zhouhan had additional comments. | 17:41 |
| numans | zhouhan, If so, please let me know and I'll address them. | 17:41 |
| dceara | numans, no problem, sorry for not spotting the issues earlier though | 17:41 |
| numans | dceara, no worries. I've never tested that scenario. I will test that now. And you added a test case. | 17:42 |
| numans | that's great. | 17:42 |
| zhouhan | numans: I will take a look for v12 | 17:42 |
| numans | I submitted v12 of the series. I think there are in decent stage now. | 17:42 |
| numans | zhouhan, thanks. | 17:42 |
| blp | The one thing that *might* turn up about OF1.5 is that it's much less tested than OF1.3. | 17:43 |
| blp | Perhaps we will turn up some bugs or points of nonconformance. I hope not. | 17:43 |
| numans | Hope not :) | 17:43 |
| flaviof | +1 | 17:43 |
| numans | I also worked on ovs idl patch which tries to return a valid txn object if it can. Thanks zhouhan for the reviews so far. I've addressed your comments | 17:43 |
| numans | and submitted v3. | 17:43 |
| numans | Request to take a look - https://patchwork.ozlabs.org/project/openvswitch/patch/20200605083029.129633-1-numans@ovn.org/ | 17:44 |
| numans | blp, not sure if you got some time for this patch. but if you can, that would be great. | 17:44 |
| blp | numans: It's on my to-do list now. | 17:44 |
| numans | blp++ | 17:44 |
| numans | thanks. | 17:44 |
| numans | I have a couple of smaller patches for review. Request to take a look | 17:44 |
| numans | https://patchwork.ozlabs.org/project/openvswitch/patch/20200601092742.550334-1-numans@ovn.org/ | 17:44 |
| numans | https://patchwork.ozlabs.org/project/openvswitch/patch/20200430081000.751498-1-numans@ovn.org/ | 17:44 |
| numans | And that | 17:44 |
| numans | that | 17:45 |
| zhouhan | numans: sure, thx! | 17:45 |
| numans | that's it from me. | 17:45 |
| numans | thanks. | 17:45 |
| zhouhan | I can go next | 17:45 |
| zhouhan | Thanks imaximets for reviewing the dp_hash recirc fix | 17:45 |
| blp | OK, I added "es" to my "Review Numan's patch" to-do item. | 17:45 |
| zhouhan | imaximets: I replied your email. blp: please take a look as well, as requested by imaximets, too :) | 17:46 |
| blp | zhouhan: I missed the context. What should I look at? | 17:47 |
| zhouhan | imaximets: we can discuss offline, but the major point I wanted make is that the problem is caused only when there is a slowpath action required by the flow, so in that case I think always computing dp_hash in slowpath for the flow wouldn't have consistency issue. | 17:47 |
| zhouhan | blp: #link https://patchwork.ozlabs.org/project/openvswitch/patch/1589527067-91901-1-git-send-email-hzhou@ovn.org/ | 17:48 |
| zhouhan | blp: this is dp_hash issue we have discussed in a previous meeting. | 17:48 |
| zhouhan | I also sent a RFC series for avoiding the flow explosion problem reported by ovn-k8s project. | 17:49 |
| imaximets | zhouhan, ok. I understand. Still looks a bit dangerous for me, need a fresh look from blp. :) | 17:49 |
| zhouhan | Thanks mmichelson and numans for reviewing it. Sorry for missing some documentation in the RFC. | 17:50 |
| zhouhan | imaximets: ok, thx | 17:50 |
| numans | zhouhan, that's fine. I think you wanted Girish to test them first. | 17:50 |
| zhouhan | numans: yeah, I wanted to send out as early as possible so that they can try. | 17:50 |
| zhouhan | that's it from me | 17:51 |
| blp | zhouhan: OK, will do. | 17:51 |
| zhouhan | blp: thx | 17:51 |
| mmichelson | OK anyone else want to give an update? | 17:52 |
| blp | imaximets: Thanks so much for coming up with a simpler RCU fix. | 17:52 |
| dceara | a oneliner from me this week: I started a thread on ovs-discuss about the current state of packet tracing in OVS/OVN. Feedback and suggestions are much appreciated: https://mail.openvswitch.org/pipermail/ovs-discuss/2020-June/050205.html | 17:52 |
| dceara | Thanks! | 17:52 |
| numans | dceara++ for working on this. | 17:53 |
| imaximets | blp: no problem. Thanks for review. | 17:53 |
| zhouhan | dceara: I saw it, that's wonderful | 17:53 |
| aginwala | thanks dceara ++ from my side too :) | 17:54 |
| blp | dceara: Is it worth discussing this here or would you prefer to keep it in email? | 17:54 |
| blp | It's a big email so sometimes I find it exhausting to try to reply to every point at once, yet I feel bad if I only reply to one point in a long series. | 17:54 |
| blp | And so sometimes that results in me just ignoring things I should help with. | 17:55 |
| dceara | blp, i'm ok either way (ML or here) | 17:55 |
| blp | A discussion, however, could be helpful, if you're up for it here. | 17:55 |
| dceara | blp, sure | 17:55 |
| blp | Last point first: it should be possible to make the output format better. | 17:56 |
| blp | IIRC, ovn-trace has multiple output formats because it has a decent internal data structure that can be serialized multiple ways. | 17:56 |
| blp | ofproto/trace used to be very primitive in that respect, but I think I redid it a few years ago to use a similar internal data structure. | 17:56 |
| blp | So it might be possible to even just mostly copy code from ovn-trace into ofproto/trace to get a json output format. | 17:57 |
| dceara | blp, yes, I looked into it too a bit and it should be relatively straightforward indeed | 17:57 |
| * numans also thinks that we can add --json option to most of the ovn (and ovs) utilities to output. That way ovn-kubernetes, ovn-scale-test etc can easily parse the output. | 17:58 | |
| blp | So, the other thing that occurs to me is that ovn-trace already knows how to talk to OVS instances, via the --ovs option. | 17:59 |
| blp | It might be better to teach it to ofproto/trace than to create a new utility to do that. | 18:00 |
| mmichelson | numans, all DB operations have json as an option. I actually used it when adding support for unidling in ovn-kubernetes | 18:00 |
| * numans is not happy with his ignorance. | 18:01 | |
| numans | mmichelson, thanks. I haven't noticed that. | 18:01 |
| mmichelson | The json schema is...interesting, to say the least :) | 18:01 |
| blp | ovn-trace, after all, already does a number of the operations listed in the description of ovn-global-trace. | 18:01 |
| blp | Currently it doesn't know how to SSH or to connect to ovs-appctl, but it could learn; any other implementation would have to do that too. | 18:02 |
| dceara | blp, ok, that's possible too. I guess we'd drop the need for ovn-detrace then (for this use case at least). | 18:02 |
| blp | dceara: yes | 18:03 |
| blp | I think that the overall idea is a good one, it puts together a lot of the stuff that is now in bits and pieces. | 18:04 |
| dceara | blp, I agree that a single ovn-trace tool would be ideal. | 18:04 |
| blp | I think the shakiest part is probably the SSH interface. I guess popen("ssh...") would be the way to do that. | 18:04 |
| flaviof | blp: dceara: so would would think that using ssh is better than using geneve? I think it makes sense so data and ctrl are completely separate, right? | 18:06 |
| numans | I agree with single ovn-trace. But if ovn-global-trace is written in python then probably it will be easier to do ssh and other stuff | 18:06 |
| dceara | blp, would it be acceptable to have the connection to the node be externally setup somehow (e.g., socat a unix connection)? | 18:06 |
| blp | flaviof: We're talking about running ofproto/trace. That has to happen over SSH because OVS only listens on a unix domain socket for that kind of request. | 18:06 |
| flaviof | blp: ack! | 18:07 |
| blp | dceara: I don't want to prescribe anything, I was just assuming SSH was the best way. | 18:08 |
| *** factor has joined #openvswitch | 18:08 | |
| imaximets | blp: theoretically we could allow appctl over ssl. | 18:08 |
| * imaximets hides | 18:08 | |
| dceara | imaximets, :) | 18:08 |
| blp | imaximets: I think you can do that today if you specify --unixctl to a pssl: socket, but I think we only support a single such socket so you'd lose the unix domain socket support if you did that. | 18:10 |
| blp | It looks like the discussion here is slowing down, please @ me if you want me. | 18:11 |
| zhouhan | dceara: you mentioned recirculation. I think that's also important for the debugging tool to be useful. It includes not only conntrack, but also dp_hash :) | 18:11 |
| zhouhan | not sure if there is a better idea other than passing an hint | 18:12 |
| imaximets | blp: it should be possible with a little magic to specify --unixctl pssl:<>,unix:<>. But anyway, I don't think it's a way good way for production deployment. | 18:12 |
| blp | imaximets: +1 | 18:13 |
| dceara | blp, popen is fine I guess, we probably need to be a bit flexible and allow the protocol to be controlled by the user though. The node running OVS might not be running an ssh server. | 18:13 |
| dceara | zhouhan, good point, we should find a solution for dp_hash too. However with dp_hash I think hints are ok. The chances that you choose a "wrong" bucket are quite small. I would expect all buckets to produce the same behavior in a sane deployment. | 18:14 |
| * numans waves and says goodbye to everyone. | 18:15 | |
| panda | numans: bye, and thanks for the reviews | 18:15 |
| zhouhan | dceara: but if we are talking about the global tracing, a wrong bucket would lead to tracing on a wrong remote (at least in the case of ECMP) | 18:15 |
| * zhouhan waves bye to numan | 18:16 | |
| dceara | numans, bye! | 18:16 |
| dceara | zhouhan, great point I didn't think of that. Do you have any other suggestions except hints? | 18:16 |
| blp | dceara: I think that an option or environment variable to specify the program to invoke would be suitable. | 18:16 |
| dceara | blp, ack | 18:17 |
| zhouhan | dceara: not yet, just brainstorming the problems :) | 18:17 |
| dceara | zhouhan, ack :) | 18:17 |
| blp | dceara: although for awful programs I sometimes end up doing PATH=/path/to/my/special/binary:$PATH to override their program choices, I don't like it ;-) | 18:18 |
| dceara | blp, what about the conntrack related part? | 18:18 |
| *** Franky_T has joined #openvswitch | 18:19 | |
| blp | dceara: Chasing the link, is the main point here that your patch hasn't been applied? Or is there something deeper? | 18:19 |
| dceara | blp, I have the impression that providing hints for conntrack (and dp_hash) would make the tool quite user "unfriendly" | 18:19 |
| dceara | blp, that patch addresses a small part of the problem | 18:19 |
| dceara | blp, that's why I didn't push for it too much. It only improves the case when we hit a ct(nat(src/dst=...)) | 18:20 |
| blp | dceara: conntrack might be hard, but maybe it depends on what assumptions we can make in the common case. | 18:20 |
| blp | dceara: In the common case, are we tracking down a problem where a packet is getting dropped? If so, then conntrack is a big ??? since it might be the thing doing the drop. | 18:20 |
| dceara | blp, maybe adding support to conntrack for "what would happen to a packet like this?" is an alternative | 18:21 |
| *** jaicaa_ has joined #openvswitch | 18:22 | |
| dceara | blp, To answer your question: I think yes, most of the times we'll be tracking down why a packet is dropped. | 18:22 |
| *** jaicaa has quit IRC | 18:22 | |
| dceara | blp, But it might be that we need to track why packets are forwarded to the wrong port (for example). | 18:23 |
| blp | So, ovn-trace and ofproto/trace mostly play "what-if" games. This is fine as far as it goes. | 18:23 |
| blp | We might need something else though. | 18:23 |
| blp | We might need something where we can inject a real packet and then attempt to actually track where it really goes in the system. | 18:24 |
| blp | NVP/NSX offer this feature to admins. I believe it gets used regularly. | 18:24 |
| mmichelson | That would be a super-useful thing, I think. | 18:25 |
| blp | Or, alternatively, a feature where we can "put a trace" on packets that match a particular specification. | 18:25 |
| blp | "All my UDP to a.b.c.d is getting dropped, what's happening?" | 18:25 |
| blp | "Where's my DNS going?" | 18:25 |
| blp | etc. | 18:25 |
| blp | There's already a way to log ACL matches, I think, this would be more advanced. | 18:25 |
| dceara | blp, If the packet hits a datapath flow it might be quite hard to determine the oflow that created it and subsequently the OVN records that inserted it. | 18:26 |
| blp | All the information to trace that back is there. | 18:26 |
| dceara | blp, hmm, OK, I had the impression that we have a many-to-one relationship from oflow to dp-flow | 18:27 |
| dceara | blp, I'll look more into that. | 18:27 |
| blp | It's not always efficient to look it up. | 18:28 |
| blp | It might be worth starting from the viewpoint, "What's the ideal debugging tool that I want to use in deployments?" and then see how close it can be approximated. | 18:28 |
| dceara | blp, ack | 18:29 |
| blp | I've always assumed we'd need a way to track real packets for debugging but I was curious how far we could take the ovn-trace model. Pretty far, it seems. | 18:30 |
| dceara | blp, thanks for all the pointers. I'll take a step back and have a look at this from the other perspective too. It might turn up to be more useful and easier to implement in the end :) | 18:30 |
| zhouhan | blp: we definitely need to track real packets, usually as a start in the debugging process. With OVN ECMP used, it is even not easy to tell which remote did the packet go. And ovs-dpctl dump-flows can't tell that in many cases. | 18:32 |
| zhouhan | tcpdump used to help much, but with tunnel (especially, STT), this step is even troublesome | 18:33 |
| zhouhan | (geneve has better support by linux, so it is more straightforward) | 18:34 |
| zhouhan | dceara: It sounds like a big project, but I hope it can be started small and progesses made incrementally :) | 18:34 |
| mmichelson | I think this is a good time to end the meeting. If people still want to talk, feel free to continue here or through e-mail | 18:36 |
| mmichelson | #endmeeting | 18:36 |
| dceara | zhouhan, blp, I guess we need to decide on a direction first. I'll think more about the tracking of real packets though and start a discussion about that alternative on the ML | 18:36 |
| openstack | Meeting ended Thu Jun 11 18:36:04 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:36 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-06-11-17.16.html | 18:36 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-06-11-17.16.txt | 18:36 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-06-11-17.16.log.html | 18:36 |
| flaviof | bye all | 18:36 |
| zhouhan | bye all | 18:36 |
| imaximets | bye | 18:37 |
| dceara | bye | 18:37 |
| *** bostondriver1 has quit IRC | 18:39 | |
| *** bostondriver has joined #openvswitch | 18:39 | |
| *** thaller has quit IRC | 18:47 | |
| *** thaller has joined #openvswitch | 18:48 | |
| *** aginwala has quit IRC | 18:56 | |
| *** dceara has quit IRC | 18:59 | |
| *** Franky_T has quit IRC | 19:01 | |
| *** dceara has joined #openvswitch | 19:06 | |
| *** inovas has joined #openvswitch | 19:13 | |
| dceara | blp, If you're still here, if we'd be able to mark a packet for "tracking" do you think forcing an upcall for such packets for every ovs action and after recirculation would be an option? | 19:13 |
| blp | dceara: Depending on the details, that might be reasonable. | 19:24 |
| dceara | blp, cool, I'll start digging more into this then. Thanks again for the guidance :) | 19:24 |
| blp | dceara: I would worry about it if there was a risk of ending up doing that to every packet that passes through the switch. If it would only happen to specially marked (e.g.) packets, it's reasonable. | 19:24 |
| dceara | blp, definitely only for specific packets | 19:25 |
| *** zhouhan_ has joined #openvswitch | 19:30 | |
| *** zhouhan has quit IRC | 19:34 | |
| *** slaweq has joined #openvswitch | 19:49 | |
| *** dceara has quit IRC | 20:15 | |
| *** slaweq has quit IRC | 20:26 | |
| *** yamamoto has joined #openvswitch | 21:26 | |
| *** yamamoto has quit IRC | 21:32 | |
| *** troulouliou_div2 has quit IRC | 21:33 | |
| *** bostondriver has quit IRC | 21:35 | |
| *** acidfoo has quit IRC | 21:36 | |
| *** acidfoo has joined #openvswitch | 21:37 | |
| *** dcbw has quit IRC | 21:37 | |
| *** JamesBenson has quit IRC | 21:38 | |
| *** acidfoo has quit IRC | 21:45 | |
| *** acidfoo has joined #openvswitch | 21:47 | |
| *** rcernin has joined #openvswitch | 21:49 | |
| *** blp has quit IRC | 22:01 | |
| *** dceara has joined #openvswitch | 22:06 | |
| *** rebrec has quit IRC | 22:14 | |
| *** yamamoto has joined #openvswitch | 23:03 | |
| *** dceara has quit IRC | 23:09 | |
| *** rcernin has quit IRC | 23:48 | |
| *** rcernin has joined #openvswitch | 23:51 | |
| *** yamamoto has quit IRC | 23:53 | |
| *** acidfu_ has joined #openvswitch | 23:58 | |
| *** acidfoo has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!