*** zhouhan_ has joined #openvswitch | 00:38 | |
*** zhouhan has quit IRC | 00:42 | |
*** markmcclain has quit IRC | 02:01 | |
*** yamamoto has joined #openvswitch | 02:01 | |
*** markmcclain has joined #openvswitch | 02:02 | |
*** dholler has quit IRC | 02:25 | |
*** yamamoto has quit IRC | 02:26 | |
*** yamamoto has joined #openvswitch | 02:27 | |
*** yamamoto has joined #openvswitch | 02:27 | |
*** dholler has joined #openvswitch | 02:38 | |
*** zhouhan_ has quit IRC | 02:43 | |
*** zhouhan has joined #openvswitch | 02:44 | |
*** rcernin has quit IRC | 02:48 | |
*** rcernin has joined #openvswitch | 03:00 | |
*** rcernin has quit IRC | 03:05 | |
*** armax has quit IRC | 03:17 | |
*** rcernin has joined #openvswitch | 03:21 | |
*** rcernin has quit IRC | 03:22 | |
*** rcernin has joined #openvswitch | 03:22 | |
*** yamamoto has quit IRC | 03:41 | |
*** yamamoto has joined #openvswitch | 03:49 | |
*** anilvenkata has joined #openvswitch | 04:57 | |
*** cpaelzer__ has joined #openvswitch | 05:17 | |
*** cpaelzer has quit IRC | 05:17 | |
*** acidfu_ has joined #openvswitch | 05:28 | |
*** acidfoo has quit IRC | 05:30 | |
*** numans has joined #openvswitch | 05:40 | |
*** cpaelzer__ is now known as cpaelzer | 05:45 | |
*** links has joined #openvswitch | 05:46 | |
*** eelco has joined #openvswitch | 06:02 | |
*** imaximets__ has joined #openvswitch | 06:06 | |
*** imaximets_ has quit IRC | 06:06 | |
*** links has quit IRC | 06:07 | |
*** links has joined #openvswitch | 06:12 | |
*** maciejjozefczyk has joined #openvswitch | 06:22 | |
*** blahdodo_ has quit IRC | 06:31 | |
*** links has quit IRC | 06:32 | |
*** blahdodo has joined #openvswitch | 06:35 | |
*** psahoo has joined #openvswitch | 06:36 | |
*** dmarchan1 is now known as dmarchand | 06:48 | |
*** mmirecki has joined #openvswitch | 06:55 | |
*** slaweq has joined #openvswitch | 07:01 | |
*** links has joined #openvswitch | 07:01 | |
*** slaweq has quit IRC | 07:06 | |
*** slaweq has joined #openvswitch | 07:07 | |
*** dceara has joined #openvswitch | 07:18 | |
*** maciejjozefczyk has quit IRC | 07:19 | |
*** maciejjozefczyk has joined #openvswitch | 07:37 | |
*** anilvenkata has quit IRC | 07:47 | |
*** anilvenkata has joined #openvswitch | 07:52 | |
*** aconstan has joined #openvswitch | 07:53 | |
*** rcernin has quit IRC | 07:54 | |
*** rcernin_ has joined #openvswitch | 07:55 | |
*** links has quit IRC | 08:00 | |
*** links has joined #openvswitch | 08:10 | |
*** rcernin_ has quit IRC | 08:20 | |
*** darkemon has quit IRC | 08:30 | |
*** darkemon has joined #openvswitch | 08:32 | |
*** yamamoto has quit IRC | 08:42 | |
*** yamamoto has joined #openvswitch | 08:42 | |
*** yamamoto has quit IRC | 09:12 | |
*** yamamoto has joined #openvswitch | 09:13 | |
*** yamamoto has joined #openvswitch | 09:23 | |
*** links has quit IRC | 09:27 | |
*** links has joined #openvswitch | 09:28 | |
*** yamamoto has quit IRC | 09:40 | |
*** psahoo has quit IRC | 09:53 | |
*** psahoo has joined #openvswitch | 10:09 | |
*** yamamoto has joined #openvswitch | 10:14 | |
*** psahoo_ has joined #openvswitch | 10:16 | |
*** yamamoto has quit IRC | 10:20 | |
*** psahoo has quit IRC | 10:20 | |
*** yamamoto has joined #openvswitch | 10:20 | |
*** yamamoto has quit IRC | 10:43 | |
*** yamamoto has joined #openvswitch | 10:44 | |
*** yamamoto has quit IRC | 10:44 | |
*** yamamoto has joined #openvswitch | 10:48 | |
*** rcernin_ has joined #openvswitch | 10:49 | |
*** yamamoto has quit IRC | 10:53 | |
*** yamamoto has joined #openvswitch | 10:54 | |
*** yamamoto has quit IRC | 10:59 | |
*** yamamoto has joined #openvswitch | 11:16 | |
*** yamamoto has quit IRC | 11:20 | |
*** yamamoto has joined #openvswitch | 11:34 | |
*** yamamoto has quit IRC | 12:12 | |
*** bostondriver has joined #openvswitch | 12:45 | |
*** donhw has quit IRC | 12:55 | |
*** apus has quit IRC | 13:04 | |
*** apus has joined #openvswitch | 13:04 | |
*** donhw has joined #openvswitch | 13:15 | |
*** armax has joined #openvswitch | 13:27 | |
*** yamamoto has joined #openvswitch | 13:28 | |
*** yamamoto has quit IRC | 13:34 | |
*** acidfu_ has quit IRC | 13:53 | |
*** billp_ has joined #openvswitch | 13:56 | |
*** dcbw has joined #openvswitch | 13:59 | |
*** billp has quit IRC | 14:00 | |
*** imaximets__ is now known as imaximets | 14:01 | |
*** KpuCko has quit IRC | 14:09 | |
*** rcernin_ has quit IRC | 14:14 | |
*** acidfu_ has joined #openvswitch | 14:51 | |
*** slaweq has quit IRC | 15:33 | |
*** links has quit IRC | 15:44 | |
*** eelco has quit IRC | 15:58 | |
*** jobewan has quit IRC | 16:09 | |
*** mmirecki has quit IRC | 16:11 | |
*** zhouhan has quit IRC | 16:37 | |
*** zhouhan has joined #openvswitch | 16:38 | |
*** armax has quit IRC | 16:48 | |
*** armax has joined #openvswitch | 16:49 | |
*** zhouhan_ has joined #openvswitch | 17:01 | |
*** zhouhan has quit IRC | 17:04 | |
*** zhouhan_ has quit IRC | 17:15 | |
*** zhouhan has joined #openvswitch | 17:16 | |
numans | Hello | 17:17 |
---|---|---|
panda | hi | 17:17 |
numans | mmichelson, meeting time ? | 17:17 |
flaviof | hi all | 17:17 |
dceara | Hi | 17:17 |
numans | Finding the tag to start the meeting. | 17:18 |
numans | #startmeeting ovn_community_development_discussion | 17:18 |
openstack | Meeting started Thu Jun 18 17:18:44 2020 UTC and is due to finish in 60 minutes. The chair is numans. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:18 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:18 |
openstack | The meeting name has been set to 'ovn_community_development_discussion' | 17:18 |
numans | Hello everyone. | 17:18 |
numans | Not sure if mmichelson is there or not. | 17:19 |
_lore_ | hi all | 17:19 |
numans | we can probably start. | 17:19 |
flaviof | o/ | 17:19 |
mmichelson | sorry I'm here | 17:19 |
mmichelson | Just got pulled away for a sec | 17:19 |
panda | o/ | 17:19 |
numans | mmichelson, I just started. All yours. | 17:19 |
mmichelson | OK, thanks | 17:19 |
mmichelson | Biggest thing is that last week we released 20.06.0 and 20.03.1 | 17:19 |
numans | thanks for the release. | 17:20 |
flaviof | woot! | 17:20 |
mmichelson | I noticed blp's patch series to OVS to remove insensitive language where possible. I think that at least in our documentation we probably should follow suit where it makes sense. | 17:20 |
numans | mmichelson, agree. I did a grep and find few instances of such words. | 17:20 |
mmichelson | So I did some searches for specific trigger words, and in documentation it's not too difficult to fix up | 17:21 |
numans | agree. | 17:21 |
mmichelson | Other than that, I've been chipping away at old patches of mine to try to get them in shape to be updated (case sensitivity in MAC and IPv6 addresses, ovs-scale-test plaintext client) | 17:21 |
mmichelson | And I've been reviewing | 17:22 |
mmichelson | That's all from me. Whoever wants to go next, feel free. | 17:22 |
numans | I can go real quick. | 17:22 |
numans | I got the ack from zhouhan for the v12 of I-P patches. | 17:22 |
numans | zhouhan thanks for the review. | 17:23 |
numans | waiting for dceara's comments if any. | 17:23 |
numans | I worked on a couple of patches and submitted for review. | 17:23 |
numans | One was to add packet marking for packets which got the router policies applied. | 17:23 |
zhouhan | numans: np | 17:23 |
numans | And did some reviews. | 17:24 |
numans | That's it from me. | 17:24 |
dceara | numans, ack, I'll try to have another look at the I-P patches tomorrow. | 17:24 |
_lore_ | can I go next? very quick | 17:24 |
numans | dceara, thanks. | 17:24 |
_lore_ | this week I mainly work on mtu issue in ovs/ovn | 17:25 |
*** mmirecki has joined #openvswitch | 17:25 | |
_lore_ | in particular if DF is not set, the sender is fragmenting the traffic after an ICMP error msg sent by OVN | 17:25 |
_lore_ | the issue is OVN still continues to send an ICMP error msg on fragmented traffic if we have connection tracking in ingress pipeline | 17:26 |
_lore_ | I figured out it is a issue in the ovs kernel datapath, I need to send the fix upstream | 17:26 |
numans | _lore_, thanks for fixing this. | 17:27 |
_lore_ | then I noticed the value we configured for check_packet_len is the frame size and not the mtu | 17:27 |
_lore_ | so I posted a patch for it | 17:27 |
_lore_ | numans: imaximets: I sent a v2 | 17:27 |
_lore_ | any comments on it? | 17:27 |
numans | _lore_, ack. | 17:27 |
numans | I don't have any. | 17:27 |
imaximets | _lore_, I didn't look yet. | 17:28 |
*** psahoo_ has quit IRC | 17:28 | |
*** Franky_T has joined #openvswitch | 17:28 | |
_lore_ | ack, actually in the current implementation we are wasting 14+4 bytes | 17:28 |
_lore_ | that's all from my side | 17:29 |
zhouhan | may I go next? | 17:31 |
numans | sure. | 17:32 |
zhouhan | we noticed another RAFT problem this week | 17:32 |
zhouhan | For some reason, one of the nodes in the cluster missed some transactions, and become inconsistent from the leader and the other node | 17:33 |
zhouhan | Restarting the node doesn't help, because the current logs are consistent with the cluster and updates can continue. | 17:34 |
numans | zhouhan, so the missed transactions are gone for ever ? | 17:34 |
zhouhan | The inconsistent part is in the snapshot, which is never going to be synced unless a install snapshot RPC is triggered, which doesn't happen usually. | 17:35 |
numans | ok | 17:35 |
zhouhan | numans: yes, for that node, the data is inconsistent for ever. So any clients connected to that server initially would get inconsistent data | 17:35 |
numans | zhouhan, its the leader ? | 17:36 |
zhouhan | only re-joining the node to the cluster would solve the issue. | 17:36 |
zhouhan | numans: no it is not the leader | 17:36 |
numans | zhouhan, ok. | 17:36 |
numans | so ovn-controller and ovn-northd will not see this inconsistency since they always connect to leader right ? | 17:37 |
dceara | zhouhan, Should we have a periodic consistency check to detect such cases earlier? | 17:37 |
zhouhan | For some time it is not even detected. However, once there is a transaction appending from the leader that need to touch the inconsistent part of the data, i.e. delete an unexisted row, the server would detect itself as inconsistent and then prevent any transaction through that node, and all the clients connected to that node would fail for ever. | 17:38 |
zhouhan | dceara: it detects as possible as it can from the server point of view. | 17:39 |
zhouhan | dceara: but it is not gracefully handled, and it still allows client to connect | 17:39 |
imaximets | zhouhan, but clients should reconnect to other correct server and sync with it. | 17:40 |
dceara | zhouhan, So is the problem that we allow new client connections even when in this state? | 17:40 |
zhouhan | The root cause of the inconsistent data is still not clear. One thing suspected to be triggering this is that the node rebooted by itself before this happens. | 17:40 |
zhouhan | dceara: that is one of the problem. But the first thing is how could the inconsitent data happen. I still have no clue. | 17:41 |
*** mmirecki has quit IRC | 17:41 | |
dceara | zhouhan, ack. | 17:42 |
zhouhan | imaximets: because of fast-resync, clients won't get the correct data unless it restarts | 17:42 |
zhouhan | so I am thinking maybe the client side detection added by dceara is still needed for such cases. | 17:43 |
imaximets | zhouhan, will the recent fix from dceara that I merged fix the issue with fast-resync in this case? | 17:43 |
dceara | zhouhan, but does the IDL detect the missing updates? | 17:43 |
imaximets | zhouhan, I mean, client will detect inconsistency eventually and disable fast-resync. | 17:44 |
zhouhan | imaximets: that fix is for conditional monitoring. In this case it is purely data inconsistency on server side, so that's not helpful | 17:44 |
imaximets | zhouhan, oh.. ok. | 17:44 |
zhouhan | imaximets: the IDL detection and disabling fast-resync (the last patch of the series) was not merged :) | 17:45 |
dceara | zhouhan, OK, but you should still see logs on ovn-controller about inexistent rows. Do you see those in your case? | 17:45 |
zhouhan | dceara: I guess it would detect, if there are transactions to trigger it. But we fixed them before it happens (by restarting the clients) | 17:46 |
imaximets | zhouhan, I understand. | 17:46 |
dceara | zhouhan, OK, I can address the comments from imaximets and send a new version of that patch then. | 17:46 |
zhouhan | I would thank Ali for reporting this issue (who may be not here in the channel today) | 17:46 |
zhouhan | That's my update :) | 17:46 |
mmichelson | OK, anybody else care to give an update? | 17:48 |
flaviof | May I go next? | 17:48 |
dceara | I just have a quick note for today: zhouhan my plan is to have a go at the lflow explosion reported by Girish for dnat_and_snat as soon as I get a chance. That's unless you started already on it. | 17:48 |
zhouhan | dceara: I am still not sure. It would help to self-correct in such situation, but we also need to make sure such problem is exposed without being hidden completely | 17:48 |
zhouhan | dceara: sure, thanks for helping on dnat_and_snat flow problem! | 17:49 |
dceara | zhouhan, that's why I was thinking of a periodic self check on the server side to see if the DBs are consistent. | 17:49 |
imaximets | zhouhan, dceara: I think we should report such issues loudly in logs with ERR log level at least. | 17:50 |
dceara | imaximets, ++ | 17:50 |
zhouhan | dceara: hmm, on server side, I am not sure what other check can be done, beside the current check when transaction detects inconsistency. | 17:51 |
*** Franky_T has quit IRC | 17:52 | |
zhouhan | dceara: in theory, the raft log should already ensure consistency. It must be a bug somewhere in some corner situation. | 17:52 |
dceara | zhouhan, I see, ok. | 17:52 |
zhouhan | imaximets: +1 for error logs | 17:52 |
dceara | zhouhan, imaximets: then i'll respin the patch and use error logs instead of the current WARN and we can continue the discussion on the ML (at least for the client side) | 17:53 |
zhouhan | imaximets: on server side, it already have error logs when it is detected, but not quite straightforward. It is only "syntax error: ..." | 17:53 |
zhouhan | dceara: sounds good | 17:53 |
imaximets | zhouhan, We might need to improve server side logs. | 17:54 |
imaximets | dceara, thanks. | 17:55 |
zhouhan | yeah, and better handling on disconnecting itself from the cluster in such case, I think | 17:55 |
*** Franky_T has joined #openvswitch | 17:55 | |
panda | zhouhan: is all this captured in some bug description ? | 17:56 |
zhouhan | panda: no, it is just here :) | 17:57 |
panda | zhouhan: ok. | 17:57 |
zhouhan | One way to detect such situation from monitoring point of view, is to compare the number of rows of particular tables, such as logical_flow and port_binding, periodically from each individual node. | 17:58 |
imaximets | panda, zhouhan: It's good that we have meeting logs. :) | 17:58 |
flaviof | ++ | 17:58 |
dceara | imaximets, zhouhan Shall we consider opening a github issue for this? | 17:58 |
mmichelson | Probably a good idea | 17:59 |
zhouhan | ++ | 17:59 |
zhouhan | We don't have an official way to track OVS bugs I guess | 17:59 |
zhouhan | Or even OVN bugs | 17:59 |
imaximets | it's usually just an e-mail thread. | 17:59 |
numans | github issues could be a starter here. | 18:00 |
mmichelson | Yeah, github issues make sense to me. THe project is on github, after all :) | 18:00 |
zhouhan | yes, email thread is good for discussion but doesn't provide a good track. Can we agree on github as bug tracking in the future? | 18:01 |
panda | zhouhan: rows may not be the right path, if you recevive two updates so the rows number remains the same, you are not detecting changes. | 18:01 |
flaviof | #action use https://github.com/ovn-org/ovn/issues as a way of tracking ovn bugs going forward | 18:01 |
zhouhan | panda: yes, it is just one indicator. When this happened, there were hundreds of rows difference in our case :) | 18:02 |
panda | zhouhan: yep ok. | 18:02 |
zhouhan | panda: of course it doesn't guarantee to detect all inconsistency by such monitoring | 18:02 |
dceara | zhouhan, maybe such a monitoring utility would be useful to have in the repo itself. It doesn't have to be 100% precise if it raises an alarm about a potential inconsistency. What do you think? | 18:03 |
zhouhan | dceara: +1 | 18:03 |
panda | I would try to use logical clocks f possible | 18:04 |
panda | but it might be a long term solution. | 18:04 |
zhouhan | or maybe implement a feature in raft to periodically compare snapshots between the servers | 18:05 |
imaximets | zhouhan, dceara: crazy idea: clinet IDL that connects to all cluster nodes at once and monitors the difference over time. | 18:06 |
panda | does the update use 2-part or 3parts commits ? | 18:06 |
zhouhan | I'd like to try to figure out the root cause after all :) | 18:06 |
mmichelson | imaximets, I was wondering if you might bring something like that up :) | 18:06 |
dceara | imaximets, that sounds cool! | 18:06 |
panda | zhouhan: yeah, taht would be the start :) | 18:06 |
imaximets | zhouhan, sure. root cause must be identified anyway. | 18:06 |
zhouhan | This was interesting discussion. Thanks all. | 18:07 |
* dceara has to run. Bye all! | 18:08 | |
* numans too. | 18:08 | |
flaviof | bye dceara ! | 18:08 |
zhouhan | (a pity that blp wasn't here) | 18:08 |
zhouhan | bye all | 18:08 |
mmichelson | bye! | 18:08 |
numans | bye | 18:08 |
panda | bye | 18:08 |
imaximets | bye | 18:08 |
dceara | Thanks! | 18:08 |
mmichelson | Seems a good place to end the meeting :) | 18:08 |
mmichelson | #endmeeting | 18:08 |
imaximets | didn't work. :) | 18:09 |
mmichelson | uhhh | 18:09 |
flaviof | maybe numans have to do it? | 18:09 |
panda | too late | 18:09 |
numans | #endmeeting | 18:09 |
openstack | Meeting ended Thu Jun 18 18:09:30 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:09 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-06-18-17.18.html | 18:09 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-06-18-17.18.txt | 18:09 |
openstack | Log: http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-06-18-17.18.log.html | 18:09 |
panda | this will be a 12hours meeting | 18:09 |
_lore_ | bye all | 18:09 |
panda | numans: thanks for the merge! | 18:09 |
panda | bye | 18:09 |
numans | flaviof, I thought any one can end the meeting | 18:10 |
numans | panda, welcome | 18:10 |
flaviof | numans: I think you have to finish what you started. ;) | 18:10 |
numans | flaviof, indeed :) | 18:10 |
numans | flaviof, Actually I was about to disappear | 18:10 |
*** factor__ has quit IRC | 18:11 | |
*** factor__ has joined #openvswitch | 18:11 | |
*** Franky_T has quit IRC | 18:16 | |
*** dholler has quit IRC | 18:22 | |
*** acidfoo_ has joined #openvswitch | 18:27 | |
*** acidfu_ has quit IRC | 18:29 | |
*** jpettit has joined #openvswitch | 18:39 | |
*** jpettit has quit IRC | 18:40 | |
*** maciejjozefczyk has quit IRC | 18:52 | |
*** zhouhan has quit IRC | 19:00 | |
dceara | imaximets, re the IDL inconsistency detection patch, based on the previous discussion during the OVN meeting, I'm wondering now if we should do an idl_retry even in cases where we could correct the inconsistency (e.g., deletion of missing row) | 19:15 |
*** zhouhan_ has joined #openvswitch | 19:16 | |
dceara | zhouhan_, not sure if you got this too so resending: <dceara> imaximets, re the IDL inconsistency detection patch, based on the previous discussion during the OVN meeting, I'm wondering now if we should do an idl_retry even in cases where we could correct the inconsistency (e.g., deletion of missing row) | 19:17 |
*** zhouhan_ has quit IRC | 19:19 | |
*** zhouhan_ has joined #openvswitch | 19:20 | |
*** zhouhan_ has quit IRC | 19:20 | |
*** zhouhan has joined #openvswitch | 19:21 | |
zhouhan | dceara: thanks, I didn't see that | 19:21 |
zhouhan | dceara: I am not what this means. The idea was to retry with last_id = 0, right? | 19:23 |
dceara | zhouhan: the patch set last_id = 0 and then called ovsdb_idl_retry which forces reconnect. But only if the update was trying to modify a missing row. In other cases, e.g., "add existing row" the code recovers by deleting and then readding | 19:26 |
dceara | the row | 19:26 |
dceara | zhouhan: I'm thinking that these cases also correspond to inconsistencies right? | 19:27 |
*** anilvenkata has quit IRC | 19:27 | |
zhouhan | ok, I see. Yes, I agree with that. | 19:27 |
dceara | zhouhan: ok, I'll give it a bit more thought before sending a new revision, thanks! | 19:28 |
zhouhan | dceara: thank you | 19:30 |
*** yamamoto has joined #openvswitch | 19:32 | |
*** yamamoto has quit IRC | 19:37 | |
*** zhouhan has quit IRC | 19:51 | |
*** mmichelson has quit IRC | 19:52 | |
*** zhouhan has joined #openvswitch | 19:52 | |
*** mmichelson has joined #openvswitch | 19:59 | |
*** zhouhan_ has joined #openvswitch | 20:05 | |
*** zhouhan has quit IRC | 20:05 | |
*** mmirecki has joined #openvswitch | 20:11 | |
*** zhouhan_ has quit IRC | 20:12 | |
*** zhouhan has joined #openvswitch | 20:13 | |
*** slaweq has joined #openvswitch | 20:23 | |
*** zhouhan has quit IRC | 20:30 | |
*** zhouhan has joined #openvswitch | 20:32 | |
*** mmirecki has quit IRC | 20:33 | |
*** mmirecki has joined #openvswitch | 20:36 | |
*** strondeak has joined #openvswitch | 20:41 | |
*** maciejjozefczyk has joined #openvswitch | 20:48 | |
*** strondeak has quit IRC | 21:14 | |
*** mmirecki has quit IRC | 21:16 | |
*** strondeak has joined #openvswitch | 21:17 | |
*** dcbw has quit IRC | 21:52 | |
*** rcernin_ has joined #openvswitch | 21:56 | |
*** zhouhan_ has joined #openvswitch | 21:58 | |
*** zhouhan has quit IRC | 21:58 | |
*** armax has quit IRC | 22:09 | |
*** rcernin_ has quit IRC | 22:13 | |
*** slaweq has quit IRC | 22:13 | |
*** aconstan has quit IRC | 22:39 | |
*** armax has joined #openvswitch | 22:55 | |
*** __lore__ has joined #openvswitch | 23:00 | |
*** _lore_ has quit IRC | 23:00 | |
*** rcernin has joined #openvswitch | 23:16 | |
*** bostondriver has quit IRC | 23:30 | |
*** yamamoto has joined #openvswitch | 23:34 | |
*** yamamoto has quit IRC | 23:39 | |
*** __lore__ is now known as _lore_ | 23:45 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!