| russellb | azbiswas: yes, i did | 00:36 |
|---|---|---|
| russellb | azbiswas: the problems i was hitting at the time were a combination of 1) a bug in ovs+conntrack support. and 2) we needed to add some custom ACLs allowing DHCP. | 00:36 |
| russellb | are you having trouble? | 00:36 |
| azbiswas | we are having trouble with DHCP requests reaching the DHCP server | 00:39 |
| azbiswas | for provider networks | 00:39 |
| azbiswas | if the dhcp namespace is the only port in the network on a host, then the DHCP request does reach it, but if there are other VMs on the same host, then it's not reaching | 00:41 |
| russellb | interesting | 00:41 |
| russellb | is this a VM reaching a remote DHCP server? | 00:41 |
| azbiswas | We have bonding enabled as well, so it could be this - https://bugzilla.redhat.com/show_bug.cgi?id=1267291 | 00:41 |
| openstack | bugzilla.redhat.com bug 1267291 in openvswitch "[Openvswitch] balance-tcp bond mode causing issues with OSP Deployments" [High,New] - Assigned to lrichard | 00:41 |
| russellb | a DHCP server on another host | 00:41 |
| russellb | ok, i'm not familiar with bonding and OVS | 00:42 |
| russellb | you could try disabling that to eliminate it from the equation | 00:42 |
| azbiswas | right | 00:42 |
| azbiswas | i'm not in charge of the environment :), i can only observe and see what's happening. | 00:43 |
| russellb | OK | 00:43 |
| russellb | any digging yet? or just looking around to see if it was a known issue? | 00:44 |
| azbiswas | the frame was reaching the br-pub but not reaching the dhcp agent unless that dhcp agent was the only port on that network in the host. That indicated a broadcast problem. | 00:45 |
| russellb | and this is using up to date networking-ovn and ovs/ovn ? | 00:45 |
| russellb | this was a problem with provider networks several weeks ago | 00:45 |
| russellb | more than 1 port on the same provider net on the same host was broken | 00:46 |
| azbiswas | it's pretty recent i.e. the ebay/han zhou patch is there | 00:46 |
| russellb | ok, sounds recent enough. | 00:46 |
| russellb | (brb) | 00:46 |
| russellb | if you know it's hitting the remote host, then the next thing i'd be doing is trying to debug the flow table ... | 00:48 |
| russellb | i'd have a client generating a DHCP request regularly (every second or something) | 00:49 |
| russellb | and then i'd watch the counters in the flow table to see what's getting hit | 00:49 |
| russellb | that usually helps me spot where a packet is going | 00:49 |
| russellb | i really like this command, or some variant ... watch -d -n 0.5 "sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | cut -f3- -d','" | 00:49 |
| russellb | you could also grep for "metadata=0x1" (or whatever the right number is) to only look at flows for a certain datapath | 00:50 |
| azbiswas | we were doing something along that line, it was hitting the broadcast flow but not the one after. | 00:52 |
| azbiswas | The watch command is something I'll keep in mind. | 00:52 |
| russellb | sounds like you've narrowed it down a good bit | 00:53 |
| azbiswas | We don't seem to hitting this in other cloud systems, so it's not consistent either. There could be other stability issues. I wanted to confirm that the conntrack/ovs bug you found was fixed. | 00:53 |
| russellb | yes, the bug i was hitting back then was fixed | 00:54 |
| russellb | azbiswas: i believe it was https://github.com/openvswitch/ovs/commit/e37b8437e915a02a88116e5ea7af1a7e716bd597 | 00:58 |
| azbiswas | russellb: thanks | 00:59 |
| russellb | np | 01:04 |
| Sam-I-Am | russellb: weeeeekennnndddd | 01:06 |
| russellb | i have beer | 01:06 |
| Sam-I-Am | but you also have computer | 01:07 |
| russellb | it's true | 01:07 |
| Sam-I-Am | so, halfway there :) | 01:07 |
| Sam-I-Am | i need to weekend hard | 01:07 |
| Sam-I-Am | its been... rough | 01:07 |
| russellb | sorry to hear that | 01:07 |
| Sam-I-Am | new bike showed up today, so there's that | 01:07 |
| russellb | nice | 01:07 |
| Sam-I-Am | going to see how it rides tomorrow | 01:07 |
| russellb | i went to a bike store today for one of those tow your kid around trailers | 01:07 |
| Sam-I-Am | oh, fun | 01:08 |
| Sam-I-Am | thats good exercise | 01:08 |
| russellb | yeah, kid was scared of it first try, but we'll keep at it :) | 01:08 |
| Sam-I-Am | i'm scared of you | 01:08 |
| russellb | makes sense | 01:08 |
| Sam-I-Am | its a thing | 01:09 |
| russellb | it has been mid-70s here | 01:09 |
| russellb | <3 | 01:09 |
| Sam-I-Am | its been lovely here | 01:10 |
| Sam-I-Am | not getting out as much as i should be | 01:10 |
| Sam-I-Am | figure now that my road bike showed up, winter will make another appearance | 01:10 |
| Sam-I-Am | what really worries me is dry march-april here leads to fires | 01:10 |
| *** chandrav has quit IRC | 01:11 | |
| Sam-I-Am | i remember riding home from work in 85 degrees in march 2012... and the whole state lit up that year. | 01:11 |
| azbiswas | There are storms all over the left coast here, so there should be residual over the rockies | 01:11 |
| Sam-I-Am | stuff has sort of gone around here | 01:12 |
| Sam-I-Am | the mountains arent doing too bad, actually | 01:12 |
| Sam-I-Am | its the plains that are dry | 01:12 |
| Sam-I-Am | russellb: i think this beer thing is a good idea | 01:12 |
| Sam-I-Am | perhaps with a side of pizza | 01:12 |
| russellb | looks like the valley should be nice for my visit next week | 01:12 |
| * azbiswas thirsty | 01:13 | |
| russellb | sunny, upper 60s/70s | 01:13 |
| russellb | i had some nachos. | 01:13 |
| Sam-I-Am | russellb: with a side of donut? | 01:13 |
| russellb | no! | 01:13 |
| Sam-I-Am | a few weeks ago the donut shop had nutella-stuffed glazed donuts | 01:13 |
| Sam-I-Am | if you're going to donut, you might as well go all in | 01:14 |
| russellb | i thought i might code a little | 01:19 |
| russellb | but... | 01:19 |
| Sam-I-Am | heh | 01:21 |
| Sam-I-Am | i'm trying to avoid my roommate while he watches the ku baylor game | 01:21 |
| Sam-I-Am | things are going to get real here | 01:21 |
| russellb | ha | 01:22 |
| russellb | i'm almost due for another beer | 01:22 |
| Sam-I-Am | i just got one | 01:22 |
| russellb | and at that point there is 0% chance of productivity | 01:22 |
| Sam-I-Am | well, i dunno about that. killing off the weak brain cells let the others work more efficiencly. | 01:23 |
| Sam-I-Am | thats why you feel smarter after drinking beer | 01:23 |
| *** asti has joined #openstack-neutron-ovn | 01:24 | |
| russellb | you're on to something | 01:24 |
| Sam-I-Am | code reviews go much better with beer | 01:24 |
| Sam-I-Am | i need to uncomputer pretty hard this weekend. bikes. airplanes. in particular, washing an airplane. its pleasantly mindless, and when i get done, i have a feeling of accomplishment. | 01:25 |
| Sam-I-Am | might break out the RC helicopters after mothballing them for the winter | 01:25 |
| *** banix has joined #openstack-neutron-ovn | 01:27 | |
| *** jckasper has joined #openstack-neutron-ovn | 01:33 | |
| russellb | Sam-I-Am: sounds fun! | 01:36 |
| *** jckasper has quit IRC | 01:38 | |
| Sam-I-Am | russellb: yep. its amazing how liberating washing and waxing an airplane is. | 01:38 |
| *** asti has quit IRC | 01:40 | |
| *** fzdarsky|afk has quit IRC | 02:22 | |
| *** fzdarsky has joined #openstack-neutron-ovn | 02:28 | |
| *** gongysh has joined #openstack-neutron-ovn | 02:43 | |
| *** s3wong has quit IRC | 03:03 | |
| *** azbiswas has quit IRC | 03:08 | |
| *** irenab has quit IRC | 03:24 | |
| *** irenab has joined #openstack-neutron-ovn | 03:28 | |
| *** azbiswas has joined #openstack-neutron-ovn | 04:09 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:17 | |
| *** azbiswas has quit IRC | 04:24 | |
| *** azbiswas has joined #openstack-neutron-ovn | 04:24 | |
| *** azbiswas has quit IRC | 04:29 | |
| *** banix has quit IRC | 04:30 | |
| *** jckasper has quit IRC | 04:31 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:32 | |
| *** jckasper has quit IRC | 04:40 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:40 | |
| *** jckasper has quit IRC | 04:41 | |
| *** jckasper has joined #openstack-neutron-ovn | 04:41 | |
| *** azbiswas has joined #openstack-neutron-ovn | 05:27 | |
| *** azbiswas has quit IRC | 05:27 | |
| *** jckasper has quit IRC | 05:31 | |
| *** jckasper has joined #openstack-neutron-ovn | 05:31 | |
| *** jckasper has quit IRC | 05:33 | |
| *** jckasper has joined #openstack-neutron-ovn | 05:33 | |
| *** armax has quit IRC | 05:38 | |
| *** jckasper has quit IRC | 06:06 | |
| *** jckasper has joined #openstack-neutron-ovn | 06:06 | |
| *** jckasper has joined #openstack-neutron-ovn | 06:07 | |
| *** jckasper has quit IRC | 06:19 | |
| *** jckasper has joined #openstack-neutron-ovn | 06:19 | |
| *** jckasper has quit IRC | 06:19 | |
| *** jckasper has joined #openstack-neutron-ovn | 06:20 | |
| *** jckasper has quit IRC | 06:38 | |
| *** jckasper has joined #openstack-neutron-ovn | 06:38 | |
| *** jckasper has quit IRC | 06:43 | |
| *** jckasper has joined #openstack-neutron-ovn | 06:45 | |
| *** gangil has quit IRC | 07:15 | |
| *** gangil has joined #openstack-neutron-ovn | 07:17 | |
| *** gangil has joined #openstack-neutron-ovn | 07:17 | |
| *** jckasper has quit IRC | 07:27 | |
| *** jckasper has joined #openstack-neutron-ovn | 07:28 | |
| *** gongysh has quit IRC | 07:29 | |
| *** jckasper has quit IRC | 07:32 | |
| *** gangil has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-neutron-ovn | 10:19 | |
| *** yamamoto has quit IRC | 10:19 | |
| *** yamamoto has joined #openstack-neutron-ovn | 11:20 | |
| *** yamamoto has quit IRC | 11:22 | |
| *** yamamoto has joined #openstack-neutron-ovn | 11:22 | |
| *** yamamoto has quit IRC | 11:22 | |
| *** jckasper has joined #openstack-neutron-ovn | 11:29 | |
| *** jckasper has quit IRC | 11:33 | |
| *** yamamoto has joined #openstack-neutron-ovn | 11:50 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:22 | |
| *** yamamoto has quit IRC | 12:36 | |
| *** banix has joined #openstack-neutron-ovn | 12:52 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:14 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** numans has joined #openstack-neutron-ovn | 13:41 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:43 | |
| *** yamamoto has quit IRC | 13:43 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:44 | |
| *** numans has quit IRC | 13:46 | |
| *** banix has quit IRC | 13:50 | |
| *** asti has joined #openstack-neutron-ovn | 13:54 | |
| *** yamamoto has quit IRC | 13:58 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:59 | |
| *** asti has quit IRC | 14:07 | |
| *** banix has joined #openstack-neutron-ovn | 14:09 | |
| *** banix has quit IRC | 14:23 | |
| *** banix has joined #openstack-neutron-ovn | 14:24 | |
| *** banix has quit IRC | 14:56 | |
| *** yamamoto has quit IRC | 15:11 | |
| *** yamamoto has joined #openstack-neutron-ovn | 15:12 | |
| *** yamamoto has quit IRC | 15:16 | |
| *** irenab has quit IRC | 15:27 | |
| *** irenab has joined #openstack-neutron-ovn | 15:29 | |
| *** openstackgerrit_ has quit IRC | 15:30 | |
| *** jckasper has joined #openstack-neutron-ovn | 15:30 | |
| *** jckasper has quit IRC | 15:35 | |
| *** yamamoto has joined #openstack-neutron-ovn | 15:40 | |
| *** openstackgerrit_ has joined #openstack-neutron-ovn | 15:43 | |
| *** obondarev has joined #openstack-neutron-ovn | 16:08 | |
| *** obondarev_ has quit IRC | 16:12 | |
| *** markmcclain has quit IRC | 16:12 | |
| *** markmcclain has joined #openstack-neutron-ovn | 16:14 | |
| *** gangil has joined #openstack-neutron-ovn | 16:27 | |
| *** gangil has joined #openstack-neutron-ovn | 16:27 | |
| *** gangil has quit IRC | 16:31 | |
| *** armax has joined #openstack-neutron-ovn | 16:39 | |
| *** yamamoto has quit IRC | 17:00 | |
| *** dlundquist has quit IRC | 17:32 | |
| *** yamamoto has joined #openstack-neutron-ovn | 18:00 | |
| *** yamamoto has quit IRC | 18:11 | |
| *** armax has quit IRC | 19:01 | |
| *** jckasper has joined #openstack-neutron-ovn | 19:32 | |
| *** jckasper has quit IRC | 19:37 | |
| *** banix has joined #openstack-neutron-ovn | 20:02 | |
| *** banix has quit IRC | 20:06 | |
| openstackgerrit | Amitabha Biswas proposed openstack/networking-ovn: Reduce number of OVSDB row changes during ACL updates https://review.openstack.org/282488 | 22:14 |
| *** jckasper has joined #openstack-neutron-ovn | 23:34 | |
| *** jckasper has quit IRC | 23:38 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!