| *** bdeschenes has joined #openstack-operators | 00:00 | |
| *** racedo has joined #openstack-operators | 00:02 | |
| *** VW has joined #openstack-operators | 00:06 | |
| *** bdeschenes has quit IRC | 00:09 | |
| *** VW has quit IRC | 00:11 | |
| *** ducttape_ has joined #openstack-operators | 00:11 | |
| *** rmcall has quit IRC | 00:14 | |
| *** rmcall has joined #openstack-operators | 00:15 | |
| *** rmcall has quit IRC | 00:24 | |
| *** rmcall has joined #openstack-operators | 00:25 | |
| *** itsuugo has quit IRC | 00:25 | |
| *** itsuugo has joined #openstack-operators | 00:28 | |
| *** cdelatte has quit IRC | 00:29 | |
| *** racedo has quit IRC | 00:32 | |
| *** ducttape_ has quit IRC | 00:34 | |
| *** Apoorva_ has joined #openstack-operators | 00:37 | |
| *** Apoorva_ has quit IRC | 00:41 | |
| *** itsuugo has quit IRC | 00:41 | |
| *** Apoorva has quit IRC | 00:41 | |
| *** ducttape_ has joined #openstack-operators | 00:42 | |
| *** itsuugo has joined #openstack-operators | 00:43 | |
| *** markvoelker has joined #openstack-operators | 00:45 | |
| *** markvoelker has quit IRC | 00:49 | |
| *** itsuugo has quit IRC | 00:49 | |
| *** itsuugo has joined #openstack-operators | 00:52 | |
| *** maticue has quit IRC | 00:53 | |
| *** ducttape_ has quit IRC | 01:02 | |
| *** itsuugo has quit IRC | 01:07 | |
| *** itsuugo has joined #openstack-operators | 01:08 | |
| *** itsuugo has quit IRC | 01:18 | |
| *** itsuugo has joined #openstack-operators | 01:19 | |
| *** itsuugo has quit IRC | 01:23 | |
| *** itsuugo has joined #openstack-operators | 01:25 | |
| *** saneax is now known as saneax-_-|AFK | 01:28 | |
| *** VW has joined #openstack-operators | 01:39 | |
| *** r-daneel has quit IRC | 01:42 | |
| *** VW has quit IRC | 01:44 | |
| *** markvoelker has joined #openstack-operators | 01:46 | |
| *** britthouser has quit IRC | 01:49 | |
| *** markvoelker has quit IRC | 01:50 | |
| *** alop has joined #openstack-operators | 02:00 | |
| *** ducttape_ has joined #openstack-operators | 02:03 | |
| *** ducttape_ has quit IRC | 02:08 | |
| *** itsuugo has quit IRC | 02:16 | |
| *** itsuugo has joined #openstack-operators | 02:16 | |
| *** itsuugo has quit IRC | 02:21 | |
| *** itsuugo has joined #openstack-operators | 02:22 | |
| *** itsuugo has quit IRC | 02:27 | |
| *** mv_ has joined #openstack-operators | 02:28 | |
| *** itsuugo has joined #openstack-operators | 02:29 | |
| *** mv_ has quit IRC | 02:29 | |
| *** mv_ has joined #openstack-operators | 02:30 | |
| *** mv_ is now known as roadst4r | 02:30 | |
| *** itsuugo has quit IRC | 02:34 | |
| *** itsuugo has joined #openstack-operators | 02:34 | |
| *** itsuugo has quit IRC | 02:44 | |
| *** itsuugo has joined #openstack-operators | 02:45 | |
| *** markvoelker has joined #openstack-operators | 02:46 | |
| *** itsuugo has quit IRC | 02:50 | |
| *** itsuugo has joined #openstack-operators | 02:50 | |
| *** markvoelker has quit IRC | 02:51 | |
| *** alop has quit IRC | 02:58 | |
| *** itsuugo has quit IRC | 02:59 | |
| *** itsuugo has joined #openstack-operators | 03:00 | |
| *** david-lyle has quit IRC | 03:03 | |
| *** itsuugo has quit IRC | 03:05 | |
| *** itsuugo has joined #openstack-operators | 03:05 | |
| *** itsuugo has quit IRC | 03:10 | |
| *** itsuugo has joined #openstack-operators | 03:11 | |
| *** ducttape_ has joined #openstack-operators | 03:11 | |
| *** rmcall_ has joined #openstack-operators | 03:21 | |
| *** rmcall has quit IRC | 03:22 | |
| *** rmcall_ is now known as rmcall | 03:22 | |
| *** itsuugo has quit IRC | 03:27 | |
| *** itsuugo has joined #openstack-operators | 03:28 | |
| *** cartik has joined #openstack-operators | 03:28 | |
| *** itsuugo has quit IRC | 03:33 | |
| *** itsuugo has joined #openstack-operators | 03:34 | |
| *** ducttape_ has quit IRC | 03:35 | |
| *** itsuugo has quit IRC | 03:43 | |
| *** itsuugo has joined #openstack-operators | 03:44 | |
| *** itsuugo has quit IRC | 03:49 | |
| *** itsuugo has joined #openstack-operators | 03:51 | |
| *** itsuugo has quit IRC | 03:56 | |
| *** itsuugo has joined #openstack-operators | 03:57 | |
| *** itsuugo has quit IRC | 04:04 | |
| *** armax has quit IRC | 04:05 | |
| *** itsuugo has joined #openstack-operators | 04:06 | |
| *** itsuugo has quit IRC | 04:11 | |
| *** itsuugo has joined #openstack-operators | 04:13 | |
| *** itsuugo has quit IRC | 04:20 | |
| *** itsuugo has joined #openstack-operators | 04:22 | |
| *** itsuugo has quit IRC | 04:26 | |
| *** itsuugo has joined #openstack-operators | 04:28 | |
| *** ducttape_ has joined #openstack-operators | 04:36 | |
| *** ducttape_ has quit IRC | 04:41 | |
| *** itsuugo has quit IRC | 04:43 | |
| *** itsuugo has joined #openstack-operators | 04:44 | |
| *** itsuugo has quit IRC | 04:49 | |
| *** itsuugo has joined #openstack-operators | 04:51 | |
| *** itsuugo has quit IRC | 04:59 | |
| *** itsuugo has joined #openstack-operators | 05:00 | |
| *** saneax-_-|AFK is now known as saneax | 05:03 | |
| *** admin0 has joined #openstack-operators | 05:14 | |
| *** itsuugo has quit IRC | 05:15 | |
| *** itsuugo has joined #openstack-operators | 05:15 | |
| *** itsuugo has quit IRC | 05:20 | |
| *** itsuugo has joined #openstack-operators | 05:21 | |
| *** itsuugo has quit IRC | 05:28 | |
| *** itsuugo has joined #openstack-operators | 05:29 | |
| *** itsuugo has quit IRC | 05:34 | |
| *** cartik has quit IRC | 05:34 | |
| *** admin0 has quit IRC | 05:36 | |
| *** itsuugo has joined #openstack-operators | 05:36 | |
| *** itsuugo has quit IRC | 05:43 | |
| *** itsuugo has joined #openstack-operators | 05:45 | |
| *** itsuugo has quit IRC | 05:56 | |
| *** itsuugo has joined #openstack-operators | 05:57 | |
| *** ducttape_ has joined #openstack-operators | 06:03 | |
| *** rcernin has joined #openstack-operators | 06:07 | |
| *** ducttape_ has quit IRC | 06:08 | |
| *** itsuugo has quit IRC | 06:10 | |
| *** itsuugo has joined #openstack-operators | 06:11 | |
| *** itsuugo has quit IRC | 06:16 | |
| *** itsuugo has joined #openstack-operators | 06:18 | |
| *** sudipto_ has joined #openstack-operators | 06:34 | |
| *** sudswas__ has joined #openstack-operators | 06:36 | |
| *** AlexeyAbashkin has quit IRC | 06:44 | |
| *** pilgrimstack has joined #openstack-operators | 06:46 | |
| *** AlexeyAbashkin has joined #openstack-operators | 06:46 | |
| *** rarcea has joined #openstack-operators | 07:04 | |
| *** ducttape_ has joined #openstack-operators | 07:04 | |
| *** admin0 has joined #openstack-operators | 07:05 | |
| *** ducttape_ has quit IRC | 07:09 | |
| *** pilgrimstack has quit IRC | 07:13 | |
| *** matrohon has joined #openstack-operators | 07:28 | |
| *** pilgrimstack has joined #openstack-operators | 07:45 | |
| *** bsilverman has joined #openstack-operators | 08:05 | |
| *** itsuugo has quit IRC | 08:13 | |
| *** itsuugo has joined #openstack-operators | 08:14 | |
| *** itsuugo has quit IRC | 08:35 | |
| *** itsuugo has joined #openstack-operators | 08:36 | |
| *** sudipto_ has quit IRC | 08:38 | |
| *** sudswas__ has quit IRC | 08:38 | |
| *** sudswas__ has joined #openstack-operators | 08:40 | |
| *** derekh has joined #openstack-operators | 08:41 | |
| *** I has joined #openstack-operators | 08:41 | |
| *** I is now known as Guest56480 | 08:42 | |
| *** itsuugo has quit IRC | 08:43 | |
| *** itsuugo has joined #openstack-operators | 08:44 | |
| *** VW has joined #openstack-operators | 08:46 | |
| *** godog has quit IRC | 08:48 | |
| *** godog has joined #openstack-operators | 08:48 | |
| *** itsuugo has quit IRC | 08:51 | |
| *** VW has quit IRC | 08:51 | |
| *** itsuugo has joined #openstack-operators | 08:52 | |
| *** itsuugo has quit IRC | 08:57 | |
| *** itsuugo has joined #openstack-operators | 08:58 | |
| *** ducttape_ has joined #openstack-operators | 09:06 | |
| *** ducttape_ has quit IRC | 09:11 | |
| *** pilgrimstack has quit IRC | 09:11 | |
| *** pilgrimstack has joined #openstack-operators | 09:14 | |
| *** arif-ali has quit IRC | 09:17 | |
| *** itsuugo has quit IRC | 09:18 | |
| *** arif-ali has joined #openstack-operators | 09:20 | |
| *** itsuugo has joined #openstack-operators | 09:20 | |
| *** sticker__ is now known as sticker | 09:22 | |
| *** ig0r_ has joined #openstack-operators | 09:23 | |
| *** itsuugo has quit IRC | 09:25 | |
| *** itsuugo has joined #openstack-operators | 09:26 | |
| *** pilgrimstack has quit IRC | 09:28 | |
| *** itsuugo has quit IRC | 09:31 | |
| *** itsuugo has joined #openstack-operators | 09:32 | |
| *** itsuugo has quit IRC | 09:40 | |
| *** itsuugo has joined #openstack-operators | 09:42 | |
| *** itsuugo has quit IRC | 09:47 | |
| *** itsuugo has joined #openstack-operators | 09:48 | |
| *** bdeschenes has joined #openstack-operators | 09:50 | |
| *** markvoelker has joined #openstack-operators | 09:52 | |
| *** itsuugo has quit IRC | 09:55 | |
| *** markvoelker has quit IRC | 09:56 | |
| *** itsuugo has joined #openstack-operators | 09:57 | |
| *** matrohon has quit IRC | 10:05 | |
| *** itsuugo has quit IRC | 10:09 | |
| *** itsuugo has joined #openstack-operators | 10:11 | |
| *** matrohon has joined #openstack-operators | 10:17 | |
| *** itsuugo has quit IRC | 10:19 | |
| *** itsuugo has joined #openstack-operators | 10:21 | |
| *** bdeschenes has quit IRC | 10:24 | |
| *** itsuugo has quit IRC | 10:31 | |
| *** itsuugo has joined #openstack-operators | 10:33 | |
| *** admin0 has quit IRC | 10:38 | |
| *** itsuugo has quit IRC | 10:43 | |
| *** itsuugo has joined #openstack-operators | 10:43 | |
| *** itsuugo has quit IRC | 10:48 | |
| *** itsuugo has joined #openstack-operators | 10:49 | |
| *** markvoelker has joined #openstack-operators | 10:53 | |
| *** itsuugo has quit IRC | 10:54 | |
| *** itsuugo has joined #openstack-operators | 10:55 | |
| *** markvoelker has quit IRC | 10:57 | |
| *** itsuugo has quit IRC | 11:00 | |
| *** itsuugo has joined #openstack-operators | 11:00 | |
| *** ducttape_ has joined #openstack-operators | 11:08 | |
| *** ducttape_ has quit IRC | 11:12 | |
| *** pilgrimstack has joined #openstack-operators | 11:12 | |
| *** Guest56480 has quit IRC | 11:27 | |
| *** sudswas__ has quit IRC | 11:27 | |
| *** bdeschenes has joined #openstack-operators | 11:29 | |
| *** itsuugo has quit IRC | 11:36 | |
| *** itsuugo has joined #openstack-operators | 11:38 | |
| *** simon-AS559 has joined #openstack-operators | 11:39 | |
| *** itsuugo has quit IRC | 11:43 | |
| *** itsuugo has joined #openstack-operators | 11:44 | |
| *** markvoelker has joined #openstack-operators | 11:50 | |
| *** itsuugo has quit IRC | 11:54 | |
| *** zigo has quit IRC | 11:54 | |
| *** itsuugo has joined #openstack-operators | 11:56 | |
| *** zigo has joined #openstack-operators | 11:58 | |
| *** zigo is now known as Guest83601 | 11:59 | |
| *** itsuugo has quit IRC | 12:02 | |
| *** Guest83601 has quit IRC | 12:03 | |
| *** itsuugo has joined #openstack-operators | 12:04 | |
| *** ducttape_ has joined #openstack-operators | 12:08 | |
| *** ducttape_ has quit IRC | 12:09 | |
| *** ducttape_ has joined #openstack-operators | 12:09 | |
| *** zigo_ has joined #openstack-operators | 12:12 | |
| *** itsuugo has quit IRC | 12:16 | |
| *** rarcea has quit IRC | 12:16 | |
| *** zigo_ has quit IRC | 12:17 | |
| *** maticue has joined #openstack-operators | 12:17 | |
| *** itsuugo has joined #openstack-operators | 12:17 | |
| *** zigo_ has joined #openstack-operators | 12:18 | |
| *** kstev has joined #openstack-operators | 12:20 | |
| *** itsuugo has quit IRC | 12:22 | |
| *** itsuugo has joined #openstack-operators | 12:24 | |
| *** armax has joined #openstack-operators | 12:30 | |
| *** rarcea has joined #openstack-operators | 12:31 | |
| *** itsuugo has quit IRC | 12:35 | |
| *** itsuugo has joined #openstack-operators | 12:36 | |
| *** Guest56480 has joined #openstack-operators | 12:43 | |
| *** sudswas__ has joined #openstack-operators | 12:43 | |
| *** itsuugo has quit IRC | 12:45 | |
| *** ducttape_ has quit IRC | 12:46 | |
| *** itsuugo has joined #openstack-operators | 12:47 | |
| *** zigo_ has quit IRC | 12:48 | |
| *** zigo_ has joined #openstack-operators | 12:51 | |
| *** armax has quit IRC | 12:51 | |
| *** armax has joined #openstack-operators | 12:56 | |
| *** david-lyle has joined #openstack-operators | 12:57 | |
| *** itsuugo has quit IRC | 13:06 | |
| *** itsuugo has joined #openstack-operators | 13:08 | |
| *** armax has quit IRC | 13:21 | |
| *** bryan_att has joined #openstack-operators | 13:24 | |
| *** itsuugo has quit IRC | 13:27 | |
| *** itsuugo has joined #openstack-operators | 13:29 | |
| *** VW has joined #openstack-operators | 13:33 | |
| *** roadst4r has quit IRC | 13:33 | |
| *** itsuugo has quit IRC | 13:34 | |
| *** itsuugo has joined #openstack-operators | 13:36 | |
| *** ducttape_ has joined #openstack-operators | 13:46 | |
| *** uxdanielle has joined #openstack-operators | 13:50 | |
| *** jamesdenton has joined #openstack-operators | 13:50 | |
| *** ducttape_ has quit IRC | 13:51 | |
| *** ducttape_ has joined #openstack-operators | 13:52 | |
| *** pilgrimstack has quit IRC | 13:57 | |
| *** rarcea has quit IRC | 13:57 | |
| *** makowals_ has joined #openstack-operators | 13:59 | |
| *** ducttape_ has quit IRC | 14:00 | |
| *** matrohon has quit IRC | 14:00 | |
| *** makowals has quit IRC | 14:01 | |
| *** r-daneel has joined #openstack-operators | 14:01 | |
| *** paramite has joined #openstack-operators | 14:04 | |
| *** vinsh has quit IRC | 14:20 | |
| *** saneax is now known as saneax-_-|AFK | 14:20 | |
| *** snecklifter has joined #openstack-operators | 14:21 | |
| *** mriedem is now known as mriedem_away | 14:22 | |
| *** ducttape_ has joined #openstack-operators | 14:26 | |
| *** cdelatte has joined #openstack-operators | 14:49 | |
| *** catintheroof has joined #openstack-operators | 14:50 | |
| *** mfisch has joined #openstack-operators | 14:51 | |
| *** mfisch has quit IRC | 14:51 | |
| *** mfisch has joined #openstack-operators | 14:51 | |
| *** marst has quit IRC | 14:54 | |
| catintheroof | Hi, i'm having a weird problem with openstack kilo (ubuntu 14.04, ovs 2.3.1) on a node i have a FIP namespace that is connected against 3 qrouters, from the FIP namespace, against a specific qrouter i have ping packet loss of all the vms attached against that qrouter (and of course from the outside) but i have no packet loss againt the vms running on the other two qrouters, did someone knows what might be happening or how ca i | 14:54 |
|---|---|---|
| catintheroof | troubleshoot something like this ? | 14:54 |
| *** vinsh has joined #openstack-operators | 14:57 | |
| *** rcernin has quit IRC | 15:07 | |
| *** arahal_ has joined #openstack-operators | 15:09 | |
| *** admin0 has joined #openstack-operators | 15:11 | |
| *** r-daneel has quit IRC | 15:13 | |
| *** makowals_ has quit IRC | 15:14 | |
| *** zigo_ is now known as zigo | 15:14 | |
| *** makowals has joined #openstack-operators | 15:14 | |
| *** makowals has joined #openstack-operators | 15:17 | |
| *** VW has quit IRC | 15:25 | |
| *** mriedem_away is now known as mriedem | 15:25 | |
| *** VW has joined #openstack-operators | 15:25 | |
| *** marst has joined #openstack-operators | 15:27 | |
| catintheroof | Hi, i'm having a weird problem with openstack kilo (ubuntu 14.04, ovs 2.3.1) on a node i have a FIP namespace that is connected against 3 qrouters, from the FIP namespace, against a specific qrouter i have ping packet loss of all the vms attached against that qrouter (and of course from the outside) but i have no packet loss againt the vms running on the other two qrouters, did someone knows what might be happening or how ca i | 15:40 |
| catintheroof | troubleshoot something like this ? | 15:40 |
| *** arahal_ has quit IRC | 15:46 | |
| *** piet has joined #openstack-operators | 15:49 | |
| *** simon-AS559 has quit IRC | 15:53 | |
| bsilverman | 100% packet loss or intermittent? | 15:54 |
| *** itsuugo has quit IRC | 15:55 | |
| *** itsuugo has joined #openstack-operators | 15:57 | |
| *** sorrison has quit IRC | 15:58 | |
| catintheroof | bsilverman, intermitent | 16:01 |
| *** arahal_ has joined #openstack-operators | 16:01 | |
| bsilverman | hmmm, I would check the routing tables to make sure there's no weird routes being injected/conflicting | 16:02 |
| catintheroof | bsilverman, where? on the qrouter namespace ? | 16:02 |
| bsilverman | On the VM | 16:03 |
| bsilverman | make sure there is nothing competing with the default routes | 16:03 |
| catintheroof | bsilverman, i have a route that goes to another tunneled net | 16:04 |
| catintheroof | bsilverman, should i delete it just to test ? | 16:04 |
| bsilverman | affirmative | 16:05 |
| *** mars_toktonaliev has joined #openstack-operators | 16:09 | |
| *** marst has quit IRC | 16:12 | |
| jlk | check MTU levels as well | 16:16 |
| *** itsuugo has quit IRC | 16:18 | |
| *** itsuugo has joined #openstack-operators | 16:20 | |
| *** rcernin has joined #openstack-operators | 16:20 | |
| *** danielitus has joined #openstack-operators | 16:25 | |
| catintheroof | bsilverman, jlk deleted the route on the vm, still all the same | 16:28 |
| bsilverman | Did you check the MTU | 16:28 |
| catintheroof | bsilverman, should i check the mtu inside the VM's ethernet ? | 16:29 |
| catintheroof | bsilverman, and then ? | 16:29 |
| catintheroof | bsilverman, MTU is 1450 | 16:30 |
| bsilverman | what is it on the router | 16:30 |
| bsilverman | or I should say, do the router and VM match | 16:31 |
| catintheroof | bsilverman, the route interface where the default GW of the vm is ? | 16:31 |
| catintheroof | bsilverman, it says MTU 1500 | 16:31 |
| bsilverman | What release are you running? Is there link congestion? Are you doing QOS? What is the load on the server, is there a resource issue outside of bandwidth? | 16:33 |
| catintheroof | bsilverman, ubuntu 14.04, openstack kilo packages for ubuntu, no link congestion, since only those two vms attached on that router fails, the resto of the 15 vms runnin in other 3 qrouters, works perfect on same host, not doing QOS, load is 0.19, no other resource issue | 16:34 |
| bsilverman | Are you running l3-agent in HA mode? | 16:35 |
| *** derekh has quit IRC | 16:40 | |
| *** zigo has quit IRC | 16:41 | |
| *** zul has quit IRC | 16:44 | |
| *** zul has joined #openstack-operators | 16:45 | |
| catintheroof | DVR | 16:49 |
| catintheroof | bsilverman, ^ | 16:49 |
| *** zigo has joined #openstack-operators | 16:51 | |
| *** zigo is now known as Guest18656 | 16:52 | |
| *** Guest18656 has quit IRC | 16:56 | |
| bsilverman | oh | 16:57 |
| bsilverman | well, that's a horse of a different color. | 16:57 |
| bsilverman | how many compute nodes/instances do you have running? | 16:58 |
| *** zigo_ has joined #openstack-operators | 16:59 | |
| *** leadfoot has quit IRC | 17:00 | |
| catintheroof | 5 | 17:06 |
| bsilverman | 5 compute nodes or 5 iinstances? | 17:07 |
| *** zigo_ has quit IRC | 17:07 | |
| *** Apoorva has joined #openstack-operators | 17:08 | |
| *** zigo_ has joined #openstack-operators | 17:11 | |
| catintheroof | bsilverman, 5 compute nodes | 17:20 |
| catintheroof | bsilverman, avg of 15 instances per compute | 17:20 |
| catintheroof | this only happens on 2 instances on a single compute node | 17:20 |
| catintheroof | on a single qrouter | 17:20 |
| bsilverman | exactly 2? | 17:24 |
| bsilverman | are they part of the same tenant? | 17:25 |
| bsilverman | on a single compute node | 17:25 |
| bsilverman | hmmmm | 17:25 |
| bsilverman | Are you seeing any problems from any other instances, on the FIP namespaces pinging/netcat | 17:29 |
| bsilverman | Can you externally run 100 ICMP packets to those instances on that compute node with zero packet loss? | 17:30 |
| catintheroof | bsilverman, i can't ping from the qrouter or externally without loosing packets | 17:31 |
| bsilverman | on all instances or just 2 | 17:31 |
| catintheroof | bsilverman, just these two | 17:32 |
| catintheroof | bsilverman, on that compute node, on that qrouter | 17:32 |
| bsilverman | and you are using VXLAN right, not VLAN | 17:33 |
| catintheroof | bsilverman, exactly | 17:33 |
| bsilverman | have you tried pinging another instance from the affected instance? | 17:35 |
| catintheroof | bsilverman, on another flotaing ? or on the same tunneled net ? | 17:36 |
| bsilverman | It all goes from the tap to the qbr to the br-int to the qr but I'd be interested to see what happens. | 17:36 |
| bsilverman | Is this only happening on the FIP? | 17:36 |
| catintheroof | bsilverman, nope, it happens at the qrouter level, so i can try to ping to another tunneled ip | 17:37 |
| bsilverman | yes, no need to bring in snat etc. | 17:37 |
| bsilverman | we are trying to boil it down to the most simple use case. | 17:37 |
| bsilverman | If you ping across tenant networks, or within the same tenant network it will use the linux bridge, integration bridge, and qrouter | 17:39 |
| catintheroof | bsilverman, oka, gimme a seccond | 17:39 |
| catintheroof | bsilverman, pinging from within one of those two vm's to tunneled ips on the same computen node and cross compute nodes, gives me no package loss | 17:47 |
| catintheroof | bsilverman, here goes another one, from within the qrouter, if i tcpdump on the qbr interface, on the qvo, qvb and tap interface, i see icmp request on all 4 interfaces, but the echo reply comes from the tap, to the qvb then qvo but never reaches qbr | 17:50 |
| bsilverman | hmmm | 17:52 |
| catintheroof | bsilverman, :O | 17:54 |
| bsilverman | are your netmasks right on these 2 instances? | 17:54 |
| bsilverman | there seems to be something wrong with routing here. | 17:55 |
| catintheroof | bsilverman, yeahp, all of them are /24 | 17:55 |
| *** itsuugo has quit IRC | 17:57 | |
| *** itsuugo has joined #openstack-operators | 17:58 | |
| *** Guest56480 has quit IRC | 17:59 | |
| *** sudswas__ has quit IRC | 17:59 | |
| bsilverman | can you ip netns exec ip a on the qrouter | 17:59 |
| bsilverman | and show me the output | 18:00 |
| catintheroof | bsilverman, sure | 18:01 |
| catintheroof | bsilverman, http://pastebin.com/Xyeu0Dim | 18:03 |
| bsilverman | have you done a tcpdump on the qvo and looked for the packets? | 18:05 |
| catintheroof | bsilverman, yes, i see everything ok, i just dont see some echo replies on the qbr | 18:08 |
| bsilverman | hmmm maybe check out the ARP tables on the qbr, see if there are double entries? | 18:09 |
| *** ig0r_ has quit IRC | 18:09 | |
| *** dmsimard is now known as dmsimard|afk | 18:10 | |
| bsilverman | you could take a look at the traffic flow via: ovs-ofctl dump-flows br-int | 18:10 |
| bsilverman | Without being on your system and digging around I | 18:13 |
| *** armax has joined #openstack-operators | 18:13 | |
| bsilverman | I'm running out of ideas | 18:13 |
| bsilverman | Look at: sudo ovs-appctl fdb/show br-int | 18:15 |
| bsilverman | You'll see the ports, VLANs MAC and Age in that table | 18:15 |
| bsilverman | see if that matches what you're seeing elsewhere. | 18:15 |
| catintheroof | bsilverman, let me look | 18:16 |
| catintheroof | bsilverman, i can see a double mac entry on the qbr | 18:17 |
| bsilverman | 2 MACs on one port? | 18:17 |
| catintheroof | bsilverman, yes they are | 18:18 |
| bsilverman | yeah, that is problematic | 18:18 |
| catintheroof | bsilverman, what does it means ? | 18:18 |
| catintheroof | that newtron gave same mac address to another vm elsewhere ? | 18:19 |
| bsilverman | Are you seeing the same MAC on 2 instances, or the same MAC on 2 qrouter ports? | 18:19 |
| *** piet has quit IRC | 18:20 | |
| *** admin0 has quit IRC | 18:20 | |
| bsilverman | Either one is bad | 18:21 |
| bsilverman | heh | 18:21 |
| *** piet has joined #openstack-operators | 18:22 | |
| bsilverman | In most use cases, you need distinct MAC addresses for each port and for each VM interface, otherwise you have L2 confusion which will lead to L3 issues. | 18:22 |
| catintheroof | bsilverman, in this case, the same mac address is set on the tap and qbr of the vm | 18:23 |
| *** mars_toktonaliev has quit IRC | 18:25 | |
| bsilverman | If you want to trace a specific traffic flow: Do this, ovs-appctl ofproto/trace br-int [in_port=#] dl_src=SRC_MAC | 18:26 |
| bsilverman | so you'll need to get the port number your instance is on and the MAC you want to trace | 18:26 |
| bsilverman | If you compare the broken instances to the good ones, do the TAPs and the qbr have the same MAC? | 18:27 |
| bsilverman | on both | 18:27 |
| bsilverman | good and bad | 18:27 |
| catintheroof | bsilverman, the bad one, has the same mac on the tap and the qbr, but the good ones, has the same mac on the qbr and qvb | 18:30 |
| *** nicodemus_ has joined #openstack-operators | 18:32 | |
| bsilverman | what is the mac address on the qvb of the bad one? What does it map to? | 18:35 |
| bsilverman | Just a heads up, I've got a hard stop for the day in about 10 minutes so as much as I am enjoying this, I will have to quit. | 18:36 |
| *** snecklifter has quit IRC | 18:36 | |
| catintheroof | bsilverman, it maps to the TAP interface of the vm, which is the mac that neutron says the vm has | 18:37 |
| bsilverman | that's on the good or the bad or both? | 18:38 |
| bsilverman | I guess what I'm trying to determine is if there are any MAC irregularities across the good ones versus the bad. | 18:38 |
| catintheroof | bsilverman, got it, gimme a sec | 18:40 |
| catintheroof | bsilverman, the mac that neutron says the bad one should have, is on the TAP and the QBR, the mac neutron says the good one should have, is only on the TA | 18:42 |
| bsilverman | yeah, that's probably your issue, the ARP table is causing confusion over where to send those ICMP packets. | 18:43 |
| bsilverman | Why this happened is another question entirely | 18:44 |
| catintheroof | bsilverman, that's what i was about to say | 18:44 |
| catintheroof | bsilverman, can i try to set the qbr the same mac of the qvb ? and let only the TAP of the bad vm has it ? | 18:46 |
| catintheroof | bsilverman, and then retry ? | 18:46 |
| *** dminer has joined #openstack-operators | 18:46 | |
| bsilverman | That is probably possible, I just don't have the process for that memorized. | 18:47 |
| bsilverman | Gotta run... best of luck. | 18:47 |
| catintheroof | bsilverman, thanks for the amazing troubleshooting | 18:47 |
| catintheroof | bsilverman, same to you | 18:48 |
| *** maishsk has joined #openstack-operators | 18:49 | |
| *** arahal_ has quit IRC | 18:58 | |
| *** mihalis68 has quit IRC | 19:02 | |
| *** itsuugo has quit IRC | 19:04 | |
| *** itsuugo has joined #openstack-operators | 19:04 | |
| *** rocky_g has joined #openstack-operators | 19:06 | |
| *** itsuugo has quit IRC | 19:18 | |
| *** itsuugo has joined #openstack-operators | 19:20 | |
| *** maishsk has quit IRC | 19:23 | |
| *** bdeschenes has quit IRC | 19:29 | |
| *** zigo_ is now known as zigo | 19:30 | |
| *** itsuugo has quit IRC | 19:35 | |
| *** itsuugo has joined #openstack-operators | 19:37 | |
| *** itsuugo has quit IRC | 19:43 | |
| *** itsuugo has joined #openstack-operators | 19:45 | |
| *** snecklifter has joined #openstack-operators | 19:50 | |
| *** snecklifter has quit IRC | 19:54 | |
| *** mars_toktonaliev has joined #openstack-operators | 19:58 | |
| *** mars_toktonaliev has quit IRC | 20:01 | |
| *** mars_toktonaliev has joined #openstack-operators | 20:01 | |
| *** mars_toktonaliev has quit IRC | 20:02 | |
| *** marst has joined #openstack-operators | 20:04 | |
| *** VW has quit IRC | 20:07 | |
| *** itsuugo has quit IRC | 20:12 | |
| *** itsuugo has joined #openstack-operators | 20:14 | |
| *** VW has joined #openstack-operators | 20:26 | |
| *** ducttape_ has quit IRC | 20:34 | |
| *** admin0 has joined #openstack-operators | 20:34 | |
| *** cdelatte has quit IRC | 20:37 | |
| *** paramite has quit IRC | 20:43 | |
| *** ducttape_ has joined #openstack-operators | 20:44 | |
| *** ducttape_ has quit IRC | 20:45 | |
| *** rocky_g has quit IRC | 20:46 | |
| *** itsuugo has quit IRC | 20:50 | |
| *** itsuugo has joined #openstack-operators | 20:52 | |
| *** dmsimard|afk is now known as dmsimard | 21:00 | |
| *** kstev has quit IRC | 21:01 | |
| *** piet has quit IRC | 21:02 | |
| *** piet has joined #openstack-operators | 21:02 | |
| *** piet has quit IRC | 21:06 | |
| *** markvoelker has quit IRC | 21:07 | |
| *** rocky_g has joined #openstack-operators | 21:07 | |
| *** itsuugo has quit IRC | 21:10 | |
| *** itsuugo has joined #openstack-operators | 21:11 | |
| *** itsuugo has quit IRC | 21:16 | |
| *** itsuugo has joined #openstack-operators | 21:17 | |
| *** jamesdenton has quit IRC | 21:21 | |
| *** kstev has joined #openstack-operators | 21:27 | |
| *** rcernin has quit IRC | 21:34 | |
| *** VW has quit IRC | 21:36 | |
| *** zul has quit IRC | 21:36 | |
| *** liverpooler has quit IRC | 21:37 | |
| *** bradjones has quit IRC | 21:37 | |
| *** Trident has quit IRC | 21:37 | |
| *** amitkqed has quit IRC | 21:37 | |
| *** admin0 has quit IRC | 21:37 | |
| *** dminer has quit IRC | 21:37 | |
| *** rmcall has quit IRC | 21:37 | |
| *** Dyon has quit IRC | 21:37 | |
| *** baffle has quit IRC | 21:37 | |
| *** DuncanT has quit IRC | 21:37 | |
| *** thingee has quit IRC | 21:37 | |
| *** AlexeyAbashkin has quit IRC | 21:37 | |
| *** mriedem has quit IRC | 21:37 | |
| *** dbecker has quit IRC | 21:37 | |
| *** blair has quit IRC | 21:37 | |
| *** mjrichardson has quit IRC | 21:37 | |
| *** tdasilva has quit IRC | 21:37 | |
| *** sshank has quit IRC | 21:37 | |
| *** bradm has quit IRC | 21:37 | |
| *** julian1 has quit IRC | 21:37 | |
| *** B_Smith has quit IRC | 21:37 | |
| *** karlamrhein has quit IRC | 21:37 | |
| *** alanmeadows has quit IRC | 21:37 | |
| *** cburgess has quit IRC | 21:37 | |
| *** dmanchad has quit IRC | 21:37 | |
| *** kragniz has quit IRC | 21:37 | |
| *** purp has quit IRC | 21:37 | |
| *** aspiers has quit IRC | 21:37 | |
| *** beddari has quit IRC | 21:37 | |
| *** melwitt has quit IRC | 21:37 | |
| *** nikhil has quit IRC | 21:37 | |
| *** yankcrime has quit IRC | 21:37 | |
| *** rocky_g has quit IRC | 21:37 | |
| *** mfisch has quit IRC | 21:37 | |
| *** arif-ali has quit IRC | 21:37 | |
| *** bsilverman has quit IRC | 21:37 | |
| *** mestery has quit IRC | 21:37 | |
| *** stevemar has quit IRC | 21:37 | |
| *** doffm has quit IRC | 21:37 | |
| *** mdorman_ has quit IRC | 21:37 | |
| *** uxdanielle has quit IRC | 21:37 | |
| *** ircuser-1 has quit IRC | 21:37 | |
| *** albertom has quit IRC | 21:37 | |
| *** jroll has quit IRC | 21:37 | |
| *** bloatyfloat has quit IRC | 21:37 | |
| *** amit213 has quit IRC | 21:37 | |
| *** Brainspackle has quit IRC | 21:37 | |
| *** hughsaunders has quit IRC | 21:37 | |
| *** zeus has quit IRC | 21:37 | |
| *** j^2 has quit IRC | 21:37 | |
| *** pasquier-s has quit IRC | 21:37 | |
| *** fyxim has quit IRC | 21:37 | |
| *** khappone has quit IRC | 21:37 | |
| *** HenryG has quit IRC | 21:37 | |
| *** peterjenkins has quit IRC | 21:37 | |
| *** mgagne has quit IRC | 21:37 | |
| *** sfrode has quit IRC | 21:38 | |
| *** AlexeyAbashkin has joined #openstack-operators | 21:38 | |
| *** mriedem has joined #openstack-operators | 21:38 | |
| *** sshank has joined #openstack-operators | 21:38 | |
| *** dbecker has joined #openstack-operators | 21:38 | |
| *** blair has joined #openstack-operators | 21:38 | |
| *** mjrichardson has joined #openstack-operators | 21:38 | |
| *** tdasilva has joined #openstack-operators | 21:38 | |
| *** bradm has joined #openstack-operators | 21:38 | |
| *** julian1 has joined #openstack-operators | 21:38 | |
| *** B_Smith has joined #openstack-operators | 21:38 | |
| *** karlamrhein has joined #openstack-operators | 21:38 | |
| *** alanmeadows has joined #openstack-operators | 21:38 | |
| *** kragniz has joined #openstack-operators | 21:38 | |
| *** cburgess has joined #openstack-operators | 21:38 | |
| *** dmanchad has joined #openstack-operators | 21:38 | |
| *** purp has joined #openstack-operators | 21:38 | |
| *** aspiers has joined #openstack-operators | 21:38 | |
| *** beddari has joined #openstack-operators | 21:38 | |
| *** melwitt has joined #openstack-operators | 21:38 | |
| *** nikhil has joined #openstack-operators | 21:38 | |
| *** yankcrime has joined #openstack-operators | 21:38 | |
| *** kelv has quit IRC | 21:39 | |
| *** sfrode has joined #openstack-operators | 21:40 | |
| *** uxdanielle has joined #openstack-operators | 21:40 | |
| *** ircuser-1 has joined #openstack-operators | 21:40 | |
| *** albertom has joined #openstack-operators | 21:40 | |
| *** jroll has joined #openstack-operators | 21:40 | |
| *** bloatyfloat has joined #openstack-operators | 21:40 | |
| *** amit213 has joined #openstack-operators | 21:40 | |
| *** Brainspackle has joined #openstack-operators | 21:40 | |
| *** hughsaunders has joined #openstack-operators | 21:40 | |
| *** zeus has joined #openstack-operators | 21:40 | |
| *** j^2 has joined #openstack-operators | 21:40 | |
| *** pasquier-s has joined #openstack-operators | 21:40 | |
| *** fyxim has joined #openstack-operators | 21:40 | |
| *** khappone has joined #openstack-operators | 21:40 | |
| *** HenryG has joined #openstack-operators | 21:40 | |
| *** peterjenkins has joined #openstack-operators | 21:40 | |
| *** mgagne has joined #openstack-operators | 21:40 | |
| *** simonmcc has quit IRC | 21:40 | |
| *** karlamrhein has quit IRC | 21:40 | |
| *** fyxim has quit IRC | 21:40 | |
| *** zul has joined #openstack-operators | 21:41 | |
| *** liverpooler has joined #openstack-operators | 21:41 | |
| *** bradjones has joined #openstack-operators | 21:41 | |
| *** Trident has joined #openstack-operators | 21:41 | |
| *** amitkqed has joined #openstack-operators | 21:41 | |
| *** serverascode has quit IRC | 21:41 | |
| *** nikhil has quit IRC | 21:41 | |
| *** admin0 has joined #openstack-operators | 21:41 | |
| *** dminer has joined #openstack-operators | 21:41 | |
| *** rmcall has joined #openstack-operators | 21:41 | |
| *** Dyon has joined #openstack-operators | 21:41 | |
| *** baffle has joined #openstack-operators | 21:41 | |
| *** thingee has joined #openstack-operators | 21:41 | |
| *** rocky_g has joined #openstack-operators | 21:42 | |
| *** mfisch has joined #openstack-operators | 21:42 | |
| *** arif-ali has joined #openstack-operators | 21:42 | |
| *** bsilverman has joined #openstack-operators | 21:42 | |
| *** mestery has joined #openstack-operators | 21:42 | |
| *** stevemar has joined #openstack-operators | 21:42 | |
| *** doffm has joined #openstack-operators | 21:42 | |
| *** mdorman_ has joined #openstack-operators | 21:42 | |
| *** amit213 has quit IRC | 21:44 | |
| *** pasquier-s has quit IRC | 21:45 | |
| *** VW has joined #openstack-operators | 21:48 | |
| *** simonmcc has joined #openstack-operators | 21:50 | |
| *** bdeschenes has joined #openstack-operators | 21:53 | |
| *** kelv has joined #openstack-operators | 21:56 | |
| *** rocky_g has quit IRC | 21:59 | |
| *** karlamrhein has joined #openstack-operators | 22:05 | |
| *** DuncanT has joined #openstack-operators | 22:08 | |
| *** pasquier-s has joined #openstack-operators | 22:09 | |
| *** admin0 has quit IRC | 22:11 | |
| *** fyxim has joined #openstack-operators | 22:13 | |
| *** bdeschenes has quit IRC | 22:14 | |
| *** serverascode has joined #openstack-operators | 22:14 | |
| *** amit213 has joined #openstack-operators | 22:14 | |
| *** sorrison has joined #openstack-operators | 22:15 | |
| *** nikhil has joined #openstack-operators | 22:16 | |
| *** marst has quit IRC | 22:18 | |
| *** kstev has quit IRC | 22:19 | |
| *** nicodemus_ has quit IRC | 22:25 | |
| *** marst has joined #openstack-operators | 22:32 | |
| *** vinsh has quit IRC | 22:41 | |
| *** VW has quit IRC | 22:49 | |
| *** marst has quit IRC | 22:54 | |
| *** maticue has quit IRC | 23:07 | |
| *** itsuugo has quit IRC | 23:08 | |
| *** itsuugo has joined #openstack-operators | 23:10 | |
| *** itsuugo has quit IRC | 23:15 | |
| *** itsuugo has joined #openstack-operators | 23:16 | |
| *** HenryG has quit IRC | 23:28 | |
| *** HenryG has joined #openstack-operators | 23:28 | |
| *** armax has quit IRC | 23:33 | |
| *** marst has joined #openstack-operators | 23:35 | |
| *** dminer has quit IRC | 23:49 | |
| *** itsuugo has quit IRC | 23:50 | |
| *** itsuugo has joined #openstack-operators | 23:51 | |
| *** itsuugo has quit IRC | 23:56 | |
| *** itsuugo has joined #openstack-operators | 23:57 | |
| *** marst has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!