| srihas | johnsom: just a doubt 00:25 < johnsom> Hmm, ok can you paste your "netstat -rn" -> inside namespace? | 00:00 |
|---|---|---|
| srihas | johnsom: not a devstack. backed by Cisco ACI with opflex | 00:00 |
| johnsom | Ok | 00:01 |
| srihas | newton on ubuntu | 00:01 |
| johnsom | srihas The route show table command shows if there are policy based routes enabled. These don't show up in the netstat -rn. | 00:02 |
| srihas | http://paste.openstack.org/show/654192/ | 00:04 |
| srihas | johnsom: ok | 00:04 |
| johnsom | Ok, no PBR games.... | 00:08 |
| johnsom | And from inside the netns you can't directly curl the backend web servers? | 00:10 |
| srihas | johnsom: no, I cant | 00:13 |
| johnsom | Yeah, I think this is a lower level networking issue in the stack. It seems like that tap interface in the netns isn't working | 00:13 |
| srihas | johnsom: ok, I cant visualise a logical architecture of this LB setup and the way its talking to VMs. Is there any guide to understand it? | 00:15 |
| johnsom | Well, at this point it isn't the LB at all. | 00:15 |
| srihas | aha | 00:15 |
| srihas | things in ACI? | 00:15 |
| johnsom | The network namespace is created on the neutron controller host. This is like an isolate environment for networking, as if it was on another host. | 00:16 |
| srihas | yeah | 00:16 |
| johnsom | In this case there is one interface, your tapb12054f1-10 interface. This is line the nic (network card) for this other host. | 00:16 |
| johnsom | from inside that network namespace you should be able to ping it's gateway 192.168.180.1 or the web servers since they are on the same subnet. | 00:17 |
| johnsom | This isn't working, which means this nic (tap) is not working, or it's not plugged into something correctly. | 00:17 |
| srihas | I can ping GW but not the VM strangely | 00:18 |
| johnsom | So, yes, it could be an issue with how ACI and neutron are configured or working | 00:18 |
| johnsom | srihas, ok, so that is at least one good step. So maybe that gateway doesn't have a working router? | 00:19 |
| johnsom | Or bridge I guess in this case | 00:19 |
| srihas | It has a working router I assume, because I can ping one VM from another over 192 network | 00:19 |
| johnsom | I don't know how ACI and neutron work together. I have not used ACI | 00:19 |
| srihas | np | 00:20 |
| johnsom | srihas ok, so that is good. So it's likely just something wrong with this tap. | 00:20 |
| johnsom | hang on a second | 00:20 |
| srihas | ok | 00:20 |
| johnsom | I'm going to look at the code for this driver and see if I can see how it creates that tap | 00:20 |
| srihas | ok. ty | 00:21 |
| johnsom | So it uses the neutron interface driver: https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/drivers/haproxy/namespace_driver.py#L88 | 00:25 |
| johnsom | And creates it here: https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/drivers/haproxy/namespace_driver.py#L318 | 00:25 |
| Bar_ | I suspect the scenario I have suggested (occupied sec-group) won't revert nicely, for several reasons. It seems as the get_delete_load_balancer_flow was not written to handle failures. | 00:26 |
| johnsom | Bar_ No, that is not correct. | 00:26 |
| Bar_ | johnsom, would you care to explain? | 00:27 |
| johnsom | srihas So I'm guessing there is some issue between the configured neutron interface driver and the ACI environment. Sadly this is getting outside my knowledge area. You might try asking in the #openstack-neutron channel if someone can help debug that tap interface and the network topology with ACI. | 00:28 |
| johnsom | Bar_ get_delete_load_balancer_flow is written to handle failures. Very explicitly actually. | 00:28 |
| johnsom | srihas It's something with that interface driver, I just have no idea what | 00:29 |
| Bar_ | I must be missing something, I had expected the tasks to implement revert method, but there is none. | 00:29 |
| Bar_ | e.g. network_tasks.UnplugVIP | 00:30 |
| srihas | johnsom: I have a doubt, where are drivers located on my host? | 00:30 |
| srihas | johnsom: nvm, found it | 00:31 |
| johnsom | srihas It's configured in the /etc/neutron/l3_agent.ini | 00:32 |
| srihas | /usr/lib/python2.7/dist-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py | 00:33 |
| srihas | I meant this johnsom | 00:33 |
| johnsom | srihas Oh, that is the lbaas driver yes, but not the problem interface driver | 00:33 |
| srihas | interface_driver = openvswitch | 00:33 |
| srihas | ovs_integration_bridge = br-int | 00:33 |
| johnsom | Hmm, ok, so it is setting up a local openvswitch. I wonder if that is plugged into your ACI | 00:34 |
| *** rstarmer has quit IRC | 00:35 | |
| johnsom | Bar_ That is correct, if UnplugVIP fails, there is no remediation action required for that step in the flow. Though all of the other steps in that flow fire their reverts. | 00:36 |
| johnsom | https://docs.openstack.org/octavia/latest/_images/LoadBalancerFlows-get_delete_load_balancer_flow.svg | 00:36 |
| johnsom | In this case we are saying "If the unplug VIP task fails, we are in a place the code does not have tools to resolve beyond what was tried in the code." so we revert up the chain, which for that one, basically means we mark the LB in ERROR and stop. | 00:37 |
| johnsom | Bar_ VIPs are important to handle carefully as the user may have apps expecting that IP address or quota, such as a heat template, or a well known IP to their application. We don't want to do more harm than good with the VIP. | 00:38 |
| Bar_ | johnsom, so if VIP deletion fails, then LB will result in ERROR, correct? | 00:39 |
| johnsom | We could consider adding retry steps to the flow if we think these are transient failures. | 00:39 |
| johnsom | Bar_ Corrent, the LB will go into provisioning_status ERROR | 00:40 |
| johnsom | Corrent->Correct | 00:40 |
| Bar_ | yeah... so it won't revert cleanly to a useful state. that's what I was afraid of. | 00:41 |
| johnsom | Bar_ What do you mean about a "useful state" it completely reverts to the correct state of "ERROR" | 00:41 |
| Bar_ | johnsom, if I had only left the sec-group orphan, would it be as bad? | 00:41 |
| Bar_ | in other words, I'll ignore error in deletion of the security group, but I will fail to task for failure to delete the port. | 00:42 |
| *** yamamoto has joined #openstack-lbaas | 00:43 | |
| Bar_ | useful state = I can still use the LB. just as if I never had issued a deletion flow. | 00:43 |
| johnsom | Well, yes, that is exactly how it behaves. Since there is no listener on the LB (you can call delete) it is not accepting traffic on any ports anyway, so yes, it behaves the same as it did before the delete call. | 00:45 |
| johnsom | It is also indicating that there is an error that is blocking the successful delete of the LB and that resources are still in use. | 00:46 |
| *** salmankhan has quit IRC | 00:49 | |
| johnsom | It also leaves it in a state that allows the user to try again. | 00:50 |
| johnsom | Bar_ Does that help to explain that flow? | 00:53 |
| Bar_ | johnsom, if there is no problem to re-delete a loadbalancer in ERROR state, then I think I'm good. | 00:55 |
| johnsom | Bar_ Yes, the user "escape" option for any resource in "ERROR" is to call delete | 00:55 |
| johnsom | It allows them to delete and recreate any entity on the LB that fails in a way we can't automatically recover it, i.e. "ERROR". | 00:57 |
| Bar_ | johnsom, ok, thanks. Just to make sure, if I fail to delete the security group, it is a good-enough reason in your opinion, to fail to whole deletion flow, correct? | 00:58 |
| johnsom | That is our current model, because if we left it there is still an SG in use for that project. It still counts against a quota, etc. | 00:59 |
| xgerman_ | +1 | 01:01 |
| johnsom | It's getting late and I'm struggling to remember all of those scenarios. I.e. if the SG is still on the port will neutron fail to delete the port, etc. There are also subtleties in who owns which part. Is is an Octavia service account resource or a user resource. | 01:01 |
| Bar_ | johnsom, just one more scenario: if I fail to delete the SG, should I give up on trying to delete the VIP? | 01:03 |
| johnsom | Bar_ Yeah, that is the part I'm not sure right now. I would have to re-evaluate that. | 01:04 |
| johnsom | I mean, if we are going to revert the flow anyway, due to resource in use, we might as well stop there and not try to continue. | 01:05 |
| johnsom | It makes it better for the operator if they are trying to manually resolve the underlying failure. | 01:05 |
| Bar_ | ok then | 01:06 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 01:08 | |
| Bar_ | johnsom, thanks for the exlpanations. Unless there's something you wish to keep/merge from this patch, you can force-abandon it. (since i'm unable to do that) | 01:08 |
| johnsom | You can't abandon it? It's your patch??? You should be able to . | 01:09 |
| Bar_ | I have written it from another user, I'm afraid. | 01:09 |
| johnsom | Ah, multiple personalities... Grin | 01:10 |
| johnsom | Ha, ok, sure NP | 01:10 |
| *** slaweq has joined #openstack-lbaas | 01:10 | |
| Bar_ | thx | 01:11 |
| johnsom | We might want to consider adding more retry steps for transient errors. We do not do that in many, if any, of the flows. | 01:11 |
| *** AlexeyAbashkin has quit IRC | 01:12 | |
| Bar_ | we could, plus there's taskflow support for this. | 01:13 |
| *** slaweq has quit IRC | 01:15 | |
| rm_work | yeah, taskflow-level retries would be neat | 01:17 |
| rm_work | and yeah, the VIP is usually the LAST thing to go regardless of whether it's technically required, because once we lose the VIP there's no getting it back, so it's the only truly unrecoverable thing that can happen | 01:17 |
| *** yamamoto has quit IRC | 01:18 | |
| johnsom | Bar_ Yes, that is why I said in the flows... | 01:21 |
| *** Bar_ has quit IRC | 01:33 | |
| *** rstarmer has joined #openstack-lbaas | 02:01 | |
| *** rstarmer has quit IRC | 02:05 | |
| *** bbzhao has quit IRC | 02:12 | |
| *** bbzhao has joined #openstack-lbaas | 02:12 | |
| *** rstarmer has joined #openstack-lbaas | 02:15 | |
| *** yamamoto has joined #openstack-lbaas | 02:19 | |
| *** HW-Peter has quit IRC | 02:22 | |
| *** yamamoto has quit IRC | 02:28 | |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 02:35 | |
| *** rstarmer has quit IRC | 02:36 | |
| *** rstarmer has joined #openstack-lbaas | 02:46 | |
| *** rstarmer has quit IRC | 02:55 | |
| *** harlowja_ has quit IRC | 03:08 | |
| *** harlowja has joined #openstack-lbaas | 03:55 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: Add SDK list to octavia user docs https://review.openstack.org/538323 | 04:04 |
| *** fnaval has joined #openstack-lbaas | 04:04 | |
| *** armax has quit IRC | 04:25 | |
| *** armax has joined #openstack-lbaas | 04:26 | |
| *** armax has quit IRC | 04:26 | |
| *** armax has joined #openstack-lbaas | 04:27 | |
| *** armax has quit IRC | 04:27 | |
| *** armax has joined #openstack-lbaas | 04:27 | |
| *** armax has quit IRC | 04:28 | |
| *** armax has joined #openstack-lbaas | 04:28 | |
| *** armax has quit IRC | 04:28 | |
| *** armax has joined #openstack-lbaas | 04:29 | |
| *** armax has quit IRC | 04:29 | |
| *** fnaval has quit IRC | 05:22 | |
| *** gcheresh has joined #openstack-lbaas | 05:31 | |
| *** kevinbenton has quit IRC | 05:37 | |
| *** kevinbenton has joined #openstack-lbaas | 05:40 | |
| *** gcheresh has quit IRC | 06:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata https://review.openstack.org/537765 | 06:28 |
| *** harlowja has quit IRC | 06:42 | |
| *** fnaval has joined #openstack-lbaas | 06:53 | |
| *** fnaval has quit IRC | 06:58 | |
| *** slaweq has joined #openstack-lbaas | 07:12 | |
| *** slaweq has quit IRC | 07:17 | |
| *** fnaval has joined #openstack-lbaas | 07:51 | |
| *** fnaval has quit IRC | 07:55 | |
| *** aojea has joined #openstack-lbaas | 08:12 | |
| *** slaweq has joined #openstack-lbaas | 08:17 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 08:37 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Handle missing security groups on listener delete https://review.openstack.org/538408 | 08:51 |
| *** AlexeyAbashkin has quit IRC | 09:22 | |
| *** aojea has quit IRC | 09:28 | |
| *** aojea_ has joined #openstack-lbaas | 09:51 | |
| *** slaweq has quit IRC | 10:00 | |
| *** slaweq has joined #openstack-lbaas | 10:02 | |
| *** slaweq has joined #openstack-lbaas | 10:02 | |
| *** aojea_ has quit IRC | 10:03 | |
| *** aojea_ has joined #openstack-lbaas | 10:03 | |
| *** slaweq has quit IRC | 10:40 | |
| *** slaweq has joined #openstack-lbaas | 10:41 | |
| *** slaweq_ has joined #openstack-lbaas | 10:46 | |
| *** slaweq has quit IRC | 10:46 | |
| *** fnaval has joined #openstack-lbaas | 10:51 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Change "Method" to "Algorithm" in pool step https://review.openstack.org/538422 | 10:53 |
| *** fnaval has quit IRC | 10:55 | |
| *** eanderson_ has joined #openstack-lbaas | 11:09 | |
| *** eandersson has quit IRC | 11:11 | |
| *** aojea_ has quit IRC | 11:38 | |
| *** yamamoto has joined #openstack-lbaas | 11:39 | |
| *** aojea_ has joined #openstack-lbaas | 12:00 | |
| *** aojea_ has quit IRC | 12:01 | |
| *** aojea_ has joined #openstack-lbaas | 12:02 | |
| *** aojea_ has quit IRC | 12:04 | |
| *** salmankhan has joined #openstack-lbaas | 12:32 | |
| *** aojea has joined #openstack-lbaas | 12:46 | |
| *** aojea has quit IRC | 13:00 | |
| *** yamamoto has quit IRC | 13:29 | |
| *** Alex_Staf has joined #openstack-lbaas | 13:40 | |
| *** fnaval has joined #openstack-lbaas | 13:50 | |
| *** Alex_Staf has quit IRC | 13:51 | |
| *** fnaval has quit IRC | 13:55 | |
| *** yamamoto has joined #openstack-lbaas | 13:57 | |
| *** ipsecguy_ has quit IRC | 13:58 | |
| *** ipsecguy has joined #openstack-lbaas | 13:58 | |
| *** salmankhan has quit IRC | 13:58 | |
| *** fnaval has joined #openstack-lbaas | 14:51 | |
| *** yamamoto has quit IRC | 14:53 | |
| *** yamamoto has joined #openstack-lbaas | 14:54 | |
| *** fnaval has quit IRC | 14:55 | |
| *** ivve has quit IRC | 15:00 | |
| *** yamamoto has quit IRC | 15:01 | |
| *** ivve has joined #openstack-lbaas | 15:15 | |
| *** armax has joined #openstack-lbaas | 16:01 | |
| *** yamamoto has joined #openstack-lbaas | 16:09 | |
| *** yamamoto has quit IRC | 16:14 | |
| *** harlowja has joined #openstack-lbaas | 16:34 | |
| *** yamamoto has joined #openstack-lbaas | 16:43 | |
| *** yamamoto has quit IRC | 16:43 | |
| *** fnaval has joined #openstack-lbaas | 16:54 | |
| *** fnaval has quit IRC | 16:58 | |
| *** armax has quit IRC | 16:58 | |
| *** yamamoto has joined #openstack-lbaas | 17:44 | |
| *** fnaval has joined #openstack-lbaas | 17:51 | |
| *** fnaval has quit IRC | 17:55 | |
| *** yamamoto has quit IRC | 17:58 | |
| *** eanderson_ has quit IRC | 18:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements https://review.openstack.org/538474 | 18:06 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:06 | |
| *** AlexeyAbashkin has quit IRC | 18:11 | |
| *** fnaval has joined #openstack-lbaas | 18:51 | |
| *** fnaval has quit IRC | 18:56 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:07 | |
| *** AlexeyAbashkin has quit IRC | 19:11 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Handle missing security groups on listener delete https://review.openstack.org/538408 | 19:29 |
| *** gcheresh has joined #openstack-lbaas | 19:42 | |
| *** fnaval has joined #openstack-lbaas | 19:51 | |
| *** fnaval has quit IRC | 19:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/537088 | 20:10 |
| *** bbzhao has quit IRC | 20:42 | |
| *** bbzhao has joined #openstack-lbaas | 20:43 | |
| *** slaweq_ has quit IRC | 21:02 | |
| *** slaweq has joined #openstack-lbaas | 21:11 | |
| *** slaweq has quit IRC | 21:15 | |
| *** gcheresh has quit IRC | 21:29 | |
| *** rstarmer has joined #openstack-lbaas | 21:33 | |
| *** fnaval has joined #openstack-lbaas | 21:51 | |
| *** fnaval has quit IRC | 21:55 | |
| *** rstarmer has quit IRC | 22:08 | |
| *** rstarmer has joined #openstack-lbaas | 22:38 | |
| *** rstarmer has quit IRC | 22:40 | |
| *** fnaval has joined #openstack-lbaas | 22:52 | |
| *** fnaval has quit IRC | 22:56 | |
| *** rstarmer has joined #openstack-lbaas | 23:03 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Re-enable "Page Contents" and update user docs https://review.openstack.org/538526 | 23:11 |
| *** fnaval has joined #openstack-lbaas | 23:51 | |
| *** rstarmer has quit IRC | 23:55 | |
| *** fnaval has quit IRC | 23:55 | |
| *** rstarmer has joined #openstack-lbaas | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
