Tuesday, 2018-10-30

*** hongbin has joined #openstack-lbaas01:23
*** bcafarel has quit IRC01:40
*** bcafarel has joined #openstack-lbaas02:00
*** ramishra has joined #openstack-lbaas02:50
*** ramishra has quit IRC03:29
*** ramishra has joined #openstack-lbaas03:45
*** hongbin has quit IRC04:23
*** sapd1 has quit IRC04:38
*** sapd1 has joined #openstack-lbaas04:59
*** pcaruana has joined #openstack-lbaas05:36
*** pcaruana has quit IRC05:47
*** ramishra has quit IRC06:16
*** mnaser has quit IRC06:16
*** mnaser has joined #openstack-lbaas06:17
*** ramishra has joined #openstack-lbaas06:18
*** beisner has quit IRC06:30
*** beisner has joined #openstack-lbaas06:31
*** ccamposr has joined #openstack-lbaas06:40
*** ramishra_ has joined #openstack-lbaas06:59
*** ramishra has quit IRC07:01
*** tomtom001 has quit IRC07:02
*** abaindur has quit IRC07:56
*** pcaruana has joined #openstack-lbaas07:57
*** pcaruana is now known as pcaruana|elisa|07:59
*** celebdor has joined #openstack-lbaas08:10
*** sapd1 has quit IRC08:10
*** sapd1 has joined #openstack-lbaas08:12
*** numans has joined #openstack-lbaas08:41
*** yboaron has joined #openstack-lbaas08:47
*** celebdor has quit IRC09:05
openstackgerritKim Bao Long proposed openstack/octavia master: Update README by adding Mailing List and Wiki URL  https://review.openstack.org/61415809:51
*** salmankhan has joined #openstack-lbaas10:01
*** celebdor has joined #openstack-lbaas10:01
openstackgerritCarlos Goncalves proposed openstack/octavia master: Redirect disk-image-builder logs, make verbose  https://review.openstack.org/61262210:39
*** ramishra_ has quit IRC11:15
*** ramishra has joined #openstack-lbaas11:18
*** salmankhan has quit IRC11:26
*** yamamoto has quit IRC11:51
*** yamamoto has joined #openstack-lbaas11:53
*** yamamoto has quit IRC11:58
*** salmankhan has joined #openstack-lbaas12:02
*** yamamoto has joined #openstack-lbaas12:33
*** celebdor has quit IRC13:04
*** celebdor has joined #openstack-lbaas13:06
*** velizarx has joined #openstack-lbaas13:07
*** yamamoto has quit IRC13:12
*** yboaron has quit IRC14:04
*** yamamoto has joined #openstack-lbaas14:21
*** yamamoto has quit IRC14:26
*** pcaruana|elisa| has quit IRC14:33
*** fnaval has joined #openstack-lbaas14:40
*** pcaruana|elisa| has joined #openstack-lbaas14:45
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add octavia-v2-dsvm-py3-scenario-fedora-latest job  https://review.openstack.org/60038114:45
*** dolly has joined #openstack-lbaas14:46
dollyrm_work, you there and have a few minutes?14:51
johnsomdolly He is on vacation for a while, so may not reply. Is there something I can help you with?14:54
dollyjohnsom, oh! I see, well I was curious about his work on https://review.openstack.org/#/c/558962/14:54
dollyI seem to hit a bug there when the health-manager recreates an amp14:55
dollyI talked about that with him, I was just curios if he was able to reproduce it14:56
johnsomdolly He is running that, but with a bunch of other patches too. His network is L3 and he doesn't use the standard active/standby, but moves ports. So, it's a bit of a different animal14:56
johnsomAh, ok. Yeah, I think he is the only one running that, so you might need an answer from him.14:57
dollyjohnsom, I see, no worries.14:57
dollyjohnsom, do you know that the "cached_zone" field is in the amphora table ?14:59
dollyCause, this is what I'm hitting http://paste.openstack.org/show/728729/14:59
*** yamamoto has joined #openstack-lbaas14:59
dollyI've actually described quite thoroughly, and I'm not sure why the "creation of the amp-step" chooses the wrong AZ, even though the "filterer" in the step before, got the correct one.15:01
dollyIt very weird15:01
*** ianychoi_ is now known as ianychoi15:07
*** velizarx has quit IRC15:08
*** yamamoto has quit IRC15:20
*** yamamoto has joined #openstack-lbaas15:23
*** yamamoto has quit IRC15:28
*** pcaruana|elisa| has quit IRC15:35
*** tomtom001 has joined #openstack-lbaas15:40
tomtom001Hello, I'm creating load balancers in Queens and noticed that the Octavia Management network has to be in a shared state to operate correctly.  Is there a reason why it can't be an non-shared network to operate correctly?15:41
johnsomtomtom001: It should not be a shared network15:43
johnsomEverything using that network should be under the octavia user/project15:43
tomtom001when I make the network non-shared octavia containers are unable to contact the lb instances on 9443 to set them up.  If I run telent <octavia mgmt ip> 9443 the containers are able to connect, but the octavia-worker cannot.15:44
tomtom001hmmm.... ok let me check that15:45
tomtom001ok, so if I log in as the octavia user, under project service, that's where the Octavia management network is specified.  It seems to be correct from what I'm seeing15:47
*** pcaruana|elisa| has joined #openstack-lbaas15:50
tomtom001johnsom: so octavia user's default project is service, is that ok?15:51
johnsomShould be ok.  Check your service_auth settings, they should match15:52
tomtom001johnsom: Octavia user is an admin under the service project members, is that what I should be looking for?15:54
*** yamamoto has joined #openstack-lbaas15:57
tomtom001under service_auth in octavia.conf service is the project_name as well.  That should be good15:57
*** aojea_ has joined #openstack-lbaas15:57
*** pcaruana|elisa| has quit IRC16:00
*** ccamposr has quit IRC16:04
dollycgoncalves, you there ?16:10
cgoncalvesdolly, I am16:10
dollyyo! long time no hear, I've got a question for you =)16:10
cgoncalvesdolly, hit me. I'll do my best to answer16:11
dolly- > /usr/share/rhosp-director-images/octavia-amphora-13.0-20180905.1.x86_64.raw <- Whats the status on this image in the current release of OSP 13. I use the github/octavia/queens/stable-release, and when creating an amp we hit this issue http://paste.openstack.org/show/733653/16:13
dollyBut, using the image you provided me a long time ago, works good!16:13
dollycgoncalves, did you run away ;)16:23
cgoncalvesdolly, I was afk for a moment, sorry16:25
dollyno worries =)16:25
cgoncalvesdolly, the OSP-provided amp image is based on latest RHEL cloud image at build time + octavia stable queens amphora bits16:26
cgoncalvesdolly, I am not sure what the issue is from that paste. would you have the amp logs?16:27
dollyok, do you remember the image you made me a while ago ?16:27
dollyI guess I can get the logs tomorrow, need to  make sure the amp is not deleted16:28
dollyso need to update the config16:28
cgoncalvesI did one? didn't I point you to the periodically built images instead?16:28
dollyand I'm on my way home16:28
dollyoh yes, sorry16:28
dollyyeah you did16:28
dollybut yes, that image works. the image provided from rhel/osp doesn't.16:28
dollyand I was just curios why16:29
cgoncalvesdolly, does it happen all the time when you build an amp?16:30
dollycgoncalves, yeps, everytime, same place. never happens with the other image16:30
dolly(but understand that we use the github/stable/queens-version of octavia, and not the one provided by rhel)16:31
cgoncalvesok, that is weird. I would really need to see the amp-agent log16:31
cgoncalvesdolly, control plane services run queens upstream while amp from OSP, is that it?16:32
cgoncalvesok, got it16:33
cgoncalveswell, downstream we do not support that obviously ;)16:33
dollyofc :)16:33
cgoncalvesbut I could have a look once you share some logs16:33
dollyI'm just curious in why it crashes16:33
dollywith "internal server erro"16:33
dollyAnd I thought since the amp-image was updated in OSP just a month ago, I figured it was "new enough" to work with the upstream version of octavia.16:34
dollyBut guess I was wrong ;)16:34
cgoncalvesdolly, not sure in which 13.0.z version you're on16:35
cgoncalvesbut 13z2 doesn't ship with latest queens release16:35
dollyNo, thats why we build our own =)16:36
cgoncalveschecked, 13z2 ships octavia 2.0.116:36
dollyWhere do you find info about which z-release you are running ?16:36
cgoncalvesdolly, yeah so the amp is also based on 2.0.116:36
dollywe are running the "latest" available16:36
dollygot it16:36
dollyIs there no way of determining which openstack 13.z version you are running? (lets say you dont know which one you have)16:39
cgoncalvesgood question, heh. looking16:45
dollyno worries =) we have just updated everything to the latest available, i'm just a bit confused about these .z releases16:45
cgoncalvesdolly, https://bugzilla.redhat.com/show_bug.cgi?id=153356516:50
openstackbugzilla.redhat.com bug 1533565 in rhosp-release "Please include information about minor version (z stream) in rhosp-release" [Unspecified,Closed: notabug] - Assigned to mburns16:50
cgoncalvestl;dr: no way16:50
*** aojea_ has quit IRC16:53
colin-if i provided octavia with a network value for this parameter https://docs.openstack.org/octavia/ocata/main/configref.html#controller_worker.amp_boot_network_list whose definition contained an IPv6 subnet would it support that?16:56
colin-yikes i reallly linked the ocata doc, my bad16:57
cgoncalvesdolly, ok, so, I'm hearing that it will be possible soon ;)16:57
johnsomcolin- Yes, someone has done this in the past. There are two comments I will make: 1. we don't yet test this in the gates, so it *might* be broken (it's on my to-do list). 2. If you want to use link-local addresses there is an open patch with some changes that need to be made.16:58
cgoncalvesdolly, https://bugzilla.redhat.com/show_bug.cgi?id=164029117:00
openstackbugzilla.redhat.com bug 1640291 in rhosp-release "Add a maintenance release version to /etc/rhosp-release" [Low,On_qa] - Assigned to jjoyce17:00
colin-oh nice, i'll give it a whirl17:01
colin-thanks johnsom17:01
johnsomcolin- I have recently been working on getting our IPv6 house back in order.  Patch reviews would be welcome. I have gates for the VIP and members now, the next step is for the lb-mgmt-net just to make sure we don't get bit-rot there.17:03
*** velizarx has joined #openstack-lbaas17:14
*** velizarx has quit IRC17:15
*** aojea_ has joined #openstack-lbaas17:19
*** yamamoto has quit IRC17:24
colin-yeah, link-local would be useful. i'll take a look at the open changes for the patch. will also take a stab at some reviews17:26
*** aojea_ has quit IRC17:31
*** aojea_ has joined #openstack-lbaas17:32
*** aojea_ has quit IRC17:39
*** aojea_ has joined #openstack-lbaas17:40
*** aojea_ has quit IRC17:44
*** salmankhan has quit IRC17:55
*** yamamoto has joined #openstack-lbaas18:03
colin-unrelated, is there a section of the config that would disable the creation of LBs outside of RFC1918 space?18:05
johnsomcolin- Like block the creation of LBs with VIPs outside 1918?18:07
johnsomRight now, the only restrictions we have on the user specified networks/subnets is if they have access to them, and the valid VIP networks whitelist and the reserved member addresses. https://docs.openstack.org/octavia/latest/configuration/configref.html#networking.valid_vip_networks18:10
*** yamamoto has quit IRC18:12
colin-yes that's what i had in mind, that makes sense thanks18:24
colin-johnsom: does this describe the link-local changes you mentioned above? https://review.openstack.org/#/c/391204/18:47
johnsomYes, that is the one. I haven’t looked at it in a while18:48
*** abaindur has joined #openstack-lbaas19:01
*** abaindur has quit IRC19:01
*** abaindur has joined #openstack-lbaas19:01
*** aojea_ has joined #openstack-lbaas19:06
*** aojea_ has quit IRC19:26
*** aojea_ has joined #openstack-lbaas19:27
*** aojea_ has quit IRC19:30
*** aojea_ has joined #openstack-lbaas19:31
*** aojea_ has quit IRC19:52
*** pcaruana|elisa| has joined #openstack-lbaas20:12
*** pcaruana|elisa| has quit IRC20:32
openstackgerritMichael Johnson proposed openstack/octavia master: Add framework for octavia-status upgrade check  https://review.openstack.org/61288320:43
*** aojea_ has joined #openstack-lbaas20:57
openstackgerritSwaminathan Vasudevan proposed openstack/octavia master: Update osutil support for SUSE distro  https://review.openstack.org/54181121:20
*** salmankhan has joined #openstack-lbaas21:20
*** yamamoto has joined #openstack-lbaas22:10
abaindurjohnsom: Hey, I might've already asked this before, but a question about the LB mgmt network. It seems octavia can only talk to the amphora via its fixed IP on the lb mgmt network. Is there a way to use floating IPs instead?22:11
abaindurIt seems the requirement is that the lb mgmt network must only be a provider network, if it cant talk via floating ip/external network.22:12
johnsomabaindur No, we currently do not have support for adding floating IPs to the lb-mgmt-net addresses. You can specify any network, including public networks if you really want, but we don't have code to add floats to those addresses.22:13
johnsomNo, it doesn't have to be a provider network. It's simply a neutron tenant network22:13
abaindurHow does octavia talk to an isolated tenant network then? the octavia-worker process isnt plumbed into OVS or attached to a neutron port - its a process running directly on the host22:14
abaindurWe are out of VLANs to allocate a provider network that exists in a physical router. Therefore we can create an isolated VXLAN based tenant network. VMs have external access, but only via floating IP22:14
*** yamamoto has quit IRC22:15
johnsomabaindur There are many ways to do this. One is to add a port to OVS like we do in devstack. One is to route the traffic.22:15
johnsomYeah, or piggy back it on one of your existing networks. It doesn't do anything fancy/special, it is simply IP traffic with TCP and UDP in them.22:16
abaindurI'm not familiar with behavior of former (devstack). route the traffic... where? in our physical router? This would make it a "provider" network, then no? But the problem is, the tenant network is VXLAN based. We dont have hardware based VTEPS or vxlan gateways22:17
abaindurso lets say our lb_mgmt network is a VXLAN based tenant network, on
abaindurOctavia-worker resides on a host22:19
abaindurif our tenant network was VLAN based, we could tag it in our switches, and add it to our router. SO our host's networking can communicate to
abaindurBut this implies its a provider network, and that we have available vlans22:20
johnsomCan your host access your VXLAN underlay?22:23
abaindurSuppose our host has connectivity via, on vlan 4. If I make my LB mgmt network, on say vlan 8, I can add vlan 8 to our switches and add a route to 10.8/16 on physical router. I'm not sure what i'd need to do to achieve this, if was VXLAN based22:23
abaindurWe dont do VXLAN on any of our physical switches22:24
*** aojea_ has quit IRC22:24
abaindurthe encap/decap is all done by OVS on the br-tun VTEPs22:24
johnsomI was just thinking if you could add a VXLAN endpoint on your controller host (linux software based) and add it to the neutron vxlan tenant network. But it sounds like you can't do that.....22:25
johnsomIt is super hard to come up with the right decision for your environment when I don't really know much about it. However, I can describe how the lb-mgmt-net works and maybe that will spark an idea?22:27
abainduryea.... hence why i wa hoping there was a way to do this using external networks/floating IPs22:27
johnsomWell, you can  use an external network for your lb-mgmt-net. There is no restriction there. Just not floating IPs (NAT)22:27
abaindurI dont believe you can attach a VM to an external network22:28
abaindurA Provider network is really the same as an external network. Only difference is external network does NAT and you cannot attach a VM directly to it. A provider network can attach to a VM, but not do NAT22:28
johnsomSo, fundamentally the lb-mgmt-net is a neutron network, type doesn't matter to octavia. It just needs to be a network with a subnet that the octavia user can attach to the service VMs (amphora).22:29
johnsomThe tricky part is how to get your controller processes access to those addresses. (CW, HM, HK processes. API does not need access)22:29
abaindurFor reference, in neutron lbaas using AVI, uses a similar LB mgmt network that it attaches to each service engine (equivalent of amphora)22:30
abaindurthe difference is, the AVI controller is a VM, deployed in the same openstack cloud22:30
johnsomOne option is to pop it off on your network nodes by adding a port to OVS. This can also be done if you put neutron on your controller hosts.22:31
abaindurso the AVI controller is attached to the same fixed IP/neutron network22:31
abaindurhence, avi can talk directly over L2 to an isolated service engine22:31
johnsomOne option is to use a neutron router to make a routable path from the tenant network to another network in your environment that the controllers are on.22:31
abainduryep and in octavia case, its a process but not a VM. so it attaches to host networking, not OVS/neutron22:31
abaindurI guess if we packaged octavia-worker and health-manager inside a VM image, and booted it up as a "Octavia controller" VM in same environment, it would work22:32
abaindursimilar to AVI22:32
johnsomIt needs access to the rabbit and database for your cloud.  Probably exactly like avi.22:33
johnsomThe controllers also need access to the other service APIs, keystone, neutron, nova, etc.22:33
abaindurthat can be achieved by assigning a floating IP to the VM22:33
johnsomFrankly most of us run them in containers of some sort, so running in a VM should not be a problem.22:34
johnsomUgly solutions that would work if you are really stuck is to tunnel the networks. GRE/IPSec, etc. There isn't that much traffic between the amps and controllers. The highest use is the heartbeats back.22:35
abaindur"One option is to use a neutron router to make a routable path from the tenant network to another network in your environment that the controllers are on."22:39
abaindurok let me try to work that out...22:39
abaindurBack to my example, we attach our VXLAN based network, to a neutron router... suppose here we have an external network on vlan 4 (same VLAN and subnet as our host's networking)22:41
abaindurNow the amphora has octavia's IP as part of its controller_ip_port_list config... so it will try to contact 10.4. this theoretically might work?22:42
abaindurwell this would go out via SNAT22:43
*** salmankhan has quit IRC22:43
johnsomBoth sides connect via IP, so as long as there is a route in both directions it works fine. You would not want to NAT though.22:44
abaindurok,  guess it would not be ablke to be an external network22:45
abaindurwe'd have to add 10.4 vlan 4 as a provider net, and attach to same neutron router22:45
abaindur... and I suppose in our physical router, we would have to add a route saying ---> nexthop = the 10.4 IP of our neutron router?22:47
abaindurI guess we are back to same problem then though. requires us to add this network in neutron, VLAN based. we have just made it a point to point link between neutron router and physical router, rather than deploying amphora directly on it22:56
johnsomI think I am following.  Here is what I am saying:22:59
johnsom1. create the lb-mgmt-net in neutron, it can whatever type you need. Create a subnet on that network, let's say
johnsom2. Add that network will be attached to a neutron router, either by default gateway or host route.23:01
johnsomAt this point you can boot an amphora, it will get a address, and can reach the router address (gateway, etc.)23:01
johnsom3. Then on the controller hosts, let's say they are connected to, and you CW host has
johnsom4. setup routing where that the can reach a router that has a path to the neutron router. This could be a physical router, etc. It needs to know how to reach (could be multiple hops) the neutron router needs to know how to reach (again, can be multiple hops).23:04
johnsom5. boot network config in octavia.conf points to the subnet/network, controller_port_ip_list has the, etc. addresses.23:05
johnsomThat configuration works.23:06
abainduryeah, I think the problem is just with #4, specifically, "It needs to know how to reach (could be multiple hops) the neutron router needs to know how to reach (again, can be multiple hops)."23:08
abaindurIt can't go out the external network of the router, due to NAT23:08
abaindurSo the neutron router can't just attach 2 subnets - our external network and
*** salmankhan has joined #openstack-lbaas23:09
johnsomWell, it has to be so that it can provide floating IPs right?23:09
abaindurwill need to attach another network to this router, in order to route the traffic to 196.168 out of the router23:10
johnsomThat is how the NAT works for floating IPs23:10
abainduryeah, but we are not attaching floating IPs to the amphora23:10
abaindurSo it needs to leave the router without being NAT'd, still retaining the source IP23:11
johnsomYeah, which neutron can do.23:12
abaindurBut because we dont have any vxlan gateways of our own, it would have to leave via a VLAN based network23:12
abaindurThat is mainly the problem ^^23:12
abaindurburning up a VLAN, when they might be limited or not readily available23:13
johnsomI'm not sure the L2 matters here23:13
abaindurHow else would packet leave the neutron router then?23:14
johnsomYour VXLAN tenant lb-mgmt-net terminates at the neutron router. What is L2 for the otherside could be anything23:14
abaindurext-network <---> (neutron router) <---> lb-mgmt net23:14
abaindurit seems like we'd need a 3rd subnet attachment there23:14
abaindurbecause we always NAT on the external network for the router23:15
johnsomNot necessarily.23:15
abaindurwouldn't everything get SNAT'd outbound?23:15
johnsomIt has to have an address on the ext-net that you could route to23:15
johnsomNot if the routing table tells it not to SNAT it, but to forward it23:16
*** fnaval has quit IRC23:16
abaindurhmm ok. I assumed the ip rule in qrouter would redirect packet to snat namespace(because the amph does not have a floating IP), and there all outbound packets would get SNAT'd23:17
abaindurip netns exec qrouter-72471ae2-5960-464b-8be2-dfcb035383c7 ip rule23:19
abaindur46861:from lookup 1623:19
abaindur57482:from lookup 1623:19
abaindur2891579393:from lookup 289157939323:19
abaindur3232366593:from lookup 323236659323:19
abaindurThat's usually how the neutron qrouter's are implemented. If theres a floating IP, theres an exact match to source fixed IP. Otherwise everything hits the subnet. Table 2891579393 re-directs packet to SNAT, and I thought just indiscriminately performs SNAT on the packet23:20
abaindurI will have to try and confirm behavior23:21
*** yamamoto has joined #openstack-lbaas23:23
abaindurjohnsom: actually nvm. I realized this table and the ip rules are only lookedup if no route exists in the router's main table. It usually is the case as the router does not have a default route23:25
abaindurMaybe adding a static route over the ext-network would allow us to bypass the SNAT rule23:26

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!