Friday, 2015-05-15

*** madhu_ak has joined #openstack-lbaas00:03
*** madhu_ak_ has joined #openstack-lbaas00:06
xgermanso dougwigs lb create is spawning an amphora...00:07
johnsom_Cool.  I have stack going again00:08
*** madhu_ak has quit IRC00:09
xgermanmy stack’s neywork is hosed00:12
*** mestery has quit IRC00:14
*** madhu_ak_ has quit IRC00:48
*** apuimedo has quit IRC00:53
*** sbalukoff has quit IRC01:16
*** fnaval has quit IRC01:21
*** sbalukoff has joined #openstack-lbaas01:28
*** mixos has joined #openstack-lbaas01:41
*** mixos has quit IRC01:56
*** mixos has joined #openstack-lbaas02:05
*** mixos is now known as mixos-away02:05
*** mixos-away is now known as mixos02:06
*** clev has joined #openstack-lbaas02:12
openstackgerritBrandon Logan proposed stackforge/octavia: Fixes health monitor create issues
openstackgerritBrandon Logan proposed stackforge/octavia: Adding sudo permissions to SSH Driver commands
bloganrm_work: ping02:27
blogandougwig: ping02:29
bloganlol xgerman: ping02:30
dougwigblogan: super ack02:30
bloganeveryone who said something ot me earlyie: ping02:30
blogandougwig: you can review that too, but i'm not sure what adam was talking about02:30
dougwigblogan: the added line 288, shouldn't that have a sudo/02:32
bloganself._execute_command does that, by default it runs as root02:33
bloganerr adds sudo02:33
dougwigok.  odd default, though.02:34
bloganlol i just realized that02:34
bloganof course most of these will be run as root02:35
openstackgerritBrandon Logan proposed stackforge/octavia: Adding sudo permissions to SSH Driver commands
blogandougwig; ^^02:43
dougwigSec, Mac updating.  I waited so long it's like 9 patches02:48
xgermank, will check right now02:55
xgermanwill leave +A to dougwig02:56
xgermanping me if you need me again ;-)02:57
*** ajmiller has joined #openstack-lbaas02:59
openstackgerritMerged stackforge/octavia: Adding sudo permissions to SSH Driver commands
*** mixos has quit IRC03:27
*** Aish has joined #openstack-lbaas03:43
*** fnaval has joined #openstack-lbaas03:46
*** HenryG has quit IRC03:51
*** clev is now known as clev-away04:03
*** clev-away is now known as clev04:03
openstackgerritPhillip Toohill proposed stackforge/octavia: Preparing for tempest testing
openstackgerritPhillip Toohill proposed stackforge/octavia: Adds scenario test
*** clev is now known as clev-away05:12
*** blogan_ has joined #openstack-lbaas05:19
*** clev-away is now known as clev05:20
*** clev is now known as clev-away05:21
*** blogan_ has quit IRC05:28
*** clev-away is now known as clev05:31
*** HenryG has joined #openstack-lbaas05:38
*** blogan_ has joined #openstack-lbaas05:43
*** mestery has joined #openstack-lbaas05:44
*** clev is now known as clev-away05:47
*** mestery has quit IRC05:53
*** mestery has joined #openstack-lbaas05:53
*** mestery has quit IRC06:13
*** Aish has quit IRC06:34
*** blogan_ has quit IRC06:43
*** jschwarz has joined #openstack-lbaas06:45
*** jschwarz has quit IRC06:50
*** Guest19895 has quit IRC07:21
*** chlong has quit IRC07:32
*** apuimedo has joined #openstack-lbaas08:09
*** woodster_ has quit IRC08:10
*** fnaval has quit IRC08:47
*** bedis_ is now known as bedis08:54
openstackgerritAnand Shanmugam proposed openstack/neutron-lbaas: Adding code to prevent vip port deletion from port api
*** rdekel has joined #openstack-lbaas12:47
*** rm_you| has joined #openstack-lbaas12:48
*** rm_you has quit IRC12:51
*** rdekel has quit IRC12:55
*** rm_you|wtf has joined #openstack-lbaas13:13
*** rm_you| has quit IRC13:16
*** woodster_ has joined #openstack-lbaas13:41
*** apuimedo has quit IRC13:53
*** amotoki has joined #openstack-lbaas14:16
*** clev-away is now known as clev15:03
*** Aish has joined #openstack-lbaas15:22
*** Aish has quit IRC15:22
*** Aish has joined #openstack-lbaas15:28
*** mlavalle has joined #openstack-lbaas15:40
*** Aish has left #openstack-lbaas15:45
*** apuimedo has joined #openstack-lbaas15:49
*** mixos has joined #openstack-lbaas15:51
mixos          <== need one more core review. :- )15:56
*** shakamunyi has quit IRC16:00
*** barra204 has quit IRC16:00
*** madhu_ak has joined #openstack-lbaas16:03
*** sbalukoff has quit IRC16:06
johnsom_blogan So I am stuck with network issues at the moment.  If I set VIP network as public I can't talk to the amp any longer, if I use internal net I have to use a namespace to access, if I use LB network the secondary IP doesn't come up in the amp so haproxy won't bind.16:10
johnsom_My next attempt is going to be setting up another custom network and try that as the vip network16:10
*** mlavalle has quit IRC16:12
mixos@johnsom_ is it possible to use private IP for VIP and create router connecting your private net and pub net.16:12
mixosthen use floating IP assignment to your VIP ?16:13
johnsom_Yeah, floats is an option, we just aren't setting up floats in the code yet, so would prefer to have a cleaner solution.  It might come down to setting up floats manually for now16:14
*** hitalia has joined #openstack-lbaas16:27
johnsom_It looks like to me the amp doesn't have a route back16:27
johnsom_It's router is missing a route or port16:28
mixosIn openstack, I was not able to assign pub IP to a VM directly. it always require VM to have private IP so that I can associate FIP to it.16:37
mixosIf I want to connect pub IP(FIP) , a router was required. maybe Octavia works different way..16:38
*** apuimedo has quit IRC16:43
*** hitalia has quit IRC16:45
bloganjohnsom_: back16:48
bloganjohnsom_: i haven't run into the issue of the amp not being able to bind16:48
bloganmixos: thats what i've run into as well, but some deployments we shoudl be able to assign pub IP16:50
bloganbut as far as I've known the way to get a public ip in a typical devstack is to point a floatingip to a non-public network16:50
mixosblogan: yes.16:51
johnsom_So I setup another network using the same script that sets up the lb-mgmt lan.  Still the amp isn't responding on it.  I can see the request come in with tcpdump, just not seeing the response16:54
mixos@johnsin_   unless your vm setup routing rule between two different network, two private network would not know how to talk to each other.  A router is still required. In the router, you may be able to add interfaces of all of private networks.  If VIP, LB VM and pool(+backend) in same network, you would not have this issue. I guess.16:59
mixosjohnsin_ --> johnsom_17:00
johnsom_Yeah, I added a port on the router17:01
dougwigjohnsom_: for demo, you could launch a browser inside the namespace.17:03
johnsom_Yeah, that is the fall back if I can't crack this issue today17:03
*** hitalia has joined #openstack-lbaas17:12
*** Guest77307 has joined #openstack-lbaas17:17
*** sbalukoff has joined #openstack-lbaas17:19
*** Guest77307 is now known as redrobot17:20
*** hitalia has quit IRC17:24
*** redrobot has left #openstack-lbaas17:26
*** redrobot has joined #openstack-lbaas17:26
*** mestery has joined #openstack-lbaas17:26
*** redrobot has quit IRC17:29
*** amotoki has quit IRC17:33
*** redrobot has joined #openstack-lbaas17:36
bloganjohnsom_: amp isn't responding to vip>?17:36
*** redrobot is now known as Guest6319317:36
bloganargh bbiab17:36
johnsom_Yeah, I don't see the answer going out the default route interface17:37
*** Guest63193 is now known as redrobot17:39
johnsom_What I don't get is the lb-mgmt net works but lb-vip doesn't even though I used the same setup script.17:42
*** barclaac has joined #openstack-lbaas17:46
*** bharath has joined #openstack-lbaas17:55
*** openstackgerrit has quit IRC18:22
*** openstackgerrit has joined #openstack-lbaas18:23
*** shakamunyi has joined #openstack-lbaas18:37
*** barra204 has joined #openstack-lbaas18:37
*** mixos is now known as mixos-away18:44
*** mixos-away is now known as mixos18:57
johnsom_I am pretty sure it is a routing issue as if the default route is changed from to (my vip network) the vip ips start working but the lb-mgmt ip stops working19:03
johnsom_Anyway, I have scrambled my devstack networking trying things, so restacking19:04
ptoohillso having both the routes breaks it?19:07
ptoohillthe devstack plugin inserts that route for the mgmt net19:08
ptoohillI cant even stack, not sure whats going on today :/19:08
bloganjohnsom_: an alternative i may investigate is to drop the allowed address pairs implementation and just expose the amphora's port's ip on that network, so instead of the vip being a different ip it'll be teh same ip that the amphora's is on the vip network19:16
*** hitalia has joined #openstack-lbaas19:26
johnsom_Not sure that will change things.  I started up a simple web server listening on all IPs and neither of the VIP addresses responded19:36
*** Varun_Lodaya has joined #openstack-lbaas19:40
*** apuimedo has joined #openstack-lbaas19:47
*** barra204 has quit IRC19:48
*** shakamunyi has quit IRC19:48
openstackgerritGerman Eichberger proposed stackforge/octavia: Implements the haproxy amphora agent api server
bloganjohnsom_: yeah pretty sure its a routing issue on the amphora19:55
bloganjohnsom_: err maybe not19:56
xgermanpick one19:57
*** mlavalle has joined #openstack-lbaas20:06
*** mixos is now known as mixos-away20:22
*** mixos-away is now known as mixos20:39
*** mlavalle has quit IRC20:47
openstackgerritGerman Eichberger proposed stackforge/octavia: haproxy reference amphora REST API client
xgermandougwig: how do I fix this:21:31
bloganjohnsom_: figured the issue out21:35
bloganjohnsom_: not sure there is a way around it21:35
johnsom_I was thinking about doing a bridge interface onto a vip network.  Haven't had a chance to try it yet though.21:36
xgermanblogan - I am curious… so what’s the root cause21:36
bloganthe default gw on the amphora goes through the ssh interface, so when we make requests over the customer data network it attempts to reply back through that ssh interface21:39
xgermanmmh, but we bind the harpy to the vip interface?21:39
bloganxgerman: we do but any requests going to that interface will be replied back on to the mgmt interface21:41
xgermanok,mmh, this is confusing me…21:43
bloganso if you change the default route in the amphora to go over the vip network, then the requests will work, but you won't be able to ssh into the instance from the global/default namespace21:44
blogankind of a either or thing21:44
xgermanok, there must be a third way21:44
bloganwell there's ways to get around it21:45
xgermanstatic route?21:46
xgermansince we know the ips of the controller?21:46
bloganwell the problem is that we are trying to ssh over a different netowrk than we are tryign to make http requests over it21:47
bloganbut from teh same ip21:47
bloganso in my setup, from the default namesapce, i'm ssh'in ginto, which is the ip of the interface on the amphora21:48
xgermangotcha -21:48
bloganthe amphora's default gw is over the .4 interface, which works21:48
bloganwhen i curl (the lb's vip address), it gets to the interface on the amphora, but replies back on the interface bc of the default route21:49
bloganthe ssh and curl requests are both coming from the same ip
bloganso when teh amphora sees it needs to respond back to the ip, it ALWAYS goes over the mgmt interface and gw, even if it is an http request21:51
bloganwhich then has the wrong source ip21:51
xgermanmakes sense21:51
bloganfor an http request21:51
bloganso you can test this out by logging into the amphora and changing the default route to go through the vip network, and then curl works21:52
bloganbut then you have to ssh to the amphora through the router namespace or the dhcp namespace21:53
xgermanmaybe our mgmt network needs to be in a namespace21:53
blogananyway this definitely needs some more discussion21:53
bloganxgerman: thats somethign i was thinking about, we could create an interface in the global namespace with a different ip and bridge that to another interface in a namespace and then that would solve it21:54
bloganbut this is more of a how do we deploy octavia than an octavia creating the correct routes problem21:55
bloganwell we may have to think about the routes problem too21:55
blogananyway i feel much better knowing what it is now21:55
ptoohillso would we be updating drivers to change route in amp?21:55
blogansad that its not a simple solution21:55
ptoohillmaybe temp?21:56
xgermannah, I think we need to think that more throigh21:56
bloganptoohill: if we change the default route int he driver then we won't be able to ssh into the amphora unless we go through namespace21:56
ptoohillIm just thinking how to (hack if needed) to get my tests working21:56
blogangoing through namespace requires knowledge of what routers/dhcp namespaces exist21:56
blogangood thing there's a summit to talk about these things21:57
xgermanI am thinking the same21:57
ptoohillhow do you plan to do it for demo?21:57
xgermanfake it21:57
bloganjust curl over a namespace21:57
xgermanit’s all video — so we can use special effects21:57
ptoohillfair enough21:57
ptoohilllens flare21:57
ptoohillLOTS of lens flare21:57
ptoohillare you sure theres no way with flows to manipulate this? thats what ovs is for21:58
ptoohillhad same type of scenario during my ryu testing21:59
ptoohillthough, i manually built the flows and not quite sure how to do the same here21:59
xgermanyeah, you can use flows since we are on different ports (unless somebody needs an lb on 22)22:00
xgermanor 8443 on REST22:00
bloganwe would need ovs running on the amphora22:00
bloganwhich i dont want to do22:00
bloganplus its not a problemw ith octavia in my mind22:01
ptoohillwhat is it then?22:01
ptoohillnetworking in general?22:02
ptoohillwell bummer22:04
bloganah there is a way to do this without iptables, ip rules22:09
blogancan basically say any traffic you receive on this interface, go back out that interface22:09
bloganthat same interface22:09
bloganit is some setup though22:09
bloganthis is probably somethign we want anyway22:12
*** mixos has quit IRC22:24
*** mixos has joined #openstack-lbaas22:27
blogani'll try to get a patch out tonight, if i can get this working22:27
johnsom_I am about ot try hacking up a bring into a vip network22:29
*** madhu_ak has quit IRC22:29
*** apuimedo has quit IRC22:34
openstackgerritGerman Eichberger proposed stackforge/octavia: haproxy reference amphora REST API client
dougwigxgerman: you have to submit a patch against openstack/requirements22:42
xgermanI was afraid of that22:43
xgermanok, will do22:43
*** clev is now known as clev-away22:51
*** mixos has quit IRC23:09
rm_workreviewing all the discussion today and it seems to be primarily networking mumbo-jumbo O_o23:10
*** mlavalle has joined #openstack-lbaas23:15
xgermandougwig ^^23:19
*** clev-away is now known as clev23:21
*** hitalia has quit IRC23:23
*** hitalia has joined #openstack-lbaas23:26
openstackgerritmin wang proposed stackforge/octavia: Continue fixing Octavia complexity issues
*** clev is now known as clev-away23:39

Generated by 2.14.0 by Marius Gedminas - find it at!