Friday, 2017-06-09

openstackgerritAdam Harwell proposed openstack/python-octaviaclient master: Add check O346 to disallow backslash line continuation
openstackgerritAdam Harwell proposed openstack/python-octaviaclient master: Add check O346 to disallow backslash line continuation
rm_work^^ had to resubmit to cover some additionally introduced errors T_T00:02
rm_workfeel free to +2 really hard00:02
rm_workso no more get in00:02
johnsomUmm, I already had Jude fix those00:03
rm_workah did you?00:03
rm_workrebased to master and it still had some00:03
rm_workmaybe different patchset?00:03
johnsomYeah, his patch hasn't merged yet00:03
rm_workso i wonder ...00:04
openstackgerritAdam Harwell proposed openstack/python-octaviaclient master: Add check O346 to disallow backslash line continuation
openstackgerritMerged openstack/python-octaviaclient master: Fix command to include PROXY and update metavars
rm_workgot my +2's back00:05
openstackgerritAdam Harwell proposed openstack/python-octaviaclient master: Add check O346 to disallow backslash line continuation
rm_workusually works :P00:05
rm_workheh yep passes pep800:06
johnsomCoolness, unless someone has a reason why not, I will cut a client release tomorrow00:06
johnsomAre you guys going to add quota is that something I need to put on my list?00:07
rm_workMy OCD might actually be satisfied for like 5 seconds00:07
rm_workah I'm sure Jude can get to it ^_-00:07
johnsomOh, darn, sorry to pop your OCD bubble....00:07
johnsomOk, off to date night.  Catch you all tomorrow.00:08
*** armax has quit IRC00:11
openstackgerritMerged openstack/python-octaviaclient master: Add check O346 to disallow backslash line continuation
*** sticker has joined #openstack-lbaas00:22
rm_workdiltram_: !00:38
openstackgerritAdam Harwell proposed openstack/octavia master: Enable add debug account for ssh access
rm_workrebased and testing00:41
marcin123Using Ocata/Neutron/Octavia should this show non zero output: "neutron lbaas-loadbalancer-stats" ?00:50
rm_workhmmm, via neutron yes I believe so00:50
rm_workif any traffic has passed on your LB00:50
rm_workyou can check the octavia DB and see if it's being recorded there00:50
marcin123Octavia db has it in table listener_statistics, but Neutron db00:56
*** leitan has joined #openstack-lbaas00:56
marcin123table lbaas_loadbalancer_statistics has zeros, so most likely neutron call is just looking into Neutron DB00:57
leitanHi guys im back, sorry soccer game, rm_work johnsom any thoughts about my logs ?00:57
rm_workleitan: err, my thoughts are ... let me look00:58
rm_workI got busy >_<00:58
rm_work[15:55:13]  <johnsom>leitan There is something in your nova config adding the extra IPs.  grep that log for "a3c64b9d-79eb-48b0-b2fd-fb6e57ab0824".  You will see that we poll nova to see when the status goes active so there are a number of debug messages while nova is building the instance, noted by "status": "BUILD".  As the build progresses nova is adding those interfaces to the instance.  If you grep for REQ you can see00:58
rm_work how we are asking nova to build it with one network.00:58
rm_workso yeah00:59
rm_worksomething funky is happening on the nova side apparently00:59
leitanrm_work: weird, i have launched by now 100 instances attached to the lb-mgmt-network and cant reproduce the behaviour, but amphorae does this with every first LB of the couple i create01:00
rm_work2017-06-08 19:48:19.951 31 WARNING octavia.compute.drivers.nova_driver [-] Both amp_image_id and amp_image_tag options defined. Using the amp_image_id.01:02
rm_workis that on purpose for debugging or such?01:03
rm_workhopefully just planning to use amp_image_tag when you're up and running01:03
leitanrm_work: tried the tag, didnt work, went back to using image id01:05
leitanrm_work johnsom , heres my nova.conf
rm_workwhen you upload the image to glance01:07
rm_workyou have to give it the tag there01:07
leitani gave them the tags01:08
leitanill go back to try that later01:08
leitanby now i just want to consistenly build load balancers01:08
rm_worklol right yeah sorry01:12
rm_workyeah i'm not seeing anything weird01:12
leitanme neither01:12
leitani have 3 neutron apis currently serving the traffic01:13
leitanill try disabling two, and let just one running01:13
leitanmaybe is something related to a deadlock or something like that01:13
rm_workand watch logs from there, maybe see what is asking for the other ips?01:13
leitanbut i found that really weird that this doestn happen in the normal instance creation workflow01:13
rm_workmaybe try mimicking EXACTLY the boot command we make01:14
leitanill do that01:15
*** bzhao_ has quit IRC01:16
*** aojea has joined #openstack-lbaas01:23
*** aojea has quit IRC01:27
*** armax has joined #openstack-lbaas01:32
*** leitan has quit IRC01:32
*** leitan has joined #openstack-lbaas01:34
rm_workalright i'm disappearing for a bit01:38
*** fnaval has quit IRC01:50
*** fnaval has joined #openstack-lbaas01:51
leitanjohnsom: checked the line that you told me on the, its there02:00
leitanwith the ":"02:01
*** leitan has quit IRC02:44
*** KeithMnemonic1 has joined #openstack-lbaas02:49
*** KeithMnemonic has quit IRC02:53
*** fnaval has quit IRC02:54
*** yamamoto has joined #openstack-lbaas03:07
openstackgerritZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id
*** gans has joined #openstack-lbaas03:42
openstackgerritOpenStack Proposal Bot proposed openstack/octavia-dashboard master: Updated from global requirements
*** links has joined #openstack-lbaas04:20
*** yamamoto_ has joined #openstack-lbaas04:43
*** kobis has joined #openstack-lbaas04:45
*** yamamoto has quit IRC04:47
*** fnaval has joined #openstack-lbaas04:55
*** fnaval has quit IRC05:00
*** kobis has quit IRC05:02
*** Guest72612 has quit IRC05:06
*** Guest6511 has joined #openstack-lbaas05:06
*** links has quit IRC05:10
*** links has joined #openstack-lbaas05:26
*** armax has quit IRC05:29
*** gcheresh_ has joined #openstack-lbaas05:47
*** sticker has quit IRC06:33
*** JudeC has joined #openstack-lbaas06:35
*** tesseract has joined #openstack-lbaas06:53
*** fnaval has joined #openstack-lbaas06:56
*** fnaval has quit IRC07:01
*** pcaruana has joined #openstack-lbaas07:05
*** links has quit IRC07:09
*** rcernin has joined #openstack-lbaas07:09
*** kobis has joined #openstack-lbaas07:21
*** links has joined #openstack-lbaas07:25
*** kobis has quit IRC07:29
*** aojea has joined #openstack-lbaas07:31
*** yamamoto_ has quit IRC07:38
*** JudeC has quit IRC07:49
*** yamamoto has joined #openstack-lbaas07:57
*** yamamoto has quit IRC07:59
*** yamamoto has joined #openstack-lbaas07:59
*** yamamoto_ has joined #openstack-lbaas08:46
*** yamamoto has quit IRC08:47
*** fnaval has joined #openstack-lbaas08:57
*** fnaval has quit IRC09:01
*** yamamoto_ has quit IRC09:03
*** yamamoto has joined #openstack-lbaas09:07
*** yamamoto has quit IRC09:16
*** kobis has joined #openstack-lbaas09:34
*** yamamoto has joined #openstack-lbaas09:37
*** yamamoto_ has joined #openstack-lbaas09:38
*** yamamoto has quit IRC09:38
*** links has quit IRC09:51
*** gcheresh_ has quit IRC09:52
*** links has joined #openstack-lbaas10:03
*** kobis has quit IRC10:07
*** gans has quit IRC10:19
*** kobis has joined #openstack-lbaas10:33
openstackgerritSongming Yan proposed openstack/neutron-lbaas master: Fix the problem of the file mode.
*** kobis has quit IRC10:51
*** yamamoto_ has quit IRC11:42
*** yamamoto has joined #openstack-lbaas11:43
*** aojea has quit IRC11:53
*** cpuga has joined #openstack-lbaas12:22
*** cpuga_ has joined #openstack-lbaas12:30
*** cpuga has quit IRC12:34
*** catintheroof has joined #openstack-lbaas12:40
*** catintheroof has quit IRC12:45
*** catintheroof has joined #openstack-lbaas12:46
*** links has quit IRC12:48
*** aojea has joined #openstack-lbaas12:54
*** fnaval has joined #openstack-lbaas12:55
*** fnaval has quit IRC12:57
*** fnaval has joined #openstack-lbaas12:58
*** aojea has quit IRC12:59
*** yamamoto has quit IRC13:04
*** yamamoto has joined #openstack-lbaas13:04
*** aojea has joined #openstack-lbaas13:22
*** aojea has quit IRC13:26
*** aojea has joined #openstack-lbaas13:33
*** chlong has joined #openstack-lbaas13:51
*** leitan has joined #openstack-lbaas13:55
*** tesseract has quit IRC14:16
*** fishbone has joined #openstack-lbaas14:20
*** tesseract has joined #openstack-lbaas14:23
leitanhi guys, still debugging de duplicated management IP issue, still cant find anything, i see two interface_attach actions, but still dont know why14:25
leitancreated tons of vms, on the same network, same image14:26
leitanthe issue doesnt happen14:26
leitanill try to emulate the same create command that you guys send, to see if theres something there14:26
leitanif not, ill try to add a "hack" for the rest_api.driver to iterate over all the interfaces trying to configure the LB14:27
fishboneHello all, when I try to create a load balancer in the dashboard I receive an error: error source_type14:30
fishboneI am able to create load balancers in the cli, however; when created the horizon lbaasv2 panel no longer works at all14:30
leitani am also wondering why is not a stable/newton tag on the neutron-lbaas-dashboard repo14:33
xgerman_leitan we didn’t do a newton release since there haven’t been changes but we learned form that omission and are now doing a release no matter what14:37
leitanxgerman_: ok, so if i install mitaka on my newton dashboard, its the same, or do you guys recommend to use ocata version on my newton dashboard ?14:38
xgerman_mitaka is the same…14:38
xgerman_we don’t really have a strong opinion14:38
fishboneI was having this issue since liberty, I have updated through to each new openstack release and the issue persisted14:47
fishboneI have reinstalled the panel from the git repository each time14:47
fishbonenot from the master branch however; I installed from one of the stable branches, I am now on ocata14:47
leitanfishbone: whats the error on the panel ? do you have a bug report created since youre having the issue since liberty ?14:51
fishboneleitan: no I do not have a bug reported, apologies, I only receive the error: source_type and error: unable to launch load balancer14:53
fishboneleitan: there are no issues in any logs for apache or neutron-lbaasv214:53
leitanno error on the apache_error.log ? thats weird, do you have your horizon configured as DEBUG=True on your ?14:54
leitanxgerman_: ok ill try mitaka, if for some reason doesnt work ill try ocata14:55
leitani remember using master with haproxy agent-mode wihtout issues from the panel14:55
leitanfishbone: are you using octavia ? or haproxy agent mode ? or f5 ? or what service_provider ?14:56
-openstackstatus- NOTICE: The Gerrit service on is being restarted now to clear an issue arising from an unanticipated SSH API connection flood14:56
xgerman_Yeah, if you can file a bug wwe can have a look - eanderson is doing some GUI work so hopefully we can get to the bottom of this14:58
*** armax has joined #openstack-lbaas15:02
*** rcernin has quit IRC15:08
fishboneleirtan: sorry for the late reply, I am using haproxy with service_provider = LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default15:11
leitanfishbone: that should work totally fine are you installing the _1481 file ? or the _1480 when you copy to /enabled ?15:28
fishbonelieten: I copied both files15:32
fishboneleitan: I copied all of the contents of the enabled/ directory in the neutron-lbaas-dashboard/neutron_lbaas_dashboard/ folder15:32
leitanfishbone: wrong15:33
leitanyo need to copy just the _148115:33
leitancollect static, compress (if compress enabled) and restart apache15:33
leitancan you try that out ?15:33
fishboneleitan: ok, I will try that, thank you15:33
*** aojea has quit IRC15:51
leitanxgerman_ rm_work johnsom i did another finding on the duplicated interface issue, if the loadbalancer VIP is created in a flat network (currentyl my public net) the duplicated ip issue does not happen, never, BUT if i create the vip for an internal LB in a tunneled VXLAN network, the issue comes up every time15:52
xgerman_mmh, interesting…15:53
leitanxgerman_: indeed15:55
*** sanfern has joined #openstack-lbaas15:56
leitanxgerman_: i thought about hacking into here : def _translate_amphora of the , in order that the rest_api_driver choses an ACTIVE port to construct the base_url, cause all the other ports that gets attached are on state DOWN15:56
leitanthis will prevent my setup from failing15:56
leitancause today, it just uses the lb_network_ip = interface.fixed_ips[0]['ip_address']15:57
xgerman_ok, sounds good… we should focus on active ports15:57
leitanthe subzero of the fixedi_ips lists15:57
leitani think this should be always like that15:57
leitanxgerman_: sounds sort of logical in order to fill a bug and get this into master ?15:57
*** dougwig has joined #openstack-lbaas15:58
xgerman_yep, bug, patch, master15:58
leitanxgerman_: roger15:58
fishboneleitman: If I exclude either of the files from the dashboard enabled directory the lbaasv2 panel does not function properly, it seems I need both the 1480 and 1481 files16:00
leitanfishbone: that should not happen:
leitanxgerman_: bug filled16:17
openstackLaunchpad bug 1697033 in octavia "lb_network_ip should choose only ACTIVE ports" [Undecided,New]16:17
*** tesseract has quit IRC16:32
leitanxgerman_: another thing i saw, if i shutdown one instance, the healthmanager does not recreate it, cause it cant contact it, and when i try to operate with the LB everything fails cause the rest_driver is trying to contact the shutdown instance, turning this isntance on, to running state fixed the issue16:34
*** pcaruana has quit IRC16:34
xgerman_well, we had a big discussion once tat if you shut down an instance manually you know what you do and don’t want ti automgically recreated… that’s why we documented the port shhutdown in all our docs as away to force a failover16:39
*** leitan has quit IRC16:44
*** rcernin has joined #openstack-lbaas16:45
*** leitan has joined #openstack-lbaas16:47
leitanxgerman_: great, if that the expected behaviour its ok, and ill handle shutdowns myself16:48
*** blogan has joined #openstack-lbaas16:49
xgerman_yeah,  we were mulling changing that but it’s expected right now16:49
leitantrying to install the dashboard im running into the same issue as fishbone17:08
leitanif i do not copy the _1480 file, the option in the panel does not even appear17:08
leitanbut when both files are copied, i get the "source_type" error17:08
*** sshank has joined #openstack-lbaas17:12
*** aojea has joined #openstack-lbaas17:15
xgerman_mmh, johnsom + I tested the board a while ago — I can only say it ysed to work ;-)17:16
*** leitan has quit IRC17:19
*** aojea has quit IRC17:19
*** armax has quit IRC17:21
*** blogan has quit IRC17:34
*** blogan has joined #openstack-lbaas17:34
fishboneI've noticed a warning that I cannot load the haproxy driver, i have tried device_driver =, device_driver = neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver17:37
*** aojea has joined #openstack-lbaas17:45
*** leitan has joined #openstack-lbaas17:56
fishboneHello all, is there an issue with my naming convention for the device driver or is there a third party application I must install/configure18:06
marcin123Neutron/Octavia (Ocata) What is a recommended way of creating VIP with X-Forwarded-For? I do not see neutron client exposing that option18:09
rm_work`neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver` looks right18:10
leitanxgerman_ rm_work johnsom celebrated too early, it happens when the vip is on a flat network too18:10
*** gcheresh_ has joined #openstack-lbaas18:10
rm_workleitan: T_T18:10
johnsomYeah, that was after neutron client was frozen for deprecation.  You can do it via the API18:10
rm_workmarcin123: X-Forwarded-For is an insert-header thing18:11
leitanill code the patch18:11
rm_workyes, the API supports it18:11
leitanso lb_network_ip18:11
leitanis picked up only from an active port18:11
leitanso i can at least workaround the issue for now18:11
rm_workleitan: erm, rather than jumping immediately to a patch that makes it work with multiple IPs -- it shouldn't have multiple to begin with?18:11
rm_worki guess that's not necessarily bad18:12
leitanrm_work: right ... but, it only happens with octavia, and i cant get to the bottom of the reason why18:12
rm_workdid you manage to exactly duplicate our nova build?18:12
leitani see two or more instance_attach orders on nova18:12
leitanbut i dont know why18:12
leitanrm_work: not yet18:12
johnsomfishbone It will fail with the _1480 file copied.  Be sure to run all of the steps in the readme after you remove that 1480 file.
johnsomI am considering deleting the 1480 file, I just need to understand better why it was there in the first place18:16
leitanjohnsom: currently not working for me using only the _1481 file either18:18
eanderssonYou need _1450 and _148118:20
eanderssonThose are the two we are using with the Ocata version of Horizon18:20
eanderssonActually not sure why _1450 would be needed, but we have it :p18:22
eandersson... not even sure where I got that one from18:23
johnsomYeah, I was trying to figure out what 1450 is18:23
johnsomAh, ok.  1450 is the LBaaSv1 panel which is now EOL18:24
johnsomIt has been deleted from the code base is why I couldn't find it18:25
fishbonejohnsom: Yes I was going to mention I do not see the 1450 in the github repository18:25
johnsomYeah, long since dead code18:25
eanderssonWe keep our own version of Horizon that bundles all of this, so might have accidently carried over from an older version.18:26
johnsomJudeC rm_work any reason not to do a client release?18:34
*** sanfern has quit IRC18:35
eanderssonWish I had more interest in web front-end stuff - so many awesome changes that could be done to the dashboard18:35
johnsomYes, it needs some love18:35
johnsomAnd a hammer18:35
*** sanfern has joined #openstack-lbaas18:38
leitaneandersson: wow, so i need that 1450 also ?18:40
johnsomNo, 1450 is gone18:40
leitanok, so _1481 is not even registering the panel for me18:42
leitanwell johnsom rm_work xgerman_ finally fixed de dup ip issue18:50
leitanwithout touching the code18:51
xgerman_how so?18:52
*** sshank has quit IRC18:52
leitanthe issue was that in this deploy, the backend for glance, was file, not CEPH as we ussualy use, so yes, the amphorae image ... was present only on one glance api node, and are active-active load balanced ... so when a compute tried to spawn and nova-compute hits a glance node without the image, neutron allocates the ip, but failed to build the instance, on another iteration succeed, and attach multiple ports to it18:52
xgerman_ouch, that’s a tough one18:53
leitanxgerman_: this but is related to it
openstackLaunchpad bug 1609526 in OpenStack Compute (nova) "Nova creates duplicate Neutron ports on instance reschedule" [Medium,In progress] - Assigned to Liyingjun (liyingjun)18:53
leitanwithout that cleanup ... this can happen18:54
leitanbut = bug18:54
leitanits friday you know haha18:54
*** aojea has quit IRC19:05
eanderssonThat one is fun19:12
eanderssonBasically for some reason the image download locks up the whole process, causing a slow image download to delay all other http requests19:12
eanderssonpuppet sets the request timeout to 30 or something so the neutron http calls timeout (as they are stuck due to the glance download)19:13
*** blogan_ has joined #openstack-lbaas19:14
leitaneandersson: here was more that two of the three active glance-api didnt actually have the image backing file ...19:15
leitancausing a one or two reeschedules19:15
leitantill hits the "good" glance19:15
*** blogan has quit IRC19:15
eanderssonIt didn't have it at all, or it didn't have it cached?19:15
leitanand nova doesnt tell neutron to cleanup the ip alloc19:15
leitanand assigns multiple ifaces to my amphorae19:16
eanderssonYea - I mean for us there were two issues19:16
leitaneandersson: didnt have it at all19:16
eanderssonBasically for some reason the glance download is blocking all other processes from using requests (the http library)19:16
leitaneandersson: hmm thats an ugly one19:17
leitani dont known why glance exists either19:17
leitanshould be another table on nova19:18
eanderssonWe would see this a lot if an un-cached image was used to create many vms.19:18
leitaneandersson: i see19:18
leitaneandersson: thats why we maily deploy ceph as backing storage, and force the deploys to run vms as bootable volumes19:19
leitanno downloads, no wait, no lock19:19
*** aojea has joined #openstack-lbaas19:20
leitanyeah, but this non-ceph deploy played me well ...19:22
*** leitan has quit IRC19:22
*** aojea has quit IRC19:25
*** JudeC has joined #openstack-lbaas20:13
*** aojea has joined #openstack-lbaas20:21
*** gcheresh_ has quit IRC20:23
*** gcheresh_ has joined #openstack-lbaas20:23
*** gcheresh_ has quit IRC20:40
*** fnaval has quit IRC20:58
*** fnaval has joined #openstack-lbaas21:00
*** blogan_ is now known as blogan21:05
*** KeithMnemonic1 has quit IRC21:06
*** armax has joined #openstack-lbaas21:16
*** JudeC has quit IRC21:45
*** cpuga_ has quit IRC21:46
*** JudeC has joined #openstack-lbaas21:47
*** blogan has quit IRC21:50
*** chlong has quit IRC21:58
*** rcernin has quit IRC22:19
johnsomThis is good for review:
*** aojea has quit IRC22:33
*** aojea has joined #openstack-lbaas22:34
*** aojea has quit IRC22:39
*** catintheroof has quit IRC22:42
*** sanfern has quit IRC23:03
*** sanfern has joined #openstack-lbaas23:04
johnsomFYI, I just built a fresh devstack with horizon enabled, followed the installation instructions for the neutron-lbaas-dashboard using the current master (Pike) versions and it loaded right up.  No problem.  Again, it's important that you follow the readme instructions for installation.  It even creates fully functional load balancers...23:10
*** zioproto_ has joined #openstack-lbaas23:23
*** zioproto has quit IRC23:23
*** zioproto_ is now known as zioproto23:24
*** aojea has joined #openstack-lbaas23:35
*** leitan has joined #openstack-lbaas23:38
*** aojea has quit IRC23:40
*** armax has quit IRC23:41
*** armax has joined #openstack-lbaas23:42
*** armax has quit IRC23:45

Generated by 2.15.3 by Marius Gedminas - find it at!