Friday, 2015-09-04

*** ajmiller has quit IRC00:02
TrevorVjohnsom, xgerman brandon identified some issues00:02
TrevorVHe's pushing a WIP review for further testing00:02
xgermanwill have a look00:02
johnsomTrevorV Excellent, that is good news00:02
*** bharathm has quit IRC00:02
TrevorVHard to debug in my mind, but he zeroed in pretty quick.  I must learn.... :P00:03
johnsomGrin.  blogan is the network plugging master00:04
openstackgerritBrandon Logan proposed openstack/octavia: Rest API plug network interface handling
bloganlol 3 lines00:08
bloganbut there's more to be done, it still didn't bring up the interface on a 2nd member add on a different network, so i need to investigate that one a bit more, i coudl have also had my config wrong too as i was switching around a lot00:09
bloganjohnsom, xgerman: look at the review and if there's a reason behind the original code I couldn't think of let me know, I'm sure I don't ahve the context00:10
xgermannah, I just tried to mimic the SSH driver of that time00:11
bloganthis is actually another reason i preferred just doing a dhclient call to bring teh interface up, i didnt have to worry about the :0 interfaces00:11
bloganbut it won't persist across reboots either sooo its a catch 2200:11
johnsomWith my quick glance this morning, this was the code that I *thought* was the issue.  So, concur00:12
*** minwang2 has quit IRC00:13
xgermanyeah, that code hasn’t changed since Vancouver?00:13
*** diogogmt_ has joined #openstack-lbaas00:26
*** diogogmt has quit IRC00:27
*** diogogmt_ is now known as diogogmt00:27
*** TrevorV has quit IRC00:31
*** amotoki has joined #openstack-lbaas00:43
*** mlavalle has quit IRC00:49
*** KunalGandhi has joined #openstack-lbaas00:51
*** KunalGan_ has joined #openstack-lbaas00:56
*** KunalGandhi has quit IRC00:56
*** SumitNaiksatam has quit IRC01:19
*** madhu_ak has quit IRC01:25
*** clev is now known as clev-away01:58
*** ganeshna has joined #openstack-lbaas02:15
*** ganeshna has quit IRC02:31
*** sballe has quit IRC03:06
*** openstackgerrit_ has joined #openstack-lbaas03:11
denghuiHi i am denghui, i am new to octavia, but i am familiar with openstack-lbaas, now i read v0.5 of component design doc of octavia, i have a question for option2 of front-end topologies?03:17
denghui" both the active and standby amphorae can bind to the vip_addresses of all their assigned loadbalancer objects on a dummy, non-ARPing interface, both can be running all haproxy services at the same time", i don't understand how this happen?03:19
*** woodster_ has quit IRC03:19
*** clev-away has quit IRC03:28
*** bharathm has joined #openstack-lbaas03:31
*** clev has joined #openstack-lbaas03:32
*** vivek-ebay has joined #openstack-lbaas03:45
*** ganeshna has joined #openstack-lbaas04:05
*** openstackgerrit_ has quit IRC04:14
*** denghui has quit IRC04:27
*** clev is now known as clev-away05:02
*** vivek-ebay has quit IRC05:50
*** KunalGandhi has joined #openstack-lbaas05:52
*** ajmiller has joined #openstack-lbaas05:55
*** ajmiller has quit IRC05:55
*** KunalGan_ has quit IRC05:56
*** KunalGandhi has quit IRC06:04
*** numans has joined #openstack-lbaas06:17
rm_workblogan / ptoohill: Barbican devstack is broken right now due to a keystone change that merged earlier (probably why TrevorV was having trouble today -- he is super lucky with this stuff)06:36
*** ganeshna has quit IRC06:48
*** vivek-ebay has joined #openstack-lbaas06:50
*** ganeshna has joined #openstack-lbaas06:51
*** vivek-ebay has quit IRC06:55
*** ganeshna has quit IRC07:00
*** ganeshna has joined #openstack-lbaas07:04
openstackgerritStephen Balukoff proposed openstack/neutron-lbaas: Make pools independent of listeners
*** ganeshna has quit IRC07:08
*** denghui has joined #openstack-lbaas07:15
*** bharathm has quit IRC07:51
*** ganeshna has joined #openstack-lbaas08:02
*** numans has quit IRC08:32
*** ganeshna has quit IRC08:40
*** ganeshna has joined #openstack-lbaas08:44
*** bradjones has joined #openstack-lbaas08:47
*** bradjones has quit IRC08:47
*** bradjones has joined #openstack-lbaas08:47
*** numans has joined #openstack-lbaas08:50
*** numans has quit IRC09:08
*** Jet has joined #openstack-lbaas09:11
*** Jet has quit IRC09:11
*** Jet has joined #openstack-lbaas09:12
*** amotoki has quit IRC09:17
rm_workas soon as Barbican should be deploying properly again in devstack09:23
rm_work*as soon as ^^ merges09:23
*** ganeshna has quit IRC09:48
*** vjay11 has joined #openstack-lbaas09:53
*** ganeshna has joined #openstack-lbaas09:54
*** Jet has quit IRC09:59
*** vjay12 has joined #openstack-lbaas10:03
*** vjay11 has quit IRC10:04
openstackgerritBrandon Logan proposed openstack/octavia: Rest API plug network interface handling
*** Jet has joined #openstack-lbaas10:11
*** denghui has quit IRC10:11
*** vjay12 has quit IRC10:12
*** kiran-r has joined #openstack-lbaas10:35
*** bharathm has joined #openstack-lbaas10:52
*** bharathm has quit IRC10:56
*** denghui has joined #openstack-lbaas10:56
*** kiran-r has quit IRC11:09
*** kiran-r has joined #openstack-lbaas11:18
*** ganeshna has quit IRC11:18
*** Jet has quit IRC11:25
*** numans has joined #openstack-lbaas11:32
*** _kiran_ has joined #openstack-lbaas11:40
*** vjay12 has joined #openstack-lbaas11:42
*** kiran-r has quit IRC11:43
*** _kiran_ has quit IRC11:49
*** vivek-ebay has joined #openstack-lbaas11:55
*** vivek-ebay has quit IRC11:59
*** vjay12 has quit IRC12:01
*** vjay12 has joined #openstack-lbaas12:06
*** vjay12 has quit IRC12:09
*** denghui has quit IRC12:51
*** denghui has joined #openstack-lbaas12:52
*** diogogmt has quit IRC13:07
*** woodster_ has joined #openstack-lbaas13:14
*** ajmiller has joined #openstack-lbaas13:19
*** mestery has joined #openstack-lbaas13:24
*** mestery has quit IRC13:26
*** mestery has joined #openstack-lbaas13:27
*** ajmiller has quit IRC13:28
*** ajmiller has joined #openstack-lbaas13:29
*** ajmiller has quit IRC13:32
*** ajmiller has joined #openstack-lbaas13:33
*** ajmiller has quit IRC13:33
*** apuimedo|lunch has quit IRC13:58
*** sballe has joined #openstack-lbaas13:59
*** localloop127 has joined #openstack-lbaas14:06
*** ajmiller has joined #openstack-lbaas14:19
*** apuimedo has joined #openstack-lbaas14:39
*** clev-away is now known as clev14:44
*** numans has quit IRC14:49
*** mlavalle has joined #openstack-lbaas15:00
*** clev is now known as clev-away15:15
*** clev-away is now known as clev15:19
*** ganeshna has joined #openstack-lbaas15:22
xgermanblogan I think you will also need to check the templates since they have :0 hardcoded as well15:25
xgermanunless that doesn’t matter15:26
*** KunalGandhi has joined #openstack-lbaas15:51
*** ganeshna has quit IRC15:53
*** KunalGan_ has joined #openstack-lbaas15:54
*** KunalGandhi has quit IRC15:55
*** minwang2 has joined #openstack-lbaas15:58
*** vivek-ebay has joined #openstack-lbaas16:01
bloganxgerman: you do need the :0 for the plug vip bc the vip interface will have 2 IPs16:01
xgermanas I said I mimicked the ssh driver16:02
xgermanso I am not sure if it’s needed16:02
xgermanI just wanted to point out another area where it’s used16:03
*** vivek-ebay has quit IRC16:05
*** vivek-ebay has joined #openstack-lbaas16:06
*** KunalGan_ has quit IRC16:18
johnsomdengui In Active/Standby mode we are using VRRP and peer synchronization16:27
*** kiran-r has joined #openstack-lbaas16:35
*** kiran-r has quit IRC16:35
*** clev is now known as clev-away16:37
*** ajmiller has quit IRC16:39
*** ajmiller has joined #openstack-lbaas16:41
*** KunalGandhi has joined #openstack-lbaas16:45
*** KunalGandhi has quit IRC16:53
*** KunalGandhi has joined #openstack-lbaas16:55
*** ganeshna has joined #openstack-lbaas17:00
*** openstackgerrit_ has joined #openstack-lbaas17:02
*** ganeshna has quit IRC17:04
*** vivek-ebay has quit IRC17:16
*** madhu_ak has joined #openstack-lbaas17:22
*** clev-away is now known as clev17:45
*** vivek-ebay has joined #openstack-lbaas17:45
openstackgerritMerged openstack/octavia: Update Active/Standby blueprint
*** vivek-ebay has quit IRC17:46
*** vivek-ebay has joined #openstack-lbaas17:46
*** bharathm has joined #openstack-lbaas17:56
*** devlaps has joined #openstack-lbaas17:56
*** kiran-r has joined #openstack-lbaas18:21
*** kiran-r has quit IRC18:32
*** clev is now known as clev-away18:40
*** mgarza_ has joined #openstack-lbaas18:42
*** bradjones has quit IRC19:03
*** minwang2 has quit IRC19:04
*** bradjones has joined #openstack-lbaas19:10
*** bradjones has quit IRC19:10
*** bradjones has joined #openstack-lbaas19:10
*** clev-away is now known as clev19:14
dougwigfyi, i'm out all next week hunting. will have no cell service.19:22
xgermanyou know you will hurt yourself or the car19:24
xgermanbetter stay home and work :-)19:25
*** mgarza_ has quit IRC19:34
bloganjohnsom, xgerman: where do i need to put the reference to a review for the image to be built with that particular octavia review?19:38
xgermanone sec19:38
xgermanamphora-agent git /opt/amphora-agent
xgermanit’s usually best top use the same you do for the devstack19:39
blogani will19:39
bloganjsut couldnt find where that was19:39
xgermancool - fro development you can run, upload the image and adjust the reference in octavia.conf19:40
*** KunalGandhi has quit IRC19:43
*** localloop127 has quit IRC19:44
*** crc32 has joined #openstack-lbaas19:52
openstackgerritCarlos Garza proposed openstack/octavia: Adding amphora failover flows
openstackgerritCarlos Garza proposed openstack/octavia: health manager service
openstackgerritCarlos Garza proposed openstack/octavia: Implement UDP heartbeat sender and receiver
openstackgerritCarlos Garza proposed openstack/octavia: Implementing EventStreamer
*** openstackgerrit_ has quit IRC20:03
*** openstackgerrit_ has joined #openstack-lbaas20:04
openstackgerritPhillip Toohill proposed openstack/neutron-lbaas: Allow updating TLS refs
*** crc32|znc has quit IRC20:16
*** crc32|znc has joined #openstack-lbaas20:16
*** clev is now known as clev-away20:16
rm_workyeah how did that even happen20:16
rm_workno other users on the bouncer disconnected...20:17
rm_workdid you do something?20:17
crc32I think your box is owned.20:17
crc32I don't even connect with it. I just uses it as a pager20:18
crc32err connect to it20:18
rm_workweird <_<20:18
rm_workjust for giggles, checked out the box, no logins besides me, ever :P20:25
rm_workoh wait shit i'm on ETO today20:26
rm_work... damnit20:26
ptoohillget outta heere20:26
rm_worki was abruptly awoken by a failing CO1 alarm going off, and was so befuddled that i forgot until just now T_T20:27
rm_workptoohill: BTW new barbican devstack works20:27
rm_workptoohill: literally ONLY add the one line "enable_plugin"20:27
rm_workthat's it, not even an enable_service line20:27
rm_workenable_plugin barbican
ptoohilldo you have link for review?20:28
ptoohilldid that get merged already20:28
rm_workit merged20:28
rm_workeverything is good20:28
ptoohillOh sweet20:28
rm_worktheir gates are in the process of flipping over to it as wel20:28
crc32rm_work: uxtmp can be faked out after being root kitted.20:28
ptoohillIll fix that other patch then, this is a good thing!20:29
rm_worki also checked authlog, though that could be tampered20:29
crc32rm_work: yeah. The only way to be sure its clean after a suspected compromise you have to destroy and rebuild the box. :(20:30
rm_workomg something keeps locking apt on my devstack test boxes and killing the stack.sh20:30
rm_workcrc32: fortunately i don't suspect a compromise :P20:30
crc32see. Morr r00tkits20:30
rm_worklol these boxes have been alive for all of 5 minutes20:30
rm_workah it's REACH20:31
crc32rm_woprk: Thats just it. Most compromises happen when you don't suspect one.20:31
crc32do the right thing and rebuild that turkey. Its the responsible thing to do since its pretty much owned now.20:31
ptoohillrm_work:  when did this merge?20:32
rm_workptoohill: day or two ago? though I just fixed it early this morning20:32
crc32and when you rebuild it. Don't install the bouncer cause IRC may have been the attack vector used in the compromise.20:32
ptoohillI had some issue yesterday am with barbican, blew away env and it started working. Thanks again for working on this20:33
rm_workyeah, since i am not actively able to pick up a specific task, i've just been doing whatever i can to help out with getting everything else up and running20:35
rm_workhopefully we can get this stuff to actually all work reliably soon :P20:35
rm_worktesting a new devstack spin-up script for octavia20:36
rm_workand then might make an ansible playbook for it or something, where you pass a CR# and it builds20:36
openstackgerritBrandon Logan proposed openstack/octavia: Rest API plug network interface handling
rm_worklets see, what is ready for review20:37
*** clev-away is now known as clev20:38
rm_workptoohill: barbicanclient won't work as expected quite yet with new devstack, because of at least one other bug20:39
rm_work(on the CLI)20:39
rm_workpossible to work around by passing os-identity-api-version explicitly20:40
ptoohillmaybe i still had old files/set up then. Was having issues yesterday though.20:42
*** minwang2 has joined #openstack-lbaas20:42
*** nmagnezi has joined #openstack-lbaas20:46
*** openstackgerrit_ has quit IRC20:47
*** openstackgerrit_ has joined #openstack-lbaas20:47
*** openstackgerrit_ has quit IRC20:48
*** openstackgerrit_ has joined #openstack-lbaas20:49
*** openstackgerrit_ has quit IRC20:53
*** clev is now known as clev-away20:55
*** ajmiller has quit IRC21:04
*** ajmiller has joined #openstack-lbaas21:05
*** madhu_ak has quit IRC21:06
*** HenryG has quit IRC21:07
*** HenryG has joined #openstack-lbaas21:11
*** whydidyoustealmy has quit IRC21:11
*** minwang2 has quit IRC21:11
*** minwang2 has joined #openstack-lbaas21:11
*** madhu_ak has joined #openstack-lbaas21:14
*** TrevorV has joined #openstack-lbaas21:18
*** whydidyoustealmy has joined #openstack-lbaas21:18
*** Varun_Lodaya has joined #openstack-lbaas21:21
rm_workcrap, we suffer from the same issue as barbican21:22
rm_workoctavia dsvm is broken because:21:26
openstackgerritAdam Harwell proposed openstack/octavia: Switch to OSC for devstack (keystone client deprecated)
rm_workerr crap that is wrong too21:29
johnsomWow, something is using the openstack client...21:29
rm_workjohnsom: technically devstack moved to ONLY openstackclient21:29
rm_workjohnsom: `keystone` doesn't work anymore21:29
rm_workit is v2.0 only and devstack completely ditched 2.0 now21:30
johnsomWell, it still need the neutron client21:30
rm_workso i guess "openstackclient only, for keystone"21:30
johnsomYeah, that was my comment.21:30
xgermanopenstackclient is the new keystone client?21:30
rm_workerr, so i am not sure how to get a token with it <_<21:31
*** nmagnezi has quit IRC21:32
TrevorVrm_work, I thought it was just "openstack token-get" and it routed the right way or whatever21:35
rm_workah found it i think -- yep21:35
rm_workopenstack token issue21:35
openstackgerritAdam Harwell proposed openstack/octavia: Switch to OSC for devstack (keystone client deprecated)
TrevorVrm_work, so with this patch my stack should complete?21:36
rm_workTrevorV: i hope so, yes21:37
rm_worki am testing that right now21:37
TrevorVrm_work, why do you not have ",barbican" to enabled services?21:40
rm_workTrevorV: the enable_plugin line does it automatically21:40
openstackgerritMerged openstack/octavia: Set the Amphora REST driver as devstack default
rm_worki was going to make octavia do that as well I think21:40
TrevorVSo then why do we have ,octavia,o-api,o-cw?21:40
rm_work^^ because it just wasn't done21:40
TrevorVOh.  So what about neutron-lbaas then?21:41
rm_workshould be easy to fix it so it does that automatically too :P21:41
rm_workthough with those it might be due to the customizability21:41
rm_workbarbican has only one real configuration21:41
rm_workoctavia and neutron-lbaas might need to be customizable21:41
TrevorVYou're customizable.  Scrub.21:42
rm_workyes, I am21:42
rm_workthough for volume I only come in Loud21:42
TrevorVLoud and Sleeping21:48
TrevorVrm_work, I think that patch didn't fix anything21:48
TrevorVAt least not when I imported to octavia21:49
TrevorVWell, I added the ref to my localrc21:49
rm_worki am testing now21:49
rm_workit might have fixed SOMETHING, just not your issue21:49
rm_worki mean, it definitely fixes ONE issue21:49
TrevorVSure, but I'm still getting keystone failure21:51
TrevorVWhich I assume "shouldn't exist"21:51
*** KunalGandhi has joined #openstack-lbaas21:51
TrevorVwell RIP.  Its a mysql failure at this point.21:53
TrevorVrm_work, "got an unexpected keyword argument 'max'"21:53
TrevorVGotta love it.21:53
*** nmagnezi has joined #openstack-lbaas21:55
*** KunalGandhi has quit IRC21:55
*** KunalGandhi has joined #openstack-lbaas21:56
TrevorVrm_work, I'm just doing a checkout on the same devstack blogan is running so I can avoid this issue altogether right now21:57
blogangrr this failover is going to require a bit more work than anticipated, hopefully i can get it done this weekend, more hoepfully tonight21:59
*** nmagnezi has quit IRC21:59
bloganjohnsom, xgerman: there's a couple of ways to solve this, but i think under the circumstances, the best right way is to change the rest driver (and we'll probably have to do it for ssh drivers) to do the plug_vip and plug_netowrk methods to take a mac address so it knows exactly which interface is to be plugged22:06
TrevorVrm_work, same "max" error as I described from head-11 devstack... wtc22:07
xgermanok, I was worried we would need that eventually22:07
bloganor post_vip_plug and post_network_plug (its plug_vip and plug_network in the amphora agent)22:07
xgermanyeah, mac is good22:07
bloganxgerman: yeah its the right way to do it and i could hack somethign together to make the failover work but it would be almost just as much work, but right now on failover the code expects one and only one interface to be down to do the plug vip or plug network.  Since the failover just attached all the ports from the previous amphora on to the new one, there are now N down interfaces.22:08
xgermanyeah, we wanted specifying ports for a while — it always worried me that we were bringing up random ports without knowing which one was the right one22:09
bloganxgerman: and there's no way for the agent to knwo which one is supposed to be the vip port and which are the member ports22:09
blogansorry interfaces22:09
rm_workTrevorV: using my script it SEEMS to have started everything properly22:09
xgermanabdelwas added some method to get a list of all inetrfaces22:10
TrevorV"seems" he says22:10
rm_workwell, not sure how to test22:10
TrevorVYour script is run on a fresh vm.  Can't do that on my tower...22:10
rm_workbut all the services are running22:10
bloganxgerman: okay ill get on that, already have but wanted to let you know just in case you thought i might be doing something that coudl be done easier22:10
TrevorVOr on my macbook22:10
rm_workTrevorV: ah yeah you may be f'd22:10
TrevorVThat's why I hate you linking your script to me every time you do stuffs there.22:10
rm_workit was a major change recently22:10
xgermannah, it shouldn’t be too tough… I think the python lib we use reports macs so it should be quite straight forward...22:11
rm_worki mean, you may have to tear everything down locally22:11
TrevorVWell major changes should have been more scrutinized.22:11
rm_workand start over22:11
TrevorVI'm working on that lulz22:11
*** nmagnezi has joined #openstack-lbaas22:11
bloganxgerman: its meticulous bc ill have to change the amphora driver interface, some flows, some tasks, the amphora agent, the ssh driver, etc22:11
rm_workblogan: want to merge so we can have a working build22:12
xgermanmmh, yeah, it’s probably a solid 4 hours'22:12
bloganrm_work: if you merge so rest driver works with all kinds of vip and member subnet permutations22:12
rm_workbut FYI, if we HAD a gate dsvm test, it'd be broken right now<_<22:13
blogantoken issue?22:13
bloganthats a terrible name22:13
blogantoken-get was just fine damnit22:13
rm_workso removing :0 isn't breaking?22:15
xgermanwe don;t think so22:15
rm_work.. kk22:15
rm_workwill trust you22:15
bloganrm_work: no bc when a network is plugged its bc a new network got plugged into the amp22:16
xgermanI had johnsom check :-)22:16
bloganrm_work: which always means a new interface, not a new address on an existing interface22:16
rm_worktrade successful22:16
bloganyall can totally test it out with this
bloganim going to pimp it out bc its saved me a lot of time22:17
bloganexcept validate_connect function doesn't work with the multiple members, but whatever, you can do a curl on the vip22:17
xgermanwe probably need to have a PTL meeting next week to run through the bugs and triage them blogan22:19
bloganxgerman: okay22:19
openstackgerritMerged openstack/octavia: Switch to OSC for devstack (keystone client deprecated)
xgermanI am ignoring them until we have actually stuff which works :-)22:19
bloganxgerman: almost there!22:19
bloganfailover has been more complicated than initially thought, which is usually how it goes22:20
bloganadn this is just the very basic happy path too22:20
xgermanalso that sudo stuff was bad.22:20
xgermanudp sender22:21
bloganoh you mean it was tougher than initially thought too?22:21
bloganyeah, these are humongous features22:21
bloganand tough to troubleshoot22:22
bloganalright i gotta head home, ill be on later im sure22:22
bloganand throughout the weekend22:22
blogantalk to yall later22:22
openstackgerritMerged openstack/octavia: Rest API plug network interface handling
xgermani will be around all weekend to +222:23
TrevorVrm_work, you in lbaas-dev?22:28
*** nmagnezi has quit IRC22:31
TrevorVrm_work,  I cleaned up EVERYTHING...22:39
TrevorVAll openstack related packages.22:39
TrevorVThat took a day22:39
*** TrevorV has quit IRC22:50
*** vivek-ebay has quit IRC23:07
*** vivek-ebay has joined #openstack-lbaas23:07
*** Jet1 has joined #openstack-lbaas23:37
*** Jet has joined #openstack-lbaas23:42
*** ajmiller has quit IRC23:56

Generated by 2.14.0 by Marius Gedminas - find it at!