Thursday, 2017-09-28

openstackgerritAdam Harwell proposed openstack/octavia master: Correctly unlock amphora health on failover revert  https://review.openstack.org/50801700:08
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561200:10
rm_workI wish I knew why calling LOG.info("my message") inside an except block ends up ALSO logging the exception text at the end of my line00:11
johnsomThose revert tasks should never fail so need to catch everything....00:11
johnsomYeah, that is odd, I have noticed that too00:12
rm_workah k00:12
rm_workwait00:13
rm_worki totally did catch everything00:13
rm_workdidn't i?00:13
rm_workoh you literally mean00:13
rm_workException00:13
johnsomJust another except block that everything, not just the DB exceptions00:13
rm_workeh i'll just catch everything in the first block00:16
rm_workthere00:16
openstackgerritAdam Harwell proposed openstack/octavia master: Correctly unlock amphora health on failover revert  https://review.openstack.org/50801700:17
johnsomLGTM00:17
rm_workso basically this last week + this next week I'm stress-testing failovers in ACTIVE_STANDBY in a real env00:18
johnsomNice00:18
rm_worki have a good tempest test for it now00:18
rm_workbut i can't upload it anywhere because omfg the chain is so out of date00:18
rm_workand i can't rebase00:18
rm_workwaiting for that to get fixed00:19
johnsomAre you going to post a review on the upstream tempest stuff?  I know you had concerns00:19
rm_workand then ... ugh, such rebasing00:19
rm_worki'm waiting for kong to finish with the rebasing00:19
rm_workso i can actually see wtf00:19
rm_worksince what really matters is what it looks like with all of the objects and the "full" test00:19
johnsomYeah, last time it tried it, it just didn't work00:19
rm_worki'm using an old chain00:19
rm_worki wish there was a good way for me to push it up without interfering00:20
rm_workit won't let me JUST push my change00:20
johnsomWell, I burned another day on that 404 thing, but at least I could  run it to ground....00:20
rm_workI honestly might just push my stuff up and overwrite all those CRs00:22
rm_workand then go back through and undo my breakage00:22
rm_workon purpose00:22
rm_workbecause i REALLY want to push this up00:22
johnsomIs what is posted in such bad shape?00:22
rm_workfor MY patch yes00:22
johnsomwell...00:23
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/50663800:26
johnsom^^^ rm_work there are a few of those around waiting for another +200:26
johnsomhttps://review.openstack.org/#/q/openstack/neutron-lbaas-dashboard+status:open,n,z00:27
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561200:30
rm_workjohnsom: ummm so should i just go through and +A the dashboard stuff00:31
rm_worki don't really ... ... JS00:31
rm_worki mean rather, I can JS00:31
rm_workbut whatever the shit this framework is00:32
johnsomWell, either you or German need to look at it.00:32
johnsomI tested all of the octavia-dashboard stuff00:32
johnsomThose lbaas-dashboard things are pretty small00:32
xgerman_Ok. You got a list?00:33
xgerman_Might look once I am back at the hotel later...00:33
johnsomI think I need to make another filter...00:33
johnsomThese are the octavia https://review.openstack.org/#/q/(project:openstack/octavia+OR+project:openstack/octavia-dashboard+OR+project:openstack/python-octaviaclient+OR+project:openstack/octavia-tempest-plugin)+AND+status:open+AND+NOT+label:Code-Review%253C0+AND+NOT+label:Verified%253C%253D0+AND+NOT+label:Workflow%253C000:34
johnsomfew more in lbaas and lbaas-dashboard00:35
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/50663800:41
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Modify the '.gitignore' file  https://review.openstack.org/49514700:43
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Remove old unused code.  https://review.openstack.org/49800300:43
*** bcafarel has quit IRC00:44
johnsomLooks like zuul v3 starts rolling out tomorrow morning00:47
*** bcafarel has joined #openstack-lbaas00:51
*** bcafarel has quit IRC01:51
*** bcafarel has joined #openstack-lbaas01:58
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/50663802:08
*** yamamoto has joined #openstack-lbaas02:22
*** yamamoto has quit IRC02:22
*** yamamoto has joined #openstack-lbaas02:22
openstackgerritMerged openstack/octavia-dashboard master: Allow updating expected codes on health monitor  https://review.openstack.org/50769903:44
*** links has joined #openstack-lbaas03:44
*** tongl has joined #openstack-lbaas04:46
*** tongl has quit IRC04:58
*** aojea has joined #openstack-lbaas05:33
*** leitan has quit IRC05:38
*** leitan has joined #openstack-lbaas05:39
*** eezhova has joined #openstack-lbaas05:41
*** gcheresh_ has joined #openstack-lbaas05:43
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/50663805:58
*** pcaruana has joined #openstack-lbaas06:19
*** eezhova has quit IRC06:29
*** Alex_Staf has quit IRC06:30
openstackgerritWei Li proposed openstack/octavia master: change db relationship to manytomany for lb and amphora  https://review.openstack.org/50806006:32
*** rcernin has joined #openstack-lbaas07:00
*** armax has joined #openstack-lbaas07:08
*** reedip has quit IRC07:14
*** armax has quit IRC07:14
*** reedip has joined #openstack-lbaas07:28
*** armax has joined #openstack-lbaas07:36
*** armax has quit IRC07:42
*** yamamoto has quit IRC07:56
*** yamamoto has joined #openstack-lbaas08:03
*** Alex_Staf has joined #openstack-lbaas08:05
*** eezhova has joined #openstack-lbaas08:20
*** yamamoto has quit IRC08:25
*** yamamoto has joined #openstack-lbaas08:30
*** yamamoto_ has joined #openstack-lbaas08:32
*** yamamoto_ has quit IRC08:34
openstackgerritMerged openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/50663808:35
*** yamamoto has quit IRC08:35
rm_workjohnsom: ah the answer to why we get random exception info on log messages: https://docs.python.org/3/library/logging.html#logging.Logger.debug08:52
rm_work"If exc_info does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) or an exception instance is provided, it is used; otherwise, sys.exc_info() is called to get the exception information."08:52
*** yamamoto has joined #openstack-lbaas09:14
*** yamamoto has quit IRC09:17
*** salmankhan has joined #openstack-lbaas09:20
openstackgerritWei Li proposed openstack/octavia master: change db relationship to manytomany for lb and amphora  https://review.openstack.org/50806009:36
*** armax has joined #openstack-lbaas09:38
*** salmankhan has quit IRC09:39
*** salmankhan has joined #openstack-lbaas09:40
*** armax has quit IRC09:42
*** ltomasbo has quit IRC09:59
*** yamamoto has joined #openstack-lbaas10:03
*** ltomasbo has joined #openstack-lbaas10:04
*** yamamoto has quit IRC10:05
*** reedip has quit IRC10:12
*** reedip has joined #openstack-lbaas10:26
*** yamamoto has joined #openstack-lbaas10:29
*** yamamoto has quit IRC10:31
*** yamamoto has joined #openstack-lbaas10:37
*** yamamoto has quit IRC10:38
*** yamamoto has joined #openstack-lbaas10:40
*** yamamoto has quit IRC10:43
*** yamamoto has joined #openstack-lbaas10:46
*** yamamoto has quit IRC10:49
*** belharar has joined #openstack-lbaas10:54
*** ipsecguy_ is now known as ipsecguy11:28
*** yamamoto has joined #openstack-lbaas11:28
*** yamamoto has quit IRC12:03
*** reedip has quit IRC12:07
openstackgerritWei Li proposed openstack/octavia master: change db relationship to manytomany for lb and amphora  https://review.openstack.org/50806012:16
*** reedip has joined #openstack-lbaas12:21
*** pcaruana has quit IRC12:31
*** pcaruana has joined #openstack-lbaas13:00
*** atoth has joined #openstack-lbaas13:02
*** yamamoto has joined #openstack-lbaas13:03
*** ianychoi_ is now known as ianychoi13:06
*** yamamoto has quit IRC13:09
*** chlong has quit IRC13:14
*** catintheroof has joined #openstack-lbaas13:15
*** atoth has quit IRC13:19
*** leitan has quit IRC13:21
*** atoth has joined #openstack-lbaas13:34
*** leitan has joined #openstack-lbaas13:37
*** links has quit IRC14:02
*** rtjure has joined #openstack-lbaas14:35
*** chlong has joined #openstack-lbaas14:39
-openstackstatus- NOTICE: The infra team is now taking Zuul v2 offline and bringing Zuul v3 online. Please see https://docs.openstack.org/infra/manual/zuulv3.html for more information, and ask us in #openstack-infra if you have any questions.14:42
*** gcheresh_ has quit IRC14:46
*** eezhova has quit IRC15:00
*** dayou1 has joined #openstack-lbaas15:15
*** atoth has quit IRC15:15
*** atoth has joined #openstack-lbaas15:16
*** eezhova has joined #openstack-lbaas15:22
*** Alex_Staf has quit IRC15:22
*** gcheresh_ has joined #openstack-lbaas15:34
*** gcheresh_ has quit IRC15:39
*** sanfern has joined #openstack-lbaas15:40
*** atoth has quit IRC15:44
*** rcernin has quit IRC15:46
*** gcheresh_ has joined #openstack-lbaas15:49
*** belharar has quit IRC15:58
*** gcheresh_ has quit IRC16:01
*** armax has joined #openstack-lbaas16:01
*** tongl has joined #openstack-lbaas16:02
*** tongl has quit IRC16:03
sanfernGates are still failing - http://logs.openstack.org/64/490164/12/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/c19913d/console.html16:07
*** kbyrne has quit IRC16:10
*** kbyrne has joined #openstack-lbaas16:12
*** armax has quit IRC16:17
johnsomsanfern It looks like you have a bug: http://logs.openstack.org/64/490164/12/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/c19913d/logs/devstacklog.txt.gz#_2017-09-27_08_50_50_59216:29
johnsomI think you need to do a mkdir -p or equivalent to make sure the directory exists16:30
*** salmankhan has quit IRC16:37
*** Alex_Staf has joined #openstack-lbaas16:40
*** numans has quit IRC16:46
*** numans has joined #openstack-lbaas16:49
*** salmankhan has joined #openstack-lbaas16:52
openstackgerritSanthosh Fernandes proposed openstack/octavia master: [WIP] Adding exabgp-speaker element to amphora image  https://review.openstack.org/49016416:58
sanfernthanks johnsom16:58
*** Alex_Staf has quit IRC17:03
*** salmankhan has quit IRC17:07
*** eezhova has quit IRC17:15
johnsomNP17:17
*** tomtomtom has quit IRC17:23
*** SumitNaiksatam has joined #openstack-lbaas17:24
*** numans has quit IRC17:26
*** gcheresh_ has joined #openstack-lbaas17:30
*** numans has joined #openstack-lbaas17:31
*** sshank has joined #openstack-lbaas17:37
*** gcheresh_ has quit IRC17:41
dayou1Hi Johnsom, I tried to reproduce the systemctl stop devstack@o-hm fault in https://review.openstack.org/#/c/462748/. Wasn't able to do it, are there specific step18:09
dayou1s to reproduce it?18:10
*** aojea has quit IRC18:12
*** aojea has joined #openstack-lbaas18:13
johnsomdayou1 So, start it up with that patch applied normal.  You should see three processes, then do systemctl stop devstack@o-hm, look again and two of the three will still be running but now parent of 118:16
*** aojea has quit IRC18:17
johnsomrestarts of the process will fail with socket in use errors18:18
johnsomThis does not happen without the patch18:18
*** belharar has joined #openstack-lbaas18:18
*** aojea has joined #openstack-lbaas18:18
dayou1I tried apply the patch and start, then without doing anything else stop, all process are gone.18:22
dayou1Also tried to trigger a failover and seems not affecting the stop of it18:23
dayou1# RECLONE=yes18:24
dayou1# PIP_UPGRADE=True18:24
dayou1I have RECLONE and PIP_UPGRADE to keep code to the latest18:24
dayou1Linux cloudhost 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux18:25
johnsomCheck your git log and maybe even the installed code, it could be picking up the wrong code18:25
johnsomReclone will pull master18:25
dayou1stack@cloudhost:~/octavia$ git log18:25
dayou1commit a24e93d4a737959a1f70204b410ab031e2a465c718:25
dayou1Author: Joshua Harlow <jxharlow@godaddy.com>18:25
dayou1Date:   Thu May 4 14:42:25 2017 -070018:26
dayou1    Handle stopping the health manager and ejecting out of current health failovers18:26
dayou1    Also handles the `failover_amphora` not working out or being18:26
dayou1    cancelled correctly and logging the statistics of those occurences18:26
dayou1    instead of incorrectly assuming everything always works out.18:26
dayou1    Depends-On: I3d86482a2999197a60a81d42afc5ef7a6e71e31318:26
dayou1    Change-Id: I240c328bb9d0902cb2a89a2808519264764aa6b018:26
dayou1commit 1421d451eec66e140ade1b30a16a0cb8ee29baec18:26
dayou1Merge: 5c5ef89 4f7c54118:26
dayou1Author: Jenkins <jenkins@review.openstack.org>18:26
dayou1Date:   Wed Sep 20 21:47:10 2017 +000018:26
dayou1    Merge "Use flake8-import-order plugin"18:26
johnsomSomething is of then, because it failed for me.  Maybe rm_work can try18:26
*** longkb_ has quit IRC18:29
*** sshank has quit IRC18:35
*** sshank has joined #openstack-lbaas18:36
*** aojea has quit IRC18:37
*** aojea has joined #openstack-lbaas18:38
*** atoth has joined #openstack-lbaas18:38
dayou1Yes, maybe, I just tried this patch in our lab with the newton version, and it seems just reload with the patch and start and stop, no thing gets wrong there.18:41
*** aojea has quit IRC18:42
*** sshank has quit IRC18:43
*** dayou has joined #openstack-lbaas18:49
*** salmankhan has joined #openstack-lbaas18:50
*** sanfern has quit IRC18:52
*** dayou has quit IRC18:53
*** dayou_ has joined #openstack-lbaas18:53
*** dayou1 has quit IRC18:54
*** belharar has quit IRC18:55
*** dayou has joined #openstack-lbaas18:55
*** salmankhan has quit IRC18:55
johnsomSep 28 11:56:06 devstackpy27-2 octavia-health-manager[111747]: error: [Errno 98] Address already in use18:56
johnsomYeah, I get the same result on this stack host too18:56
johnsomsystemctl restart devstack@o-hm18:56
johnsomstack    111665      1  3 11:55 ?        00:00:03 /usr/bin/python /usr/local/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf18:57
johnsomstack    111666      1  0 11:55 ?        00:00:00 /usr/bin/python /usr/local/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf18:57
johnsomstack    111760      1  0 11:56 ?        00:00:00 /usr/bin/python /usr/local/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf18:57
johnsomI do have an LB booted, I don't know if that makes a difference, but I suspect not18:57
openstackgerritBar RH proposed openstack/octavia master: [WIP] Assign bind_host ip address per amphora  https://review.openstack.org/50515818:59
*** aojea has joined #openstack-lbaas19:17
*** Alex_Staf has joined #openstack-lbaas19:28
*** pcaruana has quit IRC19:31
dayouAre you using python3 or python2?19:36
dayouIt seems the address is binded by the hm listener proc, and we are chaning the hm health check proc, not sure how they affecting each other.19:37
dayouchanging19:38
*** eezhova has joined #openstack-lbaas19:41
johnsompython27 on this vm19:52
dayouCan I gain access to the stack host running this? Would like to take a look at what's going on there19:55
rm_workhard for me to stack currently :/19:56
rm_workumm, i could try that patch in my staging env maybe19:56
*** chlong has quit IRC20:03
*** catintheroof has quit IRC20:09
*** leitan has quit IRC20:26
*** sshank has joined #openstack-lbaas20:28
*** Alex_Staf has quit IRC20:31
*** sshank has quit IRC20:33
*** sshank has joined #openstack-lbaas20:33
*** eezhova has quit IRC20:35
*** belharar has joined #openstack-lbaas20:38
*** gcheresh_ has joined #openstack-lbaas20:39
*** rcernin has joined #openstack-lbaas20:42
*** gcheresh_ has quit IRC20:50
rm_workkong: how goes the rebases21:15
kongrm_work: i am working on the basic ops tempest case, a lot of debugging and testing yesterday.21:19
kongbut today, i just found there is something changed in tempest/test.py21:19
kongthy removed some code that my code is relying on21:19
kongso need some changes21:20
kongbefore i submit for review21:20
johnsomThese are the only modules we can import from tempest: https://docs.openstack.org/tempest/latest/plugin.html#stable-tempest-apis-plugins-may-use21:20
johnsomFYi...21:21
kongrm_work: actually, i almost refactored all the code Jude was working on21:21
rm_workok21:21
kongso, it's not a rebase, it's re-write21:21
rm_workk21:21
rm_workdon't worry about the WIP:Failover at the end21:21
rm_workI have completely rewritten that21:21
rm_workand i am just waiting to rebase it onto your stuff21:22
kongjohnsom: thanks, i know that list21:22
*** sshank has quit IRC21:32
*** sshank has joined #openstack-lbaas21:35
*** leitan has joined #openstack-lbaas21:36
*** leitan has quit IRC21:41
*** chlong has joined #openstack-lbaas21:41
*** aojea has quit IRC21:45
johnsomkong We have a bunch of stuff waiting on getting that baseline tempest plugin going.  Do you have an ETA when code will be pushed up and ready for review?21:45
rm_workkong: so is the patch you have up at https://review.openstack.org/#/c/486775/ close to final? or did you do further rewrites to that?21:46
kongjohnsom: don't worry, i am working full time on that21:46
kongrm_work: yeah, that's for loadbalancer operations21:46
rm_workok21:47
konghttps://review.openstack.org/#/c/492311/921:47
kongthis one is for listener21:47
rm_workright21:47
rm_workjust pool/member/hm left21:47
rm_workand then i can start to rebase my stuff21:47
johnsomOk, so I can review the load balancer one now?21:47
rm_workand see how it works out21:47
*** armax has joined #openstack-lbaas21:47
kongsure21:47
kongplease21:47
*** dulek has quit IRC21:53
kongrm_work, johnsom, there is one thing you need pay attention when running tempest, please give 'OS_TEST_TIMEOUT' config a bigggger value21:56
rm_workyeah21:56
rm_workmine is set very high21:56
rm_workmy tests run for hours ;P21:56
kongbrilliant21:56
*** belharar has quit IRC21:59
*** sshank has quit IRC22:14
*** dayou_ has quit IRC22:19
*** sshank has joined #openstack-lbaas22:20
*** catintheroof has joined #openstack-lbaas22:24
johnsomSo I'm not sure jobs are firing, at least rechecks are not.  The new zuulv3 dashboard is here: http://zuulv3.openstack.org/22:51
*** ltomasbo has quit IRC23:44
*** ltomasbo has joined #openstack-lbaas23:45
*** sshank has quit IRC23:50
*** SumitNaiksatam has quit IRC23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!