openstackgerrit | Adam Harwell proposed openstack/octavia master: Correctly unlock amphora health on failover revert https://review.openstack.org/508017 | 00:08 |
---|---|---|
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 00:10 |
rm_work | I wish I knew why calling LOG.info("my message") inside an except block ends up ALSO logging the exception text at the end of my line | 00:11 |
johnsom | Those revert tasks should never fail so need to catch everything.... | 00:11 |
johnsom | Yeah, that is odd, I have noticed that too | 00:12 |
rm_work | ah k | 00:12 |
rm_work | wait | 00:13 |
rm_work | i totally did catch everything | 00:13 |
rm_work | didn't i? | 00:13 |
rm_work | oh you literally mean | 00:13 |
rm_work | Exception | 00:13 |
johnsom | Just another except block that everything, not just the DB exceptions | 00:13 |
rm_work | eh i'll just catch everything in the first block | 00:16 |
rm_work | there | 00:16 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: Correctly unlock amphora health on failover revert https://review.openstack.org/508017 | 00:17 |
johnsom | LGTM | 00:17 |
rm_work | so basically this last week + this next week I'm stress-testing failovers in ACTIVE_STANDBY in a real env | 00:18 |
johnsom | Nice | 00:18 |
rm_work | i have a good tempest test for it now | 00:18 |
rm_work | but i can't upload it anywhere because omfg the chain is so out of date | 00:18 |
rm_work | and i can't rebase | 00:18 |
rm_work | waiting for that to get fixed | 00:19 |
johnsom | Are you going to post a review on the upstream tempest stuff? I know you had concerns | 00:19 |
rm_work | and then ... ugh, such rebasing | 00:19 |
rm_work | i'm waiting for kong to finish with the rebasing | 00:19 |
rm_work | so i can actually see wtf | 00:19 |
rm_work | since what really matters is what it looks like with all of the objects and the "full" test | 00:19 |
johnsom | Yeah, last time it tried it, it just didn't work | 00:19 |
rm_work | i'm using an old chain | 00:19 |
rm_work | i wish there was a good way for me to push it up without interfering | 00:20 |
rm_work | it won't let me JUST push my change | 00:20 |
johnsom | Well, I burned another day on that 404 thing, but at least I could run it to ground.... | 00:20 |
rm_work | I honestly might just push my stuff up and overwrite all those CRs | 00:22 |
rm_work | and then go back through and undo my breakage | 00:22 |
rm_work | on purpose | 00:22 |
rm_work | because i REALLY want to push this up | 00:22 |
johnsom | Is what is posted in such bad shape? | 00:22 |
rm_work | for MY patch yes | 00:22 |
johnsom | well... | 00:23 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/506638 | 00:26 |
johnsom | ^^^ rm_work there are a few of those around waiting for another +2 | 00:26 |
johnsom | https://review.openstack.org/#/q/openstack/neutron-lbaas-dashboard+status:open,n,z | 00:27 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 00:30 |
rm_work | johnsom: ummm so should i just go through and +A the dashboard stuff | 00:31 |
rm_work | i don't really ... ... JS | 00:31 |
rm_work | i mean rather, I can JS | 00:31 |
rm_work | but whatever the shit this framework is | 00:32 |
johnsom | Well, either you or German need to look at it. | 00:32 |
johnsom | I tested all of the octavia-dashboard stuff | 00:32 |
johnsom | Those lbaas-dashboard things are pretty small | 00:32 |
xgerman_ | Ok. You got a list? | 00:33 |
xgerman_ | Might look once I am back at the hotel later... | 00:33 |
johnsom | I think I need to make another filter... | 00:33 |
johnsom | These are the octavia https://review.openstack.org/#/q/(project:openstack/octavia+OR+project:openstack/octavia-dashboard+OR+project:openstack/python-octaviaclient+OR+project:openstack/octavia-tempest-plugin)+AND+status:open+AND+NOT+label:Code-Review%253C0+AND+NOT+label:Verified%253C%253D0+AND+NOT+label:Workflow%253C0 | 00:34 |
johnsom | few more in lbaas and lbaas-dashboard | 00:35 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/506638 | 00:41 |
openstackgerrit | Merged openstack/neutron-lbaas-dashboard master: Modify the '.gitignore' file https://review.openstack.org/495147 | 00:43 |
openstackgerrit | Merged openstack/neutron-lbaas-dashboard master: Remove old unused code. https://review.openstack.org/498003 | 00:43 |
*** bcafarel has quit IRC | 00:44 | |
johnsom | Looks like zuul v3 starts rolling out tomorrow morning | 00:47 |
*** bcafarel has joined #openstack-lbaas | 00:51 | |
*** bcafarel has quit IRC | 01:51 | |
*** bcafarel has joined #openstack-lbaas | 01:58 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/506638 | 02:08 |
*** yamamoto has joined #openstack-lbaas | 02:22 | |
*** yamamoto has quit IRC | 02:22 | |
*** yamamoto has joined #openstack-lbaas | 02:22 | |
openstackgerrit | Merged openstack/octavia-dashboard master: Allow updating expected codes on health monitor https://review.openstack.org/507699 | 03:44 |
*** links has joined #openstack-lbaas | 03:44 | |
*** tongl has joined #openstack-lbaas | 04:46 | |
*** tongl has quit IRC | 04:58 | |
*** aojea has joined #openstack-lbaas | 05:33 | |
*** leitan has quit IRC | 05:38 | |
*** leitan has joined #openstack-lbaas | 05:39 | |
*** eezhova has joined #openstack-lbaas | 05:41 | |
*** gcheresh_ has joined #openstack-lbaas | 05:43 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/506638 | 05:58 |
*** pcaruana has joined #openstack-lbaas | 06:19 | |
*** eezhova has quit IRC | 06:29 | |
*** Alex_Staf has quit IRC | 06:30 | |
openstackgerrit | Wei Li proposed openstack/octavia master: change db relationship to manytomany for lb and amphora https://review.openstack.org/508060 | 06:32 |
*** rcernin has joined #openstack-lbaas | 07:00 | |
*** armax has joined #openstack-lbaas | 07:08 | |
*** reedip has quit IRC | 07:14 | |
*** armax has quit IRC | 07:14 | |
*** reedip has joined #openstack-lbaas | 07:28 | |
*** armax has joined #openstack-lbaas | 07:36 | |
*** armax has quit IRC | 07:42 | |
*** yamamoto has quit IRC | 07:56 | |
*** yamamoto has joined #openstack-lbaas | 08:03 | |
*** Alex_Staf has joined #openstack-lbaas | 08:05 | |
*** eezhova has joined #openstack-lbaas | 08:20 | |
*** yamamoto has quit IRC | 08:25 | |
*** yamamoto has joined #openstack-lbaas | 08:30 | |
*** yamamoto_ has joined #openstack-lbaas | 08:32 | |
*** yamamoto_ has quit IRC | 08:34 | |
openstackgerrit | Merged openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/506638 | 08:35 |
*** yamamoto has quit IRC | 08:35 | |
rm_work | johnsom: ah the answer to why we get random exception info on log messages: https://docs.python.org/3/library/logging.html#logging.Logger.debug | 08:52 |
rm_work | "If exc_info does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) or an exception instance is provided, it is used; otherwise, sys.exc_info() is called to get the exception information." | 08:52 |
*** yamamoto has joined #openstack-lbaas | 09:14 | |
*** yamamoto has quit IRC | 09:17 | |
*** salmankhan has joined #openstack-lbaas | 09:20 | |
openstackgerrit | Wei Li proposed openstack/octavia master: change db relationship to manytomany for lb and amphora https://review.openstack.org/508060 | 09:36 |
*** armax has joined #openstack-lbaas | 09:38 | |
*** salmankhan has quit IRC | 09:39 | |
*** salmankhan has joined #openstack-lbaas | 09:40 | |
*** armax has quit IRC | 09:42 | |
*** ltomasbo has quit IRC | 09:59 | |
*** yamamoto has joined #openstack-lbaas | 10:03 | |
*** ltomasbo has joined #openstack-lbaas | 10:04 | |
*** yamamoto has quit IRC | 10:05 | |
*** reedip has quit IRC | 10:12 | |
*** reedip has joined #openstack-lbaas | 10:26 | |
*** yamamoto has joined #openstack-lbaas | 10:29 | |
*** yamamoto has quit IRC | 10:31 | |
*** yamamoto has joined #openstack-lbaas | 10:37 | |
*** yamamoto has quit IRC | 10:38 | |
*** yamamoto has joined #openstack-lbaas | 10:40 | |
*** yamamoto has quit IRC | 10:43 | |
*** yamamoto has joined #openstack-lbaas | 10:46 | |
*** yamamoto has quit IRC | 10:49 | |
*** belharar has joined #openstack-lbaas | 10:54 | |
*** ipsecguy_ is now known as ipsecguy | 11:28 | |
*** yamamoto has joined #openstack-lbaas | 11:28 | |
*** yamamoto has quit IRC | 12:03 | |
*** reedip has quit IRC | 12:07 | |
openstackgerrit | Wei Li proposed openstack/octavia master: change db relationship to manytomany for lb and amphora https://review.openstack.org/508060 | 12:16 |
*** reedip has joined #openstack-lbaas | 12:21 | |
*** pcaruana has quit IRC | 12:31 | |
*** pcaruana has joined #openstack-lbaas | 13:00 | |
*** atoth has joined #openstack-lbaas | 13:02 | |
*** yamamoto has joined #openstack-lbaas | 13:03 | |
*** ianychoi_ is now known as ianychoi | 13:06 | |
*** yamamoto has quit IRC | 13:09 | |
*** chlong has quit IRC | 13:14 | |
*** catintheroof has joined #openstack-lbaas | 13:15 | |
*** atoth has quit IRC | 13:19 | |
*** leitan has quit IRC | 13:21 | |
*** atoth has joined #openstack-lbaas | 13:34 | |
*** leitan has joined #openstack-lbaas | 13:37 | |
*** links has quit IRC | 14:02 | |
*** rtjure has joined #openstack-lbaas | 14:35 | |
*** chlong has joined #openstack-lbaas | 14:39 | |
-openstackstatus- NOTICE: The infra team is now taking Zuul v2 offline and bringing Zuul v3 online. Please see https://docs.openstack.org/infra/manual/zuulv3.html for more information, and ask us in #openstack-infra if you have any questions. | 14:42 | |
*** gcheresh_ has quit IRC | 14:46 | |
*** eezhova has quit IRC | 15:00 | |
*** dayou1 has joined #openstack-lbaas | 15:15 | |
*** atoth has quit IRC | 15:15 | |
*** atoth has joined #openstack-lbaas | 15:16 | |
*** eezhova has joined #openstack-lbaas | 15:22 | |
*** Alex_Staf has quit IRC | 15:22 | |
*** gcheresh_ has joined #openstack-lbaas | 15:34 | |
*** gcheresh_ has quit IRC | 15:39 | |
*** sanfern has joined #openstack-lbaas | 15:40 | |
*** atoth has quit IRC | 15:44 | |
*** rcernin has quit IRC | 15:46 | |
*** gcheresh_ has joined #openstack-lbaas | 15:49 | |
*** belharar has quit IRC | 15:58 | |
*** gcheresh_ has quit IRC | 16:01 | |
*** armax has joined #openstack-lbaas | 16:01 | |
*** tongl has joined #openstack-lbaas | 16:02 | |
*** tongl has quit IRC | 16:03 | |
sanfern | Gates are still failing - http://logs.openstack.org/64/490164/12/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/c19913d/console.html | 16:07 |
*** kbyrne has quit IRC | 16:10 | |
*** kbyrne has joined #openstack-lbaas | 16:12 | |
*** armax has quit IRC | 16:17 | |
johnsom | sanfern It looks like you have a bug: http://logs.openstack.org/64/490164/12/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/c19913d/logs/devstacklog.txt.gz#_2017-09-27_08_50_50_592 | 16:29 |
johnsom | I think you need to do a mkdir -p or equivalent to make sure the directory exists | 16:30 |
*** salmankhan has quit IRC | 16:37 | |
*** Alex_Staf has joined #openstack-lbaas | 16:40 | |
*** numans has quit IRC | 16:46 | |
*** numans has joined #openstack-lbaas | 16:49 | |
*** salmankhan has joined #openstack-lbaas | 16:52 | |
openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: [WIP] Adding exabgp-speaker element to amphora image https://review.openstack.org/490164 | 16:58 |
sanfern | thanks johnsom | 16:58 |
*** Alex_Staf has quit IRC | 17:03 | |
*** salmankhan has quit IRC | 17:07 | |
*** eezhova has quit IRC | 17:15 | |
johnsom | NP | 17:17 |
*** tomtomtom has quit IRC | 17:23 | |
*** SumitNaiksatam has joined #openstack-lbaas | 17:24 | |
*** numans has quit IRC | 17:26 | |
*** gcheresh_ has joined #openstack-lbaas | 17:30 | |
*** numans has joined #openstack-lbaas | 17:31 | |
*** sshank has joined #openstack-lbaas | 17:37 | |
*** gcheresh_ has quit IRC | 17:41 | |
dayou1 | Hi Johnsom, I tried to reproduce the systemctl stop devstack@o-hm fault in https://review.openstack.org/#/c/462748/. Wasn't able to do it, are there specific step | 18:09 |
dayou1 | s to reproduce it? | 18:10 |
*** aojea has quit IRC | 18:12 | |
*** aojea has joined #openstack-lbaas | 18:13 | |
johnsom | dayou1 So, start it up with that patch applied normal. You should see three processes, then do systemctl stop devstack@o-hm, look again and two of the three will still be running but now parent of 1 | 18:16 |
*** aojea has quit IRC | 18:17 | |
johnsom | restarts of the process will fail with socket in use errors | 18:18 |
johnsom | This does not happen without the patch | 18:18 |
*** belharar has joined #openstack-lbaas | 18:18 | |
*** aojea has joined #openstack-lbaas | 18:18 | |
dayou1 | I tried apply the patch and start, then without doing anything else stop, all process are gone. | 18:22 |
dayou1 | Also tried to trigger a failover and seems not affecting the stop of it | 18:23 |
dayou1 | # RECLONE=yes | 18:24 |
dayou1 | # PIP_UPGRADE=True | 18:24 |
dayou1 | I have RECLONE and PIP_UPGRADE to keep code to the latest | 18:24 |
dayou1 | Linux cloudhost 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | 18:25 |
johnsom | Check your git log and maybe even the installed code, it could be picking up the wrong code | 18:25 |
johnsom | Reclone will pull master | 18:25 |
dayou1 | stack@cloudhost:~/octavia$ git log | 18:25 |
dayou1 | commit a24e93d4a737959a1f70204b410ab031e2a465c7 | 18:25 |
dayou1 | Author: Joshua Harlow <jxharlow@godaddy.com> | 18:25 |
dayou1 | Date: Thu May 4 14:42:25 2017 -0700 | 18:26 |
dayou1 | Handle stopping the health manager and ejecting out of current health failovers | 18:26 |
dayou1 | Also handles the `failover_amphora` not working out or being | 18:26 |
dayou1 | cancelled correctly and logging the statistics of those occurences | 18:26 |
dayou1 | instead of incorrectly assuming everything always works out. | 18:26 |
dayou1 | Depends-On: I3d86482a2999197a60a81d42afc5ef7a6e71e313 | 18:26 |
dayou1 | Change-Id: I240c328bb9d0902cb2a89a2808519264764aa6b0 | 18:26 |
dayou1 | commit 1421d451eec66e140ade1b30a16a0cb8ee29baec | 18:26 |
dayou1 | Merge: 5c5ef89 4f7c541 | 18:26 |
dayou1 | Author: Jenkins <jenkins@review.openstack.org> | 18:26 |
dayou1 | Date: Wed Sep 20 21:47:10 2017 +0000 | 18:26 |
dayou1 | Merge "Use flake8-import-order plugin" | 18:26 |
johnsom | Something is of then, because it failed for me. Maybe rm_work can try | 18:26 |
*** longkb_ has quit IRC | 18:29 | |
*** sshank has quit IRC | 18:35 | |
*** sshank has joined #openstack-lbaas | 18:36 | |
*** aojea has quit IRC | 18:37 | |
*** aojea has joined #openstack-lbaas | 18:38 | |
*** atoth has joined #openstack-lbaas | 18:38 | |
dayou1 | Yes, maybe, I just tried this patch in our lab with the newton version, and it seems just reload with the patch and start and stop, no thing gets wrong there. | 18:41 |
*** aojea has quit IRC | 18:42 | |
*** sshank has quit IRC | 18:43 | |
*** dayou has joined #openstack-lbaas | 18:49 | |
*** salmankhan has joined #openstack-lbaas | 18:50 | |
*** sanfern has quit IRC | 18:52 | |
*** dayou has quit IRC | 18:53 | |
*** dayou_ has joined #openstack-lbaas | 18:53 | |
*** dayou1 has quit IRC | 18:54 | |
*** belharar has quit IRC | 18:55 | |
*** dayou has joined #openstack-lbaas | 18:55 | |
*** salmankhan has quit IRC | 18:55 | |
johnsom | Sep 28 11:56:06 devstackpy27-2 octavia-health-manager[111747]: error: [Errno 98] Address already in use | 18:56 |
johnsom | Yeah, I get the same result on this stack host too | 18:56 |
johnsom | systemctl restart devstack@o-hm | 18:56 |
johnsom | stack 111665 1 3 11:55 ? 00:00:03 /usr/bin/python /usr/local/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf | 18:57 |
johnsom | stack 111666 1 0 11:55 ? 00:00:00 /usr/bin/python /usr/local/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf | 18:57 |
johnsom | stack 111760 1 0 11:56 ? 00:00:00 /usr/bin/python /usr/local/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf | 18:57 |
johnsom | I do have an LB booted, I don't know if that makes a difference, but I suspect not | 18:57 |
openstackgerrit | Bar RH proposed openstack/octavia master: [WIP] Assign bind_host ip address per amphora https://review.openstack.org/505158 | 18:59 |
*** aojea has joined #openstack-lbaas | 19:17 | |
*** Alex_Staf has joined #openstack-lbaas | 19:28 | |
*** pcaruana has quit IRC | 19:31 | |
dayou | Are you using python3 or python2? | 19:36 |
dayou | It seems the address is binded by the hm listener proc, and we are chaning the hm health check proc, not sure how they affecting each other. | 19:37 |
dayou | changing | 19:38 |
*** eezhova has joined #openstack-lbaas | 19:41 | |
johnsom | python27 on this vm | 19:52 |
dayou | Can I gain access to the stack host running this? Would like to take a look at what's going on there | 19:55 |
rm_work | hard for me to stack currently :/ | 19:56 |
rm_work | umm, i could try that patch in my staging env maybe | 19:56 |
*** chlong has quit IRC | 20:03 | |
*** catintheroof has quit IRC | 20:09 | |
*** leitan has quit IRC | 20:26 | |
*** sshank has joined #openstack-lbaas | 20:28 | |
*** Alex_Staf has quit IRC | 20:31 | |
*** sshank has quit IRC | 20:33 | |
*** sshank has joined #openstack-lbaas | 20:33 | |
*** eezhova has quit IRC | 20:35 | |
*** belharar has joined #openstack-lbaas | 20:38 | |
*** gcheresh_ has joined #openstack-lbaas | 20:39 | |
*** rcernin has joined #openstack-lbaas | 20:42 | |
*** gcheresh_ has quit IRC | 20:50 | |
rm_work | kong: how goes the rebases | 21:15 |
kong | rm_work: i am working on the basic ops tempest case, a lot of debugging and testing yesterday. | 21:19 |
kong | but today, i just found there is something changed in tempest/test.py | 21:19 |
kong | thy removed some code that my code is relying on | 21:19 |
kong | so need some changes | 21:20 |
kong | before i submit for review | 21:20 |
johnsom | These are the only modules we can import from tempest: https://docs.openstack.org/tempest/latest/plugin.html#stable-tempest-apis-plugins-may-use | 21:20 |
johnsom | FYi... | 21:21 |
kong | rm_work: actually, i almost refactored all the code Jude was working on | 21:21 |
rm_work | ok | 21:21 |
kong | so, it's not a rebase, it's re-write | 21:21 |
rm_work | k | 21:21 |
rm_work | don't worry about the WIP:Failover at the end | 21:21 |
rm_work | I have completely rewritten that | 21:21 |
rm_work | and i am just waiting to rebase it onto your stuff | 21:22 |
kong | johnsom: thanks, i know that list | 21:22 |
*** sshank has quit IRC | 21:32 | |
*** sshank has joined #openstack-lbaas | 21:35 | |
*** leitan has joined #openstack-lbaas | 21:36 | |
*** leitan has quit IRC | 21:41 | |
*** chlong has joined #openstack-lbaas | 21:41 | |
*** aojea has quit IRC | 21:45 | |
johnsom | kong We have a bunch of stuff waiting on getting that baseline tempest plugin going. Do you have an ETA when code will be pushed up and ready for review? | 21:45 |
rm_work | kong: so is the patch you have up at https://review.openstack.org/#/c/486775/ close to final? or did you do further rewrites to that? | 21:46 |
kong | johnsom: don't worry, i am working full time on that | 21:46 |
kong | rm_work: yeah, that's for loadbalancer operations | 21:46 |
rm_work | ok | 21:47 |
kong | https://review.openstack.org/#/c/492311/9 | 21:47 |
kong | this one is for listener | 21:47 |
rm_work | right | 21:47 |
rm_work | just pool/member/hm left | 21:47 |
rm_work | and then i can start to rebase my stuff | 21:47 |
johnsom | Ok, so I can review the load balancer one now? | 21:47 |
rm_work | and see how it works out | 21:47 |
*** armax has joined #openstack-lbaas | 21:47 | |
kong | sure | 21:47 |
kong | please | 21:47 |
*** dulek has quit IRC | 21:53 | |
kong | rm_work, johnsom, there is one thing you need pay attention when running tempest, please give 'OS_TEST_TIMEOUT' config a bigggger value | 21:56 |
rm_work | yeah | 21:56 |
rm_work | mine is set very high | 21:56 |
rm_work | my tests run for hours ;P | 21:56 |
kong | brilliant | 21:56 |
*** belharar has quit IRC | 21:59 | |
*** sshank has quit IRC | 22:14 | |
*** dayou_ has quit IRC | 22:19 | |
*** sshank has joined #openstack-lbaas | 22:20 | |
*** catintheroof has joined #openstack-lbaas | 22:24 | |
johnsom | So I'm not sure jobs are firing, at least rechecks are not. The new zuulv3 dashboard is here: http://zuulv3.openstack.org/ | 22:51 |
*** ltomasbo has quit IRC | 23:44 | |
*** ltomasbo has joined #openstack-lbaas | 23:45 | |
*** sshank has quit IRC | 23:50 | |
*** SumitNaiksatam has quit IRC | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!