Thursday, 2018-07-12

« Wednesday, 2018-07-11 Index Friday, 2018-07-13 »
openstackgerritGerman Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep  https://review.openstack.org/54929500:10
*** longkb has joined #openstack-lbaas00:36
*** hongbin has joined #openstack-lbaas00:46
*** yamamoto has joined #openstack-lbaas00:49
*** yamamoto has quit IRC00:54
*** yamamoto has joined #openstack-lbaas01:50
*** yamamoto has quit IRC01:55
*** yamamoto has joined #openstack-lbaas02:03
*** hongbin has quit IRC02:38
*** hongbin has joined #openstack-lbaas02:38
*** fnaval has joined #openstack-lbaas02:57
*** hongbin has quit IRC03:08
*** hongbin has joined #openstack-lbaas03:12
*** ramishra has joined #openstack-lbaas04:05
*** gans has joined #openstack-lbaas04:18
*** hongbin has quit IRC04:19
*** gans has quit IRC04:23
*** links has joined #openstack-lbaas05:03
*** nmanos has joined #openstack-lbaas05:03
*** strigazi_ has joined #openstack-lbaas05:29
*** strigazi has quit IRC05:32
*** strigazi has joined #openstack-lbaas05:34
*** strigazi_ has quit IRC05:36
*** yboaron has joined #openstack-lbaas06:05
*** nmanos has quit IRC06:12
*** nmanos has joined #openstack-lbaas06:18
*** nmanos has left #openstack-lbaas06:18
openstackgerritZhaoBo proposed openstack/octavia master: UDP jinja template  https://review.openstack.org/52542006:30
openstackgerritZhaoBo proposed openstack/octavia master: UDP for [2]  https://review.openstack.org/52965106:30
openstackgerritZhaoBo proposed openstack/octavia master: UDP for [3][5][6]  https://review.openstack.org/53939106:30
*** ispp has joined #openstack-lbaas06:32
*** velizarx has joined #openstack-lbaas06:56
*** velizarx has quit IRC07:13
*** rcernin has quit IRC07:20
*** ispp has quit IRC07:21
*** velizarx has joined #openstack-lbaas07:22
*** peereb has joined #openstack-lbaas07:25
*** ispp has joined #openstack-lbaas07:30
*** kobis has joined #openstack-lbaas07:39
*** kobis has quit IRC07:44
*** yamamoto has quit IRC07:48
*** ispp has quit IRC07:50
*** kobis has joined #openstack-lbaas07:51
*** ispp has joined #openstack-lbaas07:51
*** rraja has joined #openstack-lbaas08:06
*** ispp has quit IRC08:08
*** links has quit IRC08:24
*** links has joined #openstack-lbaas08:26
*** ktibi has joined #openstack-lbaas08:31
*** ispp has joined #openstack-lbaas08:31
openstackgerritTuan Do Anh proposed openstack/octavia master: Update pypi url to new url  https://review.openstack.org/58209408:33
*** sapd has quit IRC08:35
*** sapd has joined #openstack-lbaas08:35
*** tesseract has joined #openstack-lbaas08:37
ktibiHi octavia, I can see in video from last summit, neutron-lbaas will be deprecated soon. Octavia support for plugin like with F5 ?08:42
*** yamamoto has joined #openstack-lbaas08:44
*** ispp has quit IRC08:48
*** yamamoto has quit IRC08:50
*** ispp has joined #openstack-lbaas08:53
*** yboaron has quit IRC09:03
*** links has quit IRC09:10
*** links has joined #openstack-lbaas09:12
*** kobis has quit IRC09:38
*** yamamoto has joined #openstack-lbaas09:46
*** yamamoto has quit IRC09:51
*** yamamoto has joined #openstack-lbaas10:03
*** kobis has joined #openstack-lbaas10:06
*** kobis has quit IRC10:41
*** kobis has joined #openstack-lbaas10:41
*** yboaron has joined #openstack-lbaas10:48
*** velizarx has quit IRC10:53
*** velizarx has joined #openstack-lbaas11:09
*** atoth has joined #openstack-lbaas11:17
*** longkb has quit IRC11:34
*** phuoc has quit IRC11:53
*** phuoc has joined #openstack-lbaas11:53
*** amuller has joined #openstack-lbaas12:02
*** hvhaugwitz has quit IRC12:10
*** hvhaugwitz has joined #openstack-lbaas12:10
*** ispp has quit IRC12:16
*** ispp has joined #openstack-lbaas12:22
*** atoth has quit IRC12:22
*** kobis has quit IRC12:29
openstackgerritCarlos Goncalves proposed openstack/octavia master: Translate expected package names to installed ones  https://review.openstack.org/58217012:36
cgoncalvesjohnsom, ^ should resolve the issue I found while reviewing https://review.openstack.org/#/c/577344/. if you agree, rebase yours on top of that12:41
*** atoth has joined #openstack-lbaas12:52
*** yamamoto has quit IRC13:01
*** velizarx has quit IRC13:12
*** KeithMnemonic has joined #openstack-lbaas13:14
*** velizarx has joined #openstack-lbaas13:16
*** fnaval has quit IRC13:25
*** yamamoto has joined #openstack-lbaas13:28
*** yamamoto has quit IRC13:29
*** yamamoto has joined #openstack-lbaas13:34
*** fnaval has joined #openstack-lbaas13:35
*** fnaval has quit IRC13:39
*** yamamoto_ has joined #openstack-lbaas13:41
*** yamamoto has quit IRC13:41
*** fnaval has joined #openstack-lbaas13:45
*** kobis has joined #openstack-lbaas13:47
*** links has quit IRC13:50
johnsomktibi Yes, Octavia supports provider drivers. Contact F5 for information on when their driver will be ready.14:02
ktibijohnsom, thx ;)14:02
*** kobis has quit IRC14:05
*** velizarx has quit IRC14:14
jitekaHello I've faced an issue today in my lab, trying to rotate amphora image14:20
jitekaI added the new image in glance and initiated a failover, but the new amphora VM never came healthy due to :14:20
jitekaFailover exception: Waiting for compute to go active timeout.: ComputeWaitTimeoutException: Waiting for compute to go active timeout.14:20
jitekaLooking at nova logs it appear that the amp build failed due to :14:20
jitekaNeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.14:20
jitekaI ended up with my loadbalancer in ERROR running on only 1 amp MASTER instead of 2 in ACTIVE/STANDBY14:20
jitekaI forced a failover after updating the "provisioning_status" to ACTIVE and I ended up again with only 1 amp (and as expected experienced some downtime on my lb)14:20
jitekaThen I tried another approach by deleting the last amp to see what the health-manager would do and I ended up again with only 1 amp14:20
jitekaWhat could I do in that case to come back to 1 Master and 1 Backup when I only have 1 Master ?14:20
jitekahttps://pastebin.com/2tLLrKaz14:21
*** kobis has joined #openstack-lbaas14:21
jitekaI'm running devstack stable/queens on Ubuntu 16.04.4 LTS14:21
johnsomjiteka Did you resolve the nova issue that caused the failure?14:23
johnsomjiteka We have a few patches up for review that address a few of those cases where a failover fails itself due to nova/neutron outages. Let me find a few links.14:24
johnsomThis is probably what you are hitting: https://review.openstack.org/#/c/577344/ We haven't merged this on master yet, so not backported to queens yet.14:25
johnsomAnd this one https://review.openstack.org/54898914:27
johnsomIt doesn't look like that one got backported yet either, though it is merged14:27
jitekajohnsom: I was able to create a new VM after that failure14:31
jitekajohnsom: I have the feeling that it's only on the first build using a new image14:31
*** kobis has quit IRC14:34
jitekajohnsom: did the healthcheck manager will ensure that if a lb run only on 1 amp with role MASTER, when configured for ACTIVE_STANDBY, it will trigger creation of a new BACKUP ?14:35
jitekajohnsom: to recover from such situation14:35
*** mugsie has quit IRC14:36
*** mugsie has joined #openstack-lbaas14:36
*** mugsie has quit IRC14:36
*** mugsie has joined #openstack-lbaas14:36
jitekajohnsom: worst case scenario, if that patch https://review.openstack.org/#/c/577344/ allow the heath-manager to come back in stable situation when all amps are unreachable/deleted, it could work too even if causing downtime14:41
cgoncalvesjohnsom, how is one supposed to test amphora agent code changes if DIB pulls code from git.o.o no matter what?14:48
johnsomjiteka Yes, under normal situations, the HM will rebuild either amphora should it be in failure. It will stop if the failover fails and mark it in error though.14:48
cgoncalvesand FWIW always from master14:49
johnsomjiteka If you are running Active/Backup there should be less than a second of downtime with the right tuning14:49
jitekajohnsom: yes downtime wasn't an issue with 2 amp, 1 Master and 1 backup14:50
jitekajohnsom: but here I have only 1 amp Master and I don't know how to recover from that to get a backup as heathmanager is not re-creating it14:50
johnsomIf you have the right patches (see above) you can mark the failed backup as "ACTIVE" and it will try again14:51
johnsomcgoncalves There are a few answers to that.14:51
jitekajohnsom: when I delete the MASTER, healthmanager take care of re-creating it, standalone first, then MASTER, but BACKUP creation is never re-triggered14:51
jitekajohnsom: that's the thing, I don't have any backup unfortunately, just 1 amp associated to that LB14:52
johnsomRight, if the BACKUP amp failed during a failover, because of nova/neutron failure, it will be marked as ERROR and tagged to not attempt to failover it again until an operator has fixed nova/neutron and tagged it back.14:52
johnsomWe don't want to make the nova/neutron failure worse by hitting it 10000s of times trying to restore an amp14:53
johnsomcgoncalves So, there are environment variables for DIB that override where it gets the amp agent.14:54
johnsomcgoncalves Default is pull from git master.14:55
johnsomcgoncalves In devstack we override those to a local location: https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L5814:55
johnsomthrough line 6514:55
jitekajohnsom: thanks for the explaination :)14:58
jitekajohnsom: I guess my mistake here was to delete amp while lb was in inconsistent state and hack into the db to force it as active (it was in error) as I don't see any other way to recover and coming back to ACTIVE to perform administrative task on it15:00
cgoncalvesjohnsom, ah, I see! I was not getting my changes in when building the image outside devstack. so DIB_REPOLOCATION_amphora_agent=/local/path/to/octavia ./diskimage-create.sh [...]15:00
cgoncalvesthanks!15:00
johnsomjiteka I think that should have worked, but the best option would have been to just mark it active again.  However, you probably need those two patches15:01
johnsomcgoncalves FYI, https://docs.openstack.org/diskimage-builder/latest/elements/source-repositories/README.html#override-per-source15:02
cgoncalvesappreciated!15:04
*** yboaron has quit IRC15:04
*** ispp has quit IRC15:12
*** ispp has joined #openstack-lbaas15:13
*** kobis has joined #openstack-lbaas15:19
openstackgerritMurali Annamneni proposed openstack/neutron-lbaas master: Hardcode foreignkey constraint name for lbaas_listeners  https://review.openstack.org/55779715:23
*** yamamoto_ has quit IRC15:24
*** rraja has quit IRC15:30
*** ispp has quit IRC15:32
*** kobis has quit IRC15:39
*** kobis has joined #openstack-lbaas15:44
*** peereb has quit IRC15:45
*** ktibi has quit IRC15:58
*** kobis has quit IRC15:59
*** yamamoto has joined #openstack-lbaas16:12
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix package version querying on non-dpkg distros  https://review.openstack.org/58229316:23
*** ramishra has quit IRC16:24
openstackgerritCarlos Goncalves proposed openstack/octavia master: Translate expected package names to installed ones  https://review.openstack.org/58217016:33
*** yamamoto has quit IRC16:42
*** kobis has joined #openstack-lbaas16:42
*** kobis has quit IRC16:47
cgoncalveswith my latest 2 patches on top of fix failover patch, I can now create CentOS-based HA loadbalancers17:05
cgoncalvesjohnsom, trying to failover the LB by deleting both amps, I've got only 1 amp VM up17:06
johnsomYeah, one amp in error the other not. That is the bug17:06
cgoncalvesseems that nova did not receive any request to create the second one, even though it shows up on the amp list17:06
cgoncalvesthe worker keeps trying to connect to the agent but there's no VM in the first place17:07
johnsomhmm, ok, that doesn't quiet make sense.  Check you HM log, it should show the failover aborted when it couldn't rebuild the first amp properly17:07
cgoncalvesit rebuilt the first amp properly. the second is missing17:08
johnsomDid it really finish the flow?  Check the log, it should have failed and reverted the failover flow17:09
cgoncalvesI'm checking. it hasn't reverted to anything (yet) looking at the logs17:10
cgoncalveshttp://paste.openstack.org/show/725742/17:14
cgoncalvesI did not run the lb failover explicitly. just deleted nova instances and observed17:14
johnsomcorrect17:15
*** tesseract has quit IRC17:16
*** ramishra has joined #openstack-lbaas17:23
*** ramishra has quit IRC17:37
*** yamamoto has joined #openstack-lbaas17:43
*** yamamoto has quit IRC18:00
*** kobis has joined #openstack-lbaas18:18
*** rraja has joined #openstack-lbaas18:19
*** harlowja has joined #openstack-lbaas18:24
cgoncalvesjohnsom, reached connection time out. as a result, I now have only one amp listed and LB reports operating_status ONLINE19:08
cgoncalveshttp://paste.openstack.org/show/725754/19:09
*** kobis has quit IRC19:17
*** Deknos has joined #openstack-lbaas19:21
*** kobis has joined #openstack-lbaas19:29
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix package version querying on non-dpkg distros  https://review.openstack.org/58229319:31
johnsomcgoncalves I am not sure what is going on there. I would have to see the full HM log to figure it out.19:33
openstackgerritCarlos Goncalves proposed openstack/octavia master: Fix package version querying on non-dpkg distros  https://review.openstack.org/58229319:41
openstackgerritCarlos Goncalves proposed openstack/octavia master: Translate expected package names to installed ones  https://review.openstack.org/58217019:41
*** Deknos has left #openstack-lbaas19:42
*** kobis has quit IRC19:47
cgoncalvesjohnsom, http://cgoncalves.pt/trash/openstack/journalctl-o-hm-2018-07-12.log19:56
*** amuller has quit IRC19:59
johnsomcgoncalves Is this with the patch or without? Are you running multple hm?20:03
cgoncalvesjohnsom, with your patch + my 2 patches of today. 1 HM20:05
johnsomcgoncalves and that was the start of the hm log, no other failures before that right?20:06
johnsomThe interest thing for me is that it did get past the point my code directly addresses... it is failing much later in the flow.20:06
cgoncalveswith so many testing I no longer can tell when last try started. let me re-try, sorry about that20:09
johnsomcgoncalves hold up20:09
cgoncalvesI can get older log msgs. I truncated till 3 hours ago20:10
johnsomcgoncalves when you try again, when it gets into that second retry loop for the connection, can you do a "openstack server list" and "openstack amphora list"? It almost looks like in the first part it didn't mark the second failed amp as ERROR for some reason20:10
cgoncalvesok20:11
johnsomThat second connection looks like it's trying to connect to the second failed amp, which it should not be, the second failed amp should be in "ERROR" at that point and bypassed in the VRRP code20:11
cgoncalvesjohnsom, http://paste.openstack.org/show/725764/ and http://cgoncalves.pt/trash/openstack/journalctl-o-hm-2018-07-12.log20:51
*** rraja has quit IRC20:56
*** rraja has joined #openstack-lbaas20:56
johnsomcgoncalves is that the right log link? it's the same as the last one21:16
cgoncalvesjohnsom, it is21:16
johnsomok21:17
johnsomcgoncalves How are you failing these? deleting both with nova?21:25
cgoncalvesjohnsom, deleting both with nova, yes. see http://paste.openstack.org/show/725764/21:26
johnsomSomething is really wrong, I should see https://review.openstack.org/#/c/577344/2/octavia/controller/worker/tasks/amphora_driver_tasks.py in the log, but it's not there21:26
johnsomline 6221:26
cgoncalvesone thing I can confirm you: I'm running with your patch21:29
cgoncalves[centos@rdocloud-devstack2 octavia]$ pwd21:29
cgoncalves[centos@rdocloud-devstack2 octavia]$ grep "Failed to update listeners on amphora" octavia/controller/worker/tasks/amphora_driver_tasks.py21:29
cgoncalves            LOG.error('Failed to update listeners on amphora %s. Skipping '21:29
johnsomYeah, I just don't get why that would not fire....21:29
cgoncalvesnote that I only created the LB. no listeners nor pools where created21:31
johnsomOh, hmmmm, that might be the key, there might be a bug in this if there are no listeners21:32
cgoncalves*were21:33
johnsomDarn, I am so buried with internal work right now, not sure when I can get back to that21:33
cgoncalvesOSP12? xD21:33
cgoncalvesno worries. I just wanted to help testing21:34
johnsomYeah, if I knew a week ago I would have been able to get it21:34
johnsomsigh21:34
cgoncalvesgood that I did. found other issues when on centos-based amps21:34
*** rcernin has joined #openstack-lbaas21:58
*** yboaron has joined #openstack-lbaas22:05
cgoncalvesjohnsom, creating listener before forcing failover did the trick22:11
johnsomYeah, it's a bug for scenarios with no listener. not sure if the vrrp subflow should not be running or some other issue....22:12
cgoncalvesalthough the second amp only started being created after first amp had been recovered22:12
cgoncalvesI'd have expected, I guess, both to be rebuilt simultaneously22:12
johnsomThat is correct behavior22:13
johnsomWell, there are some sequencing issues there. notably to configure the other peer we need it's ip info.22:13
johnsomIt could be done in the future with some fancy sequencing, etc. but...  fix bug before optimize the whole flow22:14
*** fnaval has quit IRC22:14
*** yboaron has quit IRC22:14
cgoncalvesok, fair enough :)22:14
*** rraja has quit IRC22:17
*** rraja has joined #openstack-lbaas22:21
*** fnaval has joined #openstack-lbaas22:30
*** rraja has quit IRC22:33
*** fnaval has quit IRC22:39
openstackgerritCarlos Goncalves proposed openstack/octavia master: Translate expected package names to installed ones  https://review.openstack.org/58217022:41
*** KeithMnemonic has quit IRC22:48
*** fnaval has joined #openstack-lbaas22:52
*** fnaval has quit IRC22:56
*** fnaval has joined #openstack-lbaas23:49
openstackgerritGerman Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep  https://review.openstack.org/54929523:58
« Wednesday, 2018-07-11 Index Friday, 2018-07-13 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!