Wednesday, 2019-08-07

*** yamamoto has quit IRC00:04
*** nmagnezi has quit IRC00:09
*** logan- has quit IRC00:12
openstackgerritMichael Johnson proposed openstack/octavia master: Change amphora logging to dedicated socket
*** logan- has joined #openstack-lbaas00:13
openstackgerritMichael Johnson proposed openstack/octavia master: Re-enable the py36 tox environment
*** mkuf_ has joined #openstack-lbaas00:16
*** mkuf has quit IRC00:18
*** ivve has quit IRC00:21
*** hongbin has joined #openstack-lbaas00:34
*** yamamoto has joined #openstack-lbaas00:52
*** ricolin has joined #openstack-lbaas01:02
*** ricolin_ has joined #openstack-lbaas01:17
*** ricolin has quit IRC01:20
*** pcaruana has quit IRC01:26
*** ricolin_ is now known as ricolin02:27
openstackgerritMerged openstack/octavia-tempest-plugin master: Sync enabled_provider_drivers config help message
*** yamamoto has quit IRC02:44
*** yamamoto has joined #openstack-lbaas02:45
*** ramishra has joined #openstack-lbaas03:27
*** psachin has joined #openstack-lbaas03:36
*** ianychoi_ has joined #openstack-lbaas03:38
*** ianychoi has quit IRC03:42
*** pcaruana has joined #openstack-lbaas03:44
*** ajay33 has joined #openstack-lbaas03:45
ajay33@rm_work I get out from the error after restarting the octavia services.03:47
ajay33Thank you very much for you help !!!03:47
*** hongbin has quit IRC03:49
*** abaindur has joined #openstack-lbaas05:07
*** abaindur has quit IRC05:11
*** abaindur has joined #openstack-lbaas05:11
*** gcheresh_ has joined #openstack-lbaas05:27
*** ramishra has quit IRC06:12
*** AlexStaf has joined #openstack-lbaas06:21
*** maciejjozefczyk has joined #openstack-lbaas06:25
*** ltomasbo has joined #openstack-lbaas06:29
*** abh0 has joined #openstack-lbaas06:33
*** AlexStaf has quit IRC06:34
abh0Hello everyone,06:45
abh0octavia doesnt talk with it.06:45
abh0by the way i want any advice except updating the version.06:47
*** mkuf has joined #openstack-lbaas06:54
*** mkuf_ has quit IRC06:56
*** yamamoto has quit IRC07:00
*** rcernin has quit IRC07:03
*** rpittau|afk is now known as rpittau07:07
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Enable overcommit_memory in test server VMs
*** ramishra has joined #openstack-lbaas07:08
*** yamamoto has joined #openstack-lbaas07:09
*** tesseract has joined #openstack-lbaas07:26
dulekcgoncalves: Seems like the presentation haven't been accepted. At least the Summit is going to be a bit less stressful.07:27
openstackgerritMerged openstack/octavia master: Add missing lib dependencies to requirements.txt
*** ivve has joined #openstack-lbaas07:34
openstackgerritGregory Thiemonge proposed openstack/octavia master: Add unit test for failover of spare amphorae
cgoncalvesdulek, I'm okay with that07:49
dulekcgoncalves: Yeah, it's always a bit of relieve, isn't it?07:50
openstackgerritMaciej Józefczyk proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT
openstackgerritMaciej Józefczyk proposed openstack/octavia master: Validate supported LB algorithm in Amphora provider drivers
cgoncalvesyeah :)07:54
cgoncalvesdulek, the schedule page isn't loading the presentation list. does it open fine for you?07:55
dulekcgoncalves: Yes, it does.07:56
cgoncalvesit doesn't for me on chrome and firefox07:57
*** cgoncalves has quit IRC08:04
*** cgoncalves has joined #openstack-lbaas08:04
*** abaindur has quit IRC08:05
*** abh0 has quit IRC08:06
*** mauro|call has quit IRC08:13
*** mauro|call has joined #openstack-lbaas08:14
*** AlexStaf has joined #openstack-lbaas08:21
*** lemko has joined #openstack-lbaas08:43
*** mauro|call has quit IRC08:58
openstackgerritGregory Thiemonge proposed openstack/octavia stable/stein: Fix listener deletion in ACTIVE/STANDBY topology
*** mauro|call has joined #openstack-lbaas09:00
*** tkajinam has quit IRC09:02
ajay33My loadbalancer is going into ERROR state.09:04
ajay33Below is the error message I am getting from octavia-worker.log:09:04
ajay332019-08-07 06:02:04.904 31795 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='', port=9443): Max retries exceeded with url: /0.5/info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4f0d6b0390>: Failed to establish a new connection: [Errno 113] No09:04
ajay33route to host',))09:04
ajay33172.16.0.0/12 is my subnet of my lb-mgmt-subnet09:05
ajay33More specifically below is my error log:09:09
ajay332019-08-07 07:38:15.319 32660 ERROR octavia.amphorae.drivers.haproxy.exceptions [-] Amphora agent returned unexpected result code 404 with response {'error': '404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.', 'http_code': 404}09:09
*** mauro|call is now known as takamatsu09:14
*** mkuf__ has joined #openstack-lbaas09:27
*** mkuf has quit IRC09:29
*** mkuf____ has joined #openstack-lbaas09:30
*** mkuf__ has quit IRC09:32
*** mkuf____ has quit IRC09:34
*** mkuf____ has joined #openstack-lbaas09:35
*** mkuf____ has quit IRC09:40
*** mkuf____ has joined #openstack-lbaas09:46
*** mkuf____ has quit IRC09:49
*** sapd1_x has joined #openstack-lbaas09:54
*** ricolin_ has joined #openstack-lbaas10:01
*** ricolin has quit IRC10:03
*** nmagnezi has joined #openstack-lbaas10:05
*** yamamoto has quit IRC10:18
*** spatel has joined #openstack-lbaas10:27
*** spatel has quit IRC10:31
*** gcheresh_ has quit IRC10:39
*** gcheresh has joined #openstack-lbaas10:40
*** yamamoto has joined #openstack-lbaas10:41
*** yamamoto has quit IRC10:45
openstackgerritAnn Taraday proposed openstack/octavia master: Transition amphora flows to dicts
*** yamamoto has joined #openstack-lbaas10:59
*** AlexStaf has quit IRC11:06
*** AlexStaf has joined #openstack-lbaas11:20
*** lemko has quit IRC11:23
openstackgerritMerged openstack/octavia master: Fix wrong package names or versions for centos-minimal images
*** AlexStaf has quit IRC11:57
*** AlexStaf has joined #openstack-lbaas11:58
openstackgerritMaciej Józefczyk proposed openstack/octavia master: Add new algorithm SOURCE_IP_PORT
openstackgerritMaciej Józefczyk proposed openstack/octavia master: Validate supported LB algorithm in Amphora provider drivers
henriqueofHello there, my load balancers only show empty response on browser, can someone help me debug it?12:47
*** lemko has joined #openstack-lbaas12:48
*** ricolin_ is now known as ricolin12:48
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Lb flows to dicts
henriqueofThe haproxy services show all servers are down, how can I check it?13:29
*** tesseract has quit IRC13:56
*** tesseract has joined #openstack-lbaas13:57
*** tesseract has quit IRC14:00
*** tesseract has joined #openstack-lbaas14:01
*** mkuf has joined #openstack-lbaas14:13
*** mkuf has quit IRC14:15
*** spatel has joined #openstack-lbaas14:20
openstackgerritAnqi Li proposed openstack/octavia master: Implements notifications for octavia
*** Vorrtex has joined #openstack-lbaas14:43
*** altlogbot_1 has quit IRC14:46
*** altlogbot_3 has joined #openstack-lbaas14:47
*** yamamoto has quit IRC14:48
openstackgerritGregory Thiemonge proposed openstack/octavia master: Fixed typos and bad markup style in maintenance guide
*** ianychoi_ is now known as ianychoi14:54
*** yamamoto has joined #openstack-lbaas14:54
*** gcheresh has quit IRC14:55
*** yamamoto has quit IRC15:21
*** yamamoto has joined #openstack-lbaas15:21
*** sapd1_x has quit IRC15:42
*** ataraday_ has joined #openstack-lbaas15:45
rm_workcgoncalves / johnsom :
*** AlexStaf has quit IRC15:58
*** tesseract has quit IRC15:58
johnsomrm_work Meeting time?16:01
colin-we missed it by one minute :\16:01
openstackgerritMichael Johnson proposed openstack/octavia master: Change amphora logging to dedicated socket
*** tesseract has joined #openstack-lbaas16:02
*** henriqueof has quit IRC16:02
johnsom#startmeeting Octavia16:03
openstackMeeting started Wed Aug  7 16:03:48 2019 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:03
*** openstack changes topic to " (Meeting topic: Octavia)"16:03
openstackThe meeting name has been set to 'octavia'16:03
johnsomI am not exactly sure what is going on, but I will start the meeting16:04
johnsomHi everyone16:04
*** yamamoto has quit IRC16:05
johnsomOur PTL was just commenting, but then disappeared16:05
rm_workyeah whoops missed the time :D16:05
rm_workwas reviewing the patch you just updated about the log socket lol16:05
rm_workgo ahead, you're doing good :)16:06
johnsomHa, well, I don't have an agenda, do you?16:06
*** yamamoto has joined #openstack-lbaas16:06
rm_work^^ my yubikey wants you to know the meeting is going well16:06
*** yamamoto has quit IRC16:06
johnsomlol, yubikey?16:06
johnsom#topic Announcements16:07
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:07
*** yamamoto has joined #openstack-lbaas16:07
johnsomAnybody have anything this week?16:07
johnsomThe Shanghai summit schedule is posted16:08
johnsomOk, moving on then16:08
johnsom#topic Brief progress reports / bugs needing review16:08
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:08
*** maciejjozefczyk has quit IRC16:09
johnsomI have been busy working on octavia-lib provider related things recently. That main work is done, but I have some patch cleanup to do still.16:09
rm_worklots of merging going on! woo!16:10
rm_workgreat turnaround from a week ago when i was saying i was super concerned about review velocity16:10
johnsomI am going to start work on a better failover flow.16:10
johnsomYes, it looks like we have burned through 60+ patches since I setup the review list. That is awesome16:11
*** yamamoto has quit IRC16:11
ataraday_I posted some comments on about issues with retry tasks - usage of redis jobboard, will try to find something else, for now concluding refactor of db object - dicts...16:11
openstackgerritAnqi Li proposed openstack/octavia master: Implements notifications for octavia
ataraday_johnsom, I will need to get your change updated, do you mind if I rebase it?16:12
johnsomataraday_ Please do, I keep forgetting to look at that.  It's going to be a hard one to do.16:13
ataraday_johnsom, OK, thanks!16:13
johnsomAh, the data too long for column. I might have some ideas  on how to fix that.16:14
johnsomAny other updates?16:16
*** lemko has quit IRC16:17
johnsom#topic Open Discussion16:17
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:17
johnsomOther topics this week?16:18
ataraday_johnsom, if you have time, may be you left some comments on change? My only idea to try notify taskflow listener that task is in progress, though not sure it is possible...16:18
colin-i realize the original authors of the active-active L3 spec are not present to answer this but wanted to ask anyway because i was curious; when looking at the changes i don't see any mention of VRRP being used and was curious if anybody knew why not? it appears in haproxy's active-active guidelines is why i was wondering16:20
johnsomVRRP for the top of rack switches or for the amphora?16:21
*** maciejjozefczyk has joined #openstack-lbaas16:22
colin-for the amphora16:22
johnsomOpps, wrong window. lol16:22
colin-trying to go home so soon :)?16:22
*** ricolin_ has joined #openstack-lbaas16:23
johnsomWell, in active-active the amphora are all equal, they don't need to pass a VIP between amphora, the L3 layer is holding the VIP16:23
*** ivve has quit IRC16:23
colin-i think this might be a mistake on my part, i think the VRRP refers just to their Aloha virtual appliance that i'm actually unfamiliar with16:23
johnsomIn the non-L3 ToR active/active, the distributor component will use VRRP to pass the VIP around16:23
colin-so maybe disregard heh, quesiton might be half-baked16:23
*** ricolin has quit IRC16:26
johnsomOk, anything else?16:26
johnsomataraday_ That DataError, was that when you were using Redis for persistence?16:26
johnsomMust have lost her.16:29
johnsomAny other topics  today?16:29
*** ricolin_ is now known as ricolin16:29
*** psachin has quit IRC16:30
johnsomOk, thanks everyone16:30
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list:"16:30
openstackMeeting ended Wed Aug  7 16:30:39 2019 UTC.  Information about MeetBot at . (v 0.1.4)16:30
openstackMinutes (text):
ataraday_johnsom, Redis as jobboard, mysql as persistence16:30
johnsomAh, ok, thanks.16:31
rm_workwoo, nice and short16:31
ataraday_johnsom, redis need retry for claimning job, tasks info in written in mysql persistence backend16:32
johnsomYeah, I found the code. It looks like taskflow is using a "Text" column for that. Which holds up to 64k16:32
johnsomWe may just need to do a migration and bump that to a "MEDIUMTEXT" column16:33
johnsomOr get taskflow to truncate and not try to store so much16:33
ataraday_yeah, I filed a bug to taskflow about this, but I want to fix it on our side, so we won't depend on taskflow latest version...16:35
johnsomOk. We may need to fix that ourselves in taskflow.16:35
ataraday_I will check your idea with "MEDIUMTEXT" if it worked -  probably send change to taskflow16:37
ataraday_johnsom, thanks a lot for looking into it!16:39
*** tesseract has quit IRC16:39
*** ricolin has quit IRC16:40
*** rpittau is now known as rpittau|afk16:56
*** ramishra has quit IRC16:57
*** maciejjozefczyk has quit IRC16:57
*** gthiemonge has quit IRC17:10
*** altlogbot_3 has quit IRC17:17
*** altlogbot_2 has joined #openstack-lbaas17:23
*** gthiemonge has joined #openstack-lbaas17:30
*** altlogbot_2 has quit IRC17:31
*** henriqueof has joined #openstack-lbaas17:32
henriqueofMy load balancer haproxy says all the servers are down, can someone help me debug it?17:33
*** altlogbot_1 has joined #openstack-lbaas17:35
johnsomhenriqueof Hi, What does "openstack loadbalancer status show <lb_id>" show?17:44
johnsomYou can use to share output17:44
henriqueofThe haproxy logs inside the amphora says all the servers are down, the servers and their respective security groups are created by magnum.17:49
openstackgerritMerged openstack/octavia master: Fixed pool and members status with UDP loadbalancers
johnsomhenriqueof Ok, first thing is your lb-mgmt-net is not working correctly or the setting is not correct for your controllers17:54
johnsomI can see that the heartbeat messages from the amphora instance is not getting back to the controller17:54
johnsomThis will not impact traffic flowing through the load balancer however, just the status, stats, and health monitoring.17:56
henriqueofjohnsom: I am using the controller's interface I on that setting, will put the management IP.17:56
henriqueofjohnsom: I have used this same setting on another OS deployment and it works. :/17:57
*** spatel has quit IRC17:57
johnsomSomething is configured incorrectly as I can see you have health monitors, but the member still shows "NO_MONITOR" on it. This is a clear sign that the controller is not getting the messages from the amphora.17:58
openstackgerritMerged openstack/octavia master: Fixed down server issue after reloading keepalived
henriqueofDoes this affect the internal haproxy, I mean it can't see the servers?18:00
johnsomNo, as I said above, it will not impact the traffic through the load balancer.18:01
johnsomFor that issue, are you sure the service is reachable on IP port 32429? Can you reach that from outside the amphora?18:02
johnsomThe other thing to check is that is reachable from the subnet you selected when creating the member.18:05
henriqueofWill double check it, one moment.18:06
henriqueofjohnsom: just found something very odd, the nodes are connected to the same network and router, one can ping to the internet and the other can't, I added a security group allowing all traffic but still uncapable of going t the internet.18:39
henriqueofAnyway, thanks very much for your help!18:39
johnsomSure, good luck18:39
*** abaindur has joined #openstack-lbaas19:00
*** vesper11 has quit IRC19:37
openstackgerritMerged openstack/octavia master: Prevent UDP LBs to use different IP protocol versions in amphora driver
*** ivve has joined #openstack-lbaas20:15
*** ajay33 has quit IRC20:29
*** yamamoto has joined #openstack-lbaas21:03
*** yamamoto has quit IRC21:08
*** Vorrtex has quit IRC21:19
*** vesper11 has joined #openstack-lbaas21:25
*** henriqueof has quit IRC21:33
*** rcernin has joined #openstack-lbaas22:15
rm_worksmallish and have +2 from me22:31
*** ivve has quit IRC22:45
*** yamamoto has joined #openstack-lbaas23:08
abaindurhey johnsom: did something change for octavia with loadbalancer topology? we're nnot using flavors. and we definitely have loadbalancer_topology = ACTIVE_STANDBY in the config file for worker/health-manager/housekeeping23:34
abaindurbut only 1 amphora VM is spun up23:34
abaindurLB shows as ACTIVE23:34
abainduramphora table in octavia DB is showing it as STANDALONE:23:35
abaindur| 1cd6ae3d-cbba-4525-b52b-bc7418b9c21e | 009721c5-5baf-45ee-83ae-c174212994b9 | ALLOCATED | 4834c6c9-b80a-4147-a93d-ac8509c5accd |    | | | 253e3af0-6a90-4b32-a8a8-94bc5ddefbf8 | 94e4e516-fa61-41ec-95d2-8a10c27a071e | STANDALONE | 2021-08-06 23:32:01 |         0 | NULL           |       1 |          NULL | nova        | 2019-08-07 23:32:02 | 2019-08-07 23:34:03 |23:35
johnsomNo, without a flavor defined, the loadbalancer_topology setting is what is used.23:35
abaindur fc3e5dc6-01d1-b7ad-ed6e-1935810c8aad | 223:35
abainduryes i definitly have loadbalancer_topology = ACTIVE_STANDBY in [controller_worker] section23:36
johnsomCheck your worker log, on startup it dumps the running configuration, is it ACTIVE_STANDBY in there?23:37
johnsomIt's a debug message though:23:38
johnsomJul 30 10:52:23 devstack octavia-worker[19646]: DEBUG cotyledon.oslo_config_glue [-] controller_worker.loadbalancer_topology = SINGLE {{(pid=20380) log_opt_values /usr/local/lib/python3.6/dist-packages/oslo_config/}}23:38
johnsomWhat does the load_balancer table in the DB show for it?23:39
abaindur2019-08-07 16:26:18.216 20671 DEBUG cotyledon.oslo_config_glue [-] controller_worker.loadbalancer_topology = ACTIVE_STANDBY log_opt_values /opt/pf9/pf9-octavia/lib/python2.7/site-packages/oslo_config/
abaindurhere are worker logs during start of LB create23:40
abainduryes its showing SINGLE:23:40
abaindur| ccc6c00ce08a4e8794bfe692be87d802 | 4834c6c9-b80a-4147-a93d-ac8509c5accd | lb1  | NULL        | ACTIVE              | OFFLINE          |       1 | SINGLE   | NULL            | 2019-08-07 23:31:58 | 2019-08-07 23:34:11 | amphora  | NULL      |23:40
abaindur(and amphora table shows STANDALONE as pasted above)23:41
johnsomSo that SINGLE means somehow the API process was configured for SINGLE and not ACTIVE_STANDBY23:41
abainduryea as i mentioned this config isnt set in the file used by api service - that is located on a different node23:42
johnsomThere is no code that can change that after LB create.23:42
abaindursince this was in the [controller_worker] section, i thought it was only used by worker process?23:42
johnsomAh, that is why then. The API makes the call on that at creation time.23:42
johnsomYeah, this is a odd ball setting. We really haven't documented which settings are used by which processes.23:43
abaindurso then out of these configs, can you tell me whether its used by API service, or worker/health-manager/housekeeping?23:43
johnsomspare_amphora_pool_size is not used by they api23:44
colin-the first is a parameter in [nova], the second in the [house_keeping]23:45
johnsomNeither is enabled_anti_affinity23:45
abaindurwho uses that then?23:46
johnsomThose are needed by the backend processes however23:46
johnsomenable_anti_affinity should be on worker, HM, and housekeeping.23:47
abaindurBasically we have octavia-api service running on one host, and we have worker, health-manager, and housekeeping running on the other. So we have 2 sets of config files being used depending on which host23:47
johnsomYeah, same with pool size, it really should be set on all three, but primarily HK uses it23:48
johnsomYeah, that is a common pattern.23:48
abaindurok so it looks like we just need to move the loadbalancer_topology config over to the host where api is running23:48
abaindureverything else under [controller_worker] can be left on host where worker/hm/hk run?23:48
johnsomI would have to go through them one by one and look them up.23:51

Generated by 2.15.3 by Marius Gedminas - find it at!