| *** blake has quit IRC | 00:10 | |
| *** blake has joined #openstack-lbaas | 00:12 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix health manager to be spare amphora aware https://review.openstack.org/576668 | 00:12 |
|---|---|---|
| *** blake has quit IRC | 00:32 | |
| *** Swami has quit IRC | 00:33 | |
| *** longkb has joined #openstack-lbaas | 00:35 | |
| *** blake has joined #openstack-lbaas | 00:38 | |
| *** blake has quit IRC | 00:45 | |
| *** yamamoto has joined #openstack-lbaas | 00:48 | |
| *** yamamoto has quit IRC | 00:53 | |
| *** annp has joined #openstack-lbaas | 01:18 | |
| *** ramishra has joined #openstack-lbaas | 01:34 | |
| openstackgerrit | huangshan proposed openstack/python-octaviaclient master: Support backup members https://review.openstack.org/576530 | 01:34 |
| *** yamamoto has joined #openstack-lbaas | 01:49 | |
| *** yamamoto has quit IRC | 01:56 | |
| *** ianychoi_ has quit IRC | 02:27 | |
| *** ianychoi has joined #openstack-lbaas | 02:30 | |
| *** yamamoto has joined #openstack-lbaas | 02:51 | |
| *** yamamoto has quit IRC | 02:57 | |
| *** links has joined #openstack-lbaas | 03:23 | |
| *** mnaser has quit IRC | 03:42 | |
| openstackgerrit | huangshan proposed openstack/python-octaviaclient master: Support backup members https://review.openstack.org/576530 | 03:42 |
| *** dougwig has quit IRC | 03:43 | |
| *** zioproto has quit IRC | 03:44 | |
| *** fyx has quit IRC | 03:44 | |
| *** beisner has quit IRC | 03:44 | |
| *** mnaser has joined #openstack-lbaas | 03:45 | |
| *** dougwig has joined #openstack-lbaas | 03:45 | |
| *** dougwig is now known as Guest29132 | 03:46 | |
| *** zioproto has joined #openstack-lbaas | 03:46 | |
| *** fyx has joined #openstack-lbaas | 03:46 | |
| *** beisner has joined #openstack-lbaas | 03:46 | |
| *** wolsen has quit IRC | 03:47 | |
| *** amitry has quit IRC | 03:49 | |
| *** johnsom has quit IRC | 03:50 | |
| *** ctracey has quit IRC | 03:50 | |
| *** hogepodge has quit IRC | 03:50 | |
| *** Guest29132 has quit IRC | 03:50 | |
| *** bzhao__ has quit IRC | 03:51 | |
| *** beisner has quit IRC | 03:51 | |
| *** mnaser has quit IRC | 03:51 | |
| *** fyx has quit IRC | 03:51 | |
| *** zioproto has quit IRC | 03:51 | |
| *** lxkong has quit IRC | 03:51 | |
| *** coreycb has quit IRC | 03:51 | |
| *** mrhillsman has quit IRC | 03:51 | |
| *** xgerman_ has quit IRC | 03:52 | |
| *** yamamoto has joined #openstack-lbaas | 03:53 | |
| *** yamamoto has quit IRC | 03:59 | |
| *** Eran_Kuris has quit IRC | 04:11 | |
| *** openstack has joined #openstack-lbaas | 04:32 | |
| *** ChanServ sets mode: +o openstack | 04:32 | |
| *** yamamoto has joined #openstack-lbaas | 04:55 | |
| *** yamamoto has quit IRC | 04:58 | |
| *** yamamoto has joined #openstack-lbaas | 04:59 | |
| openstackgerrit | huangshan proposed openstack/python-octaviaclient master: Support backup members https://review.openstack.org/576530 | 05:13 |
| *** lxkong has joined #openstack-lbaas | 05:24 | |
| *** ctracey has joined #openstack-lbaas | 05:27 | |
| *** coreycb has joined #openstack-lbaas | 05:30 | |
| *** bzhao__ has joined #openstack-lbaas | 05:35 | |
| *** johnsom has joined #openstack-lbaas | 05:36 | |
| *** zioproto has joined #openstack-lbaas | 05:40 | |
| *** mnaser has joined #openstack-lbaas | 05:42 | |
| *** fyx has joined #openstack-lbaas | 05:43 | |
| *** beisner has joined #openstack-lbaas | 05:45 | |
| *** kobis has joined #openstack-lbaas | 05:46 | |
| *** beisner has quit IRC | 05:51 | |
| *** mnaser has quit IRC | 05:51 | |
| *** fyx has quit IRC | 05:51 | |
| *** mnaser has joined #openstack-lbaas | 05:51 | |
| *** beisner has joined #openstack-lbaas | 05:51 | |
| *** zioproto has quit IRC | 05:52 | |
| *** beisner has quit IRC | 05:56 | |
| *** mnaser has quit IRC | 05:56 | |
| *** coreycb has quit IRC | 05:57 | |
| *** johnsom has quit IRC | 05:57 | |
| *** lxkong has quit IRC | 05:58 | |
| *** bzhao__ has quit IRC | 05:58 | |
| *** ctracey has quit IRC | 05:59 | |
| *** dims has quit IRC | 06:09 | |
| *** dims has joined #openstack-lbaas | 06:10 | |
| *** mnaser has joined #openstack-lbaas | 06:10 | |
| *** beisner has joined #openstack-lbaas | 06:10 | |
| *** zioproto has joined #openstack-lbaas | 06:11 | |
| *** dims has quit IRC | 06:16 | |
| *** fyx has joined #openstack-lbaas | 06:16 | |
| *** dims has joined #openstack-lbaas | 06:17 | |
| *** mrhillsman has joined #openstack-lbaas | 06:20 | |
| *** xgerman_ has joined #openstack-lbaas | 06:20 | |
| *** Guest29132 has joined #openstack-lbaas | 06:21 | |
| *** bzhao__ has joined #openstack-lbaas | 06:21 | |
| *** coreycb has joined #openstack-lbaas | 06:21 | |
| *** lxkong has joined #openstack-lbaas | 06:21 | |
| *** coreycb has quit IRC | 06:22 | |
| *** coreycb has joined #openstack-lbaas | 06:22 | |
| *** lxkong has quit IRC | 06:22 | |
| *** lxkong has joined #openstack-lbaas | 06:22 | |
| *** ctracey has joined #openstack-lbaas | 06:22 | |
| *** johnsom has joined #openstack-lbaas | 06:22 | |
| *** amitry_ has joined #openstack-lbaas | 06:55 | |
| *** wolsen has joined #openstack-lbaas | 06:55 | |
| *** hogepodge has joined #openstack-lbaas | 07:00 | |
| *** ispp has joined #openstack-lbaas | 07:03 | |
| *** tesseract has joined #openstack-lbaas | 07:04 | |
| *** ispp has quit IRC | 07:18 | |
| *** nmanos has joined #openstack-lbaas | 07:23 | |
| *** nmanos has quit IRC | 07:27 | |
| *** nmanos has joined #openstack-lbaas | 07:29 | |
| *** ktibi has joined #openstack-lbaas | 07:37 | |
| *** yboaron has joined #openstack-lbaas | 07:44 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 07:49 | |
| *** rcernin has quit IRC | 08:03 | |
| *** rpittau is now known as rpittau_ | 08:05 | |
| *** rpittau_ is now known as elfosardo | 08:05 | |
| *** elfosardo is now known as rpittau | 08:05 | |
| *** ispp has joined #openstack-lbaas | 08:08 | |
| *** rpittau is now known as rpittau__ | 08:09 | |
| *** rpittau__ is now known as rpittau | 08:09 | |
| *** rpittau has quit IRC | 08:10 | |
| *** rpittau has joined #openstack-lbaas | 08:10 | |
| *** peereb has joined #openstack-lbaas | 08:10 | |
| *** pcaruana has joined #openstack-lbaas | 08:11 | |
| *** peereb has left #openstack-lbaas | 08:11 | |
| *** peereb has joined #openstack-lbaas | 08:12 | |
| *** peereb has quit IRC | 08:14 | |
| *** peereb has joined #openstack-lbaas | 08:15 | |
| *** peereb has quit IRC | 08:16 | |
| *** peereb has joined #openstack-lbaas | 08:16 | |
| *** peereb has quit IRC | 08:17 | |
| *** peereb has joined #openstack-lbaas | 08:18 | |
| *** peereb has quit IRC | 08:19 | |
| *** peereb has joined #openstack-lbaas | 08:19 | |
| *** peereb has quit IRC | 08:20 | |
| *** peereb has joined #openstack-lbaas | 08:21 | |
| *** peereb has quit IRC | 08:21 | |
| *** rcernin has joined #openstack-lbaas | 08:54 | |
| *** salmankhan has joined #openstack-lbaas | 09:02 | |
| *** cvm has joined #openstack-lbaas | 09:09 | |
| *** kobis has quit IRC | 09:21 | |
| *** aojea_ has joined #openstack-lbaas | 09:42 | |
| *** aojea_ has quit IRC | 09:47 | |
| *** kobis has joined #openstack-lbaas | 09:51 | |
| *** rcernin has quit IRC | 09:56 | |
| *** annp has quit IRC | 09:58 | |
| *** yboaron_ has joined #openstack-lbaas | 10:01 | |
| devfaz | hi, anyone here? | 10:02 |
| *** salmankhan has quit IRC | 10:04 | |
| *** yboaron has quit IRC | 10:04 | |
| *** salmankhan has joined #openstack-lbaas | 10:04 | |
| *** yamamoto has quit IRC | 10:18 | |
| *** cristicalin has joined #openstack-lbaas | 10:18 | |
| *** cristicalin has quit IRC | 10:37 | |
| *** cristicalin has joined #openstack-lbaas | 10:48 | |
| *** cristicalin has quit IRC | 10:53 | |
| *** yboaron_ has quit IRC | 10:58 | |
| devfaz | Im seeing "Amphora X health message reports X listeners when 0 expected", but im unable to locate the given amphora. ?! | 11:15 |
| *** yamamoto has joined #openstack-lbaas | 11:19 | |
| *** cristicalin has joined #openstack-lbaas | 11:20 | |
| *** yamamoto has quit IRC | 11:24 | |
| *** cristicalin has quit IRC | 11:25 | |
| *** yboaron_ has joined #openstack-lbaas | 11:29 | |
| *** longkb has quit IRC | 11:41 | |
| *** atoth has joined #openstack-lbaas | 11:42 | |
| *** yamamoto has joined #openstack-lbaas | 11:49 | |
| *** cristicalin has joined #openstack-lbaas | 11:51 | |
| *** cristicalin has quit IRC | 11:56 | |
| *** frickler has joined #openstack-lbaas | 12:00 | |
| *** yamamoto has quit IRC | 12:00 | |
| *** salmankhan has quit IRC | 12:10 | |
| *** cristicalin has joined #openstack-lbaas | 12:11 | |
| *** yboaron_ has quit IRC | 12:26 | |
| *** cristicalin has quit IRC | 12:29 | |
| *** kman has joined #openstack-lbaas | 12:29 | |
| *** crazik has joined #openstack-lbaas | 12:32 | |
| *** fnaval has joined #openstack-lbaas | 12:38 | |
| *** kman has quit IRC | 12:38 | |
| crazik | hello, two questions about LBaaSv2 with haproxy | 12:39 |
| crazik | 1. is it possible to manually reschedule to other node? | 12:39 |
| crazik | 2. I got an error about rescheduling with non-existient lb ID | 12:40 |
| crazik | I found that in lbaas_loadbalanceragentbindgs tables | 12:40 |
| crazik | table* | 12:40 |
| crazik | can I clean this table from not existing lb | 12:40 |
| crazik | and why is it not cleaning properly? | 12:40 |
| *** yamamoto has joined #openstack-lbaas | 12:51 | |
| *** kobis has quit IRC | 12:54 | |
| *** salmankhan has joined #openstack-lbaas | 13:02 | |
| *** ispp has quit IRC | 13:07 | |
| *** ispp has joined #openstack-lbaas | 13:10 | |
| johnsom | devfaz: we have done some work over the last few months to improve that logging. Basically it is an amphora reporting health that we don’t expect to be there. | 13:14 |
| johnsom | What version are you running? I wonder if we missed a backport. | 13:15 |
| devfaz | thats what I expected from the msg, but Im unable to find the given amphora in the database or even as instance. | 13:15 |
| devfaz | our prod is currently running pike, so the issue is in 1.0.2 | 13:16 |
| johnsom | Yeah, we added logging that gives the IP and amphora ID in the log message | 13:16 |
| johnsom | Ok, when I get in the office I will look to see if it didn’t get backported. | 13:17 |
| devfaz | johnsom: great, in the meantime I try to upgrade the octavia-part to master, but I think this will need some more time/testing. | 13:18 |
| johnsom | If you do a ‘openstack server list —all’ and grep for amphora you will get all of the amphora IDs, you can then compare that list to the Ready and Allocated amphora in the DB. | 13:19 |
| johnsom | Any that are in nova, but not the DB are candidates. If they only have the lb-mgmt-net in nova I usually delete them | 13:20 |
| johnsom | Something happened where Octavia orphaned that amp. Either a failover gone bad or a dirty shutdown of a controller. | 13:21 |
| *** nmanos has quit IRC | 13:23 | |
| devfaz | johnsom: there is no instance running, which is not in the database - other: all running instances are in the database. | 13:24 |
| devfaz | arg.. mom, didnt filter for ready/allocated | 13:24 |
| johnsom | Yeah, it might be in the deleted list | 13:25 |
| devfaz | johnsom: nope, no running instance which should be running. | 13:27 |
| johnsom | We have been working to improve this area of the code. Depending on our results I am half tempted to have Octavia just delete those instances. | 13:28 |
| devfaz | johnsom: think it is much easier to backport your logging-change. | 13:29 |
| johnsom | Hmm, maybe a quick scan to see which ones only have the lb-mgmt-net attached? I don’t know how many amps you have running. | 13:29 |
| devfaz | currently 39, so no very much. | 13:30 |
| johnsom | Yeah, please feel free. Like I said, I am not in the office yet so can’t do that myself yet. It is 6:30am here | 13:30 |
| devfaz | johnsom: Ok, so have a nice cup of coffee ;) | 13:31 |
| johnsom | Plus if you backport it, I can approve it. Grin | 13:31 |
| devfaz | johnsom: even in master the log-msg is the same (in the code). | 13:35 |
| devfaz | # Amphora <UUID> health message reports X listeners when Y expected | 13:35 |
| johnsom | So you have the amphora ID? The nova instances are named with it, it should be in the nova list —all | 13:37 |
| devfaz | yes, but is isnt in the list of nova instances. thats the problem. | 13:37 |
| johnsom | Ok, then I am stumped how an amphora could get booted and configured with that uuid, but not have a nova instance record. Maybe there is a qemu running that nova lost track of? | 13:39 |
| johnsom | I thought there was a patch that added the source IP address to the logs as well. Hmm | 13:40 |
| devfaz | johnsom: looks like I looked at the wrong location. Im getting the same msg on dev (master) and there is an IP in the msg. | 13:46 |
| devfaz | https://github.com/openstack/octavia/commit/1417f6f0f8cbf13fd26440ae2dc53de42873d6cc | 13:49 |
| *** ispp has quit IRC | 13:58 | |
| *** ispp has joined #openstack-lbaas | 14:01 | |
| *** ispp has quit IRC | 14:01 | |
| *** ispp has joined #openstack-lbaas | 14:01 | |
| *** ispp has quit IRC | 14:55 | |
| *** kobis has joined #openstack-lbaas | 15:00 | |
| *** kobis has quit IRC | 15:03 | |
| *** ispp has joined #openstack-lbaas | 15:06 | |
| *** kobis has joined #openstack-lbaas | 15:16 | |
| *** kobis has quit IRC | 15:19 | |
| mnaser | hey | 15:34 |
| mnaser | i think we have to bump the default # | 15:34 |
| mnaser | for local disk | 15:34 |
| mnaser | /dev/vda1 1.9G 1.7G 0 100% / | 15:34 |
| mnaser | in an amphora | 15:34 |
| johnsom | Opps, yeah, we have been trying to reduce the usage in the amps. Though centos is still pretty large. | 15:35 |
| johnsom | I know ubuntu keeps putting more stuff in the cloud image that isn't helping us.... | 15:35 |
| mnaser | i wonder if that can explain some of the health issues | 15:36 |
| johnsom | /dev/vda1 1.9G 1.5G 293M 84% / | 15:37 |
| johnsom | On a fresh amp | 15:37 |
| mnaser | yeah that doesnt leave much breathing room | 15:37 |
| johnsom | Yeah, especially on active load balancers, the flow logging will probably eat that before it rotates and compresses | 15:38 |
| *** ktibi has quit IRC | 15:38 | |
| mnaser | it is a very very active lb | 15:38 |
| *** ktibi has joined #openstack-lbaas | 15:39 | |
| *** links has quit IRC | 15:41 | |
| johnsom | For the stable branches I had to resort to https://review.openstack.org/#/c/569531/ removing snapd | 15:41 |
| johnsom | Since I couldn't backport the switch to ubuntu-minimal | 15:42 |
| johnsom | due to stable branch policy | 15:42 |
| mnaser | we'll bump it here to 20 | 15:44 |
| mnaser | but i think we should increase it upstream in os_octavia to like 10 or something | 15:44 |
| mnaser | storage is cheap... | 15:45 |
| johnsom | Ha, that is not the feedback we get. People are pissed over the 2GB | 15:53 |
| johnsom | which is thin provisioned.. ha | 15:54 |
| johnsom | There is a patch up that disables local logging as well that would help. | 15:55 |
| mnaser | that's silly, 2gb.. | 15:55 |
| mnaser | that's pennies.. | 15:55 |
| johnsom | We hoped to have those logs being shipped off the amp in Rocky too, but just didn't have the developers | 15:55 |
| johnsom | mnaser https://review.openstack.org/566741 | 15:56 |
| *** kobis has joined #openstack-lbaas | 15:56 | |
| mnaser | johnsom: yay problem #2 | 16:04 |
| mnaser | scale testing octavia | 16:04 |
| mnaser | 188.114.110.22:13195 [20/Jun/2018:16:02:31.314] <snip> 225/0/0/1/226 200 2065 - - ---- 2000/2000/1/1/0 0/0 "GET <snip> HTTP/1.1" | 16:04 |
| mnaser | 2000/2000 | 16:04 |
| mnaser | can we adjust maxconn? | 16:04 |
| johnsom | mnaser that is user setting on the listener | 16:04 |
| johnsom | mnaser --connection_limit | 16:05 |
| mnaser | ugh sweet, thank you so much | 16:05 |
| mnaser | but so here's the thing | 16:05 |
| mnaser | connection_limit=-1 | 16:05 |
| johnsom | There is an open bug about -1 being 2000 on the amphora driver. Darn, forgot about that. I need to put that on the priority bug list | 16:05 |
| mnaser | is a lie? | 16:06 |
| mnaser | haha | 16:06 |
| mnaser | yes. | 16:06 |
| mnaser | i can take care of that | 16:06 |
| johnsom | I will find the story for you | 16:06 |
| johnsom | mnaser https://storyboard.openstack.org/#!/story/1635416 | 16:07 |
| johnsom | Since we have released the API we really need to make -1 work and not change the default to 2000 | 16:08 |
| johnsom | This was an odd one because what -1 or missing means to haproxy is dependent on how it was compiled, so it varies package to package | 16:10 |
| johnsom | Thus we missed that it can sometimes mean just 2000 | 16:10 |
| johnsom | Good news is setting that listener setting will immediate resolve it | 16:10 |
| *** crazik has quit IRC | 16:11 | |
| mnaser | johnsom: are we still at a stage where this is the case? | 16:11 |
| mnaser | that was in 2016 afterall :p | 16:11 |
| johnsom | Sigh | 16:12 |
| *** yamamoto has quit IRC | 16:12 | |
| johnsom | It's been a rough few releases for us with our goals and then people disappearing. | 16:13 |
| johnsom | This one just got lost in the shuffle | 16:13 |
| mnaser | johnsom: no worries, i get that :( | 16:14 |
| mnaser | i think ideally it's defaulting to 2000 | 16:14 |
| johnsom | It is up to the packager. You can see what they compiled in default is with "/usr/sbin/haproxy -vv" | 16:15 |
| *** ispp has quit IRC | 16:15 | |
| mnaser | the docs don't seem to explain a default value | 16:15 |
| mnaser | and i think -1 is a good way to shoot yourself in the foot | 16:15 |
| johnsom | Yeah, so thanks if you take that story | 16:15 |
| johnsom | Yeah, but some vendors do support -1 as unlimited so... | 16:16 |
| mnaser | oh i know | 16:16 |
| mnaser | i have a user that wants to run at -1 | 16:16 |
| mnaser | haproxy -vv shows maxconn = 2000 for default | 16:16 |
| johnsom | -1 as an option is necessary and we need to translate that to haproxy speak. | 16:16 |
| johnsom | Yeah, that is what I have on ubuntu too | 16:16 |
| johnsom | It's the DEFAULT_MAXCONN option at compile time | 16:17 |
| mnaser | can we set -1 in the haproxy config file | 16:17 |
| johnsom | I'm not sure, but I don't think so. | 16:17 |
| *** yamamoto has joined #openstack-lbaas | 16:22 | |
| *** yamamoto has quit IRC | 16:27 | |
| *** kobis has quit IRC | 16:31 | |
| *** ramishra has quit IRC | 16:32 | |
| *** tesseract has quit IRC | 16:34 | |
| *** yamamoto has joined #openstack-lbaas | 16:37 | |
| *** yamamoto has quit IRC | 16:41 | |
| johnsom | You know it's getting serious when I need to clone a copy of the taskflow code.... | 16:43 |
| *** yamamoto has joined #openstack-lbaas | 16:52 | |
| *** yamamoto has quit IRC | 16:56 | |
| *** crazik has joined #openstack-lbaas | 16:58 | |
| *** crazik has quit IRC | 16:59 | |
| *** crazik has joined #openstack-lbaas | 16:59 | |
| *** atoth has quit IRC | 17:01 | |
| *** yamamoto has joined #openstack-lbaas | 17:07 | |
| rm_work | mnaser / johnsom: we could probably just set -1 to whatever the largest number that haproxy supports is | 17:08 |
| * rm_work shrugs | 17:08 | |
| rm_work | at some point it doesn't matter if it's exact | 17:08 |
| johnsom | Yeah | 17:08 |
| mnaser | rm_work: thanks for pushing up that queens backport | 17:09 |
| rm_work | yep :) | 17:10 |
| rm_work | now we can do Pike! :) | 17:10 |
| rm_work | (I think) | 17:10 |
| crazik | hello, two questions about LBaaSv2 with haproxy: 1. is it possible to manually reschedule to other node? 2. I got an error about rescheduling with non-existient lb ID.I found that in lbaas_loadbalanceragentbindgs table. can I clean this table from not existing lb and why is it not cleaning properly? | 17:11 |
| *** yamamoto has quit IRC | 17:12 | |
| *** atoth has joined #openstack-lbaas | 17:14 | |
| crazik | well, that was 3 questions in fact. | 17:15 |
| rm_work | crazik: you can trigger a "failover" as an administrator, which will manually force it to move the active node | 17:15 |
| rm_work | oh, wait | 17:15 |
| johnsom | rm_work He isn't using octavia | 17:16 |
| crazik | still haproxy. | 17:16 |
| crazik | octavia work in progress ;) | 17:16 |
| rm_work | are you talking about *neutron-lbaas* | 17:16 |
| rm_work | with the namespace driver? | 17:16 |
| crazik | I think so. | 17:16 |
| *** salmankhan has quit IRC | 17:16 | |
| rm_work | augh. then, no idea | 17:16 |
| crazik | blah. | 17:17 |
| johnsom | crazik 1, no I don't think so. 2. I don't know. 3. Don't know either. I don't use the namespace driver | 17:17 |
| crazik | ;) | 17:17 |
| crazik | johnsom: thank you | 17:17 |
| johnsom | Sorry, I don't think there are many folks around anymore that use that. | 17:17 |
| rm_work | do people still use that? besides crazik I guess | 17:17 |
| rm_work | yeah hmm | 17:17 |
| crazik | well, what do you mean as namespace driver? | 17:17 |
| crazik | this one which creates haproxy instances inside namespaces? | 17:18 |
| rm_work | yes | 17:18 |
| rm_work | on the agent hosts I think? | 17:18 |
| crazik | correct | 17:18 |
| crazik | so, It's my case | 17:18 |
| rm_work | yeah, that was deprecated years ago I think | 17:18 |
| crazik | lbaasv1 was deprecated | 17:18 |
| crazik | v2 wasn't at Ocata afair | 17:19 |
| rm_work | hmm, lbaasv1 was completely removed by now, but i thought we at least deprecated (not removed yet though) namespace-haproxy | 17:19 |
| rm_work | but it does make sense that it's still commonly in use | 17:19 |
| rm_work | it's *very* simple | 17:19 |
| crazik | yeah | 17:20 |
| rm_work | we just don't know much about it :( | 17:20 |
| crazik | I was trying to setup both: haproxy +ocatvia together, but had some issues. | 17:20 |
| crazik | now still thinking about way to migrate | 17:20 |
| rm_work | the entire current team started working on lbaas after it was essentially abandoned :( | 17:20 |
| rm_work | I mean, after the namespace driver was -- not the project | 17:21 |
| crazik | well, I wish I had a clean start and could use Octavia | 17:21 |
| rm_work | yeah, not always possible :( | 17:21 |
| rm_work | I know, my cloud is still on liberty :P | 17:21 |
| crazik | yeah. there is mention about migration toools on roadmap | 17:21 |
| rm_work | yeah, that is started | 17:22 |
| rm_work | there is a WIP up I believe from johnsom | 17:22 |
| crazik | any unofficial tools are available? | 17:22 |
| *** yamamoto has joined #openstack-lbaas | 17:22 | |
| rm_work | but you can also just start running Octavia as a separate service entirely, and run both at the same time | 17:22 |
| rm_work | that is what we did actually | 17:22 |
| crazik | as I said - I had some issues | 17:22 |
| rm_work | i mean, completely disconnected | 17:22 |
| rm_work | continue running neutron-lbaas with the namespace backend, and run octavia as a separate service/endpoint | 17:23 |
| rm_work | don't try to pull octavia in to neutron-lbaas | 17:23 |
| johnsom | The migration tool works for neutron-lbaas load balancers using octavia provider, but not others yet. It also does not migrate from one provider to another | 17:23 |
| *** AlexeyAbashkin has quit IRC | 17:23 | |
| rm_work | we ran them in parallel for a while, got people to migrate themselves in time, and then eventually EOL'd neutron-lbaas | 17:23 |
| crazik | I got problem with horizon to use them in parallel | 17:24 |
| rm_work | ahh yeah | 17:24 |
| rm_work | the horizon dashboards conflicting :/ | 17:24 |
| rm_work | that is a dumb problem, i am still not clear why we can't just run both dashboards >_< | 17:24 |
| rm_work | but yeah, it doesn't work :/ | 17:24 |
| crazik | ;) | 17:24 |
| johnsom | Should work, I did it once | 17:24 |
| rm_work | we don't use horizon here so it was not a problem for us | 17:24 |
| johnsom | The only odd thing was they both had the same title | 17:25 |
| crazik | next issue I had, was that amphoras needs ephemeral disk | 17:25 |
| rm_work | oh yeah, maybe it was just a problem when it was the old/new neutron-lbaas-dashboard | 17:25 |
| crazik | I have only volume-based instances | 17:25 |
| johnsom | rm_work Any way you could possibly ask harlowj a taskflow question? | 17:25 |
| johnsom | Or has he sworn off OpenStack for good now? | 17:26 |
| rm_work | he's off this week | 17:26 |
| rm_work | but certainly | 17:26 |
| *** yamamoto has quit IRC | 17:27 | |
| johnsom | Bummer, ok. Trying to figure out how subflow revert / retry works. I want a sub-flow to revert, but not revert up the parent. | 17:28 |
| rm_work | he'll be back hmmm | 17:28 |
| rm_work | next week | 17:28 |
| *** links has joined #openstack-lbaas | 17:37 | |
| *** links has quit IRC | 17:39 | |
| *** atoth has quit IRC | 17:39 | |
| *** atoth has joined #openstack-lbaas | 17:39 | |
| *** links has joined #openstack-lbaas | 17:39 | |
| *** yamamoto has joined #openstack-lbaas | 17:52 | |
| *** yamamoto has quit IRC | 17:57 | |
| *** yamamoto has joined #openstack-lbaas | 18:02 | |
| *** yamamoto has quit IRC | 18:02 | |
| *** links has quit IRC | 18:11 | |
| *** links has joined #openstack-lbaas | 18:12 | |
| *** links has quit IRC | 18:14 | |
| *** Swami_ has joined #openstack-lbaas | 18:20 | |
| *** Swami has joined #openstack-lbaas | 18:20 | |
| *** yamamoto has joined #openstack-lbaas | 18:36 | |
| *** ianychoi has quit IRC | 18:36 | |
| *** atoth has quit IRC | 18:40 | |
| *** yamamoto has quit IRC | 18:41 | |
| openstackgerrit | Merged openstack/octavia master: Align logging on oslo_log https://review.openstack.org/575931 | 18:49 |
| *** yamamoto has joined #openstack-lbaas | 18:51 | |
| *** yamamoto has quit IRC | 18:56 | |
| *** aojea_ has joined #openstack-lbaas | 19:03 | |
| *** fnaval has quit IRC | 19:03 | |
| *** yamamoto has joined #openstack-lbaas | 19:06 | |
| *** fnaval has joined #openstack-lbaas | 19:08 | |
| *** fnaval has quit IRC | 19:08 | |
| *** fnaval has joined #openstack-lbaas | 19:09 | |
| *** yamamoto has quit IRC | 19:10 | |
| *** aojea_ has quit IRC | 19:16 | |
| *** yamamoto has joined #openstack-lbaas | 19:22 | |
| *** yamamoto has quit IRC | 19:26 | |
| *** yamamoto has joined #openstack-lbaas | 19:37 | |
| *** salmankhan has joined #openstack-lbaas | 19:41 | |
| *** yamamoto has quit IRC | 19:43 | |
| *** ktibi has quit IRC | 19:57 | |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Jun 20 20:00:08 2018 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| cgoncalves | hi | 20:00 |
| nmagnezi | o/ | 20:00 |
| johnsom | Hi folks | 20:00 |
| johnsom | Another hot and sunny day here | 20:00 |
| xgerman_ | o/ | 20:00 |
| johnsom | #topic Announcements | 20:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:01 | |
| johnsom | I wanted to highlight that I have setup the Rocky priority patch/review list | 20:01 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-priority-reviews | 20:01 |
| johnsom | I am trying to update this daily with progress on the patches | 20:01 |
| johnsom | If you have some time please try to help review and / or work on patches | 20:02 |
| nmagnezi | Is it okay for others to update this as well? (In case I noticed updates) | 20:02 |
| johnsom | It is a decent sized list, but we are making some progress. | 20:02 |
| johnsom | nmagnezi Yes please! | 20:02 |
| nmagnezi | ack | 20:03 |
| johnsom | We are heading towards feature freeze and MS3 | 20:03 |
| nmagnezi | I have a question about those python 3 patches, but let's take it to the open discussion part | 20:03 |
| johnsom | MS3 and Feature freeze is the week of July 23!!!! | 20:04 |
| johnsom | Ok, sounds good. | 20:04 |
| johnsom | The only other announcement I had is about the upcoming PTG | 20:04 |
| johnsom | We will have a room for Octavia. I will be attending | 20:04 |
| johnsom | If you need travel assistance, the foundation round 1 closes July 1st. | 20:05 |
| johnsom | So apply soon | 20:05 |
| johnsom | Any other announcements today? | 20:05 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:06 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:06 | |
| johnsom | I have been busy trying to squash bugs. There have been a few little ones and tech debt. | 20:06 |
| johnsom | I am currently focused on fixing the dual amphora failure bug that causes failover to no be fully successful. | 20:07 |
| nmagnezi | Link? :) | 20:07 |
| johnsom | I have a plan, a PoC task and parallel flow. I just need to finish out the rest of the failover flow | 20:08 |
| johnsom | #link https://storyboard.openstack.org/#!/story/2001481 | 20:08 |
| johnsom | Basically if more than one amphora is down/failed, the failover flow will not fully bring up the LB. One amp will be rebuilt, but the LB will be in provisioning ERROR | 20:09 |
| *** rm_mobile has joined #openstack-lbaas | 20:09 | |
| rm_mobile | o/ | 20:10 |
| johnsom | Hopefully I can get a patch posted today for it. | 20:10 |
| * rm_mobile is running late | 20:10 | |
| nmagnezi | Put it in the prio list :) | 20:10 |
| johnsom | It is, under "Priority Patches (These need work or are WIP)" | 20:10 |
| johnsom | Line 45 | 20:10 |
| johnsom | Any other progress updates from the team? | 20:11 |
| *** rm_mobile has quit IRC | 20:12 | |
| *** rm_mobile has joined #openstack-lbaas | 20:12 | |
| johnsom | FYI, for those of you interested in the stable branch backporting, this patch would be good to backport. Someone was asking about it this morning. It will need some minor work to cherrypick though. | 20:12 |
| johnsom | #link https://review.openstack.org/#/c/561369/ | 20:12 |
| cgoncalves | not sure it was reported last week: grenade support is merged | 20:13 |
| johnsom | So if someone is motivated.... | 20:13 |
| johnsom | Yes! Excellent work. | 20:13 |
| johnsom | Anybody have comments on making that voting? | 20:13 |
| xgerman_ | if nobody os motivated I can grab that | 20:13 |
| johnsom | I think we need to have it voting and an upgrade procedure doc and we can assert our tag(s) | 20:13 |
| xgerman_ | yeah, let’s #vote on voting | 20:14 |
| cgoncalves | xgerman_, go for it. propose the backport | 20:14 |
| xgerman_ | k | 20:14 |
| johnsom | Ha, I would only call a formal vote on this if Doug was here. Just to bug him | 20:14 |
| cgoncalves | johnsom, I started the upgrade procedure doc but am out traveling this week | 20:15 |
| johnsom | I think I will look at the job history and propose a voting patch | 20:15 |
| johnsom | cgoncalves Ok, cool! | 20:15 |
| johnsom | Any other updates or should we move on? | 20:16 |
| johnsom | #topic Talk about API versioning/microversioning | 20:17 |
| *** openstack changes topic to "Talk about API versioning/microversioning (Meeting topic: Octavia)" | 20:17 | |
| johnsom | We opened this topic a few weeks ago. | 20:17 |
| * nmagnezi looks at rm_work | 20:17 | |
| johnsom | Any new thoughts on this? Have people had time to look at the reference links? | 20:18 |
| * rm_work looks at nmagnezi | 20:18 | |
| johnsom | #link https://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html | 20:19 |
| rm_work | oh right i was going to look up stuff | 20:19 |
| johnsom | #link https://docs.openstack.org/tempest/latest/microversion_testing.html#step4-separate-test-classes-for-each-microversion | 20:19 |
| * rm_work looks at the floor, which is where that task ended up | 20:19 | |
| rm_work | johnsom: oh fff no | 20:19 |
| rm_work | bleh | 20:19 |
| johnsom | As I mentioned last week, I lean towards not doing microversioning, but to increment the API version document with dot versions and only do additions to the API. | 20:20 |
| cgoncalves | I have not. I looked +1 years ago to nova on their microversion implementation. we need some sort of versioning support for rolling upgrades, for sure | 20:20 |
| xgerman_ | +! | 20:20 |
| johnsom | Yeah, it's the clients that are becoming the bigger issue | 20:20 |
| johnsom | clients don't know if they can ask for features | 20:21 |
| johnsom | Ok, so | 20:21 |
| * johnsom glares at the room | 20:21 | |
| nmagnezi | What other projects are doing when the client asks for a feature that is not supported in a specific version? | 20:22 |
| johnsom | Let's all take a look at those and have thoughts/ideas/comments ready for next week. | 20:22 |
| johnsom | Likely the same 404 we do | 20:22 |
| cgoncalves | have api versioning made to a community goal for rocky? | 20:22 |
| johnsom | No | 20:23 |
| johnsom | The goals for Rocky are no mox (we don't) and enable mutable configs | 20:23 |
| johnsom | Stein is likely going to be py3 and something I forgot. | 20:23 |
| rm_work | sometimes i think the extensions thing neutron did seems like a sane way to approach this. and then I flip my table and cry in a corner for a while. | 20:23 |
| xgerman_ | +1 as much as I hate extensions they seem a bit better than this microversioning | 20:24 |
| johnsom | Can't say I'm a huge fan of the extensions | 20:24 |
| nmagnezi | Will It make sense to do some sort of version discovery behind the scenes for specific client actions that we know not all versions support? And in that case we can mask the error and output something like "this is not supported by the current API version" | 20:24 |
| rm_work | for now i kinda just want to keep doing | 20:25 |
| rm_work | the thing | 20:25 |
| rm_work | that johnsom said | 20:25 |
| johnsom | Oh, The other Stein goal that has traction is cold upgrade | 20:25 |
| nmagnezi | For example action like amp failover that was added in queens IIRC | 20:25 |
| xgerman_ | well, if you returned 404 before you are retro-actively changing API behavior | 20:25 |
| rm_work | <johnsom>As I mentioned last week, I lean towards not doing microversioning, but to increment the API version document with dot versions and only do additions to the API. | 20:26 |
| cgoncalves | johnsom, cold upgrade: check! :) | 20:26 |
| johnsom | cgoncalves Yep! | 20:26 |
| johnsom | So, if I update my version discovery patch and update the version are we good with that? leave the path /2.0 and have the multiple dot versions? | 20:27 |
| johnsom | I think I will do that as a proposed patch. Then we can discuss again next week and make a call. | 20:27 |
| rm_work | I think so IMO | 20:27 |
| xgerman_ | +1 | 20:28 |
| nmagnezi | +1 | 20:28 |
| cgoncalves | #link https://review.openstack.org/#/c/559460/ | 20:28 |
| johnsom | Yeah, that needs fixed though. I think I don't like how it is now | 20:29 |
| johnsom | #topic Open Discussion | 20:29 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:29 | |
| johnsom | nmagnezi I think you had a py3 queston | 20:30 |
| johnsom | question | 20:30 |
| nmagnezi | yup | 20:30 |
| nmagnezi | So re: https://review.openstack.org/#/c/573348/ | 20:30 |
| *** salmankhan has quit IRC | 20:30 | |
| *** aojea has joined #openstack-lbaas | 20:31 | |
| nmagnezi | hSo re: https://review.openstack.org/#/c/573348/ | 20:31 |
| nmagnezi | once that merged | 20:31 |
| nmagnezi | Does that mean we gate on python 3 *only* ? | 20:31 |
| nmagnezi | Just double checking myself here | 20:31 |
| johnsom | It means that those jobs will run under py3, but we will still have py27 unit, functional, tempest | 20:32 |
| johnsom | So, pep8 will run under py3, cover, docs, releasenotes, and the debug. | 20:32 |
| nmagnezi | But for pep8 and cover it will run only with python3 right? | 20:32 |
| nmagnezi | yeah | 20:33 |
| johnsom | Basically they want to make py27 the exception instead of py3 being the exception as it is today | 20:33 |
| johnsom | Correct, this change moves those over to run only under py3 | 20:33 |
| rm_work | yeah A++ | 20:33 |
| johnsom | This is lining things up for the proposed Stein goal of all projects running py3 jobs as a minimum | 20:34 |
| nmagnezi | yeah. I just wonder because we still gate on python 2.x on RDO and internally | 20:34 |
| johnsom | Yeah, the important jobs: unit tests, functional, and tempest will still run both py27 and py3 to show that our code works under both | 20:35 |
| johnsom | For the near future. Eventually OpenStack will drop py2.7 support. | 20:35 |
| rm_work | in U at the earliest | 20:35 |
| nmagnezi | Got it. Thanks for the answers :) | 20:36 |
| johnsom | #link https://pythonclock.org/ | 20:36 |
| johnsom | This is being driven by the fact that python 2.7 itself is going end-of-life in a year and a half | 20:36 |
| johnsom | Well, 1 year, 6 months, 11 days, 9 hours, 22 minutes | 20:37 |
| nmagnezi | Haha | 20:37 |
| nmagnezi | I trust them to start Python 4.x just for having two versions | 20:38 |
| johnsom | I will just be happy to only have to deal with one version | 20:38 |
| johnsom | lol, yeah, probably | 20:38 |
| xgerman_ | yep | 20:38 |
| johnsom | Ok, any other discussions for today? | 20:38 |
| xgerman_ | but by then we might have rewritten OpenStack in golang | 20:38 |
| johnsom | C#? | 20:39 |
| nmagnezi | xgerman_, dougwig once wanted to port neutron-lbaas to Ruby.. :-) | 20:39 |
| rm_work | as an aside, python recommends not doing version detection :P https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection | 20:39 |
| nmagnezi | johnsom, fortran | 20:39 |
| rm_work | (related to the microversioning discussion) | 20:39 |
| *** yamamoto has joined #openstack-lbaas | 20:39 | |
| rm_work | I recommend we rewrite the amp-agent in Rust | 20:40 |
| johnsom | Hey, fortran is not a dead language.... | 20:40 |
| rm_work | Pascal? | 20:40 |
| johnsom | Oye, pascal, now we are going down hill. Might as well throw out perl or LISP | 20:40 |
| rm_work | oooh, we had to learn J in uni... | 20:40 |
| rm_work | https://en.wikipedia.org/wiki/J_(programming_language) | 20:40 |
| rm_work | APL but typable by normal humans with normal keyboards | 20:41 |
| johnsom | Sadly I have coded in all of the above except for rust | 20:41 |
| johnsom | at one point or another | 20:41 |
| xgerman_ | Rust is new | 20:41 |
| *** pcaruana has quit IRC | 20:41 | |
| johnsom | Ok, if nothing else I will let you fine folks go work on reviews. Please help with those, MS3 is coming quick. | 20:41 |
| xgerman_ | https://en.wikipedia.org/wiki/System_programming_language#Major_languages | 20:42 |
| rm_work | pretty sure http://www.cs.trinity.edu/old-index.cgi is written in J | 20:42 |
| xgerman_ | take your pick | 20:42 |
| rm_work | alright. I'm ... temporarily kinda semi-away from octavia for a few weeks, ping me for reviews or if there's patches that I need to update | 20:43 |
| rm_work | :( | 20:43 |
| xgerman_ | ok | 20:43 |
| johnsom | Ok, thanks folks! | 20:44 |
| johnsom | #endmeeting | 20:44 |
| *** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) | https://etherpad.openstack.org/p/octavia-priority-reviews" | 20:44 | |
| openstack | Meeting ended Wed Jun 20 20:44:28 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:44 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-06-20-20.00.html | 20:44 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-06-20-20.00.txt | 20:44 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-06-20-20.00.log.html | 20:44 |
| *** yamamoto has quit IRC | 20:44 | |
| *** rm_mobile has quit IRC | 21:04 | |
| *** yamamoto has joined #openstack-lbaas | 21:41 | |
| *** yamamoto has quit IRC | 21:47 | |
| *** rcernin has joined #openstack-lbaas | 22:10 | |
| *** fnaval has quit IRC | 22:23 | |
| *** aojea has quit IRC | 22:36 | |
| *** yamamoto has joined #openstack-lbaas | 22:43 | |
| *** yamamoto has quit IRC | 22:49 | |
| *** rcernin has quit IRC | 22:50 | |
| johnsom | Ok, got a working dual amp down failover. There is a bunch of flow rework that could be done to parallelize better, but I want a small patch we can backport for the bug. We can do that optimization later. | 22:56 |
| johnsom | On my workstation it's 29 seconds to LB functional again, another minute for the rest of the first amp cleanup to finish, then another 34 seconds until the second amp is fully restored. | 22:58 |
| johnsom | So about two minutes and seven seconds wall clock time to restore full redundancy after both amps were nuked. But the important number is the 29 seconds to LB functional again. | 22:59 |
| johnsom | Thirty seconds of that is retrying to connect to the secondary in case it was just a network blip. On the fence if we should retry that many times on the secondary or if we should fail it faster | 23:02 |
| *** rcernin has joined #openstack-lbaas | 23:43 | |
| *** yamamoto has joined #openstack-lbaas | 23:45 | |
| *** Swami has quit IRC | 23:47 | |
| *** Swami_ has quit IRC | 23:47 | |
| *** yamamoto has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!