| rm_work | night | 00:00 |
|---|---|---|
| rm_work | alright got distracted but back to trying to do that test | 00:00 |
| rm_work | trying to write sensu checks :/ | 00:01 |
| imacdonn | OK... lemme know if I should create a story ... I may have another thorny one coming after this one ;) | 00:02 |
| *** jdavis has joined #openstack-lbaas | 00:08 | |
| *** jdavis has quit IRC | 00:08 | |
| *** slaweq has joined #openstack-lbaas | 00:14 | |
| *** slaweq has quit IRC | 00:19 | |
| *** slaweq has joined #openstack-lbaas | 00:25 | |
| *** slaweq has quit IRC | 00:29 | |
| *** slaweq has joined #openstack-lbaas | 00:35 | |
| rm_work | yeah make stories | 00:39 |
| *** ipsecguy has quit IRC | 00:39 | |
| *** slaweq has quit IRC | 00:40 | |
| dayou | wake up and start making things :D | 00:42 |
| *** slaweq has joined #openstack-lbaas | 00:45 | |
| *** slaweq has quit IRC | 00:50 | |
| *** ipsecguy has joined #openstack-lbaas | 00:51 | |
| *** slaweq has joined #openstack-lbaas | 00:55 | |
| rm_work | imacdonn: link me the story for that when you make it | 01:00 |
| *** slaweq has quit IRC | 01:00 | |
| imacdonn | will do | 01:02 |
| *** slaweq has joined #openstack-lbaas | 01:05 | |
| *** slaweq has quit IRC | 01:10 | |
| *** slaweq has joined #openstack-lbaas | 01:16 | |
| rm_work | curious what your other thing is :) | 01:19 |
| rm_work | i don't know how many people actually use the tls stuff yet | 01:19 |
| rm_work | i have this in production but we have tls disabled :/ | 01:19 |
| rm_work | and most other deployers seem to be using neutron-lbaas still | 01:20 |
| rm_work | which has the correct logic in front, I am fairly certain | 01:20 |
| *** slaweq has quit IRC | 01:20 | |
| rm_work | imacdonn: i'm literally waiting for a story link to put up a review with a fix :) | 01:22 |
| rm_work | meanwhile my laptop lights my pants on fire running functional tests | 01:23 |
| imacdonn | oh, sorry | 01:24 |
| imacdonn | will get it done now .. too many shiny objects | 01:24 |
| *** slaweq has joined #openstack-lbaas | 01:26 | |
| rm_work | heh yeah | 01:26 |
| rm_work | i've been alternating between 3-4 things all day | 01:26 |
| *** slaweq has quit IRC | 01:30 | |
| imacdonn | rm_work: https://storyboard.openstack.org/#!/story/2001640 | 01:33 |
| imacdonn | sorry about the formatting .. haven't figured out markdown yet | 01:33 |
| rm_work | lol no worries, thanks | 01:34 |
| *** slaweq has joined #openstack-lbaas | 01:36 | |
| *** slaweq has quit IRC | 01:41 | |
| rm_work | ok | 01:45 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Properly test access to tls_refs in the API layer https://review.openstack.org/550303 | 01:45 |
| rm_work | that should take care of it ... as a first pass anyway, might want to redo the exception text or something, dunno | 01:46 |
| *** slaweq has joined #openstack-lbaas | 01:46 | |
| rm_work | of course it's going to fail all the gates because our gates are broken at the moment | 01:47 |
| rm_work | :( | 01:47 |
| rm_work | brb | 01:48 |
| imacdonn | heh ok | 01:49 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 01:49 | |
| rm_work | if you want to apply+test that patch in your env, that would be helpful | 01:50 |
| imacdonn | yes, looking at it now | 01:50 |
| *** slaweq has quit IRC | 01:51 | |
| *** sanfern has joined #openstack-lbaas | 01:53 | |
| *** AlexeyAbashkin has quit IRC | 01:53 | |
| *** slaweq has joined #openstack-lbaas | 01:56 | |
| *** dayou has quit IRC | 01:58 | |
| imacdonn | I'm currently temporarily stuck in anothet icky situation | 01:58 |
| imacdonn | an issue with my amphora (which I'll get into later) causes it to be unreachable on the network | 01:58 |
| imacdonn | by default, we retry to connect 300 times, every 15 seconds, it appears | 01:59 |
| imacdonn | so it's going to take about 1h15m to fail | 01:59 |
| imacdonn | until then, as far as I can see, I have no way to stop it | 01:59 |
| *** slaweq has quit IRC | 02:01 | |
| *** imacdonn has quit IRC | 02:02 | |
| *** imacdonn has joined #openstack-lbaas | 02:03 | |
| rm_work | lol | 02:05 |
| rm_work | that's ... accurate | 02:05 |
| rm_work | you might want to change the defaults to match your setup | 02:05 |
| rm_work | probably you did after that? :P | 02:06 |
| rm_work | it's not that big of a deal if you want to just clean it up | 02:06 |
| rm_work | if you restart the worker, it'll lose the state | 02:06 |
| rm_work | you can change the amphora record's status to ERROR in the DB | 02:06 |
| *** slaweq has joined #openstack-lbaas | 02:06 | |
| rm_work | and then you can delete the LB | 02:07 |
| rm_work | imacdonn: ^^ | 02:07 |
| rm_work | ah and change the LB's provisioning_status to ERROR also | 02:07 |
| imacdonn | I think I've gotten myself in trouble doing things like that before | 02:08 |
| rm_work | actually, just restarting the worker and changing the LB to ERROR should do it | 02:08 |
| rm_work | yeah, I know this works though | 02:08 |
| rm_work | and even if it didn't, I know how to clean up every object | 02:08 |
| rm_work | (manually) | 02:08 |
| rm_work | we basically create: one VIP port per LB, and then per amp: one VM, one mgmt port, one vrrp port, one security group | 02:09 |
| imacdonn | my usual "dig myself out of a hole" is to drop the DB and create a new one, then clean up the amps, ports, SG | 02:10 |
| rm_work | all of the names are predictable as long as you know the amp_id and the lb_id / listener_id | 02:11 |
| imacdonn | obv not suitable for production ;) | 02:11 |
| rm_work | yes | 02:11 |
| rm_work | I am assuming you're not testing this in prod :P | 02:11 |
| imacdonn | correct | 02:11 |
| *** slaweq has quit IRC | 02:11 | |
| rm_work | but, learning the manual cleanup procedure will serve you well if/when stuff does explode in prod later :) | 02:12 |
| *** dayou has joined #openstack-lbaas | 02:12 | |
| rm_work | i had an issue with my HMs (partially caused by me doing something dumb) that caused me to have to rebuild the whole amp table by hand from nova/neutron records | 02:12 |
| *** harlowja has quit IRC | 02:13 | |
| rm_work | in prod <_< it took me about 4 hours | 02:13 |
| imacdonn | sounds like fun | 02:13 |
| rm_work | it was character building :) | 02:13 |
| imacdonn | :) | 02:13 |
| rm_work | accidentally started my housekeeping process before I meant to, and a combination of two bugs (one fix merged, one still pending) caused the whole table to be wiped :/ | 02:15 |
| imacdonn | this time, it was sufficient to restart the worker and set the LB provisioning_status='ERROR' .. then was able to delete it | 02:15 |
| rm_work | yep | 02:15 |
| *** slaweq has joined #openstack-lbaas | 02:17 | |
| imacdonn | [imacdonn@home ~]$ openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container="${CONTAINER}" lb1 | 02:21 |
| imacdonn | Could not retrieve certificate: ['http://slc10jtj.dcilab.oraclecorp.com:9311/v1/containers/be0e731c-b8d9-466c-9b20-3595457187ab'] (HTTP 400) (Request-ID: req-046fe49e-ffb9-4d81-806b-6ef74ec2d809) | 02:21 |
| imacdonn | phew! | 02:21 |
| *** slaweq has quit IRC | 02:21 | |
| imacdonn | also confirmed it works with a container that it can access | 02:21 |
| imacdonn | guess should also check if the container is accessible but one of the items in it is not | 02:22 |
| *** slaweq has joined #openstack-lbaas | 02:27 | |
| *** slaweq has quit IRC | 02:32 | |
| *** slaweq has joined #openstack-lbaas | 02:37 | |
| *** fnaval has joined #openstack-lbaas | 02:37 | |
| *** slaweq has quit IRC | 02:42 | |
| *** annp has joined #openstack-lbaas | 02:44 | |
| *** slaweq has joined #openstack-lbaas | 02:47 | |
| *** slaweq has quit IRC | 02:52 | |
| *** slaweq has joined #openstack-lbaas | 02:57 | |
| openstackgerrit | Min Sun proposed openstack/octavia-dashboard master: Cannot update ssl certificate when update listener https://review.openstack.org/550313 | 03:01 |
| *** slaweq has quit IRC | 03:03 | |
| *** slaweq has joined #openstack-lbaas | 03:08 | |
| *** slaweq has quit IRC | 03:12 | |
| *** slaweq has joined #openstack-lbaas | 03:18 | |
| *** slaweq has quit IRC | 03:22 | |
| *** sanfern has quit IRC | 03:33 | |
| *** slaweq has joined #openstack-lbaas | 03:48 | |
| *** slaweq has quit IRC | 03:53 | |
| *** links has joined #openstack-lbaas | 03:59 | |
| *** slaweq has joined #openstack-lbaas | 03:59 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Add rbac support for octavia service apis https://review.openstack.org/550319 | 04:00 |
| *** slaweq has quit IRC | 04:03 | |
| *** links has quit IRC | 04:25 | |
| *** links has joined #openstack-lbaas | 04:27 | |
| *** atoth has quit IRC | 04:33 | |
| *** slaweq has joined #openstack-lbaas | 04:39 | |
| *** slaweq has quit IRC | 04:44 | |
| *** slaweq has joined #openstack-lbaas | 04:49 | |
| *** slaweq has quit IRC | 04:54 | |
| *** gcheresh has joined #openstack-lbaas | 04:57 | |
| *** slaweq has joined #openstack-lbaas | 05:00 | |
| *** gcheresh has quit IRC | 05:02 | |
| *** slaweq has quit IRC | 05:04 | |
| *** slaweq has joined #openstack-lbaas | 05:10 | |
| *** slaweq has quit IRC | 05:15 | |
| *** slaweq has joined #openstack-lbaas | 05:20 | |
| *** slaweq has quit IRC | 05:24 | |
| *** gcheresh_ has joined #openstack-lbaas | 05:47 | |
| *** irenab has joined #openstack-lbaas | 05:56 | |
| *** rcernin_ has joined #openstack-lbaas | 06:18 | |
| *** rcernin has quit IRC | 06:21 | |
| *** slaweq has joined #openstack-lbaas | 06:21 | |
| *** gcheresh_ has quit IRC | 06:23 | |
| *** slaweq has quit IRC | 06:26 | |
| *** slaweq has joined #openstack-lbaas | 06:31 | |
| *** sapd_ has joined #openstack-lbaas | 06:35 | |
| *** slaweq has quit IRC | 06:37 | |
| *** gcheresh_ has joined #openstack-lbaas | 06:37 | |
| *** sapd__ has quit IRC | 06:39 | |
| *** links has quit IRC | 06:41 | |
| *** slaweq has joined #openstack-lbaas | 06:42 | |
| *** slaweq has quit IRC | 06:46 | |
| *** gcheresh_ has quit IRC | 06:48 | |
| *** links has joined #openstack-lbaas | 06:51 | |
| *** slaweq has joined #openstack-lbaas | 06:52 | |
| *** slaweq has quit IRC | 06:57 | |
| *** gcheresh_ has joined #openstack-lbaas | 07:02 | |
| *** rcernin has joined #openstack-lbaas | 07:05 | |
| *** rcernin_ has quit IRC | 07:06 | |
| *** threestrands_ has joined #openstack-lbaas | 07:09 | |
| *** threestrands_ has quit IRC | 07:09 | |
| *** threestrands_ has joined #openstack-lbaas | 07:09 | |
| *** gcheresh_ has quit IRC | 07:10 | |
| *** threestrands has quit IRC | 07:10 | |
| *** ivve has quit IRC | 07:12 | |
| *** threestrands_ has quit IRC | 07:13 | |
| *** gcheresh_ has joined #openstack-lbaas | 07:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata https://review.openstack.org/550371 | 07:22 |
| *** gcheresh_ has quit IRC | 07:24 | |
| openstackgerrit | Carlos Goncalves proposed openstack/neutron-lbaas master: Switch to stestr and wait for LB delete on cleanup https://review.openstack.org/550282 | 07:28 |
| *** slaweq has joined #openstack-lbaas | 07:33 | |
| *** slaweq has quit IRC | 07:38 | |
| *** slaweq has joined #openstack-lbaas | 07:43 | |
| *** slaweq has quit IRC | 07:48 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Add rbac support for octavia service apis https://review.openstack.org/550319 | 07:51 |
| *** slaweq has joined #openstack-lbaas | 07:53 | |
| *** pcaruana has joined #openstack-lbaas | 07:54 | |
| *** pcaruana has quit IRC | 07:55 | |
| *** pcaruana has joined #openstack-lbaas | 07:55 | |
| *** kobis has joined #openstack-lbaas | 07:56 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 07:58 | |
| *** slaweq has quit IRC | 07:58 | |
| *** slaweq has joined #openstack-lbaas | 08:04 | |
| *** slaweq has quit IRC | 08:08 | |
| *** slaweq has joined #openstack-lbaas | 08:14 | |
| *** ivve has joined #openstack-lbaas | 08:14 | |
| *** slaweq has quit IRC | 08:18 | |
| cgoncalves | rm_work: I have awesome news (NOT!) | 08:22 |
| cgoncalves | http://logs.openstack.org/82/550282/5/check/neutron-lbaasv2-dsvm-api/e40cf42/job-output.txt.gz | 08:22 |
| *** tesseract has joined #openstack-lbaas | 08:22 | |
| cgoncalves | "Failed: 18" | 08:22 |
| cgoncalves | all "one or more ports still in use" | 08:22 |
| *** slaweq_ has joined #openstack-lbaas | 08:24 | |
| *** slaweq_ has quit IRC | 08:29 | |
| *** threestrands has joined #openstack-lbaas | 08:32 | |
| *** threestrands has quit IRC | 08:32 | |
| *** threestrands has joined #openstack-lbaas | 08:32 | |
| *** slaweq_ has joined #openstack-lbaas | 08:34 | |
| *** threestrands has quit IRC | 08:36 | |
| *** slaweq_ has quit IRC | 08:39 | |
| *** rcernin has quit IRC | 08:42 | |
| *** slaweq_ has joined #openstack-lbaas | 08:44 | |
| *** slaweq_ has quit IRC | 08:49 | |
| *** slaweq_ has joined #openstack-lbaas | 08:55 | |
| *** gcheresh_ has joined #openstack-lbaas | 08:57 | |
| *** slaweq_ has quit IRC | 08:59 | |
| rm_work | lol | 09:01 |
| rm_work | i'm fighting with pylint | 09:02 |
| rm_work | after all that talk about loving it :P | 09:02 |
| rm_work | (i still love it. sometimes lovers quarrel) | 09:02 |
| dayou | I love s-quarrel, squarrel :P | 09:04 |
| *** slaweq_ has joined #openstack-lbaas | 09:05 | |
| *** slaweq_ has quit IRC | 09:09 | |
| *** salmankhan has joined #openstack-lbaas | 09:12 | |
| *** slaweq_ has joined #openstack-lbaas | 09:15 | |
| *** slaweq_ has quit IRC | 09:19 | |
| *** slaweq_ has joined #openstack-lbaas | 09:25 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Properly test access to tls_refs in the API layer https://review.openstack.org/550303 | 09:28 |
| rm_work | imacdonn: fixed, I believe | 09:28 |
| *** salmankhan has quit IRC | 09:29 | |
| *** slaweq_ has quit IRC | 09:29 | |
| *** slaweq_ has joined #openstack-lbaas | 09:35 | |
| *** slaweq_ has quit IRC | 09:40 | |
| *** slaweq_ has joined #openstack-lbaas | 09:45 | |
| *** slaweq_ has quit IRC | 09:50 | |
| *** slaweq_ has joined #openstack-lbaas | 09:56 | |
| *** slaweq_ has quit IRC | 10:00 | |
| *** bonky has joined #openstack-lbaas | 10:05 | |
| cgoncalves | rm_work: hey. have you found the cause of the api failing tests? | 10:11 |
| rm_work | still looking | 10:14 |
| rm_work | was fighting pylint | 10:14 |
| nmagnezi_ | rm_work, i'm gonna try to locally stack a node and see if that reproduces | 10:16 |
| *** slaweq_ has joined #openstack-lbaas | 10:16 | |
| *** slaweq_ has quit IRC | 10:21 | |
| *** annp has quit IRC | 10:23 | |
| rm_work | kk | 10:26 |
| rm_work | ugh and neutron-lbaas, and run n-lbaas tempest <_< | 10:26 |
| *** slaweq_ has joined #openstack-lbaas | 10:26 | |
| *** slaweq_ has quit IRC | 10:31 | |
| *** yamamoto has quit IRC | 10:31 | |
| rm_work | gonna keep looking at code | 10:33 |
| *** salmankhan has joined #openstack-lbaas | 10:33 | |
| *** salmankhan has quit IRC | 10:34 | |
| *** salmankhan1 has joined #openstack-lbaas | 10:34 | |
| *** salmankhan1 has quit IRC | 10:36 | |
| *** slaweq_ has joined #openstack-lbaas | 10:36 | |
| *** KeithMnemonic has joined #openstack-lbaas | 10:40 | |
| *** slaweq_ has quit IRC | 10:41 | |
| *** salmankhan has joined #openstack-lbaas | 10:45 | |
| *** slaweq_ has joined #openstack-lbaas | 10:47 | |
| *** slaweq_ has quit IRC | 10:51 | |
| rm_work | gonna try somthing | 10:52 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 10:58 |
| openstackgerrit | Adam Harwell proposed openstack/neutron-lbaas master: Switch to stestr and wait for LB delete on cleanup https://review.openstack.org/550282 | 11:00 |
| rm_work | cgoncalves: just added a thing to your patch | 11:00 |
| rm_work | I have a theory about the probem | 11:01 |
| rm_work | *problem | 11:01 |
| rm_work | see the end of base.py changes | 11:01 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 11:01 |
| rm_work | I wish we already required retrying or tenacity... | 11:02 |
| rm_work | don't want to add a whole req for this test | 11:02 |
| *** ivve has quit IRC | 11:02 | |
| cgoncalves | rm_work: nice! let's see what jobs have to say about it :) | 11:03 |
| rm_work | well | 11:03 |
| rm_work | right now the jobs won't even start | 11:03 |
| rm_work | they're starting and immediately failing and going to MAX_RETRIES failure | 11:04 |
| rm_work | something must be up in Infra | 11:04 |
| cgoncalves | rm_work: yeah I'd also like to backport it so any thing that prevent us from that would be a -1... | 11:04 |
| rm_work | yeah | 11:04 |
| rm_work | the other two changes i committed recently are also backport material | 11:04 |
| cgoncalves | https://review.openstack.org/#/c/549263/ | 11:06 |
| cgoncalves | https://review.openstack.org/#/c/550303/ | 11:06 |
| cgoncalves | ^ those? | 11:06 |
| cgoncalves | I wish we could set a backport-candidate flag in gerrit reviews (can we?) | 11:07 |
| rm_work | i mean | 11:07 |
| rm_work | we can agree on one | 11:07 |
| rm_work | and maybe you can search on it | 11:07 |
| * rm_work shrugs | 11:07 | |
| rm_work | I was thinking about that today too | 11:07 |
| rm_work | maybe like | 11:07 |
| rm_work | Backport-Candidate: pike queens | 11:07 |
| cgoncalves | well, we could introduce such as project-specific ;) | 11:09 |
| cgoncalves | and then reviewers would -1/+1 that flag too | 11:09 |
| rm_work | yeah confirmed gates are down for a little bit | 11:09 |
| rm_work | they're working on it | 11:10 |
| *** salmankhan has quit IRC | 11:15 | |
| bonky | hey guys, trying to get octavia working on a pike installation (done through rhosp 12). I've come pretty far, but I'm having trouble understanding the [service_auth] section in '/etc/neutron/neutron_lbaas.conf' and the [keystone_authtoken] in octavia.conf. Should I configure both sections ? I have done that, and the error I'm getting now is from octavia-api and is as follows | 11:20 |
| *** salmankhan has joined #openstack-lbaas | 11:20 | |
| bonky | 2018-03-07 09:40:59.616 1 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to | 11:20 |
| bonky | 2018-03-07 09:41:00.299 1 ERROR wsme.api [req-319a075d-4c1c-4007-9650-0c3b7452f34e 5fc177cdfe7340399332ece2c09cd11c aff54c4fc2024c2c938def4effbba20e - default default] Server-side error: "An auth plugin is required to determine endpoint URL". Detail: Traceback (most recent call last): | 11:20 |
| bonky | I've been stuck with this for hours. | 11:20 |
| bonky | So any help would really be appritated | 11:20 |
| *** ivve has joined #openstack-lbaas | 11:21 | |
| rm_work | bonky: you should not need neutron-lbaas in Pike | 11:27 |
| rm_work | do not bother configuring it | 11:27 |
| rm_work | I think our docs are a bit out of date | 11:27 |
| *** slaweq_ has joined #openstack-lbaas | 11:27 | |
| bonky | hm, so what should I use ? (I'm sorry I find this really confusing) | 11:28 |
| bonky | Do you have any documentation on how to get octavia working on pike ? | 11:29 |
| cgoncalves | rm_work: prolly bonky wants neutron-lbaas + octavia driver since he mentioned OSP12 (red hat's downstream openstack product) | 11:29 |
| rm_work | hmmm | 11:30 |
| cgoncalves | octavia is not supported in OSP12; neutron-lbaas is | 11:30 |
| bonky | cgoncalves: yes that is what I though. | 11:30 |
| rm_work | :( | 11:30 |
| cgoncalves | *but* neutron-lbaas + octavia is not | 11:30 |
| bonky | cgoncalves: yes, I believe that is the case. Trying to get the octavia-driver to work | 11:30 |
| cgoncalves | q | 11:31 |
| cgoncalves | oops! | 11:31 |
| rm_work | in that case, if neither is officially supported, i would recommend you do not use neutron-lbaas | 11:31 |
| rm_work | and use only Octavia anyway | 11:31 |
| cgoncalves | rm_work: neutron-lbaas itself is supported | 11:31 |
| rm_work | cgoncalves: but you said n-lbaas + octavia is not | 11:31 |
| *** yamamoto has joined #openstack-lbaas | 11:32 | |
| rm_work | so if he's trying to use octavia anyway.... | 11:32 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 11:32 |
| *** slaweq_ has quit IRC | 11:32 | |
| bonky | service_provider=LOADBALANCERV2:Octavia:neutron_lbaas.drivers.octavia.driver.OctaviaDriver:default <- this is what i have in neutron.conf | 11:33 |
| cgoncalves | rm_work: if n-lbaas+octavia we support n-lbaas until it reaches out to octavia | 11:33 |
| rm_work | lol well | 11:34 |
| rm_work | that's just as useless IMO :P | 11:34 |
| cgoncalves | rm_work: no. we recently had to backport an octavia driver patch because we support n-lbaas | 11:34 |
| rm_work | err wut | 11:35 |
| rm_work | so you don't support octavia | 11:35 |
| rm_work | but you do? | 11:35 |
| cgoncalves | we support whatever is in neutron-lbaas code base | 11:35 |
| cgoncalves | octavia is out | 11:35 |
| rm_work | <_< | 11:35 |
| rm_work | but supporting neutron-lbaas+octavia but not octavia is ... not especially useful IMO | 11:36 |
| cgoncalves | https://review.openstack.org/#/c/454172/ | 11:37 |
| cgoncalves | rm_work: it is. octavia could be supported by other vendor or not supported at all | 11:37 |
| rm_work | >_> | 11:37 |
| *** slaweq_ has joined #openstack-lbaas | 11:38 | |
| *** yamamoto has quit IRC | 11:38 | |
| *** slaweq_ has quit IRC | 11:42 | |
| bonky | Hm, I'm still stuck with my initial error-message "ERROR wsme.api [req-21a5ecb9-2775-4adf-b43b-ff24891f5f9e 5fc177cdfe7340399332ece2c09cd11c aff54c4fc2024c2c938def4effbba20e - default default] Server-side error: "An auth plugin is required to determine endpoint URL".... I get this when running, "neutron lbaas-loadbalancer-create --name 1lb1011 common" | 11:43 |
| bonky | Any pointers ? | 11:43 |
| *** yamamoto has joined #openstack-lbaas | 11:45 | |
| dayou | https://github.com/openstack/openstack-ansible-os_octavia/blob/master/templates/octavia.conf.j2 | 11:47 |
| bonky | I've checked that file a million times, still cant find whats wrong | 11:48 |
| *** slaweq_ has joined #openstack-lbaas | 11:48 | |
| dayou | bonky, try check this as a reference, I remeber there was some compatibility issues for keystone | 11:48 |
| dayou | admin_tenant_name = {{ octavia_service_project_name }} | 11:49 |
| dayou | admin_user = {{ octavia_service_user_name }} | 11:49 |
| dayou | admin_password = {{ octavia_service_password }} | 11:49 |
| bonky | hm, both the sections [service_auth] *and* [keystone_authtoken] are available in octavia.conf. Do both of them need to be filled in ? I thought that '[service_auth]' section only was in '/etc/neutron/neutron_lbaas.conf' | 11:49 |
| rm_work | actually i think you're prolly missing a section in the neutron-lbaas config | 11:49 |
| dayou | Instead of project_name and username, the paramaters above solved a compatiblity issue for me | 11:50 |
| *** gcheresh_ has quit IRC | 11:50 | |
| rm_work | or neutron config | 11:50 |
| rm_work | there's two sections for auth i think | 11:50 |
| *** yamamoto has quit IRC | 11:50 | |
| bonky | in /etc/neutron/neutron_lbaas.conf i only have service_auth | 11:51 |
| bonky | you think it should be something more in there ? | 11:51 |
| bonky | in octavia I see two auth sections, but I only have one with parameters | 11:51 |
| openstackgerrit | Min Sun proposed openstack/octavia-dashboard master: Cannot update ssl certificate when update listener https://review.openstack.org/550313 | 11:52 |
| *** slaweq_ has quit IRC | 11:52 | |
| *** gcheresh_ has joined #openstack-lbaas | 11:53 | |
| rm_work | hmm, in octavia both need params | 11:53 |
| rm_work | they will often be the same | 11:53 |
| bonky | ok, im working on it now. | 11:54 |
| bonky | will try | 11:54 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Migrate to stestr https://review.openstack.org/550134 | 11:57 |
| *** gcheresh_ has quit IRC | 11:57 | |
| *** slaweq_ has joined #openstack-lbaas | 11:58 | |
| *** ivve has quit IRC | 12:00 | |
| bonky | ok, that actually got me a step further. | 12:01 |
| bonky | No more auth errores. | 12:01 |
| bonky | Sending created Load Balancer b8420cf2-f867-4053-8682-2b7b5fcddeb8 to the handler <- is the last thing I see in the octavia-api logs | 12:02 |
| *** slaweq_ has quit IRC | 12:03 | |
| bonky | No Amphora available for load balancer <- this does not sound good :( | 12:04 |
| bonky | Maybe I should just wait for OSP13 where octivia should be fully supported | 12:05 |
| bonky | 2018-03-07 12:01:57.063 19 ERROR octavia.controller.worker.controller_worker CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/ssl/certs/ssl-cert-snakeoil.pem. | 12:07 |
| *** slaweq_ has joined #openstack-lbaas | 12:08 | |
| bonky | can i create a loadbalancer without tls ? just to see if its working ? | 12:10 |
| dayou | yes, you should be able to do that | 12:13 |
| *** slaweq_ has quit IRC | 12:13 | |
| bonky | Hm, _Flow 'octavia-create-loadbalancer-flow': CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/ssl/certs/ssl-cert-snakeoil.pem. <- thats were I get stuck now. | 12:15 |
| bonky | and that file doesnt exist ,so that makes sence | 12:15 |
| bonky | but I dont want to use any tls whatsoever at this point | 12:16 |
| *** sanfern has joined #openstack-lbaas | 12:18 | |
| *** slaweq__ has joined #openstack-lbaas | 12:18 | |
| *** slaweq__ has quit IRC | 12:23 | |
| *** slaweq has joined #openstack-lbaas | 12:29 | |
| dayou | Anyone knows whether the following behavior is correct? | 12:30 |
| dayou | A user has two projects, and it chose the one of the project and send a get all loadbalancers request | 12:30 |
| dayou | then the user gets 403 | 12:30 |
| dayou | if it chose the other project and send a get all loadbalancers request, it will get 200 | 12:31 |
| *** sticker has joined #openstack-lbaas | 12:33 | |
| *** gcheresh_ has joined #openstack-lbaas | 12:33 | |
| *** slaweq has quit IRC | 12:33 | |
| *** slaweq has joined #openstack-lbaas | 12:39 | |
| rm_work | bonky: there are two things that you are mixing up | 12:39 |
| rm_work | bonky: there's TLS Termination for Listeners | 12:39 |
| rm_work | which is what you don't want (that's fine, you can disable it) | 12:39 |
| rm_work | but also we communicate to our VMs internally with cert auth | 12:40 |
| rm_work | so you absolutely need that CA cert/key | 12:40 |
| rm_work | without it we can't create Amphora (the VMs) | 12:40 |
| rm_work | so you should generate a cert/key with CA Signing capability (self-signed is fine, i doubt anyone would run this with a legit CA cert/key) | 12:41 |
| rm_work | and then set the octavia config to use it (replacing the path to that snakeoil one in the config) | 12:41 |
| rm_work | hmmm.... wtf is this: http://paste.openstack.org/show/693623/ and do I care | 12:43 |
| rm_work | it seems to be working still | 12:44 |
| *** slaweq has quit IRC | 12:44 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Properly test access to tls_refs in the API layer https://review.openstack.org/550303 | 12:44 |
| rm_work | ok so who is actually going to be around | 12:45 |
| rm_work | nmagnezi_: you are here I guess... and maybe xgerman_ will be around... will need you guys to review these gatefix patches so we can get them through | 12:46 |
| rm_work | hmm, I guess I can review too since both ended up technically being Carlos' patches :P | 12:46 |
| xgerman_ | Wut? | 12:46 |
| rm_work | I'm tentatively hopeful this'll fix it (jinx) | 12:46 |
| rm_work | xgerman_: ohai you're on already | 12:46 |
| rm_work | in octavia, https://review.openstack.org/#/c/550134/ | 12:46 |
| rm_work | and then on the n-lbaas side, the one it depends on -- https://review.openstack.org/#/c/550282/ | 12:47 |
| rm_work | hopefully both will pass zuul this time | 12:47 |
| * rm_work crosses fingers we got the issues | 12:47 | |
| rm_work | until then, all of our gates are broken | 12:47 |
| nmagnezi_ | rm_work, np at all. will look at those now | 12:48 |
| rm_work | k, though holding my breath a bit that both will pass :P | 12:48 |
| nmagnezi_ | hah | 12:48 |
| rm_work | but you can review for logic and such now :) | 12:48 |
| xgerman_ | Litmus | 12:49 |
| xgerman_ | Mobile... | 12:49 |
| *** slaweq has joined #openstack-lbaas | 12:49 | |
| nmagnezi_ | rm_work, this looks promising in zuul https://review.openstack.org/#/c/550134/ | 12:49 |
| rm_work | yeah | 12:49 |
| rm_work | i think so | 12:49 |
| *** gcheresh_ has quit IRC | 12:50 | |
| rm_work | I guess I can +A the octavia one since i didn't work on it | 12:50 |
| rm_work | will let you and german look at the n-lbaas one | 12:50 |
| rm_work | I really want the gates back | 12:51 |
| rm_work | >_< | 12:51 |
| dayou | ^^ | 12:51 |
| nmagnezi_ | rm_work, just commented "Looks good (and also passes tests in Zuul). | 12:51 |
| nmagnezi_ | Will wait with the W+1 until we get all results." | 12:51 |
| nmagnezi_ | rm_work, but... you really wanted this in.. :D | 12:51 |
| rm_work | very impatient. but, it's 5am so I guess I'll go to bed and check up on them in ~5 hours | 12:51 |
| cgoncalves | \o/ | 12:51 |
| rm_work | I mean, it has a depends-on | 12:52 |
| rm_work | so it'll have to pass | 12:52 |
| rm_work | AND the other one will have to merge | 12:52 |
| nmagnezi_ | rm_work, one day, you'll teach me how to live with no sleep what so ever | 12:52 |
| rm_work | and then it'll have to pass again | 12:52 |
| nmagnezi_ | cgoncalves, good jon! | 12:52 |
| nmagnezi_ | cgoncalves, good job! | 12:52 |
| rm_work | nmagnezi_: it's not that I get no sleep by choice :P various bosses want me to wake up for status meetings, and *I* don't want to go to sleep at normal times, so it's just kinda what happens, lol | 12:53 |
| rm_work | soon i'll be back to JST anyway | 12:54 |
| *** slaweq has quit IRC | 12:54 | |
| nmagnezi_ | rm_work, i bet you take power naps between patches.. :D | 12:54 |
| rm_work | which will be ... not significantly different as far as anyone else is concerned | 12:54 |
| rm_work | actually i am so bad at napping :( never been able to do it | 12:54 |
| rm_work | I wish I could | 12:54 |
| bonky | rm_work: ok ok, thanks for the explanation. | 12:55 |
| rm_work | I just sleep like 4h/night or so during the week and then on weekends I sleep ~12h/night | 12:55 |
| bonky | Got one step further now with my ca in place | 12:55 |
| bonky | |__Flow 'octavia-create-loadbalancer-flow': IOError: [Errno 2] No such file or directory: '/etc/octavia/certs/ca_01.pem' | 12:55 |
| cgoncalves | the only wait for rm_work to get some sleep is if we fix all bugs (including those we're not yet aware of!) and implement all sorts of features | 12:55 |
| cgoncalves | only then he can sleep | 12:55 |
| bonky | but now this | 12:55 |
| nmagnezi_ | rm_work, start like this guy in this photo: https://en.wikipedia.org/wiki/Nap | 12:55 |
| rm_work | i do love hammocks | 12:55 |
| bonky | what is ca_01.pem and why do I need it ? | 12:55 |
| bonky | :D | 12:55 |
| rm_work | never could understand sleeping in them tho | 12:55 |
| rm_work | bonky: it's just the name of the file you have in your config (prolly from default) | 12:56 |
| rm_work | you'll want to adjust that to point to a real file | 12:56 |
| rm_work | erm | 12:56 |
| rm_work | hold on, can you pastebin me that whole section of your config | 12:56 |
| rm_work | with the paths | 12:56 |
| bonky | same as the ca i just created for the internal traffic you mean ? | 12:57 |
| cgoncalves | nmagnezi_: ^^ see? he never stops | 12:57 |
| rm_work | i'm comparing to my config | 12:57 |
| bonky | I've got a meeting in a few minutes, I'll be back in 30 | 12:57 |
| bonky | post my config then | 12:57 |
| rm_work | trying to remember which cert is which | 12:57 |
| *** b_bezak has joined #openstack-lbaas | 12:57 | |
| rm_work | ok, i'll try to post some notes, may be gone by then | 12:57 |
| bonky | # client_ca = /etc/octavia/certs/ca_01.pem <- its this option | 12:58 |
| bonky | under "controller_worker" | 12:58 |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Add rbac support for octavia service apis https://review.openstack.org/550319 | 12:58 |
| rm_work | ah | 12:58 |
| rm_work | so like I have this | 12:58 |
| *** gcheresh_ has joined #openstack-lbaas | 12:58 | |
| rm_work | http://paste.openstack.org/show/693630/ | 12:59 |
| rm_work | so yes | 12:59 |
| rm_work | can be the same crt file as the ca | 12:59 |
| bonky | cool ! | 12:59 |
| bonky | I'll try it out | 12:59 |
| *** slaweq has joined #openstack-lbaas | 12:59 | |
| *** yamamoto has joined #openstack-lbaas | 13:02 | |
| rm_work | yep, the octavia one passed all the checks (the last one is in cleanup right now) | 13:03 |
| *** slaweq has quit IRC | 13:03 | |
| rm_work | xgerman_: answered your question | 13:04 |
| xgerman_ | +2’d | 13:04 |
| xgerman_ | thanks | 13:04 |
| *** yamamoto has quit IRC | 13:07 | |
| rm_work | cgoncalves: so where did you clone this change from? looking to make sure for coverage there's nothing special we have to do anymore with stestr | 13:09 |
| *** slaweq has joined #openstack-lbaas | 13:09 | |
| cgoncalves | rm_work: got inspiration from multiple other projects | 13:11 |
| *** gcheresh_ has quit IRC | 13:11 | |
| *** gcheresh_ has joined #openstack-lbaas | 13:11 | |
| rm_work | k | 13:12 |
| *** sanfern has quit IRC | 13:13 | |
| cgoncalves | e.g. https://review.openstack.org/#/c/549826/ | 13:13 |
| cgoncalves | also from cinder, zuul, nodepool, ... | 13:14 |
| *** slaweq has quit IRC | 13:14 | |
| cgoncalves | the important thing is: I took the best of each ;) | 13:14 |
| *** sanfern has joined #openstack-lbaas | 13:15 | |
| rm_work | lol k | 13:16 |
| cgoncalves | I'll work on a similar stestr patch for other octavia projects | 13:17 |
| *** slaweq has joined #openstack-lbaas | 13:19 | |
| *** sanfern has quit IRC | 13:20 | |
| *** gcheresh_ has quit IRC | 13:21 | |
| *** atoth has joined #openstack-lbaas | 13:23 | |
| *** slaweq has quit IRC | 13:24 | |
| nmagnezi_ | cgoncalves, what about https://review.openstack.org/#/c/550282/ ? | 13:27 |
| cgoncalves | nmagnezi_: what do you mean? | 13:27 |
| nmagnezi_ | cgoncalves, do we have any idea what's failing it? :\ | 13:28 |
| cgoncalves | nmagnezi_: retry_limit? CI issues | 13:28 |
| rm_work | is it failing? | 13:28 |
| rm_work | looks like it's fine | 13:28 |
| rm_work | should be all passing now | 13:28 |
| cgoncalves | so far so good | 13:28 |
| cgoncalves | I'm monitoring zuul | 13:28 |
| cgoncalves | rm_work: you're really waiting for it to verify before you go to sleep haha | 13:29 |
| *** gcheresh_ has joined #openstack-lbaas | 13:29 | |
| nmagnezi_ | oh, I see it in Zuul too | 13:29 |
| cgoncalves | you'll be in trouble if it fails :P | 13:29 |
| *** slaweq has joined #openstack-lbaas | 13:30 | |
| cgoncalves | Failed: 0 | 13:31 |
| cgoncalves | \ooooooo/ | 13:31 |
| rm_work | lol | 13:32 |
| rm_work | yes you caught me | 13:32 |
| *** ivve has joined #openstack-lbaas | 13:34 | |
| *** slaweq has quit IRC | 13:34 | |
| cgoncalves | http://logs.openstack.org/82/550282/6/check/neutron-lbaasv2-dsvm-py3x-scenario-namespace/0b166b4/job-output.txt.gz | 13:37 |
| cgoncalves | Failed: 1 | 13:37 |
| cgoncalves | Body: b\'{"NeutronError": {"detail": "", "type": "NetworkNotFound", "message": "Network f04fd51f-e0d9-4891-b753-f9972f785724 could not be found."}}\'' | 13:38 |
| cgoncalves | weird because neutron-lbaasv2-dsvm-scenario-namespace passed | 13:39 |
| *** slaweq has joined #openstack-lbaas | 13:40 | |
| *** sanfern has joined #openstack-lbaas | 13:41 | |
| rm_work | err wut | 13:43 |
| rm_work | yeah uhhhh wut | 13:44 |
| *** slaweq has quit IRC | 13:45 | |
| rm_work | asking in infra | 13:49 |
| rm_work | derp no | 13:49 |
| rm_work | i just read the job name again | 13:50 |
| rm_work | the py3x namespace DID fail | 13:50 |
| rm_work | cgoncalves: ^^ | 13:50 |
| cgoncalves | right, it failed | 13:50 |
| *** slaweq has joined #openstack-lbaas | 13:50 | |
| cgoncalves | its py27 counter-part did not (neutron-lbaasv2-dsvm-scenario-namespace) | 13:50 |
| rm_work | yeah i misread what you said T_T | 13:50 |
| rm_work | yeah that's an intermittent | 13:50 |
| cgoncalves | meh, it should be fixed... somewhere, somehow | 13:51 |
| cgoncalves | if it's an intermittent, could we recheck it? :) | 13:51 |
| rm_work | we could | 13:52 |
| rm_work | i'm looking through the code and it SHOULD be fine | 13:53 |
| rm_work | it waits for stuff... | 13:53 |
| rm_work | the neutron-lbaas <--> octavia thing can be kinda odd I think sometimes :/ | 13:53 |
| *** gcheresh_ has quit IRC | 13:53 | |
| rm_work | oh wait | 13:53 |
| rm_work | this is namespace | 13:53 |
| rm_work | so probably the process was being held open | 13:54 |
| rm_work | so it issued the restart command but haproxy was still listening for some reason and was slow to restart (which would be an async thing) | 13:54 |
| *** slaweq has quit IRC | 13:54 | |
| rm_work | so it caught one request right at the end | 13:55 |
| rm_work | i think all of this is because the gate machines are running faster now | 13:55 |
| cgoncalves | yup, that's what I wondered yesterday ;) | 13:55 |
| cgoncalves | we need slower machines for tests to pass xD | 13:55 |
| rm_work | lol whelp | 13:56 |
| rm_work | i guess i'll go to sleep now | 13:56 |
| rm_work | kicked the recheck | 13:56 |
| rm_work | hope you guys can merge while i'm sleeping | 13:56 |
| cgoncalves | I'll recheck and ask Nir to +W if it passes gate | 13:57 |
| * nmagnezi_ nods | 13:59 | |
| *** slaweq has joined #openstack-lbaas | 14:00 | |
| *** yamamoto has joined #openstack-lbaas | 14:03 | |
| *** slaweq has quit IRC | 14:04 | |
| *** yamamoto has quit IRC | 14:09 | |
| *** ivve has quit IRC | 14:09 | |
| *** slaweq has joined #openstack-lbaas | 14:10 | |
| *** slaweq has quit IRC | 14:15 | |
| *** slaweq has joined #openstack-lbaas | 14:21 | |
| *** ivve has joined #openstack-lbaas | 14:22 | |
| *** slaweq has quit IRC | 14:25 | |
| *** ivve has quit IRC | 14:29 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix kvm-centos.7 gate https://review.openstack.org/550487 | 14:29 |
| *** gcheresh_ has joined #openstack-lbaas | 14:29 | |
| rm_work | damnit seriously, namespace failed again | 14:30 |
| dayou | :) Get some rest man, we can keep an eye on the rechecking | 14:30 |
| rm_work | lol: testtools.matchers._impl.MismatchError: 25 != 0: Member server2 is balanced | 14:30 |
| rm_work | 25 now | 14:30 |
| cgoncalves | you're still here or is it your doppelganger? | 14:30 |
| rm_work | it's being slow at restarting the service | 14:30 |
| rm_work | i wonder what we can do about that honestly :/ | 14:31 |
| *** slaweq has joined #openstack-lbaas | 14:31 | |
| rm_work | yeah sleep soon | 14:31 |
| rm_work | i don't know much about the namespace driver tho, but i seriously don't know besides like ... a sleep-wait before the cleanup test there, how we would fix that | 14:31 |
| rm_work | i'm tempted to add one... | 14:32 |
| cgoncalves | I dont know what the namespace driver is/does to start with :P | 14:32 |
| rm_work | a systemd service restart is async and there's really nothing we can do about the fact that it takes a while sometimes | 14:32 |
| rm_work | it's this thing where we run a bunch of haproxy agents in cgroups on the neutron-agent host | 14:32 |
| cgoncalves | ah ok ok | 14:32 |
| *** salmankhan1 has joined #openstack-lbaas | 14:33 | |
| *** salmankhan1 has quit IRC | 14:33 | |
| rm_work | ugh if this is becoming common enough that it happens 2/3 times... we won't be able to merge this probably without a fix for this as well | 14:33 |
| rm_work | this is really dumb | 14:33 |
| *** salmankhan has quit IRC | 14:33 | |
| rm_work | why does this stuff always happen all at once in a huge mess | 14:34 |
| rm_work | three different problems | 14:34 |
| dayou | Lukily the dashboard gates are always up these days :D | 14:35 |
| *** slaweq has quit IRC | 14:35 | |
| dayou | I go to sleep now, see you guys tomorrow | 14:36 |
| nmagnezi_ | dayou, o/ | 14:36 |
| dayou | nmagnezi, o/ | 14:36 |
| dayou | :P | 14:36 |
| *** links has quit IRC | 14:37 | |
| xgerman_ | o/ | 14:38 |
| *** slaweq has joined #openstack-lbaas | 14:41 | |
| *** ivve has joined #openstack-lbaas | 14:43 | |
| *** jniesz has joined #openstack-lbaas | 14:44 | |
| *** slaweq has quit IRC | 14:45 | |
| *** fnaval has quit IRC | 14:46 | |
| *** gcheresh_ has quit IRC | 14:48 | |
| *** slaweq has joined #openstack-lbaas | 14:51 | |
| *** salmankhan has joined #openstack-lbaas | 14:52 | |
| *** slaweq has quit IRC | 14:56 | |
| rm_work | o/ | 15:00 |
| *** fnaval has joined #openstack-lbaas | 15:01 | |
| *** slaweq has joined #openstack-lbaas | 15:01 | |
| *** yamamoto has joined #openstack-lbaas | 15:05 | |
| *** slaweq has quit IRC | 15:06 | |
| *** yamamoto has quit IRC | 15:09 | |
| *** slaweq has joined #openstack-lbaas | 15:12 | |
| *** slaweq has quit IRC | 15:16 | |
| *** slaweq has joined #openstack-lbaas | 15:22 | |
| *** sanfern has quit IRC | 15:23 | |
| *** slaweq has quit IRC | 15:26 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 15:31 |
| *** slaweq has joined #openstack-lbaas | 15:32 | |
| *** slaweq has quit IRC | 15:37 | |
| *** slaweq has joined #openstack-lbaas | 15:42 | |
| *** slaweq has quit IRC | 15:46 | |
| *** slaweq has joined #openstack-lbaas | 15:52 | |
| *** slaweq has quit IRC | 15:57 | |
| cgoncalves | nmagnezi_: please +W https://review.openstack.org/#/c/550282/ :) | 16:01 |
| cgoncalves | +V anytime soon ;) | 16:01 |
| bonky | I'm hitting this issue now, 'ERROR octavia.controller.worker.controller_worker ComputeBuildException: Failed to build compute instance due to: {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500, u'details'' when it tries to deploy an amphora instance.. | 16:02 |
| bonky | Anyone got suggestions why ?= | 16:02 |
| cgoncalves | bonky: check n-api logs | 16:02 |
| *** slaweq has joined #openstack-lbaas | 16:02 | |
| cgoncalves | bonky: you could also try booting a cirros image to make sure it's not nova's fault | 16:03 |
| *** b_bezak has quit IRC | 16:03 | |
| cgoncalves | nmagnezi_: +V by zuul \o/ | 16:04 |
| *** b_bezak has joined #openstack-lbaas | 16:04 | |
| bonky | hm yes, other instances boot fione | 16:04 |
| bonky | im wondering what parameters are set to make it not find a compute node | 16:04 |
| cgoncalves | bonky: it could be that the flavor being used is not big enough | 16:05 |
| cgoncalves | bonky: nova logs should give you more info | 16:05 |
| bonky | ah, cool thanks | 16:05 |
| *** yamamoto has joined #openstack-lbaas | 16:06 | |
| *** slaweq has quit IRC | 16:07 | |
| *** b_bezak has quit IRC | 16:08 | |
| bonky | amp_flavor_id <- this parameter, in the example its an integer, but "flavor id's" according to openstack is something like, fa539e78-7457-4ee8-a1bf-c23901032c63 | 16:09 |
| *** pcaruana has quit IRC | 16:10 | |
| *** yamamoto has quit IRC | 16:12 | |
| *** slaweq_ has joined #openstack-lbaas | 16:13 | |
| *** slaweq_ has quit IRC | 16:18 | |
| xgerman_ | that’s ok we create our own flavor-id | 16:20 |
| bonky | i just thought the example parameter was confusing, like '65'. I think the example should reflect openstack's idea of "flavors id's" | 16:21 |
| *** slaweq has joined #openstack-lbaas | 16:23 | |
| *** slaweq has quit IRC | 16:27 | |
| *** KeithMnemonic has quit IRC | 16:31 | |
| *** slaweq has joined #openstack-lbaas | 16:33 | |
| *** slaweq has quit IRC | 16:38 | |
| *** sticker has quit IRC | 16:41 | |
| *** slaweq has joined #openstack-lbaas | 16:43 | |
| *** kobis has quit IRC | 16:45 | |
| *** slaweq has quit IRC | 16:47 | |
| *** slaweq has joined #openstack-lbaas | 16:53 | |
| *** slaweq has quit IRC | 16:57 | |
| *** slaweq has joined #openstack-lbaas | 16:57 | |
| *** yamamoto has joined #openstack-lbaas | 17:08 | |
| *** slaweq has left #openstack-lbaas | 17:10 | |
| *** yamamoto has quit IRC | 17:13 | |
| imacdonn | bonky: nova actually lets you specify a flavor ID, so it could be 65 | 17:25 |
| imacdonn | [imacdonn@home ~]$ openstack flavor create --id 65 --vcpus 4 --ram 4095 bonky | 17:26 |
| *** salmankhan has quit IRC | 17:41 | |
| *** salmankhan has joined #openstack-lbaas | 17:42 | |
| *** salmankhan has quit IRC | 17:43 | |
| *** salmankhan has joined #openstack-lbaas | 17:43 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 17:47 |
| *** salmankhan1 has joined #openstack-lbaas | 17:52 | |
| *** salmankhan has quit IRC | 17:55 | |
| *** salmankhan1 is now known as salmankhan | 17:55 | |
| *** pcaruana has joined #openstack-lbaas | 17:57 | |
| *** AlexeyAbashkin has quit IRC | 18:06 | |
| *** yamamoto has joined #openstack-lbaas | 18:09 | |
| *** yamamoto has quit IRC | 18:13 | |
| *** kobis has joined #openstack-lbaas | 18:17 | |
| *** salmankhan has quit IRC | 18:30 | |
| *** kobis has quit IRC | 18:47 | |
| *** tesseract has quit IRC | 18:53 | |
| *** yamamoto has joined #openstack-lbaas | 19:10 | |
| *** isantospMac has joined #openstack-lbaas | 19:14 | |
| *** yamamoto has quit IRC | 19:15 | |
| *** gcheresh_ has joined #openstack-lbaas | 19:16 | |
| *** harlowja has joined #openstack-lbaas | 19:26 | |
| xgerman_ | rm_work, johnsom yt? | 19:27 |
| *** dougwig has joined #openstack-lbaas | 19:29 | |
| xgerman_ | I had an install today (ACT-PAS) where the vrrp-ports didn’t come online; LB active, — wonder what VRRP was doing. In any case the LB should either be in PENDING or ERROR | 19:30 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:33 | |
| *** kobis has joined #openstack-lbaas | 19:36 | |
| *** kobis has quit IRC | 19:37 | |
| *** AlexeyAbashkin has quit IRC | 19:37 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:38 | |
| *** kobis has joined #openstack-lbaas | 19:41 | |
| rm_work | xgerman_: hmmm | 19:44 |
| *** gcheresh_ has quit IRC | 19:45 | |
| xgerman_ | yep, freakish and of course I am supposed to fix neutron not bringing up the port | 19:45 |
| xgerman_ | I filed a bug for good measure | 19:45 |
| rm_work | if the ports don't come up, i think unfortunately we don't see it in the flows | 19:48 |
| rm_work | because we just tell neutron to create them | 19:48 |
| rm_work | and neutron says "ok!" | 19:48 |
| rm_work | and it's all async from there | 19:48 |
| rm_work | and as long as it doesn't cause anything in our flows to break (which it wouldn't), if it fails async in neutron, then ... meh | 19:48 |
| rm_work | actually i am wondering if neutron did something recently that made ports less reliable/quick | 19:48 |
| rm_work | because we just saw ports taking longer to delete in our tests | 19:49 |
| rm_work | ah but you | 19:49 |
| *** kobis has quit IRC | 19:49 | |
| rm_work | *are probably running an older neutron anyway | 19:49 |
| rm_work | like, pike or something? | 19:49 |
| xgerman_ | Newton | 19:49 |
| rm_work | lol | 19:49 |
| rm_work | k | 19:49 |
| xgerman_ | but if I run ACTIVE-PASSIVE I would expect the thing to tell me | 19:49 |
| xgerman_ | after all it sends VRRP both ways and should be able to report an ERROR | 19:50 |
| rm_work | erm | 19:50 |
| rm_work | so until the initial connection is made | 19:50 |
| rm_work | how would it tell you | 19:50 |
| rm_work | vrrp doesn't go back to the controlplane | 19:50 |
| rm_work | it stays entirely inside the dataplane | 19:51 |
| rm_work | and there's no actual alarming up | 19:51 |
| rm_work | the way it'd tell you is by a keepalive check failing | 19:51 |
| rm_work | like, the haproxy checks it runs | 19:51 |
| rm_work | and those will pass so long as haproxy is running | 19:51 |
| rm_work | and the problem is, you don't want to alert up and force a recycle or anything of either amp because that's indistinguishable from "i'm the only Master online and I need to be serving traffic" | 19:52 |
| rm_work | so if both said "kill me" or "kill the other" and tried to take over or whatever... it'd be bad | 19:52 |
| rm_work | maybe our controlplane can check in periodically and verify that keepalived is healthy and in either master/backup with a valid connection | 19:53 |
| rm_work | brb loonch | 19:56 |
| *** ivve has quit IRC | 20:08 | |
| *** yamamoto has joined #openstack-lbaas | 20:11 | |
| *** yamamoto has quit IRC | 20:16 | |
| *** KeithMnemonic has joined #openstack-lbaas | 20:23 | |
| *** AlexeyAbashkin has quit IRC | 20:29 | |
| *** KeithMnemonic has quit IRC | 20:39 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 20:45 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 20:45 | |
| *** AlexeyAbashkin has quit IRC | 20:50 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 21:04 | |
| *** pcaruana has quit IRC | 21:12 | |
| *** yamamoto has joined #openstack-lbaas | 21:12 | |
| *** gcheresh_ has joined #openstack-lbaas | 21:16 | |
| *** yamamoto has quit IRC | 21:17 | |
| *** AlexeyAbashkin has quit IRC | 21:28 | |
| *** gcheresh_ has quit IRC | 21:29 | |
| *** isantospMac has quit IRC | 21:53 | |
| *** openstackstatus has quit IRC | 22:13 | |
| *** openstack has joined #openstack-lbaas | 22:17 | |
| *** ChanServ sets mode: +o openstack | 22:17 | |
| *** yamamoto has quit IRC | 22:18 | |
| johnsom | xgerman_: note, only one of the two neutron ports ever go active. This is normal for how neutron handles allowed address pairs. True health is always operating_status in Octavia | 22:23 |
| xgerman_ | yep, and they both were down in my world | 22:23 |
| xgerman_ | and Octavia showed ACTIVE | 22:23 |
| xgerman_ | I am talking ACTIVE-PASSIVE — I had three ports | 22:24 |
| xgerman_ | all down | 22:24 |
| xgerman_ | was expecting vip down/vrrp both up | 22:25 |
| xgerman_ | anyhow, I am more worried that neutron drops the ball and we never know | 22:26 |
| xgerman_ | aka does our responsibility include monitoring vrrp-ports or do we stop at eth.X | 22:27 |
| johnsom | Active is provisioning status, so, yeah as long as we didn’t fail that is correct. Operating status is key here | 22:28 |
| xgerman_ | and that was ONLINE | 22:29 |
| johnsom | Ok, yeah as far as we could tell all was well, so definitely an upstream neutron failure that we could not detect. The port plugged to the service vm netns must have been up, but the neutron side not | 22:31 |
| xgerman_ | yep, exactly | 22:31 |
| xgerman_ | I am pretty sure it’s an edge case *10000 but filed a bug nevertheless | 22:33 |
| xgerman_ | in my install all ports didnt come active on that network but I am not sure if there is a case where it’s more intermittent (given neutron I would put money on that) | 22:34 |
| johnsom | Yeah, it isn’t like we can ping google to check if the whole path is working. Tenant networks can’t get there | 22:34 |
| xgerman_ | yep, the good ‘ol libra fays | 22:35 |
| xgerman_ | but we can check if a port goes ACTIVE after we create it | 22:35 |
| johnsom | Well, they obviously went active for the vm side | 22:35 |
| xgerman_ | I mean check with Neutron | 22:35 |
| xgerman_ | | 6c97be4b-cf94-4880-b91e-6c248e928fc4 | octavia-lb-vrrp-6342cf32-ad1d-4d72-8ce3-27c70a756a5d | fa:16:3e:cc:a4:ba | ip_address='10.0.248.212', subnet_id='ef6afc78-705b-4abd-8c2a-916c3f2407d6' | DOWN | | 22:37 |
| *** rcernin has joined #openstack-lbaas | 22:37 | |
| johnsom | So like some external process that constantly poll neutron to make sure it doesn’t randomly go inactive after create? Hmm, we monitor the port from inside the vm, but external... I would hate to start babysitting neutron | 22:38 |
| xgerman_ | well, we can check after we create the port… but otherwise we out of luck | 22:39 |
| xgerman_ | this happened on create for me | 22:39 |
| johnsom | Ins’t that the one neutron ALWAYS reports down because it is a secondary address? Pretty sure that is normal neutron | 22:39 |
| xgerman_ | ok, different network (not broken) | 22:40 |
| xgerman_ | | 05c435ee-6f85-4554-b987-7e14d578ec8b | octavia-lb-vrrp-9fc6b0c3-0242-4bbf-a1ec-6ec30f6e593f | fa:16:3e:be:07:a6 | ip_address='192.168.0.16', subnet_id='3008b93e-3b3c-42b6-b451-664894736e7d' | ACTIVE | | 22:40 |
| xgerman_ | | 7dda1146-cc31-4709-b25b-b2c140617763 | octavia-lb-vrrp-8d44662d-62f5-46d2-8a10-958eed4a1433 | fa:16:3e:12:2c:4e | ip_address='192.168.0.13', subnet_id='3008b93e-3b3c-42b6-b451-664894736e7d' | ACTIVE | | 22:41 |
| cgoncalves | can we switch back to whatever slower gate machines we prolly had until a few days ago?? getting https://review.openstack.org/#/c/550282 verified starts to seem like the winning the lottery... | 22:41 |
| xgerman_ | only VIP port (not copied) is down | 22:41 |
| xgerman_ | mmh, we should be able to add some sleep(60) | 22:42 |
| *** openstackstatus has quit IRC | 22:42 | |
| *** openstack has joined #openstack-lbaas | 22:46 | |
| *** ChanServ sets mode: +o openstack | 22:46 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 22:46 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 22:47 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep https://review.openstack.org/549295 | 22:49 |
| *** AlexeyAbashkin has quit IRC | 22:51 | |
| rm_work | yeah i'm really tempted to add a sleep() there and be done with it <_< | 22:58 |
| rm_work | there is not much we can do about the NS driver | 22:59 |
| rm_work | hmm though why did dsvm-api fail | 22:59 |
| rm_work | wtf | 22:59 |
| rm_work | http://logs.openstack.org/82/550282/6/gate/neutron-lbaasv2-dsvm-api/c5a5cf7/job-output.txt.gz#_2018-03-07_21_31_17_816816 | 22:59 |
| rm_work | wtf is that | 22:59 |
| rm_work | "Invalid input for project_id. Reason: '' Blank strings are not permitted." | 23:00 |
| rm_work | ummmm | 23:00 |
| rm_work | that is on a test where it seems relevant | 23:00 |
| rm_work | test_create_listener_missing_tenant_id | 23:00 |
| rm_work | but how the heck would it just start randomly failing now | 23:00 |
| rm_work | and only on one test | 23:00 |
| cgoncalves | I'm for disabling gates provided that rm_work runs each patch for a week in prod before merging xD | 23:04 |
| rm_work | lol | 23:04 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [DNM] Add experimental ovsfw-scenario job https://review.openstack.org/550431 | 23:07 |
| rm_work | ummm | 23:10 |
| rm_work | does anyone know what this bash syntax means: | 23:10 |
| rm_work | ${OCTAVIA_AMP_IMAGE_SIZE:+1} | 23:10 |
| rm_work | that's not "default if undefined" | 23:10 |
| rm_work | that's .... | 23:10 |
| rm_work | "if defined, set to 1"? | 23:11 |
| rm_work | maybe? | 23:11 |
| rm_work | i'm trying to figure it out just by testing and guessing | 23:11 |
| *** threestrands has joined #openstack-lbaas | 23:12 | |
| *** threestrands has quit IRC | 23:12 | |
| *** threestrands has joined #openstack-lbaas | 23:12 | |
| *** yamamoto has joined #openstack-lbaas | 23:15 | |
| cgoncalves | ${parameter+alt_value}, ${parameter:+alt_value} | 23:19 |
| cgoncalves | If parameter set, use alt_value, else use null string. | 23:19 |
| cgoncalves | rm_work: ^ | 23:19 |
| rm_work | ok ... | 23:19 |
| rm_work | so that seems entirely wrong for where it's used | 23:19 |
| rm_work | trying to figure this out | 23:19 |
| rm_work | OH nm I see | 23:20 |
| rm_work | it's meaningless | 23:20 |
| rm_work | or rather, it's a REALLY weird way of doing this if-statement | 23:20 |
| rm_work | if [[ ${OCTAVIA_AMP_IMAGE_SIZE:+1} ]] ; then | 23:20 |
| *** yamamoto has quit IRC | 23:20 | |
| rm_work | seems like they'd be better off with like | 23:20 |
| *** yamamoto has joined #openstack-lbaas | 23:20 | |
| cgoncalves | indeed lol | 23:21 |
| rm_work | if [[ -n "$OCTAVIA_AMP_IMAGE_SIZE ]]; then | 23:21 |
| cgoncalves | [ -n $OCTAVIA_AMP_IMAGE_SIZE ]; then | 23:21 |
| rm_work | less confusing | 23:21 |
| rm_work | lol yes | 23:21 |
| cgoncalves | [ ] to be POSIX | 23:21 |
| rm_work | orl | 23:21 |
| rm_work | *orly | 23:22 |
| rm_work | k well, our devstack plugin uses [[ ]] | 23:22 |
| rm_work | not sure what the difference is | 23:22 |
| cgoncalves | it's okay. devstack officialy only supports bash | 23:23 |
| cgoncalves | there was a discussion recently around source vs . and [ ] vs [[ ]] also came up | 23:23 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix kvm-centos.7 gate https://review.openstack.org/550487 | 23:25 |
| rm_work | anyway, i don't understand how this test is failing, especially intermittently | 23:29 |
| rm_work | it should fail every time or never | 23:29 |
| imacdonn | basically, as I understand it, [ ] is supposed to be POSIX compliant, and [[ ]] allows you to use advanced (bash) features that are not part of POSIX | 23:29 |
| rm_work | either the API can take a missing project-id or it can't | 23:29 |
| imacdonn | in the above case, it seems it shouldn't make any difference | 23:30 |
| rm_work | i wonder if tempest changed something ... fff | 23:31 |
| rm_work | like, just now or somesuch | 23:32 |
| *** threestrands_ has joined #openstack-lbaas | 23:34 | |
| *** threestrands_ has quit IRC | 23:34 | |
| *** threestrands_ has joined #openstack-lbaas | 23:34 | |
| *** threestrands_ has quit IRC | 23:35 | |
| *** threestrands_ has joined #openstack-lbaas | 23:36 | |
| *** threestrands has quit IRC | 23:36 | |
| cgoncalves | at first glance, I can't find any keystone, neutron, neutron-lib or tempest patch merged in master in last couple of days that could explain these behaviors | 23:40 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 23:45 | |
| *** AlexeyAbashkin has quit IRC | 23:49 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: [WIP] Add grenade support https://review.openstack.org/549654 | 23:52 |
| cgoncalves | rm_work: 2018-03-07 23:08:34.775683 | primary | 2018-03-07 23:08:34.775 | 2018-03-07 23:07:37,323 25085 INFO [tempest.lib.common.rest_client] Request (TestListenerBasic:_run_cleanups): 404 GET http://137.74.30.69/compute/v2.1/servers/5aeb14b8-e430-401c-a043-c6bc6b0b2583 0.058s | 23:55 |
| cgoncalves | http://logs.openstack.org/82/550282/6/check/neutron-lbaasv2-dsvm-scenario-namespace/7352206/job-output.txt.gz | 23:55 |
| *** fnaval has quit IRC | 23:56 | |
| cgoncalves | there's a DELETE server before | 23:56 |
| rm_work | for the record, you can link directly to those lines if you click the timestamp in the log | 23:57 |
| cgoncalves | d'oh xD | 23:57 |
| cgoncalves | http://logs.openstack.org/82/550282/6/check/neutron-lbaasv2-dsvm-scenario-namespace/7352206/job-output.txt.gz#_2018-03-07_23_07_38_627758 | 23:57 |
| cgoncalves | ^ DELETE pool | 23:57 |
| rm_work | the 404 wasn't the problem | 23:57 |
| cgoncalves | and soon after... | 23:57 |
| cgoncalves | http://logs.openstack.org/82/550282/6/check/neutron-lbaasv2-dsvm-scenario-namespace/7352206/job-output.txt.gz#_2018-03-07_23_07_38_634810 | 23:57 |
| rm_work | the problem was a BadStatusLine | 23:58 |
| rm_work | which is an intermittent thing i get from Openstack even in production clouds | 23:58 |
| rm_work | i have never figured out what causes it | 23:58 |
| rm_work | the 404 there should be expected and handled | 23:59 |
| cgoncalves | k | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!