| *** Swami__ has joined #openstack-lbaas | 00:00 | |
| rm_work| | cool, back | 00:01 |
|---|---|---|
| johnsom | So confused at what I broke with that patch. From what I see nothing really uses what I changed... hmmm | 00:03 |
| *** Swami has quit IRC | 00:04 | |
| *** Swami_ has quit IRC | 00:04 | |
| *** Swami has joined #openstack-lbaas | 00:04 | |
| rm_work | seems straightforward? | 00:05 |
| rm_work | http://logs.openstack.org/17/433817/1/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial-nv/ca53322/console.html#_2017-02-14_18_21_05_596491 | 00:05 |
| rm_work | you changed the service from octavia to loadbalancer | 00:05 |
| rm_work | like if keystore were type "keystone" and then you changed it to type "identity" right? | 00:06 |
| rm_work | *keystone | 00:06 |
| johnsom | No, that isn't what I did... | 00:06 |
| rm_work | i thought it was? | 00:06 |
| rm_work | that's what we NEED to do... | 00:06 |
| rm_work | right? | 00:07 |
| johnsom | For neutron the service name is "neutron" and the service type is "networking", I changed ours to be "octavia" and "loadbalancing" | 00:07 |
| rm_work | right | 00:07 |
| johnsom | Where it was "octavia" "octavia" before | 00:07 |
| rm_work | for keystone, service name is keystone and type is identity | 00:07 |
| rm_work | which is how stuff looks it up in the service catalog | 00:07 |
| rm_work | by "identity" | 00:08 |
| johnsom | Right, here look at this: http://logs.openstack.org/17/433817/1/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial-nv/ca53322/logs/devstacklog.txt.gz#_2017-02-14_18_14_40_222 | 00:08 |
| johnsom | This looks right to me | 00:08 |
| rm_work | right | 00:08 |
| rm_work | so the service type is "loadbalancing" | 00:09 |
| rm_work | so now when things look it up in the service catalog, they need to look for type "loadbalancing" instead of "octavia" | 00:09 |
| rm_work | "Details: service: octavia, region: RegionOne, endpoint_type: publicURL, name: None" | 00:09 |
| rm_work | it's looking for service_type "octavia" still | 00:10 |
| *** rm_work has quit IRC | 00:11 | |
| *** rm_work| is now known as rm_work | 00:11 | |
| *** rm_work has joined #openstack-lbaas | 00:11 | |
| openstackgerrit | Merged openstack/octavia master: Add common api handlers for v1 and v2 https://review.openstack.org/433771 | 00:11 |
| johnsom | My best guess is this line is just borked: https://github.com/openstack/octavia/blob/master/octavia/tests/tempest/v1/scenario/base.py#L77 | 00:12 |
| rm_work | yes | 00:13 |
| rm_work | that one | 00:13 |
| rm_work | needs to be "loadbalancing" | 00:13 |
| rm_work | I was just trying to link that too, lol | 00:13 |
| rm_work | maybe it should be a constant for our service name... | 00:13 |
| johnsom | It doesn't even take name, so how is it supposed to find the right "loadbalancing" service? | 00:13 |
| rm_work | ?? dunno, that's just how keystone's service catalog works | 00:14 |
| rm_work | you provide a service type, get a response, use it | 00:14 |
| rm_work | I think service type is unique | 00:14 |
| rm_work | ? | 00:14 |
| johnsom | No, I think it's tempest not keystone at fault here | 00:14 |
| rm_work | err | 00:14 |
| rm_work | i mean, it's just the config | 00:14 |
| rm_work | not tempest or keystone | 00:14 |
| rm_work | keystone does what it's supposed to -- gives you a service when you ask for one | 00:14 |
| rm_work | we're just supplying the wrong service type | 00:15 |
| johnsom | Ah, maybe that is why it's volume, volume2, volume3 for cinder | 00:15 |
| johnsom | https://www.irccloud.com/pastebin/y92zF1di/ | 00:15 |
| rm_work | right | 00:15 |
| rm_work | you don't ask for nova, for instance | 00:15 |
| rm_work | you ask for compute | 00:15 |
| rm_work | and it gives you nova | 00:15 |
| rm_work | they're slot-in services | 00:15 |
| johnsom | Hmm, I would have expected "networking" to include nova and neutron and you select by service name. | 00:16 |
| rm_work | no :/ | 00:16 |
| johnsom | oddness | 00:16 |
| rm_work | I guess I just happened to run into this early in my openstack career so i never thought about it | 00:16 |
| johnsom | So, yeah, changing that could be, ummm, evil | 00:16 |
| rm_work | i assume it'd be expected | 00:17 |
| rm_work | we should be operating under "loadbalancing" | 00:17 |
| johnsom | I mean that API was never "stable" but still, it could break folks if they are using v1 | 00:17 |
| rm_work | and then we should have the client start looking for that endpoint | 00:17 |
| rm_work | and eventually deprecate looking under "networking" for LBaaS | 00:17 |
| rm_work | right but if we register as loadbalancing, they aren't going direct to us anyway | 00:18 |
| rm_work | they still go through a neutron proxy | 00:18 |
| rm_work | until the deprecation is complete | 00:18 |
| rm_work | but it allows for newer clients to go direct to our endpoint | 00:18 |
| rm_work | thus skipping the need for the stupid passthrough proxy thing | 00:18 |
| rm_work | I'm very confident that this *is* the right path forward | 00:19 |
| rm_work | and I'm not sure it'll break anything | 00:19 |
| johnsom | Right this isn't changing anything about neutron or the networking service, just the "octavia" "octavia" service we were putting in devstack | 00:19 |
| rm_work | yeah | 00:19 |
| johnsom | Ok, fixing | 00:20 |
| rm_work | and moving forward we'd recommend people do that in their clouds as well, because that's how clients will work moving forward | 00:20 |
| rm_work | and we should also deprecate /lbaas at the same time | 00:20 |
| rm_work | because ... there's no reason to have it | 00:20 |
| johnsom | Yeah, I think the plan is to support both for now | 00:21 |
| johnsom | Though /lbaas is handy to do odd l7 type things with the API | 00:22 |
| rm_work | hmm | 00:22 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix the service type for octavia https://review.openstack.org/433817 | 00:24 |
| johnsom | I checked, we never documented that in the v1 API-ref | 00:24 |
| johnsom | So, just updating our new api-ref | 00:25 |
| *** catintheroof has quit IRC | 00:29 | |
| rm_work | err | 00:29 |
| rm_work | heh k | 00:29 |
| rm_work | yeah i saw it in the new one a couple places | 00:29 |
| *** fnaval has quit IRC | 00:29 | |
| johnsom | Well, also note, the endpoints actually use the project name, not the service type. All of the endpoints are "neutron" and "octavia" | 00:32 |
| johnsom | Welll, I guess it has both there | 00:32 |
| johnsom | Hmm, Yeah, I need to rework that part of the api-ref a bit | 00:33 |
| *** Guest3904 is now known as medberry | 00:43 | |
| *** medberry has quit IRC | 00:43 | |
| *** medberry has joined #openstack-lbaas | 00:43 | |
| *** fnaval has joined #openstack-lbaas | 00:43 | |
| *** medberry is now known as med_ | 00:43 | |
| *** gongysh has joined #openstack-lbaas | 00:59 | |
| *** gongysh has quit IRC | 01:14 | |
| openstackgerrit | Sindhu Devale proposed openstack/octavia master: Add v2 load balancer type and controllers https://review.openstack.org/423543 | 01:17 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Adds v2 general API information section https://review.openstack.org/433392 | 01:28 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/432620 | 01:30 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements https://review.openstack.org/434012 | 01:32 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Adds v2 general API information section https://review.openstack.org/433392 | 01:39 |
| openstackgerrit | Merged openstack/octavia master: Remove config option "amp_network" https://review.openstack.org/432805 | 01:42 |
| *** gongysh has joined #openstack-lbaas | 02:05 | |
| xgerman | back | 02:11 |
| xgerman | agree with Adam -> new clients should go to us; octavia/loadbalancing seems sane | 02:13 |
| rm_work | johnsom: https://gist.github.com/rm-you/30da92a54d5d4123f006e6108cc23ba0 | 02:16 |
| rm_work | finally got around to writing a .... "client" for octavia v1 | 02:16 |
| rm_work | oops fixed a bug just now | 02:19 |
| johnsom | Nice! | 02:20 |
| rm_work | err and one more bug in my if statement... | 02:21 |
| rm_work | but yeah. :/ | 02:21 |
| rm_work | lack of client is T_T | 02:21 |
| johnsom | I am going to be offline this evening. Making lamb for valentines day. | 02:21 |
| rm_work | have fun! | 02:21 |
| johnsom | It is on the list for Pike | 02:21 |
| rm_work | k fixed that client code | 02:34 |
| *** fnaval has quit IRC | 02:38 | |
| *** fnaval has joined #openstack-lbaas | 02:38 | |
| rm_work | i would REALLY like to figure out single-create... | 02:40 |
| *** fnaval has quit IRC | 02:41 | |
| xgerman | :-) | 02:45 |
| rm_work | it SHOULD work, right? | 02:45 |
| rm_work | but I haven't gotten it to work yet... | 02:45 |
| xgerman | yeah, I think so | 02:45 |
| rm_work | I just need to figure out the right syntax | 02:45 |
| rm_work | ah I think I was trying to do it with Neutron | 02:45 |
| rm_work | maybe it'll actually work with Octavia | 02:45 |
| *** fnaval has joined #openstack-lbaas | 02:45 | |
| *** fnaval has quit IRC | 02:46 | |
| xgerman | neutron has it as well in someway | 02:46 |
| rm_work | well... i don't think the extension is loading properly >_> | 02:46 |
| *** yuanying has quit IRC | 02:55 | |
| xgerman | :-( | 03:00 |
| rm_work | testing to see if i can get octavia singlecreate to work | 03:00 |
| *** amotoki has joined #openstack-lbaas | 03:02 | |
| *** robcresswell_ has joined #openstack-lbaas | 03:04 | |
| rm_work | yeah that's not... great | 03:04 |
| *** coreycb` has joined #openstack-lbaas | 03:07 | |
| *** ankur-gupta-f4 has quit IRC | 03:07 | |
| *** coreycb has quit IRC | 03:07 | |
| *** robcresswell has quit IRC | 03:07 | |
| *** robcresswell_ is now known as robcresswell | 03:09 | |
| rm_work | quotas broke singe-create T_T | 03:09 |
| rm_work | filing a bug | 03:09 |
| rm_work | https://bugs.launchpad.net/octavia/+bug/1664807 | 03:18 |
| openstack | Launchpad bug 1664807 in octavia "Single-create is broken (since quotas?)" [Undecided,New] | 03:18 |
| rm_work | johnsom: ^^ :( | 03:18 |
| *** amotoki has quit IRC | 03:19 | |
| rm_work | gonna look at that tomorrow I guess :/ | 03:20 |
| rm_work | off for the night | 03:20 |
| *** ankur-gupta-f4 has joined #openstack-lbaas | 03:31 | |
| *** armax has quit IRC | 03:52 | |
| openstackgerrit | Ankur proposed openstack/octavia master: Octavia v2 API for listeners https://review.openstack.org/424744 | 04:01 |
| openstackgerrit | Ankur proposed openstack/octavia master: Add v2 load balancer type and controllers https://review.openstack.org/423543 | 04:01 |
| openstackgerrit | Ankur proposed openstack/octavia master: Introduce Octavia v2 API for pools https://review.openstack.org/405922 | 04:01 |
| *** yuanying has joined #openstack-lbaas | 04:08 | |
| *** amotoki has joined #openstack-lbaas | 04:22 | |
| *** links has joined #openstack-lbaas | 04:34 | |
| *** amotoki has quit IRC | 05:26 | |
| *** faizy has joined #openstack-lbaas | 05:28 | |
| *** amotoki has joined #openstack-lbaas | 05:36 | |
| *** amotoki has quit IRC | 05:53 | |
| *** amotoki has joined #openstack-lbaas | 06:02 | |
| *** gcheresh has joined #openstack-lbaas | 06:03 | |
| *** amotoki has quit IRC | 06:10 | |
| *** pcaruana has joined #openstack-lbaas | 06:20 | |
| *** amotoki has joined #openstack-lbaas | 06:21 | |
| *** amotoki_ has joined #openstack-lbaas | 06:37 | |
| *** amotoki has quit IRC | 06:40 | |
| *** Swami__ has quit IRC | 06:53 | |
| *** Swami has quit IRC | 06:53 | |
| *** Alex_Staf has joined #openstack-lbaas | 06:59 | |
| *** anilvenkata has joined #openstack-lbaas | 07:04 | |
| *** rcernin has joined #openstack-lbaas | 07:07 | |
| *** jsheeren has joined #openstack-lbaas | 07:28 | |
| *** eezhova has joined #openstack-lbaas | 07:28 | |
| *** Alex_Staf has quit IRC | 07:31 | |
| *** tesseract has joined #openstack-lbaas | 07:31 | |
| *** yamamoto has quit IRC | 07:38 | |
| *** krypto has joined #openstack-lbaas | 07:45 | |
| *** kobis has joined #openstack-lbaas | 07:45 | |
| *** anilvenkata has quit IRC | 07:59 | |
| *** amotoki_ has quit IRC | 08:14 | |
| *** anilvenkata has joined #openstack-lbaas | 08:15 | |
| *** eezhova has quit IRC | 08:22 | |
| *** amotoki has joined #openstack-lbaas | 08:24 | |
| *** ipsecguy has joined #openstack-lbaas | 08:28 | |
| *** links has quit IRC | 08:36 | |
| *** csomerville has quit IRC | 08:38 | |
| *** cody-somerville has joined #openstack-lbaas | 08:38 | |
| *** krypto has quit IRC | 08:41 | |
| *** yuanying has quit IRC | 08:47 | |
| *** yuanying has joined #openstack-lbaas | 08:47 | |
| *** yuanying has quit IRC | 08:47 | |
| *** links has joined #openstack-lbaas | 08:52 | |
| *** eezhova has joined #openstack-lbaas | 09:05 | |
| *** amotoki has quit IRC | 09:05 | |
| *** Alex_Staf has joined #openstack-lbaas | 09:10 | |
| *** yamamoto has joined #openstack-lbaas | 09:23 | |
| *** yamamoto_ has joined #openstack-lbaas | 09:24 | |
| *** yamamoto_ has quit IRC | 09:25 | |
| *** yamamoto_ has joined #openstack-lbaas | 09:26 | |
| *** yamamoto_ has quit IRC | 09:26 | |
| *** links has quit IRC | 09:26 | |
| *** yuanying has joined #openstack-lbaas | 09:26 | |
| *** yamamoto has quit IRC | 09:28 | |
| *** jsheeren has quit IRC | 09:32 | |
| *** jsheeren has joined #openstack-lbaas | 09:32 | |
| *** jsheeren has quit IRC | 09:33 | |
| *** jsheeren has joined #openstack-lbaas | 09:33 | |
| *** jsheeren has quit IRC | 09:33 | |
| *** jsheeren has joined #openstack-lbaas | 09:33 | |
| *** ipsecguy has quit IRC | 09:34 | |
| *** jsheeren has quit IRC | 09:34 | |
| *** jsheeren has joined #openstack-lbaas | 09:34 | |
| *** links has joined #openstack-lbaas | 09:43 | |
| *** mdavidson has joined #openstack-lbaas | 09:54 | |
| *** anilvenkata has quit IRC | 10:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements https://review.openstack.org/434012 | 10:24 |
| *** gongysh has quit IRC | 10:42 | |
| *** yamamoto has joined #openstack-lbaas | 11:53 | |
| *** Kdecherf has quit IRC | 12:09 | |
| *** Kdecherf has joined #openstack-lbaas | 12:24 | |
| *** catintheroof has joined #openstack-lbaas | 12:36 | |
| *** faizy has quit IRC | 12:39 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** yamamoto has joined #openstack-lbaas | 12:50 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** coreycb` is now known as coreycb | 12:51 | |
| *** coreycb has quit IRC | 12:52 | |
| *** jsheeren_ has joined #openstack-lbaas | 12:53 | |
| *** coreycb has joined #openstack-lbaas | 12:53 | |
| *** jsheeren_ has left #openstack-lbaas | 12:53 | |
| *** dileepr has quit IRC | 13:09 | |
| *** amotoki has joined #openstack-lbaas | 13:10 | |
| *** faizy has joined #openstack-lbaas | 13:15 | |
| *** jsheeren has quit IRC | 13:36 | |
| *** jsheeren has joined #openstack-lbaas | 13:38 | |
| *** jsheeren has left #openstack-lbaas | 13:42 | |
| *** beardedeagle has quit IRC | 13:42 | |
| *** beardedeagle has joined #openstack-lbaas | 13:43 | |
| *** beardedeagle has quit IRC | 13:47 | |
| *** jsheeren has joined #openstack-lbaas | 13:59 | |
| *** jsheeren has quit IRC | 14:09 | |
| *** matt-borland has joined #openstack-lbaas | 14:15 | |
| *** sanfern has joined #openstack-lbaas | 14:16 | |
| *** jsheeren has joined #openstack-lbaas | 14:18 | |
| *** beardedeagle has joined #openstack-lbaas | 14:26 | |
| *** links has quit IRC | 14:31 | |
| *** chlong has quit IRC | 14:37 | |
| *** Swami has joined #openstack-lbaas | 14:44 | |
| *** fnaval has joined #openstack-lbaas | 14:54 | |
| *** fnaval_ has joined #openstack-lbaas | 14:58 | |
| *** fnaval has quit IRC | 15:01 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Auto detect haproxy user_group https://review.openstack.org/429398 | 15:03 |
| diltram | morning | 15:12 |
| *** armax has joined #openstack-lbaas | 15:18 | |
| *** frickler has joined #openstack-lbaas | 15:18 | |
| frickler | so neutron-lbaas-dashboard belongs to octavia now, is that correct? and 1.0.0 is stable/mitaka, 2.0.0 is stable/ocata and there is no stable/newton existing or planned? | 15:22 |
| *** jsheeren has quit IRC | 15:24 | |
| openstackgerrit | Merged openstack/octavia master: Updated from global requirements https://review.openstack.org/434012 | 15:33 |
| diltram | frickler: yes | 15:37 |
| diltram | no there is no stable/newton because there was no progress in project | 15:37 |
| johnsom | frickler: you can run stable/mitaka with newton | 15:41 |
| *** sshank has joined #openstack-lbaas | 15:42 | |
| *** amotoki has quit IRC | 15:55 | |
| *** chlong has joined #openstack-lbaas | 16:04 | |
| *** kobis has quit IRC | 16:06 | |
| *** gcheresh has quit IRC | 16:06 | |
| frickler | o.k., I'll try that, thanks | 16:13 |
| openstackgerrit | Shashank Kumar Shankar proposed openstack/octavia master: Introduce Octavia v2 API for pools https://review.openstack.org/405922 | 16:14 |
| johnsom | Ok, spinning up a third desktop to see what I broke with single call create.... | 16:21 |
| *** faizy has quit IRC | 16:28 | |
| *** eezhova has quit IRC | 16:32 | |
| johnsom | rm_work Ping me when you have a few minutes. I can't repro your bug | 16:34 |
| johnsom | xgerman or diltram if you have a master devstack running I can give you the curl to try a single call create. Maybe you can try it too? | 16:37 |
| xgerman | I would need to boot the VM | 16:38 |
| xgerman | will go ahead | 16:38 |
| johnsom | curl -s -X POST -H Content-type:application/json -d '{"name": "test_lb", "project_id": "cf418ad823834c28b2ad3888d4b01fa8", "vip": {"subnet_id": "022f1424-6306-42fe-89de-204578eddf6f"}, "listeners": [{"name": "test_listener", "protocol": "HTTP", "protocol_port": 80, "default_pool": {"name":"test_pool1", "protocol": "HTTP", "lb_algorithm": "ROUND_ROBIN", | 16:39 |
| johnsom | "members": [{"ip_address": "172.21.1.15", "protocol_port": 80}, {"ip_address": "172.21.2.12", "protocol_port": 80}]}}]}' http://172.21.21.135:9876/v1/loadbalancers | 16:39 |
| *** dileepr has joined #openstack-lbaas | 16:42 | |
| *** rcernin has quit IRC | 16:43 | |
| *** kobis has joined #openstack-lbaas | 16:48 | |
| openstackgerrit | Sindhu Devale proposed openstack/octavia master: Octavia v2 API for listeners https://review.openstack.org/424744 | 16:53 |
| *** tesseract has quit IRC | 16:58 | |
| openstackgerrit | Nakul Dahiwade proposed openstack/octavia master: Introduce Octavia v2 API for Members https://review.openstack.org/405932 | 17:04 |
| *** eezhova has joined #openstack-lbaas | 17:09 | |
| *** faizy has joined #openstack-lbaas | 17:09 | |
| xgerman | https://www.irccloud.com/pastebin/Za6izoBt/ | 17:15 |
| xgerman | ^^ johnsom | 17:15 |
| xgerman | looks good | 17:15 |
| johnsom | Yeah, good. That is what I got too. I wonder if he wasn't doing pdb things that left a DB lock in place.... | 17:16 |
| johnsom | Thanks for confirming | 17:16 |
| openstackgerrit | Sindhu Devale proposed openstack/octavia master: Octavia v2 API for listeners https://review.openstack.org/424744 | 17:16 |
| *** csomerville has joined #openstack-lbaas | 17:21 | |
| openstackgerrit | Ankur proposed openstack/octavia master: Add v2 load balancer type and controllers https://review.openstack.org/423543 | 17:23 |
| *** chlong has quit IRC | 17:24 | |
| *** cody-somerville has quit IRC | 17:24 | |
| *** Alex_Staf has quit IRC | 17:33 | |
| *** kobis has quit IRC | 17:43 | |
| *** faizy has quit IRC | 17:45 | |
| *** kobis has joined #openstack-lbaas | 17:49 | |
| *** kobis has quit IRC | 17:50 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix the service type for octavia https://review.openstack.org/433817 | 18:07 |
| *** Swami_ has joined #openstack-lbaas | 18:17 | |
| *** Swami has quit IRC | 18:20 | |
| *** gcheresh has joined #openstack-lbaas | 18:31 | |
| *** ipsecguy has joined #openstack-lbaas | 18:35 | |
| *** Swami_ has quit IRC | 18:37 | |
| openstackgerrit | Sindhu Devale proposed openstack/octavia master: Octavia v2 API for listeners https://review.openstack.org/424744 | 18:44 |
| *** pcaruana has quit IRC | 19:03 | |
| nmagnezi | o/ | 19:16 |
| nmagnezi | diltram, welcome back! | 19:16 |
| openstackgerrit | Shashank Kumar Shankar proposed openstack/octavia master: Align Octavia API to n-lbaasv2 for L7Rules https://review.openstack.org/406336 | 19:20 |
| rm_work | johnsom: hmmm | 19:24 |
| rm_work | johnsom: no PDB here... | 19:24 |
| rm_work | it locked up in the API before it even got through to the CW | 19:24 |
| rm_work | let me try again | 19:24 |
| johnsom | Right, that is where it should happen | 19:25 |
| rm_work | yeah it's doing it | 19:25 |
| johnsom | However you should only get that error if you sent like hundreds of create calls all at once | 19:25 |
| rm_work | and looking at the code, it makes sense to me? | 19:25 |
| rm_work | err.... | 19:25 |
| johnsom | So you still have it going on? | 19:25 |
| rm_work | yes | 19:25 |
| rm_work | every time | 19:26 |
| johnsom | rm_work SHOW FULL PROCESSLIST; | 19:26 |
| rm_work | AH | 19:26 |
| johnsom | in mysql | 19:26 |
| rm_work | I didn't pass projectid? | 19:26 |
| rm_work | maybe that's different? | 19:26 |
| rm_work | I am using a token | 19:26 |
| johnsom | You should get an error that you didn't specify a project ID | 19:27 |
| johnsom | No DB deadlock | 19:27 |
| rm_work | but i'm using an auth token | 19:27 |
| rm_work | so it pulls the token from that | 19:27 |
| rm_work | err | 19:27 |
| rm_work | project_id from that | 19:27 |
| rm_work | I have keystone auth enabled | 19:27 |
| johnsom | Yeah, your log said it had a project ID anyway | 19:28 |
| rm_work | right because it pulled it | 19:28 |
| rm_work | but your curl isn't using auth | 19:28 |
| rm_work | what are you looking for in the processlist? | 19:28 |
| johnsom | Right | 19:28 |
| rm_work | just saying, you aren't exactly reproducing using my method :P | 19:28 |
| johnsom | To see which process has the DB row locked causing your error | 19:28 |
| rm_work | where would it show a lock | 19:29 |
| johnsom | I'm not sure actually, trying to google it now | 19:30 |
| johnsom | Mine shows all "sleep" | 19:30 |
| johnsom | Ah, this has comments about open locks too: show engine innodb status; | 19:31 |
| rm_work | https://gist.github.com/rm-you/4780c17e754d4b719608483b32d688c9 | 19:32 |
| *** sanfern has quit IRC | 19:32 | |
| johnsom | 1992 octavia localhost:52582 octavia Query 46 statistics SELECT quotas.project_id AS quotas_project_id, quotas.health_monitor AS quotas_health_monitor, quotas.listener AS quotas_listener, quotas.load_balancer AS quotas_load_balancer, quotas.member AS quotas_member, quotas.pool AS quotas_pool, quotas.in_use_health_monitor AS | 19:32 |
| johnsom | quotas_in_use_health_monitor, quotas.in_use_listener AS quotas_in_use_listener, quotas.in_use_load_balancer AS quotas_in_use_load_balancer, quotas.in_use_member AS quotas_in_use_member, quotas.in_use_pool AS quotas_in_use_pool \nFROM quotas \nWHERE quotas.project_id = 'e54e5a6636324fdda3f338a96e9fd60c' \n LIMIT 1 FOR UPDATE | 19:32 |
| johnsom | That is it | 19:32 |
| rm_work | right | 19:32 |
| rm_work | i assume so | 19:32 |
| rm_work | figured out you meant to do it DURING the hang :P | 19:32 |
| rm_work | (duh) | 19:33 |
| johnsom | Hmm, that isn't helpful actually. | 19:33 |
| johnsom | Yeah, I haven't done this type of stuff on mysql in a long time | 19:33 |
| johnsom | This 'show engine innodb status;" seems to give more useful info | 19:35 |
| rm_work | https://gist.githubusercontent.com/rm-you/6d90f10d393e23ef47d9b69e962ed588/raw/bef88cff5a6018adf33f363e40619e56268b7fc6/gistfile1.txt | 19:35 |
| *** chlong has joined #openstack-lbaas | 19:36 | |
| rm_work | fixed: https://gist.github.com/rm-you/6d90f10d393e23ef47d9b69e962ed588 | 19:42 |
| johnsom | SELECT pl.id ,pl.user ,pl.state ,it.trx_id ,it.trx_mysql_thread_id ,it.trx_query AS query ,it.trx_id AS blocking_trx_id ,it.trx_mysql_thread_id AS blocking_thread ,it.trx_query AS blocking_query FROM information_schema.processlist AS pl INNER JOIN information_schema.innodb_trx AS it ON pl.id = it.trx_mysql_thread_id INNER JOIN | 19:43 |
| johnsom | information_schema.innodb_lock_waits AS ilw ON it.trx_id = ilw.requesting_trx_id AND it.trx_id = ilw.blocking_trx_id; | 19:43 |
| rm_work | ... | 19:46 |
| johnsom | I have no idea why that didn't pastebin | 19:46 |
| rm_work | regardless | 19:47 |
| rm_work | the '...' was for "... what" | 19:47 |
| johnsom | https://www.irccloud.com/pastebin/tbXL6cgw/ | 19:48 |
| rm_work | oh | 19:48 |
| rm_work | you want me to run that | 19:48 |
| rm_work | ? | 19:48 |
| johnsom | That is a sql script folks say will give you info on what is blocking your query | 19:48 |
| rm_work | empty | 19:49 |
| johnsom | Yeah. I mean, I can tell you how to fix it, just restart mysql. That will close the transaction for whatever is holding the quota row lock | 19:49 |
| rm_work | oh you think something was already holding a lock, unrelated? | 19:49 |
| johnsom | Yeah | 19:50 |
| rm_work | hmm | 19:50 |
| johnsom | Somehow a previous transaction is stuck open | 19:50 |
| *** chlong has quit IRC | 19:51 | |
| johnsom | Like if you were using pdb and exited without letting the transaction close. | 19:51 |
| rm_work | restarted | 19:51 |
| rm_work | nope. | 19:51 |
| rm_work | also, never used pdb on this | 19:51 |
| rm_work | nah i see both quota checks happen | 19:52 |
| rm_work | https://gist.github.com/rm-you/f93521aea8c41e13672a9809f453c29b | 19:53 |
| rm_work | see, make the query and it makes two quota checks instantly: https://gist.github.com/rm-you/f93521aea8c41e13672a9809f453c29b | 19:53 |
| rm_work | err, same pastebin | 19:53 |
| rm_work | looking at create_load_balancer_tree | 19:56 |
| rm_work | shouldn't the quota checks be upfront anyway? | 19:56 |
| rm_work | i dunno, seems weird that it goes through objects one by one and checks quotas | 19:57 |
| johnsom | It's all or none, so we can do it as the code parses the request. It's that or parse it twice. | 19:58 |
| johnsom | I just don't understand why your database is deadlocked | 19:58 |
| rm_work | ah right it's in a transaction, duh | 19:58 |
| johnsom | mysql is not very nice about telling you what has it deadlocked | 19:59 |
| rm_work | hmmm maybe i could update the code to show me a better traceback of at least where in the code it was when it got a deadlock | 19:59 |
| johnsom | Ok, meeting time | 19:59 |
| rm_work | ah k | 19:59 |
| *** ducttape_ has joined #openstack-lbaas | 20:00 | |
| rm_work | the deadlock is on repositories.py line 501 | 20:07 |
| rm_work | first check of quota in create_load_balancer_tree | 20:08 |
| *** m-greene has joined #openstack-lbaas | 20:19 | |
| *** dileepr has quit IRC | 20:19 | |
| *** kbyrne has quit IRC | 20:22 | |
| *** _ducttape_ has joined #openstack-lbaas | 20:22 | |
| *** ducttape_ has quit IRC | 20:26 | |
| *** gcheresh has quit IRC | 20:32 | |
| *** jerrygb has joined #openstack-lbaas | 20:36 | |
| rm_work | johnsom: why does it get another lock session on line 496? | 20:39 |
| nmagnezi | rm_work, I think I managed to follow you suggestion about not duplicating the default values to the devstack plugin https://review.openstack.org/#/c/431576/4/devstack/plugin.sh :) | 20:39 |
| rm_work | the session that passed in WAS a lock session | 20:39 |
| rm_work | load_balancer.py line 124 makes a lock_session | 20:39 |
| rm_work | passes it to _create_load_balancer_graph_db | 20:40 |
| rm_work | which passes it to create_load_balancer_tree | 20:40 |
| rm_work | which ignores the existing session and makes another | 20:40 |
| rm_work | err sorry one sec | 20:40 |
| rm_work | db_prepare.create_load_balancer_tree | 20:41 |
| rm_work | that one | 20:41 |
| rm_work | err sorry | 20:41 |
| rm_work | NOT that one | 20:41 |
| rm_work | repositories.create_load_balancer_tree | 20:41 |
| johnsom | rm_work session is the DB session in the context. It is not used for locking. | 20:43 |
| rm_work | but it IS the lock_session | 20:44 |
| johnsom | Only the "lock_session" is used for locking. Both sessions go along in parallel | 20:44 |
| rm_work | trace back | 20:44 |
| rm_work | when it is passed through it IS "lock_session" | 20:44 |
| rm_work | then it's accepted as just "session" | 20:44 |
| rm_work | yep fixed | 20:45 |
| rm_work | basically, need to remove the "with session.begin" and the new lock_session | 20:45 |
| rm_work | because we're already IN a lock_session | 20:46 |
| johnsom | What????? | 20:46 |
| rm_work | screenshare? I can walk you through | 20:46 |
| johnsom | This works fine for german and I so..... | 20:46 |
| rm_work | i mean it's pretty obvious just looking at the code, to me | 20:46 |
| johnsom | Yeah, just a minute | 20:46 |
| xgerman | yeah, I used a devstack | 20:46 |
| rm_work | are you using *auth*? | 20:47 |
| rm_work | maybe it is different? | 20:47 |
| rm_work | hop on this: https://godaddy.zoom.us/j/606762661 | 20:47 |
| *** dileepr has joined #openstack-lbaas | 20:51 | |
| openstackgerrit | Shashank Kumar Shankar proposed openstack/octavia master: Introduce Octavia v2 API for pools https://review.openstack.org/405922 | 20:53 |
| -openstackstatus- NOTICE: We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear. | 20:56 | |
| rm_work | I think screenshares are really the most effective way to do stuff like this | 21:06 |
| rm_work | bar none | 21:06 |
| *** _ducttape_ has quit IRC | 21:07 | |
| *** ducttape_ has joined #openstack-lbaas | 21:08 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix quota lock on single_create with auth enabled https://review.openstack.org/434518 | 21:08 |
| rm_work | OK cool, so single-create *works*! :P | 21:17 |
| rm_work | yay | 21:17 |
| openstackgerrit | Shashank Kumar Shankar proposed openstack/octavia master: Introduce Octavia v2 API for pools https://review.openstack.org/405922 | 21:30 |
| *** ducttape_ has quit IRC | 21:40 | |
| *** ipsecguy_ has joined #openstack-lbaas | 21:51 | |
| *** ipsecguy has quit IRC | 21:52 | |
| rm_work | err, sorry if this is repeat, but my connection is wonky: | 21:53 |
| rm_work | xgerman: did cascade-delete ever make it into octavia? | 21:53 |
| xgerman | don’t know | 21:53 |
| xgerman | I think johnsom said it didn’t make it | 21:54 |
| johnsom | Yeah, I am not sure either | 21:55 |
| openstackgerrit | Sindhu Devale proposed openstack/octavia master: Octavia v2 API for listeners https://review.openstack.org/424744 | 22:12 |
| openstackgerrit | Sindhu Devale proposed openstack/octavia master: [WIP] Octavia developer quick start guide https://review.openstack.org/410419 | 22:20 |
| *** ducttape_ has joined #openstack-lbaas | 22:29 | |
| xgerman | Hu? | 22:41 |
| xgerman | https://usercontent.irccloud-cdn.com/file/VbBw1rGp/Screen%20Shot%202017-02-15%20at%205.41.33%20PM.png | 22:41 |
| xgerman | ^ johnsom, rm_work… | 22:42 |
| johnsom | No clue | 22:42 |
| xgerman | I remember some other dude to talk on Octavia we didn’t know a few years back ;-) | 22:43 |
| *** matt-borland has quit IRC | 22:43 | |
| rm_work | huh | 22:44 |
| rm_work | well, cool | 22:44 |
| rm_work | umm | 22:44 |
| johnsom | Keith Berger from HPE | 22:44 |
| rm_work | anyone in ATL? :P | 22:44 |
| johnsom | ^^^ xgerman | 22:44 |
| johnsom | https://www.meetup.com/openstack-atlanta/events/237189297/?a=socialmedia | 22:44 |
| rm_work | oh god is he demoing Liberty? >_> | 22:44 |
| xgerman | yeah, I was surprised, too, about the HPE angle | 22:44 |
| johnsom | Yeah, exactly | 22:45 |
| rm_work | please no | 22:45 |
| rm_work | can we pretend nothing actually exists before like.... newton | 22:45 |
| rm_work | or ocata really | 22:45 |
| openstackgerrit | Nakul Dahiwade proposed openstack/octavia master: Introduce Octavia v2 API for Members https://review.openstack.org/405932 | 22:45 |
| rm_work | we're just finally getting to what I'd call "usable" | 22:45 |
| johnsom | Newton was a good release. | 22:45 |
| rm_work | still a lot of stuff that's pretty breaking that we're just now finding an fixing | 22:46 |
| rm_work | i seriously feel like no one must have tried deploying this to production and using it | 22:46 |
| johnsom | Yeah, really interesting. I'm not sure I ever met Keith Berger | 22:46 |
| rm_work | because a ton of stuff was *really broken* | 22:46 |
| rm_work | or else, no one uses spares pools. or has things fail over. or ever tried single-create. lol | 22:47 |
| rm_work | actually i guess the answer is it's because i'm deploying *octavia*, not neutron-lbaas | 22:47 |
| rm_work | and I probably am the first to try to do that <_< | 22:47 |
| johnsom | Well, some of those deploying might have local fixes to stuff as well | 22:47 |
| johnsom | That maybe made it upstream in a later release | 22:48 |
| rm_work | yeah the issue I think is that I ignore anything non-octavia | 22:49 |
| xgerman | we have Mark McLain in Atlanta | 22:49 |
| rm_work | so obviously octavia wasn't deployable previously | 22:49 |
| rm_work | Yeah I was just thinking of the people actually on the project | 22:49 |
| xgerman | HPE released kilo? | 22:49 |
| rm_work | I wonder if we could get a line into that meetup | 22:49 |
| rm_work | so we could help answer questions or something | 22:50 |
| johnsom | Yeah, I would love to be a bug on the wall | 22:50 |
| rm_work | as he is probably working from some outdated info | 22:50 |
| xgerman | maybe it gets recorded | 22:50 |
| rm_work | unless he's been in here in the last few days :P | 22:50 |
| johnsom | He is in the channel... | 22:51 |
| xgerman | indeed | 22:51 |
| rm_work | kberger: o/ | 22:51 |
| rm_work | kberger: curious what your presentation tonight is going to look like :) | 22:52 |
| rm_work | and if we can be of any assistance for answering questions / etc | 22:52 |
| johnsom | Tomorrow night actually | 22:52 |
| rm_work | ah i thought it was today | 22:52 |
| rm_work | ah yeah wow i am not paying attention | 22:53 |
| xgerman | I asked in twitter if they record… let’s see | 22:54 |
| *** m-greene has quit IRC | 22:55 | |
| *** eezhova has quit IRC | 22:55 | |
| johnsom | I will ping Doug too since he is an organizer | 22:55 |
| *** m-greene has joined #openstack-lbaas | 22:56 | |
| johnsom | Ok, back to fixing the functional tests for the quotas change. I'm going to see if I can turn on my scenario test now that the project_id patch merged too. I had it marked "skip" | 22:58 |
| *** catintheroof has quit IRC | 23:00 | |
| *** beardedeagle has quit IRC | 23:11 | |
| rm_work | hmm, lots of failures on that patch due to SSH connection timeouts to cirros images >_< | 23:27 |
| rm_work | I think the scenarios are busted from something else | 23:27 |
| rm_work | also apparently i failed at pep8 | 23:27 |
| rm_work | ah unused import now? lol | 23:27 |
| rm_work | yeah actually that makes sense | 23:27 |
| johnsom | Yeah, that is all about the infra out-of-disk space issue earlier today | 23:27 |
| rm_work | that really should not have been there | 23:28 |
| rm_work | make sure you fix that | 23:28 |
| johnsom | Did already | 23:28 |
| rm_work | k | 23:28 |
| johnsom | The oddity I am looking at is I am seeing stuff not roll back right. | 23:29 |
| johnsom | So, we still have session/lock_session wrong here | 23:30 |
| rm_work | hmmmmmm | 23:31 |
| *** jerrygb_ has joined #openstack-lbaas | 23:52 | |
| *** jerrygb has quit IRC | 23:53 | |
| *** fnaval_ has quit IRC | 23:54 | |
| *** jerrygb_ has quit IRC | 23:59 | |
| *** jerrygb has joined #openstack-lbaas | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
