| *** sticker has joined #openstack-lbaas | 00:01 | |
| *** dayou has joined #openstack-lbaas | 01:26 | |
| *** aojea has joined #openstack-lbaas | 03:11 | |
| *** reedip has quit IRC | 03:13 | |
| *** aojea has quit IRC | 03:15 | |
| *** reedip has joined #openstack-lbaas | 03:25 | |
| *** gans has joined #openstack-lbaas | 03:26 | |
| *** sanfern has joined #openstack-lbaas | 03:45 | |
| *** yamamoto has joined #openstack-lbaas | 03:47 | |
| *** armax has joined #openstack-lbaas | 04:01 | |
| *** armax has quit IRC | 04:03 | |
| *** gans has quit IRC | 05:07 | |
| *** kobis has joined #openstack-lbaas | 06:06 | |
| *** rcernin has joined #openstack-lbaas | 06:47 | |
| *** dayou has quit IRC | 07:04 | |
| *** amotoki_away is now known as amotoki | 07:11 | |
| *** harlowja has quit IRC | 07:11 | |
| *** aojea has joined #openstack-lbaas | 07:21 | |
| *** tesseract has joined #openstack-lbaas | 07:36 | |
| *** dayou has joined #openstack-lbaas | 07:55 | |
| *** kobis has quit IRC | 08:54 | |
| *** kobis has joined #openstack-lbaas | 08:58 | |
| *** mdavidson has joined #openstack-lbaas | 09:04 | |
| *** diltram has quit IRC | 09:55 | |
| *** diltram has joined #openstack-lbaas | 10:01 | |
| *** diltram has quit IRC | 10:05 | |
| *** diltram has joined #openstack-lbaas | 10:10 | |
| *** ct_ has quit IRC | 10:26 | |
| *** diltram has quit IRC | 10:46 | |
| *** sanfern has quit IRC | 10:51 | |
| *** diltram has joined #openstack-lbaas | 10:53 | |
| *** atoth has joined #openstack-lbaas | 11:17 | |
| *** gcheresh_ has joined #openstack-lbaas | 11:40 | |
| *** sanfern has joined #openstack-lbaas | 11:55 | |
| *** sanfern has quit IRC | 12:00 | |
| *** sanfern has joined #openstack-lbaas | 12:01 | |
| *** Guest9640 is now known as beagles | 12:47 | |
| *** catintheroof has joined #openstack-lbaas | 13:04 | |
| *** kobis has quit IRC | 14:12 | |
| *** chlong_ has joined #openstack-lbaas | 14:24 | |
| *** cpuga has joined #openstack-lbaas | 14:29 | |
| *** armax has joined #openstack-lbaas | 14:31 | |
| *** gcheresh_ has quit IRC | 14:41 | |
| *** kobis has joined #openstack-lbaas | 14:47 | |
| *** fnaval has joined #openstack-lbaas | 14:53 | |
| *** rcernin has quit IRC | 14:57 | |
| *** armax has quit IRC | 14:58 | |
| *** armax has joined #openstack-lbaas | 15:00 | |
| johnsom | o/ | 15:10 |
|---|---|---|
| xgerman_ | o/ | 15:15 |
| xgerman_ | johnsom so the distributor data model is a mess | 15:15 |
| xgerman_ | I think what we need is: | 15:15 |
| xgerman_ | 1:many to amphora | 15:16 |
| *** chlong_ has quit IRC | 15:16 | |
| xgerman_ | and there was a motion to run a:many to load balancer | 15:16 |
| *** chlong_ has joined #openstack-lbaas | 15:30 | |
| *** kobis has quit IRC | 15:38 | |
| *** gongysh has joined #openstack-lbaas | 15:51 | |
| *** kobis has joined #openstack-lbaas | 16:00 | |
| *** aojea has quit IRC | 16:03 | |
| *** chlong_ has quit IRC | 16:06 | |
| *** kobis has quit IRC | 16:15 | |
| *** gongysh has quit IRC | 16:15 | |
| johnsom | Oye, flood of meetings this morning. That is what I get for a couple of days off... grin | 16:18 |
| xgerman_ | I don’t have meetings… so my life is easy | 16:19 |
| johnsom | So this data model issue, is it in the patches or existing? | 16:20 |
| *** chlong_ has joined #openstack-lbaas | 16:20 | |
| xgerman_ | I am making a patch — I guess it wasn’t set up right | 16:21 |
| *** harlowja has joined #openstack-lbaas | 16:37 | |
| *** chlong_ has quit IRC | 16:38 | |
| *** isantosp has quit IRC | 16:41 | |
| *** armax has quit IRC | 16:59 | |
| *** tesseract has quit IRC | 17:01 | |
| *** kobis has joined #openstack-lbaas | 17:05 | |
| *** aojea has joined #openstack-lbaas | 17:09 | |
| *** aojea has quit IRC | 17:13 | |
| *** aojea has joined #openstack-lbaas | 17:13 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin https://review.openstack.org/313006 | 17:14 |
| xgerman_ | ^^ here is my thinking so far | 17:14 |
| xgerman_ | distributor should now be more in line with jsniez | 17:15 |
| *** yamamoto has quit IRC | 17:19 | |
| johnsom | Figured out a solution to the qemu issue without rolling back to an old version.... | 17:39 |
| johnsom | https://www.irccloud.com/pastebin/Ja98EXh8/ | 17:39 |
| johnsom | Set the hw_machine_type in nova.conf | 17:39 |
| johnsom | For whatever reason it was using zesty, changing it to xenial and cirros boots again. | 17:40 |
| xgerman_ | Nice! | 17:40 |
| *** armax has joined #openstack-lbaas | 17:41 | |
| *** aojea has quit IRC | 17:48 | |
| *** gcheresh_ has joined #openstack-lbaas | 18:01 | |
| *** aojea has joined #openstack-lbaas | 18:01 | |
| *** sshank has joined #openstack-lbaas | 18:05 | |
| *** aojea has quit IRC | 18:06 | |
| *** yamamoto has joined #openstack-lbaas | 18:19 | |
| johnsom | rm_work taking a quick look at the API performance stuff. I've loaded up 400 LBs (fake), it looks like you list call takes 3.451. I'm going to run your loop and see if I can repro | 18:25 |
| *** yamamoto has quit IRC | 18:27 | |
| johnsom | https://www.irccloud.com/pastebin/klcg6DTp/ | 18:30 |
| johnsom | Pretty consistent for me | 18:30 |
| openstackgerrit | Merged openstack/octavia master: Add placement service to new-octavia-devstack.sh https://review.openstack.org/480071 | 18:30 |
| *** aojea has joined #openstack-lbaas | 18:39 | |
| *** sanfern has quit IRC | 18:40 | |
| *** sshank has quit IRC | 18:50 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin https://review.openstack.org/313006 | 18:50 |
| *** atoth has quit IRC | 19:30 | |
| *** sshank has joined #openstack-lbaas | 19:37 | |
| openstackgerrit | Jason Niesz proposed openstack/octavia master: blueprint: l3-active-active https://review.openstack.org/453005 | 19:45 |
| openstackgerrit | Carlos Puga proposed openstack/octavia master: Spec detailing Octavia service flavors support https://review.openstack.org/392485 | 19:46 |
| *** fnaval has quit IRC | 19:48 | |
| *** fnaval has joined #openstack-lbaas | 19:56 | |
| *** fnaval has quit IRC | 19:57 | |
| *** fnaval has joined #openstack-lbaas | 19:57 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin https://review.openstack.org/313006 | 20:09 |
| *** kobis has quit IRC | 20:21 | |
| *** gcheresh_ has quit IRC | 20:27 | |
| rm_work | johnsom: we only have 15 LBs and we're seeing this | 20:32 |
| rm_work | what do you see for response time | 20:33 |
| rm_work | with lower LBs | 20:33 |
| rm_work | also it appears on LB GET and pool GET | 20:33 |
| rm_work | apparently | 20:33 |
| rm_work | Min. 1st Qu. Median Mean 3rd Qu. Max. | 20:33 |
| rm_work | 1.319 1.854 2.104 2.219 2.343 34.861 | 20:33 |
| rm_work | that's for 13200 datapoints | 20:33 |
| rm_work | running with 12 threads | 20:33 |
| rm_work | i am going to do a long-running one with just one thread and see if i get similar results | 20:34 |
| rm_work | or if the number of requests matters | 20:34 |
| rm_work | this'll take a while | 20:35 |
| rm_work | johnsom: also of note, before the RBAC was added, all requests seemed to return in ~.3s | 20:38 |
| rm_work | that's 0.3 | 20:38 |
| rm_work | after, minimum time as you see is 1.3s | 20:38 |
| rm_work | that's a ridiculous hit | 20:38 |
| rm_work | i wish there was a way to DISABLE the RBAC stuff so I could verify if it's that or not | 20:39 |
| rm_work | I might make a patch and build an image where it's just ... removed from the codepaths | 20:39 |
| rm_work | although 0.5-0.6s is outside of the API | 20:40 |
| rm_work | it seems | 20:40 |
| rm_work | curl times the whole request at 1.5s when the API prints 1.0s for the request | 20:40 |
| *** aojea has quit IRC | 20:44 | |
| rm_work | HA | 20:44 |
| rm_work | http://paste.openstack.org/raw/6ve9FY9ySH9vsi0PZtli/ | 20:45 |
| rm_work | caught one | 20:45 |
| rm_work | it was after a policy reload | 20:45 |
| rm_work | which still seems to happen semi-often | 20:45 |
| *** aojea has joined #openstack-lbaas | 20:46 | |
| *** aojea has quit IRC | 20:46 | |
| *** aojea has joined #openstack-lbaas | 20:46 | |
| johnsom | So, with just one LB I get: | 20:49 |
| johnsom | https://www.irccloud.com/pastebin/bxo9MnDV/ | 20:49 |
| rm_work | ah damn i cought some counterexamples | 20:49 |
| johnsom | So, I am seeing some extremely dumb DB queries however, from this: https://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L128 | 20:50 |
| johnsom | Like 400 LBs balloons to ~4000 queries | 20:50 |
| rm_work | http://paste.openstack.org/show/614962/ | 20:51 |
| rm_work | this is the same query (obviously) ended up taking WAY longer | 20:52 |
| rm_work | and they appear to happen in groups | 20:52 |
| rm_work | ie, they will happen with one thread, but they'll happen with multiple threads and all take a long time | 20:52 |
| rm_work | so i think it's something with either the DB itself | 20:52 |
| rm_work | or the connection | 20:52 |
| rm_work | oh and the nova fix, did you put that in your local.conf somehow | 20:53 |
| *** catintheroof has quit IRC | 20:53 | |
| rm_work | http://adamharwell.com/curltimes.gz | 20:56 |
| rm_work | that was the whole dataset I got | 20:56 |
| rm_work | johnsom: soooo yeah that's I think why Jude was seeing it not matter what he tried to get from the models? | 21:00 |
| rm_work | because we were running that every time? | 21:00 |
| rm_work | but I think it's necessary? | 21:01 |
| rm_work | err | 21:01 |
| rm_work | hmm | 21:01 |
| rm_work | this is the part i'm bad at :( | 21:01 |
| *** cpuga has quit IRC | 21:27 | |
| *** cpuga_ has joined #openstack-lbaas | 21:27 | |
| *** aojea has quit IRC | 21:28 | |
| *** cpuga_ has quit IRC | 21:28 | |
| *** aojea has joined #openstack-lbaas | 21:28 | |
| *** cpuga has joined #openstack-lbaas | 21:29 | |
| *** cpuga has quit IRC | 21:33 | |
| *** aojea has quit IRC | 21:34 | |
| *** tongl has joined #openstack-lbaas | 21:39 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin https://review.openstack.org/313006 | 21:41 |
| xgerman_ | well, if that passes I am in better shape but not sure… | 21:44 |
| xgerman_ | rm_work do you want me to submit the lab for Sydney? | 21:44 |
| rm_work | I guess so | 21:44 |
| rm_work | We'll have to see if it's feasible for me to go, but i am NOT going to have an answer for that until later :/ | 21:45 |
| tongl | hi, can anyone point me to the LBaaS v2.0 API that has layer7 support? Current stable lbaasv2.0 api doc doesn't have layer7 policy and rule: https://developer.openstack.org/api-ref/networking/v2/#lbaas-2-0-stable | 21:45 |
| rm_work | I'm still waiting for clearance to go to the PTG | 21:45 |
| xgerman_ | we can always cancel | 21:45 |
| rm_work | tongl: it has it, the docs are just not there | 21:46 |
| rm_work | tongl: possibly you can follow the docs here (the API should be identical): https://developer.openstack.org/api-ref/load-balancer/v2/index.html#l7-policies | 21:47 |
| rm_work | (That is the doc for Octavia LBaaS but it should be 100% compatible with the old Neutron LBaaS v2) | 21:47 |
| *** gongysh has joined #openstack-lbaas | 21:47 | |
| tongl | @rm_work: thanks, I am writing a vendor driver. This helps. | 21:48 |
| rm_work | tongl: OK, be aware that neutron-lbaas is in feature freeze and will soon be deprecated | 21:49 |
| rm_work | tongl: we'd love to work with you on getting your driver working with Octavia | 21:49 |
| johnsom | rm_work So, yeah, this to_data_model thing is dumb for our get_all needs. | 21:50 |
| rm_work | johnsom: do you know what'd need to be done to rewrite it? | 21:51 |
| johnsom | I am going to try to create a better one. | 21:51 |
| rm_work | OK | 21:51 |
| rm_work | that might help | 21:51 |
| rm_work | though I'm not sure it's what is causing this issue | 21:51 |
| rm_work | but, maybe | 21:51 |
| johnsom | Basically we are taking the LB, walking down ALL of the objects and querying them from the DB. | 21:51 |
| tongl | @rm_work: We would like to let customer to use LBaaS v2.0 first. In the mean time, I definitely would like to work with you on the Octavia driver when it is ready. | 21:52 |
| rm_work | the way the DB works sometimes is black magic | 21:52 |
| rm_work | tongl: excellent. Yeah, I think this is the right path, just wanted you to be aware | 21:52 |
| rm_work | and that we are here to help! | 21:52 |
| *** armax has quit IRC | 21:52 | |
| johnsom | Well, the DB is something I know. It's sqlalchemy that brings the dark arts | 21:52 |
| johnsom | It should plummet our get all time significantly. | 21:53 |
| rm_work | ok, cool | 21:53 |
| rm_work | ETA? :P | 21:53 |
| rm_work | I mean is this something I should plan to review today? | 21:53 |
| rm_work | or next month | 21:53 |
| johnsom | Ummmm, today/tomorrow | 21:53 |
| rm_work | tongl: https://review.openstack.org/#/c/409398/ I have been trying to keep this up to date, but without a vendor to work with, I don't know if it will actually work properly | 21:54 |
| rm_work | tongl: so I'd love to work with you once you've got the n-lbaas driver done, to see if we can get it working in octavia | 21:54 |
| rm_work | tongl: what vendor is your driver for? | 21:54 |
| rm_work | johnsom: awesome :P | 21:54 |
| tongl | rm_work: vmware_nsx | 21:55 |
| rm_work | tongl: ah ok. | 21:57 |
| rm_work | tongl: do you work with kobis? | 21:57 |
| rm_work | isn't he vmware? | 21:57 |
| tongl | I am almost done with LBaaS v2.0 driver. I'd love to work with you to see how we can implement vmware_nsx lbaas driver to work with nsx | 21:57 |
| tongl | he is still at vmware, but he is working on other projects. | 21:57 |
| rm_work | i see | 21:57 |
| xgerman_ | rm_work I am getting error after error on that Submit preentation page | 21:58 |
| rm_work | rofl | 21:58 |
| rm_work | maybe they'll extend it | 21:58 |
| rm_work | if it's super broken | 21:58 |
| xgerman_ | is it today? | 21:58 |
| rm_work | no | 21:58 |
| rm_work | 3 or 4 days i think | 21:58 |
| *** cpuga has joined #openstack-lbaas | 21:59 | |
| xgerman_ | ok, then I will try tomorrow again | 22:02 |
| *** sshank has quit IRC | 22:26 | |
| *** aojea has joined #openstack-lbaas | 22:30 | |
| *** deep-book-gk_ has joined #openstack-lbaas | 22:34 | |
| *** deep-book-gk_ has quit IRC | 22:35 | |
| *** aojea has quit IRC | 22:36 | |
| *** catintheroof has joined #openstack-lbaas | 22:43 | |
| *** armax has joined #openstack-lbaas | 22:44 | |
| *** sshank has joined #openstack-lbaas | 23:04 | |
| rm_work | hmmmmmmm | 23:13 |
| rm_work | in my devstack, centos images start but they don't get the configured keypair installed | 23:13 |
| rm_work | :( | 23:13 |
| rm_work | cirros-0.3.5-x86_64-disk | 23:14 |
| rm_work | dunno if it's the cirros image's fault, or libvirt or nova or what | 23:14 |
| rm_work | but they're configured to use a keypair, and yet there isn't even a .ssh in ~cirros | 23:14 |
| johnsom | Yeah, I just had that happen to me today. cirros 3.4 though | 23:15 |
| rm_work | had to log in with the default password | 23:15 |
| rm_work | so maybe a nova bug? | 23:15 |
| johnsom | rm_work Up for a little experiment? | 23:21 |
| rm_work | yes | 23:21 |
| rm_work | always | 23:21 |
| johnsom | octavia/db/repositories.py line 116 | 23:21 |
| johnsom | add: | 23:21 |
| johnsom | query = query.options(joinedload('*')) | 23:21 |
| johnsom | Run your test | 23:21 |
| rm_work | ok it'll ... take me a bit | 23:22 |
| rm_work | I have to work that change through my whole pipe | 23:23 |
| rm_work | well actually... sec | 23:23 |
| *** amotoki is now known as amotoki_away | 23:43 | |
| *** sshank has quit IRC | 23:50 | |
| rm_work | errr | 23:55 |
| rm_work | I need to import something too right johnsom? | 23:55 |
| rm_work | joinedload | 23:55 |
| rm_work | I got it to where I can do quick tests | 23:55 |
| tongl | rm_work: if server doesn't get the configured keypair, most probably because something wrong with the metadata service | 23:55 |
| rm_work | yeah | 23:56 |
| rm_work | I just figured that out with help from clarkb | 23:56 |
| rm_work | my devstack didn't have n-api-meta service running | 23:56 |
| rm_work | trying again with it | 23:56 |
| *** catintheroof has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!