Monday, 2017-07-10

*** sticker has joined #openstack-lbaas00:01
*** dayou has joined #openstack-lbaas01:26
*** aojea has joined #openstack-lbaas03:11
*** reedip has quit IRC03:13
*** aojea has quit IRC03:15
*** reedip has joined #openstack-lbaas03:25
*** gans has joined #openstack-lbaas03:26
*** sanfern has joined #openstack-lbaas03:45
*** yamamoto has joined #openstack-lbaas03:47
*** armax has joined #openstack-lbaas04:01
*** armax has quit IRC04:03
*** gans has quit IRC05:07
*** kobis has joined #openstack-lbaas06:06
*** rcernin has joined #openstack-lbaas06:47
*** dayou has quit IRC07:04
*** amotoki_away is now known as amotoki07:11
*** harlowja has quit IRC07:11
*** aojea has joined #openstack-lbaas07:21
*** tesseract has joined #openstack-lbaas07:36
*** dayou has joined #openstack-lbaas07:55
*** kobis has quit IRC08:54
*** kobis has joined #openstack-lbaas08:58
*** mdavidson has joined #openstack-lbaas09:04
*** diltram has quit IRC09:55
*** diltram has joined #openstack-lbaas10:01
*** diltram has quit IRC10:05
*** diltram has joined #openstack-lbaas10:10
*** ct_ has quit IRC10:26
*** diltram has quit IRC10:46
*** sanfern has quit IRC10:51
*** diltram has joined #openstack-lbaas10:53
*** atoth has joined #openstack-lbaas11:17
*** gcheresh_ has joined #openstack-lbaas11:40
*** sanfern has joined #openstack-lbaas11:55
*** sanfern has quit IRC12:00
*** sanfern has joined #openstack-lbaas12:01
*** Guest9640 is now known as beagles12:47
*** catintheroof has joined #openstack-lbaas13:04
*** kobis has quit IRC14:12
*** chlong_ has joined #openstack-lbaas14:24
*** cpuga has joined #openstack-lbaas14:29
*** armax has joined #openstack-lbaas14:31
*** gcheresh_ has quit IRC14:41
*** kobis has joined #openstack-lbaas14:47
*** fnaval has joined #openstack-lbaas14:53
*** rcernin has quit IRC14:57
*** armax has quit IRC14:58
*** armax has joined #openstack-lbaas15:00
xgerman_johnsom so the distributor data model is a mess15:15
xgerman_I think what we need is:15:15
xgerman_1:many to amphora15:16
*** chlong_ has quit IRC15:16
xgerman_and there was a motion to run a:many to load balancer15:16
*** chlong_ has joined #openstack-lbaas15:30
*** kobis has quit IRC15:38
*** gongysh has joined #openstack-lbaas15:51
*** kobis has joined #openstack-lbaas16:00
*** aojea has quit IRC16:03
*** chlong_ has quit IRC16:06
*** kobis has quit IRC16:15
*** gongysh has quit IRC16:15
johnsomOye, flood of meetings this morning.  That is what I get for a couple of days off...  grin16:18
xgerman_I don’t have meetings… so my life is easy16:19
johnsomSo this data model issue, is it in the patches or existing?16:20
*** chlong_ has joined #openstack-lbaas16:20
xgerman_I am making a patch — I guess it wasn’t set up right16:21
*** harlowja has joined #openstack-lbaas16:37
*** chlong_ has quit IRC16:38
*** isantosp has quit IRC16:41
*** armax has quit IRC16:59
*** tesseract has quit IRC17:01
*** kobis has joined #openstack-lbaas17:05
*** aojea has joined #openstack-lbaas17:09
*** aojea has quit IRC17:13
*** aojea has joined #openstack-lbaas17:13
openstackgerritGerman Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin
xgerman_^^ here is my thinking so far17:14
xgerman_distributor should now be more in line with jsniez17:15
*** yamamoto has quit IRC17:19
johnsomFigured out a solution to the qemu issue without rolling back to an old version....17:39
johnsomSet the hw_machine_type in nova.conf17:39
johnsomFor whatever reason it was using zesty, changing it to xenial and cirros boots again.17:40
*** armax has joined #openstack-lbaas17:41
*** aojea has quit IRC17:48
*** gcheresh_ has joined #openstack-lbaas18:01
*** aojea has joined #openstack-lbaas18:01
*** sshank has joined #openstack-lbaas18:05
*** aojea has quit IRC18:06
*** yamamoto has joined #openstack-lbaas18:19
johnsomrm_work taking a quick look at the API performance stuff.  I've loaded up 400 LBs (fake), it looks like you list call takes 3.451.  I'm going to run your loop and see if I can repro18:25
*** yamamoto has quit IRC18:27
johnsomPretty consistent for me18:30
openstackgerritMerged openstack/octavia master: Add placement service to
*** aojea has joined #openstack-lbaas18:39
*** sanfern has quit IRC18:40
*** sshank has quit IRC18:50
openstackgerritGerman Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin
*** atoth has quit IRC19:30
*** sshank has joined #openstack-lbaas19:37
openstackgerritJason Niesz proposed openstack/octavia master: blueprint: l3-active-active
openstackgerritCarlos Puga proposed openstack/octavia master: Spec detailing Octavia service flavors support
*** fnaval has quit IRC19:48
*** fnaval has joined #openstack-lbaas19:56
*** fnaval has quit IRC19:57
*** fnaval has joined #openstack-lbaas19:57
openstackgerritGerman Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin
*** kobis has quit IRC20:21
*** gcheresh_ has quit IRC20:27
rm_workjohnsom: we only have 15 LBs and we're seeing this20:32
rm_workwhat do you see for response time20:33
rm_workwith lower LBs20:33
rm_workalso it appears on LB GET and pool GET20:33
rm_work   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.20:33
rm_work  1.319   1.854   2.104   2.219   2.343  34.86120:33
rm_workthat's for 13200 datapoints20:33
rm_workrunning with 12 threads20:33
rm_worki am going to do a long-running one with just one thread and see if i get similar results20:34
rm_workor if the number of requests matters20:34
rm_workthis'll take a while20:35
rm_workjohnsom: also of note, before the RBAC was added, all requests seemed to return in ~.3s20:38
rm_workthat's 0.320:38
rm_workafter, minimum time as you see is 1.3s20:38
rm_workthat's a ridiculous hit20:38
rm_worki wish there was a way to DISABLE the RBAC stuff so I could verify if it's that or not20:39
rm_workI might make a patch and build an image where it's just  ... removed from the codepaths20:39
rm_workalthough 0.5-0.6s is outside of the API20:40
rm_workit seems20:40
rm_workcurl times the whole request at 1.5s when the API prints 1.0s for the request20:40
*** aojea has quit IRC20:44
rm_workcaught one20:45
rm_workit was after a policy reload20:45
rm_workwhich still seems to happen semi-often20:45
*** aojea has joined #openstack-lbaas20:46
*** aojea has quit IRC20:46
*** aojea has joined #openstack-lbaas20:46
johnsomSo, with just one LB I get:20:49
rm_workah damn i cought some counterexamples20:49
johnsomSo, I am seeing some extremely dumb DB queries however, from this:
johnsomLike 400 LBs balloons to ~4000 queries20:50
rm_workthis is the same query (obviously) ended up taking WAY longer20:52
rm_workand they appear to happen in groups20:52
rm_workie, they will happen with one thread, but they'll happen with multiple threads and all take a long time20:52
rm_workso i think it's something with either the DB itself20:52
rm_workor the connection20:52
rm_workoh and the nova fix, did you put that in your local.conf somehow20:53
*** catintheroof has quit IRC20:53
rm_workthat was the whole dataset I got20:56
rm_workjohnsom: soooo yeah that's I think why Jude was seeing it not matter what he tried to get from the models?21:00
rm_workbecause we were running that every time?21:00
rm_workbut I think it's necessary?21:01
rm_workthis is the part i'm bad at :(21:01
*** cpuga has quit IRC21:27
*** cpuga_ has joined #openstack-lbaas21:27
*** aojea has quit IRC21:28
*** cpuga_ has quit IRC21:28
*** aojea has joined #openstack-lbaas21:28
*** cpuga has joined #openstack-lbaas21:29
*** cpuga has quit IRC21:33
*** aojea has quit IRC21:34
*** tongl has joined #openstack-lbaas21:39
openstackgerritGerman Eichberger proposed openstack/octavia master: ACTIVE-ACTIVE Topology: Initial Distributor Driver Mixin
xgerman_well, if that passes I am in better shape but not sure…21:44
xgerman_rm_work do you want me to submit the lab for Sydney?21:44
rm_workI guess so21:44
rm_workWe'll have to see if it's feasible for me to go, but i am NOT going to have an answer for that until later :/21:45
tonglhi, can anyone point me to the LBaaS v2.0 API that has layer7 support? Current stable lbaasv2.0 api doc doesn't have layer7 policy and rule:
rm_workI'm still waiting for clearance to go to the PTG21:45
xgerman_we can always cancel21:45
rm_worktongl: it has it, the docs are just not there21:46
rm_worktongl: possibly you can follow the docs here (the API should be identical):
rm_work(That is the doc for Octavia LBaaS but it should be 100% compatible with the old Neutron LBaaS v2)21:47
*** gongysh has joined #openstack-lbaas21:47
tongl@rm_work: thanks, I am writing a vendor driver. This helps.21:48
rm_worktongl: OK, be aware that neutron-lbaas is in feature freeze and will soon be deprecated21:49
rm_worktongl: we'd love to work with you on getting your driver working with Octavia21:49
johnsomrm_work So, yeah, this to_data_model thing is dumb for our get_all needs.21:50
rm_workjohnsom: do you know what'd need to be done to rewrite it?21:51
johnsomI am going to try to create a better one.21:51
rm_workthat might help21:51
rm_workthough I'm not sure it's what is causing this issue21:51
rm_workbut, maybe21:51
johnsomBasically we are taking the LB, walking down ALL of the objects and querying them from the DB.21:51
tongl@rm_work: We would like to let customer to use LBaaS v2.0 first. In the mean time, I definitely would like to work with you on the Octavia driver when it is ready.21:52
rm_workthe way the DB works sometimes is black magic21:52
rm_worktongl: excellent. Yeah, I think this is the right path, just wanted you to be aware21:52
rm_workand that we are here to help!21:52
*** armax has quit IRC21:52
johnsomWell, the DB is something I know.  It's sqlalchemy that brings the dark arts21:52
johnsomIt should plummet our get all time significantly.21:53
rm_workok, cool21:53
rm_workETA? :P21:53
rm_workI mean is this something I should plan to review today?21:53
rm_workor next month21:53
johnsomUmmmm, today/tomorrow21:53
rm_worktongl: I have been trying to keep this up to date, but without a vendor to work with, I don't know if it will actually work properly21:54
rm_worktongl: so I'd love to work with you once you've got the n-lbaas driver done, to see if we can get it working in octavia21:54
rm_worktongl: what vendor is your driver for?21:54
rm_workjohnsom: awesome :P21:54
tonglrm_work: vmware_nsx21:55
rm_worktongl: ah ok.21:57
rm_worktongl: do you work with kobis?21:57
rm_workisn't he vmware?21:57
tonglI am almost done with LBaaS v2.0 driver. I'd love to work with you to see how we can implement vmware_nsx lbaas driver to work with nsx21:57
tonglhe is still at vmware, but he is working on other projects.21:57
rm_worki see21:57
xgerman_rm_work I am getting error after error on that Submit preentation page21:58
rm_workmaybe they'll extend it21:58
rm_workif it's super broken21:58
xgerman_is it today?21:58
rm_work3 or 4 days i think21:58
*** cpuga has joined #openstack-lbaas21:59
xgerman_ok, then I will try tomorrow again22:02
*** sshank has quit IRC22:26
*** aojea has joined #openstack-lbaas22:30
*** deep-book-gk_ has joined #openstack-lbaas22:34
*** deep-book-gk_ has quit IRC22:35
*** aojea has quit IRC22:36
*** catintheroof has joined #openstack-lbaas22:43
*** armax has joined #openstack-lbaas22:44
*** sshank has joined #openstack-lbaas23:04
rm_workin my devstack, centos images start but they don't get the configured keypair installed23:13
rm_workdunno if it's the cirros image's fault, or libvirt or nova or what23:14
rm_workbut they're configured to use a keypair, and yet there isn't even a .ssh in ~cirros23:14
johnsomYeah, I just had that happen to me today.  cirros 3.4 though23:15
rm_workhad to log in with the default password23:15
rm_workso maybe a nova bug?23:15
johnsomrm_work Up for a little experiment?23:21
johnsomoctavia/db/ line 11623:21
johnsom        query = query.options(joinedload('*'))23:21
johnsomRun your test23:21
rm_workok it'll ... take me a bit23:22
rm_workI have to work that change through my whole pipe23:23
rm_workwell actually... sec23:23
*** amotoki is now known as amotoki_away23:43
*** sshank has quit IRC23:50
rm_workI need to import something too right johnsom?23:55
rm_workI got it to where I can do quick tests23:55
tonglrm_work: if server doesn't get the configured keypair, most probably because something wrong with the metadata service23:55
rm_workI just figured that out with help from clarkb23:56
rm_workmy devstack didn't have n-api-meta service running23:56
rm_worktrying again with it23:56
*** catintheroof has quit IRC23:59

Generated by 2.15.3 by Marius Gedminas - find it at!