16:00:07 <DinaBelova> #startmeeting Performance Team
16:00:08 <openstack> Meeting started Tue Aug 23 16:00:07 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:12 <openstack> The meeting name has been set to 'performance_team'
16:00:22 <DinaBelova> I think we may start with organizational questions
16:00:28 <DinaBelova> as the new time for the meeting :)
16:00:34 <DinaBelova> #topic New time for the meeting
16:01:18 <DinaBelova> so we have several options
16:01:36 <DinaBelova> 16:00 UTC (current time, really tough) vs 15:30 UTC vs 17:30 UTC vs 18:00 UTC
16:02:04 <rook> 1200 EST isn't great.... :)
16:02:20 <rook> o/
16:02:28 <DinaBelova> rook indeed
16:02:48 <DinaBelova> from what I heard from people 30 minutes earlier (15:30 UTC) seems to be the best option for now
16:02:53 <rohanion> hey everyone :)
16:03:03 <DinaBelova> rohanion o/
16:03:50 <DinaBelova> 15:30 UTC (11:30 EST, 8:30 PST) is something I've heard from AugieMena and ad_rien_ and other Inria folks at least
16:03:57 <ad_rien_> for French Inria Guys 15:30  UTC is a good choice -  (17:30 can be a second one)
16:04:09 <rook> +1 to 1530
16:04:16 <DinaBelova> rook ack
16:04:51 <DinaBelova> I want to grad confirmation from klindgren - if at least he'll be able to attend from GoDaddy side on more or less weekly basis it'll be cool
16:05:02 <DinaBelova> I know that for harlowja it'll be too early :D
16:05:15 <DinaBelova> glad -> grab **
16:05:50 <DinaBelova> although these two are having right now a daily meeting ^^, so let's wait for their response later
16:06:24 <DinaBelova> #info preliminary agreed on moving the meeting 30 minutes earlier on Tuesdays, 15:30 UTC (11:30 EST, 8:30 PST)
16:06:49 <DinaBelova> rohanion am I right that for Moscow it'll be convenient as well?
16:07:04 <DinaBelova> rohanion it'll be 18:30
16:07:13 <rohanion> yeah, no problem
16:07:21 <rohanion> 18:30 RTZ-2
16:07:30 <DinaBelova> ok, cool, I suspected this, but wanted to collect the information
16:07:48 <DinaBelova> ok, so I'll send a final update via email thread
16:08:11 <DinaBelova> so let's jump to the current progress on test plans
16:08:12 <DinaBelova> #topic Current progress on the planned tests
16:08:29 <DinaBelova> from Mirantis side we (finally) were able to move forward with labs configuration
16:09:12 <DinaBelova> our 200 nodes lab is under intensive Neutron testing for now (dataplane tests with huge enough number of Neutron objects and VMs)
16:10:05 <DinaBelova> and it looks like we finally were able to filter all non-working nodes from 500 nodes lab, so we'll have 400+ nodes Openstack cluster (installed by MOS)
16:10:28 <DinaBelova> so we'll run http://docs.openstack.org/developer/performance-docs/test_plans/control_plane/plan.html against it
16:10:37 <DinaBelova> hopefully this will start tomorrow
16:10:41 <rook> MOS ?
16:10:52 <DinaBelova> Mirantis OpenStack (way of deploying)
16:11:07 <rook> Ah MOS != Fuel ?
16:11:13 <DinaBelova> rook yep :D
16:11:35 <DinaBelova> MOS includes OpenStack packages with more fixes and backports
16:11:41 <msimonin> From inria side we made some progress on the 1000 nodes test plan
16:11:42 <DinaBelova> than usual stable/* usually have
16:11:57 <DinaBelova> but deployment tool itself is Fuel, yes
16:12:03 <DinaBelova> msimonin cool!
16:12:07 <DinaBelova> please share some details
16:12:11 <msimonin> but …
16:12:13 <msimonin> :)
16:12:16 <DinaBelova> :D
16:12:27 <msimonin> we still have some performance issue when scaling with 500+ computes
16:12:44 <DinaBelova> msimonin you use fake computes?
16:12:46 <msimonin> for the record we are using a kolla based deployment
16:12:51 <luzC> o/
16:12:55 <DinaBelova> luzC o/
16:12:57 <msimonin> DinaBelova: yes fake drivers
16:13:02 <rook> msimonin: what sort of scaling issues? Control plane?
16:13:08 <DinaBelova> msimonin ok, so kolla + fake driver
16:13:12 <DinaBelova> what kind of issues?
16:13:16 <luzC> hello Dina
16:13:31 <msimonin> actually, let me share a document that we are maintaining
16:13:40 <DinaBelova> msimonin it'll be cool
16:13:49 <msimonin> #link https://pad.inria.fr/p/FK18ZSyjFyLPITfE
16:14:31 <DinaBelova> msimonin ok, I see " very slow apis when using 500 computes"
16:14:32 <msimonin> apis + mariadb are behind the haproxy
16:14:38 <msimonin> yep
16:14:42 <msimonin> that's the issue
16:14:52 <rook> the issue is haproxy?
16:14:59 <DinaBelova> does it relate to all OpenStack REST APIs?
16:15:29 <msimonin> rook: maybe haproxy, but we need more insight
16:16:15 <DinaBelova> msimonin I suspect some parameters of HAproxy may influence here indeed on the performance, I'll agree with rook here
16:16:30 <msimonin> yes it's my feeling too
16:16:41 <msimonin> do you have some experience with haproxy ?
16:16:52 <DinaBelova> msimonin not too big sadly
16:16:53 <rook> msimonin do you know at what # of computes you start seeing the problem?
16:17:15 <msimonin> 100 computes is ok, 500 isn't
16:17:24 <rook> well you go 5x :)
16:17:30 <msimonin> sure :)
16:17:43 <DinaBelova> msimonin I can suggest you to ping alwex (Alex Shaposhnikov) and try to discuss this with him, he may have some ideas
16:17:51 <msimonin> cool, I'll do that
16:17:54 <rook> msimonin we have a monitoring solution to help track down what might be slowing down
16:18:09 <rook> using collectd/graphite/grafana
16:18:28 <rook> might provide insight into the slowness or the possible culprit.
16:18:29 <msimonin> actually we are using cadvisor/influx/grafana
16:18:45 <msimonin> but it just keep track of basics metrics
16:18:51 <DinaBelova> #info Mirantis: Neutron testing on 200 nodes in progress, 400 nodes cluster almost installed for baseline performance testing
16:19:07 <DinaBelova> #info Inria facing issues with 1000 nodes emulation experiment: https://pad.inria.fr/p/FK18ZSyjFyLPITfE
16:19:09 <rook> msimonin: sure, we break out per-process
16:20:23 <DinaBelova> msimonin I think you may try the same method rook is talking about + let's ask alwex to help
16:20:56 <DinaBelova> #action DinaBelova msimonin alwex find out what's slowing down OpenStack REST APIs in the Inria 1000 nodes experiment
16:21:16 <msimonin> rook: do you have what metrics can help tracking the issue ?
16:21:22 <msimonin> do you know*
16:21:28 <ad_rien_> Since we built our deployment framework on top of Kolla, we get the ha-proxy by default…. DinaBelova do you think it can be possible to see how you are deploying the different containers on your side ?
16:22:03 <ad_rien_> For us it is not so complex/time consuming to redeploy a set of containers on top of G5K
16:22:07 <DinaBelova> ad_rien_ I think that's possible, let's anyway communicate this with Alex as he was originating the experiment
16:22:12 <rook> msimonin: since it seems to be based on the # of computes... we could look at the msging services (rabbit?), haproxy, and possibly the individual nova services. since each of the compute nodes will be checking in on a interval.
16:22:49 <DinaBelova> ad_rien_ fyi now we're playing with k8s and fuel-ccp (containeraized control plane) to see if we can use it for the experiments and testing
16:23:16 <msimonin> rook: ack, rabbit may be the issue as well indeed
16:23:32 <rook> msimonin: honestly, it wouldn't shock me.
16:23:44 <rook> msimonin: what errors are you seeing?
16:23:49 <DinaBelova> ad_rien_ although I believe HAproxy is either not an issue at all or this can be fixed even without redeployment
16:24:29 <msimonin> rook: WARNING nova.openstack.common.loopingcall [-] task <bound method DbDriver._report_state of <nova.servicegroup.drivers.db.DbDriver object at 0x7f454c8db710>> run outlasted interval
16:24:35 <msimonin> in the nova-compute.lgo
16:24:49 <rook> yeah - could be msging for sure.
16:24:59 <msimonin> yes
16:25:11 <DinaBelova> msimonin so let's check this option as well
16:25:12 * rook isn't familiar with your control-plane setup
16:25:26 <ad_rien_> DinaBelova:  actually to be fair with your experiments, we should remove it…. ok let's see whether we can fix that point by discussing with Alex.
16:25:36 <DinaBelova> ad_rien_ ack
16:25:48 <ad_rien_> (to be faire, i.e. to provide fair comparisons)
16:26:08 <DinaBelova> ad_rien_ msimonin I'll start separated email thread with Alex
16:26:18 <msimonin> DinaBelova: ack
16:26:21 <ad_rien_> DinaBelova:  ack thanks
16:26:33 <DinaBelova> it looks like we may proceed for now
16:26:41 <rcherrueau> DinaBelova: could you also put me in CC
16:26:46 <DinaBelova> rcherrueau sure
16:26:52 <rcherrueau> thanks
16:26:59 <DinaBelova> ok, so let's jump to the OSprofiler
16:27:11 <DinaBelova> #topic OSProfiler weekly update
16:27:24 <DinaBelova> rohanion aignatev - the floor is yours
16:27:33 <aignatev> need review for https://review.openstack.org/#/c/340936/
16:27:47 <aignatev> other than that I'm not currently working with osprofiler, so no updates
16:27:50 <rohanion> ok, will look at it today
16:28:04 <aignatev> thanks!
16:28:18 <DinaBelova> aignatev I'll take a look as well, thanks
16:28:24 <aignatev> thank you
16:28:29 <DinaBelova> rohanion any updates from your side?
16:28:37 <rohanion> yep, writing :)
16:28:57 <rohanion> I'm working on enabling osprofiler in Fuel. basically I have to code a PoC that updates configuration and reloads the services.
16:29:10 <rohanion> this PoC will be turned into a Fuel plugin
16:29:34 <rohanion> Also I'm thinking about a set of profiling scenarios with single entry-point and results in table format.
16:29:38 <DinaBelova> rohanion I believe you found out that's a bit overkill solution?
16:29:52 <DinaBelova> I mean fule plugin?
16:29:55 <DinaBelova> fuel*
16:30:48 <rohanion> yes, a little. but still it will be a good option to set all the needed variables in a nice UI
16:30:49 <DinaBelova> or you decided it's not too difficult to develop this profiling plugin?
16:31:07 <DinaBelova> rohanion ack
16:31:28 <rohanion> it will be tough but I think I'll handle that.
16:31:42 <rook> bummer, will it be written to only work with Fuel?
16:31:44 <rook> not OOO?
16:31:49 <rohanion> no updates regarding osprofiler itself
16:31:54 <rohanion> rook, no
16:32:03 <rook> no to only Fuel ?
16:32:10 <rohanion> the script will work with vanilla openstack
16:32:14 <rook> ah ha cool
16:32:26 <DinaBelova> rook it's just way for us to deploy it automatically using fuel, but it workd with any OpenStack and is automated to be used with devtack
16:32:36 <rohanion> the plugin will just launch this script, sending all necessary parameters
16:32:51 <rook> cool rohanion - have a link to this work
16:33:35 <DinaBelova> ok, rohanion is this all for now?
16:33:46 <rohanion> I'll publish the changes as soon as I write the code :)
16:33:51 <rohanion> yes, that's everything
16:33:54 <DinaBelova> ok, cool
16:34:01 <DinaBelova> so we may jump to open discussion :)
16:34:06 <DinaBelova> #topic Open Discussion
16:34:45 <DinaBelova> anything else to share?
16:35:03 <rcherrueau> We really want to contribute to the performance documentation
16:35:12 <DinaBelova> rcherrueau you're really welcome :)
16:35:14 <rcherrueau> do you think it's possible
16:35:25 <DinaBelova> rcherrueau of course :)
16:35:37 <DinaBelova> that's usual enough contribution process
16:35:41 <DinaBelova> to the repository
16:35:48 <rcherrueau> for instance, do you think it's possible to add a specific subsection in the labs section with information about our grid?
16:35:52 <DinaBelova> #link https://github.com/openstack/performance-docs
16:35:57 <DinaBelova> rcherrueau for sure
16:36:03 <DinaBelova> to have all info shared
16:36:09 <rcherrueau> cool
16:36:18 <DinaBelova> and you'll mention this lab in the test plans/test results
16:36:32 <rcherrueau> pleaze what is the process to do so
16:36:47 <DinaBelova> rcherrueau you need to upload several commits
16:36:57 <DinaBelova> commit #1 - regarding the lab description
16:37:03 <DinaBelova> lemme find an example
16:37:14 <rcherrueau> yes thanks
16:37:29 <DinaBelova> rcherrueau like this https://review.openstack.org/#/c/292835/
16:37:38 <DinaBelova> but regarding your lab
16:37:58 <ad_rien_> ok DinaBelova I think we got the information we were looking for
16:38:06 <rcherrueau> ok thank you DinaBelova
16:38:11 <DinaBelova> and second one with your results, as you're going to use the same test plan as we did
16:38:39 <DinaBelova> rcherrueau https://review.openstack.org/#/c/324806/ - test results part
16:38:39 <ad_rien_> yes modulo the way we perform(ed) the experiments
16:38:47 <DinaBelova> ad_rien_ indeed
16:39:30 <DinaBelova> the overall documentation is free form, the only rule is to set test plans regarding the template  https://github.com/openstack/performance-docs/blob/master/doc/source/test_plans/template.rst
16:39:40 <DinaBelova> but you're not going to propose your own test plan :)
16:39:48 <DinaBelova> so feel free to skip my last message :)
16:39:53 <ad_rien_> ok we gonna give a look
16:40:12 <rcherrueau> OK great, that's all for us.
16:40:17 <DinaBelova> ad_rien_ sure, will take a look with great pleasure :)
16:40:43 <DinaBelova> anything else here? from my side I have nothing more to share
16:41:00 <ad_rien_> not sure whether it will be done for the next week as right now we are focusing on kolla related issues but once we get relevant results we will commit them
16:41:15 <DinaBelova> ad_rien_ ack, thanks for the update
16:41:44 <DinaBelova> ok, it looks like we're done for now
16:41:51 <DinaBelova> please wait the meeting time update email
16:42:01 <DinaBelova> I think we'll stop with 15:30 UTC
16:42:16 <DinaBelova> thanks folks for coming and participation
16:42:21 <DinaBelova> see you
16:42:25 <DinaBelova> #endmeeting