16:00:02 <DinaBelova> #startmeeting Performance Team
16:00:02 <openstack> Meeting started Tue Aug  9 16:00:02 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <openstack> The meeting name has been set to 'performance_team'
16:00:20 <DinaBelova> hey folks!
16:00:26 <DinaBelova> bvanhav indeed :)
16:00:41 <DinaBelova> going to push new release today
16:01:28 <DinaBelova> ok, let's see who's around
16:01:41 <gokrokve> o/
16:01:49 <DinaBelova> gokrokve o/
16:01:53 <rcherrueau> hi
16:01:57 <rohanion> hi o/
16:02:04 <DinaBelova> rohanion, rcherrueau nice to see you :)
16:02:21 <rohanion> same here :)
16:02:37 <DinaBelova> okay, so let's get started with usual topics
16:03:11 <DinaBelova> #topic Performance docs update
16:03:28 <DinaBelova> lots of stuff happened during last week
16:03:39 <DinaBelova> we were able to merge almost all changes on review
16:03:57 <DinaBelova> and I cleaned up a bit our LP bugs list
16:04:04 <DinaBelova> #link https://bugs.launchpad.net/performance-docs
16:04:32 <DinaBelova> so right now if bug is assigned to me you can feel free to reassign it to yourself and start work on it
16:04:36 <DinaBelova> :)
16:04:46 <DinaBelova> #link https://review.openstack.org/#/c/334964/
16:05:08 <DinaBelova> this change is going to make our life less painful next merge party
16:05:20 <rohanion> yaaay, merge party!
16:05:29 <DinaBelova> :D
16:05:45 <DinaBelova> so we won't need to rebase all changes after each merged commit
16:06:20 <DinaBelova> rohanion please find Lenya and remind him about https://bugs.launchpad.net/performance-docs/+bug/1609924 :)
16:06:20 <openstack> Launchpad bug 1609924 in OpenStack Performance Docs "500 nodes lab description is missing" [Medium,Triaged] - Assigned to Leontiy Istomin (listomin)
16:06:53 <DinaBelova> it's a bit strange we're running our tests against 500 nodes lab and no lab description is yet published here
16:07:20 <rohanion> yeah will do
16:07:54 <DinaBelova> #info next week will be dedicated to minor fixes for the bugs from the list https://bugs.launchpad.net/performance-docs
16:08:10 * rohanion bookmarked
16:08:31 <DinaBelova> so I would say that's all regarding performance docs themselves
16:08:49 <DinaBelova> #topic Current progress on the planned tests
16:09:15 <DinaBelova> so as I mentioned some meetings ago, Q3 is a logical continuation of the work done in Q2
16:09:37 <DinaBelova> so today we've continued Neutron testing
16:09:45 <DinaBelova> on 250 nodes lab
16:10:19 <DinaBelova> so more results will be coming in next 2 weeks or so
16:10:50 <DinaBelova> so that will be extended scalability tests, performance measurements under the load, etc.
16:11:06 <DinaBelova> also in parallel we're working on baseline performance testing
16:11:23 <DinaBelova> #link http://docs.openstack.org/developer/performance-docs/test_plans/control_plane/plan.html
16:11:30 <DinaBelova> against 500 nodes
16:11:45 <DinaBelova> and its comparison with 250 nodes runs we did already
16:12:04 <DinaBelova> extended reliability testing is planned later :)
16:13:30 <DinaBelova> #info working on extended Neutron testing and on baseline performance testing on 500 nodes. Reliability tests, 1000 nodes emulation and others are planned in 2 weeks or so
16:13:52 <DinaBelova> any updates on current test runs from other folks?
16:13:57 <DinaBelova> rcherrueau ?
16:14:02 <rcherrueau> Yes
16:14:12 <rcherrueau> We have a first solution that deploys a distributed OpenStack over g5k (a grid) using kolla.
16:14:25 <rcherrueau> You can find the code on github
16:14:29 <rcherrueau> #link https://github.com/BeyondTheClouds/kolla-g5k
16:14:36 <DinaBelova> ack, bookmarked
16:14:37 <kristian__> DinaBelova, hi, how are you?  good to interact again...just curious, is there an established ETA / date for osprofiler w/ Neutron and Swift?  any future plans to integrate w/ Juniper Contrail?
16:14:53 <rohanion> oh darn swift
16:14:58 <rcherrueau> We actually test with a small number of controller and go with the `fake_node` option to multiply the number of compute node.
16:14:58 <rcherrueau> 
16:15:01 <rohanion> we forgot about it T_T
16:15:11 <rcherrueau> #link https://github.com/openstack/kolla/blob/master/doc/nova-fake-driver.rst
16:15:19 <DinaBelova> kristian__ neutron is already supported :)
16:15:35 <DinaBelova> and you're right, Swift is not yet supported
16:15:46 <DinaBelova> I'd say it's up to rohanion to define the ETA
16:15:54 <kristian__> ...just confirming to make sure. and that is releasable for us, yes :)
16:16:39 <rohanion> do we have to integrate osprofiler with swift or just patch it to work with the new driver architecture?
16:16:42 <kristian__> more interested in Juniper contrail integration these days...
16:17:14 <DinaBelova> regarding the Contrail - the issue is that osprofiler is python library, so we'll be able to hack into its python client.. but not to the Juniper contrail itself using the profiler
16:18:18 <kristian__> yes so coding integration is blocked there, how about with juniper's REST API interface?
16:18:29 <DinaBelova> so right now I'm not sure how we can include contrail specific tracing details to the osprofiler trace
16:18:52 <DinaBelova> kristian__ well, I think (??) we need to check how osprofiler is working with the neurtron drivers
16:18:58 <rohanion> DinaBelova: maybe we can use osprofiler_web with juniper's REST API
16:19:10 <kristian__> good points, guys
16:19:38 <DinaBelova> rohanion - no need, we'll just wrap already wrapped stuff in the neutron drivers code - It looks like it'll be enough to check how drivers are wrapped in neutron
16:20:06 <DinaBelova> kristian__ good point anyway, let us check
16:20:19 <kristian__> np, thank you
16:20:41 <DinaBelova> #action DinaBelova rohanion check how can we estimate contrail profiling out of Neutron code using osprofiler
16:21:12 <DinaBelova> ok, so let's jump to the osprofiler itself :)
16:21:14 <rohanion> #info check if osprofiler is compatible with juniper's REST API
16:21:26 <DinaBelova> #topic OSProfiler weekly update
16:21:40 <DinaBelova> so hey! merge party finished today :)
16:21:47 <rohanion> yeah we had a great merge party last week :)
16:21:57 <DinaBelova> and I'm really glad to say I'm going to cut new release today :)
16:22:06 <DinaBelova> #info new Osprofiler release coming today
16:22:22 <DinaBelova> this will unblock lots of stuff
16:22:31 <DinaBelova> bvanhav ^^
16:22:46 <DinaBelova> currently we basically have only one major change on review
16:22:47 <rohanion> too bad I'm not in the US, we could have had a release party lol
16:22:57 <DinaBelova> #link https://review.openstack.org/#/c/340936/
16:23:01 <DinaBelova> rohanion indeed :D
16:23:11 <rohanion> Alex is away
16:23:32 <DinaBelova> I'll post the link to the openstack/releases once it'll be ready to this chat
16:23:32 <rohanion> Elasticsearch driver is almost ready
16:23:47 <DinaBelova> rohanion true, but I don't want to delay release because of this
16:24:05 <rohanion> he is going to cover it with tests and the change will be ready to merge
16:24:13 <rohanion> yes, there's no need to delay
16:24:15 <DinaBelova> I think it'll be more logical to cut one more with elasticsearch once it'll be landed
16:24:28 <DinaBelova> rohanion thanks for the information
16:24:29 <rohanion> elasticsearch does not change anything in the base architecture
16:24:34 <rohanion> u r welcome
16:24:36 <DinaBelova> indeed
16:24:48 <rohanion> will it be v1.4.0?
16:25:07 <DinaBelova> yep, I think so
16:25:14 <DinaBelova> in fact we may cut 2.0.0
16:25:17 <rohanion> because our changes do not look like a minor update
16:25:28 <rohanion> true
16:25:29 <DinaBelova> keeping in mind that arch was changed much
16:25:31 <DinaBelova> agree
16:25:35 <rohanion> 2.0.0 sounds better
16:25:45 <rohanion> so elastic will be in 2.0.1
16:25:46 <DinaBelova> #info so it'll be 2.0.0 release
16:25:51 <DinaBelova> agree
16:25:53 * rohanion hopes so
16:26:23 <bvanhav> I'll be looking forward to it no matter which release number it ends up being
16:26:30 <DinaBelova> ok, so it looks like that's all about osprofiler
16:26:34 <DinaBelova> bvanhav true :)
16:26:51 <DinaBelova> anything else to add here?
16:27:04 <rcherrueau> I have a question about osprofiler. We are interesting by measuring which methods of the database api (nova/db/api.py) is called the most often. Do you think that osprofiler is a good condidate to do that?
16:27:04 <rcherrueau> 
16:27:14 <rohanion> yes
16:27:16 <rcherrueau> cool
16:27:25 <DinaBelova> rcherrueau yes it is, but nova support was not yet merged
16:27:41 <DinaBelova> this needs to be rebased https://review.openstack.org/#/c/254703/
16:27:45 <rcherrueau> ok when the merge will take place
16:27:52 <rohanion> we already count the database and wsgi operations
16:28:11 <rohanion> ah
16:28:17 <rohanion> got your point
16:28:44 <DinaBelova> rcherrueau not in Newton for sure - it's frozen for nova
16:28:57 <rohanion> it's not the change in nova
16:29:14 <rcherrueau> ok thanks
16:29:20 <rohanion> it looks like a change in osprofiler's 'stats' field
16:29:30 <DinaBelova> rohanion it is change for nova
16:29:49 <DinaBelova> as we need to teach NOva how to use osprofiler
16:30:01 <DinaBelova> exactly the same as it was for cinder and others
16:30:11 <rohanion> ah that's what you're talking about
16:30:15 <rohanion> yes, right
16:30:17 <DinaBelova> rohanion :D
16:30:29 <rohanion> I was talking about changing the stats
16:30:50 <DinaBelova> :) I see. For Nova I was really glad to gain help from Mark
16:31:13 <DinaBelova> but this did not change the fact that it's feature freeze for Nova will end of Newton
16:31:36 <rohanion> but we can always switch to a dev branch :)
16:31:41 <DinaBelova> ok, so it looks like we're done for osprofiler
16:31:50 <rohanion> seems so
16:32:00 <DinaBelova> rohanion hehe, we can, but the question is if rcherrueau can do it :)
16:32:16 <rcherrueau> Why I cannot?
16:32:21 <DinaBelova> rcherrueau btw, what OpenStack are you running? Mitaka?
16:32:26 <rcherrueau> the dev branch is public no ?
16:32:27 <DinaBelova> rcherrueau I'm asking :)
16:32:33 <rcherrueau> yes mitake
16:32:33 <rohanion> yes
16:32:38 <rcherrueau> mitaka*
16:32:50 <rohanion> it's somewhere in review.openstack.org/...
16:32:59 <DinaBelova> rcherrueau - I can find in the history of the change the number of patch set
16:33:06 <rcherrueau> is it a problem to run osprogiler over mitaka?
16:33:12 <rcherrueau> osprofiler*
16:33:15 <DinaBelova> that was workable against mitaka
16:33:19 <rcherrueau> great
16:33:29 <DinaBelova> rcherrueau nope, we just need to identify what patch set it was
16:33:36 <DinaBelova> lemme find it and send it to you
16:33:46 <rcherrueau> thank you DinaBelova
16:33:57 <rohanion> it will work with mitaka but you won't be able to use any drivers but messaging/ceilometer
16:34:01 * notmyname reads scrollback and notices "swift"
16:34:03 <notmyname> let me know when/if you have swift questions
16:34:25 <DinaBelova> #action DinaBelova find patch set # and link to the https://review.openstack.org/#/c/254703/<n> to rcherrueau with pure Mitaka support
16:34:44 <rcherrueau> rohanion: thanks for the precision
16:34:46 <DinaBelova> notmyname ack, no so far - as we did not start working on swift support yet :)
16:35:15 <DinaBelova> #topic Open Discussion
16:35:23 <DinaBelova> ok, so let's jump to open discussion
16:35:39 <DinaBelova> anything else to share?
16:35:49 <rcherrueau> I wanna talk about the 1000 node perf
16:35:55 <rcherrueau> can I?
16:35:55 <DinaBelova> sure
16:35:57 <DinaBelova> :)
16:36:26 <rcherrueau> So just like I said before, we go with kolla for the deployment of OpenStack
16:36:54 <DinaBelova> so yeah, container-based installation
16:37:20 <rcherrueau> We uses kolla to deploy some controller nodes and then we take the advantages of the `fake_driver` option to get many compute
16:37:22 <DinaBelova> popular topic - at Mirantis we're working on CCP (containeraized control plane) as well
16:37:30 <rcherrueau> Are any one of you familiar with this option
16:37:39 <rcherrueau> #link https://github.com/openstack/kolla/blob/master/doc/nova-fake-driver.rst
16:37:39 <DinaBelova> rcherrueau I used it some time ago
16:37:47 <DinaBelova> any specific questions?
16:38:00 <DinaBelova> if i won't be able to answer, I'll ask alwex to help
16:38:10 <DinaBelova> as he did similar testing in q2
16:38:25 <rcherrueau> DinaBelova: we don't know how this option works exactly and we wanna be sure that doing some perf test with this option is equivalent to doing the same test with real node.
16:38:41 <DinaBelova> rcherrueau well, it's not in fact
16:38:48 <rcherrueau> arf ...
16:39:09 <DinaBelova> fake driver emulates all operations to the hypervizor, but not doing them in fact
16:39:33 <rcherrueau> it was a little bit expected in fact
16:39:34 <DinaBelova> so fake nodes have no libvirt running, and no REAL VMs starting
16:40:02 <DinaBelova> and this means that some specific network operations and setup is not going aas well
16:40:31 <rcherrueau> ok thanks
16:40:36 <rcherrueau> do you know if we can (easily) deploy many compute on the same physical node with kolla?
16:40:59 <DinaBelova> rcherrueau this means we're emulating the "control plane" for the hypervisor and everything underlaying, but no real VMs will be created
16:41:42 <DinaBelova> rcherrueau sadly not an expert here :( , I'm  pretty sure there will be issues with locating several compute containers on the same node
16:42:07 <DinaBelova> I think it'd be better to ask in kolla channel
16:42:21 <rcherrueau> DinaBelova: ok thanks for your responses
16:42:28 <rcherrueau> I will do so
16:42:46 <DinaBelova> rcherrueau the only 100% workable way I know to place several computes on the same physical node is to place them in the VMs
16:43:02 <DinaBelova> this is something we're going to do in next 2-3 weeks or so
16:43:09 <DinaBelova> against 500 nodes lab
16:43:10 <rcherrueau> ok
16:43:17 <rcherrueau> ah ok
16:43:29 <DinaBelova> so we'll place ~2 VMs per node and have real 1000 nodes cluster
16:43:44 <rcherrueau> ok got it
16:43:46 <DinaBelova> but i don't know how to manage this with docker
16:43:56 <DinaBelova> rcherrueau sorry for not helping much
16:44:21 <rcherrueau> no don't worry. I will ask on the mailing list
16:44:25 <DinaBelova> ack, thanks
16:44:31 <DinaBelova> ok, anything else?
16:44:43 <rcherrueau> yep
16:44:55 <DinaBelova> go ahead :)
16:45:08 <rcherrueau> We run the performance test using rally.
16:45:21 <rcherrueau> especially the `boot-and-delete` scenario
16:45:50 <rcherrueau> Which test do you use in the 500 nodes lab?
16:45:57 <DinaBelova> rcherrueau ack, can you share details regarding number of nodes, cloud topology and the results?
16:46:18 <DinaBelova> rcherrueau we are using what we're calling baseline performance test suite - http://docs.openstack.org/developer/performance-docs/test_plans/control_plane/plan.html
16:46:41 <rcherrueau> actually no, not now. We only run it over a small number of instances (less than 50).
16:47:11 <DinaBelova> rcherrueau ack -so for nova we're stressing it with booting and deleting VMs with security groups
16:47:19 <rcherrueau> ok thanks
16:47:33 <DinaBelova> as secgroups is really something that can ruin all the stuff in specific cases
16:47:54 <rcherrueau> ok
16:48:00 <DinaBelova> we've published the plugin we used here https://github.com/openstack/performance-docs/tree/master/doc/source/test_plans/control_plane/plugins
16:48:10 <DinaBelova> rcherrueau ^^
16:48:11 <rcherrueau> nice bookmarked ;)
16:48:31 <DinaBelova> so don't be surprised seeing NovaPerformancePlugin.boot_attach_and_delete_server_with_secgroups scenario :)
16:48:40 <DinaBelova> this is used from this plugin
16:48:52 <rcherrueau> OK, I will take a look
16:49:15 <DinaBelova> all other scenarios are standard
16:49:20 <DinaBelova> cool :)
16:49:57 <DinaBelova> rcherrueau anything else?
16:50:09 <rcherrueau> no
16:50:20 <DinaBelova> ok, so it looks like we're done for today :)
16:50:23 <rohanion> guys please wait for a minute :)
16:50:28 <DinaBelova> rohanion a-ha :D
16:50:30 <rohanion> Alex is going to join
16:50:45 <DinaBelova> aignatyev hey :)
16:50:51 <DinaBelova> nice to see you here :)
16:50:52 <aignatyev> hi :)
16:50:55 <rohanion> Welcome Alex!
16:51:06 <rcherrueau> hello
16:51:13 <rohanion> please meet another osprofiler monster :)
16:51:22 <aignatyev> nice to meet you all :)
16:51:22 <DinaBelova> aignatyev can you please share info about what's left to be done for https://review.openstack.org/#/c/340936/ ?
16:51:27 <DinaBelova> so we can understand the ETA
16:51:48 <aignatyev> I need to write 1-2 tests for retrieving reports from es
16:51:55 <aignatyev> there's already 1 test, but I need more
16:51:58 <aignatyev> will be done tomorrow
16:52:59 <DinaBelova> aignatyev ok, so I'll check the driver against my test lab when it'll be finalized
16:53:04 <DinaBelova> probably tomorrow as well
16:53:05 <DinaBelova> thanks
16:53:11 <aignatyev> you are welcome
16:53:14 <rohanion> I'll check it as well
16:53:19 <DinaBelova> rohanion ack
16:53:32 <DinaBelova> ok, so we had really nice discussion today
16:53:38 <DinaBelova> thanks everyone for coming
16:53:47 <DinaBelova> bye!
16:53:51 <DinaBelova> #endmeeting