16:00:03 <DinaBelova> #startmeeting Performance Team
16:00:05 <openstack> Meeting started Tue Dec  8 16:00:03 2015 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:05 * regXboi wanders in
16:00:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:09 <openstack> The meeting name has been set to 'performance_team'
16:00:14 <DinaBelova> hello everyone!
16:00:26 <DinaBelova> o/
16:00:41 <DinaBelova> so todays agenda https://wiki.openstack.org/wiki/Meetings/Performance#Agenda_for_next_meeting
16:01:06 <gokrokve> Hi!
16:01:09 <DinaBelova> o/
16:01:23 <oanufriev> hi all
16:01:30 <nihilifer> o/
16:01:34 <kun_huang> o/
16:01:40 <gokrokve> Happy Birthday Dina!
16:01:41 <DinaBelova> harlowja, SpamapS are you around? :)
16:01:45 <DinaBelova> gokrokve thank s :)
16:02:08 * regXboi wonders what IRC singing reads like
16:02:13 <DinaBelova> klindgren__, o/ are you going to attend today? :)
16:02:15 <DinaBelova> regXboi :D
16:02:52 <DinaBelova> ok, let's wait for a few more minutes
16:03:17 <DinaBelova> harlowja will be able to wake up at least :)
16:05:01 <DinaBelova> ok, so let's start with action items
16:05:04 <DinaBelova> #topic Action Items
16:05:10 <DinaBelova> last time we had few ones
16:05:14 <DinaBelova> #link http://eavesdrop.openstack.org/meetings/performance_team/2015/performance_team.2015-12-01-15.00.html
16:05:34 <DinaBelova> I would say all of them are ok, but I did not have any update from klindgren__
16:05:46 <DinaBelova> about collecting more info for nova-conductor
16:05:57 <DinaBelova> so if he won't appear we'll skip this topic later
16:06:18 <DinaBelova> also I loved document https://etherpad.openstack.org/p/hyper-scale
16:06:28 <regXboi> :)
16:06:29 <DinaBelova> regXboi - thank you sir for sharing it
16:06:49 <DinaBelova> btw I think it could be added to new performance-docs repository
16:06:59 <DinaBelova> in let's say 'research' section
16:07:28 <DinaBelova> there is no that section there yet
16:07:31 <DinaBelova> but let's ad it :)
16:07:34 <DinaBelova> add*
16:07:57 <regXboi> DinaBelova: point me at the repo/rst and I'll take the action item to make one :)
16:08:03 <DinaBelova> one moment sir
16:08:12 <SpamapS> DinaBelova: I plan to be there yes, dropping kids off at school now. :-)
16:08:14 <DinaBelova> regXboi - https://github.com/openstack/performance-docs
16:08:22 <DinaBelova> SpamapS ;)
16:08:28 <regXboi> DinaBelova: thx
16:08:30 <DinaBelova> regXboi - right now it's just clean repo
16:08:40 <DinaBelova> the first change on review right now is the following
16:08:48 <DinaBelova> #link https://review.openstack.org/#/c/254693/
16:09:02 <DinaBelova> so we'll define repository structure on fly I believe
16:09:04 <regXboi> #action regXboi to seed performance-docs repo with information from https://etherpad.openstack.org/p/hyper-scale
16:09:13 <DinaBelova> thank you sir
16:09:29 <DinaBelova> there will be at least test_plans and test_results sections there
16:09:30 <klindgren__> hi
16:09:32 <regXboi> no worries - this way I won't forget
16:09:50 <DinaBelova> so you can add what was planned to plans and what you have reached to the results
16:09:52 <DinaBelova> regXboi :)
16:09:55 <DinaBelova> klindgren__ o/
16:10:19 <DinaBelova> klindgren__ - did you have a chance to collect some more dumps?
16:10:22 <klindgren__> had to put all the boxes this morning into the recycling... it was a lot of boxes...
16:10:31 <gokrokve> regXboi: It is necessary to add measurement units tot he coef numbers.
16:10:35 <klindgren__> DinaBelova, I did not
16:10:46 <klindgren__> I dealt with a bunch of fires.
16:10:51 <klindgren__> last week
16:11:15 <klindgren__> I did try the greenlet/rpc worker and running multiple top level processes for conductor
16:11:23 <klindgren__> with pretty much 0 change
16:11:25 <DinaBelova> klindgren__ - a-ha, ok, so will it be possible for you to do that this week - dumps + screens of tops to locale the issue?
16:11:39 <DinaBelova> klindgren__ - ok, so can you please log what was tried in the etherpad?
16:11:46 <klindgren__> yep - I can get some more of those
16:11:51 <regXboi> gokrove: no - those are correlation coef from the least squares fit - they are dimensionless
16:12:08 <DinaBelova> klindgren__ - as I see there suggestions from johnthetubaguy, but I do not see what was done there
16:12:11 <DinaBelova> thanks in advance
16:12:23 <gokrokve> regXboi: Then it will be hard to interpret them
16:12:28 <klindgren__> yep - I will update the etherpad today
16:12:34 <DinaBelova> klindgren__, thank you sir
16:12:41 <gokrokve> When you have a slope angle on the graph it should have a dimention
16:12:47 <gokrokve> like msec/VM
16:13:01 <DinaBelova> #action klindgren__ fill https://etherpad.openstack.org/p/remote-conductor-performance with what was done regarding johnthetubaguy 's suggestions
16:13:05 <regXboi> gokrove: correlation coefs run between 0 and 1 - see http://mathworld.wolfram.com/CorrelationCoefficient.html
16:13:36 <regXboi> (well, actually the r^2 runs from 0 to 1, the r can be -1 to 1
16:13:37 <gokrokve> regXboi: I understand what is that. I just say it is hard to interpret them.
16:13:59 <DinaBelova> gokrokve - so can you suggest some other measurement?
16:13:59 <gokrokve> Beside the meaning that two random sequences are similar
16:14:04 <DinaBelova> regXboi used this one
16:14:12 <DinaBelova> probably something else can be chosen here
16:14:21 <regXboi> gokrove: I'm not sure how I can help you with that - it's a way to interpret the randomness of the data without plotting it
16:14:30 <regXboi> since plots don't show up in etherpad all that well :)
16:15:02 <gokrokve> If we use linear regression model as it was suggested them coefficient m - (slope angle) has meaning for users
16:15:11 <regXboi> I'm certainly open to other proposals - this is the one I know well
16:15:13 <regXboi> um
16:15:20 <regXboi> that is there is well - I call that the slope
16:15:29 <regXboi> er there *as* well
16:15:42 <regXboi> and that has units and I admit they are incomplete
16:15:50 <gokrokve> regXboi: Yes. That is what operators will understand as they can use this value for capacity planning
16:16:14 <regXboi> (I honestly thought they were self-explanatory, but will update when I put them into the dovs)
16:16:19 <regXboi> er docs (I can't type today)
16:16:44 <regXboi> btw... DinaBelova - I'm thinking of presenting the L3 data as *examples* of the methodology
16:17:10 <DinaBelova> ok, so there is slope values there, so we just need to add additional introduction info on what's that is, what params are having what dimantion
16:17:15 <DinaBelova> regXboi - yep, sure
16:17:17 <gokrokve> regXboi: Thank you! That will be great. We can use some more user friendly doc format. It is possible to use wiki.openstack.org to document these stuff with actual graphs
16:17:21 <DinaBelova> I believe it's a good one
16:17:33 <DinaBelova> gokrokve - let's use our new repo for that purpose
16:17:39 <DinaBelova> that will be built into the html
16:17:44 <DinaBelova> and that's it
16:17:53 <gokrokve> DinaBelova: Sure. repo will work as well
16:17:59 <DinaBelova> yep, ok
16:18:03 <gokrokve> Where these docs will be published?
16:18:13 <DinaBelova> gokrokve - docs.openstack.org
16:18:22 <gokrokve> Under rally project?
16:18:22 <regXboi> DinaBelova beat me to it :)
16:18:32 <DinaBelova> but I wanted to tell about that in the specific section
16:18:35 <DinaBelova> #topic New performance-docs repo
16:18:40 <DinaBelova> so let's move to it :)
16:18:49 <gokrokve> Yep. Smooth transition
16:19:09 <regXboi> we can always play the straight people - it's *so* easy :)
16:19:09 <DinaBelova> so we're having right now new repo to publish test plans, researches results, rally scenarios used, etc. etc.
16:19:15 <DinaBelova> #link https://github.com/openstack/performance-docs
16:19:29 <DinaBelova> right now it's empty repo with only infrastructure added
16:19:38 <DinaBelova> to build the docs
16:19:57 <DinaBelova> the change to create this repo was merged few days ago
16:20:06 <DinaBelova> #link https://review.openstack.org/#/c/252846/
16:20:08 <DinaBelova> here it is
16:20:24 <DinaBelova> due to the email thread
16:20:27 <DinaBelova> #link http://lists.openstack.org/pipermail/openstack-tc/2015-December/001081.html
16:20:43 <gokrokve> What is documents format?
16:20:44 <DinaBelova> it was decided that to add this info under docs.openstack.org we need a good excuse :)
16:20:48 <gokrokve> Is it RST?
16:20:52 <DinaBelova> as we're not official openstack team yet
16:20:54 <DinaBelova> gokrokve - yes
16:21:10 <DinaBelova> so that was decided to use Rally project as an umbrella
16:21:22 <DinaBelova> to have a reason to be published oficially
16:21:35 <DinaBelova> after the following change will be merged
16:21:44 <DinaBelova> #link https://review.openstack.org/#/c/252878/
16:21:59 <DinaBelova> the docs should become published on the docs.openstack.org
16:22:19 <DinaBelova> under rally section, but I believe it's more ok than having some non official repo
16:22:39 <DinaBelova> the first WIP test plan was published on review today
16:22:41 <DinaBelova> #link https://review.openstack.org/#/c/254693/
16:22:54 <gokrokve> DinaBelova: It is good to have a place to publish. Rally is ok.
16:23:15 <DinaBelova> yeah :)
16:23:44 <DinaBelova> I believe we may use this opportunity to make our effort more public
16:23:54 <gokrokve> How did you get a link to the draft of the docs at the gate?
16:24:02 <DinaBelova> regXboi - so waiting more docs :)
16:24:02 <gokrokve> Is it a part of job output?
16:24:10 <DinaBelova> gokrokve - yep
16:24:17 <DinaBelova> gokrokve - http://docs-draft.openstack.org/93/254693/2/check/gate-performance-docs-docs/652d371//doc/build/html/
16:24:27 <DinaBelova> that was the draft built
16:24:38 <regXboi> gokrove: yes - there is a *docs* job in the pipeline and it spits out draft documentation at a URL like what DinaBelova linked
16:25:00 <manand> Changing subject: I have a question regarding plugins for Rally benchmark. Does anyone contribute plugins? will it be maintained by the community?
16:25:30 <DinaBelova> manand - right now all scenarios in rally are wrapped by plugins structure
16:25:41 <DinaBelova> andreykurilin, kun_huang - please correct me if I'm wrong
16:25:59 <andreykurilin> DinaBelova: yes, everythng is right:)
16:26:02 <DinaBelova> manand - so that means that if you have the plugin merged to rally - community will maintain it
16:26:12 <manand> thanks
16:26:21 <DinaBelova> if there will be concerns, that you won't be able to fix, you'll need to keep it workable yoursellf
16:26:33 <DinaBelova> :)
16:26:38 <manand> ok
16:26:46 <DinaBelova> ok, so any questions regarding the new repo?
16:26:55 <kun_huang> manand: rally has many plugins and support different scenarios, and you could also extend your new ones ;)
16:26:57 * regXboi sharpens pencil
16:27:04 <DinaBelova> regXboi ;)
16:27:20 <kun_huang> DinaBelova: do we need data support?
16:27:39 <DinaBelova> kun_huang - do you mean where to store rare data like some logs, etc.?
16:27:43 <DinaBelova> or what in fact?
16:27:50 <kun_huang> testing plans -> real test in openstack ci
16:27:58 <Kristian__> @andreykurilin and @DinaBelova - which plugins can we use to currently monitor and will OSprofiler be more integrated w/ Rally to monitor the error/output of Rally performance tests to help find out root-cause?
16:28:04 <DinaBelova> kun_huang, a-ha, gotcha
16:28:31 <DinaBelova> kun_huang - I believe that eventually that's possible, but we need to collect some of the OpenStack-layer related test plans for instance
16:28:46 <DinaBelova> right now some of very low-level stuff will be published on the review
16:29:08 <DinaBelova> but openstack-ci needs openstack-layer-related testing I believe
16:29:25 <kun_huang> yeah, move those testing plans is already big step here
16:29:49 <DinaBelova> kun_huang - so imho we need to have some openstack-layer collection of test plans
16:29:59 <DinaBelova> workable and tested on our labs, Hauwei one
16:30:01 <DinaBelova> etc.
16:30:07 <andreykurilin> Kristian__: "will OSprofiler be more integrated 2/Rally" sure, but rally team doesn't have enough time to implement it yet
16:30:08 <DinaBelova> and then push it as a standard
16:30:10 <kun_huang> DinaBelova: no problem sure ;)
16:30:38 <DinaBelova> Kristian__ - yeah, so that's something to be done...
16:30:56 <DinaBelova> andreykurilin - btw, may you please add this item to the work items section here https://etherpad.openstack.org/p/perf-zoom-zoom
16:30:57 <DinaBelova> ?
16:31:08 <andreykurilin> DinaBelova: sure
16:31:09 <DinaBelova> that will be the great item to pick for contributors
16:31:11 <DinaBelova> thanks
16:31:21 <Kristian__> DinaBelova - thanks for the info!
16:31:33 <DinaBelova> ok, so let's go to the next topic
16:31:34 <DinaBelova> #topic Huawei lab status
16:31:37 <andreykurilin> Btw, I suppose we need to merge support of osprofiler to more openstack projects
16:31:43 <DinaBelova> kun_huang - the floor is yours, sir
16:32:06 <DinaBelova> andreykurilin - yeah.. so we need to add these point to the items as well - keystone at least
16:32:14 <kun_huang> DinaBelova: thanks for this topic
16:32:16 <DinaBelova> kun_huang - may you please share the current status?
16:32:19 <andreykurilin> DinaBelova: and nova:)
16:32:24 <regXboi> and neutron
16:32:27 <DinaBelova> andreykurilin nova is in progress
16:32:28 <andreykurilin> :D
16:32:29 <regXboi> and ...
16:32:39 <kun_huang> My plan to share some HW to test data is supported now
16:33:02 <kun_huang> there are 6 servers now
16:33:03 <andreykurilin> wow
16:33:06 <kun_huang> and more is coming
16:33:17 <andreykurilin> It's great news
16:33:27 <DinaBelova> kun_huang - the estimate is up to 40 afiar?
16:33:30 <kun_huang> my job here is to use 10 servers doing good jobs and ask more ..
16:33:38 <andreykurilin> :)
16:33:46 <DinaBelova> :)
16:33:50 <kun_huang> DinaBelova: I hope 40 after a month
16:34:12 <DinaBelova> ok, I don't see here Leontiy Istomin - there is his test plan on the review right now
16:34:22 <DinaBelova> and we'll have what regXboi will publish as well :)
16:34:42 <kun_huang> those test plans are useful for me
16:34:52 <DinaBelova> #action DinaBelova kun_huang define the queue of test plans to start testing on the Huawei lab
16:35:00 <DinaBelova> and the access as well, btw
16:35:15 <DinaBelova> kun_huang - do we have right now some doc defining the access rules, etc?
16:35:16 <kun_huang> I know that
16:35:33 <kun_huang> nope yet
16:35:44 <DinaBelova> kun_huang - ok, cool, so lets create the action item!
16:35:54 <kun_huang> I could write something basic this week
16:35:55 * DinaBelova loves writing action items
16:36:09 <regXboi> DinaBelova: there are some folks here working internally on a performance CI/CD - I'm working on (1) getting information about it upstreamed and (2) getting it in as a 3PCI
16:36:23 <DinaBelova> #action kun_huang write some simple doc describing the access to the Huawei lab
16:36:26 <DinaBelova> kun_huang thank you sir
16:36:42 <DinaBelova> regXboi - wow, that's very cool experience
16:37:02 <regXboi> I would qualify it right now as "slow progress" ...
16:37:02 <kun_huang> regXboi: super nice thing!
16:37:09 <DinaBelova> regXboi - probably can you share eventually some thoughts on how that CI should be done in our case eventually?
16:37:25 <DinaBelova> ehterpad with ideas, examples, etc. will be perfect
16:37:35 <regXboi> DinaBelova: that's the plan, I'm still working on the timing
16:37:43 <DinaBelova> regXboi - ok, cool
16:38:00 <DinaBelova> #info ask regXboi one day about the performance CI howtos
16:38:14 <DinaBelova> ok, very cool
16:38:26 <DinaBelova> thanks kun_huang one more time for your effort!
16:38:47 <kun_huang> :)
16:38:53 <DinaBelova> probably some questions regarding the lab?
16:38:58 <DinaBelova> gokrokve? :)
16:39:38 <DinaBelova> ok, cool, let's move one
16:39:40 <DinaBelova> on*
16:39:43 <DinaBelova> #topic Nova-conductor performance issues
16:39:58 <DinaBelova> let's cover that quickly just to clarify
16:40:19 <DinaBelova> klindgren__ will try to make more dumps this week for us to understand if RabbitMQ is guilty here
16:40:34 <DinaBelova> and will add more details to the https://etherpad.openstack.org/p/remote-conductor-performance
16:40:44 <DinaBelova> I would say that's pretty all for now
16:41:05 <DinaBelova> klindgren__ - am I right?
16:42:02 * DinaBelova thinks that girl having b-day cannot be wrong :)
16:42:27 <klindgren__> yep
16:42:31 <DinaBelova> :)
16:42:49 <DinaBelova> thank you sir for finding time for making all that profiling stuff during your local fires :)
16:43:00 <DinaBelova> #topic OSProfiler weekly update
16:43:22 <DinaBelova> ok, so about opprofiler
16:43:49 <DinaBelova> I'm trying to satisfy all comments that I'm seeing on the cinder and glance commits
16:44:05 <DinaBelova> it looks like I've fixed all of them for now
16:44:42 <DinaBelova> and still change to the ceilometer https://review.openstack.org/#/c/245136/ and to the osprofiler https://review.openstack.org/#/c/251343/ need to be merged
16:45:02 <DinaBelova> I hope boris-42_ will have a bit more time this week to review the osprofiler changes :)
16:45:05 <DinaBelova> thanks in advance
16:45:26 <DinaBelova> some news: I've started addin osprifiler to nova once again :)
16:45:33 <DinaBelova> #link https://review.openstack.org/#/c/254703/
16:45:33 <boris-42_> DinaBelova: hope so=)
16:45:39 <boris-42_> DinaBelova: party hard -)
16:45:45 <DinaBelova> yeah :)
16:46:04 <DinaBelova> the blueprint for this is the following
16:46:06 <boris-42_> DinaBelova: so I am going to reply on mternish comment in spec
16:46:18 <DinaBelova> boris-42_ - very cool!
16:46:21 <boris-42_> DinaBelova: about zipkin
16:46:30 <DinaBelova> #link https://blueprints.launchpad.net/nova/+spec/osprofiler-support-in-nova
16:47:01 <DinaBelova> thanks johnthetubaguy for the comments!
16:47:23 <DinaBelova> boris-42_ - ok, so will you be able to persuade dims_ to remove his -1 there? :)
16:47:46 <Kristian__> <DinaBelova> and <boris-42> looking to integrate osprofiler in our whitebox testing, because the tracing would be awesome functionality for us in aic.  do need starter docs.  are any implementation/install-specific source docs available?
16:47:47 <boris-42_> DinaBelova: it doesn't depend on me LOL
16:48:04 <boris-42_> Kristian__: soooo
16:48:15 <boris-42_> Kristian__: you need to patch some of services in openstack
16:48:31 <DinaBelova> Kristian__ - right now it's a bit compclicated as there are a lots of patches on review
16:48:40 <DinaBelova> Kristian__ - i may create the etherpad
16:48:46 <Kristian__> <DinaBelova> aha, i see
16:48:46 <boris-42_> Kristian__: we are hoping to get those patches in OpenStack
16:48:51 <DinaBelova> with the explanation of what needs to be done
16:48:54 <boris-42_> Kristian__: so it will be simple to use osprofiler
16:48:59 <regXboi> DinaBelova: is there a blueprint for integrating with Neutron yet?
16:49:08 <DinaBelova> regXboi - not yet, sir
16:49:21 <boris-42_> regXboi: so neutron guys said that there is no need in bluepritnt
16:49:26 <boris-42_> regXboi: it's enough to create bug
16:49:35 <boris-42_> regXboi: at least they said this about 2 years ago
16:49:39 <andreykurilin> :D
16:49:41 <regXboi> boris-42_: agreed - is there an RFE then?
16:49:46 <boris-42_> regXboi: it was
16:49:55 <Kristian__> <boris-42> thanks, will be patient on this one :)
16:49:56 <boris-42_> regXboi: let me try to find it
16:50:46 <boris-42_> regXboi: https://bugs.launchpad.net/neutron/+bug/1335640
16:50:46 <openstack> Launchpad bug 1335640 in neutron "Neutron doesn't support OSprofiler" [Wishlist,Expired]
16:50:47 <boris-42_> regXboi: here is it
16:51:10 <regXboi> ah, ok - it got expired
16:51:10 <DinaBelova> #link https://bugs.launchpad.net/neutron/+bug/1335640
16:51:15 <regXboi> we'll need to unexpire it
16:51:30 <boris-42_> regXboi: yeep
16:51:35 <DinaBelova> done :D
16:51:48 <regXboi> well - I went a bit further ;)
16:52:01 <DinaBelova> ;)
16:52:21 <regXboi> and put it back to new, so that the neutron drivers team can pick it up next week
16:52:35 <DinaBelova> ok, thank you sir
16:52:59 <DinaBelova> ok, so probably we can find a volunteer to implement it as well? :)
16:53:11 <regXboi> DinaBelova: I'll shepherd for the time being
16:53:47 <DinaBelova> :)
16:54:07 <boris-42_> DinaBelova: btw https://review.openstack.org/#/c/103368/
16:54:10 <boris-42_> DinaBelova: =)
16:54:11 <DinaBelova> if you'll be able to find some it'll be cool :)
16:54:36 <boris-42_> DinaBelova: it's keystone one the most important LOL
16:54:38 <DinaBelova> boris-42_ wowo
16:55:08 <DinaBelova> boris-42_ -thanks for the link
16:55:13 <DinaBelova> I'll review and try it
16:55:14 <boris-42_> DinaBelova: Uploaded	Jun 28, 2014 1:08 PM
16:55:16 <boris-42_> DinaBelova: lol
16:55:22 <DinaBelova> #link https://review.openstack.org/#/c/103368/
16:55:37 <boris-42_> DinaBelova: I am will get in age before we merge this ;)
16:55:42 <boris-42_> I will get*
16:55:45 <DinaBelova> boris-42_ - the most important thins is that it was updated only 5 weeks ago
16:55:51 <DinaBelova> not the months :D
16:56:18 <boris-42_> DinaBelova: years of works for few lines of code )
16:56:43 <DinaBelova> boris-42_ :D
16:56:50 <DinaBelova> ok, cool
16:57:02 <DinaBelova> so it looks like osprofiler topic was covered as well
16:57:08 <DinaBelova> #topic Open Discussion
16:57:14 <DinaBelova> anything else to raise?
16:57:47 <DinaBelova> thanks everyone for coming and for the progress done!
16:58:27 <DinaBelova> ok, thank you guys
16:58:32 <DinaBelova> bye!
16:58:38 <DinaBelova> #endmeeting