15:00:15 <DinaBelova> #startmeeting Performance Team
15:00:17 <openstack> Meeting started Tue Nov 24 15:00:15 2015 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:21 <openstack> The meeting name has been set to 'performance_team'
15:00:28 <DinaBelova> hello folks o/
15:00:30 <mriedem> o/
15:00:41 <bauzas> \o
15:00:44 <DinaBelova> todays agenda located here
15:00:46 <DinaBelova> #link https://wiki.openstack.org/wiki/Meetings/Performance#Agenda_for_next_meeting
15:01:12 <DinaBelova> let's wait for few more moments for others to come :)
15:01:28 <kun_huang> o/
15:01:49 * regXboi stumbles in and finds a corner to fall asleep in
15:01:51 <andreykurilin> hi Dina
15:02:12 <Augie> hello Dina
15:02:31 <DinaBelova> hello folks!
15:02:44 <DinaBelova> so let's start with action items :)
15:02:48 <DinaBelova> #topic Action Items
15:02:56 <DinaBelova> last time we had several ones
15:02:58 <temujin_> hi all
15:03:02 <DinaBelova> #link http://eavesdrop.openstack.org/meetings/performance_team/2015/performance_team.2015-11-17-15.00.html
15:03:03 <boris-42> hi
15:03:07 <DinaBelova> temujin_, boris-42  o/
15:03:22 <DinaBelova> let's start with boris-42 action item - update osprofiler spec to fit Mitaka cycle
15:03:24 <DinaBelova> that was done :)
15:03:33 <DinaBelova> boris-42 - please send the link here :)
15:04:11 <regXboi> DinaBelova, boris-42: do we expect another respin?
15:04:17 <andreykurilin> #link https://review.openstack.org/#/c/103825/
15:04:26 <DinaBelova> andreykurilin - ah, thanks - I was looking for it
15:05:33 <DinaBelova> regXboi - do you mean new era of osprofiler? :)
15:05:52 <DinaBelova> or what?
15:05:56 <rvasilets___> hi
15:06:02 <boris-42> regXboi: some kind of that=)
15:06:04 <DinaBelova> rvasilets___ o/
15:06:18 * regXboi added an ask for the next respin of 103825
15:06:28 * regXboi just wondering if it has a chance of being added
15:06:37 <DinaBelova> regXboi - I believe yes
15:06:49 <DinaBelova> last time this spec was active there were lots of +es there
15:07:00 <DinaBelova> most of the comments were regarding spelling, tc
15:07:02 <DinaBelova> etc*
15:07:21 <regXboi> ok, let me try again - I added a comment to PS7 of 103825 asking for an additional future to be added to the spec
15:07:44 <regXboi> I want to know if that will be added to PS8 or if I have to look at spinning a new patch set to modify this if it merges
15:08:11 <DinaBelova> regXboi - sadly don't know. boris-42 ?
15:09:07 * regXboi waves at dims_
15:09:30 <dims_> hi regXboi
15:09:51 <DinaBelova> regXboi - well, I don't know still. Need closer look on current PS to undertsand if that info was added
15:10:05 <regXboi> DinaBelova: no worries, we can move on
15:10:11 <DinaBelova> ok, cool
15:10:25 <DinaBelova> one more action item regarding this etherpad
15:10:28 <DinaBelova> #link https://etherpad.openstack.org/p/rally_scenarios_list
15:10:43 <DinaBelova> some info from IBM folks appeared here :)
15:10:45 <DinaBelova> cool :)
15:11:03 <DinaBelova> although still want to see it updated by yahoo for instance (pink harlowja_ )
15:11:06 <DinaBelova> ping*
15:11:24 <DinaBelova> also I'll ping some Intel folks for them to fill this etherpad as well
15:11:31 <DinaBelova> but we see the progress has started
15:11:44 <DinaBelova> and then several items on me
15:12:11 <DinaBelova> the first one about writing the email to openstack-operators mailing list about various ratios
15:12:13 <DinaBelova> #link http://lists.openstack.org/pipermail/openstack-operators/2015-November/008846.html
15:12:48 <DinaBelova> I pinged Kristian from ATT - he was not able to meet yet with ATT performance group to find out feedback on rally
15:13:11 <DinaBelova> kun_huang - and I loved the video you've shared last time very much
15:13:31 <DinaBelova> I believe eventually we need performance benchmarking like that automated in gates
15:13:35 <DinaBelova> for various projects
15:13:41 <DinaBelova> #link https://www.youtube.com/watch?v=a0qlsH1hoKs
15:13:49 <DinaBelova> for folks who did not see it yet ^^
15:14:03 <DinaBelova> it's about neutron performance benchmarking from commit to commit
15:14:03 <kun_huang> DinaBelova: +1, Kevid did a good job with rally
15:14:10 <DinaBelova> kun_huang indeed
15:14:26 <DinaBelova> so that's it with action items from last time
15:14:45 <DinaBelova> let's move forward I guess now :)
15:14:51 <DinaBelova> any objectives?
15:15:04 <DinaBelova> k
15:15:13 <DinaBelova> #topic Team Work Items
15:15:26 <DinaBelova> harlowja_ has proposed really interesting idea during last week
15:15:46 <DinaBelova> he proposed to collect list of work items for performance team
15:15:59 <DinaBelova> that new comers (or members) can pick from
15:16:10 <DinaBelova> I've added section https://etherpad.openstack.org/p/perf-zoom-zoom
15:16:11 <DinaBelova> #link https://etherpad.openstack.org/p/perf-zoom-zoom
15:16:28 <DinaBelova> called "Work Items to grab"
15:16:40 <DinaBelova> and added two osprofiler specs to work on there
15:17:06 <DinaBelova> and I'm kindly encouraging people to add bugs, issues, investigations to be done, tests to be run here
15:17:34 <DinaBelova> I believe I'll need to send the email to openstack-operators mailing list about it as well
15:18:15 <DinaBelova> regXboi, Augie, mriedem - so if you have anything hot to take a look on - please put your thoughts there
15:18:42 <DinaBelova> #action DinaBelova write an email about work items to the openstack-dev and openstack-operators
15:18:49 <Augie> Dina - OK, will do
15:18:54 <DinaBelova> Augie - thanks :)
15:18:59 <regXboi> ack
15:19:03 <mriedem> yeah, need ML, can't keep track of the bazillion etherpads
15:19:11 <DinaBelova> mriedem for sure :)
15:19:15 <DinaBelova> will do
15:19:37 <DinaBelova> I really hope that list of concrete items to work on can attract more people here
15:19:48 <DinaBelova> and some hands that can work on specific points
15:20:12 <DinaBelova> not something abstract like 'some' testing or whatever
15:20:21 <DinaBelova> any questions regarding this topic?
15:20:58 <DinaBelova> ok
15:21:21 <DinaBelova> so let's move forward (although today I feel like I'm having mostly monologue :))
15:21:30 <DinaBelova> #topic Nova-conductor performance issues
15:21:36 <DinaBelova> hot topic :)
15:21:44 <DinaBelova> ok, so about latest news
15:22:07 <DinaBelova> I've tried to reproduce this issue on my dev env - sadly, that try was not very successful :(
15:22:33 <mriedem> so one idea is running the n-api-meta service in the large ops job in the gate
15:22:55 <DinaBelova> In fact my nova-api service (containing nova-api-metadata inside) started eating CPU much faster than conductor
15:23:00 <mriedem> that job uses the nova fake virt driver though, so i'm not sure if some extra plumbing would be needed there to tickle it
15:23:03 <DinaBelova> mriedem - I believe that will be useful
15:23:22 <DinaBelova> in case of fake driver - yeah, I believe it'll be needed
15:23:54 <DinaBelova> mriedem - I'll take a look on this faking as well on my env first of all
15:24:13 <DinaBelova> and then let's see if that will be more or less easy or not
15:24:32 <DinaBelova> don't think in fact :(
15:24:37 <mriedem> wasn't SpamapS or someone talking about the nova fake virt driver yesterday?
15:24:49 <DinaBelova> mriedem - yep, we were talking about this
15:25:00 <DinaBelova> but we have feeling it won't be very easy
15:25:03 <mriedem> fwiw, adding n-api-meta to a large ops type config should be easy enough, would be a separate job maybe
15:25:14 <DinaBelova> mriedem -ack
15:25:20 <mriedem> i could poke on testing that out with a devstack-gate change, or talk to sdague about it
15:25:36 <DinaBelova> mriedem - that will be super-nice
15:25:52 <mriedem> i'll add a todo
15:26:01 <DinaBelova> mriedem thanks
15:26:40 <DinaBelova> #action mriedem testing that out with a devstack-gate  n-api-meta to a large ops type config change, or talk to sdague about it
15:27:05 <DinaBelova> fyi right now I'm testing profiling periodical task
15:27:08 <DinaBelova> for conductor
15:27:20 <DinaBelova> on my env - if it works ok, I'll ask klindgren_ to run it as well
15:27:53 <DinaBelova> I guess that will add some info here as well (as right now I see much more load on the nova-metadata-api than on the conductor itself)
15:28:20 <DinaBelova> so probably something else is happening on their cloud we just don't know about
15:28:53 <DinaBelova> ok, any additional ideas for me on what to look at here?
15:29:44 <DinaBelova> k, let's move forward
15:29:48 <DinaBelova> #topic OSProfiler weekly update
15:30:28 <DinaBelova> ok, so about profiler - we have new version of the spec proposed by boris-42 - thanks :)
15:30:52 <DinaBelova> and currently the most interesting chain on review is https://review.openstack.org/#/c/247005/
15:31:17 <boris-42> DinaBelova: heh that one is big
15:31:22 <DinaBelova> :D
15:31:26 <boris-42> DinaBelova: maybe we should split it to 2 patches
15:31:34 <DinaBelova> boris-42 - yep, possibly
15:31:35 <boris-42> DinaBelova: one that adds strucutre of drivers
15:31:44 <boris-42> DinaBelova: another that implements ceilometer on it
15:32:05 <boris-42> DinaBelova: it will be simpler to review this to logical piece
15:32:21 <DinaBelova> I want to finish first my debug of what's going wrong with the adding osprofiler timestamps to the notifications
15:32:25 <DinaBelova> and then I can do that, yep
15:32:26 <DinaBelova> thanks
15:32:54 <DinaBelova> #action DinaBelova split https://review.openstack.org/#/c/247005/ to two commits: adding drivers structure and adding ceilometer driver
15:33:16 <DinaBelova> anyway I encourage everyone here to review what's currently available
15:33:32 <DinaBelova> kun_huang, harlowja_, SpamapS  - you're welcome :)
15:34:05 <DinaBelova> and, as said, there are two more specs to work on
15:34:13 <DinaBelova> they're listed here https://etherpad.openstack.org/p/perf-zoom-zoom
15:34:26 <DinaBelova> and they still do not have any assignee :)
15:34:45 <DinaBelova> so please try working through them :)
15:35:10 <DinaBelova> any questions / ideas regarding osprofiler?
15:35:12 <DinaBelova> boris-42?
15:35:22 <boris-42> DinaBelova: reading
15:35:50 <boris-42> DinaBelova: no question about osprofiler=)
15:36:02 <DinaBelova> :D for you I've prepared option 'ideas'
15:36:04 <DinaBelova> :D
15:36:41 <DinaBelova> ok, cool :)
15:36:54 <DinaBelova> Today we're having super-fast meeting :)
15:36:57 <DinaBelova> #topic Open Discussion
15:37:14 <DinaBelova> here I just wanted to share the recent news with you, folks
15:37:35 <DinaBelova> currently performance team inside Mirantis is working on various researches
15:37:55 <DinaBelova> currently mostly around components like DB, MQ, Ceph, etc.
15:38:15 <DinaBelova> and we're preparing the testing plans to be shared with you, guys
15:38:28 <DinaBelova> I hope they will be available before the end of week
15:38:56 <DinaBelova> and then I'll kindly ask you folks to give some feedback on what else and how can be tested here
15:39:29 <DinaBelova> I'll ping kun_huang, harlowja_, SpamapS, mriedem, Augie, klindgren_ and others here and will write an email about that as well
15:40:05 <DinaBelova> your feedback will be essential - if any, we can add your items / test cases to the testing as well
15:40:37 <DinaBelova> k, any other topics, ideas or issues to be discussed?
15:41:11 <DinaBelova> I'm feeling like I was disconnected and talking with myself for a while :)
15:41:25 <Augie> Dina - that's great on the testing plans...  Do you expect to use an environment withing Mirantis for the testing?
15:41:38 <Augie> withing = within
15:41:48 <DinaBelova> Augie - yep, currently we're having not very big lab available
15:42:10 <DinaBelova> hoping to grab ~500 compute nodes close to the New Year
15:42:35 <DinaBelova> but this env will be Mirantis-available now
15:42:40 <Augie> any thoughts on potential use of that Intel/Rackspace environment?
15:44:02 <DinaBelova> Augie - in terms of Mirantis and Intel coop we're planning to use it, but right now we are not using their super-big envs :)
15:44:20 <DinaBelova> so yep, it's planned
15:44:27 <Kristian__> dina- here's latest Agile use case/feedback from my Rally development team@ t. being use case: as a test developer/executioner I want Rally to ramp up threads/users gradually over a configurable time period until reaching a target load so that tests with large loads a) do not have a sudden load impact on systems and b) a test can gradually increase until a breakpoint is found, instead of having to run subsequent tes
15:44:35 <Augie> Dina - ok
15:44:41 <DinaBelova> Kristian__ o/
15:45:16 <DinaBelova> boris-42 ^^ please take a look and comment :)
15:45:22 <Kristian__> finding breakpoints in OS deployment are crucial to us
15:45:55 <DinaBelova> Kristian__ - so you want something like growing load pattern tests - am I right?
15:47:22 <Kristian__> yes, so to paraphrase a bit- if rally detects load is too large at first, test can scale back at smaller load intervals then grow to higher load patterns until max load/errors based on SLA start forming
15:47:45 <DinaBelova> boris-42, are you around?
15:48:16 <andreykurilin> Kristian__: we have something like this in rally roadmap
15:48:20 <Kristian__> so can stay for a bit, Boris knows how to reach me....i've got an open door for him
15:48:43 <DinaBelova> andreykurilin, a-ha, can you share the estimates on when this integration is planned?
15:48:45 <Kristian__> ok.  @andreykurilin - which version is this expected in?
15:49:40 <andreykurilin> I can give you any estimations. As far as I know, there are no activities on this type of runner. Let's move this topic for next meeting
15:49:46 <andreykurilin> *I can't
15:50:10 <DinaBelova> andreykurilin, ack
15:50:46 <DinaBelova> #acion andreykurilin find out the estimates on when growing load pattern tests runner will be available in Rally
15:51:02 <DinaBelova> ok, so it looks like we're done here
15:51:03 <andreykurilin> #action andreykurilin find out the estimates on when growing load pattern tests runner will be available in Rally
15:51:07 <DinaBelova> :D
15:51:17 <DinaBelova> ok, so two action items on you :)
15:51:29 <DinaBelova> anything else to be discussed?
15:51:35 <andreykurilin> no, your action item doesn't have char "t"
15:51:38 <andreykurilin> :)
15:51:45 <DinaBelova> ah :)
15:51:48 <DinaBelova> you're right :)
15:52:00 <DinaBelova> ok, so I guess that's it for today
15:52:10 <DinaBelova> thanks everyone for coming and taking part :)
15:52:12 <mriedem> thanks
15:52:18 <DinaBelova> #endmeeting