17:27:50 <boris-42> #startmeeting Rally
17:27:51 <openstack> Meeting started Tue Jan 14 17:27:50 2014 UTC and is due to finish in 60 minutes.  The chair is boris-42. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:27:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:27:54 <openstack> The meeting name has been set to 'rally'
17:28:23 <boris-42> hughsaunders hi
17:28:26 <hughsaunders> hi
17:28:34 <boris-42> sorry I late a bit=)
17:29:14 <boris-42> os let's start
17:29:21 <boris-42> seems like nobody is here=) lol
17:30:07 <boris-42> okay I have a couple things that I would like to disucss
17:30:30 <boris-42> #topic Rally & Benchamrk engine better result collecting
17:30:58 <boris-42> So the main idea that now we are collection only times of whole scenario loop
17:31:35 <boris-42> but it could be improved to collect results of all atomic actions
17:31:45 <hughsaunders> that would be good
17:31:59 <boris-42> so we already have some kind of base classes
17:32:10 <boris-42> https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/utils.py
17:32:17 <boris-42> so with such atomic actions
17:32:23 <boris-42> we should make some decorator
17:32:35 <boris-42> that will be add on them and that will contain name of "parameter"
17:33:29 <boris-42> then we will store all data in class object (like now https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/servers.py#L38 this work)
17:33:59 <boris-42> and then get it https://github.com/stackforge/rally/blob/master/rally/benchmark/runner.py#L55
17:34:05 <boris-42> here from class instance
17:34:21 <boris-42> and push in key like full_times
17:34:37 <boris-42> hughsaunders thoughts ^
17:35:46 <hughsaunders> boris-42: I think more granular results would be useful
17:36:03 <hughsaunders> like I was attempting with scenario_specific_results
17:36:14 <boris-42> yep but that is a bit different
17:36:23 <boris-42> and could be implemented together=)
17:36:35 <boris-42> I will try to find time to retest your patch soon
17:36:43 <hughsaunders> sounds like you have a plan that could work across all scenarios?
17:36:52 <boris-42> hughsaunders yep
17:37:22 <boris-42> hughsaunders just add decorators to all "atomic" actions (like _boot_server)
17:37:37 <boris-42> that will store in class object
17:37:41 <boris-42> all timings
17:37:51 <boris-42> and then after execution of method get them
17:37:59 <hughsaunders> that makes sense
17:38:24 <boris-42> so this will be feature of scenario runner not every benchmark
17:39:06 <boris-42> okay so it seems like good idea
17:39:09 <hughsaunders> boris-42: may be able to auto-decorate with a metaclass?
17:39:22 <boris-42> not sure
17:39:31 <hughsaunders> I'm not sure either, would have to do some reading
17:39:48 <boris-42> I mean I would like to decorate only "atomic" actions
17:40:31 <boris-42> if we continue storing all such methods in couple classes
17:40:41 <boris-42> like now https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/keystone/utils.py#L35
17:40:47 <boris-42> then it is possible
17:41:28 <hughsaunders> ok, probably simpler to use a decorator as you suggested
17:41:44 <boris-42> yep to specify some beauty names
17:41:55 <boris-42> like here https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/keystone/utils.py#L41
17:41:59 <hughsaunders> anyone else here BTW?
17:42:08 <boris-42> miarmak should be=)
17:42:11 <miarmak> +
17:42:14 <miarmak> hi 2 all)
17:42:36 <hughsaunders> hi miarmak :)
17:43:27 <boris-42> so .. I think that we could continue discussing this
17:43:32 <boris-42> in other place
17:43:43 <boris-42> like ether pad but the major idea is ok
17:43:49 <boris-42> #topic Rally & Tempest
17:44:02 <boris-42> miarmak could you share your results ?
17:45:01 <miarmak> boris-42: : over what period?
17:45:12 <boris-42> miarmak about  Rally & Tempest
17:45:29 * IgorYozhikov is now away: went away...
17:46:26 <miarmak> tomorrow I'll start investigate this issue. Today and yesterday i tryed to finish all my activities in other issues
17:46:49 <boris-42> so you didn't start work around this part?
17:46:55 <boris-42> Okay I will just share for others
17:47:07 <boris-42> We are going to add some kind of cloud verification
17:47:18 <hughsaunders> I was wondering if tempest has/needs a plugin system, so stackforge projects can use that to provide their own tests
17:47:37 <boris-42> hughsaunders hmm yep I think it need
17:47:39 <miarmak> Yesterday I installed rally, but had some problems with it usage (Hugh alredy have reported this bug)
17:47:50 <boris-42> miarmak and fixed=)
17:48:07 <miarmak> oh, it's great) hughsaunders: Thanks!
17:48:25 <boris-42> so the tempest & rally integration means next one
17:48:39 <boris-42> we have at this moment part that creates "deployments"
17:48:45 <boris-42> part that runs "benchmarks"
17:49:07 <boris-42> and there is no way before running benchmark to ensure that whole cloud works properly
17:49:15 <boris-42> so we would like to add 1 more command
17:49:27 <boris-42> something like openstack-rally verify --deploy-id
17:49:46 <boris-42> that will make proper tempest config
17:49:51 <boris-42> and run tempest against cloud
17:50:16 <boris-42> miarmak hughsaunders  I think it will be nice functionality ^
17:50:34 <hughsaunders> ahh, so we don't actually need rally specific tests, we run standard tempest tests before benchmarking
17:50:46 <boris-42> not only before
17:50:49 <boris-42> when you would like
17:51:13 <hughsaunders> yeah
17:51:14 <boris-42> just "button" to run tempest against coud
17:51:17 <boris-42> cloud*
17:52:34 <boris-42> so I think it's ok?
17:52:36 <boris-42> hughsaunders ^
17:52:39 <boris-42> miarmak ^
17:52:42 <hughsaunders> I think that would be useful
17:52:59 <miarmak> I think so
17:53:07 <boris-42> ok nice
17:53:11 <boris-42> last topic for today
17:53:20 <boris-42> #topic Rally & Benchmarks Inside VMs
17:54:12 <boris-42> so the ideas is to make next steps
17:54:51 <hughsaunders> boris-42: next step: get my review to patchset 20 ;-)
17:55:02 <boris-42> lol=)
17:55:23 <boris-42> using heat install inside VMs different benchmark suits
17:55:26 <boris-42> then run them
17:55:29 <boris-42> collect info
17:55:34 <boris-42> present to user
17:55:48 <boris-42> so we can use current benchmark engine
17:56:06 <boris-42> and I don't know probably N different benchmark scenario for N different suits
17:56:42 <boris-42> each of benchmark scenario will have some (probably hardcoded heat template) that makes required installation
17:57:06 <boris-42> so we will be able to run a lot of simultaneously benchmarks in cloud
17:57:30 <boris-42> and check that there is no difference if we run with 1 active user and with N active user
17:57:47 <boris-42> hughsaunders miarmak  thoughts ^
17:58:07 <hughsaunders> that would be good, I prob won't have time to work on it soon though :(
17:58:24 <hughsaunders> also have to go at in two mins
17:58:30 <boris-42> =)
17:58:40 <miarmak> why hardcoded? We can make several heat templates and use necessary щту
17:58:46 <miarmak> one*
17:59:06 <boris-42> because every heat template is related with some benchmark suit
17:59:16 <boris-42> different benchmark suit will be in different scnearios
17:59:32 <boris-42> because processing of output data will be done in different ways
17:59:40 <boris-42> but this is all under discussion
17:59:46 <boris-42> so i will make some ether pad
17:59:50 <miarmak> oh, ok
18:00:01 <boris-42> about this and probably send to mailing list
18:00:08 <boris-42> Okay we have to end meeting
18:00:11 <boris-42> see you later
18:00:17 <boris-42> #endmeeting