17:05:35 <boris-42_> #startmeeting Rally
17:05:36 <openstack> Meeting started Tue Feb 18 17:05:35 2014 UTC and is due to finish in 60 minutes.  The chair is boris-42_. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:05:37 <boris-42_> miarmak ping
17:05:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:05:40 <openstack> The meeting name has been set to 'rally'
17:05:49 <boris-42_> Hi everybody
17:05:54 <miarmak> Hi
17:05:55 <boris-42_> seems like I have some issues
17:06:06 <boris-42_> Hi one more time hughsaunders rediskin miarmak julienvey stannie tnurlygayanov tzabal jroovers|afk jaypipes mkoderer
17:06:36 <msdubov> boris-42_ hi
17:06:41 <boris-42_> nkhare ping
17:07:03 <nkhare> boris-42_, pong
17:07:42 * boris-42_ writing todays topics
17:08:39 <tnurlygayanov___> Hi
17:10:43 <boris-42_> Okay todays topics
17:10:45 <boris-42_> 1. Benchmarking refactoring
17:10:47 <boris-42_> 2. Benchmarks in VMs
17:10:49 <boris-42_> 3. Deployments (Multinode/Bugfixes/LXC)
17:10:51 <boris-42_> 4. Results processing (Graphics)
17:10:53 <boris-42_> 5. Verification updates
17:10:55 <boris-42_> #topic 1. Benchmarking refactoring
17:11:18 <boris-42_> msdubov could you introduce us, what happened during the last week and what we are going to do on this
17:11:25 <boris-42_> msdubov I will find link to the doucment
17:12:11 <boris-42_> #link https://docs.google.com/a/mirantis.com/document/d/1LYUAHkZQD8W7dtlj2I3PDA6x67TiD3AMnSWG6ljsups/edit
17:12:22 <boris-42_> msdubov are you typing?)
17:12:39 <msdubov> boris-42_ Well I thikk it's better to see the weekly updates page on wiki :)
17:12:45 <boris-42_> msdubov nope
17:12:51 <boris-42_> msdubov it is not better
17:12:57 <boris-42_> msdubov just say in couple of words
17:13:31 <boris-42_> eyerediskin ping
17:13:49 <rediskin> sup
17:14:06 <boris-42_> msdubov is trying to say high level view of benchmarking refactoring
17:14:14 <boris-42_> msdubov pls just go through all points
17:14:25 <boris-42_> msdubov so we will be able to discuss them
17:14:28 <msdubov> boris-42_ OK wait a minute pls
17:16:42 <msdubov> Well the fisrt thing is the reimplementation of different benchmarking strategies via subclassing
17:17:19 <msdubov> This makes the code way moreclean
17:18:10 <msdubov> Also we've added support for the so-called "user" endpoints
17:18:25 <msdubov> i.e. endpoints withou admin permissions
17:18:39 <msdubov> But they are not fully integra ted yet
17:20:09 <boris-42_> msdubov okay let's speed up this discussion
17:20:39 <boris-42_> msdubov point 1 from that document
17:20:44 <boris-42_> Improving scenario input args validation:
17:21:14 <boris-42_> jroovers said that probably there will be some cases
17:21:23 <boris-42_> when you need to check for every single user
17:21:34 <boris-42_> My concern is that verification should be very quick
17:21:47 <boris-42_> and in case when we are creating 5k tmp users validation could take for a while
17:22:07 <msdubov> boris-42_ you mean different user types?
17:22:12 <boris-42_> nope
17:22:20 <msdubov> will they be really so different?
17:22:33 <boris-42_> in case of network they will be a bit different
17:22:37 <msdubov> boris-42_ what then?
17:22:39 <boris-42_> in different tenants
17:22:49 <boris-42_> so probably we should add new type
17:22:59 <boris-42_> so we will have TYPE.tennant
17:23:04 <boris-42_> tenant
17:23:11 <boris-42_> It will run for every tenant 1 time
17:23:29 <boris-42_> it should be actually used only for network validator
17:23:54 <boris-42_> so we will be able to run from 1 users, for 1 user from every tenant
17:24:05 <boris-42_> and from admin
17:24:16 <boris-42_> I think it will cover most situation and will be pretty optimal
17:24:45 <boris-42_> any thoughts?)
17:24:53 <boris-42_> miarmak rediskin ^
17:24:55 <boris-42_> hughsaunders ^
17:25:33 <msdubov> boris-42_ are there already network validators implemented?
17:25:34 <tnurlygayanov___> looks good but probably we can make it more agile
17:26:00 <tnurlygayanov___> with the ability to configure count of users/tenants and etc.
17:26:21 <boris-42_> tnurlygayanov___ could you make some case?)
17:26:32 <boris-42_> tnurlygayanov___ where this complexity will be used actually?
17:27:44 <boris-42_> tnurlygayanov___ are you typing?)
17:27:53 <tnurlygayanov___> so we can have the users with different roles in one tenant
17:28:02 <boris-42_> tnurlygayanov___ nope
17:28:10 <boris-42_> tnurlygayanov___ because we are generating them with one role
17:28:23 <tnurlygayanov___> hm, ok.
17:28:25 <boris-42_> tnurlygayanov___ in case of procreated user
17:28:32 <boris-42_> tnurlygayanov___ we have to test for all users
17:28:43 <boris-42_> tnurlygayanov___ because of roles and so on
17:28:51 <boris-42_> tnurlygayanov___ so I think this is a good step for start
17:29:02 <boris-42_> tnurlygayanov___ if we will support generating users with different roles
17:29:08 <boris-42_> tnurlygayanov___ then okay=)
17:29:33 <boris-42_> #topic 1.2  Running benchmarks with pre-created non-admin users
17:29:50 <boris-42_> ^ okay msdubov prepared all pathes, but they are waiting for improving validation
17:30:04 <tnurlygayanov___> yes, I agree, this is good for "start" and we can improve these scenarios in the future
17:30:11 <boris-42_> tnurlygayanov___ yep
17:30:26 <boris-42_> #topic 1.3  Improve scenario pluggability
17:30:27 <msdubov> Yep and I've issued a new, related, patch that refactors osclients a bit
17:30:55 <boris-42_> Okay to improve scenario pluggability we should make changes to existing confi
17:31:11 <boris-42_> msdubov tnurlygayanov___ rediskin miarmak are everybody agree with it?
17:31:15 <miarmak> I am looking this patch https://review.openstack.org/#/c/67720 and I have 1 question.
17:31:42 <boris-42_> miarmak so ask it=)
17:32:10 <tnurlygayanov___> what changes in config?
17:32:26 <miarmak> here we have endpoints in list and what if we will provide severel admins credentials? by mistake or smth like this
17:32:30 <boris-42_> tnurlygayanov___ take a look at document
17:32:39 <tnurlygayanov___> ok
17:32:39 <boris-42_> tnurlygayanov___ point 3.1
17:32:52 <boris-42_> tnurlygayanov___ new format of task config
17:33:01 <miarmak> it's not enough clear for me
17:33:08 <boris-42_> miarmak it is actually okay
17:33:10 <hughsaunders> hi all
17:33:12 <boris-42_> miarmak to provide N admins
17:33:13 <boris-42_> hughsaunders hi=)
17:33:26 <miarmak> hughsaunders: hello)
17:33:32 <boris-42_> miarmak then it will chose them randomly and run rally as a from admin
17:33:36 <hughsaunders> hey miarmak boris-42_, sorry late busy week!
17:33:47 <boris-42_> miarmak so with procreating all tmp users and so on
17:33:54 <boris-42_> hughsaunders no problems =)
17:34:00 <tnurlygayanov___> boris-42_: yes, looks good
17:34:00 <miarmak> boris-42_: oh, okey, thanks)
17:34:17 <boris-42_> hughsaunders we are discussing new task config at the moment
17:34:22 <hughsaunders> cool
17:35:01 <boris-42_> hughsaunders so what do you think about new task config?)
17:35:18 <tnurlygayanov___> boris-42_:   "cloud_config" block is unclear
17:35:29 <boris-42_> tnurlygayanov___ agree
17:35:43 <tnurlygayanov___> what we want to describe in this block?
17:35:48 <boris-42_> tnurlygayanov___ hughsaunders  miarmak msdubov variants
17:36:06 <boris-42_> we are describing all operation that will be in "init"
17:36:14 <hughsaunders> resource_creation ?
17:36:20 <boris-42_> cloud_init
17:36:21 <boris-42_> ?
17:36:44 <boris-42_> because there will be as well network setup and probably some other stuff
17:37:07 <tnurlygayanov___> setup_configuration
17:37:10 <hughsaunders> yep, so it describes the resources (users, tenants, networks, images?) that will be created for use in that run
17:37:20 <tnurlygayanov___> and need to move this block up
17:37:23 <boris-42_> hughsaunders something like thath
17:37:29 <boris-42_> tnurlygayanov___ where?
17:37:31 <boris-42_> tnurlygayanov___ why?
17:38:29 <tnurlygayanov___> need to move 'setup; block before 'scenario args' and other block to show that this action will be the fisrt
17:38:34 <tnurlygayanov___> *first
17:38:43 <boris-42_> tnurlygayanov___ it is actually JSON=)
17:38:48 <boris-42_> tnurlygayanov___ order doesn't matter
17:39:17 <boris-42_> tnurlygayanov___ but agree it will be better for wiki
17:39:25 <tnurlygayanov___> yes
17:39:29 <boris-42_> so name?
17:39:47 <boris-42_> setup_config / cloud_config / resource_config
17:39:50 <hughsaunders> I still like resource_creation, as it describes whats actually happening
17:39:57 <hughsaunders> or resource_config, that works for me also
17:40:30 <miarmak> cloud_config imho better)
17:40:39 <tnurlygayanov___> so, setup links to Python-unittests pattern )
17:41:18 <boris-42_> fuu = )lol
17:41:29 <boris-42_> we are not able to decide the right name for it=)
17:41:35 <miarmak> =)
17:41:37 <hughsaunders> load_generator --> run_configuration
17:42:44 <tnurlygayanov___> preconditions maybe.
17:43:15 <hughsaunders> tnurlygayanov___: that implies things that must be setup before the task is started
17:43:32 <hughsaunders> tnurlygayanov___: rather than resources rally will create for the task?
17:44:06 <boris-42_> so okay I added all variants
17:44:11 <boris-42_> to doc
17:44:16 <boris-42_> let discuss other things=)
17:44:22 <hughsaunders> :)
17:44:23 <boris-42_> and continue discussion of this in rally caht=)
17:44:28 <hughsaunders> boris-42_: random.choice()
17:44:36 <boris-42_> ya something like that=)
17:44:46 <tnurlygayanov___> ))
17:44:49 <boris-42_> #topic Move load_generator validation to scenario runners
17:45:13 <boris-42_> okay this is the last blocker to make scenario runners totally pluggable
17:45:31 <boris-42_> so we will add just method to scenario runner aka validate
17:45:38 <boris-42_> and send this part "load_generator"
17:45:46 <boris-42_> to it and check that it pass
17:46:06 <boris-42_> so every scenario runner will be able to override it and use own schema
17:46:12 <boris-42_> (e.g. in deployment stuff0
17:46:19 <boris-42_> I think it's ok approach?)
17:47:33 <boris-42_> somebody?)
17:47:42 <tnurlygayanov___> now we use different validators for each parameter
17:47:55 * hughsaunders looks in code
17:48:01 <tnurlygayanov___> and how it will workk after this change?
17:48:15 <boris-42_> tnurlygayanov___ now we have spaghetti
17:48:21 <boris-42_> tnurlygayanov___ in benchmark engine
17:48:23 <tnurlygayanov___> we will have only one method .validate()
17:48:28 <tnurlygayanov___> yes
17:48:42 <boris-42_> tnurlygayanov___ yep that will use specify for scenario runner json schema
17:49:00 <boris-42_> tnurlygayanov___ and probably specify validate() implantation
17:49:31 <boris-42_> tnurlygayanov___ https://github.com/stackforge/rally/blob/master/rally/benchmark/runners/continuous.py#L24
17:49:41 <boris-42_> tnurlygayanov___ in this class you will just add new method
17:49:53 <boris-42_> tnurlygayanov___ validate() that is specific for  ContinuousScenarioRunner
17:50:23 <boris-42_> tnurlygayanov___ so instance of this scenario runner will be chased if you specify type: "continuous"
17:50:30 <tnurlygayanov___> but how we will validate different test scenaries with different parameters using only one method?
17:50:36 <hughsaunders> boris-42_: ahh, I understand, so each scenario runner can validate its own config block, and therefore require different parameters
17:50:46 <boris-42_> hughsaunders yep
17:50:51 <tnurlygayanov___> hm
17:50:52 <hughsaunders> ok, makes sense to me
17:51:09 <boris-42_> tnurlygayanov___ https://github.com/stackforge/rally/blob/master/rally/benchmark/engine.py#L101-L121
17:51:19 <tnurlygayanov___> we will use different runners for scenaries with different parameters?
17:51:19 <boris-42_> tnurlygayanov___ we have this spaghetti
17:51:39 <boris-42_> tnurlygayanov___ different scneariorunners already now have different parameters
17:52:00 <boris-42_> tnurlygayanov___ continue accept always "active_users" and periodical accepts "period"
17:52:38 <boris-42_> tnurlygayanov___ so load_generator parameters are different
17:52:43 <boris-42_> tnurlygayanov___ for different load generators
17:52:51 <tnurlygayanov___> ok
17:52:59 <boris-42_> tnurlygayanov___so only laod_generator knows how to validate them
17:53:14 <boris-42_> tnurlygayanov___ so in benchmark engine we will get proper scenario runner using "type"
17:53:20 <boris-42_> tnurlygayanov___ and call validate()
17:53:20 <tnurlygayanov___> yes, now it is clear, thank you )
17:53:33 <boris-42_> so everybody agree with this chage?
17:54:02 <boris-42_> #topic  Stress execution
17:54:03 <boris-42_> 
17:54:17 <boris-42_> Okay here we have to make as well changes
17:54:20 <boris-42_> in config
17:54:25 <boris-42_> are everybody agree with them?
17:55:04 <hughsaunders> boris-42_: yep
17:55:36 <boris-42_> hughsaunders yep so about stroing results
17:55:40 <tnurlygayanov___> we will use asynchronous engine for stress tests?
17:55:41 <msdubov> boris-42_ yep
17:55:55 <boris-42_> tnurlygayanov___ nope
17:56:31 <boris-42_> tnurlygayanov___ all steps one by one
17:57:01 <tnurlygayanov___> and where we can describe the scenaries for this stress test?
17:57:03 <boris-42_> tnurlygayanov___ first you are running for N active user then for N+step
17:57:09 <boris-42_> tnurlygayanov___ ?
17:57:30 <tnurlygayanov___> hm, I found.
17:57:40 <boris-42_> tnurlygayanov___ {“start”: 2, “stop”: 10, “step”: 1,  “max_failure_rate”: 0.8}
17:57:55 <boris-42_> tnurlygayanov___ this is the same as adding in array N times
17:58:03 <boris-42_> tnurlygayanov___ full description of benchmark
17:58:19 <boris-42_> tnurlygayanov___ so it will produce N separated benchmarks
17:58:25 <tnurlygayanov___> looks like we write incorrect JSON with '|'
17:58:33 <boris-42_> tnurlygayanov___ lol
17:58:39 <boris-42_> tnurlygayanov___ it is "OR"
17:58:49 <tnurlygayanov___> ok )
17:59:09 <boris-42_> it means that in that field you are able to use different
17:59:18 <boris-42_> syntax
17:59:19 <boris-42_> so
17:59:31 <boris-42_> I think that stress execution should be stored as a one task
17:59:37 <hughsaunders> 30 seconds
17:59:46 <boris-42_> we can continue in Rally chat
17:59:59 <boris-42_> #endmeeting