17:01:08 <boris-42> #startmeeting Rally
17:01:09 <openstack> Meeting started Tue Mar 11 17:01:08 2014 UTC and is due to finish in 60 minutes.  The chair is boris-42. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:13 <hughsaunders> hi boris-42
17:01:14 <openstack> The meeting name has been set to 'rally'
17:01:26 <boris-42> hughsaunders msdubov ping
17:01:36 <hughsaunders> pong
17:01:38 <olkonami1> hi
17:01:41 <msdubov> boris-42 hi
17:03:41 <marcoemorais> hi
17:03:45 * afazekas o/
17:04:44 <boris-42> I think we could start
17:05:02 <boris-42> Sooo
17:05:21 <boris-42> thanks msdubov for latest weekly update https://wiki.openstack.org/wiki/Rally/Updates#March_10.2C_2014
17:05:54 <boris-42> #topic benchmark context
17:06:05 <boris-42> Ok it's new thing in Rally
17:06:20 <marcoemorais> msdubov: thank you!
17:06:32 <hughsaunders> msdubov: yep, great summary
17:06:37 <msdubov> :)
17:07:15 <boris-42> So we restructured code
17:07:45 <boris-42> to add support of context objects
17:08:56 <boris-42> we first of move all related code to creation users and generic cleanup
17:09:10 <boris-42> in separated classes that are used with "with statement"
17:09:40 <boris-42> then we build base class for context and introduce context object that is shared between all context classes
17:09:53 <boris-42> Now hughsaunders is working to pass this context to benchmark engine
17:10:02 <boris-42> becnahmark sceanrio**
17:10:08 <hughsaunders> yep
17:10:32 <boris-42> today we discussed with hughsaunders  this stuff
17:10:48 <hughsaunders> a copy of the context object minus the users key (See scrollback in #rally)
17:10:54 <boris-42> hughsaunders yep yep
17:11:05 <boris-42> msdubov marcoemorais and others are agree with this?
17:11:20 <boris-42> are you agree*
17:12:24 <msdubov> Yep, the code stays readable, I think
17:12:58 <boris-42> okay
17:13:03 <boris-42> so let's move
17:13:05 <boris-42> next thing is
17:13:14 <boris-42> new task config and validation
17:13:50 <boris-42> I am going to make one patch that will move validation from benchmark.engine to corresponding scenario.base, runner.base and context.base stuff
17:14:13 <boris-42> as well I will change config in the same change ^ cause it makes this part simpler
17:14:46 <boris-42> So we will be able to add new benhcmarks.runner without any changes in benchmark.engien
17:15:30 <boris-42> marcoemorais msdubov hughsaunders  ^ btw we will need to standardize output of runner
17:15:38 <hughsaunders> boris-42: will the config change be to use the new args/runner/context keys?
17:15:45 <boris-42> hughsaunders yep
17:15:50 <hughsaunders> great
17:16:11 <boris-42> hughsaunders so we will pass context content to context.base validation , runner to runner.base and args to scenario.base
17:17:57 <hughsaunders> makes sense
17:18:13 <msdubov> boris-4w What about runner output standartization?
17:18:18 <msdubov> What do you mean by this?
17:18:29 <boris-42> msdubov we should have some class ScenarioRunnerResult
17:18:52 <boris-42> msdubov that should be returned as a result of runner.run call
17:19:13 <msdubov> boris-42 Why not make it through a dictionary?
17:19:29 <boris-42> msdubov it could be
17:19:36 <boris-42> msdubov but dict + validation
17:19:55 <marcoemorais> boris-42: some keys in that should be required, like 'error'
17:20:05 <boris-42> marcoemorais yep
17:20:17 <boris-42> marcoemorais so I think it should be a subclass of dict
17:20:23 <msdubov> boris-42 Actually the results won't be too complicated to put them into a class, so I'd prefer the dict+validation combination
17:20:26 <boris-42> msdubov  marcoemorais that has schema validation
17:20:48 <hughsaunders> interesting, a self validating dict?
17:20:52 <boris-42> hughsaunders yep
17:21:12 <boris-42> hughsaunders short and powerful solution for standartization
17:21:14 <boris-42> =)
17:22:13 <msdubov> boris-42 I wonder is there any examples of using such dictionary subclasses?
17:22:18 <msdubov> Seems like a nice idea
17:22:26 <boris-42> msdubov IDK =)
17:22:44 <boris-42> so okay we could add this to our document with benchmark engine stuff
17:23:57 <boris-42> Okay and last step in context stuff
17:24:00 <boris-42> in ContextManager
17:24:20 <boris-42> that will understand what we have in context {} part and what context are required by benchmark
17:24:55 <boris-42> and then run all this stuff
17:25:05 <boris-42> and then run benchmark
17:25:29 <boris-42> so I think it's clear hughsaunders msdubov marcoemorais ?
17:25:58 <msdubov> boris-42 Yes
17:26:06 <hughsaunders> boris-42: I did wonder whether each scenario should specify which context it requires
17:26:34 <hughsaunders> maybe that is part of context config validation
17:26:46 <boris-42> hughsaunders yep it will be
17:27:07 <boris-42> hughsaunders so i mean you have benchmark like boot_runcommand
17:27:35 <boris-42> hughsaunders so it will have @context("security_group") and @context("key_pairs")
17:27:50 <msdubov> boris-42 And a couple of contexts "by default"?
17:27:53 <boris-42> hughsaunders this will be analyzed during validation (e.g. does input context works)
17:28:09 <boris-42> msdubov yep by default we have full cleanup context & user context
17:28:18 <hughsaunders> ah good, like the current validators for image and flavor
17:28:23 <boris-42> hughsaunders yep
17:28:49 <boris-42> hughsaunders so you may specify in benchmark that it requires specific context, but it don't block you to add any other in context config
17:30:01 <hughsaunders> cool
17:31:04 <boris-42> so let's move to next topic
17:31:35 <boris-42> #topic Benchmark engine optimization
17:31:57 <boris-42> marcoemorais could you share with current state of removing create_openstack_clients() ?
17:32:16 <rediskin> can we run all this stuff in separate threads, not processes?
17:32:49 <marcoemorais> boris-42: all but 1 unit test are passing
17:33:02 <marcoemorais> boris-42: I still need to handle the ssh key pair
17:33:24 <marcoemorais> boris-42: previously that was part of the osclients, but I don't think that is necessary any more
17:33:45 <hughsaunders> marcoemorais: I have a patch in review that adds context for that
17:33:50 <hughsaunders> so can be removed from osclients
17:34:04 <boris-42> marcoemorais yep I said before
17:34:12 <boris-42> marcoemorais that you should just remove ssh-keypair
17:34:30 <boris-42> marcoemorais it shouldn't be there and after hughsaunders it won't be there at all
17:34:34 <boris-42> hughsaunders share link pls
17:34:42 <marcoemorais> hughsaunders: https://review.openstack.org/#/c/78966/
17:35:17 <hughsaunders> yep
17:35:28 <hughsaunders> looks like I have some comments to address :)
17:35:55 <boris-42> hughsaunders not my=)
17:36:31 <boris-42> marcoemorais so if you remove ssh-keypair from create_openstack clients you will resolve all issues?
17:36:35 <boris-42> marcoemorais with unit tests
17:37:12 <boris-42> ?
17:37:49 <marcoemorais> boris-42: actually unit tests don't cover the ssh key pair (the one test that is failing is booting without nic, I think logic in test is faulty)
17:38:06 <marcoemorais> boris-42: key pair failure will happen at run time
17:38:06 <boris-42> marcoemorais oh that tests are all bad =(
17:38:19 <boris-42> marcoemorais we are going to have week of refactoring unit tests
17:38:27 <boris-42> marcoemorais after we finish all big refactorings
17:38:43 <kun_huang> +1 (sorry for late)
17:38:48 <boris-42> kun_huang np
17:38:58 <boris-42> so okay
17:39:11 <msdubov> boris-42 Btw can we discuss a bit what will be our guideline for UT refactoring?
17:39:22 <msdubov> Does anybody know a good guideline for that?
17:39:25 <boris-42> after "optimization topic"
17:39:27 <msdubov> Or shall we compose our own?
17:39:29 <boris-42> marcoemorais ^
17:39:32 <boris-42> msdubov ^
17:39:35 <boris-42> marcoemorais sorry*
17:39:46 <msdubov> ok
17:39:47 <boris-42> "Generic cleanup works forever"
17:40:06 <boris-42> We should add as well some kind of cleanup decorator that will accept list of affected projects
17:40:26 <boris-42> @cleanup("nova", "cinder")
17:40:27 <boris-42> or
17:40:53 <marcoemorais> boris-42: you mean like general purpose cleanup beyond what is already present in the Context?
17:41:26 <boris-42> marcoemorais I mean this https://github.com/stackforge/rally/blob/master/rally/benchmark/context/cleaner.py
17:41:33 <boris-42> marcoemorais works forever if you have 10k users
17:41:38 <boris-42> marcoemorais in keystone benchmark
17:42:02 <boris-42> marcoemorais and actually you don't need to cleanup nova and cinder if you are benchmarking keystone
17:42:21 <boris-42> marcoemorais so we should be able to specify in benchmark what cleanup to use
17:42:55 <marcoemorais> boris-42: is the issue that you deleting resources 1-by-1?
17:43:14 <boris-42> marcoemorais no issue is that you are deleting for every user
17:43:21 <boris-42> marcoemorais I mean if you have 10k users
17:43:45 <boris-42> marcoemorais you will make > 100k request to list resources in different services
17:44:34 <boris-42> marcoemorais 100k requests  even if you are able to handle 10 req/sec it will work 10k seconds ~2.5hrs
17:45:06 <boris-42> marcoemorais so you will wait 2.5hrs for nothing..
17:45:30 <boris-42> marcoemorais more services and more users => more time to wait
17:46:18 <boris-42> marcoemorais so we should be able to specify what services are affected by benchmark
17:46:27 <boris-42> marcoemorais and cleanup only them
17:47:10 <hughsaunders> boris-42: it would also be useful to be able to list resources across users/tenants so we could get a single list rather than requesting per user
17:47:15 <hughsaunders> not sure if thats possible though
17:47:33 <marcoemorais> boris-42 hughsaunders: so we need to look for openstack batch api
17:47:46 <boris-42> marcoemorais actually no
17:47:58 <boris-42> marcoemorais we know what resources are tenant based, what are user based
17:48:15 <boris-42> marcoemorais and in generic clenauper method that delete these resources we can chose what to do
17:48:25 <boris-42> marcoemorais delete one time per tenant or for every user
17:48:37 <marcoemorais> boris-42: to make sure I understand: you propose to delete resources by tenant instead of by user?
17:48:44 <boris-42> marcoemorais nope
17:49:11 <boris-42> marcoemorais if resources are common for users ( 1 pre tenant)
17:49:33 <boris-42> marcoemorais then list() and delete() them only one time
17:49:48 <hughsaunders> +1
17:49:55 <boris-42> marcoemorais now we are doing this operation for every users
17:50:07 <boris-42> marcoemorais so I mean every user will list() but only one will actually delete()
17:50:31 <boris-42> marcoemorais in case when users have own resource (e.g. VMs)
17:50:43 <boris-42> marcoemorais then we will still make list() / delete() for every user
17:51:24 <marcoemorais> boris-42: we could compile a set of resources, then iterate and delete from set instead of from user, but I still think you need to do some batching
17:52:01 <boris-42> marcoemorais 1) there is no batch delete in openstack API
17:52:13 <boris-42> 2) it's okay to delete from users user resource
17:52:51 <boris-42> I think hughsaunders proposal + clean only service that you use will resolve all issues
17:53:02 <boris-42> at least most
17:54:07 <hughsaunders> boris-42: we could store/generate expected resource names, then delete() only?
17:54:13 <marcoemorais> boris-42 hughsaunders: what about host aggregates?
17:54:42 <hughsaunders> marcoemorais: ?
17:54:51 <kun_huang> marcoemorais that is a concept only for nova?
17:56:38 <boris-42> hughsaunders it will actually requires changes in benchmark framework
17:56:44 <boris-42> hughsaunders for scenarios
17:57:24 <marcoemorais> I was thinking that host aggregates could let you manage the entire hostgroup http://api.openstack.org/api-ref-compute-ext.html#ext-os-aggregates
17:58:20 <hughsaunders> marcoemorais: thats for managing nodes rather than instances though?
17:58:37 <boris-42> hughsaunders marcoemorais  so we have to finish todays meeting
17:58:59 <marcoemorais> hughsaunders: got it for hypervisor, not guest
17:59:01 <boris-42> So any new ideas IRC chat or that document
17:59:15 <boris-42> #endmeeting