17:07:02 <boris-42> #startmeeting Rally
17:07:03 <openstack> Meeting started Tue Oct 28 17:07:02 2014 UTC and is due to finish in 60 minutes.  The chair is boris-42. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:07:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:07:06 <openstack> The meeting name has been set to 'rally'
17:07:38 <boris-42> #topic discuss DOCstring of benchmark scenarios + possible ways to auto generate doc samples
17:07:48 <boris-42> amaretskiy as you were interested in this
17:07:51 <rook> in
17:07:54 <boris-42> rook hi there
17:07:56 <rook> hey
17:08:17 <boris-42> amaretskiy so what was your idea?
17:08:28 <amaretskiy> my idea is save samples in class properties or in method properties, not in docstrings
17:08:51 <amaretskiy> advantages:
17:08:52 <amaretskiy> * no need to parse
17:08:52 <amaretskiy> * easy to check
17:08:52 <amaretskiy> * ready for using as templates
17:09:20 <boris-42> msdubov ^
17:10:00 <msdubov> amaretskiy, I definitely need some clarifications on this
17:10:15 <msdubov> amaretskiy, How would you store it - as a whole string or split?
17:10:29 <msdubov> amaretskiy, It would be great if you'd write up a small sample class
17:10:35 <msdubov> amaretskiy, To illustrate your idea
17:10:38 <tnurlygayanov> amaretskiy why we need this? you suggest to save 'help' messages and 'examples' in properties/methods instead of doc strings?
17:10:51 <msdubov> amaretskiy, As of now, it's not obvious for me why this is going to be easier to parse
17:10:57 <tnurlygayanov> doc string it is the property too.
17:11:01 <msdubov> tnurlygayanov, +1
17:11:53 <amaretskiy> for example:
17:11:53 <amaretskiy> class Foo(...):
17:11:53 <amaretskiy> """ ... """
17:11:53 <amaretskiy> SAMPLE = """{ JSON }"""
17:12:03 <amaretskiy> no need to parse
17:12:20 <amaretskiy> easy to check/validate/etc...
17:12:35 <msdubov> amaretskiy, Okay so we leave docstrings as is and just add these SAMPLE attributes?
17:12:57 <msdubov> amaretskiy, Okay, but SAMPLE is unique for the whole class
17:12:58 <amaretskiy> yes
17:13:09 <msdubov> amaretskiy, How would you store samples for each scenario?
17:13:24 <amaretskiy> if we need to add a sample for method, we can use 2 ways:
17:13:34 <boris-42> amaretskiy NO I dislike SAMPLE
17:13:55 <boris-42> amaretskiy we were talking about decorator
17:14:04 <boris-42> amaretskiy or adding to function field directly
17:14:09 <boris-42> amaretskiy as far as I remember
17:14:41 <amaretskiy> class FooScenario(...):
17:14:41 <amaretskiy> def barScenario(...): ...
17:14:41 <amaretskiy> barScenario_sample = """{JSON}"""  # 1st way
17:14:41 <amaretskiy> barScenario.SAMPLE = """{JSON}""" # 2nd way
17:15:46 <amaretskiy> yes, decorator is ok, but it seems to have extraordinary size
17:16:00 <amaretskiy> so. 3rd way - decorator
17:16:05 <amaretskiy> not docstring
17:16:40 <amaretskiy> as for me I would prefer 2nd way
17:16:53 <amaretskiy> same property name
17:16:59 <amaretskiy> for all required objects
17:17:31 <msdubov> amaretskiy, I'm a bit worried about the code getting very huge... (SLOC) Although it will be of course easier for anyone to quickly understand how a particular scenario should be configured
17:17:59 <msdubov> amaretskiy, boris-42 Maybe we could store only sample arguments, without the "runner"/"context" section?
17:18:08 <msdubov> amaretskiy, To make it a bit shorter
17:18:22 <msdubov> and then quickly compose a valid JSON sample out of that
17:18:29 <boris-42> msdubov but what about cases when it makes sense?
17:18:38 <amaretskiy> not sure, everything should be explicit
17:18:38 <msdubov> boris-42, You're right
17:19:06 <msdubov> boris-42, amaretskiy I had an idea of storing the JSON samples in doc/sample/tasks, but adding a link to those files in the docstrings
17:19:12 <amaretskiy> not sure - means - include complete data
17:19:17 <boris-42> msdubov wtf
17:19:22 <msdubov> something like :sample: doc/sample/tasks
17:19:24 <boris-42> msdubov no idea was to generate task samples
17:19:39 <msdubov> boris-42, Well we still can generate them1
17:19:40 <msdubov> !
17:19:43 <msdubov> in files
17:19:50 <boris-42> msdubov hm
17:20:00 <msdubov> boris-42, I believe that JSON samples in files are easier to navigate
17:20:06 <msdubov> than those in code
17:20:17 <msdubov> because they are stored in a neat compact way
17:20:23 <boris-42> msdubov there will be in files...
17:20:30 <boris-42> msdubov I am talking about removing them from files
17:20:36 <msdubov> boris-42, ok
17:20:38 <boris-42> msdubov i am talking about generating them from code
17:20:39 <amaretskiy> so, if we want to generate JSON, then: 1) parsing source is bad 2) templating is good - so docstrings are not good for that
17:21:11 <boris-42> amaretskiy so basically having one @base.sampe() is good choose
17:21:14 <boris-42> choice*
17:21:30 <boris-42> for "rally info" and generation of doc/samples/tasks/
17:21:51 <amaretskiy> yes, decorator is good :)
17:22:54 <msdubov> amaretskiy, boris-42 agree
17:23:40 <boris-42> Okay so then I think
17:23:47 <boris-42> we will add @sample()
17:23:55 <boris-42> btw it can be reused for Context/Runner/Scenarios
17:24:02 <boris-42> and SLA stuff
17:24:04 <tnurlygayanov> we want to generate samples based on parameters of functions/methods?
17:24:09 <boris-42> so I think it should be in common place
17:24:15 <boris-42> tnurlygayanov nope we don't want
17:24:18 <boris-42> tnurlygayanov it's impossible
17:25:45 <tnurlygayanov> ok, how we will generate samples? we need to stroe some information about each benchmark/scenario and generate samples based on this info
17:26:07 <boris-42> tnurlygayanov ...
17:26:09 <amaretskiy> property
17:26:14 <boris-42> tnurlygayanov whole discussion above was about that
17:26:24 <tnurlygayanov> ok
17:26:25 <boris-42> tnurlygayanov we will have @sample
17:26:35 <boris-42> tnurlygayanov that adds property to class/method in different cases
17:26:48 <boris-42> tnurlygayanov and that can be reused by SLA/Context/Runners/Scenarios
17:27:52 <tnurlygayanov> boris-42  ok, now it is clear :) thank you.
17:31:34 <boris-42> Okay let's move
17:31:45 <boris-42> #topic Rally info updates and future steps
17:31:47 <boris-42> msdubov go on
17:32:18 <rook> Future : way to disable cleanup  =)
17:32:29 <rook> kambiz is not here to talk more to that :D
17:32:35 <msdubov> So there are two major updates on this, both in one patch: https://review.openstack.org/#/c/126421/
17:32:39 <rook> however, i agree boris-42 it is a "nice to have"
17:32:50 <boris-42> rook I mean it will be
17:32:54 <boris-42> rook it's already in our road map
17:33:07 <boris-42> rook https://github.com/stackforge/rally/blob/master/doc/feature_request/persistence_benchmark_env.rst
17:33:15 <msdubov> 1. Typing just "rally info" or "rally info list" results in printing out a list of scenarios/engines/... that can be queried by rally info
17:33:16 <boris-42> rook this is our roadmap
17:33:27 <msdubov> 2. "rally info find <...>" now has more info about scenario groups
17:33:28 <boris-42> rook but it will take some amount of time to implement it
17:33:48 <msdubov> e.g. "rally info find NovaServers" -> you have info about what benchmark scenarios are there
17:33:53 <msdubov> in a neat table-like output
17:34:16 <msdubov> Also there is a test that checks that Rally is covered with docstrings: https://review.openstack.org/127192
17:34:34 <msdubov> As soon as we make it pass, we can be sure that "rally info" works well too :)
17:35:47 <boris-42> msdubov so
17:35:59 <boris-42> msdubov what about SLA plugins?
17:36:33 <msdubov> boris-42, What do you mean?
17:37:00 <boris-42> msdubov I mean SLA plugins
17:37:13 <boris-42> msdubov can I find them using rally info
17:37:45 <boris-42> msdubov this https://github.com/stackforge/rally/blob/master/rally/benchmark/sla/base.py#L85-L135
17:38:16 <msdubov> boris-42, So say you type "rally info find max_seconds_per_iteration"
17:38:23 <msdubov> boris-42, What do you expect to get?
17:38:27 <msdubov> boris-42, Config sample?
17:38:29 <boris-42> msdubov infromation about it
17:38:42 <boris-42> msdubov I would like to type rally info find sla
17:38:48 <boris-42> msdubov and find all sla's
17:38:57 <msdubov> boris-42, Ah I see
17:39:03 <boris-42> then type rally info find max_seconds_per_iteration
17:39:08 <boris-42> and see information about it more detailed
17:39:21 <boris-42> and probably in future samples
17:39:27 <boris-42> when we finish @sample stuff
17:39:30 <msdubov> boris-42, that actually doesn't work for scenarios/engines as well "rally info find engines" will be an invalid input
17:39:35 <msdubov> boris-42, I want fix that
17:39:41 <msdubov> boris-42, But as for SLA
17:39:48 <boris-42> msdubov ya
17:39:59 <boris-42> msdubov as well as a end user if you type rally info NovaServers
17:40:04 <boris-42> msdubov output is quite misleading
17:40:17 <msdubov> boris-42, Why?
17:40:29 <msdubov> it is just a list of scenarios with one-line description
17:40:32 <boris-42> msdubov you are getting output of "rally info"
17:40:32 <msdubov> quite compact
17:40:38 <boris-42> msdubov read one more time what I wrtoe
17:40:47 <boris-42> msdubov "rally info NovaServers"
17:40:48 <msdubov> boris-42, Of course you are getting the output of rally info
17:40:56 <boris-42> msdubov that sux
17:40:59 <msdubov> boris-42, We discussed that multiple times
17:41:07 <msdubov> boris-42, I even made a patch that fixed this...
17:41:16 <msdubov> boris-42, But it involved some dirty code
17:41:16 <boris-42> msdubov you should specify
17:41:21 <boris-42> msdubov Mike
17:41:36 <boris-42> msdubov when I type "rally info NovaServers" I would like to see ouptut
17:41:51 <boris-42> "Probably you thought to specify rally info find NovaServers"
17:42:15 <msdubov> boris-42, You forgot what I told you about the command parsers...
17:42:22 <msdubov> boris-42, They don't support such stuff
17:42:32 <boris-42> msdubov you should find the way
17:42:33 <msdubov> boris-42, I implemented that "default" method
17:42:34 <boris-42> msdubov =)
17:42:48 <msdubov> boris-42, The only way is to remove common cliutils/argparse
17:42:50 <boris-42> msdubov you should make one more IF conditions in it
17:42:57 <boris-42> msdubov in default method
17:43:14 <boris-42> msdubov like if there is third argument
17:43:18 <msdubov> boris-42, There is no default method as of now, I removed it
17:43:19 <boris-42> msdubov do helper
17:43:24 <msdubov> boris-42, Take a look at the code
17:43:40 <msdubov> boris-42, it just has a patched docstring
17:43:41 <boris-42> msdubov I don't care actually about code
17:43:48 <boris-42> msdubov I am talking about result
17:43:55 <boris-42> msdubov result is misleading for end users
17:43:59 <boris-42> msdubov it is not what we want
17:44:08 <msdubov> boris-42, Well then I don't understand you
17:44:14 <boris-42> msdubov I just said
17:44:27 <msdubov> boris-42, You suggested removing that default method and leaving this "rally info find ..." syntax as is
17:44:34 <msdubov> boris-42, And now ou require the opposite
17:45:05 <boris-42> msdubov one more time
17:45:08 <boris-42> rally info
17:45:12 <boris-42> <PRint pretty help>
17:45:18 <boris-42> rally info find NovaServers
17:45:24 <boris-42> <print info about NovaServers>
17:45:32 <boris-42> rally info NovaServers
17:45:47 <boris-42> <say you probably think that run rally info find >
17:46:01 <boris-42> ^ msdubov  what is unclear?
17:47:17 <boris-42> msdubov okay just keep it in mind
17:47:22 <boris-42> msdubov and let's move
17:47:51 <boris-42> #topic cleanup
17:47:54 <boris-42> rook =)
17:48:08 <boris-42> Okay I am working on pretty big patch that makes our cleanup system more production ready
17:48:22 <boris-42> https://review.openstack.org/#/c/129060/
17:48:30 <boris-42> I am going to finish it maybe today =)
17:48:58 <rook> boris-42++
17:49:00 <boris-42> So now cleanuping is quite simple, plugable, safe and parallel
17:49:20 <boris-42> only thing that you need is to create sublass
17:49:23 <boris-42> https://review.openstack.org/#/c/129060/16/rally/benchmark/context/cleanup/projects_resources.py
17:49:24 <boris-42> for your resoruce
17:49:45 <boris-42> everything else will be covered by cleanup stuff
17:50:01 <boris-42> without end user (dev) attention which is nice=)
17:50:10 <boris-42> rook about turning off cleanup
17:50:21 <boris-42> rook actually making it as a separated command
17:50:33 <rook> or a option in the yaml/json
17:51:29 <boris-42> rook ya maybe)
17:51:45 <boris-42> rook in any case we will find some solution
17:51:51 <boris-42> rook after few more patches=)
17:51:55 <boris-42> rook and refactorings
17:52:33 <boris-42> #topic Docker imags
17:52:50 <boris-42> So we will have job that will create docker image on every merged patch
17:53:08 <boris-42> https://review.openstack.org/#/c/131389/
17:53:14 <boris-42> which will simplify life of end users a lot
17:53:21 <boris-42> cause they won't need to install it=)
17:53:22 <boris-42> by hands
17:55:39 <boris-42> so
17:55:59 <boris-42> #topic Imprvoments in functional testing
17:56:22 <boris-42> Andrey Kurilin add patch that runs "rally verify" on every patch
17:56:30 <boris-42> And checks that at least half of tests passed
17:56:52 <boris-42> So at least we have some basic function testing for rally verify commands, that won't allow us to break everything
17:57:00 <boris-42> #topic New Rally report
17:58:00 <boris-42> amaretskiy is continue work on this part
17:58:21 <boris-42> so a lot of new improments https://review.openstack.org/#/c/128991/ that I hope we will merge soon
17:58:30 <boris-42> so we need to finish meeting
17:58:36 <boris-42> Glad to see you guys
17:58:39 <boris-42> #endmeeting