17:01:54 <boris-42> #startmeeting rally
17:01:59 <openstack> Meeting started Tue Apr 22 17:01:54 2014 UTC and is due to finish in 60 minutes.  The chair is boris-42. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:02 <openstack> The meeting name has been set to 'rally'
17:02:19 <boris-42> marcoemorais meteorfox msdubov ping
17:02:22 <msdubov> boris-42, hi
17:02:27 <meteorfox> boris-42: hi
17:02:29 <boris-42> hughsaunders ping
17:02:42 <marcoemorais> boris-42: hi
17:03:06 <boris-42> kun_huang ping
17:03:55 <boris-42> rediskin hi
17:03:58 <rediskin> hi
17:04:07 <kun_huang> boris-42 hi
17:04:13 <kun_huang> let's meeting :)
17:04:33 * boris-42 kun_huang is ready to have some meeting
17:04:38 <boris-42> ^_^
17:05:21 <boris-42> so let's start
17:05:22 <kun_huang> boris-42 chen is pregnant, so it is a little hard for her being online at this time
17:05:38 <boris-42> kun_huang oO nice =)
17:06:01 <boris-42> kun_huang ok it's good that you are here so you'll be able to share info
17:06:03 <rediskin> congratulations :)
17:06:11 <boris-42> okay lets start=)
17:06:40 <boris-42> #topic PERFORMANCE GATES FOREVER!! =)
17:06:51 <boris-42> rediskin could you share with others
17:06:59 <boris-42> rediskin with your graet work =)
17:07:30 <rediskin> now we can easyly add rally-jobs. it is now as easy as python-jobs
17:07:46 <rediskin> but
17:08:00 <rediskin> we have to make scenarios work out-of-the-box
17:08:14 <boris-42> rediskin i think it's only related to nova benchmarks
17:08:15 <rediskin> without editing image_id and other stuff
17:08:22 <boris-42> marcoemorais should finish his patch soon
17:08:25 <boris-42> marcoemorais ^
17:08:29 <marcoemorais> boris-42: ack
17:08:39 <marcoemorais> boris-42: saw your comments on review
17:08:52 <boris-42> marcoemorais we will discuss it after this topic
17:08:53 <boris-42> =)
17:09:10 <boris-42> So for others that are not related to gates
17:09:14 <boris-42> there is new gate in rally
17:09:41 <boris-42> it's called check-rally-dsvm-rally
17:10:05 <boris-42> it will run 1 task with bunch of benchmarks
17:10:13 <rediskin> this job just run tests_ci/scenarios/rally.yaml
17:10:14 <boris-42> and upload html with results
17:10:31 <rediskin> so we need to add to this yaml as many scenarios as possible %)
17:10:35 <meteorfox> boris-42: where in the tree is the code for this gate? ;)
17:10:36 <boris-42> yep so we will put in rally.yaml all benchmarks
17:10:51 <rediskin> https://review.openstack.org/#/c/86945/
17:10:52 <boris-42> meteorfox it's not in tree yet
17:11:01 <rediskin> currently gates seems broken
17:11:07 <meteorfox> ok
17:11:44 <boris-42> meteorfox here is successful run  http://logs.openstack.org/45/86945/15/check/check-rally-dsvm-rally/780bb2e/logs/devstack-gate-post-test-hook.txt.gz
17:12:01 <boris-42> meteorfox on bottom you can see table with result
17:12:04 <meteorfox> boris-42: thanks
17:12:13 <boris-42> but it's now a bit complicated
17:12:32 <boris-42> in future after this patch https://review.openstack.org/#/c/89562/
17:12:39 <boris-42> we will get html plots=)
17:12:51 <meteorfox> awesome
17:12:55 <boris-42> does anybody has any question?)
17:13:12 <marcoemorais> agree this is pretty nice
17:13:24 <kun_huang> love it
17:13:36 <msdubov> boris-42, so the output is equivalent as if we run it with the -vd option?
17:14:06 <boris-42> msdubov link was log*
17:14:06 <rediskin> msdubov: https://review.openstack.org/#/c/86945/17/tests_ci/rally-gate.sh
17:14:14 <boris-42> msdubov that allows to troubleshot
17:14:23 <boris-42> msdubov but it will be as well plots
17:14:37 <msdubov> boris-42, rediskin Yep I see
17:15:06 <boris-42> if morder or clarkb merge our patch https://review.openstack.org/#/c/89562/ =)
17:15:17 <boris-42> so okay let's just move to next topic
17:15:29 <rediskin> why merge out? merge in =)
17:15:35 <boris-42> our*
17:15:38 <boris-42> not out lol
17:15:41 <rediskin> %)
17:15:44 <boris-42> ><
17:15:56 <boris-42> #topic pre processing of args
17:16:04 <meteorfox> boris-42: will they gate have the concept of thresholds, say, like tolerating that the certain metric is +/- 10%, and have absolute limits?
17:16:20 <boris-42> meteorfox I think in half year we will be able to make something like that
17:16:25 <boris-42> meteorfox or year
17:16:26 <boris-42> =)
17:16:38 <meteorfox> boris-42: cool
17:16:45 <boris-42> meteorfox it's hard cause node pool has absolutely different nodes
17:16:59 <boris-42> and normalization in benchmarking is quite complex stuff
17:17:03 <boris-42> marcoemorais ping
17:17:09 <boris-42> marcoemorais your turn
17:17:25 <marcoemorais> boris-42: so open question is to add types
17:17:43 <marcoemorais> boris-42: types will be more generic way to apply transformation
17:17:44 <boris-42> #link https://review.openstack.org/#/c/86116/
17:18:23 <boris-42> marcoemorais so I think that Types will be simpler to organize code
17:18:27 <marcoemorais> boris-42: idea of types is to create a class which has a transform method
17:18:30 <boris-42> and it will be simpler to understand
17:19:24 <marcoemorais> boris-42: for now the only 2 Types are Image and Flavor
17:19:25 <meteorfox> marcoemorais: sorry, I still no too familiar with the code base, what kind of Types are we talking about? benchmark types, results types?
17:19:45 <marcoemorais> meteorfox: Types are objects inside of our scenario config
17:20:07 <boris-42> meteorfox we are speaking about input arguments
17:20:19 <meteorfox> gotcha
17:20:25 <rediskin> kinda OpenStack objects types, like Image, Snapshot, Volume, Network?
17:20:29 <boris-42> meteorfox https://github.com/stackforge/rally/blob/master/doc/samples/tasks/nova/boot-and-delete.json#L5-L6
17:20:45 <marcoemorais> meteorfox: given a Type — you can do a transform on that type to normalize it to be consistent with what the scenario expects
17:20:47 <boris-42> so we would like to be able to specify not only image_id
17:21:06 <meteorfox> ok, now I understand, thanks
17:21:08 <boris-42> we would like to specify image: {"id": some_id} or image: {"name": some_name}
17:21:17 <boris-42> image: {"reg": some_reg}
17:21:32 <boris-42> and now marcoemorais made patch that allows us to do any transformations
17:21:43 <boris-42> cause we are not able to use image name inside benchmark
17:22:09 <boris-42> meteorfox cause it will be huge overhead inside benchmark (e.g. if you would like jus to show() image) overhead will be almost x2
17:22:21 <boris-42> so there are 2 ways to resolve issue
17:22:29 <boris-42> more abstract
17:22:35 <boris-42> e.g. transformations
17:22:52 <boris-42> that can transfer input kwargs in any way
17:22:59 <boris-42> or "types"
17:23:09 <boris-42> that can transfer only one argument to something
17:23:14 <boris-42> meteorfox rediskin ^
17:23:46 <boris-42> As at this moment we have only Image and Flavor types and I don't think that we will have such multi arg transformations
17:23:55 <boris-42> it  is better to keep things simpler
17:23:57 <boris-42> and use just types
17:24:06 <meteorfox> ok
17:24:23 <marcoemorais> boris-42 meteorfox where in the source tree should I put types under rally/tree/master/rally/benchmark/types?
17:24:39 <boris-42> meteorfox yep
17:24:42 <msdubov> boris-42, So having types as you suggested won't make it obligatory to specify them in the docstring as well, if I understand that correctly
17:24:43 <boris-42> marcoemorais yep
17:25:00 <boris-42> msdubov probably we can reuse them
17:25:10 <boris-42> msdubov for it as well
17:27:20 <boris-42> so marcoemorais  ok?
17:27:48 <marcoemorais> boris-42 meteorfox: yes, one more small issue — for regex, we will use "first match" or fail on ambiguity?
17:28:16 <boris-42> marcoemorais validation stuff should fail
17:28:23 <boris-42> marcoemorais and say multiply images possible
17:28:32 <marcoemorais> boris-42 ok
17:28:37 <boris-42> ,marcoemorais as well for name
17:28:51 <boris-42> so next topic?
17:29:21 <boris-42> #topic tempest & rally integration
17:29:35 <boris-42> sdague probably will be interesting for you^
17:29:56 <boris-42> So actually I will just make some update
17:30:20 <boris-42> So there are 2 parts:
17:30:28 <boris-42> 1) verification part
17:30:35 <boris-42> 2) benhmarking part
17:30:40 <boris-42> In verification part
17:31:21 <boris-42> We have: 1) tempest conf generation 2) support of work with multiply clouds 3) parsing results to json and storing in rally db
17:31:41 <boris-42> 4) and base operations to work with results, e..g show/detailed/results commands
17:31:49 <boris-42> About benchmarking part
17:32:09 <boris-42> The base patch that adds possibility to run any tempest test as benchmark was merged
17:32:20 <boris-42> http://pavlovic.me/rally/tempest.html <- here is the sample of output
17:32:52 <boris-42> so actually there will be couple of improvements
17:33:29 <boris-42> A) we will mesure not only total duration (setUp, runTest, tearDown) but as well just of runTest in atomic actions
17:33:54 <boris-42> B) we will support running different tests simultaneously (this will be super usefully for testing races in tempest tests)
17:34:17 <boris-42> Actually here is the patch
17:34:19 <boris-42> https://review.openstack.org/#/c/86836/
17:34:30 <boris-42> marcoemorais meteorfox msdubov hughsaunders kun_huang rediskin any questions ^ ?
17:35:08 <msdubov> msdubov, nope
17:35:12 <msdubov> boris-42, nope
17:35:45 <marcoemorais> boris-42: high level question, what is distinction btwn use of tempest for benchmarking and use of rally scenarios for benchmarking?
17:36:05 <meteorfox> boris-42: +1 marcoemorais question
17:36:20 <boris-42> marcoemorais rally benchmarks are more benchmarks lol=)
17:36:52 <boris-42> e.g. we don't have mess of setUp/tearDown and running benchmark
17:37:02 <boris-42> and we have powerful mechanism of context classes
17:37:18 <boris-42> + we are able to collect much more results
17:37:40 <boris-42> using atomic actions and in future I'll add collecting of all python clients calls (equivalent to http calls)
17:37:49 <boris-42> that is unfortunately impossible with tempest unit tests
17:37:49 <marcoemorais> boris-42: I mean here, until rally, we did look at tempest (functional test) output as mini-benchmark, but it isn't as powerful as what we are doing with rally
17:38:19 <boris-42> marcoemorais meteorfox ^
17:39:16 <boris-42> marcoemorais you disagree with this?)
17:39:35 <meteorfox> boris-42: so why is tempest integration needed, I think that's my question. :)
17:39:46 <boris-42> meteorfox the answer is simple
17:39:50 <kun_huang> boris-42 I think marcoemorais don't know this intrgration
17:40:14 <boris-42> meteorfox 1) tempest has a bunch of validation tests (every day more)
17:40:30 <boris-42> so running it to validate cloud is I think most proper way
17:40:50 <boris-42> 2) why we need benchmarks from tempest tests?
17:41:00 <boris-42> there are couple of things
17:41:11 <boris-42> 1) helping improving tempest (removing race conditions)
17:41:21 <boris-42> 2) simple way to repeat race conditions from gates
17:41:44 <boris-42> 3) if you have something already implemented in tempest (some complex scenario) that you would like to run in rally you don't need to port it
17:41:52 <boris-42> meteorfox ^
17:42:27 <boris-42> meteorfox so as it was not hard to integrate tempest in Rally => why not?)
17:43:02 <boris-42> meteorfox I mean benchmarking part* (verification part was quite tricky)
17:43:21 <meteorfox> boris-42: ok, I see. I'm concerned with new users, won't they be confused and prefer to write the benchmarks in tempest, where instead they should use Rally scenarios?
17:44:00 <kun_huang> btw reproduce race conditions is to use 'rally task start <path-to-tempest-scen>' ?
17:45:12 <meteorfox> boris-42: I see the value of "perf testing" tempest scenario, and testing for race conditions, but not so much as for a meaningful workload on the whole system.
17:45:40 <meteorfox> boris-42: but I might not be understanding all of it.
17:45:55 <boris-42> meteorfox if user would like to use tempest to benchmark cloud we should let him
17:46:05 <boris-42> meteorfox it's his wiliness imho
17:46:15 <boris-42> meteorfox our goal is to make it simple as possible
17:46:39 <meteorfox> boris-42: cool, ok.
17:46:47 <boris-42> meteorfox why we should decide for others?)
17:47:19 <boris-42> meteorfox my goal is to make it simple and it is all
17:47:37 <meteorfox> boris-42: true, I agree. :)
17:47:40 <boris-42> meteorfox if it will be simple -> it will be popular -> many developers will start benchmark -> we will fix openstack
17:48:07 <boris-42> -> I will resolve my current task -> my boss happy -> me happy=)
17:48:13 <meteorfox> boris-42: lol
17:48:42 <boris-42> Okay let's move to next topic
17:48:52 <boris-42> #topic new output table
17:48:59 <boris-42> msdubov could you share with your ideas
17:49:02 <boris-42> about new output
17:49:15 <msdubov> boris-42 Yep
17:49:34 <msdubov> So basically the idea is to make the CLI output less messy
17:49:48 <msdubov> A reasonable first step seemingly would be to merge the 2 output tables
17:49:57 <msdubov> Namely the result table and the atomic action table
17:50:05 <msdubov> Here are 2 possible ways to do that: http://paste.openstack.org/show/76661/
17:50:33 <msdubov> Any ideas on what looks better?
17:50:45 <meteorfox> msdubov: still loading for me :/
17:51:05 <msdubov> Note that here I removed the "success/total" column
17:51:05 <kun_huang> msdubov opening it
17:51:12 <msdubov> And moved it to the parens
17:51:17 <msdubov> in "count"
17:51:42 <msdubov> Hopefully it is still clear enough, what "(0.7)" means there
17:51:58 <meteorfox> did it open for anyone?
17:52:21 <kun_huang> meteorfox html title is loaded, but content not
17:52:23 <msdubov> meteorfox,  Seems that it hangs :(
17:52:27 <boris-42> msdubov you gave us some strange link
17:52:27 <msdubov> An alternative link: http://pastebin.com/ps3bDrM9
17:52:42 <marcoemorais> meteorfox: open for me, either table seems fine, but maybe the first is a little more typical (eg total at bottom rather than top)
17:53:02 <msdubov> marcoemorais, Yes, but the last row is also the most important one...
17:53:14 <msdubov> That's why I proposed the second table
17:53:18 <boris-42> +1
17:53:21 <boris-42> I lie first one
17:53:23 <meteorfox> msdubov: I agree with marcoemorais
17:53:37 <marcoemorais> msdubov: the % success (7/10) is still a problem for me, but I don't have an idea how to improve that
17:53:40 <kun_huang> +1 for first one
17:53:53 <boris-42> yep
17:54:03 <boris-42> so everybody decided to use first one
17:54:06 <kun_huang> but which is better, 0.7 and 70%
17:54:20 <msdubov> Ok, thanks. Any ideas on what to do with success % ?
17:54:22 <boris-42> kebray 70%
17:54:28 <marcoemorais> msdubov: unless you have a row that is titled num failed or something that just has 3 in 'count' column and 'NA' in all other colums
17:54:28 <boris-42> msdubov 70%
17:54:55 <boris-42> msdubov btw do we need count at all?
17:54:56 <meteorfox> msdubov: for the order of the columns, though, wouldn't avg, min, max, 90%ile, 95%ile be a better order?
17:55:20 <boris-42> meteorfox yep agree min/agv/max is better then max/avg/min
17:55:27 <msdubov> msdubov Not sure, the order also requires discussion
17:55:30 <msdubov> boris-42, Ok!
17:55:44 <boris-42> about count it should be last column
17:55:54 <msdubov> boris-42, But meteorfox actually suggested avg/min/max
17:55:54 <boris-42> I think at least
17:56:24 <marcoemorais> msdubov: btw: enhancement would be to allow user to specify a % (some users will have a specific SLA to meet eg 98.5% of all requests <= y response time and will want rally to report that
17:56:29 <boris-42> I think min/avg/max is probably better cause it raises from left tto right
17:56:39 <msdubov> boris-42, Agree
17:56:45 <meteorfox> msdubov: I think in order in order is better, what boris-42 suggested. min/avg/max
17:56:47 <marcoemorais> boris-42: agree with count as last colum
17:57:01 <msdubov> marcoemorais, Yep, I think we'll add custom percetiles soon
17:57:03 <meteorfox> msdubov: what about %error?
17:57:19 <boris-42> yep % instead of 0.7 will be better imho
17:57:29 <msdubov> meteorfox, boris-42, Perhaps than we can make a separate column for success %?
17:57:37 <msdubov> It will have data only for the "total" row
17:57:39 <boris-42> yep
17:57:43 <boris-42> so to columns
17:57:48 <boris-42> count | success
17:57:49 <boris-42> ?
17:58:04 <msdubov> boris-42, How do you imagine that?
17:58:13 <boris-42> 2 columns*
17:58:15 <boris-42> two*
17:58:18 <msdubov> boris-42, Yep
17:58:23 <kun_huang> boris-42 success (70%) should be the first column or last?
17:58:38 <msdubov> And success % only for the last row ("total")?
17:58:53 <boris-42> msdubov it can be done for all columns
17:58:55 <boris-42> =)
17:59:06 <boris-42> msdubov okay we can discuss this in rally caht
17:59:10 <boris-42> cause time is up
17:59:12 <msdubov> boris-42, Yep
17:59:22 <boris-42> #endmeeting