17:05:00 <msdubov> #startmeeting rally
17:05:01 <openstack> Meeting started Tue Apr 29 17:05:00 2014 UTC and is due to finish in 60 minutes.  The chair is msdubov. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:05:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:05:04 <openstack> The meeting name has been set to 'rally'
17:05:11 <msdubov> So let's start out meeting
17:05:43 <msdubov> #topic Resources descriptions in scenario configs
17:06:15 <msdubov> marcoemorais, Could you please tell a bit about your recent results you published in https://review.openstack.org/#/c/86116/
17:06:49 <marcoemorais> msdubov: patch set is no longer wip
17:07:01 <marcoemorais> msdubov: biggest issue is the redundant processing in validation
17:07:15 <msdubov> marcoemorais, the thing noticed by hughsaunders and me?
17:07:26 <marcoemorais> msdubov: I earlier raised this to boris-42 so he needs to re-confirm that is what he wants
17:07:30 <marcoemorais> msdubov: yes
17:07:58 <marcoemorais> msdubov: if necessary we can re-order things, but he didn't want that
17:08:08 <msdubov> marcoemorais, hughsaunders also pointed out that even with this redundant processing the benchmarcks with 'image_exists' don't work
17:08:14 <msdubov> marcoemorais, yep, I see
17:09:01 <msdubov> marcoemorais, Just saw your recent patch update
17:09:04 <marcoemorais> msdubov hughsaunders let me take a look at that particular issue — I test with a handful of nova scenarios
17:09:07 <msdubov> marcoemorais, Does it fix that issue?
17:09:49 <marcoemorais> msdubov: I will take a closer look at image_exists
17:10:04 <msdubov> marcoemorais, Ok
17:10:16 <msdubov> marcoemorais, So basically it seems to me that here
17:10:17 <msdubov> https://github.com/stackforge/rally/blob/master/rally/orchestrator/api.py#L102-L104
17:10:18 <marcoemorais> msdubov: anyway, once this patch is merged I can add the nova scenarios to the rally conf used for gating ci https://review.openstack.org/#/c/90897/
17:10:34 <msdubov> marcoemorais, it's possible to add one line to perform args processing before validation
17:10:43 <msdubov> marcoemorais, this wouldn't make the code too dirty
17:10:55 <marcoemorais> msdubov: yes I proposed that to boris-42 but he didn't want it that way
17:10:56 <msdubov> marcoemorais, but anyway that should be discussed with participation of boris-42
17:11:08 <msdubov> marcoemorais, What was his argument?
17:11:22 <marcoemorais> msdubov: he said that validation was going to be refactored
17:12:15 <msdubov> marcoemorais, Would it be then possible to put args processing somewhere in benchmark_engine.validate() (L103) instead of benchmark_engine.run() (L104)?
17:12:23 <msdubov> marcoemorais, just as a suggestion
17:12:52 <marcoemorais> msdubov: well I have to rewrite the args passed to scenario, so it naturally fits in current location in code
17:13:08 <msdubov> marcoemorais, also true
17:13:18 <msdubov> hughsaunders, any opinions? ^^
17:17:08 <msdubov> So ok let's discuss that tomorrow, since we have several other topics to cover now
17:18:40 <msdubov> #topic Ceilometer benchmark scenarios
17:18:57 <msdubov> aswadrangnekar, Could you please tell others a bit about your work?
17:19:06 <aswadrangnekar> msdubov: sure
17:19:43 <aswadrangnekar> we are targeting to cover all scenarios for ceilometer v2 api
17:20:13 <msdubov> aswadrangnekar, are you going to cover all scenarios in one patch or in several patches?
17:20:30 <aswadrangnekar> pending scenarios are for queries and statistics
17:20:44 <aswadrangnekar> msdubov: https://review.openstack.org/#/q/status:open+project:stackforge/rally+branch:master+topic:bp/benchmark-scenarios-for-celiometer,n,z
17:21:00 <aswadrangnekar> several patches
17:21:12 <msdubov> aswadrangnekar, Yep, I see, thanks
17:21:20 <msdubov> aswadrangnekar, Great that several people are working on it
17:21:32 <msdubov> aswadrangnekar, So your patch is ready for review?
17:22:27 <aswadrangnekar> resources and meters should be up for review soon there are one scenario in each  of them
17:22:42 <msdubov> aswadrangnekar, Ok, thanks
17:22:43 <msdubov> I will take a look at it
17:22:55 <aswadrangnekar> we need to focus on queries should be ready soon
17:23:42 <msdubov> aswadrangnekar, Will require lots of review, so Rally cores should also look at these patches. Seems though like I'm now the only core developer at this meeting =)
17:24:06 <msdubov> #topic Tempest tests
17:25:02 <msdubov> andreykurilin, andreykurilin_ Could you please share your progress with Tempest? Also probably that by Olga if you know about it
17:25:31 <andreykurilin_> sure
17:25:47 <andreykurilin_> Kun left a comment about new scenarios for Tempest Benchmark(https://review.openstack.org/#/c/86836/). When I tried to fix them, I discovered bigger problem.
17:25:48 <hughsaunders> hey all, sorry was an hour off on meetime time
17:26:31 <andreykurilin_> Previous my patch(https://review.openstack.org/#/c/86337/) replaced `testr` launcher with `subunit` in verification. This helps to split test discovery and execution, but broke rally verification for "compute" and "full" sets.
17:26:32 <msdubov> hughsaunders, no problem
17:27:01 <andreykurilin_> I can't launch several tests from `compute` set via subunit without discovery.
17:27:23 <msdubov> andreykurilin_, Have you understood why that happens?
17:27:34 <andreykurilin_> a bit:)
17:27:49 <andreykurilin_> Today I chat with sdague and andreaf at #openstack-qa
17:27:57 <hughsaunders> msdubov: marcoemorais, I would prefer to remove the redundant processing from the btypes patch, but I can see that probably won't be straightforward
17:28:45 <hughsaunders> adding more redundant processing to the validators is probably quickest way to get it merged. And I do really like the feature, thanks for working on it
17:30:02 <msdubov> hughsaunders, What do you think about rewriting the scenario arguments at validation step? Do you also think that this will be too complicated?
17:30:43 <hughsaunders> msdubov: I would prefer to do the rewrite before validation, so they can stay seperate
17:30:47 <msdubov> andreykurilin_, Thanks! Do you know anything about Olga's progress with her patches? She's unfortunately absent at the moment
17:30:54 <hughsaunders> but I realise the meeting has moved on, sorry for dragging topic back
17:31:10 <msdubov> hughsaunders, Well this is a really important topic.
17:31:31 <msdubov> hughsaunders, Besides it wouldn't be great to merge a "quick" solution and the rewrite lots of code
17:32:14 <msdubov> hughsaunders, So yet another method here https://github.com/stackforge/rally/blob/master/rally/orchestrator/api.py#L102-L104
17:32:18 <msdubov> hughsaunders, before validation()?
17:32:20 <hughsaunders> I do think validation should come after translation, so that the values that will be used are checked
17:32:28 <marcoemorais> hughsaunders: so I had originally raised this point to boris-42, if I could upload a patch which does validation after processing, would that satisfy?
17:32:51 <andreykurilin_> msdubov: I ask sdague and about correct way to split test discover and execution, but sdague said that it's a bad idea:)
17:33:06 <andreykurilin_> msdibov: I do not know about Olga's progress
17:33:10 <hughsaunders> yeah, another func in benchmark engine and orchestrator api makes sense to me
17:33:22 <msdubov> andreykurilin_, Ok, thanks. Did sdague suggest something instead?
17:33:36 <andreykurilin_> msdubov, yes
17:33:52 <msdubov> hughsaunders, marcoemorais Also think that would be a good solution. So let's discuss that with boris-42 tomorrow once again
17:34:24 <hughsaunders> ok
17:34:41 <andreykurilin_> msdubov, he proposed to not split discovery and test execution.
17:35:05 <msdubov> andeykurilin_ obviously :)
17:35:15 <andreykurilin_> nsdubov,he said, that correct way to get time of test execution is to parse results of subunit stream
17:36:43 <msdubov> andreykurilin_, So great, you now have an understanding with direction to go futher. Did you realize already whether it's a hard task to parse this stream?
17:37:22 <hughsaunders> the current code uses a filter to convert it to xml (iirc) so should be able to parse that?
17:38:07 <andreykurilin_> yes, i already use it here - https://review.openstack.org/#/c/86836/
17:38:13 <msdubov> hughsaunders, andreykurilin_ Yep, I'm just not familiar with the format
17:38:50 <msdubov> andreykurilin_, In which file?
17:39:06 <andreykurilin_> `tempest_log_wrapper` from https://review.openstack.org/#/c/86836/10/rally/benchmark/scenarios/tempest/tempest.py saves results as add_atomic_actions
17:40:10 <msdubov> andreykurilin_, Yep I see
17:40:38 <msdubov> andreykurilin_, So then may we hope to see an update for "Tempest benchmarks - Part 2" soon? :)
17:41:37 <andreykurilin_> in near future I'll discuss solution of current problem with boris-42 and update my patch
17:41:58 <andreykurilin_> Also, sdague ended the conversation with phrase: "if that was addressed, and rally was reading the subunit, we could, for instance, apply the rally analysis at the end of every gate run"
17:42:13 <andreykurilin_> :)
17:42:43 <hughsaunders> awesome :D
17:43:00 <andreykurilin_> yeah
17:43:29 <msdubov> andreykurilin_, Really great
17:43:37 <msdubov> #topic CLI output
17:43:50 <msdubov> So I've prepared a patch https://review.openstack.org/#/c/89941/
17:44:04 <msdubov> That's what we discussed at one of the previous meetings
17:44:31 <msdubov> It merges the two tables (atomic actions & overall results) in the task results output into one
17:44:56 <msdubov> And also rewrites the code so that it uses the openstack.common code to draw tables (instead of PrettyTable directly)
17:45:08 <msdubov> hughsaunders, Could you review it please?
17:45:13 <hughsaunders> msdubov: will do
17:45:17 <msdubov> hughsaunders, thanks
17:45:27 <hughsaunders> msdubov: just read the commit message and the after paste looks much cleaner
17:46:01 <msdubov> hughsaunders, Unfortunately I wasn't able to split the atomic actions results and the overall results with a line
17:46:19 <msdubov> hughsaunders, Seems like PrettyTable doesn't support this kind of stuff
17:46:48 <hughsaunders> ok
17:47:09 <msdubov> Also my patch actually leaves some places written with a direct usage of PrettyTable, but this will be hopefully addressed by other patches from other developers
17:47:42 <msdubov> The next step after code refactoring will be to add the --percentile parameter to "task detailed", to make the percentiles customizable
17:48:20 <msdubov> #topic Free discussion
17:48:46 <msdubov> hughsaunders, andreykurilin_, marcoemorais Does anybody have anything to add?
17:49:21 <andreykurilin_> msdubov, nope :)
17:49:42 <hughsaunders> not from me
17:50:19 <marcoemorais> nothing from me
17:51:02 <msdubov> hughsaunders, andreykurilin_, marcoemorais, aswadrangnekar Ok, then many thanks for participation!
17:51:05 <msdubov> #endmeeting rally