14:05:50 <ikhudoshyn> #startmeeting Rally
14:05:51 <openstack> Meeting started Mon Feb 29 14:05:50 2016 UTC and is due to finish in 60 minutes.  The chair is ikhudoshyn. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:05:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:05:54 <openstack> The meeting name has been set to 'rally'
14:06:06 <ikhudoshyn> hi everybody
14:06:52 <redixin> sup
14:06:54 <amaretskiy> hi
14:07:35 <andreykurilin> o/
14:07:48 <rvasilets> o/
14:07:49 <ikhudoshyn> we got couple of topics in the agenda
14:08:15 <ikhudoshyn> the first is from Chris
14:08:39 <ikhudoshyn> stpierre, around?
14:09:23 <ikhudoshyn> my irc client autocompletion tels that he's not
14:09:37 <ikhudoshyn> so let's start with the 2nd
14:09:47 <ikhudoshyn> [amaretskiy] Let's discuss Web UI for trends report
14:09:58 <ikhudoshyn> http://logs.openstack.org/21/276721/4/check/gate-rally-dsvm-rally/0425a53/rally-plot/trends.html.gz
14:10:07 <ikhudoshyn> https://review.openstack.org/#/c/276721/
14:10:16 <ikhudoshyn> #topic [amaretskiy] Let's discuss Web UI for trends report
14:10:22 <ikhudoshyn> amaretskiy: pls start
14:10:50 <amaretskiy> we have new upcomign report "trends"
14:11:05 <amaretskiy> this report will show trends for many tasks runs
14:11:24 <amaretskiy> so we have a WIP UI http://logs.openstack.org/21/276721/4/check/gate-rally-dsvm-rally/0425a53/rally-plot/trends.html.gz
14:11:49 <amaretskiy> I would like to ask feedback, proposals, ideas about this report
14:12:03 <amaretskiy> the main questions are:
14:12:14 <amaretskiy> 1. is proposed report OK?
14:12:55 <amaretskiy> 2. Main table "Trends overview" - are columns OK? What to add?
14:13:30 <ikhudoshyn> amaretskiy: there was a question about scenarios that present in only one task
14:13:43 <ikhudoshyn> so they don't get into the report
14:13:46 <amaretskiy> 3. There is a good proposal from  ikhudoshyn regarding missed runs (will explain later in details) - I believe we discuaa it here and confirm
14:14:08 <ikhudoshyn> did you have a chance to think on it and/or discuss w/Bdfl
14:14:23 <ikhudoshyn> oops_)
14:14:32 <amaretskiy> so, let's start with 1) is proposed report OK in general? Maybe some parts should be removed/changed?
14:14:40 <ikhudoshyn> the report looks great for me)
14:15:24 <ikhudoshyn> (except TBD)
14:15:37 <amaretskiy> yes, TBD is just to ask ideas :)
14:16:02 <rvasilets> I'd like to for atomic drop down menu http://logs.openstack.org/21/276721/4/check/gate-rally-dsvm-rally/0425a53/rally-plot/trends.html.gz#/Dummy.dummy_random_atomic-2/atomic
14:16:32 <rvasilets> and possibly with some search
14:16:42 <amaretskiy> okay, let's discuss dropdown for atomics
14:17:16 <amaretskiy> we do not have a lot of atomics, probably search is not required :)
14:17:36 <ikhudoshyn> ctrl-f is good enough for me
14:17:38 <boris-42> hi everydbody
14:17:41 <ikhudoshyn> at least for now
14:17:47 <ikhudoshyn> hi boris-42
14:18:00 <boris-42> amaretskiy: you are talking about utils refactoring?
14:18:09 <ikhudoshyn> nope, trends
14:18:35 <ikhudoshyn> trends report UI, to be exact
14:18:45 <amaretskiy> boris-42: no, no refactoring
14:18:56 <amaretskiy> boris-42: I'm about UI :)
14:19:00 <boris-42> amaretskiy: ok ok
14:19:20 <amaretskiy> boris-42: trends report UI :)
14:19:24 <rvasilets> Also in what order they are printed?
14:19:48 <rvasilets> maybe there good to have sorting by key
14:19:53 <rvasilets> by name
14:20:00 <amaretskiy> atomic actions order is in their original order how they are stored in results
14:20:03 <rvasilets> and by the oeder of execution
14:20:19 <amaretskiy> they can be sorted by name but this looks not reasonable for me
14:20:39 <boris-42> amaretskiy: rvasilets let's just refactor atomic actions
14:20:50 <boris-42> amaretskiy: rvasilets instead of adding hacks about order
14:21:07 <rvasilets> okey=)
14:21:08 <amaretskiy> agreed
14:21:14 <rvasilets> agreed
14:21:17 <amaretskiy> so, there is a proposal from rvasilets
14:21:22 <boris-42> rvasilets: and adding sorting by keys will mess it up
14:21:36 <amaretskiy> to show dropdown-list for choosing atomic action
14:21:43 <ikhudoshyn> #agreed Proposed UI for trends report is OK
14:21:50 <amaretskiy> and show only single charts for selected atomic
14:22:19 <amaretskiy> boris-42 andreykurilin ikhudoshin what do you thisnk?
14:22:24 <amaretskiy> *think
14:22:58 <amaretskiy> this is about http://logs.openstack.org/21/276721/4/check/gate-rally-dsvm-rally/0425a53/rally-plot/trends.html.gz#/Dummy.dummy_random_atomic-2/atomic
14:23:34 <amaretskiy> display all atomics (as this is now) or select specific atomic by dropdown list?
14:23:46 <ikhudoshyn> since I truly believe that the best code is the one that does not exist, I would not add it before we do have a demand for the latter
14:24:07 <ikhudoshyn> lets just display all of them
14:24:11 <ikhudoshyn> as it is
14:24:53 <amaretskiy> actually changing this to dropdown (later) will not require significant efforts so this is absolutely safe to choose any variant
14:25:35 <rvasilets> okey
14:25:42 <amaretskiy> ok, let;s proceed
14:25:46 <amaretskiy> with option 2
14:25:55 <amaretskiy> 2. Main table "Trends overview" - are columns OK? What to add?
14:26:05 <amaretskiy> maybe add some extra column?
14:26:18 <amaretskiy> we can use status marks like in TBD
14:26:23 <amaretskiy> any ideas?
14:26:48 <amaretskiy> Actually we can keep only given columns and add another later
14:27:08 <amaretskiy> so if no ideas, then we can proceed :)
14:27:18 <ikhudoshyn> amaretskiy: +1
14:28:15 <rvasilets> Myabe just as in our reports passed or failed
14:28:38 <rvasilets> that is all ideas from my side)
14:28:49 <ikhudoshyn> rvasilets: that's what TBD is ))
14:28:56 <ikhudoshyn> amaretskiy: am I right?
14:29:12 <amaretskiy> rvasilets: consider that this is about N runs of report so we can probably mark "failure" for a set of runs that have at least one failed SLA
14:29:56 <ikhudoshyn> amaretskiy: that's what i'd expect
14:30:25 <rvasilets> Red - if SLA failed(or at least one SLA)
14:30:43 <amaretskiy> okay, so we have proposal
14:30:57 <amaretskiy> add a column (in place of TBD) with name "SLA"
14:31:17 <amaretskiy> and status marker - failure if at least one run has failed SLA
14:31:23 <amaretskiy> correct?
14:31:43 <rvasilets> yes, for example
14:31:56 <amaretskiy> okay
14:32:20 <amaretskiy> ikhudoshyn boris-42 please give your votes :)
14:32:28 <ikhudoshyn> +
14:32:47 <amaretskiy> great, I will add SLA
14:32:59 <amaretskiy> I guess we can proceed to 3)
14:33:01 <boris-42> ok
14:33:22 <amaretskiy> 3. There is a good proposal from  ikhudoshyn regarding missed runs
14:35:00 <ikhudoshyn> amaretskiy: whould you remind it?
14:35:38 <amaretskiy> explanation: we expect from users to specify tasks with same scenarios, however user can actually specify arbitrary or different scenarios, so we can have a user case with tasks: (A, B, C, D), (A, B, C, D), (A, B, C, E),
14:35:52 <amaretskiy> you see that we have scenario "E"
14:36:00 <amaretskiy> which appears single time
14:36:27 <amaretskiy> so trends report will even does not show it in menu - just kick it off silently
14:37:23 <amaretskiy> the idea is to show it in menu just to inform user that this scenario appeared in input data, but do not show charts (of course) because we can not build chart from single point
14:38:21 <ikhudoshyn> we have to show something when user selects it in the menu
14:38:22 <amaretskiy> so, let's approve/disapprove this: should I add single-appeared scenarios to left-side menu (and overview table with "n/a" results or smth like this) to report?
14:38:43 <amaretskiy> boris-42 ^
14:38:50 <amaretskiy> rvasilets &
14:38:55 <amaretskiy> rvasilets ^
14:39:14 <amaretskiy> andreykurilin ^
14:39:21 <ikhudoshyn> you dont even need 'n/a'  -- just show '1' number of runs in overview
14:39:38 <ikhudoshyn> min max and avg will be valid
14:39:46 <amaretskiy> ikhudoshyn: agreed
14:40:01 <rvasilets> I think that we need
14:40:09 <rvasilets> if its not hard to us
14:40:48 <amaretskiy> great, so decided to show this data (personally me agreed too)
14:41:10 <amaretskiy> Any other ideas about trends report&
14:41:14 <rvasilets> because we don't know what the strange yamls user will run. And such displaying is a good bonus
14:41:32 <amaretskiy> agreed
14:41:35 <ikhudoshyn> looks like no objections
14:42:30 <amaretskiy> okay, so let's complete topic about trends report
14:42:31 <ikhudoshyn> anything else to discuss on the topic?
14:42:37 <ikhudoshyn> ok
14:42:54 <ikhudoshyn> we actually have another one in the agenda
14:43:17 <ikhudoshyn> but since stpierre is not around I believe we'd skip it for now
14:43:21 <ikhudoshyn> ok?
14:43:21 <stpierre> i am now
14:43:29 <stpierre> i was a little late arriving, sorry
14:43:38 <ikhudoshyn> great)
14:43:43 <ikhudoshyn> #topic [stpierre] How to test complex functional cases (e.g., https://review.openstack.org/#/c/282918/)
14:43:49 <stpierre> #link https://review.openstack.org/#/c/282918/
14:43:53 <ikhudoshyn> stpierre: your turn
14:44:01 <stpierre> that's the second attempt at a patch that caused a lot of problems with 0.3.0
14:44:12 <stpierre> as you can see from the description there are three bugs linked, and at least one more mentioned in the discussion
14:44:25 <stpierre> most of those bugs have fairly trivial test cases against a running openstack
14:44:38 <stpierre> but we don't currently have a good way of executing those in gates, that i'm aware of
14:45:08 <stpierre> so, given the discussion last week around improving test coverage: 1) do we want to be adding arbitrary, potentially complex tests against a running openstack to gate? 2) if so, how?
14:46:38 <stpierre> no one has any opinions? :)
14:47:03 <ikhudoshyn> well, I believe 1) has an easy answer -- yes
14:47:45 <ikhudoshyn> the issue is that amount of such tests could be enormous
14:48:04 <stpierre> yeah. and devstack and gate are really fragile already
14:48:15 <amaretskiy> +1 for "yes" from me
14:49:02 <stpierre> "yes" seems like the obvious answer, but i'm worried that the devil is in the details. but certainly if we can make it happen without creating excessively many gate jobs (and excessively many devstacks) then it seems more reasonable
14:49:03 <ikhudoshyn> sometimes I think we miss an oldschool manual QA )
14:49:03 <andreykurilin> stpierre: there is one more bug appeared after that patch:)
14:49:08 <stpierre> hah
14:50:11 <andreykurilin> stpierre: I had a talk with redixin and he will try to add new job to rally-ci with "devstack+ssl+keystone_v3"
14:50:21 <stpierre> cool
14:50:38 <rvasilets__> We could try to build our corrent functional tests on the real db:-)
14:50:45 <stpierre> how many devstack deployment scenarios can we reasonably cover? that seems like a real concern
14:51:00 <andreykurilin> stpierre: also, I want to make one of jobs(we have 2 of them) for "rally verify" with keystone v3
14:51:00 <amaretskiy> stpierre: the main reason (afaik) why we have many dsvm is different devstack configuration (neutron vs nova-network and so on)
14:51:08 <stpierre> yeah
14:51:32 <stpierre> and i'm sure we could easily come up with reasons to have twice as many
14:52:14 <ikhudoshyn> what stops us from that actually&
14:52:16 <ikhudoshyn> ?
14:52:20 <stpierre> so maybe the current process, where it's fairly time consuming to add a new gate job, is the right process
14:52:20 <ikhudoshyn> just curious
14:52:24 <andreykurilin> btw, it will be easier to increase quality of our ci by moving all job hooks to our repo. It will provide an easy way to experiment with job opts
14:52:29 <stpierre> we have to really want to test something new in order to add a job
14:53:00 <stpierre> ikhudoshyn: my concerns would be a) the time it takes to run everything; and b) the fragility of gate. with enough jobs, you'd be practically guaranteed that at least one would experience a random transient failure
14:53:39 <ikhudoshyn> stpierre: looks like it's because the product is complex
14:53:52 <ikhudoshyn> we still have to test it
14:53:57 <stpierre> also, devstack is awful. but that's not gonna change either
14:54:10 <ikhudoshyn> true^^
14:54:24 <andreykurilin> stpierre: Is there an alternative to devstack?
14:54:44 <stpierre> i don't think there's a reasonable one for this use case
14:54:49 <ikhudoshyn> yet i think we can not afford only running tests during release preparation
14:55:09 <stpierre> no, i agree
14:55:20 <ikhudoshyn> that's the idea of CI -- always working, always green
14:55:37 <ikhudoshyn> i we have to test -- we should do
14:55:59 <ikhudoshyn> if it takes too long -- so it's big
14:56:05 <stpierre> andreykurilin: can you talk a little more about what moving job hooks to our repo would involve and how that would work?
14:57:34 <ikhudoshyn> guys, we're running out of time. let's continue on our channel
14:57:42 <stpierre> ok
14:57:44 <andreykurilin> stpierre: it should allow us to select keystone v2/v3 for each job locally. Just additional checks here https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/rally.yaml#L174-L176
14:58:07 <stpierre> for the sake of the logs, let's get the keystone v3 checks andreykurilin described earlier implemented, and we can continue to think on how to best approach this
14:58:14 <andreykurilin> also, we can try to launch partial jobs in venv and some of them system-wide
14:59:41 <ikhudoshyn> thanks everybody
14:59:51 <ikhudoshyn> #endmeeting