19:03:17 <catherine_d> #startmeeting refstack
19:03:17 <openstack> Meeting started Mon Oct  6 19:03:17 2014 UTC and is due to finish in 60 minutes.  The chair is catherine_d. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:20 <openstack> The meeting name has been set to 'refstack'
19:03:25 <zehicle> o/
19:03:30 <catherine_d> o/
19:03:31 <rockyg> o/
19:03:32 <fcarpenter> o/
19:03:39 <pvaneck> o/
19:04:02 <rockyg> agenda?
19:04:09 <juliashapovalov1> o/
19:04:24 <catherine_d> David won't be joining today ... if need we can meet again on #refsrtack at noon tomorrow
19:04:43 <zehicle> I'd like to talk about juliashapovalov1 's suggestion about the JSON file
19:05:03 <zehicle> I'm also interesting in hearing about progress on the upload client
19:05:10 <zehicle> sorry I've been so out of the loop
19:05:51 <catherine_d> I would like to discuss the supporting of other distros  with setup_env ..
19:05:53 <rockyg> also, maybe catherine_d can update us on the new api model?
19:06:03 <sslypushenko__> Hi, all!
19:06:30 <sslypushenko__> I want to discuss setup_env  to
19:06:55 <sslypushenko__> Now I'm working in improvent for this script
19:07:16 <catherine_d> sslypushenko__: great ..
19:07:41 <sslypushenko__> It will setup virtualenv with python27 if it is not availble
19:08:09 <catherine_d> that is great ..
19:08:33 <sslypushenko__> actually I have already done it and now i'm testing it
19:08:38 <rockyg> ++
19:08:58 <catherine_d> #topic setup_env to support other distros
19:09:02 <sslypushenko__> on centos6 with py26 it look like it works
19:09:22 <catherine_d> sslypushenko__: it works for Tempest?
19:09:35 <catherine_d> or just setup_env?
19:10:19 <sslypushenko__> right now I check only that virtualenv builds correctly
19:10:37 <sslypushenko__> but I hope it will be enought
19:11:08 <sslypushenko__> because vitualenv includes python2.7
19:11:41 <catherine_d> zehicle: and rocky .. that bring us to the topic whether we need to state what distro we have tested
19:11:52 <zehicle> ok
19:12:16 <rockyg> good question
19:12:30 <catherine_d> do we need to add that test statement on the refstack web site (once it is up)
19:12:32 <zehicle> I'd like some background
19:12:43 <catherine_d> ok
19:13:17 <catherine_d> We start with only end-to-end test refstack-with Ubuntu 12.05 and 13.10
19:13:18 <zehicle> catherine_d, distro or release?
19:13:27 <sslypushenko__> We need some procedure for automatic testing for all distro's
19:13:42 <catherine_d> now we add yum installation ... which means we will support
19:13:58 <catherine_d> we can support CentOS, RedHat and Federoa
19:14:01 * zehicle confused - are we talking about OpenStack distro or the Linux where the test is running?
19:14:35 <sslypushenko__> Linux
19:14:37 <catherine_d> so do we need to test (validate) all distros or just state the one we tested
19:15:11 <catherine_d> end-to-end test means test installation and run a simple Tempest test case
19:15:34 <catherine_d> sslypushenko__: had done a good job in setup_env
19:15:48 <catherine_d> now just about support statement ...
19:15:48 <sslypushenko__> almost)
19:16:34 <sslypushenko__> But now I think that maybe refstack-in-container was not so bad idea)
19:16:40 <catherine_d> this is about Linux distor (not Tempest release)
19:16:54 <catherine_d> distro
19:17:44 <rockyg> right.  what linux distros we support running refstack client on
19:18:52 * zehicle loves TCUP.
19:19:04 <zehicle> containers are helpful from a support perspective
19:19:05 <sslypushenko__> I think Ubuntu, Debian, Centos, RHEL and Fedora
19:19:21 <rockyg> SuSe?
19:19:31 <catherine_d> with Sergey's setup ... we should be able to install any distro using apt-get  and yum to setup the env ... but it does not mean refstack will work ... we need to test to validate
19:19:46 <sslypushenko__> There are some limitations for releases aslo
19:21:26 <catherine_d> maybe we should give this topic some thoughts and come back to it on an IRC at noon tomorrow?
19:22:29 <sslypushenko__> What we need to clarify?
19:23:13 <catherine_d> where we only state on the support list on the distros that we have tested
19:24:38 <catherine_d> is everyone available at noon tomorrow?
19:25:36 <sslypushenko__> I will be available
19:25:46 <rockyg> sort of.  I"ll also be following another IRC and would like to go to Chinese class.  If we move to tomorrow, we need the agenda figured out now.
19:26:15 <catherine_d> not move to tomrrow ... just the setup_env discussion for tomorrow
19:26:50 <catherine_d> since David is not here ... we need his input for that ...
19:26:52 <rockyg> still, agenda for setup_env meeting?
19:26:53 <zehicle> I can be there at noon PDT
19:27:17 <sslypushenko__> +1 for agenda for tomorrow
19:28:32 <catherine_d> Agenda ... 1) Decide on explicitely post statement of which distros we had tested  2) decide on which distro we should test
19:29:04 <rockyg> Thanks.
19:29:08 <catherine_d> 3) whether we should add Suse support to setup_env
19:29:42 <juliashapovalov2> should we consider  differentt distros versions?
19:29:47 <rockyg> I believe they are bigger in Europe
19:30:14 <catherine_d> juliashapovalov2: good topic for tomorrow tooo
19:30:48 <catherine_d> Let's move to the next topic for today ... juliashapovalov2: 's suggestopn on JSON
19:31:39 <catherine_d> #topic juliashapovalov2: 's suggestion on capability JSON
19:33:20 <juliashapovalov2> Rob only reviewed the commits
19:33:51 <juliashapovalov2> actualy they contain tests previously approved by defcore
19:34:13 <juliashapovalov2> but cleaned up according to tempest master
19:34:45 <juliashapovalov2> i beleive they should be merged as initial version and extended by other capabilities
19:34:49 <zehicle> juliashapovalov2, that work is awesome.  thanks
19:35:05 <zehicle> +1 on the merger.  I'd like catherine_d to review
19:35:37 <catherine_d> I think the first discussion is do we support one file or separated JSON files for Havana, Icehouse?
19:36:23 <juliashapovalov2> commits i've talked about change only one file
19:36:29 <juliashapovalov2> for havana
19:36:55 <juliashapovalov2> single file for openstak releases is another topic
19:37:18 <catherine_d> Since we had decided to use tempest tag release at last week IRC .... we should use a tag release to generate the fully qualified test names
19:37:26 <zehicle> I'm concerned a single file will be very difficult to manage as we go
19:37:41 <catherine_d> I agree with zehicle:
19:37:43 <zehicle> juliashapovalov2, that's the one I was hoping to discuss
19:37:48 <rockyg> question:  If we only do one file, how do we let people browse a specific release's tests?  We would need a tool.
19:37:56 <catherine_d> since we are a a very dynamic states ..
19:37:59 <zehicle> I totally see juliashapovalov2 point about having a single file
19:37:59 <juliashapovalov2> otherwise we will have a set of same long files
19:38:25 <sslypushenko__> with duplicaing info
19:38:29 <zehicle> both choices have challenges until we get UUIDs
19:38:43 <catherine_d> I thing the one file idea is good at relatively steady state
19:38:45 <zehicle> but I want to make sure the files can stand alone and be human readable
19:39:13 <juliashapovalov2> i described it in spec - we'll manage releases by json structure
19:39:22 <catherine_d> zehicle: +1at least for Havana and Icehouse
19:39:47 <juliashapovalov2> each next release will extend the parent releases
19:39:48 <sslypushenko__> The most important issue is how to avoid duplication information in test lists for different releases
19:40:04 <juliashapovalov2> i'can add more clarifications if needed
19:40:21 <zehicle> juliashapovalov2, I just dont see how people would be able to track that in json.  we'd have to build a tool
19:41:04 <zehicle> juliashapovalov2, I like the concept.  It makes sense to see the releases a cumulative
19:41:18 <zehicle> I'm worried about the practical implementation for reviewers
19:41:28 <catherine_d> the duplicate or new test cases issues  are easy to deal with,  the difficulty is identity the existing tests which are renamed
19:41:40 <zehicle> I think we could go to that approach overtime if needed
19:42:07 <catherine_d> zehicle: +1 got to it overtime
19:42:14 <juliashapovalov2> zehicle: I'm worried about the practical implementation for reviewers
19:42:14 <juliashapovalov2> as I understand, html page generator will be responsible for that
19:42:34 <zehicle> this is a messy problem no matter how we slice it - the tests & test framework were not designed with this use-case in mind
19:42:58 <zehicle> but it would be hard for reviewers to look at changes
19:43:16 <zehicle> and if we needed a tool, you could create  the deltas just as easily
19:43:26 <rockyg> anything we could do in tempest to make it more friendly? i.e., a spec that helps our stuff work if the spec is implemented?
19:43:45 * zehicle shrugs... UUIDs
19:43:49 <catherine_d> rockyg: uuid defined by Tempest
19:44:43 <catherine_d> So one  JSON or multiple JSONs for now?
19:45:53 <zehicle> we've got to get Havana right first
19:45:59 <rockyg> we are starting on icehouse and until we have tools, i think we should *start* with multiple.  If we get this worked out, we can converge.
19:46:17 <zehicle> I can see why juliashapovalov2 wants a single file
19:46:31 <zehicle> since she's looking at which files are added
19:46:37 <zehicle> which tests
19:46:46 <catherine_d> multiple JSON for now moving toward one JSON later
19:47:01 <juliashapovalov2> for me icehouse capabilities for now looks like copy-n-paste havana + add new ones
19:47:07 <sslypushenko__> May be better question is - plain lists of tests. of cumulative sets for each release?
19:47:16 <zehicle> catherine_d, +1
19:47:29 <sslypushenko__> *or cumulative...
19:47:33 <zehicle> I'd vote for plain lists
19:48:02 <sslypushenko__> Why?
19:48:12 <zehicle> because if capabilities shift
19:48:23 <zehicle> it would be very difficult to work w/ cumulative
19:48:36 <zehicle> we cannot assume that the capabilities lists are correct and wont change
19:48:40 <catherine_d> zehicle: +1
19:48:44 <zehicle> AND we cannot change the past ones that have been approved
19:49:24 <zehicle> I think that juliashapovalov2 is right for several technical reasons; HOWEVER, we have to deal with the outside process aspects of review and board approval
19:49:34 <zehicle> and those are easier to track as individual files
19:49:41 <catherine_d> zehicle: does that mean that we should use stable-havana-tempest to generate Havana test list?
19:49:45 <zehicle> juliashapovalov2, does that make sense?
19:49:59 <zehicle> catherine_d, yes, that should be the starting point
19:50:12 <zehicle> then we'd pick up the cumulative changes in the reviews
19:50:30 <catherine_d> zehicle: +1 sslypushenko__: this is why we should use multiple JSON for now
19:50:40 <zehicle> catherine_d, sorry, I thought you were asking Havana -> Ice House
19:50:45 <juliashapovalov2> if you'll explain process issues, probably :)
19:51:18 <zehicle> juliashapovalov2, we have to get the board to approve the test lists.  that's harder to track if we keep changing it
19:51:41 <zehicle> if each file is distinct, then we have very simple visibility if someone is altering board approved items
19:51:48 <zehicle> (which in some cases is OK)
19:51:57 <rockyg> Right.  And when something gets deprecated, the tests will go away in a later version
19:52:35 <juliashapovalov2> which means the list changes...
19:52:39 <catherine_d> here is what I have in mind ... Havana test list will be based on stable-havana-tempest .. Icehouse will be base on a tag release (which still need to be identified)
19:52:42 <zehicle> we have a mechanism to flag tests that are broken for vendors.  that would be an example of a test exception
19:53:13 <zehicle> and also an example of something that would impact a file that's been approved
19:53:27 <rockyg> I found a tag that QA put on right before they went "branchless".  It pretty much coincides with Icehouse (a day or two later)
19:53:30 <zehicle> IMHO, we would NOT take out a depricated test, we'd flag it instead.
19:53:34 <rockyg> I can dig it up again.
19:54:17 <catherine_d> trockyg: that is tempest-1 tag create in May, 2014
19:54:21 <rockyg> There are a ton of XML tests that vanish for icehouse
19:54:27 <catherine_d> for Icehouse right?
19:54:35 <rockyg> catherine_d: yup
19:54:44 <catherine_d> ok   ..
19:54:47 <juliashapovalov2> zehicle: havana branch have to be deprecated itself
19:55:15 <juliashapovalov2> this means that vendors won't phisically have access to deprecated tests
19:55:36 <zehicle> I thought that the same tests were available in branchless tempest
19:55:52 <rockyg> juliashapovalov2: I'm not sure.  If someone is still running it past the end of support, it's still a valid release.  Just no longer supported.
19:55:55 <juliashapovalov2> not all of them
19:56:02 <zehicle> I thought we are not running Havana from the branch but trunk
19:56:30 <zehicle> we should only use branchless tests.  if they are not ported over then we should omit them
19:56:35 <rockyg> zehicle: yes.  which is why the tests are moving all over the place.
19:56:41 <zehicle> remember: havana is advistory only.  it's ok if we have gaps
19:56:42 <juliashapovalov2> rockyg: branchless tempest supposed to be compatible with havana
19:56:56 <zehicle> the focus is on the process so vendors can start collecting data and validating
19:57:14 * zehicle throws hands up in despair
19:57:46 <catherine_d> we only have 4 mins left ... I want to give a quick update on upload client that is it is not ready ... and David is working on it
19:57:51 <rockyg> juliashapovalov2:  yes.  but the tests can be run even after the release is long gone.  It's in branchless, so they won't go away?
19:58:23 <rockyg> what about new api?
19:58:54 <juliashapovalov2> most openstack features are still there, and should be tested somehow :)
19:58:56 <catherine_d> no not thing work yet .. not old or new.
19:59:16 <rockyg> all:  we need a stable list of tests for Paris (preferrably a week or two before) so vendors can have results before Paris
19:59:56 <catherine_d> rockyg: +1 ... looks like we need more time to discuss this topic too ..
20:00:17 <zehicle> At some point, we'll just jump forward to Icehouse
20:00:25 <zehicle> good discussion.  we got a lot covered
20:00:25 <juliashapovalov2> the same time tomorrow in refstack channel?
20:00:37 <catherine_d> need to end meeting now .... we will meet at noon tomorrow  ...
20:00:40 * zehicle adds meeting to my calendar
20:00:41 <catherine_d> #refstack
20:00:58 <catherine_d> #endmeeting