17:00:19 <mtreinish> #startmeeting qa
17:00:20 <openstack> Meeting started Thu Feb 13 17:00:19 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:23 <openstack> The meeting name has been set to 'qa'
17:00:37 <mtreinish> who's here today?
17:00:49 <mlavalle> hi
17:00:52 <marun> hi
17:00:57 <vponomaryov> hi
17:00:59 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:01:05 <mtreinish> ^^^ today's agenda
17:01:13 <andreaf> hi
17:01:48 <mtreinish> sdague, dkranz, mkoderer: are you guys around?
17:01:57 <dkranz> mtreinish: here
17:02:04 <mtreinish> well let's get started
17:02:14 <mtreinish> #topic Removing nose support (mtreinish)
17:02:31 <mtreinish> So I'm not sure veryone saw but I have the patch up that will actively prevent running tempest with nose
17:02:43 <mtreinish> #link https://review.openstack.org/#/c/72040/
17:03:03 <mtreinish> this doesn't mean that py2.6 isn't supported just that nose can't be used
17:03:19 <mtreinish> and it doesn't really work right now anyway
17:03:29 <mkoderer> mtreinish: sry I am late
17:03:31 <dkranz> mtreinish: So why not fix the py26 in tox.ini rather than remove it?
17:03:39 <dkranz> mtreinish: to use testr
17:04:02 <mtreinish> dkranz: using testr there is no difference between py2.6 and py2.7 (once you have the discover patch applied)
17:04:33 <mtreinish> so there is no need for a separate py2.6 tox job
17:04:35 <dkranz> mtreinish: ok, so if we just put good doc for applying the patch then I'm ok
17:04:53 <dkranz> mtreinish: afazekas demonstrated that it works, right?
17:04:56 <afazekas> unittest2 + discovey is a dependency in addition , but otherwise it is the same
17:05:05 <marun> is the discover patch going to be integrated into the tree at some point?
17:05:05 <mtreinish> dkranz: ok I'll push out a doc update patch too
17:05:07 <mtreinish> dkranz: yeah
17:05:20 <marun> not our tree - test(*?)
17:05:28 <dkranz> mtreinish: thanks. I am in favor of this but just want to  make sure when people break they know what to do without too much trouble
17:05:33 <afazekas> s/discovery/discover/
17:05:51 <mtreinish> marun: the bug for it has been sitting there for awhile
17:05:57 <sdague> mtreinish: here, sorry, in this gate discussion
17:06:01 <mtreinish> the patch is really simple too it just changes an import
17:06:06 <mtreinish> from unittest to unittest2
17:06:10 <marun> mtreinish: which project?
17:06:33 <marun> mtreinish: I guess I'd like to know that this won't require a manual fix in perpetuity.
17:06:51 <mtreinish> marun: https://code.google.com/p/unittest-ext/issues/detail?id=79
17:07:06 <mtreinish> that's what afazekas shared before
17:07:33 <marun> oh, for unittest2?
17:08:18 <marun> anyway, carry on.  not really important to discuss here.
17:08:32 <mtreinish> ok I just wanted to let everyone know about that change
17:08:35 <afazekas> marun: there is module called  'discover' , it lives in the unittestext repo, but packaged separately
17:08:40 <mtreinish> I think we can move onto the next topic
17:08:48 <mtreinish> #topic  Blueprints
17:09:08 <mtreinish> So does anyone have any blueprints to bring up
17:09:15 <mtreinish> mkoderer: I see you have a proposal for a new one
17:09:16 <mkoderer> mtreinish: I have a proposal
17:09:20 <mtreinish> is there a link?
17:09:27 <mtreinish> or is it just an idea
17:09:31 <mkoderer> mtreinish: I didn't filed it
17:09:41 <mtreinish> ok
17:09:42 <mkoderer> but I discused it with OSSG (security group)
17:10:11 <mkoderer> mtreinish: so the idea is to build a fuzzy testing framework out of the new negative test blueprint
17:10:26 <mkoderer> maybe a mixture between stress test and negative tests
17:10:54 <mkoderer> first idea is just to create more value generators
17:11:22 <mkoderer> the OSSG is very intressted and will support me with that
17:11:43 <mkoderer> my plan is to open a blueprint tomorrow and work on the details
17:12:00 <mkoderer> anyone interested on that topic?
17:12:13 <andreaf> mkoderer: is it about avoiding malicious injection and similar?
17:12:31 <mtreinish> mkoderer: ok, it sounds like a good idea, especially if you can resue the existing test infrastructure and just add extra value from it
17:12:33 <mkoderer> andreaf: it's trying to cause a DOS
17:12:48 <mtreinish> mkoderer: would the intent to be adding this as gating
17:12:54 <mtreinish> or would it be too slow?
17:13:29 <mkoderer> mtreinish: I have the feeling that with the current state it quite easy that it fails
17:13:51 <mkoderer> so gating yes.. but later
17:13:55 <mtreinish> ok
17:14:08 <dkranz> mkoderer: That's what we said for stress too :)
17:14:16 <dkranz> We need more gate stability first
17:14:23 <mkoderer> dkranz: I know :)
17:14:39 <mkoderer> and that another project I have to care about ;)
17:14:47 <mtreinish> mkoderer: ok so draft up the bp and we can go from there
17:14:53 <mtreinish> is there anything else on this?
17:14:57 <mkoderer> mtreinish: yep I will do that
17:15:07 <mkoderer> not from my side
17:15:13 <mtreinish> mkoderer: ok
17:15:23 <mtreinish> andreaf: you have a bp listed in the agenda
17:15:34 <andreaf> mtreinish: yes
17:15:37 <mtreinish> #topic Nova v3 testing in scenario tests (andreaf)
17:15:42 <mtreinish> andreaf: ok go
17:15:54 <andreaf> mtreinish: it's not a bp I own, it's nova v3 testing
17:16:11 <mtreinish> do you have a link?
17:16:35 <andreaf> https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests
17:16:44 <mtreinish> #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests
17:16:44 <andreaf> #link https://blueprints.launchpad.net/tempest/+spec/nova-v3-api-tests
17:16:55 <andreaf> andreaf: you win :P
17:17:09 <andreaf> mtreinish: ok so the bp is about nova api v3 testnig
17:17:46 <andreaf> mtreinish: but as part of it we have now submissions for scenario nova v3 tests #link https://review.openstack.org/#/c/72858/
17:18:14 <andreaf> andreaf: I'm not fully convinced that scenario tests are for verifying multiple api versions
17:18:22 <andreaf> mtreinish: ^^
17:18:41 <andreaf> mtreinish: that's the question I wanted to raise
17:18:58 <marun> do the api's differ in functionality?
17:19:00 <mtreinish> andreaf: yeah I'm not sure either, but I don't see harm except for duplicating are slowest tests in the gate
17:19:19 <mtreinish> marun: slightly, but the scenario tests use the official clients
17:19:20 <dkranz> andreaf: I agree and don't think we should copy the scenario tests
17:19:24 <mtreinish> so that should be abstracted away
17:19:34 <dkranz> unless we think something new is being tested
17:19:40 <andreaf> mtreinish: the main difference is nova not proxying glance / cinder anymore
17:20:11 <marun> that sounds like an integration point that needs to be tested, but maybe not as part of the scenario testing
17:20:16 <dkranz> andreaf: That is an implementation change that should be covered by api tess
17:20:18 <dkranz> tests
17:20:26 <andreaf> drankz: yes indeed
17:21:05 <andreaf> mtreniish, dkranz: I think API versions shall stay in the API part of tests, or else it will become very hard to maintain scenario tests
17:21:18 <dkranz> I think the scenarios are some coverage of making sure apis don't return success but then bork the system
17:21:52 <mtreinish> andreaf: yeah I think that's good to say, but maybe we should have a config for which api version should be used with the official clients
17:22:00 <mtreinish> at least while they support more than one
17:22:20 <marun> do the clients allow the api version to be selected?
17:22:22 <andreaf> mtreinish, dkranz: but I would be in favour of the following approach: tag the test for the api version supported, and perhaps run different api versions in different jobs
17:22:32 <afazekas> marun: the nova client does
17:22:57 <mtreinish> marun: nova does, but I'm not sure how it works in the other clients (but I suspect they do too)
17:23:14 <andreaf> andreaf: ATM we only use 1 version of glance API as well in the scenario tests, and one version of cinder API
17:23:16 <marun> good to know
17:23:18 <afazekas> #link https://github.com/openstack/python-novaclient/blob/master/novaclient/client.py#L458
17:23:45 <mtreinish> andreaf: ok do you want to start a ml thread about this
17:23:54 <mtreinish> because the people doing this work aren't around now
17:24:02 <andreaf> mtreinish: ok
17:24:08 <mtreinish> but I think the direction moving forward is to make it configurable
17:24:27 <mtreinish> but not duplicate the tests
17:24:36 <dkranz> mtreinish: +1
17:24:37 <mtreinish> andreaf: is there anything else on this?
17:25:16 <andreaf> mtreinish: no
17:25:32 <mkoderer> dkranz: should we talk about negative testing?
17:25:43 <mtreinish> ok vponomaryov you've also got a bp in the agenda
17:25:49 <vponomaryov> yes
17:25:51 <vponomaryov> #link https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable
17:25:52 <dkranz> mkoderer: Not too much to say at the moment
17:25:58 <mtreinish> #topic make tempest pluggable bp
17:25:58 <mkoderer> dkranz: all right
17:26:07 <mtreinish> vponomaryov: ok you've got the floor
17:26:12 <vponomaryov> Need some extra eyes on that, and the way, that used in commits...
17:26:17 <vponomaryov> 2 commits from 3 planned:
17:26:17 <vponomaryov> #link https://review.openstack.org/#/c/62429/ (exceptions)
17:26:17 <vponomaryov> #link https://review.openstack.org/#/c/69985/ (config)
17:26:33 <vponomaryov> Main idea is concluded in pros:
17:26:47 <vponomaryov> - separate files, much less merge conflicts
17:26:47 <vponomaryov> - explicit separation by projects, no mess of exceptions or configs
17:26:47 <vponomaryov> - backward compatible, no need to change use places
17:27:36 <vponomaryov> partions of bp:
17:27:38 <vponomaryov> exceptions - only separation
17:27:38 <vponomaryov> config - separation with autoimport
17:27:38 <vponomaryov> clients -separation with autoimport
17:28:01 <mkoderer> vponomaryov: I think this is something we already have in other projects right...
17:28:27 <mkoderer> I need to have a closer look
17:28:31 <mkoderer> I can do the review
17:28:36 <mtreinish> vponomaryov: so I'm not sure I see the value of the exceptions one (we don't really have many of those)
17:29:01 <vponomaryov> it will be extended, after a while
17:29:16 <vponomaryov> new projects are appearing
17:30:28 <vponomaryov> at least, we should separate base classes and use classes
17:30:35 <mtreinish> vponomaryov: the thing is with config and clients there is a lot of copy and paste to get things setup in one file
17:30:54 <mtreinish> so I can see value with it just discovering the options and clients from a directory
17:31:14 <mtreinish> but for exceptions there aren't many and it's normally just a 1 line class that overloads the msg
17:32:11 <vponomaryov> in service clients and tests we keep base classes separately
17:32:26 <mtreinish> vponomaryov: but I think people haven't had a chance to review the proposal in too much depth
17:32:28 <vponomaryov> exceptions - all in one
17:33:01 <mtreinish> vponomaryov: so I think we handle this in review
17:33:16 <mtreinish> or outside of the meeting
17:33:27 <vponomaryov> agree
17:33:35 <mtreinish> ok then let's move on
17:33:50 <mtreinish> #topic Neutron testing
17:33:55 <mtreinish> mlavalle: are you around?
17:34:11 <mlavalle> yes iam
17:34:25 <mtreinish> are there any updates on neutron
17:34:28 <mlavalle> I am sure all know, but the Neutron gate is working now
17:34:50 <marun> stable/havana is still borked
17:35:04 <mtreinish> mlavalle: yeah and I pushed out the change that makes all the neutron jobs isolated
17:35:20 <mlavalle> the group of people contributing api tests have been active
17:35:25 <mtreinish> mlavalle: yeah it is, I think we'll have to drop the isolated jobs for the stable branch
17:35:32 <dkranz> mlavalle: Is this still smoke, or full?
17:35:33 <mtreinish> ^^^ sry marun
17:35:40 <mtreinish> dkranz: it's still smoke
17:35:41 <mlavalle> so we have a lot of code to review
17:36:23 <mlavalle> how can I help the core reviewers to go through these patches? Would a message to the ML help?
17:36:52 <mkoderer> mlavalle: just ping on irc
17:36:59 <mtreinish> mlavalle: if there was a gerrit view/query that could show them all in one place that would be nice too
17:37:03 <mtreinish> are they all on the same bp
17:37:24 <mlavalle> mtreinish: ok, I'll work on that.
17:37:25 * afazekas probably without this the havana neutron full is not able to pass: https://review.openstack.org/#/c/73301/
17:37:55 <mlavalle> we already have the bp
17:38:11 <mlavalle> that's really all I have
17:38:26 <mtreinish> mlavalle: ok
17:38:47 <mtreinish> let's move on to the next topic
17:38:50 * mlavalle eager to see marun topic
17:38:52 <mtreinish> #topic API tests in the Neutron tree (marun)
17:39:00 <mtreinish> marun: you're up
17:39:21 <marun> Hopefully interested parties have seen the patches in question and have had a chance to digest.
17:39:42 <mtreinish> marun: you started a ML thread about it too right?
17:39:54 <marun> sdague was the only one to respond on the ML as to the wisdom of delegating responsibility to Neutron for the api tests
17:40:14 <marun> My hope in submitting these patches was to get people thinking about things, so it was successful in that regard.
17:40:31 <mtreinish> marun: well I tend to agree with all of sdague's points on the ML
17:40:36 <marun> I'm not arguing for getting lax wrt api testing.
17:40:51 <marun> Whether they are in Tempest or Neutron shouldn't matter from a quality perspective as far as I am concerned.
17:41:07 <marun> There is nothing stopping openstack-qa folk from providing oversight in the neutron tree
17:41:21 <marun> Currently it's the reverse - neutron folks have to provide oversight for the tempest tree
17:41:41 <marun> I think the current approach has a fundamental problem, though.
17:41:51 <marun> i think anything short of integration testing should be a project responsibility
17:41:55 <marun> i think the danger of tempest is that if its scope creeps such that it takes on too much responsibility for project-specific testing, it will actually have a negative impact as project committers get used to not caring about certain types of testing
17:42:14 <dkranz> marun: I agree there is an issue
17:42:17 <marun> (out of sight, out of mind as it were)
17:42:30 <mtreinish> marun: there is nothing stopping neutron from having it's own api testing
17:42:45 <mtreinish> but there is value in having 2 separate tests validating the api
17:42:51 <mtreinish> because it should never change
17:43:14 <marun> mtreinish: Do you really think that 2 testing efforts are going to equal the quality of one focused one?
17:43:16 <sdague> marun: so you think that tempest core should have +2 on all core projects?
17:43:19 <dkranz> mtreinish: From a technical standpoint I am not sure that is true about having value in 2
17:43:43 <malini> hmm..sdont the api tests in the project have a different intent from the api tests in tempest?
17:43:47 <marun> sdague: I would support that if you think it's necessary.
17:43:48 <sdague> because the problem is the N-way problem
17:44:02 <dkranz> malini: We have never been clear about that.
17:44:06 <sdague> right now every project should also keep an eye on tempest
17:44:18 <marun> sdague: But I would suggest that a better solution would be isolating qa-critical tests and ensuring that only qualified reviewers have oversight on them.
17:44:18 <sdague> but if you invert it, tempest devs have to keep an eye on *all* projects
17:44:44 <marun> sdague: We manage 3rd party plugins in a similar way today.
17:45:04 <malini> dkranz: it'll be good to think/talk abt it at some point. it's something I have been struggling with when adding new test for marconi..what shoud be in tempest vs not
17:46:24 <marun> so, maybe we can discuss further on the mailing list?
17:46:27 <dkranz> malini: Yes, this was raised briefly at the last summit but we need to nail that down
17:46:42 <marun> I'd like to at least discuss the possibility of devolving responsibility for api testing to neutron.
17:46:45 <mtreinish> marun: yeah I think that would be the best place for it
17:46:58 <marun> And I haven't heard from anybody what exactly would be necessary for that to happne.
17:47:09 <marun> I don't think it's constructive to say 'NO'
17:47:18 <marun> I'd like to hear 'That could be possible IFF'
17:47:27 <marun> And then we can see if it's a possibility
17:47:43 <marun> (and maybe the answer is still no, but at least we've explored things)
17:47:44 <afazekas> IMHO tempest should not depend directly on the neutron source tree
17:47:56 <marun> afazekas: that's a NO
17:48:11 <sdague> so can we back up to the actual problem statement?
17:48:24 <sdague> because I'm not sure I understand what that is
17:48:32 <marun> repeating myself: : i think the danger of tempest is that if its scope creeps such that it takes on too much responsibility for project-specific testing, it will actually have a negative impact as project committers get used to not caring about certain types of testing
17:48:33 <sdague> I see lots of solutions being argued
17:48:58 <marun> api testing residing in tempest has put it out of neutron folks minds.  it's an afterthought.  we can't merge api tests until after code has merged
17:49:30 <sdague> marun: ok, so that's a problem of cross project dependencies
17:49:49 <sdague> which is a narrower problem, and not neutron specific
17:50:02 <sdague> the ability to tell zuul patches across repos depend on each other
17:50:08 <marun> sdague: ?
17:50:08 <sdague> so you could write them all at the same time
17:50:21 <marun> sdague: managing tests outside of a repo sucks
17:50:31 <marun> sdague: having to coordinate cross-project merges sucks
17:50:32 <SergeyLukjanov> any chance I could have 5 mins in the end of meeting for savanna testing?
17:50:44 <sdague> marun: managing 30 independent repos sucks :)
17:50:59 <mtreinish> SergeyLukjanov: it might be tight we've got 10 min left and another topic on the agenda still
17:51:00 <marun> sdague: what is the benefit, exactly, of having tempest maintain tests that are neutron-specific?  That don't actually require integration with other projects to run?
17:51:02 <sdague> and we have to coordinate cross project merges all the time
17:51:10 <mtreinish> sdague, marun: I think we're running out of time
17:51:14 <sdague> mtreinish: sure
17:51:20 <mtreinish> can we take this to the ML and/or -qa
17:51:22 <SergeyLukjanov> mtreinish, ok, will go the -qa channel ;)
17:51:33 <dkranz> I think maruns points should be discussed more
17:51:34 <sdague> marun: because you can validate your cloud with 1 project
17:51:40 <sdague> and not have to git pull all the projects
17:51:48 <afazekas> ML + next meeting
17:51:54 <mtreinish> ok so the probably the last topic for today:
17:51:57 <mtreinish> #topic Graduation requirements for incubated project w.r.t Tempest (malini)
17:52:01 <marun> sdague: why is that such a cost?  the alternative is clearly worse
17:52:10 <SergeyLukjanov> mtreinish, oh, nice topic :)
17:52:14 <sdague> marun: I don't think the alternative is clearly worse
17:52:19 <mtreinish> malini: you're up
17:52:24 <malini> thanks!
17:52:24 <sdague> and that's where we disagree, which is fine
17:52:27 <malini> To give some background, I am working on a patch that will add marconi tests to tempest.
17:52:27 <malini> https://review.openstack.org/#/c/63449/ adds a basic API test. I am hoping this will meet our graduation req as far as tempest goes. But the TC wording is really vague on what is good enough. So wanted to get some thoughts on if additional API tests & scenario tests can be a WIP while/after we graduate.
17:52:36 <marun> sdague: git pull is mechanical.  test maintenance is _not_
17:53:00 <malini> We have a few other projects getting ready to graduate, so it'll be good to make the tempest reqs clear
17:53:04 <dkranz> marun: I need to think about this more
17:53:20 <dkranz> marun: I have suggested similar in the past
17:53:24 <sdague> this has real implications for validation of OpenStack on the Defcore side as well
17:53:24 <mtreinish> malini: So personally I'm fine with the base glue being there and adding tests iteratively over time
17:53:54 <sdague> anyway, it's a longer conversation, and I honestly think we need a summit session for it, because the implications are pretty far reaching to change this
17:54:11 <mtreinish> sdague: which topic? :)
17:54:21 <sdague> sorry, marun's :)
17:54:24 <malini> I wud love a summit session for marconi ;)
17:54:36 <malini> just kidding
17:55:11 <malini> But the idea behind my topic was to make the grad reqs clearer for trmpest
17:55:25 <mtreinish> malini: as long as the basics are there then at graduation we can add the project into the integrated gate with the other project
17:55:32 <mtreinish> even if the api coverage isn't there
17:55:40 <mtreinish> full api coverage
17:55:51 <malini> cool..& fyi we have really good api coverage in the marconi functional test suite
17:55:53 <sdague> malini: yeh, the big issue is getting it devstack enabled
17:56:14 <sdague> the point being that we can then know in the basic gate if a keystone change broke marconi
17:56:26 <malini> cool…That clears up things for me
17:56:34 <sdague> so it looks like it's not running yet, right - http://logs.openstack.org/49/63449/15/check/check-tempest-dsvm-full/44af652/console.html#_2014-02-13_15_59_20_047
17:56:44 <sdague> we still need another devstack fix?
17:56:48 <malini> not yet..But I am working on it
17:56:58 <malini> it is possibly the service name I am using in tempest
17:57:05 <malini> afazekas has been helping me with this
17:57:34 <sdague> so I just +Aed this - https://review.openstack.org/#/c/73100/
17:57:58 <malini> thanks!
17:58:09 <malini> mtreinish: thts it from me for the topic
17:58:16 <mtreinish> malini: ok cool
17:58:31 <mtreinish> well with <2 min left I think we'll just end here for today
17:58:34 <afazekas> malini: https://review.openstack.org/#/c/73100/1  may cause  you need to update the tempest change again
17:58:37 <mtreinish> thanks everyone
17:59:06 <sand2rock> hi
17:59:07 <mtreinish> #endmeeting