17:00:19 <jaypipes> #startmeeting qa
17:00:20 <openstack> Meeting started Thu Mar  7 17:00:19 2013 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:23 <openstack> The meeting name has been set to 'qa'
17:00:30 <jaypipes> Good morning/evening QAers.
17:00:33 <mtreinish> morning
17:00:37 <mlavalle> hi
17:00:39 <andreaf> hi
17:00:42 <malini> hello!
17:00:42 <dwalleck> howdy
17:01:07 <jaypipes> sorry for missing last week's meeting...
17:01:27 <jaypipes> Been pleased to see a bunch of progress in code reviews and new test pushes.
17:01:42 <jaypipes> davidkranz: around?
17:01:50 <davidkranz> jaypipes: Just back.
17:01:58 <jaypipes> sdague: around?
17:02:02 <jaypipes> rohitk: evening.
17:02:10 <rohitk> jaypipes: morning :)
17:02:44 <jaypipes> OK, so dhellmann mentioned a proposed design summit session that I thought was a great idea...
17:02:54 <sdague> jaypipes: yes
17:02:55 <donaldngo_hp> hello all
17:02:57 <jaypipes> #topic Getting Started with Tempest design summit session
17:03:03 <jaypipes> donaldngo_hp, sdague: mornin.
17:03:04 <sdague> oh, right meeting :)
17:03:13 <jaypipes> #link https://etherpad.openstack.org/havana-adding-projects-to-tempest
17:03:38 <chunwang> hi
17:03:48 <jaypipes> Basic point of the session is to help Ceilometer and Heat folks get familiar with Tempest and help them start to add integration tests for Ceilo and Heat
17:04:02 <jaypipes> Which of course I think is a great idea!
17:04:22 <davidkranz> jaypipes: Yes, looks good. But first *we* need to decide what a smoke test is :)
17:04:26 <jaypipes> But, naturally, there's a bit of a learning curve for new projects learning how Tempest does things, and this session should help in that regard
17:04:38 <jaypipes> davidkranz: lol, indeed. could have a whole sesion on that ;)
17:04:48 <rohitk> and Tempest has been shaping up quite awesome in the last six months for more contributions
17:04:53 <donaldngo_hp> jaypipes: would be good for new comers to tempest as well
17:05:03 <jaypipes> FYI, design summit proposal is here:
17:05:05 <jaypipes> #link http://summit.openstack.org/cfp/details/87
17:05:12 <jaypipes> donaldngo_hp: yup, totally.
17:05:23 <sdague> jaypipes: I agree, intro to tempest would be great
17:05:38 <sdague> do we have an idea on how long our track is going to be?
17:05:43 <sdague> or the track structures in general
17:05:56 <jaypipes> donaldngo_hp: and I think that it would be really useful to devote at least half the session time to the final bullet point on the page... a walk through of adding real test cases for Ceilo and Heat
17:06:11 <jaypipes> that way we give those teams a good start
17:06:20 <jaypipes> ravikumar_hp: morning :)
17:06:22 <davidkranz> sdague: "Intro to Tempest" could be on a non-design-summit track
17:06:29 <ravikumar_hp> good morning Jay
17:06:36 <sdague> davidkranz: it could be, but that's closed
17:06:36 <jaypipes> sdague: no constraints, AFAIC
17:06:39 <donaldngo_hp> jaypipes: yea maybe get a test checked in, reviewed, and merged into tempest if possible in that session
17:07:02 <sdague> jaypipes: ok cool
17:07:22 <sdague> it's been one of those weeks where I've not been able to spend any time doing "real work"
17:07:28 <jaypipes> donaldngo_hp: yup, we could even break things into a double session, with the first part a more formal walkthrough of the existing tempest layout and concepts and the following session be actually constructing the first couple test cases for each project.
17:07:52 <jaypipes> sdague: heh, understood :)
17:08:03 <donaldngo_hp> jaypipes++
17:08:16 <mlavalle> jaypipes: ++
17:08:16 <sdague> could we decide next week's QA meeting is a design summit planning one, and give everyone a couple of days to propose remaining ideas for it?
17:08:17 <andreaf> jaypipes: +1 it's a good idea to have a formal walkthrough
17:08:38 <sdague> then we can figure out what's missing from the track that we really should have
17:08:38 <donaldngo_hp> we'll have to wait around for the jenkins run to complete 30-40 minutes
17:08:49 <jaypipes> andreaf: yeah, I think so too, so we have at least some material to save and publish uip on openstack.org for interested parties in the future.
17:09:15 <jaypipes> sdague: yes, totally. I think that's a great idea.
17:09:34 <davidkranz> sdague: Sounds good.
17:09:39 <jaypipes> #action jaypipes to send out email to dev and QA list for input on QA track and pointing to next week as the meeting to finalize
17:09:43 <malini> Pardon my ignorance..But is Tempest going to be/is the  framework for individual projects tests as well ?
17:09:53 <andreaf> jaypipes: do we have a starting point already for such documentation / session?
17:09:59 <jaypipes> malini: integration tests, not unit tests.
17:10:13 <jaypipes> andreaf: https://etherpad.openstack.org/havana-adding-projects-to-tempest
17:10:16 <sdague> malini: right, unit tests go back in the project trees directly
17:10:17 <rohitk> jaypipes: I just added a section in there that I feel could be documented
17:10:33 <jaypipes> rohitk: excellent, thx :)
17:10:37 <davidkranz> jaypipes: I think malini question is relevant.
17:10:53 <davidkranz> jaypipes: Most of the tempest tests are actually functional regression tests.
17:11:09 <davidkranz> jaypipes: There is more overlap with unit tests than would be ideal.
17:11:28 <davidkranz> I discussed this with ayoung in the context of new keystone tests.
17:11:29 <sdague> davidkranz: yeh, well I think that's design summit topic material as well
17:11:47 <davidkranz> sdague: Agreed.
17:11:50 <sdague> in reality new comers need to realize we haven't figured this all out yet :)
17:11:58 <jaypipes> davidkranz: I view them as functional integration tests because Tempest isn't mocking or assuming anything about the environment... it's just testing that the environment you run it against functions appropraiatyely.
17:11:59 <sdague> so we'll have some guidelines on the good kinds of stuff to day
17:12:11 <dwalleck> jaypipes: ++
17:12:12 <sdague> to add... that should be
17:12:25 <jaypipes> davidkranz: I agree about the overlap of regression testing and unit tests, true.
17:12:36 <jaypipes> davidkranz: if you are referring to negative tests currently in tempest
17:12:47 <ayoung> We have a couple needs/process improvements we can do on this
17:12:47 <malini> sdague: tht is good to know :) I am an openstack newbie & was having a hard time figuring out the Openstack way of doing QA
17:13:00 <davidkranz> jaypipes: Yes, and also that as both unit tests and tempest tests try to become more complete there is potential wasted effort.
17:13:02 <jaypipes> malini: embrace the chaos :)
17:13:12 <sdague> malini: yeh, we evolve it over time to get better on each release
17:13:15 <jaypipes> davidkranz: completely agreed.
17:13:26 <davidkranz> jaypipes: I will put in a summit topic for this.
17:13:34 <jaypipes> davidkranz: I was going to recommend that, thank you.
17:13:35 <sdague> ok, so lets table the rest of this for next week - for summit discussions
17:13:49 <jaypipes> ++
17:13:56 <jaypipes> sdague: you lead, sir.
17:13:58 <sdague> what about things we need to get on top of right now? like important reviews or bugs
17:14:21 <jaypipes> sdague: since I was absent last week, please go ahead and lead the meeting, since I've been away from reviews.
17:14:35 <jaypipes> sdague: sorry to put you on the spot :(
17:14:36 <donaldngo_hp> can we get this blueprint approved: https://blueprints.launchpad.net/tempest/+spec/update-expected-exception-tests
17:14:56 <sdague> oooff, as have I most of the week. :)  anyway... what about the v3 keystone tests?
17:15:14 <jaypipes> sdague: shall we consider donaldngo_hp's BP first?
17:15:16 <sdague> there were some tests in the queue, but they changed v2 to v3, so dropped v2 testing
17:15:17 <sdague> sure
17:15:44 <jaypipes> I personally have no problems supporting the BP. Glad to see agreement on doing exception assertions.
17:15:44 <sdague> I'm +1 on that blueprint
17:15:44 <ravikumar_hp> sdague: we are following folders - V2 & v3
17:16:07 <jaypipes> OK, ghow about this... does anyone OBJECT to the above blueprint?
17:16:11 <jaypipes> #link https://blueprints.launchpad.net/tempest/+spec/update-expected-exception-tests
17:16:21 <rohitk> ravikumar_hp: Is this also going to remove the usage of the exception decorators?
17:16:39 <jaypipes> donaldngo_hp: ^^
17:16:46 <sdague> rohitk: can you give an example?
17:17:06 <jaypipes> sdague: rohitk is asking whether the @raises decorator is going away.
17:17:06 <donaldngo_hp> yea what would be an example
17:17:18 <rohitk> jaypipes: right
17:17:29 <donaldngo_hp> this blueprint basically reduces 4+ lines of code into 1
17:17:55 <mtreinish> rohitk: do we have raises decorators in tempest?
17:17:57 <rohitk> I was interested to know if we should follow a standard practice
17:17:58 <jaypipes> donaldngo_hp: nah... rohitk is asking about the third way we currently assert exceptions, which is using the @raises unittest2/nose decorator
17:18:01 <sdague> rohitk: is that used somewhere?
17:18:22 <rohitk> sdague: I haven't checked lately, but just recollecting, do correct me if they're already removed
17:18:33 <sdague> rohitk: they aren't in there that I can see
17:18:51 <davidkranz> Ditto.
17:18:51 <rohitk> sdague: that's great then, thanks for correcting
17:18:53 <sdague> I think donaldngo_hp's bp is a good cleanup
17:18:58 <rohitk> +1
17:18:59 <sdague> and the right direction
17:19:15 <donaldngo_hp> maybe we can enforce this through tox of having a singlular way of doing exception assertions?
17:19:16 <sdague> I could see us doing additional things after that, but let's get this cleanup done
17:19:23 <jaypipes> OK, sounds like an approve, then? shall I approve it?
17:19:26 <donaldngo_hp> for future code submissions
17:19:29 <sdague> jamespage: +2
17:19:36 <sdague> jaypipes: +2
17:19:45 <jaypipes> done.
17:19:52 <jaypipes> donaldngo_hp: go for it, man :)
17:19:53 <donaldngo_hp> my guys can take care of this please assign it to me
17:20:08 <donaldngo_hp> cant seem to edit it
17:20:13 <sdague> donaldngo_hp: yeh, lets focus on things that we can do near term. Too often we come up with bigger plans that don't get done, this one is really straight forward, which is nice
17:20:16 <jaypipes> donaldngo_hp: done.
17:20:31 <donaldngo_hp> cool
17:20:40 <jaypipes> #topic Keystone tests v2/v3
17:20:47 <donaldngo_hp> sdague: agreed
17:20:50 <jaypipes> sdague: you had a q for ravikumar_hp
17:21:16 <sdague> yeh, ravikumar_hp, you are going to redo the patches with supporting both versions of the api, right?
17:21:43 <sdague> that was the major complaint on the previous versoin
17:21:54 <ravikumar_hp> sdague: yes. Based on the review feedback , going to resubmit
17:22:08 <sdague> ok, great
17:22:11 <ravikumar_hp> that supports both V2 & v3
17:22:45 <sdague> so I'm good on that, once we see the new reviews
17:23:07 <jaypipes> alrighty.
17:23:18 <sdague> maurosr / mtreinish: you guys have the link to the list of needed tests?
17:23:31 <mtreinish> sdague: this one https://etherpad.openstack.org/MissingTempestTests
17:23:33 <mtreinish> ?
17:23:37 <sdague> yep
17:23:41 <dolphm> i don't follow all the tempest reviews, but would appreciate being added to anything relevant :)
17:23:47 <sdague> that's worth sharing around for people looking for easy stuff
17:23:54 <sdague> dolphm: awesome, will do
17:24:21 <mtreinish> sdague: I think everything currently on it is covered or invalid
17:24:25 <mtreinish> except for some xml tests
17:24:35 <sdague> mtreinish: ok, I guess we need some more generation :)
17:25:00 <jaypipes> ++
17:25:18 <sdague> ok, honestly, that was much topic list :) sorry for not being more organized this week
17:25:19 <mtreinish> sdague: I put a link for a quick howto on the top of that page
17:25:29 <donaldngo_hp> i thought there was going to be a write up on how to generate this document?
17:25:42 <jaypipes> mtreinish: ty, very useful.
17:25:45 <mtreinish> donaldngo_hp: https://etherpad.openstack.org/CoverageAnalysisHowto it's on the top of that missing test list
17:26:11 <mtreinish> donaldngo_hp: honestly it's pretty straightforward, just a bit time consuming
17:26:43 <jaypipes> We need to do similar for Keystone and Quantum, IMHO... but I believe it would need to be manual, right mtreinish, since those projects do not include the coverage middleware, IIRC
17:27:14 <sdague> right, that's a good thing to float in maybe a cross project track at summit
17:27:17 <mtreinish> jaypipes: yeah, the coverage extension is only on nova
17:27:22 <jaypipes> kk
17:27:23 <donaldngo_hp> mtreinish: thanks, I'll try to give it a go through and update the etherpad. how often should we do this?
17:27:28 <sdague> as it would be nice to get something like the coverage extension into the other projects for thiat
17:27:35 <jaypipes> right
17:28:16 <mtreinish> donaldngo_hp: not very frequently, it shouldnt change that much once we get good coverage
17:28:38 <jaypipes> mtreinish: lol, at the rate the upstream projects add new stuff... ;)
17:28:46 <sdague> heheh
17:28:57 <sdague> yeh, we only added 20 extensions to nova this cycle :P
17:29:06 <jaypipes> right :)
17:29:34 <jaypipes> there are a bunch of quantum-related reviews in the queue..
17:30:04 <sdague> oh, quantum's a good topic. It would be nice if we could get some engagement from that team to get full tempest functional
17:30:47 <jaypipes> sdague: I've seen a few folks talking -- mnewby and a few others -- but it still seems like full tests are a ways away.
17:30:53 <sdague> yeh
17:31:13 <sdague> maybe for summit we try to get a tempest session in the quantum track?
17:31:19 <sdague> drive the conversation over there
17:31:31 <jaypipes> sdague: probably a good idea, yes.
17:31:52 <mlavalle> sdague: I attend the Quantum meeting every week. I will birng the proposal up next Monday
17:32:03 <sdague> mlavalle: thanks!
17:32:03 <jaypipes> mlavalle: thx!
17:32:51 <jaypipes> OK, well besides open reviews, which we all need to get to, are there any other open topics folks wish to discuss?
17:32:56 <jaypipes> #topic Open discussion
17:33:22 * jaypipes wonders how the runtime is doing?
17:33:25 <ravikumar_hp> jaypipes: can we add tests for incubated projects
17:33:33 <ravikumar_hp> like DbaaS
17:33:35 <sdague> jaypipes: on sec, I can tell you
17:33:42 <jaypipes> ravikumar_hp: DBaaS is incubated?
17:33:48 <ravikumar_hp> of course separate folders .
17:33:52 <sdague> ravikumar_hp: yeh, I don't think it's incubated
17:34:10 <sdague> so I'd say we stay away from that for now
17:34:12 <ravikumar_hp> I am not sure . Reddwardf , and Atlas LbaaS
17:34:28 <jaypipes> ravikumar_hp: Atlas surely is not... it's essentially being replaced with Quantum LBaaS
17:34:31 <sdague> until it's in the incubation path formally, I don't think we should be putting things in tempest
17:34:46 <sdague> there is only so much review time to go around
17:34:50 <davidkranz> sdague: Agreed, unless a project member wants to step forward.
17:35:19 <jaypipes> ravikumar_hp: I'd say doing it for DBaaS would only be slightly easier since it is, AFAIK, able to be setup via Devstack.
17:35:38 <jaypipes> ravikumar_hp: that said, I don't really feel like making the undertaking on that until incubation is formal
17:35:39 <sdague> davidkranz: even if they do, we have to put some bounds on it, because the core team still needs to take responsibility for code in tempest
17:35:46 <jaypipes> like sdague said, only so much time...
17:35:54 <ravikumar_hp> okay . we will wait
17:36:08 <sdague> jaypipes: runtime is about the same 35 - 40 mins on the hp cloud nodes
17:36:14 <jaypipes> sdague: kk
17:36:25 <jaypipes> sdague: are we not yet doing parallel testr runs?
17:36:29 <mtreinish> sdague: what about the rackspace cloud?
17:36:31 <sdague> we are not
17:36:34 <jaypipes> chris yeoh still working on that?
17:36:44 <sdague> mtreinish: rackspace isn't in the pool right now
17:36:44 <jaypipes> mtreinish: it's similar it seems
17:37:00 <jaypipes> sdague: no?
17:37:06 <jaypipes> interesting... when did that happen?
17:37:10 <mtreinish> sdague: ok
17:37:14 <sdague> jaypipes: so in order to do tempest parallel sanely, we need to get to central resource management
17:37:22 <jaypipes> sdague: yup.
17:37:33 <sdague> so that's mostly going to be design summit discussions to figure out how we finish it
17:37:39 <jaypipes> sdague: ok.
17:38:02 <jaypipes> sdague: though having tests run entirely in their own tenant should enable parallel execution, no?
17:38:04 <sdague> between now and then we're trying to refactor so things like the create_server / create_image calls go to some central resource allocation / deallocation points
17:38:23 <sdague> jaypipes: it could, but it won't be faster
17:38:31 <jaypipes> sdague: how so?
17:38:47 <sdague> because of the amount of work we do in setUpClass
17:39:02 <sdague> and testr splits at the test level, not the class level
17:39:11 <jaypipes> sdague: right, I understand that part
17:39:14 <sdague> it's a devils in the details thing
17:39:17 <jaypipes> ya
17:39:37 <sdague> so if we do some progressive refactoring of the resource allocators, so we get them all to a couple of central points
17:39:49 <afazekas> We could give an additional chance to nose parallel.
17:39:50 <sdague> we can then use those to build our own testr "scheduler"
17:40:00 <sdague> which will give us the optimizing we need
17:40:04 <jaypipes> k
17:40:25 <sdague> which, honestly, is sane regardless, because we'll have central cleanup
17:40:32 <jaypipes> afazekas: you are welcome to show a POC for that. I tried three times and ran into bugs in nose each time.
17:40:59 <sdague> yeh, the reality is moving the resource allocations to common paths will help any solution
17:41:12 <sdague> and then figured we could hash out the last bits in person at summit
17:41:28 <jaypipes> sure
17:41:56 <sdague> so, sadly, the promiss of having it for grizzly was not true
17:41:57 <afazekas> jaypipes: Ok
17:42:04 <sdague> but havana-1 seems doable
17:42:17 <jaypipes> sdague: underpromise... overdeliver.
17:42:27 <jaypipes> sdague: let's say Havana final ;)
17:42:31 <sdague> heh
17:42:31 <sdague> sure
17:43:16 <jaypipes> OK, any other topics before we break?
17:43:28 <davidkranz> jaypipes: Nope.
17:43:31 <sdague> not from me
17:44:05 <jaypipes> OK, folks have a good one. I'll start on the email about next week's meeting and the design summit.
17:44:07 <jaypipes> #endmeeting