17:03:05 <jaypipes> #startmeeting
17:03:06 <openstack> Meeting started Thu Jul  5 17:03:05 2012 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:37 * jaypipes currently doing the code review on rohit's branch.
17:04:08 <dwalleck> davidkranz: Not completely yet. I know it's a slippery slope to say which extensions are core and which are not. Unless there's a good criteria we can use
17:04:26 * dwalleck is coughing his lungs back out
17:04:46 <jaypipes> davidkranz, dwalleck: I don't have any problems per-se with dwalleck's smoke testing refactor/speedup branch. I still prefer smoke tests to be slighly different, with a sequential set of steps that follow numerous API calls (not just limited to servers or images or keypairs, etc
17:05:41 <davidkranz> jaypipes: I think that makes some sense. We have to make sure that the smoke tests keep up with new apis.
17:05:48 <dwalleck> jaypipes: Well, this was an initial, immediate solution
17:06:46 <davidkranz> dwalleck: Sure. The reason I like Jay's suggestion is that the requirement that smoke test cases have to be subsets of full coverage is limiting.
17:06:48 <jaypipes> dwalleck: oh, I totally understand (and support the patch)
17:07:16 <davidkranz> dwalleck: If we adopt the scripted actions then we can control the use of expensive functions in a better way I think.
17:07:38 <dwalleck> davidkranz: What do you mean by scripted actions?
17:07:42 <jaypipes> davidkranz: sorry, could you elaborate on "adopt the scripted actions"?
17:07:52 <jaypipes> davidkranz: oh, I think I know what you mean...
17:07:56 <davidkranz> dwalleck: Sorry, I just meant the kind of thing Jay was saying.
17:07:59 * dwalleck is getting confused. I need more cold medicine. Or less
17:08:03 <jaypipes> davidkranz: you mean "adopt practice of using sequential series of common actions"
17:08:05 <jaypipes> :)
17:08:11 <davidkranz> jaypipes: Yes :)
17:08:28 <jaypipes> dwalleck: you shouldn't live with a cat if you have allergies :P
17:08:59 <dwalleck> My concern is how we control the flow of that. I know the numerical test ordering works, but I'd like to see something more explicit if we're really going to go in and fix this
17:09:14 <dwalleck> jaypipes: Cats came with the fiance, no choice :-)
17:09:39 <jaypipes> dwalleck: I'm actually more in favour of creating separate smoke tests entirely from the positive./negative/detailed test cases we currently have...
17:10:06 <jaypipes> dwalleck: and these smoke tests would be simply sequential test methods (named test_00X_blah) that would exercise the default or common API actions
17:10:30 <dwalleck> jaypipes: Right. I think what I'm trying to ask is what they will look like
17:10:35 <dwalleck> which you just said :-)
17:10:56 <davidkranz> dwalleck: I think Jay included an example in the code Rohit took over.
17:11:07 <jaypipes> cool. well, actually, rohit's patch actually used my original smoke test branch as a base...
17:11:25 <jaypipes> so it might be worth looking into that next (after dwalleck's intermediate patch goes in)
17:11:52 <jaypipes> and once we have that, we should modify the jenkins job to fire ./run_tests.sh --smoke instead of the whole shebangh
17:11:58 <jaypipes> you guys agree with that?
17:12:08 <jaypipes> so, these steps:
17:12:18 <davidkranz> jaypipes: But we would still run the whole thing nightly, right?
17:12:22 <jaypipes> 1) get dwalleck's patch for decreasing time of smoke down
17:12:31 <jaypipes> 2) edit the jenkins job to run smoke only
17:12:42 <jaypipes> 3) work on merging in rohit's patch
17:12:55 <jaypipes> 4) set up the separate jenkins "slave jobs" that would fire the entire test suite
17:13:08 <jaypipes> that amenable to you guys?
17:13:14 <davidkranz> jaypipes: Sounds good
17:13:19 <jaypipes> dwalleck: ?
17:13:41 <davidkranz> jaypipes: What about soft reboot? It takes 2.5 minutes for me. Doesn't it kill you, and should it be in the smoke test?
17:13:55 <jaypipes> davidkranz: ah, goood point...
17:14:21 <jaypipes> dwalleck: there was one tiny mistake in your patch to fix. What about fixing the soft reboot thing while you fix that mistake?
17:14:25 <dwalleck> In general I'm on board. I have concerns with using test naming as a means of ordering
17:14:34 <jaypipes> dwalleck: basically, make sure only hard reboot is in any smoke tests...
17:14:38 <davidkranz> i don't understand how Daryl got the time down so low if he has the same issue with soft reboot.
17:14:52 <jaypipes> davidkranz: still takes around 5 minutes IIRC
17:14:55 <dwalleck> jaypipes: Not sure what you mean by fix soft reboot. it's currently commented out
17:15:05 <dwalleck> Fix as in add back in?
17:15:05 <jaypipes> hmmm...
17:15:24 <jaypipes> one sec, lemme double-check.
17:15:43 <davidkranz> dwalleck: Arg. I forgot I did that!
17:16:02 <rohitk> jaypipes: We wouldn't want to see any negative tests in smoke suite right?
17:16:13 <davidkranz> rohitk: Right.
17:16:15 <jaypipes> rohitk: correct
17:16:24 <jaypipes> davidkranz: yes, dwalleck is correct. it is skipped.
17:16:33 <jaypipes> davidkranz: verified. line 69 in test_servers_actions.py
17:24:03 <jaypipes> ok, ok.
17:24:03 <davidkranz> jaypipes: Yeah. I withdrew the skipping from my first commit but then we agreed it should be skipped because the spec is such a mess anyhway.
17:24:04 <jaypipes> yup
17:24:05 <jaypipes> alright, you three (rohit, david, daryl): are we good to try and get #1 through #3 above done TODAY?
17:24:07 * jaypipes can handle the review on rohitk's branch as well as the jenkins changes...
17:24:07 <jaypipes> in fact, I can get the jenkins changes sorted in the next hour or so...
17:24:08 <jaypipes> going to ask jeblair for some help in resetting the data on the Tempest job after it switches to run smoke tests by default
17:24:09 <davidkranz> jaypipes: Sounds good.
17:24:10 <davidkranz> What about the issue of gating Tempest itself?
17:24:12 <jaypipes> davidkranz: Tempest  is already gated, but only on pep8 :(
17:24:12 <davidkranz> We still need a way to make sure tempest doesn't get broken.
17:24:12 <davidkranz> But we can't really run the whole thing.
17:24:13 <jaypipes> davidkranz: yeah, but unfortunately, it's not so easy... but I will work with jeblair on some options
17:24:13 <jaypipes> davidkranz: the reason is because of the interdependencies in the way that the job gets triggered...
17:24:14 <jaypipes> but regardless, I will chat with the CI team and see what we can do.
17:24:14 <davidkranz> jaypipes: We may have to live with a nightly run.
17:24:14 <dwalleck> brb, cat's on fire
17:24:15 <jaypipes> lol
17:24:15 <davidkranz> jaypipes: I just want there to be clarity on code reviews.
17:24:16 <jaypipes> davidkranz: perhaps -- but let's wait and see what's possible.
17:24:16 <jaypipes> davidkranz: agreed 100%
17:24:18 <davidkranz> jaypipes: sure, let's see. BUt I think we should require that code reviewers *Not* have to think they are gonig to find every runtime bug that might happen.
17:24:19 <jaypipes> davidkranz: ++
17:24:20 <davidkranz> jaypipes: We will have to rely on the judgement of submitters that they run the right subset before checkin.
17:24:20 <jaypipes> davidkranz: let me chat with jeblair about the options here... perhaps we could have a commit to Tempest trigger the entire test suite and a commit to the core projects trigger only the smoke run? And also do nightly snapshot against full tempest?
17:24:38 <davidkranz> jaypipes: That would be ideal but I am concerned that running the entire test suite will still consume too many resources.
17:25:17 <davidkranz> jaypipes: We could have a 'slow' attribute that is skipped by everything but nightly tests. I have done that in past projects.
17:26:21 <jaypipes> davidkranz: but we don't want change to tempest passing the *tempest gate* that don't pass all tempest tests, right?
17:26:50 <jaypipes> davidkranz: the number of changes to tempest right now isn't all that much. I think I'd prefer to err on the side of correctness vs. efficiency for the tempest gate
17:27:14 <davidkranz> jaypipes: OK, let's give it a shot. We can deal with the issue later if necessary.
17:27:26 <jaypipes> indeed.
17:27:34 <jaypipes> JoseSwiftQE: any update for us, my man?
17:27:48 <JoseSwiftQE> I looked over the review just this morning
17:28:10 <JoseSwiftQE> What do you suggest for the rest client re: headers=None?
17:28:43 <jaypipes> JoseSwiftQE: I was just saying that you can remove the =None on that particular line and pass in  headers=headers
17:29:07 <JoseSwiftQE> Ah, so that way it uses in the restclient default headers?
17:29:07 <jaypipes> JoseSwiftQE: since the way it is now, you're overriding anything passed in to the function in the header param..
17:29:12 <jaypipes> right
17:29:23 <JoseSwiftQE> that makes sense, thanks.
17:29:27 <jaypipes> ya, no worries
17:29:43 <davidkranz> Should we move on to the testing Folsom features issue?
17:29:49 <JoseSwiftQE> sure, i'm done
17:29:51 <jaypipes> yes, let's.
17:30:03 <jaypipes> #topic Testing new features when they come around
17:30:12 <jaypipes> davidkranz: floor is yours
17:30:15 <davidkranz> I think the first thing is to get the ptls to tell us when things are ready for "outsider" testing.
17:30:58 <jaypipes> davidkranz: that won't happen unless we bug them :)
17:31:26 <jaypipes> davidkranz: so we need a point person in the QA team (perhaps for each core project) that will bug the PTLs (in status meetings, on IRC, on ML, etc)
17:31:48 <davidkranz> jaypipes: Right.
17:32:24 <davidkranz> jaypipes: But I hope the PTLs will quickly decide it is easier to make a habit of "releasing" features to QA when they are ready in a proactive way.
17:32:38 <dwalleck> cat longer on fire, back
17:32:52 <rohitk> davidkranz: ++
17:33:07 <jaypipes> davidkranz: agreed. in order for PTLs to decide that we should have our process (of QA validation of new features) by explicit and well-defined on the wiki.
17:33:13 <davidkranz> davidkranz: I don't have a good feel for how the PTLs view the QA situation. Are they happy with the current state of affairs?
17:33:25 <dwalleck> davidkranz: While I agree on the idea, I'm wondering if we shouldn't be a bit more agile and start working on testing as the features are being worked on
17:33:33 <jaypipes> davidkranz: totally different between core projects (which is part fo the problem)
17:33:45 <jaypipes> dwalleck: oh, I agree with that.
17:33:58 <dwalleck> But I know, baby steps
17:34:19 <jaypipes> :)
17:34:24 <davidkranz> dwalleck: I didn't mean to imply otherwise.
17:34:41 <jaypipes> all: http://etherpad.openstack.org/QaNewFeatureValidation
17:34:48 <jaypipes> let's doc it up and then transfer to the wiki...
17:34:51 <dwalleck> davidkranz: well, you said releasing to QA, it just sounded linear
17:34:55 <davidkranz> dwalleck: It's just that until a certain point, they rightly don't want to get bugged by questions about things that are still really in flux.
17:34:56 <dwalleck> fair deal
17:35:46 <davidkranz> dwalleck: Really this means when devs think an active QA involvement would be helpful.
17:35:59 <jaypipes> let's doc it up and then transfer to the wiki...
17:36:10 <jaypipes> all: http://etherpad.openstack.org/QaNewFeatureValidation
17:36:57 <davidkranz> jaypipes: OK. My concern is that in every project I have worked on, there were developers who need to be reminded of the imoprtance of early QA involvement.
17:37:07 <davidkranz> Sometime I was one of those developers :)
17:37:15 <rohitk> Tracking how many bugs Tempest tests have found in Openstack would be a good way to please(or bug) the devs
17:37:18 <jaypipes> davidkranz: sure, no disagreement from me.
17:38:00 <davidkranz> rohitk: Sure. The earlier Tempest is involved, the more bugs it will find.
17:38:31 <rohitk> we could probably tag 'tempest' in such bugs or just denote the test in the bug description(guess most of us do that already)
17:38:56 <rohitk> davidkranz: true
17:39:05 <davidkranz> rohitk: Yeah. I am not sure we need a special tag for that.
17:39:50 <davidkranz> So I think we need the following things:
17:40:09 <davidkranz> 1. Work with PTLs to identify blueprints
17:40:23 <davidkranz> 2. Assign people to follow
17:40:34 <davidkranz> 3. Add QA info to the blueprint
17:40:50 <davidkranz> 4. Document theh process.
17:40:58 <davidkranz> I think we would get a long way with just that.
17:40:58 <jaypipes> davidkranz: before 3., we need to flesh out precisely what information QA should track for each blueprint
17:41:18 <jaypipes> davidkranz: which is the reason for the questions on the etherpad :)
17:41:57 <davidkranz> jaypipes: Right. It would be incredibly helpful if each PTL could give a small amount of mindshare to this.
17:41:58 <dwalleck> I think it would be helpful if the blueprint could describe at least the acceptance criteria for functionality that should work as part of a blueprint
17:42:29 <jaypipes> davidkranz: how about we use Quantum or Glance as a guinea pig here? both of those PTLs have expressed interest in keener coop with QA
17:42:39 <dwalleck> And if not in the blueprint, then somewhere. I think if that's decided between dev and qa, then it makes the testing phase much less painful
17:42:47 <davidkranz> jaypipes: Can you bring this up at one of the weekly meetings?
17:42:56 <jaypipes> well, vishy has too, but Nova is kinda huge... better to start smaller :)
17:43:08 <jaypipes> davidkranz: I can definitely do that (as can you!) :)
17:43:19 <davidkranz> jaypipes: Agreed. Since you used to be the PTL of glance, maybe we should start with that?
17:43:28 <jaypipes> dwalleck: blueprint is the perfect place to log this kind of stuff (QA deliverables)
17:43:33 <davidkranz> jaypipes: Or we could do both.
17:43:46 <jaypipes> we just need to create a template of QA devliverables that QA team members can follow when assigned to a BP
17:44:11 <davidkranz> jaypipes: Yes, let's fill in the etherpad and then discuss more.
17:44:13 <jaypipes> davidkranz: I think perhaps it might be easier to just pick a blueprint from Nova and then use it to create the template we use
17:44:37 <jaypipes> davidkranz: let me choose one. gimme a couple minutes. dwalleck, rohitk, davidkranz in the meantime, please add tot hose etherpad questions
17:45:28 <rohitk> jaypipes: sorry, could you share the etherpad link, bit lost on that
17:45:39 <jaypipes> rohitk: http://etherpad.openstack.org/QaNewFeatureValidation
17:45:42 <rohitk> thanks
17:45:47 <jaypipes> f course :)
17:46:57 <jaypipes> dwalleck, davidkranz, rohitk: what do you guys think of this one? https://blueprints.launchpad.net/nova/+spec/task-management
17:47:24 <jaypipes> I think it's perfect. It's a) important, b) easily testable, and c) represents something that has already been a pain in the ass for the QA teasm
17:47:40 <jaypipes> and Yun is a colleague of mine :)
17:48:27 <davidkranz> jaypipes: ++
17:48:44 <rohitk> jaypipes: good candidate
17:48:47 <dwalleck> yes, that is very helpful!
17:49:15 <jaypipes> ok, excellent.
17:49:18 <davidkranz> So we create a process around this blueprint
17:49:25 <jaypipes> let's use this as the example
17:49:25 <dwalleck> Plus it should help the doc team when they want to document the functionality. win/win/win
17:49:28 <jaypipes> davidkranz: right
17:49:39 <jaypipes> dwalleck: yeah, exactly, especially this particular blueprint!
17:49:51 <davidkranz> When we are ready, we advertise it as a general mechanism and try to get more people to sign on to "own" blueprints.
17:49:54 * dwalleck still cries tears over the admin api
17:50:05 <davidkranz> That moves us in the direction of more "QA in the open".
17:50:07 <jaypipes> just asked Yun to join us...
17:50:20 <jaypipes> need to work with developer as much as possible... that's the whole point in this.
17:50:30 <jaypipes> maoy: hi! :)
17:50:37 <maoy> jaypipes: yo
17:50:44 <jaypipes> maoy: so, you've been chosen as the lucky QA winner this week.
17:50:49 <jaypipes> maoy: :)
17:50:56 <jaypipes> maoy: let me fill you in.
17:51:10 <maoy> jaypipes: is the price a new ipad? jk
17:51:25 <jaypipes> maoy: no, a new etherpad :)
17:51:26 <rohitk> https://review.openstack.org/#/c/9059/1/tempest/tests/whitebox/compute/test_servers.py kind of tests the yun's new state mgmt
17:51:45 <jaypipes> maoy: basically, the QA team (in whose weekly meeting you are now in) is really keen to work better and closer with developers working on blueprints
17:51:56 <jaypipes> maoy: for reference: http://etherpad.openstack.org/QaNewFeatureValidation
17:52:18 <jaypipes> maoy: and we have chosen your blueprint (task-management) as the blueprint to use as an example for better QA coordination with development
17:52:40 <jaypipes> maoy: since you are such a kick-ass engineer, we figured you'd be cool with being a guinea pig ;)
17:53:10 <maoy> jaypipes: i see. sounds good.
17:53:30 <jaypipes> maoy: The QA team wants to make sure that for each major blueprint, there are QA deliverables attached to the blueprint in question that represent tasks for QA and development to work on together.
17:53:37 <jaypipes> maoy: these tasks will mostly be:
17:53:49 <jaypipes> 1) Identifying any changes in API requests or responses
17:53:59 <jaypipes> 2) Identifying any change in call behaviour
17:54:12 <jaypipes> 3) Ensuring tests in Tempest fully cover the change areas
17:54:40 <jaypipes> That's basically it... but we want to be a lot more methodical about how we coordinate. thus, we're using the task-management blueprint as an example
17:54:54 <davidkranz> 4) Ensuring proper allocation of unit tests and tempests tests based on aarchitectural knowledge
17:55:01 <jaypipes> davidkranz: ++
17:55:26 <jaypipes> maoy: which we hope is OK, since most of the work on the blueprint is completed, but we still need to check that functional integration testing is fully covering the new and changed task states
17:56:09 <davidkranz> rohitk: Would you be willing to take this as you have already dived in?
17:56:19 <jaypipes> maoy: OK, so that's the story. Nothing for you to do right now -- just letting you know you will likely see us updating the blueprint with a set of tasks for QA (and some for you) to work on
17:56:29 <rohitk> davidkranz: Sure, I can
17:56:41 <maoy> jaypipes: cool.
17:56:54 <jaypipes> maoy: and hopefully, if all goes to plan, we can present the workflow at the weekly status meeting with the PTL next week.
17:58:06 <jaypipes> davidkranz: the other thing I thought of is: 5) Coordinate all findings with doc team to ensure changes to documentation correspond to changes in code
17:58:38 <davidkranz> jaypipes: Yes, Daryl snuck that in a few minutes ago.
17:58:45 <jaypipes> rock.
17:59:02 <davidkranz> The QA person can also be a more technical interface between doc team and dev.
17:59:13 <jaypipes> OK, well it's time to wrap up the meeting. I feel we got some good stuff done here today...
17:59:18 <jaypipes> davidkranz: ++
17:59:24 <rohitk> jaypipes: question
17:59:30 <jaypipes> rohitk: go for it.
17:59:47 <rohitk> jaypipes: tempest stable/essex hasn't changed since Apr 13
17:59:56 <rohitk> are we backporting core changes into it?
18:00:10 <jaypipes> rohitk: besides the patch that went in today? ;)
18:00:21 <rohitk> jaypipes: yup
18:00:23 <jaypipes> rohitk: feel free to backport changes to it.
18:00:26 <davidkranz> rohitk: I think with all the refactoring it would be pretty painful.
18:00:44 <rohitk> davidkranz: exactly, i found quite painful
18:00:52 <jaypipes> alright fellas, gonna close up this meeting...
18:00:58 <rohitk> jaypipes: okie!
18:00:59 <jaypipes> #endmeeting