17:02:18 <jaypipes> #startmeeting
17:02:19 <openstack> Meeting started Thu May  3 17:02:18 2012 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:21 <donaldngo_hp> good morning
17:02:23 <jaypipes> JoseSwiftQA: heya :)
17:02:26 <davidkranz> Daryll said he will be late or might have to miss the meeting.
17:02:29 <jaypipes> donaldngo_hp: afternoon!
17:03:20 <jaypipes> all: unfortunately, I only just now sent my email to the QA and main mailing list about the smoke test stuff, so I'm happy to move that discussion to next week since I was delayed in getting it out to everyone.
17:04:12 <jaypipes> #link http://wiki.openstack.org/Meetings/QATeamMeeting
17:04:27 <davidkranz> OK, next week it is.
17:04:34 <JoseSwiftQA> coolbeans :D
17:04:39 <jaypipes> #topic Status of the devstack tempest Jenkins job
17:05:05 <davidkranz> jaypipes: Do tell.
17:05:05 <jaypipes> OK, so there was a bunch of mess that needed to be cleaned up
17:05:11 <jaypipes> :)
17:05:41 <jaypipes> Fixes needed to be made to devstack, the devstack-gate project (part of the openstack-ci project) and to tempest itself.
17:06:05 <jaypipes> The devstack and devstack-gate fixes are now in trunk. The tempest change I just pushed about 20 minutes ago.
17:06:20 <jaypipes> In the meantime, while all this was being sorted, I disabled the Jenkins job
17:06:38 <jaypipes> Since it takes about 45 minutes to run on some of the CI environments (the HP zones...)
17:06:54 <jaypipes> Once the tempest change is approved, I will enable the job again.
17:07:06 <jaypipes> https://review.openstack.org/#/c/7067/
17:07:12 <jaypipes> that is the relevant change...
17:07:44 <jaypipes> The root of the problem was that devstack was creating the volume group with a backing file size of only 2G
17:08:01 <jaypipes> and so the volume list test was failing (since it tries to create 3 1G volumes)
17:08:10 <jaypipes> and leaving those volumes around without cleaning them up...
17:08:50 <davidkranz> OK, will look at that right after the meeting.
17:08:52 <jaypipes> There were also issues with the image tests (using the Glance API, not the compute API) that stemmed from a change in the setup of devstack's Keystone service catalog
17:09:12 <jaypipes> That required a fix in both Glance and Tempest, and those patches are also now both in the trunks
17:09:19 <donaldngo_hp> the volume list tearDownClass fix has been submitted
17:09:33 <donaldngo_hp> and also backported to stable/diablo
17:09:36 <jaypipes> donaldngo_hp: to essex :)
17:09:41 <jaypipes> yes, diablo too
17:09:44 <donaldngo_hp> awaiting review
17:09:52 <dwalleck> sorry, run a bit late
17:09:52 <jaypipes> Bottom line, we should have a fully passing Tempest gating job by end of day today.
17:10:08 <davidkranz> jaypipes: Hooray. Great work.
17:10:16 <jaypipes> and at that point, I believe we should start gating the Temepst project trunk on that job.
17:10:46 <jaypipes> All in agreement for that, I presume?
17:10:57 <Ravikumar_hp> yes Jay
17:10:59 <Ravikumar_hp> +1
17:11:03 <fattarsi> +1
17:11:06 <jaypipes> excellent.
17:11:07 <davidkranz> Agreed
17:11:12 <donaldngo_hp> agreed
17:11:42 <jaypipes> alrighty, per the agenda, let's move on to the next topic
17:11:59 <davidkranz> No blockers as far as I know.
17:12:00 <jaypipes> #topic 2. (David) Strategy for maintaining the stable/essex tempest branch:
17:12:08 <jaypipes> davidkranz: you are up my friend :)
17:12:54 <davidkranz> I think we should backport new tests if they don't have to be rewritten due to config changes and such.
17:13:16 <donaldngo_hp> +1
17:13:55 <Ravikumar_hp> Backport from Master to Essex/stable
17:13:58 <Ravikumar_hp> fine.
17:14:20 <davidkranz> We also need to make sure that tempest stable/essex keeps up with changes to the stable branch team checkins.
17:14:20 <jaypipes> davidkranz: yes, I think that is a reasonable standard
17:14:21 <Ravikumar_hp> but do we need to backport further to diablo/stable?
17:14:33 <davidkranz> Since I volunteered to be on that team I will keep an eye on that.
17:15:01 <jaypipes> Ravikumar_hp: don't *have* to, but if some team wants to manage the diablo branch, they should feel free to do so
17:15:25 <davidkranz> I would not recommend keeping diablo stable active. The stable branch team is not going to approve any change unless it is a security hole or some such.
17:15:44 <davidkranz> I mean to stable/diablo. There was only ever one release of that.
17:15:46 <Ravikumar_hp> davidkranz; +1
17:16:16 <donaldngo_hp> we need some changes in stable/diablo mainly to fix bugs in test clean up
17:16:36 <davidkranz> donaldngo_hp: That's fine. I was talking about a policy going forward.
17:16:47 <jaypipes> ++
17:16:55 <davidkranz> Of course we should be gating tempest on changes to essex/stable too.
17:17:14 <Ravikumar_hp> ++
17:17:35 <davidkranz> If there is no disagreement I think that takes care of agenda item 2.
17:17:36 <jaypipes> davidkranz: we will be.
17:17:47 <jaypipes> davidkranz: the CI jobs are all ready to run
17:18:00 <davidkranz> jaypipes: Excellent
17:18:19 <jaypipes> davidkranz: basically, devstack builds the appropriate branches of the projects based on the branch that the fix is proposed to
17:18:49 <donaldngo_hp> is the devstack builds for essex and trunk only?
17:18:59 <jaypipes> davidkranz: so, the nice thing is we don't need different jobs for different release branches... it's all one job, with environment variables switching the source of the git pulls...
17:19:15 <davidkranz> jaypipes: Great. Set it and forget it.
17:19:17 <jaypipes> donaldngo_hp: no, I think there is a diablo one too.... but I'd have to check with dtroyer
17:19:24 <jaypipes> davidkranz: right
17:19:31 <donaldngo_hp> cool
17:19:41 <jaypipes> davidkranz: OK, so I think there is agreement to both of your points?
17:19:49 <davidkranz> Seems so.
17:19:51 <jaypipes> and also to Ravikumar_hp's #4 on the agenda?
17:20:12 <davidkranz> Ditto.
17:20:23 <Ravikumar_hp> 4. (Ravi) Freezing Diablo/Stable branch as we have Essex/Stable.
17:20:39 <Ravikumar_hp> It is already taken care in previous discussion
17:20:49 <jaypipes> k. davidkranz, I'm going to assign an action item to you, then, for posting to the mailing list a summary of those decisions about stable branch maintenance for tempest.
17:21:14 <jaypipes> #action davidkranz to post decision summary to ML about stable release branch maintenance policy for Tempest
17:21:34 <jaypipes> Good to go to next item in agenda?
17:21:40 <davidkranz> jaypipes: OK, I will send it to the group first to make sure.
17:21:52 <jaypipes> davidkranz: cheers and thx!
17:22:01 <jaypipes> #topic 3. (Ravi) Status of Swift test development for Tempest
17:22:13 <jaypipes> JoseSwiftQA: ping!
17:22:27 <dwalleck> jaypipes: pong
17:22:52 <jaypipes> dwalleck: got a status on those eagerly-anticipated Swift Tempest tests? :)
17:22:58 <dwalleck> In case he doesn't pong =P
17:23:39 <dwalleck> jaypipes: Close to being converted. Gigi said to ping her, she has the timeline
17:24:07 <jaypipes> dwalleck: hmm, OK. Hoping for something to put in the post-meeting summary post to ML... ;)
17:24:09 <dwalleck> She has me focused on Nova stuff so I'm not as in the know
17:24:16 <jaypipes> dwalleck: gimme something more! :)
17:24:35 <dwalleck> Soon? :D
17:24:54 <jaypipes> lol :)
17:24:55 <JoseSwiftQA> jaypipes: Sorry, zoned out.  I've put in all the new config changes
17:24:56 <dwalleck> 2012?
17:25:03 <fattarsi> lol
17:25:07 <jaypipes> JoseSwiftQA: ah, cool.
17:25:16 <jaypipes> JoseSwiftQA: going to push to Gerrit this week?
17:25:40 <JoseSwiftQA> i'd say yes but I'm in a pessimistic mood. probably next week.
17:25:48 <jaypipes> k, good to be realistic.
17:26:10 <jaypipes> JoseSwiftQA: and about how many tests are in the making for swift? just curious..
17:26:13 * mtaylor injects https://jenkins.openstack.org/job/dev-gate-tempest-devstack-vm/400/console and runs away
17:26:16 <JoseSwiftQA> I aim to finish it up this weekend and have it tested against at least devstack
17:26:48 <JoseSwiftQA> I want to have basic api exercises for everythign in the client at least.
17:27:13 <jaypipes> JoseSwiftQA: awesome. anything the QA team can assist with?
17:27:18 <Ravikumar_hp> Great JoseSwiftQA:
17:27:35 <fattarsi> JoseSwiftQA: curious if you have anything planned for large uploads, as mentioned in https://bugs.launchpad.net/tempest/+bug/893333
17:27:36 <uvirtbot> Launchpad bug 893333 in tempest "Test large object support (>5GB)" [Medium,Confirmed]
17:27:58 <JoseSwiftQA> once I actually manage to finish writing the client, it should be fairly straight forward to write more tests.
17:28:57 <jaypipes> mtaylor: :( I disabled that job a couple days ago... someone enabled it again?
17:29:04 <JoseSwiftQA> fattarsi:  I have that written, but it's flakey
17:29:30 <JoseSwiftQA> i'll push it up anyway with many comments about caveats
17:29:42 <jaypipes> coolio.
17:30:03 <jaypipes> All: Not to jinx anything... but https://jenkins.openstack.org/job/dev-gate-tempest-devstack-vm/401/console
17:30:11 <jaypipes> tests all passing up to now...
17:30:12 <fattarsi> JoseSwiftQA: cool, I'd like to see anyway, I might be able to help
17:30:56 <mtaylor> jaypipes: oh, I enabled it because dwalleck was saying in the channel that he wanted tempest gating ... so I wanted to be able to give him a linnk to look at
17:31:11 <jaypipes> mtaylor: k, no worries...
17:31:23 <JoseSwiftQA> fattarsi: cool, i'll push everything to my fork on github once it's running, and send out an email
17:31:33 <jaypipes> JoseSwiftQA: rock on, thx man!
17:31:37 <mtaylor> jaypipes: do you guys have support for xunit test output?
17:31:38 * dwalleck refuses to comment about the positive or negative nature of that test run
17:31:41 <davidkranz> Oops. Just failed.
17:31:42 <jaypipes> mtaylor: yup.
17:32:01 <dwalleck> Well fudge
17:32:01 <fattarsi> JoseSwiftQA: awesome
17:32:05 <jaypipes> mtaylor: how do you think the graphs on the jenkins job main page are generated? ;)
17:32:10 <JoseSwiftQA> :D
17:32:10 <dwalleck> Well, good reason. A server went into error status
17:32:21 <Ravikumar_hp> JoseSwiftQa: Thanks
17:32:28 <dwalleck> Hurray for AUT bugs!
17:32:40 <JoseSwiftQA> no prob :)
17:32:51 <jaypipes> dwalleck: hey, that right there is the BEST RESULT of the tempest jenkins job we've had to date. Huzzah!
17:33:14 <jaypipes> Ran 144 tests in 531.165s -- not bad :)
17:33:20 <mtaylor> jaypipes: SO CLOSE
17:33:24 <jaypipes> NODE_PROVIDER=hpcloud-az1
17:33:31 <jaypipes> also not bad :)
17:33:37 <mtaylor> az1 is being a real bitch today
17:33:42 <jaypipes> Hooray for stable providers! :)
17:34:29 <jaypipes> OK, final agenda item before open discussion...
17:34:35 <jaypipes> #topic (Jay) Can we come to a consensus on the subset of Tempest tests that we recommend to the core projects to gate their trunks?
17:35:03 <jaypipes> I suppose this may be better to discuss on the mailing list ... but
17:35:12 <davidkranz> jaypipes: We will run everything overnight regardless, right?
17:35:19 <jaypipes> davidkranz: sure.
17:35:48 <jaypipes> davidkranz: what we're talking about is what we recommend be the set of tests whose failure will prevent a merge into a core project trunk.
17:36:15 <davidkranz> jaypipes: Right.
17:36:31 <jaypipes> davidkranz: and of course we must balance the total length of testing time with the overall breadth of coverage those tests provide...
17:36:31 <Ravikumar_hp> jaypipes: yes . may be tests are identified with existing @attr=smoke
17:36:48 <jaypipes> Ravikumar_hp: well, see my mailing list post about that particular topic ;)
17:36:55 <davidkranz> I think we should see how reliable the nightly runs are while discussing this with others.
17:37:10 <jaypipes> davidkranz: k, good point.
17:37:18 <davidkranz> The only other objection will be the time it takes.
17:37:23 <rohitk> there are about 300+ tests in tempest today
17:37:43 <davidkranz> If they were fast and reliable we would run them all on every trunk checkin.
17:37:45 <jaypipes> rohitk: there are?
17:38:09 <rohitk> just did a grep on 'def test_'|wc -l in tests
17:38:18 <Ravikumar_hp> yes. service level tests only - for example nova will run nova smoke tests - 5 or 6 tests
17:38:32 <jaypipes> davidkranz: right. but for instance, do we want a failure of, say, a test of a particular API extension to hold up merging? those are the kinds of decisions we must make...
17:38:51 <jaypipes> rohitk: ah! :)
17:38:56 <fattarsi> jaypipes: you have an idea how to separate out the 'recommended' set of tests?
17:39:16 <donaldngo_hp> it will be cool to have a smoke test run under 15 minutes and a regression nightly run that takes hours
17:39:17 <jaypipes> fattarsi: well, we can certainly use nose's @attr decorator for this.
17:39:31 <davidkranz> jaypipes: I don't think so. As long as a tempest nightly failure is considered to be an urgent issue that might be good enough.
17:39:32 <Ravikumar_hp> jaypipes: yes
17:39:38 <sdague> jaypipes: a related question, is there some sort of high level map out of the tests, especially the big holes where there is a desire for new tests?
17:39:41 <jaypipes> fattarsi: we just need to be careful about the consistency with which we use that decorator. Haven't been so consistent up to now ;)
17:39:42 <fattarsi> jaypipes: isee
17:39:43 <davidkranz> I think we should try that first.
17:40:18 <davidkranz> There is already significant overlap between tempest and various unit tests.
17:40:31 <jaypipes> sdague: unfortunately, you kind of have to check the Tempest open bug list on Launchpad right now...
17:40:45 <jaypipes> sdague: but we really should have a single "status of coverage" page. agreed...
17:40:55 <jaypipes> davidkranz: correct.
17:41:43 <sdague> jaypipes: what about actually documenting it in the repo in another text file? web pages are great, but they tend to stale relative to what's in the repo
17:41:51 <dwalleck> davidkranz: There is, but regardless of that, many bugs seem to be getting past the unit tests anyway....
17:41:54 <rohitk> jaypipes: We need documentation on using the attr decorator and the allowed/possible types, that may help test writers
17:42:10 <jaypipes> sdague: we could do that, sure... though that page can just as easily get out of date! :)
17:42:16 <jaypipes> rohitk: ++++
17:42:26 <jaypipes> rohitk: yes, that is definitely the case.
17:42:31 <dwalleck> rohitk: ++
17:42:39 <jaypipes> rohitk: I can take a stab at that one today.
17:42:41 <sdague> jaypipes: true, but if I pull tempest it would be nice to have an idea in the code about what its actually covering :)
17:42:45 <rohitk> jaypipes: thanks
17:42:49 <jaypipes> rohitk: I'll put together a doc on using it
17:43:00 <davidkranz> dwalleck: I know. Just worried that running tempest on every checkin everywhere will suck huge resources for possibly little gain relative to a nightly build with failures treated as urgent.
17:43:05 <jaypipes> sdague: :) no disagreement from me
17:43:20 <dwalleck> sdague: The problem is that you can't easily measure functional test coverage like code test coverage without instrumentation
17:44:06 <dwalleck> davidkranz: That's why we need to solve the parallezation problem. We can easily get the full test run in the 20-30 zone
17:44:39 <davidkranz> dwalleck: Yes, but it consumes the same amount of resources.
17:44:51 <jaypipes> davidkranz, dwalleck: I agree with both of you, actually. That's where the whole question of "balance" comes into play... there are tradeoffs for eveything of course. If we had a good, consistently-applied use of the @attr decorator, I think we can get the best of both worlds, and have the gate job focus on quick, important tests and have the nightly job run a series of more thorough tests
17:45:11 <donaldngo_hp> ++
17:45:26 <davidkranz> jaypipes: Agreed. Perhaps we should ask the PTLs to identify such a set that makes sense to them
17:46:14 <jaypipes> davidkranz: PTLs have too much on their plate already IMHO ;) I think we should make our best recommendation and tune it over time...
17:46:32 <dwalleck> jaypipes: ++. Sounds reasonable
17:46:37 <donaldngo_hp> yea for example uploading a 5GB file will take pretty long that is my opinion is a regression test
17:46:40 <davidkranz> OK
17:46:44 <jaypipes> davidkranz: if we notice a test is taking a long time and it may not cover an important area, we can remove it from the gate, etc
17:47:12 <davidkranz> jaypipes: Yeah. This will be a work in progress for a while if not forever.
17:47:21 <rohitk> we should probably have a longevity test suite in future
17:47:27 <jaypipes> FOREVER! :)
17:47:49 <jaypipes> rohitk: yes, I think davidkranz's stress test module would be a good basis for that.
17:47:53 <davidkranz> We will have to make the same sorts of tradeoffs for nightly streses tests.
17:48:01 <rohitk> jaypipes, davidkranz: ++
17:48:03 <jaypipes> davidkranz: yup
17:48:56 <jaypipes> alright guys, anyone got any other things to discuss?
17:49:00 <jaypipes> #topic open discussion
17:49:08 <rohitk> ping! Do we have a timeframe to get the networks client in? https://review.openstack.org/#/c/4896/3
17:49:21 <rohitk> we have a bunch of quantum tests
17:49:28 <jaypipes> rohitk: ah, yes...
17:49:47 <jaypipes> mnewby: ping
17:49:54 <davidkranz> I think we should add an item to the agenda template for "Stuff likely to get posted in the next week."
17:49:54 <mnewby> jaypipes: here
17:50:11 <jaypipes> mnewby: Hi! :)
17:50:27 <jaypipes> mnewby: so, we actually put off the discussion on smoke testing to next week...
17:50:40 <mnewby> jaypipes: hi!  I was watching the conversation for signs of the discussion.
17:50:45 <mnewby> Same time next week?
17:50:52 <jaypipes> mnewby: but I'd like to get the quantum tests in https://review.openstack.org/#/c/4896/ into tempest this week
17:51:13 <mnewby> jaypipes: They look good to me.
17:51:31 <jaypipes> mnewby: k, I will remove my -1. they were tiny nits anyway...
17:51:57 <jaypipes> mnewby: approved. off to the test pit they go.
17:52:06 <mnewby> jaypipes; I'll hopefully be talking with Daryl about the potential for automating the golden paths.
17:52:13 <jaypipes> rohitk: so there's the answer to your question ;) in about 20 minutes.
17:52:24 <jaypipes> mnewby: excellent.
17:52:47 <rohitk> jaypipes: awesome!
17:53:10 <davidkranz> Hopefully some one can review my recent stress test submission.
17:53:39 <jaypipes> davidkranz: will do today.
17:53:48 <davidkranz> jaypipes: Thanks!
17:53:51 <jaypipes> np!
17:53:53 <rohitk> i would like some suggestions on how get tests that depend on mysql-client into tempest
17:54:09 <rohitk> i have scenarios where I need to update the DB to test functionality
17:54:16 <jaypipes> rohitk: hmmm.... the whitebox tests...
17:54:18 <rohitk> yes
17:54:48 <rohitk> decorator for such tests (skip_if mysql not installed, attr type=wb)
17:54:53 <rohitk> something like that
17:54:55 <jaypipes> rohitk: please see my mailing list post about the smoke tests and also this draft merge prop: https://review.openstack.org/#/c/7069/
17:55:11 <rohitk> jaypipes: ok
17:55:23 <jaypipes> rohitk: We can add another test case class of WhiteboxTest to go along with FuzzTest and SmokeTest
17:56:21 <davidkranz> Whitebox tests may be the next frontier.
17:56:21 <rohitk> jaypipes: ++
17:56:55 <davidkranz> Like flapping a service during a stress test :)
17:57:02 <jaypipes> davidkranz: yep. I always wanted them, but wanted to complete our API coverage tests (blackbox) first... we're almost there and I think starting on whitebox is good now..
17:57:26 <rohitk> so ssh into an openstack node (not the vm) and mysql stuff would be whitebox?
17:57:30 <rohitk> right?
17:57:42 <jaypipes> rohitk: correct
17:57:43 <davidkranz> rohitk: Yes.
17:57:58 <rohitk> ok, thanks
17:58:25 <jaypipes> OK all,  we're out of time now... please feel free to comment/suggest/critique on my mailing list post about smoke tests... :)
17:58:39 <davidkranz> Bye all.
17:58:46 <rohitk> jaypipes: looking forward for that...thanks!
17:58:59 <jaypipes> davidkranz: you'll send summary to ML? and ping dwalleck about his rotation next week?
17:59:28 <jaypipes> OK, bye all!
17:59:30 <jaypipes> #endmeeting