17:03:08 <jaypipes> #startmeeting qa
17:03:09 <openstack> Meeting started Thu Nov 15 17:03:08 2012 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:11 <openstack> The meeting name has been set to 'qa'
17:03:14 <openstack> davidkranz: Error: Can't start another meeting, one is in progress.
17:03:17 <jaypipes> hehe
17:03:30 <sdague> :)
17:03:52 <davidkranz> Should we go with the agenda I posted, or is there anything else to add?
17:04:01 <jaypipes> davidkranz: nothing from me.
17:04:12 <sdague> davidkranz: I think the agenda looks good
17:04:20 <dwalleck> the posted agenda looks good
17:04:27 <sdague> the rest we can do in open discussion if things come up
17:04:36 <davidkranz> Let's start with the small ones.
17:05:02 <davidkranz> I got zero response to the message about getting dev help to review significant new api tests.
17:05:25 <jaypipes> :(
17:05:35 <sdague> davidkranz: are these existing reviews?
17:05:53 <davidkranz> The one in the agenda has been sitting in the queue.
17:06:13 <davidkranz> Is any one in our group swifty enough to review it?
17:06:25 <sdague> oh, the swift reviews
17:06:33 <davidkranz> THe last patch grew it significantly.
17:06:34 <ravkumar_hp> https://review.openstack.org/#/c/15554/ ?
17:06:38 <sdague> I honestly don't know anything about swift
17:06:58 <sdague> #topic swift tempest tests need review
17:07:01 <ravkumar_hp> davidkrnz: we will test it out and review https://review.openstack.org/#/c/15554/
17:07:24 <davidkranz> ravkumar_hp: Thanks! That is great.
17:07:42 <ravkumar_hp> we need to have theste tests in as those are critical for gating
17:07:48 <davidkranz> #topic Full tempest gate
17:07:56 <sdague> so the way we've been handling this is devstack is that we ask devs from the team to provide reviews
17:07:59 <dwalleck> I'm swiftly enough also, can take a look. I saw some comments from jaypipes, hadn't seen if those had already been addressed
17:08:09 <sdague> and if they +1 it enough then we take the word at it
17:08:45 <jaypipes> sdague: we've already asked notmyname and he gave a review on it, but no followup review IIRC.
17:09:54 <sdague> ok, I'll take a look at the review as well.
17:09:56 <davidkranz> So that will move forward now. We can ask a specific question if we have it.
17:10:40 <davidkranz> Where do we stand on getting an "acceptance test" for swift up?
17:10:47 <davidkranz> notmyname was very interested in that.
17:11:09 <ravkumar_hp> https://review.openstack.org/#/c/15554/  once checked in can be acceptance tests . right?
17:11:37 <davidkranz> ravkumar_hp: Yes, part of one.
17:11:58 <davidkranz> He was referring to a complete test that would be run to say that swift upstream was ready to be released.
17:12:21 <davidkranz> That would not be a gate test.
17:12:43 <sdague> davidkranz: honestly, I think that's probably something that needs to go back on the swift team
17:13:04 <davidkranz> Until recently it was basically RAX qa that performed that function.
17:13:29 <davidkranz> sdague: Yes, you are right.
17:13:59 <davidkranz> About the full tempest gate.
17:13:59 <ravkumar_hp> davidkranz: from hp , we will add some tests once the current one review is finished
17:14:06 <sdague> so, tempest full gate....
17:14:06 <davidkranz> ravkumar_hp: Cool.
17:14:18 <davidkranz> I added an hourly run with email to me.
17:14:24 <jose-rax> we still run those tests, as well as new ones for features as they come down the pipe.  cors headers for instance.
17:14:46 <jaypipes> jose-rax: which tests do you run?
17:15:02 <davidkranz> jose-rax: I know. We wanted tests out in the open that are acceptance tests.
17:15:07 <jaypipes> ++
17:15:40 <sdague> also +1
17:16:23 <jose-rax> pretty much everything
17:16:50 <jaypipes> jose-rax: I'm still not clear... everything that is in tempest, or something else?
17:16:55 <davidkranz> jose-rax: Gigi said at the summit that you guys were going to put the tests up so we could cooperate on maintaining/improving them.
17:16:59 <jaypipes> Rick_RS: heyo
17:17:03 <sdague> so full tempest gate... :) has there been any progress there? a couple of folks on my team are going to get some cycles in the next couple weeks, so I was going to point them in that direction.
17:17:10 <sdague> but I don't want to be duping anyone else's work
17:17:31 <jaypipes> sdague: davidkranz mentioned above that there is now an hourly full run that emails him on failures
17:17:32 <Rick_RS> jaypipes: hello
17:17:35 <davidkranz> sdague: All we need are no flakies and acceptable run time.
17:17:47 <davidkranz> There are no more volume failures as of yesterday.
17:17:52 <dwalleck> sdague: depends on what you mean by full tempest gate. I'm opening blueprints for everything myself and my guys are working on
17:18:03 <sdague> dwalleck: cool
17:18:05 <jaypipes> davidkranz: good. those were the ones that were consistently hitting me locally.
17:18:14 <sdague> davidkranz: great.
17:18:20 <sdague> davidkranz: what's the run time clocking in at now?
17:18:30 <davidkranz> jaypipes: There is still a server build flaky tha sdague is looking at.
17:18:36 <dwalleck> which is admin tests, benchmarking/combining tests, and adding a benchmarking suite
17:19:18 <sdague> dwalleck: ok, cool. If you carve those off in interesting ways we can probably get some more hands on it.
17:19:34 <davidkranz> sdague: About 25 minutes for full tempest without smoke.
17:19:41 <sdague> davidkranz: oh, that's not too bad
17:20:08 <davidkranz> smoke is another 6 minutes
17:20:17 <dwalleck> It's actually mostly done, just waiting for approvals (for admin tests at least). These are things we already have, just trying to get them submitted
17:20:25 <davidkranz> If we could parallelize we would run much faster with room to spare for more tests
17:20:31 <jaypipes> dwalleck: did you follow my response about the policy.json determining the "adminness" of API calls? :)
17:20:34 <davidkranz> dwalleck: Great!
17:20:57 <davidkranz> Let's finish Daryl's thread before moving on.
17:21:03 <jaypipes> k
17:21:07 <sdague> davidkranz: so I was kicking around an idea with mtreinish earlier about injecting log messages into nova when tests start / stop. One of the problems in looking through those logs with stack traces is having no idea what triggered them
17:21:10 <dwalleck> jaypipes: I do, and for now we can go with your suggestion. I'd like to revist it later to make things more flexible, but it'll work for now
17:21:17 <jaypipes> kk
17:21:55 <sdague> dwalleck: cool, so best guess on when we'd see these things in the review queue?
17:22:20 <davidkranz> sdague: Yeah. More instrumentation will likely be needed.
17:22:21 <sam-rax> davidkranz: we rely on the tempest swift tests for gating into our pipeline, however we have an exhaustive body of tests that are directed specifically at a deployed version of swift, not a devstack instance.
17:22:37 <dwalleck> next week is thanksgiving so that makes things a bit weird, but certainly by the meeting the week afterwards
17:22:48 <sdague> dwalleck: cool, sounds great
17:22:58 <davidkranz> sam-rax: We discussed this at the configuration testing session.
17:23:25 <Rick_RS> davidkranz: in regards to Gigi's commitment to putting the tests out in the community, we are working on it.  We are in the process of doing some cleanup before we put them out there.
17:23:36 <davidkranz> sam-rax: We just need a way to report the results of those tests back
17:24:05 <davidkranz> Rick_RS: Great. We just wanted to get an idea of where things were because notmyname was asking about it.
17:24:37 <davidkranz> #topic parallel tempest
17:24:55 <davidkranz> Is any one working on this yet?
17:25:24 <dwalleck> I haven't had much luck with testtools at the moment
17:25:32 <davidkranz> :(
17:25:46 <dwalleck> Tinkering with just simply using eventlet, have a bit more success there
17:26:19 <sdague> dwalleck: what were the testtools issues?
17:26:21 <davidkranz> dwalleck: At this point we just need something that works.
17:26:24 <dwalleck> But I thought others were working on a demo of what testtools would look like?
17:26:24 <ravkumar_hp> dwalleck: that is nosetests using evenlet?
17:26:40 <dwalleck> No, just using raw unittest with eventlet
17:28:09 <sdague> dwalleck: ok, so that's something I can have a couple folks look at soon.
17:28:43 <ravkumar_hp> dwalleck: does existing temptest tests need lot ot rework to use with eventlet?
17:29:22 <dwalleck> ravkumar_hp: Not really. Regardless of what we do, we're going to have to remove our dependencies on Nose at some point
17:29:43 <dwalleck> which is mostly skip decorators, tags, etc
17:30:27 <sdague> so mordred has been looking at converting nova over to non nose, I wonder what insights he has there.
17:31:02 <notmyname> (sorry, just got back online) sam-rax: jose-rax: what would it take to get to get your swift-specific tests more open so that others can run them?
17:31:03 <jaypipes> what testtools, testr (testrepository) gives us is more than parallel testing. It gives us ways to record previous test runs and timings as well as do regression testing locally...
17:32:39 <dwalleck> jaypipes: What type of timing does it provide? Per test timings? per request?
17:33:18 <jaypipes> per test
17:33:21 <sdague> jaypipes: ok, so probably makes sense to do a port to testtools / testr regardless to see how that goes, even if there might be other approaches
17:33:22 <jose-rax> notmyname: The idea is that they'll be part of what we put out there after we finish cleaning this stuff up.
17:33:28 <sam-rax> notmyname: We are in the process of making things cleaner, removing some rax specific things (like billing, etc...) and ready to provide to the community
17:33:41 <jaypipes> sdague: yes, I gave a shot at it about 2 weeks ago but ran into issues and never got the time to email lifeless about them.
17:34:19 <notmyname> jose-rax: sam-rax: cool. I'd love to see them. been waiting for a while ;-)
17:34:30 <sdague> jaypipes: well if you have work in progress, let me know.
17:34:45 <jaypipes> sdague: will do.
17:35:34 <davidkranz> Anything else on the previous topics?
17:35:53 <mordred> sdague: hey
17:36:08 <davidkranz> When th hourly run goes two days without failure we can decide whether to put in the full gate at 25 minutes.
17:36:23 <mordred> sdague: yes - testtools/testr/fixtures/subunit toolchain is pretty often
17:36:41 <notmyname> jose-rax: sam-rax: (we can take this to another channel if you want) what can I do to help facilitate getting the tests out?
17:36:43 <sdague> often?
17:36:43 <davidkranz> There are also easily 5 minutes of slow tests that could be turned off for most projects.
17:36:48 <mordred> dwalleck: let me know if I can be helpful
17:38:02 <Rick_RS> notmyname: we are executing on a plan to expedite this.  we can talk some more off-line if you like.
17:38:10 <dwalleck> davidkranz: "turn off" is a bit of a worrysome term. If it came to that, I think creating a secondary, non-gating job might be helpful
17:38:40 <notmyname> Rick_RS: awesome. me@not.com or in IRC. whatever is more convenient for you
17:38:40 <davidkranz> dwalleck: There is already the hourly, and nightly job.
17:39:11 <davidkranz> I just meant that some of the expensive server action scenarios could just be run on nova, for example.
17:39:29 <davidkranz> And the periodic runs of course.
17:40:23 <davidkranz> The last topic was the status of the blueprints.
17:40:57 <Rick_RS> notmyname: we can do a f2f if you want.  I sit in the compute area.
17:41:10 <notmyname> Rick_RS: I sit in San Francisco ;-)
17:41:14 <jaypipes> LOL
17:41:22 <jaypipes> lmao
17:42:01 * jaypipes buys Rick_RS a plane ticket out to SFO ;)
17:42:08 <sam-rax> notmyname: I have some open QE SDT reqs on my team if you want. :-D
17:42:26 <Rick_RS> notmyname; duh... I'll send you some information later.
17:42:33 <jaypipes> :)
17:42:45 <sam-rax> notmyname: But seriously, what Rick_RS said, we can chat later offline if you want, but we honestly are working on it, just need to extricate some things and clean it up properly before sharing.....
17:43:15 <notmyname> ok, cool. I'm really happy to hear that. please let me know how I can help. It's a huge need for others in the swift community
17:43:57 <davidkranz> sdague: I think the blueprints cover what we discussed at the summit.
17:44:32 <sdague> davidkranz: you want to take a pass at prioritizing them, and putting targets on them?
17:45:05 <davidkranz> We should take up with the ci group the issue of coordinating status of real-cluster testing out side of jenkins.
17:45:41 <davidkranz> sdague: OK, I will do that. Do you mean targets like Grizzly-1, etc?
17:45:44 <sdague> davidkranz: agreed, but that's just one of them, and I honestly think that it's lower priority than a lot of the others
17:46:02 <sdague> davidkranz: yeh, it might be nice to give ourselves goals that line up with the milestones
17:46:24 <sdague> davidkranz: what group can modify those fields?
17:46:28 <davidkranz> sdague: OK, but it doesn't make sense to give a target unless there is also an assignee :)
17:46:33 <sdague> davidkranz: agreed
17:46:42 <sam-rax> +1 that. :-)
17:46:52 <davidkranz> sdague: I presume tempest administrators?
17:46:53 <sdague> priority can be set without an assignee though
17:47:02 <jaypipes> davidkranz: tempest-drivers
17:47:07 <davidkranz> sdague: Sure.
17:47:12 <sdague> and maybe incent folks to take a look at some of them :)
17:47:20 <jaypipes> davidkranz: you are an admin so feel free to give others admin access if you want
17:47:30 <jaypipes> sdague: none of us are innocent.
17:47:35 * jaypipes snickers
17:47:40 <sdague> jaypipes: :P
17:48:02 <davidkranz> sdague: I'll add you.
17:48:07 <sdague> davidkranz: cool
17:48:44 * jaypipes would like to help more if he wasn't waist-deep in writing Chef Spec tests :(
17:49:50 <davidkranz> Anything else?
17:50:01 <jaypipes> not from me
17:50:17 <davidkranz> Counting 30 seconds.
17:50:32 <sdague> I'm good
17:51:02 <longshot> 40 seconds turkish
17:51:21 <jose-rax> 50 seconds turkish
17:51:34 <davidkranz> jaypipes: I guess you can end the meeting.
17:51:52 <jaypipes> #endmeeting