17:02:34 <davidkranz> #startmeeting qa
17:02:35 <openstack> Meeting started Thu Nov 29 17:02:34 2012 UTC.  The chair is davidkranz. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:39 <openstack> The meeting name has been set to 'qa'
17:02:39 <donaldngo> Donald here
17:03:14 <sdague> <- here
17:03:33 <davidkranz> jaypipes, dwalleck: You guys here?
17:03:47 <dwalleck> just got here
17:04:23 <davidkranz> Looks like the tempest swift tests will be part of the gate as soon as the devstack fix is approved.
17:04:57 <dwalleck> nice
17:05:00 <ravkumar_hp> davidkranz: that would be grat
17:05:06 <sdague> davidkranz: which devstack fix is that?
17:05:10 <sdague> I can take a look
17:05:14 <ravkumar_hp> we are working on some new test for Swift
17:05:16 <davidkranz> sdague: Any progress on the "instances go to ERROR" issue?
17:05:52 <sdague> davidkranz: mtreinish is looking into it today, the problem is it's hard to reproduce
17:06:04 <sdague> I expect our systems are too fast
17:06:13 <davidkranz> sdague: https://review.openstack.org/#/c/17119/
17:06:17 <sdague> so he was going to try to run it in a kvm guest to slow it down
17:06:26 <sdague> which is the way ci is run anyway
17:07:03 <davidkranz> sdague: Yeah, this is probably acting as a lame stress test in the ci setup.
17:07:22 <davidkranz> sdague: I would really like to get some stress tests in the nightly build.
17:07:26 <sdague> ok, devstack patch approved
17:07:39 <dwalleck> I'm going to kick a merge prop in today that should add the reason for a server going into error status into the logging as well, should help wiht debugging
17:07:47 <davidkranz> sdague: Thanks. We'll see if anything blows up :)
17:07:48 <sdague> dwalleck: great
17:08:07 <sdague> davidkranz: well that devstack patch was just adding another variable to config tempest, so I hope not :)
17:08:45 <davidkranz> sdague: Yeah. I just meant that those tests will now start running in all project gates.
17:09:25 <sdague> davidkranz: cyeoh on my team is starting to look at the blueprint for converting tempest to testr/testtools. So expect to start to see some activity on that one in the next week or so.
17:10:26 <davidkranz> chunwang put in a blueprint for customized-test-launcher script
17:11:18 <davidkranz> I think that blueprint needs some more information about how it integrates with parallel execution and what the main purpose is.
17:11:46 <chunwang> yes, the blueprint is a work around and improvement based on the issues currently we found during tempest execution...
17:12:03 <dwalleck> do we know yet if a testtools solution is viable? I thought someone was going to do a prototype
17:12:35 <chunwang> Actually I think it's not parallel execution, but a batch run of the tempest cases, and with the customized test case list and environment clean module
17:13:02 <sdague> dwalleck: that will be cyeoh
17:13:22 <sdague> going to assume that it's workable, and if not, he'll flag that as an issue :)
17:13:41 <ravkumar_hp> sdague: cyeoh wil analyze testtools for parallel execution also . right?
17:13:49 <sdague> yes
17:14:14 <sdague> chunwang: url for your blueprint?
17:15:05 <chunwang> https://blueprints.launchpad.net/tempest/+spec/customized-test-launcher-script
17:15:50 <dwalleck> I'm just a bit wary of diving into a solution without a full plan of attack. I'm curious though if any of this will tie us to testtools. It'd be nice if in the stripping away of nose any standard unittest test runner just worked
17:16:44 <sdague> dwalleck: fair, though I think in reality you can't really know unless you try
17:17:18 <chunwang> why the parallel exeution is so important? May I know how long will the whole tempest test cases take in your environment?
17:17:42 <sdague> if it's not any good we won't force it in just because it's on the list. :-) But this will at least make it possible to evaluate
17:18:06 <sdague> davidkranz: you have the timings for the gate?
17:18:08 <dwalleck> sdague: True. I guess what I was thinking was to perhaps start with converting say one project's tests first before doing the whole lot
17:18:53 <sdague> dwalleck: from the experience I've seen in the folks working on converting nova here, you more or less have to just do it in one batch
17:19:08 <dwalleck> chunwang: Because in a Devstack environment the tests take about 45 minutes. In a full deployed environment, especially using Windows images, it might almost take a day
17:19:13 <davidkranz> sdague: The last hourly run took just under 1000 seconds for the full part.
17:19:14 <rohitk> chunwang: I think in the longer term parallelism would prove more beneficial
17:19:41 <ravkumar_hp> dwalleck: yes. but convert one test before converting project's all tests
17:19:44 <sdague> anyway, cyeoh is in australia, so I'll proxy his updates here, because it's some ungoddly hour of the night
17:19:49 <dwalleck> The goal is to just to get more done quickly so folks are more inclined to run them
17:20:07 <davidkranz> sdague: and 200s for the smoke tests
17:20:08 <dwalleck> All fair points. I'm just a cautious one :-)
17:20:31 <sdague> dwalleck: right, which is why we've got a good review process :-)
17:20:42 <dwalleck> yup, good point
17:21:22 <chunwang> Simlar with your time...in our environemnt, it will take about 25 hours to finish the whole tests...
17:22:04 <sdague> chunwang: 25 hours? what environment is that?
17:22:09 <dwalleck> But in parallel, I'm running my Tempest tests (with real Linux/Windows images) in < 45 min, so getting there is the goal
17:22:48 <chunwang> a Essex version Openstack, with 16 cn, 1 cc...
17:24:03 <davidkranz> dwalleck: When we get parallelism for devstack gate we will likely run into https://bugs.launchpad.net/nova/+bug/1016633
17:24:04 <uvirtbot> Launchpad bug 1016633 in nova "Bad performance problem with nova.virt.firewall" [Medium,Incomplete]
17:24:08 <chunwang> but it may not similar with normal environment, becasue some of the OS image will take about 20 min to boot up at 1st time boot
17:24:34 <sdague> chunwang: gotcha
17:24:36 <davidkranz> Firing up a bunch of instances on a single compute node is slow.
17:25:12 <davidkranz> I guess we will deal with that when it happens.
17:25:15 <sdague> agreed
17:25:27 <dwalleck> Hmm, interesting.
17:25:35 <sdague> for the gate, were you going to also make it error if there were stack traces in the daemons?
17:26:18 <davidkranz> sdague: I really want to do that but there are still errors in the nova logs as far as I know.
17:26:30 <sdague> davidkranz: yes, there are.
17:26:46 <sdague> it would be great to get each one of those distinct stack traces as a nova bug
17:26:49 <davidkranz> sdague: As soon as they are clean I would go for it.
17:27:19 <sdague> seperately they are solvable, I did fix one a couple weeks ago, which was serious enough to be a folsom backport as well
17:28:57 <davidkranz> sdague: Yeah. It would be best if the nova team as a whole made this higher priority.
17:29:37 <sdague> davidkranz: yeh sure, just saying that if they get broken up as discreet issues, I can probably get folks looking at them
17:30:20 <davidkranz> sdague: I agree. I can do look at the latest logs and do that.
17:30:26 <sdague> that would be great
17:31:23 <dwalleck> I need to duck out folks. I have some updates, but I'll send those in an email this afternoon. Adios!
17:32:00 <davidkranz> I am going to take a look at the multi-node-testing blueprint.
17:32:13 <davidkranz> Any one know if anything is happening with fuzz testing?
17:32:44 <chunwang> May I know when will the quantum scripts be ready for use?
17:33:29 <rohitk> I will be jumping onto some quantum tests next week too
17:33:45 <davidkranz> chunwang: Not sure. I think Nachi Ueno is the lead ono that.
17:34:13 <davidkranz> chunwang: https://blueprints.launchpad.net/tempest/+spec/quantum-tempest
17:34:53 <davidkranz> rohitk: Great. Make sure to touch base with Nachi to avoid duplication.
17:35:06 <rohitk> davidkranz: yup :), mnewby was working on a patch I believe
17:35:34 <davidkranz> Any other topics for today?
17:35:41 <rax-Jose> ravkumar_hp:  What features are you targeting w/ your latest swift tests?
17:35:47 <rax-Jose> Just curious
17:35:47 <rohitk> davidkranz: I can't see the fuzz testing/randgen coming in anytime soon
17:36:01 <rohitk> what's the strategy to accept more negative tests?
17:36:11 <ravkumar_hp> rax-Jose: tempurl + https://blueprints.launchpad.net/tempest/+spec/add-some-functional-swift-tests
17:36:12 <ravkumar_hp> https://blueprints.launchpad.net/tempest/+spec/add-swift-security-tests
17:36:20 <rax-Jose> coolbeans, thanks.
17:37:16 <sdague> davidkranz: nothing more from me
17:37:20 <davidkranz> rohitk: Is some one working on fuzz testing at all? There isn o assignee for the blueprint.
17:38:09 <rohitk> davidkranz: I don't think so, but the old style negative API validation tests were not being accepted as they slowed down the runtime
17:38:31 <chunwang> if there is any scripts need help, I will try to help on that...
17:38:48 <davidkranz> rohitk: That is still a problem. It is also a problem that they take much longer to write than if using a negative test-generator.
17:39:06 <davidkranz> rohitk: But there is the up-front, non-distributed cost of doing that.
17:39:27 <rohitk> davidkranz: Agreed, I don't think we should afford that
17:39:28 <davidkranz> chunwang: Is proposing and creating a negative test runner something you could do?
17:39:54 <davidkranz> chunwang: There are some tools out there that do this kind of thing.
17:40:07 <sdague> rohitk: I think the important thing to is that the negative tests do more than just test fail, they need to get fails the way they expect
17:40:43 <davidkranz> sdague: A negative test runner would take care of that if you specify the space of parameters and the expected result.
17:40:50 <rohitk> sdague: Yes, more to do around bad input
17:40:56 <chunwang> ok, currently I didn't proposing more negative test myself, but if there is any existing requirement there, I will try to look into it...
17:42:27 <sdague> davidkranz: oh, that's the last thing. The coverage reporting for tempest is actually coming along by mtreinish. He's got a nova extension in final stages of review that will let us get nova coverage from an external test runner
17:42:43 <davidkranz> sdague: Great.
17:42:45 <sdague> so I'm hoping that's available in a couple of weeks as part of normal tempest runs
17:43:15 <rohitk> sdague: Sounds good!
17:43:24 <davidkranz> Last call for new issues...
17:43:25 <sdague> it doesn't really add much to the test runs time wise, so it should be something we can enable in the default runs
17:43:45 <chunwang> sdague: Good news, we also need this kind of data
17:46:49 <davidkranz> OK, see you all next week.
17:46:54 <sdague> see you then
17:47:00 <davidkranz> #endmeeting