17:01:38 <sdague> #startmeeting qa 17:01:39 <davidkranz> Here 17:01:39 <openstack> Meeting started Thu Jun 20 17:01:38 2013 UTC. The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:01:42 <openstack> The meeting name has been set to 'qa' 17:01:59 <sdague> #topic Blueprint Status 17:02:16 <sdague> please check in with any updated info on your blueprints using the #info tag 17:02:35 <sdague> we'll give everyone a minute or two on that 17:03:34 <mtreinish> sdague: no blueprint link this time? 17:03:45 <sdague> #link https://blueprints.launchpad.net/tempest 17:03:52 <sdague> sorry, I'm slow :) 17:04:08 <sdague> #link https://launchpad.net/tempest/+milestone/havana-2 17:04:24 <sdague> those are the havana 2 blueprints, which we are only 3 weeks away from, and there are a lot there 17:05:18 <davidkranz> I'm going to try to get some progress on the stress tests. 17:05:19 <sdague> davidkranz: what's need to close out the stress test one 17:05:23 <sdague> ok, great 17:05:27 <davidkranz> Volume was just added. 17:05:45 <davidkranz> sdague: The most important thing is to create the job that will run it 17:05:55 <davidkranz> sdague: And set a periodic build. 17:06:08 <davidkranz> sdague: RIght now there is nothing making sure it actually works. 17:06:29 <davidkranz> sdague: But there is an issue that these tests are really targeted to run in an environment not supported upstream. 17:06:30 <sdague> davidkranz: ok, great, is that still doable for havana-2? 17:07:03 <davidkranz> sdague: I am checking on that and will update accordingly. 17:07:07 <sdague> davidkranz: well I think we had a policy that if we can't run it in CI some how (periodic or otherwise) it can't really be in tempest 17:07:13 <sdague> because otherwise we get bit rot 17:07:31 <mtreinish> that was a big issue with the old stress tests. 17:07:44 <sdague> right, exactly 17:08:05 <davidkranz> mtreinish: Unfortunately the reason the old stress tests could not be run still remains. 17:08:12 <davidkranz> mtreinish: All the bogus log errors. 17:08:25 <sdague> davidkranz: well we can run it as a non enforcing job 17:08:56 <sdague> also, if there are specific bugs for each error in the logs, they are likely to get addressed 17:09:01 <davidkranz> sdague: We need to do that anyway but the real issue if false positives if we can't check the log for legitimat errors 17:09:15 <davidkranz> typing too fast... 17:10:00 <davidkranz> sdague: I filed a whole bunch of bugs a few months ago 17:10:09 <sdague> davidkranz: sure, but we can call out the false possitives to code around them, much like skips in our tempest code 17:10:15 <davidkranz> sdague: Some were fixed, some not. 17:10:37 <davidkranz> sdague: Yes, we can do that but it is very fragile. 17:10:38 <sdague> ok, well lets leave it at the following 17:11:03 <sdague> #action davidkranz to see if stress tests can be completed for h2 milestone 17:11:20 <davidkranz> sdague: OK 17:11:32 <sdague> mtreinish: I know I put testr later in the agenda, but you want to talk about it now, given that it's also a critical blueprint? 17:11:50 <mtreinish> sdague: sure 17:12:05 <davidkranz> mtreinish: Is there any issue with the admin tests running in parallel? 17:12:22 <davidkranz> mtreinish: There is no tenant isolation there. 17:12:30 <mtreinish> so I've got a patch pending for testr that partitions on test classes and it's giving runtime numbers that are what we expect 17:12:40 <mtreinish> davidkranz: not that I've seen in my runs 17:12:43 <davidkranz> mtreinish: Great! 17:13:30 <sdague> mtreinish: any word from lifeless yet about his evaluation on the patch? 17:13:31 <mtreinish> I'm waiting on lifeless to review the testr patch before I push out a change that moves run_tests and tox over to testr 17:13:40 <sdague> ok, cool 17:14:05 <mtreinish> once it gets merged we should be able use testr for everything 17:14:24 <mtreinish> I'm sure it will shake loose some races but we debug those as they come up 17:14:43 <sdague> yep 17:14:50 <sdague> that will be great 17:14:51 <mtreinish> but at least locally things have been fairly stable on my dev box 17:15:07 <davidkranz> mtreinish: Maybe we should keep it work in progress and rerun it a bunch of times before merging? 17:15:09 <sdague> mtreinish: you want to update that blueprint to good progress? 17:15:15 <mtreinish> if anyone cares I've put some initial numbers up here: https://etherpad.openstack.org/testr-numbers 17:15:27 <sdague> #link https://etherpad.openstack.org/testr-numbers initial tester numbers 17:15:46 <mtreinish> also the testr patch is here if anyone wants to try it: https://code.launchpad.net/~treinish/testrepository/testrepository 17:15:50 <sdague> #info https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest now Good Progress, just waiting for test patch inclusion 17:15:51 <davidkranz> mtreinish: I would hesitate to put that in the gate based on one successful run. 17:16:17 <sdague> davidkranz: yeh, probably we'd want to make a separate tox rule for a full run 17:16:28 <sdague> and run it in shadow mode for a week or two 17:16:32 <sdague> just to make sure we are good 17:16:37 <davidkranz> sdague: Sounds good to me. 17:16:41 <mtreinish> davidkranz: yeah that's a fair point 17:16:50 <mtreinish> sdague: yeah sounds like a good idea 17:17:09 <sdague> ok, anything else on blueprints? 17:17:18 <sdague> and, anyone here besides just us 3 :) 17:17:18 <mtreinish> also testr has '--until-failure' and '--analyze-isolation' which we could maybe setup as a periodic too 17:17:47 <davidkranz> mtreinish: Do you have a feel as to why the speedup falls sharply after 2x? 17:18:03 <davidkranz> mtreinish: Of course 2x would be great :) 17:18:33 <sdague> davidkranz: I think one thing that came up is that it's not using the timing database to optimize 17:18:41 <afazekas_> hi 17:18:46 <mtreinish> davidkranz: not really, I was happy with the numbers but I haven't looked into optimizing them further yet 17:18:53 <mtreinish> I'm sure there is room for improvment 17:19:06 <sdague> hey afazekas 17:19:13 <davidkranz> mtreinish: Sure, just curious. At some point it will slow just due to load. 17:19:20 <sdague> afazekas: anything from you on blueprints before we move to reviews? 17:19:21 <afazekas_> hey sdague 17:20:21 <sdague> ok, reviews 17:20:28 <sdague> #topic Reviews that need attension 17:20:54 <afazekas_> I will try to figure out why I have the issue with glance+swift+wc2, after that I will try to reproduce the real ec2 issue in a loop (as background job), at the same time I will work on the resource leak , and add some docs 17:21:03 <sdague> looks like we actually have a lot of reviews with 1 +2 on them 17:21:14 <afazekas_> s/wc2/ec2/ 17:21:16 <davidkranz> sdague: Well, there is https://review.openstack.org/#/c/33689/ :) 17:21:39 <davidkranz> sdague: I think a few are waiting for non-red-hat approval. 17:21:55 <mtreinish> davidkranz: sdague started a ml thread on that one 17:21:55 <sdague> davidkranz: sure, I'll take a look later today 17:22:25 <davidkranz> mtreinish: I know. Should be fun. We've been down this road before. Swift folks just don't agree with the API stability policy. 17:22:28 <sdague> davidkranz: yeh, on the swift one there is a thread here: http://lists.openstack.org/pipermail/openstack-dev/2013-June/010689.html 17:23:22 <sdague> ok, I need to go figure out why my patch died on pg 17:23:30 <sdague> didn't notice that one yet 17:23:41 <sdague> ok, any other reviews of note? 17:24:16 <sdague> ok, so the last thing on the agenda which we didn't yet talk about was multiple api versions - https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Weekly_QA_Team_meeting 17:24:28 <sdague> #topic Multiple API versions 17:24:57 <sdague> there was an ML thread on this, we think it's resolved enough there, or is there need for more discussion? 17:25:17 <mtreinish> sdague: that fail should just be a recheck. I opened a bug for it here: https://bugs.launchpad.net/devstack/+bug/1192990 17:25:19 <uvirtbot> Launchpad bug 1192990 in devstack "Keystone certificate error with heat" [Undecided,New] 17:25:25 <davidkranz> sdague: I don't think it is quite resolved there but we could continue the discussion there. 17:25:38 <sdague> ok, we'll take the discussion there 17:26:01 <davidkranz> My issue is how we test new and old versions that bhavior has not changed without duplicating the tests. 17:26:06 <sdague> #topic Open Discussion 17:26:14 <sdague> davidkranz: I don't think you can 17:26:30 <davidkranz> sdague: That is a big problem given the size of nova, e.g. 17:26:32 <sdague> duplicating is probably the right way from an issolation perspective, it also lets you remove old versions over time 17:26:33 <mtreinish> davidkranz: yeah there is no guarentee something won't change between versions so we just have too test both 17:26:56 <sdague> davidkranz: nova is duplicating the code paths internally for the same reason 17:27:05 <davidkranz> mtreinish: Yes but I hope we can find a way to duplicate dynamically but not statically. 17:27:21 <sdague> davidkranz: the resource structures are going to change as well 17:27:22 <afazekas_> _interface='xml' 17:27:23 <davidkranz> mtreinish: The test code will be the same if the behavior has not changed. 17:27:38 <sdague> davidkranz: the reason for the new api is because it did change 17:27:47 <mtreinish> davidkranz: not necessarily things like return codes will probably change 17:27:50 <sdague> if it didn't change, you wouldn't need a bump 17:27:56 <davidkranz> sdague: Every api changed? 17:28:01 <afazekas_> I am not sure the current '_interface' solution is the nicest way how to avoid code duplication, but it works 17:28:31 <afazekas_> the services could be responsible for the error code checking 17:28:35 <davidkranz> We can continue this on the list but a real proposal is needed. 17:28:49 <sdague> davidkranz: probably not every, but enough move around that it's easy to miss a test bug inside a complicated if / else around versions 17:28:56 <afazekas_> 'services' ie. rest_clients 17:29:09 <davidkranz> sdague: OK, I did not realize it was this extensive. 17:29:12 <sdague> things like tenant ids in urls going away 17:29:22 <davidkranz> sdague: That is a localized change. 17:29:38 <davidkranz> sdague: Obviously the rest client will change. 17:29:44 <sdague> yeh, but the problem is that you can come from 2 points of view 17:29:57 <sdague> assume it is all the same, and put complexity into the tests when it's different 17:30:09 <sdague> or assume it's all different, and factor out common bits when it's the same 17:30:30 <sdague> I feel like not knowing how same or different it will end up, option 2 is safer 17:30:41 <davidkranz> sdague: I hope it is the first. Otherwise users who upgrade will suffer a lot. 17:30:56 <sdague> davidkranz: that's why it's a major version bump 17:31:06 <sdague> and why v2 will still be in havana 17:31:23 <davidkranz> sdague: If history is a guide, once we get more stable there will be a lot of resistance to changes in major versions as well unless they are really necessary. 17:32:07 <sdague> davidkranz: honestly, I don't expect v3 to be the final word, if the interfaces don't evolve over time, they don't stay relevant 17:32:18 <sdague> but that's a meta discussion :) 17:32:28 <davidkranz> sdague: Right. 17:32:34 <sdague> ok, anything else from folks 17:33:07 <davidkranz> Nope. 17:33:11 <mtreinish> nothing from me 17:33:40 <sdague> ok, thanks all 17:33:43 <sdague> #endmeeting