22:00:48 #startmeeting qa 22:00:48 Meeting started Thu Jun 12 22:00:48 2014 UTC and is due to finish in 60 minutes. The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:49 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:51 The meeting name has been set to 'qa' 22:00:54 Hi who's here today? 22:00:59 Hi 22:01:02 hi 22:01:09 #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_June_12_2014_.282200_UTC.29 22:01:09 hi 22:01:14 ^^^ Today's agenda 22:01:21 hi 22:01:26 hi 22:01:27 hi 22:02:05 ok, lets get started 22:02:12 #topic Specs Review 22:02:14 hi 22:02:24 dkranz: you had a spec on the agenda 22:02:31 #link https://review.openstack.org/93037 22:02:45 Hi 22:02:46 mtreinish: Yes, I just wanted a decision about the xml issue per the two choices I added to the agenda 22:03:57 dkranz: I don't see how it's much more work. It's just an extra call in the xml clients 22:04:04 it's a lot of copy and paste 22:04:14 mtreinish: Yes. I guess we should do it. 22:04:29 hi 22:04:39 afazekas: late night :) 22:04:53 dkranz: heh, ok then I guess we agree. Is there anything else on this spec? 22:05:03 mtreinish: No, I think it is good to go. 22:05:18 ok, then does anyone else have a spec review that they'd like to bring up? 22:06:21 Quick chat on: https://review.openstack.org/#/c/91725/? 22:06:34 dpaterson: ok 22:06:52 A little bit related to this https://review.openstack.org/#/c/95794/ . We should print the console log an all ssh connection failure ASAP 22:07:11 So it seems that there is need for a cleanup type tool but quite different than I originally planned 22:07:22 No database cleanup for instance 22:07:51 I am going to update the spec in the next day or so but would like input before I do. 22:08:16 dpaterson: yeah having a leak detector/cleanup tool is something we've talked about for a long time 22:08:24 dpaterson: I think I put most of my thoughts in the review 22:09:17 dpaterson: you can ping me whenever on -qa if you've got specific questions when you're re-drafting it 22:09:18 So my take away from the comments I received is that the tool should help locate the root cause in OpenStack. 22:09:28 #link https://review.openstack.org/#/c/35516/ 22:09:45 My old approach for cleaning things.. 22:09:48 mtrenish: sounds good 22:10:50 yea leak detection seems more important than a cleanup tool - so we can fix the test 22:11:16 afazekas: on that spec if you're ready for reviews on you got to make it pass the docs jobs and have a lp bp to match the spec 22:11:27 cyeoh: yeah, they're tightly coupled 22:11:46 cyeoh: the old approach is easy, the other one is complicated with handling auth and multiple clients , tenant isolation ... 22:11:58 ok are there any other specs people want to bring up? 22:12:08 mtreinish: right, except we don't want to get into the habit of just always running the cleanup tool because its the easy thing to do 22:12:40 cyeoh: I think that's fine as long as reports what it's doing 22:12:46 so we can debug it 22:13:42 the point at which we identify leaked resources, running a delete on it after reporting it is no real extra effort 22:14:34 ok, if there aren't any other specs let's move on 22:14:48 #topic Blueprints 22:15:01 so Juno-1 was this week 22:15:08 we had 4 bps targetting it: 22:15:10 #link https://launchpad.net/tempest/+milestone/juno-1 22:15:11 mtreinish: do it inside a parallel test suite is complicated 22:15:46 2 of them are still under spec review so if everyone can prioritize those reviews that would be great 22:15:53 sdague, andreaf: are you guys around? 22:16:08 afazekas: no one said it was going to be easy :) 22:16:18 mtreinish: The stress one also has lots of code to review 22:16:23 mtreinish: yeh, sorry 22:16:26 I'm here... On my mobile thought 22:16:47 andreaf: Wow 22:17:04 sdague, andreaf: I was just wondering what the status of your j-1 targetted bps were 22:17:07 yeh, so all my bps are wrecked after discovering javelin wasn't really working right at summit, and that was wrecked by the gate backup 22:17:22 sdague: ok then I'll push it back to j-2 22:17:26 yeh 22:17:41 andreaf: yours was: Tempest support for testing against multiple keystone API versions 22:17:49 also, the outstanding work on branchless is basically the extensions lists 22:18:02 sdague: what more needs to be done for branchless tempest? 22:18:05 which I can give someone a quick overview on how to do it 22:18:20 dkranz: realistically I think it's just extensions 22:18:26 sdague: isn't just spin up icehouse devstack and grab the feature flags 22:18:35 then dump it in the devstack gate yaml 22:18:47 mtreinish: well the processor for that is also needed 22:18:59 so the communication path to actually get those set in tempest.conf 22:19:19 that's actually the part that will take work 22:19:48 ahh, ok I assumed that was already done 22:19:53 yeah that would take a bit more effort 22:19:58 no, we didn't need that for services 22:20:03 as that's actually set in devstack 22:20:14 everything in there so far was setable in localrc 22:20:46 so ... volunteers welcomed :) 22:20:51 ok, yeah then let's definitely push it back to j-2 22:20:55 yep 22:20:56 mtreinish: I'm back - on a laptop this time 22:21:03 andreaf: heh ok 22:21:13 I was just asking about your open j-1 bp 22:21:21 andreaf: https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests 22:21:26 should we push it back to j-2? 22:22:43 mtreinish: on my side the multi auth version is stuck on the scenario test, because of missing features in the official clients 22:23:07 mtreinish: yes 22:23:18 andreaf: ok then let's defer it, is there anything we can do to grease the wheels on getting the v3 stuff in the official clients? 22:23:38 I think that's the only one on j-1 22:24:01 andreaf: you also have an open spec review, but I'll push that one back :) 22:24:17 mtreinish: so re bp, did anyone have a chance to have a look at https://review.openstack.org/#/c/98400/ already? 22:24:28 mtreinish: oh ok 22:24:32 thanks 22:24:36 andreaf: not yet 22:24:49 andreaf: I;ll take a look after the meeting 22:25:04 ok that's all I had for bps, does anyone have anything else on the topic? 22:25:27 I wanted to ask about the tempest server /client / test repo 22:25:48 mainly about how we split work into different bps 22:25:48 andreaf: ok, can we save it for the open discussion, we've got a full agenda today 22:25:57 I know I keep pushing it back 22:25:57 mtreinish: sure np 22:26:12 ok, then let's move on 22:26:34 #topic Disabling nova v3 API tests by default 22:26:43 #link http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html 22:26:54 so if you didn't see the ML thread I started yesterday 22:27:09 I think it would be a good idea to move the v3 api tests to periodic/experimental 22:27:35 mtreinish: +1 22:27:51 I've pushed out the patches to start doing this, but I wanted to bring it up here 22:27:55 mtreinish: +1 22:28:00 btw me suggestion re scenario was to remove v3 scenario tests, not v2 ones 22:28:14 cyeoh: I don't think there are any v3 scenario tests 22:28:33 oh they probably haven't finished merging then 22:28:36 mtreinish: there are some patches for them 22:28:52 mtreinish: The one problem I see is that we made v2 depend on v3 classes a while back 22:28:55 ah, ok I must not have seen those, but I've got a big review backlog 22:29:14 I guess my concern is around regression, but if there simply isn't the capacity to do the testing anymore then they'll have to be dropped 22:29:16 mtreinish: Things like the client check change could easily cause v3 to break 22:29:35 I guess I'd flag that this is going to be an issue again when v2.1 rolls out and even more when v2.1microversions 22:29:43 dkranz: yeah we'll just have to be vigilant about the check experimental 22:29:45 so capacity issues are not going to go away 22:29:48 mtreinish: So the question is if we care about keeping the experimental/periodic working 22:30:06 I think we want at least periodic 22:30:16 cyeoh: yeah I've also started testing dropping the nova xml v2 tests 22:30:19 so we want to assure that no regression happened on v2.1 compared with what v3 is right now? is that right cyeoh ? 22:30:19 mtreinish: What if we left it in the gate for just tempest? 22:30:39 so we'll probably trade that for v2.1 microversions 22:30:49 if so maybe a grenade upgrading v3 => v2.1 once we start 2.1 + microversioning 22:30:57 mtreinish: I mean check 22:31:10 mtreinish: check not experimental for tempest 22:31:12 dkranz: it should be symmetric otherwise we'll have issues 22:31:31 mtreinish: what about having both periodic and experimental? 22:31:35 maurosr: it's still going to be configurable so we can add that to the grenade job 22:31:41 andreaf: that's what I proposed doing 22:31:51 What if we remove the v3 tests form all job and creating v3 only voting job ? 22:32:05 mtreinish: We can see what happens 22:32:08 andreaf: that doesn't solve the resource starvation which we're trying to address 22:32:11 v3 means just runs the v3 tests. 22:32:24 afazekas: ^^^ 22:32:32 afazekas: it just shifts were we use resources 22:32:37 dkranz: yeah I think let's give it a try 22:32:45 and if we have huge issues we can easily turn it back on 22:32:59 mtreinish: so with the periodic job - will the results be sent in a clearly identifiable email that we can filter on? 22:33:16 mtreinish: I predict that saying we will keep it working and removing it from the gate will turn out badly 22:33:16 cyeoh: yeah it goes to pseudo defunct openstack-qa list 22:33:33 BTW: resource starvation, do we want avoid the 'server' reuse in the test runs, or doing it more ? 22:33:49 cyeoh: also you can check here: http://logs.openstack.org/periodic-qa/ for the logs 22:34:16 mtreinish: ok. its just that anything "manual" is likely to get dropped just when we need it 22:34:28 cyeoh: yeah that's the risk 22:34:49 mtreinish: but if there some way I can get automate on my end - to at least signal a group of us when we have new failure than that's at least less bad 22:34:56 (might be able to do that through parsing the email) 22:35:36 cyeoh: yeah, I can also add a second email trigger for that job to send it somewhere else (I think) 22:35:43 anyway if we don't have the resources then we don't have any choice so I'll live with what we can get :-) 22:35:51 cyeoh: heh, ok 22:35:55 mtreinish: that would be really great! 22:36:05 we can work with -infra on figuring out a way to get what we need out of the reporting 22:36:12 mtreinish: sorry I have to drop off 22:36:13 ok 22:36:42 ok, then I'll start pushing out the rest of the patches to drop the v3 tests from the gate 22:36:49 cyeoh: honestly my other concern is that v3 wasn't really living up to the tempest standards of it's stable and versioned 22:36:59 and we were starting to hit patches that were trying to change it only in juno 22:37:06 #action mtreinish to finish pushing out patches to drop running v3 from the gate 22:37:36 sdague: well that's a good segway into the next topic 22:37:40 sdague: at this stage I just want to stop it from regressing whilst we're busy distracted with v2.1 on v3 and microversions 22:38:05 #topic Experimental tag (cyeoh) 22:38:09 cyeoh: sure, but it's experimental. I'm not sure what regressing really means at this point. 22:38:33 sdague: regressing means lots of unexpected fixups as we roll v3 features in v2.1microversions 22:39:27 so I think the experimental tag came up because in Nova we want to move to the model where new API features may be experimental for one cycle 22:40:16 because unfortunately we have a history of getting things wrong. We want to be able to signal to users that this API may change, but we think its stable (say 90% of the time we're right) 22:40:43 but having a tempest change to block *accidental* API changes would be really nice 22:40:59 to stop the situation which I've seen: 22:41:02 cyeoh: well from my tempest hat, I kind of hate that idea :) 22:41:08 1. developer makes breaking API change 22:41:17 because landing a test in tempest means you are committing to that API 22:41:18 2. sees unittest fail and *change* unittest 22:41:42 3. lots of other unittests are changed and reviewers don't notice and patch merges 22:42:01 but having a tempest test fail makes people rething what is going on (its like a safety net) 22:42:34 so that's fine, but as tempest scope has grown, I think we need to push this kind of responsibility back into the project 22:42:39 sdague: I don't see what is wrong with having an experimental api as long as it is marked as such 22:42:58 dkranz: it's a lot of extra review load when people show up wanting to change it 22:43:00 cyeoh: I get where you're coming from, but I think if you want to have tempest tests do that there should have to be consistency between all the releases the api is in 22:43:10 under the excuse that 'it was experiment!' 22:43:23 which means backporint breaking api changes to get it landed 22:43:38 branchless tempest doesn't really make this a possibility unless it's going to backport to all supported openstack branches at once 22:43:55 mtreinish: I don't think we'd want to backport the breaking change 22:44:14 we'd only want the test run against the latest version 22:44:22 cyeoh: yep, not really an option 22:44:45 if you want that, it should live in the nova tree 22:44:55 sdague: I see your POV around load on tempest and pushing this work back to the projects 22:45:03 sdague: So that is what Maru is proposing for neutron basically 22:45:10 dkranz: yes 22:45:15 which I think is really good idea 22:45:18 sdague: Which we agreed was a good idea 22:45:31 I guess I just see it as the QA project doing less for other openstack projects 22:45:32 dkranz: yes, I'm actually 100% fine with that approach 22:45:45 cyeoh: only so many cycles 22:46:06 as you are aware, most projects in openstack outnumber QA :) 22:46:21 sure, I understand. it does mean that the likelyhood of projects accidentally shooting themselves in the foot is higher :-) 22:46:35 cyeoh: only when experimental 22:46:46 right, and experimental is different 22:46:48 cyeoh: the tests would move into tempest when stable 22:47:01 stuff comes to tempest when it's something that's being committed to as a public interface 22:47:01 any chance that we keep doing it just adding a skip, changing the test once it's merged, as we do? I mean the only concern is really think twice when someone skips a test 22:47:19 maurosr: that's the current path but to unskip you need to backport the change 22:47:28 maurosr: The problem is that tempest and its reviewers are gatekeepers 22:47:36 and gatekeeping is expensive 22:47:43 maurosr: remember, we're evolving towards on tempest across all branches of all projects 22:47:44 yeah I know 22:48:07 cyeoh: backporting would be a problem from nova 2.1 pov? 22:48:28 maurosr: I'm not sure what you mean 22:48:43 v2.1 will look exactly like v2 22:49:37 ok, well we're at ~10min left so let's move on we've still got a few topics on the agenda 22:49:37 if we have discoverability to know that api is available, propose something, eventually break it cause it was wrong from the begining, then we accept that we need to skip test on tempest and backport it in nova first unstable version would solve the issue 22:49:52 honestly, the testing paradigm on micro versioning is going to require some effort as well. I think that effort is best spent there instead of chasing experimental 22:50:04 mtreinish: kk 22:50:25 maurosr, cyeoh: we can take this to -qa after the meeting 22:50:28 sure 22:50:29 #topic Neutron Testing 22:50:34 mtreinish: sure 22:50:36 mlavalle: are you around? 22:50:43 yes 22:50:56 we merged another of the api test patchsets today 22:51:07 so we only have another 2 to go of the original set 22:51:33 This week I am writing the scenarios tutorial that we are going to add to the tempest docs 22:51:36 mlavalle: so how close is the neutron full job? 22:51:50 because I'd honestly really like to cut over before we add any more neutron tests 22:51:58 so we can reclaim the smoke tag 22:52:05 sdague: +1, we really need to make that migration 22:52:22 sdague: the neutron full job is depending in some Neutron bugs 22:52:45 I will follow up with salvatore and see exactly where we are there 22:53:01 I will do that by early next week 22:53:05 mlavalle: great 22:53:49 abnd finally I am attending the LBaaS code sprint in SAT to make any adjustments to the current test 22:54:06 SAT == San Antonio 22:54:22 and for the sake of brevity that's all I have this week 22:54:36 mlavalle: ok 22:54:54 does anyone else have something to discuss about neutron testing? 22:54:58 otherwise let's move on 22:55:35 #topic Critical Reviews 22:55:49 ok does anyone have any reviews that they need to get eyes on 22:56:04 although considering the gate backup things will still be slow to get the +A probably 22:56:20 Hard coded 10.0.3.0/24 net in in the heat neutron stack was overlaping with several gate machine ip (host) https://bugs.launchpad.net/heat/+bug/1297560 most of the heat-slow test was failued becues of this. 22:56:22 Launchpad bug 1297560 in tempest "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,In progress] 22:56:39 afazekas: that's the patch you pushed out earlier today right? 22:56:41 #link https://review.openstack.org/#/c/99694/ 22:56:49 afazekas: ++! this might help heat-slow be voting again 22:56:51 mtreinish: yes 22:57:10 afazekas: ok yeah I'll take a look at that after the meeting 22:57:25 are there any other reviews? 22:58:28 ok then with ~2 min left I guess I'll open the floor 22:58:32 #topic open discussion 22:58:44 meera: for your bp that you put on the agenda you need to open a spec for it 22:58:58 meera: https://github.com/openstack/qa-specs/blob/master/README.rst 22:59:00 mtreinish: sure will do 22:59:20 meera: it should be pretty quick the specs for adding tests for a project don't need to have much detail 22:59:24 anyone get a chance to look at my stress test bp? 22:59:31 presented last week? 22:59:43 asselin: I have some draft comments I'll finish them up tomorrow 22:59:50 mtreinish, thanks 22:59:56 I think they were mostly around the formatting though 22:59:58 mtreinish: I would like to get some information on the infra/devstack changes needed to enable adding tests for the barbican service 23:00:16 meera: sure you can ask in -infra or -qa 23:00:28 ok well that's time 23:00:33 thanks everyone 23:00:42 #endmeeting