22:00:48 <mtreinish> #startmeeting qa
22:00:48 <openstack> Meeting started Thu Jun 12 22:00:48 2014 UTC and is due to finish in 60 minutes.  The chair is mtreinish. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:00:51 <openstack> The meeting name has been set to 'qa'
22:00:54 <mtreinish> Hi who's here today?
22:00:59 <dkranz> Hi
22:01:02 <masayuki_> hi
22:01:09 <mtreinish> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Proposed_Agenda_for_June_12_2014_.282200_UTC.29
22:01:09 <maurosr> hi
22:01:14 <mtreinish> ^^^ Today's agenda
22:01:21 <mlavalle> hi
22:01:26 <meera> hi
22:01:27 <dpaterson> hi
22:02:05 <mtreinish> ok, lets get started
22:02:12 <mtreinish> #topic Specs Review
22:02:14 <cyeoh> hi
22:02:24 <mtreinish> dkranz: you had a spec on the agenda
22:02:31 <mtreinish> #link https://review.openstack.org/93037
22:02:45 <dmorita> Hi
22:02:46 <dkranz> mtreinish: Yes, I just wanted a decision about the xml issue per the two choices I added to the agenda
22:03:57 <mtreinish> dkranz: I don't see how it's much more work. It's just an extra call in the xml clients
22:04:04 <mtreinish> it's a lot of copy and paste
22:04:14 <dkranz> mtreinish: Yes. I guess we should do it.
22:04:29 <afazekas> hi
22:04:39 <dkranz> afazekas: late night :)
22:04:53 <mtreinish> dkranz: heh, ok then I guess we agree. Is there anything else on this spec?
22:05:03 <dkranz> mtreinish: No, I think it is good to go.
22:05:18 <mtreinish> ok, then does anyone else have a spec review that they'd like to bring up?
22:06:21 <dpaterson> Quick chat on: https://review.openstack.org/#/c/91725/?
22:06:34 <mtreinish> dpaterson: ok
22:06:52 <afazekas> A little bit related to this https://review.openstack.org/#/c/95794/ . We should print the console log an all ssh connection failure ASAP
22:07:11 <dpaterson> So it seems that there is need for a cleanup type tool but quite different than I originally planned
22:07:22 <dpaterson> No database cleanup for instance
22:07:51 <dpaterson> I am going to update the spec in the next day or so but would like input before I do.
22:08:16 <mtreinish> dpaterson: yeah having a leak detector/cleanup tool is something we've talked about for a long time
22:08:24 <mtreinish> dpaterson: I think I put most of my thoughts in the review
22:09:17 <mtreinish> dpaterson: you can ping me whenever on -qa if you've got specific questions when you're re-drafting it
22:09:18 <dpaterson> So my take away from the comments I received is that the tool should help locate the root cause in OpenStack.
22:09:28 <afazekas> #link https://review.openstack.org/#/c/35516/
22:09:45 <afazekas> My old approach for cleaning things..
22:09:48 <dpaterson> mtrenish: sounds good
22:10:50 <cyeoh> yea leak detection seems more important than a cleanup tool - so we can fix the test
22:11:16 <mtreinish> afazekas: on that spec if you're ready for reviews on you got to make it pass the docs jobs and have a lp bp to match the spec
22:11:27 <mtreinish> cyeoh: yeah, they're tightly coupled
22:11:46 <afazekas> cyeoh: the old approach is easy, the other one is complicated with handling auth and multiple clients , tenant isolation ...
22:11:58 <mtreinish> ok are there any other specs people want to bring up?
22:12:08 <cyeoh> mtreinish: right, except we don't want to get into the habit of just always running the cleanup tool because its the easy thing to do
22:12:40 <mtreinish> cyeoh: I think that's fine as long as reports what it's doing
22:12:46 <mtreinish> so we can debug it
22:13:42 <mtreinish> the point at which we identify leaked resources, running a delete on it after reporting it is no real extra effort
22:14:34 <mtreinish> ok, if there aren't any other specs let's move on
22:14:48 <mtreinish> #topic Blueprints
22:15:01 <mtreinish> so Juno-1 was this week
22:15:08 <mtreinish> we had 4 bps targetting it:
22:15:10 <mtreinish> #link https://launchpad.net/tempest/+milestone/juno-1
22:15:11 <afazekas> mtreinish: do it inside a parallel test suite  is complicated
22:15:46 <mtreinish> 2 of them are still under spec review so if everyone can prioritize those reviews that would be great
22:15:53 <mtreinish> sdague, andreaf: are you guys around?
22:16:08 <mtreinish> afazekas: no one said it was going to be easy :)
22:16:18 <dkranz> mtreinish: The stress one also has lots of code to review
22:16:23 <sdague> mtreinish: yeh, sorry
22:16:26 <andreaf> I'm here... On my mobile thought
22:16:47 <dkranz> andreaf: Wow
22:17:04 <mtreinish> sdague, andreaf: I was just wondering what the status of your j-1 targetted bps were
22:17:07 <sdague> yeh, so all my bps are wrecked after discovering javelin wasn't really working right at summit, and that was wrecked by the gate backup
22:17:22 <mtreinish> sdague: ok then I'll push it back to j-2
22:17:26 <sdague> yeh
22:17:41 <mtreinish> andreaf: yours was:  Tempest support for testing against multiple keystone API versions
22:17:49 <sdague> also, the outstanding work on branchless is basically the extensions lists
22:18:02 <dkranz> sdague: what more needs to be done for branchless tempest?
22:18:05 <sdague> which I can give someone a quick overview on how to do it
22:18:20 <sdague> dkranz: realistically I think it's just extensions
22:18:26 <mtreinish> sdague: isn't just spin up icehouse devstack and grab the feature flags
22:18:35 <mtreinish> then dump it in the devstack gate yaml
22:18:47 <sdague> mtreinish: well the processor for that is also needed
22:18:59 <sdague> so the communication path to actually get those set in tempest.conf
22:19:19 <sdague> that's actually the part that will take work
22:19:48 <mtreinish> ahh, ok I assumed that was already done
22:19:53 <mtreinish> yeah that would take a bit more effort
22:19:58 <sdague> no, we didn't need that for services
22:20:03 <sdague> as that's actually set in devstack
22:20:14 <sdague> everything in there so far was setable in localrc
22:20:46 <sdague> so ... volunteers welcomed :)
22:20:51 <mtreinish> ok, yeah then let's definitely push it back to j-2
22:20:55 <sdague> yep
22:20:56 <andreaf> mtreinish: I'm back - on a laptop this time
22:21:03 <mtreinish> andreaf: heh ok
22:21:13 <mtreinish> I was just asking about your open j-1 bp
22:21:21 <mtreinish> andreaf: https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests
22:21:26 <mtreinish> should we push it back to j-2?
22:22:43 <andreaf> mtreinish: on my side the multi auth version is stuck on the scenario test, because of missing features in the official clients
22:23:07 <andreaf> mtreinish: yes
22:23:18 <mtreinish> andreaf: ok then let's defer it, is there anything we can do to grease the wheels on getting the v3 stuff in the official clients?
22:23:38 <andreaf> I think that's the only one on j-1
22:24:01 <mtreinish> andreaf: you also have an open spec review, but I'll push that one back :)
22:24:17 <andreaf> mtreinish: so re bp, did anyone have a chance to have a look at https://review.openstack.org/#/c/98400/ already?
22:24:28 <andreaf> mtreinish: oh ok
22:24:32 <andreaf> thanks
22:24:36 <mtreinish> andreaf: not yet
22:24:49 <mtreinish> andreaf: I;ll take a look after the meeting
22:25:04 <mtreinish> ok that's all I had for bps, does anyone have anything else on the topic?
22:25:27 <andreaf> I wanted to ask about the tempest server /client / test repo
22:25:48 <andreaf> mainly about how we split work into different bps
22:25:48 <mtreinish> andreaf: ok, can we save it for the open discussion, we've got a full agenda today
22:25:57 <mtreinish> I know I keep pushing it back
22:25:57 <andreaf> mtreinish: sure np
22:26:12 <mtreinish> ok, then let's move on
22:26:34 <mtreinish> #topic Disabling nova v3 API tests by default
22:26:43 <mtreinish> #link http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html
22:26:54 <mtreinish> so if you didn't see the ML thread I started yesterday
22:27:09 <mtreinish> I think it would be a good idea to move the v3 api tests to periodic/experimental
22:27:35 <dkranz> mtreinish: +1
22:27:51 <mtreinish> I've pushed out the patches to start doing this, but I wanted to bring it up here
22:27:55 <andreaf> mtreinish: +1
22:28:00 <cyeoh> btw me suggestion re scenario was to remove v3 scenario tests, not v2 ones
22:28:14 <mtreinish> cyeoh: I don't think there are any v3 scenario tests
22:28:33 <cyeoh> oh they probably haven't finished merging then
22:28:36 <andreaf> mtreinish: there are some patches for them
22:28:52 <dkranz> mtreinish: The one problem I see is that we made v2 depend on v3 classes a while back
22:28:55 <mtreinish> ah, ok I must not have seen those, but I've got a big review backlog
22:29:14 <cyeoh> I guess my concern is around regression, but if there simply isn't the capacity to do the testing anymore then they'll have to be dropped
22:29:16 <dkranz> mtreinish: Things like the client check change could easily cause v3 to break
22:29:35 <cyeoh> I guess I'd flag that this is going to be an issue again when v2.1 rolls out and even more when v2.1microversions
22:29:43 <mtreinish> dkranz: yeah we'll just have to be vigilant about the check experimental
22:29:45 <cyeoh> so capacity issues are not going to go away
22:29:48 <dkranz> mtreinish: So the question is if we care about keeping the experimental/periodic working
22:30:06 <cyeoh> I think we want at least periodic
22:30:16 <mtreinish> cyeoh: yeah I've also started testing dropping the nova xml v2 tests
22:30:19 <maurosr> so we want to assure that no regression happened on v2.1 compared with what v3 is right now? is that right cyeoh ?
22:30:19 <dkranz> mtreinish: What if we left it in the gate for just tempest?
22:30:39 <mtreinish> so we'll probably trade that for v2.1 microversions
22:30:49 <maurosr> if so maybe a grenade upgrading v3 => v2.1 once we start 2.1 + microversioning
22:30:57 <dkranz> mtreinish: I mean check
22:31:10 <dkranz> mtreinish: check not experimental for tempest
22:31:12 <mtreinish> dkranz: it should be symmetric otherwise we'll have issues
22:31:31 <andreaf> mtreinish: what about having both periodic and experimental?
22:31:35 <mtreinish> maurosr: it's still going to be configurable so we can add that to the grenade job
22:31:41 <mtreinish> andreaf: that's what I proposed doing
22:31:51 <afazekas> What if we remove the v3 tests form all job and creating v3 only voting job ?
22:32:05 <dkranz> mtreinish: We can see what happens
22:32:08 <mtreinish> andreaf: that doesn't solve the resource starvation which we're trying to address
22:32:11 <afazekas> v3 means just runs the v3 tests.
22:32:24 <mtreinish> afazekas: ^^^
22:32:32 <mtreinish> afazekas: it just shifts were we use resources
22:32:37 <mtreinish> dkranz: yeah I think let's give it a try
22:32:45 <mtreinish> and if we have huge issues we can easily turn it back on
22:32:59 <cyeoh> mtreinish: so with the periodic job - will the results be sent in a clearly identifiable email that we can filter on?
22:33:16 <dkranz> mtreinish: I predict that saying we will keep it working and removing it from the gate will turn out badly
22:33:16 <mtreinish> cyeoh: yeah it goes to pseudo defunct openstack-qa list
22:33:33 <afazekas> BTW: resource starvation, do we want avoid the 'server' reuse in the test runs, or doing it more ?
22:33:49 <mtreinish> cyeoh: also you can check here: http://logs.openstack.org/periodic-qa/ for the logs
22:34:16 <cyeoh> mtreinish: ok. its just that anything "manual" is likely to get dropped just when we need it
22:34:28 <mtreinish> cyeoh: yeah that's the risk
22:34:49 <cyeoh> mtreinish: but if there some way I can get automate on my end - to at least signal a group of us when we have new failure than that's at least less bad
22:34:56 <cyeoh> (might be able to do that through parsing the email)
22:35:36 <mtreinish> cyeoh: yeah, I can also add a second email trigger for that job to send it somewhere else (I think)
22:35:43 <cyeoh> anyway if we don't have the resources then we don't have any choice so I'll live with what we can get :-)
22:35:51 <mtreinish> cyeoh: heh, ok
22:35:55 <cyeoh> mtreinish: that would be really great!
22:36:05 <mtreinish> we can work with -infra on figuring out a way to get what we need out of the reporting
22:36:12 <andreaf> mtreinish: sorry I have to drop off
22:36:13 <cyeoh> ok
22:36:42 <mtreinish> ok, then I'll start pushing out the rest of the patches to drop the v3 tests from the gate
22:36:49 <sdague> cyeoh: honestly my other concern is that v3 wasn't really living up to the tempest standards of it's stable and versioned
22:36:59 <sdague> and we were starting to hit patches that were trying to change it only in juno
22:37:06 <mtreinish> #action mtreinish to finish pushing out patches to drop running v3 from the gate
22:37:36 <mtreinish> sdague: well that's a good segway into the next topic
22:37:40 <cyeoh> sdague: at this stage I just want to stop it from regressing whilst we're busy distracted with v2.1 on v3 and microversions
22:38:05 <mtreinish> #topic Experimental tag (cyeoh)
22:38:09 <sdague> cyeoh: sure, but it's experimental. I'm not sure what regressing really means at this point.
22:38:33 <cyeoh> sdague: regressing means lots of unexpected fixups as we roll v3 features in v2.1microversions
22:39:27 <cyeoh> so I think the experimental tag came up because in Nova we want to move to the model where new API features may be experimental for one cycle
22:40:16 <cyeoh> because unfortunately we have a history of getting things wrong. We want to be able to signal to users that this API may change, but we think its stable (say 90% of the time we're right)
22:40:43 <cyeoh> but having a tempest change to block *accidental* API changes would be really nice
22:40:59 <cyeoh> to stop the situation which I've seen:
22:41:02 <sdague> cyeoh: well from my tempest hat, I kind of hate that idea :)
22:41:08 <cyeoh> 1. developer makes breaking API change
22:41:17 <sdague> because landing a test in tempest means you are committing to that API
22:41:18 <cyeoh> 2. sees unittest fail and *change* unittest
22:41:42 <cyeoh> 3. lots of other unittests are changed and reviewers don't notice and patch merges
22:42:01 <cyeoh> but having a tempest test fail makes people rething what is going on (its like a safety net)
22:42:34 <sdague> so that's fine, but as tempest scope has grown, I think we need to push this kind of responsibility back into the project
22:42:39 <dkranz> sdague: I don't see what is wrong with having an experimental api as long as it is marked as such
22:42:58 <sdague> dkranz: it's a lot of extra review load when people show up wanting to change it
22:43:00 <mtreinish> cyeoh: I get where you're coming from, but I think if you want to have tempest tests do that there should have to be consistency between all the releases the api is in
22:43:10 <sdague> under the excuse that 'it was experiment!'
22:43:23 <mtreinish> which means backporint breaking api changes to get it landed
22:43:38 <sdague> branchless tempest doesn't really make this a possibility unless it's going to backport to all supported openstack branches at once
22:43:55 <cyeoh> mtreinish: I don't think we'd want to backport the breaking change
22:44:14 <cyeoh> we'd only want the test run against the latest version
22:44:22 <sdague> cyeoh: yep, not really an option
22:44:45 <sdague> if you want that, it should live in the nova tree
22:44:55 <cyeoh> sdague: I see your POV around load on tempest and pushing this work back to the projects
22:45:03 <dkranz> sdague: So that is what Maru is proposing for neutron basically
22:45:10 <sdague> dkranz: yes
22:45:15 <sdague> which I think is really good idea
22:45:18 <dkranz> sdague: Which we agreed was a good idea
22:45:31 <cyeoh> I guess I just see it as the QA project doing less for other openstack projects
22:45:32 <sdague> dkranz: yes, I'm actually 100% fine with that approach
22:45:45 <sdague> cyeoh: only so many cycles
22:46:06 <sdague> as you are aware, most projects in openstack outnumber QA :)
22:46:21 <cyeoh> sure, I understand. it does mean that the likelyhood of projects accidentally shooting themselves in the foot is higher :-)
22:46:35 <dkranz> cyeoh: only when experimental
22:46:46 <sdague> right, and experimental is different
22:46:48 <dkranz> cyeoh: the tests would move into tempest when stable
22:47:01 <sdague> stuff comes to tempest when it's something that's being committed to as a public interface
22:47:01 <maurosr> any chance that we keep doing it just adding a skip, changing the test once it's merged, as we do? I mean the only concern is really think twice when someone skips a test
22:47:19 <mtreinish> maurosr: that's the current path but to unskip you need to backport the change
22:47:28 <dkranz> maurosr: The problem is that tempest and its reviewers are gatekeepers
22:47:36 <dkranz> and gatekeeping is expensive
22:47:43 <sdague> maurosr: remember, we're evolving towards on tempest across all branches of all projects
22:47:44 <maurosr> yeah I know
22:48:07 <maurosr> cyeoh: backporting would be a problem from nova 2.1 pov?
22:48:28 <cyeoh> maurosr: I'm not sure what you mean
22:48:43 <cyeoh> v2.1 will look exactly like v2
22:49:37 <mtreinish> ok, well we're at ~10min left so let's move on we've still got a few topics on the agenda
22:49:37 <maurosr> if we have discoverability to know that api is available, propose something, eventually break it cause it was wrong from the begining, then we accept that we need to skip test on tempest and backport it in nova first unstable version would solve the issue
22:49:52 <sdague> honestly, the testing paradigm on micro versioning is going to require some effort as well. I think that effort is best spent there instead of chasing experimental
22:50:04 <sdague> mtreinish: kk
22:50:25 <mtreinish> maurosr, cyeoh: we can take this to -qa after the meeting
22:50:28 <maurosr> sure
22:50:29 <mtreinish> #topic Neutron Testing
22:50:34 <cyeoh> mtreinish: sure
22:50:36 <mtreinish> mlavalle: are you around?
22:50:43 <mlavalle> yes
22:50:56 <mlavalle> we merged another of the api test patchsets today
22:51:07 <mlavalle> so we only have another 2 to go of the original set
22:51:33 <mlavalle> This week I am writing the scenarios tutorial that we are going to add to the tempest docs
22:51:36 <sdague> mlavalle: so how close is the neutron full job?
22:51:50 <sdague> because I'd honestly really like to cut over before we add any more neutron tests
22:51:58 <sdague> so we can reclaim the smoke tag
22:52:05 <mtreinish> sdague: +1, we really need to make that migration
22:52:22 <mlavalle> sdague: the neutron full job is depending in some Neutron bugs
22:52:45 <mlavalle> I will follow up with salvatore and see exactly where we are there
22:53:01 <mlavalle> I will do that by early next week
22:53:05 <sdague> mlavalle: great
22:53:49 <mlavalle> abnd finally I am attending the LBaaS code sprint in SAT to make any adjustments to the current test
22:54:06 <mlavalle> SAT == San Antonio
22:54:22 <mlavalle> and for the sake of brevity that's all I have this week
22:54:36 <mtreinish> mlavalle: ok
22:54:54 <mtreinish> does anyone else have something to discuss about neutron testing?
22:54:58 <mtreinish> otherwise let's move on
22:55:35 <mtreinish> #topic Critical Reviews
22:55:49 <mtreinish> ok does anyone have any reviews that they need to get eyes on
22:56:04 <mtreinish> although considering the gate backup things will still be slow to get the +A probably
22:56:20 <afazekas> Hard coded 10.0.3.0/24 net in in the heat neutron stack was overlaping with several gate machine ip (host)  https://bugs.launchpad.net/heat/+bug/1297560 most of the heat-slow test was failued becues of this.
22:56:22 <uvirtbot> Launchpad bug 1297560 in tempest "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,In progress]
22:56:39 <mtreinish> afazekas: that's the patch you pushed out earlier today right?
22:56:41 <afazekas> #link https://review.openstack.org/#/c/99694/
22:56:49 <stevebaker> afazekas: ++! this might help heat-slow be voting again
22:56:51 <afazekas> mtreinish: yes
22:57:10 <mtreinish> afazekas: ok yeah I'll take a look at that after the meeting
22:57:25 <mtreinish> are there any other reviews?
22:58:28 <mtreinish> ok then with ~2 min left I guess I'll open the floor
22:58:32 <mtreinish> #topic open discussion
22:58:44 <mtreinish> meera: for your bp that you put on the agenda you need to open a spec for it
22:58:58 <mtreinish> meera: https://github.com/openstack/qa-specs/blob/master/README.rst
22:59:00 <meera> mtreinish: sure will do
22:59:20 <mtreinish> meera: it should be pretty quick the specs for adding tests for a project don't need to have much detail
22:59:24 <asselin> anyone get a chance to look at my stress test bp?
22:59:31 <asselin> presented last week?
22:59:43 <mtreinish> asselin: I have some draft comments I'll finish them up tomorrow
22:59:50 <asselin> mtreinish, thanks
22:59:56 <mtreinish> I think they were mostly around the formatting though
22:59:58 <meera> mtreinish: I would like to get some information on the infra/devstack changes needed to enable adding tests for the barbican service
23:00:16 <mtreinish> meera: sure you can ask in -infra or -qa
23:00:28 <mtreinish> ok well that's time
23:00:33 <mtreinish> thanks everyone
23:00:42 <mtreinish> #endmeeting