17:02:09 <sdague> #startmeeting OpenStack QA
17:02:10 <openstack> Meeting started Thu Mar 14 17:02:09 2013 UTC.  The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:13 <openstack> The meeting name has been set to 'openstack_qa'
17:02:24 <sdague> ok, who's here for QA meeting?
17:02:30 <mlavalle> I am
17:02:30 <ravikumar_hp> hi
17:02:31 <mtreinish> o/
17:02:31 <dwalleck> here
17:02:32 <davidkranz> here
17:02:45 <malini> here too
17:02:50 <sdague> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:02:58 <dwalleck> And sam who's looking over my shoulder
17:03:04 <sdague> threw a couple of things up on the agenda for today
17:03:11 <sdague> dwalleck: you should give him a whole computer :)
17:03:30 <chunwang> hi
17:03:30 <sdague> maybe lets start with the easy one - point #3
17:03:35 <davidkranz> Let's put the summit to last
17:03:46 <sdague> #topic cutting stable branch
17:03:53 <sdague> davidkranz: sounds good
17:04:06 <dwalleck> We're finishing something up, hopefully he'll get his soon
17:04:10 <sdague> so lets start off with the concrete one of when do we cut stable branch for grizzly
17:04:25 <sdague> davidkranz: when did that happen last time?
17:04:30 <jaypipes> doink, sorry for being late guys
17:04:36 <davidkranz> sdague: I think not until late.
17:04:46 <sdague> jaypipes: no worries
17:04:56 <davidkranz> THe problem is that the projects are rolling with respect to this issue.
17:05:27 <sdague> davidkranz: right, so the projects are in hard freeze now. And once we cut stable we could still fix issues on the stable branch
17:05:42 <sdague> but maybe we should pick a date regardless?
17:06:16 <jaypipes> sdague: ++
17:06:21 <davidkranz> THere are more useful grizzly tests going in.
17:06:32 <davidkranz> THe danger of waiting is a havana change breaks us.
17:06:51 <sdague> davidkranz: right, so how about we pick 1 week from now to branch stable
17:06:56 <davidkranz> From a test standpoint I think it would be better to wait a bit.
17:07:05 <sdague> no one should be landing anything funny in masters until after then
17:07:09 <jaypipes> davidkranz: one week from now sounds good.
17:07:12 <davidkranz> sdague: :)
17:07:22 <sdague> and we can still let people propose changes to stable after it's cut
17:07:29 <sdague> just review it with a more critical eye
17:07:29 <davidkranz> So be it.
17:07:38 <sdague> want an official vote on it?
17:07:42 <mtreinish> sdague: ++, 1 week sounds fine to me too
17:07:55 <sdague> #startvote stable cut for tempest is March 21st
17:07:56 <openstack> Unable to parse vote topic and options.
17:08:03 <jaypipes> #vote yes
17:08:11 <sdague> ok, apparently I don't know how to make the vote work right
17:08:16 <ravikumar_hp> #vote yes
17:08:18 <davidkranz> #vote yes
17:08:27 <sdague> #vote yes anyway :)
17:08:31 <jaypipes> sdague: need to include the valid vote options at end...
17:08:37 <jaypipes> hehe
17:08:48 <jaypipes> anyone *opposed* to one week from now?
17:08:50 <sdague> ok, well you'll have to teach me some time :)
17:09:14 <dwalleck> brb with some other folks
17:09:29 <sdague> ok, we'll consider that one resolved
17:09:33 <davidkranz> jaypipes: Who has the magic decoder ring to do it?
17:09:55 <jaypipes> davidkranz: to do the cut?
17:09:59 <davidkranz> Yes.
17:10:08 <jaypipes> davidkranz: we just work with the CI team, IIRC
17:10:12 <jaypipes> and maybe ttx
17:10:16 <davidkranz> jaypipes: k
17:10:22 <sdague> ok, I can take it as a todo to go figure out what we need to do
17:10:28 <jaypipes> sdague: thx
17:10:38 <sdague> #todo sdague to sort out process for stable branching on March 21
17:10:46 <sdague> #action sdague to sort out process for stable branching on March 21
17:11:04 <sdague> #topic Pending test for multiple server create adds 1 minute to gate. Is that OK? This is the "can't gate on everything issue".
17:11:24 <sdague> that one should be a reasonably concrete discussoin
17:11:28 <jaypipes> sdague: gotta link?
17:11:40 <sdague> davidkranz: do you have the link to that review?
17:11:49 <mtreinish> https://review.openstack.org/#/c/23511/
17:11:53 <jaypipes> #link https://review.openstack.org/#/c/23511/
17:11:59 <jaypipes> doh, jinx
17:12:09 <davidkranz> This test should be part of a full suite, but seems overkill for a gate.
17:12:24 <sdague> yeh, I'm on the fence. Seems like a good test to have, but I can also understand not wanting to increase the gate right now
17:12:27 <ravikumar_hp> I think it should not be a gated test
17:12:34 <davidkranz> We are looking at death by 1000 cuts for gate time.
17:12:48 <ravikumar_hp> should go into a separate folder like - stress
17:13:02 <sdague> ravikumar_hp: well I think it's a valid API test, so it should be in core tempest
17:13:14 <davidkranz> ravikumar_hp: As wel will discuss later or at the summit, these tests should be part of tempest.
17:13:15 <sdague> but it might be something we should flag as not run on normal gate jobs
17:13:24 <davidkranz> But we need a way to only gate on a subset of tests.
17:13:38 <sdague> anyway, I could go either way, davidkranz / jaypipes you guys have a pov on it
17:14:00 * jaypipes thinks if it is part of the official compute API, it needs to be in the gate.
17:14:02 <davidkranz> How about we compromise for now and drop the xml?
17:14:26 <mtreinish> jaypipes: I think it is an extension
17:14:31 <mtreinish> let me double check
17:14:35 <davidkranz> It is an extension.
17:15:02 <jaypipes> hmm...
17:15:08 <sdague> ok, so how about we hold it for now
17:15:18 <sdague> so don't put it into stable
17:15:25 <jaypipes> I don't see the test cases checking to see if the extension even is supported in the compute endpoint.
17:15:37 <sdague> yeh, that's probably a bigger issue
17:15:44 <jaypipes> if it's truly an extension, that is...
17:15:46 <davidkranz> jaypipes: We don't do that for most of the "extensions"
17:15:47 <sdague> which I don't know that we do right anywhere in tempest
17:15:55 <jaypipes> davidkranz: touche
17:16:00 <sdague> ok, so I think we're coming to the agreement to hold for now
17:16:05 <davidkranz> That issue is really for the summit.
17:16:10 <jaypipes> ya
17:16:11 <sdague> and we can reopen discussion post summit / stable
17:16:15 <sdague> I'll -2 it
17:16:16 <jaypipes> sounds good.
17:16:55 <jaypipes> sdague: next topic?
17:16:59 <sdague> #topic Blockers / Bugs
17:17:16 <sdague> any specific bugs or blockers or critical reviews we need the team to look at?
17:17:23 <davidkranz> I think we are in pretty good shape unless some one has somlething specific.
17:17:28 <sdague> (after this we'll dive into design summit topics)
17:17:45 <jaypipes> I'll have some time to get through reviews today
17:17:45 <davidkranz> sdague: Just one thing.
17:17:45 <sdague> yeh, just wanted to give folks a moment to speak up
17:18:05 <davidkranz> Should the keystone v2/v3 thing be handled in the same way as mtreinish did with images?
17:18:12 <afazekas> https://review.openstack.org/#/c/24013/ <-- rate limit handling is not good now
17:18:33 <sdague> davidkranz: I think the multiple directory approach is good
17:18:39 <sdague> so I'd be in favor of it
17:18:47 <davidkranz> ravikumar_hp: Can you recommend that?
17:19:12 <davidkranz> ravikumar_hp: Sorry to be such a pain about those submissions.
17:19:23 <sdague> afazekas: ok, I'll take a look at that one
17:19:31 <sdague> #link https://review.openstack.org/#/c/24013/
17:19:43 <ravikumar_hp> davidkranz: sorry . I missed it. what you want me to do
17:19:44 <sdague> #action need review eyes on https://review.openstack.org/#/c/24013/ to deal with rate limitting
17:19:57 <jaypipes> afazekas: that code looks excellent. I'll review fully shortly.
17:20:10 <davidkranz> ravikumar_hp: Model the v2/v3 identity tests the same way the glance tests are handling v1/v2
17:20:26 <davidkranz> ravikumar_hp: You are involved in those tests, right?
17:20:30 <ravikumar_hp> davidkranz: sure. we are resubmitting
17:20:44 <davidkranz> ravikumar_hp: Thanks!
17:20:45 <sdague> ravikumar_hp: yeh, mtreinish provided a model on doing multiple versions of the api in tree for glance, just follow that lead
17:21:00 <ravikumar_hp> sdague:ok
17:21:22 <mtreinish> ravikumar_hp: if there are any questions about it feel free to ping me too
17:21:28 <sdague> ok, any other bugs/reviews that need attention?
17:21:35 <ravikumar_hp> mtreinish: odf course
17:21:45 <andreaf> sdague: https://review.openstack.org/#/c/24433/
17:22:14 <sdague> andreaf: cool, I'll take a look later today
17:22:28 <andreaf> thanks
17:22:31 <sdague> ok, next topic, summit
17:22:37 <sdague> #topic Design Summit Sessions
17:22:52 <sdague> so there are already a number submitted at http://summit.openstack.org
17:23:07 <sdague> also mtreinish and I were brainstorming others at https://etherpad.openstack.org/havana-qa-infra-summit-brainstorm
17:23:30 <sdague> as a starting point for pruning down to what we think we really want to focus on there
17:24:38 <sdague> I'd love some commentary other thoughts on the etherpad to figure out if we don't need the sessions (it's not really a discussion, we're good with an existing solution), or if there are other thoughts about things to add
17:25:24 <davidkranz> sdague: I think it may take a few minutes for others to digest :)
17:25:31 <sdague> yep, that's fair :)
17:25:41 <sdague> I was hoping people were off reading :)
17:25:53 <RAXSam> yes still reading. :-)
17:26:29 <sdague> realize, it's a big brain dump, so think of it as scribble notes :)
17:26:36 <sdague> but it seemed better to start from that
17:26:48 <andreaf> sdague: formal processes and best practice are +++ if they make it easier to contribute tests and reduce the review cycles for new contributors (like myself ^^)
17:27:02 <jaypipes> andreaf: much agreed.
17:27:03 <ravikumar_hp> sdague: lot of topics
17:27:20 <davidkranz> Many of these are already in summit submissions.
17:27:23 <dwalleck> I like the idea in general of dashboards, especially for results/failures
17:27:27 <davidkranz> But many are not.
17:27:46 <sdague> davidkranz: yeh, I'm seeing that there is some overlap. Can you mark the dups in the etherpad?
17:27:53 <dwalleck> One of the problems we've had internally as well is handling all the data we generate
17:28:01 <davidkranz> sdague: On it.
17:28:04 <sdague> sorry, I was running low on time, so I just did the brain dump but not the dup check
17:28:25 <ravikumar_hp> sdague: not sure if we can adddress all those in summit . we need to prioritze -
17:28:30 <RAXSam> +1 the idea on moving towards working with dashboards, metrics, etc...
17:28:35 <andreaf> something that I'd like to see is whitebox tests: what is the future of that area, how to we support whitebox tests in a multinode cluster, do we want that to grow as part of tempest
17:29:07 <sdague> andreaf: yeh, right the whitebox discussion was another good one to have
17:29:19 <sdague> you want to either direct submit it, or put it on this list?
17:29:54 <andreaf> I'll put it on the list so people can comment on it
17:29:58 <dwalleck> I'm not sure how I fee about Tempest only supporting API validation. That might be okay for gating, but it's going to miss a lot of important defects
17:30:29 <sdague> dwalleck: right, that's the Tempest scope conversation, right?
17:30:41 <RAXSam> I'd also like to start thinking about things like logstash --> NoSQL --> Elastic Search for dashboarding/mining automated test logs
17:30:55 <davidkranz> dwalleck: Who suggested it should only support API validation?
17:31:08 <dwalleck> Nearly all the defects I end up finding having nothing to do with the API. They almost all have to do with the artifacts of the API call, like the instance, it's networking, etc
17:31:32 <dwalleck> sdague: right
17:31:39 <sdague> dwalleck: I agree
17:31:56 <sdague> this is more about just giving a guideline of what's in play and what's not for havana
17:32:01 <ravikumar_hp> sdague: i would to like test coverage process for new releases . right now it is adhoc based . we need to have coverage for all blueprints if applicable
17:32:10 <RAXSam> +1 on discussing Tempest Scope in general and what that focus should be
17:32:43 <RAXSam> +1 on most of our found defects involving artifacts generated by the API and the integration(s) between those artifacts
17:33:42 <jaypipes> +1 on being able to run the whitebox-type tests against a multi-node environment
17:34:32 <mlavalle> As a "newbie", I find the proposal to discuss foundational topics very useful: Best Practice Guide and Tempest Scope
17:34:35 <dwalleck> And we should probably define better what we mean by white box. I think we may all be thinking slightly differently there
17:34:55 <jaypipes> dwalleck: agreed.
17:35:10 <RAXSam> mlavalle: Hello and +1
17:35:16 <malini> mlavalle : +1 ..It will be awesome if we can have a Tempest 101
17:35:19 <sdague> yeh, whitebox is probably a topic by itself
17:35:19 <dwalleck> mlavalle: +1
17:35:58 <RAXSam> jaypipes: +1 on the mutli-node testing
17:36:24 <sdague> ok, I guess maybe it's worth a point of order to figure out how we move from the artifacts and discussions we've got going now to a schedule. Do people want some more time poking at the etherpad? How do we want to prioritize the topic lists?
17:36:35 <davidkranz> malini: I think that is what http://summit.openstack.org/cfp/details/87 is, at least in part.
17:37:22 <sdague> davidkranz / jaypipes / dwalleck you guys did this before, so curious how that next step goes
17:37:31 <davidkranz> sdague: Perhaps there should just be more summit session submissions and then we sort
17:37:39 <sdague> sure
17:37:54 <davidkranz> Thierrey said we will probably have 9-12 slots to be decided soon.
17:37:54 <jaypipes> sdague: looking over the proposed QA sessions at http://summit.openstack.org/, the only really glaring omission I think is the whitebox testing... unless dwalleck's "Beyond the API - End to End Testing of OpenStack" is really all about whitebox testing?
17:38:12 <jaypipes> dwalleck: ?
17:38:14 <mlavalle> sdague: why don't we let people vote on their 3 top topics on the etherpad over the next few days?
17:38:15 <sdague> ok, so I'd appreciate any markup in the etherpad to catch dupes
17:38:27 <sdague> mlavalle: well, I think we need them in the proper tool
17:38:30 <jaypipes> dwalleck: shall that be our whitebox testing session?
17:38:51 <mlavalle> sdague: or whatever tool is apporpriate
17:39:03 <dwalleck> jaypipes: I was referring to lower level instance validation, but I don't actually consider that to be white box
17:39:23 <jaypipes> dwalleck: ok
17:39:28 <dwalleck> But yeah, it seems like the right time to talk about that general topic
17:39:30 <jaypipes> dwalleck: just checkin :)
17:40:00 <sdague> ok, cool
17:40:04 <dwalleck> cool :)
17:40:14 <sdague> anything terrible on my list people want to throw out?
17:40:15 <davidkranz> IMO, we should be moving towards tempest being a real upstream acceptance test.
17:40:29 <davidkranz> With subsets for gating, etc.
17:40:46 <mlavalle> sdague: no.... you proposed good topics. Thanks for the effort
17:40:47 <sdague> right, well we can talk about that in the scope session
17:41:00 <jaypipes> sdague: I'm not sure that http://summit.openstack.org/cfp/details/117 belongs in the QA track
17:41:32 <davidkranz> jaypipes: Agreed. Nor design summit 101.
17:41:33 <sdague> jaypipes: I would agree with you on tat
17:41:47 <jaypipes> sdague: but there isn't an Ops track...
17:41:54 <sdague> yeh, so the qa-core team probably need to scrub
17:42:07 <jaypipes> davidkranz: yeah, loic's one should be in Process I think.
17:42:17 <RAXSam> +1 on the 117 no really belonging in the QA track (at least without better details) :-)
17:42:31 <davidkranz> jaypipes: I might well refuse it now.
17:42:39 <jaypipes> RAXSam: yeah, it's a great session proposal acutally, but just not in QA :)
17:42:43 <sdague> davidkranz / jaypipes: you guys have edit super powers on that? (I can help if you give me powers)
17:43:00 <jaypipes> sdague: no, I can not review, only view
17:43:01 <RAXSam> japypipes: Exactly...the topic is cool, just wrong slot. :-)
17:43:25 <jaypipes> ttx: any chance of getting an Operators track at the summiit?
17:43:27 <davidkranz> sdague: I think there is only one person which is me at the moment.
17:43:34 <sdague> ok
17:43:36 <jaypipes> all hail davidkranz
17:43:48 <sdague> don't make him mad, or he'll kill you session!
17:43:50 <davidkranz> jaypipes: You pushed it at me last time :)
17:44:04 <sdague> jaypipes is the master of delegation :)
17:44:14 <jaypipes> davidkranz: I am not complaining :)
17:44:50 <davidkranz> OK, so we just need people to push new sessions based on the etherpad and then we can sort.
17:45:01 <jaypipes> yep, think that's right.
17:45:03 <davidkranz> We can have one more discussion of this next week I think.
17:45:09 <jaypipes> ++
17:45:13 <davidkranz> Based on the actual submissions.
17:45:36 <davidkranz> Hopefully we will now next week how many sessions we have.
17:45:43 <davidkranz> ^^^ know
17:45:45 <sdague> ok, so. I'll take the list I had, excepting dupes, and put them in
17:45:53 <sdague> then we can prioritize next week
17:45:55 <davidkranz> sdague: Thanks.
17:45:57 <jaypipes> sdague: sounds good.
17:46:18 <sdague> ok. the last agenda topic was on best practices.
17:46:21 <davidkranz> The one question I had for now is whether the OPenStackTM thing should go in process or tempest.
17:46:30 <davidkranz> I could see it either way.
17:46:40 <jaypipes> davidkranz: Process, IMO.
17:46:46 <sdague> yeh, Process
17:46:55 <davidkranz> OK, we'll see if they accept it :)
17:47:03 <sdague> #topic Guidelines for Tempest commits
17:47:07 <sdague> last topic on agenda
17:47:17 <jaypipes> davidkranz: you talking about http://summit.openstack.org/cfp/details/16 ?
17:47:48 <davidkranz> jaypipes: No, the item in the etherpad about using tempest as a gate for openstack trademark.
17:48:02 <jaypipes> davidkranz: k, the two are similar
17:48:10 <jaypipes> davidkranz: might be worth combining.
17:48:17 <jaypipes> davidkranz: check with mordred
17:48:24 <sdague> yeh, that sounds reasonable
17:48:24 <davidkranz> jaypipes: Yes, at the implementation level.
17:48:40 <davidkranz> jaypipes: Will do.
17:48:42 <sdague> davidkranz: you going to take the todo and circle with mordred / ttx on it?
17:48:51 <davidkranz> sdague: Yes.
17:49:21 <sdague> #action davidkranz to discuss with mordred / ttx on where tempest testing for openstack trademark should go
17:49:47 <sdague> ok, last thing I wanted to make sure we brought up was just pointing lots of people (especially new folks) at - https://wiki.openstack.org/wiki/Gerrit_Workflow#Committing_Changes
17:49:55 <jaypipes> ++
17:50:01 <sdague> having sane commit messages is really useful
17:50:14 <sdague> and I've been -1ing a bunch of patches because they didn't have them
17:50:15 <mtreinish> sdague: this one too: https://wiki.openstack.org/wiki/GitCommitMessages
17:50:38 <sdague> ah, yeh, that one is better
17:50:42 <sdague> #link https://wiki.openstack.org/wiki/GitCommitMessages
17:51:01 <sdague> so if you are a new contributor, or have new contribs on your team, please go take a look at that
17:51:21 <sdague> #topic open discussion
17:51:30 <sdague> it's free for all time :)
17:51:34 <sdague> anything else from folks
17:52:03 * afazekas if I can get passport in time I will be on the summit
17:52:12 <mtreinish> afazekas: cool
17:52:35 <sdague> awesome
17:53:28 <sdague> anything else from folks?
17:53:34 <andreaf> I did not have time to follow-up anymore on enabling ssh tests, sorry about that - I'll pick it up again as soon as I can
17:53:45 <sdague> going once...
17:53:52 <sdague> going twice...
17:54:03 <dwalleck> adios!
17:54:07 <sdague> ok, lets call it. Thanks all
17:54:12 <sdague> #endmeeting