17:01:16 <sdague> #startmeeting qa
17:01:17 <openstack> Meeting started Thu May 16 17:01:16 2013 UTC.  The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:21 <openstack> The meeting name has been set to 'qa'
17:01:26 <jaypipes> sdague: I was thinking more along Voltron lines... :)
17:01:34 <sdague> ok, who's here for QA meeting
17:01:37 <sdague> jaypipes: awesome :)
17:01:38 <jaypipes> o/
17:01:38 <mlavalle> I am
17:01:43 * tkammer waves
17:01:58 <jaypipes> davidkranz is away...
17:02:15 <afazekas> hi
17:02:16 <sdague> appologies for not pulling an agenda together in advance, was down in NYC at a Linux Foundation event the last couple of days
17:02:22 <jaypipes> no worries.
17:02:24 <sdague> so here's my thoughts on agenda
17:02:35 <sdague> 1) blueprints
17:02:49 <sdague> 2) any critical reviews/bugs to look at
17:02:52 <sdague> 3) open meeting
17:03:02 <sdague> #topic blueprints for H1
17:03:36 <sdague> #link https://launchpad.net/tempest/+milestone/havana-1
17:04:00 <sdague> for me, the launchpad cleanup is coming along good. will finally complete that tomorrow
17:04:10 <jaypipes> great job.
17:04:17 <afazekas> +1
17:04:19 <sdague> the tempest restructure is about 1/3 in, 1/3 in review, and 1/3 to go
17:04:29 <sdague> currently blocked on this volumes race we uncovered
17:04:43 <jaypipes> sdague: yeah, I saw that. anything I can do to help?
17:04:52 <sdague> which giulivo and jgriffith are helping get to the bottom of
17:05:00 <giulivo> I've got it sdague
17:05:06 <sdague> jaypipes: right now, I don't think so. we've got all the right folks on it
17:05:08 <sdague> giulivo: awesome
17:05:13 <jaypipes> excellent.
17:05:14 <giulivo> the scheduler updates the volume status every minute, which is too long for the tests execution
17:05:21 <jaypipes> ah
17:05:22 <sdague> giulivo: nice find
17:05:40 <sdague> giulivo: so do we have a way to tune that for gate?
17:05:48 <jaypipes> Is Bagashree Lokare here, by any chance?
17:05:49 <giulivo> look for "Received volume service update" in the sched log
17:05:55 <giulivo> not sure, probably it is configurable behaviour
17:06:07 <giulivo> haven't checked yet
17:06:13 <sdague> ok, cool. well I'll count that as problem sorted, we just need to get the right fix together
17:06:16 <sdague> nice job on that
17:06:34 <sdague> any other blueprints people are working for for H1 that they want to report in on?
17:07:34 <sdague> ok... I'll take that as a no :)
17:07:59 <sdague> next week we'll plan to do rollcall on that, because any non completed (or close ones) we'll need to push to h2
17:08:00 <afazekas> IMHO https://blueprints.launchpad.net/tempest/+spec/ssh-auth-strategy this should be approved
17:08:04 <jaypipes> sdague: hoping that Shree will set a priority and status on her 5 blueprints
17:08:31 <sdague> afazekas: sure
17:08:38 <afazekas> https://blueprints.launchpad.net/tempest/+spec/add-basic-heat-tests this too
17:09:02 <sdague> afazekas: can you ping that author to but the detailed specification in the wiki instead of etherpad?
17:09:09 <sdague> etherpads are too easy to get borked
17:09:24 <sdague> then I'll happily move it to an approved state
17:09:26 <jaypipes> afazekas: ++ on ssh auth strategy. But I would love to see the blueprint title by more descriptive of the actual task. As it sounds, it seems like the blueprint is about the Keystone endpoints, but it's about SSHing into a VM (whitebox testing)
17:09:36 <jaypipes> RAX-Sam: hola.
17:09:47 <jaypipes> RAX-Sam: wish Daryl a happy birthday from us. :)
17:09:48 <RAX-Sam> Hey Jay
17:09:51 <jaypipes> belated...
17:10:00 <sdague> afazekas: I'm good with https://blueprints.launchpad.net/tempest/+spec/add-basic-heat-tests as well
17:10:03 <afazekas> sdague: I'll ask him
17:10:05 <RAX-Sam> :-D He is here just having trouble connecting to Freenode... :-)
17:10:28 <sdague> afazekas: thanks
17:10:28 <jaypipes> ah :)
17:10:44 <sdague> ok, any other blueprint updates?
17:11:20 <sdague> ok, next topic
17:11:31 <sdague> #topic critical reviews
17:11:56 <sdague> any reviews we need to jump on that aren't getting enough eyes right now?
17:12:12 <giulivo> doh had two blueprints to ask about: https://blueprints.launchpad.net/tempest/+spec/add-snapshot-tests
17:12:21 <giulivo> and https://blueprints.launchpad.net/tempest/+spec/set-gate-attribute
17:12:39 <jaypipes> sdague: yeah, I also still have a few bluepriont-related things..
17:12:41 <giulivo> the second one looks tricky as it gets into the labeling 'discussion'
17:12:56 <sdague> jaypipes: oops, sorry for jumping the gun :)
17:13:03 <jaypipes> no worries :)
17:13:04 <sdague> #topic blueprints
17:13:30 <jaypipes> sdague: https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests seems to be overlapped with your in-progress https://blueprints.launchpad.net/tempest/+spec/tempest-repo-restructure
17:13:31 <sdague> giulivo: I think we have enough rought concensus on the gate attribute for now. If we need to make a change in the future, we can
17:13:47 <sdague> jaypipes: partially
17:13:56 <sdague> I think that blueprint was for some new scenario tests
17:13:56 <afazekas> so how to use the smoke flag ?
17:14:07 <afazekas> Do we need services related flags ?
17:14:09 <jaypipes> sdague: and https://blueprints.launchpad.net/tempest/+spec/quantum-quota-basic-tests and https://blueprints.launchpad.net/tempest/+spec/quantum-quota-extension-test seem to also be identical.
17:14:23 <sdague> jaypipes: yes, there are massively duplicated quantum ones
17:14:33 <jaypipes> sdague: on the scenario tests, then, I would like to see more details on the former.
17:14:37 <sdague> mlavalle, were you going to try to condense those
17:15:04 <mlavalle> sdague: Yes, I was too busy this week, moving my family from Houston to San Antonio
17:15:24 <mlavalle> sdague: But I will follow up with this
17:15:25 <sdague> jaypipes: I'm happy with that as feedback to the author on the scenario tests before it moves to approved, would you like to provide it?
17:15:28 <sdague> mlavalle: thanks
17:15:58 <jaypipes> sdague: OK to approve https://blueprints.launchpad.net/tempest/+spec/add-logging-configuration? It's in code review now...
17:16:12 <sdague> jaypipes: +1 on that
17:16:15 <jaypipes> sdague: yes on scenario.
17:16:22 <jaypipes> will provide feedback on whiteboard.
17:16:43 <sdague> #action jaypipes to provide feedback on https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests
17:16:50 <sdague> great, thanks
17:17:00 <sdague> we lost one of giulivo's
17:17:08 <jaypipes> sdague: logging config BP approved.
17:17:08 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/add-snapshot-tests
17:17:13 <sdague> jaypipes: thanks!
17:17:38 <sdague> I'm good with approving https://blueprints.launchpad.net/tempest/+spec/add-snapshot-tests
17:17:47 <sdague> any objections?
17:17:57 <jaypipes> looking...
17:18:00 <giulivo> it is "slow"
17:18:14 <sdague> giulivo: we may have to not put the 'gate' tag on it
17:18:16 <giulivo> I think this could be experimental for the labeling thing
17:18:23 <giulivo> yeah indeed
17:18:30 <jaypipes> sdague: I renamed it to add-volume-snapshot-tests
17:18:36 <jaypipes> sdague: need to be specific :)
17:18:38 <sdague> jaypipes: great
17:18:44 <jaypipes> sdague: and +1 from me.
17:18:45 <sdague> yes, that's goodness
17:18:45 <afazekas> giulivo: did you turned of the secure delete option ?
17:18:51 <dwalleck> ahh, better :-) I was confused there for a sec too
17:18:53 <jaypipes> IIRC, giulivo is already pretty much done with that
17:19:00 <sdague> afazekas: secure delete is off in the gate
17:19:26 <afazekas> without secure delete it is not too slow AFAIK
17:19:45 <jaypipes> giulivo, sdague: k, updated status of add-volume-snapshot-tests
17:19:53 <sdague> jaypipes: great, thanks
17:20:09 <sdague> other blueprint issues .... ?
17:20:26 <jaypipes> well, just the Shree ones, but doesn't look like Ravi is here to provide feedback
17:20:31 <jaypipes> so no :)
17:21:16 <jgriffith> sdague: giulivo I'm looking at adding a forced check of capacity update on your race condition
17:21:24 <sdague> ok, I'll take a todo to send out an email asking about H1 status by the end of the week
17:21:50 <sdague> #action sdague to email openstack-qa to get remaining updates on blueprint (especially quantum ones) this week
17:22:07 <sdague> jgriffith: nice, thanks
17:22:37 <sdague> ok... so probably enough on the blueprints front
17:22:43 <jaypipes> ya, sounds like it.
17:22:45 <sdague> #topic outstanding important reviews
17:23:03 <sdague> now is the time to pimp reviews that need more eyes
17:23:49 <sdague> anyone have items?
17:24:26 <afazekas> https://review.openstack.org/#/c/28505/
17:24:37 <sdague> I did a run down through the review queue earlier in the week, and hit my opinions on most of them. The gate tags I still need to take a look on, but I'll do that after we get through the cinder bug
17:25:16 <sdague> afazekas: ok, I'll take a look at that
17:25:41 <afazekas> sdague: the cinder is probably configuration issue about the periodic tasks (and with the jitter config)
17:26:00 <sdague> any other reviews?
17:26:11 * jaypipes actually had some time this week to do some reviews \o/ will do some more today.
17:26:11 <sdague> going once....
17:26:18 <sdague> jaypipes: awesome
17:26:27 <sdague> #topic open discussion
17:26:42 <sdague> ok, any other topics people want to bring up?
17:26:49 <jaypipes> this gate tag...
17:26:57 <sdague> fire away
17:27:06 <jaypipes> could you give us a quick summary of the discussion so far and any decisions made?
17:27:39 <sdague> so... since grizzly rc1 our tempest run time has grown from 35 - 40 mins to 45 - 50 mins
17:27:48 <sdague> and continues to grow as new tests come in
17:28:08 <jaypipes> yes, noted.
17:28:20 <sdague> we've gotten grumbles and pushback from some of the nova team about the gate getting too long to merge complicated changesets
17:28:28 <sdague> which might have rebase conflicts
17:28:46 <sdague> so a knob that we could have, is make an explicit gate tag
17:28:55 <jaypipes> sdague: sorry, how does having a rebase conflict have to do with tempest runtime?
17:29:04 <afazekas> can we run multiple  nosetests process in the same time ?
17:29:26 <sdague> jaypipes: because when you have a 5 patch series in zuul
17:29:33 <sdague> competing with other people's 5 patch series
17:30:08 <dwalleck> afazekas: You should be able to
17:30:08 <sdague> anything that causes a gate reset, be it a bad merge, a flakey test, a flakey thing in infra, gets compounded
17:30:17 <jaypipes> sdague: k, understood.
17:30:20 <sdague> it's O(1) in theory, but not in practice
17:30:27 <jaypipes> gotcha
17:30:39 <sdague> it also means that time to first feedback on a patch, for reviewers to get to it quick, is long
17:30:50 <dwalleck> If you programatically loaded the test groups and spin up nosetest processes for each module, you'd get some pretty good relief
17:31:02 <jaypipes> sdague: so the solution to this is to improve the runtime using parallelization (poor man's or otherwise), no?
17:31:03 <sdague> afazekas: there are lots of other approaches
17:31:21 <sdague> jaypipes: parallel clearly gets us wins
17:31:23 <afazekas> It is the lowest cost approach
17:31:28 <jaypipes> dwalleck: ah, hi there :) happy belated b-day!
17:31:48 <dwalleck> jaypipes: thanks man!
17:31:53 <sdague> I guess the question is will parallel, poor man's or otherwise, ever guaruntee that we stay ahead of desired test growth?
17:32:23 <jaypipes> sdague: so, I've recommended this a couple times, but it seems to me splitting the gate into an XML and a JSON run would give us approximately a 50% reduction in runtime
17:32:33 <sdague> jaypipes: well, it wouldn't :)
17:32:38 <sdague> it will help
17:32:39 <jaypipes> sdague: and that seems like it would be very easy to do.
17:32:52 <afazekas> jaypipes: we have other than just nova tests
17:33:07 <sdague> I'm totally cool with anyone else running after any other approaches
17:33:08 <jaypipes> afazekas: sure, but big chunks of time are in nova compute tests :)
17:33:15 <sdague> jaypipes: less than you think :)
17:33:23 <afazekas> We should split in competent and/or directory  bases
17:33:42 <sdague> the gate tag was something we could get to quickyl
17:33:42 <jaypipes> sdague: my concern with the gate tag is that it means one more thing we need to keep track of -- kind of like skips.
17:33:52 <sdague> jaypipes: sure
17:34:01 <sdague> long term it just opens a question
17:34:11 <jaypipes> but I suppose things have gotten to the point where something needs to be done ASAP.
17:34:20 <jaypipes> so I will support any solutions, like you said.
17:34:24 <sdague> do we restrict content in tempest to what can run in a gate time limit
17:34:54 <sdague> I like the idea of having another knob so we can say it doesn't need to be
17:35:00 <jaypipes> sure.
17:35:16 <giulivo> that, combined with a full daily run
17:35:22 <sdague> giulivo: exactly
17:35:37 <sdague> because I do believe that any code in tempest does need to run at least once a day
17:35:43 <sdague> otherwise it bit rots
17:35:53 <sdague> eg: the old stress tests
17:35:59 <giulivo> maybe at that point we could even turn the know to 15mins
17:36:08 <giulivo> s/know/knob/
17:36:12 <jaypipes> agreed, sdague
17:36:13 <sdague> giulivo: right, but we can make that decision later
17:36:25 <sdague> right now we don't have a tool to even debate that
17:36:36 <sdague> anyway, so the gate tag seemed prudent and easy
17:36:59 <sdague> however if someone else wants to sign up for solving it a different way by H1, I'll back off of promoting it
17:37:08 <sdague> but we are talking about needing a solution for H1
17:37:18 <giulivo> still maybe there should be some agreement on when to use smoke on a testcase?
17:37:20 <sdague> because we're running long right now
17:37:33 <sdague> giulivo: yes, we need some time auditing things
17:37:53 <giulivo> oh so setting two barriers you mean
17:38:14 <sdague> my proposal is gate tag now. Spin up periodic jobs again for full. (all H1)
17:38:14 <jaypipes> sdague: understood. I would volunteer to work on a the XML/JSON split, but unfortunately, I cannot promise to make the 5/30 deadline.
17:38:24 <jaypipes> sdague: sounds like a good plan for H1
17:38:30 <RAX-Sam> I think that parallelization (poor man's or otherwise) will help short term, but it doesn't solve a long term problem. Ultimately I figure it should be a multi-part solution. I.E. Poor man's parallelization, split XML & JSON, maybe carve up further into multiple jobs (gate job, daily job, etc...) being very careful about tests that create expensive resources, etc...
17:38:31 <sdague> then push for real parallel for H2 with testr
17:38:35 <afazekas> sdague: Somehow we should document what covered by which test case in order to select correctly test cases for a shorter run
17:38:56 <sdague> then see what other optimizations we can make for H3 to get as much in the gate as we can
17:39:11 <jaypipes> RAX-Sam: agreed. my poor man's parallel == Split XML and JSON runs ;)
17:39:17 <sdague> afazekas: you volunteering?
17:39:20 <jaypipes> really poor man ;)
17:39:23 <sdague> heh
17:39:33 <giulivo> afazekas, ++ !
17:39:38 <sdague> hey, poor man's parallel is fine, I just haven't seen volunteers for it :)
17:40:07 <jaypipes> sdague: I will give a stab at PMP with XML/JSON
17:40:12 <sdague> jaypipes: awesome
17:40:16 <sdague> thanks!
17:40:18 <jaypipes> sdague: #action me.
17:40:34 <jaypipes> I don't suppose it will be any code patches to tempest itself... just devstack-gate
17:40:37 <sdague> #action jaypipes to do PiMP XML / JSON tests
17:40:41 <sdague> :)
17:40:42 <jaypipes> hehe
17:41:32 <sdague> jaypipes: actually, now that the d-g running is in tox, you'll need to do it in tempest + zuul config
17:41:40 <sdague> that's actually probably something to fyi folks on
17:41:57 <afazekas> sdague:after I  can figure out what is the normal format of doing that, yes. But first I should add same doc stings to the base classes
17:42:10 <sdague> devstack-gate is now just calling tox -e'....' into tempest to run tests
17:42:19 <sdague> so we don't have to patch d-g when we move things around
17:42:27 <jaypipes> sdague: no problem. I have enough experience with that in th epast couple weeks doing the chef cookbook stuff ;)
17:42:46 <sdague> currently we have -esmoke and -efull
17:42:58 <giulivo> afazekas, docstrings in the base classes for which purpose?
17:43:00 <afazekas> jaypipes: the 'challenge' is the proper log out output
17:43:02 <sdague> jaypipes: coolio
17:43:15 <jaypipes> afazekas: I wouldn'
17:43:21 <jaypipes> t be changing any log output...
17:43:42 <sdague> yeh, it would be separate jobs
17:43:47 <afazekas> giulivo: documenting the functions used by the test cases
17:44:02 <sdague> ok, additional topics?
17:45:15 <sdague> going once...
17:45:29 <sdague> going twice...
17:45:33 <afazekas> http://lists.openstack.org/pipermail/openstack-qa/2013-February/000219.html
17:45:34 <jaypipes> sold.
17:45:59 <jaypipes> afazekas: suggestion...
17:46:24 <jaypipes> afazekas: that email is likely too long to have an effective conversation about all the points. I would recommend splitting into separate emails for each point.
17:46:43 <afazekas> jaypipes: ok
17:46:53 <jaypipes> afazekas: possibly putting that original email in an etherpad and referring to it from the followup single-topic emails.
17:46:54 <sdague> afazekas: yeh, we've gotten active on our ML recently, so I think if it came in new in chunks, we'd get a good discussion
17:47:02 <jaypipes> ++
17:47:20 <jaypipes> afazekas: because they are all good points, just got lost in the shuffle too easily in a big email like that
17:47:39 <afazekas> I see :)
17:47:39 <sdague> ok, with that I think I'll call it a meeting. We can take additional topics to #openstack-qa or the mailing list
17:47:44 <sdague> #endmeeting