17:04:27 <jaypipes> #startmeeting qa
17:04:28 <openstack> Meeting started Thu May  2 17:04:27 2013 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:04:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:04:32 <openstack> The meeting name has been set to 'qa'
17:04:33 <openstack> davidkranz: Error: Can't start another meeting, one is in progress.
17:04:44 <jaypipes> jinx
17:05:02 <jaypipes> davidkranz: you want to lead? go for it :)
17:05:18 <davidkranz> Agenda at https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:05:46 <davidkranz> Before we get to that, I noticed in my recent stress submission that pep8 failed.
17:06:10 <davidkranz> There are a bunch of stacktraces in pep8 runs for tempest even when jenkins says it is ok. ANy one know about that?
17:06:19 <jaypipes> no, not I
17:06:38 <davidkranz> http://logs.openstack.org/27950/1/check/gate-tempest-pep8/3105/console.html.gz
17:07:13 <davidkranz> Also, what do you run in a tempest dir to get all the stuff that jenkins is diong for the pep8 test?
17:08:07 <davidkranz> I did './run_tests.sh -N -p' but that passes and jenkins is still failing pep8.
17:08:29 <jaypipes> davidkranz: looks like it failed, not succeeded... are you sure gerrit reported it a success?
17:08:49 <davidkranz> jaypipes: Gerrit reported failure but I got success when I ran it before submitting.
17:08:52 <jaypipes> davidkranz: does -p in run_tests.sh actually look in the stress/ directory though?
17:09:19 <davidkranz> jaypipes: Ah, perhaps that is the problem. Maybe I have to add it to some config.
17:10:06 <davidkranz> Let's move on.
17:10:15 <davidkranz> Any issues with reviews?
17:10:32 <jaypipes> davidkranz: I see this:
17:10:32 <jaypipes> [flake8]
17:10:32 <jaypipes> ignore = E125,H302,H404
17:10:32 <jaypipes> show-source = True
17:10:32 <jaypipes> exclude = .git,.venv,.tox,dist,doc,openstack,*egg
17:10:42 <jaypipes> in tox.ini
17:11:03 <jaypipes> davidkranz: so, AFAICT, tox -> flake8 should be picking up the stress/ directory... :(
17:11:10 <davidkranz> jaypipes: Then I think "new_stress" would have been included. I will have to investigate more.
17:11:22 <jaypipes> ya
17:11:45 <davidkranz> Does any one have any comments about stress tests?
17:12:26 <ravikumar_hp> davidkranz: Is it configurable parameters
17:12:40 <ravikumar_hp> to run stress test?
17:12:57 <ravikumar_hp> like time , # of VMs etc
17:13:26 <davidkranz> ravikumar_hp: It could be. I haven't gone there yet. Just the basic idea was for reactions to the approach.
17:13:54 <davidkranz> ravikumar_hp: Right now you provide those options in each test script.
17:14:05 <davidkranz> ravikumar_hp: There could/should be a more general configuration mechanism.
17:14:42 <davidkranz> #topic Blueprints
17:15:04 <davidkranz> I will create a blueprint for stress tests.
17:15:40 <davidkranz> Has every one looked at the blueprints and claimed those to be worked on for Havana?
17:15:58 <afazekas> davidkranz: The stack traces probably caused by inspection or __import__ used by the "flake8".
17:16:18 <davidkranz> afazekas: Should I take this up with infra?
17:16:18 <ravikumar_hp> davidkranz: I marked milestones and claimed some for our team
17:16:24 <davidkranz> ravikumar_hp: Thanks.
17:16:44 <afazekas> davidkranz: yes
17:17:55 <davidkranz> Sean was planning to soon obsolete the blueprints that are not claimed. They can always be claimed later.
17:18:13 <davidkranz> But we wanted a manageable number to keep track of.
17:19:19 <afazekas> do we really need multiple blueprints for example for keystone v3 api ?
17:19:26 <jaypipes> no
17:19:36 <davidkranz> afazekas: They should be cleaned up.
17:19:47 <afazekas> Do we really want blueprints just with  the list of test cases ?
17:20:18 <davidkranz> afazekas: It it will take more than a very small effort, then yes I think we do.
17:20:44 <davidkranz> afazekas: Part of the purpose of blueprints is to make sure there is not duplication of effort.
17:20:48 <afazekas> Even smaller effort if you  just send the patch :)
17:21:04 <davidkranz> afazekas: Unless some one else sends it first.
17:21:25 <davidkranz> It only takes a minute to create a blueprint.
17:21:36 <davidkranz> There is effort in tracking blueprints though.
17:21:37 <afazekas> The test cases in a blueprint can be done by multiple person
17:22:11 <davidkranz> afazekas: True, but that should be coordinated by the blueprint assignee.
17:22:41 <davidkranz> It is work to keep track of blueprnits which brings up the next subject which was the idea to have a "PTL" for the QA project.
17:22:45 <afazekas> How the tempest bugs fits into the picture ?
17:23:24 <davidkranz> afazekas: I think we are talking about a fairly traditional model for tracking bugs and features.
17:23:52 <davidkranz> afazekas: There is a gray area of course for a few missing test cases or something like that.
17:24:30 <davidkranz> Does any one have any thoughts about a "PTL"?
17:24:51 <davidkranz> The idea is to have some one take on ownership of tracking bugs and blueprints.
17:25:18 <davidkranz> And coordinating suggestions for improvements to how we do things.
17:27:49 <davidkranz> OK, anything else to discuss today?
17:29:27 <jaypipes> any word on CloudCafe?
17:30:30 <afazekas> jaypipes: Does it have more published  test cases ?
17:30:58 <jaypipes> afazekas: not sure. hadn't last time I checked last Friday...
17:31:10 <jaypipes> was hoping to hear from dwalleck or SamDanes
17:32:40 <afazekas> We should have some procedure for coopering on non-gating test tools anyway.
17:33:50 <afazekas> For example on test cases which expects multiple node
17:34:01 <davidkranz> afazekas: That is really a matter of multiple groups of people agreeing to use a common infrastructure.
17:34:22 <davidkranz> afazekas: The ci group is working on multiple node for jenkins.
17:35:17 <jhenner> do you think it would be possible to add multinode to devstack?
17:35:34 <davidkranz> mordred: ping
17:36:33 <jhenner> I mean, would it be difficult to make devstack able to spawn vms for testing?
17:37:14 <davidkranz> clarkb, fungi : Could one of you comment on the status of multi-node devstack in jenkins ^^^^^
17:38:18 <davidkranz> Guess they are all busy now.
17:38:52 <davidkranz> I'll ping them later and see if I can find something to send out to the qa list.
17:39:04 <giulivo> one 'not in the agenda' question
17:39:08 <mordred> sup davidkranz
17:39:09 <afazekas> davidkranz: thank you
17:39:30 <giulivo> are there plans for inclusion/creation of tests revolving around heat/ceilometer ?
17:39:37 <davidkranz> mordred:  ^^^^ discussnio about multi-node in jenkins
17:39:52 <mordred> great. multi-node jenkins is something I'm quite keen on having
17:39:53 <fungi> davidkranz: i don't think there's any major status updates on multi-node devstack, but i've not been directly involved
17:40:07 <fungi> oh, mordred's here
17:40:11 <davidkranz> mordred: Is the status or plan for this being tracked some where?
17:40:13 <mordred> pleia2 is the main one working on that, but she's getting married right now
17:40:29 <mordred> I believe so - lemme find the bug
17:41:28 <davidkranz> giulivo: This was discussed at the summit with the heat and ceilo folks. I had  a short IRC chat with heat the other day.
17:41:53 <afazekas> mordred: Is the currently planned the multi-node solution for spiting the test to multiple node, or to have multitude openstack cluster ?
17:41:58 <davidkranz> giulivo: I think they will ping us when they are ready to engage.
17:42:08 <mordred> afazekas: whose are two different things
17:42:35 <mordred> afazekas: splitting the test to run portions of the tempest tests on different nodes is one thing that testr will be the first step in accomplishing
17:42:56 <mordred> afazekas: having the cloud against which tempest is running be installed on multi-nodes is a different thing
17:43:09 <mordred> both, I believe, are important - but have a different set of challenges. :)
17:44:05 * mordred cant' find the CI bug for multi-node right now - it's in there somewhere, I'll ping you with it davidkranz when I find it
17:44:24 <davidkranz> mordred: NP. Thanks.
17:45:02 * afazekas is considering creating a test runner which can work faster on a single node
17:45:04 <mordred> I've also been having good chats with rainya about getting her infra guys hooked in with our infra guys, so hopefully that'll have some good results on our ability to more forward on some of these thigns
17:45:12 <mordred> afazekas: that should be testr
17:45:16 <mordred> I would imagine
17:46:01 <mordred> nova tests increased by 5x speed on my laptop when we moved to testr ... but it's not a straightforward move
17:46:17 <mordred> as I believe jaypipes will be more than happy to back me up on
17:46:27 <afazekas> mordred: yes, but they CPU sensitive unittests
17:46:39 <jaypipes> right.
17:47:01 <mordred> afazekas: I'm just saying- before we go off making new test runners, let's finish the current test runner migratoin that we're working on
17:47:24 <mordred> since it's a lot of work, and there are more than one reason we want to accomplish the testr one
17:49:23 <afazekas> mordred: first we should place script in tempest which start the actually used runner with the actual directory structure
17:49:38 <afazekas> Both will be changed soon
17:50:00 <mordred> I'm confused by what you mean
17:50:44 <afazekas> https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L223 , https://blueprints.launchpad.net/tempest/+spec/tempest-repo-restructure
17:52:02 <afazekas> Now we cannot do restructure and changing the gate script at the same time, with a single commit
17:52:25 <afazekas> The logic around  L223 should be in the tempest repo
17:52:57 <afazekas> Implemented as a  ./run_tests.sh --gate
17:56:20 <jaypipes> afazekas: the gate does not use run_tests.sh
17:56:26 <jaypipes> afazekas: it uses tox.ini
17:56:36 <afazekas> it should use in the tempest case
17:56:37 <jaypipes> afazekas: or calls nosetests directly
17:56:56 <afazekas> if we creating a venv we are not really using the recent python libraries
17:57:07 <jaypipes> afazekas: no venv for tempest.
17:58:40 <mordred> so, I agree that the extra config of how nose is invoked there is a problem
17:59:04 <mordred> is the restructure goig to put cli into tempest?
17:59:25 <jaypipes> mordred: it already is, AFAIK
17:59:50 <mordred> k. weird. yeah - in _this_ case - I thnk a run_tests.sh should probably be used
18:00:07 <mordred> but it should have big bold letters around it saying that it's a hack to solve historical issues
18:00:18 <mordred> and will go away in the future once we've solvle them
18:00:24 <mordred> and that we all feel bad about it
18:00:58 <openstack> bdpayne: Error: Can't start another meeting, one is in progress.
18:01:06 <jaypipes> #endmeeting