16:59:41 <sdague> #startmeeting qa
16:59:42 <openstack> Meeting started Thu Aug  1 16:59:41 2013 UTC and is due to finish in 60 minutes.  The chair is sdague. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:59:45 <openstack> The meeting name has been set to 'qa'
16:59:54 <sdague> ok, who's around for the QA meeting?
16:59:59 <dkranz> Here
17:00:02 <adalbas> here
17:00:18 <giulivo> here
17:00:25 <mtreinish> hi
17:00:46 <psedlak> hi
17:00:50 <sdague> #link - https://wiki.openstack.org/wiki/Meetings/QATeamMeeting
17:00:58 <afazekas> hi
17:01:03 <sdague> #topic Blueprints
17:01:16 <sdague> ok, status on key blueprints
17:01:22 <sdague> lets start on parallel tempest
17:01:27 <sdague> mtreinish: you're up
17:01:44 <mtreinish> so testr is making progress. I've been pushing a lot of code to try and fix the races we're seeing
17:01:50 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
17:02:10 <mtreinish> right now I'm stuck with on az create server one which still shows up periodically after I added locking
17:02:19 <mtreinish> that is the most common fail right now
17:02:40 <dkranz> mtreinish: I just saw it again with my skip slow patch
17:02:52 <afazekas> mtreinish: Do you have link to one fail ?
17:02:55 <mtreinish> I'm also working on adding tenant isolation to scenario which will fix a security group issue there
17:03:05 <mtreinish> dkranz: yeah it happens ~30-40% now
17:03:10 <mtreinish> without locking it was every run
17:03:22 <mtreinish> afazekas: one sec let me pull up the link
17:03:40 <sdague> mtreinish: do we do any agregate manip in scenario tests?
17:03:40 <afazekas> mtresnins: after this change I could add per process isolation : https://review.openstack.org/#/c/38641/
17:04:06 <mtreinish> afazekas: http://paste.openstack.org/show/42787/
17:04:27 <mtreinish> sdague: not that I remember but I'd have to double check to be sure
17:05:00 <sdague> mtreinish: is that the final big bug?
17:05:07 <mtreinish> afazekas: what do you mean by per process isolation?
17:05:11 <sdague> or are there others hidden behind that one?
17:05:25 <afazekas> mtreinish: to create user/tenant on every worker process only
17:05:57 <mtreinish> sdague: hopefully, looking at the build trend: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-testr-full/buildTimeTrend its the only one I've been seeing
17:05:58 <afazekas> mtreinish: I could be extendable to use predefined list of users, when you are using tempest without admin rights
17:06:09 <mtreinish> but there were some other ones that I saw when I first turned it on
17:06:20 <afazekas> List probably means a pair of users per worker process
17:06:33 <sdague> afazekas: so that seems like a different optimization, I'd rather not distract from testr right now
17:06:50 <mtreinish> but I haven't seen them in over a week
17:06:51 <sdague> as we have different per class tenant isolation in there
17:07:03 <adalbas> i have been investigating one of those mtreinish pointed, i'm collecting info to open a bug if that comes again to happen
17:07:06 <sdague> ok, we'll probably have to push parallelism
17:07:20 <sdague> to shake more out, we had to do that with nova
17:07:31 <afazekas> sdague: the change  has some modeling related improvement anyway
17:07:36 <sdague> I was regularly running with 16 process to make it fall apart
17:07:56 <mtreinish> sdague: yeah but first we've got to sort out the az failure I started working on it with jog0 last night
17:08:14 <giulivo> I'd like to join the forces and try the testr parallelization myself; can I ask what is the gerrit submission I should checkout?
17:08:25 <sdague> afazekas: right, I get that, but right now we have a very limitted number of weeks to get testr in, so focus should be narrowly on that. We can make model improvements post H3
17:08:27 <mtreinish> and I haven't been able to reproduce it locally
17:08:52 <dkranz> giulivo: It's already in. Just run the tox for full that uses testr
17:09:01 <mtreinish> giulivo: sure, it's actually already running I can go over the details with you after the meeting in -qa
17:09:02 <afazekas> sdague: AFAIK keystone v3 is also a goal
17:09:31 <sdague> afazekas: we have to rank goals
17:09:36 <afazekas> ok
17:09:37 <sdague> otherwise we don't get them done
17:09:56 <sdague> testr is going to be really key to the last 2 week merge rush for all the projects
17:10:03 <sdague> we can do v3 after H3
17:10:21 <sdague> that's why it's currently the only critical blueprint
17:10:25 <dkranz> sdague: What does "do v3" mean?
17:10:36 <mtreinish> dkranz: keystone v3 api
17:10:39 <sdague> spend time on landing keystone v3 api tests
17:11:02 <dkranz> sdague: I don't understand why we can't do that while Matt is working on testr.
17:11:08 <giulivo> thanks mtreinish, dkranz I see that is testenv:all
17:11:36 <sdague> dkranz: there is only so much time to go around, so if people are doing that, they aren't helping make sure testr can land
17:11:42 <mtreinish> giulivo: and testenv:testr-full, testr-full runs nonvoting on the check queue and all runs periodic
17:12:07 <dkranz> sdague: What people? We can't have 10 people working on getting testr in.
17:12:20 <ravikumar_hp> mtreinish: Can you please put readme in api folder
17:12:34 <sdague> dkranz: actually, I don't think that's true
17:12:42 <sdague> testr at this point is about chasing race bugs
17:12:54 <sdague> that's very parallelizable
17:13:15 <dkranz> sdague: OK. What should we look at?
17:13:28 <mtreinish> ravikumar_hp: there is one: https://github.com/openstack/tempest/blob/master/tempest/api/README.rst
17:13:40 <sdague> get testr parallel up, push it hard see where the flakey bits fall out
17:14:02 <mtreinish> dkranz: yeah it's about running it locally and debugging where it fails
17:14:28 <dkranz> mtreinish: OK, but let's file bugs so we are not duplicating effort.
17:14:38 <ravikumar_hp> mtreinish: i mean for running testr
17:14:41 <mtreinish> the jenkins job is to help with finding things in a gate like environment and it's found quite a few. I've documented some here: https://etherpad.openstack.org/debugging-testr-tempest
17:15:14 <mtreinish> ravikumar_hp: oh I can push out an update for the main readme to do that later today
17:15:32 <mtreinish> ravikumar_hp: but run_tests.sh -t or tox -etestr-full will run it in parallel
17:15:46 <ravikumar_hp> mtreinish: Thanks
17:16:18 <sdague> yep, if we can hang the bugs off the blueprint, that would be good
17:16:18 <mtreinish> dkranz: I agree, adalbas said he was going to add a tag for testr parallel so we can easily track this
17:16:31 <mtreinish> yeah the blueprint too
17:16:40 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
17:16:46 <adalbas> sure
17:17:04 <sdague> dkranz: mtreinish told me over lunch you had the interesting idea of switching over to testr single thread for all the jobs now
17:17:22 <psedlak> sdague: could you please also linkify those urls for debugging-testr-tempest and info those commands for running it? :)
17:17:37 <dkranz> sdague: Yes, because I did not want to maintain nose selection for slow, heat, etc.
17:17:54 <dkranz> sdague: It seems like testr is solid enough and gives advantages over nose
17:17:56 <sdague> dkranz: I like it
17:18:04 <sdague> can you propose the patch for that?
17:18:07 <dkranz> It's junk parallel that is broken
17:18:17 <dkranz> sdague: Yes, as soon as the slow skip goes through.
17:18:30 <dkranz> sdague: It seems to be working so I will unmark it WIP after the meeting.
17:18:31 <sdague> cool
17:18:48 <sdague> psedlak: um... do you have the links? :)
17:18:57 <mtreinish> #link https://etherpad.openstack.org/debugging-testr-tempest
17:19:03 <sdague> great
17:19:19 <psedlak> :) ok, i was thinking that only the chair can do it, so sorry :)
17:19:26 <sdague> #info concerted push on getting testr parallel working before H3
17:19:27 <mtreinish> #info run tempest with testr parallel using: run_tests.sh -t or tox -etestr-full
17:19:55 <sdague> #action dkranz to propose testr non parallel for all tempest job runs, getting us off nose
17:20:03 <sdague> cool
17:20:12 <sdague> this is all going to be awesome when it lands
17:20:20 <sdague> and drop average gate times in half
17:20:56 <dkranz> sdague: Excellent
17:21:12 <sdague> ok, other blueprints?
17:21:26 <adalbas> mtreinish, the tag you mentioned before is the one we add in the bugs, right?
17:21:39 <dkranz> sdague: I am also going to put some focus on getting heat in.
17:21:54 <mlavalle> sdague: I can report on mine
17:22:01 <sdague> mlavalle: go for it
17:22:05 <mtreinish> adalbas: yeah but I forgot that we can just link it to the speed-up-tempest blueprint
17:22:10 <mlavalle> https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
17:22:10 <sdague> #link https://blueprints.launchpad.net/tempest/+spec/fix-gate-tempest-devstack-vm-quantum-full
17:22:27 <mlavalle> sdague: this blueprint involves a list of several items
17:22:38 <adalbas> mtreinish, yes, it makes sense
17:22:40 <mlavalle> sdague: each item requires the following: fix something in a tempest test case, fix something in nova and potentially fix something in Neutron / Neutron client
17:22:56 <mlavalle> sdague: right now I am working on the quotas item. You and I have been talking over Gerrit about the Tempest piece. Last Sunday I pushed the corresponding patchsets for review in nova. You gave a -1 in one of them (not complaining
17:23:16 <mlavalle> sdague: after quotas, I will start with the floating ips items and so on…. I will knock them down one by one
17:23:18 <sdague> mlavalle: ok, what's the review again? maybe I should look again
17:23:22 <sdague> mlavalle: cool
17:23:27 <sdague> thanks much for this
17:23:59 <mlavalle> sdague: so we can expedite his
17:24:20 <mlavalle> each time I send patches to nova or Neutron, I'll let you know, so you can help me
17:24:26 <dkranz> sdague: https://review.openstack.org/#/c/35724/
17:24:36 <sdague> mlavalle: sounds great
17:24:47 <dkranz> mlavalle: What about the neutron smoke job?
17:24:59 <mtreinish> mlavalle: I have a concern with adding neutron tests in that the neutron jenkins jobs aren't working at all. So adding more tests we really aren't running them
17:25:03 <dkranz> mlavalle: It fails but I don't see any errors in the log.
17:25:17 <mtreinish> dkranz: it fails during the devstack exercises I believe
17:25:18 <sdague> dkranz: it fails really early, I did bring that up in the neutron channel
17:25:28 <sdague> markmclain said he'd take a look
17:25:35 <dkranz> mtreinish: Yes, we need to get the neutron gate jobs working first.
17:25:36 <mlavalle> dkranz, mtresinish: that job you are talking about is what I am fixing
17:25:46 <dkranz> mlavalle: OK, great.
17:25:58 <sdague> mlavalle: ok great
17:26:25 <mtreinish> mlavalle: ok but that review adds tests. I don't think we can merge that until the job is fixed
17:27:06 <sdague> mtreinish: no we can merge
17:27:09 <sdague> it's running
17:27:16 <sdague> so it's not like we're not executing it
17:27:23 <mlavalle> mtreinish: yeah, it's running
17:27:25 <sdague> we're just not voting with it
17:27:36 <mlavalle> sdague: correct
17:27:40 <sdague> I'm ok on that front, as it's not the same bitrot issue
17:27:44 <sdague> I just +Aed it
17:27:59 <mlavalle> sdague: thanks
17:28:13 <mtreinish> sdague: no but the jenkins jobs with neutron doesn't even get to tempest. so how can we be running the tests?
17:28:15 <mlavalle> that's all
17:28:32 <sdague> mtreinish: oh, right... well
17:28:35 <sdague> I +Aed it
17:29:15 <sdague> anyway, moving on
17:29:19 <sdague> other blueprints?
17:29:20 <afazekas> mlavalle: next week, I would like to spend more time on the smoke gate instability issues , If you have any info about these issues let me know
17:29:23 <dkranz> It would be nice to get rid of exercises
17:29:35 <sdague> dkranz: yeh, that's a post H3 thing
17:29:35 <mlavalle> afazekas: will do
17:29:46 <dkranz> sdague: Of course
17:29:54 <sdague> I think it just needs an audit to make sure scenario >= exercises
17:30:06 <sdague> and if so, we disable exercises
17:30:18 <sdague> and if not, we enhance scenario tests
17:30:39 <sdague> there is now a scenario test for horizon, which is great
17:30:50 <sdague> so we know that you can actually log into horizon
17:31:39 <sdague> ok next topic
17:31:44 <sdague> #topic Critical Reviews (sdague)
17:31:53 <afazekas> sdague: nowadays,  we are skiping too many test because of instability issues, it ruins our coverage
17:31:56 <sdague> ok, what critical reviews are out there right now that need eyes?
17:32:13 <sdague> afazekas: ok, lets come back at the end
17:32:19 <afazekas> https://review.openstack.org/#/c/39417/
17:32:20 <sdague> lets give people time for critical reviews
17:33:00 <mtreinish> afazekas: I have issues with that actually but we should take it to the list I feel
17:33:10 <mtreinish> because test runners can easily exclude tests
17:33:15 <mtreinish> and testr just does it with a regex
17:33:59 <dkranz> mtreinish: I discovered that testr is a pain because it lets you OR regexes but not AND them
17:33:59 <afazekas> mtreinish: we have about 20 bugs, it will be a long regexp :)
17:34:26 <mtreinish> afazekas: that patch is not a replacement for the skip decorator
17:34:28 <sdague> afazekas: yeh, lets take that to the list
17:34:46 <sdague> remember, we're using openstack-dev now with the [qa] tag
17:34:46 <dkranz> mtreinish: That makes testr regexp ugly. See my 'slow' tag patch for an example.
17:34:54 <mtreinish> its so he can selectively exclude tests that we don't skip because of an environmental difference
17:35:06 <dkranz> mtreinish: That is a valid use case.
17:35:13 <dkranz> mtreinish: The question is just what the best way is
17:35:14 <sdague> yeh, I'm leaning towards mtreinish on this one, because there is a way to do it already
17:35:29 <sdague> but let's take it to the list and discuss there, because I think ian needs to be in it
17:35:57 <sdague> and this approach probably has other implications for other efforts, like how the cells gate job is going to be approached
17:36:24 <sdague> as they've got this issue that cells doesn't implement a lot of nova features
17:36:42 <mtreinish> dkranz: well, we can always talk to lifeless about adding an exclude option to testr
17:36:47 <dkranz> sdague: Perhaps we should ask lifeless about allowing regexp exclusion from the command line
17:36:55 <dkranz> mtreinish: :)
17:36:56 <sdague> #action move discussion on https://review.openstack.org/#/c/39417/ to the mailing list
17:37:14 <sdague> ok, other reviews that need attention?
17:38:11 <sdague> going once
17:40:02 <sdague> ok, moving on
17:40:17 <afazekas> 1 sec
17:40:22 <sdague> ok, not moving on
17:40:43 <afazekas> https://review.openstack.org/#/c/38995/
17:41:23 <afazekas> 'InvalidOkCode' exception can simplify the assertion in the test code
17:41:46 <afazekas> We just need to assert the OK codes in rest clients
17:42:08 <sdague> ok, I'll take a look post meeting
17:42:13 <sdague> any other reviews?
17:42:14 <afazekas> ok
17:42:27 <sdague> ok, next topic
17:42:39 <sdague> #topic Call for more Core reviewers (dkranz)
17:43:09 <dkranz> We need more core reviewers
17:43:10 <sdague> so our core count remains at 6, and from the numbers those 6 are still the most active reviewers
17:43:33 <sdague> I would really like to add more folks, but we do really need folks to be doing at least the level of work as existing cores
17:43:38 <dkranz> Most of the current reviewers are from Red Hat or IBM
17:43:42 <sdague> and having good -1 judgement
17:44:05 <sdague> so consider this an extended call for "please review more, even if you aren't a core member" :)
17:44:27 <dkranz> sdague: We should send this to the list
17:44:28 <sdague> because the road to becoming a core member is to basically already be doing the work of one, then we add you
17:44:30 <sdague> agreed
17:44:46 <sdague> #action sdague to send call for more reviewers to the list
17:45:23 <sdague> ok, I think that's the agenda
17:45:28 <sdague> #topic Open Discussion
17:45:43 <mlavalle> sdague: I'll start doing reviews once I feel comfortable that I am making progress with the Neutron gate job
17:46:26 <sdague> mlavalle: great, thanks!
17:46:41 <sdague> #link http://lists.openstack.org/pipermail/openstack-dev/2013-August/012855.html
17:47:01 <sdague> in case folks haven't seen that, I've been heads down trying to unify our requirements testing so we don't wedge the gate
17:47:05 <sdague> hopefully that lands today
17:47:14 <sdague> we're actually down to a tempest requirements issue now
17:47:40 <sdague> any other things from folks?
17:48:32 <mtreinish> nothing from me
17:48:41 <sdague> anyone else
17:49:00 <dkranz> Please review https://review.openstack.org/#/c/39716/
17:49:08 <dkranz> That's it from me.
17:49:19 <mlavalle> mlavalle: nothing from me
17:50:05 <sdague> ok, great
17:50:10 <sdague> let's call it a meeting
17:50:19 <sdague> #endmeeting