17:01:38 <dwalleck> #startmeeting
17:01:39 <openstack> Meeting started Thu May 10 17:01:38 2012 UTC.  The chair is dwalleck. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:00 <rohitk> Jay??
17:02:01 <dwalleck> #topic Action items: getting the smoke test branch in gerrit
17:02:15 <dwalleck> Which Jay has done :) Pretty slick stuff
17:02:32 <rohitk> yes, im waiting for that to go through :)
17:02:50 <davidkranz> What needs to happen to get it in?
17:03:17 <dwalleck> One more review technically. I just wanted to give it enough time for folks to see things and get comfortable
17:03:29 <fattarsi> as it is I think it will conflict with my recent review
17:03:40 <rohitk> dwalleck ++
17:03:55 <jaypipes> o/
17:04:02 <fattarsi> \o
17:04:18 <dwalleck> Once everyone's good, we should be good to go
17:04:52 <dwalleck> Any more questions/thoughts on the smoke tests?
17:05:05 <rohitk> Yes!
17:05:18 <rohitk> Would we be getting rid of the decorators
17:05:26 <rohitk> after having the base Smoke class?
17:06:06 <jaypipes> rohitk: my thought was yes, because the decorators can be reused for other things (like positive vs negative, etc)
17:06:22 <jaypipes> rohitk: and the base Smoke test class can automatically decorate its test methods with a smoke attr
17:06:27 <dwalleck> rohitk: That depends. Personally, I internally use decorators to break my test groups down into finer grained groups, but we can leave that up to individual groups to tag if we like
17:06:33 <rohitk> frankly I am not too comfortable with the attr decorators
17:06:54 <jaypipes> rohitk: they are not consistently applied right now
17:06:58 <rohitk> unless, they are used consistently
17:07:01 <rohitk> right
17:07:12 <jaypipes> rohitk: and having the base classes decoarate automatically solves that problem..
17:07:21 <rohitk> jaypipes +1
17:07:29 <dwalleck> Well, we're setting standards now (and we should probably document them), and we'll follow them from this point forward
17:07:36 <jaypipes> rohitk: and allows the attrs to be more specifically used for targeting other things
17:07:41 <jaypipes> dwalleck: +++
17:08:01 <dwalleck> can't fix the past, just the future :)
17:08:11 <Ravikumar_hp> right
17:08:12 <rohitk> hmm
17:08:19 <jaypipes> dwalleck: for instance, I'd like to have a @attr(bug=XXXX) standard where we can run tests based on a failing bug report, etc
17:08:36 <dwalleck> We can even have further discussions about what attrs make sense so that we're consistent
17:08:41 <dwalleck> jaypipes: ++
17:08:51 <dwalleck> definitely agree
17:08:56 <jaypipes> dwalleck: and I'd rather not have to do: @attr(type='smoke', cls='positive', bug=XXXX) :)
17:09:06 <jaypipes> gets very verbose ;)
17:09:07 <rohitk> jaypipes: that would be helpful. tag tests based on test type/ bug linkage, etc
17:09:12 <jaypipes> rohitk: right
17:09:32 <dwalleck> Sounds good to me
17:09:54 <jaypipes> coolio. I will add that to the merge prop for the smoke tests (the auto-decorate thing...)
17:10:18 <jaypipes> davidkranz: FYI, stress test merge prop review done
17:10:38 <dwalleck> And is everyone okay with using the @attr(bug=XXXX) for now?
17:10:47 <JoseSwiftQA> I like it
17:10:48 <davidkranz> jaypipes: Great. I'll patch it after the meeting.
17:10:52 <Ravikumar_hp> i like it
17:10:59 <jaypipes> dwalleck: I am, obviously :)
17:11:13 <dwalleck> Well there you have it then, done and done :)
17:11:15 <fattarsi> this would decorate each test?
17:11:34 <jaypipes> fattarsi: it would decorate tests that were specifically hindered by a bug upstream
17:11:35 <dwalleck> fattarsi: Only tests that expose/exercise a bug
17:11:54 <jaypipes> fattarsi: what dwalleck said :)
17:11:57 <rohitk> those test also need to be skipped right?
17:11:57 <fattarsi> ok cool
17:12:09 <jaypipes> rohitk: up to when they are fixed, yep
17:12:24 <rohitk> jaypipes: got it
17:12:26 <dwalleck> skipped or failed, but there was still some discussion around which would be preferred...
17:12:32 <jaypipes> rohitk: the bug=XXX decorator is more of an easy way to "run the test to check if bug XXX is now fixed"
17:12:44 <rohitk> ok
17:13:17 <dwalleck> Okay, on we go then
17:13:24 <davidkranz> dwalleck: I think we need to skip if we are gating trunk, at least until tempest is embraced by other teams.
17:13:32 <jaypipes> davidkranz: that is correct.
17:13:46 <dwalleck> right
17:13:50 <dwalleck> #topic Outstanding code reviews
17:13:57 <jaypipes> Ravikumar_hp: would you mind communicating with rajalakshmi about holding off on the volume filters merge prop for now?
17:14:24 <Ravikumar_hp> jaypipes: sure . right now she is working on some other task
17:14:26 <jaypipes> Ravikumar_hp: dwalleck commented correctly on that merge prop that the API hasn't actually caught up to the proposed filtering functionality yet.
17:14:32 <jaypipes> Ravikumar_hp: gotcha
17:14:34 <dwalleck> davidkranz: I'm looking at yours today as well. I think I also reviewed anything that was still pending a review
17:14:38 <Ravikumar_hp> so we will hold on volme attachment test
17:14:42 <jaypipes> k
17:15:01 <davidkranz> I would like to see the volumes attach stuff that has the drive letter problem go in. For now it could just use vdk,vdq,vdx until we have a better fix. That is probably safe.
17:15:13 <jaypipes> fattarsi: did you catch my comment on #openstack-dev about assigning you to https://bugs.launchpad.net/tempest/+bug/997685?
17:15:14 <uvirtbot> Launchpad bug 997685 in tempest "tests.identity.test_roles.RolesTest.test_role_create_blank_name Fails" [High,Confirmed]
17:15:15 <davidkranz> There is a volume stress test coming that will use this code.
17:15:21 <jaypipes> davidkranz: ++
17:15:44 <fattarsi> jaypipes: yes, is there a bug filed in keystone about this?
17:15:54 <dwalleck> And the more folks who look at this, the better: https://review.openstack.org/#/c/6359/
17:15:54 <fattarsi> jaypipes: now that I look I cannot find one
17:16:20 <jaypipes> fattarsi: I don't know yet if it is a bug in Keystone or not :)
17:16:34 <jaypipes> dwalleck: will do that shortly.
17:17:00 <dwalleck> awesome
17:17:15 <davidkranz> dwalleck: I like the ssh thing but the issues about getting an address and ssh credentials is still a problem.
17:17:21 <rohitk> There are quite a few negative scenarios in keystone where expected Error codes are not returned
17:17:26 <fattarsi> jaypipes: in the meantime you think I should just that test until confirmed?
17:17:36 <davidkranz> I can't run it now and the ubuntu images do not accept user/password to ssh as far as I can tell.
17:17:36 <fattarsi> jaypipes: then it won't be the only test failing
17:17:59 <dwalleck> davidkranz: But it's more of a deployment problem, not a test problem. That's why I added an attr for them, because of that very situation
17:18:01 <davidkranz> If it is going in without solutions to those problems there needs to be a config to skip the ssh part.
17:18:17 <davidkranz> dwalleck: How do I use the attr to turn it off?
17:18:28 <jaypipes> fattarsi: no, let's find out if it really is a mismatch of spec /bug in Keystone and work with the Keystone folks on a fix. In the meantime, sure, we can do a @skip("Bug XXX not fixed in Keystone"), sure
17:18:32 <davidkranz> dwalleck: Sorry for my python/nose lameness.
17:19:06 <dwalleck> -a type!=ssh, which probably isn't the most...fun to look at. A config might be a better option, like we've done with resize
17:19:18 <davidkranz> dwalleck: ++
17:19:37 <dwalleck> I'm just very anxious to get this branch in as I have quite a large chunk of code I can submit once it's in
17:19:43 <dwalleck> Cool, I'll do that
17:19:46 <davidkranz> dwalleck: OK with me.
17:19:47 <rohitk> dwalleck++
17:19:55 <jaypipes> dwalleck: ++
17:19:59 <dwalleck> And default it to not run, just in case
17:20:04 <jaypipes> ya
17:20:22 <dwalleck> Speaking of merge props....
17:20:27 <dwalleck> #topic Swift Tests
17:20:29 <davidkranz> dwalleck: We might need another version of the ssh config to use keys. But that can wait.
17:20:43 <dwalleck> Jose?
17:20:48 <JoseSwiftQA> what up
17:20:55 <JoseSwiftQA> ah, yes
17:20:58 <dwalleck> JoseSwiftQA: you are =P
17:21:04 <JoseSwiftQA> what I want to push is mostly code complete, needs some very minor additions + pep8 attention.  Should have it submitted by day's end with dwalleck's help.
17:21:36 <davidkranz> JoseSwiftQA: Are you doing anything with swift ACLs?
17:21:41 <dwalleck> Which will be great to have in :) Good job man
17:21:59 <jaypipes> JoseSwiftQA: nice work.
17:22:06 <JoseSwiftQA> Not in tempest, yet.  It's technically just adding metadata, but I want to add 'helper' functions for all that kind of stuff too
17:22:19 <JoseSwiftQA> possibly in a 'middleware' client of some sort?
17:22:33 <jaypipes> JoseSwiftQA: how would that work?
17:22:33 <davidkranz> JoseSwiftQA: OK. When you figure out what the spec for that stuff is, please let us know :)
17:23:05 <JoseSwiftQA> for things like tempurl generation, you have to do some stuff with passwords, keys, hmac etc that are tedious and static
17:23:23 <JoseSwiftQA> so I want to write a helper function to do that, but it really shouldn't live in object, container, or account
17:24:00 <jaypipes> k
17:24:17 <dwalleck> I'll make sure he survives the first commit process :)
17:24:30 <JoseSwiftQA> ^^main concern numero uno :D ^^
17:24:30 <uvirtbot> JoseSwiftQA: Error: "^main" is not a valid command.
17:24:53 <dwalleck> #topic Documenting/Reporting functional test coverage
17:25:01 <dwalleck> Tricky stuff....
17:25:22 <dwalleck> But I saw this, and I generally like the concept for tracking functional test coverage http://code.google.com/p/test-analytics/wiki/AccExplained
17:26:13 <dwalleck> This feels right because it's a type of coverage you can manage regardless of where the dev cycle is
17:26:13 <jaypipes> interesting
17:26:47 <dwalleck> So for example, for Nova I came up with attributes of functional, secure, robust, and responsive
17:26:47 <egallen> /buffer #openstack-metering
17:27:13 <dwalleck> Which is much more descriptive than just positive/negative
17:27:37 <jaypipes> dwalleck: but what is the definition of responsive? :)
17:28:02 <jaypipes> dwalleck: do we decorate methods with an expected time to complete?
17:28:11 <dwalleck> jaypipes: Good question!
17:28:23 <jaypipes> dwalleck: something like @attr(ttc<=2.0)
17:28:41 <dwalleck> jaypipes: oh no no, I didn't mean to use these as decorators necessarily (though we could)
17:28:44 <jaypipes> dwalleck: or just create a new decorator like @complete_in_less_than(2.0)
17:28:56 <Ravikumar_hp> do we fail the test if ttc is not met
17:29:05 <jaypipes> Ravikumar_hp: good question
17:29:21 <dwalleck> I meant from a higher level, if someone asks us right now how much of each application Tempest covers...well, I can make up numbers :)
17:29:42 <davidkranz> Ravikumar_hp: I think the situations with testing on "real deploys" and virtual infrastructure are very different in this regard.
17:29:44 <jaypipes> dwalleck: sorry to get you off track... what were your thoughts on how to apply ACC to Tempest tests?
17:30:24 <dwalleck> But if I can categorize what components/capabilities to attributes, I can show happy colored heatmaps that show where I have the least/most testing
17:31:02 <dwalleck> This still isn't perfect, but it helps more than me telling my management I have x smoke tests, y positive, z negative
17:31:27 <dwalleck> Maybe I say I have 0 tests under nova api stability...they might freak :)
17:31:32 <jaypipes> ++
17:31:42 <dwalleck> But if I just say I have positive or negative tests, there's no context
17:32:36 <dwalleck> I just wanted to bring this up. It's definitely not a finished thought yet, but out of everything I could think of, it was the thing I hated the least :)
17:32:42 <jaypipes> :)
17:32:58 <dwalleck> food for thought
17:33:02 <dwalleck> #topic Development Blueprints for Folsom release, and test coverage for those implementations
17:33:16 <dwalleck> Ravikumar_hp: you're up!
17:33:32 <Ravikumar_hp> may be this is question:
17:33:32 <Ravikumar_hp> How we are tracking progress of development blueprint and making progress on addressing/adding testcases for those blueprint tasks
17:34:16 <Ravikumar_hp> basically we want to address all blueprints by sharing work
17:34:32 <jaypipes> Ravikumar_hp: you're talking about which blueprints?
17:34:57 <jaypipes> Ravikumar_hp: new stuff coming in Folsom in Nova/Glance/Swift, etc?
17:34:59 <Ravikumar_hp> Folsom release - blueprints and developmane based on those blueprints
17:35:04 <jaypipes> I see now...
17:35:30 <jaypipes> Ravikumar_hp: first thing we need is a decent list of those blueprints that are currently in progress.
17:36:03 <davidkranz> Ravikumar_hp: I would like to make sure we don't have tempest spending a lot of time duplicating stuff that is covered by unit tests that will go with these new projects.
17:36:11 <davidkranz> How do we draw that line?
17:36:39 <jaypipes> davidkranz: by ensuring that the developer does unit tests and QA does the functional test?
17:37:11 <davidkranz> jaypipes: Right. But sometimes it is not so easy.
17:37:40 <davidkranz> jaypipes: In pre-openstack life I always had QA people working more closely with the dev team than we seem to have here.
17:38:08 <Ravikumar_hp> daidkranz: ++ . i see a gap here.
17:38:33 <davidkranz> Ideally, each blueprint would have a test plan which would greatly help with this issue.
17:39:17 <davidkranz> That plan could talk about what non-unit tests were needed.
17:39:26 <dwalleck> So if I can map blueprints to stories my developers are playing, I can do some mapping as I can
17:39:42 <jaypipes> davidkranz: I think that the QA team just needs to be more aggressive in working with the developers on functional and integration tests (and test plans) while development is going on (and right after development is complete)
17:40:01 <Ravikumar_hp> Developement can add those (test plan - non unit tests )in blueprints
17:40:44 <davidkranz> jaypipes: I agree, but the result needs to be written down somewhere
17:41:35 <jaypipes> Ravikumar_hp: why wouldn't the QA team add that to the buleprints?
17:42:16 <Ravikumar_hp> sure . should we involve development to review that?
17:42:47 <jaypipes> Ravikumar_hp: yes, of course. get the dialog going now rather than later...
17:42:55 <jaypipes> the dialog can be on the blueprints and IRC of course...
17:43:55 <Ravikumar_hp> jaypipes: ok
17:44:36 <dwalleck> #topic open discussion
17:44:44 <dwalleck> What else folks?
17:44:58 <davidkranz> I will be on vacation the next two weeks.
17:44:58 <jaypipes> nothing from me
17:45:08 <jaypipes> davidkranz: enjoy!
17:45:13 <dwalleck> lucky!
17:45:27 <jaypipes> Ravikumar_hp: you are next week's QA Captain, FYI... http://wiki.openstack.org/QACaptainRotation
17:45:38 <Ravikumar_hp> Ok
17:45:41 <davidkranz> I would like to look at the auto-reclaim of resources when I get back.
17:46:11 <dwalleck> davidkranz: I agree. I'm still bouncing implementations around in my head
17:47:13 <davidkranz> dwalleck: OK. That's all from me.
17:47:21 <dwalleck> Going once?
17:47:27 <dwalleck> twice?
17:47:32 <dwalleck> #endmeeting