16:04:06 <jaypipes> #startmeeting
16:04:07 <openstack> Meeting started Wed Oct 26 16:04:06 2011 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:04:13 <jaypipes> http://wiki.openstack.org/Meetings/QATeamMeeting#preview
16:04:25 <jaypipes> http://wiki.openstack.org/Meetings/QATeamMeeting
16:04:37 <jaypipes> #topic nati to give status on unit testing in Nova efforts
16:04:43 <jaypipes> nati: you're up :)
16:04:57 <nati> Yes we moved to git
16:05:04 <jaypipes> \o/
16:05:30 <nati> There are 100 bugs in my lab (japanese) I shoud translate and submit it to community in next week
16:05:39 <nati> and unit-test code
16:06:03 <nati> And I fixed assignee of the bugs
16:06:10 <nati> That's all from me :)
16:06:16 <jaypipes> nati: that's a lot of work for just you! can you delegate part of them to another?
16:06:41 <nati> Ah it is ok :)
16:06:54 <jaypipes> nati: alright :)
16:07:08 <jaypipes> nati: how many bugs have been submitted to Nova so far from the QA team?
16:07:19 <jaypipes> nati: and how many patches from those bugs have landed in trunk so far?
16:07:42 * jaypipes puts on Pointy-Hair-Boss hat...
16:07:44 <nati> I think about 40, and 4 patches. My team start working for patating
16:07:57 <jaypipes> nati: patating?
16:07:58 <nati> Currently they forcus on unit test writing
16:08:12 <nati> write patches (sorry typo)
16:08:16 <jaypipes> ah, patching :) sorry
16:08:46 <jaypipes> nati: OK, that's great progress. Are you working with vishy to target bugs to specific milestones?
16:08:57 <nati> Ah, not yet
16:09:31 <jaypipes> nati: OK, well how about you and I set aside an hour or so this afternoon to work on that with vishy?
16:10:06 <nati> It sounds great.. but I have appointments on today's afternoon..  How about tommorow?
16:10:35 <jaypipes> nati: that's fine too. I will ping you tomorrow.
16:10:41 <nati> Thanks!
16:10:51 <jaypipes> nati: alright, anything else to bring up on unit testing?
16:11:16 <jaypipes> #action jaypipes and nati to work with vishy to target QA bugs to milestones on Friday
16:11:38 <nati> Fryday? Not tommrow?
16:11:50 <jaypipes> oh, crap, sorry, yes Thursday :)
16:11:54 <nati> :)
16:11:57 <jaypipes> #action jaypipes and nati to work with vishy to target QA bugs to milestones on Thursday
16:12:13 <jaypipes> OK then, moving on....
16:12:27 <jaypipes> #topic westmaas to give status update on integration tests
16:13:14 <westmaas> howdy!
16:13:51 <westmaas> so we have started merging some small things into openstack-integration-tests to get to a point where they can run against an installation
16:14:04 <dwalleck> westmaas is in San Antonio. Howdys are a necessity :)
16:14:24 <westmaas> good progress being made there although nothing supports keystone yet
16:14:29 <nati> howdy!
16:15:02 <westmaas> we also tried to distill the debate on the mailing list into a list of requirements so that we can make the right choice about how to write the tests
16:15:05 <jaypipes> westmaas: good news. so, is there an action plan for bringing in the other main suites (backfire, zodiac and stacktester specifically?)
16:15:21 <nati> Ah I created
16:15:23 <nati> https://docs.google.com/spreadsheet/ccc?key=0Ap9P99ymj9wLdEx2OEtyODNKOXpWODNObmpyR29LLUE#gid=0
16:15:25 <nati> Test list
16:15:42 <jaypipes> westmaas: would it be best to assign each suite to an individual? or have one person do that?
16:15:52 <nati> Sorry I misunderstand. You mean requirement to test frameworks.
16:16:00 <westmaas> I was just writing up the email just before this but didn't quite get there - would you like to see what we have thus far here, or just save that for the list so the wider community can comment?
16:16:42 <westmaas> jaypipes: Yes, I like that from an analysis and then action point once we decide on the direction those should go
16:17:14 <jaypipes> westmaas: sure, please do share
16:18:00 <westmaas> http://paste.openstack.org/show/2907/
16:18:29 <jaypipes> westmaas: gotcha.
16:18:47 <jaypipes> westmaas: SSH verification for the white-box testing, right? not the black-box testing...
16:18:50 <westmaas> just started writing so its very rough :)
16:18:56 <jaypipes> westmaas: no worries, it's a start!
16:19:14 <westmaas> jaypipes: no, you can ssh to an instance you create to make sure its actually up, it actually rebooted, etc
16:19:21 <jaypipes> westmaas: ah, gotcha.
16:19:29 <westmaas> not ssh to control infrastructre
16:19:33 <jaypipes> westmaas: yes, sorry, was thinking of SSH verification of compute states...
16:19:37 <westmaas> yep
16:19:51 <westmaas> will make that clear when I send it :)
16:20:20 <jaypipes> westmaas: anyway, so what would be cool is to have an action plan for bringing in *one* of (zodiac, backfire, stacktester) using the principles outlined on your document.
16:20:47 <dwalleck> I have zodiac pretty much ready to bring in. I can help westmaas with stacktester also
16:20:53 <jaypipes> westmaas: it would be nice to pick one of those 3 suites and aim to have it ported into the main unifiied integration test project by next week (or two weeks?)
16:20:54 <westmaas> yep, sounds good, daryl and I will work to propse an action plan with backing code for clarity
16:21:03 <westmaas> next week should be fine
16:21:14 <jaypipes> westmaas: I say let's get one good achievement first, then start on another suite...
16:21:25 <jaypipes> westmaas: sounds like Zodiac would be a good one to start with?
16:21:42 <wwkeyboard> The work will decrease considerably with each suite, due to the duplication.
16:21:43 <westmaas> sure
16:21:50 <jaypipes> westmaas: if you agree, I'll cook up a blueprint for it and target it...
16:21:56 <westmaas> #agree
16:22:05 <jaypipes> dwalleck: 1 week or 2? :)
16:22:24 <dwalleck> jaypipes: One week for this initial shot should be enough
16:22:31 <jaypipes> dwalleck: got it.
16:22:47 <jaypipes> #action jaypipes to create blueprint for zodiac inclusion... target to next week.
16:23:17 <jaypipes> westmaas: may we discuss the items under "potentially controversial"?
16:23:36 <jaypipes> westmaas: I'd like to move those to "Non-controversial" if we can :)
16:23:48 <westmaas> sure
16:24:03 <westmaas> python is not contreversial I suppose
16:24:05 <westmaas> so done
16:24:26 <nati> #agree
16:24:38 <westmaas> using novaclient vs non-novaclient vs some combination of both is potentially convtroversial
16:24:59 <nati> non-novaclient is needed for irregular input test
16:25:04 <westmaas> I am opposed to using only novaclient, because I am concerned about masking errors in the api
16:25:12 <westmaas> nati: good point
16:25:16 <nati> Yes
16:25:23 * jaypipes would prefer BOTH methods.
16:25:31 <westmaas> wwkeyboard suggested using novaclient for setup and teardown
16:25:32 <jaypipes> novaclient + httplib2 straight calls.
16:25:43 <nati> #agree
16:25:50 <westmaas> jaypipes: to what degree though?
16:25:51 <jaypipes> westmaas: sure, that's fine IMHO
16:26:00 <jaypipes> westmaas: novaclient for setup is fine I think...
16:26:05 <westmaas> do we repeat all tests using both methods?
16:26:06 <dwalleck> I'm not sure how I feel about mixing novaclient and non-novaclient calls in the same test
16:26:16 <jaypipes> westmaas: and then testing the API with both httplib2 calls and with novaclient.
16:26:22 <jaypipes> dwalleck: not in the same test, no.
16:26:58 <jaypipes> I was imagining two entirely different tests, one using an httplib2 client and the other using novaclient
16:27:15 <jaypipes> or are we supposed to refer to it as openstack-client now? ;)
16:27:22 <nati> Ideally, we should have both of novaclient and non-novaclient for a test senario
16:27:22 <dwalleck> jaypipes: Gotcha. That's what I was imaging as well
16:27:27 <wwkeyboard> jaypipes: I don't see what that buys us.
16:27:38 <wwkeyboard> If we are just running the same API twice, it should work the same twice.
16:27:40 <jaypipes> ok cool. so, westmaas looks like we can push that to non-controversial :)
16:27:52 <dwalleck> I see the novaclient tests more as of testing novaclient itself. The non-client tests are really testing the API
16:27:55 <wwkeyboard> I would rather spend the extra effort to making one suite more comprehensive.
16:27:57 <GavinB> we need to have a tested server + novaclient combination
16:28:17 <westmaas> and a thought I *just* had and so haven't decided if I like it yet is to use openstack-client for "smoke tests" to quickly check that the installation works, and that the client works with it
16:28:20 <jaypipes> wwkeyboard: yes, but one test may identify bugs in novaclient, which are as important to identify as bugs in the straight HTTP interface IMHO
16:28:32 <westmaas> and our detailed tests use httplib2
16:28:33 <donaldngo_hp> what about the python libraries?
16:28:47 <westmaas> donaldngo_hp: yes, that's what we really mean, not using it from the console
16:28:59 <jaypipes> donaldngo_hp: httplib2 and novalicent are both the python libs :)
16:29:00 <GavinB> cli/lib + server incompatibility is bad for customers
16:29:26 <rohitk> we could use httplib2 for tests novaclient cannot reach, after identifying them
16:29:29 <nati> httplib2 looks very stable
16:29:41 <rohitk> most tools use httplib2 except backfire
16:29:48 <jaypipes> nati: it is. glance's functional tests use it extensively.
16:30:28 <jaypipes> OK, so can we agree that we want tests written that use BOTH the novaclient Python library AND the straight HTTP calls through httplib2?
16:30:30 <jaypipes> #agree
16:30:35 <nati> #agree
16:30:55 <GavinB> agree (and python-novaclient -> nova cli tool works also, yes ?)
16:31:03 <rohitk> #agree
16:31:12 <jaypipes> GavinB: yes
16:31:19 <dwalleck> #agree
16:31:28 <donaldngo_hp> #agree
16:31:52 <westmaas> #agree
16:31:57 <jaypipes> OK, I should note that both types of tests don't need to be done by next week, dwalleck :) Only the httplib2 ones for Zodiac, I'm presuming...
16:32:20 <dwalleck> jaypipes: Right, gotcha
16:32:23 <donaldngo_hp> so the unix command line interface is that another category of tests?
16:32:55 <nati> Maybe , we should have to QA about clients
16:33:39 <jaypipes> dwalleck, westmaas: what do you say to donaldngo_hp's question?
16:33:58 <westmaas> I would leave command line tests out of this suite, personally
16:34:09 <westmaas> I want to test the lib that the cli uses in this suite
16:34:14 <dwalleck> Hmm, isn't that novaclient also? or is that a different implementation?
16:34:15 <jaypipes> donaldngo_hp: personally, I would be interested in that type of test, but the other types (python-novaclient lib calls and httplib2 calls) would be my top priorities.
16:34:19 <westmaas> prob is we call both python-novaclient
16:34:32 <westmaas> the cli and the lib
16:34:35 <jaypipes> heh, right :)
16:35:05 <donaldngo_hp> so python-novaclient = command line interface and python library?
16:35:06 <jaypipes> though to be honest, the CLI client is a very thin wrapper over the python lib (except for zones!)
16:35:11 <rohitk> it's the same thing ? nova CLI tool calls novaclient lib
16:35:27 <jaypipes> rohitk: yes, EXCEPT for zone management...
16:35:30 <westmaas> donaldngo_hp: yes
16:35:41 <donaldngo_hp> so is it 100% that if we cover the novaclient lib the cli calls work?
16:35:48 <rohitk> i think what's imp is how errors are seen on CLI
16:35:58 <jaypipes> donaldngo_hp: everything except zones IIRC.
16:36:17 <westmaas> rohitk: agreed.  I think that might belong in the CLI codebase itself
16:36:18 <donaldngo_hp> there are also windows clis too right?
16:36:31 <jaypipes> I say we focus on the python library calls (not the CLI calls) for the first round of test cases.
16:36:37 <westmaas> donaldngo_hp: none that are part of openstack projects
16:36:38 <jaypipes> donaldngo_hp: same thing. it's python.
16:36:42 <nati> I agree with Jay.
16:36:50 <rohitk> jaypipes: #agree
16:36:51 <nati> Each client depents on libs
16:36:52 <westmaas> jaypipes: also agree
16:36:59 <donaldngo_hp> but yes i agree that httplib and the python libraries should be priority
16:37:02 <jaypipes> not to say we can't add CLI testing later...
16:37:13 <jaypipes> OK, sounds like that is agreed upon.
16:37:31 <westmaas> ok, will represent that in the email I send out today
16:37:44 <jaypipes> westmaas: awesome, thx!
16:37:48 <nati> westmaas++
16:38:04 <jaypipes> ok dokey, let's move to open discussion...
16:38:07 <jaypipes> #topci open discussion
16:38:12 <nati> HI I have a topic
16:38:13 <jaypipes> #topic open discussion
16:38:21 <nati> https://blueprints.launchpad.net/openstack-qa/+spec/integration-test-scenario
16:38:25 <nati> https://docs.google.com/spreadsheet/ccc?key=0Ap9P99ymj9wLdEx2OEtyODNKOXpWODNObmpyR29LLUE#gid=0
16:38:44 <nati> I created list of Apis and point of tests
16:39:05 <nati> How about to check API coverage using this spreadsheet?
16:39:17 <jaypipes> wow, nice work, nati
16:39:27 <dwalleck> Nice, very good stuff
16:39:30 <nati> Me and Ravi work on WADL this week
16:39:38 <nati> I geneated this list from WADL
16:39:49 <jaypipes> great work. :)
16:39:55 <nati> Now we have WADL for nova, keystone, glance, quantum
16:40:02 <jaypipes> I would have no problem using this as a basis for test cases.
16:40:09 <nati> Thanks
16:40:09 <westmaas> sweet
16:40:12 <jaypipes> dwalleck, westmaas, wwkeyboard: thoughts?
16:40:33 <nati> And Rohit analize the current test and deplicaitons https://docs.google.com/spreadsheet/ccc?key=0AgufSlaYow2GdDIxU3BvbXY5SDEwbzRWZndTRDR6a1E
16:41:12 <dwalleck> Hmm, I think so. nati, could you perhaps send a short description of what data each of the columns represents?
16:41:27 <rohitk> yes, i wanted to gather an inventory of what we have, based on last time's discussion
16:41:32 <nati> I see.  For API list?
16:41:36 <jaypipes> rohitk: cool.
16:42:07 <westmaas> rohitk: great, thanks
16:42:39 <dwalleck> nati: Yes. It seems like we have columns for all inputs, and most of them I think I understand, but for example I'm not sure about a stress test column for each input
16:42:41 <westmaas> rohitk: except nova smoketests does have ec2 support :)
16:42:54 <dwalleck> But I'm also sure I'm just not understanding everything :)
16:43:03 <rohitk> yes, it uses boto
16:43:03 <jaypipes> nati: I think also, we would need to know the expected result of each call, no?
16:43:05 <rohitk> ??
16:43:08 <rohitk> my bad :-)
16:43:18 <nati> dwalleck: gotcha, I'll add more note
16:44:24 <jaypipes> nati: looks like you've been having lots of fun with WADL ;)
16:44:33 <nati> jaypipes: Yes. I think so too. I almost finised to create api list. next is test scenario which includes input and output expectations.
16:44:55 <nati> jaypipes: Noooo I hate WADL... It awful to write. But I love IDL
16:45:47 <nati> jaypipes: I like machine readable data format ;)
16:45:50 <jaypipes> ok, very nice work nati. please do work with dwalleck and westmaas and make sure you are sharing all your findings...
16:46:02 <westmaas> nati: we will do the same
16:46:25 <nati> jaypipes: westmaas: gotacha
16:46:27 <dwalleck> same here
16:46:47 <westmaas> do we definitely want to call this project openstack-integration-tests?  is it too late to change?  is it worth it to change the name?
16:46:51 <rohitk> how can we designate the nova-smoketests as part the integration-tests?
16:47:06 <jaypipes> westmaas: what's your proposed new name? :)
16:47:11 <westmaas> rohitk: I think we need to work with novacore to move it out?
16:47:11 <dwalleck> haha, I was actually thinking the same thing :)
16:47:30 <westmaas> openstack-functional-tests perhaps?  I'm open to other ideas :)
16:47:40 <dwalleck> I kind of like westmaas's Stacktester name
16:47:40 <westmaas> openstack-system-tests
16:47:57 <jaypipes> westmaas: any of those is fine by me...
16:47:57 <rohitk> dwalleck: me too, sounds perfect
16:48:04 <westmaas> stacktester is easier to type :)
16:48:09 <nati> StackMonkey (just a idea..)
16:48:12 <jaypipes> hehe
16:48:17 <rohitk> lol
16:48:18 <dwalleck> And it is what it sounds like :)
16:48:30 <jeblair> we can change the name.  it is a bit of a PITA, so, uh, let's try not to do it too often or without good reason. :)
16:48:38 <jaypipes> So.... stacktester then?
16:48:46 <jaypipes> or openstacktester? ;)
16:49:20 <jaypipes> jeblair: only change it once, we promise :)
16:49:38 <rohitk> #shakespeare : what's in a name...
16:49:39 <nati> I think stacktester is good. But if we do it now, some confusion ocuurs
16:49:42 <dwalleck> Maybe a vote via the list so that everyone gets a fair chance?
16:49:47 <westmaas> jeblair: once per wek
16:49:48 <jaypipes> dwalleck: ++
16:49:55 <westmaas> week*
16:50:00 <jaypipes> westmaas: want to send out a separate email on the name>
16:50:01 <jaypipes> ?
16:50:04 <westmaas> sure
16:50:16 <westmaas> to the openstack list or the openstack qa list?
16:50:25 <rohitk> do we have a poll feature on our wiki?
16:51:02 <nati> rohitk++
16:51:04 <jeblair> launchpad does
16:51:10 <jaypipes> westmaas: main list
16:51:23 <westmaas> jaypipes: k
16:51:31 <jeblair> https://launchpad.net/~openstack/+poll/e-release-naming
16:51:36 <jeblair> ^ example poll
16:51:37 <uvirtbot> jeblair: Error: "example" is not a valid command.
16:51:43 <westmaas> haha
16:51:46 <westmaas> jeblair: cool, thanks
16:51:50 <jeblair> ^ example error message
16:51:51 <uvirtbot> jeblair: Error: "example" is not a valid command.
16:52:04 <westmaas> lol
16:52:18 <rohitk> jeblair: cool,
16:52:51 <jaypipes> OK all, are we ready to end the meeting?
16:53:06 <westmaas> raise your hand if not!
16:53:14 <nati> Yup Thanks for your leading discuss!
16:54:19 <dwalleck> done done
16:54:25 <jaypipes> #endmeeting