16:03:38 <jaypipes> #startmeeting
16:03:39 <openstack> Meeting started Wed Oct 19 16:03:38 2011 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:03:39 <patelna> yes
16:03:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:03:57 <jaypipes> #topic Action items from last week...
16:04:08 <jaypipes> one sec, grabbing last week's link..
16:04:42 <jaypipes> http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-10-12-16.05.log.txt
16:04:49 <donaldngo_hp> hello
16:04:54 <jaypipes> http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-10-12-16.05.txt
16:05:13 <jaypipes> jeblair: ping
16:05:49 <jaypipes> OK, let me go over all the stuff I failed to get done this past week... :(
16:05:50 <rohitk> is anyone from Grid Dynamics here?
16:05:57 <rohitk> ok
16:06:04 <jaypipes> jaypipes to grab all other test suites code and put into openstack-integrated-tests project, each in its own subdirectory.
16:06:13 <jaypipes> unfortunately, I did not get around to this. :(
16:06:31 <patelna> any ETA?
16:06:34 <jaypipes> I should have time to do this today, however, as past week been working on glance stuff.
16:06:43 <jaypipes> patelna: today/EOB tomorrow at latest.
16:06:46 <nati2> OK, rohitk: can you do that?
16:06:50 <patelna> ok...thanks
16:06:56 <jaypipes> dwalleck_: afternoon :)
16:07:06 <dwalleck_> Sorry I'm late, couldn't get connected for some reason
16:07:15 <rohitk> nati2: ok
16:07:21 <jaypipes> jaypipes to get this meeting listed on the openstack meetings ics calendar
16:07:21 <dwalleck_> Gigi is shoulder surfing me also
16:07:30 <patelna> ah...nice
16:07:34 <rohitk> jaypipes: Is this the structure? http://paste.openstack.org/show/2689/
16:07:36 <jaypipes> ^^ also failed to get that done... will do right after this meeting with ttx
16:07:37 <uvirtbot> jaypipes: Error: "^" is not a valid command.
16:07:50 <jaypipes> rohitk: yep
16:08:00 <jaypipes> dwalleck_: cool, no worries
16:08:10 <rohitk> jaypipes: cool
16:08:14 <nati2> jaypipes: ok, please assign this action for rohit
16:08:36 <patelna> do anyone know if we have tests for Swift
16:08:40 <wwkeyboard> jaypipes: I added a new test runner that should simplify moving to that new directory structure
16:08:50 <jaypipes> #action jaypipes to create blueprint for integration test merging and assign to rohitk
16:09:00 <jaypipes> patelna: integration tests or .. ?
16:09:02 <donaldngo_hp> https://github.com/openstack/openstack-integration-tests (Kong) is this the framework that we are agreeing to utilize and develop moving forward?
16:09:04 <nati2> thanks
16:09:11 <patelna> yes...swift integration tests
16:09:27 <jaypipes> donaldngo_hp: no. kong is one of many... we are integration about 6 of them.
16:09:32 <jaypipes> patelna: not that I know of, no.
16:09:35 <Ravikumar_hp> also swift functional tests
16:10:02 <jaypipes> Ravikumar_hp: depends what you mean by functional tests :)
16:10:13 <dwalleck_> jaypipes: Have we decided what the structure of the tests/test framework is going to be going forward? This is something I'm really itching to help with
16:10:37 <dwalleck_> I have a lot of spare time and nothing to do but design, develop and test :-)
16:10:43 <jaypipes> dwalleck_: wwkeyboard and westmaas are working on that. I would advise getting involved in the code reviews for openstack-integration-tests. :)
16:11:21 <jaypipes> dwalleck_: westmaas is leading the effort to complete a best practice test runner. I would definitely hope you collaborate with wwkeyboard (Aaron Lee) and Gabe on it!
16:11:34 <dwalleck_> jaypipes: Sure, of course
16:11:54 <patelna> lJaypipes: how can we ensure we get all the tests consolidated for core products - swift, nova & glance
16:12:03 <jaypipes> I believe jeblair sent out an email about membership for integration tests....
16:12:42 <jaypipes> patelna: integration tests are going into the openstack-integration-tests project. Those tests will stress the integration points between the core projects.
16:12:58 <jaypipes> patelna: and each project has zero, some, or a lot of its own functional tests.
16:13:33 <jaypipes> patelna: the thing that is missing is really integration tests against a cluster running a bunch of openstack projects... and that is what the openstack-integration-tests project is attempting to solve.
16:13:35 <patelna> today, we need to talk about this...
16:13:51 <rohitk> jaypipes: did jeblair create this group on LP/gerrit?
16:13:52 <jaypipes> patelna: please, let's. what specific questions do you have?
16:13:54 <dwalleck_> jaypipes: By functional tests, do you mean black-box style or white box style? I wasn't aware there was a seperate black box test suite for Nova or other projects
16:13:58 <donaldngo_hp> jaypipes: right now i see only Kong in the integration test project. just be clear we are going to add more frameworks into this repo?
16:13:59 <nati2> openstack-integration-tests project++
16:13:59 <jaypipes> rohitk: yes
16:14:11 <jaypipes> dwalleck_: blackbox
16:14:20 <patelna> consolidate all the tests that are still not contributed back to the openstack
16:14:26 <zykes-> what meeting is going now ?
16:14:32 <patelna> blackbox tests for core products
16:14:34 <jaypipes> dwalleck_: and Glance has black box tests. /glance/tests/functional/
16:14:42 <rohitk> zykes: openstack-qa
16:14:47 <patelna> how about swift?
16:15:17 <jaypipes> patelna: one sec...
16:15:19 <dwalleck_> donaldngo_hp: I would hope as opposed to adding more frameworks, we'd be consolidating them to a single unified platform
16:15:34 <rohitk> dwalleck_: +1
16:15:53 <GavinB> Has anyone defined what we would like to see in a suitable framework ?
16:16:02 <GavinB> I have some definite opinions :)
16:16:15 <donaldngo_hp> dwalleck: i agree a single framework. but each of these "frameworks" currently develop differ in their approach
16:16:15 <nati2> dwalleck_:  Even if so we should share our effort at first :)
16:16:22 <rohitk> GavinB: I too was of that opinion that we need to have a list (checklist) of features needed in that framework
16:16:32 <rohitk> or pull in the best from the rest
16:16:33 <jaypipes> patelna: Swift has "functional tests" (tests/functional/) but AFAICT, they are not black box. In other words, they don't spin up Swift servers and test the public API. They use internal interfaces.
16:16:41 <dwalleck_> gavinB: As do I. I've passed around a few things internally, as well as loosely discussed it on the list
16:17:22 <patelna> Got it...
16:17:24 <jaypipes> GavinB: we don't have a lack of frameworks :) We have too many of them... we just need to settle on one or combine the best of breed and focus on writing tests, not test frameworks :)
16:17:25 <rohitk> dwalleck_, GavinB: Test data generation and data sources are high on my list
16:18:06 <patelna> this needs to be actioned then...adding more swift tests on a runnign server?  Gigi - can u take this?
16:18:10 <nati2> rohitk: Would you write blueprint about Test data generation and data sources?
16:18:29 <patelna> Nati2 +1
16:18:37 <GavinB> jaypipes: Totally agree - but that argues for a flexible platform that can cope with tests in many forms ... so pretty high level.
16:18:56 <donaldngo_hp> I think a good framework to start with is Kong it tests the core REST APIs
16:18:57 <dwalleck_> rohitk: And for me, the test code structure and design is what I'm most worried about. I think that's going to be the key to the scalability and strength of the suite going forward
16:18:59 <jaypipes> GavinB: sure, though I think the first step is just to get *something* going...
16:18:59 <Ravikumar_hp> also test results in blueprint
16:19:13 <rohitk> nati2: Sure, will need to discuss the specifics and scope
16:19:21 <jaypipes> we can get into analysis paralysis trying to create the Uber-Framework and actually not get any test writing done :)
16:19:37 <Ravikumar_hp> 1) Nose 2) KONG to standarize api & functional /integarion
16:19:59 <dwalleck_> Kong doesn't really provide a framework. It's just directly using http requests
16:20:00 <patelna> Ravikumar_hp +1
16:20:01 <GavinB> jaypipes: Agree there also. We should select something and get moving - but recognise it will not solve all our needs
16:20:02 <jaypipes> I actually don't care what it is :) I just want it done.
16:20:23 <dwalleck_> When I'm saying framework, I mean more along the lines of Stacktester or Zodiac did in terms of design
16:20:23 <jaypipes> wwkeyboard: is westmaas around?
16:20:24 <nati2> Before test code we need specs and test scenario
16:20:26 <GavinB> dwakkeck_ +1
16:20:30 <patelna> agree with jaypipes...we need tests that we can run
16:20:38 <GavinB> dwallek_ +1 (sorry !)
16:20:40 <nati2> We didn't have API specs which we can use yet
16:20:47 <wwkeyboard> jaypipes: He is in a different office.
16:20:53 <jaypipes> wwkeyboard: oh, sorry...
16:21:00 <dwalleck_> nati2: I think we have docs for the API 1.1 spec
16:21:14 <nati2> Yes, I checked it in this week
16:21:26 <rohitk> jaypipes: soren's branch is fine enough to begin contributing to, however the finer issues of 'no novaclient', etc remain
16:21:36 <nati2> But it is not cover all features in the code
16:21:45 <westmaas> hey, sorry, bit late
16:21:56 <rohitk> hi westmaas
16:22:01 <jaypipes> rohitk: well that's why we're bringing in the other test suites... some of them like backfire use the novaclient. others use httplib2 calls.
16:22:01 <nati2> Many parameters are not documented and we only have examples
16:22:03 <patelna> nati2 - should we do traceability matrix on that API spec to ensure we have tests?
16:22:10 <nati2> patelna++
16:22:18 <nati2> spec and traceability matrix on that API spec
16:22:32 <jaypipes> OK everyone.... hold up a minute... this is getting a bit freeform... let me try to give some structure to this meeting.
16:22:37 <jaypipes> EVERYBODY STOP. :)
16:22:39 <patelna> that's the 1st step
16:22:43 <nati2> jaypipes++
16:22:48 <jaypipes> Let us get one thing done at a time.
16:22:57 <jaypipes> #topic integration tests status.
16:23:10 <jaypipes> westmaas: Gabe, would you care to provide a quick status update?
16:23:46 <westmaas> sure, daryl aaron and I met just after the last meeting, and we talked through a few options, and finally settled on a lightweight runner
16:24:00 <notmyname> jaypipes: patelna: whoever: swift's functional tests do create instances of the servers and test public APIs
16:24:03 <westmaas> the goal of the runner is to give a unified entry point to run all suites in the project as of now
16:24:20 <westmaas> with the expectation that those suites will be folded into a common suite as quickly as possible
16:24:41 <westmaas> its really just a bash script that runs another script in each directory, really really straightforward
16:25:21 <jaypipes> notmyname: the functional tests use internal objects (liek swift.Account). They are not black box -- i.e. they don't do straight HTTP calls.
16:26:02 <westmaas> next step is a two step process I think.  1) Identifying what we are testing right now
16:26:14 <westmaas> 2) Select how we should test moving forward
16:26:37 <Ravikumar_hp> westmass: 3) identify the test gaps. missing tests
16:26:53 <jaypipes> westmaas: OK, thx for the update. Question: would you prefer that rohitk simply copy/paste the other suites into directories, or copy in the test cases and in the process, convert them to use the new test runner/client?
16:26:57 <westmaas> sounds good
16:26:59 <rohitk> westmaas: 'all suites in the project' means that the script would identify and execute the test suites pulled in from various frameworks?
16:27:01 <patelna> westmass; Ravikumar_hp +1
16:27:30 <rohitk> jaypipes: that's what I implied
16:27:35 <rohitk> thnx
16:27:37 <westmaas> just bringing them in is fine for a first step, since we haven't selected how to move forward
16:27:58 <westmaas> rohitk: they just look in each directory for a certain script name, nothing more
16:27:59 <wwkeyboard> westmaas: ++
16:28:09 <nati2> westmaas: ++ nice start
16:28:12 <jaypipes> westmaas: k. is there a wiki or etherpad you're using with wwkeyboard and dwalleck_ to track thoughts on how you'd like to move forward?
16:28:15 <rohitk> oh, k
16:28:17 <wwkeyboard> Just get them in one place, and running will be a start
16:28:35 <patelna> should we organize dir structure and consolidate in one place
16:28:42 <westmaas> patelna: yes soon
16:28:46 <westmaas> jaypipes: no, not yet
16:28:59 <patelna> is that waht Rohit will do?
16:29:01 <rohitk> patelna: i'll work on that
16:29:02 <westmaas> jaypipes: we can do that, but honestly, I think the most important thing is an inventory of what we have
16:29:12 <jaypipes> westmaas: no probs, and agreed.
16:29:18 <patelna> agree - we need inventory
16:29:25 <westmaas> jaypipes: in terms of both what is covered and how its covered
16:29:34 <nati2> westmaas++  for inventory
16:29:50 <jaypipes> ++
16:29:53 <rohitk> westmaas: getting the various suites together is alright, but i doubt that they would all run from the unified script
16:29:56 <westmaas> rohitk: if you bring them in, can you also add to this page: http://wiki.openstack.org/openstack-integration-test-suites?highlight=%28integration%29%7C%28tests%29
16:30:06 <donaldngo_hp> each test framework has configuration that needs be done editing config files creating objects for swift and images for nova. How will we deal with that?
16:30:06 <rohitk> since they were designed for different runners - nose, dtest, etc
16:30:19 <donaldngo_hp> its not a plug and run type of thing
16:30:38 <rohitk> donaldngo_hp: exactly
16:30:40 <jaypipes> donaldngo_hp: the config files will depend on what system the tests are run against.
16:30:48 <westmaas> rohitk, donaldngo_hp: right now that's all being kept inside each test suites directory
16:30:49 <wwkeyboard> donaldngo_hp: For now they well need to be configured in their own dirs
16:30:51 <nati2> rohitk: we should discuss it with reading test code.
16:30:58 <jaypipes> donaldngo_hp: would be good to have example configs, but more than that I'm not sure will be useful.
16:31:23 <AntoniHP> donaldngo_hp: that is not as difficult, tests we run now try to detect environment they are in and the proceed accordingly
16:31:29 <wwkeyboard> I think it will be easier/more prudent to move the missing functionality into the One True Framework than try and configure the rest automatically.
16:31:37 <patelna> can we action this to westmass, Rohit and donald?
16:31:57 <westmaas> another piece of inventory is what each suite cares about in terms of configuration - my guess is it is all very common with different names, all of which will inform the One True suite
16:33:20 <dwalleck_> westmaas: I would think as we consolidate tests, we should probably be able to consoldate configurations
16:33:22 <jaypipes> OK, so AntoniHP, dwalleck_, rohitk: please all of you go to your Gerrit settings, click Watched Projects, and make sure you have openstack/openstack-integration-tests under your project list, and that you check all the checkboxes for notification!
16:33:48 <jaypipes> donaldngo_hp: you too :)
16:33:48 <rohitk> jaypipes: gotcha
16:33:57 <donaldngo_hp> roger that
16:34:02 <dwalleck_> will do
16:34:19 <nati2> jaypipes: please include me :)
16:34:28 <AntoniHP> ok
16:34:39 <jaypipes> nati2: it's up to you! you have to edit your settings in Gerrit yourself ;)
16:34:51 <jaypipes> westmaas, all: OK, are there other issues you'd like to talk about re: integration tests?
16:34:59 <nati2> gotcha! I misunderstood :)
16:35:04 <jaypipes> np
16:35:21 <rohitk> jaypipes: yes!
16:35:22 <patelna> yes...approach to add more coverage?
16:35:51 <jaypipes> patelna: you mean approach to track the "inventory of tests" that cover project functionality?
16:35:55 <rohitk> not sure if many of us have  really looked at lettuce, westmaas, have you?
16:36:15 <patelna> jaypipes - yes + add gaps
16:36:20 <westmaas> patelna: I think we need to triage getting things running and a generic approach, I'm not sure this meeting is the right forum to plan beyond that
16:36:28 <westmaas> patelna: at least until we have those in place
16:36:38 <westmaas> patelna: although I 100% agree its super important
16:36:52 <jaypipes> patelna: I think it would be useful to log bugs in the openstack-qa project on launchpad for those gaps, similar to how nati-san's team is doing for the unit test coverage
16:37:07 <dwalleck_> rohtik: I think moving to BDD would be a huge shift, that's something we'd really have to look into
16:37:13 <patelna> agree
16:37:23 <jaypipes> patelna: agree on BDD or bugs in LP? ;)
16:37:32 <rohitk> dwalleck_: i thought so too, it might be kind of late for that
16:37:34 <westmaas> rohitk: I have not looked
16:37:38 <Ravikumar_hp> jaypipes : or blueprints to add/fill test gaps?
16:37:43 <patelna> jaypipes bugs on LP
16:38:36 <jaypipes> Ravikumar_hp: I think bugs would be better... blueprints are more for large things... of course, we could track "Find Gaps in Swift <-> Glance Integration" in a blueprint and then have multiple bugs for the individual gaps identified, etc...
16:38:44 <jaypipes> but I think bug reports is a good first start.
16:39:20 <jaypipes> patelna: OK, well we made you an admin on the openstack-qa project, so you can add, assign, and target bugs in that project. I'll work with you offline to set some up to get you familiar with the proces.
16:39:45 <patelna> ok Jaypipes
16:39:51 <jaypipes> alright, shall we move on to a status report from nati2 on unit testing progress?
16:39:53 <nati2> topic "integration tests status." is finised?
16:39:59 <dwalleck_> But to some degree we're going to need to make some shift. We need to have one agreed-upon way of of developing these
16:40:02 <dwalleck_> okay, done :0
16:40:04 <dwalleck_> :)
16:40:09 <jaypipes> :), ok...
16:40:11 <wwkeyboard> that topic will never be done
16:40:19 <jaypipes> #topic status report from nati2 on unit tests
16:40:21 <wwkeyboard> :)
16:40:24 <jaypipes> nati2: you're up!
16:40:41 <nati2> ah, my irc stopped
16:41:03 <nati2> I added details of blueprint https://blueprints.launchpad.net/openstack-qa/+spec/nova-manager-api-policy
16:41:13 <nati2> https://blueprints.launchpad.net/openstack-qa/+spec/nova-input-value-validation-policy
16:41:25 <nati2> For input validation, we have a Idea.
16:41:45 <nati2> We are developing struts like input validation framework for nova
16:41:51 <nati2> We'll share it
16:42:05 <jaypipes> nati2: could you elaborate a bit more on that?
16:42:13 <nati2> for blueprint?
16:43:18 <nati2> OK, I'll add more detail for each blueprints
16:43:50 <jaypipes> nati2: :) sorry, I meant could you explain to the group here a bit more about what those blueprints are about.
16:44:08 <nati2> I see
16:45:02 <nati2> For QA, our workflow is like this.  Make a spec clear, analyze gap between specs and code , write test code and fix
16:45:31 <nati2> But for coding level, we have no resource to make the spec clear. So we set policies.
16:45:50 <nati2> Nova-manager-api-policy is policy for the use of manger methods
16:46:27 <nati2> See this bug https://bugs.launchpad.net/openstack-qa/+bug/872445 (it is already solved for essex)
16:46:29 <uvirtbot> Launchpad bug 872445 in nova "nova.compute.manager calls methods in nova.volume.manager directly" [Undecided,Fix committed]
16:46:51 <nati2> In this case, the compute manager directly calls volume manager without using it's API.
16:47:07 <nati2> These codings should be avoided
16:47:15 <nati2> Is this make sence?
16:47:37 <jaypipes> nati2: yes, much better, thx.
16:47:48 <jaypipes> nati2: question, though: Why is the Nova bug there marked Fix Committed?
16:47:48 <nati2> OK next nova-input-value-validation-policy
16:48:02 <nati2> Because it is fixed for essex trunk
16:48:17 <nati2> not for diablo stable
16:48:32 <patelna> so u are requestign for backport into stable?
16:49:56 <nati2> Ah, it is problem. We are lacking workflow for bug backport
16:50:40 <nati2> ttx manages bug for essex
16:50:43 <patelna> @Jaypipes?  Do we have this process - any idea...backport from esses to stable branch?
16:51:09 <patelna> sorry typo esses = essex
16:52:15 <jaypipes> nati2: no, I'm wondering why it is marked Fix Committed, but there is no link or reference to the commit that fixed it...
16:52:50 <nati2> jaypipes: ah, ttx marked it as fix commied without link
16:53:16 <jaypipes> patelna: the "process" is to propose the patch for merging into the stable-diablo branch. For these types of things, if the patch just adds test cases, I'm not sure it's a high priority backport, but if the patch fixes a known high priority bug, it should definitely be proposed for merging into stable-diablo
16:53:35 <jaypipes> nati2: OK, no worries. I'm just curious to see if you and your team have started using Gerrit
16:54:00 <jaypipes> nati2: I've been concerned to see a number of bazaar branches put up on Launchpad instead of git branches to Gerrit...
16:54:03 <patelna> Thanks Jaypipes
16:54:28 <jaypipes> patelna: no worries. probably something worth putting here: http://wiki.openstack.org/StableBranch
16:54:31 <nati2> Yes we will going to use git and gerrit. Thanks Jay. I shared how to use it for my japan team
16:55:10 <jaypipes> nati2: ok, excellent. pls do let me or jeblair know if anyone needs assistance with understanding git/gerrit.
16:55:26 <jaypipes> nati2: final question for you on the unit testing bugs...
16:55:45 <jaypipes> nati2: Are you going to be updating the status of the openstack-qa bugs? (hint, hint) :)
16:56:07 <nati2> Ah, I should do it. Thanks
16:56:15 <jaypipes> nati2: right now, they are mostly New status and unassigned...
16:56:25 <jaypipes> nati2: OK, I'll bug you about that later :)
16:56:42 <jaypipes> alright, any further questions for nati-san on unit testing?
16:56:48 <nati2> Yes! It my action item. There are large queue on me to translate japanese bug to english ;(
16:56:55 <jaypipes> hehe, indeed
16:57:13 <rohitk> nati2: I'll be there to help you
16:57:24 <jaypipes> OK, all, we have 3 minutes left...
16:57:34 <jaypipes> #topic Open Discussion
16:57:51 <jaypipes> anybody have anything to bring up? any concerns?
16:58:10 <nati2> Yes I have two
16:58:26 <Ravikumar_hp> jaypipes: is there any docs on best practices for bugs/blueprint and git/gerrit
16:59:08 <jaypipes> Ravikumar_hp: git/gerrit here: http://wiki.openstack.org/GerritWorkflow and http://wiki.openstack.org/GerritJenkinsGithub
16:59:23 <nati2> ah, we should write it. jaypipes: do you have time today? How about to create some screencast with me?
16:59:29 <Ravikumar_hp> jaypipes: thanks
16:59:33 <jaypipes> Ravikumar_hp: bugs here: http://wiki.openstack.org/BugTriage
16:59:45 <jaypipes> Ravikumar_hp: and I'll put something together about blueprints and bugs for the qa team.
16:59:59 <nati2> It was a little bit difficult to understand http://wiki.openstack.org/GerritWorkflow.. But may be screencast is too much
17:00:05 <jaypipes> nati2: not today, but tomorrow, yes.
17:00:06 <rohitk> westmaas: any pointers on the location of the entry point script?
17:00:24 <jaypipes> nati2: also, jeblair may be able to come to the office to help with that.
17:00:45 <nati2> jaypipes: Thanks! I'll access to the  jeblair k
17:00:59 <westmaas> rohitk: just merged today
17:01:01 <nati2> I added a blueprint to API specs
17:01:02 <nati2> https://blueprints.launchpad.net/openstack-qa/+spec/api-specs
17:01:04 <nati2> and
17:01:06 <jaypipes> OK all, I have to run... I will send a short summary to the ML about our meeting. Goodbye fo rnow!
17:01:12 <jaypipes> #endmeeting