16:03:38 #startmeeting 16:03:39 Meeting started Wed Oct 19 16:03:38 2011 UTC. The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:03:39 yes 16:03:40 Useful Commands: #action #agreed #help #info #idea #link #topic. 16:03:57 #topic Action items from last week... 16:04:08 one sec, grabbing last week's link.. 16:04:42 http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-10-12-16.05.log.txt 16:04:49 hello 16:04:54 http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-10-12-16.05.txt 16:05:13 jeblair: ping 16:05:49 OK, let me go over all the stuff I failed to get done this past week... :( 16:05:50 is anyone from Grid Dynamics here? 16:05:57 ok 16:06:04 jaypipes to grab all other test suites code and put into openstack-integrated-tests project, each in its own subdirectory. 16:06:13 unfortunately, I did not get around to this. :( 16:06:31 any ETA? 16:06:34 I should have time to do this today, however, as past week been working on glance stuff. 16:06:43 patelna: today/EOB tomorrow at latest. 16:06:46 OK, rohitk: can you do that? 16:06:50 ok...thanks 16:06:56 dwalleck_: afternoon :) 16:07:06 Sorry I'm late, couldn't get connected for some reason 16:07:15 nati2: ok 16:07:21 jaypipes to get this meeting listed on the openstack meetings ics calendar 16:07:21 Gigi is shoulder surfing me also 16:07:30 ah...nice 16:07:34 jaypipes: Is this the structure? http://paste.openstack.org/show/2689/ 16:07:36 ^^ also failed to get that done... will do right after this meeting with ttx 16:07:37 jaypipes: Error: "^" is not a valid command. 16:07:50 rohitk: yep 16:08:00 dwalleck_: cool, no worries 16:08:10 jaypipes: cool 16:08:14 jaypipes: ok, please assign this action for rohit 16:08:36 do anyone know if we have tests for Swift 16:08:40 jaypipes: I added a new test runner that should simplify moving to that new directory structure 16:08:50 #action jaypipes to create blueprint for integration test merging and assign to rohitk 16:09:00 patelna: integration tests or .. ? 16:09:02 https://github.com/openstack/openstack-integration-tests (Kong) is this the framework that we are agreeing to utilize and develop moving forward? 16:09:04 thanks 16:09:11 yes...swift integration tests 16:09:27 donaldngo_hp: no. kong is one of many... we are integration about 6 of them. 16:09:32 patelna: not that I know of, no. 16:09:35 also swift functional tests 16:10:02 Ravikumar_hp: depends what you mean by functional tests :) 16:10:13 jaypipes: Have we decided what the structure of the tests/test framework is going to be going forward? This is something I'm really itching to help with 16:10:37 I have a lot of spare time and nothing to do but design, develop and test :-) 16:10:43 dwalleck_: wwkeyboard and westmaas are working on that. I would advise getting involved in the code reviews for openstack-integration-tests. :) 16:11:21 dwalleck_: westmaas is leading the effort to complete a best practice test runner. I would definitely hope you collaborate with wwkeyboard (Aaron Lee) and Gabe on it! 16:11:34 jaypipes: Sure, of course 16:11:54 lJaypipes: how can we ensure we get all the tests consolidated for core products - swift, nova & glance 16:12:03 I believe jeblair sent out an email about membership for integration tests.... 16:12:42 patelna: integration tests are going into the openstack-integration-tests project. Those tests will stress the integration points between the core projects. 16:12:58 patelna: and each project has zero, some, or a lot of its own functional tests. 16:13:33 patelna: the thing that is missing is really integration tests against a cluster running a bunch of openstack projects... and that is what the openstack-integration-tests project is attempting to solve. 16:13:35 today, we need to talk about this... 16:13:51 jaypipes: did jeblair create this group on LP/gerrit? 16:13:52 patelna: please, let's. what specific questions do you have? 16:13:54 jaypipes: By functional tests, do you mean black-box style or white box style? I wasn't aware there was a seperate black box test suite for Nova or other projects 16:13:58 jaypipes: right now i see only Kong in the integration test project. just be clear we are going to add more frameworks into this repo? 16:13:59 openstack-integration-tests project++ 16:13:59 rohitk: yes 16:14:11 dwalleck_: blackbox 16:14:20 consolidate all the tests that are still not contributed back to the openstack 16:14:26 what meeting is going now ? 16:14:32 blackbox tests for core products 16:14:34 dwalleck_: and Glance has black box tests. /glance/tests/functional/ 16:14:42 zykes: openstack-qa 16:14:47 how about swift? 16:15:17 patelna: one sec... 16:15:19 donaldngo_hp: I would hope as opposed to adding more frameworks, we'd be consolidating them to a single unified platform 16:15:34 dwalleck_: +1 16:15:53 Has anyone defined what we would like to see in a suitable framework ? 16:16:02 I have some definite opinions :) 16:16:15 dwalleck: i agree a single framework. but each of these "frameworks" currently develop differ in their approach 16:16:15 dwalleck_: Even if so we should share our effort at first :) 16:16:22 GavinB: I too was of that opinion that we need to have a list (checklist) of features needed in that framework 16:16:32 or pull in the best from the rest 16:16:33 patelna: Swift has "functional tests" (tests/functional/) but AFAICT, they are not black box. In other words, they don't spin up Swift servers and test the public API. They use internal interfaces. 16:16:41 gavinB: As do I. I've passed around a few things internally, as well as loosely discussed it on the list 16:17:22 Got it... 16:17:24 GavinB: we don't have a lack of frameworks :) We have too many of them... we just need to settle on one or combine the best of breed and focus on writing tests, not test frameworks :) 16:17:25 dwalleck_, GavinB: Test data generation and data sources are high on my list 16:18:06 this needs to be actioned then...adding more swift tests on a runnign server? Gigi - can u take this? 16:18:10 rohitk: Would you write blueprint about Test data generation and data sources? 16:18:29 Nati2 +1 16:18:37 jaypipes: Totally agree - but that argues for a flexible platform that can cope with tests in many forms ... so pretty high level. 16:18:56 I think a good framework to start with is Kong it tests the core REST APIs 16:18:57 rohitk: And for me, the test code structure and design is what I'm most worried about. I think that's going to be the key to the scalability and strength of the suite going forward 16:18:59 GavinB: sure, though I think the first step is just to get *something* going... 16:18:59 also test results in blueprint 16:19:13 nati2: Sure, will need to discuss the specifics and scope 16:19:21 we can get into analysis paralysis trying to create the Uber-Framework and actually not get any test writing done :) 16:19:37 1) Nose 2) KONG to standarize api & functional /integarion 16:19:59 Kong doesn't really provide a framework. It's just directly using http requests 16:20:00 Ravikumar_hp +1 16:20:01 jaypipes: Agree there also. We should select something and get moving - but recognise it will not solve all our needs 16:20:02 I actually don't care what it is :) I just want it done. 16:20:23 When I'm saying framework, I mean more along the lines of Stacktester or Zodiac did in terms of design 16:20:23 wwkeyboard: is westmaas around? 16:20:24 Before test code we need specs and test scenario 16:20:26 dwakkeck_ +1 16:20:30 agree with jaypipes...we need tests that we can run 16:20:38 dwallek_ +1 (sorry !) 16:20:40 We didn't have API specs which we can use yet 16:20:47 jaypipes: He is in a different office. 16:20:53 wwkeyboard: oh, sorry... 16:21:00 nati2: I think we have docs for the API 1.1 spec 16:21:14 Yes, I checked it in this week 16:21:26 jaypipes: soren's branch is fine enough to begin contributing to, however the finer issues of 'no novaclient', etc remain 16:21:36 But it is not cover all features in the code 16:21:45 hey, sorry, bit late 16:21:56 hi westmaas 16:22:01 rohitk: well that's why we're bringing in the other test suites... some of them like backfire use the novaclient. others use httplib2 calls. 16:22:01 Many parameters are not documented and we only have examples 16:22:03 nati2 - should we do traceability matrix on that API spec to ensure we have tests? 16:22:10 patelna++ 16:22:18 spec and traceability matrix on that API spec 16:22:32 OK everyone.... hold up a minute... this is getting a bit freeform... let me try to give some structure to this meeting. 16:22:37 EVERYBODY STOP. :) 16:22:39 that's the 1st step 16:22:43 jaypipes++ 16:22:48 Let us get one thing done at a time. 16:22:57 #topic integration tests status. 16:23:10 westmaas: Gabe, would you care to provide a quick status update? 16:23:46 sure, daryl aaron and I met just after the last meeting, and we talked through a few options, and finally settled on a lightweight runner 16:24:00 jaypipes: patelna: whoever: swift's functional tests do create instances of the servers and test public APIs 16:24:03 the goal of the runner is to give a unified entry point to run all suites in the project as of now 16:24:20 with the expectation that those suites will be folded into a common suite as quickly as possible 16:24:41 its really just a bash script that runs another script in each directory, really really straightforward 16:25:21 notmyname: the functional tests use internal objects (liek swift.Account). They are not black box -- i.e. they don't do straight HTTP calls. 16:26:02 next step is a two step process I think. 1) Identifying what we are testing right now 16:26:14 2) Select how we should test moving forward 16:26:37 westmass: 3) identify the test gaps. missing tests 16:26:53 westmaas: OK, thx for the update. Question: would you prefer that rohitk simply copy/paste the other suites into directories, or copy in the test cases and in the process, convert them to use the new test runner/client? 16:26:57 sounds good 16:26:59 westmaas: 'all suites in the project' means that the script would identify and execute the test suites pulled in from various frameworks? 16:27:01 westmass; Ravikumar_hp +1 16:27:30 jaypipes: that's what I implied 16:27:35 thnx 16:27:37 just bringing them in is fine for a first step, since we haven't selected how to move forward 16:27:58 rohitk: they just look in each directory for a certain script name, nothing more 16:27:59 westmaas: ++ 16:28:09 westmaas: ++ nice start 16:28:12 westmaas: k. is there a wiki or etherpad you're using with wwkeyboard and dwalleck_ to track thoughts on how you'd like to move forward? 16:28:15 oh, k 16:28:17 Just get them in one place, and running will be a start 16:28:35 should we organize dir structure and consolidate in one place 16:28:42 patelna: yes soon 16:28:46 jaypipes: no, not yet 16:28:59 is that waht Rohit will do? 16:29:01 patelna: i'll work on that 16:29:02 jaypipes: we can do that, but honestly, I think the most important thing is an inventory of what we have 16:29:12 westmaas: no probs, and agreed. 16:29:18 agree - we need inventory 16:29:25 jaypipes: in terms of both what is covered and how its covered 16:29:34 westmaas++ for inventory 16:29:50 ++ 16:29:53 westmaas: getting the various suites together is alright, but i doubt that they would all run from the unified script 16:29:56 rohitk: if you bring them in, can you also add to this page: http://wiki.openstack.org/openstack-integration-test-suites?highlight=%28integration%29%7C%28tests%29 16:30:06 each test framework has configuration that needs be done editing config files creating objects for swift and images for nova. How will we deal with that? 16:30:06 since they were designed for different runners - nose, dtest, etc 16:30:19 its not a plug and run type of thing 16:30:38 donaldngo_hp: exactly 16:30:40 donaldngo_hp: the config files will depend on what system the tests are run against. 16:30:48 rohitk, donaldngo_hp: right now that's all being kept inside each test suites directory 16:30:49 donaldngo_hp: For now they well need to be configured in their own dirs 16:30:51 rohitk: we should discuss it with reading test code. 16:30:58 donaldngo_hp: would be good to have example configs, but more than that I'm not sure will be useful. 16:31:23 donaldngo_hp: that is not as difficult, tests we run now try to detect environment they are in and the proceed accordingly 16:31:29 I think it will be easier/more prudent to move the missing functionality into the One True Framework than try and configure the rest automatically. 16:31:37 can we action this to westmass, Rohit and donald? 16:31:57 another piece of inventory is what each suite cares about in terms of configuration - my guess is it is all very common with different names, all of which will inform the One True suite 16:33:20 westmaas: I would think as we consolidate tests, we should probably be able to consoldate configurations 16:33:22 OK, so AntoniHP, dwalleck_, rohitk: please all of you go to your Gerrit settings, click Watched Projects, and make sure you have openstack/openstack-integration-tests under your project list, and that you check all the checkboxes for notification! 16:33:48 donaldngo_hp: you too :) 16:33:48 jaypipes: gotcha 16:33:57 roger that 16:34:02 will do 16:34:19 jaypipes: please include me :) 16:34:28 ok 16:34:39 nati2: it's up to you! you have to edit your settings in Gerrit yourself ;) 16:34:51 westmaas, all: OK, are there other issues you'd like to talk about re: integration tests? 16:34:59 gotcha! I misunderstood :) 16:35:04 np 16:35:21 jaypipes: yes! 16:35:22 yes...approach to add more coverage? 16:35:51 patelna: you mean approach to track the "inventory of tests" that cover project functionality? 16:35:55 not sure if many of us have really looked at lettuce, westmaas, have you? 16:36:15 jaypipes - yes + add gaps 16:36:20 patelna: I think we need to triage getting things running and a generic approach, I'm not sure this meeting is the right forum to plan beyond that 16:36:28 patelna: at least until we have those in place 16:36:38 patelna: although I 100% agree its super important 16:36:52 patelna: I think it would be useful to log bugs in the openstack-qa project on launchpad for those gaps, similar to how nati-san's team is doing for the unit test coverage 16:37:07 rohtik: I think moving to BDD would be a huge shift, that's something we'd really have to look into 16:37:13 agree 16:37:23 patelna: agree on BDD or bugs in LP? ;) 16:37:32 dwalleck_: i thought so too, it might be kind of late for that 16:37:34 rohitk: I have not looked 16:37:38 jaypipes : or blueprints to add/fill test gaps? 16:37:43 jaypipes bugs on LP 16:38:36 Ravikumar_hp: I think bugs would be better... blueprints are more for large things... of course, we could track "Find Gaps in Swift <-> Glance Integration" in a blueprint and then have multiple bugs for the individual gaps identified, etc... 16:38:44 but I think bug reports is a good first start. 16:39:20 patelna: OK, well we made you an admin on the openstack-qa project, so you can add, assign, and target bugs in that project. I'll work with you offline to set some up to get you familiar with the proces. 16:39:45 ok Jaypipes 16:39:51 alright, shall we move on to a status report from nati2 on unit testing progress? 16:39:53 topic "integration tests status." is finised? 16:39:59 But to some degree we're going to need to make some shift. We need to have one agreed-upon way of of developing these 16:40:02 okay, done :0 16:40:04 :) 16:40:09 :), ok... 16:40:11 that topic will never be done 16:40:19 #topic status report from nati2 on unit tests 16:40:21 :) 16:40:24 nati2: you're up! 16:40:41 ah, my irc stopped 16:41:03 I added details of blueprint https://blueprints.launchpad.net/openstack-qa/+spec/nova-manager-api-policy 16:41:13 https://blueprints.launchpad.net/openstack-qa/+spec/nova-input-value-validation-policy 16:41:25 For input validation, we have a Idea. 16:41:45 We are developing struts like input validation framework for nova 16:41:51 We'll share it 16:42:05 nati2: could you elaborate a bit more on that? 16:42:13 for blueprint? 16:43:18 OK, I'll add more detail for each blueprints 16:43:50 nati2: :) sorry, I meant could you explain to the group here a bit more about what those blueprints are about. 16:44:08 I see 16:45:02 For QA, our workflow is like this. Make a spec clear, analyze gap between specs and code , write test code and fix 16:45:31 But for coding level, we have no resource to make the spec clear. So we set policies. 16:45:50 Nova-manager-api-policy is policy for the use of manger methods 16:46:27 See this bug https://bugs.launchpad.net/openstack-qa/+bug/872445 (it is already solved for essex) 16:46:29 Launchpad bug 872445 in nova "nova.compute.manager calls methods in nova.volume.manager directly" [Undecided,Fix committed] 16:46:51 In this case, the compute manager directly calls volume manager without using it's API. 16:47:07 These codings should be avoided 16:47:15 Is this make sence? 16:47:37 nati2: yes, much better, thx. 16:47:48 nati2: question, though: Why is the Nova bug there marked Fix Committed? 16:47:48 OK next nova-input-value-validation-policy 16:48:02 Because it is fixed for essex trunk 16:48:17 not for diablo stable 16:48:32 so u are requestign for backport into stable? 16:49:56 Ah, it is problem. We are lacking workflow for bug backport 16:50:40 ttx manages bug for essex 16:50:43 @Jaypipes? Do we have this process - any idea...backport from esses to stable branch? 16:51:09 sorry typo esses = essex 16:52:15 nati2: no, I'm wondering why it is marked Fix Committed, but there is no link or reference to the commit that fixed it... 16:52:50 jaypipes: ah, ttx marked it as fix commied without link 16:53:16 patelna: the "process" is to propose the patch for merging into the stable-diablo branch. For these types of things, if the patch just adds test cases, I'm not sure it's a high priority backport, but if the patch fixes a known high priority bug, it should definitely be proposed for merging into stable-diablo 16:53:35 nati2: OK, no worries. I'm just curious to see if you and your team have started using Gerrit 16:54:00 nati2: I've been concerned to see a number of bazaar branches put up on Launchpad instead of git branches to Gerrit... 16:54:03 Thanks Jaypipes 16:54:28 patelna: no worries. probably something worth putting here: http://wiki.openstack.org/StableBranch 16:54:31 Yes we will going to use git and gerrit. Thanks Jay. I shared how to use it for my japan team 16:55:10 nati2: ok, excellent. pls do let me or jeblair know if anyone needs assistance with understanding git/gerrit. 16:55:26 nati2: final question for you on the unit testing bugs... 16:55:45 nati2: Are you going to be updating the status of the openstack-qa bugs? (hint, hint) :) 16:56:07 Ah, I should do it. Thanks 16:56:15 nati2: right now, they are mostly New status and unassigned... 16:56:25 nati2: OK, I'll bug you about that later :) 16:56:42 alright, any further questions for nati-san on unit testing? 16:56:48 Yes! It my action item. There are large queue on me to translate japanese bug to english ;( 16:56:55 hehe, indeed 16:57:13 nati2: I'll be there to help you 16:57:24 OK, all, we have 3 minutes left... 16:57:34 #topic Open Discussion 16:57:51 anybody have anything to bring up? any concerns? 16:58:10 Yes I have two 16:58:26 jaypipes: is there any docs on best practices for bugs/blueprint and git/gerrit 16:59:08 Ravikumar_hp: git/gerrit here: http://wiki.openstack.org/GerritWorkflow and http://wiki.openstack.org/GerritJenkinsGithub 16:59:23 ah, we should write it. jaypipes: do you have time today? How about to create some screencast with me? 16:59:29 jaypipes: thanks 16:59:33 Ravikumar_hp: bugs here: http://wiki.openstack.org/BugTriage 16:59:45 Ravikumar_hp: and I'll put something together about blueprints and bugs for the qa team. 16:59:59 It was a little bit difficult to understand http://wiki.openstack.org/GerritWorkflow.. But may be screencast is too much 17:00:05 nati2: not today, but tomorrow, yes. 17:00:06 westmaas: any pointers on the location of the entry point script? 17:00:24 nati2: also, jeblair may be able to come to the office to help with that. 17:00:45 jaypipes: Thanks! I'll access to the jeblair k 17:00:59 rohitk: just merged today 17:01:01 I added a blueprint to API specs 17:01:02 https://blueprints.launchpad.net/openstack-qa/+spec/api-specs 17:01:04 and 17:01:06 OK all, I have to run... I will send a short summary to the ML about our meeting. Goodbye fo rnow! 17:01:12 #endmeeting