Wednesday, 2011-12-07

*** markvoelker has quit IRC00:02
*** wwkeyboard has left #openstack-meeting00:13
*** zul has joined #openstack-meeting00:15
*** rnirmal has quit IRC00:17
*** somik_ has quit IRC00:19
*** jakedahn has quit IRC00:26
*** dragondm has quit IRC00:27
*** sleepsonthefloo has quit IRC00:29
*** AlanClark has quit IRC00:31
*** anotherjesse has quit IRC00:38
*** dendrobates is now known as dendro-afk00:48
*** dendro-afk is now known as dendrobates00:48
*** gyee has quit IRC00:54
*** reed has joined #openstack-meeting00:55
*** adjohn has joined #openstack-meeting01:01
*** ohnoimdead has quit IRC01:13
*** jmckenty has quit IRC01:13
*** _adjohn has joined #openstack-meeting01:14
*** adjohn has quit IRC01:14
*** _adjohn is now known as adjohn01:14
*** SumitNaiksatam has quit IRC01:15
*** dendrobates is now known as dendro-afk01:16
*** beekhof_ has quit IRC01:19
*** beekhof has quit IRC01:20
*** dwcramer has joined #openstack-meeting01:20
*** lloydde has quit IRC01:22
*** zns has joined #openstack-meeting01:31
*** rohitk has quit IRC01:33
*** dendro-afk is now known as dendrobates01:36
*** rohitk has joined #openstack-meeting01:40
*** salv has quit IRC01:43
*** vladimir3p has quit IRC01:47
*** beekhof has joined #openstack-meeting01:55
*** novas0x2a|laptop has quit IRC01:58
*** beekhof_ has joined #openstack-meeting01:58
*** dwcramer has quit IRC02:19
*** CDY has quit IRC02:23
*** mcohen_ has joined #openstack-meeting02:25
*** mcohen has quit IRC02:29
*** mcohen_ is now known as mcohen02:29
*** dwcramer has joined #openstack-meeting02:32
*** shang has quit IRC02:38
*** shang has joined #openstack-meeting02:51
*** mcohen has quit IRC02:52
*** mcohen has joined #openstack-meeting02:53
*** mcohen has quit IRC02:54
*** mcohen has joined #openstack-meeting02:54
*** rohitk has quit IRC03:07
*** bcwaldon has joined #openstack-meeting03:11
*** jog0_ has joined #openstack-meeting03:18
*** novas0x2a|laptop has joined #openstack-meeting03:21
*** adjohn has quit IRC03:28
*** adjohn has joined #openstack-meeting03:40
*** mcohen has quit IRC03:41
*** nati2 has quit IRC03:46
*** nati2 has joined #openstack-meeting03:47
*** rohitk has joined #openstack-meeting03:49
*** dprince has joined #openstack-meeting03:50
*** rnirmal has joined #openstack-meeting03:50
*** dprince has quit IRC03:51
*** dwcramer has quit IRC03:54
*** nati2 has quit IRC04:00
*** anotherjesse has joined #openstack-meeting04:02
*** nati2 has joined #openstack-meeting04:02
*** lloydde has joined #openstack-meeting04:03
*** adjohn has quit IRC04:04
*** bcwaldon has quit IRC04:05
*** anotherjesse has quit IRC04:11
*** donaldngo_hp has joined #openstack-meeting04:11
*** donaldngo_hp has quit IRC04:12
*** troytoman-away has quit IRC04:13
*** donaldngo_hp has joined #openstack-meeting04:16
*** troytoman-away has joined #openstack-meeting04:17
*** donaldngo_hp has quit IRC04:17
*** donaldngo_hp has joined #openstack-meeting04:18
*** donaldngo_hp has quit IRC04:18
*** nati2 has quit IRC04:25
*** nati2 has joined #openstack-meeting04:26
*** bencherian has quit IRC04:31
*** jog0_ has left #openstack-meeting04:43
*** jog0_ has quit IRC04:43
*** jog0 has joined #openstack-meeting04:52
*** jog0 has left #openstack-meeting04:54
*** adjohn has joined #openstack-meeting05:00
*** nati2_ has joined #openstack-meeting05:13
*** dolphm has joined #openstack-meeting05:15
*** nati2 has quit IRC05:17
*** danwent has joined #openstack-meeting05:18
*** _adjohn has joined #openstack-meeting05:19
*** adjohn has quit IRC05:20
*** _adjohn is now known as adjohn05:20
*** rnirmal has quit IRC05:21
*** shang has quit IRC05:24
*** shang has joined #openstack-meeting05:24
*** novas0x2a|laptop has quit IRC05:25
*** novas0x2a|laptop has joined #openstack-meeting05:28
*** bencherian has joined #openstack-meeting05:41
*** nati2_ has quit IRC05:52
*** mjfork has quit IRC05:55
*** bencherian has quit IRC06:06
*** lloydde has quit IRC06:08
*** lloydde has joined #openstack-meeting06:08
*** lloydde has quit IRC06:08
*** jog0_ has joined #openstack-meeting06:16
*** jog0_ is now known as jog006:18
*** dolphm has quit IRC06:23
*** jog0 has quit IRC06:28
*** jog0 has joined #openstack-meeting06:29
*** reed has quit IRC06:54
*** jog0 has quit IRC06:56
*** danwent has quit IRC07:26
*** GheRivero_ has joined #openstack-meeting07:54
*** GheRivero_ has quit IRC08:15
*** debo-os has quit IRC08:17
*** zns has quit IRC09:04
*** adjohn has quit IRC09:07
*** adjohn has joined #openstack-meeting09:11
*** novas0x2a|laptop has quit IRC09:19
*** novas0x2a|laptop has joined #openstack-meeting09:23
*** novas0x2a|laptop has quit IRC09:34
*** novas0x2a|laptop has joined #openstack-meeting09:35
*** darraghb has joined #openstack-meeting09:56
*** yamahata has joined #openstack-meeting10:07
*** adjohn has quit IRC10:12
*** shang has quit IRC10:29
*** novas0x2a|laptop has quit IRC10:46
*** novas0x2a|laptop has joined #openstack-meeting10:47
*** novas0x2a|laptop has quit IRC11:33
*** novas0x2a|laptop has joined #openstack-meeting11:34
*** dwcramer has joined #openstack-meeting12:50
*** novas0x2a|laptop has quit IRC12:57
*** novas0x2a|laptop has joined #openstack-meeting12:59
*** dwcramer has quit IRC13:08
*** zns has joined #openstack-meeting13:18
*** zns has quit IRC13:26
*** mjfork has joined #openstack-meeting13:37
*** dolphm has joined #openstack-meeting13:39
*** dolphm has quit IRC13:44
*** smoser has quit IRC13:44
*** zns has joined #openstack-meeting13:56
*** lloydde has joined #openstack-meeting14:09
*** Adri2000 has quit IRC14:14
*** hggdh has quit IRC14:18
*** Adri2000 has joined #openstack-meeting14:24
*** troytoman-away is now known as troytoman14:27
*** mattray has joined #openstack-meeting14:49
*** hggdh has joined #openstack-meeting14:51
*** dwcramer has joined #openstack-meeting14:51
*** mdomsch has joined #openstack-meeting14:55
*** lloydde has quit IRC15:02
*** debo-os has joined #openstack-meeting15:06
*** dolphm has joined #openstack-meeting15:08
*** zns1 has joined #openstack-meeting15:08
*** zns has quit IRC15:11
*** jmckenty has joined #openstack-meeting15:24
*** dolphm has quit IRC15:26
*** novas0x2a|laptop has quit IRC15:28
*** novas0x2a|laptop has joined #openstack-meeting15:29
*** lloydde has joined #openstack-meeting15:36
*** bencherian has joined #openstack-meeting15:39
*** zns1 has quit IRC15:40
*** dolphm has joined #openstack-meeting15:42
*** vladimir3p has joined #openstack-meeting15:45
*** debo-os has quit IRC15:45
*** reed has joined #openstack-meeting15:49
*** jaypipes has quit IRC15:53
*** derrald has joined #openstack-meeting15:54
*** derrald has left #openstack-meeting15:54
*** CDY has joined #openstack-meeting15:55
*** mahmoh1 has joined #openstack-meeting15:57
*** rnirmal has joined #openstack-meeting15:58
*** edconzel has joined #openstack-meeting15:58
*** edconzel has left #openstack-meeting15:58
*** adjohn has joined #openstack-meeting16:03
*** lloydde has quit IRC16:11
*** hggdh has quit IRC16:13
*** hggdh has joined #openstack-meeting16:14
*** mahmoh1 is now known as mahmoh16:16
*** jaypipes has joined #openstack-meeting16:17
*** mdomsch has quit IRC16:19
*** bcwaldon has joined #openstack-meeting16:31
*** bencherian has quit IRC16:35
*** debo-os has joined #openstack-meeting16:43
*** wwkeyboard has joined #openstack-meeting16:45
*** donaldngo_hp has joined #openstack-meeting16:46
*** Ravikumar_hp has joined #openstack-meeting16:47
*** novas0x2a|laptop has quit IRC16:47
*** coli has joined #openstack-meeting16:49
*** vandana has joined #openstack-meeting16:49
*** zns has joined #openstack-meeting16:57
*** jmckenty has quit IRC17:00
jaypipesmorning QAers.17:00
bcwaldonI think westmaas is afk for this meeting17:01
openstackMeeting started Wed Dec  7 17:01:40 2011 UTC.  The chair is jaypipes. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic.17:01
*** lloydde has joined #openstack-meeting17:01
jaypipeslooks like a number of folks aren't here yet... let's give a couple minutes...17:02
lloyddeis there a QA meeting now?17:02
jaypipeslloydde: yep...17:02
lloyddeoh, just answered my question17:02
lloyddeThanks, I've been having connectivity problems.17:03
jaypipesrohitk: mornin.17:04
jaypipesOK, let's get started I guess..17:04
jaypipes#topic Tempest status17:05
*** openstack changes topic to "Tempest status"17:05
jaypipesDaryl, Rohit and myself spent time this weekend getting Tempest a bit more orderly...17:05
*** RickL has joined #openstack-meeting17:05
jaypipesThere were issues with the configuration, with auth_url, and a number of other things that were preventing a simple nosetests -x tempest from working.17:05
donaldngo_hpthats awesome!17:06
jaypipesI'm still working through trying to get a clean run of Tempest. So far, I've found many of the test cases have side-effects17:06
jaypipeswhich is kinda annoying and I'm working on fixing that up.17:06
jaypipesDaryl also completed the storm -> tempest renaming and so at least we're done with that milestone and merge hell17:07
jaypipeswhat we need now is a concerted effort to get tempest stabilized and gating trunk..17:07
jaypipesand that means a couple things:17:07
Ravikumar_hpjaypipes: yes. i pulled code yesterday and saw the changes, but like like config looks for storm folder or so17:08
jaypipes1) We need lieutenants for each of the major pieces...  these folks should be responsible for carefully going over a section of the test cases17:08
jaypipesRavikumar_hp: that's been fixed :)17:08
jaypipesRavikumar_hp: pull origin master...17:08
Ravikumar_hpok. thanks17:08
jaypipes2) We need to work with jeblair and mtaylor to get Tempest runnable against a devstack deployed clusster17:09
jaypipes#2 is what I've been struggling to fix up on my local machine... and I've been reporting bugs to horizon and tempest steadily for the past 4 days ;)17:09
*** sleepsonthefloo has joined #openstack-meeting17:10
jaypipesI will take the image test cases. I'm looking for volunteers to focus on the server tests.17:10
rohitkon 1) we have started working on that and filing bugs17:10
jaypipesAnd if someone wants to lead the flavors tests, that might be an easy one to start with17:10
jaypipesrohitk: yep, I've noticed.17:11
jaypipesrohitk: still I'd like to have a go-to person for each major component of testing17:11
rohitkwork-in-progess though17:11
rohitkah, yes, i'll get back to you on that shortly17:11
jaypipesrohitk: and it's now 1 week out from the E2 milestones and we're still a ways away from a gateable solution.17:12
*** nati has joined #openstack-meeting17:12
natiHI sorry late17:12
jaypipesnati: np17:12
jeblairregarding #2, we're ready to run tempest against stable/diablo, and we can start gating on that soon.  the next thing we would expect to do is to try to get devstack configuring milestone-proposed, and then run tempest there.17:12
jaypipesjeblair: yup. though tempest doesn't even run cleanly yet...17:13
jeblaironce we can run something on trunk, we're likely to do that post-commit for a while before gating.17:13
jaypipesjeblair: sure17:13
jeblairyep, just a quick update from my perspective.  we're ready to consume tempest when you think it's ready for us. :)17:13
jaypipesjeblair: awesomeness17:13
jeblair(on stable/diablo)17:14
jaypipesjeblair: I expect it will be at least another week.... just judging from the number of failures I'm seeing17:14
jaypipesI think the biggest issue I've seen with the test cases so far is that most of them contain side-effects .. and some even depend on the side effects of other tests cases, which is not good.17:15
natijeblair: Is there any doc or memo for your configurations?17:15
rohitkhas anyone got a 100% pass result with the tests?17:15
jaypipesSo, I'm going to try to get some documentation up on today that highlights how to avoid side effects in test cases17:15
*** bencherian has joined #openstack-meeting17:16
natijaypipes: I think some test case could  be depend each other.17:16
jaypipesrohitk: not me.... plus, there's a bunch of stuff you need to do to a stock devstack deployment to even get past one or two tests -- namely, you need to remove the silly ratelimit middleware17:16
jaypipesnati: no. otherwise there is no way to get decent parallelization in the testing...17:17
jeblairnati: it's devstack with the localrc in that file17:17
jaypipesnati: all test cases need to create, query, and then delete fixtures that the tests look for17:17
rohitki have got most tests to re-run after fixing the side-effects as you mentioned, request qaers file bugs if you see inconsistencies17:17
natijeblair: Thanks17:17
jaypipesrohitk: cool. you push code yet?17:17
rohitkhave fixes in pipeline, maybe by today/tomorrow17:18
natijaypipes: Ideally, it is true. But when we  start writing test, it is not effective way to implement17:18
natijaypipes: I think we should introduce @depentent decorator such as backfire17:18
jaypipesnati: I will respectfully disagree ;)17:18
rohitkwell, me too, dependencies just build up, they should be handle in setup(), teardown()17:19
natijaypipes: hmm, or we should reuse test method for another test17:19
jaypipestest cases should test a series of related actions...17:20
natijaypipes: Current code looks like test case == action. May be, this is another problem17:21
jaypipesso what I am referring to is the (current) practices of doing a test_get_image_metadata(), then test_set_image_metadata() and a test_delete_image_metadata(). Those tests all hit a class-level image and assume state changes from other test methods have occurred. Those should just be a single test method IMHO17:21
rohitkjaypipes: agree, also @depends_on decorator would be useful when the test depends on external resources, say for ex, creation of key-pairs, libvirt  but test shouldnt depend on "each other"17:22
jaypipesand that single test method can be run in parallel if it creates its own fixture, tests against that fixture, and cleans up after itself.17:22
jaypipesrohitk: sure, agreed17:22
natijaypipes: if so, how about another test case which use test_get_image_metadata?17:23
*** coli has quit IRC17:23
donaldngo_hpthats what I struggle with in our framework. we have super long end to end scenarios17:23
jaypipesnati: then that test case is testing a series of actions that should create an image fixture, use the image metadata, etc, and then clean up after itself.17:24
jaypipesdonaldngo_hp: you struggle with test methods that are too long?17:24
*** adjohn has quit IRC17:24
jaypipesdonaldngo_hp: or struggle with test methods that interrelate with each other?17:24
natijaypipes: I got it. Basically, I agree with you to avoid dependency. I wanna discuss test actoin reuse also.17:24
donaldngo_hpsome of the tests cases require repeated actions so you have test name like test_gettoken_create_container_upload_file17:25
jaypipes#action jaypipes to put together some example tests in QA docs17:25
donaldngo_hpand test_gettoken_create_container_upload_file_delete_file_delete_container17:25
jaypipesdonaldngo_hp: but the second test method covers everything from the first... so the first is redundant and should be removed...17:26
donaldngo_hpthis is a scenario that i just made up but sometimes its needed to run tests in series but treat them as a group17:27
jaypipesSometimes, for longer integration tests like we're dealing with in Tempest, I prefer to have a "use case name" and name the test based on the use case. For example:17:27
jaypipesA use case for a user that creates a snapshot of a server, stores that image in Glance, then later deletes the snapshot.17:28
*** nati2 has joined #openstack-meeting17:28
jaypipesI might call that use case "snapshot_archive_delete" and make the test method called test_snapshot_archive_delete()17:28
jaypipesand add comments in the docstring or top of the module that detail the use cases the tests in the module are testing17:29
Ravikumar_hpjaypipes: that test makes call to common functions ... assembed from blocks17:29
jaypipesIn this way, you start to get away from the (unittest-focused) behaviour of splitting everything into tiny little test methods -- which is great for unit testing but not so much for functional testing17:29
jaypipesRavikumar_hp: sure, absolutely, but those building blocks should operate on local state, not a shared class state, as much as possible...17:30
jaypipesRavikumar_hp: pretty much the only thing I think should be in the class-level state should be the openstack.Manager and config stuff.17:30
AntoniHPwell, not really for us - this high resolution of tests means that we can produce more meaningful bug reports17:30
*** nati2 has quit IRC17:30
jaypipesAntoniHP: could you elaborate on that?17:31
AntoniHPyour test case is more useful as three tests with side effects17:31
jaypipesRavikumar_hp: and authentication tokens/users should be in class-level state too...17:31
*** nati2 has joined #openstack-meeting17:31
jaypipesAntoniHP: why?17:31
AntoniHPeasier to report, as failures are more visible17:32
jaypipesAntoniHP: not sure I agree with you on that...17:32
AntoniHPit is more useful to report that storing in glance failed, then that images do not work17:32
natiAntoniHP: I agree.  API A:OK API B:OK  API A and API B:NG. I think your point is this17:33
AntoniHPthe whole point of automation is making development faster, not to make ideal test cases17:33
jaypipesAntoniHP: that has nothing to do with whether you have test methods own their own fixtures...17:33
AntoniHPyes, but nosetest was created for unittests, which in fact we do not run17:34
AntoniHPi think good compromise would be to refer as a case to set of nosetests cases17:34
Ravikumar_hpJaypipes: yes. even in integration testing to some extent  we need to modularize17:34
jaypipesAntoniHP: sorry, you lost me a bit on that last sentence... could you rephrase?17:34
donaldngo_hpwould it be beneficial for all of us to schedule a remote desktop meeting to go over these issues so we can actualy discuss these topics in practice17:35
AntoniHPi think a test case should consist of set of sequences of single nosetest tests17:35
Ravikumar_hpcommon functions . and the integration tests will be written like create_container (test1), add_file (test1.pdf, test1) , delete_file () , delete_container17:36
AntoniHPfor example a test case should be a class, and should always be run as class17:36
Ravikumar_hpAntoniHP: ++17:36
*** nati has quit IRC17:36
AntoniHPinstead of single nose test mapped  to single test case17:36
Ravikumar_hpwithin test case class , make a call to methods which are mostly common functions17:37
jaypipesAntoniHP: well then we are using the wrong tool then, if that is how we want to structure things... nosetest (and Python's unittest2 module on which it is based) would require that we name our methods like test_001_some_action(), test_002_next_action()17:37
rohitkall that nosetests does is discover and run based on certain patterns, are fixtures within the realm?17:37
AntoniHPjaypipes: yes, we use imprefect tool, but there are libraries that allow us to modify this tool to suit our needs17:38
jaypipesrohitk: all I'm trying to say is that I believe fixtures should be created and destroyed within test methods. AntoniHP is advocating for having class-level fixtures that test methods modify.17:38
AntoniHPjaypipes: advantage with nosetests it already fits good with reporting, autodiscovery etc17:38
AntoniHPlast but not least, we already try to make all tests be working with nose17:39
jaypipesAntoniHP: it actually already works well for parallelization, too, if we design test cases and methods properly ;)17:39
nati2I think we are saying same thing as diffrent way17:39
jaypipesnati2: no, I'm not so sure. I think AntoniHP is saying that test fixtures should remain in class-level scope. right, AntoniHP?17:39
nati2Class implementation or method implementation is not important. IMO17:40
nati2Action implementation and test-case should be separated. Isn't that?17:40
AntoniHPjaypipes: yes, maybe not for all, but I think that would be good option17:40
jaypipesnati2: if test methods in a test case class can be run in parallel, that is important, no?17:40
*** dwcramer has quit IRC17:40
donaldngo_hpi really like how kong deals with the sequencing of test flows within a test class. we could declare a class as tests.FunctionalTest and still be able to run our test classes in parallel but tests within a test class are run in sequence17:40
jaypipesAntoniHP: that is how the existing Tempest test case classes work...17:41
nati2 I think  class-level scope should be converted to one method17:41
jaypipesnati2: ?17:41
nati2So they are not difference17:41
*** dwalleck has joined #openstack-meeting17:41
nati2 jaypipes:  I think it is same, one class has one method.17:41
jaypipesdonaldngo_hp: :) I don't like having to name all my tests 098_blah, 097_blah :)17:41
AntoniHPI think one class, many methods in sequence17:42
rohitkAntoniHP: is your concern specific to larger system test cases?17:42
nati2jaypipes: Big method could not be run in parallel.17:42
dwalleckWhew, finally17:42
jaypipesnati2: sure it could! just make sure the test method creates and destroys its own fixtures.... that's my whole point.17:42
nati2AntoniHP: Yes. But if these method depends each other, it look like one bit method same as Jay mentioned.17:42
jaypipesdwalleck: welcome :) we are having a rousing discussion about how to structure test cases and test case methods.17:43
AntoniHPnati2: yes, but it is better to have them as multiple methods17:43
rohitkIMO for functional/medium tests, fixture handling by individual test methods works fine17:43
dwalleckexplicit test dependencies can make for very flaky tests17:43
AntoniHPnati2: because it gives more detail in reporting17:43
dwalleckjaypipes: Just my type of discussion :)17:43
nati2AntoniHP:  We can also define helper method for test. so it is same. IMO17:43
jaypipesdwalleck: to summarize, AntoniHP (and I think Ravikumar_hp and donaldngo_hp) are in favour of class-level fixtures (like we have now, but are being misused IMHO) and I am in favour of test methods creating and destroying thei own fixtures.17:44
*** zns has quit IRC17:44
nati2AntoniHP:  Ah, reporing is another point17:44
nati2I think class method level isolation is good for reporting17:44
nati2jaypipes: How about that? It is a kind of big method17:45
dwalleckHmm...if/when we move to parallelization, class level fixtures will create some challenges17:45
jaypipesdwalleck: that was my point exactly.17:45
nati2dwalleck: If we write big method. it is same17:45
dwalleckFrom a best practices standpoint, other than basic connectivity, I'm always for a test maintaining its own data17:45
dwallecknati2: How so?17:45
jaypipesdwalleck: test *case* or test *method*? :)17:46
nati2For instance,  class X has test_A, test_B,test_C,  and test_X which do ( call_A,call_B,call_C)  is same17:46
dwalleckA long test should not have a negative effect on paralleziation17:46
dwalleckjaypipes: Well that's very specific :) When I say test case, I believe I mean test method based on what you're saying17:47
nati2dwalleck: we can have many smoll test class as test case17:47
*** dolphm_ has joined #openstack-meeting17:47
jaypipesnati2: if test_A, test_B, and test_C all create their own fixtures, they can be run in parallel. But that's NOT the same thing as testing that the *sequence of A, B, and C* happened in a correct fashion.17:47
dwalleckSo right now we have test classes, each with many test methods, which to me are individual test cases17:47
*** adjohn has joined #openstack-meeting17:47
AntoniHPmy proposition is test case to be one test class with many nosetest/unittest test methods inside17:47
donaldngo_hpjaypipes: i think the 001, 002, naming faciltates test steps in a test case. and the test class will be the test case17:48
jaypipesAntoniHP: we already have that... that's how it is today.17:48
rohitkAntoniHP: that is exactly how the unittests work today17:48
AntoniHPand we are happy with that?17:48
nati2jaypipes: Then we could have class XA test_A, Class XB test_B,17:48
dwalleckdonaldngo_hp: I'm confused. Why would we split a test case across many test methods?17:48
*** dolphm has quit IRC17:48
jaypipesAntoniHP: and unfortunately, you need to know the side effects that each method has to understand the beginning state of another test method :(17:48
donaldngo_hpdwallect: for resolution and modularity17:49
dwalleckPerhaps we all have different definitions of what a test case is. To me, a test case is a very atomic thing. If I'm asserting on many things, that becomes a test scenario17:49
AntoniHPjaypipes: yes but I do not see anything wrong about that17:49
donaldngo_hpyour test case would piece together calls to modules17:49
Ravikumar_hpjaypipes: unit tests can create test data , and remove data. but integration tests depend on othe rcomponents, we struggle a bit17:50
*** adjohn has quit IRC17:50
dwalleckdonaldngo_hp: I would have to see an example understand better. I think I see what you're saying, but to me I think I normally handle that as a helper methods to a test class17:50
Ravikumar_hpdonaldngo_hp: integration tests case has to be assembly . calls to modules (functions)17:50
dwalleckIf I have common code I want to share between tests17:50
dwalleckSo one step backwards: what problem are we trying to solve?17:51
dwalleckRight now I think we've reached a best practices discussion, on which it sounds like we all have our own opinions :)17:52
jaypipesdwalleck: we are trying to decide whether side-effects (changes of state to class-level scope/variables) is OK.17:52
nati2dwalleck: I agree for "If I have common code I want to share between tests"17:53
nati2I think reporting is only discussion point17:53
dwalleckjaypipes: Ahh, then we're stuck. To me, side effects are a bad thing. In my opinion, no one test case/method should assume anything, other than provided reference data17:53
jaypipesdwalleck: I agree with you.17:54
dwallecknati2: So what are you looking for from reporting?17:54
AntoniHPI half agree17:54
AntoniHPI agree that test case should not assume17:54
*** jog0 has joined #openstack-meeting17:54
AntoniHPI do not agree that method should not assume17:54
nati2dwalleck: AntoniHP's point is that if we implement a test case as a test class, we can get detailed report from nose result.17:55
jaypipesit's a matter of scope... AntoniHP and others think the atomic unit is the test case class. dwalleck and I think the atomic unit is the test method.17:55
dwalleckSorry, when I say test case, I mean test method. When you say test case, are you thinking test class?17:55
nati2AntoniHP: Is this right?17:55
AntoniHPi thik jaypipes just explained what i mean much better that i did17:55
*** reed has quit IRC17:55
dwalleckI think maybe I get it. Let me try an example:17:55
dwalleckSo if there was a test class called "Create Server", it would have many test methods, each which asserts very specific things about the test creation process17:56
jaypipesdwalleck: right.. it would have methods named test_001_create_instance(), test_002_check_instance_active(), etc17:56
Ravikumar_hpi would create test_Create_Serrver with couple of methods17:57
dwalleckSo the passing of each test method has a single assertion, each which tells you something specific about that process17:57
jaypipesdwalleck: right.17:57
AntoniHPdwalleck: yes17:57 I see the fundamental different. I do the same thing, except I handle that at the test method level17:57
*** jog0 has left #openstack-meeting17:57
jaypipesdwalleck: right... that's the crux of this point.17:57
Ravikumar_hp1) create_server(a), 2) Create_server(b) 3) create_server(invalid)17:57
jaypipesHaving said my opinions, it sounds like dwalleck and I might be in the minority here, so I'd be willing to change direction on this. But, we need to make a decision TODAY on this and stick to it...17:57
dwalleckI don't mind to not see the results of each individual assertion normally, I just want to see where it failed17:58
dwalleckRavikumar_hp: #3 is what I would have a problem with. Now you're not testing something about that specific test case17:58
jaypipesdwalleck: I agree with you... but I can change if the team wants to. Would you be willing to go the test_001_blah, test_002_next_blah route?17:58
rohitkdwalleck: for your test_001 and test_002, is it accessing common code "create_server()"? or ist the creat done in each test17:59
dwalleckIf #1 was assert_response_code_valid, #2 verify password works, etc, I could possibly understand that17:59
rohitki just want to understand where atomic unit lies17:59
jaypipesrohitk: for the test_001, test_002, the tests would be run in numeric order and would assume state changes from previous test method calls.17:59
dwalleckrohitk: Yes. That's the point of having the servers clients and such17:59
dwalleckWe would end up with very, very many test classes this way17:59
rohitkjaypipes: ah, then that's not too good17:59
jaypipesAntoniHP: did I state the above correctly?18:00
jaypipesRavikumar_hp: you too ^^18:00
AntoniHPactuall implementation of such sequence could be in form of test generator, rather  than numbers18:00
dwalleckMy concern is that I often run single test cases. I'm not very comfortable with having to run a whole test class to get a single result18:00
AntoniHPthis way also setUp and tearDown are called once18:00
dwalleckIt becomes a time sink18:00
dwalleckso if the problem is reporting, why not modify nose to give us a result for each assertion?18:01
rohitkif tests are dependent then we should be prepared for wasted nightly runs, one failed caused a whole lot many to fail18:01
AntoniHPif there are many small classes, one can run each class safely18:01
dwalleckrohtik: ++18:02
*** gyee has joined #openstack-meeting18:02
jaypipesdwalleck: that's something you can change in your routine, though, right? simple case of nosetests tempest.tests.test_server_create vs. nosetests tempest.tests.test_server.ServerTest.test_server_create18:02
AntoniHPI do not mean all tests to be dependent but small portions of them18:02
dwalleckAntonioHP: But it comes incredibly verbose18:02
*** dolphm has joined #openstack-meeting18:02
rohitkAntoniHP: in that case those tests should be intelligent enough to recover from side-effects18:02
dwalleckjaypipes: You could if each test method in a test class didn't depend on the state created by previous test methods18:03
dwalleckBut what is being suggested is that it should18:03
jaypipesdwalleck: yes, I understand...18:03
rohitkAntoniHP: i can foresee your requirement for writing larger end-end system tests where knowledge of state is needed, but we just need one example to start off with18:04
AntoniHProhitk: yes, propably they should SKIP if previous part of sequence failed18:04
dwalleckWe seem to be at an impasse...18:04
jaypipesin general, I'm only interested in the stuff that *failed*. I don't care about what passed... So, for me, showing the failed assertBlah() result is what I'm interested in, not necessarily seeing 001 ... OK, 002 .. OK, 003 ... OK...18:04
rohitkAntoniHP: right18:04
AntoniHProhitk: this way in reporting we see error where it happened and where tests where skipped18:05
*** dolphm_ has quit IRC18:05
jaypipesOK, we might be at an impasse here...18:05
AntoniHPoften I do not run my tests, and often I do not even report on bugs from my tests - ability of tests to self explain, skip, fail or error where needed is essential18:05
rohitkjaypipes: agree, but for functional tests, I would also like to have the report/logs containing the "Test data" that was used18:06
dwalleckrohtik: Why wouldn't you know what test data you used?18:06
donaldngo_hpi like seeing passes as well as failures18:06
dwalleckIsn't it specified implicitly?18:06
rohitkbe it a passed/failed test18:06
*** debo-os has quit IRC18:06
jaypipesrohitk: you should check out the glance functional integration tests then :) you will be pleasantly surprised18:06
rohitkdwalleck: im just saying that test data should be explicitly available in reports18:06
rohitkjaypipes: gotcha18:07
jaypipesrohitk: it should be, yes. But it should be done in a way like self.assertEquals(x, y, "x != y. Test data: %s" % my_test_data)18:07
jaypipesOK, so what to do... :)18:08
dwalleckYeah, we have to be consistant18:08
dwalleckAnd we're way over on time :)18:09
dwalleckExamples? perhaps?18:09
jaypipeswell, I don't want AntoniHP and donaldngo_hp to not write test cases in Tempest because they don't like the way the test case atomicity... I want everyone participating. :)18:10
Ravikumar_hpjaypipes: yes,18:10
jaypipesyeah, examples and code speak... I'll put together examples of the two styles and we'll vote on them on the mailing list. Does that sound ok?18:10
nati2I think decorator could be solve18:10
Ravikumar_hpsounds good18:10
donaldngo_hpsounds good18:10
nati2By using decorator we can use both of class way and method way18:11
*** RickL has quit IRC18:11
AntoniHPyes, let discuss it on the list18:11
jaypipesok. AntoniHP, Ravikumar_hp, I will shoot you the examples before I send to ML to double-check I "got it right"18:11
donaldngo_hpi also like the idea of having a working session with a remote deskop and code on someone's screen18:11
dwalleckI think examples, discussion, and then voting is reasonable18:11
jaypipesdonaldngo_hp: worked well for you and me last time! :)18:11
AntoniHPbut decorators are quite complicated18:11
dwalleckBut I hope we can come to some agreement so everyone gets what they need18:11
jaypipesOK, I'll get those examples done today and expect an email from me.18:11
jaypipesdwalleck: yep.18:12
Ravikumar_hpok Jay.18:12
jaypipesalrighty... good discussions everyone.18:12
*** openstack changes topic to "Openstack Meetings: | Minutes:"18:12
openstackMeeting ended Wed Dec  7 18:12:25 2011 UTC.  Information about MeetBot at . (v 0.1.4)18:12
openstackMinutes (text):
*** Ravikumar_hp has quit IRC18:12
*** nati2 has quit IRC18:15
*** bencherian has quit IRC18:16
*** vladimir3p has quit IRC18:18
*** mahmoh has left #openstack-meeting18:20
*** dwalleck has quit IRC18:23
*** donaldngo_hp has quit IRC18:27
*** darraghb has quit IRC18:28
*** jmckenty has joined #openstack-meeting18:31
*** novas0x2a|laptop has joined #openstack-meeting18:32
*** debo-os has joined #openstack-meeting18:37
*** dolphm has quit IRC18:38
*** adjohn has joined #openstack-meeting18:39
*** reed has joined #openstack-meeting18:39
*** mdomsch has joined #openstack-meeting18:44
*** vladimir3p has joined #openstack-meeting18:45
*** donaldngo_hp has joined #openstack-meeting18:45
*** rohitk has quit IRC18:48
*** dolphm has joined #openstack-meeting18:53
*** dolphm has quit IRC18:56
*** bencherian has joined #openstack-meeting19:00
*** nati has joined #openstack-meeting19:01
*** nati has quit IRC19:07
*** dolphm has joined #openstack-meeting19:11
*** dendrobates is now known as dendro-afk19:13
*** dwcramer has joined #openstack-meeting19:14
*** debo-os has quit IRC19:14
*** dwcramer has quit IRC19:19
*** dolphm has quit IRC19:28
*** debo-os has joined #openstack-meeting19:29
*** _adjohn has joined #openstack-meeting19:30
*** sleepsonthefloo_ has joined #openstack-meeting19:31
*** reed_ has joined #openstack-meeting19:31
*** sleepsonthefloo has quit IRC19:32
*** sleepsonthefloo_ is now known as sleepsonthefloo19:32
*** adjohn has quit IRC19:32
*** _adjohn is now known as adjohn19:32
*** reed has quit IRC19:32
*** dwcramer has joined #openstack-meeting19:33
*** dwalleck has joined #openstack-meeting19:38
*** novas0x2a|laptop has quit IRC19:39
*** novas0x2a|laptop has joined #openstack-meeting19:40
*** zns has joined #openstack-meeting19:40
*** bcwaldon has quit IRC19:40
*** dwalleck has quit IRC19:44
*** GheRivero_ has joined #openstack-meeting19:47
*** jakedahn has joined #openstack-meeting19:53
*** debo-os has quit IRC19:53
*** AlanClark has joined #openstack-meeting19:54
*** _adjohn has joined #openstack-meeting19:56
*** adjohn has quit IRC19:57
*** _adjohn is now known as adjohn19:57
*** adjohn has quit IRC19:59
*** vandana has quit IRC20:12
*** dwcramer has quit IRC20:14
*** adjohn has joined #openstack-meeting20:23
*** dwcramer has joined #openstack-meeting20:26
*** dolphm has joined #openstack-meeting20:27
*** dolphm_ has joined #openstack-meeting20:29
*** dolphm has quit IRC20:31
*** dwalleck has joined #openstack-meeting20:34
*** jmckenty has quit IRC20:41
*** jmckenty has joined #openstack-meeting20:42
*** dolphm_ has quit IRC20:43
*** dwcramer has quit IRC20:44
*** mdomsch_ has joined #openstack-meeting20:48
*** danwent has joined #openstack-meeting20:57
*** dolphm has joined #openstack-meeting21:02
*** zns1 has joined #openstack-meeting21:14
*** dwcramer has joined #openstack-meeting21:16
*** zns has quit IRC21:17
*** zns has joined #openstack-meeting21:19
*** zns1 has quit IRC21:22
*** dolphm has quit IRC21:24
*** zns has quit IRC21:30
*** zns has joined #openstack-meeting21:31
*** jmckenty has quit IRC21:31
*** dwalleck has quit IRC21:32
*** danwent has quit IRC21:33
*** zns1 has joined #openstack-meeting21:33
*** danwent has joined #openstack-meeting21:33
*** reed_ is now known as reed21:34
*** zns has quit IRC21:35
*** debo-os has joined #openstack-meeting21:41
*** dwalleck has joined #openstack-meeting21:51
*** dwcramer has quit IRC21:51
*** dwalleck has quit IRC21:52
*** dwalleck has joined #openstack-meeting21:53
*** lloydde has quit IRC21:56
*** mdomsch has quit IRC21:57
*** jmckenty has joined #openstack-meeting22:03
*** dwalleck has quit IRC22:18
*** dwcramer has joined #openstack-meeting22:18
*** lloydde has joined #openstack-meeting22:21
*** GheRivero_ has quit IRC22:21
*** rnirmal has quit IRC22:29
*** lloydde has quit IRC22:30
*** rohitk has joined #openstack-meeting22:35
*** jmckenty_ has joined #openstack-meeting23:00
*** jmckenty has quit IRC23:00
*** lloydde has joined #openstack-meeting23:01
*** troytoman is now known as troytoman-away23:02
*** lloydde has quit IRC23:15
*** dolphm has joined #openstack-meeting23:16
*** lloydde has joined #openstack-meeting23:18
*** bencherian has quit IRC23:24
*** mattray has quit IRC23:33
*** nati2 has joined #openstack-meeting23:35
*** AlanClark has quit IRC23:39
*** heckj has joined #openstack-meeting23:41
*** dwcramer has quit IRC23:46
*** dolphm has quit IRC23:50
*** zns1 has quit IRC23:56
*** jakedahn has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!