17:05:35 <dwalleck> #startmeeting
17:05:36 <openstack> Meeting started Wed Dec 21 17:05:35 2011 UTC.  The chair is dwalleck. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:05:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:05:43 <dwalleck> Nice!
17:05:57 <dwalleck> #topic Project status
17:05:58 <nati2> #topic example tests
17:06:02 <notmyname> btw, ElJose_ and longshot are rackspace cloud files (ie swift) qa people
17:06:05 <dwalleck> err
17:06:15 <dwalleck> #topic example test
17:06:30 <nati2> Ah ok! Let's talk about  Project status first :)
17:06:37 <dwalleck> So did that email make it out? I saw some weird bounced response
17:06:56 <AntoniHP> what was subject?
17:06:59 <dwalleck> I was also on a pile of cold medicine last night, so I'm hoping things were clear :)
17:07:01 <rohitk> i read it
17:07:11 <rohitk> and I liked your proposal
17:07:13 <dwalleck> We'll stick with with the example test cases
17:07:29 <dwalleck> And the discussion of test class structure going forward
17:07:47 <nati2> Gotca!
17:08:00 <Ravikumar_hp> dwalleck: i do not see that email
17:08:33 <AntoniHP> I have not seen it too, did it go to openstack general list?
17:08:39 <rohitk> dwalleck: however there is a one issue
17:08:43 <dwalleck> Did that sound reasonable? nati2, AntoniHP, I was specifically aiming for an approach that met your goals
17:09:00 <dwalleck> It went to openstack-qa-team
17:09:21 <donaldngo_hp> i got it. it was in my junk email box
17:09:27 <donaldngo_hp> check there guys
17:09:46 <dwalleck> rohtik: What issue did you find?
17:09:53 <Ravikumar_hp> yes. it is junk email box . found it
17:10:15 <rohitk> a test server was created for ServerTests in setup() and we are kind of pushed to use that server throughout the test methods in that class
17:10:38 <rohitk> what if I want to create more test servers with different params
17:10:47 <nati2> dwalleck: I understand this is point " Tests do not change the expected state of the resources under test"
17:11:03 <nati2> dwalleck: Then we do setup in setup method
17:11:32 <dwalleck> rohitk: Correct. But I thought that was the point that people wanted was that they wanted to make many assertions on one action, and see each of those results
17:11:39 <rohitk> of course it's just an example that you have stated, I just think that it reduces the flexibility to write various combinations in a single class
17:12:01 <rohitk> dwalleck: yes, as per the last meeting
17:12:14 <dwalleck> So the one thing I forgot to mention was that I could only see this being used for primary test cases for something like create server
17:12:31 <rohitk> dwalleck: yes, so the top level positive primary tests
17:12:35 <dwalleck> If you want to do some specific tests, you would do all of that within one test method
17:13:14 <dwalleck> But if you want to setup one scenario and test many, many things about it, or know that you will expand testing of it later, this something like this might be a good approach
17:13:46 * AntoniHP is reading list archive
17:13:47 <rohitk> dwalleck: Also I would not be able to run delete_server tests before the __verify_tests, as they would fail
17:14:17 <dwalleck> rohitk: Correct. I did note that delete server would needed to be tested elsewhere
17:14:21 <rohitk> dwalleck: It's a better than previous approach to have class level fixture though
17:15:07 <dwalleck> rohitk: I agree, class level fixtures are the ideal situation
17:15:29 <dwalleck> That will help when we move to parallel execution
17:15:37 <rohitk> dwalleck: +1
17:16:14 <dwalleck> So, what I would like to suggest is to see some examples from any interested person of how they would think this would work best
17:16:22 <nati2> Let's me confirmed delete server test case?
17:16:26 <dwalleck> This is just my opinion, and I'd like to see many more :)
17:16:43 <AntoniHP> it makes sense, however why call to create server is in setup rather then a test?
17:17:39 <nati2> dwalleck: If I wanna test delete server (test_delete_server), we should create a server inside test_delete_server ?
17:18:10 <dwalleck> AntoniHP: To allow for parallel execution of the tests. If the server is already created, and all that must be done is validations on the create response, the state of the server, or for example ping
17:18:15 <rohitk> AntoniHP: Check the previous pastebins you can understand the difference in the existing and proposed, IMP is an improvisation
17:18:18 <dwalleck> Then all this can be done at once
17:18:27 <rohitk> *IMO
17:18:49 <dwalleck> nati2: Correct. You would write a single test method for a case like that
17:19:31 <nati2> dwalleck: I got it
17:19:32 <dwalleck> nati2: Or, it could be a test class if you want to do many assertions about deletion of the server
17:20:08 <AntoniHP> but I think the idea would be that parallel execution would run  signle test class, then only thing that would benefit from parallel execution is creation and deletion of server
17:20:12 <dwalleck> for example, delete the server in setup, save the response in the setup, and the tests could be test response code, test if you get a 404 for a GET for that server, etc
17:20:35 <nati2> dwalleck: How about prohibit overwrite of values? like this http://pastebin.com/WRa3GqTS
17:21:19 <nati2> dwalleck: "and the tests could be test response code," I like this idea!! dwalleck++
17:21:28 <dwalleck> nati2: You could certainly do that. I really hope I wouldn't have to write defensive code in my tests, but it's certainly possible
17:22:09 <nati2> dwalleck:  I think to we should have method to check this policy
17:22:32 <nati2> dwalleck: Some test method may break parallel execution
17:22:46 <dwalleck> AntoniHP: I'm not completely sure I follow. If I think I understand, what you're saying is that parallel execution would still have to wait for the class setup to run, and that's right
17:23:17 <dwalleck> That's the trade off. But then you only have to wait for one server to build, not 5. It's an improvement
17:23:21 <AntoniHP> dwalleck: I think a signle process should execute a single class and one class should never be split among many processes
17:24:08 <dwalleck> nati2: I see your point...sorry, Python isn't my primary coding language, but don't they have some concept like Java's "final" keyword?
17:25:04 <dwalleck> AntoniHP: So what you're saying if that I have 10 test classes, I would spawn 10 test threads, each executing one class?
17:25:07 <rohitk> dwalleck: we use that in config.py
17:25:27 <rohitk> where we define methods as properties
17:25:40 <AntoniHP> dwalleck: no more than 10, but yes, no class methods should be executed in parallel
17:25:59 <nati2> dwalleck: like this https://github.com/openstack/tempest/blob/master/tempest/config.py#L23
17:26:19 <dwalleck> rohitk: That has the same effect, but I was hoping it could be as simple as "final image_ref..."
17:27:00 <dwalleck> AntoniHP: Ahh, now I understand. Why would you not want each test class running at the same time?
17:27:26 <rohitk> nati2++
17:27:41 <donaldngo_hp> so we will have class names like test_create_delete_server.py in antonis case. in darryls case we will have test_server.py and have test_delete_server method inside along with other methods to test server api. both can be ran in parallel
17:27:49 <AntoniHP> dwalleck: if there are 100s of scenarios then running them all at once could be a bottleneck
17:27:58 <donaldngo_hp> we just have more classes
17:27:59 <nati2> dwalleck: It says no final keyword http://stackoverflow.com/questions/802578/final-keyword-equivalent-for-variables-in-python
17:28:16 <dwalleck> nati2: Ugh, weird Python =P
17:28:36 <nati2> dwalleck: But we can write read only class
17:28:41 <AntoniHP> it is finally I think
17:28:42 <dwalleck> nati2: Right
17:29:46 <AntoniHP> also python does not have real multithreading, and with multiprocessing class code would be executed multiple times anyway
17:30:57 <dwalleck> AntoniHP: I think I fell off the wagon.  Could you tell me what you think a test class should look like and do?
17:31:36 <AntoniHP> http://pastebin.com/WRa3GqTS]
17:31:38 <AntoniHP> http://pastebin.com/WRa3GqTS
17:32:17 <dwalleck> AntoniHP: One of my teammates I believe has an improved multithreaded execution method. I still need to look at it
17:32:32 <AntoniHP> nosetests uses multiprocessing
17:33:13 <dwalleck> AntoniHP: And then you want each test to run in order always, correct?
17:33:40 <AntoniHP> I would like each test within a class to be exectued in order
17:34:00 <AntoniHP> however each class, containing a sequence of tests, be able to execute in parallel
17:34:40 <dwalleck> AntoniHP: Okay, now I'm clear
17:35:42 <dwalleck> Are you not concerned about the implicit dependencies that creates?
17:36:54 <dwalleck> I'm trying to understand how creating ordered tests helps. To me that complicates the tests by quite a bit
17:37:37 <AntoniHP> to be clear lets call test something that is test, and ntest something that is test in nosetest software
17:37:57 <AntoniHP> so I would like test to be convention a class containing ntests
17:37:58 <dwalleck> The way I've always handled things before is to have no test depend an anything other that data it retrieves or is created in a fixture
17:38:48 <AntoniHP> the reason behind it is it helps with reporting by increasing resolution of data that nosetests return back
17:39:52 <AntoniHP> it is also much easier to follow by humans (IMO), and in larger teams this is of high importance
17:40:11 <dwalleck> Hmm, then the last example I provided did not meet it's goal. The purpose of that example was to meet the desire to have more clear reporting
17:41:06 <dwalleck> So for the result of one test, you would like to see the results of many assertions, correct?
17:41:09 <AntoniHP> I would see it as ntest_1 - make REST call to create server and store reponse somewhere, if call failed mark it
17:41:28 <AntoniHP> ntest_2 verify reponse or if no response skip test
17:41:30 <AntoniHP> etc
17:42:31 <dwalleck> So I think the difference we have is the granularity we want from results of our tests
17:42:43 <dwalleck> Or even the granularity of our tests
17:43:02 <AntoniHP> that is right
17:43:41 <dwalleck> So what you're saying is that create server is a test, which has many test cases, which will all be part of one test class
17:43:41 <AntoniHP> I propose to disccuss it more on the list, rather then take more time on this meeting
17:43:46 <dwalleck> Is that right?
17:43:48 <donaldngo_hp> Antoni isnt your approach like how Kong was?
17:43:48 <dwalleck> I agree
17:43:57 <dwalleck> I don't think we're going to solve this today
17:43:58 <donaldngo_hp> i like Kong btw
17:44:37 <AntoniHP> ok, I will write some emails on during Christmas, I was following wrong mainiling list until today :-(
17:45:02 <dwalleck> Because I really don't follow this approach. But lets keep discussing this. I'm sure we can come to some type of middle ground
17:45:40 <dwalleck> Yes, I think lots of code examples and even result examples would be great!
17:46:05 <dwalleck> And this will give us all some time to mull things over
17:46:24 <dwalleck> Most of my team is already on vacation, and I know they'll want to chime in as well
17:47:00 <AntoniHP> agree
17:47:18 <dwalleck> Okay, so we'll table this discussion for the moment
17:47:27 <dwalleck> #topic General discussion
17:47:58 <dwalleck> Right now I don't have much else to report. I'm trying to make sure whatever work my team is doing is updated in Launchpad
17:49:10 <dwalleck> Does anyone else have any other topics they want to bring up?
17:49:36 <dwalleck> Going once?
17:49:47 <dwalleck> Twice?
17:50:01 <AntoniHP> Merry Christmas :-) Happy New Year
17:50:07 <dwalleck> have a great holiday!
17:50:10 <dwalleck> #endmeeting