17:00:52 <jaypipes> #startmeeting
17:00:53 <openstack> Meeting started Wed Nov 30 17:00:52 2011 UTC.  The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:01:11 <dwalleck> jaypipes: Sounds good. Actually when I went to gerrit this morning, it shows my commit. strange
17:01:25 <jaypipes> westmaas, dwalleck: How's progress on identifying the missing functional tests?
17:01:48 <dwalleck> Missing as in the ones not committed yet?
17:02:09 <jaypipes> dwalleck: and missing as in "we should be testing this set of functional commands but we aren't" ;)
17:02:24 <dwalleck> I have one branch submitted for image meta tests.
17:03:04 <jaypipes> dwalleck: k, will review that one shortly.
17:03:05 <dwalleck> I have at least 3 other suites I need to bring over, but I wanted to make some of my team's work more generic so it works for everyone
17:03:17 <jaypipes> dwalleck: timeframe on that?
17:03:41 <dwalleck> Right now they're making some assumptions on what IP addresses an environment has
17:04:02 <jaypipes> k
17:04:03 <dwalleck> End of day friday would be safe. It's just some minor refactoring, and porting over the work from this week
17:04:06 <Ravikumar_hp> jaypipes: where we will put the list of missing tests
17:04:18 <jaypipes> Ravikumar_hp: that's what I'm trying to determine ;)
17:04:22 <dwalleck> I've started it as bugs for openstack integration
17:04:33 <jaypipes> dwalleck: k.. lemme grab link.
17:04:50 <jaypipes> #link https://bugs.launchpad.net/openstack-integration-tests
17:04:57 <dwalleck> I have the list on my team's Redmine. I started porting it over to bugs, but the holidays interrupted me
17:05:31 <dwalleck> That I'll commit to finishing by end of day today
17:06:12 <jaypipes> dwalleck: sounds good. I'll help with the assignment and prioritization on LP.
17:06:27 <dwalleck> And this list will only contain bugs for functionality we feel hasn't been tested yet for the sake of brevity
17:06:33 <jaypipes> dwalleck: just FYI, the reason it's important is to prevent duplication of effort with Ravikumar_hp's team...
17:06:45 <nati2> Hi Sorry late
17:06:54 <dwalleck> jaypipes: Totally understood
17:07:03 <jaypipes> I will also create milestones for the openstack-integration-tests project...
17:07:18 <jaypipes> #action dwalleck to complete redmine -> LP bug migration
17:07:30 <jaypipes> #action jaypipes to create milestones in openstack-integration-tests project on LP.
17:08:04 <jaypipes> OK, continuing on integration tests topic, I'd like to discuss the documentation we're working on...
17:08:07 <dwalleck> Our team also brought up a point I wanted to bring up here. Their concern was that some of our tests might be considered almost white box, so I wanted to bring that topic here as well
17:08:14 <dwalleck> Maybe at the end
17:08:47 <jaypipes> dwalleck: sure, let's definitely discuss that right after docs.
17:08:51 <jaypipes> #link http://qa.openstack.org/
17:09:24 <jaypipes> So, I put up some placeholder stuff and dwalleck has graciously filled out some instructions... I'm going to be uploading them shortly after this meeting.
17:09:35 <jaypipes> They will be going here:
17:09:37 <jaypipes> #link http://qa.openstack.org/integration.html
17:10:01 <jaypipes> Daryl has written details about the directory structure of tempest and how to write a new test case.
17:10:15 <nati2> Cool
17:10:25 <jaypipes> These instructions should be helpful to anyone interested in writing new test cases (Ravi, anotherjesse, etc)
17:10:52 <Ravikumar_hp> jaypipes++
17:11:06 <jaypipes> And FYI, for anyone who would like to contribute to the documentation efforts in QA, please do this:
17:11:31 <nati2> Is there branch for openstack-qa-doc?
17:11:36 <jaypipes> Go to your Gerrit Settings area and make sure you have added openstack-dev/openstack-qa to your Watched Projects list
17:11:48 <jaypipes> nati2: github.com/openstack-dev/openstack-qa
17:11:59 <jaypipes> #link http://github.com/openstack-dev/openstack-qa
17:12:01 <nati2> jaypipes: Thanks
17:12:10 <jaypipes> It is managed in Gerrit like the other core projects are.
17:12:15 <jaypipes> So please do contribute
17:12:34 <jaypipes> OK, dwalleck, let's discuss white box
17:12:41 <dwalleck> Right
17:13:26 <dwalleck> So as part of our tests for Nova, we've found that we've had to SSH into the server instances we've created to fully verify some actions
17:13:47 <nati2> dwalleck: I agree.
17:14:09 <dwalleck> That immediately raised the white box flag for some people, but others feel that since the instance itself is a public facing entity, it is still black box
17:14:19 <rohitk> dwalleck: and the security groups and rules need to be created manually, right?
17:14:56 <dwalleck> It's a tricky spot. We've already ran into a few bugs where Nova has lied to us via the API about what its done :)
17:15:18 <dwalleck> rohtik: Those are things my team hasn't hit yet
17:15:44 <jaypipes> dwalleck: I would say that for the initial set of black box tests, we should steer clear of SSH'ing into the instances to verify state. At least for the top-priority tests, we are concerned with "does the API call take input and return output the way it is spec'd"
17:15:54 <dwalleck> So for what goes into Tempest, how do you all feel?
17:16:05 <dwalleck> That's fair
17:16:11 <AntoniHP> maybe looking at console output would a way to verify without depnding on network
17:16:37 <nati2> AntoniHP: It also needed
17:16:45 <dwalleck> AtonioHP: Which console output?
17:16:57 <rohitk> #agree with jay for basic verification
17:17:04 <jaypipes> dwalleck: I think AntoniHP is referring to VNC console output
17:17:04 <AntoniHP> VM console, euca-get-console-output
17:17:09 <nati2> And also, I wanna check server logs for negative case.
17:17:37 <dwalleck> Ahh, I see
17:17:52 <AntoniHP> like to check if metadata has been correctly fetched, or all devices are detected in VM
17:18:00 <nati2> tempest also coulde be used for deployment setting check. So it is useful if we can check VM without NW
17:18:17 <dwalleck> Yes, there's a lot of different ways to go deeper. I just wanted to make sure what I'm committing now is generic enough for all to use
17:18:24 <jaypipes> AntoniHP: I don't have a problem with checking console output per-se, but I believe that should be a separate test case type. I think the test cases in tempest right now should stay at the "validate the API" level, and we should complete the set of those tests before diving deeper. But that's just my opinion...
17:19:13 <nati2> jaypipes: I agree
17:19:13 <dwalleck> jaypipes: ++. We have seperate stories for console testing (which, if anyone has any thoughts on automating, I'd love to hear it :)
17:19:21 <jaypipes> AntoniHP: plus, IME, if an operation goes wrong that is only visible in white-box testing (and not in the output from an API call), I think that's a bug, too! :)
17:19:39 <dwalleck> Also, the SSH approach does not work for Windows instances of course
17:19:58 <dwalleck> Which is a whole other issue for another day
17:20:03 <jaypipes> dwalleck: ya :)
17:20:44 <rohitk> SSH would be required for many use cases, checking attached volumes, injected files etc, but those should probably fall under a different set of tests
17:20:50 <jaypipes> So, to be clear, I don't think anyone is against white-box testing approaches (like SSH or console output checks), but we're saying we should first get the API-only tests completed.
17:20:58 <dwalleck> So for now, I'll leave the SSH calls out of this
17:21:05 <dwalleck> And we can revisit later
17:21:10 <AntoniHP> jaypipes: it is asynchronous process, API calls just succeed if they are acceptable by server
17:21:45 <nati2> We should have API-only tests milestone
17:21:52 <jaypipes> nati2: ++
17:22:02 <jaypipes> nati2: dwalleck and I are going to get those done today.
17:22:07 <jaypipes> nati2: the milestones...
17:22:09 <nati2> jaypipes: Thanks
17:22:15 <AntoniHP> jaypipes: it is not different then polling database later to see if changes have been reflected
17:22:55 <jaypipes> AntoniHP: sure, but a follow up call (for instance "show me the result of this prior reservation call" should return an error if an error happened. If the only way to determine an error happened is to log into a box, that's a bug :)
17:23:15 <jaypipes> AntoniHP: I think we understand each other :)
17:23:37 <jaypipes> AntoniHP: I'm just suggesting those *type* of tests be in a separate area than "tempest"
17:23:51 <jaypipes> s/area/directory or module/
17:24:34 <nati2> I think white-box test should be in separate directory  or module on tempest in future
17:24:41 <AntoniHP> jaypipes: IMHO shoing console output falls into "how me the result of this prior (..) call"
17:24:45 <jaypipes> FYI, http://qa.openstack.org/integration.html now contains dwalleck's updates
17:25:16 <rohitk> cool
17:25:30 <jaypipes> AntoniHP: ok. what do others think? does console output fall into a separate category than SSHing to verify results?
17:26:07 <rohitk> IMO, yes since that is a different API call
17:26:12 <jaypipes> in other words, should we add methods in tempest base test classes to connect to a VNC console and verify output?
17:26:16 <dwalleck> To generalize, we could have a set of white-ish box tests which, for lack of a better term, test the side effects of API calls
17:26:48 <nati2> dwalleck: #agreed
17:27:08 <nati2> What's discussion point?
17:27:09 <jaypipes> dwalleck: no, I think AntoniHP is suggesting that in addition to validating the return of an API call, for some methods (like run_instance, etc), we do an assert in the test cases in tempest that connect to VNC and verify output.
17:27:25 <jaypipes> AntoniHP: did I summarize that correctly?
17:27:35 <AntoniHP> I meant accessing text-file where nova stores output from console using API call, rather than going to VNC
17:28:04 <AntoniHP> I understand that when I call this from boto library, all that I talk to i API server
17:28:05 <nati2> AntoniHP: Yes, VNC not needed to check console output
17:28:10 <jaypipes> AntoniHP: ok. that would indeed be easier (still checking the same output, though, of course...)
17:28:33 <Ravikumar_hp> yes. vnc NOT needed
17:28:51 <jaypipes> dwalleck: thoughts? add in methods in server actions test cases that validate console output?
17:28:56 <AntoniHP> but there is no dependency on networking, unlike SSH and/or VNC
17:29:09 <jaypipes> dwalleck: AntoniHP is correct that this is a public API output that would be checked (not internal state)
17:29:36 <nati2> jaypipes: If someone write the test code, it should be acceped
17:29:57 <jaypipes> nati2: yes... we're just having a bit of a theoretical/philosophical debate here ;)
17:30:09 <nati2> jaypipes: gotcha
17:30:27 <dwalleck> jaypipes: I think I'm still on the wagon. By validating the console output, I'm just a bit unclear of where that occurs
17:30:30 <jaypipes> nati2: not about *whether* to accept the test code, but *where* to put it :)
17:30:34 <dwalleck> Is it logged within the server instance itself?
17:30:48 <jaypipes> dwalleck: there is an output file, yes
17:31:06 <jaypipes> dwalleck: so UIAM, we'd SSH into the server instance and read that output text file.
17:31:08 <nati2> jaypipes: I got it.
17:31:11 <jaypipes> AntoniHP: right?
17:31:25 <dwalleck> jaypipes: Ahh, then that's clear now. That sounds reasonable
17:31:28 <nati2> jaypipes: Console output is one of API result, so it should be go to the black-box,IMO
17:31:35 <AntoniHP> I think not, this is accessed by nova-compute code and provided to API server
17:31:49 <jaypipes> nati2, AntoniHP: ah.... was not aware of that!
17:31:59 <jaypipes> is this for both EC2 *and* OpenStack API?
17:32:12 <nati2> jaypipes: I'm not sure, EC2 has that output
17:32:24 * jaypipes checks real quick... one sec
17:32:30 <AntoniHP> it basically call : please send me contents of specifica file from compute node
17:32:51 <rohitk> there is an OSAPI equivalent too
17:32:54 <nati2> IMO, some discussion point is mixing now
17:32:57 <rohitk> do a GET on console
17:33:30 <jaypipes> rohitk: is it an undocumented extension? I can't see anything about it at http://docs.openstack.org/api/openstack-compute/1.1/content/
17:33:35 <nati2> 1 Console output checking is while or black ? 2 Where we put while-box test 3 milestone
17:33:52 <bcwaldon> sorry I showed up late, but what's the question here?
17:33:52 <jaypipes> nati2: we have decided console output checking is "black"
17:34:07 <nati2> jaypipes: gotcha
17:34:25 <dwalleck> bcwaldon: Talking about whether accessing an instance crosses the line between white and black box testing
17:34:38 <jaypipes> bcwaldon: to summarize, we were discussing where the white-box vs. black-box boundary was regarding methods of validating input and output of public interfaces
17:34:41 <rohitk> jaypipes: I did not see documentation for it but recall from an ML /<server_id>/consoles/<console_id>
17:34:46 <bcwaldon> ok, and the question about consoles?
17:34:57 <bcwaldon> to be clear, consoles is an undocumented extension (as jaypipes said)
17:35:06 <jaypipes> bcwaldon: is verifying console output black box testing? we decided "yes" after some discussion.
17:35:07 <AntoniHP> to my knowledge there is no call in basic openstack api for getting console output
17:35:10 <bcwaldon> and the interface *will* be changing
17:35:16 <nati2> #https://servers.api.openstack.com/v1.1/{tenantId}/servers/{serverId}/consoles
17:35:34 <nati2> I suppose console for Openstack API was an extension and undocumented
17:35:43 <bcwaldon> nati2: exactly
17:36:00 <jaypipes> alright, so that answers that question for now...
17:36:05 <bcwaldon> it probably shouldnt be tested in its current state
17:36:14 <jaypipes> but it brings up another big issue: EC2 vs OpenStack API.
17:36:15 <nati2> bcwaldon: Ah sorry it was not in contrib directory,, but it is still undocumented
17:36:22 <nati2> #https://docs.google.com/spreadsheet/ccc?key=0Ap9P99ymj9wLdEx2OEtyODNKOXpWODNObmpyR29LLUE#gid=0
17:36:30 <nati2> My API list covers that
17:37:04 <jaypipes> ok, everyone hold up for a second... let me summarize the discussion on this so far and get a vote from everyone. PLEASE WAIT.
17:37:25 <jaypipes> Please #agree if the following statement is correct:
17:38:06 <jaypipes> We agree that verifying console output is black-box testing and that we should aim to add methods to functional test cases (where applicable) that verify console output via API calls.
17:38:23 <nati2> #agree
17:38:26 <rohitk> #agree
17:38:27 <Ravikumar_hp> #agree
17:38:29 <AntoniHP> #agree
17:38:30 <dwalleck> #agree
17:38:33 <donaldngo_hp> #agree
17:38:36 <jaypipes> Please #agree if the following statement is also correct:
17:39:36 <jaypipes> Currenttly EC2 supports console output API calls, but OpenStack API needs an extension. We shall document this in the integration test suite and have the afore-mentioned console output methods check to see if the OpenStack API extension is enabled before trying to gather console output in the OpenStack API tests
17:39:52 <bcwaldon> #agree
17:39:55 <AntoniHP> #agree
17:40:01 <dwalleck> #agree
17:40:02 <Ravikumar_hp> #agree -
17:40:06 <donaldngo_hp> #agree
17:40:13 <nati2> Ah OpenStack API don't need extension for console
17:40:30 <nati2> I checked the code, console was not in extension directory
17:40:35 <Ravikumar_hp> #info - we are writting OS API extension to support ec2 console ouput
17:40:53 <nati2> I'm not sure it is extension or not. But there is not docs in OS API
17:41:15 <bcwaldon> nati2: this is a best-case scenario, we will have to wait until consoles is documented as an extension
17:41:19 <Ravikumar_hp> it is NOT yet in OS API extension
17:41:23 <jaypipes> nati2: ok, thank you for checking that. may I assign you a task for finding out the facts about this extension?
17:41:36 <annegentle> I just logged a bug 898266 about the missing docs for console
17:41:37 <uvirtbot> Launchpad bug 898266 in nova "API command not documented - {tenantId}/servers/{serverId}/consoles" [Undecided,New] https://launchpad.net/bugs/898266
17:41:48 <jaypipes> wow, that was fast! :)
17:41:59 <jaypipes> nati2: may annegentle assign that to you?
17:42:09 <bcwaldon> keep in mind it needs to first be converted to an extension, then it can be documented
17:42:12 <annegentle> if it's not in the contrib dir that's odd
17:42:14 <nati2> jaypipes: gotcha, I'll talk with anne
17:42:14 <jaypipes> ah.
17:42:19 <annegentle> nati2: sounds good
17:42:19 <jaypipes> nati2: k, thx
17:42:19 <bcwaldon> and it seems like Ravikumar_hp is handling the first part
17:42:41 <nati2> One question,  who can decide it is extension or not?
17:42:51 <bcwaldon> nova-core and the nova-api team
17:43:07 <jaypipes> #action nati-san to determine current (and planned) status of the console output API extension/feature in OS API and work with annegentle to document it
17:43:18 <rohitk> question: get-console is there in OSAPI
17:43:20 <Ravikumar_hp> it has to be extension.
17:43:24 <nati2> OK I'll ask it on mailing list
17:43:29 <rohitk> are we talking about moving it to an extension?
17:43:52 <bcwaldon> rohitk: yes
17:43:58 <bcwaldon> rohitk: it cannot exist in its current state
17:44:24 <rohitk> bcwaldon: ok
17:44:29 <jaypipes> OK, can I ask (for expedience's sake) that we take that particular discussion offline and report status back on ML?
17:44:41 <bcwaldon> absolutely :)
17:44:47 <Ravikumar_hp> yes
17:44:53 <jaypipes> alrighty, we have another big issue to discuss...
17:45:11 <jaypipes> #topic EC2 API tests not currently written in tempest
17:45:18 <jaypipes> so....
17:45:40 <jaypipes> while dwalleck's team's focus is on the OS API, other teams need to focus on the EC2 API.
17:45:58 <jaypipes> and I'd like tempest to be able to run both sets of API tests of course
17:46:10 <bcwaldon> jaypipes: have you brought this to the attention of the ec2 api team?
17:46:20 <jaypipes> bcwaldon: one sec ;)
17:46:25 <nati2> bcwaldon++
17:46:47 <jaypipes> obviously, the next logical step would be to find individuals on the QA team that are interested in writing the EC2 test cases in Tempest to match the OS API ones...
17:47:02 <jaypipes> any takers?
17:47:29 <jaypipes> wow, hold on everybody! one at a time! ;)
17:47:34 <dwalleck> jaypipes: I need to look at the EC2 API, but how different is functionality? Are the workflows the same?
17:47:44 <bcwaldon> jaypipes: sounds like you should ask the ec2 api team
17:47:47 <dwalleck> If so, I might have an idea here
17:47:53 <jaypipes> dwalleck: no, quite different. especially around auth and other things
17:47:58 <dwalleck> Bah
17:48:05 <bcwaldon> humbug
17:48:30 <jaypipes> the alternative is to find an existing lib that stresses the EC2 API calls? boto have one perhaps?
17:48:42 <dwalleck> I have zero to test against, but I wouldn't mind giving it a try
17:48:57 <dwalleck> At least for some basic smoke tests
17:49:21 <jaypipes> dwalleck: well, to be fair, your charter is the OS API. It's what RAX will be running... so I'd prefer to have folks who have a stake in a production EC2 deploy on OpenStack...
17:49:53 <jaypipes> dwalleck: and let's not forget, nova-smoketests already does test the EC2 API (though it is white-box, not black-box, IIRC)
17:50:01 <bcwaldon> jaypipes: correct
17:50:17 <dwalleck> jaypipes: That's fair. I just didn't want to exclude the EC2 folks from Tempest
17:50:34 <jaypipes> OK, then we'll just have to backburner the EC2 API black box testing for now. I will talk with the nova EC2 team to check for interest in helping on test writing.
17:50:42 <nati2> Is there a stake in a production EC2 deploy on OpenStack? EC2 team?
17:50:57 <jaypipes> #action jaypipes to write nova EC2 team to get help on writing black box EC2 API test cases
17:51:19 <jaypipes> nati2: that's what I'm trying to figure out :) I suspect Canonical will be a resource there...
17:51:29 <nati2> jaypipes: Sounds cool
17:51:42 <bcwaldon> jaypipes, nati2: there were a lot of strong voices at the summit in favor of ec2
17:52:02 <jaypipes> OK, so let's open the floor up for discussion. Anybody have things to bring up?
17:52:09 <jaypipes> #topic open discussion
17:52:18 <nati2> I wanna make sure
17:52:39 <nati2> White box test will be accepted on tempest?
17:52:45 <dwalleck> So I've been tinkering with implementing logging
17:52:48 <jaypipes> nati2: no
17:52:56 <AntoniHP> I have a question, is there openstack qa mailing list?
17:53:13 <jaypipes> AntoniHP: no, main mailing list, just post with [QA] in subject
17:53:25 <bcwaldon> there's a launchpad team, so there has to be a ML
17:53:29 <jaypipes> AntoniHP: there may be eventually, but not right now...
17:53:35 <rohitk> question: when are we going to mv to "tempest" in the github repo of openstack-integration-tests
17:53:38 <jaypipes> bcwaldon: trying to keep things unified for the time being
17:53:41 <dwalleck> What I've been logging so far is just requests and responses at info, and response failures as errors.
17:53:43 <nati2> jaypipes: So some guy who needs white box test should create new project?
17:53:50 <bcwaldon> jaypipes: ok
17:53:51 <jaypipes> rohitk: good question... jeblair?
17:54:10 <jaypipes> nati2: or put in a separate directory (like kong is)
17:54:12 <jeblair> hi
17:54:21 <dwalleck> And with my next commit, I'll do the storm -> tempest transition
17:54:28 <jaypipes> nati2: eventually, we'll combine white-box stuff, too, but the priority right now is black box...
17:54:30 <rohitk> dwalleck: thanks
17:54:45 <jaypipes> jeblair: need to map out a time to migrate openstack-integration-tests to tempest...
17:54:45 <nati2> jaypipes: Ah I got it.  templest/white-some-thing.  I agree with you about priority
17:54:46 <Ravikumar_hp> nati2: white box tests can go unit tests . Right?
17:54:52 <jaypipes> nati2: yes, exactly.
17:54:59 <rohitk> i also feel the top-level repository name should change too, any ideas
17:55:04 <nati2> Ravikumar_hp: No it is not unit test
17:55:12 <dwalleck> Ravikumar_hp: They're not really unit tests since they go through the API
17:55:18 <jeblair> yep.  i'm going to get back to work on that now
17:55:22 <jaypipes> rohitk: it was going to be tempest, right?
17:55:26 <donaldngo_hp> why do we consider sshing into a vm with public ip a white box test?
17:55:30 <jeblair> yes, tempest
17:55:33 <dwalleck> It's a pseduo white/black/? type test
17:55:33 <rohitk> yes, openstack-integration-tests --> tempest
17:55:57 <jeblair> who is the best person to schedule timing with?
17:56:12 <jaypipes> donaldngo_hp: it's what you *do* with that SSH connection that determines if it's a white-box or black box test ;)
17:56:29 <jaypipes> jeblair: dwalleck and me
17:56:56 <jeblair> jaypipes: cool, let's chat after meeting
17:57:13 <jaypipes> donaldngo_hp: I think anything that checks some program state that you would need knowledge of the internal workings of an application would be considered "white box"
17:57:17 <dwalleck> jaypipes: I sort of agree. To me, the real absolute white box line is if I directly access a compute node or the Xen API
17:57:27 <dwalleck> Then I'm really exploring behind the wall
17:57:46 <jaypipes> donaldngo_hp: but I agree, the lines can be blurred at times
17:58:16 <donaldngo_hp> imo sshing is no different then calling an api endpoint to spin up the vm
17:59:02 <donaldngo_hp> also SSHing seems to be a better check since the end user will need to so this as opposed to an api call
17:59:09 <donaldngo_hp> to check logs
17:59:42 <jaypipes> donaldngo_hp: SSHing into the instance, sure... SSHing into the host/dom0, not so much :) that's all I'm sayin.
18:00:19 <dwalleck> donaldngo_hp: The only problem our team has run into with the SSH methodology is that it only works for Linux. That's still going to be a vast majority of instances, but if you try to run the tests that way with a Windows instance, no luck
18:00:40 <jaypipes> donaldngo_hp: just clarifying that anything that checks the state of something that requires internal knowledge of the platform implementation (such as querying libvirt or XenAPI directly) would be white-box to me...
18:00:52 <dwalleck> Though I'm still trying to think of a clever way to inject a SSH server into a windows instance on creation :)
18:00:58 <nati2> jaypipes: StackMonkey will do it, (kill process or something). So it should be separate directlry
18:01:22 <jaypipes> nati2: yes, separate from the black-box tests currently in /tempest
18:01:28 <jaypipes> or /storm...
18:01:34 <jaypipes> soon to be /tempest
18:01:41 <nati2> jaypipes: gotcha
18:01:51 <jaypipes> OK, going to wrap up meeting...
18:01:53 <jaypipes> #endmeeting