17:03:05 <davidkranz> #startmeeting qa
17:03:06 <openstack> Meeting started Thu Feb 28 17:03:05 2013 UTC.  The chair is davidkranz. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:09 <openstack> The meeting name has been set to 'qa'
17:03:32 <davidkranz> #topic Reviews
17:03:46 <giulivo> hi
17:03:51 <afrittoli> hi
17:03:58 <davidkranz> Welcome everyone
17:04:20 <davidkranz> Does any one know the plan for the identity v3 tests?
17:04:49 <davidkranz> I see Sean submitted a qa session about this.
17:05:07 <sdague> I did... :)
17:05:10 <sdague> ?
17:05:13 <mtreinish> I though that I did...
17:05:21 <sdague> yeh, I think mtreinish did
17:05:24 <davidkranz> mtreinish: Sorry about that.
17:05:27 <ravikumar_hp> davidkranz: we need to follow same methodology like our clients Version like Glance v1 , V2
17:05:51 <sdague> ravikumar_hp: we don't actually have a strategy for glance v2... we don't test it
17:05:56 <mtreinish> ravikumar_hp: there is no glance v2 testing right now
17:06:03 <davidkranz> ravikumar_hp: Yes, we decided last week that the configured uri should be the base, without the version.
17:06:05 <sdague> that's what mtreinish is working on figuring out a sane way to do
17:06:17 <davidkranz> The client will look for the 300 response and act accordingly.
17:06:37 <davidkranz> I think the issue os which versions we test and how is slightly different.
17:06:51 <davidkranz> ^^^of
17:07:20 <sdague> davidkranz: on the subject of design sessions, I'm going to have a bunch, but if it's ok I'm going to punt on doing them right away to try to close some release bugs first
17:07:50 <davidkranz> The question is whether we should review and accept the identity v3 tests before the client is configured so they can run, or make them able to run first.
17:07:58 <davidkranz> I would prefer the latter.
17:08:01 <davidkranz> sdague: NP
17:08:29 <sdague> davidkranz: I think it's ok to review them now. We can then refactor for simplicity later
17:08:40 <ravikumar_hp> davidkranz: I agree
17:08:54 <ravikumar_hp> sdague: +1
17:09:19 <ravikumar_hp> davidkranz: Most of V3 is complete and we need to have tests
17:09:27 <davidkranz> ravikumar_hp, sdague : OK, but let's make them always skip rather than skip if v3 is not configured so there are no troubles when they are enabled.
17:09:38 <sdague> I think we've tended in the past to want to wait for infrastructure before we add features, but some times the features are ready and the infrastructure takes time.
17:09:44 <sdague> davidkranz: yeh, agreed.
17:09:53 <sdague> actually, what's needed in devstack to flip on v3?
17:09:59 <ravikumar_hp> davidkranz: agree
17:10:14 <sdague> honestly, the two will gate each other.
17:10:32 <davidkranz> sdague: The auth_uri needs to be the base and we ask keystone for supported versions.
17:10:40 <sdague> ok
17:10:50 <davidkranz> I think that can be done in Tempest first, and then able the tests.
17:11:17 <davidkranz> I don't see why they gate each other. Supposedly v3 is already enabled in devstack keystone.
17:12:20 <davidkranz> ravikumar_hp: I think there is code in the kestone functional tests that provide an example. That's what ayoung told me.
17:12:42 <davidkranz> Any other issues with reviews?
17:12:50 <ravikumar_hp> davidkranz: ok . we will update tests that are in review state
17:13:01 <davidkranz> ravikumar_hp: Thanks.
17:13:01 <ayoung> davidkranz, yes see keysteon/tests/test_v3*
17:14:20 <ravikumar_hp> ayoung: Thanks
17:15:00 <davidkranz> I think we are getting reviews done in a timely matter.
17:15:09 <Nithya> Is rescue feature enabled by default in devstack? I have presently disbled the rescue tests and there is a question by Chris on having them disabled https://review.openstack.org/#/c/22024/
17:15:10 <sdague> yeh, things are moving pretty good
17:15:22 <davidkranz> The only old ones in the queue are the keystone v3 we just discussed.
17:15:59 <sdague> Nithya: if it's an extension, all extensions are loaded by default
17:16:11 <sdague> so no need for a config
17:16:16 <davidkranz> Nithya: How about just giving it a try?
17:16:33 <sdague> so this brings up another good point, auto skipping
17:16:38 <Nithya> I will do that. Thanks David and Sean
17:16:40 <sdague> we do too much of that
17:17:02 <davidkranz> sdague: Yeah.
17:17:03 <sdague> we should have a single flag for extension tests which is "require all extension tests" or not
17:17:10 <sdague> not a per extension flag
17:17:22 <davidkranz> sdague: +1
17:17:36 <davidkranz> sdague: We don't want to support complex configuration for this.
17:17:43 <sdague> because we end up auto skipping when things are broken otherwise
17:17:49 <davidkranz> People can modify their test runner params if theyh want something more.
17:17:58 <sdague> I think there is an ec2 issue around that today
17:18:41 <sdague> so I think we'll need a config purge at some point, maybe that's a session
17:18:48 <davidkranz> sdague: I think a fix for that may have gone in today.
17:19:14 <davidkranz> sdague: Yes, or at least a discussion.
17:19:15 <afrittoli> we could at least tag the tests when they target a specific extension
17:19:18 <sdague> because it's easy enough to query nova about the enabled extensions
17:19:25 <afrittoli> this would make it easier to filter tests
17:19:45 <sdague> yeh, well we need a summit session on tagging and test organizing as well :)
17:19:52 <mtreinish> afrittoli: they should already be split up into separate files for the most part right?
17:19:55 <sdague> because we're going to need features added to testr
17:19:56 <afrittoli> sdague:=1
17:19:59 <afrittoli> +1
17:20:05 <mtreinish> then it's just an exclude option
17:20:13 <sdague> yeh, the exclude option works
17:20:18 <afazekas> As I remember in our bug tracker there was some, request for having tempest to working in a partially configured environment.
17:20:34 <sdague> afazekas: yeh, that's probably true.
17:20:51 <afazekas> sdague: do want to refuse them ?
17:21:31 <davidkranz> afazekas: I think there just needs to be a way to identify the tests of a specific "optional" configuration.
17:21:32 <sdague> afazekas: well what I envision is a single flag which says "allow tempest to skip extensions that aren't active"
17:21:39 <sdague> which we default to no
17:21:51 <davidkranz> sdague: Sounds reasonable.
17:22:03 <afazekas> sdague: ok
17:22:12 <ravikumar_hp> sdague: how to flag extensions that are not active
17:22:13 <afrittoli> I think we need to support  tempest running outside of the community gate / periodic tasks. I mean everyone has to configure it's own test suites / selections / configurations, but the framework should facilitate this
17:22:15 <sdague> and then let tempest make the query to get_extensions on nova to get the list
17:22:30 <sdague> ravikumar_hp: the extension list is REST api call in nova
17:22:32 <afrittoli> I think the more we do this the more contributions we can get
17:23:04 <sdague> afrittoli: so it's fine to also allow for that, but the gate should be the priority at this point
17:23:15 <sdague> because there is plenty to make this good for the gate
17:23:26 <SHREE-HP> <afrittoli>: +1 for the tempest config
17:24:08 <sdague> and the last thing I want to do is make tempest this giant config explosion to suite the incompatible needs of vendor clouds :)
17:24:33 <afrittoli> sdague: +1 I agree
17:25:06 <afrittoli> but we may need to provide some level of abstractions then, some classes which can be rewritten by vendors
17:25:16 <afrittoli> like nova does with drivers / plugins
17:25:31 <sdague> afrittoli: I'm not sure I agree on the statement of rewrite
17:25:34 <afazekas> afrittoli: they can create test classes :)
17:25:37 <sdague> adds, sure
17:25:59 <sdague> but public extensions and server creation in core openstack should act that same
17:26:06 <afrittoli> I'm thinking of use cases like "how to get access to a VM"
17:26:21 <sdague> oh, the whitebox stuff
17:26:38 <sdague> yeh, sure, but lets just get api coverage above 30% first :)
17:26:50 <afrittoli> depending on how you have nova configured you will need a keypair, or a SG, and you'll have to use the tenant nw or the public network or else
17:26:54 <sdague> there is lots of low hanging fruit there
17:27:04 <sdague> afrittoli: so that would make a good session
17:27:09 <afrittoli> sdague: ok fair enough
17:27:14 <sdague> cross vendor whitebox access
17:27:48 <sdague> I'm willing to concede that we need some flexibility there
17:28:24 <afrittoli> sdague: session for the summit you mean
17:29:03 <sdague> afrittoli: yes, summit session
17:29:14 <afazekas> or just someone should create some blueprint  about, what he needs to hook and why..
17:29:27 <sdague> I guess that's a worth while topic, who in the channel is planning on going to summit?
17:29:42 <sdague> to figure out which conversations we can take there, and which we need to have outside of there
17:29:45 <mlavalle> sdague: I am going to the Summit
17:29:48 <afrittoli> sdague: I'll be there
17:29:53 <sdague> <= going
17:30:08 <donaldngo_hp> sdague: me
17:30:14 <mtreinish> sdague: I am
17:30:16 * afazekas unknown
17:30:20 <ravikumar_hp> I am
17:30:34 <davidkranz> I will be there.
17:31:42 <ayoung> dolphm, it was test_trust_crud  that failed when run in the larger body of tests.  I am just reposted the trusts patch as well as my version of the tests.
17:31:53 <ayoung> I got all successes when I ran this way:
17:32:01 <davidkranz> ayoung: wrong channel...
17:32:17 <sdague> :)
17:32:38 <davidkranz> Any one know where we stand on parallel tempest?
17:32:50 <sdague> davidkranz: yeh, I was chatting with cyeoh on that
17:33:02 <davidkranz> sdague: Do tell.
17:33:17 <davidkranz> #topic Parallel tempest
17:33:21 <sdague> so the next steps are pretty big, so we decided it was better to hold until summit to figure out a few things
17:33:41 <sdague> because to use testr for this we'll need to evolve it to understand our resource tracking
17:34:03 <sdague> otherwise parallel will just make us slower
17:34:29 <sdague> lifeless is on board on this in theory, but I think the details need to be hammered at summit
17:34:58 <davidkranz> sdague: OK. So there is no point in others trying to run it then?
17:35:08 <davidkranz> sdague: For now.
17:35:18 <sdague> so you could turn on testr today, it's about 2 days worth of patches, but it wouldn't be a win until we teach it how to be smart about our resources
17:35:25 <sdague> yeh, not really at the moment
17:35:41 <davidkranz> sdague: OK, good to know.
17:35:55 <sdague> that's part of my desire to clean up and consolidate the resource tracking inside tempest prior to summit
17:36:04 <sdague> as it will make having this conversation a lot simpler
17:36:16 <afazekas> sdague: we need limit the credential isolation to per process, instead of per class
17:36:33 <afazekas> we need a to add a TestSuite
17:36:38 <sdague> as there will be a small number of choke points in tempest where all our spawned resources come from
17:36:44 <sdague> afazekas: right, true
17:36:51 <afazekas> and after thet we can share resources between xml / json pairs
17:37:14 <sdague> afazekas: so who at redhat do we need to poke to get you to portland :)
17:37:20 <sdague> would love to have you there in these conversations
17:37:38 <sdague> I honestly expect we'll be able to sit and hack some code with lifeless while we're there
17:38:19 <afazekas> sdague: I did not asked yet, but if get negative answer I will tell you :)  (And I even don't have a passport)
17:39:23 <sdague> afazekas: please ask :)
17:40:18 <davidkranz> Any other topics?
17:40:25 <giulivo> guys, I'm a new arrival at tempest; I'd like to get some understanding of the status of the tests for cinder and swift and more in particular to ask if there is any relevant blueprint/bug I could start to look at; can you suggest any resource to start with?
17:40:54 <sdague> giulivo: the best thing to do is just start reading the relevant test trees
17:41:10 <sdague> giulivo: and come over to #openstack-qa, people can point you in the right direction over there
17:41:19 <sdague> davidkranz: no other topics from me
17:41:22 <davidkranz> sdague: What about areas identified in the coverage list?
17:41:32 <davidkranz> mtreinish:   ^^^
17:41:39 <sdague> davidkranz: mtreinish can speak to that... but we only have coverage on nova
17:41:48 <sdague> because you need an extension
17:42:18 <davidkranz> sdague: OK, but that info might point to new tests that could be added.
17:42:42 <davidkranz> New topics going once...
17:42:43 <mtreinish> davidkranz: yeah it's only in nova right now. I haven't had time to expand the list (https://etherpad.openstack.org/MissingTempestTests) in a while
17:43:20 <mtreinish> davidkranz: I'm writing up a quick howto on how I go through the reports (it's honestly pretty straightforward)
17:43:33 <davidkranz> mtreinish: OK, great.
17:43:56 <donaldngo_hp> mtreinish: how are generating this list of the missing tests?
17:44:04 <davidkranz> New topics going twice...
17:44:16 <mlavalle> davidkranz: Can I get a review for https://review.openstack.org/#/c/22927/?
17:45:11 <mtreinish> donaldngo_hp: I go through the coverage reports and compare it to the tempest code. I'm writing a howto I'll send a link to the ml and post on openstack-qa when I'm done.
17:45:11 <davidkranz> mlavalle: This should have a quantum domain expert review.
17:45:40 <davidkranz> mlavalle: Can you grab some one?
17:45:44 <donaldngo_hp> mtreinish: cool thanks
17:45:52 <mlavalle> davidkranz: sure, thanks
17:46:15 <davidkranz> mlavalle: I'll take a look after that.
17:46:25 <mlavalle> thanks
17:47:59 <SHREE-HP> any updates on the negative tests & fuzz logic develoment? the last we heard is rackspace was plannign to propose solution..
17:50:02 <davidkranz> SHREE-HP: Don't know. Perhaps you could query on the mailing list.
17:50:40 <sdague> ok, sounds like we're wrapping up?
17:52:38 <afazekas> bye
17:55:47 <sdague> davidha: ok #endmeeting?
17:56:16 <davidkranz> #endmeeting