16:01:28 <hogepodge> #startmeeting defcore
16:01:29 <openstack> Meeting started Wed May 18 16:01:28 2016 UTC and is due to finish in 60 minutes.  The chair is hogepodge. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:31 <slogan> jayahn: you can contact me at syd.logan@broadcom.com
16:01:33 <openstack> The meeting name has been set to 'defcore'
16:01:41 <hogepodge> #topic rollcall
16:01:43 <hogepodge> o/
16:01:43 <dwalleck> o/
16:02:06 <Rockyg> o/
16:02:06 <gema> o/
16:02:08 <catherineD> o/
16:02:13 <luzC> o/
16:02:14 <AlanClark> o/ (wanting to listen in)
16:02:21 <hogepodge> Note that markvoelker and eglute are both out again this week, but will be returning soon.
16:02:39 <hogepodge> #link agenda https://etherpad.openstack.org/p/DefCoreLunar.3
16:03:05 <hogepodge> #topic test spec
16:03:17 <hogepodge> gema has written up a test spec proposal
16:03:24 <gema> o/
16:03:25 <gema> yep
16:03:28 <hogepodge> #link https://review.openstack.org/#/c/317531/
16:03:47 <gema> I think I captured the overall consensus of the comments, but please review
16:03:52 <gema> we may be missing something
16:04:01 <gema> (it feels short)
16:04:04 <dwalleck> will do, I'll have a look today
16:04:08 <gema> dwalleck: thanks
16:04:21 <Rockyg> me too.
16:04:25 <gema> Rockyg: thanks
16:04:38 <gema> hogepodge: I don't have anything else to discuss
16:04:49 <Rockyg> also, we can add more details later.  this can be a living doc.
16:05:02 <VanL> I didn't see anything about atomicity
16:05:11 <gema> VanL: dwalleck requested to rephrase
16:05:13 <hogepodge> Anyone else have any comments on this for the meeting? Otherwise, leave reviews. I expect that we'll need to iterate on it.
16:05:13 <gema> and I did
16:05:25 <catherineD> should commends be added to the review patch and not  the etherpad from now on?
16:05:32 <gema> catherineD: yes, please
16:05:35 <catherineD> commends --> comments
16:05:35 <hogepodge> catherineD: yeah, comments should be in the review
16:05:54 <dwalleck> I think "targetted to test and validate one capability" was the better translation of atomic
16:06:09 <gema> dwalleck: I think so to, so went with it on the final version
16:06:14 <gema> too*
16:06:15 <hogepodge> VanL: the work 'atom' has caused heartburn, but we can probably find a way to work it back in
16:06:58 <VanL> I like the "targeted...." language, but "atomic" has an outside meaning that I like and I think is useful in this context
16:07:13 <gema> VanL: the problem is that it is overloaded
16:07:18 <gema> means different things to different people
16:07:23 <catherineD> I am not sure how DefCore would differentiate tests vs scenario tests --> to me they are all tests to test a certain capability
16:07:39 <hogepodge> gema: VanL: would it be useful to add a definition, either to the spec or to the defcore lexicon?
16:07:41 <gema> catherineD: we would when we choose the tests for the capabilities
16:07:58 <VanL> hogepodge: Yes, I think that would be helpful
16:08:32 <hogepodge> catherineD: If I understand correctly, we're not choosing scenario tests right now, only api tests. Scenario would be a future addition
16:08:35 <gema> hogepodge: probably, but I like the new wording better so I'd like to hear (in a gerrit comment) how VanL wants to change the wording to include atomicity again
16:08:40 <VanL> catherineD: I think that the way to address scenario vs functional tests is by separating out the fixture/teardown from the "test"
16:08:43 <catherineD> gema: how?
16:09:02 <gema> catherineD: by reading the tests and understading them (I thought that's what we are supposed to do)
16:09:08 <shamail> Hi everyone o/
16:09:10 <VanL> So a scenario test includes >1 capability/API in the "test" portion
16:09:16 <gema> shamail: o/
16:09:23 <dwalleck> scenario tests also have a very specific meaning in Tempest terms
16:09:41 <catherineD> dwalleck: + exactly ...
16:10:15 <dwalleck> There are CRUD tests that exist that are not in the scenario tests package, but one could argue are scenarios
16:10:15 <catherineD> so since this is a spec we should be clear on  our terminology
16:10:22 <woodster_> o/
16:10:39 <gema> catherineD: absolutely, so how do we make that part clearer
16:10:46 <Rockyg> So, a glossary is key/\.
16:11:01 <dwalleck> But I like the idea of handling those distinctions in the spec. It will save us future headaches  :-)
16:11:21 <VanL> I think it makes for a clear distinction
16:11:33 <VanL> An "atomic" test involves only 1 API
16:11:46 <catherineD> dwalleck: exactly...
16:11:46 <VanL> Any test with >1 API is a "scenario" test
16:12:03 <gema> VanL: but is it?
16:12:17 <Rockyg> any test that *validates* more than one api...
16:12:18 <gema> VanL: a scenario test is one that covers end to end functionality
16:12:20 <catherineD> VanL: in that case the spec would be written that a test can teat one API or more
16:12:51 <hogepodge> I still think it's an ideal that we can't necessarily reach. Any API test that checks creation or deletion will require multiple API calls, including listing.
16:12:52 <catherineD> get rid of the headache of tests vs scenario tests
16:13:07 <VanL> gema, catherineD: You can string together N atomic items (per definitions above) to address CRUD, end-to-end, etc.
16:13:40 <catherineD> hogepodge: foundamental issue here is do we allow testing of multiple APIs in a test
16:13:46 <VanL> hogepodge: That is why setup/teardown are programmatically excluded from the definition of "test"
16:13:50 <gema> can we put this conversation on gerrit so I can get a better idea on what I should be writting?
16:13:59 <gema> I have the impression we are going around in circles again
16:14:03 <VanL> And can be swapped as needed
16:14:04 <hogepodge> But I don't want tests that reach outside the intent of the test, say image creation relying on volume operations. That would be a clear violation of what we're trying to accomplish
16:14:13 <dwalleck> My original point was that the test itself (not including fixtures) should test one API
16:14:25 <gema> dwalleck: mine was that too
16:14:30 <dwalleck> Going by the tradition Arrange/Act/Assert model
16:14:53 <Rockyg> dwalleck, gema ++
16:15:16 <gema> hogepodge: that's a different point I think, I will capture it (but make sure it ends up in gerrit)
16:15:20 <dwalleck> So that the actual test function is doing one thing. I think we're all saying the same thing, but we may need to clarify setup vs actual test functions
16:15:24 <gema> hogepodge: it's important
16:16:03 <gema> are we at a point where, relying only in required capabilities, we can set up and tear down all the tests?
16:16:35 <catherineD> dwalleck: gema: so in the case of CRUD of compute ... we would have 4 tests 1) create server  2) list server  3) update server 4) remove ?  .. that is 4 tests to test CRUD capability
16:16:54 <gema> catherineD: yes
16:17:09 <gema> catherineD: ideally there would be some test trying to create a server that fails
16:17:15 <gema> some wrong update call
16:17:15 <gema> etc
16:17:21 <gema> to make it better coverage
16:17:41 <catherineD> the 4 tests are independance
16:17:47 <gema> yes
16:18:09 <dwalleck> catherineD: That is normally how I would do it, but at the summit I think we discussed making a special case for basic CRUD activity
16:18:32 <gema> dwalleck: and that would be the scenario tests?
16:18:36 <catherineD> then do we still need to explicitely spell out item 3 in the spec ..
16:19:07 <woodster_> catherineD: just making sure...so all four tests in your example would start with one or more created servers, then each tear them down again correct?
16:19:20 <gema> woodster_: ideally
16:19:53 <dwalleck> gema: Not necessarily. I know Neutron and I think some compute tests in the tempest.api package that do CRUD
16:19:59 <woodster_> gema: cool make sense. Tests might take longer that way of course, but isolation is essential
16:20:32 <gema> dwalleck: I still don't know why we are mixing CRUD apis with tests
16:20:47 <hogepodge> we have some tests that pair up capabilities, particularly create delete https://github.com/openstack/defcore/blob/1acbbb20060a1484f9a798c276c0109e1229f79c/2015.07.json#L207
16:21:11 <gema> dwalleck: to me a good scenario test would be one that crosses boundaries between APIs
16:21:12 <hogepodge> (I just grabbed that as a random example based on name)
16:21:16 <dwalleck> hogepodge: yes, that :-)
16:21:19 <catherineD> hogepodge: so that test violate our spec?
16:21:25 <gema> dwalleck: set up VM, assign floating ip, make sure you can connect, etc
16:22:04 <catherineD> since it is one test that tests more than one API
16:22:06 <hogepodge> qa might argue that pairing that doesn't significantly change the scope of the testing and improves gate performance (as a straw man argument)
16:22:09 <dwalleck> gema: I agree, but we're mixing terms here. Tempest has a specific definition of a scenario test. I'm wanting to make sure the spec is clear
16:22:27 <gema> dwalleck: ok, you want me to use the same terminology as tempest then?
16:22:35 <hogepodge> so it would violate the word of the spec, but not the spirit of what we're trying to do imo
16:22:38 <dwalleck> If we're using the general concept of a scenario test or a Tempest scenario test
16:22:39 <gema> or are we trying to make things better in the tempest arena as well?
16:23:07 <gema> hogepodge: agreed, that test verifies two different capabilities
16:23:25 <Rockyg> gema, would love that, but that's heavy lifting ;-)
16:23:35 <VanL> dwalleck: We should define what we mean for *our* purposes, then we can explain how that relates to tempest as appropriate
16:23:38 <gema> Rockyg: I love crossfit :P
16:23:45 <catherineD> hogepodge: gema: so once our spec merged ... would DefCore remove all tests that violate the test spec?
16:24:04 <dwalleck> gema: Not necessarily. I think the Tempest team doesn't necessarily like their scenario tests. I just want the spec to be clear so people don't get confused. Adding "scenario" that to a lexicon for the doc would help
16:24:06 <gema> catherineD: we can fix them
16:24:17 <gema> catherineD: once we agree on what a good test looks like
16:24:21 <hogepodge> catherineD: I don't know. I think it would present a case for flagging them, but I would wait until a vendor requested it
16:24:23 <catherineD> gema: ok
16:24:34 <gema> catherineD: which I don't think we are very close to yet :)
16:25:11 <gema> dwalleck: ok, will try to make that clearer
16:25:16 <catherineD> gema: that is what I would like to understand ... once the spec is in ... we need to abide to the spec ..
16:25:31 <dwalleck> gema: No worries!
16:25:47 <gema> catherineD: sure, we can either say from now on all the tests we add need to abide by the rules or we can say, we have all these tests taht don't meet the standard and need fixing
16:25:55 <gema> catherineD: either way works, I think
16:26:12 <gema> catherineD: eventually we will get there
16:26:34 <catherineD> back to the spec ... if we allow item 3 in the patch ... we would accept https://github.com/openstack/defcore/blob/1acbbb20060a1484f9a798c276c0109e1229f79c/2015.07.json#L207 by calssifying it as a "scenario" test?
16:26:37 <hogepodge> gema: I don't want to throw out tests that are valuable on technicalities.
16:26:46 <gema> hogepodge: me neither
16:26:47 <hogepodge> gema: and artificially weaken the standard
16:26:49 <Rockyg> gema,  ++
16:26:55 <gema> hogepodge: I want to fix them
16:26:57 <gema> if they need fixing
16:27:08 <gema> hogepodge: and if we feel they don't need fixing, we've done something wrong on the spec
16:27:35 <Rockyg> Fix or replace
16:27:38 <gema> catherineD: all those examples are useful, can you add them to gerrit
16:27:39 <dwalleck> gema: I think we will need feedback from the QA team on the fixing bits. I'd like to make sure we're in alignment there
16:27:44 <gema> so I can think them through?
16:27:52 <gema> dwalleck: yep
16:28:07 <VanL> We need to understand where we want to end up in order to understand what is broken, the way in which it is broken, and the steps needed to fix
16:28:18 <catherineD> My concern is with item 1, 2 and 3 included in the spec .... we could be business as usual because we could classify the current tests as a "atomic" test or "scenario" test
16:28:19 <hogepodge> It's why I feel that testing of capabilities in the same grouping (CRUD) can coexist (even if on a formal basis we feel a bit icky about it)
16:28:25 <catherineD> gema: will do
16:28:30 <dwalleck> It might be good to have oomichi and others review the spec as well
16:28:38 <hogepodge> dwalleck: +1
16:28:39 <shamail> hogepodge: +1
16:28:45 <gema> catherineD: and also what you think is wrong with those tests, so that I get a better understanding
16:28:58 <gema> catherineD: you got the experience with people running into trouble with them
16:29:12 <catherineD> gema: yup
16:29:15 <VanL> hogepodge: -1, I think that takes us places that it would be better not to go. I don't like ickiness, as that translates to "brokenness" for me
16:29:17 <gema> catherineD: +1 thanks
16:29:46 <gema> VanL: what do you mean?
16:30:15 <VanL> Quoting hogepodge: "It's why I feel that testing of capabilities in the same grouping (CRUD) can coexist (even if on a formal basis we feel a bit icky about it)"
16:30:32 <VanL> Emphasizing my aversion to the ickiness
16:30:35 <gema> ahh, ok
16:30:37 <gema> gotcha
16:30:53 <Rockyg> So, if we define scenario test well, other apis in a tes are there for the specific purpose of the scenario.
16:31:25 <hogepodge> Rockyg: and maybe that's the answer, it goes away if we somehow define crud as a scenario
16:31:34 <VanL> I would be ok with a CRUD test that was a scenario test built out of otherwise-tested atomic bits
16:31:36 <hogepodge> and admit crud scenarios
16:31:53 <gema> and forget about all other scenario tests?
16:31:57 <VanL> CRUD as scenario I'm ok with.
16:32:24 <Rockyg> no, we will need a number of scenario tests besides crud
16:32:30 <gema> ok
16:32:56 <hogepodge> gema: no, not forget all others, but I'm trying to see how to focus efforts and evaluate the current test list
16:33:25 <gema> hogepodge: trying to shoehorn the current tests into the spec, i.e. weaken the spec , is the wrong way around imo
16:33:41 <Rockyg> like, add disk space to a vm is actually a scenario or part of one.
16:33:43 <gema> hogepodge: we can be flexible on how to deal with the existing tests that don't meet the spec either
16:33:51 <hogepodge> gema: we're enforcing a current standard, and I don't want to throw that work away
16:33:52 <catherineD> One thing we want to be clear is that there is not thing wrong on the Tempest tests ... It is just that some of the Tempest tests DefCore used is not written for interop testing ..
16:34:00 <gema> hogepodge: we don't need to
16:34:14 <hogepodge> gema: if the spec is too tightly written it can be used as a tool to argue for that
16:34:36 <dwalleck> catherineD: I wouldn't even say they're wrong for interop. They just may not target one capability
16:34:42 <gema> hogepodge: ok, I see your point
16:34:44 <Rockyg> maybe we should write a grandfather clause into the spec.
16:35:02 <Rockyg> use wha we have until we have conforming tests
16:35:18 <VanL> No, no grandfather support
16:35:22 <dwalleck> But I thought part of that was handled when we defined some individual capabilities as crud
16:35:35 <hogepodge> This is good discussion, but I'm going to time cap to 40 past the hour so we can reach some of the other topics.
16:35:38 <VanL> dwalleck: Noooo!
16:35:49 <gema> hogepodge: +1
16:36:18 <dwalleck> Then I misunderstood and need more caffeine :-)
16:36:26 <hogepodge> dwalleck: that capability refactoring hasn't been approved
16:36:28 <gema> I wil wait for catherineD's examples and comments on gerrit before proposing a change
16:36:38 <hogepodge> (indeed, not even defined yet...)
16:36:43 <dwalleck> hogepodge: gotcha
16:36:47 <Rockyg> if we don't have grandfather, then defcore needs a waiver process to allow tests, cycle by cycle until we have a conforming test to replace it.
16:37:04 <VanL> Rockyg: Ok
16:37:17 <VanL> But I want to push us towards rationalized tests
16:37:31 <gema> VanL: stop throwing new words in, what are rationalized tests
16:37:32 <dwalleck> Rockyg: That's why we need to get the QA team involved. They would need to agree with our assessment
16:37:35 <catherineD> Rockyg: +1 ... if spec is merged we need to abide to it ...
16:38:01 <Rockyg> gema, ++ ;-)
16:38:02 <catherineD> maybe with a time target too ..else not much useful with the spec
16:38:06 <gema> hogepodge: who can I add from QA to the review?
16:38:06 <hogepodge> gema: tests that don't bleed way out of their intended scope I think?
16:38:08 <woodster_> I'd be good to have canonical reference examples of in-spec tests at some point
16:38:17 <VanL> gema: Not trying to define a new term, just using standard meaning
16:38:31 <gema> VanL: not standard where I live :)
16:38:32 <Rockyg> catherineD, that's why cycle by cycle.  Needs new waiver each cycle
16:38:46 <gema> VanL: I think I got it from hogepodge's explanation
16:38:51 <hogepodge> gema: oomichi and mtreinish
16:38:56 <gema> hogepodge: ack, will do
16:39:18 <hogepodge> Ok, wrapping up the discussion. Thanks everyone, this is really valuable.
16:39:28 <Rockyg> gema, maybe we need to have a suggested readking list for non-testers
16:39:30 <hogepodge> Please, comment further on the review in gerrit
16:39:45 <catherineD> hogepodge: +1 I think so ... thx gema: for  submitting the patch
16:39:51 <gema> yw!
16:40:16 <hogepodge> #topic Schema (1.4)
16:40:32 <hogepodge> These are ready for review #link https://review.openstack.org/#/c/311265/
16:40:40 <hogepodge> #link https://review.openstack.org/#/c/317800/
16:41:06 <hogepodge> The first is formalization of the 1.4 schema, along with minor guideline changes to match the formalization
16:41:15 <hogepodge> Along with linting and validation
16:41:43 <hogepodge> The second makes linting and validation voting jobs in the gate to help make sure we don't accidentally break the guidelines in updates
16:42:01 <hogepodge> Please, check my work. :-D
16:42:16 <hogepodge> Any comments or questions?
16:42:20 <catherineD> hogepodge: I know it is  format change for rst to json ..   I think it would be easier for RefStack if we make it versio n1.5 instead of remaining on 1.4
16:42:45 <hogepodge> catherineD: ok, I can do that
16:42:54 <catherineD> do we remove the rst version from now on?
16:43:16 <hogepodge> catherineD: I think we should retain them for, but don't feel strongly about it
16:43:18 <catherineD> hogepodge: thx that would help
16:43:22 <Rockyg> deprecation?  Keep it for a cycle?
16:43:52 <hogepodge> I'm mainly concerned with updates (particularly when we have merge conflicts) that break the guidelines
16:43:54 <Rockyg> or two?  follow dev standards on deprecation
16:44:05 <Rockyg> hogepodge,
16:44:12 <Rockyg> ++
16:44:35 <hogepodge> Yeah, we can remove them beginning 2017 from the repo too
16:44:40 <catherineD> hogepodge: definitely retain the existing rst ... not sure we need for the future
16:45:02 <hogepodge> no rst for the future. the json schema is formal and gateable
16:45:06 <shamail> hogepodge: I think you might need to switch line 5402 to after 5404 in this one? https://review.openstack.org/#/c/317800/4/zuul/layout.yaml
16:45:41 <catherineD> since the old rst  files will be used to render the guidlelines prior to 2015.07 ...
16:45:57 <hogepodge> catherineD: ok
16:46:02 <shamail> sorry, meant move 5402 and 5407 to the top of their sections.. Jenkins failed on that patch.
16:47:02 <catherineD> hogepodge:  thanks .. I will add those comments on the patch  ...
16:47:04 <hogepodge> shamail: I'll take a look
16:47:18 <hogepodge> any other comments?
16:47:20 <shamail> Added comments to the review.
16:47:40 <hogepodge> #topic Schema (2.0)
16:47:59 <hogepodge> #link https://review.openstack.org/#/c/310621/
16:48:32 <hogepodge> This one needs a lot more work. Cover intent, and also needs to be formalized along the lines of 1.4/1.5
16:48:55 <hogepodge> If I understand correctly, it captures higher level metadata about the guideline itself.
16:49:23 <hogepodge> catherineD: do you know andrey's irc nick so he can comment?
16:49:39 <catherineD> and also update it to JSON
16:50:14 <hogepodge> If everyone can take a look and leave reviews, that would be helpful for guiding future discussion on the direction to take the guidelines in
16:51:02 <hogepodge> Ok, moving on.
16:51:06 <catherineD> Andrey's IRC andrey-mp
16:51:15 <catherineD> will let andrey-mp: know
16:51:19 <hogepodge> #topic Interop Issues Report
16:51:34 <hogepodge> just a reminder, leave comments on this #link https://etherpad.openstack.org/p/DefCoreInteropReport
16:51:56 <hogepodge> Mark and Egle are preparing this as a report to the board and community. We discussed it a lot in Austin
16:52:12 <hogepodge> Comments?
16:52:33 <hogepodge> #topic Work in Progress
16:53:16 <hogepodge> #link markvoelker has added some tests for neutron that we'll want to use for api discovery https://review.openstack.org/#/c/311747/
16:53:56 <hogepodge> Please review
16:54:26 <hogepodge> I still need to work on refactoring (where possible) the advisory tests that require admin
16:54:34 <hogepodge> And also do doc generation
16:54:50 <hogepodge> Does anyone have links to work they want reviewed?
16:55:35 <hogepodge> Ok, open discussion for the last few minutes.
16:55:41 <hogepodge> #topic Open Discussion
16:55:43 <catherineD> hogepodge: tempest.api.image.v2.test_images_member and tempest.api.image.v2.test_images_member_negative in advisory still need 2 users
16:55:58 <hogepodge> catherineD: I'll take a look at those
16:55:59 <catherineD> hogepodge: do we flag them ?
16:56:06 <catherineD> hogepodge: ok
16:56:27 <hogepodge> the name "member" suggests it'll require two users, so likely need to be removed (if they're advisory, flag if required)
16:56:54 <catherineD> I think we can merge https://review.openstack.org/#/c/300608/
16:57:16 <dwalleck> That feature may be tricky. I think that's the tenant visibility flag that lets you share images
16:57:20 <hogepodge> catherineD: can any be rewritten? that's something I wanted to look in to
16:57:51 <Rockyg> hogepodge, ++
16:57:54 <hogepodge> (action item for me)
16:58:17 <catherineD> hogepodge: it takes time to update tempest tests ... meanwhile user still waisting their time to debug tests that we know won't work without admin
16:58:32 <catherineD> by flagging them  will save user's debug time
16:58:48 <hogepodge> catherineD: since they're advisory, I'd like just a bit more time to see if we can make that list smaller
16:59:09 <hogepodge> We're out of time.
16:59:21 <hogepodge> Thanks everyone for the productive meeting. We can continue in #defcore
16:59:56 <hogepodge> #endmeeting