16:01:14 <eglute> #startmeeting defcore
16:01:15 <openstack> Meeting started Wed Apr 20 16:01:14 2016 UTC and is due to finish in 60 minutes.  The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:18 <openstack> The meeting name has been set to 'defcore'
16:01:20 <openstack> markvoelker: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
16:01:24 <eglute> #chair markvoelker
16:01:25 <openstack> Current chairs: eglute markvoelker
16:01:43 <gema> o/  (half attending only, also in another meeting)
16:01:45 <hogepodge> o/
16:02:02 <eglute> hello everyone! here is the agenda: #link https://etherpad.openstack.org/p/DefCoreRing.20
16:02:05 <docaedo> o/
16:02:06 <catherineD> o/
16:02:30 <hogepodge> meetbot seems to be having issues
16:02:40 <eglute> hogepodge how so? seems to work?
16:02:51 <markvoelker> #topic Finalize next.json as a solid draft to present during the summit
16:03:09 <markvoelker> hogepodge: I think that may have been Egle and I typing startmeeting at the same time. =)
16:03:15 <hogepodge> ah ha
16:03:33 <markvoelker> So, we have a couple of patches left to land
16:03:41 <markvoelker> #link https://review.openstack.org/#/c/290592/
16:04:00 <markvoelker> hogepodge, did you want to talk about the role issue mentioned in the etherpad for this one?
16:04:22 <hogepodge> There is a SwiftOperator role that is required for the tests, and for the ACL capabilities.
16:05:25 <hogepodge> We wanted some clarification of if this violated the "no admin" rule. Turns out that the SwiftOperator role is meant more as an organizational "admin" role, rather than a system "admin" role. It doesn't grant privileges beyond managing ACLs
16:06:04 <hogepodge> Which to me, makes it non-admin in the context of defcore. We would expect a user in a public cloud being granted this role to fully take advantage of Swift's features
16:06:33 <hogepodge> I checked on this with notmyname yesterday during discussion of the update
16:06:36 <markvoelker> I'd agree (hence my +2 on the patch).  I researched a couple of public clouds just to be sure and it's expectd.
16:07:24 <markvoelker> Anything other discussion on this one?
16:07:30 <catherineD> yes
16:07:51 <catherineD> are the tests added from tempest plugin?
16:08:02 <hogepodge> catherineD: no, those are from tempest
16:08:11 <catherineD> hogepodge: thx
16:08:51 <eglute> sounds good to me. glad notmyname had a chance to review and provide feedback
16:08:57 <markvoelker> Ok, absent any last minute objections I'll merge this today.
16:09:02 <eglute> thank you markvoelker
16:09:13 <markvoelker> #link https://review.openstack.org/#/c/290271/ cinder scoring
16:09:30 <hogepodge> I rebased and fixed the patch this morning
16:09:36 <eglute> thanks hogepodge
16:09:47 <markvoelker> Ok, cool.  I saw the new patchset go up this morning but haven't had a chance to look at it yet
16:10:01 <hogepodge> There will be a merge conflict. I'll resolve once the first one lands.
16:10:14 <eglute> it looks good to me
16:11:07 <markvoelker> Not much dissent on this one so far I think, so I'll plan on landing this too.
16:11:54 <markvoelker> Ok, I think that's all for capabilities that we have remaining
16:12:21 <markvoelker> One other review to point out for this Guideline: https://review.openstack.org/#/c/306229/
16:12:39 <markvoelker> #link https://review.openstack.org/#/c/306229/ Add cutoff_score field to next.json
16:13:04 <markvoelker> We added a cutoff_score field in the last rev of the Guideline schema, but haven't actually added that field to next.json yet
16:13:14 <markvoelker> This patch does that and also updates the scoring script to make use of it
16:13:19 <hogepodge> LGTM
16:13:37 <markvoelker> It'll need a little rebase, but I'll let the other scoring patches land before doing that
16:13:56 <eglute> markvoelker i think it also needs to be added to 2016.01
16:14:04 <markvoelker> eglute: can do
16:14:06 <eglute> since it also uses 1.4 schema
16:14:25 <markvoelker> Ok, moving on?
16:14:25 <eglute> thank you markvoelker
16:14:28 <eglute> yes
16:14:33 <markvoelker> #topic Closing the coverage gap
16:14:41 <catherineD> Just content update in 2016.01 riht? no schema update?
16:14:51 <markvoelker> catherineD: correct
16:14:55 <markvoelker> #link https://review.openstack.org/#/c/306229/
16:15:19 <markvoelker> As we discussed last week, there's currently no Guideline that covers Mitaka and we have someone wanting to get a logo agreement for a Mitaka-based product
16:15:21 <catherineD> RefStack uses schema to render Guideline so we are sensitive to schema change
16:15:29 <eglute> catherineD the same change would go on 2016.01.json https://review.openstack.org/#/c/306229/2/next.json
16:15:48 <eglute> we just missed that field in .json files
16:16:06 <eglute> here is the schema: #link https://github.com/openstack/defcore/blob/master/doc/source/schema/1.4.rst
16:16:33 <catherineD> eglute: RefStack should be OK thax
16:16:53 <eglute> glad to hear it!
16:17:00 <markvoelker> This patch changes 2016.01 to allow it to cover Mitaka.  The Board will need to approve that change.
16:17:46 <eglute> i think we have a consensus on the fomat... any objections?
16:17:52 <markvoelker> It looks like we have rough consensus on the change from our end, so if folks could please add their final +1 or suggest any further changes they want, we'll ask the Board for approval on Sunday.
16:18:07 <eglute> #link https://review.openstack.org/#/c/306614/
16:18:18 <mfranc213> davidsha: pong
16:18:27 <mfranc213> hello david!
16:19:05 <hogepodge> mfranc213: we're in a meeting. please use another channel
16:19:21 <markvoelker> hmmm, that link is wrong, one moment
16:19:45 <markvoelker> #link https://review.openstack.org/#/c/306614/ Close coverage gap
16:19:45 <eglute> which link?
16:21:08 <markvoelker> Egle: the one in the etherpad for this topic.  It was pointing to the cutoff_score review, not the close coverage gap review.
16:21:10 <markvoelker> Just fixed it
16:21:18 <eglute> thank you
16:21:34 <markvoelker> Anything further folks want to discuss on this one?
16:21:40 <eglute> looks good to me...
16:22:35 <markvoelker> Ok then, let's move on.
16:22:48 <hogepodge> just clarification on if we use future release at time of guideline approval or at release time, and codify adding that without needing to go back to the board
16:22:48 <markvoelker> #topic DefCore Spec
16:23:27 <markvoelker> hogepodge: my plan was to do it at approval time so we don't have to keep bugging the Board (hence the change both to 2016.01 and next.json)
16:23:29 <eglute> hogepodge i think we will use future at guideline approval
16:23:33 <hogepodge> thanks
16:23:43 <hogepodge> carry on  :-D
16:23:58 <markvoelker> #link http://lists.openstack.org/pipermail/defcore-committee/2016-April/001085.html ML discussion
16:24:09 <markvoelker> #link https://etherpad.openstack.org/p/DefCoreSpec-draft Spec draft
16:24:53 <gema> will go back to that soon, please add any ideas/comments to it :)
16:24:53 <markvoelker> Several folks have been adding notes to the pad, which is a good start.  I think we're probably in a good position to take some time at the Summit during the working session to see if we can come up with a firm draft and put it up in Gerrit.
16:25:21 <markvoelker> So unless there are objections, I'll plan on adding it to the agenda.
16:25:25 <eglute> +1
16:25:28 <gema> markvoelker: ok, I won't be in the summit so someone else will have to drive that one :D
16:25:45 <hogepodge> gema: I can help out with that
16:25:52 <gema> hogepodge: +1
16:25:57 <gema> thanks
16:26:04 <markvoelker> gema: Bummer, we will miss you.  But yes, we'll get it covered. =)
16:26:14 <hogepodge> gema: thank you, it's a great start
16:26:19 <gema> markvoelker: cool :D I will add all I can think of beforehand then
16:26:28 <markvoelker> gema: yes please
16:26:36 <markvoelker> (and that goes for the rest of you lot as well =p)
16:26:50 <luzC> ok
16:26:59 <markvoelker> #action please make any additional comments on the etherpad ahead of the Summit so we can have a productive work session
16:27:32 <markvoelker> Anything folks want to discuss on this right now?  We have a little available time today I think.
16:28:34 <markvoelker> Ok, hearing none...
16:28:41 <luzC> quick question any news about heat tempest plugin priority?
16:28:43 <eglute> i think it is a good start
16:29:13 <catherineD> there will be a heat work session https://www.openstack.org/summit/austin-2016/summit-schedule/events/9247
16:29:28 <eglute> would it be worthwhile to discuss atomicity?
16:29:36 <eglute> right now there is no definition in the spec
16:29:41 <catherineD> please note the session is now at 3:30 - 4:10 on Wednesday
16:30:10 <eglute> sorry, there is some definition, getting ahead of myself
16:30:13 <VanL> There is a definition of sorts
16:30:22 <eglute> catherineD thank you... did not see that.
16:30:34 <eglute> i will be sending calendar invite to the working group session
16:30:35 <VanL> A functional test is atomic if, after excluding fixture setup and cleanup, exactly one aspect of an API is exercised.
16:31:10 <VanL> And there is some discussion in the document about "should" vs "must"
16:31:28 <catherineD> eglute: please do .. They reschedule the time so please update you calendar ... I put the new link on the DefCore etherpad
16:32:24 <luzC> catherineD thanks, I'll add it to my calendar too
16:32:52 <eglute> VanL i like the "must" part for atomicity
16:34:02 <eglute> hogepodge what do you think so far of the atomicity?
16:34:22 <markvoelker> I'll just note that when considering applying something as a "must" it would probably be good to look at our existing required tests and see how many of them would meet the bar we're creating
16:34:38 <hogepodge> It's a complicated issue. I don't want to see a huge set of tests invalidated because of the requirement.
16:34:48 <markvoelker> That's not to say we should or shouldn't use must, just that it would be informative.
16:35:04 <hogepodge> We also have capabilities that are group as CRUD, so I could see tests that do CRUD
16:35:18 <eglute> we could say that going forward we will aim for tests to meet the specs. and start working on non-atomic first
16:35:37 <eglute> hogepodge i think those would be scenario tests, no?
16:35:38 <hogepodge> By starting at should, we get a path to refining the guideline without massive disruption.
16:35:54 <VanL> I don't think that a "CRUD" capability starts by doing all of CRUD
16:36:15 <hogepodge> Oh yes, and scenario tests, most definitely not atomic.
16:36:19 <VanL> It means that we have a "create" test, a "read" test, etc
16:36:56 <VanL> hogepodge: I don't think that anyone is disputing non-atomic scenario tests
16:36:59 <hogepodge> Plus theres the issue of the fixtures, which seems to be addressed by the proposed definition.
16:38:27 <hogepodge> I am opposed to any definition that a) is an attempt to flag and remove a large number of existing tests without attempting to "fix" or replace them, and b) is used as a tool to "justify" a fork of tempest as our primary test suite.
16:38:49 <hogepodge> so starting with should gives us a criteria to work on with the qa team
16:39:24 <VanL> A refactoring of any functional test to be one or more atomic tests would improve the test suite for all purposes, not just for defcore
16:40:08 <eglute> hogepodge i dont think the spec would be a blank permission to do either. i would propose that we grandfather existing tests and then work to improve them.
16:40:16 <markvoelker> hogepodge: Fair.  We also have the option of not enforcing the spec immediately, but rather using it as a guidepost for discussions on refactoring.
16:40:17 <eglute> but we do need to improve the tests
16:41:02 <markvoelker> Ok, good things to note in the pad and bring up for discussion at the Summit
16:41:05 <hogepodge> I'm not saying there's no room for improvement.
16:41:32 <markvoelker> 20 minutes left, so perhaps we should move on to the last couple of topics for today?
16:41:34 <catherineD> So atomic test would include fixture and setup ... does the test fail if the fixture or setup/cleanup fail?
16:41:53 <eglute> catherineD good question, it should be addressed in the spec
16:41:56 <VanL> Test only fails if the test fails
16:42:13 <hogepodge> A test should only pass if it passes.
16:42:27 <catherineD> but if cleanup fail ... test may not be in a good condititon for futher testing
16:42:28 <hogepodge> we treat skips as failures
16:42:46 <VanL> Fixtures would be swappable. The test code itself is inviolate
16:43:15 <catherineD> we havve a real life test case right now in the cinder volume mounting area
16:43:30 <eglute> catherineD can you link to it?
16:44:25 <VanL> catherineD: Explain
16:44:25 <catherineD> The volume mount test passes but the cleanup by  unmounting the volume fails
16:44:50 <eglute> catherineD do we consider that a test failure?
16:45:07 <catherineD> eglute: https://bugs.launchpad.net/nova/+bug/1565859
16:45:08 <openstack> Launchpad bug 1565859 in OpenStack Compute (nova) "Can't detach SVC volume from an instance; guest detach device times out" [Undecided,Invalid]
16:45:26 <catherineD> so right  now the test count as fail... but should we?
16:45:42 <catherineD> Failure is by status provided by Tempest
16:45:52 <catherineD> so from tempest point of view it fails
16:46:02 <VanL> So I would characterize that as a "Pass" on the test, but it would point to a separate need for an "unmount" test and capability if we want to ensure that
16:46:10 <eglute> this is excellent point... looking at this example, test should not be considered as failed
16:46:13 <catherineD> but from DefCore atomic test feature testing point of view it should pass
16:46:43 <eglute> right.. i think unmount should be a separate test
16:47:08 <hogepodge> the failure still pointed out a problem with the cloud, a buggy and old driver, and the problem was fixed when the downstream error was fixed, so the test was a success in identifying a real problem
16:47:35 <VanL> So it may be a good QA test, but it is a bad defcore test
16:47:36 <hogepodge> so yes, split the tests to catch it, but it doesn't make the test immediately invalid. It served a purpose
16:47:40 <eglute> bugs will happen in tests and in frameworks, so we need to see what would be easiest for someone trying to certify
16:47:52 <catherineD> so from technical mechanical testing point of view  .. it is hard to define pass/fail
16:47:54 <hogepodge> no, it's not, because it would have caught an interoperability problem
16:48:24 <eglute> hogepodge we cannot expect everyone trying to certify try to fix all the issues before they certify
16:48:26 <hogepodge> but yes, it can be improved.
16:48:48 <catherineD> hogepodge: so in that case the test should count as fail?
16:48:55 <VanL> A bug is not necessarily an interop problem, it is a bug
16:49:04 <eglute> i agree with VanL
16:49:07 <hogepodge> catherineD: yes, because downstream had a problem
16:49:11 <VanL> QA is not the same as defcore
16:49:17 <eglute> +1
16:49:25 <catherineD> hogepodge: but the capability we are testing is mount ...
16:49:28 <hogepodge> it's a problem if features don't work for end users
16:49:35 <catherineD> if the testing is unmount then I agree
16:49:43 <VanL> hogepodge: And the mount worked for the end user.
16:49:44 <catherineD> this is the point about test atomic
16:49:45 <hogepodge> it's unfair to end users to flag away implementation bugs and call yourself interoperable
16:50:05 <eglute> it is unfair to the everyone to lump everything in one thing
16:50:11 <VanL> No, this is about improving the test suite and making it say what we mean
16:50:18 <catherineD> if we count that test case as fail ... we vilolate what we just define as "atomic"
16:50:41 <eglute> DefCore should not test the tests or the testing framework
16:50:42 <hogepodge> if you can't manage a fixture it's still a problem
16:50:51 <eglute> DefCore should test interoperability
16:50:52 <VanL> I think what happens is you identify the need for an unmount test
16:50:57 <hogepodge> especially if the capability falls under crud
16:51:21 <VanL> Then each of "CRUD" needs to be tested seperately
16:51:23 <hogepodge> yes, the correct action is to split out the tests and reassign as necessary
16:51:27 <eglute> hogepodge i think that is a different case. crud and scenario tests are separate from atomic tests.
16:51:28 <hogepodge> but it's still a failure
16:51:41 <VanL> A fixture failure is not a failure
16:51:51 <markvoelker> Ok gang, 10 minutes. =)  I think I hear you all saying "we should find ways to improve tests" and that we have a lot of discussing to do about how to get there, which we can hopefully have some good discussion about in Austin.  Fair?
16:51:52 <hogepodge> and in that case, the failure was an implementation bug
16:51:53 <VanL> If the fixture can be swapped and the test succeeds
16:51:54 <catherineD> so we are saying taht fixture/setup/cleanup failure counts as test failure?
16:52:27 <VanL> catherineD: fixture/setup/cleanup *should not* count as test failure
16:52:28 <eglute> catherineD i think that is what hogepodge is saying. I disagree with that
16:52:42 <hogepodge> and in my experience over the last year, vendors flag tests but then don't fix the tests they flag
16:52:48 <catherineD> eglute: +1
16:52:57 <eglute> hogepodge that is a separate issue we should address
16:53:03 <hogepodge> VanL: if the test isn't run to completion it's counted as a fail
16:53:33 <markvoelker> Gotta call time folks.  We still have two more topics today.
16:53:39 <markvoelker> #topic Austin summit
16:53:44 <catherineD> this is an issue of how we define "atomic" test
16:54:00 <markvoelker> #link https://etherpad.openstack.org/p/refstack-newton-summit Summit etherpad
16:54:26 <markvoelker> Please make sure your topics get added to that pad.  I'll work on finalizing the working session agendas by the end of the week.
16:54:34 <eglute> that is refstack link
16:54:39 <markvoelker> whoops
16:54:50 <markvoelker> #link https://etherpad.openstack.org/p/DefCoreRing.AustinSummit2016
16:54:57 <markvoelker> (thanks elgute)
16:55:30 <markvoelker> There are also some other sessions that are likely to interest DefCore folks listed in today's meeting etherpad
16:55:31 <eglute> (but also add topics to refstack, on refstack etherpad)
16:55:43 <catherineD> eglute: thanks :-)
16:55:55 <markvoelker> If you have others, feel free to stick them on there
16:56:05 <hogepodge> I've arranged a joint DefCore/QA working session, following the DefCore working session.
16:56:14 <eglute> thanks hogepodge
16:56:40 <eglute> please be sure QA understands what defcore is
16:57:02 <markvoelker> Ok, we already hit the cutoff score topic, so I'll jump straight o our last topic for the day:
16:57:11 <markvoelker> #topic DefCore in Launchpad
16:57:25 <markvoelker> hogepodge: take it away
16:57:43 <hogepodge> completed the action item from last week, this was in response for the need to manage blueprints
16:58:12 <hogepodge> we didn't need any special support from infra or the TC, but we do have an administrators group for it on launchpad.
16:58:24 <hogepodge> I'll add egle to it, and anyone else that we feel is appropriate to add.
16:58:34 <eglute> thank you hogepodge
16:58:46 <hogepodge> (I don't have any issues with adding anyone who wants to be on it, but I started with co-chairs)
16:59:01 <markvoelker> hogepodge: that looks fine as a starter list.  We can discuss mechanics later.
16:59:12 <markvoelker> (as we're down to our last minute)
16:59:34 <markvoelker> Any final thoughts before we switch back to #openstack-defcore?
17:00:05 <markvoelker> #endmeeting