16:01:04 <markvoelker> #startmeeting defcore
16:01:05 <openstack> Meeting started Wed Apr  6 16:01:04 2016 UTC and is due to finish in 60 minutes.  The chair is markvoelker. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:09 <openstack> The meeting name has been set to 'defcore'
16:01:13 <eglute> o/
16:01:16 <markvoelker> #chair hogepodge
16:01:17 <gema_> o/
16:01:18 <openstack> Current chairs: hogepodge markvoelker
16:01:18 <catherineD> o/
16:01:22 <markvoelker> #chair eglute
16:01:22 <openstack> Current chairs: eglute hogepodge markvoelker
16:01:25 <Rockyg> o/
16:01:30 <hogepodge> o/
16:01:34 <dwalleck> o/ (in another meeting, but here!)
16:01:45 <markvoelker> #link https://etherpad.openstack.org/p/DefCoreRing.18 Today's Agenda
16:02:49 <markvoelker> Please take a quick look through the agenda and make sure we're not missing anything folks
16:03:05 <markvoelker> #topic Straw Man Proposal for DefCore Owning DefCore Tests in Tempest
16:03:18 <markvoelker> #link https://review.openstack.org/#/c/301879/ Strawman proposal
16:03:50 <markvoelker> hogepodge posted this yesterday so I'll let him speak to it. =)
16:04:21 <markvoelker> As a general note, you are all (as usual) encouraged to please provide your input in Gerrit.
16:04:41 <hogepodge> The blueprint has a couple of intents. The first is to solidify the relationship with the QA team by marking defcore tests, approved by the board, as important.
16:04:42 <markvoelker> hogepodge: around?
16:04:44 <markvoelker> doh
16:04:47 <markvoelker> yes you are =)
16:05:25 <hogepodge> We've had friction in the past about test names changing, functionality changing, and playing a bit of catch up and writing tooling to work around the changes.
16:06:37 <hogepodge> I'm not sure it's a part of the proposal that would pass, especially using a defcore named tags. This is in part because tempest doesn't necessarily want outside influence on the project like this, but they're not 100% opposed to it either. Particularly with regards to tags that indicate a property
16:06:53 <hogepodge> do defcore-required may be bad? interop may be good?
16:07:06 <hogepodge> It's starting a dialogue there, which I feel is important.
16:07:11 <markvoelker> ++
16:07:35 <hogepodge> Secondly it's encoding some of the requirements we talked about at the previous meeting for a spec that defines "interop" test
16:07:35 <dwalleck> hogepodge:: ++, dialog is good
16:08:10 <hogepodge> I tried to write down the minimal things we could reach consensus on, but is a work in progress. I would expect that an official spec would be part of the defcore repository.
16:08:46 <eglute> hogepodge i do not recall us having consensus on tagging
16:09:06 <hogepodge> Again, it's trying to encode a relationship, saying that we take responsibility for maintaining, along with qa, what an interop test looks like.
16:09:10 <dwalleck> hogepodge: Does it really encode though? It's a decorator, but it doesn't enforce anything directly
16:09:36 <hogepodge> dwalleck: it puts meaning in code, backed by a spec.
16:10:06 <dwalleck> But it does start a dialog. It does add a burden to Tempest reviewers to understand all the things that interop means (single user, etch)
16:10:14 <hogepodge> finally, there is talk of forking tempest. I am strongly opposed to that, and personally feel it is harmful to both our aims as a standard body and to the community at large
16:10:24 <eglute> we do not have the specs yet. this is jumping ahead and making assumptions what the spec would look like
16:10:40 * docaedo arrives a little late to lurk :)
16:11:19 <hogepodge> eglute: we do have an informal spec. For example, tests are not admin. That is a requirement. They must be api, that is also a requirement.
16:12:19 <dwalleck> hogepodge: But that's a spec for what attributes a Tempest test should have to be an interop test, which is different from defining an actual spec for capabilities
16:12:35 <eglute> hogepodge we agreed to work on formal specs first before proceeding further. otherwise we will run into issues again where the test dictates what the spec should be, and not the other way around
16:13:01 <dwalleck> The second being the one I thought we had been talking about at the midcycle, but if I'm mistaken, that's okay :-)
16:13:54 <hogepodge> the one we talked about at the midcycle?
16:14:08 <eglute> i agree with dwalleck here, we need to start with defcore specs, not tempest specs
16:15:51 <gema> not sure how much time I can commit to this, but I can have a go at trying to start a spec
16:16:06 <gema> I see the tagging as useful regardless, though
16:16:10 <gema> like an orthogonal thing
16:16:22 <eglute> gema that would be good. perhaps you could work with dwalleck since he already started on some analysis
16:16:24 <hogepodge> gema: that would be great
16:16:42 <gema> eglute: yep, that was going to be my first task, catch up with dwalleck :D
16:16:52 <Rockyg> ++
16:17:01 <eglute> #action gema and dwalleck to start working on a defcore spec
16:17:08 <gema> dwalleck: o/
16:17:09 <dwalleck> gema: Absolutely, I'd be glad to contribute
16:17:17 <dwalleck> !
16:17:20 <gema> ok, we'll start this week and see how it goes :D
16:17:24 <hogepodge> eglute: I would very much like to be involved with what a spec looks like too. As the person who has worked most closely with qa, I'd like us to maintain a good relationship on that.
16:17:43 <eglute> hogepodge i dont think anyone will be excluded!
16:17:44 <gema> hogepodge: how about I start a conversation on our mailing tomorrow
16:17:47 <gema> and we make it collaborative
16:18:12 <gema> then I put the result together on some sort of commit
16:18:18 <dwalleck> But which spec are we talking about? The spec for what defines a capability, or the specs of what makes a good interop test?
16:18:30 <gema> I thought the spec for defcore
16:18:36 <markvoelker> dwalleck: We have Board-approved definitions of Capabilities
16:18:45 <gema> test spec, that is
16:18:48 <VanL> dwalleck: Spec for what makes a good interop test
16:19:01 <Rockyg> VanL, right
16:19:22 <VanL> We have a couple principles that everyone agreed on, IIRC:
16:19:24 <dwalleck> Gotcha, wanted to make sure we were using the same terminology
16:19:27 <VanL> 1. No admin
16:19:46 <VanL> 2. (From Chris:) Two paid accounts not needed.
16:19:52 <VanL> 3. Atomic
16:20:18 <VanL> I think those were the only ones that we got to before we moved on
16:20:25 <VanL> I would also suggest:
16:20:36 <Rockyg> 4. available for at least 3 releases
16:20:42 <VanL> 4. The test matches the public documentation
16:20:46 <hogepodge> VanL: I object to you using language like "that everyone agreed on". It over simplifies and may not accurately represent my views.
16:21:06 <gema> hogepodge: I am going to go back to the midcycle notes tomorrow and summarize our discussion on this
16:21:13 <gema> then frame the conversation
16:21:16 <hogepodge> VanL for example, 3. Atomic, may differ from dwalleck's
16:21:23 <VanL> Rockyg: Yes, but that is a capabilities requirement, not a interop spec.
16:21:28 <markvoelker> One suggestion: let's be careful to not overlap with the Board-approved Criteria we already have.
16:21:36 <markvoelker> #link http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst Criteria
16:21:50 <hogepodge> or 2, when I say two tenant isolated accounts I mean two tenant isolated accounts, not paid accounts.
16:21:59 <markvoelker> We don't want to essentially have two sources of truth about those.
16:22:14 <eglute> +1 markvoelker
16:22:15 <catherineD> hogepodge: ++ ... Atomic is the reason causing the issues we have with today's test set
16:22:19 <dwalleck> hogepodge: that's a fair example. Then we need to drill down into those bits and talk them through and come to a community understanding
16:22:43 <VanL> hogepodge: Please don't personalize this. I am trying to honestly represent and summarize what I understood, and if the summary isn't accurate, then by all means chime in
16:22:46 <hogepodge> dwalleck: yes. I agree
16:23:05 <gema> VanL: no worries, fortunately for everyone my memory is bad, so I will go back and read the notes
16:23:08 <gema> x)
16:23:16 <VanL> I was mostly trying to quote from /summarize the notes
16:24:08 <dwalleck> My opinions are my own, but as someone who works in the community and has invested time, I have to share my thoughts, which doesn't mean anyone has to agree
16:24:22 <gema> dwalleck: ++
16:24:35 <markvoelker> gema: hogepodge: dwalleck: do you guys have what you need to proceed on drafting something up?
16:24:45 <dwalleck> I spent a great deal of time working on Tempest and then later on CloudCafe, so I have some strong opinions about tests :)
16:24:48 <gema> markvoelker: I think so
16:24:51 <dwalleck> Yes
16:25:00 <gema> dwalleck: me too
16:25:03 <gema> we'll have fun :D
16:25:05 <VanL> FYI: https://etherpad.openstack.org/p/DefCoreSpring2016MidCycle
16:25:24 <dwalleck> Gemma: yup!
16:25:34 <VanL> Also: https://etherpad.openstack.org/p/DefCoreSpecProposals
16:25:51 <markvoelker> Great.  I'll look forward to reviewing it then. =)  We're 25 minutes in and have several other agenda items--ready to move on?
16:26:17 <eglute> markvoelker +1
16:26:24 <Rockyg> ++
16:26:27 <dwalleck> Yup
16:26:32 <markvoelker> #topic Capabilities with no tests (or only admin tests)
16:26:49 <markvoelker> #link https://review.openstack.org/#/c/299491/
16:26:58 <markvoelker> #link https://review.openstack.org/#/c/300608/
16:27:13 <gema> I'd like to start capturing all of those and make a list
16:27:16 <gema> of tests to be created
16:27:36 <gema> I have a couple on keystone as well
16:27:52 <markvoelker> We've identified some advisory tests which turn out to require admin creds.  We can't use those in Guidelines, so they'll have to go.
16:28:08 <markvoelker> The biggest question is what to do about it, because the underlying features genereally don't require admin
16:28:13 <markvoelker> So, two things:
16:28:28 <markvoelker> 1.)  I'm going to spend some time looking for other tests of the same capabilities that don't require admin
16:28:56 <markvoelker> 2.)  I think we should have a chat with the project teams/PTL's/QA teams about potentially changing those tests to not need admin
16:29:52 <markvoelker> Those are sort of the natural outcomes...the bigger question is what to with the tests in the meantime since it's uncertain how quickly we could get the tests modified (if that's even realistic...there may be good reasons for using admin)
16:30:04 <catherineD> https://review.openstack.org/#/c/300608/  is an easy one to review  and merge... because the capability affected still have some tests in it after the admin tests are  flagged or removed..
16:30:19 <markvoelker> Obviously we've got a few months before the Guideline comes up for approval here, so there's potential for us to get things done before then.
16:31:08 <markvoelker> Failing that, we could consider leaving the relevant Capabilities in advisory state for another cycle so we don't have to restart the clock by dropping them and then having them go through another advisory cycle
16:31:23 <eglute> thank you catherineD for calling those tests out. i agree with markvoelker that we need better tests, but also agree that no idea how long this will take
16:31:55 <VanL> How much real-world vetting have the advisory capabilities had? Finding admin-requiring tests makes me nervous that they have not been properly vetted by enough people out in the community
16:32:01 <eglute> i think leaving them advisory would be ok, provided we start working on getting better tests somehow
16:32:11 <markvoelker> eglute: from a cursory look at a few of them, I think it may be realistic to improve some of these before the Guideline goes up the board, for what it's worth.
16:32:25 <catherineD> VanL: I have verified all the tests in the advisory section
16:32:36 <catherineD> and thus found the one with admin requires
16:32:42 <markvoelker> VanL: the scoring for them is in the scoring sheet.  They all stacked up to criteria like widely deployed.
16:33:03 <eglute> markvoelker i would be really hesitant to make them required if we are going to change the tests
16:33:11 <hogepodge> markvoelker: qa has a history of helping sort the issues out, maybe not in time for a guideline though. They could stay advisory
16:33:23 <catherineD> Thera are also some test records in RefStack that tests the entire API Tempest test that we can use to verify
16:33:40 <markvoelker> eglute: why so?
16:33:55 <VanL> markvoelker: Sure, but we run into the problem of the *capability* being widely deployed, but the tests to show a capability not being representative or otherwise having problems.
16:33:56 <hogepodge> markvoelker: otherwise they would be immediately flagged, with the action to be to fix or remove. That seems an odd way to start, but it's definitely a workable solution
16:33:59 <markvoelker> oh, you mean make the required and then change the tests?  Yeah, definitely.
16:34:08 <markvoelker> I'm saying we change the tests before we make them required.
16:34:32 <markvoelker> Remember that the next Guideline won't get approved for several months yet.  So there's a pretty good window of time here.
16:34:37 <hogepodge> markvoelker: yes, I think that's better (or write new tests if admin is required)
16:34:40 <catherineD> markvoelker: we do have a patch today to move advisory to required
16:34:40 <dwalleck> But if a test defines the capability, how can we add a test that we plan on immediately refactoring?
16:34:43 <eglute> markvoelker people that are/could be testing agains this as advisory might be blindsided by new tests. in theory, new tests should test the same stuff, but in reality they might not be exactly the same
16:35:28 <catherineD> at the mimimum we should flagged those admin tests .... if not removed
16:35:35 <dwalleck> It gives you a placeholder for intent, but I'm not sure what other value it provides
16:35:37 <markvoelker> eglute: I see the concern.  I think in most cases though, all we'd need to do is modify the existing tests to use ordinary credentials rather than admin ones.  E.g. we're not fundamentally changing what's being tested.
16:35:38 <eglute> i think keeping them as advisory is a good compromise
16:36:05 <dwalleck> ahh, admin part. Missed that, never mind :)
16:36:06 <catherineD> eglute: keeping the capability but flagged or removed the admin tests ?
16:36:16 <markvoelker> catherineD: keep in mind that even if the patch moving them to required lands, that doesn't actually mean anything until the Board votes on the Guideline in August
16:36:24 <eglute> keeping the capability as advisory rahter than making it required
16:36:34 <eglute> catherineD ^^
16:36:38 <markvoelker> catherineD: until then it's just a draft.  But I'd be fine with dropping those tests from moving right now just for clarity.
16:37:16 <catherineD> markvoelker: agreed but if may make the required in next.json and the advisory in 2016.01 mis-match
16:37:42 <catherineD> markvoelker: agree for dropping ... I think it is cleaner
16:37:45 <hogepodge> markvoelker: eglute: catherineD: it's pretty easy to tell if you're changing essential functionality. I saw this in the multi-tenant tests. It was clear multi-tenancy was not required on all but two tests.
16:38:01 <catherineD> and send a good message the defcore does not require admin
16:38:12 <markvoelker> Ok, so I think what I'm hearing here is:
16:39:03 <markvoelker> #action markvoelker submit new patchset for https://review.openstack.org/#/c/294287/ to leave the admin-requiring tests as advisory
16:39:12 <eglute> +1
16:39:24 <markvoelker> #action markvoelker (and anyone else interested) look for alternative tests for the same capabilities
16:39:30 <eglute> +1
16:39:58 <markvoelker> #action markvoelker (and anyone else interested) investigate the feasibility of modifying the tests to not need admin
16:40:02 <catherineD> markvoelker: do we leave the admin tests or remove the test but leave the capability?
16:40:40 <catherineD> we really should not have any admin tests now that we identify them
16:40:41 <markvoelker> catherineD: I'm planning to leave them for the moment since it seems likely we're going to try to get some of them changed to not need admin.  If that doesn't happen, we have several more months to drop them
16:41:18 <eglute> that sounds like a good plan to me
16:41:44 <markvoelker> But if folks feel strongly one way or the other, I'm not opposed to dropping and re-adding them.  Just seems to make a bit more work for later.
16:42:04 * markvoelker looks at clock
16:42:19 <eglute> i am ok with dropping as well. the only thing i would be opposed to is making these tests required for 2016.08
16:42:28 <markvoelker> Ok, so I'll go that route and if anyone has strong opinions about dropping them now or not then voice them in gerrit and we'll iterate there. =)
16:42:39 <markvoelker> Move on?
16:42:41 <eglute> thank you markvoelker!
16:42:42 <eglute> yes
16:42:58 <markvoelker> #topic RefStack update to use Tempest tag 10.0.0 as default
16:43:06 <markvoelker> catherineD: the floor is yours
16:43:13 <catherineD> thx
16:43:46 <catherineD> so we will update RefStack to tempest tag 10.0.0 which is the tag for Mitaka
16:44:08 <catherineD> currently RefStack still running with a Sept 11, 2015 tempest vestionb
16:44:17 <catherineD> it is kind of old ...
16:44:40 <eglute> catherineD do you foresee any issues with the new tag?
16:45:42 <catherineD> eglute: so we have taken care of the test names change by https://review.openstack.org/#/c/297993/  ...
16:46:06 <eglute> ok
16:46:19 <catherineD> there are also some change in tempest conf that the user have to pay attention to
16:46:54 <catherineD> I know that a lot of the user is using master copy of tempest .... those users will not be affected
16:47:04 <hogepodge> tempest is moving to a model where credentials and resources are defined outside of tempest.conf. It's a good change, but one that may have short term impact
16:47:05 <markvoelker> cathrineD: this looks like a pretty routine update to me.  So, seems fine.  I'll see if I can run that version through my lab, but I don't foresee any real issues.
16:47:45 <markvoelker> hogepodge: Yep, saw that too.  Perhaps something to note in our instructions.
16:47:46 <catherineD> question is for DefCore to identify new tests to add ... should defcore also use the latest test list?
16:48:17 <hogepodge> catherineD: do you mean on aliasing?
16:49:03 <catherineD> alias is for the existing defcore test ... but if defcore going to add new tests ... defcore should use update to date test list
16:49:35 <markvoelker> catherineD: hogepodge: that's covered here: https://review.openstack.org/#/c/294287/
16:49:37 <markvoelker> oops
16:49:44 <markvoelker> I mean http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/schema/1.4.rst#n85
16:50:16 <markvoelker> So basically for a new Guideline (next.json) we use the current name of the test as it appears in Tempest.
16:50:30 <markvoelker> If there's an older name that's been changed, we move that to the alias field in next.json
16:50:55 <catherineD> for the new tests .. if defcore use new test lists for its tests  we will avode alias
16:50:57 <markvoelker> I think the same principle applies to new tests we're adding too
16:51:05 <catherineD> avode --> avoid
16:51:38 <catherineD> new tests here means the test name that has never appeared in defcore json
16:51:57 <markvoelker> catherineD: Well, usually, probably so.  But possibly not always.  Because if someone's using an older version of Tempest it might have an older test name.
16:52:17 <markvoelker> So conceivably we may need to add an alias even though we haven't required that test in the past.
16:52:32 <markvoelker> I suspect that'll be a pretty rare case in practice though
16:52:33 <catherineD> if someone is testing a new OpenStack release they should not use old json
16:53:20 <markvoelker> Well, we don't bound what version of tempest is used.  So it's not so much using old json as it is using an older Tempest (which might be necessary due to bugs, etc).
16:53:32 <catherineD> for example for Mitaka the require tests would jump from 100+ to 200+ because of the advisory tests
16:53:39 <eglute> because defcore covers several releases, they might be using older json
16:53:52 <catherineD> if certify for Mitaka they must use 2016.08
16:54:27 <catherineD> agree .. for older release the tests will be covered by aliases
16:54:54 <catherineD> but the newly defcore added tests does not need to be burdened with aliases
16:55:01 <eglute> true
16:55:15 <markvoelker> cathrineD: correct, they need to use 2016.08.  That Guideline might have a test called "ABC", which is what Tempest calls it today.  But Tempest might have changed it to "ABC" from "ZYX" the week before we added it to the Guideline.
16:55:44 <markvoelker> So if someone is using an older version of Tempest, they'll see "ABC" skipped because it's not found.  They might therefore reasonably request an alias be added.
16:55:51 <catherineD> markvoelker: in that case it will cover by aliases becayse the test already exists in defcore
16:56:26 <catherineD> think about the tests that are not in any defcore json today
16:56:44 <catherineD> those are the tests that we should use from new test list
16:57:00 <catherineD> anyway back to refstack update default tempest
16:57:17 <catherineD> I guess it is a go ?
16:57:18 <hogepodge> three minutes
16:57:27 <markvoelker> catherineD: go from me
16:57:28 <eglute> yes, catherineD lets go ahead with new version
16:57:35 <hogepodge> want to make sure people look at gema's identity capabilities patch
16:57:41 <gema> yes please
16:57:47 <gema> no need to take time here, though :D
16:57:49 <eglute> #topic keystone scoring
16:58:02 <eglute> everyone, please review gema's patch
16:58:02 <catherineD> thank you ... yea after all that tempest ins 6 month old  ...as old as the openstack release :-)
16:58:09 <markvoelker> #link https://review.openstack.org/#/c/297847/ Keystone scoring patch (new rev)
16:58:24 <eglute> thank you catherineD!
16:58:34 <eglute> and thank you gema, I will review the patch today
16:58:40 <gema> eglute: thanks!
16:58:47 <eglute> if any questions, lets take them to the defcore channel
16:59:13 <markvoelker> With that we're about out of time.  We skipped the advisory->required topic, but we actually already discussed it a bit earlier and I have an AI to submit a new patchset on it
16:59:21 <markvoelker> Thanks everyone!
16:59:24 <eglute> thanks!
16:59:25 <markvoelker> #endmeeting