16:00:45 <eglute> #startmeeting defcore
16:00:45 <openstack> Minutes:        http://eavesdrop.openstack.org/meetings/monasca/2016/monasca.2016-03-02-15.00.html
16:00:45 <witek> bye
16:00:46 <openstack> Minutes (text): http://eavesdrop.openstack.org/meetings/monasca/2016/monasca.2016-03-02-15.00.txt
16:00:47 <openstack> Log:            http://eavesdrop.openstack.org/meetings/monasca/2016/monasca.2016-03-02-15.00.log.html
16:00:48 <openstack> Meeting started Wed Mar  2 16:00:45 2016 UTC and is due to finish in 60 minutes.  The chair is eglute. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:53 <openstack> The meeting name has been set to 'defcore'
16:00:55 <hogepodge> o/
16:00:58 <dwalleck> o/
16:01:04 <markvoelker> o/
16:01:05 <purp> o/
16:01:05 <eglute> #chair markvoelker
16:01:06 <openstack> Current chairs: eglute markvoelker
16:01:19 <eglute> #topic agenda
16:01:22 <eglute> #link https://etherpad.openstack.org/p/DefCoreRing.14
16:01:33 <eglute> Hello Everyone! please review the agenda
16:01:34 <catherineD|2> o/
16:02:02 <eglute> #topic Feedback from running DefCore tests in CI
16:02:11 <eglute> dwalleck, the floor is yours!
16:02:14 <leecalcote> o/
16:02:24 <dwalleck> eglute: Thanks!
16:03:00 <dwalleck> So I've been working with folks inside Rackspace for awhile now on continuous DefCore testing. It's been....fun :)
16:03:45 <dwalleck> But there were some common themes that have come up that might be of value to a larger audience
16:04:12 <gema> o/
16:04:50 <dwalleck> The majority of the feedback was in some way centered on Tempest results
16:05:49 <dwalleck> People really, really don't like that when you run the subset of DefCore tests, that the pass/fail numbers don't add up to the total of tests to be run for DefCore
16:06:05 <purp> dwalleck: why?
16:06:22 <brunssen> o/
16:06:31 <gema> because of the tempest.conf
16:06:35 <gema> you can configure tests out
16:06:45 <dwalleck> purp: If there's a failure in a test fixture, that counts as one failure rather than the number of tests to be run in the class
16:07:38 <purp> dwalleck: ... thus the numbers don't sum, leaving you feeling empty and cold. Got it.
16:07:46 <SammyD> purp: Exactyl
16:07:56 <eglute> that does sound like a lot of fun...
16:08:17 <dwalleck> So after some manual auditing, we wrote a quick additional subunit parser to check the list of tests executed against the one used to run the tests
16:08:28 <dwalleck> https://github.com/arithx/subunit_verify
16:08:42 <dwalleck> This is a hack, but it gave people warm fuzzies :)
16:08:43 <hogepodge> dwalleck: we have room in the defcore repository for tools like that
16:08:59 <markvoelker> #link https://github.com/arithx/subunit_verify parser to check the list of tests executed against the one used to run the tests
16:09:09 <dwalleck> hogepodge: That would be great!
16:09:24 <SammyD> +1 hodgepodge
16:09:27 <catherineD|2> dwalleck: DefCore pass/fail shoukd be checked against the DefCore test list and not from pass/fail based on subunit file.
16:10:15 <dwalleck> I also know that if I had product teams push their results to RefStack, alot of that would be sorted out. However, having a dozen product teams sending results to RefStack would be a challenge to understand
16:10:18 <hogepodge> My opinion is that the repository can house tools to make testing life easier. The defcore chairs have the final say on what goes, though. refstack may be a more appropriate place for some tools also.
16:11:01 <dwalleck> I had never thought about how to use RefStack properly if you have multiple entities from one provider trying to use it all at once
16:11:05 <eglute> i am ok with having additional tools!
16:11:50 <markvoelker> dwalleck: you might be pleased to note that the RefStack folks have thought about that. =)  There are some specs out for review on vendor/product relationships that catherineD can probably point you to.
16:11:51 <dwalleck> hogepodge: Totally agree. I figured the "where" could be sorted out as long as the "why" made sense
16:12:04 <SammyD> A tools repo to improve life is a great start. Over time, key functionality from random tools should be migrated into RefStack or a similar all in one client though. :-)
16:12:31 <SammyD> ^re-useable not reliable
16:12:39 <rockyg> o/
16:13:12 <dwalleck> markvoelker: That's good to hear :) I'll have to ping catherineD|2 next week with some of the specifics
16:14:22 <dwalleck> The last bit of feedback, bluntly, is that people just don't seem to understand Tempest test failures, i.e. what part of capability is actually broken when a test fails
16:14:49 <dwalleck> This has given me gray hairs over the last 6 months :)
16:15:10 <eglute> i thought you said it was fun? :)
16:15:35 <dwalleck> To give you an example, our public cloud has a number of failures due to certain extra properties being returned in API responses
16:15:43 <purp> Maybe dwalleck is aiming for Silver Fox look, so the grey hairs are fun?
16:15:44 <rockyg> fun like being in the middle of a hurricane...
16:15:53 <eglute> so given this feedback, what do you think the solution would be? use the tool you worked on for people that want warm fuzzies?
16:16:31 <dwalleck> This causes a large number of test failures, but all with the same root cause. I had to write some aggregator tooling for results so that people didn't think the sky was falling :)
16:16:37 <eglute> also, anyone else testing regularly have similar (or different) feedback?
16:17:05 <markvoelker> Sounds like maybe you'd also like to see some improvements to the test failure messages ("test failed because stuff I don't know about was detected in the JSON respnonse")?
16:17:06 <dwalleck> But even with that, I had a hard time explaining that the entire capability (create server, etc) was not broken, but just an aspect of it
16:17:35 <SammyD> mvoelker: Exactly. That would be a place to start
16:17:46 <dwalleck> Part of what I'm seeing is that the Tempest tests as-is are not quite atomic enough. It would be nice if a failure was self-explanitory
16:18:09 <catherineD|2> dwalleck: Failure due to extra info response is the purpose if interops test right?
16:18:37 <gema> extra info won't necessarily break people's apps, lack of info would
16:18:38 <SammyD> I could debate that. Failure is fine. Failure shoudl be explicit though
16:18:46 <markvoelker> catherineD|2: yes, according to the test authors/TC
16:19:02 <SammyD> gema: +1
16:19:02 <dwalleck> catherineD|2: True, but me delivering a result to a product manager saying "the create capability is broken" isn't specific enough, and causes panic
16:19:06 <markvoelker> catherineD|2: but I think the larger point was that it was difficult to tell that that's why the test failed
16:19:30 <dwalleck> And is misleading because the functionality of the capability worked
16:19:53 <purp> dwalleck I think that's the key
16:20:00 <dwalleck> I don't disagree that it should be a failure. I would argue that we could make aspects of it more atomic
16:20:12 <catherineD|2> agree that debugging on failure is not easy
16:20:25 <purp> I'd say that a test which fails due to extra info in a response is either too fragile or should fail because it's specifically testing to prove the info isn't there.
16:20:39 * purp is contrary
16:20:41 <gema> purp: +1
16:20:43 <SammyD> purp: my point as well
16:21:03 <SammyD> part of what dwalleck is talking about when he says the tests need to be more atomic. :-)
16:21:04 <purp> Either you're specifically testing to see that the info is excluded, or your test is too fragile and breaking for poor reasons.
16:21:09 <dwalleck> purp: The challenge is that the response checking is built into the Tempest clients, and therefore into every test as an implicit check
16:21:10 <rockyg> Yeah.  Tempest was not designed to return info on option by option basis
16:21:28 <SammyD> ^what dwalleck just said
16:21:33 <eglute> hogepodge what have you been hearing from people that are testing/certifying?
16:21:34 <dwalleck> An explicit check for response conformity, etc, would be much easier to understand
16:21:51 <SammyD> in current implementation *any* test will fail if there are extra params in the repsonse, regardless of whether or not the functionality actually works
16:22:05 <dwalleck> eglute: ++. I'm really interested to hear if others have had similar thoughts
16:22:13 <purp> Okay, so sadly, then, I'd say the Tempest parser is either too strict for purpose (not reflective of real world) or we're expecting to enforce a response interface that is not allowed to include any extra info.
16:22:21 <purp> I'd view the latter as a poor design choice.
16:22:30 <eglute> +1
16:22:32 <hogepodge> Configuration is always a problem. Pointing to the official tempest configuration guide solves a lot of problems.
16:22:33 <SammyD> +1 purp
16:22:43 <rockyg> Part of the problem is that there is no "standards" for writing api tests (or any others) for that matter, other than it tests something the devs care about.
16:23:03 <markvoelker> purp: SammyD: Yeah, that's the explicit intent of the authors.  I can point you to the discussion on this a few months ago when that change was made to Tempest, but let's do that offline so as not to distract from the larger point: atomicity and better failure messages
16:23:09 <hogepodge> Usually vendors get permanently stuck when they've made downstream modifications or are backing their block storage with ceph.
16:23:26 <SammyD> mvoelker: agreed, thanks. :-)
16:23:28 <dwalleck> hogepodge: But what about interpreting results? Have you had much feedback in that area?
16:23:45 <rockyg> So, unless/until there are more standards as to what comprises the test, or how the test should respond/report, we'll see this crop up.
16:23:56 <purp> markvoelker: before we offline, want to understand. I believe you just said that strict parsing is an intentional design choice.
16:24:18 <hogepodge> They get confused about what needs to be run. Oftentimes I'll get support tickets on required but flagged tests (the terminology is not helpful)
16:24:43 <markvoelker> purp: correct.  A decision was made that adding extra data to API repsonses is no bueno and that vendors should not change the API.
16:24:43 <dwalleck> I didn't mean to soapbox this long, but I was hoping this would be helpful feedback :) I'll be in Austin next week and I can show the specifics and some of the other tooling I've put together
16:25:00 <hogepodge> when fixtures fail, it's usually trackable to common problems
16:25:20 <purp> markvoelker: I'm sad to hear that. Okay.
16:25:36 <markvoelker> dwalleck: This is good feedback!  Would you consider writing up a short summary in the etherpad (or somewhere else) so we can refer back to it?
16:25:37 <purp> eglute markvoelker: do we have a session for this in Austin?
16:25:43 <rockyg> I can dig up the mail thread about that, given enough time. I can post to ML>
16:25:52 <eglute> we have a session to talk about tests in general, yes
16:25:57 <dwalleck> markvoelker: Absolutely! I'll do that
16:26:11 <markvoelker> purp: The API response thing?  No, it was a decision made by nova-core IIRC.
16:26:39 <markvoelker> and/or other projects
16:26:49 <hogepodge> Feedback is always good. We shouldn't consider the standard or our policies as inflexible (although as the person administering the program I do need to enforce the it)
16:26:52 <purp> markvoelker: Sorry, meant the tests and responses.
16:27:10 <rockyg> Yeah. more strict validation of expected return info.
16:27:35 <SammyD> mvoelker: I don't agree or disagree with the paramater checking, just that it is not explicit when it fails due to the way it is implemented. :-)
16:27:38 <markvoelker> purp: Oh!  Sort of.  There's an hour on the tentative schedule to talk about the possibility that Tempest isn't meeting all our needs
16:27:39 <hogepodge> that is to say, we learn from what we have and evolve the standards and guideline
16:28:02 <markvoelker> SammyD: yep, I hear ya. =)
16:28:29 <rockyg> SammyD: file a bug against the test?  What was the response?
16:28:30 <eglute> I propose that we move the rest of this discussion to the midcycle. Thanks dwalleck, would be great if you could do a write up, as markvoelker suggested
16:29:07 <SammyD> rockyg: Couldn't file a bug against the test, there is no test for the API responses. It's a global default validation in the client. :-(
16:29:17 <markvoelker> I think some of this dovetails into the topic we currently have at 2:30 on Day 2...
16:29:32 <markvoelker> #link https://etherpad.openstack.org/p/DefCoreSpring2016MidCycle First pass at agenda for midcylce
16:29:33 <rockyg> Ah.  Yup.  I feel your pain.
16:29:33 <dwalleck> eglute: no problem, thanks!
16:29:36 <eglute> right, lets talk about the midcycle
16:29:43 <eglute> #topic midcycle
16:29:57 <rockyg> Huawei next cycle will hit same problem.
16:30:29 <eglute> if you have not responded yet regarding your restaurant preferences, please do so today before 2PM CST: http://doodle.com/poll/x3n9yvdws9miyrck
16:31:05 <eglute> Also, please take a look at the agenda: https://etherpad.openstack.org/p/DefCoreSpring2016MidCycle
16:31:23 <eglute> thanks markvoelker for preliminary schedule setup
16:31:33 <markvoelker> So, regarding the agenda:
16:31:56 <markvoelker> You'll notice it's pretty packed. =)  I made a first pass at it including all the topics that were in the suggested topics list from the pad
16:32:00 <purp> Would like to swap the 9:30a and 10a sessions on the first day.
16:32:11 <markvoelker> But I think there are probably some candidates for pruning in there so we can free up more time for longer discussions
16:32:30 <markvoelker> You'll notice a couple of themes for each day
16:32:41 <eglute> right, i think the test discussion is strategic and we need to have it on day 1
16:32:49 <markvoelker> Those are sort of arbitrary, just a way to bundle together related topics
16:32:51 <eglute> and i am guessing it will take more than 1 hour
16:32:57 <eglute> right, thanks markvoelker
16:33:27 <markvoelker> Egle and I were thinking of swapping the Outcomes and Tests sessions so we can tackle Tests on Day 1
16:33:58 <markvoelker> And also probably dropping the Publishing DefCore Information and Docs session since it really just seems like a to-do list item we can tackle in 5 minutes
16:34:10 <markvoelker> Any objections to either of those things?
16:34:39 <eglute> works for me
16:34:44 <gema> +1
16:35:08 <markvoelker> #action markvoelker to swap Outcomes and Tests sessions, drop Publishing session
16:35:24 <markvoelker> Ok, any other items that look like targets for pruning?
16:36:22 <eglute> might need to move user feedback to the next day based on Edgar Magana's availability
16:36:51 <markvoelker> eglute: I was just about to come to that and tackle purp's request in one fell swoop =)
16:37:08 <eglute> :)
16:37:25 <eglute> also, I am worried that some of these discussions will take longer than we have the time
16:37:30 <markvoelker> Ok, before we get there: if anyone else has suggestions for stuff you don't think is urgent enough to warrant time at the midcycle, drop me a line and we'll discuss
16:38:12 <markvoelker> So, purp: I actually originally had those swapped, but one of those sessions will have Edgar Magana from the User Committee joining us remotely from California (where it'll be pretty early)
16:38:16 <catherineD|2> markvoelker: please allow time for RefStack discusssion (one hour) under Refstack demo
16:38:33 <markvoelker> And actually we just heard from him that he'll need a later timeslot anyway, so stand by while we tinker with that some more
16:38:38 <purp> markvoelker understood, and I think that session precedes any rational discussion of operating model
16:39:10 <markvoelker> purp: Sure, I think we can work that out.
16:39:16 <eglute> i would like to move Scoring: Promoting Advisory Capabilities discussion to before actual scoring
16:39:25 <purp> markvoelker cool. Otherwise, I'm happy to roll with whatever.
16:40:11 <markvoelker> eglute: That's easily done, the scoring discussions can really be in almost any order unless they require remote attendance from anyone.
16:40:26 <eglute> thanks markvoelker
16:40:46 <markvoelker> #action markvoelker Move Promoting Advisory Capabilities to earlier in the day
16:40:47 <eglute> hogepodge are there any updates on the licensing changes?
16:41:33 <hogepodge> eglute: I can check in again. It's with the lawyers and exec staff right now
16:41:46 <markvoelker> #action markvoelker rework Value From DefCore session to accommodate remote attendee
16:41:50 <eglute> thanks hogepodge, we can always cover it during the regular meeting
16:42:25 <markvoelker> Any other shuffling folks are interested in after looking at the first pass?
16:43:09 <markvoelker> If not, I'll rework the agenda with the feedback you've all given so far and send out an update to the ML this afternoon for further iteration
16:43:14 <eglute> thanks markvoelker
16:43:23 <markvoelker> One more big important thing....
16:43:52 <eglute> also, everyone, please note that the schedule starts at 8:30 rather than my original invite at 9:00 AM. does 8:30 work for everyone?
16:43:52 <markvoelker> It was a little hard to tell who added some topics to the suggested topics list, so you'll notice that some of the sessions on the strawman agenda don't have a designated leader
16:44:10 <purp> +1 eglute
16:44:21 <markvoelker> If you added a topic to the list and want to lead it and are not listed as the leader in the agenda, please add yourself!
16:44:31 <gema> +1
16:44:36 <rockyg> argh.  I get in 1am that morning, but Ill make.
16:44:37 <markvoelker> For those that don't pick up a leader, I'll either handle them myself or split them up with Egle.
16:45:16 <hogepodge> markvoelker: +1
16:45:20 <markvoelker> Regarding the 8:30 thing: please note that 8:30-9 is just introductions, agenda review, and coffee
16:45:30 <markvoelker> So if you're a little late, that's probably not the end of the world
16:45:48 <gema> nobody wants to be late for coffee with such an agenda
16:46:11 <eglute> any other people have concerns about starting at 8:30 AM? i don't like mornings in general, but that's a different issue :D
16:46:13 * markvoelker just subtly put brussen on the hook for making sure there is copious coffee, but he's a smart cookie and probably already thought of that
16:46:26 <eglute> :D
16:46:56 <eglute> any other comments regarding midcycle planning?
16:47:12 <markvoelker> eglute: two quick ones
16:47:49 <markvoelker> We have an official dinner planned...anyone interested in informal dinner/drinks gatherings Monday evening or Wednesday evening (for those who get in early/leave later)
16:48:22 <markvoelker> IF so, please drop your name in the DefCoreRing.14 etherpad and if there's enough interest we'll try to coordinate
16:49:00 <markvoelker> eglute: ok, done now. =)
16:49:14 <dwalleck> I'm always down for drink....err food :)
16:49:28 <eglute> #topic refstack
16:49:32 <purp> Heh. I have drinks plans in South Austin Monday, would welcome dinner.
16:49:34 <eglute> catherineD|2, go ahead!
16:49:39 <catherineD|2> eglute: thx
16:50:06 <eglute> purp -- put your name in https://etherpad.openstack.org/p/DefCoreRing.14 line 38!
16:50:16 <catherineD|2> The first topic is about prevent uploading of duplicate test results to RefStack
16:50:48 <catherineD|2> #link RefStack bug  https://bugs.launchpad.net/refstack/+bug/1498159
16:50:49 <openstack> Launchpad bug 1498159 in refstack "refstack-client should help user to avoid test duplication" [Medium,In progress] - Assigned to David Liu (lzbj)
16:51:50 <catherineD|2> it seems like the only way to ensure that duplicate test results are not uploaded to RefStack to enforce testing done with RefStack client and upload the result immediately after the test is done
16:52:31 <catherineD|2> vs today ... test results can be uploaded afterward
16:53:05 <eglute> i think it is ok that the tests can be updated later
16:53:17 <eglute> i prefer to have the flexibility
16:53:42 <markvoelker> Hmm...wonder if this could be solved by computing a SHA on the test results + logs or something?
16:53:44 <purp> And I wouldn't assume that they're always run from an internet-connected machine.
16:53:53 * markvoelker is thinking out loud which is often dangerous
16:54:04 <purp> markvoelker: I was thinking UUID identifier in the beginning of test run
16:54:05 <markvoelker> purp: ++
16:54:05 <catherineD|2> eglute: since we allow test can be uploaded later we have no good ways to ensure that duplicate test results uploaded to RefStack
16:54:25 * purp assumes ++ was for internet-connection-optional
16:54:30 <markvoelker> purp: that would have to be built into tempest though if we don't require that refstack-client be used, right?
16:54:46 <purp> markvoelker it would, but would help overall debuggability
16:54:53 <purp> (if that's a word)
16:55:17 <catherineD|2> This is just one aspect of the bigger issues of test integrity
16:55:33 <purp> markvoelker catherineD|2 perhaps both: UUID for human readability, and a SHA on the upload to point out duplicates.
16:55:39 <markvoelker> purp: Yeah, I was thinking a SHA on the results/logs gives a unique identifier that's independent of the test runner
16:55:56 <markvoelker> E.g. you're essentially creating a unique identifier at the end rather than at the start
16:56:18 <rockyg> catherineD|2, ++  But, we've been operatibg ob trust since the beginning.
16:56:19 <catherineD|2> markvoelker: purp: subunit file or RefStack JSON file is a text file ... it is easy to change
16:56:31 <purp> markvoelker fair, and I think both would be useful.
16:56:44 <purp> 4 min warning
16:56:55 * markvoelker glances at the clock and wonders if we should take the brainstorming over to #openstack-defcore so Catherine can talk about other stuff
16:56:59 <catherineD|2> it the SHA is created at refstack-client ... the "malicious" user can really recreated the SHA
16:57:18 <catherineD|2> I think this can be a topic at the midcyle
16:57:34 <eglute> thanks catherineD|2!
16:57:46 <catherineD|2> next move on to the second topic of RefStack
16:57:52 <eglute> i will be in the defcore irc after this as well
16:57:55 <rockyg> So, from my perspective, we shouldn't worry too much with what tempest is as the moment and work to build in better accountability through enhancing/adding checks to tempest in future.
16:58:16 <catherineD|2> RefStack vendor registration question in http://lists.openstack.org/pipermail/defcore-committee/2016-February/001035.html
16:58:23 <eglute> we are out of time... if you can stay a bit longer, please move to the #openstack-defcore
16:58:38 <eglute> thanks everyone, see you next week!
16:58:42 <catherineD|2> hogepodge:  do we allow "Vendors registering with RefStack not being in OpenStack MarketPlace"
16:58:45 <eglute> #endmeeting