16:01:44 <markvoelker> #startmeeting interopwg
16:01:44 <openstack> Meeting started Wed Jun 14 16:01:44 2017 UTC and is due to finish in 60 minutes.  The chair is markvoelker. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:47 <openstack> The meeting name has been set to 'interopwg'
16:01:48 <eglute> o/
16:01:50 <markvoelker> o/
16:02:04 <markvoelker> #chair eglute hogepodge
16:02:05 <openstack> Current chairs: eglute hogepodge markvoelker
16:02:18 <markvoelker> #link https://etherpad.openstack.org/p/InteropVertigo.4 Today's Agenda
16:02:38 <hogepodge> o/
16:02:44 <zhipeng> o/
16:02:53 <markvoelker> Please take a quick look at today's agenda and amend as necessary
16:03:33 <markvoelker> #topic Denver PTG
16:04:10 <markvoelker> The next PTG is in Denver, Colorado, USA September 11-15
16:04:10 <eglute> As last PTG, we will be sharing a room with RefStack. Time to book hotel room if you are coming!
16:04:41 <Rockyg> o/
16:04:45 <eglute> Interop/RefStack will be Monday/Tuesday
16:04:52 <markvoelker> If you're considering attending, you can find all the info you need at:
16:04:56 <markvoelker> #link https://www.openstack.org/ptg Main PTG site
16:05:05 <mguiney> o/
16:05:07 <markvoelker> See also hotel block here:
16:05:14 <markvoelker> #link http://www.marriott.com/meeting-event-hotels/group-corporate-travel/groupCorp.mi?resLinkData=the%20OpenStack%20Project%20Teams%20Gathering%5Edensa%60fntfnta%60149.00%60USD%60false%604%609/7/17%609/19/17%608/20/17&app=resvlink&stop_mobi=yes Hotel block for PTG
16:05:44 <markvoelker> Any questions/comments on the PTG?
16:05:45 <hogepodge> There will also be a board meeting the Sunday prior to the PTG, and you may want to plan accordingly if you want to attend that.
16:07:13 <markvoelker> Ok, hearing nothing further, let's move on
16:07:20 <markvoelker> #topic 2017.08 Guideline
16:07:40 <markvoelker> #link https://review.openstack.org/#/c/467493/ non-admin token validation test
16:08:09 * markvoelker watches mguiney push a new patchset
16:08:29 <mguiney> heh. sorry, just saw a tiny typo error that was causing pep8 failute
16:08:32 <mguiney> *falure
16:08:52 <markvoelker> No worries. =)  Anything we need to discuss on this?  Looks like it's just waiting for reviews.
16:09:47 <markvoelker> Ok, moving on to:
16:10:01 <markvoelker> #link https://review.openstack.org/#/c/467528/ catalog standardization test
16:10:24 <markvoelker> THis is one we had decided was probably too early for 2017.08 but we'd mention in the BoD meeting
16:10:45 <mguiney> yep, still working on that.
16:11:51 <markvoelker> OK then, that just leaves us with this:
16:11:57 <markvoelker> #link https://review.openstack.org/#/c/472848/ Create 2017.08 Guideline
16:12:40 <eglute> thanks markvoelker for creating it. looks like i left out moving one capability to required
16:13:00 <markvoelker> eglute: Ok, I can push a new patcheset to correct that.
16:13:09 <eglute> thank you markvoelker
16:14:51 <markvoelker> Anything else?  If not, I think this just needs folks to review.  Should be pretty straightforward, and even after we land it, it can be patched until the BoD meeting.
16:15:15 <eglute> no, looks good to me
16:15:44 <markvoelker> OK then, I'll get the new patchset up today and we'll see if we can land it.
16:16:02 <markvoelker> #action markvoelker respond to comment on 2018.07 patch
16:16:12 <markvoelker> Anything else on 2017.08?
16:17:02 <markvoelker> #topic ML Discussion on Tempest tests, plugins, etc
16:17:42 <markvoelker> Last week we mentioned that there are some discussions about Tempest and test locations on the ML's, and new interop programs were brought up
16:18:14 <markvoelker> There seemed to be some lack of clarity about what we were thinking, so I volunteered to take a stab at writing up what we're thinking about for the vertical and add-on programs
16:18:33 <markvoelker> I did that here:
16:18:36 <Rockyg> Thanks for doing that.
16:18:48 <markvoelker> #link https://review.openstack.org/#/c/472785/ Create rough draft of extension programs
16:18:54 <markvoelker> And sent it out to the ML's here:
16:19:09 <markvoelker> #link http://lists.openstack.org/pipermail/openstack-dev/2017-June/118182.html ML message about future interop programs
16:19:55 <eglute> thank you markvoelker!
16:19:58 <markvoelker> There's been a little bit of feedback so far, but I'd specifically like to invite the rest of you to chime in on gerrit if there are things you want to discuss in the draft
16:20:17 <markvoelker> (or if you just want to make fun of my terrible diagrams)
16:20:20 <eglute> i will re-review again, but after reading it once it looked good to me
16:20:47 <eglute> markvoelker i imagine once we go forward with this foundation will have someone create new diagrams as needed
16:21:47 <markvoelker> Good heavens, I sure hope so...I am many things, but a graphic artist isn't one of them. =)  Still, I thought having a rough sketch would be helpful for understanding.
16:22:09 <eglute> heheh i think those diagrams are great!
16:22:15 <eglute> :D
16:22:49 <eglute> there is a board meeting next week
16:23:16 <eglute> and i would like us to present the draft to the board, if you think it is ready for that
16:23:28 <eglute> just so they can review it before it is presented for final aproval
16:23:31 <Rockyg> maybe we should ask the tc for a tc meeting to discuss?
16:23:49 <Rockyg> or ask on the dev ml?
16:23:56 <markvoelker> Sure--although I am traveling next week and may or may not be able to join the call.  Will need to check schedule.
16:24:51 <eglute> Rockyg it was already sent to the dev mailing list, as well as ops
16:25:16 <eglute> markvoelker ok, we can discuss offline regarding your schedule
16:25:29 <Rockyg> I know the initial mail was, but I'm wondering if we should do more to get the tc involved.
16:25:56 <markvoelker> Rockyg: I know at least some of the TC folks are aware of it (cdent chimed in on IRC the other day), so if the TC would like to discuss it they're welcome to. I expect they'll let us know if they'd like to discuss in a meeting and have us be there to take questions (and I'm happy to do so).
16:26:19 <Rockyg> Thanks.
16:26:46 <markvoelker> And Sean McGinnis actually commented in gerrit too, which is great.
16:27:01 <eglute> great suggestions Rockyg! and thanks markvoelker for following up on all the comments
16:27:38 <Rockyg> Figure we should give the TCs at least a couple of weeks ;-)
16:27:57 <markvoelker> Ok, anything else on this to discuss today?
16:28:57 <markvoelker> Ok, moving on...
16:29:15 <eglute> we need more reviews on this, otherwise, no :)
16:29:26 <Rockyg> yup
16:29:47 <markvoelker> #action everyone please review new programs draft
16:29:58 <markvoelker> #topic 2.0 Schema
16:30:13 <markvoelker> #link https://review.openstack.org/#/c/430556/ Initial work on 2.0 schema
16:30:59 <markvoelker> hogepodge has picked up work on this again and I've been working through it in conjuction with the extension program draft
16:31:15 <eglute> thank you hogepodge
16:31:23 <eglute> i have not had a chance to look at the newest patch!
16:31:35 <hogepodge> Yes, I need reviews to make sure I'm not missing anything, or don't have any glaring errors
16:31:38 * eglute looking now
16:31:44 <markvoelker> hogepodge: I've got a couple of suggestions that I'm working through mentally locally while setting out the add-on and vertical program definitions
16:31:45 <hogepodge> suggestions, etc. It's a dense patch
16:32:00 <hogepodge> markvoelker: cool.
16:32:02 <markvoelker> I Think I just need to get in a couple more cycles on those and then push them up in gerrit
16:32:16 <markvoelker> Overall though I think it captures the essence of what we need
16:32:22 <hogepodge> I'd like to get cranking on a schema and conversion of existing guidelines as soon as possible, in part to help refstack
16:32:37 <hogepodge> yeah
16:32:53 <hogepodge> should match up with the extensions doc you sent up last week too
16:33:23 <markvoelker> hogepodge: Also, one question that crossed my mind this week...assuming we get it done in time, do we want 2017.08 in 2.0 format?
16:33:40 <markvoelker> We can cross that bridge if/when we come to it, but just occurred to me last week that it might be ready in time.
16:33:52 <markvoelker> I suspect not though, b/c we need to give RefStack time to adjust too....
16:34:06 <markvoelker> I'm open to suggestion though. =)
16:34:28 <hogepodge> I'd like us to transition as quickly as possible, but only if we feel ready.
16:34:28 <eglute> if we want to ask for approval for new programs in august, then we would need to
16:34:29 <eglute> but
16:34:37 <eglute> we can ask for approval in september board meeting
16:34:49 <hogepodge> eglute: +1, yes, I'd like something to be available for that too
16:35:19 <eglute> if we aim for september approval, then no need to rush 2017.08 update
16:35:41 <markvoelker> Ok, let's see how far we can get and we'll make a decision as we get close to time
16:35:42 <Rockyg> Wouldn't a new program be advisory for 2017.08 no matter what?
16:36:00 <markvoelker> In the meantime:
16:36:07 <hogepodge> Rockyg: yes, the plan is advisory in August and possibly required in January (for extensions)
16:36:12 <markvoelker> #action everyone review schema 2.0 patch
16:36:20 <eglute> Rockyg yes, new program would be advisory, but we can also have a 2017.09 guideline. no restrictions on how many guidelines per year
16:36:34 <hogepodge> Please, be extremely critical. I want it to be correct.
16:36:38 <Rockyg> Ah, true.  Thanks, eglute
16:36:53 <hogepodge> We don't get a second chance at our second chance. :-D
16:37:28 <markvoelker> OK, anything else on this?
16:37:30 <eglute> hogepodge we just increase the number and call it a new chance ;)
16:37:53 <Rockyg> 2.1
16:38:08 <Rockyg> or would it be 2.0.1?)
16:38:33 <markvoelker> #topic Mandatory submission of test results
16:39:30 <markvoelker> I think the bullets in the etherpad say it all, but does anyone want to walk us through this one?
16:39:55 <markvoelker> (I didn't add this item to the agenda =p)
16:40:13 <eglute> i copied over from last week's agenda
16:40:22 <eglute> it is one item we didnt get to discuss in Boston
16:40:42 <eglute> I think we need to work on making test results required
16:40:52 <hogepodge> I don't know if I should lead it, but I can make my argument.
16:41:05 <markvoelker> I'll take a stab at it then:
16:41:15 <eglute> go for it hogepodge
16:41:27 <hogepodge> It's extremely easy to falsify test results. Keeping the output of the test run allows for forensics in case something is wrong.
16:41:29 <markvoelker> Basically we've been discussing making it mandatory that vendors submit results of all tests to RefStack rather than just required ones
16:41:47 <hogepodge> Ah, I guess I have a different view of the topic. :-P
16:41:49 <markvoelker> One benefit being that we get more data about what's supported and not across products
16:42:16 <zhipeng> do we certify on all tests as well ?
16:42:34 <Rockyg> zhipeng, no.  Just the guidelines
16:42:44 <eglute> zhipeng no, we would certify only on required tests.
16:43:33 <markvoelker> Right, the idea here was just that when we're considering including a new test in the next Guideline, we could go see that "wow, half of vendors who are currently OpenStack Powered failed that test, maybe we should look at it a little closer..."
16:43:52 <eglute> yeah, that would be great if we had that data
16:45:04 <markvoelker> Another requirement we'd been discussing was what hogepodge was just alluding to, though I think we hadn't decided exactly what artifacts we'd expect to be submitted
16:45:13 <markvoelker> (hogepodge maybe you remember better than me?)
16:45:39 <hogepodge> the testrepository data
16:45:56 <hogepodge> or the tempest.conf (without the accounts.yaml)
16:46:00 <hogepodge> both were discussed
16:47:38 <annegentle> o/ heyo super late since I was double-booked, need to read scrollback :)
16:48:03 * eglute waves to annegentle
16:48:40 <Rockyg> hogepodge, what's the most likely ways the tests are gamed and what do we need to identify that?
16:49:34 <hogepodge> Rockyg: anyone can just send up a json list of passed tests and get a passing result, without actually running the tests
16:49:41 <hogepodge> it's a major design flaw of the program, imo
16:49:46 <markvoelker> Rockyg: I think that's hard to pin down...at the end of the day we're talking about plaintext files that can be falsified with any text editor.
16:50:07 <hogepodge> having the testrepository data makes generation of the json file independently verifiable
16:50:10 <markvoelker> We've relied mostly on legalese (in the trademark agreement) and community policing in the past
16:50:14 <hogepodge> (by the foundation)
16:50:48 <hogepodge> it also helps with debugging (I get a lot of requests about "why is my cloud not passing?")
16:50:59 <Rockyg> i bet ;-)
16:51:12 <eglute> i think we have sufficient reasons to want all the results
16:51:32 <eglute> what would need to update to start asking for them?
16:51:39 <hogepodge> for example, one vendor sent in results where they modified tempest to remove a skip decorator. My conversations with them would have been a lot easier if I could see that the test actually ran in the output.
16:51:41 <Rockyg> So, what's the average size of the data set?  would we ccompress it or???
16:51:57 <markvoelker> Right, there were some concerns about what might be exposed there...e.g. passwords (in the configs) or sensitive stuff in the logs
16:52:06 <hogepodge> I can't speak to that in this meeting, but it's a text file.
16:52:10 <markvoelker> I think those are solvable
16:52:18 <hogepodge> if a password is exposed it's a tempest bug
16:52:32 <markvoelker> We probably just need to have somebody play point on pinning that stuff down and putting together a proposal
16:52:43 <Rockyg> I remember back a ways where refstack discussed stripping sensitive info before send full results.
16:52:57 <hogepodge> data would not be publicly available, and access to the database would be controlled by the infra team in the same way that all sensitive data is managed (with refstack entrypoints for foundation users)
16:53:11 <Rockyg> We might even have the bundler code in the refstack repo
16:53:57 <Rockyg> So, insead of a jason file, the testrun log was expected to be sent.
16:54:46 <markvoelker> hogepodge: would you be willing draft up a proposal on this so we can move toward ironing out details and formalizing it?
16:55:00 <hogepodge> markvoelker: sure
16:55:21 <markvoelker> #action hogepodge to work on drafting a proposal on what data we should require vendors to submit
16:55:29 <annegentle> hogepodge you're the man!
16:55:56 <eglute> thank you hogepodge!
16:56:00 <markvoelker> Ok, anything else on this topic today?
16:56:07 <eglute> not from me!
16:56:34 <markvoelker> #topic Vertical programs
16:56:49 <markvoelker> Just a couple of minutes left, so I'll keep it short
16:57:12 <markvoelker> See above for link to draft of vertical programs
16:57:43 <markvoelker> In the meantime, to get something a little more concrete we need to start putting together a very preliminary list of capabilities for an NFV program
16:58:19 <markvoelker> This is basically just a first pass, so we can flush out some of the real issues and figure out how to deal with them
16:58:29 <markvoelker> If anyone wants to help with that, let me know.
16:59:06 <markvoelker> And with that we're down to just a minute left...any last words folks?
16:59:17 <eglute> thanks markvoelker, i will re-read it again
16:59:18 <eglute> thanks everyone!
16:59:32 <eglute> and thanks markvoelker for writing up add-ons/vertical programs draft
16:59:42 <hogepodge> thanks everyone!
16:59:43 <Rockyg> thanks and review!  I will do it now
17:00:06 <markvoelker> #endmeeting