16:00:12 <markvoelker> #startmeeting interopwg
16:00:15 <openstack> Meeting started Wed Jun  7 16:00:12 2017 UTC and is due to finish in 60 minutes.  The chair is markvoelker. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:19 <openstack> The meeting name has been set to 'interopwg'
16:00:23 <Rockyg> o/
16:00:30 <markvoelker> #chair hogepodge
16:00:31 <openstack> Current chairs: hogepodge markvoelker
16:00:35 <markvoelker> o/
16:00:38 <hogepodge> o/
16:00:53 <markvoelker> #topic agenda
16:00:58 <markvoelker> #link https://etherpad.openstack.org/p/InteropVertigo.3 Today's Agenda
16:01:15 <markvoelker> Feel free to tack on anything we're missing from the agenda this week folks
16:01:24 <catherineD> o/
16:01:24 <zhipeng_> o/
16:01:25 <luzC> o/
16:01:45 <markvoelker> Without further ado...
16:01:50 <markvoelker> #topic 2017.08 Guideline
16:02:13 <markvoelker> I think we're about ready to cut the new JSON document.
16:02:27 <mguiney> o/
16:02:33 <markvoelker> I'm happy to do that this week, but there are one or two patches that we might want to decide on first in order to avoid refactoring
16:03:09 <markvoelker> We'll talk through both of those in just a moment, but barring any issues getting those decided: any objections to creating the 2017.08 doc this week?
16:04:03 <hogepodge> it looks like the patch for token checking is ready to land in tempest
16:04:08 <hogepodge> thanks to mguiney for that work
16:04:54 <hogepodge> still working on the catalog check patch (although it's moving forward at this point), so we might want to consider the status of that
16:06:16 <markvoelker> I think the token patch probably isn't a blocker for cutting the new JSON doc though, since we don't have a patch against interop yet (waiting for it to land in tempest first).  E.g. no patch = no need refactoring, just submit a new patch against 2017.08.
16:06:43 <markvoelker> Make sense?
16:06:48 <hogepodge> sure, np, just wanted to make everyone aware of the status of that work. thanks
16:06:54 * mguiney nods
16:06:54 <markvoelker> ++
16:06:57 <mguiney> makes sense
16:07:36 <markvoelker> Ok then, hearing no objections I'll take an action for cutting the new doc this week.  I'll hold off a few days to see if we can get one or two more patches decided, which we'll discuss shortly
16:07:58 <luzC> ++
16:08:03 <Rockyg> ++
16:08:13 <markvoelker> #action mvoelker Create 2017.08 doc this week (hopefully after we decide on 462860& 460372)
16:08:29 <markvoelker> Which brings us to...
16:08:35 <markvoelker> #topic Outstanding patches
16:08:53 <markvoelker> #link https://review.openstack.org/#/c/467493/ non-admin token validation test
16:09:20 <markvoelker> We'd discussed previously that we'd be ok with this being a late-ish addition to 2017.08 since we've been discussing it for a while now
16:10:44 <markvoelker> We could use a couple of Tempest core reviews on it though...anyone we can poke/buy a beer for to help out?
16:11:25 <mguiney> who would be the best people to poke, you think?
16:11:30 <Rockyg> I'm sure there are some that could be poked.
16:11:45 <luzC> I'll ping Matt
16:12:00 <Rockyg> just slide over to the qa channel and post the link and ask.
16:12:08 <mguiney> ahhhh yes
16:12:22 <mguiney> i actually already did that, but it might have been too late in the day
16:12:29 <luzC> cool
16:13:07 <markvoelker> Ok.  Let's see what happens then.  Anything else to discuss on this patch?
16:13:31 <markvoelker> Ok, moving along then...
16:13:36 <Rockyg> matt should be on at the moment.  11am where he is
16:13:41 <markvoelker> #link https://review.openstack.org/#/c/467528/ Catalog standardization test
16:14:20 <markvoelker> I don't think this one is holding up 2017.08 work either.
16:14:31 <markvoelker> We'd discussed that this is something we probably want to advertise as soon as possible though
16:14:59 <markvoelker> So even if it doesn't make it into 2017.08 we might include some verbage to the BoD about it as a future capability or something
16:15:44 <markvoelker> Personally I'd like to have it sooner rather than later, but given the redness of CI it may just need a little more bake time. =)
16:16:17 <markvoelker> Thanks again for working on this mguiney!
16:16:42 <markvoelker> Any updates or things we need to talk about on this one?
16:16:43 <mguiney> yeah, it just needs some more debug
16:17:13 <mguiney> i also have an admin version in the works, but i want to work out the kinks on this one so that the next one doesnt have so many quirks in the process
16:17:21 <mguiney> but that's about all
16:17:55 <markvoelker> Ok then, on to...
16:18:14 <markvoelker> #link https://review.openstack.org/#/c/458716/ Create tools to increase ease of scoring
16:18:48 <markvoelker> I actually haven't had time to play with this yet as I've been away for a bit, so I'll sign up to get that done this week
16:19:15 <markvoelker> luzC had some comments--were those addressed in the latest patchset?
16:19:33 <markvoelker> Also we'd talked about adding inline help
16:20:14 <markvoelker> AGain, this is not particularly urgent since we're more or less done with scoring for a while, so it's fine for this to hang out for a bit if you are otherwise occupied mguiney
16:20:23 <luzC> yes, I think the comments were addressed and I wanted to test it out, I'll review it later today
16:20:32 <markvoelker> luzC: great, thanks
16:20:37 <mguiney> ahhhh yes, i forgot to do that bit of cleanup work
16:20:54 <mguiney> i will add that right after meeting
16:21:06 <markvoelker> Sweet
16:21:18 <markvoelker> Ok, anything else on this?
16:21:55 <markvoelker> Hearing nothing, onward to...
16:22:10 <markvoelker> #link https://review.openstack.org/#/c/462860/ Add Aliases For VolumeV2 Test Cases Part 1
16:22:25 <zhipeng_> sorry for the lack of progress last week
16:22:42 <zhipeng_> was boggled with many other stuff, but I think I could wrappit up this week :)
16:23:02 <zhipeng_> catherine kindly pointed out an example, and I think I know what went wrong
16:23:23 <zhipeng_> will push a update later today
16:23:35 <markvoelker> zhipeng_: No worries, happens to the best of us.  We can land this after 2017.08 is done since it's an alias, but would just save you a little refactoring effort if we get it done first.
16:23:36 <markvoelker> Thanks!
16:23:40 <zhipeng_> again thx for lucZ and catherine's comment
16:23:49 <zhipeng_> yep understood :)
16:24:02 <markvoelker> Ok, anything further on this one?
16:24:29 <markvoelker> #link https://review.openstack.org/#/c/463944/ Remove test_delete_active_server
16:24:49 <markvoelker> Mostly just an FYI on this one: since last week it was abandoned since it looks like we're keeping the requisite tempest test after all
16:25:25 <zhipeng_> cool
16:25:40 <markvoelker> #link https://review.openstack.org/#/c/460372/ Flagging Regarding Public Cloud Subnet
16:26:20 <zhipeng_> I could make a quick update patch on this one
16:26:26 <markvoelker> In looking at this one, it looks like the rationale for flagging/removing the tests is more about the fact that the ports being created/updated are in different subnets than about the port update itself
16:26:43 <markvoelker> In which case it may just be a case where we need to refactor the tests a bit since the port update is the bit we care most about I think
16:26:48 <zhipeng_> should I keep the flagging for the next.json ?
16:26:53 <zhipeng_> this is the only question I have now
16:27:04 <zhipeng_> instead of deletion , right ?
16:27:29 <markvoelker> zhipeng_: That would be my thinking, yes (and we need to create a bug to work on the test).  Anyone else have thoughts here?
16:27:43 <zhipeng_> okey understood :)
16:28:06 <zhipeng_> this will be a quicker update than the previous one :P
16:28:18 <Rockyg> Sounds right to me.  and test of the feature is important
16:29:20 <markvoelker> Ok.  Just FYI, it might be useful to browse around and see if there are already other tests we should be using for this, too.  I suspect not much has changed, but you never know.
16:29:25 <markvoelker> Ok, anything else on this one?
16:30:02 <markvoelker> OK, anything else on outstanding patches in general?
16:30:03 <zhipeng_> got it
16:30:58 <markvoelker> #topic Discussion of Tempest tests, plugins, etc
16:31:28 <markvoelker> #link http://lists.openstack.org/pipermail/interop-wg/2017-June/000147.html Mailing list thread on many discussions around Tempest and interop and plugins
16:31:56 <markvoelker> I've been away recently so I'm still working on getting back up to speed, and there's a lot to digest here.
16:32:20 <markvoelker> rockyg: Would you like to give a quick intro for folks who are new to this discussion(s)?
16:33:16 <Rockyg> Let
16:33:33 <Rockyg> s see if I can explain some of it.
16:34:02 <Rockyg> The whole thing started with tempest in tree tests and tempest plugins.
16:34:51 <Rockyg> Then someone interested in the  IWG addon program commented on needing new tests added in tree but QA not allowing it.
16:35:27 <Rockyg> Then all sorts of fud about 30-40 TM programs and not enough QA bodies, yada, yada.
16:36:13 <Rockyg> And some pointers to TC resolutions on what is "base" or "core" or "approved" vs what is already in IWG.
16:37:08 <Rockyg> Nostly it points out that devs are very disconnected from IWG, we aare disconnected from them and QA is really strapped.
16:37:40 <Rockyg> Some bad assumptions on the part of some in dev. and maybe some from us.
16:38:14 <zhipeng_> there were discussions on heat tempest plugin retirement from in-tree (move to be maintained by heat team), also related right ?
16:38:33 <Rockyg> I think we need to let them know whether we can work with the tempest plugin model.
16:39:24 <Rockyg> That is very related.  They are also considering moving all projects to plugin model and the question is how to ensure quality of IWG test
16:39:29 <catherineD> We need clearify whether the tests for the addon is required to be in Tempest...
16:39:42 <Rockyg> that, too.
16:39:43 <markvoelker> Ok.  Maybe the easiest thing to do is for me to send out a broadcast to interwg, openstack-dev, and whoever else about our current thinking on those things, and let you all chime in too?
16:39:46 <hogepodge> As I understand it. Anything considered core, that is, corresponds to OpenStack Powered Platform, will be required to have tests in tree in Tempest
16:39:59 <hogepodge> Extensions will remain in-tree.
16:40:22 <catherineD> if everything (current program and addon ) is  requried  to be in Tempest ... the not much the plugin can be useful
16:40:28 <markvoelker> hogepodge: that is the current state of affairs due to the TC resolution, yes.  It does not govern add-ons or vertical since those were not even a twinkle in our eyes when the resolution was drafted.
16:40:31 <hogepodge> The TC members I've spoken with have said they will update the TC resolution about in tree tests to reflect the new schema style.
16:40:32 <Rockyg> I think we should get our position straight then hold a join meeting in openstack-meeting-crossproject that is well advertised
16:40:56 <zhipeng_> +1
16:41:21 <hogepodge> when I say extensions will remain in tree, I mean in-project
16:41:31 <catherineD> extenstion and addon are the same thing right?
16:41:47 <hogepodge> yes, I'm using the name extension because it's a better name :-D
16:41:50 <markvoelker> My current thinking is that the idea of add-on program tests being required to live in the tempest tree doesn't jive well with reality, so in-project-tree is fine
16:42:14 <hogepodge> I don't think anyone disagrees with that.
16:42:19 <markvoelker> Great
16:42:32 <markvoelker> Same for verticals...I'm not opposed to those living outside tempest at all.
16:42:33 <Rockyg> Yea.  The TC is willing to work with us.  We just have some really stretched hin QA folks that are still working to accept the new reality
16:42:46 <Rockyg> ++ on all of that.
16:42:50 <catherineD> I think we should standardize on our terms. It would be less confusiing to the communitee
16:42:52 <hogepodge> I think QA is on board with this too, tbh
16:43:13 <Rockyg> I also think there is a lack of awareness that some here are doing test vetting and test writing work.
16:43:38 <Rockyg> If we are vetting the tests, too, then QA might feel better about the in-project placements
16:44:16 <hogepodge> Test lists for extensions should come from project leaders, which is how I've been handling it
16:44:31 <Rockyg> QA is just so overworked at the moment, that anyone proposing new work for them makes them react negatively at first
16:44:57 <luzC> also the change that is happening for the tempest plugins is that now the plugin will live in its own repo (outside the project repo)... due to packaging issues, and because tempest is branchless, etc. etc
16:45:25 <luzC> so is it ok for IWG that test cases live on the project plugin repository
16:45:28 <catherineD> Is  this our position: extension (add-on) and the current program are governed by the TC resolution .. vertical is not as such vertical tests can use plug in?
16:45:41 <Rockyg> This could also give us back versioned test ;-)  if they are in project repos
16:45:47 <luzC> not the project tree itself?
16:45:51 <markvoelker> hogepodge: ++, I think one of the big intents for add-on^H^H^H^H^H extension programs is to get the individual projects thinking about what interop means for their projects rather than trying to delegate it to a central body
16:46:24 <Rockyg> ++
16:46:41 <markvoelker> catherineD: I'm of the opinion that extensions an verticals are both ok to be in-project-tree rather than in-tempest-tree.  Neither are govered by the existing TC resolution anyway.
16:47:25 <zhipeng_> ++
16:47:34 <Rockyg> I'm also thinking we should maybe have a *real* session at PTG getting all the devs interested onboarded wih what we do and iron out any bumps
16:47:36 <catherineD> markvoelker: if we all agree ... those should be document somewhere
16:47:48 <Rockyg> Resolution.  Like the TC
16:47:55 <catherineD> so we can use that as reference to clear up the confusion
16:48:17 <Rockyg> Or, really, somewhere in our process docs or hacking....
16:48:27 <catherineD> Rockyg: ++
16:48:41 <markvoelker> Yeah, I'm increasingly thinking that another ML thread is actually probably the wrong way to do this.  Maybe I should just submit a doc patch to our repo outlining what we're thinking and then send out ML messages inviting comment.
16:48:42 <Rockyg> I think Process docs.  Chapter on extensions
16:48:53 <Rockyg> ++
16:49:18 <markvoelker> Any objections to that plan?
16:49:25 <Rockyg> Having a base that is not changing to reference will help the list discussion
16:49:26 <catherineD> My impression is that hogepodge: would like the extension tests (chosen by the project team) but reside in tempest tree... I may be mistaken
16:49:55 <Rockyg> I think hogepodge would be okay with either.
16:50:02 <hogepodge> catherineD: no, extension tests can live anywhere.
16:50:21 <Rockyg> thanks, hogepodge
16:50:55 <catherineD> hogepodge: thanks for clarifying ...
16:51:43 <markvoelker> Ok, I will do my best to draft up something later today or tomorrow and send it out then.
16:52:09 <markvoelker> #action markvoelker Draft up current thinking on extension/vertical programs, post doc patch, send emails
16:52:21 <catherineD> next do we care whether the plugin code is in the project tree or on its own repos ... personally, I do not mind either way .. and that is one of the discussion point too
16:52:47 <Rockyg> thanks so much!  we will need to respond quickly to review it if we want any honing before the ML discussion.
16:53:04 <hogepodge> qa teams wants tests in their own repos out of project tree
16:53:16 <hogepodge> but some projects are pushing back against that
16:53:45 <Rockyg> If it's in project tree, the tests will be versioned.  If plugin tree, not necessarilyl
16:53:51 <markvoelker> catherineD: from a refstack perspective we can handle either, right?
16:54:11 <hogepodge> It's just tempest plugin either way from what I understand. That's the refstack requirement.
16:54:43 <catherineD> yea we can handle plugin and really do not care the location ...
16:54:48 <markvoelker> hogepodge: exactly.  There's a bit of a mechanical concern in terms of "how do I download the tests" (e.g. which repos to clone) but that seems trivial in my mind.
16:54:50 <luzC> I think for reftack is better to have it test cases separated otherwise when installing the plugin you have to install the project dependencies, etc
16:55:09 <markvoelker> So I think my opinion here is that I
16:55:24 <catherineD> luzC: ++
16:55:25 <markvoelker> 'm pretty unopinionated about whether they live in a separate repo or not. =p
16:56:04 <markvoelker> It's really a question that has more impact on the proejcts and QA than on us, so feels like a thing they should hash out and let us know what they decide.
16:56:57 <Rockyg> ++
16:57:09 * markvoelker glances at clock
16:57:34 <markvoelker> Ok, in the interest of time I'll propose that I write it up to that effect and people with stronger opinions (which may be you!) can add comments to the review
16:57:45 <Rockyg> ++
16:57:56 <hogepodge> markvoelker: this is related to the efforts https://review.openstack.org/#/c/430556/
16:58:00 <luzC> what about the quality of the test cases, or tracking the changes on different repos... is that still business as usual for us?
16:58:03 <hogepodge> I've resumed work on it
16:58:33 <hogepodge> everyone please review and add comments. I'll start drafting up the official schema and translation of v1 -> v2 after comments have been received
16:58:57 <markvoelker> luzC: I generally think we want the projects to be aware of what criteria we're using and decide on tests for themselves.  We can be a final approval layer, but I would like the projects themselves to be most vested.
16:59:33 <markvoelker> hogepodge: thanks for that.  I am way overdue on this one, will do my best to get on it shortly.
16:59:38 <catherineD> markvoelker: agree .. especially the project team will be the one choosing the tests
16:59:39 <Rockyg> luzC, ++  that's the reason for ptg session
17:00:04 * markvoelker has got to stop spending so much time on planes
17:00:06 <markvoelker> And with that, I'm afraid we are out of time.
17:00:16 <markvoelker> #endmeeting