15:00:14 <bswartz> #startmeeting manila
15:00:14 <openstack> Meeting started Thu Aug 13 15:00:14 2015 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <bswartz> hello all
15:00:18 <openstack> The meeting name has been set to 'manila'
15:00:19 <ganso_> hello
15:00:22 <toabctl> hi
15:00:25 <dustins> \o
15:00:37 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:00:45 <vponomaryov> Hello
15:00:58 <bswartz> also a quick reminder before we get started
15:01:15 <bswartz> we've got 3 weeks until feature freeze, and 1 week until feature proposal freeze
15:01:50 <bswartz> hopefully everyone understand what feature proposal freeze means
15:02:16 <bswartz> ameade: here?
15:02:26 <bswartz> #topic Consistency Groups discussion
15:02:30 <ameade> o/
15:02:58 <ameade> so I just wanted to make sure everyone has been paying attn to https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups
15:03:03 <ameade> It should be up to date
15:03:08 <ameade> and patches will be coming soon
15:03:21 <ameade> one thing I wanted to discuss is doing this as an extension
15:03:23 <bswartz> ameade: that's the wiki you shared at the midcycle right?
15:03:28 <ameade> bswartz: yeah
15:03:59 <bswartz> ameade: extensions are bad! (according to jaypipes)
15:04:07 <xyang1> Hi
15:04:24 <bswartz> we've discussed the possibility of marking the APIs as @experimental, once we have microversions
15:04:25 <jaypipes> bswartz: um, yeah, API extensions are baaad, m'kay.
15:04:54 <ameade> I think we should do as an extension for Liberty since that makes deployers able to turn it off and we dont have the experimental stuff all the way implemented
15:05:03 <bswartz> ameade: is there a reason to prefer extensions over microversioned APIs for this feature?
15:05:43 <ameade> we have to get the microversion stuff done and there isn't a lot of time?
15:05:47 <bswartz> ameade: you mean some of the APIs won't even work? or they'll be missing?
15:06:01 <bswartz> ameade: microversions have been in gerrit for a long time
15:06:06 <bswartz> just waiting for merge
15:06:16 <ameade> bswartz: no i mean experimental api decorators dont exist yet
15:06:17 <vponomaryov> ameade: microversions are Ok now, small nits left
15:06:20 <bswartz> https://review.openstack.org/#/c/207228/
15:06:41 <vponomaryov> ameade: you can use it as dependency
15:06:51 <ameade> iirc it doesnt have anything about experimental apis in it
15:06:58 <bswartz> ameade: adding @experimental should be super easy
15:07:16 <bswartz> you're correct that @experimental is not part of microversions currently
15:07:40 <ameade> also, being an experimental api means we have to have generic driver support right?
15:07:48 <bswartz> but I propose adding it as a follow on -- probably less than 100 lines of new code to do that
15:08:37 <ameade> what do we gain making it experimental?
15:08:38 <bswartz> ameade: no -- whether it's experimental or an extension has nothing to do with generic driver support
15:08:48 <dustins> bswartz: What's the benefit of a follow on as opposed to the @expermiental decorator?
15:08:57 <bswartz> @experimental would essentially be a documentation thing telling users that it's NOT guaranteed to exist in future version
15:08:59 <bswartz> versions
15:09:31 <akerr> bswartz: don't you get that same implication with an extension?
15:09:52 <bswartz> akerr: yes but extensions are not versioned and we want to kill them with fire
15:10:35 <bswartz> and currently we have some "extensions" which are part of the core API, so we can't actually remove them without causing breakage
15:10:57 <bswartz> I think the problem is that extensions are *supposed* to be optional but they're actually not, so they just cause more problems than they solve
15:11:10 <bswartz> that's why I want to eliminate them the sooner the better
15:11:19 <akerr> but this would be completely optional, which is why we were asked to make it an extension in the first place
15:12:02 <bswartz> I get that the feature will be optional for backends to implement
15:12:18 <bswartz> backends can simply not support the API if they don't want to
15:12:38 <ameade> but that means users can hit an endpoint that will never work
15:12:46 <bswartz> however the REST APIs will be part of manila, in the next microversion after 1.0
15:13:10 <bswartz> ameade: that will be true when the feature is not experimental too
15:13:41 <bswartz> if someone deploys manila with a non-cg-supporting-backend a year from now, all of the CG APIs will just return not supported
15:13:53 <bswartz> that's by design I thought
15:14:12 <ameade> i think ideally we'd have versioned extensions and it would always be an extension
15:14:31 <ameade> yeah they could always change the policies like in cinder if they want to block then endpoints that way
15:14:52 <bswartz> I don't think anyone has volunteered to implemented versioned extensions
15:15:07 <bswartz> it was the lack of such a feature that drove us to want to eliminate extensions in the first place
15:15:23 <bswartz> and microversions is the path forward for us and the rest of openstack
15:15:26 <ameade> that's why i said ideally :P
15:15:56 <ameade> abusing microversions is scary in and of itself by the way but that's another discussion
15:16:02 <bswartz> abusing how?
15:16:37 <bswartz> I believe that the combination of microversions and experimental tags should allow us to support everything we need to going forward
15:17:16 <ameade> what's the logical difference between an extension and an experimental api?
15:17:38 <bswartz> experimental APIs are versioned
15:17:41 <akerr> why do experimentals need versions?  They by definition don't need to maintain compatibility
15:17:53 <toabctl> there could also be a "extension" tag instead of experimental.then youhave versioned extensions.
15:18:07 <bswartz> so if we have version 2.15, and we want to add a new experimental API, we increment the version to 2.16 and add it, but with the @experimental tag
15:18:33 <bswartz> when we decide that it was a bad experiment, we increment the version to 2.17 and remove the API, or replace it with a better version
15:18:55 <bswartz> extensions give the client no way to know what version they're talking to
15:19:08 <akerr> experimental implies unstable, whereas extension implies optional.  They're not interchangeable
15:19:17 <bswartz> you have to rename the extension to change it in any material way
15:19:29 <bswartz> akerr: I agree
15:19:46 <cknight> akerr: +1
15:19:49 <toabctl> akerr: yeah. true
15:19:58 <dustins> akerr: +1
15:20:11 <vponomaryov> akerr: killer argument! =)
15:20:11 <bswartz> My proposal is that no part of the manila API should be optional -- the API is the API, and if a particular installation of Manila can't support some APIs there better be a way for the client to find that out
15:20:20 <ameade> akerr: +1
15:21:09 <bswartz> that's why I'm very keen on things like public capabilities in the share_type for stuff like snapshots (now that they're optional), for share_server/share_networks (added in kilo) and for share replication (planned for Mitaka)
15:21:10 <ganso_> akerr: +1
15:21:31 <ameade> at the midcycle we were just gonna do it as a core feature i thought, then it got switch to an extension, and now we want to switch it back
15:21:43 <bswartz> My view of extensions is that they should be things out of tree
15:22:13 <cknight> An experimental tag is a logical extension to the microversions work.  I would volunteer to do that, but I doubt I could do it by FPF.
15:22:22 <ameade> i want a FFE if we have to go that route
15:22:23 <bswartz> if an administrator wants to "extend" his Manila instance with special sauce he wrote, then he should be able to extend the API, but not contribute that upstream
15:23:23 <bswartz> okay
15:23:49 <dustins> ameade: +1, it seems like a lot of effort to get done before FF
15:23:50 <bswartz> ameade: isn't changing things from an extension to a core API a relatively small change?
15:24:07 <ameade> bswartz: gotta wait on the experimental tag
15:24:14 <ameade> gotta change the client and the tempest tests
15:24:38 <bswartz> okay I'm confused about why @experimental is more than a few minutes of work
15:24:57 <bswartz> but we can table that and get back to consistency groups
15:25:20 <bswartz> other than how the APIs are presented to end users, is there anything we need to discuss about consistency groups?
15:25:56 <ameade> i've got nothing else, the design doc is up there and the code is coming soon
15:26:24 <bswartz> ameade: given the FPF date, I suggest pushing it in its current form, even if it's currently implemented as an extension
15:27:02 <bswartz> we can figure out the API side in the next 2 weeks and change it as needed based on microversions, and experimental
15:27:06 <ameade> bswartz: that sounds good, that should only affect the API patch anyways, the db and scheduler changes should be fine as is
15:27:20 <bswartz> yeah I really think that's a minor concern
15:27:46 <bswartz> the point of the FPF date is to give reviewers time to review and provide feedback
15:27:53 <cknight> bswartz: OK, I'll work on @experiemental.  Microversions should merge this week (hopefully).
15:27:54 <bswartz> and most of the code has nothing to do with the REST APIs
15:28:19 <bswartz> well let's talk about experimental real quick since we're all here
15:28:26 <bswartz> #topic experimental APIs
15:28:33 <bswartz> so my proposal is simple
15:28:49 <bswartz> add a new empty decorator called @experimental
15:29:07 <bswartz> it would layer on top of microversions
15:29:50 <bswartz> basically if an API has the tag, it documents to users that we make no guarantees about that API existing or changing in future versions
15:30:15 <bswartz> once we're happy with an API, we can simply remove the @experimental tag and it becomes set in concrete like the existing rest API
15:30:40 <bswartz> adding/remove @experimental status on any API would cause the microversion to bump
15:31:27 <bswartz> this gives us a path to deprecate certain APIs without deprecating the rest of the API
15:31:52 <bswartz> of course any API can be deprecated by raising the minimum microversion that we support
15:32:35 <bswartz> but for reasons that are hopefully obvious to everyone, we will very rarely raise the minimum version
15:32:51 <bswartz> any thoughts/concerns about this proposal?
15:32:55 <dustins> bswartz: so we could wind up using the @experimental tag as a way to deprecate APIs?
15:33:05 <cknight> dustins: no
15:33:21 <bswartz> dustins: the thinking is that it allows us to release new features and obtain user feedback without commiting ourselves to an implementation
15:33:34 <bswartz> the hope is that it results in features shipping faster
15:33:39 <dustins> cknight: Okay, I thought so, just wanted to clarify my understanding
15:33:53 <cknight> bswartz: The @experimental tag is an implementation detail that may have a simpler solution, but everything else makes sense.  I volunteer.
15:33:59 <bswartz> because we can avoid the step where we argue for 3 months about the exact API implementation
15:34:17 <dustins> cknight: +1
15:35:21 <bswartz> anyone against experimental APIs?
15:36:02 <bswartz> okay!
15:36:30 <vponomaryov> bswartz: wait
15:36:36 <bswartz> vponomaryov: ?
15:36:49 <vponomaryov> bswartz: if we make some tmporary thing influence our versions
15:37:06 <vponomaryov> bswartz: then after its removal we get broken versions
15:37:25 <bswartz> vponomaryov: so the proposal for microversions is to increment the version with *every* change
15:37:37 <vponomaryov> bswartz: right, but keeping them always
15:37:40 <bswartz> we could have possibly 2 dozen microversion bumps in a single release
15:38:00 <cknight> bswartz: correct.  Nova has seen about 12 in Liberty.
15:38:08 <bswartz> liberty may be version 1.5 and mitaka could be 1.25
15:38:19 <cknight> http://docs.openstack.org/developer/nova/api_microversion_history.html
15:38:53 <bswartz> so there will inevitably be some microversions with very few or no actual users
15:39:02 <bswartz> the point of microversions is that that's harmless
15:39:03 <vponomaryov> bswartz: then each release should not have any "experimental things
15:39:12 <vponomaryov> bswartz: only in dev time
15:39:23 <cknight> bswartz: most users will send nothing and automatically get 1.0
15:39:38 <bswartz> vponomaryov: there's 2 things to consider
15:39:39 <cknight> bswartz: but as we add new stuff, we'll bump the version requested by manilaclient
15:40:01 <cknight> bswartz: so CLI users follow the latest by default
15:40:18 <bswartz> vponomaryov: microversions is designs to support people who run bleeding edge -- some people run unreleased code in production -- it's not recommended but it's supported
15:41:14 <bswartz> vponomaryov: releases are important because we create stable branches as those places and most people end up running the stable branch instead of master, but if experimental features never make it into stable branches then the value of them is lost
15:41:21 <vponomaryov> bswartz: I mean following: if we removed something, then server with latest code would not know some part of "supported" version
15:42:26 <bswartz> I believe it's important to have experimental features in the stable branches because that's how we get actual user feedback on new features that are evolving
15:42:47 <bswartz> it's not realistic to expect every new feature to get designed and implemented within a single 6 month release cycle
15:43:22 <bswartz> and I see that problem causing paralysis in other projects where "big" stuff never gets done
15:43:56 <bswartz> I want to enable big stuff to happen over multiple releases
15:44:23 <bswartz> and @experimental is the key to enabling that in my view
15:44:47 <cknight> bswartz: +1  Let's do big stuff.
15:44:47 <bswartz> btw the Linux kernel has experimental feature too, and they get released and later removed/revised
15:45:40 <bswartz> perhaps this topic deserved a mail thread
15:45:44 <bswartz> deserves*
15:46:06 <xyang1> Will experimetal not get a version number until it becomes supported?
15:46:15 <bswartz> I think people still don't understand the full value of microversions
15:46:34 <bswartz> xyang1: experimental APIs will be versioned along with everything else
15:46:55 <xyang1> Then we could have gaps?
15:46:59 <bswartz> remember that the microversion for the _whole_API_ increments every time you chance anything in any REST API
15:47:15 <xyang1> If we stop supporting it
15:47:26 <bswartz> only for experimental stuff
15:47:28 <cknight> xyang1: that's important, because experimental APIs can change between versions as we conduct the experiments!
15:48:09 <toabctl> so as I understand it: if we remove an as experimental declared api, we increase the microversion and that's it.
15:48:10 <bswartz> the vast majority of REST APIs will not change
15:48:18 <bswartz> toabctl: yes
15:49:01 <vponomaryov> bswartz: and codebase of server will not have support of these versions
15:49:11 <vponomaryov> bswartz: xyang1 asked about it as "gaps"
15:49:30 <vponomaryov> bswartz: that were used for this "temporary" feature
15:49:33 <toabctl> vponomaryov: odebase of server will not have support for the experimental feature, but still for the version
15:49:35 <bswartz> with microversions it's completely legal to remove an API and increment the version -- callers of the new version will see that API is gone -- however if callers request the older version, then the server is still obligated to support the API as it was in that version
15:50:02 <bswartz> @experimental relieves the requirement to continue supporting the old API after it's removed
15:50:06 <vponomaryov> bswartz: but server will not have code!
15:50:17 <bswartz> vponomaryov: with microversions it actually will
15:50:24 <ameade> this is what i mean by scary
15:50:32 <xyang1> So what I mean is we could support 1, 2, and 5, but not 3 and 4?
15:50:36 <toabctl> bswartz: so you want to have experimental code even if it was removed with a new version?
15:50:47 <akerr> so the code base will be bloated with old version of implementations everywhere?
15:50:49 <bswartz> vponomaryov: with microversions you can never remove any old APIs until you increase the minimum microversion supported by the server
15:50:52 <ameade> if you increment micro versions for every little change then you have crazy code complexity
15:51:05 <bswartz> akerr: Yes -- and that's no worse than what we have today
15:51:14 <bswartz> ameade: no not really
15:51:29 <bswartz> ameade: because everything that didn't change doesn't need to be touched
15:51:30 <ameade> bswartz: elaborate
15:51:48 <bswartz> I think sevaral of you may have missed cknights explanation of microversions at the midcycle
15:52:02 <bswartz> too bad we didn't record that
15:52:07 <akerr> that sounds like a maintenance nightmare to keep all the old implementations around
15:52:20 <ameade> even if it's just lots of little parts of things
15:52:28 <bswartz> akerr: yes, and what we have now is also a maintenance nightmare
15:52:43 <vponomaryov> bswartz: so, is there really enough gain of microversions in that case?
15:53:06 <cknight> vponomaryov: yes, because we can evolve APIs as needed.  that's really hard now.
15:53:11 <bswartz> okay, I sense that the microversions can of worms has opened again
15:53:19 <toabctl> :-)
15:53:25 <akerr> how far back do we support the apis?  Will there be tempest tests for all supported versions?
15:53:30 <ameade> i am only concerned with abusing them, if we change something from returning a 403 to a 409 or something then that means just a little bit of code complexity
15:53:30 <bswartz> we may need to spend some time educating everyone one what they actually do
15:53:42 <cknight> bswartz: I already did :-)
15:53:44 <toabctl> akerr: until we bump the major version. in that case we can remove old crap
15:53:48 <bswartz> however we had another topic for this week that was actually on the agenda unlike this one
15:53:48 <dustins> cknight: could you hit up the mailing list with a cliff notes version of what you went over at the midcycle for those of us that missed it?
15:54:10 <bswartz> cknight: I'm sorry but a refresher seems needed
15:54:19 <bswartz> I want to give dustins a chance
15:54:28 <cknight> bswartz: The best writeup comes from Nova: #link http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html
15:54:28 <dustins> I'll make it quick :D
15:54:29 <xyang1> I think it is probably better for us to read the doc
15:54:30 <bswartz> #topic Manila Tempest Audit
15:54:42 <bswartz> dustins: sorry you've only got 5 minutes
15:54:57 <dustins> should be all I need (hopefully) :P
15:55:25 <dustins> So, I've been going through the Manila Tempest tests and I'm finding some that violate some of Tempest's best practices
15:55:57 <dustins> One big one being test independence, the shares_actions tests create shares on Class initialization and reuse them in all tests
15:56:14 <dustins> which could be a problem if a test fails and puts the share/etc into a bad state
15:56:15 <bswartz> dustins: now that openstack is encouraging projects to move their functional tests out of tempest and back into the various projects, how will we enforce testing best practices across projects?
15:56:41 <bswartz> are we going to rely on QA liasons in each project keeping the faith?
15:56:53 <dustins> bswartz: Just because they're moving the tests out of Tempest doesn't mean their guidelines are suddenly invalidated
15:56:56 <akerr> i think thats a separate discussion bswartz
15:57:06 <bswartz> yeah but the enforcement will get a lot harder
15:57:30 <dustins> Then we bring that upon ourselves (and I don't mind being that liason)
15:57:31 <bswartz> I'm all for conforming, but openstack-qa doesn't review our tests
15:57:50 <vponomaryov> dustins: it is toooo expensive to create separate share for each test that requires share
15:57:56 <akerr> Lack of outside enforcement isn't a valid reason to lower our standards
15:58:02 <bswartz> dustins: https://wiki.openstack.org/wiki/CrossProjectLiaisons#QA
15:58:07 <dustins> akerr: +100
15:58:37 <ameade> lets focus on specific issues as they come up and try to do a better job
15:58:37 <dustins> vponomaryov: But having multiple tests fail because of an issue in one makes it difficult to find where the true problem lies
15:58:49 <cknight> dustins: +1
15:59:03 <cknight> vponomaryov: How much slower are the gate tests if we create a share per test?
15:59:27 <bswartz> cknight: that's my question too
15:59:30 <bswartz> I worry they'll get 10x slower
15:59:54 <ganso_> they are already displaying timeouts ocasionally
15:59:58 <vponomaryov> cknight: hundreds percent
16:00:18 <dustins> What's the point of doing them faster if by doing so we make the results less useful to the folks that fix the bugs they find?
16:00:20 <ameade> it's easy to find out if dustins writes the patch
16:00:37 <cknight> ameade:  +1 :-)
16:00:38 <bswartz> let's take this tempest discussion over to #openstack-manila, we need to vacate this room
16:00:39 <dustins> ameade: I have no problem with that
16:00:43 <akerr> ameade: +1
16:00:46 <angdraug> o/
16:00:46 <bswartz> I want to keep having this discussion
16:00:48 <vponomaryov> ameade: it will have timeout failure ^_^
16:00:49 <ameade> dustins: not to sign you up :P
16:00:54 <evgenyl> hi
16:00:55 <vponomaryov> ameade: from CI
16:01:00 <akerr> vponomaryov: that can be bumped
16:01:02 <bswartz> #endmeeting