15:00:49 <bswartz> #startmeeting manila
15:00:49 <openstack> Meeting started Thu Feb  2 15:00:49 2017 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:53 <openstack> The meeting name has been set to 'manila'
15:00:54 <bswartz> hello all
15:00:57 <dustins> \o
15:00:58 <vponomaryov> Hello
15:00:58 <tbarron> hi
15:01:02 <markstur> hello
15:01:23 <ganso> hello
15:01:28 <Yogi1> Hi
15:01:31 <bswartz> gouthamr might be a few minutes late -- he's interviewing a candidate atm
15:01:44 <rhagarty> \o
15:02:10 <xyang2> hi
15:02:38 <bswartz> #topic announcements
15:03:07 <bswartz> the target date for RC1 is today
15:03:15 <bswartz> #link https://launchpad.net/manila/+milestone/ocata-rc1
15:03:37 <bswartz> I count 17 open bugs on this list
15:03:38 <gouthamr> hello o/
15:04:17 <vkmc> o/ hiii
15:04:19 <bswartz> today I'm going to start retargetting bugs to pike
15:04:23 <kaisers> hi
15:04:53 <bswartz> because while I'm okay with not doing RC1 today, it does need to get done in the next couple of days
15:05:15 <bswartz> also a reminder
15:05:28 <bswartz> PTG is in 2.5 weeks!
15:05:38 <bswartz> keep adding topics to the etherpad
15:05:47 <bswartz> #link https://etherpad.openstack.org/p/manila-pike-ptg-topics
15:06:04 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:06:14 <bswartz> oops looks like an item was left off of this
15:06:36 <bswartz> fixed
15:06:41 <bswartz> #topic Microversions
15:06:47 <bswartz> rhagarty you're up
15:07:00 <rhagarty> ok...
15:07:19 <bswartz> #link https://review.openstack.org/#/c/420119
15:07:28 <toabctl> hi
15:07:35 <rhagarty> from manila_ui perspective, was hoping to just make single call to get min support based on installed client and service
15:08:37 <rhagarty> I think right now, the call to get microversion support is just a call to manila endpoint
15:09:25 <bswartz> rhagarty: what are you not able to get from the current endpoint?
15:09:29 <bswartz> I'm unclear on what's missing
15:09:35 <rhagarty> what if the installled client is older and doesn't provide support?
15:10:16 <vponomaryov> rhagarty: what is wrong with this -> https://github.com/openstack/python-manilaclient/blob/c0fdf827/manilaclient/api_versions.py#L30
15:10:21 <bswartz> how old?
15:10:30 <rhagarty> user has older client that doesn't support new feature added in 2.33, but endpoint service says we support 2.52.
15:10:36 <vponomaryov> rhagarty: you have versions range in client and get it from server
15:10:39 <bswartz> anything so old that it doesn't speak microversions will be stuck with the v1 API anyways
15:11:03 <vponomaryov> rhagarty: don't see problem at all
15:11:35 <rhagarty> ok - let me take a look...
15:12:19 * bswartz doesn't see a problem either
15:12:27 <rhagarty> and remember, I am talking from manila_ui perspective... goal would be not to show feature. Getting an error is not the best solution...
15:12:32 <bswartz> I hope this is just a misunderstanding about the version API
15:12:47 <rhagarty> ok - thanks
15:13:06 <vponomaryov> rhagarty: it is question of manila UI behaviour in using of existing client functionality
15:13:12 <bswartz> rhagarty: I completely agree -- the UI must be capable of greying out or hiding functionality from the user if it's not going to work
15:13:27 <tbarron> rhagarty: do you want conditional behavior in the UI depending on max microversion supported in the python-manilaclient?
15:13:36 <bswartz> letting the user try something which will always fail is a dumb user experience
15:14:02 <dustins> and infuriating to the user
15:14:03 <rhagarty> tbarron: yes
15:14:29 <tbarron> so we have UI that supports share groups but client that does not for example
15:14:40 <bswartz> what?!?!
15:14:43 <tbarron> customer has
15:14:48 <ganso> tbarron: the client has to support first
15:14:58 <ganso> tbarron: manila-UI uses client, does not make rest calls directly
15:14:58 <bswartz> that particular case should be prevented using proper requirements
15:15:17 <bswartz> the UI should simply have a dependency on the client version that supports what it needs
15:15:24 <tbarron> I'm just trying to get the problem or perceived problem exposed clearly.
15:15:28 <bswartz> it would be idiotic to allow the UI to run with a too-old client version
15:15:42 <rhagarty> bswartz:  how is that prevented?
15:15:48 <ganso> bswartz: like, older than the UI itself?
15:15:50 <tbarron> rhagarty: correct if that's the wrong idea
15:15:53 <bswartz> rhagarty: requirements.txt
15:16:05 <gouthamr> yeah, but the requirements need a fix then.. https://github.com/openstack/manila-ui/blob/master/requirements.txt#L14
15:16:14 <vponomaryov> rhagarty: also strange to hear "too old" when whole openstack has "requirements.txt" file describing all dependencies
15:16:15 <bswartz> everyone understands that packages depend on minimum versions of other packages
15:16:26 <bswartz> if you can't satisfy those dependencies then you can't use the package
15:16:36 <gouthamr> we say anything greater than 1.12.0 is acceptable..
15:16:37 <bswartz> it's not like these things are installed on different machines -- it's all local
15:16:40 <gouthamr> which is weird..
15:16:58 <gouthamr> we've upper constraints for releases, but not milestones when features get in
15:17:00 <tbarron> gouthamr: +1
15:17:00 <vponomaryov> gouthamr: why?
15:17:19 <bswartz> the fix is to bump up he minimum client version
15:17:26 <vponomaryov> gouthamr: while lib keeps being backwards compatible why it is weird?
15:17:33 <bswartz> I can't think of ANY good reason to use an old client version with a new UI version
15:18:02 <vponomaryov> bswartz: own fork?
15:18:12 <bswartz> what fork?
15:18:20 <vponomaryov> of client
15:18:33 <bswartz> who cares about forks
15:18:34 <vponomaryov> downstream fork of upstream client
15:18:35 <toabctl> then it's the problem of the people who did the fork
15:18:45 <gouthamr> vponomaryov: the lib being the manilaclient?
15:18:49 <bswartz> toabctl: +9000
15:18:50 <tbarron> they forked themselves
15:18:52 <vponomaryov> bswartz; you said "think", I mentioned one of possible
15:19:03 <bswartz> I said good reason
15:19:08 <vponomaryov> ok )
15:19:14 <bswartz> people who fork code are on their own
15:19:34 <bswartz> typically forks are done by smart people, such as downstream package maintainers
15:19:36 <toabctl> manila-ui/ocata should simply require manilaclient/ocata
15:19:42 <bswartz> yes
15:19:50 <toabctl> which is done via g-r
15:20:06 <bswartz> I completely agree
15:20:12 <tbarron> that seems right
15:20:33 <vponomaryov> that is why I don't see any reason in proposed feature of UI project such as "support range of microversions at once"
15:20:36 <bswartz> so my stance here is that manila-ui should be able to depend on a completely modern version of manilaclient -- there should never be a version mismatch between those
15:21:05 <bswartz> however manila-ui should be prepared to deal with possibly older or newer versions of the server
15:21:28 <vponomaryov> bswartz: server should be of the same release
15:21:33 <bswartz> microversions exist to help with compabitility between client version and server versions
15:21:42 <bswartz> not between application versions and client versions
15:22:01 <rhagarty> bswartz: agreed
15:22:05 <gouthamr> okay, so we need to bump the requirements file for ocata.. i.e, release python-manilaclient 1.x and require >=1.x
15:22:08 <bswartz> vponomaryov: on most cases yes, but we added microversions to cover the small set of cases where that's not true
15:22:21 <bswartz> gouthamr: +1
15:22:24 <bswartz> okay can we move on?
15:22:32 <rhagarty> bswartz: yes, thanks
15:22:43 <bswartz> #topic Share groups bugs
15:23:16 <bswartz> so in the conversion from consistency groups to share groups, some functionality was lost
15:23:38 <vponomaryov> and some existing bugs were discovered too
15:23:38 <bswartz> and there are some bugs in the core of share groups, but according to vponomaryov those are waiting for review
15:24:01 <ganso> bswartz: others are being postponed to be fixed in Pike...
15:24:09 <bswartz> #link https://bugs.launchpad.net/manila/+bug/1660321
15:24:09 <openstack> Launchpad bug 1660321 in Manila "share-group-snapshot-create is not passing share as part of snapshot" [High,In progress] - Assigned to Valeriy Ponomaryov (vponomaryov)
15:24:10 <bswartz> #link https://bugs.launchpad.net/manila/+bug/1660319
15:24:11 <bswartz> #link https://bugs.launchpad.net/manila/+bug/1659023
15:24:11 <bswartz> #link https://bugs.launchpad.net/manila/+bug/1661266
15:24:12 <bswartz> #link https://bugs.launchpad.net/manila/+bug/1661268
15:24:12 <openstack> Launchpad bug 1660319 in Manila "share continues to be member of group after migration" [High,In progress] - Assigned to Rodrigo Barbieri (rodrigo-barbieri2010)
15:24:13 <openstack> Launchpad bug 1659023 in Manila "Consistent Snapshots are broken in the NetApp cDOT driver" [Medium,Triaged] - Assigned to Ben Swartzlander (bswartz)
15:24:14 <openstack> Launchpad bug 1661266 in Manila "Share group types have no public extra specs" [Undecided,New]
15:24:15 <openstack> Launchpad bug 1661268 in Manila "consistent_snapshots group extra spec should allow "pool" and "host" consistency" [Undecided,New]
15:24:22 <gouthamr> ouch
15:24:32 <bswartz> sorry for paste storm
15:24:54 <gouthamr> the bugs hurt more  :P
15:25:02 <bswartz> anyways I'm concerned about the problems users may have upgrading from newton to ocata, and from ocata to pike
15:25:09 <vponomaryov> first two ready to review/merge
15:25:42 <bswartz> I think we should consider disabling share group APIs for the ocata release, and turning them on in pike
15:26:23 <vponomaryov> It has been experimental feature
15:26:34 <vponomaryov> why it should be disabled?
15:26:35 <bswartz> our main objective (my main objective at least) was to remove the experimental cgroup APIs which we've known needed to be removed for a long time, and that's done
15:26:46 <tbarron> I certainly don't want to be handling escalated customer calls on this stuff.
15:27:10 <tbarron> We aren't testing it downstream end-to-end.
15:27:13 <tbarron> Yet.
15:27:17 <bswartz> mainly I want to make sure our messaging is clear about what is usable and what should be avoided
15:27:24 <ganso> vponomaryov: I think it is better to be disabled than to explain erroneous behaviors to customers
15:27:27 <vponomaryov> tbaron: disable it in your deployments, what is the problem then?
15:27:29 <bswartz> I don't want to remove the code -- it's good code and we should build on it
15:27:44 <bswartz> but I'm proposing turning it dark for just this release, since we have no time to resolve remaining issues
15:28:18 <bswartz> vponomaryov: alternatively we could disable it through policy.json upstream, and people who wanted to experiment with it could re-enable it
15:28:22 <gouthamr> disable how? make 'em unroutable? or set the policy so deployers/users can enable at their own risk?
15:28:24 <tbarron> vponomaryov: maybe, but we want to minimize differences between upstream and downstream.
15:28:54 <vponomaryov> tbarron: policy.json is configuration file
15:29:03 <gouthamr> i am concerned about the share-group id being in the shares model
15:29:05 <vponomaryov> tbarron: I am not talking about soruce code change
15:29:09 <gouthamr> and us disabling APIs
15:29:10 <bswartz> mainly I don't want people thinking that they're going to have smooth sailing if they're using cgroups on newton and they upgrade
15:29:18 <ganso> gouthamr: maybe we should have a microversion bump for this
15:29:19 <bswartz> I would rather have them wait and upgrade to pike
15:29:48 <ganso> bswartz: if they do that, they'll miss all other ocata features
15:29:49 <bswartz> gouthamr: that's another bug we need to file and fix
15:30:08 <bswartz> ganso: only customers who actually use and care about cgroups
15:30:17 <gouthamr> the only driver that supported consistency groups (CG snapshots) was cDOT/NetApp
15:30:26 <bswartz> ganso: hopefully that set of customers is the empty set, but we can't be sure
15:30:50 <ganso> gouthamr, bswartz: we got multi-vendor clouds out there, and we can't have 2 versions of manila on a cloud
15:32:11 <bswartz> are there any technical issues with disabling the APIs through policy.json?
15:32:11 <bswartz> ganso: I'm not sure what your point is
15:32:32 <ganso> bswartz: if some cloud is already using CGs, then it is highly unlikely to upgrade to ocata
15:32:47 <bswartz> ganso: and what do you propose we do to fix that?
15:33:06 <bswartz> ganso: I'm saying that we have no good options here
15:33:31 <gouthamr> +1 disable, but can we have an ML post to dev/operators please?
15:33:36 <bswartz> either people give up on cgroups for 1 release or they stay behind or they have unfixed bugs to face
15:33:57 <vponomaryov> -1 for disable
15:34:07 <ganso> bswartz: yea we don't have... existing shares in CGs will still exist, and even though they cannot be deleted or new ones created, they must remain usuable
15:34:12 <ganso> bswartz: s/usuable/usable
15:34:13 <vponomaryov> experimental feature + easy to disable manually
15:34:26 <bswartz> vponomaryov: what is your proposal to prevent issues around consistency groups?
15:34:28 <tbarron> easy to enable manually
15:34:31 <tbarron> :D
15:34:43 <gouthamr> vponomaryov: we're not doing enough to protect users from the pitfalls of these "experimental" features
15:34:57 <ganso> +1 disable
15:35:12 <vponomaryov> bswartz: issues exist in every feature, disable anything that is not going to be used in prod
15:35:25 <vponomaryov> bswartz: I woudl do so
15:35:43 * gouthamr remembers someone mentioning "Bob in IT sets the experimental flag in the API request and boom"
15:36:16 <bswartz> my feeling is that experimental features are features we believe are usable but reserve the right to change based on user feedback
15:36:37 <bswartz> features that are under development and not ready to use at release time should be turned dark
15:36:45 <markstur> +1 disable
15:36:52 <vponomaryov> bswartz: experimental means "no guarantee"
15:37:05 <markstur> My biggest concern would be for customers that were using CGs.  I don't have any of those.
15:37:19 <bswartz> vponomaryov: there's never a guarantee, even on supported features
15:37:28 <gouthamr> markstur: not sure if we have any, but that's no excuse. :)
15:37:32 <bswartz> Apache license specifically states NO WARANTEE
15:38:05 <gouthamr> :( whoa. bad precedents here
15:38:19 <bswartz> however we owe it to our users to communicate what we think is good and what we think needs more work
15:38:42 <markstur> gouthamr: right but that path to share groups via bug fixes vs skipping a release might be a better call for someone with a CG impl
15:39:02 <vponomaryov> bswartz: and remember feedback from last summit -> "It is experimental? No thanks, we will wait until it becomes stable"
15:39:31 <tbarron> until the API is stable is different than until it works
15:39:31 <ganso> markstur: maybe some fixes we can backport, and then users could turn the feature on manually, once their bugfix has been backported
15:39:43 <bswartz> anyone else against disabling?
15:39:58 <bswartz> ganso: I like that idea but I suspect the needed bugfixes will involve database migrations
15:40:01 <markstur> so we could backport a disable of the disabling too?
15:40:05 <gouthamr> ganso: these fixes affect the database..
15:40:05 <gouthamr> ganso: as far as i see..
15:40:10 <gouthamr> naw man..
15:40:17 <bswartz> markstur that would piss of distros
15:40:28 <bswartz> piss off*
15:40:29 <tbarron> CGs are a safety feature, ensuring data integrity.  I wouldn't use them till they are solid.
15:40:39 <xyang2> bswartz: can't these remaining bugs be fixed and backported?
15:40:48 <ganso> bswartz, gouthamr: yes, but what if we have some of them fixed now with db migrations? but this is likely riskier
15:40:54 <markstur> did someone say distros shmistros?  wasn't me. love distros
15:41:07 <bswartz> xyang2: as I said, yes, unless they involve database migrations or anything that looks like a new feature
15:41:30 <gouthamr> ganso: we'll still review and fix them..
15:41:47 <ganso> gouthamr: yes, review, but likely not test enough
15:41:55 <markstur> but it sounds like we're stuck w/o a share groups that is ready to release
15:41:55 <bswartz> is anything likely to break if we change policy.json to disable the APIs?
15:42:05 <gouthamr> ganso: sadly
15:42:38 <bswartz> we have another agenda item I want to get to
15:43:22 <xyang2> gouthamr: NetApp CI actually passed on the groups functional tests.  Are those not working?
15:43:31 <gouthamr> xyang2: we never ran them
15:43:45 <xyang2> gouthamr: I saw them in the logs
15:43:45 <vponomaryov> gouthamr: ))
15:43:51 <ganso> gouthamr: :O
15:44:02 <xyang2> gouthamr: http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/64/355264/26/check/manila-cDOT-ss/75f34c0/console.txt
15:44:03 <vponomaryov> xyang2: i think netapp driver uses default approach as all other drivers do
15:44:35 <gouthamr> xyang2: ah.. sorry, i meant we didn't run any "CG" tests
15:44:47 <xyang2> gouthamr: ok
15:44:48 * tbarron logs for future discussion: lots of big changes merging at FF, not enough time to test end-to-end and review.
15:45:00 * gouthamr +100
15:45:25 <bswartz> okay let's move on
15:45:25 <vponomaryov> tbarron: big features are being tested for years
15:45:38 <bswartz> #topic LVM revert to snapshot
15:45:40 <vponomaryov> tbarron: and constantly improving
15:45:51 <tbarron> vponomaryov: yes, we are getting better
15:45:54 <bswartz> this came up yesterday in the channel with ganso
15:46:08 <bswartz> there's a bug in LVM's revert to snapshot support
15:46:39 <bswartz> I attempted to fix it and noticed that the nfs helper code is gross
15:46:48 <bswartz> wait let me step back
15:46:48 <tbarron> vponomaryov: here is another example, I +2ed revert to snapshot and didn't catch this bug.
15:47:09 <tbarron> vponomaryov: we won't catch them all of course
15:47:25 <bswartz> more background: the bug in LVM is that the share must be temporarily exported in order for the unmount to proceed so we can revert the LV
15:47:35 <bswartz> s/exported/unexported/
15:47:56 <ganso> bswartz: else it gets stuck in lv merge state... were you able to unstuck it?
15:48:14 * ganso wonders if "unstuck" is a real word
15:48:21 <bswartz> at the very least the driver interface needs the manager to pass down all the access rules for that share so it can safely unexport/re-export the share
15:48:53 <bswartz> ganso: unstucking it isn't the issue -- it should never have gotten to that stage
15:49:21 <bswartz> anyways I want to clean up the nfs helper code and fix this bug at the same time
15:49:32 <bswartz> but that would be a reltively large change this late in the release
15:49:33 <ganso> bswartz: if it is not possible to unstuck and recover the data, it makes the issue level increase to critical... I wasn't able to recover data
15:49:59 <bswartz> my proposal here is to remove the revert_to_snapshot_support flag from LVM
15:50:05 <bswartz> and fix this in pike
15:50:24 <vponomaryov> revert feature is not experimental, right?
15:50:25 <bswartz> the downside is that it leaves us with a test coverage hole for the new feature we merged
15:50:34 <bswartz> vponomaryov: correct
15:50:53 <bswartz> the only drivers that would support the feature in ocata would be netapp and hnas (I think)
15:50:59 <vponomaryov> then yes, either disable or fix
15:51:25 <bswartz> personally that doesn't bother me, as we will fix the bug in pike and have first party driver support again
15:51:42 <bswartz> but there's a risk of backports to ocata breaking the revert feature (a very small risk)
15:51:52 <bswartz> and the only way we would catch those is through 3rd party CI
15:51:59 <tbarron> do we set a bad precedent/incentives?
15:52:28 <bswartz> tbarron: I don't think so
15:52:55 <bswartz> the requirement that we have first party driver support for the feature wasn't really violated, it was just too buggy to release
15:53:07 <tbarron> well, I can get a feature in despite not having reliable first party test/success for it.
15:53:25 <tbarron> feature velocity vs stability
15:53:47 <bswartz> tbarron: what remedy would you prefer?
15:53:55 <bswartz> fix the LVM driver even if it's gross?
15:54:00 <bswartz> disable the feature entirely?
15:54:22 <tbarron> agree we don't have any great alternatives now ...
15:54:22 <ganso> bswartz: I'd prefer fix even if it is gross, backport the good fix asap
15:54:51 * tbarron and again, I was part of moviing this one through feature freeze, am not casting stones.
15:55:04 <bswartz> I'm okay with the gross fix approach, but I wanted to warn you guys bout it
15:55:06 <bswartz> about* it
15:55:10 <ganso> bswartz: even though my argument against share groups was that we could not test enough, I am in for fixing now, reviewing it, testing it (it is small) and shipping in ocata
15:55:41 <bswartz> okay I'll continue work on this
15:55:57 <bswartz> we're gonna skip the PTG topic today due to lack of time
15:56:00 <bswartz> #topic open discussion
15:56:06 <bswartz> anything else critical for today?
15:56:27 <bswartz> regarding the 17 open bugs, please get those fixes merged asap
15:56:55 <bswartz> bugs without fixes will get retargeted today unless I hear from the owner
15:57:14 <bswartz> and owners of bugs with fixes not ready to merge will be hearing from me
15:58:02 <bswartz> okay it looks like we're done
15:58:11 <bswartz> thanks all
15:58:21 <bswartz> #endmeeting