15:03:51 <bswartz> #startmeeting manila
15:03:52 <openstack> Meeting started Thu Apr  6 15:03:51 2017 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:53 <gouthamr> dustins: haha just did
15:03:53 <bswartz> hello all
15:03:56 <vponomaryov> Hello
15:03:56 <openstack> The meeting name has been set to 'manila'
15:03:59 <gouthamr> hello o/
15:04:00 <dustins> gouthamr: lol
15:04:00 <tbarron> hi
15:04:01 <zhongjun_> hello
15:04:01 <ganso> hello
15:04:03 <dustins> \o
15:04:05 <bswartz> sorry I got distracted in a hallway conversation
15:04:30 <gouthamr> too late for a coup
15:04:39 <markstur> hi
15:04:52 <xyang1> hi
15:04:53 * bswartz reminds himself to hide out in the 30 minutes before this meeting so nobody can distract him
15:05:11 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:05:43 <bswartz> so I didn't have any topics other than the specs to review and priorities
15:05:47 <bswartz> tom proposed an agenda item
15:05:49 <kaisers> hi
15:05:59 <dustins> gouthamr: Next time lemme know when you're gonna go all coup d'etat  over here :)
15:06:01 <bswartz> let's cover tom's topic first, then we'll address specs
15:06:12 <tbarron> #link http://lists.openstack.org/pipermail/openstack-dev/2017-April/114945.html
15:06:18 <gouthamr> dustins: shhh...
15:06:19 <tbarron> probably quick
15:06:28 <bswartz> #topic Concurrency bugs with LVM driver
15:06:37 <bswartz> #link http://lists.openstack.org/pipermail/openstack-dev/2017-April/114945.html
15:06:56 <tommylikehu> tbarron: do you mean we have concurrency bugs in lvm?
15:07:01 <tbarron> just checking on what lvm concurrency bugs we have filed and if any are known to pertain to running multiple lvm backends on the same node at the same time.
15:07:02 <bswartz> okay the actual issues are in the "helpers" used by generic and LVM
15:07:28 <bswartz> #link https://github.com/openstack/manila/blob/master/manila/share/drivers/helpers.py#L170
15:07:56 <tbarron> in gate we run with London and Paris lvm backends, both with the same IP set for lvm_share_export_ip
15:08:14 <bswartz> the way the NFS helper works is by manipulating rules using "exportfs" and copying files around to make the rules permanent
15:09:23 <tbarron> bswartz: so we need some synchronization there I guess
15:09:26 <bswartz> I would like to do something more cooperation-friendly like using exports.d if possible
15:10:26 <bswartz> one of the challenges is that some distros (*cough* Ubuntu *cough*) have ancient NFS tools and might not support what we need to do this right
15:10:27 <tbarron> are these concurrency issues due to multiple backends or even if we have concurrent operations within a backend?
15:10:53 <bswartz> tbarron: honestly I'm not sure -- I've never observed a race condition personally
15:10:56 <tbarron> i.e. process or thread issues?
15:10:56 <ganso> tbarron: there were some bugs that pertain to that fact of "multiple backends on the same node at the same time", but some have been addressed
15:11:13 <bswartz> I observed other kinds of problems in this code and when I inspected it I immediately got worried about races
15:11:41 <tbarron> ok, so this is an area we need to investigate and file bugs in
15:12:00 <tbarron> not an area where we have long outstanding unsolved bugs already filed
15:12:05 <ganso> tbarron: there was a bug that the /etc/exportfs file was being handled, saved and overwritten by separate driver instances (multibackend), so access rules were being lost due to race conditions... that was fixed through locking
15:12:12 <tbarron> tommylikehu: ^^^ agree?
15:12:20 <bswartz> but in fairness to authors of that NFS helper code I havent' done the investigation to prove that there are problems, and in fairness to ubuntu I haven't checked to see if the NFS tools are too old to use a better approach like /etc/exports.d
15:12:22 <tbarron> ganso: k, ty
15:12:43 <tbarron> bswartz: ok, that's what I wanted to know
15:13:25 <bswartz> my ideal solution here would be one file per share
15:13:39 <bswartz> all of them stored in /etc/exports.d
15:13:57 <bswartz> and manila just updates one file at a time -- no locking needed
15:14:02 <tbarron> that's basically what rraja does (in difft location) for ganesha
15:14:31 <tbarron> with uuids in the files so there's no namespace conflict
15:14:39 <tommylikehu> sorry, I was in handling another issues just now, trying to catch up with you
15:14:43 <bswartz> right
15:15:22 <tbarron> tommylikehu: just knew you were asking about concurrency issues and lvm and wanted to make sure you didn't know of others that weren't mentioned
15:15:38 <tommylikehu> I am not sure
15:15:43 <bswartz> tbarron: so to wrap up this topic, did you plan to work on any of these issues? or are you looking for someone to volunteer?
15:16:11 <tbarron> bswartz: I can work them as I hit them and am willing to see what I can do if others hit them
15:16:28 <tbarron> but we need to file launchpad bugs to get them beyond anecdotal
15:16:29 <tommylikehu> ganso, the extending share on lvm driver is always failing randomly, is that one possibly could due to this bug?
15:16:43 <ganso> tommylikehu: I don't think so
15:16:51 <ganso> tommylikehu: bswartz has hit that bug as well
15:16:54 <tommylikehu> ganso: ok
15:17:02 <bswartz> yeah, I would like to invest my own time in rewriting these helpers but it's not a super high priority for me currently
15:17:04 <ganso> tommylikehu: it may be due to delays when mounting/unmounting
15:17:23 <tommylikehu> ganso:  you mean the delay in lvm?
15:17:28 <ganso> tommylikehu: yes
15:17:42 <tommylikehu> ganso: that could be a bug..
15:17:48 <vponomaryov> bswartz: isn't ubuntu old package is a blocker for "rewriting"?
15:17:57 <ganso> bswartz: tommylikehu is hitting a similar/same bug as you've hit when you fixed the revert_to_snapshot bug in lvm driver
15:18:04 <bswartz> vponomaryov: need to investigate that
15:18:54 <tommylikehu> ganso: I checked the code in cinder, but didn't find a big difference
15:19:20 <bswartz> what code in cinder?
15:19:28 <bswartz> cinder doesn't do reverts does it?
15:19:39 <tommylikehu> extending when volume has snapshot
15:19:51 <tommylikehu> bswartz: it's not your case
15:20:09 <bswartz> okay
15:20:14 <bswartz> so anything else on this topic?
15:20:21 <tbarron> not from me
15:21:07 <bswartz> #topic specs
15:21:17 <bswartz> #link https://review.openstack.org/#/q/status:open+project:openstack/manila-specs
15:21:20 <tommylikehu> bswartz: can you paste the link you are trying to fix here?
15:21:31 <bswartz> so just a reminder, spec freeze is 1 week away
15:21:50 <bswartz> we have a lot of specs in good shape
15:22:10 <bswartz> my major concern is that we need to prioritize and not overcommit out review resources
15:22:14 <bswartz> s/out/our/
15:22:42 <bswartz> zhongjun has 8 specs (!)
15:22:47 <zhongjun_> we didn't merge one spec in pike :)
15:22:50 <bswartz> are all of these really important?
15:23:16 <zhongjun_> most of these are important =)
15:23:26 <bswartz> yeah I know some of this is carryover from earlier releases
15:23:52 <bswartz> I just know that if we approved all 8, it's unlikely we'd get the dev/review bandwidth to merge all 8 features in pike
15:24:07 <bswartz> so we need to make a judgement call about where to draw the line
15:24:23 <bswartz> all of this is good stuff but some is better than the rest
15:24:57 <bswartz> zhongjun_: you could help by sharing with us your prioritizations for the stuff you've proposed
15:25:28 <zhongjun_> ok
15:25:35 <bswartz> and I think I need to talk to the core reviewers about how much we're all willing to commit to in Pike
15:25:46 <tommylikehu> agreed
15:26:11 <zhongjun_> ensure share, infinite share, Support retrieve shares filtered by export-location,  Filter things out providing flexible way to API, dynamic reconfiguration
15:26:15 <bswartz> I think I'd be happiest if we merged some but not all of these specs
15:27:16 <zhongjun_> How about this prioritization
15:27:35 <bswartz> backup not on the list?
15:27:55 <bswartz> it also helps if we focus on smaller stuff
15:28:08 <zhongjun_> I hope it could on the list
15:28:08 <bswartz> 5 small features might be easier to deal with than 1 large feature
15:28:19 <bswartz> in terms of review impact
15:29:03 <bswartz> zhongjun_: I'd love to have share backup too, but nobody thinks it's a small feature and if we commit to it and fail to deliver, then everyone is unhappy
15:29:06 <zhongjun_> but you said there are few memebers in pike
15:30:03 <bswartz> yes we've lost some reviewer bandwidth
15:30:28 <bswartz> fortunately we're getting new community members at the same time older members are moving on
15:30:32 <zhongjun_> we could do something that we can do
15:30:43 <bswartz> but it takes time for new people to reach the level of core reviewer
15:31:49 <zhongjun_> good news
15:33:19 <bswartz> so core reviewers, do you want to work out the cut line here during the meeting, or would you rather I talk to people individually this afternoon?
15:34:09 * markstur wonders if it is easy enough to draw a line now
15:34:12 <gouthamr> okay with either.. it'd be good to loop zhongjun_ in because she's working on more stuff that needs reviews.
15:34:32 <ganso> bswartz: maybe we could have the etherpad where people sign up, and then items that do not have people signed up may be a good indicative of what can be cut out
15:34:36 <tbarron> and it will be very late in China "this afternoon"
15:34:40 <tommylikehu> gouthamr:  +1
15:34:58 <xyang1> +1 with etherpad
15:35:09 <bswartz> tbarron: we have zhongjun_'s priorities
15:35:12 <tommylikehu> +1 xyang1
15:35:20 <tbarron> zhongjun_: tommylikehu can you mark which of these overlap with similar cinder proposals?
15:35:31 <vponomaryov> yeah, etherpad would work great
15:35:37 <bswartz> it comes down to how much we can handle and what the core team wants to prioritize
15:35:46 <bswartz> I knew I should have done an etherpad weeks ago
15:35:52 * bswartz curses himself
15:35:57 <bswartz> one momeny
15:36:00 <tbarron> zhongjun_: tommylikehu I'd like the projects not to diverge on some of this stuff unless there is good reason
15:36:01 <bswartz> moment*
15:36:23 <tommylikehu> tbarron:  two of them I think
15:36:29 <markstur> some rough prioritization would be nice. It'd help us find that cut-off / stretch-goals line
15:36:39 <tbarron> dynamic config and filter by LIKE?
15:36:52 <bswartz> #link https://etherpad.openstack.org/p/manila-pike-spec-review-focus
15:36:54 <tommylikehu> oh, plus user message
15:37:06 * markstur suspects tbarron was sent here to spy on us for cinder
15:37:07 <tbarron> tommylikehu: oh yeah, that one :D
15:37:23 <zhongjun_> Okay, good idea
15:40:13 <jprovazn> https://review.openstack.org/452097 is infinite share,  not user messages
15:40:13 <zhongjun_> maybe we could extend specs deadline, beacuse we have more time in pike than ocata
15:42:29 <bswartz> zhongjun_: if theres one or 2 specs we want to grant extensions on I'm happy to do that
15:42:51 <bswartz> however a blanket extension is just kicking the can down the road
15:43:22 <markstur> blanket extension is called Q
15:43:24 <ganso> bswartz: I believe we have to designate the hi-pri specs before people sign up to other ones, correct?
15:44:13 <bswartz> the only one I considered high priority was ipv6
15:44:24 <markstur> I agree if we find something where it's harder to get the spec done, but important enough to keep it in-plan
15:44:28 <bswartz> does someone want to nominate more specs high priority?
15:44:31 <tommylikehu> ensure share should be one of them
15:44:51 <bswartz> tommylikehu: why?
15:44:57 <zhongjun_> tommylikehu:+1
15:45:17 <bswartz> is it important enough to drop everything and focus on just that?
15:45:27 <zhongjun_> It is a common feature, it can solve some problems
15:45:34 <bswartz> the status quo doesn't seem that terrible
15:45:41 <bswartz> it's a problem to be sure, and we can do better
15:45:59 <bswartz> but high priority means (to me) that it's absolutely essential
15:46:47 <vponomaryov> gouthamr: where is "quota for replicas" spec?
15:47:02 <bswartz> should we vote? on ensure share high priority?
15:47:14 <gouthamr> vponomaryov: didn't write one. is one needed? it's only fixing a bug for missing quotas for that resource
15:47:37 <vponomaryov> gouthamr: why not?
15:47:40 <gouthamr> vponomaryov: i don't agree that adding a new resource into the existing quota system needs a spec :)
15:47:43 <tommylikehu> bswartz:  we can have a try
15:47:43 <bswartz> gouthamr: a spec can't hurt, it would be super short
15:48:12 <vponomaryov> gouthamr: it will be APIImpact
15:48:14 <gouthamr> bswartz: lite-specs? :P glance has that..
15:48:52 <gouthamr> vponomaryov: certainly APIImpact.. i'll write one up if necessary..
15:49:05 <bswartz> #startvote Should ensure_share be a high priority spec for Pike? yes, no
15:49:06 <openstack> Begin voting on: Should ensure_share be a high priority spec for Pike? Valid vote options are yes, no.
15:49:07 <openstack> Vote using '#vote OPTION'. Only your last vote counts.
15:49:21 <tommylikehu> #vote yes
15:49:25 <bswartz> #vote no
15:49:30 <tbarron> #vote no
15:49:33 <zhongjun_> #vote yes
15:49:33 <xyang1> #vote no
15:49:34 <gouthamr> #vote no
15:49:35 <markstur> #vote no
15:49:37 <tommylikehu> we had a lot of complaint from our product team
15:49:38 <dustins> #vote no
15:49:40 <markstur> mabye
15:49:40 <vponomaryov> #vote no
15:49:45 <tommylikehu> ok.......
15:49:49 <kaisers> #vote no
15:49:50 <gouthamr> tommylikehu: we'll still work on it.. :)
15:49:54 <kaisers> sry
15:50:02 <vponomaryov> tommylikehu: voting is about priority
15:50:12 <tbarron> tommylikehu: doesn't mean we can't do it, I just am not dropping everything else to sort it out :D
15:50:26 <bswartz> tommylikehu: we get that's it's high priority for you, but the community has its own priorities
15:50:28 <tommylikehu> vponomaryov: tbarron , that's clear
15:50:29 <bswartz> #endvote
15:50:30 <openstack> Voted on "Should ensure_share be a high priority spec for Pike?" Results are
15:50:31 <gouthamr> tommylikehu: get your product team here and they can overturn these votes with majority :P
15:50:32 <openstack> yes (2): tommylikehu, zhongjun_
15:50:33 <openstack> no (8): bswartz, markstur, gouthamr, vponomaryov, tbarron, xyang1, kaisers, dustins
15:51:09 <zhongjun_> gouthamr: next time :D
15:51:32 <markstur> we didn't vote it out of Pike, right?
15:51:45 <tbarron> markstur: right
15:51:48 <bswartz> okay I'm going back to the etherpad to order the specs
15:52:14 <markstur> we just singled that one out to make it try harder?
15:53:02 <gouthamr> markstur: we want to ensure we have priorities about ensuring shares in pike
15:53:53 <bswartz> markstur: the question whas if anything other than ipv6 should be marked high priority
15:54:06 <bswartz> the other question the core team needs to answer is how many specs to merge period
15:54:28 <markstur> she surely should ensure share she shouts  ( gouthamr )
15:55:52 * gouthamr googles that and first hit: "Trouble": Song by Never Shout Never . well
15:57:02 <bswartz> okay I moved the 3 that zhongjun_ didn't list as priority below the line
15:57:21 <bswartz> I'm still not comfortable with the remaining list honestly
15:57:46 <bswartz> I think the dynamic reconfiguration feature will have pretty high impact
15:57:56 <zhongjun_> Could you please put back the "Enhance update shares"
15:58:00 <bswartz> and if we take that off the list then the rest doesn't look too bad
15:58:13 <bswartz> zhongjun_: like that?
15:58:17 <gouthamr> bswartz: enhance update shares is tied to ensure share
15:58:31 <markstur> feel free to change those lines as you like (wording or whatever)
15:58:39 <zhongjun_> It just separated from ensure share =)
15:58:57 <bswartz> aha, markstur is the leafy green color with no name
15:59:15 <markstur> verdant
15:59:23 <gouthamr> green
15:59:46 <bswartz> alright we at the and of our time in this room
16:00:01 <bswartz> this is a good start at a cutline
16:00:23 <bswartz> core reviewers, please ping me if you want to make further changes to this list
16:00:46 <bswartz> otherwise, focus on reviewing stuff above the line
16:00:53 <bswartz> #endmeeting