15:00:32 <bswartz> #startmeeting manila
15:00:33 <openstack> Meeting started Thu Apr 27 15:00:32 2017 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:37 <openstack> The meeting name has been set to 'manila'
15:00:38 <bswartz> hello all
15:00:41 <dustins> \o
15:00:41 <gouthamr> hello o/
15:00:44 <jprovazn> hi
15:00:44 <ganso> hello o/
15:00:47 <vponomaryov> hello
15:00:50 <zhongjun> hi
15:00:50 <tbarron> hi
15:00:55 <toabctl> hi
15:01:21 <vkmc> hi o/
15:01:25 <xyang2> hi
15:01:32 <tommylikehu> hi
15:02:07 <bswartz> markstur cknight: courtesy ping
15:02:41 <bswartz> #topic announcements
15:03:00 <bswartz> not much from me today
15:03:16 <bswartz> today is the extended deadline for 2 more specs that we gave extensions to last week
15:03:34 <bswartz> we can discuss those more in this meeting (usage monitoring in particular)
15:03:49 <bswartz> vponomaryov had something to share though
15:04:03 <vponomaryov> yeah
15:04:33 <vponomaryov> just to notify that I am going to ganso's club of retired people from manila
15:04:44 * tbarron slams his head on the wall
15:04:45 <tommylikehu> !!!
15:04:46 <openstack> tommylikehu: Error: "!!" is not a valid command.
15:04:47 <ganso> O_O
15:04:48 <toabctl> oh
15:04:49 <dustins> O_O
15:04:58 <zhongjun> oh!!!
15:04:59 <tommylikehu> :(((((((((((((((((((((
15:05:08 <vponomaryov> since middle of June
15:05:11 <zhongjun> :((
15:05:31 <bswartz> yeah I was very sad to hear this
15:05:33 <ganso> :(
15:05:49 <bswartz> vponomaryov has been an essential developer for most of the live of the project
15:06:15 <toabctl> vponomaryov, do you stop working for mirantis ? or are you going todo something different there?
15:06:28 <vponomaryov> toabctl: different
15:06:28 <jprovazn> sorry to hear (actually read) that :(
15:07:18 <xyang2> vponomaryov: oh no!
15:07:23 <dustins> vponomaryov: Best of luck in your future endeavors, man, sorry to hear that
15:07:24 <tbarron> vponomaryov: I can't see how it could possibly be as much fun as manila.
15:07:43 <vponomaryov> thank you for nice words )
15:07:47 <tbarron> I hope it has broken gates for you to fix.
15:07:50 <xyang2> vponomaryov: anyone else from Mirantis still work on Manila?
15:08:04 <vponomaryov> xyang2: don't think so
15:08:05 <zhongjun> Would you like to keep working on manila?
15:08:16 <vponomaryov> zhongjun: yes, would like
15:09:06 <bswartz> for those that don't know, Mirantis's involvement in Manila was part of a partnership between NetApp and mirantis back when we were trying to jumpstart the project with talented developers in 2013
15:09:42 <ganso> bswartz: so is that partnership over?
15:09:47 <xyang2> bswartz: are you trying to say this is NetApp's fault?:)
15:09:52 <bswartz> mirantis's business has changed over time though
15:10:36 <tommylikehu> intel and rackspace
15:10:41 <bswartz> I don't know about the partnership, but this particular part of it was ended despite my attempts to continue it
15:10:54 <gouthamr> and NetApp probably feels that Manila has been jump started (opinions not endorsed by employer)
15:12:07 <vponomaryov> and, compared to ganso, I cannot say I will be able to spend time for manila activities
15:12:32 <xyang2> bswartz, gouthamr: will you two continue to work on Manila
15:12:33 <vponomaryov> because amount of free time is unknown yet
15:12:52 <bswartz> gouthamr: yeah back when we started the project we never really imagined that mirantis would be involved for this long, but it was a terrific partnership
15:13:14 <ganso> vponomaryov: so, unless something turns up for you to continue to work on manila, we will be 100% short of a core
15:13:15 <bswartz> and whatever happens in the future, we all owe a huge thanks to vponomaryov for what he's done so far
15:13:29 <cknight> bswartz: +100!
15:13:35 <gouthamr> xyang2: would love to :)
15:13:36 <toabctl> +100!
15:13:55 <ganso> bswartz: +1000
15:13:56 <zhongjun> +++++
15:14:02 <gouthamr> vponomaryov: you'll be a tough act to follow... but thanks for teaching us along the way
15:14:15 <vponomaryov> =)
15:14:33 <bswartz> regarding vponomaryov's status, he'll remain on the core reviewer team and he'll be welcome to participate however he is able to going forward
15:14:54 <tbarron> he's kinda young looking for an emeritus professor
15:15:13 <bswartz> if vponomaryov ends up going a completely different direction and doesn't return, we'll eventually remove him like we did with u_glide
15:15:15 <xyang2> tbarron: +1:)
15:15:15 <vponomaryov> tbarron: hard childhood )0
15:15:22 <zhongjun> thanks vponomaryov for your goooood ideas
15:15:36 <vponomaryov> bswartz: not only u_glide
15:15:52 <vponomaryov> =)
15:16:41 <bswartz> anyways vponomaryov is the owner of a number of things, and while we have him for another ~6 weeks, he won't be able to finish everything he owns so many things will need new owners
15:16:57 <ganso> vponomaryov: well, I will need to find somebody else to argue on the IRC channel now :P
15:17:10 <vponomaryov> oh, yeah, one additional note, I will be on vacation since 1st of May and till 9th
15:17:28 <tbarron> slacker
15:17:29 <vponomaryov> so, it is about 5 weeks
15:17:36 <tommylikehu> vponomaryov: where is your destination
15:18:22 <tbarron> vponomaryov is too smart to answer
15:18:23 <vponomaryov> tommylikehu: not sure I understand your question
15:18:25 <bswartz> there are some essential items, like the generic driver and zfs drivers which definitely need new owners, and vponomaryov has told me he thinks he can shepherd all the share-group patches until they merge
15:18:37 * vkmc joins the convo late
15:18:40 <xyang2> vponomaryov: your next adventure?
15:18:43 <bswartz> vponomaryov: he's asking if you're traveling anywhere on your vacation
15:18:49 <vkmc> vponomaryov, so sorry to hear that you are moving to some other project :(
15:18:53 <tommylikehu> bswartz: )
15:19:43 <vponomaryov> xyang2, bswartz: vacation is not a journey to somewhere
15:19:49 <vponomaryov> have lots of local deeds
15:20:04 <bswartz> vponomaryov: in the USA we call that a "staycation"
15:20:15 <bswartz> a vacation where you stay home
15:20:16 <xyang2> vponomaryov: I meant do you know what you will be working on after Manila?
15:20:25 <vponomaryov> xyang2: not yet
15:20:59 <vponomaryov> bswartz: then yes, it is staycation )
15:21:05 <bswartz> okay well that's enough of the sad news, let's move on to the meeting agenda
15:21:13 <bswartz> #agenda https://wiki.openstack.org/wiki/Manila/Meetings
15:21:23 <bswartz> #topic Share usage monitoring
15:21:31 <bswartz> #link https://review.openstack.org/#/c/459150/
15:22:12 <bswartz> so, I'm embarrassed to say that last week when we discussed usage monitoring, I didn't consider the fact that polling would be involved until I read the updated spec
15:22:29 <bswartz> and I've been thinking all week about any way possible to avoid polling
15:22:46 <bswartz> unfortunately I haven't come up with anything
15:23:09 <bswartz> so we need to recognize that if we want this feature, it has a built in scalability problem
15:23:54 <bswartz> in retrospect, this is why we avoided implementing it in the beginning, because share usage was definitely something we considered at the start of the project in 2013
15:24:36 <bswartz> given where the project is now, I think it's worth doing the feature with monitoring if necessary
15:24:50 <bswartz> but we need to be mindful of the impact of periodic polling
15:25:14 <bswartz> I like the fact that the interval will be admin controlled, so the admin can decide if we wants fast or slow polling
15:25:39 <bswartz> a small deployment might want to gather usage statistics every few minutes, while a large deployment might only do it once a day
15:25:46 <tbarron> or none and just trigger the poll without periodic control in manila
15:26:00 <bswartz> tbarron: what would be the point of that?
15:26:05 <tommylikehu> tbarron:  how
15:26:28 <bswartz> how is on-demand monitoring any better than slow automatic monitoring?
15:26:30 <tbarron> I might want to query in smaller batches spaced out over the day rather than doing all shares on the backend at once\
15:26:47 <bswartz> tbarron: I thought about that but there's reasons not to
15:26:53 <tbarron> ?
15:27:03 <tbarron> depends on my business I think
15:27:10 <bswartz> consider that for many backends, it may be vastly more efficient to query all the usage information at once
15:27:21 <tbarron> ack, and for some it may not
15:27:22 <bswartz> rather than to ask one at a time
15:27:41 <tbarron> for some I may be running 'du -s' on each share in the batch.
15:27:54 <bswartz> tbarron: I can see it being the same, but how could all at once be worse than one at a time?
15:28:14 <tbarron> the manager will wait less time on each call
15:28:25 <zhongjun> bswartz: Why we don't want  add periodic task when we need monitor?
15:28:30 <tbarron> the call manager to driver is synchronous
15:28:44 <bswartz> zhongjun: I'm in favor of the periodic task
15:28:57 <vkmc> you could perform the query manually, when needed...
15:29:06 <bswartz> tbarron: as the spec is written, the manager asks the driver for all the shares in a single call
15:29:11 <tbarron> I'm not against it, and she has a way to turn it off.
15:29:30 <bswartz> and I like the current proposal, but I see your objection
15:29:30 <tbarron> it asks for usage for a list of shares, which it supplies, as the spec is written
15:29:51 <bswartz> the problem is that we'd need an entirely different mechanism to do what you propose and space the shares out over the whole dat
15:30:02 <gouthamr> tbarron: i presume the list of shares will be gathered for that host..
15:30:09 <tbarron> the spacing out can be done outside of manila
15:30:30 <gouthamr> tbarron: in the multi-share server case, that's kinda helpful
15:30:36 <bswartz> tbarron: okay you're thinking the driver will have it's own magic inside to gather the data incrementally?
15:30:50 <tbarron> If I'm a cloud op, I can query manila for the shares on each backend, then run through that list at whatever granularity I want.
15:31:13 <tbarron> bswartz: I thought of incremental logic in the driver but I think that's overkill.
15:31:13 <vponomaryov> tbarron: so, you want to have API for it?
15:31:19 <tbarron> vponomaryov: ack
15:31:25 <bswartz> okay so this brings us to the other important point -- storing the data
15:31:44 <bswartz> the spec as written proposes storing the data in memcached
15:31:55 <bswartz> but memcached is a lossy storage system
15:31:58 <gouthamr> manila gather-usage-stats --filters user_id=xyzzy,name~=test
15:31:59 <gouthamr> :)
15:32:30 <tbarron> gouthamr: +1
15:32:45 <zhongjun> gouthamr: haha
15:32:57 <bswartz> memcached can choose to evict data if it fills up, making it a poor choice for storing data that the manila API will depend on
15:33:19 <zhongjun> Do we need automatic in cloud? or just new api?
15:33:33 <bswartz> A more obvious choice IMO would be to store the usage data in the shares table itself
15:33:55 <bswartz> on the query side, that adds almost no cost when invoking a share details API
15:34:02 <vponomaryov> +1 bswartz , memcached solves only load issues
15:34:19 <vponomaryov> it should not be the only place where we store data
15:34:38 <vponomaryov> DB should have it and we should cache it in memcache, only that way
15:34:40 <bswartz> the only downside I can see (to storing in the DB) is the constant write traffic from the share manager as it periodically collects and updates usage
15:34:41 <tbarron> zhongjun went to memcached b/c I argued that that has performance impact scaling issue.
15:34:58 <tbarron> that ^^ storing usage in db
15:35:16 <tommylikehu> tbarron +1
15:35:29 <bswartz> however the cost of that write traffic is arguably less than the cost of collecting the information from the backends (at least in cases where a du -s is involved)
15:35:29 <ganso> bswartz: if we have cache we could write to DB in bursts
15:35:42 <gouthamr> if interval is configurable the usage numbers might be stale and inappropriate for user consumption
15:35:42 <tbarron> she started with storing in DB tables
15:35:53 <bswartz> ganso: better yet we could simply not update the ones that haven't changed
15:36:04 <ganso> bswartz: +1000
15:36:20 <tbarron> bswartz: and she has proposed the compare and only update if changed
15:36:55 <vponomaryov> tbarron: current proposal considers memcache as the only source of that data
15:37:07 <vponomaryov> tbarron" s/current/current patch set/
15:37:10 <bswartz> so I'm pretty happy with the existing driver interface design, but my current thinking is that we should ignore memcached entirely in this spec, because it's an optimization to solve a problem which may not exist
15:37:25 <tbarron> vponomaryov: I understand, I'm just pointing out that some of these ideas have been proposed along the way.
15:37:26 <bswartz> classic premature optimization
15:38:01 <bswartz> and if optimization is needed, we will still need stable storage behind memcached, so it's better to figure that out now
15:38:19 <tbarron> I'm not sure that the issue of performance issues that may scale with the number of shares on the backend is a premature optimization issue.
15:38:22 <zhongjun> bswartz: +1 yes, we could use other way to store info, even if we don't install memcached
15:38:27 <tbarron> It is a design issue.
15:38:52 <bswartz> tbarron: the scaling issue cuts across all parts of the design
15:39:03 <tbarron> +1000
15:39:14 <bswartz> ideally we want to avoid doing work on behalf of shares that haven't been touched by anybody
15:39:48 <bswartz> unfortunatey manila isn't in a position to know which shares are "in-use" vs unused, so polling seems to be unavoidable
15:40:07 * tbarron isn't meaning to argue that we should use memcached though
15:40:19 <bswartz> perhaps backends can know which shares are untouched and use that as part of an optimization when collecting usage information
15:40:29 <vponomaryov> tbarron: actually, it is good to use it even out of scope of this feature
15:40:38 <bswartz> but that's below the driver interface and not our concern
15:41:07 <tbarron> we might want to prototype it in one of our drivers though
15:41:31 <tbarron> best practice if you have to check each share separately
15:41:35 <bswartz> tbarron: I'm pretty sure the prototype will involve a system("df") call
15:42:28 <tbarron> I was thinking stat will be cheaper than du -s
15:43:31 <bswartz> tbarron: you mean statfs?
15:43:37 <zhongjun> tbarron:  stat?
15:43:42 <bswartz> stat is for individual inodes
15:44:11 <tbarron> yeah, I was thinking stat on the root of the share but that may not get what is needed
15:44:37 <tbarron> zhongjun: 'stat' system call or equivalent, but nm, I am taking us off track
15:44:41 <bswartz> statfs() is the venerable linux system call intended for this purpose
15:44:58 <tbarron> bswartz: ack
15:45:06 <bswartz> it has many bugs, so beware ;-)
15:45:24 <bswartz> s/bugs/errata/
15:45:30 <tbarron> :D
15:46:15 <bswartz> okay so in the interest of moving on, can we agree that we should replace memcached with a simple new database column in the spec? and we can assess DB performance issues at another time?
15:46:31 <bswartz> I suspect that DB performance issues will be the least of our scaling issues regarding this feature
15:46:54 <vponomaryov> DB support is a must
15:47:11 <gouthamr> okay.. as long as we're controlling who sees that column by policy.
15:47:13 <vponomaryov> memcache support should be separate effort
15:47:31 <gouthamr> when the GET /shares API is invoked..
15:47:36 <bswartz> gouthamr: what security concern do you have?
15:47:43 <bswartz> this isn't exactly secret information
15:47:55 <gouthamr> it's not a security concern as much as it's unpredictable, unlike size.
15:47:58 <tbarron> storing volatile info in this kind of DB seems weird to me, gotta say
15:48:42 <tbarron> I think ceilometer does specialized stuff for these kinds of data
15:48:50 <gouthamr> so as a user, i'd much rather rely on looking at a mounted share for the size.. administrators controlling the polling information may know that the information is as stale as they let it be
15:49:05 <vponomaryov> tbarron: it is not something like user sessions that can be lost without any problems
15:49:44 <tbarron> as an admin I want a timestamp with the data and I want to refresh it if it's too stale and use it if it's good enough
15:49:52 <tbarron> as a user I want the truth
15:49:54 <tbarron> now
15:50:11 <bswartz> tbarron: you raise a good point
15:50:26 <tbarron> vponomaryov: only if I care what the usage was "then".  I can always get it now at more cost by doing a direct driver query.
15:50:47 <bswartz> the information is volatile, and users who want the true value can always obtain it themselves by mounting the share
15:50:49 <ganso> gouthamr: maybe we should name the field "last_usage_reading" or something along these lines... else I believe it will sparkle confusion about the inaccuracy of the usage value
15:51:02 <zhongjun> gouthamr: but if you have many shares, you could looking at share usage size quickly in the list
15:51:39 <bswartz> however if manila will ever use these numbers for any decision making, then we need them stored somewhere so the admin/user can see what manila things that value is, however incorrect the value may be
15:51:46 <vponomaryov> tbarron: so, you propose to have also timestamp when usage record was updated?
15:51:48 <bswartz> s/things/thinks/
15:52:06 <tbarron> vponomaryov: i think so, any downside?
15:52:14 <vponomaryov> tbarron: no, I like it
15:52:18 <vponomaryov> ))
15:52:31 <bswartz> downside is that the timestamp will *always* change ensuring writes to that DB column all the time
15:52:48 <tbarron> right
15:52:49 <gouthamr> bswartz: not if you define another timestamp for this purpose
15:52:53 <bswartz> again it's probably not a huge issue, but it's more expensive than just storing the GB value
15:53:15 <tbarron> so i think ceilometer was made for this kind of thing :D
15:53:52 <bswartz> tbarron: so your proposal is, make the share manager collect this data, and emit it into the ether but don't store it at all?
15:53:56 <gouthamr> ganso: +1 size_used sounds like it's always accurate
15:54:10 <tbarron> bswartz: I think it's worth considering.
15:54:16 <bswartz> yeah I agree
15:54:19 <gouthamr> +1 i like the idea
15:54:55 <bswartz> zhongjun: what is the use case for making the data available through the share API? why not just send it to ceilometer or some other metrics collection engine?
15:54:58 <zhongjun> tbarron:  We could store a inexact timestamp
15:54:59 <tbarron> we have to get the info, that's one problem.  We have to transmit it to ceilometer anyways, that's another.
15:55:05 <vponomaryov> tbarron: it means, any admin that wants to bill for usage will require ceilometer onboard?
15:55:28 <tbarron> Then whether we also build our own persistence so we can run without ceilometer and get usage, that's another.
15:55:32 <ganso> if there is nobody to consume the data, then the data has no value... a polling mechanism that stores in the DB will always discard the value at the polling rate
15:55:52 <tbarron> vponomaryov: that's the downside unless we build something like ceilometer in manila.
15:56:01 <gouthamr> ganso: if nobody will consume it, we'll have a way to turn it off
15:56:01 <bswartz> tbarron: let's not to dothat
15:56:08 <bswartz> gouthamr: +1
15:56:27 <bswartz> the collection itself is disabled by default, it will only run if the admin specifically configures it
15:56:30 <zhongjun> bswartz: I wrote it in the spec, admin could want to know it for charge
15:56:43 <bswartz> and presumably the admin would configure a consume for the data
15:56:52 <ganso> gouthamr: I meant that for this the ceilometer approach makes more sense, if someone will consume it, should be through ceilometer, it is less expensive and storing it in our DB a value that nobody might read
15:56:56 <gouthamr> ganso: also, i think the option will be "turn it on if necessary", it's off by default.
15:57:02 <bswartz> zhongjun: but the admin could go to ceilometer's API for that, right?
15:57:19 <bswartz> ganso: +1
15:57:31 <gouthamr> ganso: +1
15:58:12 <bswartz> I really don't like storing volatile data which might be read by nobody and will usually be inaccurate
15:58:38 <bswartz> ceilometer is designed exactly for this kind of thing
15:58:44 <zhongjun> bswartz:  but if the admin don't install the ceilometer
15:58:59 <bswartz> zhongjun: then they leave the collection turned off
15:59:04 <ganso> zhongjun: then I'd say the feature cannot be enabled
15:59:09 <bswartz> or use gnochi (sp?) instead
15:59:21 <vponomaryov> agree
15:59:31 <gouthamr> zhongjun: it needn't be ceilometer, any consumer to our telemetry notifications..
15:59:37 <bswartz> okay I sense we're in agreement here about the data storage
15:59:49 <bswartz> it will probably take more than a few hours to update the spec
16:00:03 <bswartz> so let's give this one a few more days and keep providing gerrit feedback
16:00:06 <bswartz> I think we're close
16:00:12 * gouthamr #TIL  gnocchi = soft dough dumplings
16:00:16 <bswartz> we have to end this meeting though
16:00:24 <bswartz> thanks all
16:00:31 <bswartz> #endmeeting