21:00:20 <vilobhmm11> #startmeeting quotas-wg
21:00:24 <openstack> Meeting started Mon Mar 28 21:00:20 2016 UTC and is due to finish in 60 minutes.  The chair is vilobhmm11. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:27 <openstack> The meeting name has been set to 'quotas_wg'
21:00:35 <vilobhmm11> hi all
21:02:45 <mc_nair> hey
21:03:19 <vilobhmm11> hi mc_nair
21:03:30 <vilobhmm11> lets wait few min for others to join
21:03:41 <ninag> hi
21:03:42 <nikhil> Courtesy quotas-wg meeting reminder: nikhil, vilobhmm, DuncanT, mc_nair, ninag
21:03:49 <nikhil> amrith ^
21:03:53 <vilobhmm11> hi ninag, nikhil
21:04:03 <nikhil> hey vilobhmm11
21:04:24 <DuncanT> Hi
21:04:39 <vilobhmm11> hello DuncanT
21:04:45 <vilobhmm11> all lets start then
21:04:46 <nikhil> amrith may join in a min
21:04:50 <vilobhmm11> ok
21:05:14 <amrith> sorry, hadn't connected to this ...
21:05:20 <nikhil> do we have a agenda today vilobhmm11 or should I quickly look up items from last mtg?
21:05:36 <nikhil> amrith: np, we've the full house now
21:05:43 <vilobhmm11> nikhil : i can go over the latest changes to https://review.openstack.org/#/c/284454/7
21:05:43 <nikhil> vilobhmm11: ^
21:05:46 <vilobhmm11> but you can start
21:06:04 <nikhil> vilobhmm11: please go ahead, actually that was the first item from that list :)
21:06:12 <vilobhmm11> from what happened in past meeting; since i was nto well last week could not join
21:06:20 <vilobhmm11> ok :)
21:06:39 <vilobhmm11> so we have been getting good feedback on https://review.openstack.org/#/c/284454/7/specs/delimiter-cross-project-quota-enforcement.rst
21:06:57 <vilobhmm11> before making any further changes wanted to give a bigger picture of what is the proposed approach here
21:07:15 <vilobhmm11> so if we check https://review.openstack.org/#/c/284454/7/specs/delimiter-cross-project-quota-enforcement.rst line # 116
21:07:59 <vilobhmm11> any project who wan;ts to consume quota will need to create an instance of the quota engine
21:08:17 <vilobhmm11> and quota engine will provide simple api's like
21:08:26 <vilobhmm11> #1. check_quota ( Query API)
21:08:46 <vilobhmm11> #2. assign_quota (Update API
21:09:12 <vilobhmm11> which internally will be implemented by the quota engine in atransaction safe manner
21:09:53 <vilobhmm11> so the quota engine will need the name of the project to get intialized
21:10:47 <vilobhmm11> please note that the data for quotas will still be kept with the respective projects and the quota entity(delimiter) won't like to maintain it [Thats what I have proposed here; we can discuss more if people don;t agree]
21:10:53 <vilobhmm11> advantages while doing so :-
21:11:07 <vilobhmm11> #. delimiter does not need to worry about data migration
21:11:18 <vilobhmm11> #. respective projects store quota data locally
21:11:45 <vilobhmm11> what delimiter provides is a common set of models that can be operated upon in a transaction safe manner
21:12:14 <DuncanT> I've been rather worried by the lack of reservations, and people saying we don't need them - the more I look at the cinder code, the more I think we do need them
21:12:16 <nikhil> I like this idea
21:12:20 <vilobhmm11> better way to put is *common set of models which can be used to enforce quotas in a transaction safe manner*
21:12:49 <vilobhmm11> DuncanT : we can get to that later ; first we should agree upon lib/service debate
21:12:49 <amrith> will delimiter allow oversubscription?
21:12:57 <vilobhmm11> amrith : yes
21:13:02 <amrith> vilobhmm11, will wait till we get there.
21:13:11 <nikhil> vilobhmm11: do we have the third name for that oversubscription?
21:13:14 <nikhil> :)
21:13:21 <amrith> On the subject of service/library, I think we should stick with library
21:13:25 <ninag> +1 on not having a quota service
21:13:33 <vilobhmm11> if you see https://review.openstack.org/#/c/284454/7/specs/delimiter-cross-project-quota-enforcement.rst #75 we have a proposal there
21:13:44 <nikhil> agreed on the library way
21:13:56 <vilobhmm11> nikhil : not yet :P floating sounds ok but not to the mark
21:14:03 <amrith> has anyone made a strong argument for service?
21:14:22 <nikhil> vilobhmm11: ha! no worries, we will take what we've then
21:14:31 <nikhil> amrith: there was one in the ML
21:14:36 <vilobhmm11> amrith : please go through http://osdir.com/ml/openstack-dev/2016-03/msg01582.html
21:14:40 <vilobhmm11> for details
21:14:40 <nikhil> it has to do with the UX aspects of it
21:14:49 <nikhil> Boris specifically
21:15:10 <nikhil> also, PWG is building on user stories that indicate the req for UX and a service would help there
21:15:41 <nikhil> but my proposal to PWG liaison for quotas and he  agreed to it was: we start off small as lib and then may be in future cycles..
21:15:55 <amrith> is he (Boris) going to join here?
21:15:56 <nikhil> we implement a service that will be able to proxy per project calls
21:16:16 <nikhil> I did not get a ack from him indicating so
21:16:49 <vilobhmm11> +1 nikhil imho we can eventually go to a service  (*if needed*) but for now i think its better to start as a lib as there are new set of problems to solve if its a service
21:17:22 <nikhil> yeah, the biggest concern is about atomicity (and of course the upgrades too)
21:17:55 <nikhil> amrith: but I think we'd get the folks who have strong inclination on this being service on the same page as the rest of WG
21:18:07 <nikhil> preferably before the summit
21:18:36 <vilobhmm11> by being a lib we need not worry about data migration ; what we could focus on is transactional guarantees (which is of at most importance for something like quotas) and provide easy to use API for users
21:18:46 <vilobhmm11> like check_quota ; assign_quota
21:18:59 <vilobhmm11> and lets the quota engine do the magic underneath :)
21:19:42 <nikhil> vilobhmm11: amrith: what do you think about updating the alternatives section of the spec to indicate service option and why the spec prefers lib?
21:20:10 <vilobhmm11> the logic to understand project heirarchies like  non-hiearchical or hierarchical will be taken into consideration by delimiter
21:20:11 <amrith> nikhil, I think that's a good alternative
21:20:29 <amrith> I (personally) think the library option is a good start
21:20:45 <vilobhmm11> nikhil : makes sense. I have an updated draft ready will push the changes for alternative section with that
21:20:51 <amrith> but, wanted to make sure that the alternatives were also well spoken for. the ml seemed to be thin on that side.
21:21:41 <vilobhmm11> nikhil, ninag, mc_nair, amrith, DuncanT : So then looks like all of us are on same page and would be nice to start off with delimiter being a LIBRARY for now
21:21:53 <vilobhmm11> amrith : sure
21:22:32 <nikhil> amrith: and I doubt if ML will be so far thorough :/
21:22:44 <nikhil> vilobhmm11: sounds like a good plan
21:22:51 <amrith> understood. that makes sense
21:23:09 <vilobhmm11> with the focus of delimiter be on is transactional guarantees (which is of at most importance for something like quotas) and provide easy to use API for users and the logic to understand project heirarchies like  non-hiearchical or hierarchical will be taken into consideration by delimiter
21:23:10 <amrith> let's move ahed with the option of listing service as an alternative with reasons why not as you propose.
21:24:13 <nikhil> ++
21:24:20 <vilobhmm11> nikhil : thats it from my side; unless anyone has some specific questions about the spec would let you take it forward;
21:24:38 <vilobhmm11> cool we have an agreement :)
21:24:44 <nikhil> vilobhmm11: I think we need to write that down as a mission statement on the repo that will be created ;)
21:25:21 <vilobhmm11> nikhil : For sure :)
21:25:30 <vilobhmm11> will do
21:26:30 <amrith> ++
21:27:24 <vilobhmm11> nikhil, DuncanT : we can talk about reservations now ?
21:27:31 <amrith> I have a question re: repo
21:27:40 <vilobhmm11> please go ahead
21:27:41 <amrith> is this going to be a project unto itself, or a part of (say) oslo?
21:27:57 <amrith> oslo.delimiter
21:28:03 <amrith> as opposed to openstack/delimiter
21:28:15 <nikhil> vilobhmm11: I've added a few agenda items for which we nee ~15mins https://etherpad.openstack.org/p/quotas-wg-meeting-agenda
21:29:07 <vilobhmm11> ok
21:29:11 <nikhil> amrith: I'd a chat with dims on this and oslo doesn't want this as it's part. they have accomplished their objective of common code. the preference is for a separate independent repo
21:29:20 <amrith> ok, thanks
21:29:28 <amrith> project it is
21:29:47 <nikhil> I'm afraid to call it a project but sure :)
21:30:20 <nikhil> not all projects have a service (like infra) but it's a pretty common misunderstanding
21:30:41 <nikhil> vilobhmm11: thoughts?
21:30:56 <amrith> ok, not an issue (that isn't the biggest issue at this stage, I guess)
21:31:01 <nikhil> (I was just carrying the older context into this discussion)
21:31:15 <vilobhmm11> nikhil : you r right projects like tooz/taskflow
21:31:29 <vilobhmm11> started as independent projects and then got incubated into oslo
21:31:40 <vilobhmm11> tooz/taskflow are not services though
21:32:04 <nikhil> amrith: sorry for being overly conservative. I am trying to get us unstuck from all possible roadblocks :)
21:32:17 <nikhil> vilobhmm11: sounds good
21:32:28 <amrith> nikhil, good course of action
21:32:51 <vilobhmm11> my thought is to start it simple and eventually grow by focussing on the core peice (providing transaction gurantess)
21:32:59 <nikhil> ++
21:33:40 <thingee> or take UX team under the big tent which doesn't have deliverables
21:33:57 <nikhil> :)
21:35:16 <nikhil> now that thingee is here, may be we can get a quick vote on whether we should propose this (quota) topic for CPL meeting tomorrow?
21:35:39 <vilobhmm11> thingee : welcome ! nikhil : sure
21:35:49 <nikhil> DuncanT: ninag: amrith: mc_nair: ^
21:36:06 <ninag> Sure
21:36:30 <thingee> I think it's healthy to communicate back to the general team to make sure efforts are going a good direction.
21:36:40 <nikhil> vilobhmm11: Let's take the silence as yes.
21:36:43 <thingee> since not every project is represented in this meeting it seems
21:36:47 <DuncanT> Yes from me
21:36:57 <mc_nair> +1
21:37:13 <nikhil> thingee: ++ (I think it was merely a matter of when rather than if)
21:37:17 <amrith> +1
21:37:24 <amrith> on the communicate to CPL tomorrow
21:37:37 <thingee> also just like release liaisons, I'm going to send out an email for projects with new ptls to make sure cross-project spec liaisons are refreshed if necessary.
21:37:48 <thingee> so it'll be good in case we have newcomers.
21:38:22 <nikhil> thanks for the heads up, thingee.
21:38:43 <vilobhmm11> thingee : sure; thanks for heads up
21:39:16 <nikhil> vilobhmm11: DuncanT: I won't interrupt with any more request until 5 mins before the mtg closing time. we can talk about reservations now if you all want.
21:39:33 <nikhil> may be better to use # topic to indicate the flow
21:39:41 <vilobhmm11> also one more thing thingee, nikhil : we got a slot in the cross project design session https://etherpad.openstack.org/p/newton-cross-project-sessions so something to look forward to
21:40:06 <vilobhmm11> mc_nair, ninag, DuncanT, amrith : ^^
21:40:14 <amrith> yes, looking at etherpad
21:40:29 <nikhil> vilobhmm11: yes, about that. Great idea, good proactive input and we'd discuss that in a few more details next monday.
21:40:54 <DuncanT> The reservations things...
21:40:55 <vilobhmm11> DuncanT : you can explain us the problems with and without reservation ?
21:41:04 <nikhil> #topic reservations
21:41:22 <nikhil> vilobhmm11:  you need to do that (only chairs can change topic)
21:41:26 <DuncanT> So we use a reservation because we want to report quota-not-available and similar in the API
21:41:35 <DuncanT> so we can give immediate feedback
21:41:48 <vilobhmm11> #reservation
21:41:52 <DuncanT> but we only know if we've successfully used quota in the c-vol code
21:41:55 <vilobhmm11> #topic reservations
21:42:07 <DuncanT> which is the other side of an rpc
21:42:21 <DuncanT> and might fail, silently (crash, rabbit restart, etc)
21:42:33 <DuncanT> reservations give us that right now
21:42:45 <DuncanT> not sure how to keep those sematics withotu them
21:43:33 <DuncanT> I've been meaning for ages to make reservations more verbose (include the volume id and the request id) for debugging and logging
21:44:37 <DuncanT> Can anybody explain how to recover from silent failures without reservations?
21:45:12 <vilobhmm11> DuncanT : what if without doing #1. reservation #2. commit we did #1. commit directly (based on actual resource usage) and if that didnt work do a rollback….If we impose expire time with every quota commit made won't it be the same
21:45:52 <DuncanT> vilobhmm11: you can't track individual commits though
21:46:00 <vilobhmm11> if silent failures happen the expire time will be triggered eventually
21:46:05 <nikhil> yes and the issue on the ML raised was that a 2 phase commit is really hard to achieve
21:46:50 <vilobhmm11> nikhil, DuncanT : even if its done as part of a transaction ?
21:46:52 <amrith> nikhil, I think the ML comment was questioning whether you want to hold a txn open for the time it takes for the underlying srevice to do it's thing
21:47:10 <DuncanT> vilobhmm11: you can't do a transaction since it's very async
21:47:15 <amrith> I have a proposal for how to address this without the long comments and txn's.
21:47:33 <vilobhmm11> DuncanT : i see
21:47:35 <DuncanT> amrith: I'm all ears
21:47:37 <amrith> as part of the 'reservation' process, if the library (delimiter) returned a token
21:47:49 <amrith> and the token was one that would expire in a certain amount of time.
21:47:59 <amrith> then if the service didn't finish up in time, the reservation expires
21:48:17 <amrith> if it does fihins, it comes back and tells the library that reservation <number> completed.
21:48:24 <amrith> the code flow would look like this
21:48:31 <amrith> for <operation> get token
21:48:33 <amrith> do operation
21:48:45 <amrith> report that <operation> with <token> is done
21:48:57 <amrith> in practice, the reservation would put things into an 'in-flight' table.
21:49:00 <amrith> so if you have quotas
21:49:05 <DuncanT> amrith: That works for us I think, as long as the quota is logically consumed for the time the token is live, yeah
21:49:09 <amrith> you have committed quotas and inflight quotas
21:49:32 <amrith> and then when you are asked to check for quota, you use committed quotas and inflight quotas along with overcommit to make your decision
21:49:45 <amrith> when a completion is reported, you move from 'inflight' to 'committed'
21:50:05 <amrith> periodically a janitor will throw away in-flight stuff (which is timestamped) so you can cleanup from time to time
21:50:15 <DuncanT> Being able to hang some metadata off the token (resource id, request id) would be useful for debugging, but that sounds pretty sematically close to what we have, so it should be easy to work with
21:50:26 <amrith> DuncanT, logically consumed modulo oversubscription, I think.
21:51:01 <amrith> my interest (very personally) is that I'd like to have cross-project quotas
21:51:08 <nikhil> amrith: that makes sense. I was trying to tie this together with Jay's message from yesterday..
21:51:17 <DuncanT> amrith: cross-project quotas?
21:51:18 <amrith> I'm wondering how we fold that into the design for per-project-library based qutas.
21:51:22 <amrith> DuncanT, yes.
21:51:24 <amrith> consider this
21:51:34 <amrith> Trove is a consumer of services from many core services
21:51:45 <amrith> lets just look at cinder and nova
21:51:54 <amrith> I'd like a reservation token for cinder and nova resources
21:52:35 <amrith> and then be able at a later stage (say when trove is done with a database) to go to the quota mechanism and just say "the token you gave me for reservation XYZ (potentially months ago) is now done, free it up"
21:52:47 <DuncanT> amrith: Got you. Not a guarantee the operation will work, but it at least solves the quota issues
21:53:01 <amrith> so on some granularity (per database, per cluster, ...) I'd like to associate a resource token
21:53:15 <DuncanT> amrith: You shouldn't need to do that, since the resource will actually exist to hold the quota though, right?
21:53:19 <amrith> and use taht to free up the resources cleanly (in the quota context)
21:53:34 <DuncanT> amrith: or do you want to speculatively hold quota for a month? That seems a bit odd
21:53:37 <amrith> whether the actual resource will be provisioned successfully or freed successfully, quota's can't guarantee.
21:53:43 <amrith> not speculatively
21:53:46 <amrith> here's the workflow
21:54:03 <amrith> user comes to Trove and says, give me a MongoDB cluster, 7 nodes, each with these amounts of disk
21:54:12 <amrith> so config servers and query routers have little disk
21:54:17 <amrith> data nodes have a lot of disk
21:54:24 <amrith> we get a reservation token
21:54:43 <amrith> then make reservations based on those tokens.
21:54:52 <amrith> then we attempt to get the actual resources
21:55:00 <amrith> if they succeed, I commit the reservations.
21:55:03 <amrith> the token remains.
21:55:14 <amrith> six months later, the cluster has to be expanded
21:55:16 <DuncanT> That's not a good design in my view
21:55:20 <amrith> we use the same token to extend it
21:55:31 <DuncanT> Once you've created the resource, the quota token is meaningless
21:55:37 <amrith> finally three years later, the cluster goes away, we should be able to free all resources based on that token
21:55:57 <DuncanT> That's resource grouping, totally independent of quota
21:56:05 <nikhil> ah, so your requirement is for ensuring that you can guarantee if a new cluster can be built or not and commit to that fact upfront?
21:56:17 <nikhil> I think last few lines didn't indicate this
21:56:22 <amrith> nikhil, that's one
21:56:23 <DuncanT> Reusing the quota token makes no sense since you need to check and reserve more quota anyway
21:56:41 <amrith> bug I want to use the token as a tracking mechanism for all resources held for a purpose.
21:56:43 <DuncanT> Unless you are grabbing more quota than you plan on using immediately?
21:56:52 <amrith> maybe that's the part that duncant is pointing out is bad.
21:57:19 <DuncanT> I'd call that resource tagging or similar, and say it is entirely authoginal to quotas
21:57:31 <amrith> ok
21:57:31 <DuncanT> orthogonal, sorry
21:58:02 <amrith> ok, that makes sense
21:58:18 <vilobhmm11> i think any service who plans to consume quota from IaaS layer will have this use case say for example Trove, Magnum, Heat
21:58:42 <vilobhmm11> we are running out of time
21:58:48 <nikhil> yeah
21:58:52 <vilobhmm11> i think this topic need more discussion
21:58:55 <vilobhmm11> and thought
21:59:17 <amrith> I see DuncanT's point, let me think some more about this and see if I can take the two apart
21:59:30 <amrith> and make quota's and resource tagging independent.
21:59:34 <vilobhmm11> may be DuncanT, amrith would be great to start a ML thread on this and take communities opinion so that we have some direction
21:59:38 <amrith> maybe we can table it for now
21:59:43 <amrith> sure, we can do that ...
21:59:49 <nikhil> ++
22:00:00 <amrith> I added something to the agenda :)
22:00:05 <vilobhmm11> #action : DuncanT, amrith to send ML thread and start discusssion regarding reservations
22:00:19 <nikhil> yes, we'd amrith :)
22:00:26 <vilobhmm11> alrite folks thanks for attending have a great week
22:00:28 <nikhil> vilobhmm11: let's run over for a couple more mins :)
22:00:30 <DuncanT> Writing a really dump prototype ASAP would be good too, start trying to hack it into the cinder, nova, glance code, see what fits
22:00:44 <amrith> before we go, could we talk about a time for next meeting?
22:00:49 <nikhil> vilobhmm11:  or we can discuss it outside of the meeting in the same channel
22:00:53 <amrith> or is there another group that needs this channel?
22:01:02 <vilobhmm11> nikhil, amrith : lets discuss outside
22:01:04 <DuncanT> Yes please talk about times, it is 1 am here!
22:01:11 <nikhil> DuncanT: oops
22:01:13 <DuncanT> What channel for discussions?
22:01:16 <nikhil> DuncanT: have a good night
22:01:20 <nikhil> same channel
22:01:24 <nikhil> #endmeeting