21:00:20 #startmeeting quotas-wg 21:00:24 Meeting started Mon Mar 28 21:00:20 2016 UTC and is due to finish in 60 minutes. The chair is vilobhmm11. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:27 The meeting name has been set to 'quotas_wg' 21:00:35 hi all 21:02:45 hey 21:03:19 hi mc_nair 21:03:30 lets wait few min for others to join 21:03:41 hi 21:03:42 Courtesy quotas-wg meeting reminder: nikhil, vilobhmm, DuncanT, mc_nair, ninag 21:03:49 amrith ^ 21:03:53 hi ninag, nikhil 21:04:03 hey vilobhmm11 21:04:24 Hi 21:04:39 hello DuncanT 21:04:45 all lets start then 21:04:46 amrith may join in a min 21:04:50 ok 21:05:14 sorry, hadn't connected to this ... 21:05:20 do we have a agenda today vilobhmm11 or should I quickly look up items from last mtg? 21:05:36 amrith: np, we've the full house now 21:05:43 nikhil : i can go over the latest changes to https://review.openstack.org/#/c/284454/7 21:05:43 vilobhmm11: ^ 21:05:46 but you can start 21:06:04 vilobhmm11: please go ahead, actually that was the first item from that list :) 21:06:12 from what happened in past meeting; since i was nto well last week could not join 21:06:20 ok :) 21:06:39 so we have been getting good feedback on https://review.openstack.org/#/c/284454/7/specs/delimiter-cross-project-quota-enforcement.rst 21:06:57 before making any further changes wanted to give a bigger picture of what is the proposed approach here 21:07:15 so if we check https://review.openstack.org/#/c/284454/7/specs/delimiter-cross-project-quota-enforcement.rst line # 116 21:07:59 any project who wan;ts to consume quota will need to create an instance of the quota engine 21:08:17 and quota engine will provide simple api's like 21:08:26 #1. check_quota ( Query API) 21:08:46 #2. assign_quota (Update API 21:09:12 which internally will be implemented by the quota engine in atransaction safe manner 21:09:53 so the quota engine will need the name of the project to get intialized 21:10:47 please note that the data for quotas will still be kept with the respective projects and the quota entity(delimiter) won't like to maintain it [Thats what I have proposed here; we can discuss more if people don;t agree] 21:10:53 advantages while doing so :- 21:11:07 #. delimiter does not need to worry about data migration 21:11:18 #. respective projects store quota data locally 21:11:45 what delimiter provides is a common set of models that can be operated upon in a transaction safe manner 21:12:14 I've been rather worried by the lack of reservations, and people saying we don't need them - the more I look at the cinder code, the more I think we do need them 21:12:16 I like this idea 21:12:20 better way to put is *common set of models which can be used to enforce quotas in a transaction safe manner* 21:12:49 DuncanT : we can get to that later ; first we should agree upon lib/service debate 21:12:49 will delimiter allow oversubscription? 21:12:57 amrith : yes 21:13:02 vilobhmm11, will wait till we get there. 21:13:11 vilobhmm11: do we have the third name for that oversubscription? 21:13:14 :) 21:13:21 On the subject of service/library, I think we should stick with library 21:13:25 +1 on not having a quota service 21:13:33 if you see https://review.openstack.org/#/c/284454/7/specs/delimiter-cross-project-quota-enforcement.rst #75 we have a proposal there 21:13:44 agreed on the library way 21:13:56 nikhil : not yet :P floating sounds ok but not to the mark 21:14:03 has anyone made a strong argument for service? 21:14:22 vilobhmm11: ha! no worries, we will take what we've then 21:14:31 amrith: there was one in the ML 21:14:36 amrith : please go through http://osdir.com/ml/openstack-dev/2016-03/msg01582.html 21:14:40 for details 21:14:40 it has to do with the UX aspects of it 21:14:49 Boris specifically 21:15:10 also, PWG is building on user stories that indicate the req for UX and a service would help there 21:15:41 but my proposal to PWG liaison for quotas and he agreed to it was: we start off small as lib and then may be in future cycles.. 21:15:55 is he (Boris) going to join here? 21:15:56 we implement a service that will be able to proxy per project calls 21:16:16 I did not get a ack from him indicating so 21:16:49 +1 nikhil imho we can eventually go to a service (*if needed*) but for now i think its better to start as a lib as there are new set of problems to solve if its a service 21:17:22 yeah, the biggest concern is about atomicity (and of course the upgrades too) 21:17:55 amrith: but I think we'd get the folks who have strong inclination on this being service on the same page as the rest of WG 21:18:07 preferably before the summit 21:18:36 by being a lib we need not worry about data migration ; what we could focus on is transactional guarantees (which is of at most importance for something like quotas) and provide easy to use API for users 21:18:46 like check_quota ; assign_quota 21:18:59 and lets the quota engine do the magic underneath :) 21:19:42 vilobhmm11: amrith: what do you think about updating the alternatives section of the spec to indicate service option and why the spec prefers lib? 21:20:10 the logic to understand project heirarchies like non-hiearchical or hierarchical will be taken into consideration by delimiter 21:20:11 nikhil, I think that's a good alternative 21:20:29 I (personally) think the library option is a good start 21:20:45 nikhil : makes sense. I have an updated draft ready will push the changes for alternative section with that 21:20:51 but, wanted to make sure that the alternatives were also well spoken for. the ml seemed to be thin on that side. 21:21:41 nikhil, ninag, mc_nair, amrith, DuncanT : So then looks like all of us are on same page and would be nice to start off with delimiter being a LIBRARY for now 21:21:53 amrith : sure 21:22:32 amrith: and I doubt if ML will be so far thorough :/ 21:22:44 vilobhmm11: sounds like a good plan 21:22:51 understood. that makes sense 21:23:09 with the focus of delimiter be on is transactional guarantees (which is of at most importance for something like quotas) and provide easy to use API for users and the logic to understand project heirarchies like  non-hiearchical or hierarchical will be taken into consideration by delimiter 21:23:10 let's move ahed with the option of listing service as an alternative with reasons why not as you propose. 21:24:13 ++ 21:24:20 nikhil : thats it from my side; unless anyone has some specific questions about the spec would let you take it forward; 21:24:38 cool we have an agreement :) 21:24:44 vilobhmm11: I think we need to write that down as a mission statement on the repo that will be created ;) 21:25:21 nikhil : For sure :) 21:25:30 will do 21:26:30 ++ 21:27:24 nikhil, DuncanT : we can talk about reservations now ? 21:27:31 I have a question re: repo 21:27:40 please go ahead 21:27:41 is this going to be a project unto itself, or a part of (say) oslo? 21:27:57 oslo.delimiter 21:28:03 as opposed to openstack/delimiter 21:28:15 vilobhmm11: I've added a few agenda items for which we nee ~15mins https://etherpad.openstack.org/p/quotas-wg-meeting-agenda 21:29:07 ok 21:29:11 amrith: I'd a chat with dims on this and oslo doesn't want this as it's part. they have accomplished their objective of common code. the preference is for a separate independent repo 21:29:20 ok, thanks 21:29:28 project it is 21:29:47 I'm afraid to call it a project but sure :) 21:30:20 not all projects have a service (like infra) but it's a pretty common misunderstanding 21:30:41 vilobhmm11: thoughts? 21:30:56 ok, not an issue (that isn't the biggest issue at this stage, I guess) 21:31:01 (I was just carrying the older context into this discussion) 21:31:15 nikhil : you r right projects like tooz/taskflow 21:31:29 started as independent projects and then got incubated into oslo 21:31:40 tooz/taskflow are not services though 21:32:04 amrith: sorry for being overly conservative. I am trying to get us unstuck from all possible roadblocks :) 21:32:17 vilobhmm11: sounds good 21:32:28 nikhil, good course of action 21:32:51 my thought is to start it simple and eventually grow by focussing on the core peice (providing transaction gurantess) 21:32:59 ++ 21:33:40 or take UX team under the big tent which doesn't have deliverables 21:33:57 :) 21:35:16 now that thingee is here, may be we can get a quick vote on whether we should propose this (quota) topic for CPL meeting tomorrow? 21:35:39 thingee : welcome ! nikhil : sure 21:35:49 DuncanT: ninag: amrith: mc_nair: ^ 21:36:06 Sure 21:36:30 I think it's healthy to communicate back to the general team to make sure efforts are going a good direction. 21:36:40 vilobhmm11: Let's take the silence as yes. 21:36:43 since not every project is represented in this meeting it seems 21:36:47 Yes from me 21:36:57 +1 21:37:13 thingee: ++ (I think it was merely a matter of when rather than if) 21:37:17 +1 21:37:24 on the communicate to CPL tomorrow 21:37:37 also just like release liaisons, I'm going to send out an email for projects with new ptls to make sure cross-project spec liaisons are refreshed if necessary. 21:37:48 so it'll be good in case we have newcomers. 21:38:22 thanks for the heads up, thingee. 21:38:43 thingee : sure; thanks for heads up 21:39:16 vilobhmm11: DuncanT: I won't interrupt with any more request until 5 mins before the mtg closing time. we can talk about reservations now if you all want. 21:39:33 may be better to use # topic to indicate the flow 21:39:41 also one more thing thingee, nikhil : we got a slot in the cross project design session https://etherpad.openstack.org/p/newton-cross-project-sessions so something to look forward to 21:40:06 mc_nair, ninag, DuncanT, amrith : ^^ 21:40:14 yes, looking at etherpad 21:40:29 vilobhmm11: yes, about that. Great idea, good proactive input and we'd discuss that in a few more details next monday. 21:40:54 The reservations things... 21:40:55 DuncanT : you can explain us the problems with and without reservation ? 21:41:04 #topic reservations 21:41:22 vilobhmm11: you need to do that (only chairs can change topic) 21:41:26 So we use a reservation because we want to report quota-not-available and similar in the API 21:41:35 so we can give immediate feedback 21:41:48 #reservation 21:41:52 but we only know if we've successfully used quota in the c-vol code 21:41:55 #topic reservations 21:42:07 which is the other side of an rpc 21:42:21 and might fail, silently (crash, rabbit restart, etc) 21:42:33 reservations give us that right now 21:42:45 not sure how to keep those sematics withotu them 21:43:33 I've been meaning for ages to make reservations more verbose (include the volume id and the request id) for debugging and logging 21:44:37 Can anybody explain how to recover from silent failures without reservations? 21:45:12 DuncanT : what if without doing #1. reservation #2. commit we did #1. commit directly (based on actual resource usage) and if that didnt work do a rollback….If we impose expire time with every quota commit made won't it be the same 21:45:52 vilobhmm11: you can't track individual commits though 21:46:00 if silent failures happen the expire time will be triggered eventually 21:46:05 yes and the issue on the ML raised was that a 2 phase commit is really hard to achieve 21:46:50 nikhil, DuncanT : even if its done as part of a transaction ? 21:46:52 nikhil, I think the ML comment was questioning whether you want to hold a txn open for the time it takes for the underlying srevice to do it's thing 21:47:10 vilobhmm11: you can't do a transaction since it's very async 21:47:15 I have a proposal for how to address this without the long comments and txn's. 21:47:33 DuncanT : i see 21:47:35 amrith: I'm all ears 21:47:37 as part of the 'reservation' process, if the library (delimiter) returned a token 21:47:49 and the token was one that would expire in a certain amount of time. 21:47:59 then if the service didn't finish up in time, the reservation expires 21:48:17 if it does fihins, it comes back and tells the library that reservation completed. 21:48:24 the code flow would look like this 21:48:31 for get token 21:48:33 do operation 21:48:45 report that with is done 21:48:57 in practice, the reservation would put things into an 'in-flight' table. 21:49:00 so if you have quotas 21:49:05 amrith: That works for us I think, as long as the quota is logically consumed for the time the token is live, yeah 21:49:09 you have committed quotas and inflight quotas 21:49:32 and then when you are asked to check for quota, you use committed quotas and inflight quotas along with overcommit to make your decision 21:49:45 when a completion is reported, you move from 'inflight' to 'committed' 21:50:05 periodically a janitor will throw away in-flight stuff (which is timestamped) so you can cleanup from time to time 21:50:15 Being able to hang some metadata off the token (resource id, request id) would be useful for debugging, but that sounds pretty sematically close to what we have, so it should be easy to work with 21:50:26 DuncanT, logically consumed modulo oversubscription, I think. 21:51:01 my interest (very personally) is that I'd like to have cross-project quotas 21:51:08 amrith: that makes sense. I was trying to tie this together with Jay's message from yesterday.. 21:51:17 amrith: cross-project quotas? 21:51:18 I'm wondering how we fold that into the design for per-project-library based qutas. 21:51:22 DuncanT, yes. 21:51:24 consider this 21:51:34 Trove is a consumer of services from many core services 21:51:45 lets just look at cinder and nova 21:51:54 I'd like a reservation token for cinder and nova resources 21:52:35 and then be able at a later stage (say when trove is done with a database) to go to the quota mechanism and just say "the token you gave me for reservation XYZ (potentially months ago) is now done, free it up" 21:52:47 amrith: Got you. Not a guarantee the operation will work, but it at least solves the quota issues 21:53:01 so on some granularity (per database, per cluster, ...) I'd like to associate a resource token 21:53:15 amrith: You shouldn't need to do that, since the resource will actually exist to hold the quota though, right? 21:53:19 and use taht to free up the resources cleanly (in the quota context) 21:53:34 amrith: or do you want to speculatively hold quota for a month? That seems a bit odd 21:53:37 whether the actual resource will be provisioned successfully or freed successfully, quota's can't guarantee. 21:53:43 not speculatively 21:53:46 here's the workflow 21:54:03 user comes to Trove and says, give me a MongoDB cluster, 7 nodes, each with these amounts of disk 21:54:12 so config servers and query routers have little disk 21:54:17 data nodes have a lot of disk 21:54:24 we get a reservation token 21:54:43 then make reservations based on those tokens. 21:54:52 then we attempt to get the actual resources 21:55:00 if they succeed, I commit the reservations. 21:55:03 the token remains. 21:55:14 six months later, the cluster has to be expanded 21:55:16 That's not a good design in my view 21:55:20 we use the same token to extend it 21:55:31 Once you've created the resource, the quota token is meaningless 21:55:37 finally three years later, the cluster goes away, we should be able to free all resources based on that token 21:55:57 That's resource grouping, totally independent of quota 21:56:05 ah, so your requirement is for ensuring that you can guarantee if a new cluster can be built or not and commit to that fact upfront? 21:56:17 I think last few lines didn't indicate this 21:56:22 nikhil, that's one 21:56:23 Reusing the quota token makes no sense since you need to check and reserve more quota anyway 21:56:41 bug I want to use the token as a tracking mechanism for all resources held for a purpose. 21:56:43 Unless you are grabbing more quota than you plan on using immediately? 21:56:52 maybe that's the part that duncant is pointing out is bad. 21:57:19 I'd call that resource tagging or similar, and say it is entirely authoginal to quotas 21:57:31 ok 21:57:31 orthogonal, sorry 21:58:02 ok, that makes sense 21:58:18 i think any service who plans to consume quota from IaaS layer will have this use case say for example Trove, Magnum, Heat 21:58:42 we are running out of time 21:58:48 yeah 21:58:52 i think this topic need more discussion 21:58:55 and thought 21:59:17 I see DuncanT's point, let me think some more about this and see if I can take the two apart 21:59:30 and make quota's and resource tagging independent. 21:59:34 may be DuncanT, amrith would be great to start a ML thread on this and take communities opinion so that we have some direction 21:59:38 maybe we can table it for now 21:59:43 sure, we can do that ... 21:59:49 ++ 22:00:00 I added something to the agenda :) 22:00:05 #action : DuncanT, amrith to send ML thread and start discusssion regarding reservations 22:00:19 yes, we'd amrith :) 22:00:26 alrite folks thanks for attending have a great week 22:00:28 vilobhmm11: let's run over for a couple more mins :) 22:00:30 Writing a really dump prototype ASAP would be good too, start trying to hack it into the cinder, nova, glance code, see what fits 22:00:44 before we go, could we talk about a time for next meeting? 22:00:49 vilobhmm11: or we can discuss it outside of the meeting in the same channel 22:00:53 or is there another group that needs this channel? 22:01:02 nikhil, amrith : lets discuss outside 22:01:04 Yes please talk about times, it is 1 am here! 22:01:11 DuncanT: oops 22:01:13 What channel for discussions? 22:01:16 DuncanT: have a good night 22:01:20 same channel 22:01:24 #endmeeting