16:59:15 #startmeeting quotas-wg 16:59:15 Meeting started Tue Apr 19 16:59:15 2016 UTC and is due to finish in 60 minutes. The chair is vilobhmm11. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:59:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:59:19 The meeting name has been set to 'quotas_wg' 16:59:26 hello everyone 17:00:07 hello everyone 17:00:10 o/ 17:00:41 vilobhmm11: can you please highlight us on the channel, I've the courtesy reminder list added to the agenda etherpad.. thanks in advance! 17:00:41 hi so have updated the agenda for today 17:01:03 #link https://etherpad.openstack.org/p/quotas-wg-meeting-agenda 17:01:10 sure 17:01:11 nikhil, vilobhmm, DuncanT, mc_nair, ninag, amrith 17:01:45 nikhil : thanks for the link 17:01:49 yw! 17:01:58 hey :) 17:02:02 will wait for few min 17:02:04 hey raildo 17:02:30 hello all 17:02:38 :) 17:03:23 alrite lets start 17:03:35 Hi 17:03:45 #topic Meetup at Summit 17:04:14 Nikhil bought up a good point last meeting or an e-mail conversation that we should meetup on Monday 17:04:34 before we session on Tuesday 04/26 17:05:40 #link http://doodle.com/poll/e4nq9tc77bdv8zwr 17:05:41 we can meet at room 4a reserved for brown bag 17:05:41 Seems like a good plan 17:05:58 please vote! 17:06:06 nikhil : thanks again ! :) 17:06:14 np 17:07:18 sorry, am timeslicing with a phone call 17:08:05 as per the timings on the doodle we won't be getting the room reserved for brown bag but we can all meet near that room and then find a common one 17:08:23 otherwise it would be very hard to find each other 17:08:49 vilobhmm11: yeah, true 17:09:01 what's the best mode of communication here? email? 17:09:24 Email for me 17:09:33 email works for me, IRC won't most of the time 17:09:38 nikhil : email works for me too 17:09:45 cool 17:10:07 so please vote on the doodle 17:10:11 Thanks all. will email then. 17:10:38 we can gather votes and find a good time that works for everyone 17:10:44 #topic Product group getting started with requirements. 17:11:08 (it'd be great if harlowja can join us too given he is leading the CP session) 17:11:13 I've some substantial worries about this and scope creep 17:11:25 :) 17:11:38 nikhil : he will join; we discussed and i will be leading the CP session 17:11:47 Things like reserving quota for heat and similar - at least for cinder, that's pointless 17:11:57 vilobhmm11: ah k, thanks for the update. 17:12:03 harlowja : ping 17:12:44 DuncanT: reserving or enforcing ? :) 17:12:50 or both 17:12:52 nikhil : did request sdauge to for an update on the chair for the quota session ; should see an update soon… 17:13:10 vilobhmm11: excellent, thanks. I was about to ping you offline. 17:13:19 nikhil: what's the point of a reservation if it is anything other than enforcing? 17:13:30 DuncanT , nikhil : will come to reservation in a bit; its a long topic …lets cover smaller ones quickly…thanks! 17:13:42 ok 17:13:46 irc://irc.freenode.net:6667/#topic : Product group getting started with requirements. 17:13:51 irc://irc.freenode.net:6667/#topic Product group getting started with requirements 17:13:57 vilobhmm11: This isn't reservations in the currnet code sense :-) 17:14:04 but is this particular one tied to PW 17:14:07 ok :) 17:14:33 shamail and other have started gathering use-cases and requirement about how people use quota and do capacity management 17:14:44 is there a reference document for the scope? and for worries from DuncanT? 17:14:47 We've lots of terms beign thrown around, and we've still got no glossary - each mailing list thread picks its own terminology 17:15:19 a BIG +1 on "defining terms" 17:15:23 this will help to strengthen the use-cases problems with current quota mgmt 17:15:31 (please excuse my typing, I've substantial damage to two fingers and so it's rather tricky to type well and fast) 17:16:02 actually, that can be the additional info section in the spec and people _should_ refer to it while emailing 17:16:09 coming back to nikhil and DuncanT :) 17:16:17 I'm a strong proponent of matching terms to existing usage, e.g. reservation should be the code construct we already have in three projects 17:16:32 #topic No reservation 17:16:40 DuncanT : +1 17:17:17 So first question: what do we mean by 'reservation'? 17:17:17 I'm curious to know how we can handle quotas without reservations; the risk is that quotas are not a constraint if you don't have reservations (I THINK, AND MAY BE WRONG) 17:17:38 DuncanT, if that's a quesiton to me, I described it in my mail to the ML 17:17:41 having said that what it funcationaly translates to in every project and in delimiter can mean different things (if at all they are used) 17:17:46 * amrith shamlessly plugs email to the ML :) 17:17:55 heh 17:18:00 amrith: Got a link to your mail, please? 17:18:06 amrith : first of all thanks for the ML 17:18:30 http://openstack.markmail.org/thread/at3phitv3oupxupi 17:18:42 I got one comment from Jay, didn't realize it till now 17:18:44 have to read it 17:18:57 his comments are always very insightful 17:19:06 yup 17:19:08 they are 17:19:15 in short let me brief everyone 17:19:31 on why i am saying that even going with no-reservation can be an option 17:19:33 I think the advantage (and consequential disadvantages for "the other set") is that reservation is super useful if your assets/resources "DO have" a task potentially in process that is not highly elastic or distributed. 17:19:42 I strongly disagree with Jay about rate limiting not being a quota thing, for the record. I'll write a reply to him soon 17:20:34 vilobhmm11, I'm going to wait to hear what you say about no-reservations and hold off on reading Jay's comments 17:20:52 In the case of cinder, a reservation is just consumed quota attached to a request i nprogress. 17:21:00 in progress 17:21:30 exactly DuncanT, it's the "in process" part that's significant 17:21:36 because the race is outside of the DB 17:21:37 DuncanT : rate limiting or rate of request mgmt can be done even by external sources like repose http://www.openrepose.org/ 17:21:42 to elaborate on my point: Glance will not need reservation for storing images or number of images or members etc. I think the DB transaction semantics is enough. 17:21:48 nikhil: There shouldn't be a race 17:22:13 amrith : sure…in a min going through duncanT and nikhils comments 17:22:31 nikhil: Assuming the semantics are implemented without bugs, there are no race differences between getting a reservation and getting commited quota before returning from the create request 17:23:08 I see 17:23:24 so, we are talking about reservation vs. quota-commit+rollback 17:23:31 there's the third case 17:23:34 The only difference between a reservation and a commited quota usage is that the reservation can be cancelled if the consuming process dies (plus they are handy for some debugging, though not as handy as they could be) 17:23:43 that involved simply quota-commit on success 17:24:26 You need to(Semantically) quota commit before you create the resource, or the quota is useless against DoS and similar 17:24:27 * amrith is having a hard time demultiplexing conversations 17:24:54 DuncanT: not necessarily 17:25:07 for example: image-create call in glance 17:25:18 it's a simple record / row creation in the DB 17:25:25 the only thing that we need is: 17:25:33 1. create a image , update quota table 17:25:45 Plus storage used.... 17:25:51 2. once quota is full, keep returning non availability to the users 17:26:01 storage is different resource 17:26:17 so I am breaking things down for sanity sake 17:26:21 1. # images 17:26:36 2. amount of data stored for all images in a cloud account 17:26:50 3. # members on an image / account (depending on usage) 17:27:08 for #1, all we need quota commit on success 17:27:26 #2 is image upload for which it's a "task" or potentially long running process 17:27:28 nikhil: If I can create db resources (even db records, in principle) without quota check, then in theory I can bomb the system with parallel requests 17:28:07 nikhil: Glance is a pretty weird API, but the principle stands. If I have a quota of 10, it should be impossible for me to create more than 10 'things' 17:28:29 nikhil: I should never have got back a 200 or a 202 from the 11th request 17:28:37 Glance API is that way as Nova needs it so 17:29:06 nikhil: I can strongly disagree with that, but this isn't the venue -grab me over a beer 17:29:19 and to your point of 10 resources, if we implement test and set for quota check in glance we don't need reservation (As the transaction can be limited to DB only) 17:30:41 yeah, I don't know what you are referring to with the specifics of Glance API. If you ask me , cinder shouldn't be using glance (& use something like GLARE) in the first place but then again .. not the place for that :) 17:31:13 alrite first of all DuncanT : I agree with you on the point of *timelines for reservation and quota commit* nikhil : just to elaborate more you can think of reservation and few quota related tables as *cache* tables for actual resource consumption. They are nothing more than that! 17:31:32 amrith : ^^ 17:31:44 vilobhmm11: exactly 17:31:50 reservation and few quota tables are cache tables 17:31:50 and that's a race for glance 17:32:08 vilobhmm11, I've totally lost the train of the conversation 17:32:19 actual resource consumption happens in actual resource tables like nova.instances in nova or glance.images in glance 17:32:20 The one thing about cinder and nova reservations currently is that they can auto-uncommit if the system breaks down and the creation request is lost. That is badly done at the moment but might be a desirable property 17:32:28 I don't know who is replying to whom/what and I'm trying to untangle the threads 17:32:41 nikhil: Why is there a race? 17:33:04 amrith : just read my last comment you will get a gist of it 17:33:08 also, I think we're talking about the specifics of a specific service or a set of services, rather than talking about what a generic set of requirements may be for a bunch of services. 17:33:18 vilobhmm11, I read it; I don't understand why we're talking about tables. 17:33:22 that's an implementation detail 17:33:33 what I want to understand is what we're going to QUOTA'ify. 17:33:40 is it (for example) some predetermined set of resources 17:33:49 or can each consumer of thelibrary define their own set of resources 17:34:10 amrith : for existing project there are well defined set of resources right 17:34:15 amrith: Consumers need to define their own resource set. That's a hard requirement for cinder 17:34:35 DuncanT, I'll let you respond to that point that Jay makes (a static set of resources) 17:34:44 ok, then there's a question on the kinds of quotas we want 17:34:45 amrith: e.g. volume types (which are arbitrary in name and number) have individual quotas currently 17:34:46 upper bound 17:34:47 DuncanT : for existing projects too …like nova has ram, cpu, mem as resources 17:34:48 lower bound 17:34:49 rate 17:35:03 DuncanT : for existing projects too ? 17:35:21 then there's the question of what to do about resizes; should they be supported in the library or not 17:35:50 We need to support all of the quota operations used in projects today, or we won't get adopted 17:35:56 amrith : let me go over jay's reply and get back 17:36:05 then, are these quotas to be multi-tenant or system wide 17:36:13 vilobhmm11, this isn't about jay's email 17:36:15 amrith, DuncanT, nikhil :strongly agree on getting the scope right for the lib 17:36:16 I haven't read it 17:36:19 Resizes are supported in projects right now (e.g. cinder), ergo we need to wupport them 17:36:37 vilobhmm11: yes, and we need to meet personally for that! :)) 17:36:48 nikhil : ++ 17:36:59 then there's the question of how we intend dealing with the use-case where a user (say me) may have a quota of 5 instances and comes and makes two concurrent requests for 3 instances each 17:37:06 the first one we look at says 3 < 5, great 17:37:18 but since no instances have been actually created, the others says 3 < 5, great 17:37:25 and now we get 6 instances 17:37:31 and 6 > 5 17:37:46 we can figure out what we want to call it; reservations, preallocations, intents, ... 17:37:53 amrith: That is already handled - creating a reservation consumes quota. The first to grab the quota gets created, the second doesn't 17:38:19 vilobhmm11: I need 5 mins for open discussion 17:38:23 amrith: We can't go backwards from that sematic (we can fix the races that each project has that break those semantics) 17:38:30 nikhil : sure 17:39:37 so DuncanT what you are saying is that this new library is constrained by what existing projects are doing? 17:39:42 the timeline when the reservation is commited and when the quota commit happens are the same…also the validation applies at both places…amrith : the use case you mentioned above 17:39:49 amrith: That semantic *can* be implemented without any kind of reservation though, as long as you're on a single DB. You put the thing in the DB table in some 'creating' state (cinder and nova already do that) and count those as existing for quota purposes 17:40:24 DuncanT, depending on the isolation level of your database, that may or may not work 17:40:25 DuncanT : yup +1 17:41:00 sorry vilobhmm11, happy to sketch it out on a whiteboard and show you why not 17:41:06 amrith : question for you…what benefits you think that reservation is bringing right now and we might end up loosing it if it is removed ? 17:41:25 amrith : sure 17:41:40 I believe that the business that a service does is multi-step 17:41:48 and that is surely the case for something like trove 17:42:01 and therefore, being able to support multiple steps is essential 17:42:20 and having a database transaction be long lived and assuming that this will work is fraught with danger 17:42:37 so I'm viewing reservations as short lived database operations that help you break up the long lived database transaction 17:42:39 amrith: That is *not* what reservations (as the term is currently used in the code) is all about 17:43:06 ok, what does a reservation mean in the current usage in code 17:43:11 amrith: reservations are not an externally visible thing 17:43:24 amrith: They're a handle on a block of consumed quota 17:43:29 That's it 17:43:41 I think I'm describing the same thig 17:43:42 thing 17:43:52 not externally visible: yes 17:44:02 a handle on a block of consumed quota: yes 17:44:09 something that is 'in flight' 17:44:18 Ok, I'm just not understanding you then 17:44:24 to make a database instance I need an instance and a cinder volume 17:44:30 i make a reservation for cinder 17:44:34 then a reservation for an instance 17:44:36 No. 17:44:45 if the latter fails, I release the former 17:44:47 Definitely not. 17:44:58 sorry, I don't know what you mean by no; that's the use-case in Trove :) 17:45:09 There is no concept of trove 'making a reservation' for cinder 17:45:14 Currently 17:45:34 Reservations are just a construct inside cinder - no external consumer sees them 17:45:36 actually there is, but ... 17:45:42 here is what I mean 17:45:51 You do what you wish in cinder, that's a black box to me 17:45:53 Trove asks cinder to create a volume. That's the only operation it has 17:45:57 here are the limits that trove has 17:46:05 no user may have more than 10 db volumes 17:46:11 no user may have more than 10 db instances 17:46:19 remember that the user has a nova quota of 1000 instances 17:46:25 and a cinder quota of 500 volumes 17:46:34 trove has its own limits ON CINDER RESOURCES 17:46:42 that Trove has to enforce 17:46:50 MY point being, the word 'reservation' has a meaning in the code now. You're using it to mean something slightly different, and that is causing communication difficulties 17:47:02 sorry, I'm using it to mean the same thing 17:47:08 in the trove context 17:47:17 I need to make a reservaton for a cinder volume 17:47:21 and a reservation for an instnace 17:47:25 I don't tell cinder about it 17:47:27 you don't know 17:47:31 I make my own reservations 17:47:43 once I make my reservations and they all succeed, then it means the user has the quotas 17:47:50 then I go call cinder and ask for a volume 17:47:54 you could still fail the request 17:47:57 no guarantees 17:48:00 then I go ask nova 17:48:00 Gah, ok, so you're talking about trove's counting of cinder volumes? That was entirely not clear 17:48:06 it could succeed or fail 17:48:11 sorry 17:48:27 that was Trove's reservation for a cinder resource; not something cinder gets to know about 17:48:52 as I said in my email, "Such a hypothetical service may also consume resources from other services that it wishes to track, and impose limits on." 17:48:55 You could just commit all those reservations in trove, and release them later in exactly the same way as a delete - there's no need for a reservation 17:49:06 vilobhmm11: my point on defining terms and the ill consequences of over use of the same word .. :) (in practice) (no offense to anyone, just pointing out example of mis communications) 17:49:09 Just consume the resource 17:49:23 BAcking out a reservation is semantically the same as deleting it 17:49:53 nikhil : sure…but few terms which come over time should def be documented in glossary..point noted will take care 17:50:05 Sorry DuncanT how do you intend that I do that? 17:50:20 "just commit all those ..." 17:51:03 amrith: start transaction; count (volumes where tenant id = me); if count < quota insert into volumes table state = creating 17:51:14 Then do the same for the nova table 17:51:18 DuncanT, great 17:51:23 now let me tell you why that won't work 17:51:25 vilobhmm11: yes, agreed on glossary (true!, we can't really predict the future) :) 17:51:38 to begin, what's the database isolation level? I assume read-committed? 17:51:50 or would you like more than that? 17:51:55 would you like repeatable read 17:51:58 or serializable? 17:52:03 take your pick, let me know 17:52:08 I'm assuming read committed 17:52:12 do we agree? 17:52:23 * amrith waits 17:52:39 amrith: Honestly, I can only pick what is currently used in cinder, I've not looked the the terms for teh semantics I want for some time 17:52:46 amrith : read-commited means read whatever is written last right 17:52:51 NO 17:52:56 what ever was committed last 17:52:58 let me explain 17:53:00 starting state 17:53:03 0 instances 17:53:06 0 volumes 17:53:08 quota 5 of each 17:53:10 by written i mean commited 17:53:13 request 1: 3 instances 17:53:18 ok 17:53:21 begin transaction request 1 17:53:30 transaction 1: how many instances: answer 0 17:53:40 0 is less than 5, insert 3 in table 17:53:47 request 2: 3 instances please 17:53:52 begin transaction request 2 17:54:00 transaction 2: how many instance? 17:54:02 answer: 0 17:54:11 please give me 3 ... 17:54:14 insert 3 in table 17:54:20 transaction 1: do something about volumes 17:54:22 commit 17:54:28 tranaction 2: do something about volumes 17:54:29 commit 17:54:31 net result 17:54:33 6 instances 17:54:43 amrith: It is my understanding that one of those transations should fail 17:54:51 not if you run read committed 17:54:56 amrith: Or at least can be constructed that way 17:55:10 for that you increase isolation level for the db 17:55:14 to repeatable read 17:55:17 amrith: yes, that was the argument I was making earlier but in non-defined terms 17:55:17 or serializable 17:55:21 amrith : yes won't compare and swap or test and set happen at the db level 17:55:21 then performance issues begin 17:55:39 amrith: Ok, I now understand your point. Thank you. 17:55:41 vilobhmm11, yes, if you have the right isolation level 17:55:57 amrith: I also need to go examine some cinder db atomics code 17:56:33 I think the openstack general rule on sql-like DBs is read committed 17:56:34 and please read: http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-model.html 17:56:37 barring nova in some cases 17:56:39 the default is read-committed for mysql 17:56:55 yes, if you run in something more, be ready for deadlocks galore 17:57:17 anyway, I'll reply to jay on this aspect 17:57:28 amrith: So I think our best way forward is to define a python interface and its semantics and then figure out how to implement it (and stress test it)? 17:57:32 amrith: in glance, we explicitly implemented test&set for some update operations 17:57:48 amrith : but in openstack context for now we are mostly concered about sql db only no *nosql* in openstack right…so sql based i.e. mysql is read-commited by default.. 17:57:51 not sure how much of that is upstream (gah) 17:58:35 nikhil: We have something similar, but I think they only operate on one table and use update...where in our case 17:58:37 alrite guys we are running out of time 17:58:37 howdy 17:58:55 :) 17:58:56 hello harlowja…just wrapping up 17:59:03 I need a few mins pls 17:59:10 #topic open discussion 17:59:11 i'm around :-P 17:59:11 but on phone, ha 17:59:13 thanks 17:59:14 yes nikhil 17:59:14 nikhil gets 1 min 17:59:17 :) 17:59:19 ya, i'll lead, but u guys need to drive ;) 17:59:21 pong 17:59:22 harlowja : np 17:59:23 I was just thinking.. 17:59:32 glance midcycle is in boston/cambridge 17:59:36 june 15-17 17:59:51 I know amrith is in that area (from twitter) ? 17:59:59 would it be possible for others to join 18:00:09 may be the 14th for quotas? 18:00:18 I will be happy to attend but 15-17 is ... I hae a board meeting on 15, I can do 16 and 17 18:00:22 seems to me that summit will be too short for long discussions 18:00:27 I can do 14th 18:00:36 but I'd prefer 16th or 17th 18:00:48 That's a long flight for me, I'm unlikely to get funding 18:00:56 umm, let me ask shamail too 18:01:05 I can arrange for remote attendance 18:01:16 all I need is your full attention for half a day 18:01:18 remote is certainly doable 18:01:33 nikhil : need to check about funding..otherwise can join via hangouts 18:01:38 It will have to be morning though - I'm at GMT+2/3 18:01:39 should we move to #openstack-quota? 18:01:45 something that you can block in your .cal and let mgmt know unavailability for that time 18:01:57 sure 18:02:00 sure 18:02:02 we can discuss this at the summit 18:02:04 food for thought 18:02:09 lets move to #openstack-quota 18:02:10 thanks all! 18:02:14 thanks! 18:02:15 #endmeeting