17:03:31 #startmeeting quotas-wg 17:03:31 Meeting started Tue Apr 5 17:03:31 2016 UTC and is due to finish in 60 minutes. The chair is vilobhmm11. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:03:33 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:03:36 The meeting name has been set to 'quotas_wg' 17:03:38 p/ 17:03:41 o/* 17:03:54 hi raildo, amrith 17:04:06 will wait for few more min before others join 17:04:07 hi vilobhmm11 :) 17:05:11 hi 17:05:15 so lets starts 17:05:17 start 17:05:30 #topic agenda 17:05:35 #link https://etherpad.openstack.org/p/quotas-wg-meeting-agenda 17:06:37 I have to confess that I've not read the latest version of the spec. 17:06:47 as part of action item for last week we decided to drop an e-mail to the ML for getting more insights into reservations and are they really needed or not 17:07:08 amrith, DuncanT : ^^ 17:08:10 amrith : sure np, I can go over it in a min 17:08:39 since i didn't see a conversation i started one yesterday http://www.gossamer-threads.com/lists/openstack/dev/53889 17:08:56 please feel free to share your opinions on this ML 17:09:21 vilobhmm11: on the spec, I didn't find anything about the db migration between our current solution and this new service/lib 17:09:33 so that we get more insights on this topic and hopefully can come to a conclusion in next week or so 17:09:38 vilobhmm11: this will oln works for new deploys? 17:09:46 will only* 17:10:05 raildo : sure will come to it in a min l; let me just finish the reservation part 17:10:17 vilobhmm11: sure 17:10:36 vilobhmm11, I read the ml and I didn't respond because it seemed more to be targeted at the projects like nova/cinder/... 17:10:42 Hi, sorry I'm late 17:10:48 np DuncanT 17:11:41 I saw the thread, and the reply about nested quotas is probably a stronger point than my usual one 17:11:45 amrith : its not ; the title itself says cross-project but irrespectiev of that please share your opinion so that we as a community can get benefit from it 17:12:33 DuncanT : are you talking in context of reservations ? 17:12:39 nowadays, reservation it's really important in a big deploy. we now that the current solution have a lot of issues related to query atomicity... so we can have race conditions and other erros 17:12:44 vilobhmm11: Yes 17:13:39 and the service can use the reservetion to retry/rollback, instead of just say? "hey, we can't enforce the quota" 17:13:58 vilobhmm11: My usual reason is that volume workers (or the scheduler) can go down, causing the thing processing the reservation to be lost, but in every case we have something in the DB that needs fixing up anyway, so we can release the quota there 17:14:45 raildo: Why does than need reservations, rather than just atomically decrementing the quota in API, which is way easier to get right (at least without nested quotas) 17:15:11 DuncanT, isn't the issue with keeping the reservation and allocation in sync with the actual deployment? 17:15:24 i.e, would you want to hold a transaction open for the time it takes to provision an instance? 17:15:29 that's my undertanding 17:15:32 no? 17:15:36 DuncanT: I'm not defending the reservation use, I'm just trying to explain how it's used today 17:16:06 DuncanT: the problem is this "atomically drecrementing" doesn't work properly 17:16:09 amrith: You can just commit the quota change when the API request is accepted and release it on failure, rather than bothering with the reservation 17:16:13 with our current solution 17:16:26 raildo: Atomic reservatioins are harder than atomic decrement 17:16:51 DuncanT, amrith : imho the quota consumption from the quota_usages table and change of state for the resource should be one transaction 17:17:02 DuncanT, I don't follow that last statement (decrements are harder) 17:17:50 DuncanT, raildo, amrith : all of these are good points would request you to reply to ML so that we can have the conversation going on and take a better decision. 17:18:13 #link https://bugs.launchpad.net/nova/+bug/1301532 17:18:14 Launchpad bug 1301532 in OpenStack Compute (nova) "Quotas can be exceeded by making highly parallel requests" [Medium,Confirmed] 17:18:19 amrith: update quota.available = N-1 where quota = N and project = foo; in a transation, retrying if it comes back matching 0 records (since the available quota changed 17:19:04 raildo: I'm more familiar with the cinde rbugs, where the summing of reservations isn't atomic 17:19:59 amrith: Anything with reservations is going to involve more records under the transaction, never fewer, so will always be at least as hard to commit 17:20:06 raildo : thats there but does reservation guarantee fairness always ? can't we do without it ? do we really need it ? is what i am coming to which is what the ML is discussing about http://www.gossamer-threads.com/lists/openstack/dev/53889 17:20:20 DuncanT: nether reservations and nether commit (independent of nested quota) 17:21:16 DuncanT : so do you mean to say we don't need reservations ? 17:21:22 atleast in cinder 17:21:46 vilobhmm11: imo, no, we don't need reservation, but if we are removing this phase we need explain better what part of the new solution are making this work 17:22:21 vilobhmm11: I don't thing reservations improve fairness, no. A reservation is sematically the same as having consumed the quota. Rollback of nested quota is the only usecase so far, but we can always release whatever we would have released in the delete case 17:22:26 raildo : sure we will do that 17:22:37 * amrith reads ... head hurts 17:22:43 first need to decide if we need one 17:22:44 vilobhmm11: I wonder if nested quotas on a busy system will livelock the db though? 17:22:59 vilobhmm11: but then I don't see reservations solving that either 17:24:57 DuncanT : ok…by fairness mean reservation help to to reserve a set of resources beforehand and hence eventually preventing any other upcoming request (serial or parallel) to exceed quota if because of original request the project might have reached the quota limits. 17:25:39 #action Lets discuss more about reservation on ML. Would request everyone to share there opinion here http://www.gossamer-threads.com/lists/openstack/dev/53889 17:25:53 +1 17:26:03 raildo : coming back to your question 17:26:10 vilobhmm11: To do that, you need to do an atomic sum of the existing quota usage + the existing reservations... that's no easier than just atomically consuming the quota. Same effect too as long as you can guarentee quota release on failure 17:26:38 vilobhmm11: The lack of atomics in the current implementation is a critical problem 17:28:24 DuncanT : that what i am saying quota consumption + resource state change should be under one transaction. There are many unknown on this topic and hence would like us to chart it out. 17:28:50 vilobhmm11: Why resource state change? Which state change? 17:29:06 DuncanT : in-use => active 17:29:41 vilobhmm11: A cinder volume can go through several state changes on the way to in-use 17:30:21 vilobhmm11: As long as you give the quota back on failure, there's no need or reason for it to be in the same transaction that I can see 17:31:37 raildo : last week we decided to that it makes more sense to design delimeter as a lib than a service. Doing so has advantage that we need not worry abt data migration as athe respective projects will own quota relaed information 17:32:30 vilobhmm11: Something still needs to get the data out of whatever format it is currently in and into whatever format the library needs 17:32:40 raildo : does that make sense? 17:33:02 DuncanT : yes and the engine will take care of it 17:33:28 vilobhmm11: Which engine? Sorry, I've missed something here 17:33:48 The Quota Engine as proposed in spec 17:33:51 https://review.openstack.org/#/c/284454/ 17:34:05 Duncant : would request you to go through the spec https://review.openstack.org/#/c/284454/ if you haven't done so 17:34:33 where I have covered in detail about the various aspects of the library "Delimiter" 17:34:54 vilobhmm11: I'll go through again before further comment, thanks. I don't remember enough of it 17:35:29 DuncanT : np 17:36:31 so DuncanT, amrith, raildo : thats it from my end. Do you guys have anything more to talk about ? 17:36:52 I will post a new revision of the spec by today eod 17:36:57 vilobhmm11, I'm going to respond to the ML and read the scrollback again. 17:37:03 then I'll read the new spec 17:37:07 amrith : sure thanks! 17:37:13 also, we can ignore the action item re: new meeting time. 17:37:16 sorry agenda item 17:37:39 vilobhmm11: Ok, I've re-read the spec, I don't think it covers the data migration issues in enough detail, there are loads of issues I can see. 17:38:14 vilobhmm11: I'll put them in an email, I don't thing they're going to be easy to detail on IRC 17:39:04 DuncanT : do you still think we might come across data migration issues even if delimeter is a library ? sure DuncanT that would really help 17:40:10 vilobhmm11: sounds good to me 17:40:22 vilobhmm11: It being a library doesn't change much - there's still existing data, being accessed in real rime (including reservations etc) by the existing code, that you have to be able to live upgrade from with parallel accesses still happening by the old code 17:41:05 DuncanT : I think the data migration scripts in respective projects can take care of it…the conversion of data format and the computation based of it would be the responsibility of the engine. 17:41:13 DuncanT : lets go step by step 17:41:50 #1. quota_engine = delimiter.QuotaEngine() 17:41:59 vilobhmm11: Migration script can't do bulk data conversion, since it locks the db for too long on a live system 17:42:02 create engine 17:43:55 DuncanT : ok but how its being done right now…don't we see the problems in current design ? 17:44:46 DuncanT : would be nice to go through the example on a step by step basis 17:45:05 vilobhmm11: Yes, there are issues, but there's no longer a 'stop everything' point during upgrade - we are live upgrade now, so changing data formats is much, much harder 17:45:12 vilobhmm11, sorry, have to run; just got tapped on the shoulder by someone who has to speak with me. 17:45:49 amrith : sure… 17:45:52 np 17:46:35 vilobhmm11: I'm happy to go through an example, I'm just not sure what the problems will be until we know the rough requirements of the new code 17:47:02 DuncanT : so if you can comment on the spec with all these problem and a step by step example as in how we might run into it without the library having the control of data 17:47:11 would be happy to address and enhance the spec 17:47:52 Sure. It's going to be vague though, until the new library is firmed up a bit 17:48:43 DuncanT : it shouldn't be vague ; it should be based of what you have experience before or what problems you see in the current code which can be improved upon 17:49:11 #topic open-dicussion 17:49:39 since we are running out of time would like to see if anyone has any comments suggestion 17:49:41 vilobhmm11: Reservations or not in the new code? That changes lots of things. Exactly how are we going to store hieracical quota data? That changes a few things. 17:51:45 DuncanT : hence I wanted to chart it out first ; will "delimiter" - quota lib need reservation or not based on our prior experience and code understanding…lets see if the ML helps 17:53:34 raildo, DuncanT : if we don't have anything more to discuss we should be good then we can meet again next week. Feel free to update the agenda https://etherpad.openstack.org/p/quotas-wg-meeting-agenda 17:53:48 vilobhmm11: nothing from my side 17:53:53 Will do, thanks 17:54:04 thanks raildo, DuncanT 17:54:05 Nothing more for now 17:54:14 see you again next week 17:54:44 #action Discuss more on reservation on ML thread 17:54:54 #endmeeting