17:03:31 <vilobhmm11> #startmeeting quotas-wg
17:03:31 <openstack> Meeting started Tue Apr  5 17:03:31 2016 UTC and is due to finish in 60 minutes.  The chair is vilobhmm11. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:36 <openstack> The meeting name has been set to 'quotas_wg'
17:03:38 <raildo> p/
17:03:41 <raildo> o/*
17:03:54 <vilobhmm11> hi raildo, amrith
17:04:06 <vilobhmm11> will wait for few more min before others join
17:04:07 <raildo> hi vilobhmm11 :)
17:05:11 <vilobhmm11> hi
17:05:15 <vilobhmm11> so lets starts
17:05:17 <vilobhmm11> start
17:05:30 <vilobhmm11> #topic agenda
17:05:35 <vilobhmm11> #link https://etherpad.openstack.org/p/quotas-wg-meeting-agenda
17:06:37 <amrith> I have to confess that I've not read the latest version of the spec.
17:06:47 <vilobhmm11> as part of action item for last week we decided to drop an e-mail to the ML for getting more insights into reservations and are they really needed or not
17:07:08 <vilobhmm11> amrith, DuncanT : ^^
17:08:10 <vilobhmm11> amrith : sure np, I can go over it in a min
17:08:39 <vilobhmm11> since i didn't see a conversation i started one yesterday http://www.gossamer-threads.com/lists/openstack/dev/53889
17:08:56 <vilobhmm11> please feel free to share your opinions on this ML
17:09:21 <raildo> vilobhmm11: on the spec, I didn't find anything about the db migration between our current solution and this new service/lib
17:09:33 <vilobhmm11> so that we get more insights on this topic and hopefully can come to a conclusion in next week or so
17:09:38 <raildo> vilobhmm11: this will oln works for new deploys?
17:09:46 <raildo> will only*
17:10:05 <vilobhmm11> raildo : sure will come to it in a min l; let me just finish the reservation part
17:10:17 <raildo> vilobhmm11: sure
17:10:36 <amrith> vilobhmm11, I read the ml and I didn't respond because it seemed more to be targeted at the projects like nova/cinder/...
17:10:42 <DuncanT> Hi, sorry I'm late
17:10:48 <vilobhmm11> np DuncanT
17:11:41 <DuncanT> I saw the thread, and the reply about nested quotas is probably a stronger point than my usual one
17:11:45 <vilobhmm11> amrith : its not ; the title itself says cross-project but irrespectiev of that please share your opinion so that we as a community can get benefit from it
17:12:33 <vilobhmm11> DuncanT : are you talking in context of reservations ?
17:12:39 <raildo> nowadays, reservation it's really important in a big deploy. we now that the current solution have a lot of issues related to query atomicity... so we can have race conditions and other erros
17:12:44 <DuncanT> vilobhmm11: Yes
17:13:39 <raildo> and the service can use the reservetion to retry/rollback, instead of just say? "hey, we can't enforce the quota"
17:13:58 <DuncanT> vilobhmm11: My usual reason is that volume workers (or the scheduler) can go down, causing the thing processing the reservation to be lost, but in every case we have something in the DB that needs fixing up anyway, so we can release the quota there
17:14:45 <DuncanT> raildo: Why does than need reservations, rather than just atomically decrementing the quota in API, which is way easier to get right (at least without nested quotas)
17:15:11 <amrith> DuncanT, isn't the issue with keeping the reservation and allocation in sync with the actual deployment?
17:15:24 <amrith> i.e, would you want to hold a transaction open for the time it takes to provision an instance?
17:15:29 <amrith> that's my undertanding
17:15:32 <amrith> no?
17:15:36 <raildo> DuncanT: I'm not defending the reservation use, I'm just trying to explain how it's used today
17:16:06 <raildo> DuncanT: the problem is this "atomically drecrementing" doesn't work properly
17:16:09 <DuncanT> amrith: You can just commit the quota change when the API request is accepted and release it on failure, rather than bothering with the reservation
17:16:13 <raildo> with our current solution
17:16:26 <DuncanT> raildo: Atomic reservatioins are harder than atomic decrement
17:16:51 <vilobhmm11> DuncanT, amrith : imho the quota consumption from the quota_usages table and change of state for the resource should be one transaction
17:17:02 <amrith> DuncanT, I don't follow that last statement (decrements are harder)
17:17:50 <vilobhmm11> DuncanT, raildo, amrith : all of these are good points would request you to reply to ML so that we can have the conversation going on and take a better decision.
17:18:13 <raildo> #link https://bugs.launchpad.net/nova/+bug/1301532
17:18:14 <openstack> Launchpad bug 1301532 in OpenStack Compute (nova) "Quotas can be exceeded by making highly parallel requests" [Medium,Confirmed]
17:18:19 <DuncanT> amrith: update quota.available = N-1 where quota = N and project = foo; in a transation, retrying if it comes back matching 0 records (since the available quota changed
17:19:04 <DuncanT> raildo: I'm more familiar with the cinde rbugs, where the summing of reservations isn't atomic
17:19:59 <DuncanT> amrith: Anything with reservations is going to involve more records under the transaction, never fewer, so will always be at least as hard to commit
17:20:06 <vilobhmm11> raildo : thats there but does reservation guarantee fairness always ? can't we do without it ? do we really need it ? is what i am coming to which is what the ML is discussing about http://www.gossamer-threads.com/lists/openstack/dev/53889
17:20:20 <raildo> DuncanT: nether reservations and nether commit (independent of nested quota)
17:21:16 <vilobhmm11> DuncanT : so do you mean to say we don't need reservations ?
17:21:22 <vilobhmm11> atleast in cinder
17:21:46 <raildo> vilobhmm11: imo, no, we don't need reservation, but if we are removing this phase we need explain better what part of the new solution are making this work
17:22:21 <DuncanT> vilobhmm11: I don't thing reservations improve fairness, no. A reservation is sematically the same as having  consumed the quota. Rollback of nested quota is the only usecase so far, but we can always release whatever we would have released in the delete case
17:22:26 <vilobhmm11> raildo : sure we will do that
17:22:37 * amrith reads ... head hurts
17:22:43 <vilobhmm11> first need to decide if we need one
17:22:44 <DuncanT> vilobhmm11: I wonder if nested quotas on a busy system will livelock the db though?
17:22:59 <DuncanT> vilobhmm11: but then I don't see reservations solving that either
17:24:57 <vilobhmm11> DuncanT : ok…by fairness mean reservation help to to reserve a set of resources beforehand and hence eventually preventing any other upcoming request (serial or parallel) to exceed quota if because of original request the project might have reached the quota limits.
17:25:39 <vilobhmm11> #action  Lets discuss more about reservation on ML. Would request everyone to share there opinion here http://www.gossamer-threads.com/lists/openstack/dev/53889
17:25:53 <amrith> +1
17:26:03 <vilobhmm11> raildo : coming back to your question
17:26:10 <DuncanT> vilobhmm11: To do that, you need to do an atomic sum of the existing quota usage + the existing reservations... that's no easier than just atomically consuming the quota. Same effect too as long as you can guarentee quota release on failure
17:26:38 <DuncanT> vilobhmm11: The lack of atomics in the current implementation is a critical problem
17:28:24 <vilobhmm11> DuncanT : that what i am saying quota consumption  + resource state change should be under one transaction. There are many unknown on this topic and hence would like us to chart it out.
17:28:50 <DuncanT> vilobhmm11: Why resource state change? Which state change?
17:29:06 <vilobhmm11> DuncanT : in-use => active
17:29:41 <DuncanT> vilobhmm11: A cinder volume can go through several state changes on the way to in-use
17:30:21 <DuncanT> vilobhmm11: As long as you give the quota back on failure, there's no need or reason for it to be in the same transaction that I can see
17:31:37 <vilobhmm11> raildo : last week we decided to that it makes more sense to design delimeter as a lib than a service. Doing so has advantage that we need not worry abt data migration as athe respective projects will own quota relaed information
17:32:30 <DuncanT> vilobhmm11: Something still needs to get the data out of whatever format it is currently in and into whatever format the library needs
17:32:40 <vilobhmm11> raildo : does that make sense?
17:33:02 <vilobhmm11> DuncanT : yes and the engine will take care of it
17:33:28 <DuncanT> vilobhmm11: Which engine? Sorry, I've missed something here
17:33:48 <vilobhmm11> The Quota Engine as proposed in spec
17:33:51 <vilobhmm11> https://review.openstack.org/#/c/284454/
17:34:05 <vilobhmm11> Duncant : would request you to go through the spec https://review.openstack.org/#/c/284454/ if you haven't done so
17:34:33 <vilobhmm11> where I have covered in detail about the various aspects of the library "Delimiter"
17:34:54 <DuncanT> vilobhmm11: I'll go through again before further comment, thanks. I don't remember enough of it
17:35:29 <vilobhmm11> DuncanT : np
17:36:31 <vilobhmm11> so DuncanT, amrith, raildo : thats it from my end. Do you guys have anything more to talk about ?
17:36:52 <vilobhmm11> I will post a new revision of the spec by today eod
17:36:57 <amrith> vilobhmm11, I'm going to respond to the ML and read the scrollback again.
17:37:03 <amrith> then I'll read the new spec
17:37:07 <vilobhmm11> amrith : sure thanks!
17:37:13 <amrith> also, we can ignore the action item re: new meeting time.
17:37:16 <amrith> sorry agenda item
17:37:39 <DuncanT> vilobhmm11: Ok, I've re-read the spec, I don't think it covers the data migration issues in enough detail, there are loads of issues I can see.
17:38:14 <DuncanT> vilobhmm11: I'll put them in an email, I don't thing they're going to be easy to detail on IRC
17:39:04 <vilobhmm11> DuncanT : do you still think we might come across data migration issues even if delimeter is a library ? sure DuncanT that would really help
17:40:10 <raildo> vilobhmm11: sounds good to me
17:40:22 <DuncanT> vilobhmm11: It being a library doesn't change much - there's still existing data, being accessed in real rime (including reservations etc) by the existing code, that you have to be able to live upgrade from with parallel accesses still happening by the old code
17:41:05 <vilobhmm11> DuncanT : I think the data migration scripts in respective projects can take care of it…the conversion of data format and the computation based of it would be the responsibility of the engine.
17:41:13 <vilobhmm11> DuncanT : lets go step by step
17:41:50 <vilobhmm11> #1. quota_engine = delimiter.QuotaEngine()
17:41:59 <DuncanT> vilobhmm11: Migration script can't do bulk data conversion, since it locks the db for too long on a live system
17:42:02 <vilobhmm11> create engine
17:43:55 <vilobhmm11> DuncanT : ok but how its being done right now…don't we see the problems in current design ?
17:44:46 <vilobhmm11> DuncanT : would be nice to go through the example on a step by step basis
17:45:05 <DuncanT> vilobhmm11: Yes, there are issues, but there's no longer a 'stop everything' point during upgrade - we are live upgrade now, so changing data formats is much, much harder
17:45:12 <amrith> vilobhmm11, sorry, have to run; just got tapped on the shoulder by someone who has to speak with me.
17:45:49 <vilobhmm11> amrith : sure…
17:45:52 <vilobhmm11> np
17:46:35 <DuncanT> vilobhmm11: I'm happy to go through an example, I'm just not sure what the problems will be until we know the rough requirements of the new code
17:47:02 <vilobhmm11> DuncanT : so if you can comment on the spec with all these problem and a step by step example as in how we might run into it without the library having the control of data
17:47:11 <vilobhmm11> would be happy to address and enhance the spec
17:47:52 <DuncanT> Sure. It's going to be vague though, until the new library is firmed up a bit
17:48:43 <vilobhmm11> DuncanT : it shouldn't be vague ; it should be based of what you have experience before or what problems you see in the current code which can be improved upon
17:49:11 <vilobhmm11> #topic open-dicussion
17:49:39 <vilobhmm11> since we are running out of time would like to see if anyone has any comments suggestion
17:49:41 <DuncanT> vilobhmm11: Reservations or not in the new code? That changes lots of things. Exactly how are we going to store hieracical quota data? That changes a few things.
17:51:45 <vilobhmm11> DuncanT : hence I wanted to chart it out first ; will "delimiter" - quota lib need reservation or not based on our prior experience and code understanding…lets see if the ML helps
17:53:34 <vilobhmm11> raildo, DuncanT : if we don't have anything more to discuss we should be good then we can meet again next week. Feel free to update the agenda https://etherpad.openstack.org/p/quotas-wg-meeting-agenda
17:53:48 <raildo> vilobhmm11: nothing from my side
17:53:53 <DuncanT> Will do, thanks
17:54:04 <vilobhmm11> thanks raildo, DuncanT
17:54:05 <DuncanT> Nothing more for now
17:54:14 <vilobhmm11> see you again next week
17:54:44 <vilobhmm11> #action Discuss more on reservation on ML thread
17:54:54 <vilobhmm11> #endmeeting