16:01:19 <schwicke> #startmeeting hierarchical_multitenancy
16:01:20 <openstack> Meeting started Fri Oct  9 16:01:19 2015 UTC and is due to finish in 60 minutes.  The chair is schwicke. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:23 <openstack> The meeting name has been set to 'hierarchical_multitenancy'
16:01:27 <schwicke> hi, all
16:01:31 <ericksonsantos> Hi :)
16:01:40 <sajeesh> Hi all :-)
16:01:48 <iurygregory> Hello
16:02:00 <schwicke> We have one main topic today
16:02:07 <raildo> p/
16:02:10 <raildo> o/*
16:02:34 <schwicke> #topic hierarchical quota and overcommitting
16:02:57 <schwicke> not 100% sure how to structure that
16:03:35 <ericksonsantos> I was away because I was sick, but I've read the discussion about it.
16:03:42 <schwicke> The current implementation allows this only at the root level, is that correct ?
16:03:43 <sajeesh> schwicke ...the first question is that , whether to allow it or not ?
16:03:46 <ericksonsantos> I totally agree with Tim Bell.
16:03:53 <sajeesh> I mean in Nova
16:04:19 <raildo> the last time that I talked with johnthetubaguy, we was thinking in create two API calls
16:04:40 <raildo> schwicke: yes, on nova and on cinder too
16:05:09 <sajeesh> ok
16:05:39 <schwicke> I also agree that for this first round we should be consistent with what has been implemented for Cinder
16:05:48 <schwicke> Else this will be confusing to users.
16:06:30 <schwicke> in Cinder it is currently not allowed, correct ?
16:06:34 <sajeesh> I think John was interested in over commit ...Am I right ?
16:06:39 <raildo> schwicke: kind of... be consistent doesn't mean taht we don't need support overcommit
16:06:40 <ericksonsantos> schwicke, not for subprojects
16:06:57 <sajeesh> what about root project ?
16:07:07 <raildo> schwicke: this was the default behavior for the projects before HMT
16:07:21 <raildo> today, this works on neutron, cinder, magnum and nova
16:07:42 <raildo> so we can do that for every root project
16:07:53 <sajeesh> Currently , in nova I have coded like that only
16:08:02 <johnthetubaguy> raildo: hi can I help answer questions on my thoughts here?
16:08:25 <raildo> johnthetubaguy: sure
16:08:27 <sajeesh> Hi John ..we were talking about over commit
16:08:36 <johnthetubaguy> my basic worry is about us changing the general pattern for quotas
16:09:16 <johnthetubaguy> so many/most users overcommit their quota, its just there as a safety net
16:09:40 <johnthetubaguy> as a user, I would expect hierarchical quota to be the same idea as the regular quota
16:09:52 <raildo> johnthetubaguy: ++
16:10:01 <iurygregory> ++
16:10:09 <johnthetubaguy> now I see a use case for not having over-commit, and that should just work, you give people less quota, thats cool
16:10:44 <johnthetubaguy> I could see a case where we add extra things to make not allowing over-commit, in different ways, but thats a different feature really
16:11:07 <schwicke> For the root projects it makes sense.
16:11:09 <johnthetubaguy> my take, is by default, I would like us to keep allowing overcommit
16:11:14 <schwicke> I have some doubts for sub-projects though.
16:11:21 <sajeesh> schwicke ++
16:11:42 <ericksonsantos> johnthetubaguy, I see... so, I think we have to also support over-commit in Cinder, right? In order to keep consistency.
16:11:50 <sajeesh> johnthetubaguy: for subprojects too ?
16:12:01 <schwicke> Let's say the resource provider sets some quota on the root project.
16:12:02 <johnthetubaguy> ericksonsantos: that would be ideal, thats true
16:12:10 <ericksonsantos> johnthetubaguy, nice
16:12:12 <sajeesh> johnthetubaguy++1
16:12:29 <raildo> why not? It's really weird have different behaviours for projects in different levels...
16:12:35 <schwicke> then he creates some sub-projects and delegates the administration of those
16:12:35 <johnthetubaguy> sajeesh: so I am talking about subprojects and projects really, I expect them to work the same way
16:12:36 <raildo> this is bad design
16:12:49 <sajeesh> johnthetubaguy: ok
16:13:04 <johnthetubaguy> raildo: so to keep consistency, an openstack spec would be a good thing here
16:13:29 <sajeesh> ++1
16:13:40 <raildo> johnthetubaguy:you mean a cross project spec for every service that ahve quota control?
16:13:43 <raildo> have*
16:13:48 <johnthetubaguy> yeah
16:13:58 <raildo> johnthetubaguy: makes sense...
16:14:05 <johnthetubaguy> but lets not get ahead of ourselves here
16:14:13 <sajeesh> johnthetubaguy: makes sense ++1
16:14:16 <johnthetubaguy> I am just saying thats a good approach to get come consistency
16:14:28 <johnthetubaguy> we should probably do that, but anyways, lets get back to overcommit
16:14:35 <raildo> ok
16:15:01 <johnthetubaguy> so, there is a bit I don't quite get here, sorry for these dumb questions
16:15:21 <johnthetubaguy> so we have tenant A as a parent of tenant B and C
16:15:24 <schwicke> my worry is that if overcommit is allowed at all levels then the the upper levels loose control over the quota, making them meaningless
16:15:47 <schwicke> ok
16:15:52 <johnthetubaguy> so whats the starting position here
16:15:59 <sajeesh> ok
16:16:03 <schwicke> let's say A got some quota by the resource provider
16:16:05 <johnthetubaguy> they all have the default quota?
16:16:21 <johnthetubaguy> so before that
16:16:30 <sajeesh> A is will having default quota
16:16:37 <johnthetubaguy> A, B, C all have the default quota right?
16:16:37 <raildo> johnthetubaguy: the default quota for subprojects will be zero
16:16:46 <sajeesh> B and C will be having zero default quota in the begining
16:16:57 <johnthetubaguy> so how is a project and a sub project different?
16:17:12 <johnthetubaguy> that are all just tenants for nova
16:17:19 <johnthetubaguy> so I guess the starting point is
16:17:20 <sajeesh> johnthetubaguy: only root projects are different
16:17:33 <ericksonsantos> johnthetubaguy, we get this information in keystone
16:17:38 <johnthetubaguy> right, so we have to reconfigure nova at this point
16:17:51 <johnthetubaguy> so nova has a default tenant of zero?
16:17:58 <johnthetubaguy> oops, default quota of zero?
16:18:10 <ericksonsantos> johnthetubaguy, yes, just for subprojects
16:18:11 <sajeesh> johnthetubaguy: except for root projects
16:18:21 <raildo> johnthetubaguy: hum... got it, we are just setting default quota to zero, because the allocated quota...
16:18:22 <sajeesh> johnthetubaguy:yes
16:18:30 <schwicke> for root projects everything is as it is now
16:18:40 <sajeesh> ++1
16:18:53 <schwicke> The difference between sub-projects and is who manages them.
16:19:10 <sajeesh> johnthetubaguy: root projects will behave like the projects in DbQuotaDriver
16:19:44 <johnthetubaguy> well, thats a change from today I guess, right now everyone just gets the default quota regardless right?
16:19:55 <ericksonsantos> johnthetubaguy, yes
16:19:58 <johnthetubaguy> nova just sees user tokens for a specific tenant
16:20:02 <sajeesh> yes
16:20:19 <johnthetubaguy> OK, so the first change is Nova defaults sub tenants quota to zero
16:20:28 <sajeesh> yes
16:20:31 <raildo> johnthetubaguy: in this still happen with hierarchical multitenancy, the token is scoped olny for a project
16:20:50 <sajeesh> raildo:+1
16:21:14 <schwicke> +1
16:21:35 <johnthetubaguy> agreed the token is still for a single tenant
16:21:40 <johnthetubaguy> I guess thats my issue here in some ways
16:21:50 <johnthetubaguy> so let me describe what I was expecting I guess...
16:21:55 <sajeesh> ok
16:21:57 <johnthetubaguy> everyone gets a default qutoa
16:22:06 <johnthetubaguy> sub tenant or otherwise
16:22:26 <johnthetubaguy> actually, thats dumb I guess
16:22:43 <johnthetubaguy> ... so who creates the sub tenant usuaully?
16:22:49 <johnthetubaguy> then root tenant?
16:23:09 <sajeesh> we start from the root tenant
16:23:27 <schwicke> the owner of a project can create sub-tenants and determine their quota
16:23:28 <johnthetubaguy> sorry, s/then/the/
16:23:46 <johnthetubaguy> schwicke: right makes sense, just getting that straight in my head
16:24:13 <johnthetubaguy> so we could default to having no restriction on subtenant, they just consume quota of the parent
16:24:29 <schwicke> exactly !
16:24:59 <johnthetubaguy> in that world, the subtenant is simply used for scoping resources, etc, and organising, which is cool
16:25:25 <schwicke> sometimes it is easier to look at a real live example.
16:25:51 <johnthetubaguy> so in my head, that means each subtenant, basically has the same quota as the parent, except its consumes quota from that tenant, because its not a root project
16:26:28 <raildo> johnthetubaguy: but in the other world, my as a cloud provider want improve his profit by making better use of the cloud resources, doing overcommit below the hierarchy...
16:26:50 <sajeesh> johnthetubaguy: but we do check that sum of children quota will not exceed parent quota
16:26:54 <johnthetubaguy> so in this world, I assume the root has an overcommited quota anyways
16:27:11 <johnthetubaguy> sajeesh: agreed the current code doesn't, I guess I am saying it should
16:27:12 <raildo> johnthetubaguy: and a cloud provider (or project admin) can be present in a parent project (and not always a root project)
16:27:15 <sajeesh> johnthetubaguy: in that case yes it is
16:27:32 <sajeesh> johnthetubaguy:ok got your point
16:27:51 <johnthetubaguy> so lets back up
16:28:17 <johnthetubaguy> I guess I am expecting many root tenants, A and A' just acting like today, get there own default quota
16:28:30 <sajeesh> ok
16:28:34 <vilobhmm_> hi all
16:28:48 <johnthetubaguy> then can create sub teants B, C and B' C' that can consume their quota with them
16:28:52 <johnthetubaguy> thats cool, life is happy
16:29:09 <johnthetubaguy> in this case, B C just consume quota from A, no checks of their own
16:29:12 <vilobhmm_> joining late sorry bt that
16:29:25 <johnthetubaguy> same B' C' just consume quota of A' no extra checks
16:29:26 <ericksonsantos> johnthetubaguy, right
16:29:35 <schwicke> the trick here is that the person who creates A and a'  is not the same who creates B and b'
16:29:42 <johnthetubaguy> schwicke: +1
16:29:49 <ericksonsantos> schwicke, +1
16:29:51 <johnthetubaguy> OK, so we have the setup
16:29:54 <sajeesh> +1
16:29:58 <johnthetubaguy> that seems the good default here
16:30:03 <raildo> and we are not considering the quota for user =/
16:30:09 <ericksonsantos> johnthetubaguy, that's how we are doing in Cinder
16:30:57 <johnthetubaguy> so at some point A wants to limit B and C, becuase B just used up all the quota leaving A and C with no quota
16:31:30 <johnthetubaguy> seems fine, so A saying B can't use more than 1/4 of my quota, same with C
16:31:36 <vilobhmm_> joththetubag : yes https://review.openstack.org/#/c/205369/ this is the implementation for cinder nested quota driver
16:31:49 <schwicke> correct
16:32:06 <johnthetubaguy> OK, so the nova spec seems very different to this world
16:32:13 <sajeesh> yes
16:32:18 <johnthetubaguy> well, thats wrong
16:32:43 <ericksonsantos> johnthetubaguy, so, how do you think it should work in nova?
16:32:53 <johnthetubaguy> seems like I want what cinder has, for the Nova one, if I am understanding this all correctly
16:33:18 <ericksonsantos> oh, nice
16:33:20 <schwicke> the current way of working in nova is included as a sub-case which is when you have only root projects.
16:34:02 <johnthetubaguy> so we have per user quota I guess, is this the bit thats causing worries?
16:34:32 <ericksonsantos> yes, we are not sure about how to handle per user quotas
16:34:39 <raildo> johnthetubaguy: Do you expect this behaviour in the current DBQuotaDriver, right? in other words, a user doesn't need change any configuration for this works.
16:35:00 <johnthetubaguy> raildo: not sure if I understand that question yet
16:35:17 <johnthetubaguy> ericksonsantos: so just thinking about per user quotas, after reading this: https://blueprints.launchpad.net/nova/+spec/per-user-quotas
16:35:27 <johnthetubaguy> honestly, a user just consumes the project quota
16:35:49 <johnthetubaguy> it sounds like just another level or hierarcy, for users within the tenant
16:36:20 <raildo> johnthetubaguy: we have the dbquotadriver, today right? we can change this drive to include the hierarchy information, as we made on Cinder, or we can create a new driver, and the user should have to change the driver in nova.conf for this works.
16:36:21 <johnthetubaguy> similar to just having nested teants, except when you list your servers all users see each others servers in the same tenant
16:36:23 <ericksonsantos> johnthetubaguy, hm... so, we won't need to change it, right?
16:36:45 <johnthetubaguy> ericksonsantos: unsure, I suspect the code is not as clean as all that
16:37:02 <johnthetubaguy> ericksonsantos: but in theory, the behaviour can be the same, logically speaking :)
16:37:17 <sajeesh> johnthetubaguy: for nested projects in nova , users can be treated like sub projects right ?
16:37:25 <ericksonsantos> johnthetubaguy, I see... thanks for clarifying that
16:37:32 <johnthetubaguy> sajeesh: roughly yes, its the same deal
16:37:45 <sajeesh> johnthetubaguy:ok
16:37:48 <johnthetubaguy> so massive thank you for walking through all that
16:38:05 <johnthetubaguy> my head is much clearer on this now
16:38:14 <johnthetubaguy> I think we need the spec updating so it does that quite simple walk through
16:38:19 <raildo> awesome :D
16:38:30 <schwicke> +1
16:38:31 <johnthetubaguy> so there is one other issue here...
16:38:31 <sajeesh> ok
16:38:36 <sajeesh> johnthetubaguy:+1
16:38:42 <raildo> actuallyI updated the spec, to be bery closer to the cinder implementation
16:38:48 <raildo> but we can fix some points
16:39:00 <johnthetubaguy> the slight issue is that quotas are fairly broken at the moment
16:39:02 <raildo> to be closer*
16:39:26 <johnthetubaguy> they race like crazy, and the fix up logic, is messy, and no one really understands the code enough to fix it
16:39:26 <ericksonsantos> :(
16:39:48 <raildo> johnthetubaguy: we can put this as a next step for us
16:39:51 <johnthetubaguy> now honestly, thats mostly why its proving hard to get people to review and think about this
16:40:04 <sajeesh> raildo:++1
16:40:09 <johnthetubaguy> we do have a spec up with ideas to totally simplify the system
16:40:28 <johnthetubaguy> if I was given a choice, I would say, simplify it, then add the new features into the simpler system
16:40:47 <raildo> following that idea to create a cross project spec for nested quota in every service :)
16:41:10 <johnthetubaguy> thats me be honest, thats my preference here, fix quotas, then add the nesting
16:41:25 <johnthetubaguy> there is a horrible idea that just popped into my head....
16:41:32 <vilobhmm_> johnthetubaguy : nested quota just adds a heirarchy structure for quota management…
16:41:57 <raildo> johnthetubaguy: do you have some opened bugs related to this?
16:42:02 <johnthetubaguy> so generally, its best to fix something before you extend it, thats all really
16:42:04 <ericksonsantos> johnthetubaguy, can you send us  a link for this spec for us to take a look at?
16:42:35 <schwicke> +1
16:42:36 <johnthetubaguy> raildo: honestly, there are a few, but its mostly that there are lots of edge cases, and the quotas in production seem to drift out of sync with reality in odd ways
16:42:37 <raildo> johnthetubaguy: I think that we can goinf fixing some poing when we are adding nested quota...
16:42:52 <johnthetubaguy> so the fix here, is a re-write of the concept, thats the issue
16:42:57 <johnthetubaguy> let me find the spec for that...
16:43:04 <sajeesh> ok
16:43:29 <johnthetubaguy> https://review.openstack.org/#/c/182445/2/specs/backlog/quotas-reimagined.rst,cm
16:43:39 <schwicke> if the idea is a complete re-design would it not be better to do that from the beginning with support for nesting ?
16:43:44 <johnthetubaguy> we had a session on the last summit discussing all these quota issues, and that kinda of came out of that
16:44:01 <johnthetubaguy> schwicke: thats an option yes, I would go for that
16:44:10 <sajeesh> schwicke++1
16:44:21 <schwicke> what are the time scales for this ?
16:44:29 <johnthetubaguy> we could have a whole new quota driver, that takes the new approach
16:44:46 <sajeesh> johnthetubaguy ++1
16:44:49 <johnthetubaguy> the current problem, is I have been totally unable to get anyone to work on quotas to fix them up
16:45:18 <johnthetubaguy> so the timescale is currently towards the "unknown point in the future" rather than next cycle, or the one after
16:45:32 <johnthetubaguy> we have ideas, but not enough willing folks to work on the fixes right now
16:45:44 <raildo> I think that the addition of nested quota in nova in mitaka, will be a great feature
16:45:48 <johnthetubaguy> so I like the idea of doing nested from the beginning
16:46:01 <raildo> and I think that we can start working on this re-design without remove this feature for now
16:46:02 <johnthetubaguy> as a separate driver, written from scratch
16:46:15 <sajeesh> yes
16:46:50 <vilobhmm_> johnthetubag : qq so is fixing quota infrastruture a dependency for making nested quota in nova ? or nested quota for nova can go in mitaka and redesign in future release ? or both the efforts can go in parallel ?
16:47:20 <johnthetubaguy> so the problem is quotas don't work properly, I really don't like extending the broken thing
16:47:50 <johnthetubaguy> now we could add a new thing that supports nesting from the beginning
16:48:04 <sajeesh> ok
16:48:05 <johnthetubaguy> as suggested above, but that might be a big tasks
16:48:11 <johnthetubaguy> so the idea goes like this...
16:48:12 <sajeesh> yes
16:48:19 <schwicke> I'm a bit worried about the time lines to implement this.
16:48:31 <vilobhmm_> johnthetubag : thats true..its a huge task
16:48:36 <johnthetubaguy> have a db table with an entry for every quota "usage"
16:48:43 <raildo> and if we think in compatibility with cinder, this will not wappen, since the cinder code is very similar to the nova code
16:48:44 <johnthetubaguy> use a compare and swap style update
16:49:00 <johnthetubaguy> so add in the usage, then check if you have gone over, if you have gone over remove the usage
16:49:05 <raildo> so we might have to re-design quota for other service... and this is a huge risk
16:49:26 <johnthetubaguy> simple optimistic concurrency control stuff, basically
16:49:27 <raildo> happen*
16:49:39 <raildo> johnthetubaguy: lol
16:50:06 <johnthetubaguy> yeah, "simple" and concurrency in the same sentence, thats just wrong :)
16:50:24 <johnthetubaguy> the other idea is one about a cloths/washing line
16:50:26 <ericksonsantos> haha
16:50:43 <johnthetubaguy> basically, in the API layer, you have to pass a quota check, just like a policy check
16:50:49 <johnthetubaguy> there is a line in the sand you must pass
16:51:06 <johnthetubaguy> the checks ensure you don't have too much quota when you pass that line and consume the quota
16:51:23 <johnthetubaguy> when you delete the server, you decrement the qutoa (even if the delete fails)
16:51:36 <johnthetubaguy> i.e. we remove all the rollback complexity
16:51:56 <johnthetubaguy> (there is a dance around resize you need, but its the same idea there)
16:52:11 <johnthetubaguy> anyways, with all that, in place, we have a really simple system
16:52:35 <johnthetubaguy> and I think the nested stuff, should be quite easy to add into that system, its just a few more checks in the check phase
16:52:41 <johnthetubaguy> anyways, that spec tried to describe all that
16:52:57 <johnthetubaguy> so... I should shut up now
16:53:06 <raildo> haha
16:53:08 <johnthetubaguy> I guess the question is, how do we move forward...
16:53:17 <raildo> johnthetubaguy: ++
16:53:18 <johnthetubaguy> I think we update that spec, so we match cinder
16:53:33 <sajeesh> johnthetubaguy: thanks a lot for joining the meeting
16:53:39 <johnthetubaguy> I think we probably cut and paste that into the openstack specs repo, so we get cross project alignment
16:54:09 <johnthetubaguy> we then spend some time at the summit (in the unconference I think) to look at what the community things is the best approach
16:54:37 <johnthetubaguy> from our discussion here, sounds like Cinder has a nice simple system for nested, that is basically what we have with users already in nova
16:54:42 <johnthetubaguy> sounds the same anyways
16:54:49 <raildo> johnthetubaguy: Do you want that we change this spec, that you sent to match with the cinder, or the spec that we are moving for mitaka?
16:55:16 <johnthetubaguy> the sticking point is, fix quotas then add features, or add features then fix, I prefer the first one, but I don't have anyone to help with that yet
16:55:35 <johnthetubaguy> raildo: so I think we rework that moved spec to it matches cinder
16:55:43 <raildo> johnthetubaguy: ok
16:55:50 <sajeesh> johnthetubaguy ++1
16:55:56 <johnthetubaguy> raildo: that might just be a cut and paste of the cinder spec, with s/cinder/nova s/volumes/servers/
16:56:03 <schwicke> my feeling is that there is some sympathy for the approach here
16:56:07 <raildo> johnthetubaguy: ++ haha
16:56:44 <johnthetubaguy> schwicke: so I love the feature, I want the feature in Nova, the problem is how we make that happen in a sustainable way
16:57:47 <johnthetubaguy> so that was super helpful for me, thanks for letting my wreck your meeting here
16:57:50 <schwicke> sure
16:58:02 <schwicke> thanks a lot for joining!
16:58:05 <ericksonsantos> johnthetubaguy, thank you :)
16:58:05 <johnthetubaguy> does everyone feel like we have a path to move forward?
16:58:09 <sajeesh> johnthetubaguy: thanks a lot :-)
16:58:10 <schwicke> I also thing it was very useful
16:58:10 <raildo> thanks johnthetubaguy :)
16:58:17 <iurygregory> thanks =)
16:58:23 <schwicke> time is out
16:58:24 <sajeesh> johnthetubaguy: sure many things are clear now
16:58:29 <johnthetubaguy> hey, happy to help, will read the logs before I go through that spec :)
16:58:38 <vilobhmm_> thanks ! :)
16:58:41 <raildo> we have to finish the meeting :(
16:58:53 <schwicke> yep
16:58:59 <schwicke> #endmeeting