15:00:42 <n0ano> #startmeeting gantt
15:00:44 <openstack> Meeting started Tue Jun 17 15:00:42 2014 UTC and is due to finish in 60 minutes.  The chair is n0ano. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:47 <openstack> The meeting name has been set to 'gantt'
15:00:51 <n0ano> anyone here to talk about the scheduler?
15:00:55 <bauzas> n0ano: o/
15:00:59 <tianst> 哦/
15:01:04 <tianst> o/
15:01:06 <toan-tran> o/
15:01:12 <mspreitz> here
15:01:29 <jgallard> hi
15:01:32 <n0ano> tianst, I like your first response (if I only knew what it meant :-)
15:01:47 <tianst> :)
15:02:15 <n0ano> let's get started, I'll try and make it short today
15:02:23 <bauzas> :)
15:02:25 <n0ano> #topic forklift status
15:02:34 <bauzas> right :)
15:02:34 <n0ano> bauzas, anything good to report?
15:02:48 <bauzas> well, some updates :)
15:03:07 <bauzas> after a looong discussion with johnthetubaguy, decision has been made to simplify the propoal
15:03:18 <bauzas> so, we remove use of object for it
15:03:33 <bauzas> and scheduler doesn't have to manage its own view of services
15:03:50 <bauzas> so, the update is just coming in, and the DB got updated
15:04:01 <bauzas> https://review.openstack.org/82778
15:04:14 <bauzas> it's up to review, no work left
15:04:15 <n0ano> bauzas, that will still lead to clean, easily separatable interfaces, right?
15:04:33 <bauzas> well that's my point on a dependent patch
15:04:47 <bauzas> atm, this code is a library run by the compute host
15:05:14 <bauzas> so that only provides a way to "isolate" db accesses
15:05:34 <bauzas> but speaking of that, computes are still directly updating db
15:05:45 <bauzas> for scheduler compute_nodes table
15:06:09 <n0ano> computes updating the DB is not a problem, it's the scheduler access we need to isolate
15:06:25 <bauzas> n0ano: let me rephrase
15:06:46 <bauzas> computes are updating still updating a DB table
15:06:53 <bauzas> related to the scheduler
15:07:34 <bauzas> the proposal for letting Scheduler the only access to compute_nodes is here : https://review.openstack.org/97232
15:07:58 <mspreitz> bauzas: which table are you complaining about?
15:08:06 <bauzas> mspreitz: compute_nodes
15:08:10 <n0ano> ah, that to me is unrelated to splitting the scheduler out into the gantt tree, that's my main focus
15:08:19 <bauzas> mspreitz: which is the mother of all the scheduling decisions
15:08:21 <marun> http://omrasdfsdfshttp://omreunionproject.org/http://omreunionproject.org/eunionproject.org/
15:08:58 <bauzas> n0ano: that's related to it :)
15:09:19 <bauzas> if we split out the scheduler, we will need to provide a client library for compute nodes
15:09:43 <toan-tran> bauzas: +1
15:09:52 <n0ano> the client library either sends an api to the schduler or updates the DB, in either case it's unrelated to gantt code
15:10:01 <bauzas> if we leave the situation as it is, Nova conductor is still updating DB's compute_nodes
15:10:12 <mspreitz> so the debate here is really about where the split is
15:10:40 <bauzas> that's unrelated to gantt code, not data ;)
15:10:43 <toan-tran> n0ano: it's related in the sense that we have to use some call from *some service* to scheduler
15:11:00 <toan-tran> maybe nova-api
15:11:46 <n0ano> I think an API is fine but I just don't see it as a necessary precursor to splitting the scheduler code into gantt
15:12:42 <bauzas> mmm
15:13:00 <bauzas> I agree with the thing we draw a line
15:13:23 <bauzas> with compute not directly talking to conductor, but scheduler rather
15:13:46 <bauzas> but that scheduler.client portion of code will be run by Nova
15:14:04 <bauzas> while it is part of a Gantt code
15:15:46 <n0ano> so, to be clear, we need the 82778 change, do we also think that 97232 is required before we try and split the code out?
15:15:47 <toan-tran> bauzas: I'm confused, are you talking about the database access? or schuduling request?
15:17:06 <bauzas> toan-tran: I'm talking about nova computes running a Gantt code to update DB
15:17:17 <bauzas> n0ano: well, we can try with 82778 only
15:17:32 <n0ano> bauzas, that would be my hope
15:17:46 <bauzas> n0ano: but the problem is that Nova will need to import Gantt as a whole
15:18:20 <toan-tran> bauzas: that's not https://review.openstack.org/#/c/82778/ is about
15:18:20 <n0ano> bauzas, that would kind of defeat the purpose, why would nova need all of gantt?
15:19:20 <toan-tran> bauzas: ok so it's https://review.openstack.org/#/c/97232/
15:19:28 <bauzas> toan-tran: right
15:19:42 <bauzas> n0ano: I mean the scheduler client code has to be left in Nova
15:19:52 <bauzas> n0ano: and not be part of the fork
15:20:05 <bauzas> n0ano: that's a client library
15:20:36 <n0ano> the client library, yes we can do that as a second effort, as long as we split the scheduler proper out that the main goal
15:20:54 <n0ano> we do need the client library but we can do that later
15:21:04 <bauzas> n0ano: sorry, it seems we are not on the same page
15:21:06 <bauzas> :)
15:21:12 <bauzas> I'm trying to reformulate
15:21:21 <bauzas> so, my concern is that
15:21:35 <bauzas> if we say that all scheduler.* namespace becomes gantt
15:21:50 <bauzas> that's an external project
15:22:10 <bauzas> you ok?
15:22:40 <n0ano> I believe so, yes
15:22:45 <bauzas> ok
15:23:27 <bauzas> so, if Nova still has to call Scheduler for either select_destinations() or update_resource_stats(), it will need to import a certain portion of code
15:23:35 <bauzas> by importing, I mean Python importing
15:23:46 <bauzas> from ... import client
15:24:16 <bauzas> so we can't impose nova to declare gantt as a dependency
15:24:46 <bauzas> so, I'm just saying that nova.scheduler.client has to be left where it is
15:25:01 <bauzas> you ok ?
15:25:11 <toan-tran> bauzas: *where*?
15:25:18 <n0ano> we have to think about that, importing from a different project causes problems with testing
15:25:42 <bauzas> n0ano: right, that's why nova.scheduler.client will still be nova.scheduler.client
15:25:51 <bauzas> and not become gantt.client
15:26:00 <toan-tran> bauzas: if you're talking about cross-project call, then it's better in common or oslo
15:26:15 <bauzas> toan-tran: trust me, that's not that common :)
15:26:28 <bauzas> anyway
15:26:37 <bauzas> I'm just saying that :
15:26:50 <bauzas> A/ the scheduler.client code will remain in Nova
15:27:03 <bauzas> B/ all scheduler.* except scheduler.client will become Gantt
15:27:21 <bauzas> so the dependency is far less strong
15:27:23 <n0ano> well, the ultimate goal is to have a `python-ganttclient', like other projects, and isolate the client access that way
15:27:27 <toan-tran> bauzas: ok, I think I understand what you're saying
15:27:37 <bauzas> n0ano: agreed, that will be future plan
15:27:55 <toan-tran> basically you want to create an *interface* that scheduler listent to update db
15:28:02 <bauzas> n0ano: at the moment, we only have that nova.scheduler.client proposed in 82778 to be the last standing man
15:28:09 <toan-tran> and create something inside nova-compute to use that interface
15:28:25 <toan-tran> but instead of using an API, you use python call
15:28:55 <toan-tran> is that right?
15:29:14 <bauzas> toan-tran: I'm just thinking of a way to have a clean interface now, before having a python client :)
15:29:41 <bauzas> so, if we say nova.scheduler.client is the last standing man in Nova
15:29:53 <toan-tran> bauzas: I'm agreed that we would have to make some channel from nova to gantt to update DB
15:30:09 <toan-tran> I'm just worried that the channel is low-level function call
15:30:10 <bauzas> that means that we need to have compatibility in between nova.scheduler.client code and Scheduler
15:30:28 <toan-tran> it should be temporary
15:30:59 <bauzas> toan-tran: if we want a python-ganttclient we need a gantt-api :)
15:31:13 <n0ano> bauzas, +1
15:31:18 <toan-tran> bauzas: +1
15:31:31 <n0ano> the question is do we need python-ganttclient before we can make the split
15:31:40 <bauzas> n0ano: nope
15:31:47 <bauzas> n0ano: RPC API is enough
15:32:04 <n0ano> then I'm good, stepwise development works for me
15:32:17 <bauzas> n0ano: hence https://review.openstack.org/#/c/97232/
15:32:40 <n0ano> actually, that leads to the action for all
15:33:01 <n0ano> #all to review https://review.openstack.org/#/c/82778 and https://review.openstack.org/#/c/97232/
15:33:10 <bauzas> n0ano: thanks
15:33:34 <n0ano> I think we should move on
15:33:40 <bauzas> n0ano: as I'm just trying to explain, 97232 is the only piece of code needed to have a clear interface in between Gantt and Nova
15:34:09 <n0ano> bauzas, +1
15:34:18 <bauzas> n0ano: we can move on without that piece of code, but then we will need to manage compatibilty at the DB level
15:34:42 <n0ano> #topic policy based scheduler
15:35:01 <toan-tran> bauzas: I suggest you put the roadmap somewhere, to know which patch corresponds to which piece of the roadmap
15:35:09 <n0ano> toan-tran, I believe this is your topic, I noticed there was some reviews (fairly negative) on your BP
15:35:21 <bauzas> n0ano: could you please undo ? there is a WIP patch for isolate-scheduler-db
15:35:50 <n0ano> #topic code forklift
15:35:59 <n0ano> sorry, got ahead of myself
15:36:05 <tianst> isolate scheduler db, I put a daft patch , just want to make sure something,  one :if i did the right thing? two：need Depends bauzas's patch and split the API update to one patch?
15:36:24 <tianst> n0ano :0
15:36:31 <tianst> :)
15:36:43 <bauzas> tianst: well, I made some comments
15:36:54 <bauzas> tianst: maybe we could discuss that later on
15:36:59 <tianst> bauzas : I see thanks
15:37:08 <bauzas> tianst: btw, your topic name is incorrect
15:37:11 <n0ano> I haven't had a chance to review it yet, but I will later today
15:37:27 <bauzas> tianst: can't find it thru gerrit https://review.openstack.org/#/q/status:open+topic:bp/isolate-scheduler-db,n,z
15:37:34 <tianst> n0ano ,thanks
15:37:47 <bauzas> tianst: you made an excellent job
15:38:00 <bauzas> tianst: I'm just worried about the fact the bp is not validated yet
15:38:10 <tianst> bauzas: it's a draft, i will update tomorrow
15:38:11 <bauzas> https://review.openstack.org/#/c/89893/
15:38:19 <bauzas> tianst: yep, I know
15:38:37 <bauzas> tianst: would you mind if I propose another patchset ?
15:38:37 <n0ano> bauzas, I'm assuming the BP will get validated and I don't want to just sit and wait for that
15:39:08 <bauzas> n0ano: yey, but I had to do many back-and-forths with the sched-lib
15:39:29 <bauzas> n0ano: I don't want to have the same situation for isolate-sched-db
15:39:44 <n0ano> bauzas, hopefully this one is slightly less controversial (I'm an optimist)
15:39:52 <bauzas> n0ano: I'm also :)
15:40:11 <bauzas> so, tianst, would you mind if I'm proposing another patchset ?
15:40:24 <bauzas> so we could co-author it
15:40:31 <tianst> nop
15:40:32 <bauzas> that will be very quick
15:40:37 <bauzas> ok cool
15:41:02 <tianst> :)
15:41:07 <n0ano> OK, anything else about the forklift?
15:41:35 <n0ano> #topic policy based scheduler
15:41:41 <bauzas> well, we're running out of time :)
15:41:44 <bauzas> yep
15:41:57 <n0ano> toan-tran, anything you want to say today?
15:42:06 <toan-tran> i'll be quick
15:42:15 <toan-tran> I'm rewritting the proposal
15:42:35 <toan-tran> last time I made a mistake presenting usecases one by one
15:42:55 <toan-tran> it's easy to have patch for one usecase, but if we're keep doing that it's complicated
15:43:22 <toan-tran> I will re-present it with the whole picture, it's better to see the usefulness of this bp
15:43:36 <bauzas> toan-tran: the problem is that your proposal is big
15:43:45 <toan-tran> yes, the bp was proposed several months ago
15:43:50 <bauzas> toan-tran: so it should be managed by a single bp
15:43:53 <toan-tran> and it was the problem
15:44:05 <toan-tran> so I'm trying to break to down into pieces
15:44:06 <bauzas> should *not
15:44:10 <bauzas> right
15:44:18 <n0ano> toan-tran, yeah, it looks like you have to justify things a bit, breaking it up into smaller pieces might be easier
15:44:31 <toan-tran> n0ano: +1
15:44:31 <bauzas> maybe the rationale should still reside in a Wikipage
15:44:48 <mspreitz> I have an observation and a suggestion
15:44:53 <bauzas> you can just provide as many blueprints as you need
15:45:01 <bauzas> related to that
15:45:02 <toan-tran> bauzas: some old text is here: https://docs.google.com/document/d/1gr4Pb1ErXymxN9QXR4G_jVjLqNOg2ij9oA0JrLwMVRA/edit
15:45:13 <toan-tran> mspreitz: go ahead
15:45:51 <mspreitz> I have reviewed https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOriIQB2Y/edit# and the central idea there is that what you are adding is support for provider policy...
15:46:11 <mspreitz> When we get into policy-driven scheduling, there are two sources for policy : the user and the provider
15:46:44 <mspreitz> What I see your BP about is enabling the provider to make some fine-grained declarations of provider policy
15:46:53 <mspreitz> Leave the rest out of it
15:46:58 <mspreitz> That's a way to focus
15:47:18 <mspreitz> full stop
15:48:11 <toan-tran> mspreitz: well at first the idea is to have an engine driven by policy
15:48:28 <toan-tran> regardless of who the policy is provided by
15:48:46 <toan-tran> so we will have provider's polciy, and user's policy
15:48:58 <mspreitz> In some sense today's scheduler is driven by policy.  Both you and Yathi are working with user policy being given today
15:49:09 <toan-tran> but that's for later, right now I'm focusing on the engine itself
15:49:15 <mspreitz> I myself and Yathi want richer user policy input, but that's a separate issue
15:49:52 <mspreitz> So here is the factoring I see...
15:50:21 <mspreitz> One part is about letting the provider make fine-grained policy statements; those can, as outlined in https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOriIQB2Y/edit# , be parameters of today's scheduler
15:50:24 <toan-tran> mspreitz: +1, the obstacle for me with user's policy is that we don't have an API for user to delare that
15:50:41 <toan-tran> so I'm limited with provider policy only
15:50:52 <mspreitz> Other parts are about changing how the scheduler deals with incoming policy
15:50:58 <bauzas> toan-tran: I think that once we succeed in getting Gantt out
15:51:01 <toan-tran> but that's the thing I see in your proposal
15:51:13 <bauzas> toan-tran: the first item will be to provide an API
15:51:42 <bauzas> toan-tran: I have some experience of Pecan and WSME, I will be able to help
15:51:42 <toan-tran> bauzas: +1
15:51:51 <mspreitz> Yathi and I have been working on the API for user policy, I see you as contributing API for provider policy, let us not mix the two up
15:52:14 <toan-tran> mspreitz: right
15:52:14 <mspreitz> they are different things
15:52:22 <bauzas> toan-tran: mspreitz: I'm almost sure Nova cores won't want to extend Nova API for these purposes
15:52:38 <toan-tran> for now  I'm focusing on the engine itself, not the policy part
15:52:38 <bauzas> toan-tran: mspreitz: that needs to land on Gantt
15:52:44 * johnthetubaguy is lurking, in case someone had a question
15:52:48 <toan-tran> bauzas: +1
15:52:59 <mspreitz> Yes, I thought everyone agreed that Gantt comes first
15:53:18 <n0ano> toan-tran, note, focusing on the eingine might be your problem, it's hard to justify the engine without a user of it
15:53:20 <mspreitz> As for extending API to accept *more* user policy, yes that is not settled yet
15:53:21 <bauzas> mspreitz: by saying that, you implicitely accept the dependency on a Gantt API
15:53:41 <toan-tran> n0ano: that's why I'm working on its presentation :)
15:54:13 <toan-tran> mspreitz: I guess the API part should be talked once we have some discussion of gantt API
15:54:19 <bauzas> johnthetubaguy: we had some discussion abotu https://review.openstack.org/97232
15:54:37 <bauzas> johnthetubaguy: let's discuss it in #openstack-nova once that meeting ends up
15:54:47 <mspreitz> bauzas: I think Nova's client-facing API for accepting user policy does not necessarily have to expose Gantt; consider the proposals that Yathi and I and others have made; the implementation, OTOH, will involve Gantt
15:54:47 <johnthetubaguy> bauzas: ack
15:55:17 <toan-tran> FYI, I'm looking ath the work in Congress
15:55:33 <toan-tran> they're proposing some really interesting approach on policy part
15:55:35 <bauzas> mspreitz: what if Nova refuses to extend Nova API for adding user policies ?
15:56:03 <toan-tran> if it becomes official, we can benefit from congress API & policy language
15:56:22 <bauzas> toan-tran: I agree with you, but coming from a Stackforge project, that's tough now
15:56:37 <bauzas> toan-tran: we can't bet on a project to become integrated
15:56:44 <mspreitz> Refusing an API for accepting more user policy input = refusing to cope with more user policy.  If that's the community's decision, I'll probably be taking my marbles elsewhere.
15:57:16 <toan-tran> mspreitz: now now don't be on defensive :)
15:57:43 <toan-tran> I think that's the need (that's why I do it :) )
15:57:53 <n0ano> OK, we're running out of time and I do have one quick open
15:58:03 <toan-tran> I just see some work before it can be merged
15:58:04 <bauzas> toan-tran: so I would prefer to have an API accepting policies in Gantt and a possible backend that would be Congress
15:58:05 <n0ano> #topic opens
15:58:07 <mspreitz> I'm just saying that if the community decides to not work on a topic, then I will not try to work with this community on that topic
15:58:22 <toan-tran> bauzas: +1
15:58:42 <bauzas> mspreitz: well, I'm not saying the community will push back
15:59:03 <mspreitz> bauzas: you do not have to say it, it has already been said and done.  But not settled.
15:59:03 <n0ano> my BP https://review.openstack.org/#/c/97903/ for changing the compute node updates to the DB has been approved, expect to see code to implement it real soon now
15:59:06 <mspreitz> On to opens
15:59:07 <bauzas> mspreitz: but maybe it could be said that Nova is not the right place for it
15:59:24 <bauzas> n0ano: cool
15:59:31 <n0ano> reviews of the code, as always, are appreciated
15:59:38 <bauzas> :)
16:00:03 <n0ano> that's about all the time we have, barring any last minute opens
16:00:27 <bauzas> nope
16:00:30 <n0ano> OK, tnx everyone, we'll talk again next week.
16:00:31 <bauzas> thanks for your work
16:00:35 <n0ano> #endmeeting