15:03:39 <n0ano> #startmeeting gantt
15:03:40 <openstack> Meeting started Tue May 27 15:03:39 2014 UTC and is due to finish in 60 minutes.  The chair is n0ano. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:43 <openstack> The meeting name has been set to 'gantt'
15:03:47 * n0ano really should type into the right window
15:04:03 <bauzas> hurraaaaa
15:04:07 * doron here
15:04:33 <n0ano> I started the meeting in openstack-dev and was saying bad things about no one joining :(
15:04:34 <bauzas> so, I think we can start
15:04:40 <n0ano> anyway
15:04:49 <n0ano> #Juno summit recap
15:04:49 <bauzas> mspreitz and toan-tran said hello
15:05:15 <n0ano> just so everyone knows, the etherpads from the friday sched sessions were:
15:05:20 <n0ano> forklift
15:05:20 <n0ano> https://etherpad.openstack.org/p/juno-nova-gantt-apis
15:05:20 <n0ano> no-db scheduler
15:05:20 <n0ano> https://etherpad.openstack.org/p/juno-nova-no-db-scheduler
15:05:20 <n0ano> simultaneout scheduling for server groups
15:05:21 <n0ano> https://etherpad.openstack.org/p/juno-nova-scheduling-server-groups
15:05:25 <n0ano> hints for VM lifecycle
15:05:27 <n0ano> https://etherpad.openstack.org/p/juno-nova-scheduler-hints-vm-lifecycle
15:05:48 <bauzas> we also discussed in -dev ML
15:05:56 <bauzas> lemme find the link
15:06:06 <n0ano> we can do a brief rundown of the sessions:
15:06:36 <n0ano> forklift - forge ahead, clean up the interfaces in nova first and then split the code out
15:06:41 <bauzas> #link http://lists.openstack.org/pipermail/openstack-dev/2014-May/035415.html
15:07:12 <n0ano> nodb - the memcached is used as a journal of the changes (not what I expected and created a large bit of discussion)
15:07:35 <n0ano> simultaneous scheduler for server groups - should look into maybe merging this into climate
15:07:51 <bauzas> s/Climate/Blazar now :)
15:08:20 <n0ano> n0ano, sigh, can't tell the players without a score card (baseball reference :-)
15:08:41 <mspreitz> and some of them have funny names
15:08:47 <n0ano> hings for VM lifecycle - good idea, issue is where to store the hings
15:09:02 <bauzas> right
15:09:15 <n0ano> that's my summary of results, anyone want to expand any of that?
15:09:38 <toan-tran> yeah
15:09:42 <bauzas> the detailed analysis is in the link I gave previously
15:09:46 <toan-tran> on VM lifecycle
15:10:01 <toan-tran> IMO it's really necessary
15:10:17 <toan-tran> and the solution that Jay Lau proposed sounds well
15:10:30 <toan-tran> just want to know when the bp starts :)
15:10:31 <bauzas> toan-tran: there was a consensus about the need
15:10:47 <n0ano> bauzas, +1
15:10:47 <mspreitz> Which is Jay's solution?
15:10:49 <bauzas> toan-tran: the discussion was about how to persist it
15:10:59 <mspreitz> +1 on th need
15:11:09 <toan-tran> mspreitz: stores hints in db
15:11:24 <toan-tran> whether instance metadata  or something else
15:11:26 <bauzas> mspreitz: the idea was to store hints in  compute_nodes table
15:11:31 <bauzas> right
15:11:48 <bauzas> the discussion was about the possible interest of system_metadata
15:11:54 <toan-tran> I got another thought
15:12:14 <toan-tran> since this hint can be used with Instance Group
15:12:31 <toan-tran> so that user can add VMs into groups
15:12:57 <toan-tran> it's not bad iea to have it stored with instance group
15:13:11 <toan-tran> s/iea/idea
15:13:18 <bauzas> toan-tran: hints are not only for groups
15:13:31 <johnthetubaguy> at the session, people didn't like the idea of every VM having a group, but its an option for sure
15:13:37 <n0ano> what bauzas said, I would think the hings should be VM specific
15:13:51 <n0ano> s/hings/hints/
15:13:57 <mspreitz> n0ano? should be?  Are!  right?
15:13:58 <toan-tran> OK, so idea reversed:
15:13:58 <johnthetubaguy> n0ano: +1 for now it has to be really
15:14:18 <toan-tran> make Instance group use hints ?
15:14:28 <bauzas> https://review.openstack.org/#/c/88983/ is the nova-spec
15:14:43 <mspreitz> toan-tran: how?
15:15:08 <toan-tran> mspreitz: sorry ?
15:15:17 <mspreitz> I am asking what you mean
15:15:25 <mspreitz> expand on your remark pls
15:15:37 <toan-tran> I mean we can use hint to make instance group policy persistant
15:15:57 <bauzas> the proposed change is planned to be located in instances table
15:16:00 <mspreitz> I thought Instance Grp policy is already persistent
15:16:20 <toan-tran> right now the problem of instance group, IMO, is that whenever user wants to create a VM, it has to connect with the group's current policy
15:16:43 <toan-tran> thus from my point of view, instance group' s policy and hint is the same
15:16:55 <n0ano> toan-tran, are you getting confused, we need to make the hints for a VM persisten, how the instance groups uses those hints is up to the instance group code
15:16:58 <mspreitz> toan-tran: I'm confused, that's the point of Inst grp
15:17:06 <bauzas> n0ano: heavy +1
15:17:30 <mspreitz> Storing hints in instances table makes sense to me
15:17:44 <toan-tran> well, I'm saying that if we think of how new VM should be created for a group
15:17:58 <bauzas> mspreitz: that's what's proposed in the spec
15:18:11 <garyk> i am not sure i follow
15:18:19 <bauzas> toan-tran: please address your concerns there : https://review.openstack.org/#/c/88983/17/specs/juno/persist-scheduler-hints.rst
15:18:25 <garyk> the spawn today uses the hint to indicate which instance group to ise
15:18:30 <garyk> sorry, use
15:18:45 <garyk> the group is persistent.
15:18:56 <mspreitz> bauzas: I have not had a chance to read that patch yet, I presume it follows the spec (store in instances table)
15:19:09 <n0ano> I thought the only issue from the summit was whether to store the hings in a new instance table column or a new table
15:19:13 <bauzas> mspreitz: you presume well
15:19:21 <toan-tran> n0ano: no argument here
15:19:32 <mspreitz> So I am +1 on the spec and patch
15:19:44 <toan-tran> I just think of how instance group can profit from persistent hint
15:19:57 <n0ano> toan-tran, so is you only concern `how` the instance group code uses that data
15:20:06 <toan-tran> n0ano: yes
15:20:06 <bauzas> time is precious, and we still need to discuss about other summit topics
15:20:13 <bauzas> could we move on ?
15:20:22 <toan-tran> bauzas: i'm done :)
15:20:36 <n0ano> I think we've discovered we're in violent agreement, so sure
15:20:41 <bauzas> :)
15:20:47 <n0ano> bauzas, you had another point?
15:20:53 <bauzas> I just wanted to discuss about no-db
15:20:58 <bauzas> and sched-forklift
15:21:12 <bauzas> but sched forklift can wait for the next topic
15:21:22 <bauzas> I just saw YorikSar joining
15:21:27 <bauzas> YorikSar: ping ?
15:21:29 <n0ano> is boris-42 or one of his team members available?
15:21:29 <YorikSar> bauzas: Hi :)
15:21:36 <bauzas> YorikSar: hi :)
15:21:39 <YorikSar> n0ano: I'm here
15:21:46 <n0ano> YorikSar, can we talk about the no-db scheduler work
15:21:48 <bauzas> YorikSar is the current owner of the no-db patches
15:21:53 <YorikSar> Sure.
15:21:58 <bauzas> YorikSar was not present at Summit
15:22:08 <bauzas> so we need to make sure we're on the same page
15:22:26 <bauzas> YorikSar: has boris-42 discussed with you about the results of the session ?
15:22:26 <YorikSar> As I understood from what boris-42 told me, I need to make the new host state delivering scheme optional.
15:22:37 <iqbalmohomed> At the summit, there was some confusion/questions about the 60 second sync ups ..
15:22:44 <n0ano> I was confused when the summit session indicated the memcache would be a tranascation journal and not a state cache, is that still the case
15:23:01 <YorikSar> And for the rest of it people were agreed upon it.
15:23:08 <bauzas> YorikSar: well, that's a point, the second one was about journalising updates
15:23:22 <bauzas> n0ano: +1
15:23:34 <toan-tran> n0ano: +1
15:23:38 <YorikSar> iqbalmohomed: We can have pluggable backends that store all data used by Synchronizer.
15:23:39 <n0ano> bauzas, yeah, I think there was negative agreement on the journaling issue
15:23:50 <YorikSar> n0ano: ^
15:24:12 <YorikSar> It includes smth like a journal but is not limited to it.
15:24:46 <n0ano> I don't understand why you have a journal at all, why not a state table
15:25:00 <YorikSar> So just like with SQL backend we can ask for all new updates and fetch only records that were changed with those updates.
15:25:03 <iqbalmohomed> regarding the journal ... was the idea that there is a periodic checkpoint that is done?
15:25:33 <mspreitz> YorikSar: changed since <what> ?
15:26:05 <YorikSar> Since last sync action. Sync is done before every read/
15:26:52 <mspreitz> There are multiple consumers, right?  So it's since <this client's> last sync, right?
15:26:59 <iqbalmohomed> ok ... i guess the sync = the checkpoint
15:27:29 <YorikSar> So most common case will be: user spawns an instance, scheduler fetches list of updates since last its request and fetches records touched by those updates.
15:27:39 <YorikSar> mspreitz: Yes.
15:27:47 <toan-tran> ok so basically there are two storage: one for the whole state and another for changes
15:27:49 <toan-tran> ?
15:27:52 <YorikSar> iqbalmohomed: Well... You can name it that way.
15:28:03 <YorikSar> toan-tran: No.
15:28:19 <mspreitz> YorikSar: So there is some way of identify the client or its last sync.  What is that?
15:28:24 <YorikSar> Update is just a record that some Record has been changed, no data.
15:28:43 <n0ano> YorikSar, ?? (now I'm confused)
15:29:11 <YorikSar> mspreitz: Every client stores a token of the latest update it have seen.
15:29:43 <YorikSar> So it asks backend "give me all records that were changed by updates with token greater that this one"
15:29:45 <iqbalmohomed> When a new client comes in (e.g. during crash recovery), how does it catch up?
15:29:49 <mspreitz> YorikSar: token = sequence number, works because updates are totally ordered ?
15:29:59 <YorikSar> mspreitz: Yes
15:30:09 <YorikSar> iqbalmohomed: It fetches all records from the backend
15:30:19 <YorikSar> And a latest update's token.
15:30:57 <mspreitz> So there is one central thing that is totally ordering updates.  Is that a scalability problem?
15:31:39 <YorikSar> mspreitz: For SQL backend it's just an autoincremented primary key.
15:32:08 <YorikSar> mspreitz: For memcached it's a key that will be inc()'ed for every new update.
15:32:11 <bauzas> YorikSar: IIUC, the debate was about the need of implementing checkpoints with memcache
15:32:20 <toan-tran> YorikSar: how big the data it has to sends ? it's quite a problem if the client is far behind
15:32:50 <bauzas> YorikSar: because that was a logic that memcache could handle on its own, no need to implement it at software level
15:32:57 <YorikSar> bauzas: I'm not sure I understand what's a checkpoint in this context.
15:33:37 <bauzas> correct me if I'm wrong, but you're implementing transactions at the implementation level ?
15:33:40 <YorikSar> toan-tran: If the client is far behind (lost next update after his last one) it'll eventually refetch all records from backend.
15:34:26 <YorikSar> bauzas: No, with memcached there is no transactions.
15:34:31 <bauzas> YorikSar: the point that has been raised was about why implementing it and not leave memcache holding this?
15:35:00 <n0ano> OK, everyone take a deep breath and let's pause for a minute...
15:35:08 <bauzas> n0ano: :)
15:35:22 <YorikSar> Let me throw in some spec I've written before summit
15:35:22 <n0ano> I think there is a `lot` of confusion about the details of this design...
15:35:26 <YorikSar> #link https://review.openstack.org/#/c/92128/2/specs/juno/no-db-scheduler.rst
15:35:52 <YorikSar> It feels like I'm retelling some parts of it.
15:36:03 * mspreitz wants to study before speaking more
15:36:15 <n0ano> YorikSar, thank you, that's what I was about to ask for, let's all read that spec and come back next week with some understanding and raise any issues we have
15:36:22 <mspreitz> +1
15:36:24 <bauzas> n0ano: +2
15:36:35 <YorikSar> n0ano: Great idea :)
15:36:37 <toan-tran> n0ano: +3
15:37:00 <n0ano> YorikSar, BTW, thank for doing this, we're really not trying to beat up on you, there's just lots of confusion
15:37:01 <YorikSar> I hope I'll be able to address some of John's comments there.
15:37:12 <YorikSar> n0ano: I understand.
15:37:29 <iqbalmohomed> +1
15:37:30 <johnthetubaguy> YorikSar: cool, its just the optional bits that makes me −2 at this point, I love the idea in general
15:37:36 <YorikSar> I guess noone noticed that spec after John -2'd it
15:37:41 <bauzas> n0ano: indeed, we're just reporting discussion from the Summit
15:38:02 <n0ano> moving on, any other summit comments before we move to forklift efforts?
15:38:07 <bauzas> n0ano: maybe we could put an action
15:38:15 <YorikSar> n0ano: I totally understand.
15:38:35 <n0ano> #action everyone to read the no-db spec and come back for detailed discussion next week
15:38:42 <bauzas> sweet
15:38:46 <YorikSar> johnthetubaguy: Yes, I've just been waiting or summit to pass before getting back to the spec
15:39:14 <johnthetubaguy> YorikSar: cool, no problem, ping me if I miss an updated, happy to help review that
15:39:18 <n0ano> #topic forklift efforts
15:39:25 <bauzas> soooo
15:39:28 <n0ano> bauzas, can you provide a brief summary
15:39:30 <YorikSar> johnthetubaguy: Sure. Will do.
15:39:57 <bauzas> there are 3 bps I'm owning for that
15:40:04 <bauzas> 2 are still specs to review
15:40:18 <bauzas> the most important one is to isolate the sched db
15:40:27 <bauzas> https://review.openstack.org/89893
15:40:39 <bauzas> I made some changes, based on Summit feedback
15:40:51 <bauzas> I'm planning to begin working on that by next week
15:41:04 <bauzas> that one is really huge to work on
15:41:15 <bauzas> so, work items are important
15:41:27 <bauzas> people interested in contributing can review the spec
15:41:43 <bauzas> in order to make I'm not forgetting a crucial piece of work
15:41:47 <johnthetubaguy> bauzas: the currently approved blueprint has all its patches up for review now?
15:41:48 <mspreitz> Can you pls post a pointer to all 3 bps?
15:41:50 <bauzas> s/make/make sure
15:41:59 <bauzas> mspreitz: sure
15:42:23 <bauzas> #link https://review.openstack.org/82133 (scheduler-lib validated)
15:42:46 <bauzas> #link https://review.openstack.org/89893 (isolate-sched-db review in progress)
15:43:12 <bauzas> #link https://review.openstack.org/94916 (prep_resize to move to conductor - review in progress)
15:43:29 <bauzas> these are the 3 bps I'm owning
15:43:35 <bauzas> sched-lib is targeted for juno-1
15:43:46 <bauzas> I'll speak about it right after
15:43:54 <n0ano> bauzas, when you say `review in progress`, is that the BP or the patches to implement the BP?
15:43:59 <bauzas> isolate-sched-db is planned for juno-3
15:44:09 <mspreitz> https://review.openstack.org/#/c/94916/ is a change with no associated BP
15:44:11 <bauzas> spec is currently in review
15:44:33 <bauzas> mspreitz: my bad, missed the bp link
15:44:51 <bauzas> mspreitz: bp/move-prep-resize-to-conductor
15:45:01 <bauzas> mspreitz: you can find it as bp name
15:45:09 <bauzas> mspreitz: but will amend the patch
15:45:15 <mspreitz> bauzas: thanks
15:45:20 <bauzas> mspreitz: that one is quite small to do
15:45:36 <bauzas> mspreitz: the most important efforts will go to https://review.openstack.org/89893 implementation
15:45:53 <bauzas> hence the juno-3 target
15:46:06 <bauzas> so, I pledge you to review the spec and comment it
15:46:31 <bauzas> juno-2 is beginning in 2 weeks
15:46:36 <johnthetubaguy> bauzas: about this one: https://blueprints.launchpad.net/nova/+spec/scheduler-lib
15:46:46 <johnthetubaguy> bauzas: are all the patches up for that now?
15:47:15 <bauzas> johnthetubaguy: the current patch is https://review.openstack.org/82778
15:47:33 <bauzas> johnthetubaguy: it's just missing unittests for sched-lib
15:47:37 <johnthetubaguy> bauzas: do you expect any more patches?
15:47:57 <johnthetubaguy> ah, OK, its not complete yet, fair enough
15:47:59 <bauzas> johnthetubaguy: I will do another one for patching the calls of select_destinations
15:48:09 <johnthetubaguy> just its targeted at juno-1
15:48:25 <bauzas> johnthetubaguy: it will be ended by tomorrow
15:48:52 <bauzas> johnthetubaguy: I nearly finished to write the tests, will submit them today or tomorrow
15:49:03 <bauzas> johnthetubaguy: so you can review it
15:49:11 <johnthetubaguy> bauzas: OK, so all patches hopefully up by tomorrow then, thats cool
15:49:22 <bauzas> johnthetubaguy: yup
15:49:43 <johnthetubaguy> once is ready, you can move to needscodereview blueprint state, which is cool, thanks
15:49:46 <bauzas> johnthetubaguy: so that next week will be focused on delivering new patchsets if any requested
15:49:57 <bauzas> johnthetubaguy: ok will do
15:51:03 <n0ano> bauzas, maybe a couple of actions, bauzas to complete the one patch, everyone to do reviews?
15:51:07 <bauzas> so, as I said, I'm looking for support on reviews for both the isolate-sched-db spec and the sched-lib patch
15:51:16 <bauzas> n0ano: would love to
15:51:23 <toan-tran> bauzas: +1
15:51:47 <n0ano> #action all to review isolate-sched-db spec & sched-lib patch
15:52:00 <bauzas> thanks
15:52:23 <n0ano> #action bauzas to complete tests for scheduler-lib patch
15:52:36 <bauzas> cool
15:52:55 <n0ano> bauzas, BTW, tnx for the effort, it's much appreciated
15:53:01 <iqbalmohomed> In the last few mins, could we do a quick recap of the simultaneous server group scheduling discussion?
15:53:04 <bauzas> your welcome
15:53:14 <n0ano> #topic opens
15:53:18 <bauzas> sure, I'm don
15:53:20 <bauzas> done
15:53:30 <n0ano> iqbalmohomed, sure, go ahead
15:53:32 <bauzas> iqbalmohomed: what do you want to know ?
15:53:47 <iqbalmohomed> So what I got was that we need to talk to the climate folks first
15:53:58 <iqbalmohomed> figure out how that integrates with GANTT
15:54:13 <iqbalmohomed> and then, we'll be ready to get started
15:54:23 <iqbalmohomed> but i didn't hear strong objection to the capability
15:54:40 <iqbalmohomed> any other thoughts?
15:54:40 <bauzas> iqbalmohomed: that's right
15:54:49 <n0ano> iqbalmohomed, that's what I heard
15:54:50 <bauzas> iqbalmohomed: I'm one of the Climate folks, btw.
15:55:20 <mspreitz> So quick question: when I looked at the API a month ago, I was disappointed to find no way to make a reservation without creating a VM at the same time.  Is that still true?
15:55:20 <doron> iqbalmohomed: climate is now blazar- it was renamed....
15:55:25 <bauzas> iqbalmohomed: the idea is to see how to integrate that idea in Climate and see what pieces are missing in Nova
15:55:58 <iqbalmohomed> cool .. will follow up in the climate discussion group and report back here
15:56:01 <bauzas> mspreitz: you're right, we're not having a no-op plugin
15:56:18 <bauzas> mspreitz: but that's not a huge amount of work
15:56:35 <mspreitz> We want some separation.  Discussion to follow, this meeting is about to end.
15:56:36 <bauzas> iqbalmohomed: you can join #openstack-blazar
15:56:50 <bauzas> iqbalmohomed: for discussion
15:56:50 <iqbalmohomed> bauzas: thanks ... do
15:57:12 <n0ano> mspreitz, do you want me to queue this up for next weeks agenda?
15:57:28 <mspreitz> not sure yet, let us discuss with Blazar folks first
15:57:36 <n0ano> mspreitz, sounds good
15:57:40 <iqbalmohomed> thx all
15:57:44 <toan-tran> btw why blazar?
15:57:54 <bauzas> I promise it won't be the Blazaar :)
15:58:09 <toan-tran> bauzas: :)
15:58:15 <n0ano> Winding down for today, any last minute comments?
15:58:28 <bauzas> toan-tran: I leave you check Wikipedia :)
15:58:49 <bauzas> toan-tran: Novas and Blazars are common objects in space :)
15:58:58 <toan-tran> yeah super-nova IMO
15:59:10 <n0ano> OK, tnx everyone, talk again next week
15:59:14 <n0ano> #endmeeting