14:05:47 <gampel> #startmeeting Smaug
14:05:47 <openstack> Meeting started Tue Jan 26 14:05:47 2016 UTC and is due to finish in 60 minutes.  The chair is gampel. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:05:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:05:51 <openstack> The meeting name has been set to 'smaug'
14:06:20 <gampel> Hi everyone
14:06:33 <yinweimac> hi
14:06:36 <xiangxinyong456> hi
14:06:40 <zengyingzhe> o
14:06:49 <liuan> hi
14:06:51 <chenzeng> hello
14:08:00 <gampel> #topic Design status
14:09:47 <zengyingzhe> preparing to implement the protectables, if you guys have no more comments.
14:11:09 <chenzeng> operationengine design bp has submitted. also the implementation of timetrigger. i have started the development of rpc interfaces of operationengine
14:11:09 <zengyingzhe> Anyone else?
14:11:42 <yinweimac> yingzhe, how about the project protectable, is there any API to show/list project object in openstack?
14:12:44 <zengyingzhe> that depends on if we provide the project protectable
14:12:47 <yinweimac> Protection Service design has committed patch set 8 for review, where I have added namespace interfaces in BankPlugin according to Eran's comment
14:13:35 <yinweimac> yingzhe, you can confirm it with Eran
14:14:03 <yinweimac> I think so, since smaug allow user to choose project as a resource to be protected
14:14:33 <zengyingzhe> They might still have some problems about the exposed method from protection plugin as protect, restore...
14:14:41 <yinweimac> seems network in Eran side is not stable
14:14:50 <saggi> Yes
14:14:53 <saggi> it's ok
14:14:58 <saggi> the bot is recording everything
14:14:59 <yinweimac> he's always joining and leaving
14:15:08 <saggi> he'll catch up
14:15:10 <saggi> please continue
14:15:15 <zengyingzhe> yinweimac, yes, there is a topic to discuss what resource we support.
14:15:53 <saggi> zengyingzhe: I pushed a list to gerrit. With the the basic protectables we want to support and their dependency.
14:16:25 <yinweimac> saggi, I raised an issue to see whether we should have protection plugin to build work flow tasks
14:16:54 <yinweimac> Basically, I prefer plugins only implement simple functions
14:17:05 <gampel> I am back sorry
14:17:19 <zengyingzhe> saggi, that's too much, can we concentrate on some of them, the most important now?
14:17:43 <yinweimac> I don't want to involve protection plugin to care about our execution component, like task flow, nor the persistence logic, like checkpoint/bank
14:18:17 <gampel> yinweimac: it makes senses
14:18:47 <saggi> zengyingzhe: There are only 6 of them, and Keystone::Project is just metadata. Manila::Share is low priority
14:19:24 <yinweimac> so is it possible to make the 'get_prev_task' and 'get_post_task' out of protection plugin?
14:19:27 <saggi> I don't see us launching without VM, Net Topology, Cinder and Glance
14:19:33 <zengyingzhe> yinweimac, agree with plugin not handling persistence logic.
14:19:55 <saggi> zengyingzhe, yinweimac: One thing at a time please :)
14:20:06 <saggi> Are we OK about the protectables list?
14:20:13 <yinweimac> ok, you guys go first
14:20:20 <yinweimac> I'm ok with it
14:20:30 <saggi> zengyingzhe: You said that there are too much in the list
14:20:39 <zengyingzhe> yes
14:21:04 <saggi> This list ? https://review.openstack.org/#/c/262264/7/doc/images/available_protectables.svg
14:21:05 <zengyingzhe> Perhaps we don't have enough people to implement all of these.
14:22:25 <zengyingzhe> So can we pick some of them, such as VM, cinder, the most important ones for now.
14:22:30 <saggi> zengyingzhe: As I said we can remove Manila::Share from the initial list but the rest seem essential to me. Don't you think ?
14:23:04 <yinweimac> yingzhe, if you find in short of human resource, we have Dr. hu who just joined us to help on several of protectables
14:23:07 <saggi> How will we manage VM network ?
14:23:16 <t4nk603> i am Rong Hu
14:23:39 <yinweimac> hi, Rong
14:24:04 <yinweimac> Just recommend you to help on protectable implementation
14:24:12 <zengyingzhe> If we got enough time and human resource, then it's ok to me
14:24:35 <yinweimac> saggi, what do you think about managing VM network
14:24:35 <zengyingzhe> but we should estimate the workload of these plugins
14:24:55 <yinweimac> yes, let's scoping it now
14:26:06 <saggi> gampel agrees that the protectables listed are essential
14:26:22 <saggi> We can't backup a VM without it's storage and network information
14:26:57 <yinweimac> yes, those protectables are must have
14:26:57 <zengyingzhe> saggi, does that mean we should also implement protection plugins for those resource type?
14:27:28 <saggi> zengyingzhe: Yes, but remember that only Cinder\Glance actually contain data. The rest is just metadata.
14:27:49 <zengyingzhe> OK, got it. my topic is done. go next one
14:27:56 <saggi> So the plugins will be relatively simple from backup perspective. The only real work will be for restore.
14:28:13 <gampel> I think that this is the list of must have
14:28:23 <saggi> yinweimac: Please articulate your concerns about the task and plugins
14:28:39 <yinweimac> ok
14:28:45 <saggi> gampel, please change the topic to "Protection Plugins and Tasks"
14:30:08 <yinweimac> currently we have get_prev_task and get_post_task in protection plugin interface
14:30:41 <yinweimac> which means we need build task and task flow (dependencies among tasks) inside protection plugin implementation
14:31:27 <yinweimac> inside task execute() implementation, it even needs care about persistence logic, since we need persist the protection result into checkpoint/bank
14:31:48 <yinweimac> which I think makes the protection plugin implementation a complex
14:32:09 <yinweimac> I'm thinking if it's possible to follow cinder driver routine
14:32:31 <yinweimac> where driver only implement simple functions, and the data to be persisted
14:32:57 <saggi> yinweimac: The problem is that we don't want to force the plugins to a specific flow.
14:33:20 <yinweimac> So we don't expose our execution and persistence logic to protection plugins
14:33:45 <saggi> The plugins will have to handle persistent logic for anything outside the bank anyway.
14:34:15 <saggi> And will have to handle transactioning that. Meaning they will have to leave information in the bank before they put stuff anywhere else.
14:35:00 <yinweimac> what i don't prefer is to have plugin to persist things into bank
14:35:19 <saggi> But it will have to do that to manage it's own transactions.
14:35:39 <saggi> Since we don't want it to do anything it can't clean up from information in the bank
14:36:02 <saggi> that means persisting some transaction information in the bank before and after the actual data transfer.
14:36:10 <yinweimac> it could provide interface to clean up its resources
14:36:31 <saggi> I'll try and explain the use case better
14:36:31 <yinweimac> i don't think so
14:36:39 <yinweimac> ok, go ahead
14:39:22 <saggi> Let's say I have a plugin that backs up the volumes to some NFS share somewhere. If it starts copying and we crash there is no way for us to know that there is partial information in the share. What the plugin is supposed to do is save some information in the bank like 'copy started to = nfs://folder/volume' and then start copying. That way if we crash we know that we have partial data in the NFS share. After we finish copying we will change it to
14:39:46 <saggi> This means the plugin needs low level access and the ability to create temporary information in it's section of the bank.
14:40:14 <saggi> Since if we back up to multiple locations or to multiple files we might need more complex flows.
14:41:09 <yinweimac> saggi, this could be solved by task flow engine itself
14:42:12 <yinweimac> first, to recycle in-progress garbage, it could be implemented in task.revert() function
14:42:35 <yinweimac> plugin only provides interface to recycle it
14:42:35 <saggi> This is only valid at runtime
14:42:49 <saggi> It will still need to persist data in the bank.
14:42:54 <yinweimac> second, if you want the engine to resume the status last time it crashes
14:43:02 <saggi> We have to give plugins access to put information there
14:43:03 <yinweimac> wait, i haven't finished yet
14:43:10 <saggi> sorry
14:43:14 <yinweimac> np
14:43:24 <yinweimac> i print slowly
14:43:59 <yinweimac> if you want the engine to resume the status last time it crashes, we can give the engine a persistent storage
14:44:12 <yinweimac> which will persist the status of the engine itself
14:44:38 <yinweimac> so it could resume its status and the task flow status and the task status when it crashes
14:44:51 <gampel> yinweimac:  but the task could have multi step not known to the engine
14:46:10 <yinweimac> I think the granularity of task is equal to the granularity of resources we built in resource graph
14:46:49 <yinweimac> otherwise, we need split the resource in resource graph with smaller granularity
14:47:56 <gampel> yinweimac: i mean a even a backup task of volume could have muti step this depends on the backup provider features
14:48:11 <yinweimac> for volume backup example, create backup will return a backup id at once
14:48:34 <gampel> yinweimac: whats the reasoning to push this responsibility to the engine ?
14:48:56 <yinweimac> that' what plugin gives back as return data, we persist it in bank but doesn't care the content of the return data
14:49:25 <yinweimac> push what responsibility to engine?
14:50:10 <yinweimac> I mean, the task building and persistence logic should be the work of protection service, not the plugin implementation
14:50:53 <gampel> yinweimac:  maybe i miss some thing because of all the disconnects , lets take it to our channel and continue with the other items, what do you think  ?
14:51:44 <saggi> yinweimac: The engine will be responsible for all the common elements. But we still leave the actual data flow logic to the plugin.
14:52:03 <gampel> #action yinweimac & Saggi & gampel define who is responsible  persistence logic
14:52:04 <yinweimac> I prefer not
14:52:28 <yinweimac> actually I wrote a mail to Eran
14:52:40 <yinweimac> Eran, could you pls. fwd it to saggi?
14:52:52 <yinweimac> we can talk about it in mail offline
14:53:19 <gampel> ok i will lets continue this discussion in our channel
14:53:21 <yinweimac> gample, it's late here.
14:53:37 <yinweimac> ok, let's use our channel
14:53:44 <gampel> lets do it tomorrow morning
14:53:53 <gampel> we do not have much time left
14:54:12 <yinweimac> np
14:54:14 <yinweimac> sure
14:54:48 <gampel> There are few tasks that are open for  coding like protectable
14:55:08 <chenzeng> by the way, please give more time to review operationengine design and implementation. thanks.
14:55:21 <zengyingzhe> gampel, yes, i'm working on it.
14:55:30 <gampel> lets try all to finalize all the open review  this week
14:55:55 <chenzeng> it is great.
14:56:25 <yinweimac> Bin is working on implement swift bank plugin and Liu an is doing provider registry work
14:57:00 <yinweimac> shall we talk in our own channel?
14:57:15 <gampel> this is great we should mean while we work on the design start sending patches on the items that are all ready defined
14:57:17 <gampel> yes
14:57:43 <gampel> Thanks every one and sorry for all my disconnects thanks saggi for covering
14:57:47 <gampel> Thanks every one and sorry for all my disconnects thanks saggi for covering
14:58:15 <xiangxinyong456> you are welcome
14:58:25 <gampel> lets continue in our channel
14:58:39 <gampel> #endmeeting