13:59:28 <saggi> #startmeeting Smaug Biweekly
13:59:29 <openstack> Meeting started Tue Jan 12 13:59:28 2016 UTC and is due to finish in 60 minutes.  The chair is saggi. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:59:33 <openstack> The meeting name has been set to 'smaug_biweekly'
14:00:06 <saggi> #topic roll call
14:00:16 <chenying_> Hi
14:00:16 <saggi> Hi everyone
14:00:20 <zhonghua-lee> hi
14:00:26 <zengyingzhe> o
14:00:54 <saggi> gampel is mid-flight so he will be missing this meeting
14:01:04 <saggi> are we waiting for anyone?
14:01:10 <saggi> Yinwei?
14:01:19 <zengyingzhe> Let's begin?
14:01:33 <zhonghua-lee> is there any topic related yinwei?
14:01:54 <saggi> Protection Service tasks
14:02:22 <zengyingzhe> Talk other topics first
14:02:25 <smcginnis> o/
14:02:26 <saggi> sure
14:02:28 <zhonghua-lee> the first one?
14:02:38 <saggi> #topic API, Last call for comments
14:02:52 <saggi> So I want to finalize the API soon
14:03:17 <zengyingzhe> There are some put methods to add.
14:03:18 <zhonghua-lee> is it the last version now?
14:03:35 <zengyingzhe> Please see Eran's comment.
14:03:47 <saggi> No, I have some minor comments from gampel.
14:04:27 <saggi> But I hope it's the last iteration
14:05:03 <saggi> smcginnis, I don't know if you are here but I would appreciate you giving it a second look.
14:05:16 <smcginnis> saggi: I will try to do that later today.
14:05:21 <saggi> thanks
14:05:41 <smcginnis> saggi: No problem. I'm travelling, so I might not get to it right away.
14:05:48 <smcginnis> But it's on my list now. :)
14:05:50 <chenying_> smcginnis: Thanks.
14:06:17 <saggi> So if people have any questions or comments please post them. Even if you don't understand something. It's very important as changing APIs is hard.
14:06:32 <zengyingzhe> sure
14:06:44 <smcginnis> saggi: Very true!
14:07:02 <chenying_> I see
14:07:32 <zengyingzhe> Next topic?
14:07:48 <chenying_> API service
14:07:53 <chenying_> Api resource controller design is in progress.
14:08:02 <saggi> #topic API Service
14:08:10 <chenying_> I plan to do api service design and resource controller development.
14:08:43 <zhonghua-lee> chenying_: wo can work together
14:08:52 <saggi> chenying_: please coordinate with yinwei about serializers and other integration between the API and the protection service.
14:09:07 <chenying_> Ok
14:09:58 <saggi> I see some stuff that appear to be duplicate effort between you two
14:10:33 <zhonghua-lee> saggi: you mean chenying_ and yinwei?
14:10:39 <saggi> yes
14:10:54 <saggi> Stuff that could be moved to common code between the services
14:11:04 <saggi> chenying_: All in all you are doing a good job
14:11:16 <saggi> chenying_: Is there anything you need from anyone else?
14:11:38 <chenying_> She has explained that serializers in manager is for bankplugin.
14:12:53 <saggi> I understand. Than we might need something that will also work for the plugins. I will think about it.
14:13:18 <chenying_> I will coordinate with yinwei about it.
14:13:41 <saggi> Good, you will need to coordinate the schema transfer and other information from the checkpoint.
14:14:05 <saggi> Should we move to Pluggable Proection Provider Q/A
14:14:22 <zengyingzhe> yes
14:14:44 <saggi> #topic Pluggable Proection Provider Questions and Answers
14:15:00 <saggi> I posted the design doc about the pluggable protection provider.
14:15:22 <saggi> This will be the reference implementation and recommended way to extend Smaug.
14:15:47 <saggi> For cases where you need small modifications to the existing backup.
14:15:59 <saggi> Are there any questions about the design?
14:16:10 <zengyingzhe> Continue the discussion this afternoon. create_checkpoint needs to add a checkpoint info parameter?
14:16:26 <zengyingzhe> right?
14:16:55 <saggi> create_checkpoints get's a checkpoint info
14:17:04 <saggi> In the post request
14:17:30 <zengyingzhe> But there gotta be a way to pass this info.
14:17:44 <saggi> zengyingzhe: I don't understand the question.
14:17:44 <zengyingzhe> to create_checkpoint()
14:18:38 <zengyingzhe> Let
14:18:49 <saggi> zengyingzhe: Oh
14:18:57 <zengyingzhe> Let's talk this latter
14:18:57 <saggi> Now I understand what you are talking about
14:19:17 <zengyingzhe> saggi, yes?
14:19:23 <saggi> zengyingzhe: You are correct I need to change this in the diagram.
14:19:38 <saggi> zengyingzhe: I kept thinking you are talking about the API diagram.
14:19:47 <saggi> Completely my mistake.
14:19:52 <saggi> I'm very sorry
14:19:57 <saggi> I will fix it
14:20:36 <zengyingzhe> :) Don't be sorry. I explained it unclearly
14:21:09 <saggi> So please everyone, make sure you understand it. It's going to be the heart of Smaug so your comments are very important.
14:21:52 <saggi> Yinwei, and zengyingzhe are already doing a good job finding my mistakes :)
14:22:17 <zengyingzhe> And another one. I see create_checkpoint() is for protect operation, does there need other interfaces for restore or enable?
14:22:49 <saggi> zengyingzhe: Yes, it's missing. I thought I already did that but I guess I didn't.
14:23:12 <zengyingzhe> saggi, thanks.
14:23:22 <saggi> Should we move to protection Service tasks
14:23:48 <saggi> ?
14:23:59 <zengyingzhe> yinwei
14:25:18 <saggi> We'll start without her
14:25:34 <saggi> #topic Protection Service tasks
14:25:34 <zhonghua-lee> it seems she is not here
14:25:55 <saggi> I've added tasks to the etherpad https://etherpad.openstack.org/p/smaug-bootsrap
14:26:23 <zengyingzhe> what's this topic about?
14:26:28 <saggi> This are things that we are more or less closed on how we want to do them
14:26:58 <saggi> So if you have time to do some coding. You should pick something from the list.
14:27:36 <chenzeng> can we move to operationengine?
14:27:39 <saggi> The Protectables are low handing fruit as you can make a small tool and just make sure you can query openstack etc
14:28:03 <saggi> #topic Operation Engine design status\
14:28:09 <saggi> #topic Operation Engine design status
14:28:22 <chenzeng> today, I have finished the coding and simple tests of TimeTrigger. I will submit it this week. next, I will modify the design of OperationEngine, and finish the remaining coding.
14:28:32 <chenzeng> one question. according to the discussing with saggi, OperationEngine need to track the operation. does it mean some operations like delete operation need finish its logic when executing it.
14:29:34 <saggi> Delete is simple as it just means to mark the checkpoint as deleted. Another pass will actually free all the resources asynchronously.
14:29:56 <saggi> This is because we want to assume atomic deletes.
14:30:04 <saggi> s/assume/assure
14:30:32 <chenzeng> yes, i aggree.
14:31:16 <chenzeng> so, we need to encapsulate the logic into the operation class, right?
14:31:35 <saggi> What logic?
14:32:12 <saggi> The actual deletion pass, or GC (Garbage Collection) pass is done in Smaug independently from the operation.
14:32:21 <chenzeng> the operation
14:33:13 <saggi> So the operation just needs to make sure it maked all the required checkpoints as deleted. This should be in the operation. For example if we need to mark 10 checkpoints and we crashed after the 5th we need to continue once we restart.
14:33:59 <chenzeng> ok, understand.
14:34:30 <saggi> When creating multiple checkpoints. We also need to track those.
14:34:50 <zengyingzhe> So OperationEngine needs to record the progress of one operation to DB?
14:35:36 <saggi> Yes. It needs to make sure it handles crashes in the middle of operations or cases where the trigger was missed.
14:36:12 <chenzeng> what is the meaning of " trigger was missed"
14:36:42 <saggi> You have a trigger at 00:00 and the host was down between 23:59 and 00:01.
14:36:55 <saggi> In that case you will miss that daily backup
14:37:23 <chenzeng> i think operation class need to make sure the logic be finished correctly.
14:37:37 <saggi> Yes, and make sure it actually started
14:38:23 <saggi> And if you were down until 14:00 than you might not want to backup since it will stress the system during high load. So timed triggers should have a window of operation.
14:38:31 <saggi> user defined window of operation
14:39:33 <chenying_> saggi I have a question about Operation action.
14:39:50 <chenying_> In pluggable_protection_provider.pu this file. I note that enum Operation  ony have protect, enable,disable,restore these action , but do not have 'delete'. Why?
14:40:21 <chenying_> As a user he have a requirement deleting the backup data of a volume resource.
14:40:26 <saggi> chenying_: I think I missed it
14:40:49 <chenying_> OK  I see
14:41:06 <saggi> chenying_: Good catch!
14:41:34 <saggi> #topic open discussion
14:41:35 <chenzeng> saggi: can we use the crontab format for the timetrigger ?
14:41:44 <chenying_> I will do some review about the protection_provider later.
14:42:27 <chenzeng> i don't understand the rfc2445
14:43:28 <saggi> http://www.kanzaki.com/docs/ical/rrule.html
14:44:21 <chenzeng> saggi:thanks
14:44:36 <saggi> chenzeng: But you are right, crontab is simpler.
14:45:14 <saggi> I say we will start with crontab and see if users ask for the more complex features of ical\rrule
14:45:15 <chenzeng> saggi:will we use crontab, or support both
14:45:22 <zengyingzhe> rfc2445 is complex indeed.
14:46:03 <saggi> chenzeng: OK, I agree with you. crontab for now.
14:46:20 <smcginnis> saggi: Speaking from experience from working on a data protection product in the past, user protection rules usually end up being more complex than cron can represent.
14:46:21 <chenzeng> saggi:ok
14:46:34 <saggi> smcginnis: hmmm
14:46:43 <smcginnis> But that said, I think it's probably simpler to start with cron and make it more complex later if that's what's needed.
14:46:59 <smcginnis> Probably better not to go for the more complex scheme right away. ;)
14:47:22 <saggi> chenzeng, smcginnis: We'll add a format specifier in the trigger. crontab only for v1
14:47:29 <smcginnis> saggi: +1
14:47:48 <chenzeng> saggi:ok
14:48:11 <saggi> #action add format specifier for timed triggers
14:48:48 <saggi> Anything else? We have 10 more minutes!
14:49:04 <chenzeng> saggi:in order to make sure the consistance of operation, do you have any good ideas?
14:50:15 <saggi> Always persist before doing anything. Make sub operations that are either idempotent or sub operations you can always verify if they occurred or not.
14:50:47 <saggi> chenzeng: We could talk in #openstack-smaug about specific operations if you like
14:51:07 <saggi> OK, that I think we are done.
14:51:09 <chenzeng> saggi:ok
14:51:26 <saggi> Thanks everyone
14:51:31 <smcginnis> Thanks!
14:51:35 <chenying_> Thanks
14:51:42 <chenzeng> thanks
14:51:51 <zengyingzhe> Bye
14:51:55 <saggi> I hope that next time yinwei will join us.
14:52:07 <saggi> #endmeeting