14:00:28 <saggi> #startmeeting smaug 14:00:29 <openstack> Meeting started Tue May 3 14:00:28 2016 UTC and is due to finish in 60 minutes. The chair is saggi. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:33 <openstack> The meeting name has been set to 'smaug' 14:00:38 <saggi> Hi everyone 14:00:40 <saggi> Who's here? 14:00:44 <chenying_> hi 14:00:47 <xyang1> hi 14:00:52 <yuval> hey 14:00:53 <xiangxinyong> hi 14:00:54 <yinweimac> hi 14:01:13 <sgotliv> o/ 14:01:14 * smcginnis is lurking 14:01:20 <saggi> :) 14:01:31 <sgotliv> saggi, I am lurking too 14:02:00 <saggi> OK, so let's start. 14:02:10 <saggi> #topic OpenStack Summit update 14:02:29 <saggi> So I was the only one from that Smaug that was able to get it to the summit. 14:02:57 <saggi> I thought it went well. 14:03:10 <saggi> I tried to go around and shove Smaug in to people's faces 14:03:22 <saggi> I hope I wasn't to obnoxious. 14:03:25 <xiangxinyong> :) 14:03:25 <smcginnis> Lot's of interest and good things said about the project from what I heard. 14:03:51 <saggi> I got to speak to a lot of people and there is some interest stirring. 14:04:03 <saggi> There are a lot of people I need to touch base with 14:04:22 <saggi> but the general feeling is that people understand that this is a piece that is currently missing. 14:04:50 <xyang1> saggi: hi, I talked to you at the cinder session. will need to talk more later 14:05:11 <saggi> xyang1: sure, no problem 14:05:16 <saggi> I will send a link to the mailing list with the vbrownbag presentation I did. 14:05:35 <saggi> People tell me it's good. I still can't bare looking at my own recordings. 14:05:53 <smcginnis> ;) 14:06:02 <saggi> I got some good feedback from Cinder people. 14:06:07 <xiangxinyong> saggi: great work 14:06:28 <saggi> There are still point of integration that need to get ironed out. 14:06:57 <saggi> Mainly a way for us to understand more about types. Since now the are (purposefully) opaque. 14:07:32 <saggi> And this will cause problem if we ever need to do something vendor specific. Since we will need to pick inside and see which vendor is driving what type. 14:07:49 <xyang1> we are designing replication api in cinder, so good to discuss about how to integrate with smaug 14:08:21 <saggi> Every vendor I spoke with (admittedly not a lot of them) tell me to manually poke around the host and voume and guess what is installed. 14:08:54 <saggi> xyang1: Are there any special irc meetings for that. Or is it in the general Cinder IRC meeting? 14:09:28 <xyang1> saggi: we discussed about it on Friday and wednesday 14:09:54 <saggi> I was on Wednesday but I couldn't make it on Friday. I'll look up the meeting notes. 14:10:00 <xyang1> I have specs, will update them after getting feedback from the summit 14:10:08 <saggi> But I was asking about any on going tracking of the feature. 14:10:27 <xyang1> saggi: we talked about them on Friday afternoon 14:11:16 <smcginnis> saggi: So far just weekly meetings and midcycles. No special meetings planned for it. 14:11:27 <saggi> xyang1: I know, I was going to get there but I there were problems with my flight\luggage I had to sort out. 14:11:42 <saggi> smcginnis: OK, it's on us to be on top of it. 14:12:02 <xyang1> saggi: ok, let me get the spec link 14:12:13 <smcginnis> saggi: We can try to keep you updated on any changes. 14:12:19 <saggi> smcginnis: Thanks 14:12:33 <smcginnis> saggi: But if there is something smaug needs exposed to be able to integrate well, let us know and we can work out a way to enable it. 14:13:11 <xyang1> https://review.openstack.org/#/c/229722/ 14:13:11 <saggi> smcginnis: I'll look up the spec today if it's updated. 14:13:22 <xyang1> this needs to be updated though 14:13:51 <saggi> xyang1: I added myself as a reviewer so I'll get an email when it's updated. 14:14:14 <xyang1> I'd like to describe the flow in the spec on how smaug can use this api 14:14:27 <saggi> xyang1: Go ahead. 14:14:36 <xyang1> ok, thanks 14:15:52 <chenying_> A good news. So far, The integration testing of cinder backup plugins has done. about volume backup, restore and delete action. 14:16:19 <xyang1> chenying_: are those in review or merged? 14:16:21 <saggi> chenying: Are you using v2 or v3? 14:16:41 <chenying_> some patches are in review 14:16:54 <saggi> chenying, please post a link 14:17:03 <chenying_> <saggi>: you means cinder api V2? 14:17:09 <saggi> yes 14:17:21 <chenying_> yes we use cinder api V2 14:17:43 <smcginnis> We've actually moved on the v3 now with the addition of microversion in our API. 14:18:41 <xyang1> saggi: I scanned your repo, it looks like currently there is code for backup and restore, but not for replication yet. is that right? 14:18:51 <saggi> Yes 14:18:52 <chenying_> Ok I will start the work using cinder api v3. 14:19:39 <saggi> chenying, There are some minor things to keep in mind when using microversions though. I'll speak to you about it afterwards. 14:19:59 <chenying_> Ok 14:20:07 <saggi> xyang1: We want to have the full cycle working with good testing and then start adding more features. 14:20:19 <saggi> cycle==backup->restore 14:20:22 <xyang1> saggi: sure 14:21:00 <saggi> I also, had some conversation with Freezer's PTL. I think I gave a strong arguments about them trying to integrate instead of them trying to do everything. We chose to schedule a specific meeting and see 14:21:08 <saggi> how we can integrate. 14:21:36 <xiangxinyong> saggi: integrate with freezer? 14:21:47 <xyang1> saggi: I also heard that they want to do everything thenselves 14:22:01 <saggi> xyang1: I know :) I tried to be political. 14:22:06 <xyang1> saggi: your approach is better 14:22:47 <saggi> xyang1: I am trying to convince them to concentrate on guest assistance 14:22:55 <yinweimac> they want to do what by themselves? for eg? 14:23:16 <saggi> They currently bypass cinder and use their own implementation for backup\replication 14:23:22 <chenying_> https://review.openstack.org/#/q/project:openstack/smaug some patches about the plugins are open. Welcome to review them. 14:23:42 <chenying_> We have plans to integrate with freezer 14:23:53 <chenying_> also about okka 14:23:59 <saggi> and now they also want to do monitoring. Which I think is a bit of spreading a bit too thin. 14:24:13 <saggi> guest agent\assitance is a whole different story 14:24:49 <saggi> I spoke with some operators and vendors and they don't really agree about having an agent or doing it through ssh 14:26:17 <saggi> It's something we need to talk to them about or maybe someone else. I don't know if this should be in our scope or not. 14:26:25 <yinweimac> yes, customers care about security issue. agentless is more preferred 14:27:04 <saggi> I suggested using something similar to Ansible (or just Ansible) that deploys code, runs it and then cleans up afterwards. 14:27:20 <saggi> But this is something I need actual operators to way in. 14:28:51 <saggi> Since it's less technological and more preference standpoint. In the end we will need to run privileged code in the guest. 14:29:02 <saggi> No way around it. 14:29:28 <saggi> Unless people don't want application level consistency 14:29:32 <yinweimac> what they want agent to work for? 14:29:46 <yinweimac> monitor failure status? 14:29:51 <yinweimac> ok 14:29:52 <saggi> Before backup 14:29:54 <yinweimac> got it 14:29:55 <saggi> Tell DB to flush all transactions 14:30:04 <saggi> Get their application to flush in memory infor 14:30:15 <saggi> Application level stuff 14:30:48 <saggi> This is classic use case for Smaug as it requires coordination between multiple OpenStack entities. 14:31:24 <sgotliv> saggi, did you have a chance to talk to the Trove community about backups? 14:32:37 <saggi> No, I did not have a chance to speak with them. 14:33:03 <sgotliv> they may be interested too 14:33:06 <saggi> I did speak briefly with the Manilla guys since they are now working on their backup apis. 14:33:57 <saggi> sgotliv: They are not targeted for Smaug v1. (unless we get more resources) but I'll try and see what they think about the project. 14:33:57 <xyang1> saggi: anyone working on manila plugins in smaug? 14:33:58 <sgotliv> and Trove consider to start using Cinder and Manila to backup databases 14:34:41 <yinweimac> xyang1: not yet 14:35:00 <saggi> xyang1: No, we don't have enough resources for that. We wanted to push them to version 1 but we don't see how we can't get that many things stable. 14:36:05 <saggi> xyang1: https://wiki.openstack.org/wiki/File:Smaug-available-protectables.svg 14:36:54 <yinweimac> but that would be a good suggestion and we will take a look at it later, right? saggi? both manila and trove I mean 14:37:50 <saggi> yinweimac: They were in the original tree. If we get more people early enough in the cycle than we'll add more protectables. 14:38:55 <saggi> I prefer stability right now more than features. 14:38:58 <xiangxinyong> saggi: do you hear about ekko? http://git.openstack.org/cgit/openstack/ekko/ 14:39:04 <saggi> Respect is something you earn 14:39:26 <xyang1> saggi: are you using cinder backup and restore APIs? 14:39:37 <yinweimac> yes 14:39:38 <saggi> xyang1: Yes 14:39:51 <xyang1> saggi: great 14:39:55 <yinweimac> currently the cinder plugin is based on cinder backup/restore 14:40:10 <xyang1> yinweimac: thanks, I'll take a look 14:41:04 <saggi> I also met up with sgotliv and Sean Cohen which told me about some use cases they have from users at Red Hat. 14:41:26 <saggi> It seems like we could be a good fit. 14:41:39 <sgotliv> saggi, absolutely 14:43:26 <saggi> There are a lot more stuff that happened. But we're running out of time. I'll compose an email to the team so you are all up to date. yuval already has some stuff to figure out how to solve. 14:43:38 <saggi> #topic Status updates 14:43:58 <saggi> I know that you all worked on getting the full backup\restore running by this week 14:44:07 <saggi> How is that doing? 14:44:09 <saggi> going 14:44:20 <saggi> Are we ready for shipping :) 14:44:21 <yinweimac> it works now 14:44:38 <saggi> nova\cinder\neutron ? 14:44:52 <yinweimac> in shenzhen lab we have integrated smaug with hybrid cloud, and both nova/cinder works 14:45:14 <yinweimac> there we don't need restore network, just binding floating ip will do 14:45:46 <saggi> Are you binding them through restore options? 14:45:53 <yinweimac> and I guess in chengdu lab, chenying also have got one smaug with devstak work. Just only cinder there. 14:46:02 <yinweimac> yes 14:46:26 <yinweimac> you mean restore parameters? 14:46:33 <saggi> yes 14:46:49 <yinweimac> no, that's same floating ip we protected 14:47:00 <saggi> Oh, right since they are proxied 14:47:14 <yinweimac> parameter is the feature we need further enhanced 14:47:22 <saggi> are there tests for everything? 14:47:32 <yinweimac> almost 14:48:02 <yinweimac> create plan, create trigger, schedule operation, create checkpoint, list checkpoint, restore checkpoint, etc. 14:48:13 <saggi> awsome! 14:48:56 <chenying_> chenhuayi is developing the restore action of neutron plugin. After that, we will start integrating test with neutron plugin. 14:49:12 <saggi> great 14:49:22 <saggi> #topic Task Flow improvement 14:49:34 <yinweimac> yes, we will have two categories of plugins, one for original openstack, the other for hybrid cloud 14:49:45 <saggi> That's the plan 14:50:13 <saggi> yuval did a lot of good work getting tasks to work like we want them to 14:50:44 <yuval> hey all :) 14:50:46 <yuval> well: 14:50:50 <yuval> Regarding the operations (protect, restore, delete) workflow 14:51:06 <yuval> In the current design, resource tasks start in order of their dependency, but they might invoke async actions, which can complete after their parent actions are invoked. 14:51:31 <yuval> As you might recall, I sent the suggestion for the workflow to the mailing list 14:51:50 <yuval> The purpose of the design is to let Protection Plugin writers the ability to perform the steps required for the operation in the correct order and in parallel, if possible 14:52:12 <saggi> BTW, async actions are polled actions. actions that we wait on someone else (like cinder backup) and the worflow host doesn't invke itself. 14:52:26 <yuval> correct, thanks 14:52:41 <yuval> The workflow issue is highly important. The design need to support all use-cases and we need to complete it quick, as all Protection Plugins depend on it 14:53:33 <yuval> the suggestion I sent includes two types of activities: SyncActivity and ParallelActivity 14:53:47 <yuval> After discussion with Saggi and his impressions from the summit, I believe there is a need for an additional type of activity, which operates before dependent resources Activity begin 14:54:14 <yuval> (this can be used, for example, to tell the guest agent to complete transactions and flush) 14:54:31 <yuval> This concludes for 3 types of activities, which differ by time they are invoked in respect for other resource activities: 14:55:01 <yuval> 14:55:02 <yuval> PreActivity (which operates before dependent resources activities are invoked. for example, tell guest agent to flush) 14:55:06 <yuval> WorkerActivity / ParallelActivity (which operates in parallel to other resources WorkerActivities. can be used to perform heavy lifting) 14:55:09 <yuval> PostActivity / SyncActivity (which operates after dependent resources completed all their activities. can be used to perform work which depend on child resources status) 14:55:22 <yuval> 14:55:25 <yuval> You can view an example of such design (currently lacking the PreActivity, and names may differ): https://review.openstack.org/#/c/310385 https://review.openstack.org/#/c/310386 https://review.openstack.org/#/c/310498 14:55:43 <yuval> </p> :) 14:56:34 <saggi> pre\post are used to bookend some actions like quiesce and unquiesce (if this is the proper term) 14:57:33 <saggi> It's important that (A) people review this so it gets in (B) Everyone understands how it works since all the plugins will use this. 14:58:01 <saggi> Changing this interface later effects all plugins. It's important we get this right. 14:58:53 <saggi> yinweimac, I especially want a review from you. 14:58:54 <yuval> I'll send an updated document tomorrow 14:59:38 <saggi> This is the last minute. So I'd like to thank all the new people that joined us today. We hope you come back for the next meeting. 14:59:52 <yinweimac> sorry for the late response, I was busy with our tdr3 demo and incomming demo next week 15:00:00 <saggi> We do want to make it weekly and have one time for US and one time for China. 15:00:03 <yinweimac> I will try to get time to check it 15:00:25 <yinweimac> that's good 15:00:40 <saggi> smcginnis, xyang1, sgotliv, thank you again for helping. 15:00:51 <xyang1> saggi: thank you 15:00:53 <smcginnis> saggi: +1 15:01:13 <saggi> #endmeeting