16:05:26 <daemontool> #startmeeting thu-04-02-2016 16:05:27 <openstack> Meeting started Thu Feb 4 16:05:26 2016 UTC and is due to finish in 60 minutes. The chair is daemontool. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:05:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:05:30 <openstack> The meeting name has been set to 'thu_04_02_2016' 16:05:38 <daemontool> Hi Team 16:05:42 <ddieterly> hi 16:05:45 <reldan_> hi 16:05:51 <daemontool> today topics are avaialble here https://etherpad.openstack.org/p/freezer_meetings 16:06:26 <daemontool> we have today a specific etherpad for the tenant backup topic here https://etherpad.openstack.org/p/tenant-backup 16:06:45 <daemontool> #topic Define common metadata for all backups 16:07:12 <daemontool> so we need to define a metadata that will be used by all the backup methods 16:07:31 <daemontool> ie. file based, incremental, cindernative, novanative and so on 16:07:58 <daemontool> we have so far a schema example available here https://etherpad.openstack.org/p/tenant-backup 16:08:43 <daemontool> reldan, we do not use anymore the old medata, right? 16:08:59 <reldan_> Nope, it was only swift based 16:09:18 <reldan_> so it is obsoleted and deleted 16:09:41 <daemontool> ok 16:09:58 <reldan_> Actually about tenant backup 16:10:14 <reldan_> Who is the driver? Who defines requireents? 16:10:24 <reldan_> requirements 16:11:08 <daemontool> reldan, Igood question 16:11:31 <reldan_> Because otherwise we can create something, that is very good but not very practical 16:11:33 <daemontool> I think, one way of defining basic requirements 16:11:49 <daemontool> is to get them from our Companies 16:12:00 <daemontool> so from HP I think someone should talk with Arun 16:12:17 <daemontool> from 99Cloud we can get some from EinstCrazy and zhangjn 16:12:36 <reldan_> Yes, exactly - we need a guy who wants “tenant backup” and ask - how do you see this feature? how are you going to use it? 16:12:37 <daemontool> I'll take the requirements from Ericsson 16:12:46 <daemontool> yes 16:12:54 <daemontool> I'd say that as a quick win 16:12:59 <daemontool> for volumes backups 16:13:19 <daemontool> we can integrate with the cinder volumes incrementals feature 16:13:28 <daemontool> available from Liberty 16:13:30 <zhangjn> volumes backups online 16:14:01 <daemontool> zhangjn, by online you mean, without downtime right? 16:14:03 <reldan_> For example I have question - if we have many volumes, we cannot implement backup immidiatele. All backups will have different time. How we specify the time of tenant backup then? 16:14:04 <daemontool> ok 16:14:08 <daemontool> I'm adding the requirements 16:14:11 <daemontool> on etherpad 16:14:34 <daemontool> reldan, probably we can manage that from the scheduler? 16:16:05 <reldan_> Yes, let’s say we started our backup at 2:00 and we have 4 volumes - 1G, 2G, 100G, 1K G - so backups very completed at 2:00:30, 2:00:50, 2:04:00, 2:14:00 16:16:31 <reldan_> But for tenant backup we need to specify time - should it be 2:00? 16:17:12 <zhangjn> Maybe define every volume backup time 16:17:39 <ddieterly> how about start time and end time for each volume? 16:17:39 <reldan_> Or what to do if by some reason we can do backup for 3 of 4 our volumes. And 4-th returns error 16:17:46 <zhangjn> Use can config every volume or vm backup time 16:17:49 <zhangjn> user 16:18:11 <reldan_> Yes, exactly - here a lot of questions and it will be great to some product owner - who knows how it should be from practical point of view 16:18:12 <daemontool> ddieterly, we do not know the end time :( 16:18:18 <reldan_> to have 16:18:55 <dmellado> daemontool: would it be possible to have some kind of estimate for that? 16:19:03 <reldan_> If we have no such person, I can imagine some metadata format - but I doupt it will be very good in real life 16:19:22 <daemontool> we use the cinder api, so it's cinder dependant 16:19:30 <daemontool> it depends on the volume size 16:19:47 <daemontool> but I'm not sure we can do an estimate 16:19:55 <daemontool> it depends on the current load of the cinder storage nodes also 16:20:13 <daemontool> on how many backup are happening etc 16:20:16 <daemontool> so 16:20:18 <dmellado> I see 16:20:20 <daemontool> on which case 16:20:30 <daemontool> the volumes that belongs to a tenant 16:20:36 <daemontool> should be taken at the same time? 16:20:55 <zhangjn> backup_start_time and backup_end_time in our schema 16:21:01 <daemontool> ok 16:21:06 <dmellado> even if they do start at a given time we won't be sure if they end at the same time 16:21:12 <dmellado> so if they are thought as ' a block ' 16:21:23 <dmellado> maybe we'd have to wait for the slowest one to end 16:21:58 <daemontool> I think we need to define probably a timeout, that would be backup_end_time? 16:23:57 <daemontool> dmellado, ++ 16:24:09 <daemontool> reldan, your question is about what we do if one volume fail? 16:24:25 <daemontool> so here we have the same challenge of parallel backups right? :) 16:24:27 <reldan_> Yes 16:24:34 <reldan_> Yes, exaclty 16:24:40 <daemontool> we solved that by defining a policy 16:24:41 <daemontool> like 16:25:06 <daemontool> strict: if one fail, the job session fail, and all the volumes backups needs to be retaken 16:25:25 <reldan_> Yes, but we need it in metadata 16:25:36 <daemontool> reldan, yes 16:25:39 <daemontool> ok 16:25:58 <reldan_> So probably we will have corrupted tenant backups 16:26:03 <daemontool> so we can have a key called 16:26:13 <openstackgerrit> Pierre Mathieu proposed openstack/freezer: Fixing database config file parsing https://review.openstack.org/276333 16:26:18 <daemontool> failure_policy: {strict, flexible} ? 16:26:24 <reldan_> Yes 16:26:44 <reldan_> Let’s say we want to restore tenant backup now, and the latest one contains only 3 volumes of 4 16:26:56 <reldan_> And we have previous backup with 4 of 4 16:27:02 <reldan_> What I should do for restore? 16:27:44 <daemontool> so that backup 16:27:48 <daemontool> to have 3 of 4 16:28:07 <daemontool> it mean the user set failure_polic: 'flexible' 16:28:12 <reldan_> yes 16:28:15 <daemontool> policy 16:28:22 <reldan_> should we restore 3 of 4? 16:28:32 <reldan_> or should we restore previous one with 4 of 4? 16:28:32 <daemontool> yes 16:28:38 <daemontool> 3 of 4 16:28:41 <reldan_> or should ve restore one from old and 3 from new? 16:28:55 <daemontool> I think 3 of 4 16:29:11 <daemontool> because the backup session if OK according to the policy 16:29:27 <daemontool> and is just the previous backups available from the restore date set by the user 16:29:42 <daemontool> we have failure_policy 16:29:47 <daemontool> it semplify 16:29:51 <daemontool> the whole thing 16:30:03 <daemontool> because if you have 'strict' 16:30:10 <reldan_> But what I should do if I wnat to restore all 4? 16:30:11 <daemontool> 3 of 4 is not valid 16:30:36 <reldan_> Let’s say I have 1 web node, 2 sql node (master - slave), 1 processing node 16:30:38 <dmellado> wouldn't that lead to data issues? that would happen if any data from changes in between 16:30:44 <daemontool> in that case the user shoudl have the the 'failure_policy': 'strict' 16:30:49 <reldan_> If I restore 3 of 4 - it will be not very good for me 16:31:10 <daemontool> reldan, yes I agree 16:31:14 <daemontool> so in that case you set 16:31:21 <daemontool> 'failure_policy': 'strict' 16:31:24 <reldan_> I suppose we need to have a mechanism to choose backup for restore not only by datetime 16:31:34 <daemontool> yes 16:31:49 <reldan_> like in my sql: select last full backup to this date 16:31:50 <zhangjn> yes 16:32:04 <daemontool> that's a requirement 16:32:10 <daemontool> ++ 16:32:29 <openstackgerrit> Pierre Mathieu proposed openstack/freezer: Fixing database config file parsing https://review.openstack.org/276333 16:32:32 <reldan_> How we are going to store our metadata? In mysql? 16:32:42 <daemontool> reldan, one sec 16:32:47 <daemontool> let me add the requirement 16:33:21 <daemontool> ok 16:33:23 <zhangjn> mysql backup we can use xtrabackup tools. 16:33:57 <daemontool> zhangjn, we have been analyzing that, we can talk about that off line if you want 16:34:05 <daemontool> I'll tell you the whole story 16:34:14 <daemontool> :) 16:34:28 <zhangjn> OK 16:34:36 <daemontool> reldan, I think, 16:34:41 <daemontool> we need to store the metadata 16:34:59 <daemontool> in the API and also in the media storage we set 16:35:09 <daemontool> for cindernative we do not have a media storage to set 16:35:29 <daemontool> but I think, we should store it on the freezer-api 16:35:34 <daemontool> cause from that we can compute metrics 16:35:38 <daemontool> and show data from the web ui 16:36:08 <daemontool> for now, I wouldn't solve which db are we going to use as api backup other than elastic search 16:36:11 <reldan_> Should freezer-agent know about api? 16:36:14 <daemontool> cause that's story itself 16:36:21 <daemontool> the scheduler knows 16:36:45 <daemontool> the agent return the metadata 16:36:47 <daemontool> to the scheduler 16:36:51 <reldan_> Ok ) 16:36:55 <daemontool> and the scheduler upload it to the api 16:36:59 <daemontool> as we are doing now 16:37:04 <daemontool> we have that already 16:37:04 <reldan_> So who is the guy responsible for metadata format? 16:37:19 <openstackgerrit> Pierre Mathieu proposed openstack/freezer: Fixing database config file parsing https://review.openstack.org/276333 16:37:28 <daemontool> I think we need 2 dudes for that 16:37:37 <daemontool> 1 backup the other :) 16:37:47 <daemontool> well, it's more ha 16:37:55 <zhangjn> ha 16:38:00 <dmellado> daemontool: gotta run, would you put the 'proposed schedule' for the source code overview in etherpad? 16:38:07 <daemontool> dmellado, yes 16:38:11 <daemontool> thanks a lot for your inputs 16:38:20 <dmellado> np, thanks! ;) 16:38:32 <daemontool> reldan, zhangjn are you ok doing that? 16:38:35 <reldan_> I believe it should be someone who knows requirements. 16:38:38 <daemontool> vannif, needs to do that also 16:38:42 <daemontool> zhangjn, and vannif ? 16:38:46 <daemontool> me? 16:39:04 <daemontool> it should be someone that works on the agent 16:39:10 <daemontool> and someone that works on the api 16:39:23 <daemontool> the requierments we can define that together 16:39:30 <daemontool> as we need to collect them from our companies also 16:39:33 <reldan_> I can create metadata - but it may be just useless for guy who actually is going to use tenant backup 16:40:16 <daemontool> is vannif around there? 16:40:27 <daemontool> he did quite a good job last time with scheduler metadata 16:40:38 <reldan_> Nope he is not here 16:40:44 <daemontool> ok 16:40:54 <daemontool> so I can do it for now 16:41:08 <daemontool> reldan, and zhangjn if you wants to do it too 16:41:12 <daemontool> you are welcoem 16:41:22 <zhangjn> we have a holiday(Spring Festival). 16:41:26 <reldan_> Thank you :) 16:41:35 <daemontool> ok reldan we'll do it 16:41:42 <reldan_> Just because now we have only that 16:41:46 <reldan_> As a tenant, I need to use Freezer to backup all my data and metadata from an OS Cloud and restore it at my convenience. 16:41:52 <reldan_> And it is good 16:42:09 <reldan_> But it would be great a real person, who wants this because he is really running something 16:42:25 <reldan_> and it is really important for him to have tenant backup 16:42:35 <reldan_> Like guy who runs e-commerce application 16:43:18 <daemontool> reldan, can you ask to Arun there? 16:43:20 <reldan_> Otherwise it is more like - we need a tenant backup just because it is cool 16:43:21 <daemontool> that's his job 16:43:50 <reldan_> Oh, I don’t have lync on my comp. Probably ddieterly can 16:44:12 <daemontool> ddieterly, can you have a conversation with Arun 16:44:12 <zhangjn> I have some customer want this function. 16:44:27 <daemontool> and get the requirements for the backup as a service in general? 16:44:29 <reldan_> zhangjn: Great. Could you gather requirements? 16:44:34 <daemontool> I'll do the same 16:44:57 <ddieterly> yes 16:45:12 <daemontool> also Pierre can have some requirement 16:45:15 <daemontool> Slashme 16:45:19 <daemontool> ddieterly, thanks a lot 16:45:54 <zhangjn> Yes, I will gather requirements. 16:46:00 <reldan_> Yes, sure - let’s gather requirements first. If we can, otherwise we will implement something that may not very usefull for our customers. I’m afraid this situation 16:46:03 <daemontool> we have a product owner here 16:46:05 <reldan_> Thank you 16:46:14 <openstackgerrit> Pierre Mathieu proposed openstack/freezer: Fixing database config file parsing https://review.openstack.org/276333 16:46:16 <daemontool> ok 16:46:20 <zhangjn> I will push our guys to join in this project. 16:46:34 <daemontool> zhangjn, lol thank you :) 16:46:46 <daemontool> now 16:46:51 <reldan_> Currently I can start to add —force and —increment to cindernative 16:46:58 <daemontool> ok let's move forward 16:47:01 <daemontool> reldan, that's good 16:47:09 <zhangjn> Some POC will start after Spring Festival 16:47:11 <daemontool> I think zhangjn can do that 16:47:19 <daemontool> reldan, with your guidance? 16:47:31 <daemontool> so he'll take familiarity with the base code? 16:47:34 <reldan_> Yes, sure - it shouldn’t be very difficult 16:47:48 <reldan_> zhangjn: please contact me anytime 16:47:53 <daemontool> ok, please arrange a meeting 16:48:07 <zhangjn> ok, I will 16:48:10 <daemontool> ty 16:48:12 <daemontool> zhangjn, when the Spring Festival 16:48:16 <daemontool> will ends up? 16:49:01 <daemontool> ok let's move forward 16:49:04 <daemontool> to the next topic 16:49:29 <daemontool> #topic Source code walkthru definition 16:49:34 <zhangjn> Feb 6 ~ Feb 15 16:49:55 <daemontool> zhangjn, ok so we'll arrange the source code walkthough session 16:50:02 <daemontool> after the 15th of Feb? 16:50:14 <zhangjn> yes 16:50:17 <daemontool> ok 16:51:01 <daemontool> let's move forward 16:51:03 <daemontool> to 16:51:15 <zhangjn> I have a beginer in freezer. learn and learn, :) 16:51:16 <daemontool> #topic Mitaka-3 release 16:51:24 <daemontool> ok :) 16:51:28 <daemontool> so 16:51:33 <daemontool> I'm doing that currently 16:51:46 <daemontool> there are few issues, like adding modules to global-requirements.txt 16:52:02 <daemontool> fixing devstack api plugin (it's required by something I can't remember) and the tagging 16:52:53 <daemontool> we should be good for M3 timeline at 16:53:26 <daemontool> 13th of Feb 16:54:06 <daemontool> 8-12th Feb 16:54:09 <daemontool> need to verify 16:54:14 <daemontool> anyway that is important 16:54:24 <openstackgerrit> Pierre Mathieu proposed openstack/freezer-web-ui: Using a smarter way to get freezer-api URL https://review.openstack.org/275804 16:54:37 <daemontool> moving forward 16:54:50 <daemontool> #topic Change meeting to 11am GMT every Thu to facilitate people from Asia to participate 16:55:12 <daemontool> anyone wants to submit a patch for that to the openstac-infra/meetings repo? 16:55:28 <daemontool> I can do that 16:55:43 <reldan_> great 16:55:45 <daemontool> so the topic are over 16:55:58 <daemontool> there's anything else you want to discuss? 16:56:03 <zhangjn> great 16:56:08 <daemontool> anyone wanted to say somethign and didn't had the chance? 16:56:22 <daemontool> something more on the volumes meta? 16:56:32 <daemontool> reldan, we need to set a meeting 16:56:41 <ddieterly> 11 AM GMT is like 4AM for me 16:56:41 <ddieterly> yikes 16:56:46 <daemontool> with frescof_ 16:56:48 <daemontool> mmmhhh 16:56:49 <daemontool> ok 16:56:55 <daemontool> that's the issue 16:57:02 <daemontool> how can we do this at a time 16:57:08 <daemontool> that is suitable for US and Asia? 16:57:25 <ddieterly> 2 PM GMT? 16:57:26 <daemontool> 1PM ? 16:57:41 <daemontool> zhangjn, ? 16:57:42 <zhangjn> 4 AM ? 16:57:50 <zhangjn> or PM? 16:57:50 <daemontool> lol 16:57:53 <daemontool> am 16:58:09 <daemontool> in us 11AM GMT is 4AM 16:58:29 <daemontool> ddieterly, 1 PM sound? 16:58:35 <zhangjn> GMT 4 AM? 16:58:35 <ddieterly> early 16:58:53 <ddieterly> that's 6AM my MST 16:58:58 <daemontool> how is this problem solved 16:59:06 <daemontool> in the other teams? 16:59:17 <ddieterly> i could do 2 PM GMT which is 7 AM MTD 16:59:27 <daemontool> zhangjn, is that ok? 16:59:31 <ddieterly> what is that in Japan? 16:59:33 <daemontool> it will be 10pm in china 16:59:57 <ddieterly> who is in china? 17:00:02 <daemontool> zhangjn, and EinstCrazy 17:00:16 <ddieterly> how is 10pm for them? 17:00:25 <daemontool> they will work on baas 17:00:30 <daemontool> mainly cinder and nova 17:01:11 <ddieterly> if i have to, i can do 1PM GMT 17:01:49 <daemontool> 1.30 pm that would be 9.30 pm ? 17:01:51 <daemontool> 6.30? 17:02:39 <zhangjn> I think 2 PM is ok 17:03:15 <daemontool> zhangjn, ddieterly thanks for your flexibility 17:03:28 <daemontool> so 2PM GMT is the new time 17:03:37 <daemontool> I'll send a patch 17:04:33 <zhangjn> ddieterly ,thank you :) 17:04:35 <daemontool> #endmeeting thu-04-02-2016