14:03:12 <m3m0_> #startmeeting 'Freezer meeting 11-02-16'
14:03:12 <openstack> Meeting started Thu Feb 11 14:03:12 2016 UTC and is due to finish in 60 minutes.  The chair is m3m0_. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:03:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:03:16 <openstack> The meeting name has been set to '_freezer_meeting_11_02_16_'
14:04:00 <m3m0_> hello guys, raise your hand if you are joining the freezer meeting
14:04:02 <m3m0_> o/
14:04:27 <ddieterly> o/
14:04:35 <daemontool> o/
14:05:06 <m3m0_> let's wait 2 more minutes
14:05:09 <daemontool> ok
14:05:23 <daemontool> reldan,  \o
14:05:43 <reldan> daemontool: \o/
14:06:31 <m3m0_> I don't know if frescof_ is around but the first topics relates to him
14:07:01 <m3m0_> slashme not around as well so let's move forward with the second topic
14:07:04 <daemontool> yes I understand slashme and szaher have some thought about it
14:07:05 <daemontool> ok
14:07:14 <m3m0_> #topic Add options to not remove files before the restore (Deklan)
14:07:36 <daemontool> ddieterly, that is related to the issue you had
14:08:09 <daemontool> does it make sense the mail response I sent you on the reason why files in the directory are delete before doing a restore?
14:08:10 <m3m0_> ddieterly is there a commit or bp or bug related to this that we can review?
14:08:29 <daemontool> although I think we have to provide some more options to configure that
14:09:10 <m3m0_> just for everyone to have context on this, the problem is that while doing restore freezer removes the content of the directory to be restored first?
14:09:17 <ddieterly> i am sending a follow up email
14:09:29 <daemontool> ddieterly,  ty please
14:09:32 <ddieterly> the problem is that the backup came from /tmp/freezer-test/...
14:09:32 <daemontool> wirte that to a bp or bug
14:09:46 <ddieterly> then the restore was to /users/dieterld/
14:09:51 <daemontool> ok
14:09:58 <ddieterly> all files in /users/dieterld were removed
14:10:04 <ddieterly> that does not seem correct
14:10:27 <ddieterly> it seems catastrophic
14:11:14 <daemontool> I understand that
14:11:22 <daemontool> the thing is that if you want to do that
14:11:45 <daemontool> you have to backup /home/user with freezer
14:11:49 <daemontool> I mean
14:11:53 <daemontool> I udnerstand your case
14:11:57 <daemontool> bbut in production environment
14:12:01 <daemontool> for real world application
14:12:08 <daemontool> by not removing files lead in the past
14:12:12 <daemontool> to many issues
14:12:24 <daemontool> but we can provide options to do that
14:12:25 <daemontool> for sure
14:13:16 <daemontool> ddieterly,  the options we have are
14:13:23 <daemontool> 1) delete the directory before restore
14:13:44 <daemontool> 2) override only the files that exists in both the restore directory and in the backup
14:13:51 <m3m0_> so far the behaviour is that if you restore, freezer expects the path to be empty, otherwise the consistency of the backup/restore could be compromised
14:14:03 <daemontool> yes that's the issue
14:14:06 <daemontool> you backup a whole directory
14:14:11 <daemontool> you restore a whole directory
14:14:17 <daemontool> but we need to have mechanism
14:14:21 <daemontool> to avoid that
14:14:33 <ddieterly> you can say 'override existing files?'
14:14:37 <daemontool> to avoid ddieterly  case
14:14:46 <m3m0_> or --force
14:14:49 <ddieterly> but, don't delete files in the target directory
14:15:12 <daemontool> currently is not possible, but is very easy to provide
14:15:18 <ddieterly> no one expects files to be deleted when a restore happens
14:15:54 <daemontool> for applicatinos yes
14:15:58 <daemontool> let's say
14:16:03 <daemontool> you backup /var/lib/mysql
14:16:15 <daemontool> then disaster happens
14:16:31 <daemontool> then you restore /var/lib/mysql
14:16:39 <daemontool> what do you expect to have?
14:16:48 <daemontool> the same content of when you took the backup?
14:17:03 <daemontool> or the same with /var/lib/jenkins
14:17:13 <daemontool> where the xml files are read by the application when it starts
14:17:14 <ddieterly> if i want a clean directory before the backup, then i should manually delete all fles
14:17:23 <ddieterly> the restore should not do that itseld
14:17:36 <ddieterly> 'before the restore'
14:17:45 <daemontool> so , for instance
14:18:05 <daemontool> you execute the backups and you do not dereference symbolic links
14:18:20 <daemontool> then you restore
14:18:34 <daemontool> and the links with data of previous data
14:18:44 <daemontool> points to current files
14:18:55 <daemontool> and the application doesn't work
14:19:10 <daemontool> I agree with you
14:19:16 <daemontool> that we need to provide flexibility
14:19:32 <m3m0_> but we need to agree in a default behaviour
14:19:33 <daemontool> but removing the directory content before restoring
14:19:52 <daemontool> I think it make sense for applications in production
14:20:16 <daemontool> ddieterly, what default behavior do you recommend?
14:20:33 <daemontool> override only existing files?
14:20:36 <ddieterly> don't automatically delete anything
14:20:50 <ddieterly> have a flag to allow everything to be deleted
14:21:09 <ddieterly> have a flag to allow overriding existing files
14:21:39 <daemontool> that didn't worked in the past
14:21:48 <daemontool> not deleting I mean
14:21:54 <daemontool> for mongodb
14:22:02 <daemontool> ou have the journal logs in /var/lib/mysql
14:22:15 <daemontool> if you do not remove the directory before
14:22:17 <ddieterly> default behavior should be to just restore the files and if a file already exists, throw an error and stop unless the user has explicitly stated to override existing files
14:22:33 <m3m0_> freezer-agent --action restore --restore-abs-path /home/u/ --soft-restore ?
14:22:56 <daemontool> ok ddieterly  can you write a bp for that?
14:23:09 <ddieterly> yea, there should be an option to totally wipe out what is already in the directory
14:23:18 <ddieterly> sure
14:23:25 <m3m0_> I don't agree with that
14:23:49 <daemontool> I do not agree either... I've suffered that when I was working in ops in the past
14:23:51 <daemontool> but
14:23:53 <daemontool> let's have the bp
14:24:01 <daemontool> and we can discuss further
14:24:10 <daemontool> sound?
14:24:32 <m3m0_> the priority is to restore to a previous state, not deal with complex scenarios in which we can have different verions of files
14:24:33 <ddieterly> ?
14:24:51 <vannif> maybe leaving the existing files around is a good default behavior for "fs" mode. The default for modes like "mysql" could be to wipe out the destination before extracting files
14:24:56 <m3m0_> but we should warn the user that the directory is not empty
14:25:24 <daemontool> for sure we need to be more explicit expalning that in the options
14:25:27 <daemontool> and documentation
14:25:30 <m3m0_> to abort or continue the restore process
14:26:10 <daemontool> vannif,  for application that writes journals in the file system using aotmic writes (pretty much any nosql db)
14:26:20 <daemontool> the mode is fs
14:26:33 <daemontool> so if you do not delete that journals log
14:26:37 <daemontool> when you restart the app
14:26:39 <slashme> m3m0_: please let's not have anything interactive. It's hell to automate
14:26:41 <daemontool> it will reply them
14:27:06 <daemontool> replay
14:27:11 <m3m0_> agree on that, so we need to find a way
14:27:18 <daemontool> so you have a mixed that on the restore
14:27:20 <daemontool> from old and new
14:27:27 <m3m0_> or add new flags to deal with this
14:27:38 <daemontool> we need to provide more flexibility for sure
14:27:47 <daemontool> as ddieterly case r real
14:28:28 <daemontool> so let's have a bp
14:28:30 <m3m0_> we need a bp to define the correct behaviour on this
14:28:53 <daemontool> yes the only thing we are diverding is the default behaviour
14:29:07 <daemontool> s/diverding/diverging/
14:29:49 <daemontool> can we move forward?
14:30:07 <m3m0_> ddieterly: we should, ddieterly agree on this?
14:30:14 <ddieterly> yes
14:30:25 <m3m0_> do you want to provide a bp?
14:30:33 <ddieterly> ok
14:30:39 <daemontool> ty
14:31:04 <m3m0_> ty
14:31:33 <m3m0_> next #topic 'Define common metadata for all backups'
14:31:41 <daemontool> reldan, we had to do this
14:31:44 <daemontool> and we didn't
14:31:57 <m3m0_> #topic Define common metadata for all backups
14:32:03 <daemontool> so early next week we need to have 1 hour conversation
14:32:08 <daemontool> to define it
14:32:11 <reldan> Yes, do we have a requiremens? I remember that it should be the first step?
14:32:26 <m3m0_> we want to have a midcycle meetup as well
14:32:36 <daemontool> I've sent a request to the product owner
14:32:37 <daemontool> here
14:32:53 <daemontool> ddieterly,  did you manage to get some requirements from Arun?
14:33:02 <ddieterly> yes
14:33:07 <ddieterly> one sec please
14:33:10 <daemontool> ok
14:33:42 <ddieterly> The quick ones I can think of -from upstream perspective (not in priority order J )
14:33:42 <ddieterly> -          ES backup (logs and Freezer jobs)
14:33:43 <ddieterly> -          Metering (esp Billing related ones)
14:33:44 <ddieterly> -          Storage Quota management ( I am not sure how much we have here)
14:33:46 <ddieterly> -          Vertica DB
14:33:48 <ddieterly> -          Swift data backup
14:33:50 <ddieterly> -          Container backup
14:34:12 <m3m0_> swift data backup?
14:34:26 <ddieterly> Here is the complete requirement backlog
14:34:27 <ddieterly> https://jira.hpcloud.net/issues/?filter=26722
14:34:41 <ddieterly> yea, backup swift, i guess
14:34:43 <daemontool> ok we need that added in our wiki
14:34:47 <m3m0_> mmm
14:34:51 <reldan> Container backup?
14:34:53 <daemontool> let's collect the requirements
14:34:57 <reldan> Are we going to backup swift?
14:35:01 <daemontool> and then we discuss them in details
14:35:05 <m3m0_> I hope not
14:35:26 <daemontool> I don't know, we asked for requirements, that's what we got :)
14:35:30 <ddieterly> yea, backup swift to swift ;-)
14:35:42 <m3m0_> but adding more modes in freezer should require an improvement on the plugin layer
14:35:44 <reldan> Is it requirements for tenant backup?
14:35:55 <reldan> Just I see metering
14:35:56 <daemontool> good question
14:36:04 <reldan> And don’t sure that it is a part of tenant backup
14:36:05 <daemontool> reldan,  I think so
14:36:07 <szaher_> Yes, we need badly the plugin layer
14:36:39 <slashme> szaher_: +100
14:36:55 <daemontool> ok please write a bp for that
14:37:05 <szaher_> Ok
14:37:40 <ddieterly> are there any reqs from ericcson?
14:37:51 <daemontool> re: backup modes abstraction, make it pluggable to easily add new backup modes
14:37:54 <daemontool> ddieterly, not yet
14:38:13 <daemontool> I've requested it to the product owner
14:38:44 <m3m0_> for metering and Storage Quota management we should rely on other components
14:39:55 <daemontool> ddieterly,  do you mind create a new page on https://wiki.openstack.org/wiki/Freezer and add those?
14:40:00 <daemontool> I can do it if you are busy
14:40:09 <daemontool> we all add ours there
14:40:19 <daemontool> and then we translate that to a roadmap
14:40:24 <ddieterly> the reqs from arun, you mean?
14:40:29 <daemontool> yes
14:40:32 <ddieterly> sure
14:40:37 <daemontool> thanks a M
14:40:45 <szaher_> for the pluggable layer should we have a different bp for db, engine, mods, storage ?
14:41:13 <m3m0_> yes, that could be better
14:41:20 <reldan> So I propose to postpond creation of common metadata for all backups untill we will have at least some requirements
14:41:34 <daemontool> szaher_, I think so
14:41:42 <daemontool> reldan,  ok
14:41:42 <reldan> The last thing we want - is support many versions of schema
14:42:18 <daemontool> but I think we can start defining the meta for tenant resources backups?
14:43:16 <daemontool> ok we'll talk about that offline
14:43:18 <daemontool> next?
14:43:24 <m3m0_> #topic Python34 jobs
14:43:25 <slashme> daemontool: I think we need the plugin layer for the tenant backup
14:43:44 <daemontool> slashme, ++
14:43:46 <reldan> daemontool: we can define the meta, but without requirements we will create v1. After receiving requirements we will have v2 and etc
14:44:19 <daemontool> so we are blocked by the pluggable mode
14:44:21 <daemontool> ok
14:44:34 <daemontool> szaher_, please write the bp asap
14:44:43 <daemontool> so we can start with the implementation
14:44:43 <m3m0_> that should be the priority before moving forwrd
14:44:44 <szaher_> Ok
14:44:59 <daemontool> bp both on launchpad and gerrit
14:45:06 <daemontool> review and acceptance in gerrit
14:45:38 <daemontool> m3m0_,  next?
14:45:40 <szaher_> that is another important thing we need to priorities things
14:45:47 <m3m0_> and please update the etherpad with those bp
14:46:08 <m3m0_> the next topic is python 34 jobs
14:46:13 <m3m0_> who is involved in this?
14:46:16 <m3m0_> reldan?
14:46:24 <daemontool> reldan, slashme and me
14:46:28 <daemontool> mainly reldan
14:46:48 <m3m0_> is there any updates? because I'm getting -1 in my patches for freezer :)
14:46:56 <reldan> Yes, I suddenly have found that we have python 34 in infra, but our code is not py3 complient at all
14:47:49 <daemontool> that's a big priority
14:48:23 <daemontool> so on kilo python34, docs and dsvm are disabled
14:48:25 <reldan> I’m going to send fix today. But I have to skip several tests
14:48:43 <daemontool> reldan,  let's see this offline
14:48:53 <daemontool> on the project chan
14:49:11 <m3m0_> so should we move with another topic?
14:49:15 <daemontool> we have 10 minutes left
14:49:17 <daemontool> I think so
14:49:45 <daemontool> we can keep talking on #openstack-freezer if we want after the meeting ends here
14:50:10 <m3m0_> ok, etherpad updated
14:50:14 <m3m0_> #topic Source code walkthrough
14:50:25 <m3m0_> who requested this topic?
14:50:41 <daemontool> this is important for new commiters
14:50:49 <daemontool> like the guys from 99Cloud
14:50:56 <daemontool> but we need to finish
14:51:10 <daemontool> we need to wait the Spring Festival
14:51:13 <m3m0_> so we need to set a time and place for this for people to join
14:51:14 <daemontool> end
14:51:30 <m3m0_> do you know the guys?
14:51:37 <daemontool> yes we can set the times for next week
14:51:41 <daemontool> on IRC
14:51:43 <daemontool> yes
14:52:00 <daemontool> EinstCracy and zhangjn
14:52:07 <m3m0_> ok, are you going to do this?
14:52:12 <daemontool> so we have to find a suitable time for ddieterly  to
14:52:17 <daemontool> we are going to do this
14:52:20 <ddieterly> :-)
14:52:22 <daemontool> for the components we know more
14:52:33 <m3m0_> no I mean, set the time with them
14:52:41 <daemontool> ah yes
14:52:51 <daemontool> let's talk about it offline on the freezer chan
14:52:56 <m3m0_> perfect :) sure
14:52:59 <m3m0_> last topic then
14:53:26 <m3m0_> sorry last -1 :P
14:53:30 <m3m0_> #topic Mitaka-3
14:53:40 <daemontool> this is needed https://review.openstack.org/#/c/260950/
14:53:55 <daemontool> in order to ublock this https://review.openstack.org/#/c/271072/
14:54:15 <daemontool> because I doubt pytest will be added to global0-requirements
14:54:25 <daemontool> so py34 fix is really needed
14:54:51 <m3m0_> ok, then that should be priority number 1
14:55:04 <daemontool> if there are no comment let's go next
14:55:06 <daemontool> ++
14:55:19 <m3m0_> #topic Midcycle meetup
14:55:38 <m3m0_> slashme do you want to propose someting here?
14:56:02 <slashme> Yes: Let's do a midcycle meetup
14:56:25 <daemontool> slashme, ++ before or after the summit?
14:56:34 <m3m0_> IRC, skype, galway?
14:56:41 <slashme> We have a few item on our list that require strong thinking before implementation
14:56:50 <daemontool> yes
14:56:51 <daemontool> ok
14:56:54 <slashme> I think IRL is better
14:57:11 <daemontool> ok for me
14:57:19 <m3m0_> when?
14:57:23 <slashme> Question is : Can daemontool and ddieterly make it to Galway
14:57:28 <daemontool> I can
14:57:31 <daemontool> ddieterly,  too
14:57:32 <daemontool> haha
14:57:38 <daemontool> HP Galway can host it
14:57:51 <slashme> That would be the idea.
14:57:54 <ddieterly> its possible
14:57:55 <daemontool> ok
14:58:04 <ddieterly> i'll check
14:58:05 <m3m0_> guys we are running out of time
14:58:09 <slashme> Then let's see the organisation offline
14:58:10 <ddieterly> when is it?
14:58:10 <daemontool> let's think about possible suitable dates and discuss moer details
14:58:19 <daemontool> on the next meeting
14:58:29 <daemontool> ddieterly,  we have to define that including you too
14:58:34 <ddieterly> ok
14:58:43 <daemontool> so this is a next weeking topic for the agenda
14:58:44 <daemontool> next
14:58:54 <m3m0_> #agree
14:59:01 <daemontool> #agreed
14:59:08 <daemontool> ok
14:59:19 <daemontool> so now we need to discuss the DR bp
14:59:33 <daemontool> I understand there are discordant feelings about it
14:59:41 <m3m0_> we can do it in #openstack-freezer channel or next week
14:59:57 <daemontool> let's do it after this meeting on freezer chan
15:00:09 <m3m0_> ok guys, time's over
15:00:16 <m3m0_> thanks all for participating
15:00:17 <m3m0_> #endmeeting