14:01:50 <m3m0> #startmeeting freezer
14:01:51 <openstack> Meeting started Thu Mar 31 14:01:50 2016 UTC and is due to finish in 60 minutes.  The chair is m3m0. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:55 <openstack> The meeting name has been set to 'freezer'
14:02:00 <reldan> o/
14:02:10 <m3m0> as always, topics are available here: https://etherpad.openstack.org/p/freezer_meetings
14:02:26 <m3m0> who's here for the freezer meeting? hands in the air :)
14:02:28 <m3m0> o/
14:02:32 <ddieterly> o/
14:02:55 <m3m0> let's wait for more ppl to join
14:03:21 <slashme> Hi
14:03:30 <slashme> o/
14:04:07 <zhangjn> PTL
14:04:11 <zhangjn> come on
14:05:09 <m3m0> guys before start the meeting, feel free to modify the agenda and add or edit topics
14:05:45 <m3m0> #topic new PTL
14:06:10 <m3m0> well the first thing is to congratulate slashme in his new role
14:06:34 <ddieterly> slashme congrats!
14:06:35 <m3m0> under his ruling he will provide food and water to freezer developers
14:06:47 <slashme> ;)
14:06:49 <szaher> slashme: Congratulations :) :)
14:07:33 <zhangjn> slashme: congrats!
14:07:39 <zhangjn> :):)
14:07:47 <m3m0> can we move forward?
14:07:54 <yangyapeng> get_backup_args
14:08:04 <yangyapeng> Congratulations
14:08:46 <daemontool> o/
14:08:54 <zhangjn> Make money
14:09:31 <zhangjn> freedom
14:10:15 <slashme> As said in my candidacy, my goal for freezer during the newton release cycle will be to improve Freezer stability, user experience, documentation and interoperability. I really look forward to work with all of you and I think we couldn't have a better team to work on Freezer. :)
14:10:16 <vannif> hi
14:10:53 <m3m0> wiser words haven't been spoken
14:10:55 <slashme> I'm also very happy to see more and more people joining the dev team, this is a very good sign.
14:11:25 <m3m0> next topic
14:11:38 <m3m0> #topic integration tests
14:11:52 <m3m0> ddieterly: would you like to comment on this?
14:12:05 <ddieterly> sure
14:12:30 <ddieterly> the freezer repo and the freezer-api repo both have a gate that runs tempest tests
14:12:59 <ddieterly> this review has good examples of how to write tempest tests https://review.openstack.org/#/c/297360/
14:13:19 <ddieterly> so, the infrastructure is all there and there are examples for people to follow
14:13:33 <ddieterly> there should be no excuses for not writing tests now ;-)
14:14:03 <slashme> How are we going to manage integration testing for the freezer-agent? I see a lot of example in other project on how to test api's, but how will we test an "executable" ?
14:14:29 <daemontool> we could test the api with the agent
14:14:33 <daemontool> and the scheduler
14:14:34 <ddieterly> you can write the tests without using the tempest rest apis
14:14:52 <slashme> bash/python ?
14:14:55 <ddieterly> just write code under the tempest plugin that follows the examples
14:15:22 <ddieterly> the import thing is that there is a gate that will invoke the tests
14:15:28 <daemontool> slashme, yes similarly on how the jobs are crated in hlm
14:15:29 <ddieterly> you can write any code you like
14:15:39 <slashme> So tempest would still be executing them. Seems perfect then.
14:15:49 <ddieterly> yes
14:16:08 <ddieterly> you can also invoke any other code using the post/pre test hooks in the gate
14:16:52 <ddieterly> that code can be outside the tempest tests dir
14:17:31 <ddieterly> you all now have the power :-)
14:17:49 <daemontool> thanks for that ddieterly
14:18:01 <ddieterly> daemontool my pleasure
14:18:50 <m3m0> ddieterly thanks a lot :)
14:19:29 <daemontool> next?
14:19:41 <m3m0> sure
14:19:53 <m3m0> #topic mitaka branch for ui
14:20:03 <m3m0> slashme any input on this?
14:20:10 <m3m0> daemontool
14:20:18 <daemontool> well we have to do it :)
14:20:21 <daemontool> next week?
14:20:30 <m3m0> sure, that could be great
14:20:38 <m3m0> so we can start the newton development
14:20:40 <slashme> Mar 28-1 	R-1 	Final RCs and intermediary releases 	TC member self-nomination
14:20:40 <slashme> Apr 4-8 	R-0 	Mitaka release 	TC member election
14:21:16 <daemontool> we can work on newton even now
14:21:23 <daemontool> master actually is newton
14:21:29 <daemontool> stable/mitaka is mitaka
14:21:56 <m3m0> but the ui needs that branch and then we can move to newton
14:22:28 <m3m0> we can do that next week
14:22:45 <m3m0> next topic?
14:23:29 <m3m0> #topic pending reviews
14:23:36 <m3m0> guys please review this:
14:23:40 <m3m0> https://review.openstack.org/#/c/297119/
14:23:40 <m3m0> https://review.openstack.org/#/c/295220/
14:23:40 <m3m0> https://review.openstack.org/278407
14:23:40 <m3m0> https://review.openstack.org/280811
14:23:41 <m3m0> https://review.openstack.org/291757
14:23:41 <m3m0> https://review.openstack.org/296436
14:23:44 <daemontool> ok
14:23:52 <daemontool> I think we need to change something
14:23:55 <daemontool> in the gate job
14:24:18 <daemontool> we need to differentiate for job execute on master
14:24:23 <daemontool> and previous releases
14:24:28 <daemontool> I'm not sure how to do this
14:24:38 <daemontool> we need to have a conversation with infra
14:24:41 <daemontool> in the infra channel
14:24:45 <daemontool> to understand
14:25:50 <m3m0> agree on that, do you want to have that conversation with them?
14:27:24 <daemontool> I can do it, it's not super urgent
14:27:29 <daemontool> but sooner the better
14:27:56 <daemontool> there's something about the incremental engine
14:28:05 <daemontool> I'd like to get some feedback for
14:28:16 <daemontool> (if we are ok with the previous topic)
14:28:48 <daemontool> tar recognize the data backup file without the need of the metadata
14:29:00 <m3m0> #topic tar metadata
14:29:11 <daemontool> now, how can freezer recognize which engine was used
14:29:17 <daemontool> for the backup
14:29:23 <daemontool> options are
14:29:33 <daemontool> 1) read that in the metadata
14:29:48 <daemontool> 2) the user has to set --incremental-engine or --engine
14:29:50 <daemontool> then restoring
14:29:59 <daemontool> 3) add few bytes in the rsync data blob
14:30:11 <daemontool> like freezer_rsync_v0.1
14:30:25 <daemontool> 4) change the current object/directory structure name
14:30:33 <daemontool> prepending tar or rsync at the beginning
14:30:38 <daemontool> like rsync_hostname
14:30:42 <daemontool> and rsync_metadata
14:30:44 <daemontool> same for tar
14:30:51 <daemontool> any thought?
14:31:33 <m3m0> if we are going to store the metadata in the storage as well could be a great place to write in the metadata
14:32:02 <m3m0> because for backups made with the agent we don't have support from the api
14:32:26 <daemontool> ok
14:32:33 <daemontool> each option has pro and cons
14:32:33 <reldan> I would prefer naming in sub-folders or backup name
14:32:42 <daemontool> reldan,  so #4 ?
14:32:55 <reldan> yes, and I don’t see any drawback for this approach
14:33:12 <slashme> Well, non-backward compatible
14:33:14 <daemontool> that was the one I thought at the beginning
14:33:30 <daemontool> the only one that is backward compatible
14:33:33 <reldan> slashme: any approach is not backward compatible
14:33:35 <daemontool> is #3 I think
14:33:46 <slashme> Yes but it is very bad
14:33:51 <reldan> :(
14:33:57 <daemontool> tar do that for the metadata file
14:33:57 <reldan> It is sad, yes
14:34:02 <slashme> You need to get the object first to have the metadata
14:34:05 <daemontool> is you open the metadata
14:34:11 <daemontool> at the very beginning
14:34:22 <daemontool> there-s GNU TARv1.27 something
14:34:34 <daemontool> but it's also
14:34:36 <daemontool> complicated
14:34:42 <daemontool> to write
14:35:23 <daemontool> cause you have to place the parsing after the data went out of the pipe
14:35:29 <slashme> I also think #4 is the best option.
14:35:38 <daemontool> so #4?
14:35:52 <daemontool> it's important, cause we need to modify also the remove part
14:36:16 <daemontool> ok
14:36:58 <slashme> And if using #4, do we go for : engine_backupname_hostname_timestamp_level or engine/backupname_hostname_timestamp_level
14:37:32 <daemontool> or engine/hostname/backupname/timestamp/level ?
14:38:06 <daemontool> it would be different code for swift
14:38:29 <daemontool> reldan,  you had an idea about this if I remember well?
14:40:01 <reldan> I would prefer sub-folders sub-containers
14:40:07 <reldan> some logical tree from storage
14:41:57 <daemontool> let's do this
14:42:00 <daemontool> let's finish rsync
14:42:04 <daemontool> basically working
14:42:16 <daemontool> and then asap change the object/file format
14:42:39 <daemontool> ?
14:42:48 <daemontool> thoughts?
14:43:12 <daemontool> let me know in case
14:43:35 <daemontool> one more thing, it's important to generate metadata
14:43:41 <daemontool> for the current cinder and nova backup modes
14:43:49 <daemontool> wheter they are native or not
14:43:53 <daemontool> we need to save some metadata
14:45:09 <daemontool> the baas needs to be improved
14:46:00 <m3m0> vannif: any other input for the backup metadata?
14:46:42 <vannif> i don't like too much the idea of using paths to store information
14:47:56 <reldan> vannif: any other idea. if only you have is filesystem
14:48:14 <daemontool> a path is somewhat
14:48:18 <daemontool> already there
14:48:32 <daemontool> but I'm not sure we can use a different approach with swift
14:48:44 <daemontool> unless we use SLO rather DSO
14:48:55 <daemontool> that's a whole new discussion anyway
14:48:58 <daemontool> :)
14:49:12 <daemontool> slashme, we really need to add some metadata
14:49:24 <daemontool> to the backup done on VMs and Volumes
14:49:28 <daemontool> reldan,  ^^
14:49:32 <daemontool> very important
14:50:00 <m3m0> guys we have 10 min left
14:50:04 <daemontool> ok
14:50:09 <reldan> two ways - have something in database, in tree or hashmap
14:50:12 <reldan> second - path
14:50:20 <reldan> first - database, index
14:50:25 <slashme> You mean backup metadata (stored in the api) or engine metadata (stored in a storage media)
14:50:41 <daemontool> backup metadata
14:50:57 <daemontool> that will be displayed by the python-freezerclient
14:50:59 <daemontool> and the web ui
14:51:42 <slashme> Okay, What should we add ?
14:52:10 <daemontool> if now we execute a backup of a cindervolume
14:52:12 <daemontool> with cinder native
14:52:19 <daemontool> it's not recorded anywhere
14:52:22 <daemontool> afaik
14:52:24 <slashme> What do we store right now ? Just the volume/instance id ?
14:52:30 <daemontool> I think nothing
14:52:39 <daemontool> we need that
14:52:43 <daemontool> volume/instance id, timestamp
14:53:00 <reldan> Yes, we need common approach
14:53:03 <daemontool> that basic info at least
14:53:06 <reldan> for all types of backup
14:53:10 <daemontool> yes
14:53:18 <daemontool> it starts to be critical now
14:54:45 <daemontool> slashme,  probably we should define the critical things were we should focus on?
14:55:07 <slashme> For newton:
14:55:36 <slashme> 1st: freezer-agent plugins layers
14:56:16 <daemontool> ok
14:56:44 <m3m0> guys 3 min left
14:57:00 <daemontool> anyway let's define that
14:57:02 <slashme> Integration tests. We can't conitue like this where every time we try a feature, it is broken
14:57:08 <daemontool> ok
14:57:39 <slashme> I'd say these are our top priority. Let's define the rest in an etherpad
14:57:45 <daemontool> ok
14:57:48 <daemontool> baas
14:58:03 <daemontool> so many users ask me for baas and dr
14:58:15 <daemontool> baas top
14:58:50 <m3m0> first we need real multitenancy
14:58:59 <slashme> Then our main blocker to having baas is oslo policy integrtation
14:59:03 <m3m0> guys let's move to #openstack-freezer channel
14:59:07 <daemontool> ok
14:59:09 <daemontool> slagle,  ok
14:59:13 <m3m0> #endmeeting