Monday, 2015-12-07

*** zhonghua is now known as zhonghua-lee06:26
*** openstackgerrit_ has joined #openstack-freezer06:51
openstackgerritKenji Yasui proposed openstack/freezer: Delete python bytecode before every test run  https://review.openstack.org/25405007:44
*** reldan has joined #openstack-freezer10:09
*** openstackgerrit_ has quit IRC10:23
*** daemontool has joined #openstack-freezer10:43
daemontoolMorning10:43
daemontoolall: we should start planning a tenant based backup and restore mechanism10:43
daemontoolhelpful and also a hot topic that everybody needs for the BaaS10:44
m3m0we've been thinking about that daemontool10:48
m3m0but so far we only backup files, we need to create a new mode that interacts more with mysql10:49
daemontoolwhy10:50
daemontoolwhy?10:50
daemontoolops twive sorry10:50
daemontools/twive/twice/10:50
m3m0in mysql we backup the whole directory10:50
daemontoolwe can execute the tenant based backup using the services api10:50
daemontooland use the same api for restore10:50
m3m0then if that's the case let's do it with the api :)10:51
daemontoolthen the service in question will write correctly the data to the db10:51
daemontoolI'm just thinking loud10:51
daemontoolbut I think our life ie way easier10:51
m3m0can we restore in another instance?10:51
daemontoolif we do through the services API10:51
daemontoolsomething like http://paste.openstack.org/show/avJHQbg5ty92baqnknfH/10:52
daemontoolthat steps10:52
daemontoolI think that is critical for Back up as a service10:52
daemontoolquestion is10:53
daemontoolwhere they'd be executed10:53
daemontoolform where10:53
m3m0we have the same question for cinder and nova backups made from the ui10:54
daemontoolfrescof: reldan  Slashme  ^^10:54
m3m0who will execute those10:54
daemontoolour approach is distributes10:54
daemontooldistributed10:54
daemontoolto scale10:54
daemontoolso always VMs should do that10:54
daemontoolnow VMs can be dedicated or not, that's user choise10:54
m3m0that's the approach we follow right now10:55
m3m0ok ok10:55
*** szaher_ has joined #openstack-freezer10:55
daemontoolbut I think the freezer-scheduler and agent should do that10:55
*** szaher has joined #openstack-freezer10:55
daemontoolafter all is a sequence of orchestrated jobs10:55
daemontooli.e. keystone user data export, cinder volumes snapshots, vms snapshots, upload to the storage media, etc10:56
daemontooleach step can be a job10:56
daemontoolthat is managed by the scheduler10:56
m3m0then following the same logic an agent should do the tenant backup as well10:56
daemontoolyes10:56
daemontoolI think so10:56
daemontoolafter all the credentials and the tenant information are located in the node where the agent is running10:56
daemontoolszaher:  ^^10:57
daemontoolwhere's Vanni?10:57
m3m0everywhere10:57
daemontooleverywhere but here :)10:57
m3m0kidding, is here :)10:57
daemontoolok if we can join irc10:57
daemontoolwe can have this discussion even now10:57
daemontoolfrescof: is there around?10:58
m3m0in fact the tenant backup was proposed long time ago :)10:58
daemontoolyep10:58
m3m0before it was cool :)10:58
daemontoolwell, we couldn't make it, anyway10:58
daemontoolbefore :)10:59
daemontoolnow many things are OK10:59
daemontoolit would be good even doing something backup all_tenants10:59
daemontoolin once10:59
*** emildi has quit IRC11:00
m3m0and a single tenant as well11:00
daemontoolreldan: what do you think?  ^^11:00
daemontoolyes11:00
daemontooladmin user in keystone can do all_tenants11:01
szaheryes daemontool :) Vanni is here around :)11:01
reldandaemontool: about backing up all tenant like you describe here http://paste.openstack.org/show/avJHQbg5ty92baqnknfH/?11:01
daemontoolsomethign liek that11:01
daemontoolis just a first thought11:01
daemontooljust to have something to discuss on11:01
daemontoolwe can use a different approach11:01
daemontoolas usual11:01
*** vannif has joined #openstack-freezer11:02
reldanHow about cindernative?11:02
reldanBecause it is actually the only way of doing incremental cinder backups?11:03
daemontoolreldan:  that can be an option11:03
daemontoolI'd say, let's first identify11:04
daemontoolwhat to backup11:04
daemontooland where to get the data11:04
daemontoolthen we can decide how and where to store11:04
daemontoolI think we should download the data and upload them to the storage media we want11:04
daemontoolso we can fully support disaster recovery11:04
daemontoolbut that's just my opinion11:04
reldanYes, but if you have a very big volume11:05
reldanlike 100GB11:05
daemontoolor 1T11:05
reldanto do a backup everyday and reupload an image11:05
daemontoolit's huge11:05
daemontoolyes11:05
reldaneven if you have no changes at all11:05
daemontoolok let's think about it11:07
daemontoollet's use all the native backup services api then11:07
daemontoolat least for the beginning11:07
daemontoolthen we can improve11:07
daemontoolbut let's think it throughtly11:08
daemontoolbefore11:08
daemontoollike couple of days11:08
daemontoolwe need to have a hangout meeting I think11:08
*** emildi has joined #openstack-freezer11:08
reldanOkey. I feel that we can have application consistent backups and volume/directory backup11:08
daemontoolalso we need to start talking with Nova and Cinder to converge the backup efforts11:08
daemontoolreldan:  ++11:09
reldanWe should have some entities, I feel that one entity is application (MySql, MongoDB … ), second is snapshot mechanism (cinder snapshot, lvm, shadow, …) third is backup mechanism (tar, rsync, …)11:10
reldanAnd there are different combinations of any three of that.11:11
reldanGuys, please review: https://review.openstack.org/#/c/253117/ https://review.openstack.org/#/c/253106/ https://review.openstack.org/#/c/247840/11:11
daemontoolreldan:  yes11:12
reldanSo if you ask to make a full backup of tenant you should have some document with description - how do you want to backup all your resources11:12
reldanDo you need an application level and what do you prefer tar or rsync, what is downtime should be ...11:13
daemontoolI think the tenant based backups are done using crash consistent  approach11:15
daemontoolat least at the beginning11:15
daemontoolI have to go now11:15
daemontoolI'll be back in 1 hour11:15
daemontoollet's talk about it11:16
daemontoolre: https://review.openstack.org/#/c/253106/11:16
reldanSure!11:16
daemontoolso now that options does not have any effect at all?11:16
daemontoolcause that was the only way to list the backups from swift11:16
daemontoolnow we have no way whatsoever to list backups objects11:16
daemontoolunless we use the swift client11:16
daemontooland the approach is different for ssh and local media11:17
daemontoolwe need also to solve that asap11:17
daemontoolI'll be back in 1 hour laterz11:17
reldandaemontool: I believe it shouldn’t be a part of freezer to list remote directories or show files in container11:17
Slashme+1 reldan11:20
SlashmeThe freezer agent should only be a backup-mashine11:20
SlashmeListing should go in the python-freezerclient11:20
Slashmedaemontool: https://review.openstack.org/#/c/251710/ why did you +2 this ? What is the point of that change ?11:21
*** szaher_ has quit IRC11:37
*** zhonghua-lee has quit IRC11:42
*** zhonghua-lee has joined #openstack-freezer11:42
*** reldan has quit IRC11:48
*** reldan has joined #openstack-freezer12:08
*** samuelBartel has joined #openstack-freezer13:09
*** zhonghua-lee has quit IRC13:14
*** zhonghua-lee has joined #openstack-freezer13:14
*** jokke__ is now known as jokke_13:25
szaherall please, we need to finalize this for me to start working on the agent as well https://etherpad.openstack.org/p/Freezer_arguments13:28
openstackgerritMemo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard  https://review.openstack.org/25418913:29
szaherall: please, review this https://review.openstack.org/#/q/status:open,n,z13:29
*** reldan has quit IRC13:37
*** samuelBartel has quit IRC13:37
m3m0could you send the correct url szaher?13:38
szahersorry for that m3m0 :) here is the correct one https://review.openstack.org/#/c/252568/13:39
m3m0we talked about this last week, you need to specify a bp or a bug for each commit13:40
szaherwe need to merge that before adding more features to the scheduler to avoid merge conflict13:40
szahertake another look, there is a blueprint for that commit13:40
openstackgerritMemo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard  https://review.openstack.org/25418913:41
*** samuelBartel has joined #openstack-freezer13:42
*** reldan has joined #openstack-freezer13:47
daemontoolreldan: Slashme  m3m0 frescof  vannif szaher  ping14:00
reldanI’m here14:00
szaherme too14:00
daemontoolok14:00
daemontoolsamuelBartel:  hi14:00
daemontoolwe need to talk about the tenant based backup and backup list14:00
Slashmedaemontool: Yup ?14:00
daemontoolwe need at least to write a bp about it14:01
daemontoolidentifying what are we going to backup14:02
daemontooland how (even multiple options)14:02
daemontooland where14:03
reldanI actually don’t quite understand the use case. So we have a user and this user have an account. And he wants to be able to restore everything in the same way? Or what?14:03
daemontoolso14:04
daemontoolthe use case is the backup as a service14:04
daemontoolwhere a tenant wants to backup all the resources (i.e. data and settings) that belongs to him/her14:05
samuelBarteldaemontool, hi14:05
daemontoolthat same resources needs to be restored14:05
daemontoolon the same cloud platform or a different one14:05
reldanOkey, how about network and public IP?14:05
reldanshould we be able to restore them as well?14:05
daemontoolsamuelBartel:  did you get the chance to see that now we are an official openstack service? :)14:06
daemontoolsamuelBartel:  thanks for everything you did so far, didn't get the chance to say thanks to you14:06
daemontoolreldan: good question14:06
daemontoolI think we should backup also the neutron data14:06
daemontoolfwaas, lbaas, vpnaas14:06
vannifyes. the BaaS is a fundamental feature IMO14:07
daemontoolit needs to be implemented also from the web ui and api of course14:08
vannifdefinitely14:08
reldanIt’s very very big task14:08
reldanWe should split it14:09
vannifthat's the main interface for normal cloud users14:09
reldanOtherwise it is untrackable from my point of view14:09
* vannif agrees with reldan14:09
szaherI think it may be a good start to use modular mode agent14:12
szahermod14:13
daemontoolI'm wondering if splitting the agent can cause confusion and/or learning curve issue to the users....14:14
daemontoolall: we need to provide a technical proposal14:29
daemontoolto the Nova Team14:29
daemontoolabout a possible approach/integration with freezer and nova instances backups / restore14:29
daemontoolreldan: are you in?14:30
daemontool:)14:30
samuelBarteldaemontool, great, I miss that information14:30
daemontoolSlashme: m3m0 vannif  frescof szaher samuelBartel  ^^14:30
samuelBartelyou have really made a great job14:31
reldanI don’t know. We need to create a proof of concept I feel. Because they have backups, but it’s not what we want I suppose14:31
samuelBartelcongrats all14:31
reldanSo we should say - Hi guys, we need this and this and it can be done that way14:31
daemontoolreldan:  yes, also we can probaly provide a better solution14:32
daemontoolto do what they do14:32
daemontoolso they currently are snapshotting and uploading the image to glance14:32
reldanYes, but they have access to the hypervisor and we have no14:32
daemontoolmy question is14:32
daemontoolwhy the data needs to pass through nova14:33
daemontoolsorry14:33
daemontoolthrough glance14:33
daemontoolto execute a backup and store the data somewhere?14:33
daemontoolreldan:  if we provide a good solution14:33
daemontoolperhaps they can use the freezer-agent14:33
daemontoolto execute the vms backup14:33
reldanWhat do you mean by nova backup? Do you mean ephimeral disk only? Or all attached volumes as well?14:34
daemontoollet's think before the vm image file14:34
samuelBarteldaemontool, I hope to have dedicated time to contribute again in the near future but I change team and I am more focused on NFV feature for the moment14:35
daemontoolbut the final vision would to backup vm image file + attached volumes + neutron networks + etc14:35
daemontoolsamuelBartel:  ah... interesting, so you are working on Neutron now14:35
daemontool?14:35
vannifI'd like to be quite general here: 1) what do we mean for backup of a VM. ephemeral disk, network state, mounted volumes. 2) different use cases: backup of a stopped vm, backup of a live VM without suspension (nova backup suspends the vm while the snapshotted image is created).14:36
daemontoolvannif: I'm referring to 114:36
vannifmoreover: data integrity. differenciate between the integrity of the image files (to be able to boot the vm again, for instance), and integrity of the data *inside* the VM, which, I think, requires some interaction with what is *inside* the VM (be it the freezer-scheduler or a qemu-agent)14:38
daemontoolvannif: the idea would be to place the freezer-agent between the nova api and vm, then yes we need to talk to the hypervisor14:39
samuelBarteldaemontool, not directly, the use case is more to work on vnf manager for nfv such as vCPE, vCDN and so on14:39
daemontoolsamuelBartel:  ok14:39
daemontoolso we probably need to implement an additional backup method on Nova14:40
daemontoolwhere the use can use the native mode14:41
daemontoolor freezer14:41
daemontoolif freezer is used14:41
daemontoolthe agent will talk with the hypervisor14:41
daemontooland so on14:41
daemontoolsomething like, I'd like to know what frescof think about this14:41
daemontoolor also another option14:46
daemontoolwould be to execute the backups on glance14:46
daemontoolwe need to elaborate an approach14:47
daemontooland get in the backup/restore process of nova vms and cinder volumes14:48
daemontoolour agent can definetely provide better features for backup and restore14:48
daemontoolthan the current ones14:48
reldanI feel that we are trying to solve very different tasks. One of them - creation backups of single machine from the inside. The second - OpenStack resources backup.14:50
reldanAnd it is really different tasks14:50
daemontoolreldan:  yes, we are already doing that14:50
daemontoolboth14:50
reldanI am not sure that it should be mixed14:51
daemontoolwhere it is mixed?14:51
daemontoolthe code do you mean?14:51
daemontoolthe code can be splitted, that's not an issue14:51
reldanYou see, we have gtar backup and rsync backup for a directory. It can be used without OS at all. And it has sense.14:52
daemontoolyes14:52
reldanFrom other case we would like to do a backup of complex structure - like several VM, several Volumes, Network ...14:52
daemontoolafter all, everything is a file14:53
daemontoolif you think about it14:53
daemontoolfor kvm14:53
daemontoolthe hypervisor contains everything14:53
daemontoolall the necessary information14:53
daemontoolmy question14:53
daemontoolhow would be easier our life if we can execute actions directly on the hypervisor?14:54
daemontoolor at least we would get quite significant flexibility14:54
reldanYes and no. Let’s imagine that I have an online shop. I have 2 VM with mysql with master-slave replication and cinder volumes. 2 VM with nginx with frontend application on php, connected with mysql instances. And I would like to be able to make a backup, then create the same vm’s and volumes (let’s say for testing purposes)14:55
reldanSo I need at some point to create a backup at the same time on all nodes14:56
reldanI need syncronization - and it is quite different from backuping a file on one disk14:56
reldanPlus I would like to restore network14:56
reldanAnd it is much more difficult14:57
reldan1) I need a way do describe my environement (vm’s, volumes, networks, connection between each of them). 2) I need a way to recreate my environement 3) I need to restore data, attach instance back, fix IP for master/slave replication14:59
reldanAnd it’s quite different from gtar backup - now it contains a lot of application specific, toplology specific information15:00
daemontoolthe difference would be15:06
daemontoolon making the snapshots at the same time15:07
daemontooland the possible interactions with the hypervisor15:07
openstackgerritEldar Nugaev proposed openstack/freezer: Removing obsolete parameters from freezer args (swift related).  https://review.openstack.org/25310615:15
daemontoolvannif: do we have currently a way to show the backups from the API?15:15
daemontoolthe list of taken backups15:15
openstackgerritEldar Nugaev proposed openstack/freezer: Removing old integration tests  https://review.openstack.org/25311715:16
daemontoolreldan:  I'd split the problem15:20
daemontoolfor now, I'd focus on the best approach15:20
daemontoolto backup the vms15:21
vannifno. the backup metadata is in the db but the freezer-scheduler does not have any browse/search tool in place to retrieve that information15:23
reldandaemontool: Ephemeral or volume based?15:23
reldanProbably we shouldn’t care about ephemeral15:24
*** zhonghua-lee has quit IRC15:25
*** zhonghua-lee has joined #openstack-freezer15:26
daemontoolreldan: ephemeral may contains15:29
daemontoolimportant informations15:29
daemontoollike users15:29
daemontoolsettings15:30
daemontoolconfig15:30
reldanBut ephemeral disk can be erased even after rebooting15:30
reldanSo you should keep only caches and nonsensetive information there15:31
reldanData on ephemeral storage ceases to exist when the instance it is associated with is terminated. Rebooting the VM or restarting the host server, however, will not destroy ephemeral data. In the typical use case an instance's root filesystem is stored on ephemeral storage. This is often an unpleasant surprise for people unfamiliar with the cloud model of computing.15:31
reldanOh, I see it is different in OS and AWS15:31
daemontoolso the challenge is, how to backup all the rsources belonging to one tenant15:32
daemontooland if there's anything that can facilitate our job by doing actions on the comput node or glance or anything else15:42
daemontoolall:  had a quick conversation with jokke_  and the Nova PTL15:43
daemontoolit turns out most probably the easies way to achieve this for nova vm is to leverage glance15:44
daemontoolso like the approach we are using currently15:44
daemontoolreldan:  do you remember if the cinder compress the volumes when they are snapshotted and converted to an image?15:49
reldanYes, but how we can have incremental backup in this case?15:49
daemontoolnot on glance15:50
daemontoolonly on the media storage after swift processing15:50
daemontooland only after the block based incremental15:50
jokke_reldan: snapshot is incremental already15:50
daemontoolis implemented on freezer15:50
jokke_iirc15:51
reldanSnapshots are - yes, but we cannot sotre it on swift15:51
reldanstore15:51
jokke_reldan: so when you backup that glance image, which is result of snapshot that is automatically incremental backup15:51
jokke_you don't need to do the incremental part yourself as the snapshotting took care of it already15:52
daemontooljokke_:  that's for cinder or nova?15:52
reldanFor cinder snapshot is just a delta and cannot be used without a volume15:53
reldanFor nova volume based - same15:53
jokke_reldan: initial snapshot can be used to create the initial volume and then multiple snapshots can be built on top of that ... same applies to the nova snapshots that are not volume based (well they have the base image always)15:55
reldanSure, we can use snapshots and it will be incremental. But the current approach was - to creation a snapshot, creation volume, volume -> glance image, glance image -> download, upload to swift or local storage or ...15:58
reldanSure we can use snapshots and backups in cinder and nova without freezer15:58
*** dschroeder has joined #openstack-freezer15:59
*** daemontool has quit IRC16:06
openstackgerritMemo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard  https://review.openstack.org/25418916:28
openstackgerritMemo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard  https://review.openstack.org/25418917:04
*** samuelBartel has quit IRC17:31
*** reldan has quit IRC18:01
*** szaher has quit IRC18:35
*** vannif_ has joined #openstack-freezer19:16
*** vannif has quit IRC19:16
*** reldan has joined #openstack-freezer20:40
*** reldan has quit IRC21:24

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!