| *** dschroeder has quit IRC | 00:30 | |
| *** reldan has quit IRC | 00:45 | |
| *** yangyapeng has joined #openstack-freezer | 01:05 | |
| *** EinstCrazy has joined #openstack-freezer | 01:09 | |
| *** yangyapeng has quit IRC | 01:43 | |
| EinstCrazy | anyone can review https://review.openstack.org/#/c/264654/ for me? thanks | 01:52 |
|---|---|---|
| *** c00281451_ has joined #openstack-freezer | 05:13 | |
| *** c00281451__ has quit IRC | 05:17 | |
| *** yangyapeng has joined #openstack-freezer | 05:54 | |
| *** Slashme_ has joined #openstack-freezer | 08:24 | |
| *** Slashme_ has quit IRC | 08:32 | |
| *** zhangjn has quit IRC | 08:43 | |
| *** zhangjn has joined #openstack-freezer | 08:48 | |
| *** daemontool_ has joined #openstack-freezer | 09:08 | |
| *** daemontool_ is now known as daemontool | 09:08 | |
| daemontool | EinstCrazy, re: https://review.openstack.org/#/c/264654/3 done | 09:09 |
| EinstCrazy | Thanks | 09:09 |
| daemontool | someone else needs to give +A still | 09:09 |
| *** daemontool__ has quit IRC | 09:11 | |
| daemontool | all, can anyone review this change please https://review.openstack.org/#/c/264654/3 | 09:13 |
| *** reldan has joined #openstack-freezer | 09:14 | |
| *** c00281451_ has quit IRC | 09:18 | |
| *** c00281451_ has joined #openstack-freezer | 09:18 | |
| daemontool | slashme, very important | 09:22 |
| daemontool | we need to provide an api endpoint | 09:22 |
| daemontool | that allows to do upgrades | 09:22 |
| daemontool | m3m0, if the python-freezerclient code ready to be published to pypi? | 09:36 |
| daemontool | I can do it, as long as we are sure it works | 09:36 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Refactoring of mysql, msserver and mongo code. https://review.openstack.org/280602 | 09:45 |
| daemontool | reldan, ping | 09:56 |
| reldan | daemontool: yes | 09:56 |
| daemontool | all, so finally I'm restart working on the rsync stuff | 09:57 |
| daemontool | wanted to discuss some details | 09:57 |
| reldan | daemontool: yes, sure | 09:57 |
| daemontool | how do we differenciate the incremental_type ? | 09:57 |
| daemontool | with incremental_type? :) | 09:57 |
| daemontool | so incremental_type>: {'tar', 'rsync', n} | 09:57 |
| daemontool | ? | 09:57 |
| reldan | daemontool: currently we have only incremental_type, and only tar | 09:58 |
| daemontool | yes | 09:58 |
| daemontool | exactly | 09:58 |
| daemontool | so we need to make that extendable&pluggable now | 09:58 |
| reldan | but rsync is incremental as well | 09:59 |
| daemontool | yes | 09:59 |
| reldan | for non_incremental we can have level === 0 | 09:59 |
| daemontool | so --incremental-type rsync | 09:59 |
| daemontool | --incremental-type tar? | 09:59 |
| daemontool | same for config | 09:59 |
| daemontool | incremental_type | 09:59 |
| daemontool | ? | 09:59 |
| daemontool | we need to provide an interface to the user where the user can decide if the incremental is block based or file based | 10:00 |
| daemontool | (inode change based) | 10:00 |
| reldan | daemontoll: I don’t know. Incremental-type sounds not very clear, but I have no other options | 10:00 |
| reldan | backup-engine | 10:01 |
| daemontool | incremental mode? | 10:01 |
| reldan | but i don’t know | 10:01 |
| reldan | daemontool: I have doubts about word incremental. | 10:01 |
| daemontool | slashme, frescof yangyapeng EinstCrazy m3m0 sc zhangjn any thought? | 10:01 |
| daemontool | ok | 10:01 |
| slashme | Engine | 10:02 |
| reldan | daemontool: because we can imagine some non-incremental mode | 10:02 |
| slashme | https://etherpad.openstack.org/p/freezer_new_archi | 10:02 |
| daemontool | insn't engine too general? | 10:02 |
| daemontool | incremental_engine ? | 10:02 |
| slashme | We can have non-incremental, non-compressive, non-encrypting engine | 10:03 |
| EinstCrazy | incremental is supported in the cinder backup now | 10:03 |
| slashme | The engine takes data and apply whatever operations it wants to to it | 10:03 |
| slashme | and sends it the the storage | 10:03 |
| daemontool | slashme, ok, my point is, that we need to provide to the user an easy way to say | 10:03 |
| daemontool | I want to do backup using tar | 10:03 |
| daemontool | or I want to do backup using rsync block changes | 10:04 |
| daemontool | so if you have to use freezer as a user | 10:04 |
| daemontool | and you need to define that | 10:04 |
| slashme | engine=tar / engine=rsync | 10:04 |
| daemontool | what option would you like to have | 10:04 |
| daemontool | ok | 10:04 |
| slashme | engine=noop | 10:04 |
| daemontool | ok, what do you think if we prepend a prefix to the engine word | 10:05 |
| slashme | By noop I mean no operation on the data. They would be sent as is | 10:05 |
| daemontool | so if we have more engines | 10:05 |
| daemontool | we can differentiate? | 10:05 |
| daemontool | ok | 10:05 |
| daemontool | or engine only is ok_ | 10:05 |
| daemontool | ? | 10:05 |
| slashme | Please, check the etherpad I just sent. We already made a big amount of thinking about this | 10:05 |
| slashme | We can worry about the naming convention later | 10:06 |
| daemontool | ok | 10:06 |
| daemontool | so engine | 10:06 |
| daemontool | now | 10:06 |
| daemontool | next thing | 10:06 |
| slashme | Or data_engine | 10:06 |
| daemontool | we need a way to distinguish | 10:06 |
| daemontool | the metadata between tar and rsynx | 10:06 |
| slashme | and we would have snapshot_engine, storage_engine, ... | 10:06 |
| daemontool | rsync | 10:06 |
| daemontool | so now we prepend the metadata as prefix of the file | 10:07 |
| daemontool | I think we can prepend rsync_ or tar_ | 10:07 |
| daemontool | to identify metadata files generated by tar or rsync | 10:08 |
| EinstCrazy | I think the mode to be a meaningful name? such as backup_mode? | 10:09 |
| daemontool | EinstCrazy, like adding the mode to the backup file names? | 10:10 |
| slashme | It needs to work for backup AND restore. | 10:11 |
| daemontool | probably a good idea, but we still need to define | 10:11 |
| daemontool | how to identify | 10:11 |
| daemontool | the metadata generated with tar or rsync | 10:11 |
| daemontool | I was thinking about prepending | 10:11 |
| daemontool | tar_meta or rsync_meta | 10:12 |
| daemontool | to the metadata file | 10:12 |
| daemontool | or we can have | 10:12 |
| yangyapeng | This is easy to read | 10:12 |
| daemontool | the same file | 10:12 |
| daemontool | and parse the content | 10:12 |
| daemontool | but for tar | 10:12 |
| daemontool | and also for rsync | 10:12 |
| daemontool | the content is binary | 10:12 |
| daemontool | I'd go for engine prefix | 10:13 |
| daemontool | we can change it later | 10:13 |
| daemontool | just need this to move forward | 10:13 |
| daemontool | and start with | 10:13 |
| daemontool | (by later I mean before merging the code :) ) | 10:13 |
| daemontool | ok so | 10:14 |
| daemontool | engine=(tar,rsync) | 10:15 |
| daemontool | and (tar_meta|rsync_meta) | 10:15 |
| daemontool | ? | 10:15 |
| daemontool | I'm starting with that | 10:15 |
| daemontool | it is also important to delete/retain backups | 10:15 |
| daemontool | slashme, one thing we should implement is a better way to retain our old backups | 10:16 |
| daemontool | now we remove backup oldr than x | 10:16 |
| daemontool | or older of certain date | 10:16 |
| daemontool | on top of that we should probably also provide a feature like | 10:16 |
| daemontool | retain 1 backup for the last 12 months, even if older than date or n days | 10:16 |
| slashme | Makes sense | 10:16 |
| daemontool | do you have bandwidth to do it? | 10:17 |
| daemontool | or are you interested? | 10:17 |
| daemontool | or anyone else is interested? | 10:17 |
| daemontool | like retention_engine | 10:18 |
| daemontool | :) | 10:18 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Refactoring of mysql, msserver and mongo code. https://review.openstack.org/280602 | 10:23 |
| slashme | daemontool: I don't see how this could be decorelated from the data_engine. | 10:37 |
| daemontool | ok | 10:42 |
| daemontool | let's put it in the data_engine then | 10:43 |
| daemontool | the only think | 10:43 |
| daemontool | we need to have same code to manage delete and retention for backups stored on the storage media | 10:43 |
| daemontool | and also for backup taken using cinder and nova api | 10:43 |
| daemontool | reldan, all this is the modified pyrsync code http://paste.openstack.org/show/488924/ | 10:44 |
| reldan | daemontool: Good! | 10:45 |
| daemontool | that is way faster and memory efficient than https://github.com/isislovecruft/pyrsync/blob/master/pyrsync.py | 10:45 |
| daemontool | working on that now | 10:45 |
| daemontool | after the first implementation | 10:45 |
| daemontool | if we can manage to send blocks through pipes | 10:46 |
| daemontool | rather yielding | 10:46 |
| daemontool | and have multiprocess/threads | 10:46 |
| daemontool | I'm quite sure it will be way faster | 10:46 |
| daemontool | the only slow part will be the math computation for rolling checksum | 10:46 |
| daemontool | but that can be improved with pypy a lot | 10:46 |
| reldan | daemontool: Great | 10:47 |
| reldan | If you need any help with engine - I can help | 10:47 |
| reldan | guys, can you give me +workflow please here https://review.openstack.org/#/c/286592/ | 10:47 |
| daemontool | let me put together the first code | 10:47 |
| daemontool | and we'll review it together | 10:47 |
| daemontool | what I think is high priority from your side | 10:48 |
| daemontool | is the vm incremental backups (I know we need the design and arch details : ) ) | 10:48 |
| daemontool | and the volumes incrementals | 10:48 |
| daemontool | too | 10:48 |
| reldan | daemontool: good plan! | 10:48 |
| daemontool | if we can work in parallel on, rsync incrementals, | 10:48 |
| daemontool | retention | 10:48 |
| daemontool | volumes and instances incrementals | 10:49 |
| daemontool | and dsvm tests | 10:49 |
| daemontool | in short time we'll be in a way better position | 10:49 |
| daemontool | reldan, if you try that pyrsync code when you have time | 10:50 |
| daemontool | you'll see | 10:50 |
| daemontool | that you have a binary file | 10:50 |
| daemontool | than you edit that file | 10:50 |
| daemontool | add few random code in the middle | 10:50 |
| daemontool | and the alg only will yield that block | 10:50 |
| daemontool | there are some limitations tho | 10:50 |
| daemontool | but it works | 10:50 |
| reldan | daemontool: I remember that code actually. It was not very fast, but worked | 10:51 |
| daemontool | reldan, yes | 10:51 |
| daemontool | I've tried that with pypy | 10:51 |
| daemontool | and is a lot faster | 10:51 |
| daemontool | the math part | 10:51 |
| daemontool | this https://github.com/isislovecruft/pyrsync/blob/master/pyrsync.py#L162 and this https://github.com/isislovecruft/pyrsync/blob/master/pyrsync.py#L173 | 10:52 |
| daemontool | functions | 10:52 |
| daemontool | are the slow part | 10:52 |
| *** daemontool_ has joined #openstack-freezer | 11:15 | |
| openstackgerrit | Merged openstack/freezer: lvm snapshot unique name https://review.openstack.org/286705 | 11:15 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Refactoring of mysql, msserver and mongo code. https://review.openstack.org/280602 | 11:15 |
| daemontool_ | pycharm5 rocks | 11:16 |
| *** daemontool has quit IRC | 11:17 | |
| *** EinstCrazy has quit IRC | 11:24 | |
| *** yangyapeng has quit IRC | 11:30 | |
| *** daemontool_ has quit IRC | 11:39 | |
| *** daemontool_ has joined #openstack-freezer | 11:39 | |
| openstackgerrit | Merged openstack/freezer: Fix metadata curr_level https://review.openstack.org/286592 | 11:41 |
| *** reldan has quit IRC | 11:43 | |
| *** reldan has joined #openstack-freezer | 12:05 | |
| jith_ | hi all, I tried to use freezer for local backup. I used | 12:16 |
| jith_ | sudo freezerc --path-to-backup original/sample.txt --backup-name backup --storage local --container freezer_backups --action backup | 12:16 |
| jith_ | I found | 12:16 |
| jith_ | 1)No file is created in the given container path(freezer_backups), but a folder named freezer_backups is created in –path-to-backup folder(original) and it contains a file with bkp extension | 12:16 |
| jith_ | 2) when I tried to restore it with verbose option I got the following error” | 12:16 |
| jith_ | [*] Critical Error: [*] No matching backup name "hostname_backup" found | 12:16 |
| jith_ | I am using the stable/kilo version | 12:17 |
| *** EmilDi has joined #openstack-freezer | 12:35 | |
| reldan | jith_: Hi | 12:37 |
| reldan | jith_: I see. Yes, it is bug. But it is expected bug | 12:38 |
| reldan | jith_: For —container for storage local you should use an absolute path to the directory where you want to store your backups | 12:38 |
| reldan | jith_: You have provided relative path and it was created from current directory (yes, it is wrong and should be fixed) | 12:39 |
| reldan | jith_: so please provide absolute path | 12:40 |
| reldan | daemontool_: Do you know what is wrong with py34? https://review.openstack.org/#/c/280602/ | 12:58 |
| daemontool_ | reldan, looking | 13:01 |
| reldan | daemontool_: actually now it seems that this path have a merge conflict problem | 13:02 |
| reldan | but it looks like some problem with github or gerrrit | 13:02 |
| daemontool_ | reldan, try to rebase the change | 13:02 |
| daemontool_ | there are some conflicts to fix | 13:02 |
| reldan | yes, resolving conflicts now | 13:02 |
| daemontool_ | ok | 13:03 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Refactoring of mysql, msserver and mongo code. https://review.openstack.org/280602 | 13:06 |
| *** yangyapeng has joined #openstack-freezer | 13:19 | |
| *** yangyapeng has quit IRC | 13:22 | |
| reldan | Guys, please review it https://review.openstack.org/#/c/280602/ | 13:43 |
| daemontool_ | reldan, mod shouldn'd be called mode? | 13:45 |
| reldan | daemontool_: I don’t know. I had a review from Saad: “Can we rename Hook to Mod ? Can we use six and abc to abstract this class, just to enforce the Mod developers to follow the design guide lines and implements these functions “ | 13:46 |
| daemontool_ | ok | 13:46 |
| daemontool_ | but if we are referring to modes | 13:46 |
| reldan | daemontool_: But if you suppose it should be mode | 13:46 |
| daemontool_ | the correct world in english is mode | 13:46 |
| reldan | daemontool_: Sure, let me fix it | 13:47 |
| daemontool_ | s/world/word/ | 13:47 |
| daemontool_ | is it that? | 13:47 |
| daemontool_ | --mode right? | 13:47 |
| *** yangyapeng has joined #openstack-freezer | 13:48 | |
| reldan | yes | 13:49 |
| slashme | +1 let's use mode | 13:50 |
| *** EmilDi has quit IRC | 13:50 | |
| reldan | done | 13:50 |
| reldan | daemontool_ , slashme: done | 13:51 |
| daemontool_ | ok | 13:51 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Refactoring of mysql, msserver and mongo code. https://review.openstack.org/280602 | 13:52 |
| daemontool_ | m3m0, can you test if https://review.openstack.org/#/c/280602/11 work as expected on windows? | 13:57 |
| daemontool_ | it should, just to double check | 13:57 |
| m3m0 | let me check daemontool | 13:59 |
| *** daemontool_ is now known as daemontool | 13:59 | |
| m3m0 | BTW I have vagrant images for windows | 13:59 |
| m3m0 | windows 7 8.1 and 10 | 13:59 |
| m3m0 | to test freezer | 13:59 |
| daemontool | brilliant | 13:59 |
| daemontool | we should have that test to execute automatically with dsvm | 13:59 |
| daemontool | getting the images from a github repo | 14:00 |
| daemontool | like the one we are using for the binaries | 14:00 |
| m3m0 | I'm not so sure if we can upload those images to github | 14:00 |
| daemontool | ok | 14:00 |
| m3m0 | but let me check and on top of that is 4.5gb each | 14:00 |
| daemontool | what's the size_ | 14:00 |
| daemontool | ? | 14:00 |
| m3m0 | :( | 14:00 |
| daemontool | ah ok | 14:00 |
| daemontool | mmhhh | 14:00 |
| daemontool | :( | 14:00 |
| m3m0 | just give me a few min, I'm loading again the vms | 14:03 |
| daemontool | ok | 14:04 |
| daemontool | reldan, that is a good change... | 14:06 |
| daemontool | (thumbsup) | 14:06 |
| *** EinstCrazy has joined #openstack-freezer | 14:06 | |
| reldan | daemontool: Thank you! | 14:06 |
| reldan | daemontool: next step is extract lvm and shadow to classes | 14:07 |
| daemontool | that's anothe rgood one | 14:17 |
| *** EmilDi has joined #openstack-freezer | 14:28 | |
| reldan | Guys, please review https://review.openstack.org/#/c/280602/ Jenkins has +! | 14:31 |
| reldan | +1 | 14:31 |
| daemontool | reldan, yes, let's have m3m0 doing the tests on windows there | 14:32 |
| daemontool | EmilDi, hi | 14:32 |
| reldan | sure, thank you! | 14:32 |
| EmilDi | daemontool: hi | 14:33 |
| *** yangyapeng has quit IRC | 14:51 | |
| *** yangyapeng has joined #openstack-freezer | 14:52 | |
| *** yangyape_ has joined #openstack-freezer | 14:56 | |
| *** yangyapeng has quit IRC | 14:57 | |
| *** yangyape_ has quit IRC | 15:19 | |
| *** yangyapeng has joined #openstack-freezer | 15:19 | |
| *** yangyape_ has joined #openstack-freezer | 15:26 | |
| *** yangyape_ has quit IRC | 15:29 | |
| *** yangyapeng has quit IRC | 15:30 | |
| *** dschroeder has joined #openstack-freezer | 16:05 | |
| *** daemontool_ has joined #openstack-freezer | 16:12 | |
| *** daemontool has quit IRC | 16:13 | |
| *** smekel__ has joined #openstack-freezer | 16:13 | |
| *** smekel_ has quit IRC | 16:15 | |
| *** smekel__ is now known as smekel_ | 16:15 | |
| *** EinstCrazy has quit IRC | 16:19 | |
| *** reldan has quit IRC | 17:02 | |
| *** yangyapeng has joined #openstack-freezer | 17:14 | |
| *** yangyapeng has quit IRC | 17:19 | |
| openstackgerrit | Deklan Dieterly proposed openstack/freezer-api: Introduce tempest tests https://review.openstack.org/287369 | 18:29 |
| *** ddieterly has joined #openstack-freezer | 18:31 | |
| *** zhangjn has quit IRC | 18:37 | |
| *** zhangjn has joined #openstack-freezer | 18:38 | |
| *** EmilDi has quit IRC | 18:47 | |
| daemontool_ | ddieterly, that's a good one (thumbsup) | 18:51 |
| *** daemontool_ has quit IRC | 19:10 | |
| ddieterly | thanks | 19:48 |
| *** dschroeder has quit IRC | 20:34 | |
| *** dschroeder has joined #openstack-freezer | 20:38 | |
| *** yangyapeng has joined #openstack-freezer | 20:50 | |
| *** yangyapeng has quit IRC | 20:57 | |
| openstackgerrit | Deklan Dieterly proposed openstack/freezer-api: Introduce tempest tests https://review.openstack.org/287369 | 21:14 |
| *** yangyapeng has joined #openstack-freezer | 22:41 | |
| *** yangyapeng has quit IRC | 22:46 | |
| openstackgerrit | Deklan Dieterly proposed openstack/freezer-api: Introduce tempest tests https://review.openstack.org/287369 | 22:54 |
| *** ddieterly has quit IRC | 23:39 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!