Friday, 2016-03-25

« Thursday, 2016-03-24 Index Saturday, 2016-03-26 »
*** yangyapeng has quit IRC00:00
*** clsacramento has quit IRC00:26
*** reldan has quit IRC00:28
*** clsacramento has joined #openstack-freezer00:29
*** reldan has joined #openstack-freezer00:33
*** zhangjn has quit IRC00:41
*** zhangjn has joined #openstack-freezer00:47
*** yangyapeng has joined #openstack-freezer01:04
*** EinstCrazy has joined #openstack-freezer01:05
*** EinstCrazy has quit IRC01:07
*** EinstCrazy has joined #openstack-freezer01:09
*** yangyapeng has quit IRC01:10
*** yangyapeng has joined #openstack-freezer01:23
*** reldan has quit IRC01:23
*** reldan has joined #openstack-freezer01:24
*** reldan has quit IRC01:36
*** reldan has joined #openstack-freezer02:23
*** reldan has quit IRC03:00
*** EinstCra_ has joined #openstack-freezer04:05
*** EinstCrazy has quit IRC04:06
*** yangyape_ has joined #openstack-freezer05:23
*** yangyapeng has quit IRC05:23
*** yangyape_ has quit IRC05:24
*** EinstCra_ has quit IRC05:30
*** EinstCrazy has joined #openstack-freezer05:33
*** kelepirci has joined #openstack-freezer05:58
*** EinstCrazy has quit IRC06:44
*** EinstCrazy has joined #openstack-freezer06:48
*** EinstCrazy has quit IRC07:12
*** EinstCrazy has joined #openstack-freezer07:21
*** c00281451_ is now known as chenzeng07:26
openstackgerritdaiki kato proposed openstack/freezer: Fix typos  https://review.openstack.org/29756508:04
*** EinstCrazy has quit IRC08:24
*** EinstCrazy has joined #openstack-freezer08:34
*** reldan has joined #openstack-freezer08:55
*** EinstCra_ has joined #openstack-freezer09:16
*** EinstCrazy has quit IRC09:17
*** EinstCrazy has joined #openstack-freezer09:18
*** EinstCra_ has quit IRC09:19
*** EinstCrazy has quit IRC09:20
*** EinstCrazy has joined #openstack-freezer09:21
*** daemontool has joined #openstack-freezer09:22
daemontoolhi09:23
*** EinstCrazy has quit IRC09:23
*** EinstCr__ has joined #openstack-freezer09:23
*** EinstCrazy has joined #openstack-freezer09:24
*** Einst____ has joined #openstack-freezer09:26
*** EinstCr__ has quit IRC09:27
daemontoolreldan,  re https://review.openstack.org/#/c/296404/09:27
daemontoolwhat do you think if we call the options as 'overwrite' or 'skip_old_file'09:28
daemontoolwithout prepending tar as specific option09:28
daemontoolso we have the same option for any engine09:29
*** EinstCrazy has quit IRC09:29
openstackgerritEldar Nugaev proposed openstack/freezer: Added --tar-overwrite parameter for removing all files from restore directory. Added --tar-skip-old-files for don't overwriting existing files. Default value --tar-overwrite=False Default value --tar-skip-old-files=False  https://review.openstack.org/29640409:29
reldandaemontool: Hi09:32
reldandaemontool: Probably yes, but let’s first implement at least one additional implementation of engine09:32
reldandaemontool: And then let’s check what is common between them09:33
daemontoolreldan, ok, but then we have to change the other options09:33
reldandaemontool: can we have the same behaviour for rsync09:33
daemontoolyes09:33
daemontoolthat's the idea09:33
daemontoolbecause then09:33
reldandaemontool: second problem - it is not very clear what ‘overwrite’ means for me at least09:33
reldanif I have overwrite=False does it the same as skip-old-files=True09:34
reldanFrom common logic yes, but in terms of tar - not09:34
reldanWe also make agreement that cinder or nova are engines09:35
reldanBut for cinder and nova overwrite or skip-old-files have no sense at all09:36
daemontoolskip_old_file don't take any action if the file exists in the fs09:36
daemontoolI don't think the option for a backup solution make sense in general09:36
reldanSlashme said that he wants this option09:36
daemontooloverwrite do not skip, and do not remove the file prior09:36
daemontoolthe restore09:36
daemontoolit just rewrite the file09:37
reldanYes, but probably overwrite should be “clean”09:37
daemontoolI understand slashme said to not remove the content of the whole09:37
daemontooldirectory09:37
daemontoolprior restore09:37
daemontool?09:37
daemontoolslashme, ?09:37
reldanHe wants overwrite and skip-old-files09:37
reldanMy first patch contained only —overwrite09:38
reldanBut he said that he also would like to have —skip-old-files and it was not hard for me to implement09:38
daemontoolit's not09:38
daemontoolok09:38
daemontoolone sec09:38
daemontoollet me see09:39
daemontoolthe code09:39
daemontoolso I think the issue is with --unlink-first09:39
daemontoolwhen tar execute that on .09:39
daemontool     -U, --unlink-first09:40
daemontool           remove each file prior to extracting over it09:40
daemontoolbut if you do that on .09:40
daemontoolprobably remove the content of the whole directory prior the restore09:40
daemontoolIt's just a theory09:40
daemontoolwe need to remove09:40
daemontooland see09:40
daemontoolif the files are removed09:40
reldanI checked —overwrite and withouth overwrite it leaves new files untouched09:41
reldanwith —overwrite - removes all new files09:41
daemontoolcheck with slashme  because in the comments he says otherwise09:42
daemontoolis that the same case?09:42
reldanI will check again, but yesterday it worked for me09:46
*** reldan has quit IRC09:48
*** EinstCrazy has joined #openstack-freezer09:56
*** Einst____ has quit IRC09:59
slashmedaemontool: I tried removing the --unlink-first as well, with no luck.10:04
slashmeI tried to strip more and more things from the command10:06
slashme/bin/tar -z --incremental --extract --ignore-zeros --warning=none --directory /root/restore_dir/10:06
slashmeThis still removes my file10:06
*** EinstCrazy has quit IRC10:14
daemontoolif you remove overwrite and unlink-first still removes all the files in the directory?10:15
*** reldan has joined #openstack-freezer10:22
slashmeYup10:25
slashmeThis is with a patched liberty version10:25
slashmeGoing to try with master and the patch.10:25
slashmeBecause it does not make any sense...10:26
daemontoolslashme,  ok, it does remove the files even if the command is used directly on the shell?10:28
daemontoolkelepirci, good job with the documentation :)10:36
daemontoolI think10:39
daemontoolwe need to add also the sparse option10:39
daemontoolattributes and files acl10:39
daemontoolm3m0,  https://github.com/openstack/python-freezerclient10:40
daemontool:)10:40
slashmedaemontool: When I execute from the CLI, no problems10:44
slashmecat padawan-ccp-c0-m1-mgmt_plop_1458845634_0/1458845634/67108864/00000000 | tar -z --extract --ignore-zeros --directory /home/stack/restore_dir/10:45
slashmels restore_dir/10:45
slashmefile1  file2  file3  file410:45
daemontoolok10:51
m3m0could someone help me to review? https://review.openstack.org/#/c/291757/11:07
openstackgerritSaad Zaher proposed openstack/freezer: Use correct type for freezer arguments  https://review.openstack.org/29763011:18
*** jonaspf has joined #openstack-freezer11:20
openstackgerritSaad Zaher proposed openstack/freezer: Use correct type for freezer arguments  https://review.openstack.org/29763011:25
daemontoolm3m0,  if you added few comments on https://review.openstack.org/#/c/291757/ please check if  they make sense or not11:49
m3m0yep, it improves the messages11:54
m3m0thanks :)11:54
slashmedaemontool: After removing one flag at the time, it appears that the one that trigger the purge of the directory is --incremental ...11:54
daemontooland with --listed-incremental=/dev/null ?11:55
*** reldan has quit IRC11:59
daemontoolwe need to get rid of the tar and related binaries...12:00
daemontoolfrom ftp://ftp.gnu.org/old-gnu/Manuals/tar/html_node/tar_83.html12:03
daemontool --incremental (-G) in conjunction with --extract (--get, -x) causes tar to read the lists of directory contents previously stored in the archive, delete files in the file system that did not exist in their directories when the archive was created, and then extract the files in the archive.12:03
daemontoolThis behavior is convenient when restoring a damaged file system from a succession of incremental backups: it restores the entire state of the file system to that which obtained when the backup was made. If --incremental (-G) isn't specified, the file system will probably fill up with files that shouldn't exist any more.12:03
openstackgerritMemo Garcia proposed openstack/freezer-web-ui: UX improvements  https://review.openstack.org/29175712:07
slashmeThat's what we constated.12:08
slashmeI guess that should be an option for freezer.12:08
slashmeOnly problem , raised by reldan:12:13
slashmeIf you have:12:13
slashmeLevel1: file1 file2 file312:13
slashmeLevel2: file2 file312:13
slashmeRestore dir: file412:13
slashmeWhen restoring with --incremental, you get:12:13
slashmefile2 file312:13
slashmeWhen restoring without --incremental, you get12:13
slashmefile1 file2 file3 file412:13
daemontoolwell files needs to be removed12:14
daemontoolI mean if a file is removed between backup levels12:14
slashmeBut if we keep the --incremental behaviour by default, we will keep on having people complaining that they accidentaly wiped the content of their /home/ directory12:14
daemontoolthen it is right to remove it12:14
daemontoolI understand but12:15
slashmeIt is right to remove file1, but not file412:15
daemontoolyes12:15
daemontoolI think12:15
daemontoolif we restore the files that has been removed in the middle12:15
daemontoolusers can have big problems when restoring12:15
daemontooltar people thought the same as we thought after all12:16
daemontoolwe can add an option called --force12:17
daemontoolor --force-overwrite12:18
daemontoolare the files removed12:19
daemontooleven by using12:19
daemontool--listed-incrementals=/dev/null12:19
daemontoolin place of --incrementals12:19
daemontool?12:19
slashmechecking12:19
daemontoolif --reastore-abs-path directory is not empty, the restore stops12:20
daemontooland you need to use --force-overwrite12:20
daemontoolwe can do that12:20
slashmesame thing with --listed-incrementals12:21
daemontoolok12:21
daemontoolwhat do you think about the --force-overwrite option?12:21
slashmeWe have two solutions12:21
slashmeAnyway, by default, restore should be without the --incremental flag. and we should add some kind of --force-overwrite flag.12:25
slashmeOption1: We just restore even if directory is not empty (and end up with file1)12:25
slashmeOption2: We Error if restore_abs_path is not empty, no restore happen and you have to restart with --force-overwrite12:25
slashmeOption3: We Warning but still do the restore12:25
slashmeI'm not sure what I prefer12:25
slashmeLet's get some opinions12:26
slashmereldan m3m0 frescof vannif szaher ^^12:26
daemontoolI think the restore should be with --incremental and it does not allow to do the restore if the directory is not empty or --force-overwrite is used12:26
daemontoolthe goal of this12:27
daemontoolis to prevent users12:27
daemontoolto delete the content of the directory12:27
daemontoolthey are restoring on12:27
daemontoolwithout they knows12:27
*** reldan has joined #openstack-freezer12:28
daemontoolI don't think we should restore without removing the files the have been removed between levels12:28
daemontoolbut yes, let's see what others think also12:29
daemontoolthat was just my opinion12:29
daemontoolreldan,  ping12:29
reldanI’m here12:30
reldandaemontool12:30
openstackgerritMerged openstack/freezer: Use correct type for freezer arguments  https://review.openstack.org/29763012:30
slashmedaemontool: I agree12:32
daemontoolreldan, I'm doing some code reorganization for rsync12:33
daemontoolbefore going forward as it will be easier12:33
daemontoolto udnerstand12:33
daemontoolI'm using queues now12:33
daemontooldo you think I should just use threading queues12:33
daemontoolor use RichQueue, the one you wrote12:34
daemontoolI'm going to use Threading queues for now I already know  it, then when reviewing you can advise if it would be the case of changing it?12:35
daemontoolor use directly righqueues?12:35
daemontools/righ/rich/12:35
reldandaemontool: I am not very experienced with python threading. I have created RichQueue to be able close queue12:36
reldanand don’t get synchronious stop12:37
reldanif my queue is full12:37
reldanput and get to RichQueue are not blocking with retries12:37
reldanLet’s say I have writer and reader and would like to stop reader if writer got exception12:38
daemontoolok12:38
reldanSo if you can implement your staff without it, let’s do that way12:39
daemontoolok12:55
*** reldan has quit IRC12:57
openstackgerritMerged openstack/freezer-web-ui: UX improvements  https://review.openstack.org/29175712:57
*** reldan has joined #openstack-freezer13:21
m3m0slashme ping pong13:40
slashmeyup13:49
m3m0please review a backport to liberty : https://review.openstack.org/#/c/297690/13:58
m3m0new backport https://review.openstack.org/29769314:14
m3m0another backport https://review.openstack.org/29769614:14
*** ddieterly has joined #openstack-freezer14:17
*** ddieterly is now known as ddieterly[away]14:30
*** ddieterly[away] is now known as ddieterly14:34
*** ddieterly is now known as ddieterly[away]14:35
*** dschroeder has joined #openstack-freezer14:37
*** ddieterly[away] is now known as ddieterly14:40
slashmedaemontool: What about --overwrite [never, yes, merge]14:54
slashmeBefore restore, we check if restore-abs-path is empty. If not:14:54
slashmeIf overwrite never: restore does not happen and stops with an error14:54
slashmeIf overwrite yes: restore happens with --incremental (we loose file4)14:54
slashmeIf overwrite merge: restore happens without --incremental (we have file1 in excess)14:54
slashmeWhat would be the default behaviour. Frescof is ruling for merge. I'm not sure14:54
openstackgerritMemo Garcia proposed openstack/python-freezerclient: Filter jobs by client or custom filters  https://review.openstack.org/29771414:55
reldanToo complex for me. I would prefer two modes: 1) Remove everything and restore, I don’t care 2) If directory for restore is not empty - throw exception, I don’t want rewrite14:59
*** yangyapeng has joined #openstack-freezer15:05
daemontoolslashme,  I think is complex15:10
daemontoolmerge is not easy to understand15:11
daemontoolI think, when the user restore, he/she wants to restore as fast as possible15:12
slashmeOkay. So let's implement --overwrite (boolean) and we will see later if addind the merge makes any sense. I kind of think it does. But it'll be part of another commit and option.15:13
daemontoolwhy not add --force-overwrite15:13
daemontoolif the directory is not empty you have to use it15:13
daemontoolsimple15:13
slashmeI don't have any preference between --overwrite and --force-overwrite.15:14
daemontoolit would works exactly as it works now15:14
daemontoolthe only code to add if a check for the directory empty15:15
slashmeyep15:15
daemontoolraise exception is empty15:15
daemontoollog15:15
daemontools/is/if/15:15
daemontoolthen the user have 2 choices15:15
daemontool1) delete the content of the directory15:15
daemontool2) create a directory and restore there15:15
daemontool3) use --force-overwrite15:15
daemontooland have the same behavior we have now15:15
slashmeI agree15:15
daemontoolwell that are 3 choises15:16
daemontool:D15:16
daemontoolnot 215:16
daemontoollol15:16
daemontoolwe don't have much choices15:17
daemontoolas tar remove the directory content15:17
daemontooland for instance15:17
daemontoolin a use case15:17
daemontoolwhere the user is restoring 200GB15:17
daemontoolor 500GB like it was happening in the public cloud15:17
daemontoolit takes time15:17
daemontooland you need to restore15:17
daemontoolit does not delete the data that has been delete between levels15:18
daemontooland the space is filled up15:18
daemontooland you cannot restore15:18
daemontoolthen yes as a user I'd be very pissed off15:18
slashmeYup15:19
slashmeOkay. Let's go with that.15:19
daemontoolwe need to add one option, one if, one log and one raise15:20
daemontooland we are good15:20
slashme--overwrite or --force-overwrite ? From my experience, non-empty directory represent most usecases.15:20
slashmeMaybe we can save the user a few keystroke15:20
daemontoolthen --overwrite15:21
daemontoolit's OK15:21
daemontoolif that check can be a function in utils better15:21
daemontoolso it can be used in both tar and rsync15:21
slashmeMakes sense15:22
*** ddieterly is now known as ddieterly[away]15:38
*** ddieterly[away] is now known as ddieterly15:38
openstackgerritEldar Nugaev proposed openstack/freezer: Added --overwrite parameter for removing all files from restore directory. Default value --overwrite=False  https://review.openstack.org/29640415:55
slashmedaemontool: ping15:59
daemontoolyes16:00
slashmeCan you check your pm16:02
daemontoolpm == private mail?16:04
slashmeOops send it to daemontool_16:05
slashmeNeed review on https://review.openstack.org/#/c/296404/ please. It is the overwrite patch16:22
*** reldan has quit IRC16:36
*** ddieterly is now known as ddieterly[away]16:45
daemontoolslashme,  I'm not sure we should backport that change16:46
daemontoolto liberty16:46
*** ddieterly[away] is now known as ddieterly16:48
slashmeHP requirement... If you really insist, I'll manage on internal repos, but I think we should, we need to maintain liberty for some time and people will keep coming to us after having delete important stuff.16:49
*** ddieterly has quit IRC16:52
openstackgerritMerged openstack/freezer: Added --overwrite parameter for removing all files from restore directory. Default value --overwrite=False  https://review.openstack.org/29640417:17
*** reldan has joined #openstack-freezer17:18
slashmeBackports:17:23
slashme UI: https://review.openstack.org/#/c/29769017:23
slashme UI: https://review.openstack.org/#/c/29769317:23
slashme UI: https://review.openstack.org/#/c/29769617:23
slashme Agent: https://review.openstack.org/#/c/29776617:23
slashmeMitaka:17:30
slashme https://review.openstack.org/29777817:30
slashme https://review.openstack.org/29777917:30
slashme https://review.openstack.org/29778017:30
slashme https://review.openstack.org/29778217:30
daemontoolok but in the future we should really avoid to do that17:30
slashmeI totaly agree, we shouldn't backport feature.17:34
slashmeBut from my point of view, this one is a bug-fix, as people come to us saying this is a bug in Freezer.17:34
daemontoolso there's a but in tar also17:34
daemontoolcause we do just what tar does17:35
daemontools/but/bug/17:35
daemontoolbut in general, we shouldn't change default behaviour of past stable release17:35
daemontoolcause if people have jobs for the restore, then it wont work17:36
slashmeFreezer is not tar, freezer leverages tar in a smart way. Also this is not the default behaviour of tar, just the one you get when using --incremental17:36
slashmeI know for the jobs, it was part of my concern when doing the backport17:37
slashmeWe need to backport the "Start to introduce tempest tests." to liberty if we want to be able to merge anything on that branch.17:39
slashmehttps://review.openstack.org/29778617:40
*** reldan has quit IRC17:49
*** joanspf_ has joined #openstack-freezer17:51
*** jonaspf has quit IRC17:55
*** joanspf_ has quit IRC17:58
*** jonaspf has joined #openstack-freezer17:59
openstackgerritFausto Marzi proposed openstack/freezer: WIP block based incremntal support - rsync  https://review.openstack.org/29046118:07
daemontoolI'm going off18:08
daemontoolenjoy your Easter18:08
daemontoolbye18:08
slashmeYou too :)18:09
*** daemontool has quit IRC18:12
*** reldan has joined #openstack-freezer18:15
openstackgerritDeklan Dieterly proposed openstack/freezer-api: Add Freezer API Tempest Tests  https://review.openstack.org/29736018:24
openstackgerritDeklan Dieterly proposed openstack/freezer-api: Add Freezer API Tempest Tests  https://review.openstack.org/29736019:14
*** jonaspf has quit IRC21:10
openstackgerritDeklan Dieterly proposed openstack/freezer-api: Add Freezer API Tempest Tests  https://review.openstack.org/29736021:48
openstackgerritDeklan Dieterly proposed openstack/freezer-api: Add Freezer API Tempest Tests  https://review.openstack.org/29736022:01
*** reldan has quit IRC22:22
*** reldan has joined #openstack-freezer22:32
*** reldan has quit IRC23:15
*** dhellmann has quit IRC23:24
*** reldan has joined #openstack-freezer23:24
*** reldan has quit IRC23:36
*** dschroeder has quit IRC23:42
*** yangyapeng has quit IRC23:51
« Thursday, 2016-03-24 Index Saturday, 2016-03-26 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!