| *** ddieterly has joined #openstack-freezer | 00:04 | |
| *** ddieterly is now known as ddieterly[away] | 01:11 | |
| *** ddieterly[away] is now known as ddieterly | 01:11 | |
| *** ddieterly is now known as ddieterly[away] | 01:12 | |
| *** EinstCrazy has joined #openstack-freezer | 01:13 | |
| *** EinstCrazy has quit IRC | 01:34 | |
| *** EinstCrazy has joined #openstack-freezer | 01:34 | |
| *** ddieterly[away] has quit IRC | 02:12 | |
| *** EinstCrazy has quit IRC | 04:50 | |
| *** EinstCrazy has joined #openstack-freezer | 05:08 | |
| *** EinstCra_ has joined #openstack-freezer | 05:59 | |
| *** EinstCrazy has quit IRC | 06:02 | |
| *** daemontool has joined #openstack-freezer | 06:37 | |
| daemontool | Morning | 06:49 |
|---|---|---|
| *** daemontool has quit IRC | 07:30 | |
| *** daemontool has joined #openstack-freezer | 07:41 | |
| *** reldan has joined #openstack-freezer | 07:41 | |
| daemontool | reldan, ping | 07:45 |
| *** EinstCra_ has quit IRC | 08:15 | |
| *** EinstCra_ has joined #openstack-freezer | 08:15 | |
| *** reldan has quit IRC | 08:24 | |
| daemontool | slashme, ping | 08:31 |
| *** reldan has joined #openstack-freezer | 08:49 | |
| reldan | daemontool: Hi! How are you? | 08:49 |
| daemontool | all good | 08:53 |
| daemontool | for the restore... | 08:53 |
| daemontool | I think I need to add a small header at the beginning of the binary blob | 08:54 |
| daemontool | otherwise I cannot identify if the backup is done with rsync or tar | 08:54 |
| daemontool | did you see the first bytes | 08:55 |
| daemontool | in the tar metadata with listed-incremental? | 08:55 |
| daemontool | something like that | 08:55 |
| daemontool | like | 08:55 |
| daemontool | freezer_rsync-1.0 | 08:55 |
| daemontool | so when I download the data for restore | 08:56 |
| daemontool | I can read those bytes | 08:56 |
| daemontool | and execute the restore accordingly | 08:56 |
| reldan | daemontool: Yes, or we can store this information to metadata | 08:57 |
| reldan | daemontool: I even prefer approach with metadata | 08:59 |
| daemontool | reldan, yes | 09:11 |
| daemontool | but then we need the metadata to restore | 09:11 |
| daemontool | with tar that is not required now | 09:12 |
| reldan | Yes, you are right. So we have 3 options: 1) Add signature to file 2) Wait for metadata 3) During restore provide engine manually (like now we should provide compressiona algorithm) | 09:13 |
| reldan | I would prefer 3-rd for now and then second | 09:14 |
| daemontool | otherwise we need to modify the names | 09:14 |
| daemontool | incrementalengine_hostname_.... | 09:14 |
| daemontool | incrementalengine_metadata_hostname.... | 09:14 |
| daemontool | the file names | 09:14 |
| reldan | Yes, we are going to have it, but not in names - in sub-folders | 09:15 |
| reldan | like engine/backup_name(or id)/timestamp_of_level_0/timestamp_level/{data-file engine-metafile freezer-metafile} | 09:15 |
| daemontool | ok | 09:15 |
| daemontool | I have a long meeting today | 09:16 |
| daemontool | vannif, ping | 09:16 |
| daemontool | vannif, per chance, do you think we can give the source code walkthrought tomorrow? | 09:16 |
| daemontool | laterz | 09:16 |
| *** daemontool has quit IRC | 09:17 | |
| reldan | daemontool: I send a PR for restore https://review.openstack.org/#/c/298817/ | 09:17 |
| reldan | daemontool: But then gave me +2 then uploaded some path, then abandoned it | 09:17 |
| reldan | then have created two https://review.openstack.org/#/c/298867/ and https://review.openstack.org/#/c/298869/ | 09:19 |
| reldan | daemontool: Your comment here is good https://review.openstack.org/#/c/298869/ , and should work. But here is potental bug with sslerror during getting stream | 09:20 |
| reldan | in this case it will produce wrong stream | 09:20 |
| *** vannif has quit IRC | 09:29 | |
| *** kelepirci has joined #openstack-freezer | 09:30 | |
| *** vannif has joined #openstack-freezer | 09:31 | |
| openstackgerrit | Memo Garcia proposed openstack/python-freezerclient: Display message for empty lists https://review.openstack.org/299248 | 09:41 |
| szaher | Hey guys, Anyone can add +A here plz https://review.openstack.org/#/c/298859/ ? | 09:50 |
| EinstCra_ | Hello guys, Anyone can review https://review.openstack.org/298588 and https://review.openstack.org/298628 ? | 10:07 |
| openstackgerrit | Einst Crazy proposed openstack/freezer: Restore cindernative volume pass right params and call right api https://review.openstack.org/299277 | 10:27 |
| *** EinstCra_ has quit IRC | 10:28 | |
| slashme | Ahhrrrrr, sorry for the mess Eldar on the swift client patch | 10:33 |
| slashme | I'll restore it... | 10:33 |
| reldan | slashme: No problem at all :) If you need any help - I can create new pull request or reopen old one | 10:34 |
| openstackgerrit | Merged openstack/freezer: Fix creation of jobs with stopped state https://review.openstack.org/298859 | 10:41 |
| *** daemontool has joined #openstack-freezer | 10:57 | |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Recreation swift client for each chunk during swift incremental restore https://review.openstack.org/298817 | 11:06 |
| *** ccesario has quit IRC | 11:08 | |
| openstackgerrit | Ferhat Ozkasgarli proposed openstack/freezer: New Better & Well Formatted Docs with Sphinx https://review.openstack.org/296448 | 11:42 |
| *** ddieterly has joined #openstack-freezer | 11:46 | |
| *** openstackgerrit has quit IRC | 11:47 | |
| *** openstackgerrit has joined #openstack-freezer | 11:48 | |
| *** ddieterly is now known as ddieterly[away] | 11:52 | |
| *** EinstCrazy has joined #openstack-freezer | 11:59 | |
| *** kelepirci has quit IRC | 12:00 | |
| *** ddieterly[away] is now known as ddieterly | 12:03 | |
| *** ccesario has joined #openstack-freezer | 12:07 | |
| *** ddieterly is now known as ddieterly[away] | 12:15 | |
| *** daemontool has quit IRC | 12:27 | |
| *** daemontool has joined #openstack-freezer | 12:32 | |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** openstackgerrit has joined #openstack-freezer | 12:33 | |
| slashme | FYI, We are impacted by this tar behaviour when using incremental: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=648048 | 12:38 |
| openstack | Debian bug 648048 in tar "tar: incremental restore fails when a directory is renamed as a name of a deleted directory" [Important,Open] | 12:38 |
| daemontool | slashme, ah, ok | 12:40 |
| daemontool | freezer mitaka-rc2 https://review.openstack.org/#/c/299368/ | 12:45 |
| daemontool | slashme, why on https://review.openstack.org/#/c/298869/ are we recreating the swiftclient for each chunk | 12:49 |
| *** ddieterly[away] has quit IRC | 12:50 | |
| slashme | daemontool: 1 sec, I'm getting the traces | 13:09 |
| daemontool | slashme, np take your time | 13:10 |
| daemontool | unfortunately that is going to slow down the process | 13:11 |
| daemontool | as for each chunk, a new token will be requested | 13:11 |
| slashme | http://paste.openstack.org/show/492436/ | 13:14 |
| *** openstackgerrit has quit IRC | 13:18 | |
| *** openstackgerrit has joined #openstack-freezer | 13:19 | |
| daemontool | so if we recreate the swift client and it works | 13:30 |
| daemontool | it is probably because | 13:30 |
| daemontool | the client object is closed by some other process? | 13:30 |
| daemontool | reldan, ping | 13:32 |
| reldan | daemontool: I suppose just because of expiration | 13:33 |
| daemontool | expiration of the ssl session or the token? | 13:34 |
| reldan | daemontool: I actually don’ know exactly | 13:34 |
| daemontool | if it does not fail for every chunk, | 13:34 |
| daemontool | let's try something like this https://review.openstack.org/#/c/298869/1/freezer/storage/swift.py | 13:34 |
| daemontool | in the comment | 13:34 |
| daemontool | that method is used for restore, right? | 13:35 |
| daemontool | def backup_blocks(self, backup): | 13:35 |
| reldan | daemontool: we can write something like that | 13:36 |
| reldan | try: | 13:36 |
| daemontool | cause like that is going to slow down considerably the restore | 13:37 |
| reldan | chunks = self.swift().get_object …. | 13:37 |
| reldan | except SSLError: | 13:37 |
| reldan | chunks=self.swift().create_swift().get_object | 13:37 |
| reldan | And then | 13:38 |
| reldan | for chunk in chunks: | 13:38 |
| reldan | yield chunk | 13:38 |
| reldan | daemontool: Good? | 13:38 |
| daemontool | mmhhhh | 13:38 |
| daemontool | what's chunks? | 13:38 |
| daemontool | a list of what? | 13:38 |
| daemontool | of just binary data? | 13:38 |
| reldan | yes | 13:38 |
| daemontool | binary data? | 13:39 |
| reldan | binary data | 13:39 |
| reldan | like we have code now | 13:39 |
| daemontool | so with python, a for loop on a binary data | 13:39 |
| reldan | but we split it in two | 13:39 |
| reldan | 1) get chunks with try | 13:39 |
| reldan | 2) yield each chunk | 13:39 |
| daemontool | iterate 1 byte at the time | 13:39 |
| EinstCrazy | Hello guys, Anyone can review https://review.openstack.org/298588 and https://review.openstack.org/298628 ? | 13:39 |
| reldan | daemontool: the same way it works now | 13:39 |
| reldan | daemontool: https://review.openstack.org/#/c/298817/4/freezer/storage/swift.py | 13:40 |
| daemontool | mmhhh nope, because get_object avoid that | 13:40 |
| daemontool | that is what resp_chunk_size is for | 13:40 |
| daemontool | reldan, did you see the code in my comment_ | 13:40 |
| daemontool | ? | 13:40 |
| reldan | daemontool: yes | 13:41 |
| daemontool | well it would restart the backup | 13:41 |
| reldan | daemontool: check comment here https://review.openstack.org/#/c/298817/4/freezer/storage/swift.py | 13:41 |
| reldan | my comment | 13:41 |
| daemontool | but I'd like to understand why the ssl error is happening... | 13:41 |
| reldan | Is it good for you? | 13:41 |
| daemontool | checking | 13:42 |
| reldan | daemontool: Me niether, I cannot reproduce it. But Pierre says that it fixes his problem | 13:42 |
| slashme | It does fix the problem | 13:42 |
| daemontool | reldan, yes | 13:42 |
| daemontool | if chunks is an iterator | 13:43 |
| daemontool | it's OK | 13:43 |
| daemontool | otherwise not | 13:43 |
| daemontool | can you print len(chunk) inside the for loop? | 13:43 |
| *** ddieterly has joined #openstack-freezer | 13:43 | |
| reldan | daemontool: Great. So in this case we can solve both of our problem. We can be sure, that our stream is correct or fail. | 13:43 |
| reldan | daemontool: yes, it is equal to chunk size | 13:43 |
| daemontool | reldan, ok | 13:43 |
| daemontool | ah perfect | 13:43 |
| daemontool | :) | 13:43 |
| reldan | :) | 13:43 |
| slashme | Testing that | 13:45 |
| daemontool | I'd even put that inside a while True | 13:46 |
| daemontool | in case while chunks is iterated and the keystone token expires it would be caught | 13:46 |
| reldan | caemontool: We cannot do this, otherwise we repeat some chunks in stream | 13:47 |
| reldan | daemontool | 13:47 |
| daemontool | yes | 13:47 |
| daemontool | that's true | 13:47 |
| daemontool | it would be complicated | 13:47 |
| daemontool | to manage | 13:47 |
| daemontool | with counters etc | 13:48 |
| daemontool | I think your solution is brilliant | 13:48 |
| openstackgerrit | Einst Crazy proposed openstack/freezer: Restore cindernative volume pass right params and call right api https://review.openstack.org/299277 | 13:49 |
| reldan | daemontool: Thank you! | 13:50 |
| *** ddieterly is now known as ddieterly[away] | 13:55 | |
| *** ddieterly[away] is now known as ddieterly | 14:00 | |
| *** EinstCrazy has quit IRC | 14:07 | |
| *** sohail__ has joined #openstack-freezer | 14:30 | |
| daemontool | hi sohail__ | 14:30 |
| daemontool | reldan, vannif slashme al, sohail__ works in Ericsson he's doing a thesis | 14:31 |
| daemontool | interested on working in openstack and contributing on freezer | 14:31 |
| sohail__ | Hi All | 14:32 |
| daemontool | slashme, ^^ | 14:33 |
| reldan | Hi soahil__ | 14:33 |
| sohail__ | Hi reldan | 14:33 |
| daemontool | sohail__, would you like to test our documentation? | 14:33 |
| daemontool | about what you do not understand | 14:33 |
| vannif | hi sohail__ | 14:33 |
| daemontool | and so on | 14:33 |
| daemontool | so you can improve it? | 14:33 |
| daemontool | :) | 14:33 |
| daemontool | or feel free to ask to the guys | 14:33 |
| daemontool | for some task to do | 14:34 |
| sohail__ | Yes I can do that | 14:34 |
| openstackgerrit | Deklan Dieterly proposed openstack/freezer-api: Add Freezer API Tempest Tests https://review.openstack.org/297360 | 14:34 |
| ddieterly | sohail__ you can write tests | 14:34 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Recreation swift client for each chunk during swift incremental restore https://review.openstack.org/298817 | 14:34 |
| daemontool | sohail__, needs to improve his python skills | 14:35 |
| daemontool | :) | 14:35 |
| sohail__ | I know very little about python | 14:35 |
| daemontool | time to improve that dude :) | 14:35 |
| ddieterly | do you want to learn python? | 14:35 |
| sohail__ | yes | 14:35 |
| ddieterly | ok, then writing tests is the best way to learn | 14:35 |
| sohail__ | If you guide me I can do my best | 14:39 |
| slashme | Hi sohail__ Welcome :) | 14:42 |
| slashme | +1 ddieterly tests is a great way of stepping into python and freezer | 14:43 |
| slashme | Don't hesitate to ask question :) | 14:43 |
| *** ddieterly is now known as ddieterly[away] | 14:49 | |
| daemontool | I think sohail__ needs some more guidance | 14:50 |
| daemontool | sohail__, try to install the freezer-agent | 14:51 |
| daemontool | and execute some backup on your desktop | 14:51 |
| daemontool | following th edocumentation | 14:51 |
| daemontool | and let us know what's not right with it | 14:51 |
| daemontool | so the doc can be improved | 14:51 |
| daemontool | and you learn things | 14:51 |
| daemontool | after few days | 14:51 |
| daemontool | you can do some other task if that is OK for you | 14:52 |
| slashme | Yup. That's a first step. After a few days, trying to install a devstack and the rest of the freezer infra would be insteresting as well. | 14:52 |
| *** ddieterly[away] is now known as ddieterly | 14:52 | |
| slashme | This would give you a good understanding of the global concepts before starting to look at the code. | 14:53 |
| daemontool | sohail__, I think you need to understand about openstack-infra ci | 14:54 |
| daemontool | also | 14:54 |
| slashme | yes | 14:56 |
| slashme | If you already know git, then this link explain how code managment work in OpenStack: http://docs.openstack.org/infra/manual/developers.html | 14:57 |
| *** ddieterly is now known as ddieterly[away] | 15:12 | |
| *** ddieterly[away] is now known as ddieterly | 15:15 | |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Recreation swift client for each chunk during swift incremental restore https://review.openstack.org/298817 | 15:16 |
| *** dschroeder has joined #openstack-freezer | 15:21 | |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Recreation swift client for each chunk during swift incremental restore https://review.openstack.org/298817 | 15:22 |
| slashme | daemontool: tested that fix. It works. I narrowed the error to only request.exception.SSLError If in the future see other error causing the same behabiour, we will add them here. | 15:23 |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Recreation swift client for each chunk during swift incremental restore https://review.openstack.org/298817 | 15:25 |
| slashme | Definitly nicer than recreating the client every chunk. :) | 15:26 |
| sohail__ | Hi daemontool and ddieterly thanks for your help | 15:27 |
| sohail__ | if you give the documentation that i can use for backup and restore. | 15:27 |
| sohail__ | i can use it to test | 15:28 |
| ddieterly | sohail__ the doc is in the readme file | 15:28 |
| openstackgerrit | Deklan Dieterly proposed openstack/freezer-api: Add Freezer API Tempest Tests https://review.openstack.org/297360 | 15:43 |
| *** ddieterly is now known as ddieterly[away] | 16:10 | |
| slashme | Waiting for review: | 16:12 |
| slashme | - SSL Master: https://review.openstack.org/#/c/298817/ | 16:12 |
| slashme | - SSL Mitaka: https://review.openstack.org/#/c/298869/ | 16:12 |
| slashme | - SSL Liberty: https://review.openstack.org/#/c/298867/ | 16:12 |
| slashme | - Booleans Mitaka: https://review.openstack.org/#/c/297780 | 16:12 |
| slashme | - Cinder 1 : https://review.openstack.org/298588 | 16:12 |
| slashme | - Cinder 2 : https://review.openstack.org/298628 | 16:12 |
| slashme | - Percent LVM: https://review.openstack.org/#/c/298745/ | 16:12 |
| *** ddieterly[away] is now known as ddieterly | 16:16 | |
| *** daemontool has quit IRC | 16:18 | |
| *** openstack has joined #openstack-freezer | 17:07 | |
| *** ddieterly is now known as ddieterly[away] | 17:08 | |
| *** reldan has joined #openstack-freezer | 17:20 | |
| *** sohail__ has quit IRC | 17:24 | |
| daemontool | ddieterly[away], https://github.com/openstack/freezer/blob/master/freezer/job.py#L93 | 17:24 |
| daemontool | and https://github.com/openstack/freezer/blob/master/freezer/job.py#L103 | 17:25 |
| *** reldan has quit IRC | 17:32 | |
| slashme | daemontool: Which level for the SSL error in the logs. Debug ? | 17:34 |
| daemontool | LOG.exception I think | 17:40 |
| daemontool | as it is generated by an exeption | 17:40 |
| slashme | I don't think exception is a loggin level. | 17:43 |
| slashme | Level are Critical, Error, Warning, Info, Debug | 17:43 |
| daemontool | isn't this available in oslo.log? https://docs.python.org/2/library/logging.html#logging.Logger.exception | 17:44 |
| daemontool | if not sorry | 17:44 |
| daemontool | log.error? | 17:44 |
| daemontool | like the example here http://stackoverflow.com/questions/5191830/best-way-to-log-a-python-exception | 17:45 |
| daemontool | but we do not have to use it necessarly | 17:45 |
| daemontool | Error or Warning | 17:45 |
| slashme | You are right, exception is available. | 17:46 |
| slashme | Debug: Shows everything and is likely not suitable for normal production operation due to the sheer size of logs generated | 17:48 |
| slashme | Info: Usually indicates successful service start/stop, versions and such non-error related data | 17:48 |
| slashme | Audit: (An OpenStack invented, level, still not sure what we should be doing with it) | 17:48 |
| slashme | Warning: Indicates that there might be a systemic issue; potential predictive failure notice | 17:48 |
| slashme | Error: An error has occurred and an administrator should research the event | 17:48 |
| slashme | Critical: An error has occurred and the system might be unstable; immediately get administrator assistance | 17:48 |
| slashme | Exception will generate a ERROR level | 17:48 |
| jokke_ | exception is ERROR level log with stack trace | 17:49 |
| daemontool | warning or exception | 17:49 |
| daemontool | are good for me | 17:49 |
| slashme | I'd go for warning | 17:49 |
| daemontool | o | 17:49 |
| daemontool | k | 17:49 |
| slashme | thx | 17:49 |
| daemontool | we whould have a feedback from reldan about this https://review.openstack.org/#/c/298588/ | 17:52 |
| slashme | Yup | 17:53 |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Recreation swift client for each chunk during swift incremental restore https://review.openstack.org/298817 | 17:56 |
| *** reldan has joined #openstack-freezer | 18:02 | |
| slashme | daemontool: Do you understand why this one fails ? https://review.openstack.org/298895 | 18:04 |
| *** daemontool has quit IRC | 18:07 | |
| *** ddieterly[away] has quit IRC | 18:08 | |
| *** ddieterly has joined #openstack-freezer | 18:11 | |
| *** ddieterly is now known as ddieterly[away] | 18:11 | |
| *** ddieterly[away] is now known as ddieterly | 18:13 | |
| *** ddieterly is now known as ddieterly[away] | 18:15 | |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Adding the freezer logo to the README https://review.openstack.org/299567 | 18:16 |
| *** ddieterly[away] is now known as ddieterly | 18:18 | |
| openstackgerrit | Pierre Mathieu proposed openstack/freezer: Adding the freezer logo to the README https://review.openstack.org/299567 | 18:18 |
| slashme | We officialy have a logo ! | 18:27 |
| slashme | https://wiki.openstack.org/wiki/FreezerLogo | 18:27 |
| ddieterly | that's great | 18:39 |
| ddieterly | er... c'est magnifique! | 18:40 |
| *** ddieterly is now known as ddieterly[away] | 18:54 | |
| *** ddieterly[away] is now known as ddieterly | 18:55 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** openstackgerrit has joined #openstack-freezer | 20:48 | |
| *** reldan has quit IRC | 20:57 | |
| *** ddieterly is now known as ddieterly[away] | 22:13 | |
| *** ddieterly[away] is now known as ddieterly | 22:15 | |
| *** ddieterly is now known as ddieterly[away] | 22:24 | |
| *** ddieterly[away] is now known as ddieterly | 22:25 | |
| *** ddieterly is now known as ddieterly[away] | 22:26 | |
| *** ddieterly[away] has quit IRC | 22:39 | |
| *** ddieterly has joined #openstack-freezer | 23:17 | |
| *** ddieterly is now known as ddieterly[away] | 23:30 | |
| *** ddieterly[away] is now known as ddieterly | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!