| *** mmcardle has joined #openstack-swift | 00:00 | |
| *** mwstorer has quit IRC | 00:03 | |
| *** mmcardle has quit IRC | 00:04 | |
| gholt | clayg: zaitcev: creiht: redbo: anymore? Doesn't look like anyone's called is_deleted with the timestamp kwarg since we open-sourced Swift. My best guess is that it was to be used by whoever was doing reclamation to decide when to remove old deleted containers; but then that logic got pulled out to the db_replicator.py when that refactor happened (pre open sourcing). | 00:06 |
|---|---|---|
| zaitcev | gholt: thanks | 00:07 |
| redbo | code archaeology | 00:07 |
| * portante wonders when folks will start getting advanced degrees in that discipline ... | 00:08 | |
| clayg | lol @ http://en.wikipedia.org/wiki/Software_archaeology | 00:08 |
| gholt | We probably still have the pre open source git repo around somewhere; that would be fun, heheh. | 00:08 |
| zaitcev | okay, PYTHONPATH=$(pwd) ./.unittests and committing | 00:09 |
| portante | tox -e pep8? | 00:09 |
| clayg | portante: that's what the gate is for? | 00:09 |
| portante | ;) | 00:09 |
| portante | clayg: many a problem with code has been found via a simple pep8 check | 00:10 |
| *** _bluev has quit IRC | 00:10 | |
| portante | zaitcev: have you check code coverage with the unit tests? | 00:10 |
| openstackgerrit | A change was merged to openstack/swift: update setup.py with pbr version https://review.openstack.org/86101 | 00:11 |
| openstackgerrit | A change was merged to openstack/swift: Update callback with proper bytes transferred https://review.openstack.org/84610 | 00:11 |
| openstackgerrit | A change was merged to openstack/swift: Attempt to ensure connect always timesout https://review.openstack.org/85457 | 00:11 |
| openstackgerrit | A change was merged to openstack/swift: merge master to EC branch https://review.openstack.org/85919 | 00:11 |
| gholt | I hope redbo wasn't staring at me again while I was in my coma. | 00:11 |
| redbo | I don't have to, I have lots of pictures of you when I get lonely | 00:12 |
| *** _bluev has joined #openstack-swift | 00:13 | |
| redbo | though your face isn't in most of them | 00:13 |
| clayg | whoa | 00:14 |
| redbo | it's not what you think. I just take pictures of his crotch. | 00:14 |
| * clayg struggles to pick himself up off the floor | 00:17 | |
| gholt | Yeah, it's kinda creepy. Oh, who am I kidding? It's really creepy. | 00:17 |
| portante | awkward | 00:21 |
| *** _bluev has quit IRC | 00:25 | |
| *** RockKuo has joined #openstack-swift | 00:28 | |
| *** RockKuo_ has joined #openstack-swift | 00:28 | |
| *** matsuhashi has joined #openstack-swift | 00:31 | |
| zaitcev | well, I ended Seitokai Yakuindomo today, so it's nothing | 00:38 |
| zaitcev | and yeah... mem_backend.py is at around 60% for both a and c | 00:38 |
| zaitcev | needs a look-over | 00:39 |
| *** zaitcev has quit IRC | 00:40 | |
| *** shri1 has quit IRC | 00:40 | |
| *** mkollaro has quit IRC | 00:56 | |
| *** matsuhashi has quit IRC | 00:56 | |
| *** acoles has quit IRC | 01:01 | |
| *** mmcardle has joined #openstack-swift | 01:01 | |
| *** jamie_h has quit IRC | 01:03 | |
| *** matsuhashi has joined #openstack-swift | 01:05 | |
| *** mmcardle has quit IRC | 01:06 | |
| *** acoles has joined #openstack-swift | 01:10 | |
| *** ChanServ sets mode: +v acoles | 01:10 | |
| *** haomaiw__ has joined #openstack-swift | 01:12 | |
| *** haomaiwa_ has quit IRC | 01:12 | |
| *** _bluev has joined #openstack-swift | 01:26 | |
| *** nosnos has joined #openstack-swift | 01:29 | |
| *** _bluev has quit IRC | 01:30 | |
| *** changbl has joined #openstack-swift | 01:34 | |
| *** simpleAJ has joined #openstack-swift | 01:35 | |
| simpleAJ | Hi | 01:36 |
| simpleAJ | I am trying to understand interaction between different swift components when I upload / download file | 01:36 |
| simpleAJ | while downloading | 01:36 |
| simpleAJ | http://saio:8080/v1/AUTH_test/test/amey | 01:37 |
| simpleAJ | curl -i http://saio:8080/v1/AUTH_test/test/amey -X GET -H "X-Auth-Token: AUTH_tk5c38b3f3f8204062a61248370d4b9733" | 01:37 |
| simpleAJ | is it talking to object server or proxy server? | 01:37 |
| *** haomaiw__ has quit IRC | 01:47 | |
| *** haomaiwa_ has joined #openstack-swift | 01:47 | |
| clayg | both? | 01:52 |
| clayg | client is talking to proxy, proxy is talking to object server | 01:53 |
| *** mmcardle has joined #openstack-swift | 02:03 | |
| *** simpleAJ has quit IRC | 02:06 | |
| *** haomaiw__ has joined #openstack-swift | 02:06 | |
| *** mmcardle has quit IRC | 02:07 | |
| *** haomaiwa_ has quit IRC | 02:09 | |
| *** zhiyan is now known as zhiyan_ | 02:11 | |
| *** zhiyan_ is now known as zhiyan | 02:12 | |
| *** judd7 has quit IRC | 02:14 | |
| *** praveenkumar has joined #openstack-swift | 02:21 | |
| *** simpleAJ has joined #openstack-swift | 02:25 | |
| *** _bluev has joined #openstack-swift | 02:27 | |
| *** _bluev has quit IRC | 02:31 | |
| *** saurabh_ has joined #openstack-swift | 02:44 | |
| *** saurabh_ has joined #openstack-swift | 02:44 | |
| *** _bluev has joined #openstack-swift | 02:45 | |
| openstackgerrit | Zhang Hua proposed a change to openstack/swift: Add profiling middleware in Swift https://review.openstack.org/53270 | 02:46 |
| *** gyee has quit IRC | 02:47 | |
| saurabh_ | Hi there, I have a question:- During ring balancing after adding a new node, some partitions are moved from existing nodes to new node, is there any partition movement between existing nodes | 02:48 |
| *** _bluev has quit IRC | 02:49 | |
| *** simpleAJ has left #openstack-swift | 02:51 | |
| clayg | they're can be if some of the existing nodes already have positive balance | 02:52 |
| clayg | *their | 02:52 |
| clayg | *there acctually, wow | 02:53 |
| *** fifieldt has joined #openstack-swift | 02:56 | |
| glange | nice :) | 02:56 |
| clayg | glange: fingers moving faster than the brain apparently | 02:57 |
| praveenkumar | Is complete object move to memory when we create/update/delete a object metadata because I am using python-swiftclient to check object creation and metadata operation so when any metadata operation occur, I can see memory usage drastically change? | 02:57 |
| clayg | praveenkumar: so POSTs to objects are really COPYs under the hood - but no, the entire object should never be in memory, and cirtainly the object is never transfered to the client. | 02:59 |
| clayg | You may just be seeing that POST is heavy weight operation, there's a config option to do something different that works pretty ok except with content-type changes and container-sync. | 03:00 |
| clayg | # object_post_as_copy = true | 03:00 |
| clayg | ^ default | 03:00 |
| clayg | object_post_as_copy is sometimes called "fast POST" | 03:01 |
| clayg | er... object_post_as_copy = false is sometimes... | 03:01 |
| clayg | glange: just shut me up already... | 03:01 |
| glange | clayg will be here all week! try the veal :) | 03:02 |
| praveenkumar | clayg: ah so if object_post_as_copy=true then POST's are heavy weight operations, I may need to read more about API options to set it to false instead making change in default config file. | 03:03 |
| clayg | praveenkumar: well that particular option is up to the deployer, it's not really part of the API | 03:03 |
| *** mmcardle has joined #openstack-swift | 03:04 | |
| praveenkumar | clayg: oh, so let assume if deployer set this option true on a box which have not much memory to handle 1Gb's objects POST's operations then he will surly get *defined error* in this case? | 03:06 |
| *** mmcardle has quit IRC | 03:08 | |
| clayg | yeah... i mean I guess there's an amplification with COPY requests, so it's probably more likely to overwhelm a deployment targeting that verb than say PUT. | 03:09 |
| clayg | but ultimately there's not any more memory required to support a PUT than a COPY - so I'm not quite sure what you're seeing | 03:10 |
| clayg | your memeory usage during a POST should look a lot like if your client did that same PUT | 03:10 |
| praveenkumar | clayg: alright, let me paste a sample code and memory change, may be it will help. | 03:11 |
| *** matsuhashi has quit IRC | 03:12 | |
| clayg | praveenkumar: maybe? I'm just saying if you see memory usage "drastially change" from a POST to say a HEAD, that makes sense. But if that memory usage matches roughly what'd you get from PUT then it makes sense, so you'll have to capture both. | 03:14 |
| clayg | a PUT or a COPY - COPY is acctually a little better for that test... | 03:14 |
| praveenkumar | clayg: alright. | 03:14 |
| * praveenkumar needs some time to check what clayg suggested. | 03:15 | |
| *** Guest_ has joined #openstack-swift | 03:16 | |
| *** cheri has joined #openstack-swift | 03:18 | |
| *** Fin1te has joined #openstack-swift | 03:24 | |
| *** Fin1te has quit IRC | 03:25 | |
| *** nosnos has quit IRC | 03:26 | |
| praveenkumar | clayg: so when I am doing a POST operation to create metadata of a object(file.txt) which is 1GB in size I am getting http://fpaste.org/92757/01413513/, so is it the same issue which we are discussing right? | 03:30 |
| clayg | praveenkumar: would have been very helpful to see the before, but the logs are the only place that would really explain why you got a 503 | 03:44 |
| *** _bluev has joined #openstack-swift | 03:45 | |
| *** _bluev has quit IRC | 03:50 | |
| praveenkumar | clayg: http://fpaste.org/92758/13970159/ -> logs snippet for error. | 04:00 |
| *** chandan_kumar has joined #openstack-swift | 04:04 | |
| *** mmcardle has joined #openstack-swift | 04:05 | |
| *** zhiyan is now known as zhiyan_ | 04:06 | |
| clayg | praveenkumar: yeah that's not memory - the 507 is because you don't have the disk to store the intermediate value (COPY will do a GET streamed to PUT followed by a DELETE) | 04:07 |
| clayg | so while the request is in flight you have two copies of the object | 04:07 |
| clayg | 507 == ERROR Insufficient Storage | 04:07 |
| clayg | i guess you really need to have disk to hold six replicas... | 04:08 |
| praveenkumar | clayg: yes right, I checked the mounted disk size and it's 1.1G which is not sufficient, thanks will add more space. | 04:09 |
| *** mmcardle has quit IRC | 04:10 | |
| *** zackf has joined #openstack-swift | 04:12 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/python-swiftclient: Add some bash helpers for auth stuff https://review.openstack.org/86224 | 04:14 |
| clayg | ^ i but all of the it looks like I did into that, feel free to push over it or -1 it so it'll expire or whatever | 04:15 |
| clayg | wow! "I put all of the time it looks like I did into that" - gawd I'm so lazy, I can't even manage to type out a half assed self deprecating joke about how lazy I am... | 04:16 |
| *** saurabh_ has quit IRC | 04:22 | |
| *** matsuhashi has joined #openstack-swift | 04:24 | |
| *** nosnos has joined #openstack-swift | 04:25 | |
| *** ppai has joined #openstack-swift | 04:31 | |
| *** zackf has quit IRC | 04:33 | |
| *** matsuhashi has quit IRC | 04:42 | |
| *** _bluev has joined #openstack-swift | 04:46 | |
| *** matsuhashi has joined #openstack-swift | 04:50 | |
| *** _bluev has quit IRC | 04:51 | |
| *** erlon has quit IRC | 04:52 | |
| *** chandan_kumar has quit IRC | 05:00 | |
| *** mmcardle has joined #openstack-swift | 05:07 | |
| *** mmcardle has quit IRC | 05:11 | |
| *** chandan_kumar has joined #openstack-swift | 05:14 | |
| *** mmcardle has joined #openstack-swift | 05:23 | |
| *** mmcardle1 has joined #openstack-swift | 05:25 | |
| *** mmcardle has quit IRC | 05:25 | |
| *** mmcardle1 has quit IRC | 05:29 | |
| *** zhiyan_ is now known as zhiyan | 05:31 | |
| *** _bluev has joined #openstack-swift | 05:47 | |
| *** _bluev1 has joined #openstack-swift | 05:49 | |
| *** _bluev has quit IRC | 05:49 | |
| *** _bluev1 has quit IRC | 05:53 | |
| *** mlipchuk has joined #openstack-swift | 05:57 | |
| *** ppai has quit IRC | 06:00 | |
| *** Guest_ has quit IRC | 06:05 | |
| *** Guest_ has joined #openstack-swift | 06:06 | |
| *** cheri has quit IRC | 06:08 | |
| *** ppai has joined #openstack-swift | 06:17 | |
| *** ashish_ has joined #openstack-swift | 06:20 | |
| *** psharma has joined #openstack-swift | 06:37 | |
| *** ashish_ has quit IRC | 06:43 | |
| *** haomaiw__ has quit IRC | 06:47 | |
| *** _bluev has joined #openstack-swift | 06:50 | |
| *** _bluev has quit IRC | 06:54 | |
| *** Guest_ has quit IRC | 07:05 | |
| *** Guest_ has joined #openstack-swift | 07:09 | |
| *** haomaiwang has joined #openstack-swift | 07:10 | |
| *** ppai has quit IRC | 07:12 | |
| *** cheri has joined #openstack-swift | 07:15 | |
| *** Guest_ has quit IRC | 07:16 | |
| *** ppai has joined #openstack-swift | 07:24 | |
| *** Gue______ has joined #openstack-swift | 07:26 | |
| *** matsuhashi has quit IRC | 07:29 | |
| *** matsuhashi has joined #openstack-swift | 07:31 | |
| *** nshaikh has joined #openstack-swift | 07:35 | |
| *** matsuhashi has quit IRC | 07:35 | |
| *** Gue______ has quit IRC | 07:39 | |
| *** Gue______ has joined #openstack-swift | 07:39 | |
| *** yuan has joined #openstack-swift | 07:40 | |
| *** matsuhashi has joined #openstack-swift | 07:49 | |
| *** tanee-away is now known as tanee | 07:50 | |
| *** tanee is now known as tanee-away | 07:50 | |
| openstackgerrit | Takashi Kajinami proposed a change to openstack/swift: Add timestamp checking in AccountBroker.is_status_deleted https://review.openstack.org/86248 | 07:50 |
| *** Guest_ has joined #openstack-swift | 07:51 | |
| *** tanee-away is now known as tanee | 07:51 | |
| *** Gue______ has quit IRC | 07:53 | |
| *** Guest_ has quit IRC | 07:53 | |
| *** Guest_ has joined #openstack-swift | 07:54 | |
| *** Guest_ has quit IRC | 07:54 | |
| *** cheri has quit IRC | 08:02 | |
| *** cheri has joined #openstack-swift | 08:02 | |
| openstackgerrit | Takashi Kajinami proposed a change to openstack/swift: Add timestamp checking in AccountBroker.is_status_deleted https://review.openstack.org/86248 | 08:02 |
| *** nacim has joined #openstack-swift | 08:06 | |
| *** gadb has joined #openstack-swift | 08:16 | |
| *** d89 has joined #openstack-swift | 08:17 | |
| *** ashish_ has joined #openstack-swift | 08:21 | |
| gadb | hi. | 08:22 |
| gadb | as far as I know, data can be isolated with the concept of each zones. | 08:24 |
| gadb | but when i upload object then at first it is not. | 08:25 |
| gadb | atfter a while object moved each zone | 08:25 |
| gadb | what demon do that job | 08:26 |
| gadb | ? | 08:26 |
| gadb | object-server ? | 08:26 |
| omame | object-replicator? | 08:26 |
| gadb | i think too, but i see the replicator source code . | 08:26 |
| *** foexle has joined #openstack-swift | 08:26 | |
| gadb | but i can not find logic | 08:26 |
| omame | oh, object-auditor then :) | 08:26 |
| gadb | right? | 08:27 |
| *** _bluev has joined #openstack-swift | 08:27 | |
| gadb | ok. i will be check auditor source code. but | 08:27 |
| gadb | thx. your answer | 08:28 |
| omame | np | 08:28 |
| *** jamie_h has joined #openstack-swift | 08:29 | |
| *** gvernik has joined #openstack-swift | 08:36 | |
| gvernik | hi. does anyone knows why Jenkins failed on test_sleeper (test.unit.obj.test_auditor.TestAuditor) ? (https://review.openstack.org/#/c/85650/) | 08:37 |
| *** nshaikh has left #openstack-swift | 08:42 | |
| *** _bluev1 has joined #openstack-swift | 08:44 | |
| *** _bluev has quit IRC | 08:44 | |
| *** foexle has quit IRC | 08:46 | |
| *** foexle has joined #openstack-swift | 08:47 | |
| *** chandan_kumar has quit IRC | 08:49 | |
| *** _bluev has joined #openstack-swift | 08:57 | |
| *** _bluev1 has quit IRC | 09:00 | |
| *** foexle has quit IRC | 09:01 | |
| *** chandan_kumar has joined #openstack-swift | 09:02 | |
| *** tanee is now known as tanee-away | 09:06 | |
| *** foexle has joined #openstack-swift | 09:08 | |
| *** gadb has quit IRC | 09:10 | |
| *** tanee-away is now known as tanee | 09:15 | |
| *** haomaiw__ has joined #openstack-swift | 09:18 | |
| *** haomaiwang has quit IRC | 09:20 | |
| *** nshaikh has joined #openstack-swift | 09:24 | |
| *** nshaikh has quit IRC | 09:25 | |
| *** haomaiw__ has quit IRC | 09:39 | |
| *** haomaiwa_ has joined #openstack-swift | 09:40 | |
| *** zhiyan is now known as zhiyan_ | 09:43 | |
| *** matsuhas_ has joined #openstack-swift | 09:45 | |
| *** matsuhashi has quit IRC | 09:45 | |
| *** matsuhas_ has quit IRC | 09:45 | |
| *** matsuhashi has joined #openstack-swift | 09:46 | |
| *** ppai has quit IRC | 09:47 | |
| *** haomaiw__ has joined #openstack-swift | 09:55 | |
| *** haomaiwa_ has quit IRC | 09:57 | |
| *** ppai has joined #openstack-swift | 10:00 | |
| *** chandan_kumar_ has joined #openstack-swift | 10:03 | |
| *** foexle has quit IRC | 10:04 | |
| *** foexle has joined #openstack-swift | 10:04 | |
| *** chandan_kumar has quit IRC | 10:06 | |
| *** Trixboxer has joined #openstack-swift | 10:11 | |
| *** matsuhashi has quit IRC | 10:12 | |
| *** ashish_ has quit IRC | 10:14 | |
| *** matsuhashi has joined #openstack-swift | 10:16 | |
| *** tanee is now known as tanee-away | 10:17 | |
| *** tanee-away is now known as tanee | 10:20 | |
| *** jamie_h has quit IRC | 10:31 | |
| *** jamie_h has joined #openstack-swift | 10:31 | |
| *** mkollaro has joined #openstack-swift | 10:31 | |
| *** chandan_kumar_ has quit IRC | 10:32 | |
| openstackgerrit | A change was merged to openstack/swift: Add missing constraints to /info https://review.openstack.org/86015 | 10:32 |
| *** praveenkumar has quit IRC | 10:33 | |
| *** mmcardle1 has joined #openstack-swift | 10:38 | |
| *** jamie_h has quit IRC | 10:41 | |
| *** jamie_h has joined #openstack-swift | 10:46 | |
| *** dosaboy has quit IRC | 10:49 | |
| *** dosaboy has joined #openstack-swift | 10:50 | |
| *** chandan_kumar_ has joined #openstack-swift | 10:58 | |
| *** matsuhashi has quit IRC | 11:05 | |
| *** matsuhashi has joined #openstack-swift | 11:05 | |
| *** RockKuo has quit IRC | 11:09 | |
| openstackgerrit | Alistair Coles proposed a change to openstack/swift: Add tests and comments re constraints in /info https://review.openstack.org/86286 | 11:15 |
| acoles | cschwede: notmyname: ^^ tweaks we discussed re constraints in /info. original patch got approved before i managed to update, sorry should have WIP'd | 11:17 |
| *** matsuhashi has quit IRC | 11:19 | |
| _bluev | we're having some issues with Gerrit / Jenkins - failures with the check-grenade-dsvm tests. can anyone help with that ? | 11:27 |
| _bluev | https://review.openstack.org/#/c/85453/ | 11:27 |
| _bluev | Its's related to a new dependancy on python-swiftclient that check-grenade-dsvm seems to be unaware of | 11:28 |
| portante | acoles: will look at it in a bit | 11:33 |
| acoles | portante: thanks | 11:34 |
| *** matsuhashi has joined #openstack-swift | 11:36 | |
| *** matsuhashi has quit IRC | 11:39 | |
| *** mmcardle1 has quit IRC | 11:42 | |
| *** mmcardle has joined #openstack-swift | 11:43 | |
| *** praveenkumar has joined #openstack-swift | 11:44 | |
| *** joeljwright has joined #openstack-swift | 11:49 | |
| *** ppai has quit IRC | 11:50 | |
| *** foexle has quit IRC | 11:56 | |
| *** foexle has joined #openstack-swift | 11:58 | |
| *** zhiyan_ is now known as zhiyan | 12:10 | |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Update swift-container-info to be storage policy aware https://review.openstack.org/86304 | 12:15 |
| *** erlon has joined #openstack-swift | 12:19 | |
| openstackgerrit | A change was merged to openstack/swift: Support for content-length in the upload object method for internal client. https://review.openstack.org/85650 | 12:34 |
| *** nacim has quit IRC | 12:37 | |
| *** JuanManuelOlle has joined #openstack-swift | 12:37 | |
| *** nacim has joined #openstack-swift | 12:57 | |
| *** psharma has quit IRC | 13:03 | |
| *** JuanManuelOlle has quit IRC | 13:04 | |
| *** tdasilva has joined #openstack-swift | 13:13 | |
| *** nosnos has quit IRC | 13:14 | |
| *** cheri has quit IRC | 13:21 | |
| *** chandan_kumar_ has quit IRC | 13:46 | |
| *** changbl has quit IRC | 13:47 | |
| *** joeljwright has quit IRC | 13:51 | |
| *** chandan_kumar_ has joined #openstack-swift | 13:53 | |
| *** joeljwright has joined #openstack-swift | 13:55 | |
| *** mmcardle has quit IRC | 13:57 | |
| *** gvernik has quit IRC | 14:07 | |
| *** Longgeek_ has joined #openstack-swift | 14:14 | |
| *** mmcardle has joined #openstack-swift | 14:24 | |
| *** zaitcev has joined #openstack-swift | 14:24 | |
| *** ChanServ sets mode: +v zaitcev | 14:24 | |
| *** saju_m has joined #openstack-swift | 14:26 | |
| *** bill_az has joined #openstack-swift | 14:28 | |
| *** joeljwright has quit IRC | 14:29 | |
| *** bill_az_ has joined #openstack-swift | 14:33 | |
| *** joeljwright has joined #openstack-swift | 14:33 | |
| *** bill_az has quit IRC | 14:33 | |
| *** mmcardle has quit IRC | 14:36 | |
| *** erlon has quit IRC | 14:42 | |
| *** piyush1 has joined #openstack-swift | 14:43 | |
| *** zackf has joined #openstack-swift | 14:47 | |
| *** byeager has joined #openstack-swift | 14:49 | |
| *** saju_m has quit IRC | 14:58 | |
| *** Longgeek_ has quit IRC | 14:58 | |
| *** mmcardle has joined #openstack-swift | 15:00 | |
| *** lpabon has joined #openstack-swift | 15:02 | |
| *** nacim has quit IRC | 15:23 | |
| *** nacim has joined #openstack-swift | 15:32 | |
| notmyname | _bluev: I'm not sure why that has a problem. I'll ask around | 15:34 |
| tdasilva | notmyname: good morning | 15:36 |
| notmyname | tdasilva: good morning | 15:36 |
| tdasilva | notmyname: I remember a few weeks ago in during a openstack-meeting it was decided that icehouse will be released without SP and that SP would be released shortly after, is that still the plan? | 15:37 |
| creiht | for some measure of "shortly" :) | 15:37 |
| tdasilva | lol | 15:37 |
| tdasilva | is there an estimate? | 15:37 |
| *** mwstorer has joined #openstack-swift | 15:38 | |
| notmyname | tdasilva: yes. I'm hoping for the first SP patches to be proposed to master next week. clayg and peluse are working hard on it, and we should also have an update from them at today's meeting | 15:38 |
| tdasilva | notmyname: thanks! | 15:40 |
| notmyname | also, the current plan is to have a series of patches with dependencies proposed, block the first from landing, get the whole series approved, then unblock the first, thus allowing them to all land temporally close. | 15:42 |
| *** mmcardle has quit IRC | 15:42 | |
| notmyname | but that's just the "best idea that's been proposed so far". there are certainly other ways to do it | 15:42 |
| *** mmcardle has joined #openstack-swift | 15:43 | |
| *** zhiyan has quit IRC | 15:45 | |
| tdasilva | notmyname: ok, this makes sense to me. | 15:45 |
| tdasilva | notmyname: at what point would we stop using the ec feature branch? | 15:45 |
| tdasilva | notmyname: right after this first batch of patches? | 15:46 |
| portante | creiht: you around? | 15:46 |
| notmyname | tdasilva: the patches proposed to master will not be from the feature/ec branch (ie not a merge from that branch). the patch proposed to master will be in better logical chunks with a cleaner history | 15:46 |
| creiht | portante: what's up? | 15:47 |
| notmyname | tdasilva: at that point, feature/ec will at least be reset. we need to discuss as a group if the long-running feature branch is a good idea or not | 15:47 |
| creiht | the review about moving the functions? | 15:47 |
| portante | yes | 15:48 |
| portante | you can read my mind | 15:48 |
| portante | scary | 15:48 |
| portante | what was that movie, scanners? | 15:48 |
| creiht | heh | 15:48 |
| portante | did my comments resonate with you? | 15:49 |
| creiht | I get what you are saying | 15:50 |
| tdasilva | notmyname: thanks for the comments, grabbing some lunch now, but will be back for the meeting | 15:50 |
| portante | creiht: thanks, so did it sway your position, or should I rework what I am doing for solving the in-process functional tests problem? | 15:52 |
| creiht | so it still isn't clear to me if this is required to solve another issue? | 15:52 |
| creiht | or is it purely for form | 15:53 |
| creiht | portante: I guess I would prefer it to be in one patch just so it is more clear why we are doing things | 15:55 |
| creiht | otherwise it just feels like we are moving code around for moving code around's sake | 15:55 |
| creiht | but that's just me :) | 15:56 |
| creiht | and my allergies have been killing me, so it might make me tempermental :) | 15:56 |
| *** judd7 has joined #openstack-swift | 15:58 | |
| *** mlipchuk has quit IRC | 15:59 | |
| anticw | redbo: debian recent, i can try with debug yes | 15:59 |
| notmyname | _bluev: ok, I have an answer for you | 16:01 |
| *** gvernik has joined #openstack-swift | 16:01 | |
| notmyname | _bluev: short version: you need to add the futures dependency to the stable/havana branch of the openstack requirements repo | 16:02 |
| notmyname | if python-swiftclient adds a new dependency on master, then that dependency also must be backported to the supported stable branch of the requirements repo | 16:02 |
| *** zhiyan has joined #openstack-swift | 16:03 | |
| *** changbl has joined #openstack-swift | 16:05 | |
| notmyname | conversation explaining this for anyone who's interested: https://gist.github.com/anonymous/b366b2d078290777858b | 16:06 |
| *** gvernik has quit IRC | 16:07 | |
| *** foexle has quit IRC | 16:08 | |
| *** chandan_kumar_ has quit IRC | 16:08 | |
| notmyname | thus resulting in python-*client's master branch being usable by still-supported stable openstack integrated releases | 16:09 |
| _bluev | notmyname: thanks :-) | 16:13 |
| *** gyee has joined #openstack-swift | 16:17 | |
| *** byeager has quit IRC | 16:18 | |
| *** tanee is now known as tanee-away | 16:18 | |
| *** dmb__ has joined #openstack-swift | 16:27 | |
| portante | creiht: I can combine the eventlet patch with the move of the functions to the individual tests | 16:28 |
| portante | do you think I just combine that with the in-process test changes as well? | 16:28 |
| portante | or is that better kept separately? | 16:28 |
| portante | so you have allergies to code movement? ;) | 16:28 |
| *** tanee-away is now known as tanee | 16:29 | |
| *** tanee is now known as tanee-away | 16:29 | |
| *** tanee-away is now known as tanee | 16:29 | |
| creiht | portante: that doesn't matter as much to me | 16:35 |
| portante | okay, I'll update in a bit | 16:35 |
| *** byeager has joined #openstack-swift | 16:37 | |
| zaitcev | cschwede: seen this https://blueprints.launchpad.net/swift/+spec/container-alias | 16:38 |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: In-process swift server for functional tests https://review.openstack.org/66108 | 16:39 |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: Use eventlet instead of threading for timeout https://review.openstack.org/85782 | 16:39 |
| zaitcev | portante: ouch, I'm pointing out a mis-spelling | 16:41 |
| *** mmcardle has quit IRC | 16:41 | |
| portante | zaitcev: where? | 16:41 |
| zaitcev | portante: I'll post review in a moment | 16:42 |
| portante | k | 16:42 |
| portante | thx | 16:42 |
| zaitcev | but I'm a very shallow reviewer | 16:42 |
| zaitcev | can't imagine why John invited me to core | 16:42 |
| zaitcev | eventlet.hubs.use_hub(get_hub()) -- I see this construct all the time but why is this not the default in eventlet | 16:43 |
| *** pberis has quit IRC | 16:44 | |
| portante | there is a description of why in utils, I believe | 16:44 |
| portante | creiht, redbo, other from rackspace might remember, but epoll had some kind of one in a million billion hang, that was very hard to debug | 16:45 |
| *** joeljwright has quit IRC | 16:45 | |
| zaitcev | "This prevents strange race conditions for those who want to use both Twisted and Eventlet separately." -- oh god you crazy ecosystem you | 16:45 |
| redbo | It was losing a socket close event once every gazillion times under load, and when that socket's file descriptor got reused, eventlet freaked out because it was still waiting for an event from it in another coro. But that was forever ago and hasn't been retested. | 16:49 |
| zaitcev | Thanks | 16:49 |
| openstackgerrit | A change was merged to openstack/swift: Add tests and comments re constraints in /info https://review.openstack.org/86286 | 16:54 |
| *** ashish_ has joined #openstack-swift | 16:54 | |
| zaitcev | You know what really happened, BTW. I saw the get_hub() and somehow misthought it as eventlet's function by the same name, but we actually refer to our function in swift.common.utils. | 16:55 |
| zaitcev | maybe just put 'selects' there since it's just the test and does not need 10,000 descriptors. | 16:56 |
| zaitcev | Switched to branch "review/peter_portante/inprocfunc" | 16:56 |
| zaitcev | [zaitcev@guren swift-fetch]$ ./.functests | 16:56 |
| zaitcev | SSSSSSSSSSSSSSEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE........S.... | 16:56 |
| zaitcev | uh-oh | 16:56 |
| portante | are you on a SAIO? | 16:57 |
| zaitcev | no | 16:57 |
| zaitcev | well, it's a tempauth, so outght to be like SAIO | 16:57 |
| portante | let me know what the error messages say | 16:57 |
| *** nacim has quit IRC | 16:59 | |
| *** piousbox has joined #openstack-swift | 17:04 | |
| *** piyush1 has quit IRC | 17:07 | |
| *** shri has joined #openstack-swift | 17:12 | |
| *** byeager has quit IRC | 17:13 | |
| *** csd has joined #openstack-swift | 17:13 | |
| *** piousbox has quit IRC | 17:19 | |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: Base class for storage servers https://review.openstack.org/86109 | 17:20 |
| *** tanee is now known as tanee-away | 17:21 | |
| *** chandan_kumar_ has joined #openstack-swift | 17:24 | |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: In-process swift server for functional tests https://review.openstack.org/66108 | 17:27 |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: Use eventlet instead of threading for timeout https://review.openstack.org/85782 | 17:27 |
| *** chandan_kumar_ has quit IRC | 17:34 | |
| *** zhiyan is now known as zhiyan_ | 17:37 | |
| *** d89 has quit IRC | 17:41 | |
| *** byeager has joined #openstack-swift | 17:44 | |
| *** chandan_kumar_ has joined #openstack-swift | 17:47 | |
| *** byeager has quit IRC | 17:52 | |
| *** byeager has joined #openstack-swift | 17:56 | |
| *** ashish_ has quit IRC | 17:56 | |
| *** JuanManuelOlle has joined #openstack-swift | 18:05 | |
| *** lpabon has quit IRC | 18:06 | |
| notmyname | 50 minutes until the team meeting | 18:09 |
| *** piyush1 has joined #openstack-swift | 18:11 | |
| *** piyush2 has joined #openstack-swift | 18:12 | |
| *** piyush1 has quit IRC | 18:15 | |
| *** gvernik has joined #openstack-swift | 18:19 | |
| *** jamie_h has quit IRC | 18:22 | |
| *** shri1 has joined #openstack-swift | 18:22 | |
| *** shri has quit IRC | 18:23 | |
| *** scohen has joined #openstack-swift | 18:25 | |
| *** ashish_ has joined #openstack-swift | 18:28 | |
| ashish_ | Hey everyone could anyone tell me in which directory in ubuntu do I need to run the ssbench commands. | 18:29 |
| swifterdarrell | ashish_: unless you use relative paths to files given on the command-line, it shouldn't matter | 18:32 |
| ashish_ | swiftdarrel I did not get your point | 18:33 |
| ashish_ | swiftdarrel I tried got the following error. http://paste.openstack.org/show/75441/ | 18:35 |
| swifterdarrell | ashish_: that's wild! The command you're after is probably ssbench-master. See https://github.com/swiftstack/ssbench#example-simple-single-server-run for some examples | 18:37 |
| swifterdarrell | ashish_: also that whole README is filled with super-relevant information | 18:37 |
| cschwede | zaitcev: thanks (for the container-alias blueprint). Yes, I thought it would be more simple :) Thinking how to continue on this. | 18:38 |
| ashish_ | swiftdarrel Thanks.Please give me a minute to try. | 18:39 |
| zaitcev | cschwede: I am thinking about some reconciliation rules... Like losing alias if a container ever turns up, even tombstoned. But honestly I did not think enough. | 18:39 |
| *** shri1 has quit IRC | 18:41 | |
| cschwede | zaitcev: i’ll think more about this and will collect some ideas in the blueprint. so far one of the challenges is how to notify the user in case something went wrong; ie destination is missing | 18:44 |
| wer | my deletes are getting slower and slower :) is it worth using db_preallocation for my containers if I have massive amounts of deletes? Or in general? | 18:44 |
| ashish_ | swiftdarrel I tried but got the following error http://paste.openstack.org/show/75442/ | 18:46 |
| creiht | wer: db_preallocation is used to prevent file fragmentation of the db on spinning disk | 18:51 |
| wer | creiht: think that would help my things getting slower by the day? I suspect fragmentation cause I am not increasing the number of object :/ | 18:52 |
| wer | basically I know I have a hotspot on that container... And I am going to shard it. But I don't like that it gets slower each day :/ And was looking for anything else that might be a problem. | 18:53 |
| creiht | well if it is already fragmented, turning it on isn't going to change anything for that db | 18:54 |
| creiht | you could try rebuilding the db to a new file to create a non-fragmented copy of it | 18:54 |
| *** jamie_h has joined #openstack-swift | 18:54 | |
| wer | creiht: oh! So I am creating a brand new cluster. which I will migrate to. | 18:54 |
| creiht | ahh | 18:54 |
| creiht | if you have your containers on spinning disks, then you do want preallocation enabled | 18:54 |
| creiht | if you are using ssds, you don't | 18:55 |
| ashish_ | cschwede Could help me with a ssbench error http://paste.openstack.org/show/75442/ | 18:55 |
| wer | makes sense. thanks again creiht :) | 18:55 |
| *** csd has quit IRC | 18:56 | |
| wer | does preallocation happen when I create the container? IE if I enable that, and restart the container-server, and remove readd the container will that do what I think it does? | 18:56 |
| creiht | yes | 18:56 |
| wer | ossm! | 18:56 |
| creiht | You could remove one copy | 18:57 |
| creiht | actually | 18:57 |
| creiht | it would likely rsync it over | 18:57 |
| creiht | if you delete one and run replication | 18:57 |
| creiht | I think | 18:57 |
| creiht | :) | 18:57 |
| wer | interesting. | 18:57 |
| creiht | wer you can use some xfs command line tools to see how fragmented the file is | 18:59 |
| creiht | I think xfs_bmap maybe | 18:59 |
| notmyname | meeting time in #openstack-meeting | 18:59 |
| creiht | yeah you can use that to see what blocks a file occupies | 19:00 |
| creiht | will at least know if it is fragmented, how bad, and if anything you do improves that | 19:00 |
| wer | derp. is it swift-get-nodes to find the actual database. | 19:00 |
| cschwede | ashish_: hmm, some local permission problems? tried to run it with sudo? | 19:00 |
| ashish_ | cschwede Thanks.I will give it a try. | 19:01 |
| creiht | wer: yes | 19:01 |
| wer | totally doing this. | 19:01 |
| clayg | ohai | 19:03 |
| *** peterbusque has joined #openstack-swift | 19:03 | |
| *** swills has joined #openstack-swift | 19:03 | |
| swills | so how crazy would it be to put swift data on ZFS? | 19:03 |
| *** mlipchuk has joined #openstack-swift | 19:04 | |
| wer | gah this container is just very large. | 19:05 |
| *** byeager has quit IRC | 19:06 | |
| ashish_ | cschwede sudo resolves it now I ge the following error.Do I need to pass the storage tokan as well. ERROR:http://paste.openstack.org/show/75444/ | 19:06 |
| *** csd has joined #openstack-swift | 19:06 | |
| *** chandan_kumar_ has quit IRC | 19:07 | |
| cschwede | ashish_: please use „http://127.0.0.1:8080/auth/v1.0“ for the url | 19:07 |
| ashish_ | cschwede Thanks.That resolves the error But now I got a different error. http://paste.openstack.org/show/75446/ | 19:10 |
| wer | creiht: I didn't follow you on the whole delete and then replicate idea. | 19:12 |
| *** dmb__ has quit IRC | 19:12 | |
| wer | 95 chunks in this fragmentation btw. It's 2.5 G | 19:13 |
| cschwede | ashish_: are your credentials correct? accountname is admin, username is admin and secret also? | 19:13 |
| ashish_ | cschwede the credentials are correct. | 19:14 |
| cschwede | ashish_: your url is wrong: http://127.0.0.1:8080/v1.0/ - should be http://127.0.0.1:8080/auth/v1.0/ | 19:14 |
| creiht | wer: if you delete the container db, container replication will rsync one of the containers from another server to the one you deleted it from | 19:16 |
| creiht | hopefully with less fragmentation | 19:16 |
| *** mlipchuk has quit IRC | 19:17 | |
| ashish_ | cschwede thanks a lot now my ssbench works perfectly | 19:17 |
| cschwede | ashish_: you’re welcome, nice that it works for you now! | 19:18 |
| ashish_ | cschwede are the containers and objects in my admin account being tested in this benchmark | 19:19 |
| wer | creiht: that make me nervous. I had strange results early on with deleting things. I could "change" things and they got fixed, via replication, but I have deleted things only to never have them get replaced..... that was with object replication so maybe I didn't wait long enough :/ | 19:19 |
| creiht | yeah object replication is a little different | 19:20 |
| *** chandan_kumar_ has joined #openstack-swift | 19:20 | |
| creiht | I would also stop the deletes while you do that if you can | 19:20 |
| creiht | just trying to brainstorm easy ways to rebuild the db | 19:20 |
| wer | yeah I'm all curious now :) I 'll have to try that once I'm in a better position. | 19:21 |
| swifterdarrell | ashish_: (sorry for the delay; stepped otu for lunch) | 19:21 |
| creiht | wer: actually what would be easier | 19:22 |
| creiht | 1.) stop your deletes | 19:22 |
| creiht | 2.) stop one of the container servers | 19:22 |
| swifterdarrell | ashish_: that code path gets hit when SUDO_UID and SUDO_GID are in the environment; is that true and your current user is not the superuser? | 19:22 |
| creiht | 3.) mv the container db file, and then copy it back to the original name | 19:22 |
| creiht | that should give you a less fragmented file | 19:23 |
| ashish_ | swiftdarrel no issues cschwede got all my doubts solved. cschwede Thanks. | 19:23 |
| wer | derp. Is that just xfs magic? | 19:23 |
| swifterdarrell | ashish_: sweet! | 19:23 |
| wer | Or just new better "creation" versus all the updating. | 19:23 |
| *** ryanez has joined #openstack-swift | 19:25 | |
| creiht | well when it creates the new file, it creates it all in one continuous or near continuous block | 19:26 |
| _bluev | is anyone aware of a reason why the content-length in a GET proxy log message would be less than the size of the object. The status of the message is 200. User reported a truncated file. We are running Grizzly 1.8.0 | 19:27 |
| ryanez | Hi installed swift on a vm according to http://pastebin.com/7umP7EB6. I used a loopback device. Is there a guide for stopping swift, resize the loopback devic, and restarting swift? What portions of the install do I have to redo? | 19:28 |
| wer | right. The copy. A rebalance on the container could do it too probably.... hrm. | 19:28 |
| creiht | _bluev: sounds likely that it got uploaded truncated | 19:29 |
| ashish_ | swiftdarrel .Now I am able to run the ssbench.But I have a doubt are the objects in my swift actually being tested by ssbench | 19:29 |
| _bluev | creiht: I should have said - the filename matched in historic logs shows a larger size than the later 200 GET responce | 19:30 |
| notmyname | _bluev: creiht: or perhaps a bad replica was loaded from disk and quarantined after the read. I don't remember what status code that used to be logged as | 19:30 |
| wer | creiht: think that may be an OK way to unfragment occasionally? Just rebalance the container ring? I don't have a ton of containers... and I follow the same giant ring structure..... | 19:30 |
| creiht | wer: yeah dunno, could be worth a try | 19:31 |
| creiht | ryanez: you can start/stop swift with swift-init all start/stop | 19:31 |
| creiht | or swift-init main start/stop if you aren't running the background services | 19:32 |
| creiht | re-create the loopback device the saize you want | 19:32 |
| creiht | run the resetswift script | 19:32 |
| creiht | and you should be good to go | 19:32 |
| ryanez | Awesome! Thanks creiht! | 19:33 |
| creiht | that will delete any current data you have | 19:33 |
| creiht | just to be clear :) | 19:33 |
| creiht | ryanez: -^ | 19:33 |
| ryanez | no problem. This is just for a docker container | 19:33 |
| creiht | _bluev: yeah not sure off the top of my head then, first step might be to look at the three replicas to see if they are all the same, and see if they are correct or truncated | 19:34 |
| *** openstackgerrit has quit IRC | 19:34 | |
| creiht | swills: some have mused about running swift on zfs | 19:34 |
| creiht | don't think it would be crazy, but might take a little work | 19:34 |
| swills | creiht: oh? i kinda have it working already | 19:36 |
| _bluev | creiht notmyname thanks guys - will check both of those possibilities | 19:36 |
| creiht | swills: well there you go then :) | 19:37 |
| swills | creiht: main thing was telling it not to check mountpoint, although technically i could have made it a mountpoint... | 19:37 |
| swills | creiht: my question was more about performance and efficiency | 19:37 |
| creiht | swills: I don't think anyone has gotten much further than you | 19:37 |
| swills | creiht: scary. ;) | 19:37 |
| swills | yeah, i actually have it running in jails on zfs on FreeBSD | 19:38 |
| creiht | hehe | 19:38 |
| swills | so that's fun... | 19:38 |
| creiht | well good to hear that it works | 19:38 |
| swills | but i was asking because of the stuff i was reading about using xfs and such | 19:38 |
| swills | and there being code to check that files aren't corrupt, which zfs also does for you | 19:39 |
| creiht | right | 19:39 |
| swills | all i really did to get it working was install the port http://www.freshports.org/databases/py-swift/ | 19:39 |
| swills | and read the docs, configure, get it going, etc. | 19:39 |
| creiht | oh nice :) | 19:39 |
| swills | there's the client too http://www.freshports.org/databases/py-swiftclient/ | 19:39 |
| swills | anyway, i was just wondering, how much do people really stick to all that stuff about non-redunant disks, xfs, etc.? | 19:40 |
| swills | cause i have a need for a deployment with many disparate nodes but not a ton of data | 19:40 |
| swills | so i was thinking a smaller number of more redundant nodes | 19:41 |
| swills | and maybe performance doesn't matter all that much because it's all cached in ram anyway, etc. | 19:41 |
| creiht | swills: well you are in uncharted territory, but I'm sure there are others that would like to hear more as you go | 19:41 |
| swills | ok. :) | 19:42 |
| *** gvernik has quit IRC | 19:42 | |
| *** openstackgerrit has joined #openstack-swift | 19:42 | |
| creiht | I think the team from softlayer did a lot of work to get swift to work on freebsd, but I think they ended up switching to linux just because it was easier | 19:42 |
| creiht | hopefully they might be able to chime in | 19:42 |
| swills | ok | 19:42 |
| swills | i wonder if there was any real work involved, or just removing linuxisms | 19:43 |
| swills | "omg, eth0!" | 19:43 |
| creiht | lol | 19:43 |
| creiht | I don't remember exactly | 19:43 |
| creiht | there is also certainly opportunity to optimize swift to work better with freebsd/zfs | 19:44 |
| creiht | and hopefully that will get easier over time with the disk abstraction | 19:44 |
| swills | sure | 19:44 |
| creiht | so the xfs specific bits could have a zfs equiv plugin | 19:45 |
| swills | what are the xfs specific bits? | 19:45 |
| zaitcev | the zero-file is the most important | 19:47 |
| swills | (in case it wasn't clear, my jail/zfs implementation was for testing, i'm kinda trying to figure out if it's viable for prod) | 19:47 |
| creiht | yeah so for example, if xfs get corrupted it creates a zero-byte file | 19:48 |
| creiht | the auditor can do a quick scan for zero-byte files, and validate them | 19:48 |
| zaitcev | there's a patch to auditor to work around it, but it was stalled for a while | 19:48 |
| swills | filesystem corruption, interesting... ;) | 19:49 |
| creiht | lol | 19:49 |
| zaitcev | https://review.openstack.org/11452 | 19:49 |
| *** scohen has quit IRC | 19:49 | |
| creiht | swills: I know that *never* happens on zfs ;) | 19:49 |
| *** chandan_kumar_ has quit IRC | 19:49 | |
| zaitcev | yeah, just the whole node fails to reboot because it can't get a pool to start | 19:49 |
| creiht | but that said, some of the process could be written in a way to take advantage of zfs' characteristics | 19:49 |
| swills | *nod* | 19:50 |
| creiht | I think things should mostly work | 19:50 |
| wer | creiht: no fragmentation now. I'll see if the deletes improve. | 19:50 |
| creiht | wer: cool | 19:50 |
| wer | yeah neat :) | 19:50 |
| *** scohen has joined #openstack-swift | 19:50 | |
| clayg | I don't think i can acctually get tox tests to run on python-swiftclient right now? (tox.ConfigError: ConfigError: substitution key 'posargs' not found) | 19:51 |
| *** byeager has joined #openstack-swift | 19:51 | |
| creiht | swills: some things are written in a way that follows how xfs works | 19:52 |
| zaitcev | I think auditor is the only one that matters. | 19:52 |
| swills | hmm, ok, like what? could i expect anything to break? | 19:52 |
| creiht | swills: not likely | 19:52 |
| creiht | but for example | 19:52 |
| swills | i know nothing of xfs, fwiw | 19:52 |
| creiht | file metadata writes in xfs are atomic | 19:52 |
| zaitcev | Everything else works perfectly on ext4 | 19:53 |
| creiht | so we don't fsync when doing writes to metadata things (like setting xattrs since they are stored in the inode) | 19:53 |
| *** ryanez has left #openstack-swift | 19:53 | |
| creiht | swills: yeah these things are subtle | 19:53 |
| swills | great, fsync is the devil. | 19:53 |
| creiht | heh | 19:53 |
| creiht | but we do fsync the writes of the objects | 19:54 |
| swills | oh, dang | 19:54 |
| creiht | we also try to prealloc the size of the object if it is available | 19:54 |
| creiht | so that it doesn't cause fragmentation in xfs | 19:54 |
| swills | shouldn't have any effect on zfs, i think | 19:54 |
| creiht | swills: but all of that said, things should mostly work | 19:55 |
| creiht | if they don't, please let us know :) | 19:55 |
| swills | *nod* they seem to | 19:55 |
| creiht | it would also be interesting to hear how performance goes for you | 19:55 |
| swills | oh, it's aweful. :) | 19:55 |
| creiht | and if you run into problems, let us know | 19:55 |
| creiht | heh | 19:55 |
| swills | but that's because it's all just running on my desktop for testing | 19:55 |
| creiht | oh | 19:56 |
| creiht | hehe | 19:56 |
| swills | all the jails are on one box on one set of mirrored zfs disks | 19:56 |
| creiht | hah | 19:56 |
| *** byeager has quit IRC | 19:56 | |
| swills | so, you know, low expectations | 19:56 |
| ashish_ | swiftdarrel I have objects in the range 1 -16 kb ie for only tiny but still i get the operations on medium and small.Even though I have none of the objects in their range | 19:56 |
| swills | but it seemed functional | 19:56 |
| creiht | swills: if you can run the functional and probe tests, then you are doing pretty good | 19:57 |
| swills | tests! that's a good idea! how do i do that? | 19:57 |
| ashish_ | swiftdarrel this is my test run output http://paste.openstack.org/show/75447/.Please explain | 19:57 |
| swills | i just tested with the client uploading some files, downloading them, etc. | 19:57 |
| *** piousbox has joined #openstack-swift | 19:57 | |
| creiht | swills: which instructions did you follow to set up? | 19:58 |
| clayg | hrmm... tox version 1.6 seems to be working better | 19:58 |
| swills | http://docs.openstack.org/developer/swift/howto_installmultinode.html | 19:58 |
| swills | mostly those, i think | 19:58 |
| swills | with a detour into some of the others to figure some things out | 19:59 |
| creiht | swills: in the swift codebase, there is a swift/test/sample.conf | 20:00 |
| creiht | (not sure where that ends up in the freebsd install) | 20:00 |
| creiht | copy that to /etc/swift/test.conf | 20:00 |
| creiht | of course translate my linuxisms to freebsdisms :) | 20:01 |
| swills | it's mostly prefixing everythign with /usr/local | 20:01 |
| swills | :) | 20:01 |
| creiht | edit that file to make sure the auth endpoint is correct, and create the test users needed | 20:01 |
| creiht | then you should be able to run the function tests which are /swift/.functests | 20:02 |
| creiht | that will go a long ways to testing stuff | 20:02 |
| creiht | http://docs.openstack.org/developer/swift/development_saio.html | 20:02 |
| creiht | that describes a bit of a simpler setup that most of us dev on | 20:02 |
| swills | ok, for auth, i don't have keystone | 20:02 |
| swills | let me see... | 20:02 |
| creiht | and auto sets up stuff for testing | 20:03 |
| *** byeager has joined #openstack-swift | 20:03 | |
| creiht | swills: what are you using for auth? | 20:03 |
| creiht | the default settings are setup for the simpleauth | 20:03 |
| swills | tempauth | 20:03 |
| creiht | or tempauth | 20:03 |
| swills | that user_system_root = testpass .admin stuff | 20:04 |
| creiht | sorry name has changed a couple of times :) | 20:04 |
| creiht | yeah | 20:04 |
| creiht | if you search for the proxy-server.conf on that page | 20:04 |
| creiht | under filter:tempauth | 20:05 |
| creiht | if you copy all of those over, then you should have all the test accounts needed for default settings | 20:05 |
| swills | ok | 20:05 |
| creiht | swills: I *think* at that point, you should be able to run the .functests | 20:08 |
| creiht | I don't think you will be able to run the probe tests with your current setup | 20:09 |
| swills | i can run the tests from a linux VM or something if necessary | 20:10 |
| *** praveenkumar has quit IRC | 20:10 | |
| creiht | I don't think it should be necessary | 20:10 |
| *** openstackstatus has quit IRC | 20:15 | |
| swills | ok, finally got it all running again... | 20:15 |
| swills | i did this like a month ago as proof of concept, didn't get anywhere really as far as $work and hardware goes... so I shut it down... now i've got the go ahead so i'm figuring a few things out, need to start things back up, etc. | 20:16 |
| *** openstackstatus has joined #openstack-swift | 20:16 | |
| *** scohen has quit IRC | 20:18 | |
| swills | reading this dev doc, makes me wish i wrote some notes... | 20:21 |
| *** ashish_ has quit IRC | 20:21 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/python-swiftclient: Add some bash helpers for auth stuff https://review.openstack.org/86224 | 20:23 |
| *** scohen has joined #openstack-swift | 20:23 | |
| *** Mikalv has joined #openstack-swift | 20:38 | |
| openstackgerrit | Peter Portante proposed a change to openstack/swift: Base class for storage servers https://review.openstack.org/86109 | 20:38 |
| *** bmac423 has joined #openstack-swift | 20:40 | |
| *** lpabon has joined #openstack-swift | 20:40 | |
| bmac423 | Can anyone help with swift-informant? I'm not seeing the stats in graphite, although I seem to have everything configured correctly per the openstack cookbook. Anything I should look for? | 20:42 |
| _bluev | is the content-length for a GET in the object server main log the actual number of byte read from the disk, or is it the metadate content-length ? | 20:44 |
| _bluev | bmac423: do you have a statsd server up and running ? | 20:45 |
| bmac423 | I do | 20:45 |
| bmac423 | I can manually send stats to it | 20:45 |
| bmac423 | e.g. echo "foo:1|c" | nc -u 127.0.0.1 8125 | 20:48 |
| *** G________ has joined #openstack-swift | 20:50 | |
| *** lpabon has quit IRC | 20:50 | |
| bmac423 | test passes | 20:51 |
| bmac423 | Ran 15 tests in 0.013s | 20:51 |
| bmac423 | swift-informant-master/test/test_informant.py | 20:52 |
| pandemicsyn | bmac423: if you tcpdump on the stats port or run your version of statsd in a verbose/debug mode do you see any udp traffic from your proxy ? | 20:53 |
| *** tdasilva has left #openstack-swift | 20:54 | |
| openstackgerrit | A change was merged to openstack/swift: Set permissions on generated ring files https://review.openstack.org/85546 | 20:55 |
| *** G________ has quit IRC | 20:56 | |
| bmac423 | pandemicsyn: tcpdump is silent unless I manually send something | 20:57 |
| pandemicsyn | weird, and you've got it in your pipeline near the front and specified a port and ip right ? | 21:01 |
| bmac423 | it's currently last in the pipeline, does that make a difference? | 21:01 |
| peluse | clayg: you there? | 21:02 |
| pandemicsyn | last , like last in the list ? that'll definitely cause a problem, because the requests wont ever actually reach informant | 21:02 |
| bmac423 | ha, I see. Let me move it up to the front and give it a shot | 21:03 |
| _bluev | Please can anyone help me to understand these log entries from the time a user received a truncated file. First the PUT. Then the doomed GET | 21:03 |
| _bluev | Apr 9 15:56:56 SM-X9DBL36B-S-3-PWP71-GB object-server 10.32.37.73 - - [09/Apr/2014:14:56:56 +0000] "PUT /sdi1/1655169/AUTH_91b3483a7d744f57b5359983b31b95d1/Mouse/_versions/5/_project/514.avb" 201 - "-" "tx1eb33c0e7aad47f4ae14e42d874d5962" "openstack.net/1.1.2.1" 0.1150 | 21:03 |
| _bluev | Apr 9 16:04:37 DE-R210II-S-3-x proxy-server 127.0.0.1 127.0.0.1 09/Apr/2014/15/04/37 GET /v1/AUTH_91b3483a7d744f57b5359983b31b95abc/Mouse/_versions/5/_project/514.avb HTTP/1.0 200 - openstack.net/1.1.2.1 7e0ba34ee77b4ee4b6903b161dbe4c3f - 524288 - txc306f5e1a27645b9aa424edbe081631c - 49.7439 - | 21:03 |
| _bluev | Apr 9 16:04:37 DE-R210II-S-3-x proxy-server ERROR with Object server 10.32.37.85:6000/sdi1 re: Trying to read during GET: ChunkReadTimeout (45s) (txn: txc306f5e1a27645b9aa424edbe081631c) (client_ip: 127.0.0.1) | 21:03 |
| _bluev | Apr 9 16:03:50 SM-X9DBL36B-S-3-x object-server 10.32.37.73 - - [09/Apr/2014:15:03:50 +0000] "GET /sdi1/1655169/AUTH_91b3483a7d744f57b5359983b31b95d1/Mouse/_versions/5/_project/514.avb" 200 3245061 "-" "txc306f5e1a27645b9aa424edbe081631c" "openstack.net/1.1.2.1" 0.6062 | 21:04 |
| _bluev | Cluster is all running 1.8.0 and node_timeout is 45seconds | 21:04 |
| clayg | peluse: sure | 21:05 |
| _bluev | Here's the proxy server message from the PUT, this time showing the content-length: | 21:07 |
| _bluev | Apr 9 15:56:56 DE-R210II-S-3-PWP71-GB proxy-server 127.0.0.1 127.0.0.1 09/Apr/2014/14/56/56 PUT /v1/AUTH_91b3483a7d744f57b5359983b31b95d1/Mouse/_versions/5/_project/514.avb HTTP/1.0 201 - openstack.net/1.1.2.1 7e0ba34ee77b4ee4b6903b161dbe4c3f 3245061 - - tx1eb33c0e7aad47f4ae14e42d874d5962 - 0.1540 - | 21:07 |
| notmyname | _bluev: looking. | 21:07 |
| notmyname | _bluev: there's a couple of things you can do to help people look at it | 21:08 |
| notmyname | _bluev: first, use paste.openstack.org (or some other pastebin site) | 21:08 |
| *** csd_ has joined #openstack-swift | 21:08 | |
| notmyname | _bluev: second, notice the transaction id in those log lines (eg tx1eb33c0e7aad47f4ae14e42d874d5962 in the last one) | 21:08 |
| notmyname | _bluev: all log lines associated with a single request will have the same transaction id. so you can grep your logs (on every node) for that transaction id and get everything about it | 21:09 |
| _bluev | notmyname: good point re paste, thank-you. Put them here: http://paste.openstack.org/show/75449/ | 21:10 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Documentation https://review.openstack.org/85824 | 21:10 |
| notmyname | _bluev: thanks | 21:11 |
| *** csd has quit IRC | 21:11 | |
| notmyname | _bluev: the PUT seems off. there should be more than one storage node log line | 21:11 |
| _bluev | I will grab the other two lines | 21:11 |
| notmyname | thanks | 21:12 |
| notmyname | _bluev: so to sum up, it seems that the PUT logs 3245061 bytes and the GET logs only 524288 bytes | 21:12 |
| peluse | FYI anyone wanting to start reading through the raw content that will become Storage Policy docs, please feel free to start taking a look at the above patch and commenting. Right now its a mix of end user stuff and internals. | 21:12 |
| peluse | notmyname: let me know how you feel on the level of detail in the 'under the hood' section - no diagrams or pics as of yet, just trying to get the text in there and see how it plays | 21:13 |
| notmyname | _bluev: although it looks like the GET to the object server did the right thing | 21:13 |
| notmyname | peluse: it's at the top of https://wiki.openstack.org/wiki/Swift/PriorityReviews now | 21:14 |
| *** G________ has joined #openstack-swift | 21:14 | |
| peluse | clayg: I just added a few TODO cards on Trello for places I noticed were still un-SP'd so to speak. Was going to knock one out, had you looked at ContainerSync at all? | 21:14 |
| notmyname | peluse: I'll take a look later | 21:14 |
| peluse | gracias | 21:14 |
| *** scohen has quit IRC | 21:14 | |
| *** openstackstatus has quit IRC | 21:14 | |
| peluse | notmyname/clayg: also note that account reaper is still TODO, was accidnetally moved to DONE a while back. SHould be small though | 21:15 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/python-swiftclient: Add some bash helpers for auth stuff https://review.openstack.org/86224 | 21:16 |
| *** scohen has joined #openstack-swift | 21:16 | |
| clayg | peluse: fun fun! | 21:16 |
| *** JuanManuelOlle has quit IRC | 21:16 | |
| *** csd_ is now known as csd | 21:16 | |
| clayg | peluse: I think I have not looked at ContainerSync | 21:17 |
| notmyname | _bluev: but the error on the read is important in this case, and would explain what you are seeing | 21:17 |
| peluse | clayg: caught on consolidated diff - cool to see the big picture now. | 21:18 |
| peluse | clayg: no biggy, I'll take care of them and let you know if I have issues | 21:18 |
| notmyname | _bluev: BTW, swift 1.11 included a fix you would like https://github.com/openstack/swift/blob/master/CHANGELOG#L170 (auto server-side retry on this type of failure) | 21:18 |
| notmyname | _bluev: is the bug you are seeing repeatable? does it happen on every request? | 21:19 |
| _bluev | _bluev: not on every request, no. I've double checked the transactions and have updated http://paste.openstack.org/show/75450/ | 21:20 |
| * notmyname wonders what a swift-backed pastebin would look like | 21:20 | |
| *** scohen has quit IRC | 21:20 | |
| pandemicsyn | notmyname: heh we use one | 21:21 |
| notmyname | pandemicsyn: cool | 21:21 |
| pandemicsyn | http://paste.ronin.io | 21:21 |
| pandemicsyn | thats the stripped down version | 21:21 |
| clayg | acphacphapchaph github link 404's | 21:21 |
| pandemicsyn | (soonish) | 21:22 |
| pandemicsyn | because im lazy | 21:22 |
| clayg | NOW *is* soonish | 21:22 |
| *** openstackstatus has joined #openstack-swift | 21:22 | |
| pandemicsyn | clayg: https://www.youtube.com/watch?v=gNIwlRClHsQ | 21:22 |
| _bluev | notmyname: that server-side read timeout recovery looks excellent, yes. | 21:23 |
| notmyname | _bluev: ya, it's one of those things that is like "how are we just getting this now?" | 21:24 |
| notmyname | (thanks dfg!) | 21:24 |
| *** judd7 has quit IRC | 21:32 | |
| _bluev | notmyname: so if I understand.- the proxy server is streaming the content through on the GET request. the object server likely has a disk issue or some other problem and we reach node_timeout, whereby the proxy error is logged. The user has received part of the file but the flow stops. The proxy in 1.8.0 does not try to recover. | 21:33 |
| notmyname | _bluev: that is correct. ther eis one other thing I'm looking at in your logs, though | 21:36 |
| _bluev | notmyname: assuming the last field in the ammount of the time taken, the object-server log for the GET looks to be bizarrely short to me | 21:37 |
| notmyname | _bluev: well, what's confusing me is the IP addresses there. the PUT goes to one (?!) server (.73). but the error is with an object server on .85. why is that? | 21:38 |
| _bluev | notmyname: DE-R210II-S-3-x is the proxy server and is .73. The three object servers in the PUT section at the top are .82 .25 .85. - they are three separate machines | 21:42 |
| *** Trixboxer has quit IRC | 21:46 | |
| notmyname | _bluev: ah, my mistake. that's the remote address, not the local one | 21:46 |
| _bluev | notmyname: :-) | 21:46 |
| notmyname | _bluev: so you're right on track. there's 2 things I'd do next | 21:46 |
| notmyname | _bluev: first, check to see if there are other errors on .85. maybe it's a bad NIC or drive or something else that needs to be replaced | 21:47 |
| notmyname | _bluev: second, if there isn't a bad drive, then check the on-disk object on .85 and see what's up with it | 21:47 |
| _bluev | notmyname: Thank you - that sounds like a good idea. The last piece of the puzzle for me is - why does the user receive HTTP 200 - is that because the HTTP status code is determined before a single bit of payload has been sent ? | 21:48 |
| notmyname | _bluev: and if those don't turn anything up, then could it be something that is happening under load? is .85 getting overwhelmed? | 21:48 |
| notmyname | _bluev: yes. headers are sent first, so we can't rewind what's already been sent to the client | 21:49 |
| notmyname | _bluev: which is (another reason) why the etag is nice. the client can check that the bytes received match the value in the header | 21:49 |
| *** piyush2 has quit IRC | 21:49 | |
| bmac423 | Getting this error now: LookupError: No section 'informant' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/swift/proxy-server.conf | 21:50 |
| _bluev | notmyname: .85 has had a few issues - it's disks are about 2.5yrs old and the replication to push data off of it of course means it has a lot of work to do. load avg is around 7 normally but does spike to 15 | 21:51 |
| notmyname | _bluev: how many cores? load average of 15 is fine if you've got 16 cores ;-) | 21:52 |
| _bluev | notmyname: 4 cores :-( | 21:52 |
| notmyname | _bluev: so it could simply be a hardware bottleneck or it could be a drive issue | 21:54 |
| pandemicsyn | bmac423: do you have a typo in your [filter:informant] maybe ? | 21:54 |
| pandemicsyn | [filter:informant] section* | 21:54 |
| bmac423 | I don't think so... | 21:55 |
| _bluev | notmyname: i'm betting on disk , but yes - could be NIC possibly or generally too much load. | 21:55 |
| bmac423 | [filter:informant] use = egg:informant#informant statsd_host = | 21:55 |
| *** bmac423 has quit IRC | 22:07 | |
| *** jamie_h has quit IRC | 22:15 | |
| *** peterbusque has quit IRC | 22:15 | |
| *** peterbus_ has joined #openstack-swift | 22:15 | |
| *** G________ has quit IRC | 22:26 | |
| *** _bluev has quit IRC | 22:28 | |
| *** openstack has joined #openstack-swift | 22:34 | |
| *** mkollaro has quit IRC | 22:44 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Expose container info on deleted containers. https://review.openstack.org/86465 | 22:53 |
| clayg | notmyname: ^ | 22:53 |
| notmyname | clayg: thanks | 22:53 |
| notmyname | clayg: look at that sexy topic | 22:53 |
| peluse | ahhh | 22:56 |
| notmyname | clayg: peluse: https://wiki.openstack.org/wiki/Swift/PriorityReviews | 22:57 |
| peluse | Expose is a good work I think | 22:57 |
| * clayg looks up virility | 22:57 | |
| clayg | peluse: is that really the word you ment? RE: Diskfile directory structure "external modules have no virility into that detail" | 22:58 |
| peluse | really, I did that? :) | 22:59 |
| peluse | pretty sure that was mean to be visibility. I types all that in an email so it would get spellchecked as I went | 22:59 |
| notmyname | http://thesaurus.com/browse/virility I guess you could use "manhood" | 22:59 |
| peluse | hahah | 22:59 |
| *** byeager has quit IRC | 22:59 | |
| peluse | I say we keep it :) | 23:00 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Put X-Timestamp in object 404 responses https://review.openstack.org/86467 | 23:01 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Container Sync https://review.openstack.org/86469 | 23:02 |
| clayg | peluse: wow, is that it? | 23:03 |
| peluse | clayg: unit testing could be a bit better. it only tests policy 0 now but other than that yes | 23:03 |
| peluse | clayg: not sure which tests to manually extend cross policies. Who is a good container_sync person to ask about the various test cases? | 23:04 |
| notmyname | peluse: gholt | 23:04 |
| peluse | cool, thanks. Will hit hit up tomorrow. | 23:05 |
| peluse | clayg: I also kept using some of the names (and hardcoded 0 in unit tests) until we get all this stuff done then we can do one swoop to update variable names, etc | 23:06 |
| clayg | peluse: that's fine | 23:07 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/python-swiftclient: Add some bash helpers for auth stuff https://review.openstack.org/86224 | 23:09 |
| peluse | clayg: saw the doc comments, thanks! Will update tomorrow. No clue how to correctly format, etc., so jsut focusing on content now and maybe someone else who know this sphincter thing can help :) | 23:09 |
| clayg | peluse: totally! | 23:09 |
| *** bobf has joined #openstack-swift | 23:19 | |
| *** changbl has quit IRC | 23:21 | |
| clayg | peluse: aww man i was looking at the unified diff and saw get_data_dir and get_async_dir - we should pull that bit out of the container-obj-put-409 patch and get it on feature/ec | 23:29 |
| peluse | yeah, I agree | 23:32 |
| peluse | want me to do it? | 23:33 |
| *** _bluev has joined #openstack-swift | 23:34 | |
| peluse | also, wrt container sync there might be some more to do there beyond test code (checking if not handling new scenarios) but I'll chat with gholt and see what he thinks too | 23:34 |
| *** zackf has quit IRC | 23:34 | |
| clayg | peluse: sure1 | 23:35 |
| bobf | does the feature/ec branch contain the swift-ring-builder for multiple policies. After modifying the swift.conf and remakerings to handle storage policy. the output appears not to generate object-<x>.builder as expected. Am I missing a step for multiple policy enablement? | 23:36 |
| peluse | clayg: cool, will do | 23:37 |
| peluse | bobf: you need to add the policy index when you use the builder command, did you check our the emerging docs yet? | 23:37 |
| bobf | pointer to doc? | 23:37 |
| peluse | https://review.openstack.org/#/c/85824/ | 23:38 |
| bobf | ok thanks | 23:38 |
| notmyname | bobf: to be fair, those were pushed publically about 2 hours ag o;-) | 23:38 |
| peluse | pretty raw right now - just posted initial content a few hrs ago so expect lots of changes. the basics for getting going should work though. if you have feedback, would love to hear it! | 23:38 |
| peluse | Yeah, I know - earlier comment wasn't meant to be an RTFM type thing :) | 23:39 |
| bobf | :) | 23:39 |
| notmyname | lol. peluse is "helpful" by telling people, "that's totally documented in the proposed patch that I pushed up 30 minutes ago! go read the docs!" | 23:39 |
| peluse | yeah yeah :) you know that's now what I meant... should have said "if you can, please take this opportunity to check out the emerging documentation"... | 23:40 |
| *** mwstorer has quit IRC | 23:42 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!