| *** itlinux has joined #openstack-swift | 00:09 | |
| *** itlinux_ has joined #openstack-swift | 00:13 | |
| *** itlinux has quit IRC | 00:15 | |
| *** rchurch has joined #openstack-swift | 00:54 | |
| *** rchurch_ has quit IRC | 00:55 | |
| *** openstackgerrit has joined #openstack-swift | 01:01 | |
| openstackgerrit | Merged openstack/swift master: py3: port proxy container controller https://review.openstack.org/638329 | 01:01 |
|---|---|---|
| openstackgerrit | Merged openstack/swift master: py3: port proxy account controller https://review.openstack.org/637653 | 01:01 |
| *** rchurch has quit IRC | 01:10 | |
| *** rchurch has joined #openstack-swift | 01:11 | |
| clayg | i made this - in case you need to know what your async_pendings are upto -> https://gist.github.com/clayg/249c5d3ff580032a0d40751fc3f9f24b | 01:17 |
| supamatt | clayg: forked! | 01:19 |
| openstackgerrit | Tim Burke proposed openstack/swift master: docs: clean up SAIO formatting https://review.openstack.org/640914 | 01:37 |
| openstackgerrit | Merged openstack/python-swiftclient master: Update release to 3.7.0 https://review.openstack.org/640819 | 01:39 |
| kota_ | good morning | 02:20 |
| openstackgerrit | Merged openstack/swift master: Clean up func tests ahead of py3 https://review.openstack.org/640519 | 02:27 |
| *** psachin has joined #openstack-swift | 02:39 | |
| *** chocolate-elvis has joined #openstack-swift | 02:42 | |
| *** itlinux_ has quit IRC | 03:06 | |
| *** itlinux has joined #openstack-swift | 03:12 | |
| *** itlinux has quit IRC | 03:21 | |
| *** itlinux has joined #openstack-swift | 03:25 | |
| *** gyee has quit IRC | 03:54 | |
| *** _david_sohonet has quit IRC | 04:03 | |
| kota_ | rledisez: I'll be at office until tonight and online as possible so if you're ready to push a merge commit to feature/losf branch, I'll be able to help any time in your morning | 04:53 |
| notmyname | tdasilva: thanks for taking care of the changelog. it landed so I updated https://review.openstack.org/#/c/640549/ | 04:59 |
| patchbot | patch 640549 - releases - swiftclient 3.7.0 release - 2 patch sets | 04:59 |
| *** chocolate-elvis has quit IRC | 04:59 | |
| *** pcaruana has joined #openstack-swift | 05:52 | |
| *** pcaruana has quit IRC | 06:07 | |
| *** ccamacho has quit IRC | 06:21 | |
| *** e0ne has joined #openstack-swift | 06:24 | |
| *** e0ne has quit IRC | 06:33 | |
| *** itlinux has quit IRC | 06:34 | |
| *** e0ne has joined #openstack-swift | 06:46 | |
| *** e0ne has quit IRC | 07:07 | |
| *** thyrst has left #openstack-swift | 07:31 | |
| *** hseipp has joined #openstack-swift | 07:43 | |
| admin6 | Hi team, some help is welcomed on dealing with a server full in my Erasure coding ring. I’m a bit puzzled about the behavior of the object-reconstructor that only try to rebalance data to the server that is already full. I’ve strongly reduced the weight of all disks on this server but the rebalance still try to put most of the datas on them. | 07:57 |
| *** e0ne has joined #openstack-swift | 08:17 | |
| *** pcaruana has joined #openstack-swift | 08:19 | |
| kota_ | admin6: could you try handoff_only = True mode in object-server.conf? https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L353 | 08:19 |
| kota_ | that will help you to move the handoffs from weight reduced nodes to the new nodes. | 08:19 |
| kota_ | and also multiple workers might help to speed up the transfer, https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L333 | 08:20 |
| kota_ | if you have Swift newer enough to use the stuffs. | 08:21 |
| *** tkajinam has quit IRC | 08:22 | |
| *** pcaruana has quit IRC | 08:25 | |
| *** ccamacho has joined #openstack-swift | 08:29 | |
| admin6 | kota: thanks a lot for your suggestion. I’m on swift v2.17. I think it should be supported in this version (I’ll check). I’ll try it and see if it helps. | 08:31 |
| *** pcaruana has joined #openstack-swift | 08:37 | |
| *** pcaruana has quit IRC | 08:44 | |
| rledisez | hi kota_. i just arrived at work | 08:45 |
| kota_ | rledisez: good morning! | 08:46 |
| kota_ | rledisez: did you see my gist for the procedure to get master merged? | 08:47 |
| rledisez | yes, i'll try it right now | 08:47 |
| kota_ | b | 08:50 |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift feature/losf: Merge remote-tracking branch 'remotes/origin/master' into merge-master https://review.openstack.org/640955 | 08:52 |
| rledisez | kota_: ^ I had no conflicts | 08:53 |
| kota_ | good | 08:53 |
| rledisez | if tests are passing, I'll +2 / +A it | 08:53 |
| kota_ | perfect | 08:54 |
| *** cschwede has joined #openstack-swift | 08:57 | |
| *** ChanServ sets mode: +v cschwede | 08:57 | |
| *** e0ne has quit IRC | 08:58 | |
| *** e0ne has joined #openstack-swift | 09:00 | |
| *** pcaruana has joined #openstack-swift | 09:01 | |
| *** cschwede has quit IRC | 09:03 | |
| *** cschwede has joined #openstack-swift | 09:07 | |
| *** ChanServ sets mode: +v cschwede | 09:07 | |
| admin6 | kota: I have plenty of messages concerning .lock files like : object-reconstructor: 10.10.2.55:7000/s05z1ecd04/105712 Unexpected response: ":ERROR: 0 '8 seconds: /srv/node/s05z1ecd04/.lock’". | 09:21 |
| admin6 | kota: in the object-server logs, 99+% of PUT logs concern the full server, with ssync subrequest failed with 507 error | 09:23 |
| kota_ | perhaps, you hit https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L170 ? | 09:23 |
| kota_ | oh my.... it sounds like the destination is full enough? | 09:24 |
| admin6 | kota: and in object-reconstructor, very few part of "Removing partition" messages concern the full server, | 09:24 |
| *** cschwede has quit IRC | 09:25 | |
| kota_ | admin6: no idea currently. To know what happens, we should check the ring stat, device usage, then, check if we've managed the ring correctly. | 09:30 |
| admin6 | kota: yes the destination is full. and the strange behavior is that the most I’ve reduced the weight of disks on the concerned server (which now is full), the most data was written to it | 09:31 |
| admin6 | kota: what do you means by ring stat? verbose ring dispersion report? | 09:33 |
| * kota_ is meaning `swift-ring-builder <builder file>` | 09:37 | |
| kota_ | if the ring was configured (reduce the device weight and assigned partitions are decreased), the reconstructor in the node should move the partitions very actively. | 09:40 |
| admin6 | kota: that’s what I did, but the behavior was the opposit, and the most I reduces the weight of the disks, the more this server was filled :-( | 09:53 |
| admin6 | kota: would you be kind enough to have a look at my ring config ? https://pastebin.com/iTHxebmH | 09:55 |
| kota_ | admin6: one possible reason was, there are already a tons of handoffs for the full node in other nodes, then, the handoffs are back to the full nodes when it has free space. | 09:55 |
| kota_ | which is full disk? | 09:57 |
| admin6 | kota: all disk of server 10.10.1.53 | 09:57 |
| kota_ | admin6: looking at line 322, 10.10.1.53 still keeps 41.19 % of partitions. is it intended for you? | 09:59 |
| kota_ | i'm feeling it's still enough partition to be filled up as full, maybe? | 10:00 |
| kota_ | and looking at the dispersion report from Line 224, it seems like we still have room for rebalance? | 10:02 |
| admin6 | kota: no, it is not intended at all. and I’m not sure what exactly means this percentage and how to reduce it. | 10:02 |
| kota_ | I cannot make sure what the weight on each disk is according to but the weight: 3700 looks not smaller enough than other nodes, some of devices have smaller weight... | 10:05 |
| kota_ | and the balance looks not to fit with the weight too, so my bet is calling `swift-ring-builder <builder file> rebalance` to follow the weight balance. | 10:09 |
| admin6 | kota: my weigths are globally related to size of disk : 4000 for a 4TB disk. disks on server 10.10.1.53 also are 4TB and I’ve reduced them to 3700, thinking that could be enough to reduce their usage. | 10:09 |
| admin6 | kota: smaller weights are disk that are currentlty gradually intergated into the ring | 10:10 |
| kota_ | i see | 10:10 |
| kota_ | admin6: could you calculate how much data is in a partition? | 10:12 |
| admin6 | kota: don’t think if I’m right but each time I do a rebalance, I try not to change more than 8% of the total ring weight, then wait for the dispersion report to return at least 99% of object copies found. | 10:13 |
| kota_ | the 4TB disks with 4000 weight has around 14504 partitions, then, 3700's has 14127 partitions so just around 380 partitions is the difference with other 4TB disks | 10:14 |
| admin6 | I’ve currently 700TB stored in this ring divided by 262144 partitions, that means 2,7GB per partition | 10:16 |
| kota_ | hmmm | 10:25 |
| admin6 | kota: server 10.10.1.54 has disk flled with 3520GB for 14504 partitions. Server 10.10.1.53 has disk flled with 3866GB for 14127 partitions | 10:29 |
| kota_ | that sounds like your object size stored in Swift is in dispersion :/ | 10:31 |
| kota_ | and if a partition can store 2.7GB, 14504 partitions mean that the server can store 39TB that seems different from your intension. | 10:34 |
| kota_ | at most. | 10:34 |
| admin6 | kota: I’m not sure what you means by "object size stored is in dispersion". unless it was sarcasm ;-) for the partition size, isn’t their something related to the number of fragment a file is divided in ?, because on server 10.10.1.53 for example, 3866GB divided by 14127 partitions means 274MB per partition, not 2,7GB | 10:39 |
| kota_ | I don't think it's being from actual stored data. it's from design how operator want to control the stored data. When you think to create 700TB logical cluster with 262144 partitions, a partition has the meaning that has the capacity with 2.7GB in average. | 10:44 |
| kota_ | but the size of each partition should be "average", so that if the real object size in-coming from users would be distributed, some of partitions may gets more size rather than average. | 10:46 |
| kota_ | anyway, the partition size is from your cluster design and total real disk volume size, so I have no idea how much size of each partition is in your cluster. | 10:48 |
| kota_ | but if the partition average size * the number of partitions is bigger than the real disk size, the disk will keep full unless you decrease assigned partitions less than the real volume size. | 10:49 |
| kota_ | it's so strategic. | 10:50 |
| admin6 | kota: but if I sum the number of partitions of each disks, I obtain 3145728 which is 12 time more than 262144 which is the numer of partition in the ring, ang it seems somewhere logical as my erasure coding ring is a 9+3 | 10:50 |
| admin6 | so the partition at the disk level has not the same meaning as the ring level ? | 10:51 |
| kota_ | 9 + 3 meaning 12 pieces of fragments so 12 times is correct, i think. | 10:51 |
| kota_ | so if assuming 700TB actual (not logical) disk volumes in your cluster, 227MB for a partition. | 10:53 |
| kota_ | it seems like around 3.2TB, mathmatically. | 10:55 |
| kota_ | with 14127 partitions. | 10:55 |
| admin6 | so 227MB * 14127 partitions = 3207GB, even with 10% overload; I should not be over 3527GB per disk, but they are full with 3866GB and reconstructor still want to write more data only into these disks :( | 10:58 |
| kota_ | admin6: that could happen, if you're running for a long time with the full disk. I think the reconstructor in the full disk node .53 actively push the data but in-coming handoff from other nodes might be faster. | 11:00 |
| kota_ | so if my thought would be correct, my suggestion is `run handoff_only = True mode only in the full disk node` | 11:01 |
| admin6 | kota: ok, I can try that option. | 11:02 |
| admin6 | kota: could you tell me if I’m right or not, thinking that it may not be safe to push a new rebalance if the dispersion report is at 95/96% ? | 11:04 |
| admin6 | kota: lastly, I thank you a lot for your time spent with me, really appreciated :) | 11:05 |
| kota_ | admin6: no worries, what's 95/96% mean? | 11:06 |
| admin6 | each time I do a rebalance, I try not to change more than 8% of the total ring weight, then wait for the dispersion report to return at least 99% of object copies found. Currently my dispersion report is : Queried 2621 objects for dispersion reporting, 45s, 0 retries - There were 1417 partitions missing 0 copies. - There were 1074 partitions missing 1 copy. - There were 127 partitions missing 2 copies. - There were 3 | 11:08 |
| admin6 | partitions missing 3 copies. - 95.75% of object copies found (30115 of 31452). That’s this 95,75% I’m talking about. | 11:08 |
| kota_ | i see | 11:11 |
| admin6 | I fixed 8% percent max because it means about 1 fragment per object on my 9+3 EC. | 11:13 |
| kota_ | ring weight and dispersion is relative but not completely tied up because Swift doesn't move more than 1 replica for a partition on one rebalance. | 11:13 |
| kota_ | basically | 11:14 |
| kota_ | but I can not say 'yes it's safe' because you already 3 parts missing 3 copies | 11:14 |
| kota_ | you already have | 11:14 |
| kota_ | that means, the rebalance call might cause to make a partition missing 4 copies in the primary locations. | 11:15 |
| kota_ | that will be unfortunately causing 503 (or 404?) errors temporaly | 11:16 |
| admin6 | kota: that’s very clear. and than means swift is safer a more smarter than I tought. (or than me ;-) ) | 11:17 |
| *** e0ne has quit IRC | 11:32 | |
| openstackgerrit | Merged openstack/swift feature/losf: Merge remote-tracking branch 'remotes/origin/master' into merge-master https://review.openstack.org/640955 | 11:34 |
| *** e0ne has joined #openstack-swift | 11:42 | |
| *** ccamacho has quit IRC | 12:17 | |
| *** early` has quit IRC | 12:26 | |
| *** early` has joined #openstack-swift | 12:29 | |
| *** admin6_ has joined #openstack-swift | 14:21 | |
| *** admin6 has quit IRC | 14:24 | |
| *** admin6_ is now known as admin6 | 14:24 | |
| *** cschwede has joined #openstack-swift | 14:28 | |
| *** ChanServ sets mode: +v cschwede | 14:28 | |
| clayg | timburke: do you think a rebase would help https://review.openstack.org/#/c/637662/ or does it just need a recheck? | 14:44 |
| patchbot | patch 637662 - swift - Simplify empty suffix handling - 2 patch sets | 14:44 |
| *** pcaruana has quit IRC | 14:57 | |
| *** e0ne has quit IRC | 15:02 | |
| *** ccamacho has joined #openstack-swift | 15:03 | |
| *** e0ne has joined #openstack-swift | 15:13 | |
| *** openstackgerrit has quit IRC | 15:28 | |
| *** pcaruana has joined #openstack-swift | 15:42 | |
| *** psachin has quit IRC | 16:54 | |
| *** ccamacho has quit IRC | 16:55 | |
| -openstackstatus- NOTICE: Gerrit is being restarted for a configuration change, it will be briefly offline. | 17:11 | |
| *** jistr|sick is now known as jistr | 17:14 | |
| notmyname | python-swiftclient 3.7.0 has been tagged. this is our release in the stein cycle | 17:16 |
| *** e0ne has quit IRC | 17:18 | |
| *** hseipp has quit IRC | 17:25 | |
| *** itlinux has joined #openstack-swift | 17:35 | |
| clayg | rledisez: have you ever experimented with pulling out the post-revert-replicate request? lp bug #1818709 | 17:43 |
| openstack | Launchpad bug 1818709 in OpenStack Object Storage (swift) "object replicator update_deleted post ssync REPLICATE request considered harmful" [Undecided,New] https://launchpad.net/bugs/1818709 | 17:44 |
| clayg | rledisez: IIRC you also use SSYNC on your replicated clusters - and there as well it seems more and more like a bad IO trade off | 17:44 |
| clayg | N.B. we mainly use rsync replication for replicated storage policies and for me there's no obvious way to get away from REPLICATE requests - but with SSYNC it seems like a cheap win? | 17:45 |
| clayg | ... unless I'm missing something? | 17:45 |
| *** gyee has joined #openstack-swift | 17:54 | |
| *** e0ne has joined #openstack-swift | 18:19 | |
| *** itlinux has quit IRC | 18:42 | |
| *** itlinux has joined #openstack-swift | 18:47 | |
| *** SkyRocknRoll has joined #openstack-swift | 19:11 | |
| *** e0ne has quit IRC | 19:17 | |
| *** SkyRocknRoll has quit IRC | 19:19 | |
| *** e0ne has joined #openstack-swift | 19:37 | |
| *** itlinux has quit IRC | 19:56 | |
| *** itlinux has joined #openstack-swift | 20:09 | |
| *** itlinux has quit IRC | 20:15 | |
| *** pcaruana has quit IRC | 21:10 | |
| *** e0ne has quit IRC | 21:13 | |
| *** e0ne has joined #openstack-swift | 21:26 | |
| *** e0ne has quit IRC | 21:30 | |
| *** rchurch_ has joined #openstack-swift | 21:45 | |
| *** rchurch has quit IRC | 21:47 | |
| *** mvkr has quit IRC | 22:25 | |
| *** openstackgerrit has joined #openstack-swift | 22:51 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3token: Add note about config change when upgrading from swift3 https://review.openstack.org/641153 | 22:51 |
| *** tkajinam has joined #openstack-swift | 22:54 | |
| tdasilva | timburke: why the check for v < 2: https://review.openstack.org/#/c/636748/3/setup.py@109 ? | 23:26 |
| patchbot | patch 636748 - pyeclib - Use liberasurecode_get_version() - 3 patch sets | 23:26 |
| timburke | in case we ever make a v2 | 23:27 |
| timburke | plus, that's what the old code seemed to be *trying* to do, so... may as well keep at it? | 23:30 |
| tdasilva | timburke: got it, sounds good! thanks | 23:37 |
| openstackgerrit | Merged openstack/pyeclib master: Use liberasurecode_get_version() https://review.openstack.org/636748 | 23:40 |
| mattoliverau | morning | 23:44 |
| timburke | o/ mattoliverau | 23:46 |
| openstackgerrit | Merged openstack/swift master: docs: clean up SAIO formatting https://review.openstack.org/640914 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!