| timburke | if we land https://review.openstack.org/#/c/640518/ and https://review.openstack.org/#/c/640552/ ... i've got a patch locally that builds on them and gets test/functional/tests.py to pass (if you hack up your stdlib). it does some weird things, so it'd be good for us to look at and start thinking about what's involved... | 00:25 |
|---|---|---|
| patchbot | patch 640518 - swift - py3: fix copying unicode names - 1 patch set | 00:25 |
| patchbot | patch 640552 - swift - Stop monkey-patching mimetools - 2 patch sets | 00:25 |
| *** gyee has quit IRC | 00:26 | |
| *** gyee has joined #openstack-swift | 00:27 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Refactor write_affinity DELETE handling https://review.openstack.org/641527 | 00:45 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: failing test for write_affinity POST https://review.openstack.org/641528 | 00:45 |
| clayg | I think the refactor change is reasonable cleanup - I'd love a sanity +A - esp the two tests where the names seemed backward to me? | 00:46 |
| *** gyee has quit IRC | 01:11 | |
| *** rcernin has quit IRC | 01:31 | |
| *** rcernin has joined #openstack-swift | 01:32 | |
| *** rchurch has joined #openstack-swift | 01:36 | |
| *** rchurch_ has quit IRC | 01:38 | |
| *** psachin has joined #openstack-swift | 02:42 | |
| *** [diablo] has quit IRC | 03:12 | |
| *** [diablo] has joined #openstack-swift | 03:14 | |
| *** [diablo] has quit IRC | 03:19 | |
| *** [diablo]9 has joined #openstack-swift | 03:19 | |
| *** timss- has quit IRC | 04:26 | |
| *** kazsh has quit IRC | 04:26 | |
| *** kukacz has quit IRC | 04:26 | |
| *** tdasilva has quit IRC | 04:26 | |
| *** patchbot has quit IRC | 04:26 | |
| *** sorrison has quit IRC | 04:26 | |
| *** timss- has joined #openstack-swift | 04:53 | |
| *** kazsh has joined #openstack-swift | 04:53 | |
| *** tdasilva has joined #openstack-swift | 04:53 | |
| *** kukacz has joined #openstack-swift | 04:53 | |
| *** patchbot has joined #openstack-swift | 04:53 | |
| *** sorrison has joined #openstack-swift | 04:53 | |
| *** card.freenode.net sets mode: +v tdasilva | 04:53 | |
| *** paladox has quit IRC | 04:56 | |
| *** rcernin has quit IRC | 04:56 | |
| *** paladox has joined #openstack-swift | 04:59 | |
| *** rcernin has joined #openstack-swift | 05:02 | |
| *** e0ne has joined #openstack-swift | 06:32 | |
| *** e0ne has quit IRC | 07:17 | |
| *** e0ne has joined #openstack-swift | 07:21 | |
| *** pcaruana has joined #openstack-swift | 07:25 | |
| *** rcernin has quit IRC | 07:27 | |
| *** thurloat2 has joined #openstack-swift | 07:31 | |
| *** thurloat has quit IRC | 07:32 | |
| *** [diablo]9 has quit IRC | 07:42 | |
| *** [diablo] has joined #openstack-swift | 07:44 | |
| *** hseipp has joined #openstack-swift | 08:03 | |
| *** e0ne has quit IRC | 08:04 | |
| *** tkajinam has quit IRC | 08:12 | |
| *** ccamacho has joined #openstack-swift | 08:13 | |
| *** hseipp has quit IRC | 09:03 | |
| *** hseipp has joined #openstack-swift | 09:14 | |
| *** ccamacho has quit IRC | 10:06 | |
| *** zaitcev_ has joined #openstack-swift | 10:14 | |
| *** ChanServ sets mode: +v zaitcev_ | 10:14 | |
| *** zaitcev has quit IRC | 10:18 | |
| admin6 | Hi team, still facing my disks full problem on an EC 9+3 ring (only on one node). I have identified lots or directories on my disks with partition numbers that has nothing to do there (not listed in the list_parts in the ring for this disk, not part of the main 12 replica, neither the first 12 handoff of the partition). Objects that these fragment are part of, has their 12 main fragment existing on the appropriate place | 10:20 |
| admin6 | Option handoffs_only doesn’t help as it seems not to detect correctly that these fragment could be removed. These directories represent about 25% of the disk space used. | 10:20 |
| admin6 | as this node has its 30 disks full at 100%, I urgently need to reduce the usage of the disks. As an emergency process, do you see real big risks if I manually delete these directories ? | 10:24 |
| *** e0ne has joined #openstack-swift | 10:25 | |
| admin6 | I swear I’ll not consider you as repsonsible for the mess if you answer ‘yes’ ;-) | 10:36 |
| rledisez | admin6: before deleting data, try to run the reconstructor manually to see what it says. something like : | 10:57 |
| rledisez | swift-object-reconstructor /etc/swift/object-server.conf -d sda -p 42 -o -v | 10:57 |
| rledisez | replace sda and 42 par the full disk and the partition that should not be here | 10:57 |
| admin6 | rledisez: Merci. the reconstructor does’nt seem to detect handoffs : https://pastebin.com/S3TVTWAp and the data are still there after the reconstructor run. | 11:06 |
| *** ccamacho has joined #openstack-swift | 11:31 | |
| admin6 | rledisez: a bit more details if you have time to take a look : https://pastebin.com/aZvuuhnk | 11:38 |
| *** ybunker has joined #openstack-swift | 12:00 | |
| ybunker | hi all, from yesterday i got the containers being replicated :) yeay! ... but.. not so yeay lol haha, because accounts are still not able to replicate..., also i notice that the newly nodes that were added to the cluster, are not freeing space from the rest.. and im worry because one of the nodes is at 98% of full condition | 12:02 |
| *** thurloat22 has joined #openstack-swift | 12:39 | |
| *** thurloat22 has quit IRC | 12:41 | |
| *** thurloat2 has quit IRC | 12:42 | |
| *** mugsie has joined #openstack-swift | 12:45 | |
| rledisez | admin6: remind me your swift version please | 12:54 |
| rledisez | no, nevermind. I was thinking of something else, it does ont apply to reconstructor | 12:56 |
| rledisez | (there is a bug where -p make the reconstructor does nothing, i must report this) | 12:57 |
| rledisez | just in case, if you can try to re-run the reconstructor with only -d | 12:58 |
| admin6 | rledisez: I use swift v2.17 | 13:03 |
| rledisez | i mean -d -o -v (so without -p :)) | 13:04 |
| admin6 | rledisez: same behavior witouh the partition number. however, I got a really fast answer, is that normal ? | 13:09 |
| rledisez | admin6: it can be very fast when there is nothing to do ;) | 13:19 |
| rledisez | can you rerun after setting the log_level to debug | 13:19 |
| rledisez | in object-server.conf: log_level = DEBUG | 13:19 |
| rledisez | it might gives more info | 13:19 |
| rledisez | I'm not sure if -v enable the debug logs | 13:20 |
| admin6 | DEBUG only provide one more line : Spawned worker 9495 with {'override_devices': ['sde'], 'override_partitions': []} (and no, -v doesn’t enable debug log) | 13:25 |
| admin6 | rledisez: did you have a chance to look at https://pastebin.com/aZvuuhnk to confirm I’m not mad ? | 13:27 |
| rledisez | hum, at that point, I usually go add some print() in the code to find out what is going on, but that's up to you :) | 13:27 |
| rledisez | if you decide to delete some data, please check that every file you delete is somewhere else in your cluster (with the same name/md5) | 13:27 |
| rledisez | yeah, I checked, and it seems I'm as mad as you ;) | 13:28 |
| admin6 | rledisez; ok, thanks :D | 13:30 |
| admin6 | I’m unfortunately not familiar enough with the code, to put print() debugs at useful breaking points… | 13:31 |
| *** ccamacho has quit IRC | 13:49 | |
| *** psachin has quit IRC | 14:24 | |
| ybunker | anyone? clayg? :) | 14:51 |
| *** admin6 has quit IRC | 16:11 | |
| notmyname | good morning | 16:12 |
| ybunker | hi | 16:14 |
| *** e0ne has quit IRC | 17:07 | |
| *** hseipp has quit IRC | 17:28 | |
| *** gyee has joined #openstack-swift | 17:57 | |
| *** irclogbot_2 has joined #openstack-swift | 18:03 | |
| *** e0ne has joined #openstack-swift | 18:19 | |
| *** psachin has joined #openstack-swift | 18:23 | |
| *** e0ne has quit IRC | 18:24 | |
| timburke | tdasilva, https://review.openstack.org/#/c/635040/ | 18:25 |
| patchbot | patch 635040 - swift - Include some pipeline validation during proxy-serv... - 1 patch set | 18:25 |
| *** e0ne has joined #openstack-swift | 18:29 | |
| *** e0ne has quit IRC | 18:33 | |
| *** pcaruana has quit IRC | 18:46 | |
| *** pcaruana has joined #openstack-swift | 19:05 | |
| *** pcaruana has quit IRC | 19:33 | |
| *** ybunker has quit IRC | 19:51 | |
| *** rchurch_ has joined #openstack-swift | 20:15 | |
| *** rchurch has quit IRC | 20:17 | |
| *** psachin has quit IRC | 20:17 | |
| *** e0ne has joined #openstack-swift | 20:35 | |
| * clayg timburke: on p 640552 isn't the supported way to do this just to set MessageClass on our Protocol/Handler such that we can make get_content_type do whatever we want? | 21:04 | |
| patchbot | https://review.openstack.org/#/c/640552/ - swift - Stop monkey-patching mimetools - 2 patch sets | 21:04 |
| clayg | https://docs.python.org/2/library/basehttpserver.html#BaseHTTPServer.BaseHTTPRequestHandler.MessageClass | 21:04 |
| timburke | clayg, https://github.com/python/cpython/blob/v2.7.15/Lib/mimetools.py doesn't have a get_content_type :-( | 21:06 |
| clayg | ybunker try deleting the hashes.pkl from the part dir and re-run it - or look at the data in the part dir - might be hitting a bug like lp bug #1706321 | 21:06 |
| openstack | Launchpad bug 1706321 in OpenStack Object Storage (swift) "reconstructor does not delete empty partition that previously had non-durable data" [High,Fix released] https://launchpad.net/bugs/1706321 | 21:06 |
| clayg | er... ^ admin6 you might try messing with https://gist.github.com/clayg/7a975ef3b34828c5ac7db05a519b6e8a | 21:07 |
| timburke | i guess maybe we could have a subclass that does parsetype for py2 and get_content_Type for py3? i can try it out... | 21:07 |
| timburke | not sure it'll really feel much less brittle... i can try it, though. i think i've got a stash that started to try to do that... | 21:08 |
| clayg | I was just reading https://github.com/eventlet/eventlet/blob/v0.24.1/eventlet/wsgi.py#L651 and wondering what exactly self\.headers is when I noticed that BaseHTTPSever had a MessageClass attr that was documented | 21:13 |
| clayg | maybe it's not really supported - but since we already pass in a Protocol I figured hacking in our own class for headers would be fine as long as it follows the protocol of the super class - quacks like a duck! | 21:14 |
| clayg | I think that's slightly better than trying to manipulate the logic flow of a caller by pre-setting some attr it looks at to short circut | 21:14 |
| *** e0ne has quit IRC | 21:14 | |
| clayg | maybe it's not so easy to say what the supported interface of the headers object is really supposed to be? parsetype vs get_content_type? 😬 | 21:15 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Stop monkey-patching mimetools https://review.openstack.org/640552 | 21:21 |
| timburke | aight, see what you think of *that* | 21:21 |
| clayg | ybunker so do account-server/1.conf and account-server/2.conf use the same devices root? Like - they use different *ports* - I feel like if they use the same devices root a request to /sdX/<part>/... would go the same path (/srv/node/sdX/accounts/<part>/...) on either server? That's kinda weird.. | 21:21 |
| *** zaitcev_ is now known as zaitcev | 21:24 | |
| *** e0ne has joined #openstack-swift | 22:01 | |
| mattoliverau | morning | 22:01 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Refactor write_affinity DELETE handling https://review.openstack.org/641527 | 22:01 |
| zaitcev | I think it's possible to have several instances on the same /srv/node, as long as device names are globally unique. It's just a fragile setup, until the first mistake. | 22:25 |
| *** e0ne has quit IRC | 22:34 | |
| *** e0ne has joined #openstack-swift | 22:35 | |
| clayg | zaitcev: yes, I think it might work - not sure if that's what's intended tho | 22:35 |
| *** e0ne has quit IRC | 22:36 | |
| *** rcernin has joined #openstack-swift | 22:41 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Fix how we UTF-8-ify func tests https://review.openstack.org/641855 | 22:46 |
| timburke | :-( | 22:46 |
| *** tkajinam has joined #openstack-swift | 22:58 | |
| timburke | so i was trying to make heads or tails of the quoting in p 639263 ... thought i'd take a stab at getting test/functional/test_dlo.py to run under py3 ... and bumped into ^^^ ... but after addressing *that*, i kinda really want p 571905 to land before digging in too far... | 23:33 |
| patchbot | https://review.openstack.org/#/c/639263/ - swift - py3: port dlo - 1 patch set | 23:33 |
| patchbot | https://review.openstack.org/#/c/571905/ - swift - Fix quoting for large objects - 4 patch sets | 23:33 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!