| mattoliverau | kota_: moring and happy new year! | 00:01 |
|---|---|---|
| kota_ | mattoliverau: how was going for your vacation? | 00:03 |
| mattoliverau | kota_: it was great, am well rested, might take me a while to get my brain back into gear though :P | 00:04 |
| kota_ | mattolivearu: me too :) | 00:05 |
| *** takashi has joined #openstack-swift | 00:58 | |
| *** wanghua has joined #openstack-swift | 01:14 | |
| *** chlong has joined #openstack-swift | 01:25 | |
| *** haomaiwang has joined #openstack-swift | 02:12 | |
| *** rminmin has joined #openstack-swift | 02:19 | |
| *** haomaiw__ has joined #openstack-swift | 02:34 | |
| *** haomaiwang has quit IRC | 02:34 | |
| *** haomaiw__ has quit IRC | 03:01 | |
| *** 6A4ABI07L has joined #openstack-swift | 03:01 | |
| *** rminmin has quit IRC | 03:03 | |
| *** daniel_ has joined #openstack-swift | 03:16 | |
| *** lcurtis has joined #openstack-swift | 03:35 | |
| *** Jeffrey4l_ has joined #openstack-swift | 03:41 | |
| *** links has joined #openstack-swift | 03:54 | |
| *** 6A4ABI07L has quit IRC | 04:01 | |
| *** haomaiwang has joined #openstack-swift | 04:01 | |
| notmyname | hello, world | 04:11 |
| *** daniel_ has quit IRC | 04:12 | |
| notmyname | figured I'd get a head start by checking email/IRC tonight instead of spending half the day tomorrow doing it | 04:14 |
| *** yatin has joined #openstack-swift | 04:28 | |
| mattoliverau | notmyname: good afternoon! And Happy new year! | 04:35 |
| notmyname | mattoliverau: thanks! and to you too! | 04:35 |
| notmyname | mattoliverau: how were your holidays? | 04:35 |
| mahatic | hello, good morning | 04:38 |
| notmyname | hi mahatic | 04:39 |
| mahatic | notmyname: hello! New year wishes! | 04:40 |
| notmyname | thanks | 04:40 |
| mahatic | notmyname: how were holidays? | 04:44 |
| notmyname | great. very relaxing for the most part. | 04:45 |
| mahatic | nice | 04:45 |
| notmyname | took me several days to actually get disconnected and not sit around with IRC and emails open ;-) | 04:45 |
| notmyname | but after that, it was very nice ;-) | 04:45 |
| mahatic | It ws virtual holidays for me too ;) | 04:45 |
| mahatic | heh I am sure :D | 04:45 |
| kota_ | notmyname, mahatic: happy new year o/ | 04:46 |
| mahatic | was* | 04:46 |
| mahatic | kota_: thanks, happy new year too! | 04:46 |
| *** yatin_ has joined #openstack-swift | 04:49 | |
| mattoliverau | notmyname: aweseome, saw my parents up the coast for the week. Relaxing. yours? | 04:51 |
| mattoliverau | mahatic: morning, and happy new year :) | 04:52 |
| notmyname | mattoliverau: great. did your wife get a lot of belly pats from the family? | 04:52 |
| *** SkyRocknRoll has joined #openstack-swift | 04:52 | |
| *** yatin has quit IRC | 04:52 | |
| mahatic | mattoliverau: hello, thanks, you too! :) | 04:52 |
| mattoliverau | Yeah :) | 04:52 |
| *** yatin_ has quit IRC | 04:54 | |
| *** yatin has joined #openstack-swift | 04:55 | |
| *** SkyRocknRoll has quit IRC | 04:58 | |
| *** ppai has joined #openstack-swift | 04:58 | |
| *** haomaiwang has quit IRC | 05:01 | |
| *** klrmn has quit IRC | 05:01 | |
| *** haomaiwa_ has joined #openstack-swift | 05:01 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:10 | |
| *** Jeffrey4l_ has quit IRC | 05:37 | |
| *** Jeffrey4l_ has joined #openstack-swift | 05:50 | |
| *** trifon has joined #openstack-swift | 05:51 | |
| *** lcurtis has quit IRC | 05:54 | |
| *** haomaiwa_ has quit IRC | 06:01 | |
| *** haomaiwa_ has joined #openstack-swift | 06:01 | |
| *** ChubYann has quit IRC | 06:13 | |
| *** yarkot has joined #openstack-swift | 06:19 | |
| *** wer has joined #openstack-swift | 06:54 | |
| *** chlong has quit IRC | 06:55 | |
| *** haomaiwa_ has quit IRC | 07:01 | |
| *** haomaiwang has joined #openstack-swift | 07:01 | |
| *** yatin has quit IRC | 07:06 | |
| *** yatin has joined #openstack-swift | 07:08 | |
| *** Jeffrey4l_ has quit IRC | 07:11 | |
| *** yarkot has quit IRC | 07:15 | |
| *** venkat has joined #openstack-swift | 07:19 | |
| *** Jeffrey4l_ has joined #openstack-swift | 07:29 | |
| *** yatin has quit IRC | 07:31 | |
| *** Jeffrey4l_ has quit IRC | 07:59 | |
| *** haomaiwang has quit IRC | 08:01 | |
| *** haomaiwa_ has joined #openstack-swift | 08:01 | |
| *** arnox has joined #openstack-swift | 08:09 | |
| *** rledisez has joined #openstack-swift | 08:09 | |
| *** Jeffrey4l_ has joined #openstack-swift | 08:12 | |
| *** arnox has quit IRC | 08:14 | |
| *** arnox has joined #openstack-swift | 08:15 | |
| *** jordanP has joined #openstack-swift | 08:20 | |
| kota_ | anyone? | 08:20 |
| mahatic | kota_: what's up? | 08:21 |
| kota_ | mahatic: o/ i have a question for Swift :) | 08:21 |
| *** eranrom has joined #openstack-swift | 08:22 | |
| kota_ | mahatic: i am reviewing a swift3 patch from timburke | 08:22 |
| mahatic | ah I see | 08:22 |
| kota_ | mahatic: that patch is doing to make a copy request with *Range* header. | 08:22 |
| kota_ | mahatic: but *right now* i am not sure that would be supported officially | 08:22 |
| kota_ | like | 08:23 |
| mahatic | kota_: COPY with Range header? | 08:23 |
| kota_ | mahatic: yes | 08:23 |
| kota_ | curl -H 'X-Auth-Token: xxx' http://host/v1/a/c/o2 -H 'X-Copy-From: /c/o' -H 'Range: bytes=1-' -XPUT | 08:24 |
| kota_ | that worked in my local lab environment. | 08:24 |
| kota_ | but i didn't find the docs | 08:24 |
| kota_ | and currently not sure can we use "Range" header for *PUT* request | 08:25 |
| kota_ | i am worried that it *might* broke HTTP specification... | 08:25 |
| mahatic | kota_: I think PUT range is not yet there, but range header in COPY, not sure - looking | 08:25 |
| kota_ | not sure. | 08:25 |
| kota_ | s/might broke/might break/ | 08:26 |
| kota_ | mahatic: yeah, i think so. i am wondering with your boat. | 08:26 |
| *** yatin has joined #openstack-swift | 08:27 | |
| mahatic | kota_: PUT with range header does seem to break http spec. hence no Range header on COPY. But fast-post may solve it? | 08:30 |
| *** Jeffrey4l__ has joined #openstack-swift | 08:32 | |
| mahatic | oh it doesn't I suppose, it's probably not related | 08:33 |
| kota_ | mahatic: i don't think fast-post forcuses on that problem | 08:33 |
| mahatic | kota_: yeah | 08:33 |
| *** Jeffrey4l_ has quit IRC | 08:33 | |
| kota_ | yup, and i also found some docs about the range | 08:33 |
| kota_ | on *RFC* | 08:34 |
| kota_ | https://tools.ietf.org/html/rfc7233 | 08:34 |
| kota_ | section 3.1 | 08:34 |
| kota_ | The "Range" header field on a GET request modifies the method.... | 08:34 |
| mahatic | kota_: I am looking at it too, I thought you were asking on swift docs, my bad | 08:34 |
| kota_ | no worries | 08:35 |
| kota_ | Is the conclusion that "it seems to break the rfc"? | 08:36 |
| mahatic | kota_: yup, looks like it "A server MUST ignore a Range header field received with a request method other than GET." | 08:37 |
| kota_ | mahatic: ok, thanks. | 08:37 |
| kota_ | mahatic: i will raise up this issue to next irc meeting | 08:37 |
| mahatic | kota_: sure, I still am not aware of the swift3 patch though :) | 08:38 |
| kota_ | and if no one has concens to fix (disallow some requests like PUT with range), i will push a patch. | 08:39 |
| mahatic | great | 08:41 |
| mahatic | sounds good | 08:41 |
| *** peterlisak has joined #openstack-swift | 08:48 | |
| mattoliverau | kota_: ranged copy is new, so wont be in any stables swifts yet.. that might make it hard for swift3 | 08:55 |
| kota_ | mattolivearu: newly supported? | 08:56 |
| kota_ | perhaps, should we use "If-Range" instead of pure "Range"? | 08:59 |
| kota_ | reading... | 08:59 |
| kota_ | reading RFC... | 09:00 |
| *** links has quit IRC | 09:00 | |
| *** haomaiwa_ has quit IRC | 09:01 | |
| *** haomaiwang has joined #openstack-swift | 09:01 | |
| mahatic | ranged copy is supported? | 09:01 |
| kota_ | mahatic: i don't think but...mattoliverau may know something... | 09:03 |
| mahatic | if-range is probably not the one - it sends over the missing parts if the obj is unchanged or will send the entire of it | 09:03 |
| *** venkat has quit IRC | 09:03 | |
| kota_ | mahaic: exactly | 09:03 |
| *** links has joined #openstack-swift | 09:04 | |
| kota_ | my bad | 09:05 |
| *** km_ has quit IRC | 09:05 | |
| mattoliverau | kota_: sorry that was more of a question. How is swift3 as a middleware released. If the underlying swift install doesn't support a feature then you can' just use that feature or do swift3 releases match swift releases. | 09:06 |
| mattoliverau | I've seen something about ranged copies, it's either a landed patch or one in review.. (or I dreamt it while on holiday) :P | 09:06 |
| kota_ | mattoliverau: swift3 didn't do that yet. | 09:06 |
| kota_ | mattoliverau: https://review.openstack.org/#/c/255069/8/swift3/controllers/multi_upload.py | 09:07 |
| mattoliverau | by ranged copies I mean ranged GET resulting in a new object | 09:07 |
| *** ppai has quit IRC | 09:07 | |
| kota_ | timburke proposed that patch and I found *current Swift* allows such a request. | 09:08 |
| mattoliverau | k, so it's landed then, but it might only be in master. so probably wont work in liberty or kilo | 09:08 |
| kota_ | mattoliveau: really? swift3 gate are testing on kilo stable but jenkins shows +1... | 09:09 |
| mattoliverau | which is why I was asking how swift3 releases work, ie, if people just download master and use that on a swift stable they might have trouble using that feature. | 09:09 |
| kota_ | mattolivearu: can i have the commit to allow the range copy? | 09:09 |
| mattoliverau | oh really, I didn't think it was that long ago | 09:10 |
| mattoliverau | sure, let me find it | 09:10 |
| kota_ | mattoliverau: yup, waiting :) | 09:11 |
| *** jistr has joined #openstack-swift | 09:12 | |
| *** joeljwright has joined #openstack-swift | 09:12 | |
| *** ChanServ sets mode: +v joeljwright | 09:12 | |
| mattoliverau | kota_: maybe I'm dreaming or confusing some other patch, cause I can't find it, and on the other hand why would'nt a ranged COPY work seeing as thats just a GET then a PUT. | 09:15 |
| *** venkat has joined #openstack-swift | 09:15 | |
| kota_ | mattoliverau: i think it might ok that COPY with range because it will request for source object like GET | 09:18 |
| mattoliverau | kota_: yup, I'm thinking so too | 09:18 |
| kota_ | mattoliverau: but PUT -H 'X-Copy-From: xxx' is obviously for the destination object | 09:19 |
| mattoliverau | yeah, the PUT path would hopefully ignore that header | 09:20 |
| kota_ | not sure but it seems that the range affects to downstream constraint. | 09:20 |
| kota_ | but right now the PUT -H 'X-Copy-From' with range works well :/ | 09:21 |
| kota_ | (i mean the request will make a partial copied object) | 09:21 |
| kota_ | that's my question, "Is it expected behavior?" | 09:22 |
| kota_ | mattolivearu: i think we are in same boat "the PUT path hopfully ignore (or disallow) range header" | 09:24 |
| kota_ | if we want to do that, use COPY or make a new header for swift apis, imo. | 09:25 |
| mattoliverau | kota_: multi-ranged GETs return a multipart mime, so not sure if that would work.. though I've never tested it. | 09:26 |
| kota_ | mattoliverau: exactly, i also think i would see somes issue for COPY with multi-range at the time of EC work... | 09:28 |
| kota_ | s/somes issue/some issues/ | 09:28 |
| kota_ | mattolivearu: I will continue to work around there thanks o/ | 09:30 |
| mahatic | kota_: but that isn't the issue right - PUT is accepting Range and need to know if that should happen or not is what you're looking for? | 09:30 |
| mattoliverau | mahatic: yeah, I that's his question, I'm just thinking out loud :) | 09:32 |
| mahatic | mattoliverau: right, okay :) I'll shut it :P | 09:34 |
| mattoliverau | A PUT puts or overwrites an object (Ie you can't append), so Range means nothing to it, so should be ignored. So I don't see it as a problem if a Range header is added, so long as it is ingored (like it would be like adding any x header that swift doesn't know about). So long as it is ignored in the PUT path. | 09:36 |
| mattoliverau | mahatic: nah speak up, I could very well be wrong ;) | 09:36 |
| mattoliverau | tho if there is an RFC somewhere saying you can't then I'll shut up :P | 09:38 |
| mahatic | mahatic: heh. yeah, Range is ignored in a regular PUT anyway | 09:43 |
| mahatic | in Swift | 09:44 |
| mahatic | mattoliverau: * | 09:47 |
| kota_ | mattoliverau, mahatic: thanks! | 09:49 |
| *** goodygum has joined #openstack-swift | 09:50 | |
| *** jmccarthy has joined #openstack-swift | 09:54 | |
| *** Jeffrey4l__ has quit IRC | 09:58 | |
| *** haomaiwang has quit IRC | 10:01 | |
| *** haomaiwang has joined #openstack-swift | 10:01 | |
| ahale | a range'd put sounds more like what i'd expect a patch req to do, if they existed in swift | 10:01 |
| *** openstackgerrit has quit IRC | 10:02 | |
| *** openstackgerrit has joined #openstack-swift | 10:03 | |
| *** Jeffrey4l__ has joined #openstack-swift | 10:11 | |
| *** aix has joined #openstack-swift | 10:15 | |
| *** jistr has quit IRC | 10:24 | |
| *** jistr has joined #openstack-swift | 10:30 | |
| *** jistr has quit IRC | 10:30 | |
| *** haomaiwang has quit IRC | 11:01 | |
| *** 17WABG26P has joined #openstack-swift | 11:01 | |
| *** kei_yama has quit IRC | 11:07 | |
| *** joker_ has joined #openstack-swift | 11:13 | |
| *** daemontool has joined #openstack-swift | 11:29 | |
| *** yatin has quit IRC | 11:36 | |
| *** jistr has joined #openstack-swift | 11:41 | |
| *** Jeffrey4l__ has quit IRC | 11:43 | |
| *** 17WABG26P has quit IRC | 12:01 | |
| *** haomaiwang has joined #openstack-swift | 12:01 | |
| *** chlong has joined #openstack-swift | 12:02 | |
| *** ntt has joined #openstack-swift | 12:04 | |
| ntt | Hi, it is possible to change the replication network in a running swift cluster? | 12:04 |
| *** joeljwright has quit IRC | 12:05 | |
| openstackgerrit | Merged openstack/swift: Add round-trip encrypter/decrypter unit tests https://review.openstack.org/251606 | 12:06 |
| openstackgerrit | renminmin proposed openstack/swift: Byte-quotas should allow object re-upload https://review.openstack.org/263227 | 12:07 |
| *** silor has joined #openstack-swift | 12:07 | |
| *** joeljwright has joined #openstack-swift | 12:10 | |
| *** ChanServ sets mode: +v joeljwright | 12:10 | |
| *** CaioBrentano has joined #openstack-swift | 12:22 | |
| *** silor1 has joined #openstack-swift | 12:26 | |
| *** silor has quit IRC | 12:28 | |
| *** silor1 is now known as silor | 12:28 | |
| *** links has quit IRC | 12:34 | |
| *** SkyRocknRoll has quit IRC | 12:39 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:42 | |
| *** zul has quit IRC | 12:47 | |
| *** zul has joined #openstack-swift | 12:47 | |
| *** jistr has quit IRC | 12:50 | |
| kota_ | hmm...all conditional headers look to be sent to the source object and ignored on PUT destination. | 12:54 |
| *** hseipp has joined #openstack-swift | 12:58 | |
| *** venkat has quit IRC | 12:58 | |
| kota_ | semantically If-Match allowed on PUT in RFC but If-Modified-Since isn't. | 12:59 |
| kota_ | Swift looks like not to both, tho. | 13:00 |
| *** haomaiwang has quit IRC | 13:01 | |
| *** haomaiwang has joined #openstack-swift | 13:01 | |
| *** jistr has joined #openstack-swift | 13:02 | |
| *** dosaboy_ is now known as dosaboy | 13:05 | |
| *** mariusv has joined #openstack-swift | 13:05 | |
| *** mariusv has joined #openstack-swift | 13:05 | |
| *** haomaiwang has quit IRC | 13:06 | |
| *** dustins has joined #openstack-swift | 13:06 | |
| *** 17WABG31A has joined #openstack-swift | 13:09 | |
| *** proteusguy has quit IRC | 13:16 | |
| *** vinsh_ is now known as Vinsh | 13:19 | |
| *** chlong has quit IRC | 13:20 | |
| *** Jeffrey4l__ has joined #openstack-swift | 13:24 | |
| *** 17WABG31A has quit IRC | 13:28 | |
| *** proteusguy has joined #openstack-swift | 13:29 | |
| *** peterlisak has left #openstack-swift | 13:31 | |
| *** chlong has joined #openstack-swift | 13:32 | |
| *** daemontool has quit IRC | 13:33 | |
| *** daemontool has joined #openstack-swift | 13:33 | |
| *** daemontool has quit IRC | 13:33 | |
| *** daemontool has joined #openstack-swift | 13:34 | |
| *** dslevin_ has joined #openstack-swift | 13:41 | |
| openstackgerrit | James Nzomo proposed openstack/python-swiftclient: Fix upload to pseudo-dir passed by <container> arg https://review.openstack.org/263259 | 13:43 |
| *** dslevin_ has quit IRC | 13:50 | |
| *** dslevin_ has joined #openstack-swift | 13:51 | |
| *** dslevin_ has quit IRC | 13:51 | |
| *** dslevin_ has joined #openstack-swift | 13:52 | |
| *** dustins has quit IRC | 13:54 | |
| *** links has joined #openstack-swift | 13:56 | |
| *** haomaiwang has joined #openstack-swift | 14:06 | |
| *** links has quit IRC | 14:07 | |
| *** breitz has joined #openstack-swift | 14:30 | |
| *** jistr has quit IRC | 14:30 | |
| *** jistr has joined #openstack-swift | 14:31 | |
| *** SkyRocknRoll has quit IRC | 14:33 | |
| *** jlvacation is now known as jlvillal | 14:40 | |
| *** tamizh_geek_ is now known as tamizh_geek | 14:56 | |
| *** tamizh_geek has left #openstack-swift | 14:57 | |
| *** tamizh_geek has joined #openstack-swift | 14:57 | |
| *** haomaiwang has quit IRC | 15:01 | |
| *** blmartin has joined #openstack-swift | 15:01 | |
| *** haomaiwa_ has joined #openstack-swift | 15:01 | |
| *** dustins has joined #openstack-swift | 15:05 | |
| *** tsg has joined #openstack-swift | 15:13 | |
| *** SkyRocknRoll has joined #openstack-swift | 15:14 | |
| *** shakamunyi has quit IRC | 15:15 | |
| *** barra204 has quit IRC | 15:15 | |
| pdardeau | good morning! happy new year! | 15:27 |
| *** petertr7 is now known as petertr7_away | 15:29 | |
| pchng | happy new year :) | 15:29 |
| *** shakamunyi has joined #openstack-swift | 15:29 | |
| *** mragupat has joined #openstack-swift | 15:30 | |
| *** petertr7_away is now known as petertr7 | 15:34 | |
| openstackgerrit | Azhagu Selvan SP proposed openstack/swift: Removed the unnecessary comment about FakeRing._get_parts method reported in bug #1488704 https://review.openstack.org/263307 | 15:36 |
| openstack | bug 1488704 in OpenStack Object Storage (swift) "FakeRing does fake get_part anymore" [Low,New] https://launchpad.net/bugs/1488704 - Assigned to Aniruddha Singh Gautam (aniruddha-gautam) | 15:36 |
| openstackgerrit | Azhagu Selvan SP proposed openstack/swift: Removed the unnecessary comment about FakeRing._get_parts method reported in bug #1488704 https://review.openstack.org/263307 | 15:43 |
| openstack | bug 1488704 in OpenStack Object Storage (swift) "FakeRing does fake get_part anymore" [Low,New] https://launchpad.net/bugs/1488704 - Assigned to Aniruddha Singh Gautam (aniruddha-gautam) | 15:43 |
| wbhuber | happy new year, swift'ers! | 15:45 |
| *** mtreinish has quit IRC | 15:46 | |
| *** mtreinish has joined #openstack-swift | 15:48 | |
| *** eranrom has quit IRC | 15:48 | |
| *** minwoob has joined #openstack-swift | 15:54 | |
| *** tsg has quit IRC | 15:57 | |
| *** tsg_ has joined #openstack-swift | 15:57 | |
| *** corvus is now known as jeblair | 15:58 | |
| *** haomaiwa_ has quit IRC | 16:01 | |
| *** haomaiwang has joined #openstack-swift | 16:01 | |
| *** klrmn has joined #openstack-swift | 16:14 | |
| *** aix has quit IRC | 16:26 | |
| *** trifon has quit IRC | 16:31 | |
| tamizh_geek | on swift master, I couldn't do a pip install -r requirements.txt | 16:32 |
| tamizh_geek | I get this - ValueError: ("Expected ',' or end-of-list in", "dnspython>=1.12.0;python_version<'3.0'", 'at', ";python_version<'3.0'") | 16:32 |
| *** diazjf has joined #openstack-swift | 16:32 | |
| acoles | tamizh_geek: make sure pip, setuptools and pbr are all up to date | 16:35 |
| *** zaitcev has joined #openstack-swift | 16:41 | |
| *** ChanServ sets mode: +v zaitcev | 16:41 | |
| *** garthb has joined #openstack-swift | 16:46 | |
| *** mragupat has quit IRC | 16:49 | |
| *** mragupat has joined #openstack-swift | 16:50 | |
| *** barra204 has joined #openstack-swift | 16:51 | |
| *** haomaiwang has quit IRC | 16:51 | |
| *** shakamunyi has quit IRC | 16:51 | |
| *** 21WAAO7OX has joined #openstack-swift | 16:54 | |
| notmyname | good morning | 16:56 |
| *** 21WAAO7OX has quit IRC | 17:01 | |
| *** haomaiwang has joined #openstack-swift | 17:01 | |
| notmyname | +1 for email subject line honesty. "Checking in - I'm a recruiter" | 17:03 |
| *** dustins has quit IRC | 17:03 | |
| *** diazjf has quit IRC | 17:06 | |
| *** klrmn has quit IRC | 17:07 | |
| *** joeljwright has quit IRC | 17:09 | |
| pdardeau | notmyname: i'd like to get your opinion on something re: device count enhancement when you have some time | 17:11 |
| *** diogogmt has joined #openstack-swift | 17:11 | |
| *** gyee has joined #openstack-swift | 17:14 | |
| notmyname | pdardeau: what's up? | 17:19 |
| pdardeau | notmyname: i heeded your advice and built support for varying size of device id in ring | 17:20 |
| notmyname | oh cool! | 17:20 |
| pdardeau | my question is related to configuration part | 17:20 |
| pdardeau | i'm wondering whether a setting that automatically shifted | 17:21 |
| pdardeau | between 2 and 4 bytes would be useful (or desirable)? | 17:21 |
| *** CaioBren_ has joined #openstack-swift | 17:21 | |
| notmyname | pdardeau: what are you thinking? where or when would you want to change it? | 17:21 |
| *** CaioBren_ has joined #openstack-swift | 17:22 | |
| pdardeau | so, here's the dilemma that i see. if it's explicitly set to 2, then it all has to be recreated when needing to go beyond 65536 devices | 17:22 |
| pdardeau | if it's explicitly set to 4 bytes, then extra memory is used (unnecessarily) when less than 65536 devices are in ring | 17:23 |
| *** jistr has quit IRC | 17:23 | |
| pdardeau | so the big question is: can a ring configured for using 2 bytes per device id ever transition to using more than 65536 devices | 17:24 |
| *** yarkot has joined #openstack-swift | 17:24 | |
| pdardeau | and: can a ring configured for using 4 bytes per device id ever 'shrink' down to 2 bytes if number of devices contracts to below 65536 devices | 17:24 |
| *** CaioBrentano has quit IRC | 17:25 | |
| pdardeau | so i'm considering the possibility of having 3 valid values for configuration: 2,4, or 'auto' | 17:25 |
| pdardeau | where 'auto' means to automagically expand/contract as number of devices goes above (or drops below) 65536 | 17:26 |
| pdardeau | does any of that make sense? | 17:26 |
| notmyname | yeah. I need to get my brain up to speed again :-) | 17:28 |
| pdardeau | another way of looking at the question: should amount of storage used per device id be explicitly configured or should it automatically adapt as needed? | 17:28 |
| *** rledisez has quit IRC | 17:28 | |
| notmyname | my gut reaction is that it should be auto. without another config option | 17:28 |
| *** rledisez has joined #openstack-swift | 17:28 | |
| pdardeau | ok. i didn't have a good feel for what would be the more sensible option. | 17:30 |
| *** rledisez has quit IRC | 17:32 | |
| *** jordanP has quit IRC | 17:39 | |
| *** diazjf has joined #openstack-swift | 17:40 | |
| gmmaha | good morning!! | 17:40 |
| *** petertr7 is now known as petertr7_away | 17:41 | |
| pdardeau | gmmaha: good morning. and happy new year! | 17:44 |
| *** lcurtis has joined #openstack-swift | 17:44 | |
| pdardeau | :1 | 17:45 |
| gmmaha | happy new year pdardeau :) | 17:45 |
| pdardeau | tabfail | 17:45 |
| *** hseipp has quit IRC | 17:47 | |
| *** arnox has quit IRC | 18:00 | |
| *** haomaiwang has quit IRC | 18:01 | |
| *** haomaiwang has joined #openstack-swift | 18:01 | |
| *** acoles is now known as acoles_ | 18:02 | |
| *** klrmn has joined #openstack-swift | 18:03 | |
| *** mragupat has quit IRC | 18:05 | |
| *** mragupat has joined #openstack-swift | 18:06 | |
| *** mragupat has quit IRC | 18:08 | |
| *** mragupat has joined #openstack-swift | 18:08 | |
| *** badari has joined #openstack-swift | 18:09 | |
| *** yarkot has quit IRC | 18:25 | |
| *** openstackgerrit has quit IRC | 18:32 | |
| *** openstackgerrit has joined #openstack-swift | 18:32 | |
| *** petertr7_away is now known as petertr7 | 18:36 | |
| *** silor has quit IRC | 18:39 | |
| *** silor has joined #openstack-swift | 18:40 | |
| clayg | pdardeau: i still don't understand who is asking for a ring with more than 65K devices - it's just nuts | 18:43 |
| *** ChubYann has joined #openstack-swift | 18:44 | |
| pdardeau | clayg: i kinda feel the same way, but i don't have any firsthand knowledge of large swift clusters | 18:44 |
| pdardeau | clayg: iirc, RAX requested it | 18:45 |
| clayg | pdardeau: i'm sure cloud files has not strong needs to increase devices in a ring beyond some thousands - 10's of thousands is not realistic in a single storage policy | 18:46 |
| clayg | IMHO | 18:46 |
| *** eranrom has joined #openstack-swift | 18:46 | |
| *** diazjf has quit IRC | 18:47 | |
| clayg | ... how about optomizing rebalance so it doesn't take many many hours and thousands of megabytes of ram to rebalance a ring with partpower > 24? a) that would be useful to everyone b) i'd be on of *many* pre-req to having a good time with +10K devices in a ring | 18:49 |
| *** SkyRocknRoll has quit IRC | 18:53 | |
| notmyname | clayg: I agree, but one small point is that the 65k limit is/was hit because of holes in the ring instead of actually having 65k devices. so it's larger clusters + age. (IIRC this was info from ahale) | 18:54 |
| clayg | has anyone made it through the latest jespen/aphyr on rethinkdb? | 18:55 |
| notmyname | I hadn't seen it yet | 18:55 |
| clayg | notmyname: ok, well that's still then - the result of that problem should be we need to figure out why it's not re-using device id's - who wants to carry around all those None's anyway? | 18:55 |
| ahale | yeah thats a problem, i havent looked on recent code .. but looks like its still gonna happen with https://github.com/openstack/swift/blob/master/swift/common/ring/builder.py#L339 | 18:56 |
| *** diazjf has joined #openstack-swift | 18:56 | |
| clayg | ahale: yeah seems a lot easier to fix that with index(None) if -1 then len() + 1 | 18:57 |
| *** doxavore has joined #openstack-swift | 19:00 | |
| *** haomaiwang has quit IRC | 19:01 | |
| *** haomaiwang has joined #openstack-swift | 19:01 | |
| doxavore | Running swift 2.2.2, after 2 drives failed and were subsequently removed from the ring (and ring has been rebalanced/distributed), a drive on a separate node has filled up and doesn't seem to be draining except for very short periods where I'll see ~200MB free up. (A system rebalance is still in progress.) Is there a safe way to speed up the deletion of unneeded files on that drive to bring the system back to a | 19:04 |
| doxavore | state that isn't extremely slow/timing out? | 19:04 |
| *** zhill has joined #openstack-swift | 19:05 | |
| *** dslevin has quit IRC | 19:06 | |
| *** sc68cal has quit IRC | 19:11 | |
| *** dslevin has joined #openstack-swift | 19:12 | |
| *** sc68cal has joined #openstack-swift | 19:14 | |
| clayg | doxavore: you can simulate this -> https://review.openstack.org/#/c/215867/ - by setting handoffs_first to true and handoff_delete the replica_count - 1 ; then just keep restarting replicators every so often | 19:14 |
| clayg | doxavore: was the cluster already near capacity? Did you *replace* the failed devices? | 19:15 |
| clayg | doxavore: sometimes its better to just "swap" the failed devices w/o a rebalance vs. rebalancing all their parts off of them, then rebalancing a bunch of new/different parts back on? | 19:15 |
| clayg | doxavore: you should also up your rsync max_connections - more i/o will get tied up in replication so your request latency may take a hit | 19:16 |
| doxavore | i just removed the failed drives from the ring entirely (i don't make it to this facility very often, and we see request latency shoot up when we run with some failed drives left in the ring.) the cluster is running at about 75% capacity, across ~75 drives | 19:18 |
| pdardeau | clayg, notmyname, ahale: based on current thinking, is there still a need to allow for more than 65K device slots in ring? | 19:20 |
| *** openstackstatus has quit IRC | 19:20 | |
| *** openstackstatus has joined #openstack-swift | 19:23 | |
| *** ChanServ sets mode: +v openstackstatus | 19:23 | |
| ahale | i dunno really - yeah probably - im not convinced that migrating to more storage policies and maintaining that is a great alternative | 19:23 |
| ahale | something to defrag the devs would be good though and push that problem further away for us at least. whether thats a new command or something done when deleting idk. | 19:24 |
| ahale | and i kinda thing builder should do more checking, so it wont assign device id's that it will error trying to rebalance. warn instead or something | 19:25 |
| torgomatic | clayg: filling in dev-id holes is definitely the way to go IMO | 19:34 |
| *** changbl has quit IRC | 19:35 | |
| *** joeljwright has joined #openstack-swift | 19:35 | |
| *** ChanServ sets mode: +v joeljwright | 19:35 | |
| *** wer has quit IRC | 19:38 | |
| *** wer has joined #openstack-swift | 19:38 | |
| notmyname | pdardeau: IMO, it's good to remove any particular device id limit in swift. but it's probably not the first thing to alleviate immediate problems | 19:44 |
| notmyname | but how complicated is the patch overall? | 19:44 |
| pdardeau | notmyname: changes to ringbuilder.py, builder.py, and ring.py and 3 unit tests | 19:47 |
| openstackgerrit | James Nzomo proposed openstack/python-swiftclient: Fix upload to pseudo-dir passed by <container> arg https://review.openstack.org/263259 | 19:54 |
| *** daemontool has quit IRC | 19:59 | |
| notmyname | pdardeau: sure, but is it a lot of changes? tricky changes? | 20:00 |
| *** haomaiwang has quit IRC | 20:01 | |
| hrou | clayg, acoles_, torgomatic, all - For symlinks (namely the handling of POSTs) we wanted to rehash a few of the original proposals to fully understand the inconsistency issues involved (even if it just amounts to documenting them I think it'll have some value : - ), would you mind taking a look at the following etherpad and inlining any thoughts / questions / comments you may have ! https://etherpad.openstack.org/p/swift_symlink_post_strategy | 20:01 |
| *** haomaiwang has joined #openstack-swift | 20:01 | |
| *** blmartin has quit IRC | 20:01 | |
| notmyname | pdardeau: to me it's a cost-benefit thing. someday, sure, I'd like to have no dev_id limits. so how does your current patch to do that compare to stuff like filling in dev-id holes (which should also be done)? | 20:01 |
| *** blmartin has joined #openstack-swift | 20:03 | |
| *** cdelatte has joined #openstack-swift | 20:05 | |
| Zyric | Good Morning:) | 20:15 |
| *** tristanC has joined #openstack-swift | 20:18 | |
| *** robefran has joined #openstack-swift | 20:19 | |
| *** aix has joined #openstack-swift | 20:22 | |
| onovy | hi. can someone explain me dependencies between pyeclib, liberasurecode and libjerasure? | 20:45 |
| notmyname | Zyric: hello | 20:45 |
| onovy | and swift of course | 20:45 |
| notmyname | onovy: swift requires pyeclib. which uses the c library liberasurecode. liberasurecode has some simple stuff in it, but faster/better algorithms and libraries can be used. it's pluggable. | 20:46 |
| notmyname | one library which works is jerasure | 20:46 |
| onovy | and libjerasure? | 20:46 |
| notmyname | another is ISA-L | 20:46 |
| onovy | hmm | 20:46 |
| onovy | so swift needs pyeclib only? | 20:46 |
| onovy | and pyeclib needs liberasure, which needs libjerasure? | 20:47 |
| onovy | because unit tests is failing for me if libjerasure is not installed | 20:47 |
| notmyname | no, libjerasure isn't needed (with current versions of liberasurecode) | 20:47 |
| onovy | so pyeclib needs only liberasure | 20:48 |
| onovy | and libjerasure is optional | 20:48 |
| notmyname | yes | 20:48 |
| *** eranrom has quit IRC | 20:48 | |
| notmyname | that's the way it's supposed to work | 20:48 |
| *** joeljwright has quit IRC | 20:49 | |
| onovy | is it recommended to have libjerasure? | 20:50 |
| notmyname | no. or not as compared to anything else | 20:51 |
| notmyname | basically there are 2 free options I know of, and you should use one of them. jerasure and ISA-L | 20:51 |
| onovy | and how liberasure choose which algo is used? | 20:51 |
| onovy | or pyeclib chooses? | 20:52 |
| onovy | or swift? :) | 20:52 |
| notmyname | based on the config in swift.conf for the storage policy | 20:52 |
| notmyname | eg: ec_type = jerasure_rs_vand | 20:52 |
| notmyname | or ec_type = isa_l_rs_vand | 20:52 |
| onovy | and liberasure without jerasure OR ISA-L will work? | 20:52 |
| onovy | with swift, not theoretically | 20:52 |
| notmyname | yes, but you're more limited to the erasure codes you can use. IIRC you can only do xor codes in liberasurecode. I dont' think you can do reed-solomon codes | 20:53 |
| onovy | hmm | 20:53 |
| onovy | Failure: PolicyError (Error creating EC policy (pyeclib_c_init ERROR: Invalid arguments. Please inspect syslog for liberasurecode error report.), for index 0) ... ERROR | 20:54 |
| onovy | liberasurecode[15792]: liberasurecode_backend_open: dynamic linking error libJerasure.so.2: cannot open shared object file: No such file or directory | 20:54 |
| onovy | when running unit tests | 20:54 |
| notmyname | current HEAD of master? | 20:55 |
| onovy | 2.5.0 | 20:56 |
| notmyname | ah, ok | 20:56 |
| onovy | in unit tests is: ECStoragePolicy(3, 'ec', ec_type='jerasure_rs_vand', | 20:56 |
| notmyname | so yeah, that's expected there | 20:56 |
| onovy | so unit tests NEEDS jerasure | 20:56 |
| notmyname | not anymore, but it did in 2.5.0 | 20:56 |
| *** PsionTheory has joined #openstack-swift | 20:56 | |
| onovy | hmmm | 20:56 |
| onovy | ok :) | 20:56 |
| onovy | and will 2.5.0 swift work without jerasure? | 20:57 |
| onovy | so it's only unit tests deps? | 20:57 |
| notmyname | yes | 20:57 |
| onovy | ok, so it's build deps | 20:57 |
| *** haomaiwang has quit IRC | 21:01 | |
| onovy | testing... | 21:01 |
| *** haomaiwang has joined #openstack-swift | 21:01 | |
| onovy | notmyname: fixed, thanks! http://anonscm.debian.org/cgit/openstack/swift.git/commit/?id=940cd2648c13c4fd2c8bb578752c25e344b74837 | 21:04 |
| *** minwoob has quit IRC | 21:09 | |
| pdardeau | notmyname: i wouldn't say that the patch is complex or tricky, but it does not currently have any changes for reuse of holes or any defrag capabilities | 21:14 |
| mattoliverau | morning | 21:16 |
| pdardeau | mattoliverau: good morning | 21:16 |
| *** petertr7 is now known as petertr7_away | 21:20 | |
| *** trifon has joined #openstack-swift | 21:20 | |
| *** silor has quit IRC | 21:22 | |
| clayg | I think "supporting > 65K devices in a ring" is not a great goal. We should should aim to support >65K devices in a *replicated storage policy in swift used in production*. Fixing the ring sometime after we have a replicated storage policy working in production with 10's of thousands of devices should be trivial compared to the other work we did to get there. | 21:29 |
| clayg | I didn't know about ahale's reusing device id's thing - but that's just a bug with how we remove devices and burn device id's - garbage collection is a thing - we should just fix that. | 21:29 |
| notmyname | clayg: yeah, I think that's a decent rephrasing | 21:30 |
| clayg | notmyname: well sans all my attitude - I think my heart is in the right place | 21:31 |
| notmyname | lol | 21:31 |
| ahale | but saying "just add more storage policies" without saying "and then identify your large and growing containers, migrate them, maintain balance so ring0 stays below 64k forever while monitoring usage across all policies and remember to review distribution" is a bit like saying "screw ops". whats so bad about 66k devices vs 63k ? | 21:32 |
| notmyname | ahale: oh yeah, I'm totally with you that storage policies aren't the way to grow beyond a drive limit. at least not "policies" as they're implemented today | 21:33 |
| pdardeau | notmyname: here's a high-level synopsis of current patch | 21:39 |
| pdardeau | created a new version of ringdata. new version has device id size (in bytes) | 21:39 |
| pdardeau | that piece of data added to serialization/deserialization | 21:39 |
| pdardeau | that piece of data is needed to know how to deserialize the device ids (whether they're 2 bytes or 4 bytes each) | 21:40 |
| pdardeau | when new devices are added, builder looks to see if the count will cross the 65k boundary (if currently using 2 bytes) | 21:41 |
| pdardeau | if so, device id's have to be reconstructed using 4 bytes instead of 2 | 21:42 |
| pdardeau | was thinking to also modify remove device to see if 65k is crossed going other way and if so, can rewrite using 2 bytes per dev id instead of 4 | 21:42 |
| notmyname | pdardeau: was the 2 vs 4 byte choice simpler than something like "bits = log2(max_dev_id) + 1" so it uses however many it needs? | 21:48 |
| *** badari has quit IRC | 21:48 | |
| pdardeau | notmyname: yes, i believe so | 21:49 |
| notmyname | pdardeau: but that's kinda getting into the weeds. the important thing is if this is the best way to solve the problem at hand. which is dev count + holes in the ring being more than 64k | 21:49 |
| pdardeau | notmyname: agreed | 21:50 |
| *** robefran has quit IRC | 21:51 | |
| pdardeau | notmyname: the aspect of the holes is new to me | 21:52 |
| peluse | pdardeau: notmyname: the >65K thing came up from rax as part of the cloud ofr all stuff... seems like filling in holes is a good thing regarldess. whether it solves their immediate problem or not, I don't know but pdardeau is basically operating off of a requirement that they put forward. we can certainly ask them if that stops the bleeding or not | 21:56 |
| notmyname | peluse: yup. I agree | 21:57 |
| peluse | notmyname: cool... if it does then we should take care the holes thing and go revisit the priority list again, maybe actually icnreasing >65K falls off.... | 21:57 |
| *** diogogmt has quit IRC | 21:57 | |
| peluse | we'll check for sure | 21:57 |
| *** mragupat_ has joined #openstack-swift | 21:58 | |
| *** trifon has quit IRC | 21:59 | |
| peluse | pdardeau: there's a meeting Thu this week, i'll add a few folks to the email you sent to Matt and I and see if we can get an answer that way... (or at least find out hwo had 1st hand knowledge of the problem) | 21:59 |
| *** badari has joined #openstack-swift | 21:59 | |
| pdardeau | peluse: thanks! sounds like a plan | 21:59 |
| *** haomaiwang has quit IRC | 22:01 | |
| *** haomaiwang has joined #openstack-swift | 22:01 | |
| *** mragupat has quit IRC | 22:01 | |
| *** changbl has joined #openstack-swift | 22:06 | |
| onovy | notmyname: can you look to https://review.openstack.org/#/c/229532/ pls? | 22:27 |
| *** robefran has joined #openstack-swift | 22:36 | |
| *** darrenc is now known as darrenc_afk | 22:39 | |
| *** darrenc_afk is now known as darrenc | 22:54 | |
| *** blmartin has quit IRC | 22:58 | |
| *** mragupat_ has quit IRC | 22:59 | |
| *** diazjf has quit IRC | 22:59 | |
| *** haomaiwang has quit IRC | 23:01 | |
| *** haomaiwang has joined #openstack-swift | 23:01 | |
| *** km__ has joined #openstack-swift | 23:04 | |
| *** zhill has quit IRC | 23:12 | |
| *** kei_yama has joined #openstack-swift | 23:23 | |
| notmyname | just reran my community stats for the first time. looks like we broke 500 contributors sometime in the last two weeks | 23:24 |
| *** doxavore has quit IRC | 23:30 | |
| mattoliverau | \o/ | 23:37 |
| *** breitz has quit IRC | 23:55 | |
| *** breitz has joined #openstack-swift | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!