| *** annegentle has joined #openstack-swift | 00:00 | |
| *** petertr7_away is now known as petertr7 | 00:03 | |
| *** annegentle has quit IRC | 00:05 | |
| clayg | notmyname: good answer on the object data cache question | 00:08 |
|---|---|---|
| notmyname | thanks | 00:08 |
| clayg | notmyname: you could have mentioned that you can turn off the fadvise thing with the config option to tune it for small objects | 00:09 |
| notmyname | yeah, I thought about it but left it out. I think it makes it extra confusing, especially considering how badly it's named | 00:10 |
| clayg | notmyname: your probably right | 00:10 |
| notmyname | but yes, I specifically thought of our recent conversations on that very config as I was typing it ;-) | 00:10 |
| clayg | :) | 00:10 |
| notmyname | ok, I need to grab some dinner before my next flight | 00:11 |
| * notmyname out | 00:11 | |
| *** petertr7 is now known as petertr7_away | 00:13 | |
| peluse | notmyname, hey I know there's a youtube video out there (that you star in) but figured this was faster. About how many devs and how long from T0 to get Swift developed and functioning in its first production cluster? | 00:29 |
| peluse | oh man | 00:30 |
| peluse | just read the flight comment.... argh | 00:30 |
| peluse | found it... https://www.youtube.com/watch?v=Dd7wmJCDh4w | 00:31 |
| clayg | OMG he's so YOUNG! | 00:33 |
| ctennis | that's what I was gonna say! | 00:33 |
| clayg | baby face | 00:34 |
| clayg | ha! i look exactly the same -> https://www.youtube.com/watch?v=Dd7wmJCDh4w&feature=youtu.be&t=29 | 00:35 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Make ssync_sender a better HTTP client https://review.openstack.org/195458 | 00:36 |
| *** dmorita has joined #openstack-swift | 00:37 | |
| *** kota_ has joined #openstack-swift | 00:37 | |
| *** ChanServ sets mode: +v kota_ | 00:37 | |
| kota_ | morning | 00:38 |
| mattoliverau | kota_: morning | 00:39 |
| redbo | The first commit to swift was Jun 28, 2009, but it was just me and chmouel. gholt and chuck came in mid-august, notmyname in september, glange in november. We started putting customers on it mid-May 2010. | 00:43 |
| clayg | redbo: NICE! | 00:44 |
| clayg | redbo: hey wouldn't happen to know how eventlet monkey patching works do you? | 00:44 |
| redbo | not really. I looked at your gist and it was confusing. | 00:46 |
| clayg | redbo: yeah it's sorta bullshit | 00:50 |
| clayg | redbo: bet this kind of crap doesn't happen in go :'( | 00:50 |
| redbo | I know, you just learn to live with however it works, or copy the whole source code into your repo and change the part you need. | 00:51 |
| clayg | heh | 00:53 |
| *** wbhuber has joined #openstack-swift | 00:54 | |
| openstackgerrit | Merged openstack/swift: go: restructure cmd/hummingbird.go https://review.openstack.org/188939 | 01:01 |
| *** annegentle has joined #openstack-swift | 01:01 | |
| *** annegentle has quit IRC | 01:06 | |
| *** david-ly_ has joined #openstack-swift | 01:06 | |
| *** blmartin has joined #openstack-swift | 01:07 | |
| *** david-lyle has quit IRC | 01:09 | |
| *** wbhuber has quit IRC | 01:10 | |
| blmartin | Good evening all | 01:15 |
| blmartin | mattoliverau, I posted all those bugfix pull requests last night before I caught some sleep. Have you had a chance to peruse them? | 01:18 |
| mattoliverau | I can't remember.. I think so.. Let me double check :) | 01:18 |
| blmartin | thanks! Sorry I kind of threw them into so many review and posted them at the same time. I'll try to even it out in the future | 01:19 |
| mattoliverau | Maybe not completely, I'll look at em now. And no stress, that's for all the hard work :) | 01:21 |
| blmartin | I stated on some of the container backend tests today. It looks like build_full_shard_trie is not used anywhere and is mostly replaced by build_shard_trie | 01:23 |
| blmartin | do you have grand designs for that function or is now a good time to trim the code? | 01:25 |
| *** jasondotstar has joined #openstack-swift | 01:33 | |
| openstackgerrit | Alexandra Settle proposed openstack/swift: Correcting minor grammatical errors https://review.openstack.org/195825 | 01:35 |
| mattoliverau | asettle: ^ nice work :) | 01:36 |
| asettle | YES | 01:36 |
| asettle | WOO | 01:36 |
| asettle | WORKED | 01:36 |
| *** gar has joined #openstack-swift | 01:41 | |
| *** gar has joined #openstack-swift | 01:41 | |
| *** jrichli has joined #openstack-swift | 01:42 | |
| *** amoturi has quit IRC | 01:45 | |
| *** jasondotstar has quit IRC | 01:59 | |
| kota_ | gerrit gate seems to get broken? | 02:00 |
| kota_ | recent pushed patches failed at check-swift-dsvm-functional. | 02:02 |
| *** doxavore has joined #openstack-swift | 02:02 | |
| *** aluria has quit IRC | 02:03 | |
| *** dosaboy has quit IRC | 02:04 | |
| *** wolsen has quit IRC | 02:05 | |
| *** wolsen has joined #openstack-swift | 02:05 | |
| *** dosaboy has joined #openstack-swift | 02:06 | |
| *** aluria has joined #openstack-swift | 02:06 | |
| *** jasondotstar has joined #openstack-swift | 02:10 | |
| asettle | It's not even working for me | 02:15 |
| kota_ | oh, that's for docs. | 02:18 |
| mattoliverau | kota_: yeah, there has been a httplib2 CVE that limits the number of HTTP headers which is breaking things :( | 02:19 |
| mattoliverau | kota_: clay was looking into monkey patching the _MAX_HEADERS in the new httplib to solve the prob | 02:20 |
| mattoliverau | see scroll back if your interested | 02:20 |
| kota_ | mattoliverau: makes me sense, thx, I'm looking at back log. | 02:21 |
| asettle | Ah jeez there's a lot more jenkins checks than I'm used to. | 02:23 |
| openstackgerrit | Matthew Oliver proposed openstack/swift-specs: Add Spec Lifecycle Rules to readme https://review.openstack.org/190066 | 02:24 |
| *** annegentle has joined #openstack-swift | 02:24 | |
| kota_ | mattoliverau: nice | 02:25 |
| mattoliverau | kota_: ^^ Does that make more sense from your point of view? | 02:25 |
| kota_ | mattoliverau: I'm looking at *now* | 02:25 |
| kota_ | mattoliverau: perfect, thanks a lot! | 02:26 |
| mattoliverau | I still use the term patch in rule 4, but I think in that case it makes sense. | 02:26 |
| mattoliverau | kota_: no thank you :) | 02:26 |
| kota_ | +2 added | 02:28 |
| *** d0ugal has quit IRC | 02:29 | |
| *** d0ugal has joined #openstack-swift | 02:29 | |
| *** d0ugal is now known as Guest5335 | 02:30 | |
| mattoliverau | kota_: thanks! | 02:30 |
| *** wbhuber has joined #openstack-swift | 02:31 | |
| *** doxavore has quit IRC | 02:33 | |
| *** annegentle has quit IRC | 02:39 | |
| *** jasondotstar has quit IRC | 02:42 | |
| openstackgerrit | Merged openstack/swift-specs: Add Spec Lifecycle Rules to readme https://review.openstack.org/190066 | 02:42 |
| kota_ | nice thiago, that absolutely land fast. | 02:44 |
| *** lpabon has quit IRC | 02:47 | |
| *** asettle is now known as asettle-afk | 02:49 | |
| *** esker has joined #openstack-swift | 02:54 | |
| *** esker has quit IRC | 02:54 | |
| *** esker has joined #openstack-swift | 02:55 | |
| *** wbhuber has quit IRC | 02:56 | |
| mattoliverau | thanks tdasilva and kota_ :) | 02:57 |
| *** gyee has quit IRC | 02:58 | |
| *** annegentle has joined #openstack-swift | 03:40 | |
| *** asettle-afk is now known as asettle | 03:42 | |
| *** annegentle has quit IRC | 03:45 | |
| *** xnox has quit IRC | 03:47 | |
| *** sudorandom has quit IRC | 03:47 | |
| blmartin | q | 03:47 |
| blmartin | whoops, this isn't pdb | 03:47 |
| *** CrackerJackMack has quit IRC | 03:48 | |
| *** david-ly_ is now known as david-lyle | 03:51 | |
| *** asettle has quit IRC | 03:52 | |
| *** sudorandom has joined #openstack-swift | 03:53 | |
| *** xnox has joined #openstack-swift | 03:53 | |
| *** CrackerJackMack has joined #openstack-swift | 03:54 | |
| charz | mattoliverau | 04:36 |
| mattoliverau | charz: yo | 04:37 |
| *** jrichli has quit IRC | 04:37 | |
| blmartin | oh man mattoliverau, you got to the change set before I could even post a comment. I applaud your speed | 04:38 |
| charz | mattoliverau: did you know where is the monkey patching (_MAX_HEADERS) for httplib? | 04:38 |
| charz | mattoliverau: is this https://review.openstack.org/#/c/195458/ | 04:40 |
| mattoliverau | blmartin: what can I say, I'm obviously spying on you :P | 04:40 |
| blmartin | D: | 04:41 |
| *** annegentle has joined #openstack-swift | 04:41 | |
| mattoliverau | charz: I'm not sure clayg has finished looking at it. I think he was having eventlet patching pain. knowing clayg he's probably silent in channel, not because he's sleeping but cause he's beating his head against the code. | 04:43 |
| mattoliverau | so no, that one isn't it. | 04:43 |
| mattoliverau | ahh clayg sent an email to the powers that be: https://lists.secondlife.com/pipermail/eventletdev/2015-June/001213.html | 04:44 |
| charz | mattoliverau: got it, thanks. | 04:44 |
| mattoliverau | so that's probably where we are stuck atm | 04:45 |
| blmartin | Good luck and good night guys. I'm going to go catch some Zs. | 04:45 |
| charz | blmartin: nite | 04:46 |
| *** blmartin has quit IRC | 04:46 | |
| *** annegentle has quit IRC | 04:46 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:15 | |
| *** silor has joined #openstack-swift | 05:28 | |
| *** ppai has joined #openstack-swift | 05:38 | |
| hugokuo | a question regarding to the fi of a partiton-disk. Is it by design to have same fi of any object's fa on the same partition-disk ? | 05:41 |
| *** annegentle has joined #openstack-swift | 05:42 | |
| kota_ | hi, hugo, maybe, no? | 05:42 |
| kota_ | all fi(s) depend on node index of the ring. | 05:43 |
| kota_ | if the fa's partition is different each other, different disks should be assigned, AFAIK. | 05:44 |
| hugokuo | kota_: k. same as my understand. | 05:45 |
| hugokuo | thx | 05:45 |
| hugokuo | That means which the particular fa of a fi must be on a disk which is calculated during build/rebalance ring. It's assigned at very beginning while the ring is created | 05:46 |
| *** annegentle has quit IRC | 05:47 | |
| hugokuo | so that the Example in this http://docs.openstack.org/developer/swift/overview_replication.html#hashes-pkl is impossible. a43 - fi 2 , b23 - fi 1 . It's not gonna happen. | 05:48 |
| kota_ | it seems failure case, right? | 05:51 |
| kota_ | something like handoff | 05:51 |
| charz | kota_: make sense | 05:51 |
| kota_ | but I'm still considering the case would be really occur. | 05:52 |
| *** links has joined #openstack-swift | 05:55 | |
| hugokuo | kota_: unless inconsistency rings on different node | 05:55 |
| kota_ | hugokuo: hmm... I guess handoff node can have this situation easily because... | 05:56 |
| charz | kota_: wait, I'm wrong. like hugokuo said, a43 - fi 2. that's impossible have another fi handoff will point to a43. | 05:57 |
| kota_ | at PUT, node2 for a43 failed to connect, it will make a43 at handoff | 05:57 |
| kota_ | and then | 05:57 |
| kota_ | next PUT for b23, if node1 failed, it will make b23 fa at handoff | 05:58 |
| kota_ | charz: exactly the node already had a43 doesn't allow to land another index fa for a43 unless the node is primary | 05:59 |
| kota_ | charz: but this case is a43 and b23 objects are completely different, right? | 06:00 |
| kota_ | charz: that's based on the path /account/container/object | 06:00 |
| kota_ | s/that's/the hash is/ | 06:01 |
| hugokuo | they're different object but place in same partition ID. And fi follows the device ID. | 06:03 |
| kota_ | hugokuo: right | 06:04 |
| kota_ | basically they are almost same fi in the primaries. | 06:05 |
| kota_ | unless the objects are not placed handoffs | 06:05 |
| hugokuo | hmm... if two primary devices are down, would it place two fi(different object but same partition) in same handoff node's partition | 06:06 |
| hugokuo | I think not tho | 06:06 |
| hugokuo | let me do some interesting test now | 06:06 |
| kota_ | k, let me know someting found :) | 06:09 |
| kota_ | it's interesting question | 06:09 |
| hugokuo | kota_: confirmed. you are correct. The handoff device has the chance to include different fi of fa for different object. It happens while different primary device unavailable while the request be PUT in. | 06:20 |
| hugokuo | interesting | 06:20 |
| hugokuo | That means ls | 06:21 |
| kota_ | sounds good, thanks for your confirmation :) | 06:22 |
| *** SkyRocknRoll has quit IRC | 06:28 | |
| *** Guest5335 is now known as d0ugal | 06:34 | |
| *** d0ugal has quit IRC | 06:34 | |
| *** d0ugal has joined #openstack-swift | 06:34 | |
| *** annegentle has joined #openstack-swift | 06:43 | |
| *** annegentle has quit IRC | 06:48 | |
| charz | kota_: Which process will take care handoff of ec while one of primary devices is broken or unmount? reconstructor? | 06:57 |
| kota_ | no, proxy only for now. | 06:58 |
| kota_ | that's difference from replicated, right now. | 06:59 |
| kota_ | proxy-server will make a handoff when primary device broken. | 07:00 |
| charz | kota_: ok, but that's only for new coming objects. right? | 07:00 |
| kota_ | chars: yes | 07:00 |
| kota_ | s/chars/charz/ sorry | 07:00 |
| charz | kota_: for existed objects that's no way to reconstruct fa to another handoff | 07:01 |
| kota_ | charz: yes, at least kilo-stable. | 07:01 |
| charz | kota_: re-call it, that's already have a bug in lp. | 07:02 |
| kota_ | maybe, clayg, acoles and peluse are working for that. | 07:02 |
| kota_ | charz: i think so... | 07:03 |
| *** proteusguy has quit IRC | 07:03 | |
| charz | kota_: thank. | 07:03 |
| *** proteusguy_ has joined #openstack-swift | 07:03 | |
| *** acoles_away is now known as acoles | 07:14 | |
| acoles | good morning | 07:15 |
| kota_ | acoles: good morning :) | 07:16 |
| acoles | hey kota_ hows it going? | 07:16 |
| cschwede | Good morning! | 07:16 |
| kota_ | works is good but I still have cold a bit. | 07:17 |
| kota_ | cschwede: morning | 07:17 |
| acoles | kota_: sorry to hear that, not nice | 07:17 |
| acoles | cschwede: hi! | 07:17 |
| kota_ | acoles: no worries | 07:17 |
| *** bkopilov has quit IRC | 07:23 | |
| *** ppai has quit IRC | 07:29 | |
| *** rledisez has joined #openstack-swift | 07:38 | |
| *** proteusguy_ has quit IRC | 07:46 | |
| *** proteusguy has joined #openstack-swift | 07:46 | |
| *** therve has joined #openstack-swift | 07:49 | |
| therve | Hey | 07:49 |
| therve | clayg, Around? Looking at that httplib issue | 07:49 |
| *** ppai has joined #openstack-swift | 08:12 | |
| *** kun_huang has joined #openstack-swift | 08:15 | |
| mattoliverau | clayg: http://paste.openstack.org/show/321297/ | 08:16 |
| kun_huang | check-swift-dsvm-functional failed (http://logs.openstack.org/66/195866/2/check/check-swift-dsvm-functional/bcaec5b/console.html) | 08:17 |
| kun_huang | it seems not a random failure | 08:17 |
| kun_huang | is anyone working on this? | 08:17 |
| mattoliverau | therve: ^^ here is a hacky way to get around the _MAXHEADERS eventlet issue (seeing as your looking at it). Its a mjor hack, but was just trying to figure out whats happening | 08:18 |
| mattoliverau | kun_huang: yeah we all know, there has been a patch to httplib which is breaking things | 08:18 |
| mattoliverau | kun_huang: we tried monkey_patching _MAXHEADERS but that doesn't work when using eventlet.. well in my testing. Hopefully we can get it fixed ASAP. Thanks for taking the time to report it :) | 08:20 |
| mattoliverau | kun_huang: unless your an eventlet expert then your welcome to take a shot :) | 08:21 |
| mattoliverau | K, I'm calling it a day, I'm suppose to take the wife out tonight, and I'll get in trouble if I dont stop working soon :P | 08:22 |
| cschwede | mattoliverau: have a nice evening & weekend! | 08:23 |
| kun_huang | mattoliverau: have a nice night | 08:23 |
| therve | mattoliverau, You could use eventlet.patcher.import_patched to make it a bit cleaner | 08:26 |
| mattoliverau | therve: cool, I know nothing about eventlet.. just playing with whats broken | 08:28 |
| haypo | therve: hi. have fun with monkey patching: https://lists.secondlife.com/pipermail/eventletdev/2015-June/001213.html "[Eventletdev] how to monkey patch the monkey patchers" :-) | 08:29 |
| therve | Yeah... | 08:30 |
| cschwede | therve: mattoliverau: nice, this looks like it works. based on the sample from clayg: http://paste.openstack.org/show/321301/ | 08:31 |
| *** ppai has quit IRC | 08:31 | |
| therve | cschwede, +1 | 08:31 |
| *** nexusz99 has joined #openstack-swift | 08:41 | |
| *** nexusz99 has quit IRC | 08:42 | |
| *** proteusguy has quit IRC | 08:42 | |
| *** annegentle has joined #openstack-swift | 08:44 | |
| mattoliverau | cschwede: awesome, nice one! | 08:47 |
| *** annegentle has quit IRC | 08:49 | |
| mattoliverau | works on my HTTPResponse(socket) tests too. Nice so looks like we have a work around. | 08:51 |
| mattoliverau | clayg: FYI - ^^ Read this section of scroll back | 08:52 |
| mattoliverau | K, I really need to go, wife wont wait much longer, it is friday night :) Have a great weekend all. | 08:53 |
| *** joeljwright has joined #openstack-swift | 08:54 | |
| *** ChanServ sets mode: +v joeljwright | 08:54 | |
| kota_ | mattoliverau: great work, have a nice week end! | 08:54 |
| cschwede | i’m going to submit a patch for the headers thingy | 09:00 |
| *** BAKfr has quit IRC | 09:01 | |
| *** BAKfr has joined #openstack-swift | 09:03 | |
| *** breitz has quit IRC | 09:05 | |
| *** breitz has joined #openstack-swift | 09:06 | |
| mattoliverau | cschwede: awesome do it man! #onPhone #Don'tTellWife :P | 09:08 |
| *** husanu8 has joined #openstack-swift | 09:09 | |
| *** jordanP has joined #openstack-swift | 09:10 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 1024 https://review.openstack.org/195940 | 09:15 |
| *** husanu8 has quit IRC | 09:18 | |
| *** kota_ has quit IRC | 09:19 | |
| *** dmorita has quit IRC | 09:20 | |
| *** husanu1 has joined #openstack-swift | 09:23 | |
| *** nexusz99 has joined #openstack-swift | 09:24 | |
| *** husanu1 has quit IRC | 09:28 | |
| *** nexusz99 has quit IRC | 09:29 | |
| *** nexusz99 has joined #openstack-swift | 09:33 | |
| kun_huang | cschwede: failed again | 09:34 |
| *** BAKfr has quit IRC | 09:34 | |
| kun_huang | https://jenkins01.openstack.org/job/check-swift-dsvm-functional/495/console | 09:34 |
| cschwede | kun_huang: yes, but this time in the tests, the serverprocess are already ok. working on it | 09:34 |
| cschwede | oh, that looks different than my local env. checking | 09:35 |
| *** BAKfr has joined #openstack-swift | 09:39 | |
| nexusz99 | Q: there are only 2 options in policy_type in Storage Policies Configuration? (replication, erasure_code) then why we need policy name for duplicate policy_type? i think we only need 2 policy and share them other containers. | 09:40 |
| *** annegentle has joined #openstack-swift | 09:45 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 1024 https://review.openstack.org/195940 | 09:47 |
| *** annegentle has quit IRC | 09:50 | |
| cschwede | nexusz99: there might be more policy types in the future, that are neither replication nor erasure_code | 09:51 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 1024 https://review.openstack.org/195940 | 09:55 |
| mattoliverau | nexusz99: you need policy_names because you still might use more then 1 of each type of policy, so yes duplicates. You may have a policy what will only place things in one region, you may have another that keeps more replicas then another, you may have a bunch of faster disks or ssds and all of which you can have a different policy for each of them. Or as cschwede mentioned, in the future there will most definitely be | 09:56 |
| mattoliverau | more policy_types. Storage policies open up many more use cases in the same clluster. | 09:56 |
| nexusz99 | cschwede: mattoliverau: thanks for reply. now i understand why we need policy_names. thank you! | 09:58 |
| clayg | sys.modules['__patched_module_httplib'] is genious - kudos mattoliverau | 10:01 |
| clayg | and cschwede & therve for the suggested cleanups | 10:01 |
| cschwede | clayg: i have a wip patch up, https://review.openstack.org/#/c/195940/ - looks like the latest patchset fixes the issue | 10:05 |
| mattoliverau | clayg: ta, just following where you left off, got lost in eventlet code and debugged the hell out of it. And therve and cschwede for their fixing up my brute force results.. so team effort :) | 10:05 |
| cschwede | clayg: hope the patch is ok for you, wanted to make some progress to unlock the gate | 10:05 |
| clayg | cschwede: great work - i've got it checked out now | 10:06 |
| cschwede | ok, the gate-swift-tox-func already passed this time :) https://jenkins05.openstack.org/job/gate-swift-tox-func/882/console | 10:09 |
| cschwede | hope the rest succeeds also | 10:09 |
| clayg | cschwede: omm i had to patch functests/swift_test_client too | 10:12 |
| clayg | :\ | 10:12 |
| clayg | let me try again w/o | 10:12 |
| cschwede | what the… why did the commit message update used patchset #1, and not #2 as base? argh! | 10:14 |
| cschwede | clayg: that was included in patchset #2 | 10:14 |
| cschwede | clayg: let me fix that, once the test for patchset #2 finished on the gate | 10:14 |
| clayg | yeah i love that no one even discusses dropping the test to only validate 99 user-metadata keys or somethign? | 10:19 |
| clayg | 100 just isn't a lot | 10:19 |
| clayg | maybe we're all jaded - if you don't have to squint and point to count the zeros it's not a big number | 10:19 |
| cschwede | :/ https://jenkins04.openstack.org/job/check-swift-dsvm-functional/460/console | 10:23 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 1024 https://review.openstack.org/195940 | 10:27 |
| cschwede | ah, that’s the patchset without the patched testclient. next try | 10:27 |
| cschwede | i’m still wondering why the commit msg update used patchset #1 as base, and not #2. got lost somehow... | 10:27 |
| *** links has quit IRC | 10:36 | |
| openstackgerrit | Charles Hsu proposed openstack/swift: Fix reconstructor stats mssage. https://review.openstack.org/195275 | 10:46 |
| *** nexusz99 has quit IRC | 10:48 | |
| *** nexusz99 has joined #openstack-swift | 10:49 | |
| *** geaaru has joined #openstack-swift | 10:54 | |
| *** mariusv has joined #openstack-swift | 11:00 | |
| *** mariusv has quit IRC | 11:00 | |
| *** mariusv has joined #openstack-swift | 11:00 | |
| *** mariusv has quit IRC | 11:01 | |
| *** links has joined #openstack-swift | 11:17 | |
| *** links has quit IRC | 11:19 | |
| *** nexusz99 has quit IRC | 11:19 | |
| *** delattec has quit IRC | 11:31 | |
| *** cdelatte has quit IRC | 11:31 | |
| *** ppai has joined #openstack-swift | 11:34 | |
| *** kei_yama has quit IRC | 11:36 | |
| *** aix has quit IRC | 11:41 | |
| *** annegentle has joined #openstack-swift | 11:47 | |
| *** annegentle has quit IRC | 11:52 | |
| *** aix has joined #openstack-swift | 12:08 | |
| *** proteusguy has joined #openstack-swift | 12:22 | |
| *** proteusguy_ has joined #openstack-swift | 12:24 | |
| *** proteusguy has quit IRC | 12:24 | |
| *** proteusguy_ has quit IRC | 12:24 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 128 https://review.openstack.org/195940 | 12:26 |
| *** doxavore has joined #openstack-swift | 12:28 | |
| *** aix has quit IRC | 12:28 | |
| *** ppai has quit IRC | 12:31 | |
| *** doxavore has quit IRC | 12:35 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 128 https://review.openstack.org/195940 | 12:36 |
| *** NM has joined #openstack-swift | 12:40 | |
| tdasilva | good morning! | 12:44 |
| *** tellesnobrega has joined #openstack-swift | 12:45 | |
| *** thurloat_isgone is now known as thurloat | 12:59 | |
| *** tellesnobrega has quit IRC | 13:04 | |
| *** tellesnobrega has joined #openstack-swift | 13:08 | |
| *** jrichli has joined #openstack-swift | 13:10 | |
| *** cdelatte has joined #openstack-swift | 13:25 | |
| *** delattec has joined #openstack-swift | 13:25 | |
| cschwede | All: patch 195940 fixes the issue on the gate. Some reviews needed though :) | 13:30 |
| patchbot | cschwede: https://review.openstack.org/#/c/195940/ | 13:30 |
| cschwede | clayg: notmyname: tdasilva: acoles: ^^^ | 13:30 |
| *** acampbell has joined #openstack-swift | 13:34 | |
| *** cazino has joined #openstack-swift | 13:38 | |
| acoles | cschwede: thanks, see my review, did i misunderstand? | 13:40 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS to 128 https://review.openstack.org/195940 | 13:43 |
| *** petertr7_away is now known as petertr7 | 13:46 | |
| tdasilva | oh, i see | 13:47 |
| tdasilva | oops, wrong window | 13:47 |
| *** annegentle has joined #openstack-swift | 13:48 | |
| cschwede | acoles: holy smoke! well, good to have you as a reviewer, thx! it’s fixed now | 13:49 |
| tdasilva | cschwede: are there any tests for checking the max number of headers? | 13:52 |
| *** wbhuber has joined #openstack-swift | 13:52 | |
| *** annegentle has quit IRC | 13:53 | |
| cschwede | tdasilva: well, no, at least not directly. but if you set the constraint to 100 you will notice that a lot of functests fail, even if you’re not using py2.7.9 or later | 13:53 |
| cschwede | i wanted to focus on unblocking the gate first, so my first approach were only two lines; the patch has grown a little bit ;) | 13:54 |
| tdasilva | ok | 13:54 |
| *** annegentle has joined #openstack-swift | 13:55 | |
| acoles | cschwede: i haven't been following along - what changed to cause gate to fail? | 13:59 |
| cschwede | acoles: python 2.6.9 and 3.2.6. there is a new limit of 100 headers in httplib | 13:59 |
| *** proteusguy has joined #openstack-swift | 14:01 | |
| acoles | cschwede: oic. i have a vague memory of seeing this 100 limit hurt us before, but can't remember what the situation was | 14:01 |
| acoles | cschwede: maybe it was in swiftclient. idk | 14:02 |
| cschwede | acoles: i think so, it’s the same case for swiftclient. you can’t use it with more than ~ 90 metadata values; but since that’s the default of swift there won’t be many users that hit this limit | 14:03 |
| acoles | yup | 14:04 |
| acoles | cschwede: did you consider making MAX_HEADER_COUNT = MAX_META_COUNT + N (e.g. N=38) rather than another configurable? | 14:07 |
| acoles | seems the two config options are coupled | 14:07 |
| *** annegentle has quit IRC | 14:07 | |
| cschwede | hmm, thought of it. do you mean to remove the operator setting completely? so MAX_HEADER_COUNT = MAX_META_COUNT + N is always set? or just a default if it is not set, and using a custom value if set? | 14:09 |
| acoles | well, i was just concerned that human-error could set MAX_HEADER_COUNT=128 and MAX_META_COUNT=150 | 14:11 |
| *** annegentle has joined #openstack-swift | 14:11 | |
| *** annegentle has quit IRC | 14:11 | |
| tdasilva | so maybe only make MAX_META_COUNT configurable and then calculate MAX_HEADER_COUNT internally | 14:12 |
| tdasilva | with the constant 'N=38'? | 14:12 |
| acoles | and also that as swift adds new headers we may need the mAX_META_COUNT to increase but usersmay have configured the 'default' 128 | 14:12 |
| acoles | tdasilva: thats what i was wondering yes | 14:12 |
| acoles | then we can increase N if needed in future without worrying that someone has actually configured a fixed value | 14:13 |
| *** annegentle has joined #openstack-swift | 14:13 | |
| *** annegentle has quit IRC | 14:13 | |
| *** amoturi has joined #openstack-swift | 14:14 | |
| * acoles wonders what the worst case header count is for swift API | 14:14 | |
| peluse | MooingLemur, FYI I'll be on shortly after lunch... will look for you then | 14:14 |
| tdasilva | acoles, cschwede: seems like a good idea | 14:14 |
| cschwede | good point. i’m also wondering about N | 14:14 |
| cschwede | N=32? | 14:14 |
| cschwede | i somehow like 2^x numbers | 14:14 |
| tdasilva | :) | 14:14 |
| acoles | cschwede: i wonder why ;) | 14:15 |
| cschwede | so, in my tests the biggest amount of headers i saw was 104; that was 90 from meta_count plus some others. so if we select 32 as default we should be on the safe side, no? | 14:15 |
| acoles | cschwede: seems reasonable. | 14:16 |
| acoles | cschwede: tdasilva : hmmm, what about third party middleware headers? | 14:16 |
| cschwede | forbid 3rd party middlewares 8) | 14:17 |
| acoles | yeah! | 14:17 |
| *** joeljwright has quit IRC | 14:17 | |
| cschwede | so, a default of 32 plus the option to override this? | 14:17 |
| tdasilva | cschwede: well..the way for an op to override would be with MAX_META_COUNT, right? | 14:18 |
| tdasilva | cschwede: or do you mean for 3rd party middleware? | 14:18 |
| acoles | so ... max_header_count = MAX_META_COUNT + 32 + (configurable extra_header_count) | 14:18 |
| acoles | that way you can't configure too low | 14:19 |
| acoles | but you can increase | 14:19 |
| acoles | assuming we check that extra_header_count is > 0 | 14:19 |
| cschwede | that sounds great! | 14:19 |
| tdasilva | but then we are back at having two configurable options :/ | 14:20 |
| cschwede | tdasilva: well, no? only one new, that is 0 by default, but can be increased only? | 14:21 |
| acoles | tdasilva: yes. but at least one option cannot conflict with the other. | 14:21 |
| acoles | tdasilva: cschwede we could even not document it :P | 14:22 |
| cschwede | that will get back to us sometime | 14:22 |
| acoles | i was only kidding ;) | 14:23 |
| tdasilva | lol | 14:27 |
| *** lcurtis has joined #openstack-swift | 14:29 | |
| *** blmartin has joined #openstack-swift | 14:40 | |
| *** lpabon has joined #openstack-swift | 14:41 | |
| blmartin | Good Morning all | 14:41 |
| *** jlhinson has joined #openstack-swift | 14:42 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Increase httplib._MAXHEADERS https://review.openstack.org/195940 | 14:43 |
| cschwede | acoles: tdasilva: like this? I can add a test as a followup, but want to unbreak the gate first | 14:43 |
| anteaya | does anyone know the status of the swift-dsvm-functional job? | 14:44 |
| anteaya | clayg notmyname mattoliverau ^^ | 14:44 |
| anteaya | its status is about to change as per -qa | 14:44 |
| *** jordanP has quit IRC | 14:45 | |
| cschwede | anteaya: we’re working on it, see patch 195940 | 14:46 |
| patchbot | cschwede: https://review.openstack.org/#/c/195940/ | 14:46 |
| cschwede | the last patchset is hopefully the one that is going to be merged | 14:46 |
| anteaya | can someone join infra and share this information? | 14:47 |
| cschwede | i’ll do that | 14:47 |
| anteaya | it is pertinent to the current conversation | 14:47 |
| anteaya | thank you | 14:47 |
| acoles | cschwede: +2'd | 14:49 |
| tdasilva | cschwede: +2+A | 14:49 |
| acoles | cschwede: nice work | 14:49 |
| cschwede | acoles: tdasilva: thx a lot! | 14:49 |
| tdasilva | cschwede: thank you! | 14:50 |
| cschwede | anteaya: thanks for pinging us! | 15:04 |
| anteaya | cschwede: thanks for the team work on that patch and being so responsive | 15:05 |
| anteaya | :) | 15:05 |
| anteaya | let's see if we can get all this sorted out | 15:05 |
| cschwede | yes, hopefully that issue is solved real soon now. i’m keeping my fingers crossed for the running tests :) | 15:06 |
| anteaya | :) | 15:09 |
| * anteaya also crosses her fingers | 15:09 | |
| *** DericHorn-HP has joined #openstack-swift | 15:23 | |
| *** zul has quit IRC | 15:28 | |
| *** zul has joined #openstack-swift | 15:33 | |
| *** ajiang has left #openstack-swift | 15:34 | |
| *** tellesnobrega has quit IRC | 15:34 | |
| *** minwoob_ has joined #openstack-swift | 15:35 | |
| *** minwoob has quit IRC | 15:36 | |
| *** DericHorn-HP has quit IRC | 15:36 | |
| *** tellesnobrega has joined #openstack-swift | 15:38 | |
| *** ajiang has joined #openstack-swift | 15:43 | |
| *** blmartin_ has joined #openstack-swift | 15:45 | |
| *** Fin1te has joined #openstack-swift | 15:46 | |
| cschwede | what the? check-tempest-dsvm-full failed on the check job, but passed on the gate job? | 15:48 |
| *** blmartin has quit IRC | 15:49 | |
| acoles | :/ | 15:50 |
| cschwede | ok, it was a timeout | 15:50 |
| openstackgerrit | Minwoo Bae proposed openstack/swift: After the .durable has been written, fsync the directory. https://review.openstack.org/184113 | 15:50 |
| *** amoturi has quit IRC | 15:55 | |
| *** zaitcev has joined #openstack-swift | 15:57 | |
| *** ChanServ sets mode: +v zaitcev | 15:57 | |
| *** gar1 has joined #openstack-swift | 15:57 | |
| *** gar has quit IRC | 15:57 | |
| * cschwede afk for ~ 2 hours, keep babysitting patch 195940 by mail | 16:04 | |
| patchbot | cschwede: https://review.openstack.org/#/c/195940/ | 16:04 |
| *** Fin1te has quit IRC | 16:05 | |
| acoles | notmyname: clayg peluse tdasilva ^^ i gotta leave too can you keep an eye on this and recheck if necessary | 16:05 |
| *** janonymous_ has joined #openstack-swift | 16:06 | |
| *** acoles is now known as acoles_away | 16:10 | |
| *** DericHorn-HP has joined #openstack-swift | 16:11 | |
| *** vinsh has joined #openstack-swift | 16:24 | |
| *** haypo has left #openstack-swift | 16:25 | |
| *** rledisez has quit IRC | 16:26 | |
| *** bill_az has joined #openstack-swift | 16:26 | |
| *** tellesnobrega has quit IRC | 16:28 | |
| *** nadeem has joined #openstack-swift | 16:28 | |
| openstackgerrit | Merged openstack/swift: Increase httplib._MAXHEADERS https://review.openstack.org/195940 | 16:28 |
| *** nadeem has quit IRC | 16:28 | |
| *** nadeem has joined #openstack-swift | 16:29 | |
| *** gyee has joined #openstack-swift | 16:31 | |
| *** tellesnobrega has joined #openstack-swift | 16:35 | |
| *** petertr7 is now known as petertr7_away | 16:39 | |
| *** cazino has left #openstack-swift | 16:51 | |
| bill_az | We are experimenting with swift multi-site replication and have a few questions. Is there anyone who has experience using this with multiple storage policies? | 17:02 |
| *** tellesnobrega has quit IRC | 17:03 | |
| bill_az | notmyname: I watched your "going global" video (https://www.youtube.com/watch?v=LpmBRqevuVU), and still have questions | 17:05 |
| bill_az | notmyname: We are experimenting with 2 regions, and 2 storage policies per region, default policy is local replication only, and policy 1 is replication across both regions | 17:06 |
| bill_az | notemyname: we see objects placed correctly to both regions, but the gets fail on remote site | 17:07 |
| bill_az | notmyname: call to container_info on remote site does not find container (or does not know storage policy for request) and so uses the default ring | 17:08 |
| bill_az | notmyname: is there detailed documentation available about how to configure global replication with different replication settings in several storage policies? | 17:09 |
| cschwede | ok, patch 195940 merged, gate is unstuck. time for the weekend :) thx everyone who helped out with that! | 17:14 |
| patchbot | cschwede: https://review.openstack.org/#/c/195940/ | 17:14 |
| anteaya | could some swift type folks review this patch which removes a non-voting job from the swift gate, please? https://review.openstack.org/#/c/196134/1 | 17:21 |
| anteaya | non-voting jobs shouldn't be in the gate | 17:21 |
| *** nexusz99 has joined #openstack-swift | 17:23 | |
| *** EmilienM is now known as EmilienM|brb | 17:25 | |
| *** petertr7_away is now known as petertr7 | 17:28 | |
| *** annegentle has joined #openstack-swift | 17:31 | |
| *** dhellmann has joined #openstack-swift | 17:33 | |
| *** annegentle has quit IRC | 17:36 | |
| *** amoturi has joined #openstack-swift | 17:39 | |
| *** elmiko has left #openstack-swift | 17:40 | |
| dhellmann | I'm running into some failures with a swift functional test job on a change that shouldn't be related to swift at all. I can't seem to find actual tracebacks in the error log for the job, though. Is there anyone available familiar with the check-swift-dsvm-functional job who can give me a hand poking around? | 17:42 |
| dhellmann | patch is https://review.openstack.org/#/c/195724/ and failure logs are http://logs.openstack.org/24/195724/1/check/check-swift-dsvm-functional/c60644b/logs/devstack-gate-post_test_hook.txt.gz | 17:43 |
| MooingLemur | peluse: smae, I should be back between noon and 1 | 17:45 |
| MooingLemur | MST of course :P | 17:45 |
| *** theoharr has joined #openstack-swift | 17:46 | |
| *** theoharr has quit IRC | 17:46 | |
| cschwede | dhellmann: that should be fixed by patch 195940 - could you recheck your patch? | 17:46 |
| patchbot | cschwede: https://review.openstack.org/#/c/195940/ | 17:46 |
| dhellmann | cschwede: oh, cool, thanks! | 17:47 |
| dhellmann | cschwede: in the mean time, where would I have wanted to look for tracebacks from those test failures? | 17:47 |
| dhellmann | I saw the errors reported, but no tracebacks from the tests, in the console log | 17:48 |
| cschwede | dhellmann: in the screen-s-proxy-log; in your case in http://logs.openstack.org/24/195724/1/check/check-swift-dsvm-functional/c60644b/logs/screen-s-proxy.txt.gz | 17:48 |
| cschwede | dhellmann: „HTTPException: got more than 100 headers“ | 17:49 |
| dhellmann | cschwede: ah, I was expecting to find the test runner reporting something more detailed about how the test had errored out (not necessarily that error about the headers, but something) | 17:49 |
| cschwede | dhellmann: if it’s not a change in swift itself, but a swift job that fails it is most likely related to the proxy | 17:49 |
| dhellmann | I'll keep that in mind for next time, thanks | 17:50 |
| cschwede | so in this case it was simply bad luck - python has a new limit of 100 headers, and there were tests that sent 104 headers | 17:50 |
| cschwede | dhellmann: you’re welcome! | 17:50 |
| *** EmilienM|brb is now known as EmilienM | 18:04 | |
| *** theoharr has joined #openstack-swift | 18:04 | |
| clayg | cschwede: great work | 18:09 |
| cschwede | clayg: thx, but you deserve a big thx as well for figuring out what was broken! | 18:11 |
| swifterdarrell | hahaha, this reminds me of that one time when I had to go to the .h file and change #define SOME_ARBITRARY_LIMIT 1024 to #define SOME_ARBITRARY_LIMIT 1025. That was fun. I miss that. | 18:11 |
| *** DericHorn-HP has quit IRC | 18:12 | |
| *** DericHorn-HP has joined #openstack-swift | 18:13 | |
| *** theoharr has left #openstack-swift | 18:16 | |
| *** DericHorn-HP has quit IRC | 18:17 | |
| *** DericHorn-HP has joined #openstack-swift | 18:18 | |
| clayg | jrichli: my browser crashed - i'm sure i could dig it out of history - but if you have it handy - can you link me the unified crypto patch again? | 18:22 |
| jrichli | clayg: https://review.openstack.org/#/c/157907/ | 18:26 |
| *** nexusz99 has quit IRC | 18:27 | |
| *** minwoob_ has quit IRC | 18:27 | |
| clayg | merci | 18:28 |
| *** Fin1te has joined #openstack-swift | 18:29 | |
| *** minwoob has joined #openstack-swift | 18:29 | |
| jrichli | clayg: things get sorta complicated in order to make functests happy :-) | 18:30 |
| jrichli | like the "tail" concept we ended up going with because we couldn't be successful moving the metadata length checks | 18:31 |
| clayg | jrichli: i thought that all worked now? | 18:31 |
| *** annegentle has joined #openstack-swift | 18:32 | |
| jrichli | the solution you and I came up with fixed the functest we were working on, but broke another. i will have to dig up the info. its all recorded in the trello board | 18:32 |
| clayg | charz: lp bug #1468708 has me *very* confused | 18:35 |
| openstack | Launchpad bug 1468708 in OpenStack Object Storage (swift) "ssync receiver get an odd SSYNC command and raise a exception" [Undecided,New] https://launchpad.net/bugs/1468708 | 18:35 |
| *** annegentle has quit IRC | 18:37 | |
| *** geaaru has quit IRC | 18:39 | |
| openstackgerrit | Merged openstack/swift: Fixed Errors in Swift-Form Post middleware section rst. https://review.openstack.org/195154 | 18:40 |
| *** silor1 has joined #openstack-swift | 18:47 | |
| *** silor has quit IRC | 18:48 | |
| *** annegentle has joined #openstack-swift | 18:51 | |
| *** Fin1te has quit IRC | 18:52 | |
| *** Fin1te has joined #openstack-swift | 19:02 | |
| clayg | wbhuber: you make any progress on the proxy failure test? charz found another lp bug #1469094 that's very closely related | 19:03 |
| openstack | Launchpad bug 1469094 in OpenStack Object Storage (swift) "Missing .durable files of FAs can cause rebuildable objects to be inaccessible" [High,Confirmed] https://launchpad.net/bugs/1469094 | 19:03 |
| wbhuber | clayg: i was about to unlink the disks before uploading the object, but slowly a progress overall. | 19:04 |
| wbhuber | perhaps charz and i can work together on the bug | 19:04 |
| clayg | maybe! | 19:05 |
| clayg | minwoob: how are you doing today!? | 19:08 |
| *** janonymous_ has quit IRC | 19:14 | |
| minwoob | clayg: I submitted a patch set this morning for one that I'm trying to close up, and will be finishing up the duplicates test for the Reconstructor bug. | 19:18 |
| minwoob | clayg: As of now, I think I have things under control :) | 19:19 |
| clayg | awesome! let me know if you get stuck - or when you're ready for a review | 19:19 |
| minwoob | Okay. Will do. | 19:20 |
| *** annegentle has quit IRC | 19:20 | |
| *** blmartin_ has quit IRC | 19:21 | |
| *** Fin1te has quit IRC | 19:24 | |
| jrichli | clayg: I updated a trello card to explain why we went with the "tail" idea for metadata limits. https://trello.com/c/0LUlx7Wr/26-fix-what-breaks-when-encrypter-sets-user-metadata-values-that-exceed-max-size | 19:25 |
| *** linuxgeek_ has joined #openstack-swift | 19:26 | |
| clayg | torgomatic: peluse: tdasilva: zaitcev: high priority fix for ec that's fairly trivial (IMHO) https://review.openstack.org/#/c/195457/ | 19:31 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix ValueError in ssync_receiver https://review.openstack.org/195457 | 19:34 |
| clayg | updated commit to reference bug # | 19:34 |
| zaitcev | reloaded | 19:35 |
| clayg | there's some interplay of multiple issues - charz is going to have to keep digging - but that patch and it's dependency should allow him to get back on master | 19:35 |
| zaitcev | >>> int('') | 19:36 |
| zaitcev | ValueError: invalid literal for int() with base 10: '' | 19:36 |
| zaitcev | urk | 19:36 |
| clayg | sigh :'( | 19:36 |
| zaitcev | okay, I'm not getting it. It's inside the if get():. Would both None and '' fall inside the if? I think so... | 19:37 |
| clayg | the valueerror in the bug log was just the disconnect - the *reason* for the disconnect was the ValueError invalid literal for int() with base 10: 'None' (which was on the other side of the connection) | 19:37 |
| clayg | the *reason* for the None - was the unmount | 19:37 |
| zaitcev | oh, it's a string that containes the word 'None | 19:37 |
| clayg | 'None' does fall into the if - '' does not | 19:37 |
| zaitcev | holy mother how did that happen | 19:38 |
| clayg | how did what happen - the 'None'? | 19:38 |
| clayg | putheader | 19:38 |
| zaitcev | oh | 19:38 |
| clayg | and the missing test which the patch adds ;) | 19:38 |
| redbo | I really wish wsgi defined what happens when a client disconnects early. | 19:39 |
| zaitcev | clayg: what about the line above? If there could be a connection without node index, could there be one without frag index? And what happens with that 'None'? | 19:40 |
| clayg | there should *not* be one without a frag index | 19:41 |
| clayg | but we shouldn't be putting 'None' on the wire | 19:41 |
| clayg | zaitcev: why aren't you just reviewing this in gerrit ;) | 19:41 |
| * clayg tires to find the line zaitcev is looking at | 19:41 | |
| zaitcev | well, whatever. it looks good | 19:42 |
| clayg | zaitcev: no I think a '' default would be way better there - nice catch | 19:42 |
| *** jrichli has quit IRC | 19:44 | |
| clayg | zaitcev: acctually - that would have blow up ssync on a replication policy (like you'd get if a primary device was 507'd) | 19:45 |
| clayg | zaitcev: good fucking catch! | 19:45 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Make ssync_sender a better HTTP client https://review.openstack.org/195458 | 19:47 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix ValueError in ssync_receiver https://review.openstack.org/195457 | 19:47 |
| clayg | torgomatic: zaitcev: thanks - and sorry for the double work | 19:47 |
| clayg | i'll ping you guys again when jenkins finishes with it - i want to rename the ssync "rsync" test to ssync "replicated" (doh!)_ | 19:50 |
| clayg | might as well run probetests while I wait | 19:51 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Make ssync_sender a better HTTP client https://review.openstack.org/195458 | 19:51 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix ValueError in ssync_receiver https://review.openstack.org/195457 | 19:51 |
| clayg | what is the deal with "colorama" | 19:55 |
| zaitcev | SCOTUS made gay marriage legal in all 50 today | 19:56 |
| zaitcev | (if that's what you mean) | 19:56 |
| clayg | lol - no - that's not what I ment | 19:56 |
| clayg | when i don't have colorama installed i see this trying to run nose -> /usr/local/lib/python2.7/dist-packages/nose/plugins/manager.py:395: RuntimeWarning: Unable to load plugin openstack.nose_plugin = openstack.nose_plugin:Openstack: colorama | 19:57 |
| *** nexusz99 has joined #openstack-swift | 19:57 | |
| clayg | but like - if I don't have it installed - why is it trying to load it? | 19:58 |
| *** nexusz99 has quit IRC | 20:02 | |
| clayg | who knew -> | 20:02 |
| clayg | vagrant@saio:~$ sudo | 20:02 |
| clayg | err... sudo pip install --upgrade openstack.nose-plugin | 20:02 |
| *** thurloat is now known as thurloat_isgone | 20:05 | |
| openstackgerrit | Minwoo Bae proposed openstack/swift: EC Reconstructor: Do not reconstruct existing fragments. https://review.openstack.org/193279 | 20:11 |
| minwoob | clayg: I went ahead and posted what I have for bug 1452553. | 20:13 |
| openstack | bug 1452553 in OpenStack Object Storage (swift) "don't rebuild existing fragments" [Critical,In progress] https://launchpad.net/bugs/1452553 - Assigned to Minwoo Bae (minwoob) | 20:13 |
| clayg | minwoob: great! what's not finished - can I help! | 20:14 |
| minwoob | clayg: It is similar to the test for the case where reconstruct_fa finds itself and does not fail. | 20:14 |
| minwoob | clayg: The one for duplicates is failing, though. | 20:14 |
| clayg | minwoob: cool - i can look at it if that'd be helpful | 20:14 |
| minwoob | clayg: I'm still looking at it, but yeah, if you'd like to take a look at it as well, that would be great. | 20:15 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Make ssync_sender a better HTTP client https://review.openstack.org/195458 | 20:17 |
| clayg | minwoob: i see test_reconstruct_fa_finds_duplicate_does_not_fail failing? | 20:19 |
| minwoob | Yes -- that's the one that I just added. | 20:19 |
| minwoob | And everything else seems to be fine. | 20:20 |
| clayg | idk - so rebuild should only need ec_ndata fragments | 20:21 |
| clayg | minwoob: try this -> https://gist.github.com/clayg/eb77a5e69812b6ea25eb | 20:23 |
| clayg | well - that might not be what you were going for | 20:24 |
| clayg | that just changes the order | 20:24 |
| clayg | the broken_body is what's being rebuilt right? | 20:24 |
| minwoob | In this test case, I don't think it matters which fragment is being rebuilt -- it's just to make sure that the reconstructor doesn't fail when there's 2 (or more) of the same fragment found. | 20:25 |
| minwoob | It should not matter if that's even the fragment that needs to be rebuilt, right? | 20:25 |
| minwoob | At least, that's what the code in the reconstructor itself checks for. | 20:26 |
| *** amoturi has quit IRC | 20:27 | |
| minwoob | To be precise, actually, the reconstructor first checks whether the fragment to be rebuilt already exists, and then it checks for duplicates in the responses from the other nodes. | 20:27 |
| clayg | hrmm... well if the frag_index == fi_to_rebuild it'll skip it for a different reason | 20:27 |
| minwoob | Yeah. | 20:28 |
| peluse | MooingLemur, you there? | 20:33 |
| minwoob | clayg: In regards to the gist you posted above, it does work. | 20:35 |
| clayg | minwoob: weeeeeelllll it *passes* - but does it test the code we're trying to test? | 20:37 |
| *** annegentle has joined #openstack-swift | 20:37 | |
| openstackgerrit | Minwoo Bae proposed openstack/swift: EC Reconstructor: Do not reconstruct existing fragments. https://review.openstack.org/193279 | 20:38 |
| minwoob | clayg: It brings us closer, so I went ahead and updated it. | 20:39 |
| MooingLemur | peluse: I am now.. it's been busier than expected :) | 20:39 |
| peluse | MooingLemur, BRB | 20:39 |
| peluse | MooingLemur, hey, you wanna call me real quick and we can catch up a bit more efficiently? | 20:48 |
| MooingLemur | sure | 20:49 |
| peluse | 480 554 3688 | 20:49 |
| clayg | minwoob: no it's a *false* closeness ;) | 20:50 |
| peluse | bah, stupid IP phone isn't working | 20:50 |
| MooingLemur | silence | 20:50 |
| MooingLemur | :) | 20:50 |
| peluse | 480 334 4630 | 20:50 |
| clayg | minwoob: i think there's some pep8 failures too? | 20:58 |
| minwoob | clayg: The 79 characters issue ... I've fixed those. | 20:59 |
| minwoob | clayg: I'll upload the latest in a sec here. | 20:59 |
| clayg | minwoob: ok - try this then -> https://gist.github.com/clayg/eb77a5e69812b6ea25eb | 20:59 |
| clayg | after pulling out the broken frag - we put some number of duplicates at the front of the connection list - it can't be more than parity - 1 because we can only tolerate parity missing and still rebuilt - and we're already missing the one we're rebuilding | 21:00 |
| minwoob | Hmm... the case that I got working was for having 1 extra duplicate. | 21:02 |
| minwoob | Is it better to go with parity - 1? | 21:02 |
| *** CaioBrentano has quit IRC | 21:03 | |
| clayg | minwoob: it's the maximum amount of failure that would tolerable - so it's a stronger test I think to push the limit - if this breaks having only duplicate would also likely break - i'm not sure if there is a strong correlation the other way | 21:04 |
| clayg | minwoob: I didn't realize you had a fix | 21:04 |
| clayg | minwoob: i'm sure whatever you cooked up with also be acceptable | 21:04 |
| minwoob | clayg: If that's the case, then I'll go ahead and test that. | 21:06 |
| minwoob | The one you have does seem like a stronger test. | 21:09 |
| openstackgerrit | Minwoo Bae proposed openstack/swift: EC Reconstructor: Do not reconstruct existing fragments. https://review.openstack.org/193279 | 21:12 |
| *** theoharr has joined #openstack-swift | 21:16 | |
| clayg | torgomatic: you around? | 21:16 |
| torgomatic | clayg: yeah | 21:17 |
| clayg | sweet! | 21:17 |
| clayg | i'm thinking about lp bug #1457691 | 21:18 |
| openstack | Launchpad bug 1457691 in OpenStack Object Storage (swift) "Insufficient Fragments occurred on EC GET object" [High,Confirmed] https://launchpad.net/bugs/1457691 - Assigned to Bill Huber (wbhuber) | 21:18 |
| clayg | and lp bug #1469094 | 21:18 |
| openstack | Launchpad bug 1469094 in OpenStack Object Storage (swift) "Missing .durable files of FAs can cause rebuildable objects to be inaccessible" [High,Confirmed] https://launchpad.net/bugs/1469094 | 21:18 |
| clayg | ... it's all related | 21:18 |
| wbhuber | clayg: how did you confirm that it's all related? | 21:18 |
| clayg | I'm trying to think of what interfaces I might want to grow on the object-server | 21:18 |
| clayg | wbhuber: well - i tried to convey this earlier - it's not like this was a "bug" that was missed | 21:19 |
| clayg | we *know* there's situations where we can have all the datas "out there" but the proxy doesn't know how to find them | 21:19 |
| *** jlhinson has quit IRC | 21:22 | |
| anteaya | this fix merged: https://review.openstack.org/#/c/195940 | 21:23 |
| anteaya | it is working to your satisfaction? | 21:23 |
| *** petertr7 is now known as petertr7_away | 21:23 | |
| *** openstack has joined #openstack-swift | 21:26 | |
| torgomatic | clayg: what if we had the object server convey, on GET response, the timestamps+indices of all the FAs it had *and* added a header on GET requests that would force it to a specific timestamp+index? Is that sufficient to get where you want to be? | 21:28 |
| clayg | anteaya: wfm | 21:32 |
| clayg | anteaya: that was *so* 4 hours ago | 21:33 |
| anteaya | clayg: heh | 21:33 |
| anteaya | thanks | 21:33 |
| clayg | torgomatic: maybe? I was worried when talking to a handoff we'd have to ask - what do you have - ok - can I have this one - and then later if it turns out the handoff is the *only* guy with index-2 and we asked him for index-3 :\ | 21:33 |
| *** DericHorn-HP has quit IRC | 21:34 | |
| clayg | torgomatic: not that we can't ask the same guy for two things | 21:34 |
| torgomatic | clayg: we can run two connections to that same node though | 21:34 |
| torgomatic | heh, exactly | 21:34 |
| clayg | torgomatic: but you had one time said something like "x-if-frag-not-in-list: 0, 2, 4, 6" | 21:34 |
| clayg | torgomatic: where the response would be basically "sure, here's something useful" | 21:35 |
| clayg | and in our loop we just say "while not_enough: ask_the_next_guy_for_something_useful" | 21:35 |
| torgomatic | clayg: yeah, that'd probably work better; that way we could have the fetchers all update a common set of what they found already | 21:35 |
| clayg | torgomatic: yeah definately! | 21:35 |
| clayg | torgomatic: ok - so but if the responses have "all the avaialables" (this is particuarlly useful when dealing with under-replicated durables) | 21:36 |
| clayg | how do you decide if you should ask next-primary/handoffs or go back to a node you already have a connection to? | 21:36 |
| clayg | torgomatic: I guess it's not super common for a node to have more than one frag | 21:37 |
| clayg | torgomatic: definately might have a non-durable frag and serve a stale timestamp - in that case you better go back to him | 21:37 |
| openstackgerrit | Merged openstack/swift: Fix ValueError in ssync_receiver https://review.openstack.org/195457 | 21:38 |
| clayg | torgomatic: like instead of node_iter being a list you consume - we want a queue that you can stick things into | 21:38 |
| torgomatic | clayg: yeah, maybe we start off with continuing into the handoffs because it's easy, and if we end up with a non-trivial number of requests running out of object servers to try, then go write the backtracking code | 21:38 |
| clayg | torgomatic: ahhhh - good call | 21:39 |
| clayg | minwoob: I think i'm still getting pep8 errors? | 21:39 |
| *** annegentle has quit IRC | 21:39 | |
| torgomatic | we just have to make sure to log it and statsd it when it happens, so we can actually know something about its rate of occurrence | 21:40 |
| minwoob | clayg: I'll take a look at what's going on. | 21:40 |
| *** annegentle has joined #openstack-swift | 21:40 | |
| clayg | torgomatic: *another* good point! | 21:41 |
| minwoob | Where do you see that? | 21:41 |
| clayg | minwoob: locally | 21:41 |
| clayg | minwoob: i ran vtox | 21:41 |
| clayg | minwoob: maybe i'm confused | 21:41 |
| clayg | minwoob: if you don't see pep8 errors - you can ignore me until jenkins makes the call | 21:41 |
| clayg | minwoob: swift/obj/reconstructor.py:252:80: E501 line too long (88 > 79 characters) | 21:43 |
| *** silor1 has quit IRC | 21:43 | |
| clayg | ^ I *think* i'm on the latest patch set? | 21:43 |
| minwoob | clayg: Patch set 6 is the latest. I fixed the ones that occured on one of the previous patches. | 21:43 |
| minwoob | clayg: Wait, that's in the reconstructor. | 21:44 |
| minwoob | You're right. Let me go fix that. | 21:44 |
| clayg | ok | 21:44 |
| openstackgerrit | Greg Lange proposed openstack/swift: go: add targeted replication https://review.openstack.org/196259 | 21:46 |
| *** nexusz99 has joined #openstack-swift | 21:46 | |
| minwoob | So | 21:47 |
| minwoob | While running the test suite again just as a sanity check, | 21:47 |
| minwoob | we should discuss how to handle the abort-reconstruct scenario, when an existing fragment is found. | 21:48 |
| minwoob | I think the best way to handle that would be to open up a new LP bug, and new gerrit review. | 21:48 |
| minwoob | So that this current one can go ahead and get merged. | 21:48 |
| minwoob | I'll tackle it, but may have to do it when I get back. | 21:49 |
| clayg | minwoob: sure np - please do see about opening a new bug - I think you might be able to just raise DiskFileError('found fragment') - let ssync skip it? | 21:49 |
| openstackgerrit | Minwoo Bae proposed openstack/swift: EC Reconstructor: Do not reconstruct existing fragments. https://review.openstack.org/193279 | 21:50 |
| *** nexusz99 has quit IRC | 21:51 | |
| minwoob | clayg: Okay - Thanks for the head start. | 21:54 |
| *** acampbell has quit IRC | 21:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 22:04 |
| clayg | minwoob: patch set 7 wfm - well give folks the day to catch up - did you say you'll be out next week? | 22:13 |
| *** NM has quit IRC | 22:14 | |
| minwoob | clayg: I'm just about to step out of the office soon. I'll be back as usual on Monday, possibly online over the weekend. | 22:17 |
| minwoob | Either way, my next line item is the abort-reconstruction scenario. | 22:17 |
| clayg | minwoob: oh - great - well have a nice weeked! | 22:17 |
| clayg | minwoob: ok - please open a bug for it! | 22:17 |
| minwoob | clayg: Will do. | 22:17 |
| clayg | minwoob: thanks again! | 22:18 |
| minwoob | clayg: Have a good weekend! | 22:18 |
| *** DericHorn-HP has joined #openstack-swift | 22:18 | |
| *** DericHorn-HP has quit IRC | 22:19 | |
| *** DericHorn-HP has joined #openstack-swift | 22:40 | |
| *** annegentle has quit IRC | 22:41 | |
| *** nexusz99 has joined #openstack-swift | 22:58 | |
| *** annegentle has joined #openstack-swift | 23:01 | |
| *** nadeem has quit IRC | 23:02 | |
| *** nexusz99 has quit IRC | 23:02 | |
| *** lcurtis has quit IRC | 23:12 | |
| *** pgbridge has quit IRC | 23:12 | |
| *** ekarlso has quit IRC | 23:18 | |
| *** ekarlso has joined #openstack-swift | 23:18 | |
| *** annegentle has quit IRC | 23:19 | |
| *** gyee has quit IRC | 23:21 | |
| *** DericHorn-HP has quit IRC | 23:27 | |
| *** pberis has joined #openstack-swift | 23:33 | |
| *** pberis has quit IRC | 23:54 | |
| *** nadeem has joined #openstack-swift | 23:58 | |
| openstackgerrit | Merged openstack/swift: Add one more test for ssync_receiver https://review.openstack.org/194518 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!