| *** two_tired has joined #openstack-swift | 00:21 | |
| *** two_tired has quit IRC | 00:35 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: port account/container replicators https://review.openstack.org/614656 | 00:47 |
|---|---|---|
| *** tdasilva has quit IRC | 01:42 | |
| seongsoocho | wow! swift prometheus exporters !! It's amazing. | 01:57 |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: py3: adapt the account server completely https://review.openstack.org/613505 | 02:16 |
| openstackgerrit | Merged openstack/swift master: Use correct headers in reconstructor requests https://review.openstack.org/614298 | 02:39 |
| openstackgerrit | Merged openstack/swift master: added note about double url quoting https://review.openstack.org/614612 | 02:39 |
| *** psachin has joined #openstack-swift | 03:10 | |
| *** threestrands has joined #openstack-swift | 03:46 | |
| openstackgerrit | Merged openstack/swift master: Remove empty directories after a revert job https://review.openstack.org/611325 | 04:34 |
| *** threestrands has quit IRC | 06:52 | |
| *** gkadam has joined #openstack-swift | 06:58 | |
| *** pcaruana|elisa| has joined #openstack-swift | 07:40 | |
| *** pcaruana|elisa| has quit IRC | 07:59 | |
| *** pcaruana has joined #openstack-swift | 08:05 | |
| *** e0ne has joined #openstack-swift | 09:43 | |
| *** e0ne has quit IRC | 10:32 | |
| openstackgerrit | Merged openstack/swift master: py3: adapt common/db_replicator.py https://review.openstack.org/612553 | 10:34 |
| *** e0ne has joined #openstack-swift | 10:36 | |
| *** pcaruana has quit IRC | 11:05 | |
| *** gkadam has quit IRC | 11:46 | |
| *** gkadam has joined #openstack-swift | 11:50 | |
| *** pcaruana has joined #openstack-swift | 11:52 | |
| *** gkadam_ has joined #openstack-swift | 12:04 | |
| *** gkadam has quit IRC | 12:07 | |
| openstackgerrit | Chuck Short proposed openstack/swift-bench master: Add python35 support https://review.openstack.org/614749 | 12:16 |
| openstackgerrit | Chuck Short proposed openstack/swift-bench master: Add python35 support https://review.openstack.org/614749 | 12:34 |
| *** e0ne has quit IRC | 12:55 | |
| *** gkadam_ has quit IRC | 12:57 | |
| *** e0ne has joined #openstack-swift | 13:40 | |
| *** mark-mcardle has joined #openstack-swift | 13:55 | |
| *** e0ne_ has joined #openstack-swift | 14:09 | |
| *** e0ne has quit IRC | 14:10 | |
| *** gkadam has joined #openstack-swift | 14:13 | |
| *** mvkr has quit IRC | 14:40 | |
| *** gkadam has quit IRC | 15:09 | |
| *** mvkr has joined #openstack-swift | 15:11 | |
| *** kukacz has quit IRC | 15:12 | |
| *** psachin has quit IRC | 15:13 | |
| *** itlinux has quit IRC | 15:13 | |
| *** kukacz has joined #openstack-swift | 15:19 | |
| *** gyee has joined #openstack-swift | 15:33 | |
| *** e0ne_ has quit IRC | 15:34 | |
| *** e0ne has joined #openstack-swift | 15:34 | |
| notmyname | good morning | 15:59 |
| *** itlinux has joined #openstack-swift | 16:14 | |
| *** e0ne has quit IRC | 16:26 | |
| *** e0ne has joined #openstack-swift | 17:31 | |
| *** e0ne has quit IRC | 17:47 | |
| *** e0ne has joined #openstack-swift | 17:59 | |
| *** e0ne has quit IRC | 18:02 | |
| *** e0ne has joined #openstack-swift | 18:05 | |
| *** e0ne has quit IRC | 18:07 | |
| *** mvkr has quit IRC | 18:47 | |
| *** itlinux has quit IRC | 19:01 | |
| *** pcaruana has quit IRC | 19:05 | |
| *** itlinux has joined #openstack-swift | 19:39 | |
| *** mvkr has joined #openstack-swift | 19:51 | |
| *** e0ne has joined #openstack-swift | 20:11 | |
| *** itlinux has quit IRC | 20:14 | |
| *** itlinux has joined #openstack-swift | 20:14 | |
| *** e0ne has quit IRC | 20:28 | |
| timburke | so... how do we feel about https://github.com/openstack/swift/blob/2.19.0/test/unit/__init__.py#L71-L74 ? 'cause we still rewrite HASH_PATH_PREFIX/HASH_PATH_SUFFIX *all over the place* in tests, and rarely if ever with a mock.patch(...) | 20:40 |
| timburke | i'm kinda tempted to set them both in test/unit/__init__.py, rip out all the other updating, and fix the tests to be OK with the new values | 20:41 |
| timburke | (modulo some tests in test_utils.py that are exercising hash_path) | 20:41 |
| zaitcev | dunno | 20:43 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: port account/container replicators https://review.openstack.org/614656 | 20:52 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Clean up HASH_PATH_* patching https://review.openstack.org/614867 | 20:52 |
| timburke | eh, i'll do the lower-risk thing for now, anyway | 20:52 |
| *** fatema_ has joined #openstack-swift | 20:54 | |
| DHE | question. I have a 10+3 EC policy and to the best of my knowledge all codes are intact, but I'm having a very high incidence of my proxy server doing reconstruction rather than just pulling the 10 data fragments and reassembling them. it's hurting performance. suggestions to improve it? | 21:34 |
| *** fatema_ has quit IRC | 21:35 | |
| notmyname | DHE: what do you mean by the proxy doing reconstruction. the default behavior is to get the num_data fragments and reconstruct on that (instead of fetching num_data+num_parity) | 21:41 |
| timburke | DHE: what's your ec_type? i know isa-l offers some optimization on intel hardware, fwiw | 21:41 |
| timburke | notmyname: if even one data frag is slow to respond, we'll go fetch a parity, reconstruct, and performance will suffer | 21:42 |
| notmyname | sure, but that performance hit is because the frag is slow to respond, not because of EC math (right?) | 21:42 |
| timburke | that brings me to my next question, DHE: what's your node_timeout? | 21:43 |
| timburke | i'd expect that the node_timeout would dominate the performance hit... if it doesn't, i feel like it's probably set too low | 21:44 |
| DHE | not set, so default | 21:54 |
| timburke | for both? | 21:55 |
| DHE | node_timeout not set, ec type liberasurecode_rs_vand | 21:55 |
| DHE | the cluster is 2 servers: dedicated PAC and one dedicated ACO with 30 spinning drives for objects. | 21:55 |
| timburke | ...is it a prod cluster? if it were me, i'd really want to try to switch that out for isa-l... let me see if i can find some performance numbers to back that up... | 21:58 |
| DHE | no it's a lab. but I am beating on it with real data for the heck of it. | 21:58 |
| notmyname | yeah, isa-l or jerasure are going to be *much* better than using libec's implementation | 21:59 |
| DHE | the CPU is just slow enough that EC rebuilds hit ~800 megabits on a core, so I can see on the realtime bandwidth and CPU usage display when it's doing direct reassembly and doing EC reconsturction... | 22:00 |
| DHE | (where "slow" is 3.6 GHz) | 22:00 |
| timburke | maybe try it with a new storage policy first. dunno how long it'd take you to rebuild the env. but definitely try something other than libec for the backend -- we mainly have that as a proof-of-concept, unencumbered implementation; others are *much* better | 22:01 |
| notmyname | DHE: https://01.org/intel®-storage-acceleration-library-open-source-version | 22:01 |
| notmyname | wow. intel's lawyers messed that url up | 22:02 |
| notmyname | ok, the real url is linked off of https://software.intel.com/en-us/storage/ISA-L | 22:02 |
| DHE | hahaha | 22:03 |
| zaitcev | The intel® URL worked for me. The wonders of UTF-8. | 22:03 |
| notmyname | yeah, it "works" but my irc client breaks the link at the ® | 22:03 |
| DHE | it renders in my irc client, the symbol works in firefox, but the copy/paste suffered a fail somewhere along the way | 22:04 |
| notmyname | and it's more satisfying to blame lawyers than my irc client ;-) | 22:04 |
| notmyname | I guess https://github.com/01org/isa-l is the best url, though :-) | 22:04 |
| zaitcev | I dingus-clicked on it in Hexchat. | 22:04 |
| * DHE the conn_timeout and node_timeout, let's see if this helps in the short term... | 22:11 | |
| DHE | even though it's a lab, I've put data on it I'd rather not lose given the choice | 22:12 |
| DHE | *raised | 22:13 |
| timburke | hmm... also, do you have insight into how much data is misplaced? if a primary was overloaded during the PUT, we could write a data frag to a handoff, basically guaranteeing that we'd use a handoff in the proxy until the object-reconstructor gets it back where it belongs | 22:15 |
| DHE | I don't think anything's ever been moved. I've never changed any hardware or rebuilt the rings since original creation and there's only the 1 object server machine. worst case it's rebooted once or twice. | 22:16 |
| DHE | last reconstruct pass showed 3 known broken objects (maybe reboot related?) | 22:17 |
| *** threestrands has joined #openstack-swift | 22:17 | |
| timburke | :-( i think I must've gotten rid of the graphs i made a while back... at least i was smart enough to put up https://gist.github.com/tipabu/b614587d2e978df8438c0250cf353ebc and https://gist.github.com/tipabu/454d8859d997110ba0acbca36ab6b7ec so i could make them again | 22:30 |
| timburke | ...but i think i'm too lazy to do that *right now* | 22:30 |
| DHE | 3x8 + 0? | 22:36 |
| timburke | i was experimenting with various alternatives while reviewing the ec duplication patch | 22:37 |
| timburke | 3x8+0 was a terrible, terrible plan :-) | 22:37 |
| DHE | I'm trying to think of whether that's a safer bet mathematically than just 3 way replication and realizing I don't want to do the numbers... :) | 22:38 |
| timburke | definitely noticeably faster to rebuild though! (that is, *when* it could rebuild...) | 22:38 |
| timburke | i'm thinking probably not... durability should come out about the same? performance seems likely to suffer though, as you need so many backend connections | 22:40 |
| DHE | in this particular scenario I'm concerned about CPU usage since that's where I'm bottlenecked right now.... | 22:40 |
| DHE | definitely going to use the intel stuff when we go live... | 22:40 |
| mattoliverau | Morning | 22:45 |
| mattoliverau | Yeah ec will be more proxy cpu bound. So more proxies would help share that load | 22:50 |
| mattoliverau | IE hitting you lab hard with slot of data and I proxy, that has to do more work | 22:50 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Make BufferedHTTPResponse.readline block less https://review.openstack.org/614348 | 23:06 |
| DHE | mattoliverau: the catch is there doesn't seem to be a good reason for reads to require so much reconstruction by the proxy, but it is... | 23:09 |
| timburke | makes me wonder if we're prioritizing the data frags or not... our default sort method is shuffle, yeah? | 23:11 |
| mattoliverau | even if getting all data fragements in a GET, they still need to go through the ec lib to be "reconstructed", though the reconstruction should be a very simple one. | 23:11 |
| mattoliverau | yeah me might not be. need to look at the code. | 23:11 |
| timburke | *way* easier to invert and multiply the identity matrix ;-) | 23:12 |
| *** mvkr has quit IRC | 23:22 | |
| *** mvkr has joined #openstack-swift | 23:23 | |
| notmyname | timburke: I thought I remembered some testing with EC that showed it didn't matter if you used data or parity fragments (speed wise) | 23:49 |
| *** gyee has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!