| openstackgerrit | Tim Burke proposed openstack/swift master: sharding: Better-handle newlines in container names https://review.opendev.org/699892 | 01:14 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 01:14 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add versioning+sharding probe test https://review.opendev.org/700464 | 01:14 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Implement object versioning API https://review.opendev.org/673682 | 01:14 |
| *** psachin has joined #openstack-swift | 03:24 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Implement object versioning API https://review.opendev.org/673682 | 05:09 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Middleware that allow for a user to quote Etag https://review.opendev.org/700056 | 05:39 |
| openstackgerrit | Merged openstack/swift stable/train: py3: Fix s3api header casing https://review.opendev.org/697384 | 05:40 |
| openstackgerrit | Merged openstack/swift stable/train: Fix BadResponseLength error when creating symlink https://review.opendev.org/697333 | 07:48 |
| *** diablo_rojo has joined #openstack-swift | 07:52 | |
| *** diablo_rojo has quit IRC | 08:34 | |
| *** csmart has quit IRC | 11:32 | |
| *** baffle has quit IRC | 11:32 | |
| *** evrardjp has quit IRC | 11:59 | |
| *** evrardjp has joined #openstack-swift | 12:03 | |
| *** csmart has joined #openstack-swift | 12:47 | |
| *** baffle has joined #openstack-swift | 12:47 | |
| manuvakery | Hi guys if 2 out of 3 PUT request to storage nodes responds faster and the other request completed relatively slower , does it affect the overall swift upload performance . as per the logs storage nodes took 2.4134 , 1.6452, 1.8505 secs respectively and swift proxy node completed the request in 2.5778 .. this should not be the case i guess it should be max( 1.6452, 1.8505) , i have set the | 13:15 |
| manuvakery | quorum_timeout to 0.5 .. am working on tuning the cluster to get the best out of it | 13:15 |
| *** aluria has joined #openstack-swift | 14:06 | |
| *** evrardjp has quit IRC | 15:55 | |
| *** evrardjp has joined #openstack-swift | 15:55 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-swift | 17:33 | |
| timburke | manuvakery, post_quorum_timeout definitely seems like the thing you want; here's where we use it: https://github.com/openstack/swift/blob/2.23.0/swift/proxy/controllers/obj.py#L481-L494 | 17:35 |
| timburke | so we've got quorum reached at 1.85s, post_quorum_timeout should pop around 2.35s -- there's still 200ms unaccounted-for 🤔 | 17:37 |
| manuvakery | @timburke: so we can avoid the 500ms delay by decreasing post_quorum_timeout to say 0.05.. have to look for remaining 200ms | 17:41 |
| timburke | yeah. it may harm durability a bit, though (if the object's large enough, i'd expect there to still be unsent data in the proxy's socket buffer; when we kill the socket, there's no chance that the slow object-server will get a complete copy) | 17:46 |
| timburke | which would lead to increased IO due to replication | 17:47 |
| timburke | if you're doing benchmarks, i'd recommend fairly long runs (at least an hour, say) and making sure that you've got the background daemons running | 17:48 |
| timburke | at least, for the general case. maybe you've got a workload such that "write for five minutes, then the cluster's going to be idle a while" would be more representative ¯\_(ツ)_/¯ | 17:50 |
| manuvakery | Okk.. default value 0.5 makes sense in that case | 17:53 |
| manuvakery | Yeah I will run benchmark for longer time | 17:54 |
| manuvakery | @timburke: also my storage disks are able to persist data at a rate of 200MBps (avg) but Swift write speed is only 100MBps.. I think the Swift write speed is capped with the disk speed. | 18:17 |
| manuvakery | chunk_size is set to 1mb on both proxy and storage nodes | 18:19 |
| manuvakery | What do u think the bottleneck could be | 18:20 |
| timburke | how's your CPU utilization? i know rledisez has been looking at how to get rid of some MD5ing... i know *that* can be a time-suck | 18:45 |
| timburke | how many disks per node? do you have separate replication NICs? | 18:45 |
| *** psachin has quit IRC | 18:57 | |
| *** spotz has quit IRC | 19:01 | |
| *** spotz has joined #openstack-swift | 19:05 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Middleware that allows a user to have quoted Etags https://review.opendev.org/700056 | 19:36 |
| timburke | i'm debating about putting ^^^ into our recommended pipelines right away... | 19:36 |
| timburke | would certainly make it easier to write some func tests | 19:46 |
| rledisez | timburke: I don't see any reason to not do it considering that by default it changes nothing | 20:06 |
| timburke | that was part of my thinking, too | 20:06 |
| rledisez | at somepoint I started to ask myself if it would be better to quote the etag on the proxy and have a middleware etag_unquoter. because proxy would do the right thing then and the addon just keep the compatibility | 20:17 |
| timburke | fwiw, i remember running into issues with encryption when i first tried to get it into the proxy; that's why https://review.opendev.org/#/c/695131/ wound up handling it mostly in the gatekeeper | 21:40 |
| patchbot | patch 695131 - swift - Add proxy-server option to quote-wrap all ETags - 5 patch sets | 21:40 |
| *** irclogbot_3 has quit IRC | 21:47 | |
| *** irclogbot_0 has joined #openstack-swift | 21:51 | |
| *** d34dh0r53 has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!