| *** gyee has quit IRC | 00:07 | |
| *** hoonetorg has quit IRC | 03:02 | |
| *** rcernin has quit IRC | 03:14 | |
| *** hoonetorg has joined #openstack-swift | 03:15 | |
| *** rcernin has joined #openstack-swift | 03:16 | |
| *** rcernin has quit IRC | 03:24 | |
| *** rcernin has joined #openstack-swift | 03:31 | |
| *** psachin has joined #openstack-swift | 03:34 | |
| *** rcernin has quit IRC | 03:45 | |
| *** rcernin has joined #openstack-swift | 03:49 | |
| *** rcernin has quit IRC | 03:56 | |
| *** rcernin has joined #openstack-swift | 04:05 | |
| *** evrardjp has quit IRC | 04:34 | |
| *** evrardjp has joined #openstack-swift | 04:34 | |
| *** ccamacho has quit IRC | 04:53 | |
| *** m75abrams has joined #openstack-swift | 05:59 | |
| *** m75abrams has quit IRC | 05:59 | |
| *** m75abrams has joined #openstack-swift | 06:00 | |
| *** rpittau|afk is now known as rpittau | 06:20 | |
| *** ccamacho has joined #openstack-swift | 07:05 | |
| *** godog has quit IRC | 07:10 | |
| *** godog has joined #openstack-swift | 08:02 | |
| *** rcernin has quit IRC | 08:17 | |
| *** corvus has quit IRC | 08:17 | |
| *** corvus has joined #openstack-swift | 08:30 | |
| *** rpittau is now known as rpittau|bbl | 10:20 | |
| *** tkajinam has quit IRC | 10:22 | |
| *** rpittau|bbl is now known as rpittau | 12:42 | |
| *** psachin has quit IRC | 13:21 | |
| *** manuvakery has joined #openstack-swift | 13:53 | |
| *** m75abrams has quit IRC | 15:00 | |
| ormandj | we're seeing an interesting issue re: throughput. if we fire up aws cli s3 as a client, from any single location, with concurrency of 10/20/30/whatever + chunk sizing of 10M+, on 10G files for example - we get almost exactly 100MB/s. we can fire up 3 in 3 different places, and all get 100MB/s almost exactly, at the same time. any idea where the bottleneck might be? we've tested the entire network | 15:41 |
|---|---|---|
| ormandj | layer from incoming to storage and it's all north of 20gbit/s, and not hotspotting. we're not seeing high await times on more than a drive or two on the swift storage nodes (which is always the case, prod system, lots of stuff going on). we've disabled the ratelimit module to test, same problem. this is all over the s3 layer. any ideas what to check into next? we're not cpu limited as far as we can | 15:41 |
| ormandj | tell anywhere in the stack | 15:41 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: s3api: Don't do naive HEAD request for auth https://review.opendev.org/735738 | 15:58 |
| *** rpittau is now known as rpittau|afk | 16:11 | |
| clayg | ormandj: ^ maybe your container db's are maxing out on HEAD requests 😁 | 16:11 |
| clayg | or even just PUT requests 🤔 | 16:12 |
| alecuyer | ormandj: are you not reaching the CPU limit on the client (awscli) ? I remember seeing something like that and having to change client | 16:16 |
| alecuyer | ooops missed your last sentence, nevermind | 16:17 |
| *** gyee has joined #openstack-swift | 16:46 | |
| *** manuvakery has quit IRC | 16:56 | |
| ormandj | alecuyer: not | 16:59 |
| ormandj | clayg: same container for all three clients | 16:59 |
| clayg | ormandj: well, you might look at those container db's disk (or just try to use more containers to prove out if that layer is a bottle neck) | 17:27 |
| clayg | timburke: sorry i missed the quiet_boto_logging getting moved into the parent | 17:27 |
| ormandj | conainer db disks are on nvme | 17:37 |
| ormandj | they're fine | 17:37 |
| ormandj | we're starting to think it might be 'encryption' in the pipeline | 17:37 |
| ormandj | not sure how offloading for that works, but if it's single-core, then that could be it | 17:38 |
| DHE | it'll certainly be single-core per TCP session at very best | 18:21 |
| DHE | I was going to suggest might be using EC with a slow algorithm | 18:22 |
| clayg | alecuyer: so it looks like an explicit "use the stupid inline crc32" option is going to the way to make upgrades safe, you might find https://gist.github.com/clayg/df7c276a43c3618d7897ba50ae87ea9d useful (thank Tim, cc zaitcev ) | 18:45 |
| clayg | ormandj: i love me some nvme disks! but even when iop constrained I've seen lock contention put limits on req/s to sqlite databases - if you can rework the tests to use multiple containers that would prove it out; i could be wrong 👍 | 18:46 |
| clayg | i whould say "even when NOT iop constrained" - we run container db's on nvme; but we're still finding bottlenecks there on one of our largest sharded root containers when using s3api in particular | 18:47 |
| openstackgerrit | Merged openstack/swift master: s3api: Allow CompleteMultipartUpload requests to be retried https://review.opendev.org/733919 | 20:47 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Identify deleted shards as shards https://review.opendev.org/738113 | 21:30 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Address a sharder/replicator race https://review.opendev.org/738114 | 21:30 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Don't do naive HEAD request for auth https://review.opendev.org/735738 | 23:42 |
| *** gyee has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!