| *** baojg has quit IRC | 00:10 | |
| *** baojg has joined #openstack-swift | 00:11 | |
| *** gyee has quit IRC | 00:56 | |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Fix a race condition in case of cross-replication https://review.opendev.org/754242 | 00:57 |
|---|---|---|
| *** baojg has quit IRC | 00:57 | |
| *** baojg has joined #openstack-swift | 00:58 | |
| *** tdasilva_ has quit IRC | 01:07 | |
| *** tdasilva_ has joined #openstack-swift | 01:08 | |
| *** ChanServ sets mode: +v tdasilva_ | 01:08 | |
| *** baojg has quit IRC | 01:26 | |
| *** rcernin has quit IRC | 02:26 | |
| *** rcernin has joined #openstack-swift | 02:50 | |
| sorrison | What's the recommendation for filesystem types for swift these days, the docs are quite out of date ```For distros with more recent kernels (for example Ubuntu 12.04 Precise),``` | 02:52 |
| *** rcernin has quit IRC | 02:57 | |
| *** rcernin has joined #openstack-swift | 02:59 | |
| *** rcernin has quit IRC | 02:59 | |
| *** rcernin has joined #openstack-swift | 03:00 | |
| *** baojg has joined #openstack-swift | 03:26 | |
| *** psachin has joined #openstack-swift | 03:37 | |
| timburke | sorrison, still recommending xfs -- though i think rledisez/alecuyer have at least played around a bit with zfs? | 03:55 |
| sorrison | ok thanks, we're adding a bunch more nodes into our cluster so just thought I'd check, xfs is what we have now so I think we'll stick with that | 03:58 |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| openstackgerrit | Merged openstack/swift stable/ussuri: Fixing S3 credential caching region support https://review.opendev.org/753840 | 04:50 |
| openstackgerrit | Merged openstack/swift stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/752751 | 05:31 |
| openstackgerrit | Merged openstack/swift stable/ussuri: s3api: Allow lower-cased region name for AWS .NET SDK compatibility https://review.opendev.org/750535 | 06:25 |
| *** viks____ has joined #openstack-swift | 06:38 | |
| *** rcernin has quit IRC | 07:39 | |
| *** baojg has quit IRC | 08:51 | |
| *** baojg has joined #openstack-swift | 08:52 | |
| *** rcernin has joined #openstack-swift | 09:37 | |
| *** rcernin has quit IRC | 10:10 | |
| *** noonedeadpunk has quit IRC | 10:33 | |
| *** noonedeadpunk has joined #openstack-swift | 10:34 | |
| *** rcernin has joined #openstack-swift | 11:42 | |
| *** gmann is now known as gmann_pto | 12:41 | |
| *** rcernin has quit IRC | 13:21 | |
| *** tdasilva_ has quit IRC | 13:39 | |
| *** tdasilva_ has joined #openstack-swift | 13:40 | |
| *** ChanServ sets mode: +v tdasilva_ | 13:40 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Use bigger GreenPool for concurrent EC https://review.opendev.org/754412 | 14:48 |
| *** tkajinam has quit IRC | 15:00 | |
| *** djhankb has quit IRC | 15:27 | |
| *** djhankb has joined #openstack-swift | 15:27 | |
| *** thiago__ has joined #openstack-swift | 15:31 | |
| *** ChanServ sets mode: +v thiago__ | 15:31 | |
| *** tdasilva_ has quit IRC | 15:33 | |
| *** gyee has joined #openstack-swift | 15:55 | |
| ormandj | is it anticipated that the conn_timeout in proxy-server conf is impacting DELETE requests? | 16:33 |
| ormandj | we've got a node_timeout of 300s, a conn_timeout of 20s, and we're seeing DELETE requests fail with a timeout of 20s | 16:34 |
| ormandj | Sep 25 16:22:39 sw-proxy01 proxy-server: ERROR with Object server x.x.x.x:7033/sdam re: Trying to DELETE /AUTH_X/XXXX/Veeam/Archive/VEEAM/XXXXXX-XXXXXX/XXXXXXX/blocks/XXXXXX/XXXX.XXXXXX.00000000000000000000000000000000.blk: ConnectionTimeout (20.0s) (client_ip: x.x.x.x) | 16:37 |
| ormandj | another unrelated question - if something was added with full weighting, which obviously leads to a massive rebalance, can you drop the weighting again partially (based on the amount of data already rebalanced) to let the cluster settle down? or once it's set at full weight, just power through until the full rebalance is done? | 16:43 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Pass thread_locals through to frag queue feeder https://review.opendev.org/754457 | 16:57 |
| clayg | ormandj: if you use min_part_hours dropping weight might not help; and even if it did ... it's just a dice roll you don't end up moving the parts that are already on there! | 16:59 |
| clayg | ormandj: best to power through 😥 | 16:59 |
| clayg | ormandj: the connection timeout can effect any request, e.g. if a node is offline | 17:00 |
| ormandj | to be clear, is that a timeout with tcp re: accepting/acking connection, not the request itself? | 17:05 |
| ormandj | if so - we had expected servers_per_port to help with this (we have saturated drives), is '2' too low for this? | 17:05 |
| ormandj | the way the example config was written it makes it sound like conn_timeout applies to GET/HEAD/etc requests, and node_timeout applies to PUT/DELETE/etc | 17:06 |
| ormandj | we're seeing a huge number of connection timeouts to object storage nodes with drives that are relatively saturated, but i'd expect them to at least accept the request even if it blocks for a while on IO, or at least that was our hope. we are only running at servers_per_port of 2 though | 17:07 |
| clayg | i agree, i wouldn't expect a lot of connection timeouts just cause drives are saturated! 🤯 | 17:34 |
| clayg | maybe if the cpu on those nodes was *super* busy 🤔 | 17:34 |
| clayg | i'm not seeing the node in the example config for node vs conn timeout - in all methods I think we connect with a connect timeout and wait for a response (or continue) for a node timeout | 17:37 |
| clayg | we have https://bugs.launchpad.net/swift/+bug/1834212 to give operators more control over timeouts | 17:38 |
| openstack | Launchpad bug 1834212 in OpenStack Object Storage (swift) "Separate node_timeouts per backend-layer" [Undecided,New] | 17:38 |
| clayg | servers_per_port = 2 is a great place to start, we run 4 - but we also have conn_timeout 0.5 and node_timeout ~8s | 17:39 |
| *** psachin has quit IRC | 18:34 | |
| ormandj | clayg: they're not cpu saturated except container-server eating up 100% | 18:38 |
| ormandj | these nodes have tons of free cpu | 18:39 |
| ormandj | 48 core CPU E5-2650 | 18:39 |
| ormandj | all cores but the one container-server is hammering are 75% idle | 18:39 |
| ormandj | we'd love to have our timeouts that low, but we're already seeing 20s get elapsed. if that's a tcp connect timeout, then something is seriously afoul here. | 18:40 |
| ormandj | any other ideas on what might cause conn_timeout to get hit? | 18:40 |
| ormandj | (if not cpu, that is) | 18:41 |
| *** thiago__ has quit IRC | 19:13 | |
| *** thiago__ has joined #openstack-swift | 19:14 | |
| *** ChanServ sets mode: +v thiago__ | 19:14 | |
| *** thiago__ has quit IRC | 19:16 | |
| *** thiago__ has joined #openstack-swift | 19:17 | |
| *** ChanServ sets mode: +v thiago__ | 19:17 | |
| *** tdasilva_ has joined #openstack-swift | 19:31 | |
| *** ChanServ sets mode: +v tdasilva_ | 19:31 | |
| *** thiago__ has quit IRC | 19:33 | |
| openstackgerrit | Merged openstack/swift stable/train: s3api: Allow lower-cased region name for AWS .NET SDK compatibility https://review.opendev.org/750536 | 20:45 |
| *** baojg has quit IRC | 20:47 | |
| *** baojg has joined #openstack-swift | 20:48 | |
| *** cwright has joined #openstack-swift | 20:51 | |
| *** cwright has quit IRC | 22:06 | |
| *** cwright has joined #openstack-swift | 22:07 | |
| *** baojg has quit IRC | 23:21 | |
| *** baojg has joined #openstack-swift | 23:22 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!