| acoles | reid_g: there is a (brief) doc section on memcache here https://docs.openstack.org/swift/latest/deployment_guide.html#memcached-considerations which I point out mainly for the notes re sharding and max memcache entry sizes | 09:51 |
|---|---|---|
| reid_g | I think there is something fishy going on with this memcached server... I should have 7200 max connections but if I increase memcached connection limit to 16k... 32k... they are all being used. The other servers in my pool stay around 7k. Any ideas? | 13:50 |
| DHE | enumerate them? lsof -n -p $PID | 13:57 |
| reid_g | Right now I have it set to 8192 and it is showing lsof -n -p 672679 | grep -c 11211 | 13:58 |
| reid_g | 7986 | 13:58 |
| reid_g | If I changed to 32768 and wait about 30 sec I get messages in journal 'Too many open connections' | 14:00 |
| reid_g | lsof -n -p 676945 | grep -c 11211 | 14:00 |
| reid_g | 32562 | 14:00 |
| opendevreview | Alistair Coles proposed openstack/swift master: Make cmp_policy_info agree with the API behaviour https://review.opendev.org/c/openstack/swift/+/816731 | 14:22 |
| opendevreview | Alistair Coles proposed openstack/swift master: Improve storage policy index reconciliation unit tests https://review.opendev.org/c/openstack/swift/+/816892 | 14:22 |
| opendevreview | Alistair Coles proposed openstack/swift master: Re-write reconciler.cmp_policy_info() https://review.opendev.org/c/openstack/swift/+/816893 | 14:22 |
| acoles | clayg: ^^ | 14:23 |
| clayg | 🤩 | 14:23 |
| opendevreview | Alistair Coles proposed openstack/swift master: Re-write reconciler.cmp_policy_info() https://review.opendev.org/c/openstack/swift/+/816893 | 14:24 |
| DHE | what I meant was to check... where are the actual connections from? is some host connected more often than others? any foreign connections? (!!) | 14:35 |
| reid_g | ah. they are all connections from other swift-proxy servers. did a grep -v -- '->10.40.100' to show anything that shouldn't be there | 14:50 |
| clayg | @acoles "AssertionError: local policy did not change to match remote for replication row scenario no_row" - maybe the tests are still flakey? | 15:59 |
| acoles | just looking into those failures | 16:00 |
| acoles | clayg: it seems the current implementation favours older put over newer delete. IDK how to think about that. These two tests make exact same assertions: test.unit.container.test_replicator.TestReplicatorSync.test_sync_local_create_policy_over_newer_remote_create and test.unit.container.test_replicator.TestReplicatorSync.test_sync_local_create_policy_over_newer_remote_delete - is that reasonable? | 16:27 |
| clayg | so we create r1 with sp0 and r2 with sp1 (at this point r1 should win, r2 was probably a handoff - that's "newer_remote_create") - then we delete r2 (I assume r1 rejected the delete) - so sp1 still wins? Is that "newer_remote_delete"? | 16:29 |
| clayg | I think maybe we nearly almost always prefer the un-deleted spi? | 16:30 |
| acoles | no, sp0 wins (the older put) | 16:53 |
| acoles | and we could also have r1 with sp0, then r2 with sp1, then delete *r1*...and sp1 wins | 16:54 |
| acoles | as you say, undeleted always wins | 16:55 |
| acoles | which I think means that before deleting the container the reconciler would migrate object rows from sp1 to sp0 (earliest put wins), but after deleting will migrate from sp0 to sp1 (undeleted wins) | 16:57 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP container-server: set shard ranges in memcache https://review.opendev.org/c/openstack/swift/+/817294 | 18:24 |
| reid_g | it looks like we resolved our memcached problem (too many open files) by moving it to another host. Not exactly sure why... config is exatly the same as the old one | 19:43 |
| opendevreview | Clay Gerrard proposed openstack/swift master: Ignoring status_changed_at is one way to fix it https://review.opendev.org/c/openstack/swift/+/817302 | 19:52 |
| opendevreview | Tim Burke proposed openstack/swift master: memcache: Prevent possible pool exhaustion https://review.opendev.org/c/openstack/swift/+/817307 | 20:52 |
| reid_g | What have you used to benchmark swift? | 21:58 |
| opendevreview | Tim Burke proposed openstack/swift master: Ensure close socket for memcached if got timeout https://review.opendev.org/c/openstack/swift/+/338819 | 23:25 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!