| opendevreview | Tim Burke proposed openstack/swift master: slo: refactor GET/HEAD response handling https://review.opendev.org/c/openstack/swift/+/893578 | 04:24 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/swift master: Add mailmap entry for my new mail https://review.opendev.org/c/openstack/swift/+/900502 | 07:23 |
| mcape | acoles, thank you very much for your detailed advice! Will try that with newer containers that are moved to S3 recently and need to be deleted. | 08:10 |
| mcape | Unfortunately I can't apply it to my old deleted "sharding_test" container. | 08:11 |
| mcape | I did "find ./ | grep '\.db$' | xargs -I % swift-container-info %" on container nodes to find the root container db, and all I got is 5 shards, root container db is missing. | 08:11 |
| mcape | Perhaps it is because container was deleted while storage was on Queens swift version? | 08:12 |
| mcape | I also have tried to bluntly delete shards using curl -XDELETE ..../v1/.shards_AUTH_0deacb4b247b45dd8973e1e56e603ef3/sharding_test-9355909c5261f9f2a1003bc49ab68075-1544548185.28971-5 | 08:12 |
| mcape | But i'm getting 403 Permission denied. | 08:12 |
| mcape | At the moment my brightest idea is to stop container servers/replicators, remove db files from all replicas, and start services back. | 08:12 |
| mcape | However I feel that records for the shards will remain in account DB.. Is there a cleaner way to git rid of those shards? | 08:13 |
| acoles | mcape: so what exactly are the symptoms - you said you were seeing the container sharder making requests? if the root DBs have gone that sounds like maybe the sharder auditing the shard DBs, so are these GET requests to the non-existent root? | 10:09 |
| acoles | as you describe it, this would be the orphaned shards scenario we tried to avoid by not allowing the root to be reclaimed, but TBH I can't recall is that has *always* been the case or added at some point. | 10:11 |
| mcape | acoles: container-sharder does requests looking for container root, gets 404... "Failed to get shard ranges from AUTH_0deacb4b247b45dd8973e1e56e603ef3/sharding_test: Unexpected response: 404 " | 11:35 |
| mcape | also there are GET requests from container-shareder-ic that look like "/v1/AUTH_0deacb4b247b45dd8973e1e56e603ef3/sharding_test%3Fmarker%3D%26states%3Dauditing%26end_marker%3Dbfhfeee%26format%3Djson" | 11:37 |
| mcape | and there are some PUT requests to the root container that look like "PUT /device19/739/AUTH_0deacb4b247b45dd8973e1e56e603ef3/sharding_test" 404 | 11:39 |
| mcape | PUT requests are from container-sharder | 11:39 |
| opendevreview | Alistair Coles proposed openstack/swift master: sq? rename SloAttrs->RespAttrs https://review.opendev.org/c/openstack/swift/+/900518 | 12:49 |
| acoles | mccape: that makes sense. The GETs and PUTs are when the sharder audits a shard container and tries to call home to the root container. | 16:38 |
| acoles | Assuming you are confident that the root container DBs are gone forever, you could try deleting the shard containers by curling a DELETE to one of the container servers for each shard (based on the curl command you get from swift-container-info). You'll need to send an X-Timestamp header with value equal to now in seconds. The replicator will update the other DBs, and they'll eventually get reclaimed. As a sanity check, | 16:43 |
| acoles | first try to HEAD them to confirm they are not already deleted in case I have misunderstood the current state. | 16:43 |
| acoles | Maybe another core dev could cross-check my thinking? | 16:44 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: always use namespaces for updating and listing https://review.opendev.org/c/openstack/swift/+/900350 | 16:45 |
| opendevreview | Alistair Coles proposed openstack/swift master: sq: refactor test_container https://review.opendev.org/c/openstack/swift/+/900546 | 16:45 |
| opendevreview | Merged openstack/swift master: Add mailmap entry for my new mail https://review.opendev.org/c/openstack/swift/+/900502 | 18:00 |
| opendevreview | Tim Burke proposed openstack/swift master: docs: Document WSGI server process management https://review.opendev.org/c/openstack/swift/+/900256 | 22:10 |
| opendevreview | Tim Burke proposed openstack/swift master: wsgi: Reap stale workers (after a timeout) following a reload https://review.opendev.org/c/openstack/swift/+/789035 | 22:10 |
| opendevreview | Clay Gerrard proposed openstack/swift master: slo: refactor GET/HEAD response handling https://review.opendev.org/c/openstack/swift/+/893578 | 23:24 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!