| opendevreview | ASHWIN A NAIR proposed openstack/swift master: unit tests for s3api slo patch https://review.opendev.org/c/openstack/swift/+/899990 | 00:14 |
|---|---|---|
| mcape | Hey All! I need help with the proper method for deleting a sharded container, I couldn't find this info in the documentation. | 01:29 |
| mcape | Should I use the swift-manage-shard-ranges tool to first delete the shards from the root container and then delete the root container itself? | 01:29 |
| mcape | A few years back, I deleted a small sharded container that was set up for a sharding test. | 01:30 |
| mcape | Since then, the container-sharder has been making several requests per second to shards of the deleted container, that seem to be left in the database. | 01:30 |
| mcape | Is there a way to remove these remaining shards? | 01:31 |
| mcape | Thanks in advance for any help or suggestion! | 01:31 |
| opendevreview | Merged openstack/swift master: Utils: fix Namespace and ShardRange attribute encoding in py2. https://review.opendev.org/c/openstack/swift/+/899993 | 02:16 |
| acoles | mcape: it depends what you mean by 'deleted'. If you mean that you deleted all the objects and then issued a DELETE request to the container via the API, then the current state should be that you have DBs on disk for the container (the 'root', lets call it myaccount/x) *and* it's shard containers (these will be in .shards_myaccount). The sharder continues to audit these shard DBs which may be the requests you se (although | 09:48 |
| acoles | I'd expect them to be requests to the root container). You may also see some logs 'Reclaimable db stuck waiting for shrinking'. | 09:48 |
| acoles | To get rid of the root container DBs, you first need to get rid of the shards. That's because even though the root DB has no objects and is in a deleted state, unlike a normal 'deleted' container, the DB won't be cleaned up after reclaim_age, because doing so would 'orphan' the shard container DBs. | 09:51 |
| acoles | To get rid of the shards, you can use the `swift-manage-shard-ranges compact` command https://github.com/openstack/swift/blob/master/swift/cli/manage_shard_ranges.py#L1068. You'll run this on *one and only one* replica of the root container DB. It will move the objects in smaller shards into a neighbouring shard, and ultimately, when there are no neighbours left, move them into the root container. Since your shards should | 09:57 |
| acoles | be empty, there's no objects to move, so the effect is just to prune the shards until they are all gone. | 09:57 |
| acoles | The `compact` command has options to constrain how much compaction is done, so you can proceed conservatively and repeat - there's a test here that illustrates its use https://github.com/openstack/swift/blob/master/test/probe/test_sharder.py#L2993 | 10:00 |
| acoles | The `show` subcommand will show you the active shards, so you should see them disappearing as you use `compact`. | 10:03 |
| acoles | some caveats: | 10:03 |
| acoles | we've rarely done this in production. It sounds like this is a test container that you no longer care about. | 10:04 |
| acoles | after each `compact` command, the sharder and replicator daemons need to pass over all the DBs (root and container) maybe a couple of times (the command merely states the desired state, the daemons then take action to reach that state). | 10:06 |
| acoles | so be patient | 10:06 |
| acoles | run the commands on only one replica of the root DB, otherwise there is a risk of conflicting desired states. | 10:07 |
| acoles | once the shards have all gone, the `show` command should show none, the `--include_deleted` option could be used to confirm the shards are deleted. The `info` command should show that the root containers own shard range is in state ACTIVE. | 10:11 |
| acoles | The root is now ready to be reclaimed (again, assuming it has no object rows). You will still need to wait for reclaim_age for the DBs to be removed from disk. | 10:12 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: replication network aware helper for container-updater in proxy https://review.opendev.org/c/openstack/swift/+/899171 | 17:53 |
| opendevreview | Tim Burke proposed openstack/swift master: slo: refactor GET/HEAD response handling https://review.opendev.org/c/openstack/swift/+/893578 | 17:58 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: always use namespaces for updating and listing https://review.opendev.org/c/openstack/swift/+/900350 | 18:00 |
| opendevreview | Tim Burke proposed openstack/swift master: slo: refactor GET/HEAD response handling https://review.opendev.org/c/openstack/swift/+/893578 | 19:26 |
| opendevreview | Tim Burke proposed openstack/swift-bench master: Catch connection errors and count them as failures https://review.opendev.org/c/openstack/swift-bench/+/900448 | 19:38 |
| opendevreview | Tim Burke proposed openstack/swift master: wsgi: Handle multiple USR1 signals in quick succession https://review.opendev.org/c/openstack/swift/+/747496 | 20:01 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Nov 8 21:00:32 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliver | o/ | 21:01 |
| mattoliver | Although I had a very distracted week and didn't get much Swift work done.. but I'm here :) | 21:02 |
| kota | o/ | 21:03 |
| opendevreview | Tim Burke proposed openstack/swift master: slo: refactor GET/HEAD response handling https://review.opendev.org/c/openstack/swift/+/893578 | 21:03 |
| timburke | so i neglected to update the agenda again (sorry) | 21:04 |
| timburke | but the main things i've been thinking about are (1) the SLO refactors that are stacked in front of the part-number APIs | 21:05 |
| timburke | and (2) how we manage our processes (especially WSGI servers) | 21:05 |
| mattoliver | yeah, the chain seems to be getting long. And a bit daunting. How are they looking? | 21:06 |
| timburke | #topic SLO refactor | 21:06 |
| timburke | the first part has now merged, just refactoring tests | 21:06 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/896466 | 21:06 |
| timburke | the second part (which *does* impact how SLO behaves) is coming along; i've been pushing up a few fixups today | 21:07 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/893578 | 21:08 |
| timburke | i think at this point i'm happy with it; if anyone else wants to take a look, you might want to do that soon | 21:08 |
| timburke | as it is, clayg, acoles, and indianwhocodes are all listed as authors, and i'm likely to put a +2 on it once we get a clean test run | 21:09 |
| mattoliver | OK. I'll be more dilegent and make sure a corner off parts of my day to reviews. Probably healther for my mind to take a break an conext switch anyway :) | 21:10 |
| timburke | thanks! don't worry too much if you don't find time; we'll likely get it merged by next week, i think | 21:11 |
| timburke | #topic part-number APIs | 21:11 |
| timburke | on top of that refactor, we've got a fairly classic new-slo-feature, new-s3api-feature chain | 21:12 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/894570 | 21:12 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/894580 | 21:12 |
| mattoliver | yeah, kk so the ones that continue on the SLO refactor chain (for those playing along at home) | 21:13 |
| timburke | these are also coming along, though there's a a bit more i'd like to see before they merge, specifically around making sure we send back X-Parts-Count and Content-Range when appropriate | 21:14 |
| mattoliver | Cool, I'll work up the chain, as getting the refactor squared away is need first. | 21:15 |
| mattoliver | Might also do a local refactor of tracing on top and see how SLO actions look in a trace in comparision I assume I sholdn't see any difference.. but might be nice to check :hmm: | 21:17 |
| timburke | oh, nice idea! yeah, i'd expect they should all look more or less the same (at least, until you start exercising the new api) | 21:18 |
| timburke | #topic wsgi server management | 21:18 |
| mattoliver | yeah, 100% expect so. Just want to get used to doing it.. in case we see differences | 21:18 |
| timburke | i've been trying to browbeat clayg into reviewing some patches of mine that we've been carrying for a while | 21:18 |
| timburke | one's landed already | 21:19 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/837633 | 21:19 |
| timburke | the last in that chain still needs some review, though | 21:20 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/837641 | 21:20 |
| timburke | that will remove the process-tree polling from `swift-reload` in favor of a more explicit signal that the reload has completed | 21:21 |
| timburke | but there were some other patches going, too | 21:21 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/789035 | 21:21 |
| timburke | ensures old worker processes don't hang around indefinitely following a reload (we've occasionally had ops notice old workers continuing to process in-flight requests for many hours after a reload) | 21:22 |
| timburke | which reminded clay that he didn't remember how reloads work | 21:23 |
| timburke | so i wrote up some docs about it! | 21:24 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/900256 | 21:24 |
| mattoliver | oh yeah. I think remembering the process polling is only a intermediate change.. I wonder if I should review the new approach first. If we like that we could skip the ps polling patch altogether (unless its too big a patch to merge). | 21:24 |
| mattoliver | \o/ docs! awesome | 21:24 |
| timburke | too late on the process-tree polling! sorry. https://github.com/openstack/swift/blob/master/swift/cli/reload.py#L109 | 21:25 |
| mattoliver | yeah, I needed to read up on systemd stuff when I wanted to review. nice one man! I just wish gerrit could display SVGs. | 21:25 |
| mattoliver | oh well, I probably merged that so it's my bad :P | 21:26 |
| timburke | the docs preview site is pretty good: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_34c/900256/2/check/openstack-tox-docs/34c02b1/docs/overview_wsgi_reload.html | 21:26 |
| mattoliver | oh that is cool. I'd have just looked locally, but that's much easier! | 21:27 |
| timburke | i'm realizing that it maybe should be a more-general "WSGI Server Process Management" page, though, with a starting section about how graceful shutdowns work | 21:29 |
| timburke | working through all of these, clayg and i have been debating about finding a way to get rid of the socket-closer process, so i might spend some time looking into that next | 21:29 |
| timburke | oh! and there was one other patch i was reminded of: | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/747496 | 21:30 |
| timburke | currently, if you send two USR1 signals in quick succession, the second one can land in between the execv call and installing the new signal handler -- in that case, we've still got the defaul handler, which just kills the process | 21:32 |
| mattoliver | oh, yeah opps. | 21:33 |
| mattoliver | +1 to a wsgi server process page too. | 21:33 |
| timburke | that patch sets a signal mask before execv, then clears it after we've got the handler installed. unfortunately, it had to dip into ctypes to work on py2 -- but maybe i could let the bug linger on py2? | 21:34 |
| mattoliver | yeah, I think we're getting ready to drop py2, so don't go to too much effort. Like atm tracing is a noop in py2. Move to py3 if you want it. | 21:35 |
| timburke | all right, i think that's all i've got | 21:35 |
| timburke | #topic open discussion | 21:35 |
| timburke | anything else we ought to bring up this week? | 21:35 |
| mattoliver | nope, I mean I do want to work on some sharding improvements. Maybe play with high water marks in shard replication.. but just haven't had time.. maybe soon. | 21:36 |
| timburke | all right then -- we'll stop a little early | 21:38 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:38 |
| timburke | #endmeeting | 21:38 |
| opendevmeet | Meeting ended Wed Nov 8 21:38:27 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:38 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-11-08-21.00.html | 21:38 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-11-08-21.00.txt | 21:38 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-11-08-21.00.log.html | 21:38 |
| mattoliver | thanks for all your hard work timburke | 21:39 |
| opendevreview | Clay Gerrard proposed openstack/swift master: wip: -100 is MAX_BALANCE https://review.opendev.org/c/openstack/swift/+/900482 | 21:41 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Container-server: add request timing metrics for PUT/GET object/shard/container. https://review.opendev.org/c/openstack/swift/+/898700 | 23:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!