21:04:00 <timburke> #startmeeting swift
21:04:00 <opendevmeet> Meeting started Wed Jul 20 21:04:00 2022 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:04:00 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:04:00 <opendevmeet> The meeting name has been set to 'swift'
21:04:08 <timburke> who's here for the swift meeting?
21:04:22 <kota> o/
21:04:35 <acoles> o/
21:04:42 <seongsoocho> o/
21:05:01 <timburke> sorry to be late
21:05:06 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:05:07 <mattoliver> o/
21:05:13 <timburke> first up
21:05:23 <timburke> #topic sha1 deprecations
21:05:56 <timburke> we've had these patches in-flight for a while, but i think we're getting close on merging
21:06:06 <timburke> #link https://review.opendev.org/c/openstack/swift/+/845862
21:06:32 <timburke> brings sha1 back to enabled-by-default, but keeps the deprecation warning
21:06:38 <timburke> for tempurl
21:06:50 <timburke> #link https://review.opendev.org/c/openstack/swift/+/833713
21:07:25 <timburke> adds sha1 deprecation to formpost, with a lot of common-code extraction
21:08:08 <timburke> mattoliver's got a +2 on the first, and done a good bit of follow-ups on the second (thanks!)
21:08:38 <mattoliver> Well we pretty much had to do the same thing twice because they were basically the same. So we should it it property
21:08:48 <timburke> does anyone feel strongly about wanting to review either of these? otherwise, i'll plan on reviewing the formpost patch with an eye toward landing them both this week
21:10:26 <timburke> shortly after merging them, i'd like to get a release out, since we haven't had a release since march
21:10:34 <mattoliver> I'm good with that, but that probably doesn't count :p
21:11:48 <acoles> sounds like a good plan
21:12:01 <timburke> all right then
21:12:04 <kota> +1
21:12:11 <timburke> #topic backend ratelimiting
21:12:56 <timburke> i remember that we had a plan to do some testing in a staging environment -- did we ever get around to that? (mattoliver, i think?)
21:14:01 <mattoliver> oh yeah, I think our SRE team were going to test that in staging.. not sure where that got too. I'll reach out or even look up the ticket.
21:14:18 <timburke> cool, thanks -- i just want to make sure it hasn't fallen off our radars
21:14:37 <timburke> #topic stuck sharding DBs
21:15:03 <timburke> this seems to have been an area of a lot of concern since our last meeting
21:15:16 <timburke> (sorry that it's been so long -- things kept coming up)
21:15:53 <timburke> i've been doing my best to catch up on patches, maybe acoles or mattoliver could explain the observed issue(s) better, though
21:16:06 <mattoliver> This has flushed out 2 different things.
21:16:29 <mattoliver> 1. The sharder doesn't process deleted undsharded dbs even if they have shards
21:16:44 <mattoliver> 2. our is_delete method isn't really consistant
21:17:13 <mattoliver> the first one we have a patch for, and it'll mean these deleted unsharded dbs will actaully get processed
21:17:24 <mattoliver> I'll find the link in a sec (on my phone)
21:18:10 <timburke> and i think there's a 1a. corollary -- an unsharded DB with shards needs to progress to sharded since it won't replicate
21:18:22 <timburke> #link https://review.opendev.org/c/openstack/swift/+/849548
21:18:31 <mattoliver> in doing the first one, we also found that we didn't migrate the deleted_timestamp when migrating between the retiring and fresh db.
21:18:34 <mattoliver> yeah
21:18:35 <timburke> ^^^ sharder: process deleted DBs
21:18:48 <timburke> #link https://review.opendev.org/c/openstack/swift/+/849450
21:18:55 <timburke> ^^^ Sharding: Sync container_stat table with fresh db
21:19:32 <mattoliver> So those "should" allow deleted shards to be processed and become sharded.
21:20:05 <acoles> yes, so we noticed that once we fixed sharder to process deleted dbs, it would effectively undelete then :( so that needed a fix too
21:21:11 <timburke> between all three patches, though, we're pretty sure we should resolve the issue, yeah? except for a new issue acoles diagnosed today
21:21:32 <acoles> yes and yes
21:22:01 <timburke> (namely, once you've removed all assignments for a device from the container ring, the sharder stops being able to find a device where we can write shard dbs)
21:22:01 <mattoliver> We had some objects in the shards in a deleted unsharded container. So with these we now shold move to sharded state, we'll find these objects, process them, and then be really deleted.
21:22:44 <mattoliver> oh ouch.. yeah, that's a probably
21:22:48 <mattoliver> *problem
21:23:12 <mattoliver> So just need to add the device at 0 weight and the sharder will find it again
21:24:36 <timburke> now, i got the impression that there were a bunch of other patches flying around related to the investigation -- i'd appreciate it if owners could go through them this week and either abandon them or leave a comment about how we really should keep pushing on them and why
21:25:29 <mattoliver> yeah, those are bascially around 2. where is_deleted isn't a complete empty check like .empty.
21:25:34 <timburke> i tried to capture them on the agenda -- though i may have missed some (or erroneously flagged some)
21:25:45 <mattoliver> so more dicussions are needed.
21:26:09 <timburke> fair enough. i just want to have a sense of what's still an active area :-)
21:26:30 <mattoliver> but yeah we have a different patches floating around we should consolidate
21:27:18 <timburke> all right -- i only have one other item for the week
21:27:23 <timburke> #topic ring v2
21:27:28 <acoles> timburke: ack re abandoning
21:28:05 <timburke> mattoliver and i are getting pretty happy with where the head of the chain is ending up -- if anyone else wants to take a look, now's the time
21:28:58 <timburke> otherwise we'll probably just merge it when we're both happy, like the plan for the sha1 patches
21:28:59 <mattoliver> yup otherwise I'm going to press the big +A button :)
21:29:55 <timburke> it features a new on-disk format, and new apis for interacting with rings, and plans for extensions -- so i wanted to flag it up as something that might warrant more eyes
21:30:21 <timburke> #link https://review.opendev.org/c/openstack/swift/+/834261
21:30:24 <timburke> for the main patch
21:31:03 <timburke> #link https://review.opendev.org/c/openstack/swift/+/834621
21:31:19 <timburke> adds the rebalance history
21:31:39 <zaitcev> oh I see. I was just about to ask "is this really the main patch".
21:32:34 <timburke> and recently i added a repair function for what i could imagine being an easy-to-accidentally-do failure mode (unzipping then naively re-zipping a ring file)
21:32:44 <timburke> #link https://review.opendev.org/c/openstack/swift/+/849441
21:33:24 <timburke> all right, that's all i've got
21:33:30 <timburke> #topic open discussion
21:33:42 <timburke> anyone have other topics to bring up this week?
21:36:16 <timburke> all right then
21:36:27 <timburke> thank you all for coming, and thank you for working on swift!
21:36:33 <timburke> #endmeeting