21:00:06 <timburke> #startmeeting swift
21:00:06 <opendevmeet> Meeting started Wed Jul 27 21:00:06 2022 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:06 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:06 <opendevmeet> The meeting name has been set to 'swift'
21:00:15 <timburke> who's here for the swift meeting?
21:00:18 <zaitcev> o
21:00:25 <zaitcev> o7
21:00:45 <kota> hi
21:01:22 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:01:30 <timburke> first up
21:01:37 <timburke> #topic sha1 deprecations
21:01:47 <timburke> the server patches have merged!
21:01:58 <timburke> #link https://review.opendev.org/c/openstack/swift/+/845862
21:02:12 <timburke> to walk back the deprecation a bit in tempurl
21:02:25 <timburke> #link https://review.opendev.org/c/openstack/swift/+/833713
21:02:32 <mattoliver> o/ (sorry im late)
21:02:37 <timburke> to do a similar sort of deprecation for formpost
21:02:40 <acoles> o/
21:03:12 <timburke> i started on a similar patch for /info calls
21:03:14 <timburke> #link https://review.opendev.org/c/openstack/swift/+/850787
21:03:34 <timburke> but it needs some more work -- maybe i'll get to it for next week
21:04:56 <timburke> there's still one open question for me around the sha1 deprecation
21:05:16 <timburke> and that's what should happen with the client backport at https://review.opendev.org/c/openstack/python-swiftclient/+/847398
21:06:02 <timburke> now that we've walked back the deprecation process such that sha1 is still allowed by default, i'm not sure it's still necessary. i'll follow up on it
21:07:03 <timburke> anybody have any questions or comments about the patches?
21:07:45 <zaitcev> not me
21:07:58 <mattoliver> I'll revisit the client one too
21:09:15 <timburke> thanks -- i'm not completely sure what was going on with the gate -- there's a decent chance that our non-master jobs are currently busted :-/
21:09:43 <timburke> all right. let's keep the merge-train rolling
21:09:48 <timburke> #topic stuck sharding DBs
21:10:42 <timburke> nothing new has actually merged yet here, but the outstanding patches are progressing well
21:10:51 <acoles> we unstuck some dbs!
21:10:54 <timburke> #link https://review.opendev.org/c/openstack/swift/+/849548
21:11:00 <timburke> for processing deleted DBs
21:11:11 <timburke> #link https://review.opendev.org/c/openstack/swift/+/850597
21:11:21 <timburke> for trying harder to find a device to use as a handoff
21:11:52 <acoles> that one is new since last week's meeting^^^
21:12:49 <acoles> the db's we're trying to get sharded are on nodes that are retiring and turns out they had no devices with non-zero weight in the ring :( so the sharder couldn't find a device on which to cleave shard dbs
21:13:36 <acoles> the solution is to just *pick a device* for the temporary shard dbs, even if zero weight
21:14:25 <mattoliver> Yeah so progress could be made
21:14:35 <timburke> you can run into similar issues with the reconciler -- but since it would happen during the _post_replicate_hook(), we'd log the error then go ahead and delete the handoff (assuming everything sync'ed)
21:15:51 <mattoliver> The latest patchset has the fallback moved to find_local_handoff_for_part so they both should get the fallback
21:16:18 <acoles> ☝️
21:16:22 <timburke> 👍 i'll take another look this week
21:16:40 <timburke> #topic SLO resource leak
21:16:41 <mattoliver> Thanks
21:17:04 <timburke> OVH spotted a bug recently: https://bugs.launchpad.net/swift/+bug/1980954
21:17:18 <timburke> and even provided a fix
21:17:21 <timburke> #link https://review.opendev.org/c/openstack/swift/+/850357
21:18:12 <timburke> i had a thought for how we might make it a little less resource-intensive
21:18:20 <timburke> #link https://review.opendev.org/c/openstack/swift/+/850782
21:19:03 <timburke> since the original patch would drain the response iter (which could amount to several gigs) before closing
21:19:12 <timburke> thanks for helping review them both, zaitcev!
21:19:24 <zaitcev> np
21:19:37 <zaitcev> But I think I bikeshedded something there
21:19:45 <zaitcev> Oh, I remember now, n/m
21:20:34 <timburke> i feel i should point out that there is an upgrade concern for my follow-up, though: if you've still got pre-2.24.0 object nodes, the slo-delete call could 500
21:22:19 <zaitcev> Sorry, I'm not seeing the scenario.
21:22:35 <zaitcev> The first patch just adds the drain. Seems like no impact.
21:23:08 <zaitcev> The follow-up has 2 consistent actions that do not happen one without the other... And it's fully implemented on the proxy.
21:24:36 <timburke> it's a concern in the follow-up. with it, if you've got an object-server that doesn't have https://github.com/openstack/swift/commit/e8b654f3 then it won't know about X-Backend-Ignore-Range-If-Metadata-Present, so it'll respect the Range that was sent and the json.loads() will fail
21:25:51 <zaitcev> oh... that old
21:26:18 <timburke> i could probably add some special casing to have it fall back to old behavior if it sees an SLO respond with just one byte... or we could just allow the 500, and let the client retry
21:26:59 <timburke> yeah, it puts it at Feb 2020 -- part of why i wanted to flag it up and get people's opinions
21:29:48 <timburke> well, we can sort it out on the review. maybe i'll at least try writing the fallback, see how bad it gets
21:30:05 <timburke> #topic ring v2
21:30:15 <timburke> mattoliver has a +2 on
21:30:16 <timburke> #link https://review.opendev.org/c/openstack/swift/+/834261
21:30:27 <mattoliver> yeah I do
21:30:57 <timburke> anyone else want to weigh in before we go ahead and merge?
21:31:37 <mattoliver> nows the chance to have a say on the new ring on disk format.
21:32:04 <timburke> well, now, and the last two ptgs, and... ;-)
21:33:04 <timburke> maybe i'll see if i can get clayg to take a look later by threatening him to have to support a new file format he hasn't thoroughly looked at :P
21:34:01 <mattoliver> lol
21:34:07 <timburke> all right. one last thing i forgot to add to the agenda
21:34:13 <acoles> I'll try to cast an eye over it...but I can't promise
21:34:24 <timburke> thanks acoles
21:34:26 <timburke> #topic PTG in ohio
21:34:56 <timburke> unfortunately, NVIDIA recently sent out an email saying "We are suspending business travel in order to prioritize our investments, effective immediately. Travel for internal meetings, conferences, and events should be cancelled. Use virtual meetings instead."
21:35:26 <timburke> so... i don't think acoles, clayg, mattoliver, or i are likely able to go :-(
21:36:03 <kota> oh
21:36:12 <timburke> but i'd be happy to help with planning if other people are going to be able to get together there
21:36:54 <timburke> to make sure you'd have a room, etc.
21:36:59 <kota> could we plan another virtual setting of ptg for a specific project?
21:38:02 <zaitcev> I see the final touch-up for dark data watcher is slated for PTG, so is this okay to poke acoles now? Since he's not coming anyway.
21:38:35 <zaitcev> Well the ring v2 is clearly more important but I hope this is simpler
21:38:38 <zaitcev> https://review.opendev.org/c/openstack/swift/+/787656
21:38:46 <timburke> i see no reason we couldn't do a virtual meetup outside of the context of a PTG :-)
21:38:52 <acoles> +1
21:38:58 <kota> +1
21:39:19 <mattoliver> great idea
21:39:20 <zaitcev> You know
21:40:08 <zaitcev> When Covid just hit, in a few months our management started a crackdown on coworking speakeasies.
21:40:33 <zaitcev> Apparently teams would get together in locked and abandoned offices
21:40:46 <zaitcev> Or in hotels that still operated
21:41:08 <zaitcev> We should rent a cabin at Lake Tahoe.
21:41:56 <acoles> reminds me of the fallback to the airbnb in Dublin during the snow :)
21:42:03 <timburke> i'd be real tempted :-) biggest trouble is getting someone to pay for flights for acoles, kota, and mattoliver ;-)
21:44:38 <timburke> i guess the question then would be, when would you all like to do a virtual meeting? i'm guessing we'd want something along the lines of our previous vPTGs for structure...
21:47:58 <timburke> maybe i'll put together a doodle poll to pick a week in the next couple months
21:48:05 <timburke> all right, that's all i've got
21:48:08 <mattoliver> that's a good idea.
21:48:12 <timburke> #topic open discussion
21:48:23 <timburke> anything else we should bring up this week?
21:48:45 <acoles> depeding on if anyone is going to the in person ptg, we could go virtual the same week
21:49:00 <acoles> that'd save me changing my calendar ;-)
21:50:24 <timburke> now that the sha1 patches are mostly in order, i think i'll try to get a release together
21:50:45 <timburke> let me know in the next week or so if there's any other patches you think ought to land before a release
21:51:46 <mattoliver> kk
21:52:47 <timburke> ok, i'm calling it
21:52:59 <timburke> thank you all for coming, and thank you all for working on swift!
21:53:03 <timburke> #endmeeting