21:03:22 <mattoliver> #startmeeting swift
21:03:22 <opendevmeet> Meeting started Wed Mar 20 21:03:22 2024 UTC and is due to finish in 60 minutes.  The chair is mattoliver. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:03:22 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:03:22 <opendevmeet> The meeting name has been set to 'swift'
21:03:36 <mattoliver> who's here for the swift meeting?
21:03:55 <mattoliver> o/ (might as make it feel as normal as possible :P)
21:04:33 <mattoliver> As always the agenda is at
21:04:39 <mattoliver> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:05:15 <mattoliver> And I did update the agenda for those reading along later. Although I forgot to add the first topic
21:05:41 <mattoliver> #topic caracal release!
21:06:04 <mattoliver> #link https://review.opendev.org/c/openstack/releases/+/912371
21:06:04 <patch-bot> patch 912371 - releases - Release Swift for 2024.1 Caracal (MERGED) - 3 patch sets
21:06:39 <mattoliver> It's landed, so we have a release! And because we have no one else here I'll have to say it... it's the best swift release yet!
21:07:07 <mattoliver> Actually there were alot of awesome things in this release, so kudos to everyone who works on swift!
21:08:16 <mattoliver> Moving on, and most of these I'll just mention and move on, because the relevent parties aren't here today to give us real status updates. But will still mention them as they seem to be the active things running atm in swift land.
21:08:30 <mattoliver> #topic s3api: Fix handling of non-ascii access keys
21:09:17 <JayF> I confirm best swift release ever, thanks mattoliver :)
21:09:31 <mattoliver> We came across this in our prod. Seems we missed a unicode conversion to wsgi string when doing the py3 migration all those years ago. And we only just discovered it.
21:09:39 <mattoliver> lol, thanks JayF
21:09:50 <mattoliver> But we have a fix
21:09:59 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/913723
21:09:59 <patch-bot> patch 913723 - swift - s3api: Fix handling of non-ascii access keys - 1 patch set
21:10:50 <mattoliver> It's caused if someone has a non-asci character in their aws key.
21:11:24 <mattoliver> We had tests we thought covered this.. and looks like we did. But our fake app in the tests wasn't a good enough fake :(
21:11:45 <mattoliver> Anyway.. just a heads up, I expect that to land before the next meeting.
21:11:57 <mattoliver> #topic expirer grace period
21:13:53 <mattoliver> This is an older topic from the last meeting, but left it in. There is active work going on it. We have an intern working on it atm, and coming along nice. There is current discussions around maybe changing the name a little. Basically it gives us an optional grace period when expiring objects, so there can be like a soft delete.
21:14:20 <mattoliver> one of our users had a need for this. And it's optional.
21:14:51 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/874806
21:14:51 <patch-bot> patch 874806 - swift - expirer: per account and container grace period - 17 patch sets
21:15:00 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/874710
21:15:00 <patch-bot> patch 874710 - swift - support x-open-expired header for expired objects - 36 patch sets
21:15:14 <mattoliver> I'll try and remember to link first :P
21:15:26 <mattoliver> #topic cooperative tokens
21:15:38 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/890174
21:15:39 <patch-bot> patch 890174 - swift - common: add memcached based cooperative token mech... - 14 patch sets
21:15:47 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/908969
21:15:48 <patch-bot> patch 908969 - swift - proxy: use cooperative tokens to coalesce updating... - 10 patch sets
21:16:11 <mattoliver> Jianjian is leading the charge here. And it's really interesting and awesome work.
21:17:29 <mattoliver> For those who have been running swift for a while, or come to the last few PTGs we'd talked about a thundering herd problem, when there is alot of updates coming to one sharded container and the roots shard-ranges have ttl'ed out of cache.
21:17:45 <indianwhocodes> o/
21:19:20 <mattoliver> Well this uses a token in the cache as a lock that updaters can grab. If they get it then they can go to the back end and get the latest set of shardranges and then they'll place them back in cache. It's a modified version of whats called a ghetto lock. A slightly more concurrent version that better supports a distributed system like swift
21:19:40 <mattoliver> Hey indianwhocodes just you and me, so was mostly talking to myself.
21:19:52 <mattoliver> but not anymore! now I don't look as crazy :P
21:20:15 <indianwhocodes> ya scrolled up, good so far!
21:20:35 <indianwhocodes> I have currently nothing ready for reviews yet
21:20:50 <mattoliver> Anyway, the work is awesome. I hope to get in an review the chain.
21:20:55 <mattoliver> thanks :P
21:21:03 <mattoliver> ok next topic
21:21:15 <mattoliver> #topic Feature/MPU feature branch
21:21:35 <mattoliver> So we've created a feature branch! We haven't done one of those in Swift since container-sharding.
21:22:55 <mattoliver> And it's because we're working, finally, on something we've been talking about for years. We used to call it ALO, or atomic large object. Ie, a large object where users can't see the segments so we can have a 1:1 connection.
21:24:25 <mattoliver> We'll follow the MPU api somewhat. No idea what the name will end up being, swift MPU? So this will go with our DLO and SLO. And our s3api will eventually just use them. And no more weird edgecases of orphaned segments!
21:24:42 <indianwhocodes> sounds promising!
21:24:52 <mattoliver> acoles: is leading the charge. No doubt if he was here you'd get a better explanation then I can give :P
21:25:04 <mattoliver> It does!
21:25:18 <mattoliver> #topic aws-chunked transfers
21:25:46 <mattoliver> timburke: would be better at talking about these. And making progress.
21:26:03 <mattoliver> I did look at the patches in the past, I should do so again!
21:26:19 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/909049
21:26:20 <patch-bot> patch 909049 - swift - s3api: Improve checksum-mismatch detection - 5 patch sets
21:26:28 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/909800
21:26:28 <patch-bot> patch 909800 - swift - utils: Add crc32c function - 5 patch sets
21:26:40 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/909801
21:26:40 <patch-bot> patch 909801 - swift - s3api: Add support for additional checksums - 6 patch sets
21:27:08 <indianwhocodes> I have been blackbox testing tim's patch with mountpoint-s3 benchmarks
21:27:11 <mattoliver> I attempting to recreate the probetest failure of that ^ one.
21:27:26 <mattoliver> oh nice
21:27:36 <mattoliver> oh I forgot more links
21:27:45 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/909802
21:27:45 <patch-bot> patch 909802 - swift - WIP: s3api: Additional checksums for MPUs - 6 patch sets
21:27:53 <mattoliver> #link  https://review.opendev.org/c/openstack/swift/+/836755
21:27:53 <patch-bot> patch 836755 - swift - Add support of Sigv4-streaming - 15 patch sets
21:28:05 <mattoliver> man that timburke is a machine!
21:28:33 <mattoliver> yeah we need these aws-chucked transfers for mountpoint-s3 don't we.
21:28:38 <indianwhocodes> agreed.
21:28:55 <mattoliver> So indianwhocodes your a good man to test and review these :)
21:29:33 <indianwhocodes> i am looking into adding more s3api cross-compat tests to  p 909801
21:29:33 <patch-bot> https://review.opendev.org/c/openstack/swift/+/909801 - swift - s3api: Add support for additional checksums - 6 patch sets
21:30:05 <mattoliver> oh nice!
21:30:33 <mattoliver> If/when you have a patch ready, let's add it to the list the patches then :)
21:31:13 <mattoliver> Let's move on.. almost at the end of the agenda :) I don't you we're busy!
21:31:26 <mattoliver> #topic drive-full-checker
21:31:39 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/907523
21:31:40 <patch-bot> patch 907523 - swift - drive-full-checker - 34 patch sets
21:32:51 <mattoliver> I think this is awesome, and we want to get this landed at some point. One of our SRE is interested in testing it. But downstream stuff got in the way this last week or so. Looks like timburke's had a play.
21:33:04 <mattoliver> I hope to too at somepoint.
21:33:25 <mattoliver> Maybe we can play with it in a VSAIO at least.
21:33:58 <mattoliver> Hopefully I can trick sre into looking this week or so :P
21:34:16 <mattoliver> #topic s3api and slo Partnum support
21:34:27 <mattoliver> So the swift side chain landed!
21:34:34 <mattoliver> Nice work indianwhocodes !!
21:34:41 <indianwhocodes> finally.
21:34:51 <mattoliver> Now we can add support to python-swiftclient!
21:35:02 <mattoliver> #link https://review.opendev.org/c/openstack/python-swiftclient/+/902020
21:35:03 <patch-bot> patch 902020 - python-swiftclient - support part-num in python swiftClient - 15 patch sets
21:35:13 <indianwhocodes> i intend to play with the drive-full checker too if it goes ahead with mountpoint they will go hand in hand as major wins
21:35:44 <mattoliver> Oh yeah, great point!
21:35:48 <mattoliver> nice
21:36:28 <indianwhocodes> reason i say that is that it took me just 2 benchmarking jobs to fill up my vsaio, so just the amount of data we have in prod will have a significant impact!!!
21:36:30 <mattoliver> Well I'll try and take a look at the swiftclient partnum patch as soon as I can so we can get it all squared away.
21:36:45 <indianwhocodes> sounds good.
21:37:49 <mattoliver> Hey it's a jianjian , too bad he missed me talking about his awesome cooperative  token stuff, he's going to have to read the logs :P
21:38:12 <jianjian> sorry for being late
21:38:24 <jianjian> that's awesome! yes, I will read the logs. :-P
21:38:44 <mattoliver> indianwhocodes: you can increase your vsaio disk size if need be. But maybe easily filling them is what we need to testing the drive-full-checker anyway :P
21:38:57 <indianwhocodes> exactly.
21:39:26 <mattoliver> #topic Drop support for liberasurecode<1.4.0
21:39:41 <mattoliver> Last topic I have on the agenda before I open the floor.
21:40:05 <mattoliver> This is from last meeting, and I think yeah sounds good.. but I guess I should actaully go review it :P
21:40:16 <mattoliver> I see jianjian did. Nice
21:40:28 <indianwhocodes> i just did as well, lol.
21:40:38 <mattoliver> oh nice
21:41:08 <mattoliver> oh you did to, 8 mins ago!
21:41:33 <mattoliver> Well hopefully that means it'll land over the next week :)
21:41:49 <mattoliver> That's all the topics I gathered  (over the 10 mins before this meeting) :P
21:41:52 <mattoliver> So
21:41:57 <jianjian> yeah, looked good to me, the new liberasurecode also is able to replace old .so library with static functions, which is nice
21:42:30 <mattoliver> oh cool
21:42:41 <mattoliver> #topic open floor
21:43:55 <mattoliver> I've been blowing dust off 3 year old patches I had for a better auto-sharding leader-election algorithm. Still fairly basic, but better then what we have:
21:44:00 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/667030
21:44:00 <patch-bot> patch 667030 - swift - Auto-sharding: first attempt at _elect_leader - 10 patch sets
21:44:39 <mattoliver> That's basically a rebase and squash and an attempt to address some comments from 3 years ago.
21:44:42 <jianjian> wow, leader election... that's cool!
21:45:17 <mattoliver> Well it's something we always had planned. And we purposely picked a overly simple one to land sharding
21:45:56 <mattoliver> But have always told people not to use auto-sharding because it isn't production ready. But we do use auto-sharding in tests
21:46:11 <jianjian> was there a design doc? or mostly it's described in the commit message
21:46:53 <mattoliver> Auto-sharding basically means the sharder takes responsiblity not just for sharding but identifying, scanning and initiate the scanning too.
21:47:09 <mattoliver> yeah there is, and there's been alot of docs over the years.
21:47:27 <mattoliver> I've been trying to gather them all up. I'll find the current link
21:47:42 <indianwhocodes> I have tried it before on a personal swift cluster
21:49:22 <jianjian> thanks. I guess only leader can kick off a sharding with auto-sharding, what happen if leader node dies? will another node stands up to be a new leader?
21:49:51 <mattoliver> these are the problems.
21:50:00 <mattoliver> #link https://docs.google.com/document/d/1VSpmPcEt1NDhDLb8Btvfl6BwnaeGboONvltSM-ZHUVQ/edit?usp=sharing
21:50:25 <mattoliver> ^ that I think only works for nvidians, but I'll make it available to everyone when I get the chance
21:51:00 <mattoliver> There are alot of leader election options to choose. But the first version is an increment of the existing one.
21:51:03 <jianjian> 👍
21:51:29 <mattoliver> Existing one is super simple.. if your index 0 for the container (partition) then your the leader..
21:52:17 <mattoliver> So great for testing and simple, but doesn't actaully use any real ellection and its super easy to get 2 who think they're the leader because of rebalances and eventual consistency
21:53:11 <mattoliver> this newer version takes ring versions into account and only listens and gets a qourum of votes from the latest ring. Meaning handoffs (old primaries now don't get a say).
21:54:00 <mattoliver> And currently I think there is a double check. Am I leader, scan, am I still the leader, write ranges into shardranges and replicate.
21:54:20 <mattoliver> throw away the work if I'm not.
21:55:00 <mattoliver> though maybe thats inefficent. But taking the less split brainy approach
21:55:42 <jianjian> so a shard leader could stop in the middle the sharding during rebalancing, and then cancel its work
21:56:21 <mattoliver> anther approach is to elect quicker and just get better at dealing and recovering from the split-brains. But in realilty we need to solve this anyway, because it'll happen
21:56:34 <mattoliver> not in the middle of sharding.. only in scanning for shardranges.
21:57:01 <mattoliver> Once a leader has scanned and inserted the shardranges. Sharding then happens as per normal.
21:57:17 <mattoliver> but yeah, atm
21:57:50 <mattoliver> I am thinking of also adding a memcache sentinal lock or something
21:57:59 <mattoliver> as an enhancement.
21:58:14 <mattoliver> Or maybe we just need a branc new approach :)
21:58:34 <mattoliver> like I said, these are from years ago and I'm relearning what past Matt was thinking :P
21:58:54 <jianjian> lol
21:59:27 <mattoliver> Dream is one day, we have auto-sharding enabled by default in swift. And we never need to think about it.
21:59:50 <jianjian> that'll be cool
21:59:57 <mattoliver> And for us downstream, we deprecate it from the controller
22:00:07 <mattoliver> Anyway, we're at time
22:00:09 <jianjian> I also feel we need shard shrinking as well regarding to the topic of sharding
22:00:38 <jianjian> yeah
22:01:10 <mattoliver> Yeah, there is a shrinking edge case that is blocking out auto-shrinking atm too, which is another blocker. So that also needs to be solved!
22:01:12 <mattoliver> Thanks for coming and thanks for working on swift!
22:01:15 <mattoliver> #endmeeting