07:01:05 <mattoliver> #startmeeting swift
07:01:05 <opendevmeet> Meeting started Wed Sep 24 07:01:05 2025 UTC and is due to finish in 60 minutes.  The chair is mattoliver. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:01:05 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
07:01:05 <opendevmeet> The meeting name has been set to 'swift'
07:01:18 <mattoliver> Who is here for the swift meeting?
07:01:46 <cschwede> o/
07:02:01 <mattoliver> nice, hey cschwede !
07:02:13 <mattoliver> at least i wont be talking to myself :P
07:02:23 <mattoliver> As usual the agenda is at:
07:02:29 <mattoliver> #link https://wiki.openstack.org/wiki/Meetings/Swift
07:03:11 <mattoliver> I'm going to skip the bug topic, I forgot to remove it. Instead we'll move it to a topic at the PTG :)
07:03:12 <mattoliver> so
07:03:24 <mattoliver> #topic eventlet removal POC update
07:03:39 <mattoliver> #link https://review.opendev.org/q/status:open+project:openstack/swift+branch:feature/threaded
07:03:55 <mattoliver> We have a feature branch, so that might be helpful ^
07:04:22 <mattoliver> How goes things with the POC cschwede ?
07:04:40 <cschwede> No big updates this time, been on PTO last week. That said, pretty good progress before PTO - unit+functional tests passing for the account-server :)
07:05:22 <cschwede> Currently working on more fixes to eventually pass all tests without eventlet too
07:06:14 <mattoliver> nice! and fair enough with the PTO :)
07:06:34 <mattoliver> I hope the time off was awesome!
07:07:00 <cschwede> Thanks, it was - been out in the mountains :D
07:07:38 <mattoliver> I'm jealous!
07:07:41 <mattoliver> I have been meaning to give the acount-server sans eventlet a whirl, just haven't seem to find the time yet. Work is always crazy busy.
07:07:48 <mattoliver> Let's move on then
07:08:02 <mattoliver> #topic October vPTG
07:08:08 <mattoliver> The vPTG is coming up
07:08:17 <mattoliver> Oct 27-31
07:08:55 <cschwede> Wow, only one month left until PTG - time flies by!
07:08:59 <mattoliver> And as in PTGs of very old, I'll be jetlagged, because the week b4 I'll be in the US and would have only just got home. So seems fitting :P
07:09:09 <mattoliver> #link https://ptg.openinfra.org/
07:09:18 <mattoliver> #link https://etherpad.opendev.org/p/swift-ptg-gazpacho
07:09:43 <mattoliver> We now have the etherpad to fill out. I just dded the bug triage topic :P
07:09:57 <mattoliver> Still need to add others, will do that soon.
07:10:21 <cschwede> Yes, let's discuss some plans to get better with the triage - and make more progress. Thx!
07:10:23 <mattoliver> But please if there is anything you want to talk about, and cool ideas, add them so we can discuss and make swift even better!
07:10:42 <mattoliver> Yeah, like just remove old bugs :P
07:11:09 <mattoliver> Also don't forget to register for the PTG
07:11:28 <mattoliver> Thats more informational, so let's move on
07:11:39 <mattoliver> #topic ringv2 next steps
07:11:47 <mattoliver> Chain starts:..
07:11:56 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/955263
07:12:09 <mattoliver> I bring this up because it's something I'm digging into.
07:12:24 <mattoliver> And the chain has grown and about to get even longer :P
07:12:54 <mattoliver> The first few patches are just some cleaning things up. And a ring/ringdata refactor to make the later patches better
07:13:02 <mattoliver> s/better/cleaner/
07:14:14 <mattoliver> The tail of the chain is making the reconstructor smarter by looking for hashes on an old primary, and if there is don't build a sync job. Because it still has frags it hasn't moved (ie don't bother rebuilding if the old primary still has it).
07:15:02 <cschwede> Ah, that sounds like a great improvement!
07:15:05 <mattoliver> Currently I'm working on the proxy being smarter with old proxies, ie GETs to look at old primaries (kinda as a first handoff)
07:15:42 <cschwede> smarter with old _primaries_?
07:16:39 <mattoliver> yeah, sorry. Using the rings history tables, so on "reads" and if there is an old primary, check it before other handoffs.
07:17:15 <cschwede> thx, these improvements sound pretty great.
07:17:20 <mattoliver> But I've moved the last_parts_moved to the ring too, so it can also know how long ago the part was moved and should it bother checking
07:17:44 <mattoliver> thanks. They're still in flux, esp the later patches. But we can talk all about them at the PTG :)
07:18:10 <mattoliver> Speaking of other interesting patches I'm working on..
07:18:16 <mattoliver> #topic unique proxy offsets
07:19:03 <mattoliver> So clayg and acoles are working on improvments to timestamp collisions for EC in diskfile.
07:19:13 <mattoliver> And they're making it much more robust and much better.
07:19:18 <mattoliver> At the same time, I
07:20:04 <mattoliver> am working trying to stop timestamp collisions from happening at all by using a modified and simlified version of the lamport clock algorithm
07:20:31 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/959009
07:21:05 <mattoliver> Basically giving every proxy a unique offset to add to x-timestamps, so even if we have a timestamp collision, the offset automatically breaks the tie
07:21:40 <mattoliver> So far, it seems to work great. We already have Timestamp class throughout swift which supports offsets AND it's already plumbed through to diskfile
07:22:05 <mattoliver> So far all the probetests that we have to simulate the EC timestamp collisions are fixed by it too.
07:22:28 <cschwede> Just read the commit msg - really like the details in the msg, kudos
07:23:13 <mattoliver> Just need to eyes on it because the generating of a unique proxy number (automatically) is always hard
07:23:15 <mattoliver> thanks :)
07:23:59 <mattoliver> Moving on to the last topic I have
07:24:08 <mattoliver> #topic Relinker - allow clobber-hardlink-collision
07:24:53 <mattoliver> So we've been doing another part power increase on our EC policy. And had relinker's blocked and not finishing
07:25:23 <cschwede> uhh, what happened there?
07:25:45 <mattoliver> Turns out, out cluster is really big and we've had some more timestamp collisions.. and we've to got a place where the old and new object are 2 different inodes and not hardlinked.
07:26:31 <mattoliver> ie one writes the first and then hardlinks it to the 2nd location. As the first one somehow writes the first one and then fails to harklink the second because it exists.
07:26:55 <mattoliver> Then when the relinker comes along it also fails because the file is already there with a different inode.
07:27:11 <cschwede> ouch!
07:27:37 <mattoliver> There wasn't many so the SRE could manually unlink the new location and let the relinker do the hardlink again.
07:27:44 <mattoliver> But it was still manual
07:27:58 <mattoliver> Again, our timestamp collision stuff will also solve this eventually
07:28:30 <cschwede> so even at your very large cluster size this seems to be a very rare event, given that it can be fixed manually in a timely manner?
07:28:37 <mattoliver> but in the mean time, clayg has a patch to give SRE the option to clobber in the relinker
07:28:58 <mattoliver> yeah, that and we have dedicated SREs :P
07:29:08 <cschwede> #link https://review.opendev.org/c/openstack/swift/+/961761
07:29:20 <mattoliver> but yeah. enough to slow things down as to figure out why a bunch of disks never end
07:29:28 <mattoliver> oh yeah, thanks cschwede :)
07:30:05 <mattoliver> So that patch allows ops to set an option to globber the file when hardlinking. Just to make is easer for them if there is a bunch on a node that they want ot run manually
07:30:13 <mattoliver> Anyway, thats all I have
07:30:20 <mattoliver> #topic open discussion
07:30:32 <mattoliver> I've already talked about all my fun stuff :)
07:30:51 <mattoliver> Anything else to bring up?
07:31:07 <cschwede> Nothing from my side, thx
07:31:13 <cschwede> As always, thx mattoliver for all the updates and your time!
07:31:27 <mattoliver> Well then let's call it early.
07:31:38 <mattoliver> Thanks for coming and thanks for working on swift!
07:31:42 <mattoliver> Thanks cschwede
07:31:45 <mattoliver> #endmeeting