21:04:16 <mattoliver> #startmeeting swift
21:04:16 <opendevmeet> Meeting started Wed Nov 26 21:04:16 2025 UTC and is due to finish in 60 minutes.  The chair is mattoliver. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:04:16 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:04:16 <opendevmeet> The meeting name has been set to 'swift'
21:04:34 <mattoliver> whose here for the swift meeting?
21:04:50 <mattoliver> As always the agenda is at:
21:04:53 <mattoliver> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:04:56 <acoles> o/
21:05:03 <mattoliver> hey Al!
21:05:22 <mattoliver> First topic
21:05:33 <mattoliver> #topic eventlet removal
21:06:00 <mattoliver> This is still something that's happening and being spear headed by cschwede.
21:06:10 <acoles> (I'm only here for 20mins)
21:06:23 <mattoliver> We have a feature branch so you can see the current progress at:
21:06:40 <mattoliver> #link https://review.opendev.org/q/project:openstack/swift+branch:feature/threaded+is:open
21:06:57 <mattoliver> acoles: no worries, I don't think we'll have a long one. More just fyis and because we haven't had one in a while :)
21:08:07 <mattoliver> We are planning on having a meetpad (video) based swift meeting in the near future so we can talk through more eventlet removal stuff with cschwede . So look out for that. And check the agenda. I'll have a link there when it happens and will also leave messages here
21:09:00 <mattoliver> The next 2 topics are based on the ts collision work. The first is just a potential warning about EC and ts collisions
21:09:15 <mattoliver> #topic potential for EC data loss
21:09:59 <mattoliver> Basically caused by writing the same object at exactly the same time.. the likelyhood is very very low. But we've seen in.
21:10:37 <mattoliver> Latest swift has patches that better handle collisions down in diskfile, but still trying to fix it to stop happening all together.
21:11:02 <mattoliver> We have some tooling we wrote to help rebuild things. If anyone else sees this, we'd be happy to share it.
21:12:00 <mattoliver> In fact we're potentually looking at sharing a bunch of our swift "gizmo" scripts. So might be a good topic to colloaborate on where and all sharing cool swift tools so we can all benefit!
21:13:43 <mattoliver> We've been lucky but so far every timestamp collision has been with the almost 100% same object data, so rebuild is possible.
21:14:31 <mattoliver> I guess its important to mention, it becomes a problem when you have at-rest encryption too.
21:15:02 <mattoliver> Anyway, be aware and ping if you have any issues and need help, and let's talk about sharding gizmos in a future meeting
21:15:10 <mattoliver> last topic..
21:15:25 <mattoliver> #topic Timestamp collisions
21:15:52 <mattoliver> acoles and myself are working at stopping this completely at the timestamp level
21:16:20 <mattoliver> The first version just throwing in a random jitter as a quick fix. But working on a better long term solution as a follow up.
21:16:27 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/967738
21:16:44 <acoles> 🛠️
21:17:17 <acoles> the test churn is huge :(
21:17:20 <mattoliver> Because this makes the first 10 characters of the offset field apart of the timestamp, this effects a bunch of tests and parts of Swift
21:18:04 <mattoliver> So there is an effort to work through all the unit tests, and get them working (and refactored and cleaned up) with using the timestamp class over time.time()
21:18:12 <acoles> but we're making progress burning down the test changes. I think we're down form about 650 to about 230 failing tests on the patch chain
21:18:20 <mattoliver> nice!
21:19:38 <mattoliver> We also are using the first offset character as a version field, so we can future proof the problem cen fix and give our future selves more rope to improve things in the future!
21:20:10 <acoles> much of this is due to the fact that we're writing the offset/hex part of the timestamp string *always* regardless of what we put in it. Just so happens that the first cut is to put some random number in there.
21:20:18 <opendevreview> Christian Ohanaja proposed openstack/swift master: Enhances sharding_in_progress metrics in recon log  https://review.opendev.org/c/openstack/swift/+/962305
21:20:40 <acoles> I made the first character 2 for version 2. Pretty imaginative I think ;-)
21:20:41 <opendevreview> Christian Ohanaja proposed openstack/swift master: Improves recon stats cleaving context retrieval  https://review.opendev.org/c/openstack/swift/+/966139
21:20:47 <opendevreview> Christian Ohanaja proposed openstack/swift master: Implements concurrent shard processing via cpool  https://review.opendev.org/c/openstack/swift/+/966512
21:21:41 <mattoliver> yup, genius :)
21:21:42 <mattoliver> I have been playing with a version of our Timestamp class that can handle nanosecond timestamps, has similar test fall out, so test refactoring is going to help there if we decide to also take that route.
21:22:38 <mattoliver> Turns out now that we're on python 3, we get time.time_ns which is epoch in nanoseconds. So working on a patch to use that.. which is interesting. Because when converting to floats has presision issues. So using the Decimal class.
21:23:06 <acoles> mattoliver: I'm -1 on 968357: WIP timestamps: use internal timestamp format in db file names | https://review.opendev.org/c/openstack/swift/+/968357, because I think it could cause legacy sharder code to make the wrong decision about container sharding state. I don't think it's a big deal because we're not concerned about timestamp collisions when enabling sharding so the epochs can remain simple float only timestamps.
21:23:09 <mattoliver> but in theory using nanosecond resolution would mean a smaller collision window.
21:23:27 <acoles> I'm planning to back out 968357: WIP timestamps: use internal timestamp format in db file names | https://review.opendev.org/c/openstack/swift/+/968357 tomorrow, just FYI
21:23:38 <mattoliver> thanks acoles
21:24:09 <mattoliver> yeah, my first patchset didn't have the filenaming changes, because I was worried about things. So was on the fence
21:24:43 <mattoliver> but special casing tings to .normal worried me.. but seems we're doing it anyway for some timestamps
21:24:51 <mattoliver> so probably ok
21:25:34 <mattoliver> new sharders could pick up old shard dbs because the epochs match the filename, but yeah the other way round is problematic :(
21:25:53 <acoles> reclaim ages are another peculiar case: replicator code actually passes in floats, but most/all tests pass in Timestamps, so we're making the reclaim code path tolerate v2 timestamps only for tests, maybe??
21:26:52 <acoles> But I don't see any upgrade issues there so we may as well tolerate both types
21:27:19 <mattoliver> hmm, yeah. I guess when it comes to reclaim age. it just means things that don't match the jitter, will just get relclaimed the next time when the float part is beyond
21:28:15 <mattoliver> Current nanosecond patch is at:
21:28:18 <mattoliver> #link https://review.opendev.org/c/openstack/swift/+/967632
21:28:23 <acoles> yep, and I don't think we care about sub-deca-microsec precision after waiting a month to reclaim :)
21:28:45 <mattoliver> exactly :)
21:29:00 <acoles> I ended my day down in test_diskfile which mostly cleaned up ok but I'm a little concerned about the content-type timestamp handling on on disk filenames. That's also on my radar for tomorrow.
21:29:25 <mattoliver> kk, then let's go back floats only for db_state and filenames :)
21:29:44 <mattoliver> sounds good
21:30:06 <mattoliver> well I know you need to go al... so let's move on and then close the meeting
21:30:08 <mattoliver> #topic open discussion
21:30:35 <acoles> times up for me, thanks for hosting mattoliver
21:30:35 <mattoliver> any other topics to bring up or shall we just let acoles go
21:30:53 <mattoliver> nps acoles thanks for coming and for everything else!
21:31:04 <mattoliver> let's call it a meeting too
21:31:12 <acoles> ttfn
21:31:32 <mattoliver> Thanks everyone for coming and/or reading and thanks for working on swift!
21:31:35 <mattoliver> #endmeeting