| opendevreview | Merged openstack/swift master: tests: clarify timestamp formats https://review.opendev.org/c/openstack/swift/+/968181 | 03:33 |
|---|---|---|
| opendevreview | Merged openstack/swift master: tests: pass Timestamps to date_header_format https://review.opendev.org/c/openstack/swift/+/968308 | 05:11 |
| opendevreview | Matthew Oliver proposed openstack/swift master: WIP timestamps: use internal timestamp format in db file names https://review.opendev.org/c/openstack/swift/+/968357 | 05:25 |
| opendevreview | Matthew Oliver proposed openstack/swift master: WIP TombstoneReclaimer: quote string timestamp in sql query https://review.opendev.org/c/openstack/swift/+/968358 | 05:25 |
| opendevreview | Matthew Oliver proposed openstack/swift master: WIP Timestamp: add random part to offset field https://review.opendev.org/c/openstack/swift/+/967738 | 05:25 |
| opendevreview | Matthew Oliver proposed openstack/swift master: Sharder: Allow sharder to use timesamps with jitter https://review.opendev.org/c/openstack/swift/+/968277 | 05:25 |
| opendevreview | Alistair Coles proposed openstack/swift master: TombstoneReclaimer: quote string timestamp in sql query https://review.opendev.org/c/openstack/swift/+/968358 | 11:38 |
| opendevreview | Alistair Coles proposed openstack/swift master: test_db.py: clean up usage of Timestamps https://review.opendev.org/c/openstack/swift/+/968457 | 12:19 |
| opendevreview | Alistair Coles proposed openstack/swift master: TombstoneReclaimer: quote string timestamp in sql query https://review.opendev.org/c/openstack/swift/+/968358 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: test_db.py: clean up usage of Timestamps https://review.opendev.org/c/openstack/swift/+/968457 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: Add a Timestamp.zero() method https://review.opendev.org/c/openstack/swift/+/968236 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP timestamps: use internal timestamp format in db file names https://review.opendev.org/c/openstack/swift/+/968357 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: tests: container/test_backend.py cleanup https://review.opendev.org/c/openstack/swift/+/968356 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: sharder: use correct Timestamp format https://review.opendev.org/c/openstack/swift/+/968277 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP Timestamp: add random part to offset field https://review.opendev.org/c/openstack/swift/+/967738 | 14:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: test_sharder.py: add test_cleave_root_legacy_epoch https://review.opendev.org/c/openstack/swift/+/968481 | 14:11 |
| opendevreview | Ashwin Nair proposed openstack/swift master: wip: swift-manage-expirer - add cli with summary, details, balance, rebalance https://review.opendev.org/c/openstack/swift/+/953486 | 17:54 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP timestamps: use internal timestamp format in db file names https://review.opendev.org/c/openstack/swift/+/968357 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: tests: container/test_backend.py cleanup https://review.opendev.org/c/openstack/swift/+/968356 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: sharder: use correct Timestamp format https://review.opendev.org/c/openstack/swift/+/968277 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP Timestamp: add random part to offset field https://review.opendev.org/c/openstack/swift/+/967738 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: test_sharder.py: add test_cleave_root_legacy_epoch https://review.opendev.org/c/openstack/swift/+/968481 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: test_base.py: tighten up use of Timestamp https://review.opendev.org/c/openstack/swift/+/968537 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: container server: use Timestamp.zero() for default timestamps https://review.opendev.org/c/openstack/swift/+/968538 | 18:24 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP test_diskfile.py: fix Timestamp usage https://review.opendev.org/c/openstack/swift/+/968539 | 18:24 |
| opendevreview | Clay Gerrard proposed openstack/swift master: Only 500 if we race with another iv https://review.opendev.org/c/openstack/swift/+/968323 | 18:33 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Adds labeled cleaving metrics to the sharder https://review.opendev.org/c/openstack/swift/+/965421 | 19:39 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Improves recon stats cleaving context retrieval https://review.opendev.org/c/openstack/swift/+/966139 | 19:40 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Implements concurrent shard processing via cpool https://review.opendev.org/c/openstack/swift/+/966512 | 19:47 |
| mattoliver | sorry running late. Had internet issues. But seems to be working again | 21:03 |
| mattoliver | We haven't had a meeting in while. So will run one, even if its just an update on things and me talking to the void of cyber space :) | 21:04 |
| mattoliver | #startmeeting swift | 21:04 |
| opendevmeet | Meeting started Wed Nov 26 21:04:16 2025 UTC and is due to finish in 60 minutes. The chair is mattoliver. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:04 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:04 |
| opendevmeet | The meeting name has been set to 'swift' | 21:04 |
| mattoliver | whose here for the swift meeting? | 21:04 |
| mattoliver | As always the agenda is at: | 21:04 |
| mattoliver | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:04 |
| acoles | o/ | 21:04 |
| mattoliver | hey Al! | 21:05 |
| mattoliver | First topic | 21:05 |
| mattoliver | #topic eventlet removal | 21:05 |
| mattoliver | This is still something that's happening and being spear headed by cschwede. | 21:06 |
| acoles | (I'm only here for 20mins) | 21:06 |
| mattoliver | We have a feature branch so you can see the current progress at: | 21:06 |
| mattoliver | #link https://review.opendev.org/q/project:openstack/swift+branch:feature/threaded+is:open | 21:06 |
| mattoliver | acoles: no worries, I don't think we'll have a long one. More just fyis and because we haven't had one in a while :) | 21:06 |
| mattoliver | We are planning on having a meetpad (video) based swift meeting in the near future so we can talk through more eventlet removal stuff with cschwede . So look out for that. And check the agenda. I'll have a link there when it happens and will also leave messages here | 21:08 |
| mattoliver | The next 2 topics are based on the ts collision work. The first is just a potential warning about EC and ts collisions | 21:09 |
| mattoliver | #topic potential for EC data loss | 21:09 |
| mattoliver | Basically caused by writing the same object at exactly the same time.. the likelyhood is very very low. But we've seen in. | 21:09 |
| mattoliver | Latest swift has patches that better handle collisions down in diskfile, but still trying to fix it to stop happening all together. | 21:10 |
| mattoliver | We have some tooling we wrote to help rebuild things. If anyone else sees this, we'd be happy to share it. | 21:11 |
| mattoliver | In fact we're potentually looking at sharing a bunch of our swift "gizmo" scripts. So might be a good topic to colloaborate on where and all sharing cool swift tools so we can all benefit! | 21:12 |
| mattoliver | We've been lucky but so far every timestamp collision has been with the almost 100% same object data, so rebuild is possible. | 21:13 |
| mattoliver | I guess its important to mention, it becomes a problem when you have at-rest encryption too. | 21:14 |
| mattoliver | Anyway, be aware and ping if you have any issues and need help, and let's talk about sharding gizmos in a future meeting | 21:15 |
| mattoliver | last topic.. | 21:15 |
| mattoliver | #topic Timestamp collisions | 21:15 |
| mattoliver | acoles and myself are working at stopping this completely at the timestamp level | 21:15 |
| mattoliver | The first version just throwing in a random jitter as a quick fix. But working on a better long term solution as a follow up. | 21:16 |
| mattoliver | #link https://review.opendev.org/c/openstack/swift/+/967738 | 21:16 |
| acoles | 🛠️ | 21:16 |
| acoles | the test churn is huge :( | 21:17 |
| mattoliver | Because this makes the first 10 characters of the offset field apart of the timestamp, this effects a bunch of tests and parts of Swift | 21:17 |
| mattoliver | So there is an effort to work through all the unit tests, and get them working (and refactored and cleaned up) with using the timestamp class over time.time() | 21:18 |
| acoles | but we're making progress burning down the test changes. I think we're down form about 650 to about 230 failing tests on the patch chain | 21:18 |
| mattoliver | nice! | 21:18 |
| mattoliver | We also are using the first offset character as a version field, so we can future proof the problem cen fix and give our future selves more rope to improve things in the future! | 21:19 |
| acoles | much of this is due to the fact that we're writing the offset/hex part of the timestamp string *always* regardless of what we put in it. Just so happens that the first cut is to put some random number in there. | 21:20 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Enhances sharding_in_progress metrics in recon log https://review.opendev.org/c/openstack/swift/+/962305 | 21:20 |
| acoles | I made the first character 2 for version 2. Pretty imaginative I think ;-) | 21:20 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Improves recon stats cleaving context retrieval https://review.opendev.org/c/openstack/swift/+/966139 | 21:20 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Implements concurrent shard processing via cpool https://review.opendev.org/c/openstack/swift/+/966512 | 21:20 |
| mattoliver | yup, genius :) | 21:21 |
| mattoliver | I have been playing with a version of our Timestamp class that can handle nanosecond timestamps, has similar test fall out, so test refactoring is going to help there if we decide to also take that route. | 21:21 |
| mattoliver | Turns out now that we're on python 3, we get time.time_ns which is epoch in nanoseconds. So working on a patch to use that.. which is interesting. Because when converting to floats has presision issues. So using the Decimal class. | 21:22 |
| acoles | mattoliver: I'm -1 on 968357: WIP timestamps: use internal timestamp format in db file names | https://review.opendev.org/c/openstack/swift/+/968357, because I think it could cause legacy sharder code to make the wrong decision about container sharding state. I don't think it's a big deal because we're not concerned about timestamp collisions when enabling sharding so the epochs can remain simple float only timestamps. | 21:23 |
| mattoliver | but in theory using nanosecond resolution would mean a smaller collision window. | 21:23 |
| acoles | I'm planning to back out 968357: WIP timestamps: use internal timestamp format in db file names | https://review.opendev.org/c/openstack/swift/+/968357 tomorrow, just FYI | 21:23 |
| mattoliver | thanks acoles | 21:23 |
| mattoliver | yeah, my first patchset didn't have the filenaming changes, because I was worried about things. So was on the fence | 21:24 |
| mattoliver | but special casing tings to .normal worried me.. but seems we're doing it anyway for some timestamps | 21:24 |
| mattoliver | so probably ok | 21:24 |
| mattoliver | new sharders could pick up old shard dbs because the epochs match the filename, but yeah the other way round is problematic :( | 21:25 |
| acoles | reclaim ages are another peculiar case: replicator code actually passes in floats, but most/all tests pass in Timestamps, so we're making the reclaim code path tolerate v2 timestamps only for tests, maybe?? | 21:25 |
| acoles | But I don't see any upgrade issues there so we may as well tolerate both types | 21:26 |
| mattoliver | hmm, yeah. I guess when it comes to reclaim age. it just means things that don't match the jitter, will just get relclaimed the next time when the float part is beyond | 21:27 |
| mattoliver | Current nanosecond patch is at: | 21:28 |
| mattoliver | #link https://review.opendev.org/c/openstack/swift/+/967632 | 21:28 |
| acoles | yep, and I don't think we care about sub-deca-microsec precision after waiting a month to reclaim :) | 21:28 |
| mattoliver | exactly :) | 21:28 |
| acoles | I ended my day down in test_diskfile which mostly cleaned up ok but I'm a little concerned about the content-type timestamp handling on on disk filenames. That's also on my radar for tomorrow. | 21:29 |
| mattoliver | kk, then let's go back floats only for db_state and filenames :) | 21:29 |
| mattoliver | sounds good | 21:29 |
| mattoliver | well I know you need to go al... so let's move on and then close the meeting | 21:30 |
| mattoliver | #topic open discussion | 21:30 |
| acoles | times up for me, thanks for hosting mattoliver | 21:30 |
| mattoliver | any other topics to bring up or shall we just let acoles go | 21:30 |
| mattoliver | nps acoles thanks for coming and for everything else! | 21:30 |
| mattoliver | let's call it a meeting too | 21:31 |
| acoles | ttfn | 21:31 |
| mattoliver | Thanks everyone for coming and/or reading and thanks for working on swift! | 21:31 |
| mattoliver | #endmeeting | 21:31 |
| opendevmeet | Meeting ended Wed Nov 26 21:31:35 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:31 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2025/swift.2025-11-26-21.04.html | 21:31 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2025/swift.2025-11-26-21.04.txt | 21:31 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2025/swift.2025-11-26-21.04.log.html | 21:31 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Implements concurrent shard processing via cpool https://review.opendev.org/c/openstack/swift/+/966512 | 21:40 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Enhances sharding_in_progress metrics in recon log https://review.opendev.org/c/openstack/swift/+/962305 | 21:43 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Improves recon stats cleaving context retrieval https://review.opendev.org/c/openstack/swift/+/966139 | 21:45 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Implements concurrent shard processing via cpool https://review.opendev.org/c/openstack/swift/+/966512 | 21:46 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Adds labeled cleaving metrics to the sharder https://review.opendev.org/c/openstack/swift/+/965421 | 21:53 |
| opendevreview | Christian Ohanaja proposed openstack/swift master: Migrates the sharder to labeled metrics https://review.opendev.org/c/openstack/swift/+/968580 | 23:20 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!