| opendevreview | ASHWIN A NAIR proposed openstack/swift master: s3api: fix s3api mpu complete log for correct c/o https://review.opendev.org/c/openstack/swift/+/966917 | 00:12 |
|---|---|---|
| opendevreview | Alistair Coles proposed openstack/swift master: timestamps: add SimpleTimestamp for ShardRanges https://review.opendev.org/c/openstack/swift/+/968740 | 10:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP Timestamp: add random jitter to offset field https://review.opendev.org/c/openstack/swift/+/967738 | 10:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: object_server: return 400 if POST timestamp has jitter https://review.opendev.org/c/openstack/swift/+/972535 | 10:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: tolerate but round meta timestamp jitter in backend https://review.opendev.org/c/openstack/swift/+/972536 | 10:11 |
| opendevreview | Alistair Coles proposed openstack/swift master: Fix flakey proxy server test_node_timing https://review.opendev.org/c/openstack/swift/+/974215 | 17:03 |
| timburke | almost swift meeting time! | 20:55 |
| acoles | <drum roll> | 20:58 |
| mattoliver | Oh swift meeting? | 21:04 |
| timburke | #startmeeting swift | 21:07 |
| opendevmeet | Meeting started Wed Jan 21 21:07:09 2026 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:07 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:07 |
| opendevmeet | The meeting name has been set to 'swift' | 21:07 |
| timburke | heh, sorry -- got distracted | 21:07 |
| timburke | hello again! | 21:07 |
| mattoliver | lol, nps, i have a very bad track record :P | 21:07 |
| timburke | who's here for the swift team meeting? | 21:07 |
| mattoliver | great to see ya tim! | 21:07 |
| mattoliver | o/ | 21:07 |
| acoles | \o | 21:08 |
| timburke | as usual, the agenda's at | 21:08 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:08 |
| timburke | first up | 21:08 |
| timburke | #topic health update | 21:08 |
| timburke | hi! been a while since i led a meeting -- i've been busy dealing with health issues | 21:09 |
| timburke | the good news is that things seem to be leveling out a bit, so i hope to be more involved and keeping up with things better | 21:09 |
| acoles | good to hear that timburke | 21:10 |
| mattoliver | well thats good to hear! We've realy missed you | 21:10 |
| timburke | big thanks to everyone (especially mattoliver) for keeping things going without me | 21:10 |
| timburke | i suppose it's still a bit of an open question of whether i should nominate myself for PTL next cycle... i guess we'll see how things are next month | 21:11 |
| mattoliver | barely, but doing what I could, big shoes to fill. | 21:11 |
| timburke | 'cause i do feel a bit bad about leaving things in the lurch | 21:11 |
| mattoliver | lets take it a week ata time. | 21:11 |
| mattoliver | no, don't you worry about that | 21:12 |
| timburke | 👍 | 21:12 |
| acoles | we can cope with a lurch, do what's right for you | 21:12 |
| timburke | next up | 21:12 |
| timburke | #topic keystone security bug | 21:12 |
| timburke | i didn't see that this had been covered in the last few meetings, so i wanted to make sure it got called out now | 21:12 |
| timburke | there was a security bug in the way that swift and keystone communicate about s3api access. for more info see | 21:13 |
| timburke | #link https://bugs.launchpad.net/keystone/+bug/2119646 | 21:13 |
| patch-bot | Bug #2119646 - [OSSA-2025-002] Unauthenticated access to EC2/S3 token endpoints can grant Keystone authorization (CVE-2025-65073) (Fix Released) | 21:13 |
| timburke | the long and short of it is that keystone now requires auth tokens be sent by the s3api middleware, which we never did before | 21:14 |
| timburke | good news is that all the relevant patches are merged now (most/all of them merged back in november) | 21:15 |
| mattoliver | \o/ | 21:15 |
| timburke | and that includes backports to stable branches and even unmaintained/2024.1 | 21:16 |
| timburke | bad news is that we still haven't tagged any releases for any of that -- so i ought to get on it | 21:16 |
| mattoliver | ahh nice segway | 21:17 |
| timburke | for any operators that rely on s3api for keystone users, note that you'll want to upgrade (and properly configure) swift first *then* keystone, to minimize downtime for your users | 21:18 |
| timburke | but yeah, next up | 21:18 |
| timburke | #topic swift release | 21:18 |
| timburke | i put up an authors/changelog patch for swift 2.37.0 | 21:18 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/972812 | 21:19 |
| patch-bot | patch 972812 - swift - AUTHORS/CHANGELOG for 2.37.0 - 6 patch sets | 21:19 |
| timburke | thanks for taking a look at it already mattoliver | 21:19 |
| timburke | it was probably one of the harder changelogs to put together -- mostly just because i've been so out of the loop. so if you get a chance, please take a look and let me know if there's anything i should have mentioned but didn't | 21:20 |
| acoles | do you need another review? | 21:20 |
| timburke | couldn't hurt, if you don't mind | 21:20 |
| acoles | oh, ok, I'll take a look :) | 21:20 |
| timburke | i still need to rebase and reno-ize it, anyway, so there's going to be at least one more patchset :-) | 21:21 |
| timburke | but i hope to get that merged/released in the next week or so (of course, that's what i said last week, too...) | 21:21 |
| timburke | from there, i'll start doing stable releases, but that should be easier/faster | 21:22 |
| timburke | and then there was *one* thing i was still hoping to fix before the release... | 21:22 |
| timburke | #topic docker image uploads | 21:23 |
| timburke | they're broken! | 21:23 |
| timburke | have been since opendev needed to re-key the world -- for more info, see | 21:23 |
| timburke | #link https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/WBBLBI6ZS6FA6Q5ZMH4C2MWPL3WG3H24/ | 21:23 |
| mattoliver | ahh ok that explains why it broke | 21:24 |
| timburke | mattoliver has already rotated the password, so now we just need to update the secret in our .zuul.yaml | 21:24 |
| timburke | ...which has proven more difficult that expected | 21:24 |
| timburke | that was complicated by the fact that the only way to verify that i did the update right seems to be attempting to merge this fix (to a non-voting job!) | 21:26 |
| mattoliver | and we should document the procedure once we get it working, so future us aren't pulling our hair again :P | 21:26 |
| timburke | hence p 973194 merging but leaving the job busted | 21:27 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/973194 - swift - Update dockerhub secret (MERGED) - 3 patch sets | 21:27 |
| timburke | but (1) it's busted in a new way -- POST_FAILURE instead of RETRY_LIMIT and (2) i eventually figured out that we want to merge it as a fix for a *voting* job, then fast-follow with another patch to put it back to non-voting | 21:28 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/973990 | 21:28 |
| patch-bot | patch 973990 - swift - CI: Update dockerhub secret (again) - 1 patch set | 21:28 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/973991 | 21:28 |
| patch-bot | patch 973991 - swift - CI: Make swift-upload-image non-voting again - 1 patch set | 21:28 |
| mattoliver | what | 21:29 |
| timburke | i'll keep poking at it, probably reach out to folks in openstack-infra or openstack-dev | 21:30 |
| mattoliver | that seems nuts | 21:30 |
| mattoliver | lol, tanks for taking it on! | 21:30 |
| timburke | seems like the secret's getting decrypted now (yay!), but login fails -- which is weird because the same commands seem to work OMM | 21:30 |
| timburke | like i said, i'll keep poking at it. but i suppose it shouldn't really be a release blocker | 21:31 |
| timburke | next up | 21:31 |
| timburke | #topic timestamp collisions | 21:31 |
| timburke | i know acoles and mattoliver have been working on this a good bit lately -- take it away! | 21:32 |
| acoles | first we made a bit of progress understanding why we see these collisions | 21:32 |
| acoles | ]mostly thanks to Clay... | 21:32 |
| acoles | we think we have a client request stalled in a proxy accept queue for long enough that the client retries. Then the retry and the original enter the proxy concurrently and by chance get the same timestamp. That's a hypothesis at least, can't prove it. | 21:33 |
| acoles | Onto the fix... | 21:33 |
| timburke | is the hypothesis that these requests go to the same proxy, or different proxies? | 21:34 |
| acoles | the plan is to add some random 'jitter' into the existing extra hex part of the timestamp, to effectively increase the precision of the 'time'. | 21:34 |
| mattoliver | different proxies | 21:35 |
| timburke | 👍 | 21:35 |
| jianjian | to different proxies from what I can tell, but could be same proxy in theory | 21:35 |
| acoles | We've needed to do a lot of work to cleanup timestamp handling in tests to allow for them now always (almost always) having the offset part | 21:35 |
| acoles | something like 40 patches merged in this quest. | 21:36 |
| acoles | there's a few more preparatory patches to reduce the places where we assign the x-timestamp, which gets us ready to... | 21:36 |
| acoles | add jitter to request timestamp, initially only for object PUT which is where we see the problem (one step at a time rather than all request types in a big bang) | 21:37 |
| acoles | the patch chain leads to here https://review.opendev.org/c/openstack/swift/+/967738 (and coninues with some follow ons) | 21:38 |
| patch-bot | patch 967738 - swift - WIP Timestamp: add random jitter to offset field - 48 patch sets | 21:38 |
| mattoliver | I'll take a look at the new preparatory patches | 21:38 |
| acoles | I'd love to get al the prep patches out of the way so that we can start to haggle over the implementation details of the timestamp change | 21:38 |
| acoles | thanks mattoliver | 21:39 |
| acoles | I hope that all the cleanup will prove to be a good step forwards regardless. | 21:40 |
| timburke | i'll take a look, too -- been trying to approve as many of those pre-reqs as i could ;-) | 21:40 |
| mattoliver | yeah, just makes sense to use timestamp classes anyway, its swifts way of dealing with time. | 21:40 |
| timburke | all right, last up (from me, anyway) | 21:41 |
| timburke | #topic LRC scheme for liberasurecode | 21:41 |
| timburke | this has been brewing for a while -- someone at OVH wants to add locally repairable (or recoverable, depending on the literature you're following) codes | 21:42 |
| timburke | this lets you have a scheme like 8+4, but where some (say, 2) of those parities are "global" and some (the other 2) are "local" | 21:43 |
| mattoliver | ok, interesting | 21:44 |
| acoles | what's the significance of local/global? | 21:45 |
| timburke | so if you've got two regions, you send 4 data frags and one local parity to each region, then you don't need any cross-region traffic to rebuild so long as you've still got 4 of those 5 frags | 21:45 |
| mattoliver | sounds like I have some reading to do | 21:45 |
| timburke | whereas you'd normally need *8* frags to do any reconstruction at all | 21:45 |
| acoles | are all the frags the same size? | 21:46 |
| timburke | yup | 21:46 |
| acoles | but bigger then a regular 8+4 scheme?? | 21:47 |
| jianjian | what if operator wants to use 8+3 or 10+5? | 21:47 |
| timburke | downside is that you can no longer take an arbitrary set of 8 frags and be assured you can decode -- if you get 4 data frags from one region, the local parity frag for that region is redundant | 21:48 |
| timburke | jianjian, any scheme should be supported; basically there's a one more parameter to it: instead of just k (ndata) and m (nparity), there's also an l (nregions? must be <=m) | 21:49 |
| timburke | trying to get this to be something useful *in swift* is probably still a ways off -- but the liberasurecode patch seems to be getting somewhat close | 21:50 |
| timburke | #link https://review.opendev.org/c/openstack/liberasurecode/+/959280 | 21:50 |
| patch-bot | patch 959280 - liberasurecode - feature: LRC (locally repairable code backend) - 15 patch sets | 21:50 |
| jianjian | oh, got it | 21:50 |
| timburke | if you're interested in erasure coding, linear algebra, or reducing the connections needed for reconstruction, consider taking a look! | 21:51 |
| timburke | there have also been a handful of patches kicked out to liberasurecode and pyeclib as i dug into this -- thanks for taking a look at some of them, mattoliver! | 21:52 |
| timburke | all right, i think that's all i've got | 21:52 |
| timburke | #topic open discussion | 21:52 |
| timburke | anything else we should bring up this week? | 21:52 |
| mattoliver | I've got nothing else to bring up | 21:53 |
| acoles | nor me | 21:53 |
| timburke | all right, let's call it then | 21:55 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:55 |
| timburke | #endmeeting | 21:55 |
| opendevmeet | Meeting ended Wed Jan 21 21:55:47 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:55 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2026/swift.2026-01-21-21.07.html | 21:55 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2026/swift.2026-01-21-21.07.txt | 21:55 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2026/swift.2026-01-21-21.07.log.html | 21:55 |
| jianjian | bye~ | 21:55 |
| mattoliver | thanks timburke ! | 21:57 |
| opendevreview | Merged openstack/swift master: Fix flakey proxy server test_node_timing https://review.opendev.org/c/openstack/swift/+/974215 | 21:58 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!