21:01:18 <timburke_> #startmeeting swift
21:01:18 <opendevmeet> Meeting started Wed Feb  5 21:01:18 2025 UTC and is due to finish in 60 minutes.  The chair is timburke_. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:18 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:18 <opendevmeet> The meeting name has been set to 'swift'
21:01:24 <timburke_> who's here for the swift meeting?
21:01:38 <kota> hello
21:01:56 <jianjian> o/
21:02:44 <mattoliver> o/ (but might be a little distracted this morning)
21:03:15 <timburke_> i actually remembered to update the agenda!
21:03:18 <timburke_> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:40 <timburke_> first up
21:03:54 <timburke_> #topic storage policies in account listings
21:04:27 <timburke_> so Callum027 wrote up a bug & patch
21:04:30 <timburke_> #link https://bugs.launchpad.net/swift/+bug/2097074
21:04:40 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/940601
21:05:14 <timburke_> i had a chance to take a look today, and it looks really promising
21:06:08 <timburke_> we've been tracking policy info in the account's container table since we introduced storage policies way back in https://github.com/openstack/swift/commit/00a162c4
21:06:50 <timburke_> i can't think of why any reason why we wouldn't have started returning that info in listings :-/
21:07:40 <timburke_> if anyone remembers why we didn't do that (or has any other concerns about it) please check out the patch!
21:07:47 <jianjian> yes, that'll be helpful info to return
21:08:00 <mattoliver> what we have, I never noticed that before (always tracking sp in account db)
21:09:11 <timburke_> i feel like our SRE tooling currently periodically does a bunch of HEADs against every container in the cluster; i bet it'd be way better if we could switch that to just looking at account listings
21:09:45 <timburke_> next up
21:09:51 <timburke_> #topic new boto3 and aws-chunked
21:10:36 <timburke_> so new boto3 (/s3transfer/awscli/whatever) started using a chunked protocol by default for uploads
21:11:16 <timburke_> see https://github.com/boto/boto3/issues/4392 for a little more context
21:12:08 <timburke_> but it gave us some renewed interest in getting support for aws-chunked into s3api
21:12:30 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/836755
21:13:24 <timburke_> i pulled that patch out of the chain i'd put it in earlier (wherein we'd get support for additional checksums landed first)
21:13:58 <timburke_> addressed a bunch of the issues i had with it, and added a fair few new cross-compat tests
21:14:41 <timburke_> there's surely still some things to improve, but I think it's about ready for another round of review
21:16:49 <timburke_> note that i still want to get the additional checksums work merged, too (since it's the one data-integrity backstop we'd still have), but it seems reasonable to start by making sure many people's default client works again
21:17:26 * Callum027 is around if anyone wants to discuss the storage policy in GET account patch btw :)
21:18:04 <timburke_> speaking of the checksum patches...
21:18:08 <timburke_> #topic etag overrides
21:18:58 <timburke_> acoles was recently reviewing them, and realized that our tendency to stuff extra parameters into container listing etags seems brittle
21:19:28 <timburke_> he's got a chain of work to try to make it better by adding helpers; the end is at
21:19:31 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/940165
21:20:32 <timburke_> i took a brief look at that one, but need to spend some more time with the chain
21:21:04 <mattoliver> cool. Al told me we was thinking about helpers and such, I'll take a look
21:21:22 <timburke_> that's about all i've got
21:21:58 <timburke_> i keep meaning to get a release together, but my back's been giving me trouble, so i haven't been able to sit at the computer long enough to do it
21:22:06 <timburke_> #topic open discussion
21:22:18 <timburke_> anything else we should discuss this week?
21:22:26 <mattoliver> timburke_: no stress, remember to rest your back and get it better!
21:22:49 <mattoliver> I'm pulling on the ringv2 chain.
21:23:03 <mattoliver> timburke_: you had some throughts in your latest review of it?
21:23:21 <jianjian> take care, Tim. stand up desk also helps me
21:23:51 <timburke_> oh, yes! thanks for that. i did -- i'm still torn about whether to have the GzipWriter/RingWriter separation like i did...
21:24:06 <mattoliver> I kind of want to get the first few patches ready for landing (if possible). So we can take advantage of wide dev ids
21:25:06 <timburke_> stuffing some version info into the gzip comment field was clever, but i still want to try other approaches
21:25:12 <mattoliver> Yeah fair enough. I figure the important thing at this point is to be happy with the format. We can always merge the writers in the future and change how they work so long as we're happy with the on disk format.
21:25:25 <timburke_> (since i feel like there are other reasons there might be a comment)
21:26:04 <mattoliver> well pythons GzipFile just skips over the comment when reading. So feels like a not every used feature
21:26:14 <mattoliver> *very used feature
21:26:47 <mattoliver> I just happened to see the header while readying the gzip rfc
21:27:33 <timburke_> yeah, just thinking about other gzips that people might have lying around there system -- especially if we're going to try to get something into the general magic db...
21:27:56 <mattoliver> to be honest, the fname (filename) part of the header is also pretty useless for us. because it just grabs the filename when not specified and removes the .gz
21:27:57 <timburke_> not necessarily made by python at all
21:28:12 <mattoliver> yeah, fair call
21:29:01 <mattoliver> having magic print out a comment (rather then just saying there is a comment) feels like a nice magic/file enhancement in my opionon anyway
21:29:39 <mattoliver> But happy to hold off that patch
21:30:25 <timburke_> for anyone else interested
21:30:27 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/834261
21:30:44 <timburke_> is the first, main patch that introduces the new format
21:30:46 <mattoliver> that's more of a, hey I've been trying to use v2 and it would be nice to be able to easily use 'file object.ring.gz' to know what serialization format it is.. so no rush to land that one I guess.
21:31:11 <timburke_> we should probably squash in
21:31:13 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/939681
21:31:59 <timburke_> to ensure that builders always get written with at least 2-byte dev ids (to ease rollbacks in case we decide we don't actually like it)
21:32:18 <mattoliver> yeah, let's do that
21:32:41 <timburke_> and then mattoliver's been playing with libmagic/file detection stuff in
21:32:44 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/940088
21:32:46 <timburke_> and
21:32:54 <timburke_> #link https://review.opendev.org/c/openstack/swift/+/940491
21:34:25 <mattoliver> I think those last 2 are handy, but we can decide to land them later if people are worried. We could at least land the first one, then people can at least use file -z object.ring.gz and now the serialization format (because the ring filename doesn't change).
21:34:26 <timburke_> i know rcmgleite had a few questions/interested, too -- sorry that i hadn't responded to them yet
21:35:04 <timburke_> on optimal sizing -- i feel like we usually see more troubles going too small than too big. like, below 2x replica count disks, your proxies are by default willing to look at every disk in the cluster, which isn't representative of swift behaviors as you scale up
21:36:12 <timburke_> at the upper end: swift was originally designed to operate on (i want to say) 10s of PB clusters, but has seen improvements enough to manage 100s of PB, with billions of objects in a single container (thanks to container sharding)
21:36:26 <timburke_> you'll sometimes see issues in the transition from a modest cluster to a large one, though -- a part power appropriate for a 100 disk cluster (even with an eye toward growth) is a decent bit smaller than what you'd want for a 10k+ disk cluster
21:38:05 <timburke_> on .pendings vs sqlite's WAL -- mattoliver's memory matches my own; there was some kind of limitation that made it not-great for us before, but maybe it's gotten better
21:39:11 <timburke_> i'm guessing it might have to do with the fact that we're always doing DELETEs/INSERTs rather than UPDATEs for the bulk of the data?
21:40:18 <jianjian> ack, as you scale up your cluster and workload, lots of components would be under stress and different performance/latency issues would pop up.
21:41:28 <timburke_> on metadata, yeah, last write wins -- witht he timestamp set by the proxy servicing the request (so making sure your nodes are all roughly in sync is important; don't want anybody minutes off or worse)
21:41:39 <jianjian> a lot of those fixes have landed upstream, give @rcmgleite another motivation to upgrade his cluster to the latest swift
21:41:59 <timburke_> latest swift is best swift!
21:42:27 <jianjian> \o/
21:43:04 <timburke_> i wonder if i dug through https://wiki.openstack.org/wiki/Swift/Etherpads maybe i could find something about the sqlite WAL ....
21:43:49 <mattoliver> maybe, but the orig pending WAL I think predates the etherpads even
21:45:06 <mattoliver> but we have talked about experimenting with more modern sqlite WAL implementations. esp if we can have multiple write locks to sqlite wals (that's what my sharded pending gives us, if we ever decided to land it)
21:46:02 <timburke_> whoa... https://github.com/openstack/swift/commit/a44635ca / https://github.com/openstack/swift/commit/ed69db16
21:46:42 <timburke_> (both of which predate using gerrit for reviews)
21:47:21 <timburke_> all right, i think i'll call it for this week
21:47:33 <timburke_> thank you all for coming, and thank you for working on swift!
21:47:38 <timburke_> #endmeeting