| opendevreview | Tim Burke proposed openstack/swift master: tests: Fix flaky reconciler test https://review.opendev.org/c/openstack/swift/+/940755 | 00:12 |
|---|---|---|
| opendevreview | Nathan Nguyen proposed openstack/swift master: Implement heartbeat response for COPY request https://review.opendev.org/c/openstack/swift/+/940762 | 04:40 |
| mattoliver | The .pending files is our attempt at a better wal for sqlite, as when we first started using sqlite they're version wasn't great. It apparenlty has better WAL support now, so maybe an interesting research spike. But I've also worked on scaling our side (which jian mentioned). | 06:44 |
| mattoliver | timburke_: I may have to take kiddos to school in the morning as the wife may need to go out in the morning.. in case I'm a bit spotty or missing parts of the swift meeting tomorrow. | 06:45 |
| opendevreview | Yan Xiao proposed openstack/swift master: stats: Add path_parts environ for s3api https://review.opendev.org/c/openstack/swift/+/940791 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift master: Apply etag override header params to footer https://review.opendev.org/c/openstack/swift/+/940165 | 15:40 |
| opendevreview | Alistair Coles proposed openstack/swift master: Add etag override header helpers https://review.opendev.org/c/openstack/swift/+/940803 | 15:40 |
| jianjian | @rcmgleite - here are the main patches we have done to improve container performance, you can find our analysis in the commit messages. as a result of those patches, various container server latencies (HEAD, GET, PUT, avg or P99) got optimized by 3X to 100X. https://review.opendev.org/c/openstack/swift/+/888361, https://review.opendev.org/c/openstack/swift/+/888310, https://review.opendev.org/c/openstack/swift/+/888575, | 18:08 |
| jianjian | https://review.opendev.org/c/openstack/swift/+/890470 | 18:08 |
| opendevreview | Callum Dickinson proposed openstack/swift master: Add per-container storage policy to account listing https://review.opendev.org/c/openstack/swift/+/940601 | 19:35 |
| opendevreview | Clay Gerrard proposed openstack/swift master: sq: square up inconsistent timestamp "normalization" https://review.opendev.org/c/openstack/swift/+/940830 | 19:52 |
| opendevreview | Tim Burke proposed openstack/swift master: py3: Fix unicode-header-name handling https://review.opendev.org/c/openstack/swift/+/938631 | 20:51 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Fix flaky reconciler test https://review.opendev.org/c/openstack/swift/+/940755 | 20:53 |
| timburke_ | oh, right! | 21:01 |
| timburke_ | #startmeeting swift | 21:01 |
| opendevmeet | Meeting started Wed Feb 5 21:01:18 2025 UTC and is due to finish in 60 minutes. The chair is timburke_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| opendevmeet | The meeting name has been set to 'swift' | 21:01 |
| timburke_ | who's here for the swift meeting? | 21:01 |
| kota | hello | 21:01 |
| jianjian | o/ | 21:01 |
| mattoliver | o/ (but might be a little distracted this morning) | 21:02 |
| timburke_ | i actually remembered to update the agenda! | 21:03 |
| timburke_ | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:03 |
| timburke_ | first up | 21:03 |
| timburke_ | #topic storage policies in account listings | 21:03 |
| timburke_ | so Callum027 wrote up a bug & patch | 21:04 |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/2097074 | 21:04 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/940601 | 21:04 |
| timburke_ | i had a chance to take a look today, and it looks really promising | 21:05 |
| timburke_ | we've been tracking policy info in the account's container table since we introduced storage policies way back in https://github.com/openstack/swift/commit/00a162c4 | 21:06 |
| timburke_ | i can't think of why any reason why we wouldn't have started returning that info in listings :-/ | 21:06 |
| timburke_ | if anyone remembers why we didn't do that (or has any other concerns about it) please check out the patch! | 21:07 |
| jianjian | yes, that'll be helpful info to return | 21:07 |
| mattoliver | what we have, I never noticed that before (always tracking sp in account db) | 21:08 |
| timburke_ | i feel like our SRE tooling currently periodically does a bunch of HEADs against every container in the cluster; i bet it'd be way better if we could switch that to just looking at account listings | 21:09 |
| timburke_ | next up | 21:09 |
| timburke_ | #topic new boto3 and aws-chunked | 21:09 |
| timburke_ | so new boto3 (/s3transfer/awscli/whatever) started using a chunked protocol by default for uploads | 21:10 |
| timburke_ | see https://github.com/boto/boto3/issues/4392 for a little more context | 21:11 |
| timburke_ | but it gave us some renewed interest in getting support for aws-chunked into s3api | 21:12 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/836755 | 21:12 |
| timburke_ | i pulled that patch out of the chain i'd put it in earlier (wherein we'd get support for additional checksums landed first) | 21:13 |
| timburke_ | addressed a bunch of the issues i had with it, and added a fair few new cross-compat tests | 21:13 |
| timburke_ | there's surely still some things to improve, but I think it's about ready for another round of review | 21:14 |
| timburke_ | note that i still want to get the additional checksums work merged, too (since it's the one data-integrity backstop we'd still have), but it seems reasonable to start by making sure many people's default client works again | 21:16 |
| * Callum027 is around if anyone wants to discuss the storage policy in GET account patch btw :) | 21:17 | |
| timburke_ | speaking of the checksum patches... | 21:18 |
| timburke_ | #topic etag overrides | 21:18 |
| timburke_ | acoles was recently reviewing them, and realized that our tendency to stuff extra parameters into container listing etags seems brittle | 21:18 |
| timburke_ | he's got a chain of work to try to make it better by adding helpers; the end is at | 21:19 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/940165 | 21:19 |
| timburke_ | i took a brief look at that one, but need to spend some more time with the chain | 21:20 |
| mattoliver | cool. Al told me we was thinking about helpers and such, I'll take a look | 21:21 |
| timburke_ | that's about all i've got | 21:21 |
| timburke_ | i keep meaning to get a release together, but my back's been giving me trouble, so i haven't been able to sit at the computer long enough to do it | 21:21 |
| timburke_ | #topic open discussion | 21:22 |
| timburke_ | anything else we should discuss this week? | 21:22 |
| mattoliver | timburke_: no stress, remember to rest your back and get it better! | 21:22 |
| mattoliver | I'm pulling on the ringv2 chain. | 21:22 |
| mattoliver | timburke_: you had some throughts in your latest review of it? | 21:23 |
| jianjian | take care, Tim. stand up desk also helps me | 21:23 |
| timburke_ | oh, yes! thanks for that. i did -- i'm still torn about whether to have the GzipWriter/RingWriter separation like i did... | 21:23 |
| mattoliver | I kind of want to get the first few patches ready for landing (if possible). So we can take advantage of wide dev ids | 21:24 |
| timburke_ | stuffing some version info into the gzip comment field was clever, but i still want to try other approaches | 21:25 |
| mattoliver | Yeah fair enough. I figure the important thing at this point is to be happy with the format. We can always merge the writers in the future and change how they work so long as we're happy with the on disk format. | 21:25 |
| timburke_ | (since i feel like there are other reasons there might be a comment) | 21:25 |
| mattoliver | well pythons GzipFile just skips over the comment when reading. So feels like a not every used feature | 21:26 |
| mattoliver | *very used feature | 21:26 |
| mattoliver | I just happened to see the header while readying the gzip rfc | 21:26 |
| timburke_ | yeah, just thinking about other gzips that people might have lying around there system -- especially if we're going to try to get something into the general magic db... | 21:27 |
| mattoliver | to be honest, the fname (filename) part of the header is also pretty useless for us. because it just grabs the filename when not specified and removes the .gz | 21:27 |
| timburke_ | not necessarily made by python at all | 21:27 |
| mattoliver | yeah, fair call | 21:28 |
| mattoliver | having magic print out a comment (rather then just saying there is a comment) feels like a nice magic/file enhancement in my opionon anyway | 21:29 |
| mattoliver | But happy to hold off that patch | 21:29 |
| timburke_ | for anyone else interested | 21:30 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/834261 | 21:30 |
| timburke_ | is the first, main patch that introduces the new format | 21:30 |
| mattoliver | that's more of a, hey I've been trying to use v2 and it would be nice to be able to easily use 'file object.ring.gz' to know what serialization format it is.. so no rush to land that one I guess. | 21:30 |
| timburke_ | we should probably squash in | 21:31 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/939681 | 21:31 |
| timburke_ | to ensure that builders always get written with at least 2-byte dev ids (to ease rollbacks in case we decide we don't actually like it) | 21:31 |
| mattoliver | yeah, let's do that | 21:32 |
| timburke_ | and then mattoliver's been playing with libmagic/file detection stuff in | 21:32 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/940088 | 21:32 |
| timburke_ | and | 21:32 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/940491 | 21:32 |
| mattoliver | I think those last 2 are handy, but we can decide to land them later if people are worried. We could at least land the first one, then people can at least use file -z object.ring.gz and now the serialization format (because the ring filename doesn't change). | 21:34 |
| timburke_ | i know rcmgleite had a few questions/interested, too -- sorry that i hadn't responded to them yet | 21:34 |
| timburke_ | on optimal sizing -- i feel like we usually see more troubles going too small than too big. like, below 2x replica count disks, your proxies are by default willing to look at every disk in the cluster, which isn't representative of swift behaviors as you scale up | 21:35 |
| timburke_ | at the upper end: swift was originally designed to operate on (i want to say) 10s of PB clusters, but has seen improvements enough to manage 100s of PB, with billions of objects in a single container (thanks to container sharding) | 21:36 |
| timburke_ | you'll sometimes see issues in the transition from a modest cluster to a large one, though -- a part power appropriate for a 100 disk cluster (even with an eye toward growth) is a decent bit smaller than what you'd want for a 10k+ disk cluster | 21:36 |
| timburke_ | on .pendings vs sqlite's WAL -- mattoliver's memory matches my own; there was some kind of limitation that made it not-great for us before, but maybe it's gotten better | 21:38 |
| timburke_ | i'm guessing it might have to do with the fact that we're always doing DELETEs/INSERTs rather than UPDATEs for the bulk of the data? | 21:39 |
| jianjian | ack, as you scale up your cluster and workload, lots of components would be under stress and different performance/latency issues would pop up. | 21:40 |
| timburke_ | on metadata, yeah, last write wins -- witht he timestamp set by the proxy servicing the request (so making sure your nodes are all roughly in sync is important; don't want anybody minutes off or worse) | 21:41 |
| jianjian | a lot of those fixes have landed upstream, give @rcmgleite another motivation to upgrade his cluster to the latest swift | 21:41 |
| timburke_ | latest swift is best swift! | 21:41 |
| jianjian | \o/ | 21:42 |
| timburke_ | i wonder if i dug through https://wiki.openstack.org/wiki/Swift/Etherpads maybe i could find something about the sqlite WAL .... | 21:43 |
| mattoliver | maybe, but the orig pending WAL I think predates the etherpads even | 21:43 |
| mattoliver | but we have talked about experimenting with more modern sqlite WAL implementations. esp if we can have multiple write locks to sqlite wals (that's what my sharded pending gives us, if we ever decided to land it) | 21:45 |
| timburke_ | whoa... https://github.com/openstack/swift/commit/a44635ca / https://github.com/openstack/swift/commit/ed69db16 | 21:46 |
| timburke_ | (both of which predate using gerrit for reviews) | 21:46 |
| timburke_ | all right, i think i'll call it for this week | 21:47 |
| timburke_ | thank you all for coming, and thank you for working on swift! | 21:47 |
| timburke_ | #endmeeting | 21:47 |
| opendevmeet | Meeting ended Wed Feb 5 21:47:38 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:47 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2025/swift.2025-02-05-21.01.html | 21:47 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2025/swift.2025-02-05-21.01.txt | 21:47 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2025/swift.2025-02-05-21.01.log.html | 21:47 |
| opendevreview | Callum Dickinson proposed openstack/swift master: Add per-container storage policy to account listing https://review.opendev.org/c/openstack/swift/+/940601 | 22:09 |
| opendevreview | Clay Gerrard proposed openstack/swift master: sq: what is a lower normalized meta key anyway? https://review.opendev.org/c/openstack/swift/+/940840 | 22:13 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!