21:00:02 <timburke> #startmeeting swift
21:00:02 <opendevmeet> Meeting started Wed Mar 23 21:00:02 2022 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:02 <opendevmeet> The meeting name has been set to 'swift'
21:00:09 <timburke> who's here for the swift meeting?
21:00:49 <kota> o/
21:00:52 <mattoliver> o/
21:02:24 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:01 <timburke> not a whole lot for this week (which is good; i've got another meeting midway through this one :P)
21:03:08 <timburke> #topic PTG
21:03:20 <timburke> just another reminder :-)
21:03:29 <timburke> #link https://etherpad.opendev.org/p/swift-ptg-zed
21:03:43 <timburke> has development topics
21:03:52 <timburke> and if you haven't already, register at
21:03:58 <timburke> #link https://openinfra-ptg.eventbrite.com/
21:04:31 <timburke> meeting times and rooms are on the etherpad now, too
21:04:46 <mattoliver> I've convinced csmart to come and had a chat with some of our SREs and recommended they look at adding any topics they're interested in (and turn up for them).
21:05:36 <timburke> sorry again about the times kota and mattoliver -- i figure we ought to try to front-load the time slot so you guys can get to bed and not worry about missing much
21:05:50 <kota> no problem
21:06:06 <timburke> but i also didn't want to invade your saturday
21:06:13 <mattoliver> yeah no probs, I'll have an excuse for asking silly questions :P
21:06:33 <timburke> #topic py2 support in swiftclient
21:06:37 <timburke> it's gone!
21:07:23 <timburke> someone's apparently been champing at the bit to clean up the cruft from a mixed py2/py3 codebase, too, so there's been a bunch of patches landing recently
21:08:23 <mattoliver> yeah, that's been awesome
21:08:56 <timburke> if anyone wants to look at some more of that sort of thing, i pushed up https://review.opendev.org/c/openstack/python-swiftclient/+/834751 to address some things i noticed while reviewing the other patches
21:09:04 <mattoliver> I have to learn to stop leaning on six as a python crutch now :)
21:10:07 <timburke> i also rebased a few old patches i had -- my hope is that dropping py2 support will make us a little more inclined to show the client some love
21:10:21 <mattoliver> Also meant I had to rebase https://review.opendev.org/c/openstack/python-swiftclient/+/833954 already, but that's a good thing, actaully makes it cleaner, no more getattr.
21:10:30 <timburke> \o/
21:11:42 <timburke> server side, i expect we'll want to wait at least a few months before doing a py2 cleanup -- all that churn will make it really difficult to cherry-pick fixes to the stable/yoga branch if anything comes up
21:12:13 <timburke> #topic ring v2
21:13:04 <timburke> just a status update: i started tinkering with a new approach that brings in the index idea from last ptg
21:13:14 <timburke> #link https://review.opendev.org/c/openstack/swift/+/834261
21:13:26 <mattoliver> It's an awesome implementation Tim. So easy to use.
21:14:02 <mattoliver> I did a quick follow up that adding a few extra features, helper to write the version string/magic
21:14:06 <timburke> mattoliver's done a great job building on top of it, too -- we'll keep working on making that chain great in time to talk about it at *this* ptg :-)
21:14:35 <mattoliver> I've got a new last_part_table that stores index as well
21:15:16 <timburke> that's the important thing to keep in mind: what we *really want* is a way to track some ring history so we can make replication/reconstruction smarter
21:15:23 <mattoliver> and almost completed a new version of the reconstructor that uses it. And does so much earlier so we don't waste ssync connections. Hopefully get the first version of that up today.
21:15:41 <mattoliver> yup +100
21:16:28 <mattoliver> Because I have the index at hand, I only need to check if one of the neighbours/partners happen to also be the same index.
21:16:55 <mattoliver> and do things only then.
21:17:17 <mattoliver> So far it's coming out much nicer (famous last words) :P
21:18:08 <timburke> there's definitely some sneaky bits to the new format -- i do a bunch of Z_FULL_FLUSH flushes as we're writing so we can come back and just start reading, and there's this one point where i swap out the decompressor for the GzipFile so i can force it to be uncompressed so i know exactly how many bytes i'm going to write out...
21:18:35 <timburke> but on the whole, i think i drew sane boundaries around it so we mostly don't have to think about it :-)
21:18:42 <timburke> (speaking of famous last words...)
21:18:57 <mattoliver> lol
21:19:28 <mattoliver> well just so see the (de)serialize_v2 in the follow up. I think it shows it's a winner :)
21:19:34 <mattoliver> *go
21:20:19 <timburke> yeah, i was about to say -- if anyone wants to start loading it into their head, that patch is a great place to start so you can see how the RingWriter api is actually used
21:20:51 <timburke> #topic storage server ratelimiting
21:22:32 <timburke> we've been seeing issues where servers get overloaded (often trying to get one particular object) and it starts causing broader performance degradation
21:23:36 <timburke> clayg's been playing with getting some ratelimiting in there so we can just 503 before hitting disk
21:24:28 <opendevreview> Matthew Oliver proposed openstack/swift master: POC: Random hack to add some kind of disk stat based ratelimiting  https://review.opendev.org/c/openstack/swift/+/834099
21:24:30 <timburke> the idea being that we want to still have the iops to be able to serve the *other* bytes that are on this poor disk
21:25:01 <mattoliver> sorry, that got me thinking about that patch I was playing with, that I still had stuck on my laptop.
21:25:21 <timburke> and mattoliver's been looking at how to make it so ops don't need to pick some number out of the air, but make it a little more adaptive
21:27:00 <timburke> this reminds me of issues we've had with overloaded container-servers, too -- i think the main take-away is that we need to (and are working towards) have backend servers be more defensive and shed load
21:27:16 <mattoliver> Yeah I should blow some dust of the adaptive code.
21:27:34 <timburke> that's all i've got
21:27:39 <timburke> #topic open discussion
21:27:42 <mattoliver> speaking of container servers I'd love to have someone take a look at: https://review.opendev.org/c/openstack/swift/+/830551
21:28:14 <mattoliver> It seems to be a big improvement to concurrent access to the same database.
21:28:24 <timburke> yeah! i can try to take a look this week
21:28:27 <mattoliver> at least in my vsaio.
21:28:30 <mattoliver> thanks.
21:28:37 <mattoliver> Well I have another meeting to get t
21:28:38 <mattoliver> to
21:28:48 <mattoliver> So that's all I have.. for now :P
21:29:12 <kota> no topics from me, thx timburke
21:29:31 <timburke> all right
21:29:41 <timburke> thank you all for coming, and thank you for working on swift!
21:29:45 <timburke> #endtopic
21:30:12 <timburke> #endmeeting