21:00:16 <timburke> #startmeeting swift
21:00:16 <opendevmeet> Meeting started Wed Jan  5 21:00:16 2022 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:16 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:16 <opendevmeet> The meeting name has been set to 'swift'
21:00:29 <timburke> who's here for the swift meeting?
21:00:49 <kota> hi
21:01:17 <timburke> if memory serves, mattoliver is out this week
21:01:45 <timburke> clayg and acoles may or may not be around
21:02:03 <kota> oic
21:02:14 <clayg> Thanks for the ping!
21:02:33 <timburke> maybe it'll work with zaitcev too!? :P
21:02:49 <timburke> any way, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:00 <timburke> #topic object updater fairness
21:03:47 <timburke> we ran with a bunch of updater improvements over the holidays, and they seem to be working well!
21:04:30 <zaitcev> okay, good. Although your cloud is private, isn't it? So the holiday is actually not the peak load for you guys, right?
21:05:11 <timburke> true -- i'd say we had a lighter-than-usual load over the holiday
21:06:15 <timburke> but still, clearing a billion asyncs is impressive! there was another brief spike of 200-300M asyncs, but my understanding was that we could keep our previous tuning just fine and they cleared within a day or two
21:07:25 <timburke> the skips in https://review.opendev.org/c/openstack/swift/+/820962 seemed to do a good job of protecting the container servers
21:08:36 <timburke> probably still a good idea to address the deterministic-hashing issue that clayg spotted before we land it, though
21:09:06 <timburke> #topic memcache refresh
21:09:20 <clayg> Can someone do that for me?  I’m super duper lazy.
21:09:56 <timburke> clayg, i'll try to get to it
21:10:30 <clayg> Hahaha I didn’t think that would work!
21:11:02 <timburke> stuff falling out of memcache is still hurting us, so i'm interested in getting something ship-able
21:11:31 <timburke> i've got two main efforts: the one for shard ranges is simpler/cheaper: https://review.opendev.org/c/openstack/swift/+/736802
21:12:30 <timburke> the one for container info isn't *so* bad: https://review.opendev.org/c/openstack/swift/+/821921
21:12:50 <timburke> but it's got a dependency on a refactor: https://review.opendev.org/c/openstack/swift/+/821920
21:13:26 <zaitcev> Just think that maybe now we could sustain the load from Gnocchi.
21:13:46 <zaitcev> Except that Gnocchi is fortunately dead because it was killing Ceph just as well as Swift.
21:14:22 <zaitcev> (sorry that was about the updater; will try to keep up)
21:14:58 <timburke> no worries :)
21:16:08 <clayg> timburke: those are great!  I’ll dig in more. Can’t wait to see those new stats in prod.
21:16:48 <timburke> zaitcev, fwiw, you'll definitely want sharded containers ahead of worrying too much about those updater improvements -- and if we can get the set of updating shard ranges to hang around in memcache, the updater improvements may become moot
21:18:50 <timburke> the get_container_info refactor has an interesting side-effect: middleware can no longer interfere with the info cached and returned. i don't think it will impact anything, but seems worth calling out
21:20:27 <timburke> (previously, a left-in-the-pipeline middleware could make a get_container_info call with its local "app" and middleware to the right of it could impact results. with the patch, everybody gets a reference to the actually proxy-server app and uses *that*)
21:22:18 <clayg> Moddleware can manipulate memcaxge directly. It’s a “gotcha” at worst.
21:23:05 <timburke> #topic tempurl signature logging
21:23:19 <timburke> #link https://bugs.launchpad.net/swift/+bug/1685798
21:23:58 <timburke> mattoliver took a stab at an alternate approach in https://review.opendev.org/c/openstack/swift/+/822585
21:24:37 <clayg> Is Matt driving this?
21:24:51 <clayg> I’d love to see it closed.
21:25:03 <timburke> where we'd stick a new key in the request environment and middlewares could update that with whatever headers/params they want to obscure
21:26:29 <timburke> i got worried that you could bounce out (due to ratelimit, say) before we get to tempurl, and so the env wouldn't get updated
21:27:14 <timburke> (and it also bugged me that we had to make the same update *for every request*)
21:27:39 <timburke> so i tried an approach similar to register_swift_info in https://review.opendev.org/c/openstack/swift/+/823432
21:29:05 <clayg> Let’s wait and see what Matt thinks then. It looks like mostly code refactoring/quality of life - not correctness.
21:29:55 <clayg> Should we do the obvious fix and iterate on the technical investment?  I think Matt pointed out correctly: we’ll want this the next time we have to obscure something.
21:30:06 <timburke> sounds good. on the whole, i think i like the approach more than the original patch at https://review.opendev.org/c/openstack/swift/+/817476 -- it became trivial to fix up s3api
21:31:25 <timburke> that's everything i've got
21:31:26 <clayg> OIC, we’ll https://review.opendev.org/c/openstack/swift/+/823432 looks pretty thin.
21:31:39 <timburke> #topic open discussion
21:31:49 <timburke> what else should we talk about this week?
21:32:51 <clayg> Oh, but that just builds on Matt’s. You two crack me up.
21:32:53 <timburke> clayg, yeah, both matt's patch and mine need tests for the new middleware api
21:33:26 <clayg> So… we’ll keep waiting on closing the CVE 🤣
21:34:37 <timburke> it's been 4 years, what's another? :P
21:36:41 <timburke> all right, i think i'll call it then
21:36:52 <timburke> thank you all for coming, and thank you for working on swift!
21:36:56 <timburke> #endmeeting