21:00:16 #startmeeting swift 21:00:16 Meeting started Wed Jan 5 21:00:16 2022 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:16 The meeting name has been set to 'swift' 21:00:29 who's here for the swift meeting? 21:00:49 hi 21:01:17 if memory serves, mattoliver is out this week 21:01:45 clayg and acoles may or may not be around 21:02:03 oic 21:02:14 Thanks for the ping! 21:02:33 maybe it'll work with zaitcev too!? :P 21:02:49 any way, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:03:00 #topic object updater fairness 21:03:47 we ran with a bunch of updater improvements over the holidays, and they seem to be working well! 21:04:30 okay, good. Although your cloud is private, isn't it? So the holiday is actually not the peak load for you guys, right? 21:05:11 true -- i'd say we had a lighter-than-usual load over the holiday 21:06:15 but still, clearing a billion asyncs is impressive! there was another brief spike of 200-300M asyncs, but my understanding was that we could keep our previous tuning just fine and they cleared within a day or two 21:07:25 the skips in https://review.opendev.org/c/openstack/swift/+/820962 seemed to do a good job of protecting the container servers 21:08:36 probably still a good idea to address the deterministic-hashing issue that clayg spotted before we land it, though 21:09:06 #topic memcache refresh 21:09:20 Can someone do that for me? I’m super duper lazy. 21:09:56 clayg, i'll try to get to it 21:10:30 Hahaha I didn’t think that would work! 21:11:02 stuff falling out of memcache is still hurting us, so i'm interested in getting something ship-able 21:11:31 i've got two main efforts: the one for shard ranges is simpler/cheaper: https://review.opendev.org/c/openstack/swift/+/736802 21:12:30 the one for container info isn't *so* bad: https://review.opendev.org/c/openstack/swift/+/821921 21:12:50 but it's got a dependency on a refactor: https://review.opendev.org/c/openstack/swift/+/821920 21:13:26 Just think that maybe now we could sustain the load from Gnocchi. 21:13:46 Except that Gnocchi is fortunately dead because it was killing Ceph just as well as Swift. 21:14:22 (sorry that was about the updater; will try to keep up) 21:14:58 no worries :) 21:16:08 timburke: those are great! I’ll dig in more. Can’t wait to see those new stats in prod. 21:16:48 zaitcev, fwiw, you'll definitely want sharded containers ahead of worrying too much about those updater improvements -- and if we can get the set of updating shard ranges to hang around in memcache, the updater improvements may become moot 21:18:50 the get_container_info refactor has an interesting side-effect: middleware can no longer interfere with the info cached and returned. i don't think it will impact anything, but seems worth calling out 21:20:27 (previously, a left-in-the-pipeline middleware could make a get_container_info call with its local "app" and middleware to the right of it could impact results. with the patch, everybody gets a reference to the actually proxy-server app and uses *that*) 21:22:18 Moddleware can manipulate memcaxge directly. It’s a “gotcha” at worst. 21:23:05 #topic tempurl signature logging 21:23:19 #link https://bugs.launchpad.net/swift/+bug/1685798 21:23:58 mattoliver took a stab at an alternate approach in https://review.opendev.org/c/openstack/swift/+/822585 21:24:37 Is Matt driving this? 21:24:51 I’d love to see it closed. 21:25:03 where we'd stick a new key in the request environment and middlewares could update that with whatever headers/params they want to obscure 21:26:29 i got worried that you could bounce out (due to ratelimit, say) before we get to tempurl, and so the env wouldn't get updated 21:27:14 (and it also bugged me that we had to make the same update *for every request*) 21:27:39 so i tried an approach similar to register_swift_info in https://review.opendev.org/c/openstack/swift/+/823432 21:29:05 Let’s wait and see what Matt thinks then. It looks like mostly code refactoring/quality of life - not correctness. 21:29:55 Should we do the obvious fix and iterate on the technical investment? I think Matt pointed out correctly: we’ll want this the next time we have to obscure something. 21:30:06 sounds good. on the whole, i think i like the approach more than the original patch at https://review.opendev.org/c/openstack/swift/+/817476 -- it became trivial to fix up s3api 21:31:25 that's everything i've got 21:31:26 OIC, we’ll https://review.opendev.org/c/openstack/swift/+/823432 looks pretty thin. 21:31:39 #topic open discussion 21:31:49 what else should we talk about this week? 21:32:51 Oh, but that just builds on Matt’s. You two crack me up. 21:32:53 clayg, yeah, both matt's patch and mine need tests for the new middleware api 21:33:26 So… we’ll keep waiting on closing the CVE 🤣 21:34:37 it's been 4 years, what's another? :P 21:36:41 all right, i think i'll call it then 21:36:52 thank you all for coming, and thank you for working on swift! 21:36:56 #endmeeting