21:00:07 #startmeeting swift 21:00:07 Meeting started Wed Dec 15 21:00:07 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:07 The meeting name has been set to 'swift' 21:00:17 who's here for the swift meeting? 21:01:15 o/ 21:01:16 o/ 21:01:18 o/ 21:02:24 as usual, the agenda's at 21:02:25 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:37 first up 21:02:47 #topic object-updater fairness 21:03:37 you may have noticed a buynch of patches lately fiddling with how the object-updater handles ratelimiting 21:03:49 mornin 21:04:47 Alistair Coles proposed openstack/swift master: updater: per bucket deferral queues https://review.opendev.org/c/openstack/swift/+/821736 21:04:47 there's enough of them floating around now that it can be a little tricky to see what's the main thrust of the effort ;-) 21:05:07 there's one right there ^^ :D 21:05:12 lol 21:06:20 my impression is that that's the end of the main chain, is that right? and we'll likely squash/abandon most of the others? 21:06:46 there are three main ideas: 21:07:20 timburke: correct, that's the end and if we settle on the idea will be squashed with parent and grandparent 21:07:53 https://review.opendev.org/c/openstack/swift/+/820962/ sets a per-container ratelimit on updates -- and when the limit is exceeded, we drop updates to be picked up on the next cycle 21:07:57 mattoliver: this could probably be abandoned (but was very useful!) 21:08:05 https://review.opendev.org/c/openstack/swift/+/821790/2 21:08:19 yeah, kk no problem 21:08:51 I'll abandon that and the other priority queue chain seeing as yours incorparates the best of all the worlds :) 21:09:21 https://review.opendev.org/c/openstack/swift/+/821326/ and https://review.opendev.org/c/openstack/swift/+/821480/ should probably get squashed together -- they let us keep some of that deferred work in our head so we can make effective use of the time that we'd otherwise be sleeping for the sake of the configured interval 21:10:28 and https://review.opendev.org/c/openstack/swift/+/821736/ better prioritizes that deferred work so we don't burn a bunch of CPU shuffling a deque 21:12:00 overall, the expectation is that we'll have two ratelimits for updaters: one that protects us from using up all the iops searching for work to do, and another that protects the container servers from getting overloaded 21:13:25 and you can adjust the interval as something of a slider to decide how much time you want to devote to overloaded containers vs responsive containers 21:14:46 i don't have much to add to all that -- i think we've got plenty of attention on it and it's moving in a good direction. just wanted to keep everyone else in the loop, too :-) 21:15:19 #topic keeping things in memcache 21:16:27 we've seen some bad thundering-herd behaviors when items fall out of memcached -- but just increasing cache time is risky since it increases the likelihood that the cache will be stale 21:17:42 a while back i started playing with an idea to randomly skip going to memcache sometimes, so we'd get a refresh before things fall out 21:17:45 #link https://review.opendev.org/c/openstack/swift/+/736802 21:19:10 makes a bunch of sense, under high load when something expires out of cache the server can get hit quite hard. keeping it hot seems like a win. 21:19:18 I'll review the patch today 21:19:27 incorporating some of the early feedback got a little hairy, though -- needed to start digging through app.app.app... to find config values for every time a middleware wants to call get_container_info 21:19:57 i think i've got a plan, though, and it involves getting rid of pipeline_property :-D 21:20:06 do it! 21:20:06 lol, should we revisit your old less brittle pipeline_property 21:20:22 so mattoliver, i wouldn't worry about reviewing it until i push up a new chain ;-) 21:20:31 kk 21:21:02 on the same vein as this, I've been working on optionally putting error limited into memcache so things can globally share this info 21:22:05 And even playing with the updater sharing the same error limiting info, so if a container server is overloaded and say a proxy knows it, so does others rather then having to each figure it out themselves (by failing to hit it 11 times by default). 21:22:38 So if anyone gets bored: https://review.opendev.org/c/openstack/swift/+/820486 21:23:03 definitely seems interesting 21:23:13 next up 21:23:17 as an FYI, I worked on a different approach to keeping shard ranges in cache here https://review.opendev.org/c/openstack/swift/+/817294, but we had some concerns about the bleeding of concerns from proxy to container server so I am currently hoping timburke 's randomised approach works out 21:23:36 #topic tempurl signature logging 21:23:39 #link https://review.opendev.org/c/openstack/swift/+/817476 21:23:51 has anyone had a chance to revisit this patch? 21:25:03 I reviewed it at some point, but got distracted. I'll loop back and see where it's at today if you like 21:25:19 that'd be great, thanks mattoliver! 21:26:38 #topic recon and servers_per_port 21:26:53 i still need to follow up on https://review.opendev.org/c/openstack/swift/+/818881 21:28:06 but does anyone have concerns about the approach of only querying once per ip? 21:29:01 i could imagine deploying with not all disks available to all ports, but i'm not sure that anyone actually *does* 21:30:41 i guess i won't worry about it then 21:31:02 #topic request tracing 21:31:13 mattoliver, when's that talk again? 21:31:58 Let me find the link :) 21:32:21 #link https://lca2022.linux.org.au/schedule/presentation/32/ 21:32:24 next month! 21:32:41 oh thanks, yeah 15th Jan 21:33:07 Its cause I tweeted about the tracing at the PTG so now I'm turning it into a talk. 21:33:23 need anything to help prep for it? 21:33:34 Plus side is, it's getting me to clean up and fix a bunch. 21:34:23 not at this point. I don't plan on it merging anytime soon. Trying to debug why bulk and slo seem to loose the known spans. 21:35:24 debugging they're there at the start of the middleware, but gone in a function call in the same middleware.. so need to dig into how that can happen. 21:35:46 my best guess would be something with the greenthreadpools -- i seem to remember that being tricky when poking at similar ideas with zipkin 21:35:58 should i keep it on the agenda for next week? 21:36:23 yeah, I've got some interesting scafalding to get around some green pool things. 21:36:39 this seems to be different (but probably isn't once I dig in some more). 21:37:23 all right, that's all i've got 21:37:26 #topic open discussion 21:37:31 was wanting to do a bulk extract as a interesting example in the talk.. but need to get it working first :P 21:37:34 yeah, keep it in the agenda and it'll motivate me to fix it :P 21:37:36 what else should we bring up this week? 21:38:08 Nothing for me. 21:38:54 I'm good 21:39:04 oh -- speaking of next week, should we bother meeting? or should we next meet next year? 21:39:36 oh yeah, next week I'll be driving to my parents (10 hours away). so probably not around 21:39:51 So I'm ok for skipping 21:41:50 i'm voting for next year :-) 21:41:53 +1 or skipping 21:42:07 enjoy the end of the year everyone! 21:42:09 I'm ok 21:42:30 me too :-) 21:42:36 all right -- thank you all for coming, thank you for working on swift, and talk to you next year! 21:42:41 #endmeeting