21:00:44 <timburke> #startmeeting swift
21:00:44 <opendevmeet> Meeting started Wed Aug 11 21:00:44 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:44 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:44 <opendevmeet> The meeting name has been set to 'swift'
21:00:54 <timburke> who's here for the swift meeting?
21:00:59 <zaitcev> o7
21:01:03 <kota> good morning
21:01:53 <mattoliver> Morning
21:02:18 <mattoliver> o/
21:02:56 <timburke> as usual, the agenda's on the wiki
21:03:00 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:33 <timburke> first up
21:03:37 <timburke> #topic PTG
21:03:49 <timburke> just a reminder of the call for topics
21:03:52 <timburke> #link https://etherpad.opendev.org/p/swift-ptg-yoga
21:04:07 <timburke> (i need to put stuff on there, too ;-)
21:04:54 <timburke> still a couple months away, though
21:04:59 <timburke> next up
21:05:10 <timburke> #topic misplaced objects in sharded containers
21:06:04 <timburke> acoles and mattoliver have been going back and forth on this a decent bit, but it seems like we're getting a pretty good set of changes together for it
21:06:37 <timburke> thanks for splitting up the changes, mattoliver -- i think it *is* making it easier for me to take in
21:07:17 <mattoliver> Nps 😀
21:07:21 <timburke> my read on it is that there are two main patches and a handful of follow-ons that are likely to get squashed in, is that right?
21:07:25 <timburke> #link https://review.opendev.org/c/openstack/swift/+/803423
21:07:34 <timburke> #link https://review.opendev.org/c/openstack/swift/+/800748
21:08:28 <timburke> the first lets us list objects from the correct policy (from the root's perspective) even when the shards have the wrong one
21:08:40 <zaitcev> That commit message is bad.
21:08:48 <timburke> and the second makes sure the shards eventually get the right policy from the root
21:09:10 <DHE> timburke: 24 hard drives of 8TB each, 192 GB of RAM. had some xfs stack traces in hung tasks. they unfroze, but it lasted way longer than I'm comfortable with..
21:09:15 <DHE> very strange...
21:09:16 <mattoliver> yeah, their the main ones.
21:10:08 <mattoliver> There has been some follow ups to the first one, and discussion that happened while I slept, so will go over them today.
21:10:11 <zaitcev> "There is a tight coupling between a root container and it's shards the shards hold the object metadata for the root container, so are really an extension of the root." - holy cow, mattoliver, what does this even mean.
21:10:29 <mattoliver> lol
21:10:38 <timburke> mattoliver, do you feel like we've got a pretty good idea of where we want things before merging, or is more discussion needed?
21:11:34 <timburke> and yeah, a cleaned up commit message might be on that list of changes-before-merging ;-)
21:11:50 <mattoliver> I think the first (even with all it's follow ons) to pretty straight forward and in a good stable place.
21:12:22 <mattoliver> the 2nd one could have a bit of a going through. Because it actaully does a policy migration.. so justwant to be sure its right :)
21:13:07 <timburke> yeah, i still haven't loaded that one into my head enough to have a good sense of it yet
21:15:32 <timburke> all right -- we'll get some more eyes on the second one and look forward to a new patchset (or two) on the first
21:15:57 <timburke> #topic internal client bottlenecks
21:16:01 <mattoliver> zaitcev: pretty much means the root container has sharded all it's metadata into shards, and invisible to the user, so shards are really just an extension of the root. So if the root is requesting an object in a particular policy then the shards to respond accordingly.
21:16:11 <timburke> #undo
21:16:11 <opendevmeet> Removing item from minutes: #topic internal client bottlenecks
21:16:56 <mattoliver> sorry, I got a little wordy in the commit message, can try and make it more clear if it doesn't make sense.
21:17:02 <mattoliver> timburke: you can keep going :)
21:17:20 <timburke> no worries
21:17:23 <timburke> #topic internal client bottlenecks
21:18:17 <timburke> so we've seen issues with processes using internal-client where it gets CPU-bound because it can only use a single core
21:19:17 <timburke> we saw it in the context of timur's 1space wanting to move a buinch of data around, but i expect it can also happen with the reconciler or container-sync
21:20:45 <timburke> basically, whether you've got 1, 10, or 50 greenthreads going, it doesn't matter much when they're all pulling data out of an EC policy and we go off to C for a while
21:21:52 <timburke> so i had an idea to make an internal-client-alike that would be a drop-in replacement but with make_request calling out to a local-only no-auth proxy
21:22:13 <timburke> it turned out way less work than i expected!
21:22:15 <timburke> #link https://review.opendev.org/c/openstack/swift/+/803536
21:23:36 <timburke> there are some more updates i've got for that locally -- i realized that it'd be pretty self-contained so i might keep the whole thing in a single separate file for now. but i've now got streaming support, for example
21:25:15 <timburke> to support allow_modify_pipeline=False, though, i needed to get some changes in the proxy (or wsgi.py) so we could have a config option to turn it off
21:25:29 <timburke> #link https://review.opendev.org/c/openstack/swift/+/803647
21:26:23 <timburke> clayg isn't quite happy with where that is yet -- but now i'm kind of wondering why we didn't just do that to begin with
21:26:44 <timburke> the kwarg for internal-client feels weird
21:28:14 <timburke> #topic open discussion
21:28:16 <timburke> that's all i had -- what else should we talk about this week?
21:28:44 <clayg> The config option feels like a foot gun. Turn this off if you need it, but seriously don’t ever turn it off. Maybe just a note about “no client can use this proxy” 🤷‍♂️🤔
21:30:23 <timburke> i seem to remember torgomatic sometimes wanting something like it, too, though. yeah, it's a foot-gun, but sometimes i actually *do* want swift to get out of the way and let me make bad decisions
21:31:04 <zaitcev> Maybe start it with underscore or something.
21:32:49 <timburke> clayg, i can definitely make the wording in the sample conf more explicit: "You should never need to turn this off."
21:35:23 <timburke> zaitcev, i took a look at the account-server 500 patch at https://review.opendev.org/c/openstack/swift/+/743797 -- but i'm not sure that the probe test is demonstrating what we wanted
21:35:42 <zaitcev> timburke: I saw your comments and I'm working through them.
21:35:57 <timburke> 👍
21:40:56 <timburke> all right, seems like we can end a little early
21:41:00 <zaitcev> ok
21:41:46 <timburke> fwiw, i just added the meeting times to the ptg etherpad: https://etherpad.opendev.org/p/swift-ptg-yoga
21:41:54 <timburke> sorry it took me a bit to get around to that
21:41:57 <timburke> thank you all for coming, and thank you for working on swift!
21:42:02 <timburke> #endmeeting