Wednesday, 2021-08-11

opendevreviewMatthew Oliver proposed openstack/swift master: container-server: distinguish between obj returned spi vs brokers
opendevreviewAlistair Coles proposed openstack/swift master: Set X-Backend-Storage-Policy in original listing request
claygZaitcev it sounds like you totally understood how the get more nodes factory generator works 😁13:27
zaitcevclayg: On reflection it makes sense not to load the iterator with counting at eay yield.13:28
acolestimburke: ack, I agree14:29
acolestimburke: apologies, I cannot make today's swift meeting (travelling)14:29
DHEis using ext4 still discouraged over xfs? I've been having issues with xfs lately, jamming under memory pressure.16:37
timburkezaitcev, fwiw, i don't think i'd fully appreciated that until i saw clayg and cschwede__'s talk in barcelona:
zaitcevDHE: my official position is to tune VM parameters. Remember that ext4 is not guaranteed to degrade any better under load, in part because jbd is a separate process. It just uses less RAM, but fundamentally it also allocates when working.20:51
timburkeDHE, FWIW i've only ever run xfs in prod. memory is a perennial issue, especially with denser BOMs. i think my homelab at least has one disk using btrfs? but it sees so little use that basically *anything* with support for large enough xattrs would be fine20:58
timburkealmost meeting time!20:58
zaitcevsounds exciting21:00
zaitcevwe can also badmouth Al while he's away21:00
zaitcevI'm joking. He's too perfect for any of that.21:00
timburke#startmeeting swift21:00
opendevmeetMeeting started Wed Aug 11 21:00:44 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
opendevmeetThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
kotagood morning21:01
timburkeas usual, the agenda's on the wiki21:02
timburkefirst up21:03
timburke#topic PTG21:03
timburkejust a reminder of the call for topics21:03
timburke(i need to put stuff on there, too ;-)21:04
timburkestill a couple months away, though21:04
timburkenext up21:04
timburke#topic misplaced objects in sharded containers21:05
timburkeacoles and mattoliver have been going back and forth on this a decent bit, but it seems like we're getting a pretty good set of changes together for it21:06
timburkethanks for splitting up the changes, mattoliver -- i think it *is* making it easier for me to take in21:06
mattoliverNps πŸ˜€21:07
timburkemy read on it is that there are two main patches and a handful of follow-ons that are likely to get squashed in, is that right?21:07
timburkethe first lets us list objects from the correct policy (from the root's perspective) even when the shards have the wrong one21:08
zaitcevThat commit message is bad.21:08
timburkeand the second makes sure the shards eventually get the right policy from the root21:08
DHEtimburke: 24 hard drives of 8TB each, 192 GB of RAM. had some xfs stack traces in hung tasks. they unfroze, but it lasted way longer than I'm comfortable with..21:09
DHEvery strange...21:09
mattoliveryeah, their the main ones. 21:09
mattoliverThere has been some follow ups to the first one, and discussion that happened while I slept, so will go over them today. 21:10
zaitcev"There is a tight coupling between a root container and it's shards the shards hold the object metadata for the root container, so are really an extension of the root." - holy cow, mattoliver, what does this even mean.21:10
timburkemattoliver, do you feel like we've got a pretty good idea of where we want things before merging, or is more discussion needed?21:10
timburkeand yeah, a cleaned up commit message might be on that list of changes-before-merging ;-)21:11
mattoliverI think the first (even with all it's follow ons) to pretty straight forward and in a good stable place. 21:11
mattoliverthe 2nd one could have a bit of a going through. Because it actaully does a policy migration.. so justwant to be sure its right :)21:12
timburkeyeah, i still haven't loaded that one into my head enough to have a good sense of it yet21:13
timburkeall right -- we'll get some more eyes on the second one and look forward to a new patchset (or two) on the first21:15
timburke#topic internal client bottlenecks21:15
mattoliverzaitcev: pretty much means the root container has sharded all it's metadata into shards, and invisible to the user, so shards are really just an extension of the root. So if the root is requesting an object in a particular policy then the shards to respond accordingly. 21:16
opendevmeetRemoving item from minutes: #topic internal client bottlenecks21:16
mattoliversorry, I got a little wordy in the commit message, can try and make it more clear if it doesn't make sense. 21:16
mattolivertimburke: you can keep going :)21:17
timburkeno worries21:17
timburke#topic internal client bottlenecks21:17
timburkeso we've seen issues with processes using internal-client where it gets CPU-bound because it can only use a single core21:18
timburkewe saw it in the context of timur's 1space wanting to move a buinch of data around, but i expect it can also happen with the reconciler or container-sync21:19
timburkebasically, whether you've got 1, 10, or 50 greenthreads going, it doesn't matter much when they're all pulling data out of an EC policy and we go off to C for a while21:20
timburkeso i had an idea to make an internal-client-alike that would be a drop-in replacement but with make_request calling out to a local-only no-auth proxy21:21
timburkeit turned out way less work than i expected!21:22
timburkethere are some more updates i've got for that locally -- i realized that it'd be pretty self-contained so i might keep the whole thing in a single separate file for now. but i've now got streaming support, for example21:23
timburketo support allow_modify_pipeline=False, though, i needed to get some changes in the proxy (or so we could have a config option to turn it off21:25
timburkeclayg isn't quite happy with where that is yet -- but now i'm kind of wondering why we didn't just do that to begin with21:26
timburkethe kwarg for internal-client feels weird21:26
timburke#topic open discussion21:28
timburkethat's all i had -- what else should we talk about this week?21:28
claygThe config option feels like a foot gun. Turn this off if you need it, but seriously don’t ever turn it off. Maybe just a note about β€œno client can use this proxy” πŸ€·β€β™‚οΈπŸ€”21:28
timburkei seem to remember torgomatic sometimes wanting something like it, too, though. yeah, it's a foot-gun, but sometimes i actually *do* want swift to get out of the way and let me make bad decisions21:30
zaitcevMaybe start it with underscore or something.21:31
timburkeclayg, i can definitely make the wording in the sample conf more explicit: "You should never need to turn this off."21:32
timburkezaitcev, i took a look at the account-server 500 patch at -- but i'm not sure that the probe test is demonstrating what we wanted21:35
zaitcevtimburke: I saw your comments and I'm working through them.21:35
timburkeall right, seems like we can end a little early21:40
timburkefwiw, i just added the meeting times to the ptg etherpad:
timburkesorry it took me a bit to get around to that21:41
timburkethank you all for coming, and thank you for working on swift!21:41
opendevmeetMeeting ended Wed Aug 11 21:42:02 2021 UTC.  Information about MeetBot at . (v 0.1.4)21:42
opendevmeetMinutes (text):
zaitcevDHE: the floor is yours although I have no idea how to help with thread freezes.21:42
mattoliveroh cool (re: adding the times to the etherpad)21:42
DHEzaitcev: I don't have much beyond "can I try ext4 instead?"22:55
opendevreviewMatthew Oliver proposed openstack/swift master: container-server: distinguish between obj returned spi vs brokers
mattoliver^ Just making the returning the new header when x-backend-record-type is object. Next I'll squash it in. Probably didn't need to push it, but it means we have the history in gerrit πŸ€·β€β™‚οΈ23:39

Generated by 2.17.2 by Marius Gedminas - find it at!