| opendevreview | Matthew Oliver proposed openstack/swift master: container-server: distinguish between obj returned spi vs brokers https://review.opendev.org/c/openstack/swift/+/804208 | 07:12 |
|---|---|---|
| opendevreview | Alistair Coles proposed openstack/swift master: Set X-Backend-Storage-Policy in original listing request https://review.opendev.org/c/openstack/swift/+/804234 | 13:20 |
| clayg | Zaitcev it sounds like you totally understood how the get more nodes factory generator works 😁 | 13:27 |
| zaitcev | clayg: On reflection it makes sense not to load the iterator with counting at eay yield. | 13:28 |
| acoles | timburke: ack, I agree | 14:29 |
| acoles | timburke: apologies, I cannot make today's swift meeting (travelling) | 14:29 |
| DHE | is using ext4 still discouraged over xfs? I've been having issues with xfs lately, jamming under memory pressure. | 16:37 |
| timburke | zaitcev, fwiw, i don't think i'd fully appreciated that until i saw clayg and cschwede__'s talk in barcelona: https://www.openstack.org/videos/summits/barcelona-2016/forget-everything-you-knew-about-swift-rings-heres-everything-you-need-to-know-about-swift-rings | 20:49 |
| zaitcev | DHE: my official position is to tune VM parameters. Remember that ext4 is not guaranteed to degrade any better under load, in part because jbd is a separate process. It just uses less RAM, but fundamentally it also allocates when working. | 20:51 |
| timburke | DHE, FWIW i've only ever run xfs in prod. memory is a perennial issue, especially with denser BOMs. i think my homelab at least has one disk using btrfs? but it sees so little use that basically *anything* with support for large enough xattrs would be fine | 20:58 |
| timburke | almost meeting time! | 20:58 |
| zaitcev | sounds exciting | 21:00 |
| zaitcev | we can also badmouth Al while he's away | 21:00 |
| zaitcev | I'm joking. He's too perfect for any of that. | 21:00 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Aug 11 21:00:44 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| zaitcev | o7 | 21:00 |
| kota | good morning | 21:01 |
| mattoliver | Morning | 21:01 |
| mattoliver | o/ | 21:02 |
| timburke | as usual, the agenda's on the wiki | 21:02 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:03 |
| timburke | first up | 21:03 |
| timburke | #topic PTG | 21:03 |
| timburke | just a reminder of the call for topics | 21:03 |
| timburke | #link https://etherpad.opendev.org/p/swift-ptg-yoga | 21:03 |
| timburke | (i need to put stuff on there, too ;-) | 21:04 |
| timburke | still a couple months away, though | 21:04 |
| timburke | next up | 21:04 |
| timburke | #topic misplaced objects in sharded containers | 21:05 |
| timburke | acoles and mattoliver have been going back and forth on this a decent bit, but it seems like we're getting a pretty good set of changes together for it | 21:06 |
| timburke | thanks for splitting up the changes, mattoliver -- i think it *is* making it easier for me to take in | 21:06 |
| mattoliver | Nps 😀 | 21:07 |
| timburke | my read on it is that there are two main patches and a handful of follow-ons that are likely to get squashed in, is that right? | 21:07 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/803423 | 21:07 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/800748 | 21:07 |
| timburke | the first lets us list objects from the correct policy (from the root's perspective) even when the shards have the wrong one | 21:08 |
| zaitcev | That commit message is bad. | 21:08 |
| timburke | and the second makes sure the shards eventually get the right policy from the root | 21:08 |
| DHE | timburke: 24 hard drives of 8TB each, 192 GB of RAM. had some xfs stack traces in hung tasks. they unfroze, but it lasted way longer than I'm comfortable with.. | 21:09 |
| DHE | very strange... | 21:09 |
| mattoliver | yeah, their the main ones. | 21:09 |
| mattoliver | There has been some follow ups to the first one, and discussion that happened while I slept, so will go over them today. | 21:10 |
| zaitcev | "There is a tight coupling between a root container and it's shards the shards hold the object metadata for the root container, so are really an extension of the root." - holy cow, mattoliver, what does this even mean. | 21:10 |
| mattoliver | lol | 21:10 |
| timburke | mattoliver, do you feel like we've got a pretty good idea of where we want things before merging, or is more discussion needed? | 21:10 |
| timburke | and yeah, a cleaned up commit message might be on that list of changes-before-merging ;-) | 21:11 |
| mattoliver | I think the first (even with all it's follow ons) to pretty straight forward and in a good stable place. | 21:11 |
| mattoliver | the 2nd one could have a bit of a going through. Because it actaully does a policy migration.. so justwant to be sure its right :) | 21:12 |
| timburke | yeah, i still haven't loaded that one into my head enough to have a good sense of it yet | 21:13 |
| timburke | all right -- we'll get some more eyes on the second one and look forward to a new patchset (or two) on the first | 21:15 |
| timburke | #topic internal client bottlenecks | 21:15 |
| mattoliver | zaitcev: pretty much means the root container has sharded all it's metadata into shards, and invisible to the user, so shards are really just an extension of the root. So if the root is requesting an object in a particular policy then the shards to respond accordingly. | 21:16 |
| timburke | #undo | 21:16 |
| opendevmeet | Removing item from minutes: #topic internal client bottlenecks | 21:16 |
| mattoliver | sorry, I got a little wordy in the commit message, can try and make it more clear if it doesn't make sense. | 21:16 |
| mattoliver | timburke: you can keep going :) | 21:17 |
| timburke | no worries | 21:17 |
| timburke | #topic internal client bottlenecks | 21:17 |
| timburke | so we've seen issues with processes using internal-client where it gets CPU-bound because it can only use a single core | 21:18 |
| timburke | we saw it in the context of timur's 1space wanting to move a buinch of data around, but i expect it can also happen with the reconciler or container-sync | 21:19 |
| timburke | basically, whether you've got 1, 10, or 50 greenthreads going, it doesn't matter much when they're all pulling data out of an EC policy and we go off to C for a while | 21:20 |
| timburke | so i had an idea to make an internal-client-alike that would be a drop-in replacement but with make_request calling out to a local-only no-auth proxy | 21:21 |
| timburke | it turned out way less work than i expected! | 21:22 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/803536 | 21:22 |
| timburke | there are some more updates i've got for that locally -- i realized that it'd be pretty self-contained so i might keep the whole thing in a single separate file for now. but i've now got streaming support, for example | 21:23 |
| timburke | to support allow_modify_pipeline=False, though, i needed to get some changes in the proxy (or wsgi.py) so we could have a config option to turn it off | 21:25 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/803647 | 21:25 |
| timburke | clayg isn't quite happy with where that is yet -- but now i'm kind of wondering why we didn't just do that to begin with | 21:26 |
| timburke | the kwarg for internal-client feels weird | 21:26 |
| timburke | #topic open discussion | 21:28 |
| timburke | that's all i had -- what else should we talk about this week? | 21:28 |
| clayg | The config option feels like a foot gun. Turn this off if you need it, but seriously don’t ever turn it off. Maybe just a note about “no client can use this proxy” 🤷♂️🤔 | 21:28 |
| timburke | i seem to remember torgomatic sometimes wanting something like it, too, though. yeah, it's a foot-gun, but sometimes i actually *do* want swift to get out of the way and let me make bad decisions | 21:30 |
| zaitcev | Maybe start it with underscore or something. | 21:31 |
| timburke | clayg, i can definitely make the wording in the sample conf more explicit: "You should never need to turn this off." | 21:32 |
| timburke | zaitcev, i took a look at the account-server 500 patch at https://review.opendev.org/c/openstack/swift/+/743797 -- but i'm not sure that the probe test is demonstrating what we wanted | 21:35 |
| zaitcev | timburke: I saw your comments and I'm working through them. | 21:35 |
| timburke | 👍 | 21:35 |
| timburke | all right, seems like we can end a little early | 21:40 |
| zaitcev | ok | 21:41 |
| timburke | fwiw, i just added the meeting times to the ptg etherpad: https://etherpad.opendev.org/p/swift-ptg-yoga | 21:41 |
| timburke | sorry it took me a bit to get around to that | 21:41 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:41 |
| timburke | #endmeeting | 21:42 |
| opendevmeet | Meeting ended Wed Aug 11 21:42:02 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:42 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2021/swift.2021-08-11-21.00.html | 21:42 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2021/swift.2021-08-11-21.00.txt | 21:42 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2021/swift.2021-08-11-21.00.log.html | 21:42 |
| zaitcev | DHE: the floor is yours although I have no idea how to help with thread freezes. | 21:42 |
| mattoliver | oh cool (re: adding the times to the etherpad) | 21:42 |
| DHE | zaitcev: I don't have much beyond "can I try ext4 instead?" | 22:55 |
| opendevreview | Matthew Oliver proposed openstack/swift master: container-server: distinguish between obj returned spi vs brokers https://review.opendev.org/c/openstack/swift/+/804208 | 23:37 |
| mattoliver | ^ Just making the returning the new header when x-backend-record-type is object. Next I'll squash it in. Probably didn't need to push it, but it means we have the history in gerrit 🤷♂️ | 23:39 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!