21:00:34 <timburke> #startmeeting swift 21:00:34 <opendevmeet> Meeting started Wed Aug 28 21:00:34 2024 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:34 <opendevmeet> The meeting name has been set to 'swift' 21:00:43 <timburke> who's here for the swift meeting? 21:01:08 <mattoliver> o/ 21:02:13 <timburke> i guess it's just us for now then :-) 21:02:37 <timburke> thanks for covering for me while i was still recovering last week mattoliver 21:02:51 <timburke> sorry for having missed the meeting without warning 21:03:09 <mattoliver> Nps man! 21:03:12 <fulecorafa> o/ 21:04:05 <timburke> unfortunately, i still haven't updated the agenda 21:04:25 <zaitcev> aww. I was just going to check if InternalClient work was on it. 21:04:44 <timburke> zaitcev, what kind of internal client work? 21:06:17 <zaitcev> IIRC Matt wanted to move either reaper or .. something to it, from direct client. 21:06:43 <zaitcev> I hoped he'd do the same thing to the dark data watcher. 21:07:32 <timburke> oh, yeah! in relation to 21:07:34 <timburke> #link https://bugs.launchpad.net/swift/+bug/2070397 21:07:34 <patch-bot> Bug #2070397 - Account reaper do not reap sharded containers (In Progress) 21:07:36 <timburke> and 21:07:41 <timburke> #link https://review.opendev.org/c/openstack/swift/+/924034 21:07:42 <patch-bot> patch 924034 - swift - WIP: reaper: Support reaping sharded containers - 1 patch set 21:08:01 <zaitcev> Another suspicious place is the reconciler 21:08:26 <mattoliver> oh yeah, I did a patch, I probably should test it out some more.. and write tests. 21:08:43 <timburke> yeah, i think we still need to find the bandwidth to focus on it :-( 21:09:10 <timburke> i think reconciler already uses IC though: https://github.com/openstack/swift/blob/master/swift/container/reconciler.py#L378 21:09:49 <mattoliver> obviously got distracted with some other sharding internal issue that I'm now working on a new version of CLEAVING and state_changed_at patches to help solve.. but I'll defintely loop back to it. I'll try push it along this week 21:10:03 <timburke> zaitcev, or were you more worried about policy for the root vs policy for the shards? 21:10:35 <timburke> i think mattoliver also has patches kicking around related to that... we should dust them off, too 21:11:01 <mattoliver> lol, I seem to start alot of things and get distracted :( 21:11:16 <timburke> don't we all ;-) 21:12:07 <mattoliver> maybe I need to one say take a week and just focus on finishing internal clienting all the things :P (one day) 21:12:52 <mattoliver> if there is a bug or something that your hitting zaitcev I'd happily start prioritsing it for you 21:13:12 <timburke> speaking of dusting things off -- we're getting towards the end of the cycle, so i've been thinking about getting a release out, and refreshing a few patches of my own in hopes that we might land some of them 21:13:24 <zaitcev> mattoliver: sorry for the noise, I 21:13:51 <timburke> there were a couple old drop-translations patches i hadn't pushed on in a while 21:13:54 <timburke> #link https://review.opendev.org/c/openstack/swift/+/823423 21:13:54 <patch-bot> patch 823423 - swift - common: Stop translating log messages - 2 patch sets 21:14:00 <timburke> #link https://review.opendev.org/c/openstack/swift/+/823424 21:14:01 <patch-bot> patch 823424 - swift - common: Stop translating a bunch of printed messag... - 2 patch sets 21:14:12 <zaitcev> mattoliver: I just saw your work and it seemed familiar to what the watcher has to deal with. 21:15:01 <mattoliver> zaitcev: never need to be sorry. Reminding me about them is great, things can get quite chaotic at nvidia and things fall out of my head :) Your right we had the similar issues with the dark data watcher so it could use it too! 21:16:07 <mattoliver> Maybe I'll start a doc for the ideas section where we can track ic daemon improvements. And I can bring it up as a topic next PTG 21:16:38 <timburke> 👍 21:16:47 <mattoliver> timburke: cool, did I also see you getting the deprecate py2 patch ready? 21:16:55 <mattoliver> just. in. case? 21:17:51 <timburke> you did :-) i need to check in with a couple people about where we are on running py3 everywhere still, though... 21:18:06 <timburke> there was one other one i'd kind of like us to make a decision about before this next release 21:18:08 <timburke> #link https://review.opendev.org/c/openstack/swift/+/924795 21:18:09 <patch-bot> patch 924795 - swift - Remove legacy bin/ scripts - 3 patch sets 21:18:35 <mattoliver> whats the +1 to -1 ratiio? who wins? 21:18:41 <timburke> mattoliver and i are both +1 on dropping them now; packagers shouldn't need them any more 21:19:22 <timburke> but no one else has left a vote. i guess that means we can just do as we like? :P 21:19:48 <mattoliver> well it has been a few weeks :) 21:19:49 <timburke> at any rate, i wanted at least one more opportunity for anyone who might like to raise an objection to do so 21:20:41 <timburke> lacking any other feedback, i'll merge it as i'm putting together release notes 21:21:13 <mattoliver> sounds good 21:21:18 <mattoliver> Speaking of release, I've tricked csmart into finishing the account qouta follow up patch Qatar started, and will make sure it's ready before the release. 21:21:31 <fulecorafa> 👍🏻 21:21:32 <mattoliver> I'll poke him about it today :) 21:21:39 <timburke> \o/ thanks mattoliver and csmart! 21:22:10 <mattoliver> timburke: have you loaded up the priority reviews page for this release? 21:22:30 <timburke> i was just about to say i need to do that :D 21:23:05 <mattoliver> kk, I'll take a look at existing patches too 21:23:16 <timburke> other interesting bodies of work lately: 21:23:41 <timburke> we're getting really close on splitting out the stats client from our adapted loggers 21:23:43 <timburke> #link https://review.opendev.org/c/openstack/swift/+/915483 21:23:44 <patch-bot> patch 915483 - swift - Split statsd client from logger - 29 patch sets 21:24:42 <timburke> there will still be some chimera object returned by utils.get_logger, but there won't be any reference to statsd anymore in utils/logs.py 21:25:24 <mattoliver> kk, nice 21:26:08 <timburke> we also saw a timestamp collision in prod! of course, the way you notice such a thing is when you get an EC object that can't be decoded :-( 21:26:35 <mattoliver> woah, i missed that 21:26:39 <timburke> on our 8+4 policy, we had a 7/5 split for which version was written down 21:27:15 <timburke> it feels a lot like https://bugs.launchpad.net/swift/+bug/1910804 but subtly different since it was actually external-client-driven 21:27:16 <patch-bot> Bug #1910804 - Encryption doesn't play well with processes that copy cleartext data while preserving timestamps (Fix Released) 21:29:06 <timburke> fwiw, the two clients seemed to have written the same data (so the X-Object-Sysmeta-Crypto-Etag-Mac values all matched) but that still doesn't really help much :-( 21:29:44 <timburke> but that's why clayg has been throwing out some patches lately 21:29:58 <timburke> #link https://review.opendev.org/c/openstack/swift/+/927327 21:29:59 <patch-bot> patch 927327 - swift - wip: testing timestamp collison - 1 patch set 21:30:09 <timburke> #link https://review.opendev.org/c/openstack/swift/+/927329 21:30:09 <patch-bot> patch 927329 - swift - wip: detect link at collision - 1 patch set 21:30:19 <timburke> #link https://review.opendev.org/c/openstack/swift/+/927330 21:30:19 <patch-bot> patch 927330 - swift - wip: check etag in commit - 1 patch set 21:30:29 <timburke> #link https://review.opendev.org/c/openstack/swift/+/927328 21:30:29 <patch-bot> patch 927328 - swift - wip: detect ec encyption collision - 1 patch set 21:31:15 <mattoliver> he has been busy, and makes alot of sense. don't know how I missed that. kinda exciting (not for sre, but from a dev POV) :) 21:35:01 <timburke> and last, mattoliver has been busy trying to smooth out some rough edges in the sharding state machine 21:35:13 <timburke> #link https://review.opendev.org/c/openstack/swift/+/789651 21:35:13 <patch-bot> patch 789651 - swift - shrinking: Add a CLEAVING state - 13 patch sets 21:35:28 <timburke> #link https://review.opendev.org/c/openstack/swift/+/926036 21:35:28 <patch-bot> patch 926036 - swift - ShardRange: track last state change timestamp - 6 patch sets 21:35:49 <timburke> #link https://review.opendev.org/c/openstack/swift/+/926179 21:35:49 <patch-bot> patch 926179 - swift - Sharding: Add minimum_active_age to shard compaction - 2 patch sets 21:36:47 <mattoliver> yeah a bunch of stuff there :) 21:37:17 <mattoliver> The early CLEAVED state temp listing hole hit in prod 21:37:51 <mattoliver> and another it solves itself (its a known situration). this time a user noticed and caused it to become more of an issue. 21:38:21 <mattoliver> Which isn't a bad thing.. acoles and I have been wanting to fix it for a while.. and means now the priority rose :) 21:38:27 <timburke> and there's also some hope that those will prove useful not only there, but also when shrinking 21:39:20 <mattoliver> exactly, solving the early CLEAVED also happens to solve the early ACTIVE issue! which might be chosen as a shrink candidate! 21:39:46 <mattoliver> So fixing it earlier in the state machine will be a big win. 21:40:21 <mattoliver> first 2 patches are the most interesting there.. and I'll keep working on them 21:41:08 <timburke> i think there was just one other main thing i wanted to keep tracking 21:41:11 <timburke> fulecorafa, sorry, i think we still haven't taken a look at https://bugs.launchpad.net/swift/+bug/2077179 yet :-( 21:41:12 <patch-bot> Bug #2077179 - S3Api - async delete (New) 21:41:38 <fulecorafa> no worries. I myself did not have time to work on it either XD 21:41:52 <mattoliver> before figuring out it was the early CLEAVED issue I did get to blow the cobwebs off: https://review.opendev.org/c/openstack/swift/+/775066 21:41:52 <patch-bot> patch 775066 - swift - Added a swift-graph-shardranges cli script / tool - 15 patch sets 21:41:55 <timburke> that's all i've got 21:42:01 <timburke> #topic open discussion 21:42:06 <mattoliver> which is a cool tool (but I'm biased) 21:42:20 <timburke> anything else we should talk about today? 21:42:24 <mattoliver> but it does have a dep on graphviz.. 21:42:36 <timburke> mattoliver, oh nice! i keep forgetting about that 21:43:07 <mattoliver> and I don't want that to be a main dep in Swift. So that patch just detects and warns if its missing when using the tool :shrug: 21:43:26 <mattoliver> Also I created: https://review.opendev.org/c/openstack/swift/+/924820 21:43:26 <patch-bot> patch 924820 - swift - account info: Add --sync/-s option (MERGED) - 1 patch set 21:43:39 <mattoliver> oh it's merged... 21:44:04 <mattoliver> my bad, obviously my browser has cached it wrong 21:44:04 <timburke> yeah, i think that's fine -- makes perfect sense that we'd only really want graphviz on a dev or ops machine as they're debugging an issue 21:44:10 <zaitcev> Guys, did you hear that S3 is strongly consistent now? 21:44:37 <timburke> yep -- it's going to be a heavy lift for us to figure out how to get there... 21:45:46 <mattoliver> maybe oneday we'll grow container policies and can have one that is "strong" but yo'd have to live with higher latencies and what not. 21:47:05 <timburke> that was kinda my thought, too. and once we get far enough out that path that we're feeling reasonably confident about it, look at how we could migrate from the old eventually-consistent containers to the new ones 21:47:40 <timburke> we've definitely got users interested in it 21:49:01 <mattoliver> they both that different use-cases. String for those who annoyingly stuck in the past :P and eventually consistant that you can write to fast :) 21:49:08 <mattoliver> *have 21:49:17 <mattoliver> *strong 21:49:22 <mattoliver> wow, I can't type today 21:50:59 <timburke> i figure if they can turn it on for everyone, we should be able to, too ;-) 21:51:23 <timburke> it's coming up on four years now https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ 21:52:25 <mattoliver> anyway, that's all I got. We did get to run the vacuum patch in prod to get an idea of bloated container metrics which was cool.. but I have a little more work on a better way to push the metrics because we don't have guages in statsd. So maybe I'll do recon dumps.. but that doesn't need to be discussed here :) We do some some pretty bloated containers but still unsure how many there areally are :) 21:54:25 <timburke> oh yeah, i started playing with adding gauges a while back... https://review.opendev.org/c/openstack/swift/+/895737 21:54:25 <patch-bot> patch 895737 - swift - swift-recon-cron: Add async_pendings stat gauge - 1 patch set 21:55:12 <timburke> all right, i think i'll call it 21:55:25 <timburke> thank you all for coming, and thank you for working on swift! 21:55:31 <timburke> #endmeeting