| opendevreview | ASHWIN A NAIR proposed openstack/swift master: manage py2-contsraints for swift correctly https://review.opendev.org/c/openstack/swift/+/927313 | 00:26 |
|---|---|---|
| *** avanzaghi15 is now known as avanzaghi1 | 07:20 | |
| *** edausqu is now known as edausq | 07:20 | |
| opendevreview | Matthew Oliver proposed openstack/swift master: shrinking: Add a CLEAVING state https://review.opendev.org/c/openstack/swift/+/789651 | 07:51 |
| opendevreview | Shreeya Deshpande proposed openstack/swift master: Split statsd client from logger https://review.opendev.org/c/openstack/swift/+/915483 | 15:56 |
| opendevreview | Tim Burke proposed openstack/swift master: diskfile: Treat EUCLEAN like ENODATA https://review.opendev.org/c/openstack/swift/+/927425 | 18:48 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Attempt to use configured tmp in xprofile tests https://review.opendev.org/c/openstack/swift/+/926372 | 19:27 |
| opendevreview | Tim Burke proposed openstack/swift master: obj: Have handoff_delete reduce replication to handoffs https://review.opendev.org/c/openstack/swift/+/919911 | 19:47 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: manage py2-contsraints for swift correctly https://review.opendev.org/c/openstack/swift/+/927313 | 20:19 |
| opendevreview | Tim Burke proposed openstack/swift master: Drop py2 support https://review.opendev.org/c/openstack/swift/+/853590 | 20:19 |
| mattoliver | Morning | 21:00 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Aug 28 21:00:34 2024 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliver | o/ | 21:01 |
| timburke | i guess it's just us for now then :-) | 21:02 |
| timburke | thanks for covering for me while i was still recovering last week mattoliver | 21:02 |
| timburke | sorry for having missed the meeting without warning | 21:02 |
| mattoliver | Nps man! | 21:03 |
| fulecorafa | o/ | 21:03 |
| timburke | unfortunately, i still haven't updated the agenda | 21:04 |
| zaitcev | aww. I was just going to check if InternalClient work was on it. | 21:04 |
| timburke | zaitcev, what kind of internal client work? | 21:04 |
| zaitcev | IIRC Matt wanted to move either reaper or .. something to it, from direct client. | 21:06 |
| zaitcev | I hoped he'd do the same thing to the dark data watcher. | 21:06 |
| timburke | oh, yeah! in relation to | 21:07 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/2070397 | 21:07 |
| patch-bot | Bug #2070397 - Account reaper do not reap sharded containers (In Progress) | 21:07 |
| timburke | and | 21:07 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/924034 | 21:07 |
| patch-bot | patch 924034 - swift - WIP: reaper: Support reaping sharded containers - 1 patch set | 21:07 |
| zaitcev | Another suspicious place is the reconciler | 21:08 |
| mattoliver | oh yeah, I did a patch, I probably should test it out some more.. and write tests. | 21:08 |
| timburke | yeah, i think we still need to find the bandwidth to focus on it :-( | 21:08 |
| timburke | i think reconciler already uses IC though: https://github.com/openstack/swift/blob/master/swift/container/reconciler.py#L378 | 21:09 |
| mattoliver | obviously got distracted with some other sharding internal issue that I'm now working on a new version of CLEAVING and state_changed_at patches to help solve.. but I'll defintely loop back to it. I'll try push it along this week | 21:09 |
| timburke | zaitcev, or were you more worried about policy for the root vs policy for the shards? | 21:10 |
| timburke | i think mattoliver also has patches kicking around related to that... we should dust them off, too | 21:10 |
| mattoliver | lol, I seem to start alot of things and get distracted :( | 21:11 |
| timburke | don't we all ;-) | 21:11 |
| mattoliver | maybe I need to one say take a week and just focus on finishing internal clienting all the things :P (one day) | 21:12 |
| mattoliver | if there is a bug or something that your hitting zaitcev I'd happily start prioritsing it for you | 21:12 |
| timburke | speaking of dusting things off -- we're getting towards the end of the cycle, so i've been thinking about getting a release out, and refreshing a few patches of my own in hopes that we might land some of them | 21:13 |
| zaitcev | mattoliver: sorry for the noise, I | 21:13 |
| timburke | there were a couple old drop-translations patches i hadn't pushed on in a while | 21:13 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/823423 | 21:13 |
| patch-bot | patch 823423 - swift - common: Stop translating log messages - 2 patch sets | 21:13 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/823424 | 21:14 |
| patch-bot | patch 823424 - swift - common: Stop translating a bunch of printed messag... - 2 patch sets | 21:14 |
| zaitcev | mattoliver: I just saw your work and it seemed familiar to what the watcher has to deal with. | 21:14 |
| mattoliver | zaitcev: never need to be sorry. Reminding me about them is great, things can get quite chaotic at nvidia and things fall out of my head :) Your right we had the similar issues with the dark data watcher so it could use it too! | 21:15 |
| mattoliver | Maybe I'll start a doc for the ideas section where we can track ic daemon improvements. And I can bring it up as a topic next PTG | 21:16 |
| timburke | 👍 | 21:16 |
| mattoliver | timburke: cool, did I also see you getting the deprecate py2 patch ready? | 21:16 |
| mattoliver | just. in. case? | 21:16 |
| timburke | you did :-) i need to check in with a couple people about where we are on running py3 everywhere still, though... | 21:17 |
| timburke | there was one other one i'd kind of like us to make a decision about before this next release | 21:18 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/924795 | 21:18 |
| patch-bot | patch 924795 - swift - Remove legacy bin/ scripts - 3 patch sets | 21:18 |
| mattoliver | whats the +1 to -1 ratiio? who wins? | 21:18 |
| timburke | mattoliver and i are both +1 on dropping them now; packagers shouldn't need them any more | 21:18 |
| timburke | but no one else has left a vote. i guess that means we can just do as we like? :P | 21:19 |
| mattoliver | well it has been a few weeks :) | 21:19 |
| timburke | at any rate, i wanted at least one more opportunity for anyone who might like to raise an objection to do so | 21:19 |
| timburke | lacking any other feedback, i'll merge it as i'm putting together release notes | 21:20 |
| mattoliver | sounds good | 21:21 |
| mattoliver | Speaking of release, I've tricked csmart into finishing the account qouta follow up patch Qatar started, and will make sure it's ready before the release. | 21:21 |
| fulecorafa | 👍🏻 | 21:21 |
| mattoliver | I'll poke him about it today :) | 21:21 |
| timburke | \o/ thanks mattoliver and csmart! | 21:21 |
| mattoliver | timburke: have you loaded up the priority reviews page for this release? | 21:22 |
| timburke | i was just about to say i need to do that :D | 21:22 |
| mattoliver | kk, I'll take a look at existing patches too | 21:23 |
| timburke | other interesting bodies of work lately: | 21:23 |
| timburke | we're getting really close on splitting out the stats client from our adapted loggers | 21:23 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/915483 | 21:23 |
| patch-bot | patch 915483 - swift - Split statsd client from logger - 29 patch sets | 21:23 |
| timburke | there will still be some chimera object returned by utils.get_logger, but there won't be any reference to statsd anymore in utils/logs.py | 21:24 |
| mattoliver | kk, nice | 21:25 |
| timburke | we also saw a timestamp collision in prod! of course, the way you notice such a thing is when you get an EC object that can't be decoded :-( | 21:26 |
| mattoliver | woah, i missed that | 21:26 |
| timburke | on our 8+4 policy, we had a 7/5 split for which version was written down | 21:26 |
| timburke | it feels a lot like https://bugs.launchpad.net/swift/+bug/1910804 but subtly different since it was actually external-client-driven | 21:27 |
| patch-bot | Bug #1910804 - Encryption doesn't play well with processes that copy cleartext data while preserving timestamps (Fix Released) | 21:27 |
| timburke | fwiw, the two clients seemed to have written the same data (so the X-Object-Sysmeta-Crypto-Etag-Mac values all matched) but that still doesn't really help much :-( | 21:29 |
| timburke | but that's why clayg has been throwing out some patches lately | 21:29 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/927327 | 21:29 |
| patch-bot | patch 927327 - swift - wip: testing timestamp collison - 1 patch set | 21:29 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/927329 | 21:30 |
| patch-bot | patch 927329 - swift - wip: detect link at collision - 1 patch set | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/927330 | 21:30 |
| patch-bot | patch 927330 - swift - wip: check etag in commit - 1 patch set | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/927328 | 21:30 |
| patch-bot | patch 927328 - swift - wip: detect ec encyption collision - 1 patch set | 21:30 |
| mattoliver | he has been busy, and makes alot of sense. don't know how I missed that. kinda exciting (not for sre, but from a dev POV) :) | 21:31 |
| timburke | and last, mattoliver has been busy trying to smooth out some rough edges in the sharding state machine | 21:35 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/789651 | 21:35 |
| patch-bot | patch 789651 - swift - shrinking: Add a CLEAVING state - 13 patch sets | 21:35 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/926036 | 21:35 |
| patch-bot | patch 926036 - swift - ShardRange: track last state change timestamp - 6 patch sets | 21:35 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/926179 | 21:35 |
| patch-bot | patch 926179 - swift - Sharding: Add minimum_active_age to shard compaction - 2 patch sets | 21:35 |
| mattoliver | yeah a bunch of stuff there :) | 21:36 |
| mattoliver | The early CLEAVED state temp listing hole hit in prod | 21:37 |
| mattoliver | and another it solves itself (its a known situration). this time a user noticed and caused it to become more of an issue. | 21:37 |
| mattoliver | Which isn't a bad thing.. acoles and I have been wanting to fix it for a while.. and means now the priority rose :) | 21:38 |
| timburke | and there's also some hope that those will prove useful not only there, but also when shrinking | 21:38 |
| mattoliver | exactly, solving the early CLEAVED also happens to solve the early ACTIVE issue! which might be chosen as a shrink candidate! | 21:39 |
| mattoliver | So fixing it earlier in the state machine will be a big win. | 21:39 |
| mattoliver | first 2 patches are the most interesting there.. and I'll keep working on them | 21:40 |
| timburke | i think there was just one other main thing i wanted to keep tracking | 21:41 |
| timburke | fulecorafa, sorry, i think we still haven't taken a look at https://bugs.launchpad.net/swift/+bug/2077179 yet :-( | 21:41 |
| patch-bot | Bug #2077179 - S3Api - async delete (New) | 21:41 |
| fulecorafa | no worries. I myself did not have time to work on it either XD | 21:41 |
| mattoliver | before figuring out it was the early CLEAVED issue I did get to blow the cobwebs off: https://review.opendev.org/c/openstack/swift/+/775066 | 21:41 |
| patch-bot | patch 775066 - swift - Added a swift-graph-shardranges cli script / tool - 15 patch sets | 21:41 |
| timburke | that's all i've got | 21:41 |
| timburke | #topic open discussion | 21:42 |
| mattoliver | which is a cool tool (but I'm biased) | 21:42 |
| timburke | anything else we should talk about today? | 21:42 |
| mattoliver | but it does have a dep on graphviz.. | 21:42 |
| timburke | mattoliver, oh nice! i keep forgetting about that | 21:42 |
| mattoliver | and I don't want that to be a main dep in Swift. So that patch just detects and warns if its missing when using the tool :shrug: | 21:43 |
| mattoliver | Also I created: https://review.opendev.org/c/openstack/swift/+/924820 | 21:43 |
| patch-bot | patch 924820 - swift - account info: Add --sync/-s option (MERGED) - 1 patch set | 21:43 |
| mattoliver | oh it's merged... | 21:43 |
| mattoliver | my bad, obviously my browser has cached it wrong | 21:44 |
| timburke | yeah, i think that's fine -- makes perfect sense that we'd only really want graphviz on a dev or ops machine as they're debugging an issue | 21:44 |
| zaitcev | Guys, did you hear that S3 is strongly consistent now? | 21:44 |
| timburke | yep -- it's going to be a heavy lift for us to figure out how to get there... | 21:44 |
| mattoliver | maybe oneday we'll grow container policies and can have one that is "strong" but yo'd have to live with higher latencies and what not. | 21:45 |
| timburke | that was kinda my thought, too. and once we get far enough out that path that we're feeling reasonably confident about it, look at how we could migrate from the old eventually-consistent containers to the new ones | 21:47 |
| timburke | we've definitely got users interested in it | 21:47 |
| mattoliver | they both that different use-cases. String for those who annoyingly stuck in the past :P and eventually consistant that you can write to fast :) | 21:49 |
| mattoliver | *have | 21:49 |
| mattoliver | *strong | 21:49 |
| mattoliver | wow, I can't type today | 21:49 |
| timburke | i figure if they can turn it on for everyone, we should be able to, too ;-) | 21:50 |
| timburke | it's coming up on four years now https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/ | 21:51 |
| mattoliver | anyway, that's all I got. We did get to run the vacuum patch in prod to get an idea of bloated container metrics which was cool.. but I have a little more work on a better way to push the metrics because we don't have guages in statsd. So maybe I'll do recon dumps.. but that doesn't need to be discussed here :) We do some some pretty bloated containers but still unsure how many there areally are :) | 21:52 |
| timburke | oh yeah, i started playing with adding gauges a while back... https://review.opendev.org/c/openstack/swift/+/895737 | 21:54 |
| patch-bot | patch 895737 - swift - swift-recon-cron: Add async_pendings stat gauge - 1 patch set | 21:54 |
| timburke | all right, i think i'll call it | 21:55 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:55 |
| timburke | #endmeeting | 21:55 |
| opendevmeet | Meeting ended Wed Aug 28 21:55:31 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:55 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2024/swift.2024-08-28-21.00.html | 21:55 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2024/swift.2024-08-28-21.00.txt | 21:55 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2024/swift.2024-08-28-21.00.log.html | 21:55 |
| mattoliver | oh nice, well that might be useful for me.. lol tim having a patch in the wings strikes again :P | 21:55 |
| mattoliver | thanks tim, thanks everyone | 21:55 |
| timburke | yeah, the gauge itself was dead simple to implement | 21:56 |
| timburke | https://review.opendev.org/c/openstack/swift/+/895737/1/swift/common/utils/__init__.py | 21:56 |
| patch-bot | patch 895737 - swift - swift-recon-cron: Add async_pendings stat gauge - 1 patch set | 21:56 |
| mattoliver | well maybe I'll change over to that and see how it goes in my vsaio. | 21:57 |
| opendevreview | Tim Burke proposed openstack/swift master: diskfile: Treat EUCLEAN like ENODATA https://review.opendev.org/c/openstack/swift/+/927425 | 22:38 |
| opendevreview | Tim Burke proposed openstack/swift master: diskfile: Treat EUCLEAN like ENODATA https://review.opendev.org/c/openstack/swift/+/927425 | 22:58 |
| opendevreview | Merged openstack/python-swiftclient master: Authors/ChangeLog for 4.6.0 https://review.opendev.org/c/openstack/python-swiftclient/+/927304 | 23:43 |
| opendevreview | Merged openstack/swift master: tests: Attempt to use configured tmp in xprofile tests https://review.opendev.org/c/openstack/swift/+/926372 | 23:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!