| opendevreview | Tim Burke proposed openstack/swift master: Add abstract sockets for process notifications https://review.opendev.org/c/openstack/swift/+/837641 | 01:17 |
|---|---|---|
| opendevreview | Tim Burke proposed openstack/swift master: wsgi: Avoid the need for a separate socket-closer child https://review.opendev.org/c/openstack/swift/+/900957 | 01:20 |
| opendevreview | Matthew Oliver proposed openstack/swift master: WIP: reaper: use internal client https://review.opendev.org/c/openstack/swift/+/925188 | 08:19 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: merge master to feature/mpu https://review.opendev.org/c/openstack/swift/+/936973 | 10:23 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: wsgi_quote mpu symlink target https://review.opendev.org/c/openstack/swift/+/937027 | 14:08 |
| opendevreview | Merged openstack/swift feature/mpu: mpu: wsgi_quote mpu symlink target https://review.opendev.org/c/openstack/swift/+/937027 | 17:01 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: add backoff sleep to REPLICATE https://review.opendev.org/c/openstack/swift/+/934675 | 17:29 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: add backoff sleep to REPLICATE https://review.opendev.org/c/openstack/swift/+/934675 | 17:31 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: Use constraints for api-ref builds https://review.opendev.org/c/openstack/swift/+/937045 | 17:47 |
| opendevreview | Tim Burke proposed openstack/swift master: infra: Remove 'bandit' tox environment https://review.opendev.org/c/openstack/swift/+/937046 | 17:47 |
| mattoliver | morning | 21:00 |
| timburke | #startmeeting swift | 21:01 |
| opendevmeet | Meeting started Wed Dec 4 21:01:00 2024 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| opendevmeet | The meeting name has been set to 'swift' | 21:01 |
| timburke | who's here for the swift meeting? | 21:01 |
| mattoliver | o/ | 21:01 |
| nathang15 | #help | 21:01 |
| nathang15 | me | 21:01 |
| timburke | i haven't updated the agenda; still catching up a bit on what i've missed the past couple weeks | 21:02 |
| timburke | but i think i can pull a few topics out of that catch-up ;-) | 21:02 |
| timburke | #topic failing cors tests | 21:02 |
| timburke | a new eventlet release broke one of our test jobs | 21:03 |
| timburke | it was a fairly esoteric sort of a break; the content-type of HEAD requests against accounts and containers changed with eventlet 0.38.0 | 21:04 |
| timburke | clayg spearheaded the fix in p 935817 | 21:05 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/935817 - swift - Ensure correct content-type in container HEAD resp... (MERGED) - 6 patch sets | 21:05 |
| opendevreview | Matthew Oliver proposed openstack/swift master: Quarantine asyncs older then reclaim_age https://review.opendev.org/c/openstack/swift/+/527296 | 21:05 |
| mattoliver | yup, that was fun, we all learnt how much work timburke does for this project! | 21:06 |
| timburke | and i've proposed backports of that to the stable branches. i'm a little torn about the `Content-Length: 0` that's getting inserted, though | 21:06 |
| timburke | it revealed a broader problem with our CI, though: many/most of our jobs run with a weird mishmash of dependency versions | 21:07 |
| timburke | many get pinned by upper-constraints from the requirements repo, but direct dependencies are not (which meant that eventlet was getting updated despite the global u-c having 0.36.1) | 21:08 |
| timburke | *that* should get fixed by p 936872 (which also has a comment going into what's going on there a little more) | 21:09 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/936872 - swift - CI: drop pip --upgrade flag in tox.ini - 1 patch set | 21:09 |
| mattoliver | ahh, ok, thats annoying | 21:09 |
| timburke | or rather, that addresses the problem for many, but not all, jobs | 21:10 |
| timburke | there's another comment on that review that calls out the other jobs that i saw pulling in latest eventlet | 21:11 |
| mattoliver | ok so the -U would trump the contraints.. wow. | 21:11 |
| timburke | mattoliver, kind of! only for direct dependencies, which is probably why it hasn't bitten us much before: we actively try to keep that list small | 21:12 |
| timburke | earlier today i also pushed up p 937045 to add constraints to a job that was missing them | 21:14 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/937045 - swift - CI: Use constraints for api-ref builds - 1 patch set | 21:14 |
| mattoliver | ok, still unexpected behaviour can cause us a shoot ourselves in the foot unknowingly. nice find. | 21:14 |
| timburke | and i'm working on bringing our py3-constraints.txt file more in line with the requirements u-c version | 21:14 |
| mattoliver | at least now we know we can now work with a later eventlet :P | 21:15 |
| timburke | next up after that, we'll want to | 21:15 |
| timburke | (1) use constraints for all the *other* places we run `pip install` (down in ansible playbooks) | 21:16 |
| timburke | and (2) add some non-voting jobs that intentionally run unconstrained | 21:16 |
| timburke | i'm thinking unit and func-encryption tests should be sufficient, probably on py312 | 21:17 |
| mattoliver | kk, so api-ref I feel less worried about but agree that consistency is much better, because rather then problems like this biting us on the butt again. | 21:17 |
| mattoliver | Sounds like a solid plan. | 21:17 |
| mattoliver | non-voting to let us know if there are problems afoot but not gate blockers! | 21:18 |
| timburke | #topic statsd-logging coupling | 21:20 |
| timburke | looks like p 931473 finally merged! | 21:20 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/931473 - swift - Remove statds from the logs module (MERGED) - 27 patch sets | 21:20 |
| timburke | great work acoles and shreeya, and thanks for reviewing clayg | 21:21 |
| timburke | i know there was a decent bit of back-and-forth on it | 21:21 |
| timburke | not just on that patch, but p 915483 as well | 21:22 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/915483 - swift - Split statsd client from logger (ABANDONED) - 52 patch sets | 21:22 |
| mattoliver | 52 patchesets and abandoned. yeah | 21:23 |
| timburke | now that that's landed... | 21:25 |
| timburke | #topic labeled metrics | 21:25 |
| timburke | it looks like yan has rebased the chain: p 909882, p 917711, p 930918 | 21:26 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/909882 - swift - stats: API for native labeled metrics - 39 patch sets | 21:26 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/917711 - swift - Add labeled metrics to proxy-logging - 23 patch sets | 21:26 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/930918 - swift - proxy-logging: Add real-time transfer bytes counters - 10 patch sets | 21:26 |
| mattoliver | nice, one step closer to having labelled metrics support in swift | 21:27 |
| timburke | we should probably try to focus on that next so it doesn't get stalled out again (i think i last reviewed it back in July 😱) | 21:27 |
| mattoliver | we have been running these downstream too right, so they're also getting some live testing | 21:28 |
| timburke | speaking of code already running downstream... | 21:29 |
| timburke | #topic swift-reload robustness | 21:29 |
| timburke | i finally got around to addressing some of clayg's concerns on p 837641 | 21:30 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/837641 - swift - Add abstract sockets for process notifications - 13 patch sets | 21:30 |
| mattoliver | oh cool, I'll take a look | 21:31 |
| timburke | the major thing was to rip out the ancdata processing to verify that messages were coming from the expected pid; it's unlikely that anything else would be using the pid-specific address | 21:31 |
| mattoliver | kk | 21:32 |
| timburke | which also gave me an excuse to dust off p 900957 | 21:32 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/900957 - swift - wsgi: Avoid the need for a separate socket-closer ... - 2 patch sets | 21:32 |
| mattoliver | cool, oneday we'll get through your swift-reload chain :P | 21:35 |
| timburke | and speaking of server process management, i should probably also revisit p 789035 ... | 21:35 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/789035 - swift - wsgi: Reap stale workers (after a timeout) followi... - 19 patch sets | 21:35 |
| timburke | but there's always so much to review! | 21:35 |
| timburke | on that note... | 21:35 |
| timburke | #topic feature/mpu | 21:35 |
| timburke | acoles has been hard at work! | 21:35 |
| timburke | i'm pretty sure the majority of landed changes these past two weeks have been to feature/mpu | 21:36 |
| timburke | that said, i'm sure he'd appreciate some reviews on the other ones that are still open: https://review.opendev.org/q/project:openstack/swift+branch:feature/mpu+is:open | 21:37 |
| timburke | ...but i think that's about as far as i've gotten in terms of catching up ;-) | 21:39 |
| timburke | #topic open discussion | 21:39 |
| timburke | anything else we should talk about this week? | 21:39 |
| mattoliver | I've been playing with 2 things | 21:40 |
| mattoliver | p 527296 was revived | 21:40 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/527296 - swift - Quarantine asyncs older then reclaim_age - 4 patch sets | 21:40 |
| mattoliver | And I've changed it to quarantine rather then unlink. | 21:40 |
| zaitcev | It's like there's no win. | 21:41 |
| zaitcev | Except keep the cluster health up so that asyncs aren't that old. | 21:41 |
| mattoliver | Also finally managed to get the reaper to properly use an internal client. Thanks to writing a probe test, lead me to find areas where internal_clients would fail because x-backend-override-deleted isn't passed it get_info requests. | 21:41 |
| timburke | at least now we've got p 931979 ! | 21:42 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/931979 - swift - Add oldest failed async pending tracker (MERGED) - 33 patch sets | 21:42 |
| mattoliver | zaitcev: yeah true. and stops the updater from continually picking them up and useing them forever | 21:43 |
| mattoliver | probably why it sat there for 6 years :P | 21:43 |
| mattoliver | here is the shard-aware reaper using internal_client: p 925188 | 21:44 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/925188 - swift - WIP: reaper: use internal client - 5 patch sets | 21:44 |
| mattoliver | Still need to clean it up. As there was a bit of live hacking/debugging while diagnosing why the probetest kept failing. | 21:45 |
| mattoliver | but ended up being (as I said) get_infos and better account-server override-delete support. | 21:46 |
| mattoliver | turns out direct client made certain things much easier. | 21:46 |
| mattoliver | That patch also adds a 'legecy_mode' to the reaper, for a possible upgrade path.. but that's still in debate.. so still wip | 21:47 |
| mattoliver | that's all I have | 21:47 |
| jianjian | good work, Matt. | 21:48 |
| jianjian | yeah, zuul is not happy yet with the reaper patch | 21:48 |
| mattoliver | yeah, I saw that. Only got probe test working late last night.. so now I'll doublecheck everything else with a fresh brain :) | 21:50 |
| timburke | think it's just that unit tests need updating | 21:51 |
| jianjian | for the "Quarantine asyncs older then reclaim_age" patch, is there a valid case that we are going to need that async pending even after reclaim_age? | 21:51 |
| timburke | i think the idea is that the operator probably ought to dig in some and figure out what the listing is supposed to look like | 21:52 |
| jianjian | yeah, I am thinking if there is case like uploaded object is still there, and container listing still doesn't have it | 21:52 |
| jianjian | will think more and look into it | 21:54 |
| mattoliver | Clay liked the idea try again and only q if it failed | 21:54 |
| timburke | with multiple replicas trying to get the update in, i don't think dark data tends to be the issue. the trouble is usually the opposite: ghost listings, where the data's been cleaned up but container still has the listing | 21:55 |
| mattoliver | if it partially succeeds (managed to update a replica) then keep it around.. but knowing it still might get q next time round. | 21:55 |
| mattoliver | yeah, ghost listings is really what I meant | 21:56 |
| jianjian | good point, the chance of ghost listing is bigger | 21:56 |
| timburke | even if we've got an async for the delete, once we get it to the container, it'll delete the row entirely, so another replica can just send the ghost back again :-( | 21:58 |
| timburke | all right, we're about at time | 21:59 |
| mattoliver | does it delete entirely or put a tombstone (mark deleted=1) in the db | 21:59 |
| timburke | thank you all for coming, and thanks for working on swift! | 21:59 |
| timburke | mattoliver, well, if it's past reclaim age... | 21:59 |
| timburke | #endmeeting | 21:59 |
| opendevmeet | Meeting ended Wed Dec 4 21:59:49 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2024/swift.2024-12-04-21.01.html | 21:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2024/swift.2024-12-04-21.01.txt | 21:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2024/swift.2024-12-04-21.01.log.html | 21:59 |
| mattoliver | oh yeah | 21:59 |
| mattoliver | itll probably write it and then next cycle reclaim it : P | 22:00 |
| mattoliver | *next replicator cycle reclaim it. | 22:00 |
| mattoliver | so hmm yeah | 22:00 |
| opendevreview | Tim Burke proposed openstack/swift master: Bring py3-constraints.txt more in line with global u-c https://review.opendev.org/c/openstack/swift/+/937072 | 22:20 |
| opendevreview | Tim Burke proposed openstack/swift master: Bring py3-constraints.txt more in line with global u-c (part 2) https://review.opendev.org/c/openstack/swift/+/937073 | 22:20 |
| opendevreview | Tim Burke proposed openstack/swift master: Add tool for comparing py3-constraints.txt against global u-c https://review.opendev.org/c/openstack/swift/+/937074 | 22:20 |
| opendevreview | Tim Burke proposed openstack/swift master: Up-rev hacking https://review.opendev.org/c/openstack/swift/+/937075 | 22:26 |
| opendevreview | Tim Burke proposed openstack/swift master: Up-rev hacking https://review.opendev.org/c/openstack/swift/+/937075 | 22:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!