| opendevreview | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/c/openstack/swift/+/931120 | 04:06 |
|---|---|---|
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: make MPUSession.is_aborted a property https://review.opendev.org/c/openstack/swift/+/931023 | 10:26 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: minor middleware fixes https://review.opendev.org/c/openstack/swift/+/931011 | 13:10 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: complete mpu with user content-type when available https://review.opendev.org/c/openstack/swift/+/931165 | 13:10 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: minor middleware fixes https://review.opendev.org/c/openstack/swift/+/931011 | 13:14 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: complete mpu with user content-type when available https://review.opendev.org/c/openstack/swift/+/931165 | 13:14 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: use '~' delimiter for MPUId parts https://review.opendev.org/c/openstack/swift/+/931021 | 13:36 |
| opendevreview | Yan Xiao proposed openstack/swift master: stats: API for native labeled metrics https://review.opendev.org/c/openstack/swift/+/909882 | 13:59 |
| opendevreview | Yan Xiao proposed openstack/swift master: Add labeled metrics to proxy-logging https://review.opendev.org/c/openstack/swift/+/917711 | 13:59 |
| opendevreview | Merged openstack/swift feature/mpu: mpu: make MPUSession.is_aborted a property https://review.opendev.org/c/openstack/swift/+/931023 | 14:14 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: use '~' delimiter for MPUId parts https://review.opendev.org/c/openstack/swift/+/931021 | 14:26 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: minor middleware fixes https://review.opendev.org/c/openstack/swift/+/931011 | 14:58 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: complete mpu with user content-type when available https://review.opendev.org/c/openstack/swift/+/931165 | 14:58 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: modify MPU entries in container listing https://review.opendev.org/c/openstack/swift/+/928840 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: completeUpload: Parse user manifest before modifying session https://review.opendev.org/c/openstack/swift/+/929654 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: completeUpload: append part count to MPU etag https://review.opendev.org/c/openstack/swift/+/929656 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: cleanup mpu GET and HEAD response headers. https://review.opendev.org/c/openstack/swift/+/930369 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: fix listParts response headers https://review.opendev.org/c/openstack/swift/+/930483 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: mpu: pad part-number in part names to 6 digits https://review.opendev.org/c/openstack/swift/+/930741 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: WIP: func test symlink to MPU https://review.opendev.org/c/openstack/swift/+/930235 | 15:00 |
| opendevreview | Alistair Coles proposed openstack/swift feature/mpu: WIP s3api: add more MPU cross-compat tests https://review.opendev.org/c/openstack/swift/+/931187 | 15:07 |
| opendevreview | Chinemerem Chigbo proposed openstack/swift master: Make object-expirer respect internal_client_conf_path https://review.opendev.org/c/openstack/swift/+/930095 | 15:58 |
| opendevreview | Tim Burke proposed openstack/swift master: slo: Support range requests for part-number queries https://review.opendev.org/c/openstack/swift/+/906391 | 16:10 |
| opendevreview | Yan Xiao proposed openstack/swift master: proxy-logging: Add real-time transfer bytes counters https://review.opendev.org/c/openstack/swift/+/930918 | 16:59 |
| opendevreview | Merged openstack/swift feature/mpu: mpu: minor middleware fixes https://review.opendev.org/c/openstack/swift/+/931011 | 18:38 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: s3api: testing for md5 mismatch errors https://review.opendev.org/c/openstack/swift/+/931223 | 19:56 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Object-expirer: continue to process next container on listing errors https://review.opendev.org/c/openstack/swift/+/930393 | 20:07 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: s3api: testing for md5 mismatch errors https://review.opendev.org/c/openstack/swift/+/931223 | 20:13 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: s3api: testing for md5 mismatch errors https://review.opendev.org/c/openstack/swift/+/931223 | 20:52 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: s3api: refactor testing for checksum errors https://review.opendev.org/c/openstack/swift/+/931223 | 20:53 |
| kota | good morniing | 20:57 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Oct 2 21:00:57 2024 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:01 |
| zaitcev | o/ | 21:01 |
| kota | o/ | 21:01 |
| timburke | i didn't get around to updating the agenda (sorry) | 21:01 |
| timburke | but first up is still the PTG! | 21:02 |
| timburke | #topic PTG | 21:02 |
| fulecorafa | o/ | 21:02 |
| timburke | it's not far off now -- Oct 21-25 | 21:02 |
| rwmtinkywinky4 | (not a part of the team, but thought I should make an appearance!) | 21:02 |
| timburke | i'm never too particular about who's "part of the team" -- way i see it, if you want to help make swift better, you're always welcome :-) | 21:04 |
| rwmtinkywinky4 | great :D | 21:05 |
| timburke | i still need to book meeting slots for the ptg -- filling out the poll at https://framadate.org/LQOsGVVWXDhXqQUw would be appreciated | 21:05 |
| timburke | though i also know there's a strong US contingent that hasn't responded yet :-/ i poked them internally earlier today | 21:06 |
| timburke | we've also started gathering topics to discuss at | 21:07 |
| timburke | #link https://etherpad.opendev.org/p/swift-ptg-epoxy | 21:07 |
| timburke | i think i heard that cschwede might be able to make it, too -- it'd be nice to catch up a bit :-) | 21:09 |
| timburke | but i think that's about all i've got to say about the ptg | 21:10 |
| timburke | next up | 21:10 |
| timburke | #topic feature/mpu | 21:10 |
| timburke | acoles is on a roll lately! | 21:10 |
| fulecorafa | acoles has been on a streak! | 21:10 |
| timburke | i'm sure he'd appreciate more feedback if anyone has some spare review cycles | 21:11 |
| fulecorafa | Trying to find some time to review these | 21:11 |
| timburke | #link https://review.opendev.org/q/project:openstack/swift+branch:feature/mpu+is:open | 21:11 |
| timburke | to get a sense of what's currently in progress | 21:11 |
| timburke | next up | 21:14 |
| timburke | #topic object expirer | 21:14 |
| timburke | i know clayg and jianjian (and to some degree, acoles) have been thinking about this lately | 21:14 |
| indianwhocodes | o/ | 21:14 |
| fulecorafa | Do we mean the object expirer for orphas in this case? Trying to get the context | 21:17 |
| timburke | so we have a large backlog of work in the queue, and recently updated our task-containers-per-day from 100 to 10000 | 21:18 |
| timburke | the 100 is currently a hard-coded constant buried down in utils | 21:18 |
| timburke | we previously ran into issues with all expirers wanting to list all tasks in every container -- with an expirer running on each object-server, we were doing so many listings that the container servers would get overloaded | 21:20 |
| timburke | so we put up a patch to try to divide things such that any particular expirer could skip an entire container, then take more work from the containers that it *did* bother to list | 21:22 |
| timburke | in the extreme, each expirer could find the one and only container it actually cared about and work through the whole thing -- but only if we remove that hard-coded 100-containers-per-day limit | 21:24 |
| timburke | trouble was, even after doing that, we have a bunch of work still in the old queues, so what would happen was that we'd get 100 really busy expirers, and 1000+ mostly-idle ones | 21:25 |
| jianjian | yeah, Tim summarized the situation very well, thanks | 21:26 |
| timburke | and worse, a bunch of the old entries were in fact stale -- the object had been overwritten, or updated with a new x-delete-at -- and the attempt to clean up the queue went to the new 1-in-10000 container, rather than the old 1-in-100 | 21:27 |
| timburke | so we're also looking at what to do with the mass of 412s you get when X-If-Delete-At doesn't match | 21:27 |
| timburke | we think we've found a couple viable paths forward | 21:28 |
| timburke | first is to try to migrate the old queue entries to the new, expected location | 21:28 |
| jianjian | yes, thanks Tim for the suggestions for fixing 412, I am working on a patch trying to remove those old delete tasks from queue as soon as we see them | 21:28 |
| timburke | second is to try to figure out when a 412 happens because the metadata it was based upon was stale, so we can delete them rather than retry forever | 21:29 |
| jianjian | also, there is one patch under review as well, to have expirer dealing with all different listing errors, 503, 404: If, instead of this remark, my father had taken the pains to explain to me that the principles of Agrippa had been entirely exploded, and that a modern system of science had been introduced, I should certainly have thrown Agrippa aside, but the cursory glance my father had taken of my volume by no means | 21:29 |
| jianjian | assured me that he was acquainted with its contents; and I continued to read with the greatest avidity. | 21:29 |
| jianjian | sorry, that was my son 😞 | 21:30 |
| jianjian | also, there is one patch under review as well, to have expirer dealing with all different listing errors, 503, 404: https://review.opendev.org/c/openstack/swift/+/930393 | 21:30 |
| patch-bot | patch 930393 - swift - Object-expirer: continue to process next container... - 4 patch sets | 21:30 |
| timburke | jianjian, you reading frankenstein? i recently picked that up again -- been a while :-) | 21:30 |
| timburke | i was just looking for another quote earlier today: "It was my temper to avoid a crowd, and to attach myself fervently to a few." | 21:32 |
| jianjian | my son is working on his homework, yes, it is frankenstein. And I accidentally copied his copy-and-post, haha, sorry again. | 21:32 |
| kota | lol | 21:33 |
| timburke | some other interesting patches in this vein: | 21:33 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/920452 | 21:33 |
| patch-bot | patch 920452 - swift - Configurable expiring_objects_task_container_per_day - 19 patch sets | 21:33 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/918366 | 21:34 |
| jianjian | Tim is a great coder plus reader 😉 | 21:34 |
| patch-bot | patch 918366 - swift - Parallel distributed task container iteration - 22 patch sets | 21:34 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/914713 | 21:35 |
| patch-bot | patch 914713 - swift - DNM expirer: new options to control task iteration - 36 patch sets | 21:35 |
| indianwhocodes | i would also appreciate a review on | 21:36 |
| indianwhocodes | #link https://review.opendev.org/c/openstack/swift/+/916547 | 21:36 |
| patch-bot | patch 916547 - swift - fix x-open-expired 404 on HEAD?part-number reqs - 43 patch sets | 21:36 |
| jianjian | those two patches introduce a new random strategy and parallel strategy to distribute the expirer listing work evenly to every expirer, and probably will ditch random and land parallel only upstream if testing of parallel goes well in the end. | 21:37 |
| timburke | i feel like we ought to structure these a little more so we could actually review/merge some of them -- i don't particularly like hos that DNM patch is ahead of the other two -- makes me worry none of it's actually close to being able to merge | 21:37 |
| indianwhocodes | my bad, msg got prematurely sent | 21:37 |
| timburke | ok, good to know jianjian | 21:38 |
| timburke | indianwhocodes, no worries :-) | 21:38 |
| timburke | i think i'm running out of things to bring up anyway | 21:39 |
| timburke | #topic open discussion | 21:39 |
| timburke | what else should we discuss this week? | 21:39 |
| fulecorafa | So, something came up around here, wanted to get your input in | 21:39 |
| fulecorafa | The demand is to delete a container/bucket with items inside of it, without having to list all objects in container -> delete objects in batches -> delete container | 21:40 |
| timburke | does it need to be user(/client) facing? | 21:41 |
| zaitcev | Isn't it what bulk does? | 21:41 |
| rwmtinkywinky4 | we do get people grumpy about that, and have scripts to work around it | 21:41 |
| fulecorafa | Was thinking of doind something like what object expirer does, but for buckets, `X-Delete-At` and "markers" | 21:41 |
| JayF | o/ I have an item for open discussion whenever you're at the end of this thread | 21:41 |
| fulecorafa | timburke yes we'd like it client facing | 21:41 |
| zaitcev | It has both bulk upload and bulk delete. Still not a single call but at least better than the full thing. | 21:41 |
| timburke | a while back i wrote up https://github.com/openstack/swift/blob/master/swift/cli/container_deleter.py but it's intended as an operator tool | 21:41 |
| fulecorafa | zaitcev kind of. In this case we're deleting buckets with about 100 milion objects, so it takes quite a long time to delete all objects throught client scripting | 21:42 |
| fulecorafa | Some bucket deletions are taking north of 7 days running non stop | 21:42 |
| fulecorafa | timburke nice to know, will certainly take a look. Thanks for the help | 21:44 |
| fulecorafa | rwmtinkywinky4 mind sharing how you solved this? | 21:44 |
| rwmtinkywinky4 | fulecorafa: I'll see if I can share some of our deletion scripts | 21:44 |
| timburke | fulecorafa, so one way or another, it's going to take a long time to fully process. what should happen with the bucket in the mean time? do we want this to be something like an account deletion, where the container still exists but client attempts to access it get back a 410 or the like? | 21:44 |
| fulecorafa | timburke yes I'm aware. But at this point, the important thing is to make it appear to the client as if it's already been deleted | 21:45 |
| timburke | i should go back through some of our old etherpads -- i'm sure i remember people talking about making a container-reaper before | 21:45 |
| fulecorafa | From what I understand, that's what happens in object expirer right? It may still be there, not deleted, but since it is marked with `X-Delete-At` it is not listed to the user/client | 21:46 |
| timburke | fulecorafa, what should happen if they attempt to recreate it? | 21:46 |
| timburke | yup -- objects appear in listings until the expirer actually processes them | 21:46 |
| fulecorafa | That's a good question, but I recon it should just fail with a BucketAlreadyExists | 21:47 |
| jianjian | haven't tried it and read code, can we rename a container? | 21:48 |
| timburke | jianjian, nope -- gotta copy all the data, then delete the old locations | 21:48 |
| zaitcev | The name inputs into DHT directly for both containers and objects, so I highly doubt it's possible. | 21:48 |
| zaitcev | There's no inode. | 21:48 |
| timburke | bingo | 21:48 |
| fulecorafa | jianjian I think not. The "move" existant is just a copy then delete | 21:48 |
| jianjian | yeah, it's the ring. | 21:49 |
| fulecorafa | One thing to notice is that we have a workaround to share the bucket namespace between all accounts | 21:49 |
| fulecorafa | So user_b cannot create my_bucket if user_a/my_bucket exists | 21:50 |
| timburke | fulecorafa, the jobbed off delete seems reasonable enough -- i could definitely imagine a way to make that work. might even be able to piggy-back off the existing account-reaper -- have it start looking at both account and container dbs. it'd be a bit of finicky plumbing to get all the behaviors right, i think | 21:50 |
| timburke | fulecorafa, the global bucket namespace sounds super cool! i keep meaning to think more about that, but haven't | 21:51 |
| fulecorafa | Yeah, in our case we just keep a separate DB, not implemented in swift itself | 21:51 |
| fulecorafa | Sorry about that | 21:51 |
| timburke | yeah, that seems like the way to do it :-) | 21:52 |
| timburke | should be low enough write traffic that it wouldn't be too hard to manage | 21:52 |
| fulecorafa | Yep | 21:52 |
| fulecorafa | I do think it's finicky as well. But I took enough time, I think JayF wanted to chat? | 21:53 |
| fulecorafa | Thanks for the help guys! | 21:53 |
| JayF | Mine is easy and simple: just wanted to draw attention to https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/XJUTKK5ZVWEO6IFHVQ3YSH6S5HQE6NEQ/ | 21:53 |
| JayF | and request on behalf of VMT you audit your security contacts in case we need to contact someone about a security vuln | 21:53 |
| JayF | we've discovered many teams had hideously out of date groups so I'm asking folks to check | 21:53 |
| JayF | that's all but I'm happy to answer questions if you've got 'em o/ | 21:54 |
| timburke | JayF, thanks for the heads-up! there are definitely some folks listed that i wouldn't expect to be terribly active in resolving any security issues, but all the people that should be there are | 21:55 |
| JayF | alright, I'd suggest removing those folks so they aren't accidentally made privvy to secrets they don't need to know | 21:56 |
| JayF | but thanks for checking :D | 21:56 |
| timburke | i probably ought to add a topic to for the PTG: "review core membership" | 21:57 |
| JayF | Yeah. I've suggested in Ironic community that we more aggressively add cores. It's easy to revert an accidentally-merged change; it's hard to onboard someone if the core team dwindles too low. | 21:58 |
| timburke | JayF, so the coresec members are a subset of the overall cores, i take it? | 21:59 |
| JayF | I don't think it has a universal meaning other than "the first place VMT looks for humans to review security issues if they don't have a security liason" | 21:59 |
| JayF | and given our old security liason tracking page is in the wiki and is about 5 years outta date | 22:00 |
| JayF | you can see why I take the approach I am :D | 22:00 |
| timburke | i was thinking more specifically about "what does ironic do?" and "is that maybe what *we* should do?" | 22:01 |
| timburke | i suppose whoever's listed there can specifically bring in other subject-matter experts if/as needed to a given security bug | 22:02 |
| timburke | all right, we're a bit past time | 22:03 |
| timburke | thank you all for coming, and thank you for working on swift! | 22:03 |
| timburke | #endmeeting | 22:03 |
| opendevmeet | Meeting ended Wed Oct 2 22:03:26 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2024/swift.2024-10-02-21.00.html | 22:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2024/swift.2024-10-02-21.00.txt | 22:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2024/swift.2024-10-02-21.00.log.html | 22:03 |
| JayF | Ironic just has coresec set to 4 of our longest term cores | 22:03 |
| JayF | and realistically, I'm on VMT | 22:03 |
| JayF | so that's a bit of a cheat for us lol | 22:03 |
| rwmtinkywinky4 | fulecorafa: have DMed with my terrible code for cleaning containers out, but nothing particularly magic about it | 22:04 |
| fulecorafa | Thanks a lot rwmtinkywinky4 | 22:06 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Delete s3api MPU segments when expiring the manifest https://review.opendev.org/c/openstack/swift/+/800701 | 23:08 |
| opendevreview | Jianjian Huo proposed openstack/swift master: DNM expirer: new options to control task iteration https://review.opendev.org/c/openstack/swift/+/914713 | 23:08 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Configurable expiring_objects_task_container_per_day https://review.opendev.org/c/openstack/swift/+/920452 | 23:08 |
| opendevreview | Jianjian Huo proposed openstack/swift master: refactor: remove iter_task_accounts_to_expire https://review.opendev.org/c/openstack/swift/+/919639 | 23:08 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Parallel distributed task container iteration https://review.opendev.org/c/openstack/swift/+/918366 | 23:08 |
| timburke | whee! sure enough, looks like message framing issues in #2081103 | 23:55 |
| patch-bot | https://bugs.launchpad.net/swift/+bug/2081103 - s3api: Deleting the current version of an object can (sometimes?) 500 (Confirmed) | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!