| opendevreview | Merged openstack/swift-bench master: Remove nose detritus https://review.opendev.org/c/openstack/swift-bench/+/943029 | 04:59 |
|---|---|---|
| opendevreview | Matthew Oliver proposed openstack/swift master: Use SO_TIMESTAMP to track request times before they hit accept queue https://review.opendev.org/c/openstack/swift/+/944103 | 11:14 |
| clayg | timburke_: DHE: I assume we'd still only talk to handoffs when primaries miss. A POST would still generally (202, 202, 404) => but for failure-during-rebalance I could see a (ConnectionTimeout, 202, 404, 202) being better than (ConnectionTimeout, 202, 404, 404) - For read-stale-after-delete I think would hope the x-backend-timestamp from the tombstone on the primary could somehow surpress the 2XX from the stale primary | 14:08 |
| clayg | inserted into the handoffs list; but maybe we don't do that. Again, I think there's code we start adding tests to somewhere... | 14:08 |
| clayg | maybe start with a rebase 834621: ring: Add a rebalance history in the ring | https://review.opendev.org/c/openstack/swift/+/834621 | 14:08 |
| DHE | most likely, sure, you assume that your 3 good copies are probably fine. but immediately after a rebalance that is false. having the 3rd copy at least in the handoffs list should improve reliability at least a little bit... maybe making it the first handoff isn't the right answer... | 14:48 |
| timburke_ | idk, sticking an old primary in as first handoff seems like the right sort of trade-off to make, especially for a AP system. oh! what about this: stick the old primary in as first handoff on reads, and as an extra primary on (replicated) writes! | 15:25 |
| timburke_ | just need to make sure it doesn't impact the quorum requirement | 15:26 |
| DHE | I like that.. on reads you're likely to find an expected copy quickly. on writes, it's best to keep handoffs still as spread out as possible across regions/zones which is what get_more_nodes() does.. | 15:31 |
| timburke_ | continuing to write to old primaries should also reduce unnecessary rsync traffic (where today we rsync the old data with everything else and let cleanup_ondisk_files unlink it) | 15:47 |
| timburke_ | would probably want a config option for that over-replicate-on-write behavior, though -- capacity crunches are no fun | 17:29 |
| opendevreview | Alistair Coles proposed openstack/swift master: sq: move CRCHasher to checksum.py https://review.opendev.org/c/openstack/swift/+/944146 | 18:43 |
| opendevreview | Alistair Coles proposed openstack/swift master: sq? S3Request checksumming suggestions https://review.opendev.org/c/openstack/swift/+/944147 | 18:43 |
| -opendevstatus- NOTICE: One of our Zuul job log storage providers is experiencing errors. We have removed that storage target from base jobs. You should be able to safely recheck changes now. | 20:24 | |
| kota | good morning | 20:57 |
| timburke_ | o/ | 21:00 |
| timburke_ | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Mar 12 21:00:33 2025 UTC and is due to finish in 60 minutes. The chair is timburke_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke_ | who's here for the swift meeting? | 21:00 |
| kota | o/ | 21:00 |
| mattoliver | o/ | 21:01 |
| timburke_ | sorry about missing last week; time just got away from me | 21:01 |
| timburke_ | on that note, i also haven't updated the agenda 😬 | 21:02 |
| timburke_ | but we can still have some updates! | 21:02 |
| timburke_ | #topic PTG | 21:02 |
| acoles | o/ | 21:02 |
| timburke_ | reminder that the ptg is less than a month away | 21:02 |
| mattoliver | woo! need to get topics filled out then! | 21:03 |
| timburke_ | #link https://etherpad.opendev.org/p/swift-ptg-flamingo | 21:03 |
| timburke_ | we also need to choose meeting times. this time i figured, rather than list each individual hour, let's just list the blocks and we'll pick one per day that seems to work fairly well. if you can't make the whole block, that's fine; but those tables got too wide before | 21:05 |
| timburke_ | https://framadate.org/LBfzMX2I4C1JL9cZ | 21:05 |
| timburke_ | #link https://framadate.org/LBfzMX2I4C1JL9cZ | 21:05 |
| mattoliver | kk | 21:05 |
| timburke_ | and if you haven't registered already, please do | 21:06 |
| jianjian | got it | 21:06 |
| timburke_ | #link https://ptg.openinfra.dev/ | 21:06 |
| timburke_ | i look forward to seeing you there! | 21:06 |
| timburke_ | #topic swift release | 21:06 |
| timburke_ | i merged the authors/changelog patch for 2.35.0 | 21:07 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/943073 | 21:07 |
| timburke_ | y'all did a lot of great work this past cycle! | 21:07 |
| timburke_ | i should really try to get more than one release out per cycle :-/ | 21:08 |
| timburke_ | and there's a releases patch up to create the tag, though they're pushing for a major version bump | 21:09 |
| timburke_ | #link https://review.opendev.org/c/openstack/releases/+/943880 | 21:09 |
| timburke_ | we'll see if i can convince them that actually very little has changed, despite our removing py2 gating | 21:09 |
| jianjian | let us know if you need support by posting comments from us 😉 | 21:10 |
| timburke_ | otherwise there'll be another quick patch to update the version in the changelog. i'd be inclined to go to 4.0.0 (to avoid confusion with the swift3 project) | 21:10 |
| kota | :D | 21:11 |
| timburke_ | next up | 21:11 |
| timburke_ | #topic aws-chunked support | 21:11 |
| acoles | timburke_: thanks as always for all your work on the release etc | 21:11 |
| mattoliver | +1 | 21:12 |
| timburke_ | our (nvidia's) users have started to get used to the fact that boto3 doesn't work with default configs -- which isn't really where we want to be | 21:12 |
| timburke_ | makes me think of something like "there's nothing more permanent than a temporary solution" | 21:13 |
| acoles | lol | 21:13 |
| timburke_ | so we've been working on getting the right balance between adding support for aws-chunked, performing all the validation that clients are requesting, but not necessarily taking on all the work required to add full additional-checksums support | 21:15 |
| timburke_ | latest attempt looks like | 21:15 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/944073 | 21:15 |
| timburke_ | where the idea is just to validate that the bytes received match the checksum provided during PutObject and UploadPart -- but we won't write any of it down, and we won't validate them during CompleteMultipartUpload | 21:17 |
| timburke_ | acoles has already started to look further down the chain (thanks!) and i'll try to get some of his follow-ups squashed in soon | 21:18 |
| mattoliver | oh thats a good idea, so at least we're doing something with the given crc rather then just ignoring it. | 21:19 |
| acoles | I was thinking a bit more about that "not writing anything down part...that will mean that if/when we do support including crcs in the object attributes API thing, then there'd forever be some objects that we can't report the values. I do NOT want to scope creep what we're doing, but we may want to consider how hard it would be to stuff the value into sysmeta so it could be retrieved by a HEAD in future. | 21:21 |
| timburke_ | acoles, but that's no different than if they'd uploaded the object without any checksums, right? i'd be hesitant to pretend to know what we want to write down until we're actually willing to return it | 21:22 |
| acoles | I'm not so worried about ListParts because that is transient, once an MPU completes we don't need the crc value | 21:22 |
| acoles | sure its no different than an upload with no crc. But if client gave us a crc and we checked it then we might one day want to return it to the client via some API? or am I misunderstanding? | 21:23 |
| acoles | I should perhaps go and study the object attributes s3 api docs some more ! | 21:24 |
| acoles | or even play with it | 21:25 |
| timburke_ | sure -- but as a client i wouldn't expect any of my uploads to get that retroactively -- once i see the feature's in, then i'd want it to work, but not for anything that predates it | 21:25 |
| acoles | I guess that's a reasonable position. I just thought I'd flag up ...like if it proves to be low hanging fruit (except my alter-ego will scream "even low hanging fruit requires more tests etc" ) ;-) | 21:26 |
| timburke_ | yeah -- and the full-object vs composite checksums thing makes me worry about what all we need to write down | 21:27 |
| acoles | oh yeah, _that_ | 21:28 |
| acoles | ok, let's stay focussed! | 21:29 |
| timburke_ | somewhat related, i know clayg has noted some of the strange read-a-byte-then-return-an-error in this stuff -- we might want to revive https://review.opendev.org/c/openstack/swift/+/799866 sometime soon, make it so we can tell eventlet to close a connection without bothering to discard | 21:29 |
| timburke_ | #topic ring v2 | 21:30 |
| timburke_ | there's been a bit of back and forth in IRC lately about what we can do with an old-primaries table | 21:31 |
| timburke_ | DHE heard about our plan to insert old primaries at the start of the handoff list for reads | 21:32 |
| timburke_ | and i'd previously been thinking about how we might want it for at least *some* writes as well (POSTs in particular) | 21:33 |
| timburke_ | DHE had a concern about how we might return out-dated data -- which is always a bit of a risk with Swift, but it does seem like the current plan would increase that risk | 21:35 |
| mattoliver | But also the ability to just grab an old hand off if it exists. To break deadlocks. | 21:36 |
| timburke_ | so now i had a thought about adding the old primary to the start of handoffs for reads (like we've been planning) but *also* add it to the set of *primaries* for writes, causing some over-replication | 21:36 |
| mattoliver | oh interesting, so we over provision new writes. If there is an old primary. And if there's not, same behaviour as today? | 21:38 |
| timburke_ | yup | 21:39 |
| timburke_ | it should avoid DHE's worry and reduce some unnecessary rsync traffic (where the old primary would sync data just so the remote could delete it), but at a risk of not digging out from full-disks fast enough | 21:39 |
| timburke_ | so we'd probably want to make it configurable | 21:39 |
| timburke_ | all right, that's all i've got on my mind | 21:40 |
| timburke_ | #topic open discussion | 21:40 |
| timburke_ | what else should we talk about this week? | 21:40 |
| kota | I'll be absent the next week due to attendance of GTC. | 21:41 |
| timburke_ | we could try to meet up -- sorry, i think i dropped the ball on that last time you were in town | 21:44 |
| mattoliver | oh cool, GTC! | 21:44 |
| kota | np, thx. | 21:45 |
| mattoliver | I'm playing with adding the SO_TIMESTAMP socket option to swift so we can see how long a request was sitting on the accept queue. https://review.opendev.org/c/openstack/swift/+/944103 | 21:45 |
| mattoliver | In my basic socket script it works all the time. I've put it into Swift and it works sometimes :( | 21:46 |
| timburke_ | curious that it should only be sometimes :-/ how does it fail? | 21:47 |
| mattoliver | Other times I get resource not avaialbe. Using eventlet / green sockets in my script all work. But still might be something eventlet. Still playing with it to figure out what I'm missing. There are some flags to force ti to wait and to PEEK at the data. So trying some of those. | 21:47 |
| mattoliver | the socket.recvmsg() to get the ancdata just throws a [Errno 11] Resource temporarily unavailable | 21:48 |
| mattoliver | but other times it works and returns the ancdata. | 21:48 |
| mattoliver | And I'm just running the same request (swift stat) over and over. | 21:49 |
| mattoliver | Maybe something something eventlet pools and socket not available. | 21:49 |
| mattoliver | I thought maybe the socket didn't have data yet and is non-blocking.. but there is a MSG_WAITALL flag I can use but that doesn't seem to fix it. | 21:50 |
| timburke_ | curious. when you're trying it in your script, is it with greened sockets, or stdlib? | 21:50 |
| mattoliver | I started with stdlib, but moved to using eventlet.listen and greened sockets like we do in Swift. All good | 21:51 |
| timburke_ | huh. strange | 21:51 |
| mattoliver | But that's still a single "thread" so maybe I need to try firing off a bunch of threads, ie a pool | 21:52 |
| mattoliver | I did try a sleep(0) just in case I'd loose context (just trying anything atm). | 21:52 |
| mattoliver | The code I pushed last night just pass the exception, but just put in a self.log_message(f"=== Eception {str(ex)} ===") and you'll see the error | 21:53 |
| timburke_ | i wonder if with the non-blocking sockets we could get to a point where accept could return, but not all the ancdata is available yet | 21:53 |
| mattoliver | yeah, true. | 21:54 |
| mattoliver | eitherway mostly good news, just need to figure out this. | 21:54 |
| mattoliver | Also I seemed to have issues using recvmsg after we socket.makefile... but might recheck that because I did try a billion things last night, so could've been a red herring | 21:55 |
| mattoliver | Anyway, that's all I got for now. | 21:55 |
| timburke_ | hmm... i wonder if you could use that to back-date some spans for your otel work... would be interesting if we could see how long the request was sitting in the accept queue with that | 21:56 |
| timburke_ | all right, i think we can wrap up then | 21:58 |
| mattoliver | oh yeah! | 21:58 |
| mattoliver | well once I get this working, I'll add that to otel tracing ;) | 21:58 |
| timburke_ | 👍 | 21:58 |
| timburke_ | thank you all for coming, and thank you for working on swift! | 21:58 |
| timburke_ | #endmeeting | 21:58 |
| opendevmeet | Meeting ended Wed Mar 12 21:58:51 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2025/swift.2025-03-12-21.00.html | 21:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2025/swift.2025-03-12-21.00.txt | 21:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2025/swift.2025-03-12-21.00.log.html | 21:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!