| opendevreview | David Hauer proposed openstack/python-swiftclient master: make setup dependencies discoverable https://review.opendev.org/c/openstack/python-swiftclient/+/904547 | 04:42 |
|---|---|---|
| opendevreview | David Hauer proposed openstack/python-swiftclient master: Remove duplicate script entry leading to broken wheel build https://review.opendev.org/c/openstack/python-swiftclient/+/904548 | 05:17 |
| opendevreview | ASHWIN A NAIR proposed openstack/python-swiftclient master: Support part-num in python swiftClient https://review.opendev.org/c/openstack/python-swiftclient/+/902020 | 17:05 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: slo: part-number=N query parameter support https://review.opendev.org/c/openstack/swift/+/894570 | 19:17 |
| opendevreview | Clay Gerrard proposed openstack/swift master: s3api: Support GET/HEAD request with ?partNumber https://review.opendev.org/c/openstack/swift/+/894580 | 19:19 |
| opendevreview | Clay Gerrard proposed openstack/swift master: s3api: unify enforcement of configured max part num https://review.opendev.org/c/openstack/swift/+/904234 | 19:19 |
| opendevreview | Alistair Coles proposed openstack/swift master: sq: don't lie about max part number constraint https://review.opendev.org/c/openstack/swift/+/904497 | 19:19 |
| opendevreview | Alistair Coles proposed openstack/swift master: Delete s3api MPU segments when the manifest expires https://review.opendev.org/c/openstack/swift/+/800701 | 19:20 |
| opendevreview | Alistair Coles proposed openstack/swift master: s3api: add API support for optional bucket inventory https://review.opendev.org/c/openstack/swift/+/835938 | 19:20 |
| opendevreview | Merged openstack/swift master: Document allowed_digests for formpost middleware https://review.opendev.org/c/openstack/swift/+/904325 | 20:18 |
| kota | good morning and a happy new year | 20:58 |
| acoles | kota: happy new year | 21:00 |
| kota | acoles: o/ | 21:01 |
| acoles | I spoke to timburke earlier - he's done with jury duty and was planning to be here | 21:03 |
| acoles | who else is here for the swift meeting? | 21:04 |
| jianjian | happy new year | 21:04 |
| acoles | jianjian: o/ | 21:04 |
| jianjian | me :-) | 21:04 |
| kota | o/ | 21:04 |
| acoles | ok let's make a start | 21:05 |
| acoles | #startmeeting | 21:05 |
| opendevmeet | acoles: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 21:05 |
| acoles | #startmeeting swift | 21:05 |
| opendevmeet | Meeting started Wed Jan 3 21:05:14 2024 UTC and is due to finish in 60 minutes. The chair is acoles. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:05 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:05 |
| opendevmeet | The meeting name has been set to 'swift' | 21:05 |
| acoles | I did volunteer to chair in Tim's absence. I've missed a couple of previous meetings, but I can at least offer some updates on stuff I've been involved with recently | 21:06 |
| timburke | oh! sorry -- lost track of time | 21:06 |
| acoles | the agenda is here https://wiki.openstack.org/wiki/Meetings/Swift | 21:06 |
| acoles | aha! welcome back timburke | 21:07 |
| timburke | i'll let you do your updates, though ;-) | 21:07 |
| acoles | timburke: can you give us an update on eventlet? I'll let you catch your breath though and start with s3api partNumber support | 21:08 |
| timburke | 👍 | 21:08 |
| acoles | #topic s3api partNumber | 21:08 |
| acoles | we've been working on this for a while but feel like it's very close to being done | 21:09 |
| acoles | #link https://review.opendev.org/c/openstack/swift/+/894580 | 21:09 |
| acoles | recap: the partNumber API feature allows clients to GET or HEAD parts of a multipart upload | 21:10 |
| acoles | swift implements multipart uploads (MPUs) using SLO, so first we need to add support to SLO for GET and HEAD using a part number https://review.opendev.org/c/openstack/swift/+/894570 | 21:11 |
| acoles | #link https://review.opendev.org/c/openstack/swift/+/894570 | 21:11 |
| acoles | then the s3api takes advantage of the new feature in SLO | 21:11 |
| acoles | along the way we refactored the s3api unit tests to better separate the s3acl True/False test cases | 21:12 |
| jianjian | quite a patch, +1261 -40. one question, how do users know how many parts in one SLO object? before they issue s3api partnum HEAD/GET requests. | 21:13 |
| acoles | there's a few pieces still to polish - s3 restricts the maximum nmber of parts to 10000, but swift made it configurable and it defaults to 1000, so we've tried to do the right thing in terms of matching error messages that state what the partNumber range should be https://review.opendev.org/c/openstack/swift/+/904497 | 21:14 |
| acoles | jianjian: the s3api MPU etag includes the part number count, something like <etag>-<part count>, so clients can infer the number of parts from the etag | 21:15 |
| timburke | jianjian, fwiw i've seen s3 libraries do a GET with ?partNumer=1, then look at the response headers to see how many more concurrent download threads to kick off (one per part) | 21:15 |
| jianjian | i see, thanks. | 21:16 |
| acoles | unfortunately SLO does NOT have expose its part count in general (so only manifests uploaded via s3api have this form of etag) | 21:16 |
| acoles | but we could expose the part count from SLO as metadata in future (just not for existing objects IIUC) | 21:17 |
| acoles | ok, that's probably enough said on that, watch this space for a merge soon 🤞 | 21:19 |
| jianjian | okay, this is mainly for s3api | 21:19 |
| acoles | #topic container server GET namespaces | 21:20 |
| acoles | another set of patches that are hopefully near completion! | 21:20 |
| acoles | the patch chain starts with https://review.opendev.org/c/openstack/swift/+/890470 but we're planning to squash patches before merging | 21:21 |
| acoles | #link https://review.opendev.org/c/openstack/swift/+/890470 | 21:21 |
| acoles | for a while we have been using a more compact representation of shards when the proxy is caching them in memcache - we called this a Namespace - just has a name and lower/upper bounds\ | 21:22 |
| acoles | this next step is to use the compact Namespace format when the proxy fetches shard ranges from the container server | 21:23 |
| jianjian | yeah, the format stored in memcache only has name and lower | 21:23 |
| acoles | right, even more compact when serialized into memcache | 21:23 |
| acoles | so the container server GET grows support for a X-Backend-Record-Shard-Format header that can take value 'namespace', which will cause the container server to return a list of ... guess what... Namespaces :) | 21:25 |
| acoles | and the proxy changes to request this namespace format *when it wants shard ranges that will be cached*, since it only needs the compact form | 21:25 |
| jianjian | one note on the performance improvement, for a very large container which has >10K shards, a single query of all shard ranges into container DB will take more than 300 ms, but a query of all namespaces will only take about 80ms. | 21:27 |
| acoles | yes, so we've seen significant improvement in container GET request times with this change already in prod , where we have some very large containers with many shards | 21:28 |
| timburke | i'm starting to wonder if we should *require* shard range caching -- i seem to remember OVH noticing a regression in the no-cache path at one point; i don't think we have any gate tests that really functionally cover it... | 21:28 |
| acoles | timburke: the no cache path worries me a little - I do sometimes set the cache existence times to zero in my SAIO for probe tests, but not routinely I confess | 21:29 |
| acoles | BTW, that's the only way I know to exercise "no cache" path - you gotta have memcache in the pipeline for auth to work | 21:30 |
| timburke | yeah, that's part of my thinking... though i suppose i could push on https://review.opendev.org/c/openstack/swift/+/861271 more to get fernet token support into tempauth | 21:31 |
| acoles | the last patch in the chain (which may remain a separate patch) cleans up the proxy container GET path a lot, which may help reason about the two paths https://review.opendev.org/c/openstack/swift/+/901335 | 21:31 |
| acoles | #Link https://review.opendev.org/c/openstack/swift/+/901335 | 21:31 |
| acoles | hmmm, it'll be interesting to consider how we would enforce "sharding requires memcache", if that's what you mean | 21:32 |
| timburke | i mean, we already nearly require memcache for account/container info -- if you don't have it, you're going to be incredibly sad | 21:33 |
| timburke | my thinking is mainly to formalize the dependency on memcache more | 21:33 |
| jianjian | good point, we should do the same thing to shard range caching, it's much more expansive | 21:35 |
| jianjian | agree with acoles, 901335 can be a separate patch | 21:36 |
| acoles | remind, does zuul functional testing use a setup with memcache? | 21:37 |
| timburke | iirc the dsvm jobs at least do; i think maybe the in-process tests use our FakeMemcache? i'm not actually entirely sure... | 21:38 |
| timburke | they must have *some* way of storing/retrieving tokens... | 21:38 |
| timburke | sure enough, FakeMemcache: https://github.com/openstack/swift/blob/master/test/functional/__init__.py#L110-L118 | 21:39 |
| acoles | ok, maybe we can explore that more | 21:39 |
| acoles | let's move on? | 21:39 |
| acoles | #topic eventlet | 21:40 |
| clayg | https://github.com/NVIDIA/swift/blob/master/tools/playbooks/saio_single_node_setup/setup_saio.yaml#L114 | 21:40 |
| timburke | so, there's been a bunch of eventlet activity lately | 21:40 |
| timburke | among other things, including adding support for py310-py312! | 21:41 |
| timburke | which is all well and good, except when the powers that be went to update upper-constraints... our tests were failing with it | 21:41 |
| timburke | #link https://review.opendev.org/c/openstack/requirements/+/904147 | 21:42 |
| timburke | the fun thing, though, is that it wasn't a normal failure; it was a segfault! | 21:42 |
| timburke | i wrote up a bug for eventlet | 21:42 |
| timburke | #link https://github.com/eventlet/eventlet/issues/864 | 21:42 |
| timburke | and they wrote up a bug for cpython (since i didn't have time to get to that right then) | 21:43 |
| timburke | #link https://github.com/python/cpython/issues/113631 | 21:43 |
| timburke | but the response there was mostly "well, don't do that, then" | 21:43 |
| jianjian | haha | 21:44 |
| timburke | the good news is, there seems to be a new approach to greening existing locks for eventlet | 21:44 |
| timburke | #link https://github.com/eventlet/eventlet/pull/866 | 21:44 |
| timburke | i'm going to try to put it through its paces, but so far it seems promising | 21:45 |
| timburke | but it might have some implications for when we want to monkey-patch -- i remember us wanting to stop monkey-patching as an import side-effect, but we might need to | 21:46 |
| timburke | the segfault in our unit tests was also independently reported | 21:46 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/2047768 | 21:46 |
| timburke | and since there are published eventlet packages out there that can trip the segfault, it seems like we probably want to do what we can to work around it | 21:47 |
| timburke | fortunately, i think there's just one spot where we actually for-realsy monkey-patch in tests, and if we mock it out, no segfault (at least, OMM) | 21:48 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/904459 | 21:48 |
| acoles | timburke: "one spot where we actually for-realsy monkey-patch in tests" ... sorry, I'm probably being dumb, but how do we not do the monkey-patch in other unit tests | 21:50 |
| acoles | ? | 21:50 |
| jianjian | is logging the only place in swift which use locks? if we can safely switch to no lock logging, maybe we don't need greening lock anymore | 21:50 |
| timburke | acoles, i'm not actually entirely sure -- i think other tests involving run_wsgi etc. are exercising the error-checking before monkey-patching? or something. maybe i just got lucky | 21:52 |
| timburke | jianjian, there's definitely another lock in (unittest.)mock -- that's the one that started tripping the segfault. not sure about other places there may be locks lurking | 21:53 |
| jianjian | i see | 21:53 |
| timburke | on the whole, though, i think it's yet more argument for us finding a way to get off eventlet -- writing cooperative code as though it's blocking doesn't seem long-term sustainable | 21:54 |
| acoles | ok, thanks for the update timburke , and for keeping tabs on eventlet! | 21:55 |
| timburke | or at least get off of monkey-patching? maybe if we were more explicit about things, it'd be better... though at that point, it'd probably be about the same effort to rewrite it all with asyncio | 21:56 |
| acoles | last I read the community's long term plan is to move away from eventlet right? | 21:57 |
| jianjian | probably so, at least to get off of monkey-patching | 21:57 |
| timburke | yep -- and eventlet's long-term plan looks to be to move to some legacy/unmaintained status | 21:58 |
| acoles | we have a couple of minutes, so... | 21:58 |
| acoles | #topic open discussion | 21:58 |
| acoles | anything else to discuss? | 21:58 |
| timburke | i think i'll spend a bit of time poking at the py312 test failures next | 21:58 |
| acoles | seems like we're done, so as is traditional... | 22:00 |
| acoles | thank you for coming and thank you for working on swift! | 22:00 |
| acoles | and happy new year | 22:00 |
| acoles | #endmeeting | 22:00 |
| opendevmeet | Meeting ended Wed Jan 3 22:00:37 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2024/swift.2024-01-03-21.05.html | 22:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2024/swift.2024-01-03-21.05.txt | 22:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2024/swift.2024-01-03-21.05.log.html | 22:00 |
| timburke | thanks for runnign the meeting acoles! sorry again for being late | 22:00 |
| acoles | timburke: NP, better late than never (even more appropriate in a world of eventual consistency :D ) | 22:02 |
| indianwhocodes | auditing the whole meeting rn, sorry i missed the hour slot, happy new year everyone! | 22:06 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: This change unifies the enforcement of the configured maximum part number using a single method to validate partNumber. While AWS has a limit of 10000 parts, Swift's limit is configurable and defaults to 1000. https://review.opendev.org/c/openstack/swift/+/904234 | 22:31 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: s3api: unify enforcement of max part number and handle exceptions https://review.opendev.org/c/openstack/swift/+/904234 | 22:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!