| opendevreview | Alistair Coles proposed openstack/swift master: s3api: Add support for additional checksums https://review.opendev.org/c/openstack/swift/+/909801 | 11:05 |
|---|---|---|
| opendevreview | Alistair Coles proposed openstack/swift master: s3api: Additional checksums for MPUs https://review.opendev.org/c/openstack/swift/+/909802 | 11:05 |
| opendevreview | Alistair Coles proposed openstack/swift master: Add support of Sigv4-streaming https://review.opendev.org/c/openstack/swift/+/836755 | 14:43 |
| opendevreview | Clay Gerrard proposed openstack/swift master: sq: add test for parse_error https://review.opendev.org/c/openstack/swift/+/939828 | 16:59 |
| opendevreview | Tim Burke proposed openstack/swift master: splice: Stop dipping down to C get MD5 sockets https://review.opendev.org/c/openstack/swift/+/939912 | 20:54 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Jan 22 21:00:34 2025 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| rcmgleite | I am! | 21:00 |
| mattoliver | o/ | 21:01 |
| * zaitcev peeks in | 21:02 | |
| timburke | so i've had a crazy weekend due to moving houses and haven't really had a chance to update the agenda (or think much about what i'd want to call out during this meeting in general) | 21:02 |
| mattoliver | nps, we'll wing it :) | 21:03 |
| timburke | but i *do* know that there are a couple items of work that are seeing renewed interest: ring v2, and the aws-chunked chain | 21:03 |
| rcmgleite | Is there documentation about what ring v2 is? (sorry, new here) | 21:04 |
| timburke | oh, and p 853697 merged, so a whole bunch of stuff has needed rebasing! | 21:04 |
| timburke | rcmgleite, yes! it's included in the patch: https://review.opendev.org/c/openstack/swift/+/834261 | 21:05 |
| mattoliver | yup, both are important for us at least. the latter is important for any s3api users with the latest boto sdk updates | 21:05 |
| timburke | oh, right, patch-bot is still packed up... i've gotta bring some hardware back online so i've got a bouncer and whatnot... | 21:05 |
| mattoliver | ringv2 is a new serialization format for rings. The old had some issues, but there are aslo cool new imporovements like can store more devices in the ring (Which is what we really need). | 21:06 |
| mattoliver | oh, you're running patchbot yourselve timburke lol | 21:06 |
| mattoliver | *yourself | 21:06 |
| timburke | the check job also published a preview sight of the new docs; i think https://df7f3296a8ac461f2ab3-3dd5e628e5b815e2df15fc764c3bb478.ssl.cf2.rackcdn.com/834261/22/check/openstack-tox-docs/91194ff/docs/overview_ring_format.html#ring-v2 is the interesting bit | 21:07 |
| timburke | but yeah, mattoliver's right -- the big thing driving the renewed interest is going beyond the 64k device limit rings currently have | 21:09 |
| timburke | rcmgleite, welcome! what brings you here? any particular part of swift you're interested in? | 21:09 |
| rcmgleite | Hey! I'm an ex s3 engineer working for a public cloud in a brazilian company. Our object storage offer runs on swift (still yoga for now) | 21:10 |
| rcmgleite | I'm here to learn and if possible ask lots of questions! | 21:11 |
| mattoliver | oh nice!! welcome! | 21:11 |
| jianjian | nice! o/ | 21:11 |
| opendevreview | Merged openstack/swift master: docs: Fix version call-out for stale_worker_timeout https://review.opendev.org/c/openstack/swift/+/939501 | 21:11 |
| jianjian | welcome | 21:11 |
| rcmgleite | Thank you! | 21:11 |
| rcmgleite | Currently very interested in the 64k limit change of the ring (also a problem for us). and we've been working on some patches of our own that we would like to push upstream if it makes sense to you | 21:12 |
| rcmgleite | Things like object locking, middlewares that help decouple keystone from swift etc.. | 21:13 |
| timburke | oh, cool! welcome! i seem to remember notmyname talking about at least one brazilian company with a pretty large swift install... | 21:13 |
| rcmgleite | that's us for sure! Rafael (also from my company) has been joining this meeting for a while now | 21:13 |
| mattoliver | cool, more people working upstream the better! then we can all benefit from each others work | 21:14 |
| rcmgleite | @mattoliver exactly! | 21:15 |
| rcmgleite | Do you folks mind If I ask a couple of questions? Not sure how these meetings usually go | 21:15 |
| timburke | not at all, go for it! you're also welcome to ask questions any time, not just during meetings :-) | 21:16 |
| zaitcev | rcmgleite: https://gist.github.com/tipabu/435fc63f1aa76d6b346956571a393665 | 21:16 |
| rcmgleite | We are working on upgrading from yoga forward.. our first step is Zed. Anything we should be worried about with these upgrades? | 21:16 |
| rcmgleite | By reading the commits and changelog it doesn't look like so.. but wanted to be sure | 21:17 |
| timburke | no, i wouldn't expect. and fwiw, i wouldn't hesitate to upgrade swift straight from yoga to 2.34.0 (or even master) -- though you might have your own reasons for moving more slowly | 21:18 |
| rcmgleite | I think I tend to be overly paranoid... And we have a bunch of changes to swift that we never pushed upstream that get very annoying to merge when pulling new changes.. | 21:19 |
| timburke | if possible, i'd upgrade object, then container, then account, then proxy | 21:19 |
| timburke | yeah, fair enough. long-lived forks are no fun | 21:19 |
| rcmgleite | Yeah we are trying to catch up and send proper proposals so that we don't keep our fork anymore.. | 21:20 |
| mattoliver | thats why we try to push as much upstream as possible.. less maintence burden | 21:20 |
| mattoliver | \o/ | 21:20 |
| rcmgleite | is there a rational for having to choose a specific order to upgrade? | 21:20 |
| jianjian | I remember I have working on a few expirer related changes recently, and one of the goal is to be order-agnostic for the upgrade process. I think probably Tim wants to be cautious in general. | 21:23 |
| timburke | servers are kept backward compatible with older clients (at least, as best we can, and to the extent that we don't, we treat it as a bug and fix it) | 21:24 |
| timburke | so older proxy should always be able to talk to newer backend | 21:24 |
| timburke | but newer proxy might want to use features only available with newer backend | 21:24 |
| mattoliver | alot of the smarts and features are in the proxy, they can talk to older storage nodes. But it's just best practice as Jian says. And meansyou can use new features as sson as the proxies are upgraded :) | 21:24 |
| rcmgleite | oh interesting. that makes sense! Thanks for the context | 21:25 |
| timburke | the exact order of the backend upgrades is less important, but we usually try to think about them in that order (iirc) | 21:26 |
| timburke | i think for the most part it's less of a concern these days; it's been a while since we added a bit new protocol change like sysmeta or listing overrides... | 21:27 |
| rcmgleite | Second question: we want to update a bit how ranged gets work -> ie: we want storage nodes to also receive a range and be able to fetch only the exact amount of data needed from disk. I wanted to share the design with you and have proper discussions before we go full on implementing it. What's the best way to do this? Do you discuss in PR direclty? Or have a design session? | 21:27 |
| rcmgleite | Great! Thanks guys! | 21:27 |
| timburke | rcmgleite, that should already be happening -- what've you been seeing? | 21:28 |
| rcmgleite | (it's more of a general question about how designs are done within this group/project) | 21:28 |
| mattoliver | We tend to have a design doc in any form you like, we put it on the ideas page (normally), and can discuss it at a meeting or PTG. Then break out into more meetings or a something as required. It's pretty flexible. | 21:28 |
| rcmgleite | We just started discussing about this - but since we don't have internal checksums per range, My assumption is that you have to read the entire content from disk to be able to apply integrity checks on whatever storage nodes are sending back | 21:29 |
| mattoliver | #link https://wiki.openstack.org/wiki/Swift/ideas | 21:29 |
| rcmgleite | Thanks @mattoliver! | 21:29 |
| rcmgleite | I might have missed how you do it though | 21:30 |
| mattoliver | basically want the idea (or more flushed idea/design doc) that people can read / comment on. and then we can take it from there. | 21:30 |
| rcmgleite | Cool! That makes sense! | 21:30 |
| mattoliver | if its a huge feature we sometime use a feature branch in git upstream to speed up development | 21:30 |
| mattoliver | storage_polices, sharding and currently a better MPU (then slo) is currrently being worked on in this manner | 21:31 |
| rcmgleite | That's interesting! I can find those by looking at branches on the github repo? | 21:32 |
| mattoliver | yeah, but your best place to see progress is on gerrit | 21:32 |
| rcmgleite | hm, ok! | 21:32 |
| mattoliver | I use this gerrit dashboard (sorry about the big link): | 21:33 |
| mattoliver | https://review.opendev.org/dashboard/?foreach=%28project%3Aopenstack%2Fswift+OR+project%3Aopenstack%2Fpython%2Dswiftclient+OR+project%3Aopenstack%2Fswift%2Dbench+OR+project%3Aopenstack%2Fpyeclib+OR+project%3Aopenstack%2Fliberasurecode%29+status%3Aopen+NOT+label%3AWorkflow%3C%3D%2D1+NOT+label%3ACode%2DReview%3C%3D%2D2+is%3Amergeable&title=Swift+Review+Dashboard&Starred+%28by+myself%29=%28is%3Astarred%29+AND+status%3Aopen&Small+things= | 21:33 |
| mattoliver | delta%3A%3C%3D25+limit%3A10+%28NOT+label%3ACode%2DReview%2D1%2Cswift%2Dcore+OR+%28label%3ACode%2DReview%2D1%2Cswiftclient%2Dcore+AND+project%3Aopenstack%2Fpython%2Dswiftclient%29%29&Needs+Final+Approval+%28to+land+on+master%29=NOT+label%3AWorkflow%3E%3D1+NOT+label%3AWorkflow%3C%3D%2D1+NOT+owner%3Aself+label%3ACode%2DReview%3E%3D2+%28NOT+label%3ACode%2DReview%2D1%2Cswift%2Dcore+OR+%28label%3ACode%2DReview%2D1%2Cswiftclient%2Dcore+AND+ | 21:33 |
| mattoliver | project%3Aopenstack%2Fpython%2Dswiftclient%29%29+branch%3Amaster&Open+Backport+Proposals=branch%3A%5Estable%2F.%2A+status%3Aopen&Feature+Branches=branch%3A%5Efeature%2F.%2A+status%3Aopen+NOT+branch%3A%5Efeature%2Fhummingbird&Open+patches=NOT+label%3AWorkflow%3E%3D1+NOT+label%3ACode%2DReview%2D1%2Cswift%2Dcore+NOT+label%3ACode%2DReview%3E%3D2+branch%3Amaster | 21:33 |
| mattoliver | thats an old one. I thnk tim has newer probably better ones :P | 21:35 |
| timburke | note that most of the feature branches on the github repo are basically historical artifacts -- the only active feature branch today is feature/mpu | 21:35 |
| zaitcev | I save those URLs in something like this: https://slasti.zaitcev.us/zaitcev/swift/ | 21:36 |
| zaitcev | Wait | 21:36 |
| zaitcev | http://slasti.zaitcev.us/zaitcev/swift/ | 21:36 |
| zaitcev | Private CA cert, sorry. | 21:36 |
| rcmgleite | great, thx! Anyone else want to jump in before I ask a last question? | 21:37 |
| mattoliver | nah go ahead | 21:37 |
| rcmgleite | Is there any fork/initiatives that you know of that are related to strong consistency? | 21:38 |
| rcmgleite | I know swift is eventually consistent by design... so not too hopeful | 21:38 |
| rcmgleite | but it's something we've been seeing the need for more and more... | 21:38 |
| timburke | yeah, we're seeing the need more and more, too... i don't recall any forks pursuing it, though | 21:40 |
| mattoliver | not that I'm aware of. At nvidia we've been thinking about what it might look like to get more strongly consistent in container listings for example. But the cost of stong consistency at scale is latency. Thought maybe we need to have container policies where you might be able to opt-in for a different consistency model. but more just spitballing still | 21:41 |
| mattoliver | could be a great topic at a future PTGs, when we an all brainstorm together. | 21:41 |
| zaitcev | Ceph RGW is consistent. But I don't like that code of it. I worked on it on and off. It's in an-hoc C++, everything is super random. Classes, indexes... | 21:42 |
| zaitcev | Minio maybe okay but it has licensing issues. | 21:42 |
| mattoliver | yeah, I worked on ceph RGW and yeah had latency issues because of metadata strong consistency (when I worked on it) | 21:42 |
| rcmgleite | It's great to hear that the need for that is also there for you.. we might bring up some ideas soon! | 21:43 |
| mattoliver | acoles: has some bucket inventory code we run downstream to get a more consistant although historic view of bucket listings. | 21:43 |
| mattoliver | but yeah, more we can all work together on problems we face the better we can solve them and make swift better :) | 21:44 |
| jianjian | @rcmgleite, what kind of consistency you are interested in, listing or obj overwrite? | 21:45 |
| rcmgleite | both. listing can be annoying for analytics workloads but overwrites are also bad for things like bucket policy updates, locking updates etc.. | 21:47 |
| rcmgleite | it's something customers don't expect - that you say: Lock my file for deletions, they get a 200 and the next call they do is a delete and their object is gone | 21:47 |
| rcmgleite | it can totally be something we could improve in our implementation of locking (which we very much want to push upstream) | 21:48 |
| rcmgleite | but it's one of the many cases where eventual consistency is biting us | 21:48 |
| timburke | yup, makes sense | 21:49 |
| mattoliver | rcmgleite: I look forward to digging into some of these issues and seeing what you all have done! | 21:50 |
| rcmgleite | Yup! let's do that! thanks for all the information folks! really helpful! | 21:51 |
| timburke | all right, anything else we want to discuss? | 21:52 |
| jianjian | look forward to seeing patches from you, @rcmgleite | 21:52 |
| mattoliver | As for ringv2 work, I've been pushing it down the road some more, got 2 more small patches. One to save 2byte minimum forr dev in builders | 21:52 |
| timburke | i saw, thanks mattoliver! | 21:53 |
| mattoliver | and I to add a `swift-ring-builder <ring> version` command, so we have an easy way to confirm ring versions, seeing as we haven't changed the file name semantics. | 21:53 |
| timburke | yeah, that might be better than what i was doing over in https://review.opendev.org/c/openstack/swift/+/857234 | 21:54 |
| mattoliver | just playing with the code in my vSAIO and trying to get used to using it day 2 day. | 21:54 |
| timburke | i saw https://github.com/NVIDIA/vagrant-swift-all-in-one/pull/170 too :-) | 21:54 |
| mattoliver | oh yeah, ring-info, so a new utility, could also work. | 21:54 |
| mattoliver | oh yeah, if you actaully want to build v2 rings, it would be nice for vSAIO to support it. | 21:55 |
| mattoliver | although that only works if your running the branch because before that --format-version isn't a valid option. | 21:56 |
| mattoliver | ring_version=1 could just not write the --format-version option I guess.. that'll work until we change the default version in the future :hmm: | 21:57 |
| timburke | i suppose i could try to pull that out as another patch in front -- and justify it as "well, you want to be able to test loading v0 rings, don't you?" :P | 21:57 |
| mattoliver | loll, true.. but surely we're just going to land ringv2 soon right ;) | 21:58 |
| timburke | i was so sad when i noticed the "Created 4 years ago" when zaitcev linked to my gist... | 21:59 |
| timburke | all right, we're about at time | 21:59 |
| zaitcev | Our velocity is a bit low. | 21:59 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:59 |
| timburke | #endmeeting | 21:59 |
| opendevmeet | Meeting ended Wed Jan 22 21:59:56 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2025/swift.2025-01-22-21.00.html | 21:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2025/swift.2025-01-22-21.00.txt | 21:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2025/swift.2025-01-22-21.00.log.html | 21:59 |
| mattoliver | well we were good to land I think almost a year or 2 ago.. and then things just stalled. | 22:00 |
| mattoliver | thanks timburke and everyone! nice to meet you rcmgleite | 22:00 |
| timburke | low velocity can have some value -- i get a little nervous sometimes about how much movement there's been in eventlet lately | 22:01 |
| rcmgleite | Thanks folks! see u next week! | 22:04 |
| rcmgleite | Now that I'm not intruding on the meeting - I have a question about ranged gets that I was unable to find in code - given that we have only the md5 of the full object, how do we validate integrity of the bytes from a ranged get at any of the swift layers? | 22:05 |
| zaitcev | I don't think you can. But auditor re-validates them constantly. So it should be okay with a certain probability. All that's left is the transmission over the network. | 22:07 |
| rcmgleite | yeah that's what I thought..I've been thinking about adding a block framing layer that would include checksums on every X bytes of the plain text data.. Will create a doc a run that by you folks later | 22:10 |
| rcmgleite | Thanks @zaitcev | 22:11 |
| zaitcev | Doesn't SLO do it already? | 22:11 |
| zaitcev | Unless you mean page-sized segments. | 22:12 |
| zaitcev | ttyl | 22:12 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily for a restart to put some database compaction config changes into effect, and will return within a few minutes | 22:55 | |
| timburke | rcmgleite, on ranged GETs -- it took me a bit to hunt down (i'd forgotten how it works myself), but the significant bit is around https://github.com/openstack/swift/blob/master/swift/common/swob.py#L1385-L1423 -- when we've got ranges, the object-server will find those app_iter_range and app_iter_ranges functions on the diskfile reader, allowing it to seek appropriately | 23:13 |
| timburke | that required a lot more digging than i expected -- i really *wanted* the relationship to be fairly obvious around https://github.com/openstack/swift/blob/master/swift/obj/server.py#L1127-L1136 but it's not :-/ | 23:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!