21:00:09 <timburke> #startmeeting swift
21:00:10 <openstack> Meeting started Wed Dec 11 21:00:09 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:14 <openstack> The meeting name has been set to 'swift'
21:00:19 <timburke> who's here for the swift meeting?
21:00:20 <mattoliverau> o/
21:00:26 <kota_> o/
21:00:34 <seongsoocho> o/
21:00:40 <rledisez> o/
21:00:59 <tdasilva> o/
21:01:23 <timburke> hi everybody!
21:01:27 <timburke> agenda's at
21:01:30 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:01:43 <timburke> #topic V release naming
21:01:55 <timburke> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011477.html
21:02:01 <timburke> polling's open!
21:02:28 <timburke> help us name our next release :-)
21:02:36 <timburke> #link https://civs.cs.cornell.edu/cgi-bin/vote.pl?id=E_13ccd49b66cfd1b4&akey=39d9234da798e6a3
21:02:47 <timburke> note that "Vidette" was accidentally included twice -- you might want to ensure that both entries have the same ranking when submitting to make life easier for the election officials
21:03:01 <kota_> lol
21:03:44 <timburke> not much more than that; mainly just wanted to make sure everyone was aware and could let their voice be heard :-)
21:03:59 <timburke> #topic Keystone credentials API
21:04:43 <timburke> in light of the interest i heard in Shanghai about getting credential API support into swiftclient, i figured i ought to mention a recent security issue...
21:04:48 <timburke> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011529.html
21:05:53 <timburke> FWIW, we *do* use the credentials API for secret caching in s3token -- i haven't yet checked to see what sorts of recommendations we may need to make to ensure that continues working...
21:06:03 <timburke> see also...
21:06:05 <timburke> #link https://github.com/openstack/swift/commit/b0aea9360
21:07:09 <timburke> more broadly, this very much still seems like a thing swiftclient should support, though i don't know of anyone actively trying to implement it
21:07:54 <timburke> but i've also heard from some of *my* customers that they'd be interested, so... maybe i'll end up with some bandwidth for it?
21:09:13 <timburke> again, wanted to raise awareness as much as anything -- if you *do* have customers using keystone application credentials, you might want to get them patched sooner rather than later
21:09:43 <timburke> #topic end of year meetings
21:10:20 <timburke> with the holidays coming up, i figured we ought to look at which scheduled meetings it makes sense to keep
21:10:46 <timburke> as things are, we'd hit both christmas (Dec 25) and new years (Jan 1)
21:10:54 <mattoliverau> makes sense, and yeah 25th and 1st make sense
21:11:02 <timburke> but i don't think it makes any sense to actually hold meetings on those days
21:11:20 <mattoliverau> hit both the major holidays, wtg :P
21:11:38 <timburke> anyone feel otherwise? *i* certainly won't be able to host, so any dissenters would be on the hook to chair the meeting ;-)
21:11:40 * rledisez agrees to skip both of them
21:11:57 <tdasilva> +1
21:12:41 <timburke> i *think* we've still got enough going on that it makes sense to keep our meeting for next week, though
21:13:06 <timburke> plus, it'll give me another opportunity to mention the skipped meetings ;-)
21:13:39 <timburke> (sorry to kota_ (and maybe others?) about the late notice for skipping the meeting a couple weeks ago)
21:15:08 <timburke> all right, seems like we're all on the same page :-)f
21:15:42 <timburke> i'll also make sure i update the "next meeting" date on the wiki right after the meeting next week
21:16:01 <timburke> on to updates!
21:16:11 <timburke> #topic versioning
21:16:21 <timburke> clayg, tdasilva -- how's it going?
21:17:22 <tdasilva> I applied some changes to the swift versioning patch early in the week based on reviews, now currently working on a container-sync patch
21:17:51 <tdasilva> so that container-sync will skip versioned containers (until we have a better solution for it)
21:19:15 <tdasilva> the s3 versioning is also in good shape, i noticed timburke put up a change recently to make sure s3api behaves correctly if versioning is not enabled i think, still need to look at that
21:19:38 <tdasilva> my hope is to see these patches merged by next week (fingers crossed)
21:20:09 <timburke> (sounds like an action item for all of us: think about what you'd like container-sync to do with new-style versioned containers)
21:20:48 <timburke> i did! pretty sure most everything works like it did before 🤞
21:21:18 <tdasilva> i think that's all i got, any questions?
21:22:23 <tdasilva> container-sync patch without tests: https://review.opendev.org/#/c/698139/
21:22:24 <patchbot> patch 698139 - swift - WIP: prevent cont-sync when versioning enabled - 2 patch sets
21:22:35 <tdasilva> in case you want to see the direction it is going
21:23:04 <timburke> if i remember right, there were also some questions about whether we'd want the new versioning enabled by default, or maybe whether we would want to *switch it* to be on-by-default in the near-ish future -- i'd certainly be interested in other people's thoughts on *that* question
21:24:08 <mattoliverau> need to think about that.. but it is a different api.. and we want people to adopt it. Beside who wouldn't want to use versioning ;)
21:25:01 <tdasilva> yeah, i think there are two sides to consider. one is what mattoliverau said + s3api versioning depends on it. OTOH, it does make use of the new reserved name API which is very very new
21:25:40 <mattoliverau> test it with fire :P (but yeah that is the otherside)
21:26:15 <tdasilva> what could go wrong
21:27:01 <timburke> it sure *feels like* something we'd want to have on-by-default eventually -- seems (to me, anyway) in all ways better to the versioning scheme we had before. so i guess, what sort of proof-points would people like to see before we move to on-by-default?
21:28:14 <timburke> make sure the api works as intended (ie, we don't go creating some objects in the null namespace that can't be deleted through some client-facing api), make sure we play well with at least container-sync...
21:28:19 <timburke> anything else?
21:28:53 <timburke> probably look at expirer/reconciler interactions...
21:29:30 <mattoliverau> make sure sharding can shard null continers. I assume it can communicate with nulls. ie making null shards
21:29:45 <mattoliverau> I assume this may have been already tested
21:30:03 <mattoliverau> but also assum versioned containers will potentually grow large over time
21:30:52 <tdasilva> i think we looked at some probe tests for that...let me look
21:30:58 * timburke is not sure that's a good assumption...
21:32:01 <tdasilva> heh, maybe not
21:32:37 <mattoliverau> I'll give it a go then :)
21:32:46 <timburke> i suppose something like AWS's lifecycle management api would certainly be nice to help limit the growth of old versions... but since users would have to opt-in to enabling versioning, i'm not sure it should be a hard requirement that we have that before making it available by default...
21:33:39 <timburke> anyway, something to think about. i don't think we're likely to enable-by-default before the next time we all see each other face-to-face, anyway ;-)
21:33:47 <timburke> #topic lots of files
21:34:30 <timburke> rledisez, is there much new going on with this investigation?
21:35:24 <timburke> (i was debating about whether to keep it on the agenda or drop it for now and let you add it back when there's new information to share)
21:35:25 <rledisez> so alecuyer if not here today. he's currently working on some other ovh related things, for the next 3 or 4 weeks. next step for himm will be the battle zfs vs leveldb. so nothing really new for now
21:35:42 <timburke> 👍
21:36:00 <kota_> ic
21:36:03 <rledisez> i think you can drop it for now, we'll see in january when things start moving again
21:36:13 <timburke> sounds good
21:36:26 <timburke> #topic profiling
21:36:41 <timburke> i saw a watchdog patch recently...
21:36:47 <rledisez> #link https://review.opendev.org/#/c/697653/
21:36:48 <patchbot> patch 697653 - swift - Replace all "with Chunk*Timeout" by a watchdog - 5 patch sets
21:36:55 <timburke> that's the one :-)
21:37:15 <rledisez> there is some numbers in a comment that give an idea of the improvement
21:38:17 <rledisez> after that, i'll have an other patch that brings +30% download speed on EC (by removing greenthread/queue in reading fragments from object-servers), but I havce weird issues that, I'm now convinced, are related to greenthread scheduling & co… i don't know how i'll handle that
21:38:28 <rledisez> (weird issues in unit tests only)
21:39:08 <mattoliverau> wow, nice!
21:39:34 <mattoliverau> Once I finish reviewing the versions patch, I look forward to taking these for a spin :)
21:39:47 <rledisez> after that i'll have some small patches that bring small improvement (caching some stuff, etc…)
21:40:03 <rledisez> but the bug win is in the patch linked here and the next one
21:40:07 <rledisez> *big
21:40:27 <timburke> sounds great :D
21:40:52 <timburke> does anyone have any questions about the watchdog patch? i must admit, i haven't looked at it yet
21:41:35 <rledisez> tdasilva: put some in the reviews. thx for looking at it, I hope i answered them :)
21:42:24 <timburke> well, with three more items, maybe i'll keep it moving then ;-)
21:42:32 <timburke> #topic MD5 sums
21:42:36 <tdasilva> rledisez: I did a quick scan of your comments, but TBH I need to look closer to make sure I understand it well
21:42:45 <timburke> #link https://etherpad.openstack.org/p/swift-md5-optimisation
21:42:48 <tdasilva> but thanks for responding quickly
21:43:43 <rledisez> that's a first draft of my thoughts on MD5. it's very preliminary work, but i'm interested in any feedbacks
21:45:31 <timburke> interesting... i don't think i've heard of XXH before...
21:46:44 <tdasilva> rledisez: fwiw, i think azure is using crc64...
21:47:24 <rledisez> tdasilva: good to know. i was wondering if crc32 is enought given the default max object size is 5g
21:47:56 <timburke> i think i rather like the idea of breaking up the client-proxy validation (which is an API that will be hard to break) from proxy-object validation (where we have a lot more freedom)
21:48:02 <rledisez> tdasilva: i did some research and it seems it could be good enough (but i don't have solid argument for that)
21:48:40 <tdasilva> rledisez: yeah, i'm not sure either, their objects sizes are even smaller IIUC
21:48:53 <tdasilva> rledisez: what's 'MD5 + CRC32c * 3' ?
21:49:37 <rledisez> tdasilva: it's a simulation where we would do MD5 once (for the etag) then 3 times crc32c for crypto/ec (like it's done actually with MD5)
21:49:48 <rledisez> timburke: can you reformulate a bit, i'm not sure i totally got you
21:50:11 <rledisez> you prefer to break client-proxy than proxy-object?
21:50:14 <tdasilva> rledisez: but we then ended with something worse than what we have now?
21:50:55 <rledisez> tdasilva: no, right now we have MD5*4 (1.64Gbps). If we had MD5+CRC32c*3, we would have 5.65Gbps => so better :)
21:51:12 <timburke> so we care about multiple hops: client to proxy server, proxy server to object server. part of the problem for EC is that the data sent to each object server has to be hashed separately, since we can't just use a client-provided etag
21:51:35 <timburke> similar arguments with encryption on triple-replica, though the overhead isn't quite as bad
21:54:01 <mattoliverau> I guess it's just a shame we have the md5 contract with the client (the etag). other then that we can seperate things. which is what timburke is saying in the nutshell I think :)
21:54:21 <timburke> having something in the proxy be responsible for ensuring data integrity with the client while some sort of lighter-weight integrity check for the rest of the pipeline and backend requests seems like a solid win
21:54:35 <mattoliverau> +1
21:54:48 <kota_> +1
21:54:59 <rledisez> +1, we are on the same page :)
21:55:49 <timburke> the upgrade may get interesting -- but presumably we could just have an option that you don't switch on until you've upgraded all the backend servers
21:56:32 <mattoliverau> if an op ends up "providing" and alg or not, I wonder if we should start returning etag (md5) and etag-alg in the hopes one day to deprecate the first ;)
21:56:35 <rledisez> yeah, that's probably the right solution. but if you're using ssync, you might degrade nicely to moving data with MD5 I guess? but rsync would be an issue
21:56:55 <rledisez> mattoliverau: totally!
21:56:58 <kota_> probably we need to add sort of Accept: headers for the change between proxy and objext
21:57:18 <timburke> all right, running low on time. there were just a couple patched i wanted to call attention to
21:57:38 <timburke> #topic ranged requests against SLOs
21:57:42 <timburke> #link https://review.opendev.org/#/c/697739/
21:57:42 <patchbot> patch 697739 - swift - Have slo tell the object-server that it wants whol... - 3 patch sets
21:58:30 <timburke> introduces a new header, X-Backend-Ignore-Range-If-Metadata-Present, that let's SLO tell object servers that it wants the whole object if there's a X-Static-Large-Object header
21:58:54 <timburke> otherwise, ranged SLO reads tend to look like:
21:59:20 <timburke> proxy issues GET with Range which 416s (because manifest is way smaller than large object)
21:59:40 <timburke> proxy issues GET, dropping Range, which gets manifest
21:59:47 <timburke> proxy issues GET to segment data
22:00:01 <timburke> i want to get rid of that 416 back-and-forth
22:00:21 <mattoliverau> oh i see
22:00:27 <timburke> #topic RFC-compliant etags
22:00:31 <timburke> #link https://review.opendev.org/#/c/695131/
22:00:31 <patchbot> patch 695131 - swift - Add proxy-server option to quote-wrap all ETags - 5 patch sets
22:00:42 <timburke> i dusted off a 6-year-old Won't Fix bug :{
22:00:46 <timburke> er, :P
22:00:52 <mattoliverau> lol
22:00:55 <timburke> but i'm also out of time ;-)
22:01:10 <timburke> thank you all for coming today, and thank you for working on swift!
22:01:15 <timburke> #endmeeting