21:00:09 <timburke> #startmeeting swift
21:00:10 <openstack> Meeting started Wed Mar 11 21:00:09 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:14 <openstack> The meeting name has been set to 'swift'
21:00:17 <timburke> who's here for the swift meeting?
21:00:22 <seongsoocho> o/
21:00:30 <mattoliverau> o/
21:00:31 <tdasilva> o/
21:00:32 <rledisez> o/
21:00:34 <alecuyer> o/
21:01:00 <kota_> o/
21:01:19 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:01:37 <timburke> i've got a couple things to point out before we get to updates, though
21:02:07 <timburke> there's an interesting call to "consolidate" on the ML: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013236.html
21:02:57 <timburke> i'm not sure what exactly that will mean in practice, or whether we'd even be impacted by it, but i figure it's a thread to be aware of
21:03:56 <timburke> pretty sure i mentioned it in -swift, but in case you missed it, after Victoria, we'll have Wallaby: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013006.html
21:04:38 <timburke> and it looks like they've approved the schedule for Victoria: https://releases.openstack.org/victoria/schedule.html
21:05:35 <timburke> that's all i've got for general openstack news -- any questions or comments?
21:05:52 <mattoliverau> so many links to get lost reading :P
21:06:12 <timburke> hehe sorry -- i can wait a bit :-)
21:08:04 <timburke> all right, on to updates!
21:08:12 <timburke> #topic waterfall EC
21:08:35 <timburke> i don't know that clayg's actually around right now...
21:08:59 <zaitcev> Poke him harder.
21:09:10 <timburke> but he's been hard at work trying to figure out what we'd want out of something similar to concurrent gets but for EC
21:09:14 <zaitcev> I often forget about our meeting.
21:09:51 <timburke> turns out, it's a bit of a mess
21:11:24 <timburke> part of the trouble is that we've *also* got some logic where if we start hitting read timeouts, the proxy will try to find another node to read from
21:12:46 <timburke> to make that still work with EC we put each "frag lane" (as clayg's been calling them) in its own ResumingGetter
21:13:57 <timburke> but then there's *also* logic to have us spin up more than ndata frag lanes in some cases (specifically, when a single node may have more than one frag due to rebalances)
21:14:59 <timburke> so... like i said, it's a bit of a mess. it might be a bit before there's a patch that'd actually be ready to merge. and we might come asking you guys about trade-offs that we might make
21:15:27 <alecuyer> Thanks for explaining these bits already, I'm not familiar with that logic
21:15:52 <zaitcev> I don't understand why we cannot request 2 fragments from the same node in the same time. We do exactly that for PUT, don't we?
21:18:10 <timburke> we can, following https://review.opendev.org/#/c/215276/
21:18:10 <patchbot> patch 215276 - swift - Enable object server to return non-durable data (MERGED) - 43 patch sets
21:18:57 <timburke> i'm not sure that we do for PUT, though... oh, except i was saying "node" when i really meant "disk"
21:19:44 <kota_> so it means each frag-lane needs to get ndata fragments to decode each segment, hmm
21:21:12 <timburke> i was talking to clay about this a bit yesterday; if i understood him right, the models is more like each frag-lane is responsible for getting one frag ready and reading it, and we need ndata frag-lanes to respond
21:21:15 <kota_> even the segment that is decoded is just different from other frag-lane's ones like concurrent gets, isn't it?
21:22:08 <timburke> but since they're all sharing the same node_iter, it gets messy trying to spawn off extra requests
21:22:22 <kota_> hmm
21:24:09 <timburke> like, if if we have a 3+2 policy, you might spin up lanes A, B, and C ... all of which are kinda sluggish to respond. so the concurrency timeout pops and you... do what? the cheap thing is to have each lane spawn another request: A', B', C'
21:25:10 <timburke> but then you still can't respond even if you've got A, B, and A' in hand, since A and A' are in the same "lane"
21:26:04 <timburke> and meanwhile, C' must be going to a handoff, which we don't expect to have anything useful anyway
21:27:05 <timburke> otoh, you could spin up new lanes D and E... but then we lose the ResumingGetter behavior of going to another disk when there's a read timeout mid-stream
21:27:52 <kota_> i see
21:28:01 <timburke> anyway, clayg's got a couple exploratory patches up: p 709326 and p 711342
21:28:01 <patchbot> https://review.opendev.org/#/c/709326/ - swift - Enable more concurrency for EC GET - 2 patch sets
21:28:03 <patchbot> https://review.opendev.org/#/c/711342/ - swift - wip: asyc concurrent ecfragfetcher - 2 patch sets
21:29:09 <timburke> and there's probably some more history i could dig up to remind us how we got here and what was informing our design decisions at the time, so we could sanity-check that those assumptions/goals are still valid
21:29:40 <timburke> but i don't have that all collected right now -- i'll try to get some stuff together for next week or the week after, maybe start an etherpad
21:30:02 <timburke> #topic lots of small files
21:30:33 <timburke> rledisez, alecuyer how's it going? should i be getting the feature branch up-to-date?
21:31:14 <rledisez> so, last week alecuyer worked on the new key format to save CPU. it's mostly done. for the comoing week, he will work on the algorithm to select the volume where to write an incoming object
21:31:45 <rledisez> I'll take some time to try to merge master in the feature branch
21:32:03 <rledisez> I think last time I checked (few weeks ago), there were minor conflicts, nothing big IIRW
21:32:13 <timburke> sounds great! thanks for keeping us updated
21:32:39 <timburke> anything else you need from us? things to review, or even just to be thinking about?
21:32:54 <rledisez> alecuyer: ^
21:33:31 <alecuyer> I don't think so yet (although of course any advice or comment is helpful!),
21:33:52 <timburke> 👍
21:34:03 <timburke> #topic CORS
21:34:05 <alecuyer> I'll try to push what has been done so far on the feature branch once we get it rebased with romain?
21:34:18 <timburke> yeah, that sounds great
21:35:21 <timburke> so i've been getting CORS loaded back into my head -- the last time i gave it much thought was when torgomatic was poking at pulling it out to middleware in https://review.opendev.org/#/c/528106/
21:35:21 <patchbot> patch 528106 - swift - Move CORS to middleware. - 5 patch sets
21:36:10 <kota_> "updated: 2years, 2 months ago" oh
21:36:24 <timburke> at the time, i started trying to write some functional tests -- https://review.opendev.org/#/c/533028/
21:36:24 <patchbot> patch 533028 - swift - Add some functional CORS tests - 7 patch sets
21:37:36 <timburke> which have proven really useful in getting myself familiar with the browser model and how these three agents (swift, browser, and some third-party js) interact
21:38:53 <timburke> in the most-recent patchset, i've even got a test script that will spin up a webserver running on localhost, use selenium to drive a variety of browsers, run the tests, and print out some results
21:39:12 <zaitcev> wow
21:39:20 <kota_> great
21:39:20 <timburke> next up, i need to figure out how to run that as a gate job
21:40:34 <timburke> now, the end goal (for me) is to get some app that works with S3 to also work against Swift
21:41:13 <timburke> i'm not actually 100% sure what this app is, or how it's used, though :-( hopefully i'll be able to get more details on that in the next week or so
21:41:50 <timburke> but toward that end, i started poking at adding some s3api CORS tests in https://review.opendev.org/#/c/710354/
21:41:50 <patchbot> patch 710354 - swift - Add CORS func tests for s3api - 3 patch sets
21:42:50 <timburke> and fixing up s3api so they can actually pass
21:42:59 <timburke> in https://review.opendev.org/#/c/710330/ and https://review.opendev.org/#/c/710355/
21:43:00 <patchbot> patch 710330 - swift - s3api: Pass through CORS headers - 2 patch sets
21:43:01 <patchbot> patch 710355 - swift - s3api: Allow CORS preflight requests - 4 patch sets
21:43:22 <zaitcev> Maybe some kind of gallery?
21:44:47 <timburke> well, the pass/fail output looks something like http://paste.openstack.org/show/790568/
21:45:10 <kota_> looks nice
21:45:59 <timburke> i originally wanted to get it working more like/with unittest, but... that went poorly (rife with import side-effects, overall pretty terrible)
21:46:31 <timburke> i settled on TAP (https://en.wikipedia.org/wiki/Test_Anything_Protocol) mainly for its simplicity
21:47:35 <timburke> so, i've got a bunch of stuff that i think is ready for review... but my bigger question at the moment is: does anyone else have much interest in CORS?
21:48:08 <zaitcev> It used to be more important when Swift was supposed to back websites, like images or geo data.
21:48:09 <timburke> honestly, if it weren't for this one user, i probably would've continued ignoring it
21:48:30 <zaitcev> What does that user do, unless it's a secret?
21:49:03 <rledisez> we rarely got requests/complaints, but some people use it for sure
21:49:04 <timburke> i don't remember! notmyname's been the main contact -- i need to cut out the middleman ;-)
21:50:01 <rledisez> that said, I like the idea of a middleware for that
21:50:22 <timburke> basically, i'm trying to figure out whether i should be bugging you all for reviews, or just clayg and tdasilva :P
21:50:36 <rledisez> side question: does anybody already measured the cost of a middleware? (thinking of latency)
21:51:35 <timburke> i've not -- though i'd expect it will tend to vary a lot with the middleware and its scope of responsibility
21:52:42 <rledisez> timburke: of course. I was more thinking of the basic noop middleware, just the overhead of having a middleware in the pipeline
21:53:01 <rledisez> it was jus crossing my mind, it's probably negligible
21:53:06 * timburke nods
21:53:17 <timburke> extra function calls add up, for sure
21:53:37 <timburke> oh yeah, and while investigating all of this CORS stuff, i noticed that it can be used to check whether encryption is enabled or not -- https://review.opendev.org/#/c/712237/ should fix it, though
21:53:38 <patchbot> patch 712237 - swift - encryption: Expose decrypted metadata via CORS - 2 patch sets
21:54:02 <timburke> ok, want to still have some time for discussion, so
21:54:07 <timburke> #topic open discussion
21:54:20 <timburke> anything else to bring up?
21:55:06 <rledisez> just one thing on MD5 "replacement"
21:55:11 <zaitcev> I had some strange case reviewing one of rledisez' patches.
21:56:04 <rledisez> well, forget it, it's so unclear in my head right now that I can't even formulate my point. i'll think more :)
21:56:21 <zaitcev> Running  tox -e py27  has a bad case of tests interfering: it only fails when testing all-up, but not individually.
21:56:28 <timburke> rledisez, next week :-)
21:56:31 <rledisez> zaitcev: yeah, I saw your comment, thx for trying it out, I spent so much time trying to figure out what could happen
21:56:32 <zaitcev> https://review.opendev.org/704892
21:56:32 <patchbot> patch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 1 patch set
21:57:31 <timburke> zaitcev, yeah, rledisez called out the trouble before -- we're suspecting some kind of eventlet shenanigans? gotta dig in some more though; definitely a blocker to merging as-is
21:57:33 <zaitcev> It all seems to have something to do with StopIteration leaking somewhere. Seems like those queues isolated callers from it. But with yields all over I cannot track who calls what. Well, not at a glance.
21:58:03 <zaitcev> ok
21:58:28 <zaitcev> seems like a tactical scale problem, just annoyingly slippery
21:58:44 <zaitcev> well, I need to go
21:59:01 <timburke> yeah, it's getting to be that time
21:59:13 <timburke> just, one last announcement: today is the last day of SwiftStack! tomorrow, clayg, tdasilva, notmyname and myself will all be nvidians
21:59:43 <rledisez> the king is dead long live the king!
21:59:54 <timburke> it's been a crazy ride, and i can't wait to see where this next part of the journey takes us :-)
22:00:20 <rledisez> have fun, all of you, new challenges for sure :)
22:00:30 <timburke> all right
22:00:40 <timburke> thank you all for coming, and thank you for working on swift!
22:00:45 <timburke> #endmeeting