21:00:05 <timburke> #startmeeting swift
21:00:06 <openstack> Meeting started Wed Oct 23 21:00:05 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:10 <openstack> The meeting name has been set to 'swift'
21:00:14 <timburke> who's here for the swift meeting?
21:00:24 <kota_> hello~
21:00:27 <mattoliverau> o/
21:01:13 <rledisez> o/
21:01:57 <timburke> i know tdasilva won't be able ot make it
21:02:26 <rledisez> alecuyer won't be present neither
21:02:30 <timburke> and i'm guessing clayg won't either
21:02:35 <timburke> so let's begin!
21:02:49 <timburke> i can make it quick so mattoliverau and kota_ can get on with breakfast ;-)
21:02:56 <mattoliverau> :)
21:02:58 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:00 <kota_> :)
21:03:16 <timburke> #topic Shanghai
21:03:20 <timburke> #link https://etherpad.openstack.org/p/swift-ptg-shanghai
21:03:28 <timburke> it's getting so close now!
21:03:55 <timburke> i know clay's got his visa squared away, and i got mine a couple weeks ago
21:04:19 <timburke> rledisez and kota_, can't wait to see you guys there! mattoliverau, you will be missed!
21:04:28 <mattoliverau> Sorry I won't make it.
21:05:27 <timburke> no worries -- if there's anything you'd like us to talk about, feel free to put it on the etherpad and i'll do my best to represent you (short of doing voices)
21:05:43 <mattoliverau> Lol
21:06:40 <timburke> main thing for this was to just keep highlighting the etherpad
21:06:54 <timburke> #topic stable release
21:07:12 <timburke> we have a new swift!
21:07:13 <timburke> #link http://lists.openstack.org/pipermail/release-announce/2019-October/008047.html
21:07:24 <mattoliverau> \o/
21:07:37 <kota_> nice
21:08:21 <timburke> this is to make sure that our py3 support is the best we can make it. this cycle in particular, i'm going to try to be aggressive in backporting, *especially* for py3 fixes
21:08:57 <kota_> sounds good
21:09:09 <timburke> but having said that, i'm also working on getting stable releases together for stein, rocky, and even queens if i can hurry up ;-)
21:09:33 <timburke> hence the flurry of patches i've proposed lately
21:10:32 <timburke> i think stein is about ready; if anyone wants to take a look at the authors/changelog and check that i didn't miss anything, i'd appreciate it
21:10:34 <timburke> #link https://review.opendev.org/#/c/690699/
21:11:22 <timburke> on to updates!
21:11:30 <timburke> #topic object versioning
21:12:52 <timburke> quick recap: we've got three patches we're working on: one to add the null namespace, one to add a new versioning mode, and one to get s3api to use it
21:13:59 <timburke> the s3api patch had fallen behind for a while, but we recently got it rebased on top of the new versioning mode -- with all the functests passing!
21:14:09 <timburke> probably means we need to write more tests ;-)
21:14:19 <mattoliverau> Lol
21:14:51 <timburke> clay's recently gotten the versioning patch rebased on the null-namespace one
21:15:26 <timburke> i'm pretty sure by shanghai we'll have a pretty solid patch chain with tests passing and everything
21:15:54 <timburke> which should be handy reference as we discuss the design and implementation with kota_ and rledisez :-)
21:16:10 <kota_> thx
21:17:33 <timburke> one of the recent developments with the namespace patch is that we decided that any reserved name must start with a null byte -- effectively partitioning the namespace so we've got all reserved names, followed by "normal" user names
21:18:43 <timburke> this allows us to avoid a lot of the problems with delimiters and figuring out whether a "subdir" entry ought to appear when there's a mix of reserved and user-accessible names
21:19:19 <timburke> that's the main news there -- any questions or feedback?
21:19:22 <rledisez> you mean something like <null>versionning_AUTH_xxx ?
21:19:31 <kota_> \u0000 seems null
21:19:47 <timburke> yeah -- though we aren't currently using it at the account level
21:19:49 * kota_ is looking at https://etherpad.openstack.org/p/swift-null-namespace
21:20:02 <rledisez> ok, I should re-read the etherpad
21:20:26 <timburke> there would be URLs like /v1/AUTH_test/%00versioning%00<client-container>/%00<client-object>%00<timestamp>
21:21:25 <timburke> part of the reason for wanting the null namespace was so we could roll up the usage stats in the existing account DB
21:22:20 <kota_> ic
21:22:40 <timburke> all right -- losf!
21:22:45 <kota_> the objects exist in the user namespace as possible
21:23:00 <timburke> yeah, kinda :-)
21:23:10 <timburke> #topic lots of small files
21:23:22 <timburke> rledisez, how's it going?
21:23:44 <rledisez> nothing new on our side. alecuyer was busy on other topics
21:24:26 <rledisez> he worked on the leveldb state patch (detecting if leveldb is corrupted or not)
21:24:49 <rledisez> it seems it's not as easy as he though, there is not method in leveldb to get that information
21:25:38 <timburke> interesting. and a bit annoying, i suppose :-/
21:27:17 <timburke> all right. i'll keep an eye out for more patches, and *some day* i'll actually find time to get this going in my dev environent
21:28:16 <timburke> i feel like i'm not doing your work justice -- i keep saying i'll look at it but have hardly started :-(
21:28:28 <mattoliverau> does leveldb support any time of checking/scrubbing. might be a good feature request for them. Then we can work on a workaround and hopefully get someone upstream to fix it long term ;)
21:28:59 <rledisez> timburke: don't worry, there is plenty of time to do it. And alecuyer will sit next to you in shangai until you try it ;)
21:29:15 <mattoliverau> lol
21:29:15 <timburke> it'll be great :D
21:29:34 <timburke> all right
21:29:40 <timburke> #topic open discussion
21:29:43 <rledisez> mattoliverau: I can't answer, alecuyer may know that
21:29:57 <mattoliverau> nps, just thinking out load :)
21:30:20 <timburke> mattoliverau, sorry, i dropped the sharding status updating -- figured you probably wouldn't mind too much though ;-)
21:30:38 <mattoliverau> lol, nps, sorry, I haven't been to active. I was away last week. And this week ramping up in a new role.
21:30:54 <rledisez> I have one topic for open discussion today: I noted recently that on our proxy servers (doing only proxy) we will be CPU-bound before we will be network-bound. our proxies are 16 cores Xeon E5-2630 at 2.40GHz, connected in 10Gbps
21:31:16 <rledisez> I made some profiling (only on PUT for now) and found that the "with SomeTimeout()" around read/write on sockets are really consuming a lot of CPU
21:31:40 <rledisez> just for the test, I removed them and was able to reduce the CPU usage by about 35% while increasing the speed of the upload by ~20% (on an other configuration with smaller CPU)
21:31:41 <timburke> mattoliverau, glad the job's still safe :-)
21:32:08 <rledisez> I worked a bit this afternoon on a possible alternative implementation, the idea is to get a "watchdog" greenthread that will kill a socket if nothing happened in a $timeout delay. It seems it's viable
21:32:08 <rledisez> BTW, the second most consuming part is the one the enqueue/dequeue chunk for the putters. I have no idea how to patch that for now :)
21:32:44 <timburke> huh -- interesting...
21:32:48 <rledisez> is CPU usage a concern for somebody else?
21:33:28 <kota_> not to me but curious.
21:33:40 <mattoliverau> Still trying to figure out what the new role is, hopefully I'll still get to spend some of my time upstream on any project. Just not sure how much. (obvious I plan to make that Swift).. although that also means I'm not sure how much openstack releated travel support would be included (I'm assuming none) :(
21:34:01 <timburke> i'll ask around for sure -- i don't remember off hand. i *thought* we'd usually get network-bound, but maybe that's more for larger objects...
21:34:35 <mattoliverau> this is where a Mark would be handy. He'd keep an eye on things via getput and find bottle necks. But he's retired now.
21:34:52 <timburke> mattoliverau, sounds like i need to try to get an LCA talk now ;-)
21:35:15 <mattoliverau> yeah!! maybe we could add some basic perf test in ci/cd so we can get a baseline
21:35:31 <mattoliverau> then we could at least see any regressions.
21:35:43 <mattoliverau> again I'm just brain storming :)
21:36:20 <rledisez> I'll write in an ehterpad the methodolgy I followed for that test, so everyone can check if I didn't make mistake
21:36:55 <timburke> sounds good, thanks for mentioning it!
21:37:03 <mattoliverau> rledisez: good idea, if there is something we can do we should do it. we don't want no stinkin bottlenecks ;)
21:37:40 <kota_> perhaps, the info what's version of eventlet might be helpful.
21:37:59 <timburke> and object size for the PUT
21:38:08 <kota_> we're using thread switching around the chunk putter.
21:38:16 <rledisez> eventlet 0.25, it was 4G so the test was long enough
21:38:52 <kota_> ok
21:39:25 <rledisez> kota_: exactly, every read or write is wrapped in a with Timeout that trigger an eventlet scheduling & co… (I'm not an eventlet expert)
21:41:52 <timburke> yeah, gotta schedule the callback around https://github.com/eventlet/eventlet/blob/v0.25.0/eventlet/timeout.py#L65-L70
21:42:47 <timburke> rledisez, what are your object_chunk_size and client_chunk_size?
21:43:09 <rledisez> defaults (64K)
21:43:54 <timburke> might try increasing that... fwiw, i think we default to 1MB (and just tell customers to buy more RAM :P)
21:44:14 <timburke> anyway, maybe this is best left for #openstack-swift :-)
21:44:23 <rledisez> sure :)
21:44:25 <kota_> to fit the size of ec chunk?
21:44:31 <timburke> and we can let kota_ and mattoliverau get on with their mornings
21:44:44 <kota_> :P
21:44:51 <timburke> thank you all for coming, and thank you for working on swift!
21:44:56 <timburke> #endmeeting