21:00:13 <timburke> #startmeeting swift 21:00:14 <openstack> Meeting started Wed Mar 4 21:00:13 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:17 <openstack> The meeting name has been set to 'swift' 21:00:21 <timburke> who's here for the swift meeting? 21:00:28 <mattoliverau> o/ 21:00:29 <kota_> hi 21:00:32 <seongsoocho> o/ 21:00:35 <clayg> I/ 21:00:43 <alecuyer> o/ 21:00:47 <rledisez> hi o/ 21:01:04 <tdasilva> o/ 21:01:12 <timburke> agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:01:32 <timburke> i haven't got much planned, so straight to updates! 21:01:39 <timburke> #topic concurrent EC 21:01:44 <timburke> clayg, how's it going? 21:02:14 <clayg> Ok, I’ll push up my wip after the meeting. Folks can see t new approach. 21:03:02 <alecuyer> teasing us 21:03:07 <clayg> I’ll probably do a new change? Might be easier to A/B 21:03:36 <timburke> do you think it'll be in a ready-to-merge state, or still largely experimental? 21:03:48 <timburke> (asking to set expectations, as much as anything) 21:03:52 <clayg> alecuyer: no, it’s very WIP a lot of duplication that’ll need to be removed after we get it working 21:04:13 <clayg> Not ready to merge. 21:04:15 <timburke> 👍 21:04:23 <clayg> I’ll want some feedback 21:04:48 <timburke> anything else you need now? 21:05:03 <clayg> Not now. Next week. 👍 21:05:19 <timburke> sounds good 21:05:20 <timburke> #topic 503 delays 21:05:37 <timburke> i wasn't sure if there was any other discussion needed here or not 21:05:51 <clayg> I’m good on this topic. 21:06:13 <clayg> I think the whole idea is hostile to well behaved clients 21:06:21 <clayg> I’m over it. 21:07:41 <timburke> i guess the idea is that the client should ensure its own request smearing? 21:08:31 <rledisez> i still see a point in it cause as a public cloud operator, we have no control on clients 21:08:47 <timburke> my main questions are, would something as simple as sleep(random.random() * conf.max_unavailable_wait) work? defaulting it to zero, of course. and where would we want it to live? in the proxy, or somewhere in middleware? 21:09:10 <timburke> in the proxy-server app, i mean 21:09:43 <rledisez> I would say on the left of the pipeline because many things can happen without proxy-server being involved 21:10:14 * timburke nods 21:10:45 <timburke> probably ratelimit (for separation of concerns) or catch_errors, yeah? or its own thing 21:11:23 <clayg> ratelimit seems reasonable 21:11:24 <rledisez> ratelimit makes sense 21:12:40 <mattoliverau> +1 21:12:49 <timburke> oh yeah, i should look at the intereaction between ratelimit and s3api some more... 21:13:00 <clayg> Hahaha 21:13:38 <timburke> i think it's not *terrible* following https://review.opendev.org/#/c/704659/ ? 21:13:38 <patchbot> patch 704659 - swift (stable/stein) - s3api: Better handle 498/429 responses (MERGED) - 1 patch set 21:14:02 <timburke> er, https://review.opendev.org/#/c/697535/ for the master version 21:14:02 <patchbot> patch 697535 - swift - s3api: Better handle 498/429 responses (MERGED) - 1 patch set 21:14:09 <clayg> Merged! 21:14:15 <timburke> anyway 21:14:18 <timburke> #topic losf 21:14:27 <clayg> I remember when things merged. 21:14:29 <timburke> rledisez, how's it going? 21:14:42 <rledisez> fine I guess, just an update about what's going on for LOSF 21:14:59 <rledisez> as discussed at the last PTG in Shangai, the topic is not really lots of small files, but instead lots of big disks (having a consequence lots of files) 21:14:59 <rledisez> this topic is back on our short-term roadmap cause we are experimenting with new hardware (36x14TB, 2 SSD, 96GB of RAM) 21:15:08 <clayg> 🍿 21:15:38 <rledisez> we examined different option that could replace LOSF without having to maintaining them: 21:15:38 <rledisez> XFS with rtdev option => not stable 21:15:38 <rledisez> XFS with Intel CAS => not stable 21:15:38 <rledisez> ZFS with metadata device => too much space needed for metadata + few operational issues 21:15:41 <kota_> wow 21:16:25 <rledisez> in the end, we concluded that LOSF stays the best option. we made some changes that was discussed in Shangai: 21:16:25 <rledisez> - new key format so that we can save a lot of CPU when doing replication 21:16:25 <rledisez> - new option to store the LevelDB in a new path (on a fast device like SSD/NVMe) 21:16:25 <rledisez> - store metadata in the LevelDB (TODO) 21:16:46 <rledisez> and when I say we made some change, I mean alecuyer did it ;) 21:17:15 <timburke> go alecuyer! 21:17:29 <timburke> that all sounds great :D 21:17:47 <rledisez> that's it for me on LOSF 21:18:04 <alecuyer> heh, I guess :) So, I don't want to spend meeting time now on this ,but if anyone may be hitting these same issues we have, and want to guide the design/dev process, let's talk 21:18:06 <clayg> It’s back on!!! 21:18:24 <rledisez> so we're still involved in it, we just had to figure some stuff first 21:18:59 <timburke> what were the issues you were seeing with the new boxes that brought it back around? RAM constraints still? 21:19:32 <rledisez> yeah, the pattern will be the same => small files. if it was not working on 36x6TB, it will for sure not work on 36x14TB 21:19:38 <alecuyer> inodes not fitting in the VFS cache 21:19:51 <rledisez> also IO starvation because of replication because of cache miss in the VFS cache 21:20:36 <timburke> make sense 21:20:38 <rledisez> We have around 3 millions of objects per TB of disk. You can imagine the number of inode it implies (around 10M per TB of disks) 21:20:59 <rledisez> well, maybe 8M, not 10 21:21:26 <timburke> #topic open discussion 21:21:38 <timburke> anything else to bring up today? 21:22:07 <rledisez> zaitcev: I got your message, I loaded the two reviews. will have a look this week, I promess :) 21:22:33 <zaitcev> rledisez: so, what can I help you with, then? You have any reviews for me? 21:22:55 <timburke> oh, this was an interesting thread: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/012950.html 21:23:01 <zaitcev> Hopefully not the whole LoSF 21:23:20 <clayg> Do we need to auto retest failed tests in the gate before we fail the job? 21:23:25 <rledisez> zaitcev: yeah, that would be unfair trading 21:23:27 <kota_> oh what 21:23:40 <tdasilva> rledisez: are you still investigating those proxy level performance improvements? 21:23:52 <timburke> clayg, yeah, i should look into that. i'm pretty sure there's a way to configure that... 21:24:16 <rledisez> tdasilva: yes, I had to pause for the last 2 weeks, busy doing OVH-stuff, but I want to get back on it next week 21:24:56 <tdasilva> rledisez: cool, looking forward to it.... 21:24:58 <rledisez> tdasilva: I already have a review "ready" (I have a weird eventlet random failure in test). I'm now on MD5 replacement 21:25:21 <tdasilva> rledisez: can you share gerrit # ? 21:25:25 <rledisez> timburke: I admit I skipped the thread. in few words, any conclusion came out? 21:26:49 <rledisez> tdasilva: https://review.opendev.org/#/c/697653/ actually this one is OK, I just need to rebase it 21:26:49 <patchbot> patch 697653 - swift - Replace all "with Chunk*Timeout" by a watchdog - 5 patch sets 21:26:51 <timburke> no real conclusion, just putting forward an idea of spreading the PTL responsibilities more (in part due to concerns about projects not having any self-nominees) 21:27:02 <timburke> i don't expect *that* to be an issue for us ;-) 21:27:26 <rledisez> tdasilva: this one has the random failure : https://review.opendev.org/#/c/704892/ 21:27:26 <rledisez> running the test individually pass, running all of them at once fails 21:27:26 <patchbot> patch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 1 patch set 21:27:43 <timburke> right! i meant to dig into that but never got to it... 21:28:30 <zaitcev> sounds like a challenge 21:29:28 <timburke> so, i just realized we might want to poke more at libec -- see if we can avoid needing to do matrix inversion on every call to decode... 21:29:56 <timburke> should be eminently cacheable 21:30:00 <kota_> hmm 21:30:48 <timburke> https://github.com/openstack/liberasurecode/blob/master/src/backends/isa-l/isa_l_common.c#L211-L222 seems like we shouldn' 21:30:59 <timburke> t need to do it on every call 21:31:25 <tdasilva> just fyi, Vianney pinged me recently about quadiron lib patches, they are still waiting for reviews. I've been lacking on that, so if anyone has a chance 21:31:33 <tdasilva> s/lib/libec 21:32:01 <tdasilva> i think patch chain starts here: https://review.opendev.org/#/c/635603/1 21:32:01 <patchbot> patch 635603 - liberasurecode - fix: data access when having non-zero metadata size - 1 patch set 21:32:44 <timburke> my trouble is that when i start doing libec reviews, i end up wanting to rewrite sizable parts of it all ;-) 21:33:40 <rledisez> did anyone already looked into the glacier api, maybe to add it to s3api? 21:33:59 <zaitcev> Sounds ambitious. 21:34:05 <timburke> oh, and i should reach out to libphazr... i think that first patch would break them, but they don't have any CI set up with us... 21:38:15 <timburke> all right 21:38:31 <timburke> thank you all for coming, and thank you for working on swift! 21:38:36 <timburke> #endmeeting