21:00:44 <timburke> #startmeeting swift
21:00:45 <openstack> Meeting started Wed Nov 11 21:00:44 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:48 <openstack> The meeting name has been set to 'swift'
21:00:54 <timburke> who's here for the swift meeting?
21:01:25 <kota_> o/
21:02:04 <rledisez> hi o/
21:02:20 <acoles> hi
21:03:03 <clayg> \o/
21:03:46 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:04:26 <timburke> mostly just follow-up on work in progress (which reminds me, it's been a while since i read through the ML...)
21:04:52 <timburke> #topic audit watchers
21:05:35 <timburke> i know zaitcev has his +2 on there; i've been eaning to circle back on it since the ptg but haven't really
21:06:31 <timburke> is anyone else interested in reviewing it? or is it mostly a matter of "tim needs more time"?
21:08:22 <timburke> well, i'll see what i can do this coming week, then ;-)
21:08:43 <timburke> #topic cross-replication locking
21:09:14 <timburke> rledisez, i finally got around to setting up an env where i could repro it!
21:09:45 <timburke> fix looks great; just had a few follow-up thoughts that i put in https://review.opendev.org/#/c/762393/
21:09:46 <patchbot> patch 762393 - swift - Cross-SSYNC race follow-up - 1 patch set
21:10:45 <timburke> did we ever write up a second bug specifically for rsync?
21:10:58 <rledisez> yes, I quickly looked at it. the follow up totally makes sense to me
21:11:33 <rledisez> timburke: I didn't, I created only 1 bugreport for both, but as patch will close it maybe we need a second one for rsync
21:12:10 <timburke> yeah, that sounds good. do you want to write it or should i?
21:13:12 <rledisez> I will. I think it will mostly be a copy/paste; though, I don't know how to reproduce event if I'm sure it applies for rsync
21:13:21 <rledisez> *even
21:14:09 <timburke> thanks! i'll see about fiddling with my ssync env to test rsync and plan on throwing some more info on the new bug if/as needed
21:15:07 <timburke> speaking of replication...
21:15:14 <timburke> #topic post-sync rehashing
21:16:25 <timburke> so acoles approved https://review.opendev.org/#/c/758636/ and i know clayg is planning on packaging https://review.opendev.org/#/c/761801/ for our cluster in the near future
21:16:25 <patchbot> patch 758636 - swift - Have REPLICATE with suffixes just append to hashes... - 8 patch sets
21:16:27 <patchbot> patch 761801 - swift - Stop invalidating suffixes post-SSYNC - 4 patch sets
21:17:29 <timburke> iirc there was a slight concren that on-disk files may take a bit longer to get cleaned up
21:18:17 <timburke> but on the whole it seems like we've done a lot of good stuff to keep drives from filling up entirely, so it's probably not actually a problem
21:19:21 <timburke> as much as anything, i just want to call these out as changes that are coming, in case people are running near capacity
21:19:54 <timburke> #topic open discussion
21:20:07 <timburke> anything else we should talk about today?
21:23:29 <clayg> sorry distracted
21:23:43 <clayg> did we already say we want to merge the SLO async thing?
21:23:53 <clayg> https://review.opendev.org/#/c/733026/
21:23:54 <patchbot> patch 733026 - swift - Add a new URL parameter to allow for async cleanup... - 16 patch sets
21:24:51 <timburke> let's do it! i'm not scared!
21:25:11 <clayg> https://review.opendev.org/#/c/759183/ is in a similar boat except mattoliverau has some ideas to make it better - i'm sure acoles and timburke will get to those by next week
21:25:11 <patchbot> patch 759183 - swift - memcache: Refuse to error limit the last available... (ABANDONED) - 2 patch sets
21:25:52 <clayg> but pete is says he's +2 with ?asycn=on
21:26:05 <clayg> so... that'll probably happen just so everyone knows
21:26:44 <acoles> clayg: which 'similar' patch did you mean - the link is to an abandoned patch?
21:26:59 <clayg> ... but yeah, that's all I got unmerged that i'd like to see landed - well the shrinking stuff -> but mattoliverau looked at that one too, and it's still WIP
21:27:00 <clayg> https://review.opendev.org/#/c/741721/
21:27:01 <patchbot> patch 741721 - swift - WIP: Fix sharder consistency issues - 11 patch sets
21:27:01 <acoles> I'm guessing...shrinking
21:27:06 <timburke> i think we went with https://review.opendev.org/#/c/761029/ for the memcache-error-limiting stuff -- reminds me that i ought to make some changed so our config-pushing to make sure we turn off the error-limiting when only writing out a single memcached server
21:27:06 <patchbot> patch 761029 - swift - memcache: Make error-limiting values configurable (MERGED) - 4 patch sets
21:27:51 <clayg> oh yeah nix 759183 long live 761029
21:28:26 <clayg> yeah i want to merge the shriking fixes but like... after they're more baked
21:28:45 <clayg> the memcache stuff I guess is also not like "will merge any day" because we haven't shipped the new apporach I don't think
21:29:01 <clayg> so it's really only the SLO that's coming down the pipe!  cool!  I feel really good about what we're carrying
21:30:08 <timburke> supposedly we (NVIDIA) will see a bunch of deletes in the near future, so it'll be interesting to see whether we feel the need to turn allow_async_deletes back off ;-)
21:31:37 <clayg> timburke: ahhaha - or just make the expirer faster ;)
21:33:43 <timburke> fwiw, i know we're also trying to run down some issues with short reads and/or client disconnects
21:35:44 <timburke> which has led me down various rabbit-holes like why we set eventlet.wsig.WRITE_TIMEOUT when it seems nothing in eventlet actually looks at it, or second-guessing the safety of eventlet.green.threading's thread-local storage
21:36:43 <timburke> i'm sure we'll discuss it more as the investigation continues
21:38:19 <timburke> all right, i think i'll call it
21:38:30 <timburke> thank you all for coming, and thank you for working on swift!
21:38:36 <timburke> #endmeeting