21:00:44 <timburke> #startmeeting swift 21:00:45 <openstack> Meeting started Wed Nov 11 21:00:44 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:48 <openstack> The meeting name has been set to 'swift' 21:00:54 <timburke> who's here for the swift meeting? 21:01:25 <kota_> o/ 21:02:04 <rledisez> hi o/ 21:02:20 <acoles> hi 21:03:03 <clayg> \o/ 21:03:46 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:04:26 <timburke> mostly just follow-up on work in progress (which reminds me, it's been a while since i read through the ML...) 21:04:52 <timburke> #topic audit watchers 21:05:35 <timburke> i know zaitcev has his +2 on there; i've been eaning to circle back on it since the ptg but haven't really 21:06:31 <timburke> is anyone else interested in reviewing it? or is it mostly a matter of "tim needs more time"? 21:08:22 <timburke> well, i'll see what i can do this coming week, then ;-) 21:08:43 <timburke> #topic cross-replication locking 21:09:14 <timburke> rledisez, i finally got around to setting up an env where i could repro it! 21:09:45 <timburke> fix looks great; just had a few follow-up thoughts that i put in https://review.opendev.org/#/c/762393/ 21:09:46 <patchbot> patch 762393 - swift - Cross-SSYNC race follow-up - 1 patch set 21:10:45 <timburke> did we ever write up a second bug specifically for rsync? 21:10:58 <rledisez> yes, I quickly looked at it. the follow up totally makes sense to me 21:11:33 <rledisez> timburke: I didn't, I created only 1 bugreport for both, but as patch will close it maybe we need a second one for rsync 21:12:10 <timburke> yeah, that sounds good. do you want to write it or should i? 21:13:12 <rledisez> I will. I think it will mostly be a copy/paste; though, I don't know how to reproduce event if I'm sure it applies for rsync 21:13:21 <rledisez> *even 21:14:09 <timburke> thanks! i'll see about fiddling with my ssync env to test rsync and plan on throwing some more info on the new bug if/as needed 21:15:07 <timburke> speaking of replication... 21:15:14 <timburke> #topic post-sync rehashing 21:16:25 <timburke> so acoles approved https://review.opendev.org/#/c/758636/ and i know clayg is planning on packaging https://review.opendev.org/#/c/761801/ for our cluster in the near future 21:16:25 <patchbot> patch 758636 - swift - Have REPLICATE with suffixes just append to hashes... - 8 patch sets 21:16:27 <patchbot> patch 761801 - swift - Stop invalidating suffixes post-SSYNC - 4 patch sets 21:17:29 <timburke> iirc there was a slight concren that on-disk files may take a bit longer to get cleaned up 21:18:17 <timburke> but on the whole it seems like we've done a lot of good stuff to keep drives from filling up entirely, so it's probably not actually a problem 21:19:21 <timburke> as much as anything, i just want to call these out as changes that are coming, in case people are running near capacity 21:19:54 <timburke> #topic open discussion 21:20:07 <timburke> anything else we should talk about today? 21:23:29 <clayg> sorry distracted 21:23:43 <clayg> did we already say we want to merge the SLO async thing? 21:23:53 <clayg> https://review.opendev.org/#/c/733026/ 21:23:54 <patchbot> patch 733026 - swift - Add a new URL parameter to allow for async cleanup... - 16 patch sets 21:24:51 <timburke> let's do it! i'm not scared! 21:25:11 <clayg> https://review.opendev.org/#/c/759183/ is in a similar boat except mattoliverau has some ideas to make it better - i'm sure acoles and timburke will get to those by next week 21:25:11 <patchbot> patch 759183 - swift - memcache: Refuse to error limit the last available... (ABANDONED) - 2 patch sets 21:25:52 <clayg> but pete is says he's +2 with ?asycn=on 21:26:05 <clayg> so... that'll probably happen just so everyone knows 21:26:44 <acoles> clayg: which 'similar' patch did you mean - the link is to an abandoned patch? 21:26:59 <clayg> ... but yeah, that's all I got unmerged that i'd like to see landed - well the shrinking stuff -> but mattoliverau looked at that one too, and it's still WIP 21:27:00 <clayg> https://review.opendev.org/#/c/741721/ 21:27:01 <patchbot> patch 741721 - swift - WIP: Fix sharder consistency issues - 11 patch sets 21:27:01 <acoles> I'm guessing...shrinking 21:27:06 <timburke> i think we went with https://review.opendev.org/#/c/761029/ for the memcache-error-limiting stuff -- reminds me that i ought to make some changed so our config-pushing to make sure we turn off the error-limiting when only writing out a single memcached server 21:27:06 <patchbot> patch 761029 - swift - memcache: Make error-limiting values configurable (MERGED) - 4 patch sets 21:27:51 <clayg> oh yeah nix 759183 long live 761029 21:28:26 <clayg> yeah i want to merge the shriking fixes but like... after they're more baked 21:28:45 <clayg> the memcache stuff I guess is also not like "will merge any day" because we haven't shipped the new apporach I don't think 21:29:01 <clayg> so it's really only the SLO that's coming down the pipe! cool! I feel really good about what we're carrying 21:30:08 <timburke> supposedly we (NVIDIA) will see a bunch of deletes in the near future, so it'll be interesting to see whether we feel the need to turn allow_async_deletes back off ;-) 21:31:37 <clayg> timburke: ahhaha - or just make the expirer faster ;) 21:33:43 <timburke> fwiw, i know we're also trying to run down some issues with short reads and/or client disconnects 21:35:44 <timburke> which has led me down various rabbit-holes like why we set eventlet.wsig.WRITE_TIMEOUT when it seems nothing in eventlet actually looks at it, or second-guessing the safety of eventlet.green.threading's thread-local storage 21:36:43 <timburke> i'm sure we'll discuss it more as the investigation continues 21:38:19 <timburke> all right, i think i'll call it 21:38:30 <timburke> thank you all for coming, and thank you for working on swift! 21:38:36 <timburke> #endmeeting