21:00:03 <timburke__> #startmeeting swift
21:00:04 <openstack> Meeting started Wed Mar 24 21:00:03 2021 UTC and is due to finish in 60 minutes.  The chair is timburke__. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:07 <openstack> The meeting name has been set to 'swift'
21:00:10 <timburke__> who's here for the swift meeting?
21:00:20 <seongsoocho> o/
21:00:27 <kota_> o/
21:01:03 <rledisez> o/
21:01:26 <mattoliverau> o/
21:03:42 <acoles> o/
21:03:47 <timburke__> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:51 <timburke__> first up
21:03:55 <timburke__> #topic PTG
21:04:27 <timburke__> if you haven't yet, please register and respond on the doodle poll for meeting times
21:04:37 <timburke__> #link https://april2021-ptg.eventbrite.com/
21:04:42 <timburke__> #link https://doodle.com/poll/cnr3qb7y2afiad9v
21:05:39 <timburke__> and we've got an etherpad for gathering topics
21:05:41 <timburke__> #link https://etherpad.opendev.org/p/swift-ptg-xena
21:06:05 <timburke__> i really ought to book (virtual) rooms and such today
21:06:41 <mattoliverau> then doodle poll needs to be a priority.. so please do it if you haven't yet!
21:08:08 <timburke__> any questions or comments on it? less than a month away now!
21:11:01 <timburke__> all right then
21:11:08 <timburke__> on to some updates!
21:11:12 <timburke__> #topic relinker
21:11:41 <timburke__> so we did another part power at nvidia! just finished up this morning
21:12:02 <timburke__> went better than the last time -- we're definitely getting better at it
21:12:16 <timburke__> but there are still some rough edges
21:14:00 <timburke__> notably, we saw a bunch of extra (empty) handoff partitions getting created because we ran the relinker more than once, and it'd want to do a rehash on a partition that never got any links put in it
21:14:59 <timburke__> i think https://review.opendev.org/c/openstack/swift/+/781908 should deal with it -- basically, if we call get_hashes for a partition that doesn't exist, we should be able to return early
21:16:55 <timburke__> since last week, we had a bunch of great stuff land -- operators can now target specific policies and even partitions, and we've merged acoles's great refactoring to make relink and cleanup reuse a lot of the same code
21:18:21 <timburke__> still to come, mattoliverau has some more improvements to progress visibility, and i rebased my patch to parallelize the relinker
21:18:32 <mattoliverau> nice
21:19:06 <timburke__> if you're interested in part power increases, those are likely the best places to jump in!
21:19:14 <seongsoocho> cool. the relinker is getting better
21:19:23 <zaitcev> So how high are you now?
21:19:32 <zaitcev> And I don't mean marijuana
21:19:49 <timburke__> only 18. we've done it twice, and intend to do at least two more
21:19:50 <acoles> we also learnt some stuff doing this increase after a previous one. For example, we discovered we had some tombstones that had not been cleaned up in their original partitions, so during this relink we had *two* locations trying to create a link in the same new location
21:20:04 <zaitcev> Hmm. So even 18 is not enough for you?
21:20:13 <acoles> we just love doing it :D
21:20:17 <timburke__> (speaking of, https://review.opendev.org/c/openstack/swift/+/781591 is a first crack at being able to step by more than one)
21:20:21 <zaitcev> That's like 262000 partitions
21:20:58 <timburke__> if you assume 10k disks, that's still only 26/disk ;-)
21:21:24 <mattoliverau> oh cool, jump more then 1, interesting
21:22:21 <seongsoocho> one question for relink. in my case, I forgot to increase reclaim age before the relink, so there are a lots of cleanup job is failed..
21:22:59 <seongsoocho> I fi
21:23:42 <timburke__> as long as the failures are for tombstones (.ts files) i expect it should be mostly fine
21:24:30 <seongsoocho> I finished the part power job and the object replicator is running again. The failed diskfiles will be delete? Most of diskfiles are .data and it was deleted in other node during relink
21:27:11 <timburke__> the .datas -- are the objects still in listings?
21:27:36 <seongsoocho> nop. It is not in list
21:28:23 <timburke__> i think those will need to be manually cleaned up :-(
21:28:58 <timburke__> though zaitcev's dark data audit watcher could come in handy when you upgrade
21:30:02 <zaitcev> It does not reattach them though
21:30:06 <zaitcev> Can only delete
21:30:37 <zaitcev> But what the heck happened? Relinker misfired, or they were orphans even before partition power increase?
21:30:39 <timburke__> fwiw, newer versions also make it so the relinker can respect the cluster's reclaim_age by pointing it at object-server.conf
21:31:04 <zaitcev> Oh, n/m I see the comment about reclaim age.
21:31:09 <mattoliverau> sounds like they were deletes that weren't propergated fully so delete makes sense.
21:32:23 <timburke__> i'm guessing there was a delete after the increased ring went out but before the cleanup processed the partition. then a reclaim age passed. cleanup finally got to it, went to open the diskfile in the new location -- and cleaned up the tombstone before erroring
21:33:14 <timburke__> ...makes me think the relinker (when not given a config file) should use a reclaim_age that's intentionally longer than anyone would likely have it configured...
21:33:17 <zaitcev> That's going to be the first time ever watcher got useful to anyone then.
21:33:28 <zaitcev> If seongsoocho uses it.
21:33:34 <seongsoocho> yes. during the relink , the object was deleted  in other 2 nodes but not deleted in one node (because relinker use that disk too heavy , so object server return error)
21:33:41 <timburke__> peace of mind is still a valid use-case ;-)
21:33:56 <seongsoocho> zaitcev:  cool.
21:34:48 <timburke__> all right
21:34:57 <timburke__> #topic sharding and shrinking
21:35:17 <timburke__> i know mattoliverau and acoles have continued working on this
21:35:44 <mattoliverau> I've been playing with al's repair and anaylze chain, and it's pretty awesome.
21:36:08 <timburke__> and mattoliverau, you put together an etherpad about how to tell that a shard is old enough to consider shrinking, yeah?
21:36:10 <timburke__> #link https://etherpad.opendev.org/p/swift-shrinking-active-age
21:36:26 <mattoliverau> oh yeah. The current shrinking blocker is ^
21:36:53 <acoles> zaitcev pulled me up on some __hash__ kludge in that for which I am grateful - made it better
21:37:46 <acoles> I finally got some ideas about tombstone reporting up https://review.opendev.org/c/openstack/swift/+/782832 - very much WIP though
21:38:50 <mattoliverau> there is an edge case that one primary is small and shards before the others, the new shards ranges become active and then being pretty empty will be considered candidates for shrinking. The etherpad explains current progress, problems, pain points.
21:39:34 <mattoliverau> turns out what we thought should be an easy fix seems to get more and more complicated :P
21:39:35 <acoles> mattoliverau: thanks for starting the etherpad
21:40:08 <mattoliverau> So if your interested, read, ask questions and comment in the etherpad.
21:40:11 <acoles> mattoliverau: that's ditributed systems for you! ;)
21:40:19 <mattoliverau> lol yeah :)
21:40:55 <mattoliverau> in the meantime I wanted to give Als' repair stuff some love, because they'd also be great to have and have been up a long time.
21:41:38 <mattoliverau> I plugged them into my WIP shard graph visualiser tool and now you can get a green path for the best path. Which is nice when testing als changes.
21:41:39 <timburke__> sounds good. we'll be able to fix up busted sharding yet!
21:41:50 <timburke__> #topic open discussion
21:41:58 <timburke__> what else should we bring up this week?
21:42:19 <kota_> ah
21:42:33 <kota_> one thing, it is not actually related to Swift
21:43:10 <kota_> before a week from PTG, NVIDIA GTC will be held at online and I have a session to introduce my recent work.
21:43:13 <kota_> just FYI.
21:43:24 <acoles> kota_: great!
21:43:29 <timburke__> cool! congrats!
21:43:40 <seongsoocho> kota_:  cool!!!!
21:43:43 <acoles> do you have a link / date/ time ?
21:43:48 <mattoliverau> nice!
21:43:50 <kota_> it's free so you can watch the pre-recorded video
21:44:19 <kota_> the registration is at https://gtc21.event.nvidia.com/
21:45:12 <kota_> it looks like the direct link to the video is not work unless someone registered to see.
21:45:14 <acoles> kota_: thanks for sharing that
21:45:22 <kota_> :D
21:45:26 <zaitcev> kota_: thanks
21:45:29 <mattoliverau> +1
21:45:55 <zaitcev> Although I'm a little sick of everything remote.
21:46:25 <kota_> exactly, I like the face to face meeting we did in the past.
21:46:32 <timburke__> yeah, me too. with a little luck, maybe we'll be able to actually see each other in person the next time
21:46:37 <mattoliverau> yeah. but at least it makes it easier to attend conferences you might not be able to convince someone to send you too.. esp if your on the otherside of the world :)
21:47:20 <mattoliverau> but +100 for missing going to f2f conferences
21:49:14 <timburke__> all right, i think i'll call it then
21:49:25 <timburke__> thank you all for coming, and thank you for working on swift!
21:49:29 <timburke__> #endmeeting