21:00:51 <timburke> #startmeeting swift
21:00:51 <openstack> Meeting started Wed Apr  7 21:00:51 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:54 <openstack> The meeting name has been set to 'swift'
21:01:00 <timburke> who's here for the swift meeting?
21:01:17 <rledisez> o/
21:01:28 <mattoliverau> o/
21:02:53 <acoles> o/
21:03:45 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:53 <clayg> o/
21:04:08 <timburke> but first, i want to insert a topic ;-)
21:04:13 <timburke> #topic rledisez
21:05:01 <zaitcevx> o/
21:05:12 <timburke> rledisez just proposed https://review.opendev.org/c/openstack/swift/+/785276 -- it looks like he won't be working on swift much more
21:05:44 <timburke> i just wanted to say, thank you for doing so much for (and with) swift over the years
21:06:00 <rledisez> it's really been a pleasure all these years to work with you guys. I can't imagine a better team for an open source project
21:06:40 <rledisez> I'll miss Swift for sure, but I know you'll continue to make it the best object storage software ;)
21:06:50 <zaitcevx> What does it mean for Swift at OVH? Are you guys shutting it down, and if yes, what's the replacement? Something OpenIOish?
21:07:52 <rledisez> quickly: short term there is no plan to shut it down. you don't shutdown hundreds of PB easily 😅. long term it will surely be replaced by "something openioish"
21:07:53 <timburke> i'd forgotten to mention it before, but we had an internal user looking for some swift storage who said, "I like using ovh for my personal projects and I was really quite pleased with how easy it was to hook up the swift object storage to my build system" :-)
21:08:07 <acoles> rledisez: thanks for all your fantastic contributions, you'll be missed. au revoir
21:08:21 <mattoliverau> :(
21:08:48 <clayg> rledisez: thank you!
21:09:35 <zaitcevx> Thanks rledisez for all the work, and I'm bummed that queue-less request submission didn'
21:09:42 <zaitcevx> t work with py2
21:09:59 <zaitcevx> Completely stumped me.
21:10:35 <rledisez> zaitcevx: yeah, one of the most brainfuck thing I failed I think
21:11:35 <timburke> i hope we eventually cross paths again :-) always feel free to pop in and see how we're doing
21:11:43 <timburke> especially at events like...
21:11:48 <timburke> #topic ptg
21:12:01 <clayg> 🤣 nice transistion
21:12:22 <timburke> looks like we're starting to get the etherpad filled out
21:12:39 <timburke> #link https://etherpad.opendev.org/p/swift-ptg-xena
21:13:01 <timburke> i'll try to get some more stuff on there today
21:13:26 <mattoliverau> yeah I just brandumped some possible topics
21:13:56 <timburke> (though part of me feels like we could spend an *awful lot* of time just thinking hard about tsync & replication 😉)
21:14:58 <timburke> next up
21:15:13 <timburke> #topic part power increase
21:15:42 <timburke> the relinker now has a workers option! and it defaults to <# disks>
21:17:09 <timburke> i still want to play with moving more than one step at a time -- so https://review.opendev.org/c/openstack/swift/+/781591 is rebased (though it still needs tests)
21:17:38 <timburke> and i know mattoliverau is still working on getting relinker status into recon
21:18:16 <mattoliverau> nice, so it's getting better and better :)
21:18:48 <clayg> it's already pretty damn good
21:19:26 <clayg> @timburke @mattoliverau is there any doc update that would be required to help people with their first PPI (or have most of the changes been "in the backend")
21:19:49 <timburke> at nvidia, we just finished another increase and it was our smoothest one yet -- down to ~27hrs, and no hacking together of scripts to run against prod :-)
21:20:17 <mattoliverau> relinker does help you find bad disks though :P
21:21:03 <mattoliverau> clayg: oh yeah we should look at our current ppi doc, there are alot more options so we should mention them.
21:21:22 <timburke> clayg, excellent question! i think users get most of the new goodness "for free", but i'll make a note to review our existing docs. surely, it should have something about taking a config file if it doesn't already
21:22:14 <timburke> that's all i've got for that
21:22:22 <timburke> #topic sharding / shrinking
21:23:02 <clayg> on shrinking I'm most excited about counting tombstones for the purpose of shrinking 🤩 (go acoles !!!)
21:23:29 <acoles> oh yeah, I made some progress on that since last week https://review.opendev.org/c/openstack/swift/+/782832
21:23:46 <mattoliverau> nice
21:23:56 <timburke> yeah, that seems great! and we get it as a side-effect of other work we're already doing!
21:24:28 <acoles> but it messes up probe tests, some of which rely on (auto)shrinking immediately after deleting objects :( so I need to figure out the best approach to rectify that
21:25:16 <acoles> for now I added a --reclaim-age option to sharder cli so I can get rid of the tombstones immediately (With --reclaim-age=0) in the probe test
21:25:35 <timburke> we already open up the broker, right? maybe do a manual reclaim? or that ;-)
21:26:23 <acoles> it's a little frustrating how the probe tests have to work around auto-sharding doing its thing
21:26:31 <clayg> there's some other test where we say "go forward in time a reclaim age" as an explination of reclaim-age=0
21:27:10 <clayg> acoles: yeah, do we have probetests running with both auto-sharding AND using s-m-s-r directly?
21:27:37 <acoles> *not yet* ... but that was another option I considered
21:28:10 <acoles> i.e. to execute the shrinks 'manually' with s-m-s-r in place of the autoshrinking points in the test
21:28:28 <timburke> clayg, all in one test, or some tests that exercise one, and some that exercise the other? we've got the latter but not the former
21:28:49 <acoles> tes, what timburke said ^^
21:28:53 <acoles> yes*
21:29:23 <timburke> (though as i recall that required that we introduce a --no-auto-sharding option or the like)
21:29:31 <clayg> yes, I wasn't clear - that matches my understanding
21:29:41 <timburke> oh yeah, and zaitcevx proposed https://review.opendev.org/c/openstack/swift/+/784617 to have swift-recon aggregate sharding info
21:29:46 <clayg> but I guess Al is looking at breaking existing auto-shard tests 🤔
21:30:08 <timburke> does that mean you're been doing some sharding on train? how's it going?
21:30:37 <zaitcevx> That patch can be expanded for something sharding-specific, maybe some calculations or whatnot.
21:30:40 <acoles> clayg: yes, some existing auto- tests break because they don't shrink anymore :'( cos of tombstones
21:30:58 <clayg> @acoles 👍
21:31:31 <zaitcevx> timburke: I'm still experimenting with Train, but I see no showstoppers yet. Anyway, any further backports to Train get too hard. I tried to walk a little up the tree. So, bugfixes only now.
21:31:33 <clayg> so you're considering "migrating" the "broken" auto-tests to manual (do we have manual tests of shrinking already?)
21:33:14 <acoles> clayg: we do already have manual test for compact. I'm undecided how to deal with the breaking auto tests - either migrate wholly or partly to manual , or allow reclaim-age to be overridden
21:34:24 <mattoliverau> mayber the fake sharder for autosharding can have a reclaim of 0 or something short that we can sleep on, or can mock a reclaim or something.
21:34:43 <acoles> I'm leaning towards migrating partly to manual (where we need to override auto- behaviour)
21:35:55 <mattoliverau> once we have shrinking/compact working, tested and happy with it, and analyze and repair.. we'd be closer to re-vamping autosharding to maybe actaully be stable ;)
21:36:22 <mattoliverau> we almost have all the peices.. we just need to gain some confidence in them first
21:37:41 <mattoliverau> but yeah for now whatever we need (part manual or whatever) to make sure the probes pass. it's good work and tomb stones added to the mix makes it much nicer! great work
21:37:50 <acoles> I suppose the probe test failures at least add confidence that the tombstones are being considered before shrinking :D
21:38:00 <mattoliverau> lol
21:38:03 <clayg> WTG probe tests!
21:38:15 <timburke> i was just thinking that -- they're breaking because we're doing the right thing now :-)
21:38:34 <timburke> all right
21:38:44 <timburke> one more impromptu topic
21:38:51 <timburke> #topic meeting next week
21:39:13 <timburke> i'm going to be out, camping -- does anyone want to chair the meeting next week?
21:39:18 <zaitcevx> I'm going to be away next week. Well, I hope.
21:39:31 <timburke> or we just skip it :-)
21:39:44 <mattoliverau> how many is left before the PTG
21:39:46 <acoles> it's my birthday so I may (will) not come - I feel a cancellation coming here :)
21:39:49 <kota_> rledisez: thanks for working for Swift, I quickly catch up the meeting back log and found your proposed https://review.opendev.org/c/openstack/swift/+/785276
21:40:18 <timburke> it'd be our last meeting before the ptg, but otoh, we haven't had that many new topics to talk about
21:40:19 <acoles> PTG is week after right?
21:40:43 <kota_> oops, i wrote to meeting channel... sorry for the interruption :/
21:40:46 <mattoliverau> cool. Fine to skip so long as poeple fill out the etherpad on topics. That
21:40:59 <mattoliverau> that's the only thing we'd remind in the meeting next week :)
21:41:46 <timburke> and then, of course, we'll skip the one during the ptg since we'll have plenty of contact already
21:42:13 <timburke> so, next meeting: April 28th
21:42:17 <mattoliverau> yeah, well we'll be in virtual person that week :)
21:42:28 <timburke> that's all i've got
21:42:34 <timburke> #topic open discussion
21:43:29 <timburke> anything else we ought to bring up?
21:44:09 <zaitcevx> I'm good.
21:45:20 <timburke> all right
21:45:35 <timburke> thank you all for coming, and thank you for working on swift!
21:45:44 <timburke> #endmeeting