21:00:30 #startmeeting swift 21:00:31 Meeting started Wed May 12 21:00:30 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:32 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:34 The meeting name has been set to 'swift' 21:00:37 who's here for the swift meeting? 21:00:49 o/ 21:01:11 o/ 21:01:57 o/ 21:02:09 o/ 21:02:16 as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:02:31 though i only just updated it ;-) 21:03:06 #topic Python 3.10 beta 21:04:01 i just wanted to call attention to this -- it seems like there may be some work to get tests running under py310, and i don't really want it to be a fire-drill come october 21:05:31 i've started playing around with it. eventlet's not working yet, nose is busted (and at this point, unlikely to be fixed) 21:05:42 good thinking. might need to see if I can create a venv of it. 21:05:55 oh wow 21:05:58 awesome 21:06:04 python-swiftclient's fine, though -- as long as you've got some other test runner 21:06:50 good news is that distros seemed to be pretty quick on the packaging front; i got it fine on fedora and ubuntu (via deadsnakes ppa, looks like) 21:07:33 pyeclib needed an update, but it was tame enough that i went ahead and merged it when i confirmed the gate was happy with it 21:08:11 https://review.opendev.org/c/openstack/pyeclib/+/790537 - Use Py_ssize_t when calling PyArg_Parse 21:09:00 if we wont be able to use nose anymore, whats the alternative.. will be need to migrate to something like pytest.. or wait and hope someone fixes nose. or is {o,s}testr still a supported thing? 21:10:04 stestr seems to still be the "preferred" way in openstack, as best i can tell. it *also* doesn't work with py310 right now, though -- but i think the next testtools release should fix it 21:10:33 personally, i kinda like pytest -- and it's working *today* 21:10:51 kk 21:11:37 stestr is definitely alive (ostestr is totally deprecated and not used anymore) 21:11:53 tl;dr: py310 is coming, and it's likely going to require some changes which we should be on the lookout for. if anyone's interested in staying on the bleeding edge, try it out and see what's broken ;-) 21:12:27 any questions or comments? 21:12:41 timburke: thanks for the headsup 21:12:45 tosky: thanks! 21:13:40 all right, on to updates! 21:13:45 #topic sharding 21:14:09 so we've merged the tombstone counting 21:14:41 what are the next steps? acoles, mattoliverau 21:15:57 one last known piece of the shrinking jigsaw is getting a better handle on a shard being sharded enough before shrinking, which mattoliverau has been working on 21:16:09 I've been playing more with the extra ACTIVE steps based on discussions from PTG. 21:16:57 the chain starts at https://review.opendev.org/c/openstack/swift/+/789471 21:17:33 I'm hopeful that the latest idea of looking at rows cleaved will work out 21:17:40 it starts with adding rows_cleaved to the CleaveContexts so we can tell if a context is a smaller or larger 21:18:00 yeah, thanks acoles me too :) 21:18:54 I've attempting to see if adding an extra CLEAVED state will help with the edge case where fast cleaving shards become CLEAVED and therefore responsible for listings 21:19:05 producing holes until others shard. 21:19:30 it's the last in the chain.. and it's still very much a WIP and still trying it out. 21:20:50 sounds good 21:20:53 I also noticed acoles rebase a bunch of follow up sharding patches based around unification of the sharder and s-m-s-r tool 21:20:59 I have a couple of patches around sharder config, first is to share config loading between sharder and s-m-s-r 21:21:00 so I want to take a look at those 21:21:00 https://review.opendev.org/c/openstack/swift/+/778989 21:21:16 lol, same wave length :) 21:21:43 and based on that, a proposal to add absolute config values for shrink threshold etc https://review.opendev.org/c/openstack/swift/+/778990 21:22:17 yeah, that seems like it ought to reduce confusion for ops -- good plan 21:22:27 +1 21:22:42 the percent based config is convenient but I am concerned that it is harder to explain to ops the consequences of config changes that have knock-on effects 21:23:25 next up 21:23:28 #topic relinker 21:23:43 it keeps getting better! 21:24:14 i put up a change to clean up replication locks, which were preventing the old parts from getting cleaned up properly for EC policies 21:24:20 https://review.opendev.org/c/openstack/swift/+/790305 21:25:32 and while investigating that, i realized there was a bug that would let two different processes hold partition locks if we're also trying to delete those lock files 21:25:35 https://review.opendev.org/c/openstack/swift/+/791022 21:26:39 were we already deleting locks ? or is this a future-bug-waiting-to-happen? 21:27:13 I guess we already are in relinker cleanup? 21:27:26 its just the replication locks that we're adding 21:27:51 but, any other places locks get deleted? 21:28:43 we're already deleting locks -- but just during relinking 21:28:54 ok 21:30:53 after that, i think i'd prioritize https://review.opendev.org/c/openstack/swift/+/788413 - relinker: Log and recon on SIGTERM signal 21:31:19 replication "should" be on hold on the policy that's being increased.. so it shouldn't be taking locks? 21:31:37 oh yeah, mattoliverau 's relinker recon patch merged which is really useful :) 21:32:08 the partition lock is still taken during object writes, so we can update hashes.invalid 21:32:14 that ended up being more patchsets then I expected :P 21:32:23 hehe 21:33:01 i'm not as worried about the replication lock (for the reason mattoliverau pointed out), but i updated the cleanup patch to take that lock before deleting it as well 21:33:10 mattoliverau: replication locks may not be beng taken but the lock files were preventing the partition rmdir 21:34:10 next up 21:34:17 #topic stale EC frags 21:34:45 we landed https://review.opendev.org/c/openstack/swift/+/788833 - Quarantine stale EC fragments after checking handoffs 21:34:58 did we have any noteworthy follow-ups that shook out of that? 21:35:16 \o/ nice one acoles 21:35:41 I'm curious too. 21:35:58 I don't recall any follow up patches 21:36:30 ah, did someone take a crack at the TODO re. finding the actual wanted frag? 21:36:59 but that's more another topic than follow up 21:37:00 oh yeah, mattoliverau did https://review.opendev.org/c/openstack/swift/+/790374 - WIP: Single frag iter hack 21:37:29 looked fairly sane, and not as much code as i'd feared 21:37:33 not really related to stale EC frags, but I have been nerd sniped and playing with trying to use a fragment if we find it (ie duplication) and has lead to adding tracking to last primaries in the ring so rebalances in the reconstructor could be faster because we'd actually check the old primary for the fragment. 21:38:19 ring last primary tracking: https://review.opendev.org/c/openstack/swift/+/790550 21:38:26 we'll be trying out the stale EC frag quarantine this week and hoping to see a bunch of lonely old frags (and annoying error logs) disappear 21:39:17 and that previous-primary tracking seems likely to be useful in other contexts 21:40:01 be able to have single-replica policies where a rebalance doesn't automatically ensure availability issues ;-) 21:40:37 all right; next up 21:40:45 #topic dark data watcher 21:41:25 i've still not gotten around to looking at this problem yet. but i wanted to keep the topic as a reminder that i should 21:42:11 Well, we have 2 problems, AFAIK. 21:42:40 true 21:42:45 #1 is incompatibility with the sharded containers, which you tackled and then Alistair didn't like, and I don't understand what exactly he didn't like. 21:43:02 #2 is conflict with async pendings and that one I am fixing 21:43:33 Or, fixed. But I broke probe tests, so I tried to run baseline tests, and it turned out that they no longer work for me. 21:43:53 I dealt with that just today, and I'm getting to fixing up the tests. 21:44:08 Oh, and thanks for landing my antisudo thing 21:44:19 cool! there was more progress on this than i'd thought :D 21:44:31 thanks zaitcev 21:45:12 BTW. In a couple of places we're running swift-manage-shard-ranges, which turned out to be impossible anymore for me, because I do not run an installed cluster. My code is in git repo, it's not in /usr/local. I use PYTHONPATH to make it work. 21:45:39 I scratched my head a little and threw a stub into ~/bin, moved on with my life. 21:46:10 I have a feeling you guys always run probe tests in some kind of VM or at least a container anymore. 21:46:22 Maybe I should do it too. 21:46:44 That's it, thanks for listening. 21:47:16 yeah I run it in an saio which installs swift (pip install -e .).. so maybe that's why I've never had the problem 21:47:18 zaitcev: I have always run probe tests in a VM, these days using vagrant-SAIO 21:47:40 all right 21:47:42 Makes sense, thanks guys 21:47:46 #topic open discussion 21:47:56 anything else we should bring up this week? 21:48:02 Well 21:48:08 Have anyone heard of Minio? 21:48:24 heard of it, though i haven't played with it 21:48:38 A friend of mine wanted an S3, but he installed that thing instead of Swift. Me sad. 21:49:02 Apparently it's way easier to install in modern servers, it comes as a ready-to-go docker image. 21:49:53 It may not be something NVIDIA cares about, but I think we ought to think about some kind of canned single-node. 21:49:58 Better than SAIO 21:50:31 BTW, I made a post https://zaitcev.livejournal.com/262542.html 21:51:14 Then, a(nother) friend of mine wrote me and wagged his finger how NVIDIA is uppercase now tsk tsk. 21:51:30 I had to change all references. 21:52:11 So, anyway, do we have some kind of ready-made Dockerfile? I would not be surprised if it's in the tree already, I'm just not aware. 21:52:59 we do but I have never looked at it until right now ! 21:53:32 yeah, we should -- https://github.com/openstack/swift/blob/master/Dockerfile and https://github.com/openstack/swift/blob/master/Dockerfile-py3 21:54:03 tdasilva gets the blame :D 21:55:00 iirc it was useful as a test target for 1space 21:55:28 dont we have https://hub.docker.com/repository/docker/openstackswift/saio 21:55:40 that's being build by zuul 21:55:46 *built 21:56:06 tdasilva: did that, he's pretty awesome 21:56:43 yup! again, not something i've really played with, though :-( 21:56:48 i keep meaning to 21:58:09 all right -- i'm gonna call it 21:58:21 thank you all for coming, and thank you for working on swift! 21:58:30 #endmeeting