Wednesday, 2021-05-12

*** martial has quit IRC00:26
*** bbowen has joined #openstack-meeting00:28
*** bbowen_ has quit IRC00:29
*** yasufum has joined #openstack-meeting00:29
*** bbowen has quit IRC00:39
*** bbowen has joined #openstack-meeting00:39
*** dsariel has joined #openstack-meeting00:41
*** bbowen_ has joined #openstack-meeting00:44
*** bbowen has quit IRC00:45
*** bbowen__ has joined #openstack-meeting00:47
*** bbowen_ has quit IRC00:49
*** bbowen__ has quit IRC00:51
*** dsariel has quit IRC00:53
*** whoami-rajat has quit IRC01:23
*** Carlotronics has quit IRC01:26
*** Carlotronics has joined #openstack-meeting01:33
*** bbowen has joined #openstack-meeting01:59
*** bbowen has joined #openstack-meeting02:01
*** freerunner has quit IRC02:06
*** freerunner has joined #openstack-meeting02:06
*** gyee has quit IRC02:20
*** jmasud has quit IRC02:23
*** rcernin has quit IRC02:23
*** rakhmerov has joined #openstack-meeting02:32
*** evrardjp has quit IRC02:33
*** evrardjp has joined #openstack-meeting02:33
*** jmasud has joined #openstack-meeting02:59
*** rcernin has joined #openstack-meeting03:09
*** rcernin has quit IRC03:16
*** rcernin has joined #openstack-meeting03:25
*** rcernin has quit IRC03:26
*** rcernin has joined #openstack-meeting03:27
*** whoami-rajat_ has joined #openstack-meeting04:44
*** psahoo has joined #openstack-meeting04:47
*** manpreet has joined #openstack-meeting04:47
*** jmasud has quit IRC05:03
*** jmasud has joined #openstack-meeting05:09
*** lpetrut has joined #openstack-meeting05:27
*** udesale has joined #openstack-meeting05:33
*** macz_ has joined #openstack-meeting05:37
*** macz_ has quit IRC05:43
*** ralonsoh has joined #openstack-meeting06:07
*** slaweq has joined #openstack-meeting06:16
*** lpetrut has quit IRC06:21
*** rh-jlabarre has joined #openstack-meeting06:22
*** rh-jelabarre has quit IRC06:22
*** rakhmerov has quit IRC06:27
*** rakhmerov has joined #openstack-meeting06:29
*** ralonsoh has quit IRC06:48
*** ralonsoh has joined #openstack-meeting06:49
*** tinwood has quit IRC06:54
*** tinwood has joined #openstack-meeting06:57
*** jhesketh has quit IRC07:08
*** tosky has joined #openstack-meeting07:09
*** dklyle has quit IRC07:15
*** macz_ has joined #openstack-meeting07:17
*** macz_ has quit IRC07:21
*** rpittau|afk is now known as rpittau07:26
*** jmasud has quit IRC07:41
*** macz_ has joined #openstack-meeting08:09
*** macz_ has quit IRC08:13
*** ociuhandu has joined #openstack-meeting08:17
*** ociuhandu has quit IRC08:18
*** ociuhandu has joined #openstack-meeting08:18
*** jhesketh has joined #openstack-meeting08:23
*** rpittau is now known as rpittau|bbl09:07
*** ociuhandu has quit IRC09:08
*** udesale has quit IRC09:09
*** udesale has joined #openstack-meeting09:11
*** ociuhandu has joined #openstack-meeting09:27
*** dsariel has joined #openstack-meeting09:32
*** ociuhandu has quit IRC09:35
*** ociuhandu has joined #openstack-meeting09:47
*** dsariel has quit IRC09:50
*** macz_ has joined #openstack-meeting10:09
*** macz_ has quit IRC10:14
*** udesale has quit IRC10:16
*** rpittau|bbl is now known as rpittau10:19
*** ociuhandu has quit IRC10:20
*** macz_ has joined #openstack-meeting10:30
*** udesale has joined #openstack-meeting10:32
*** macz_ has quit IRC10:34
*** b1airo has quit IRC10:44
*** yasufum has quit IRC10:47
*** ociuhandu has joined #openstack-meeting10:51
*** dsariel has joined #openstack-meeting10:53
*** ociuhandu has quit IRC10:55
*** yasufum has joined #openstack-meeting11:04
*** ociuhandu has joined #openstack-meeting11:07
*** jmasud has joined #openstack-meeting11:12
*** rcernin has quit IRC11:23
*** rcernin has joined #openstack-meeting11:24
*** jmasud has quit IRC11:32
*** ociuhandu has quit IRC11:33
*** rcernin has quit IRC11:35
*** rakhmerov has quit IRC11:42
*** ociuhandu has joined #openstack-meeting11:45
*** rakhmerov has joined #openstack-meeting11:48
*** yasufum has quit IRC12:05
*** rakhmerov has quit IRC12:28
*** macz_ has joined #openstack-meeting12:28
*** psahoo_ has joined #openstack-meeting12:28
*** bbowen has quit IRC12:29
*** bbowen has joined #openstack-meeting12:29
*** psahoo has quit IRC12:32
*** macz_ has quit IRC12:33
*** macz_ has joined #openstack-meeting13:05
*** yasufum has joined #openstack-meeting13:08
*** macz_ has quit IRC13:09
*** e0ne has joined #openstack-meeting13:12
*** whoami-rajat_ is now known as whoami-rajat13:17
*** ociuhandu has quit IRC13:25
*** timburke has quit IRC13:26
*** timburke has joined #openstack-meeting13:27
*** SpamapS has quit IRC13:47
*** trandles has quit IRC13:48
*** SpamapS has joined #openstack-meeting13:49
*** ociuhandu has joined #openstack-meeting13:56
*** ociuhandu has quit IRC14:02
*** ociuhandu has joined #openstack-meeting14:12
*** ociuhandu_ has joined #openstack-meeting14:17
*** ociuhandu has quit IRC14:17
*** yasufum has quit IRC14:24
*** rakhmerov has joined #openstack-meeting15:00
*** yasufum has joined #openstack-meeting15:00
*** rakhmerov has quit IRC15:01
*** rakhmerov has joined #openstack-meeting15:03
*** macz_ has joined #openstack-meeting15:06
*** rakhmerov has quit IRC15:07
*** macz_ has quit IRC15:10
*** dklyle has joined #openstack-meeting15:20
*** mlavalle has joined #openstack-meeting15:22
*** rakhmerov has joined #openstack-meeting15:22
*** macz_ has joined #openstack-meeting15:26
*** gyee has joined #openstack-meeting15:26
*** e0ne has quit IRC15:30
*** ociuhandu_ has quit IRC15:37
*** ociuhandu has joined #openstack-meeting15:40
*** udesale has quit IRC15:47
*** jmasud has joined #openstack-meeting15:59
*** e0ne has joined #openstack-meeting16:03
*** ociuhandu_ has joined #openstack-meeting16:07
*** ociuhandu has quit IRC16:11
*** rakhmerov has quit IRC16:11
*** yasufum has quit IRC16:11
*** ociuhandu_ has quit IRC16:11
*** slaweq has quit IRC16:11
*** rpittau is now known as rpittau|afk16:12
*** rakhmerov has joined #openstack-meeting16:14
*** M0nk3Ee has quit IRC16:17
*** M0nk3Ee has joined #openstack-meeting16:17
*** rakhmerov has quit IRC16:19
*** ociuhandu has joined #openstack-meeting16:21
*** ociuhandu has quit IRC16:30
*** psahoo_ has quit IRC16:32
*** e0ne has quit IRC16:39
*** rakhmerov has joined #openstack-meeting16:49
*** jmasud has quit IRC16:54
*** rakhmerov has quit IRC16:57
*** e0ne has joined #openstack-meeting17:02
*** ralonsoh has quit IRC17:02
*** jmasud has joined #openstack-meeting17:25
*** rakhmerov has joined #openstack-meeting17:30
*** rakhmerov has quit IRC17:37
*** ociuhandu has joined #openstack-meeting17:40
*** ociuhandu has quit IRC17:46
*** rakhmerov has joined #openstack-meeting18:08
*** rakhmerov has quit IRC18:18
*** bbowen has quit IRC18:37
*** e0ne has quit IRC18:45
*** jmasud has quit IRC18:46
*** macz_ has quit IRC19:16
*** e0ne has joined #openstack-meeting19:23
*** macz_ has joined #openstack-meeting19:32
*** jmasud has joined #openstack-meeting19:41
*** ociuhandu has joined #openstack-meeting19:44
*** slaweq has joined #openstack-meeting19:48
*** bbowen has joined #openstack-meeting20:02
*** macz_ has quit IRC20:08
*** macz_ has joined #openstack-meeting20:08
*** ociuhandu has quit IRC20:12
*** rakhmerov has joined #openstack-meeting20:14
*** rakhmerov has quit IRC20:26
*** dansmith has quit IRC20:29
*** jmasud has quit IRC20:53
*** acoles has joined #openstack-meeting21:00
timburke#startmeeting swift21:00
openstackMeeting started Wed May 12 21:00:30 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
timburkeas usual, the agenda's at
timburkethough i only just updated it ;-)21:02
timburke#topic Python 3.10 beta21:03
*** openstack changes topic to "Python 3.10 beta (Meeting topic: swift)"21:03
timburkei just wanted to call attention to this -- it seems like there may be some work to get tests running under py310, and i don't really want it to be a fire-drill come october21:04
*** slaweq has quit IRC21:04
timburkei've started playing around with it. eventlet's not working yet, nose is busted (and at this point, unlikely to be fixed)21:05
mattoliveraugood thinking. might need to see if I can create a venv of it.21:05
mattoliverauoh wow21:05
timburkepython-swiftclient's fine, though -- as long as you've got some other test runner21:06
timburkegood news is that distros seemed to be pretty quick on the packaging front; i got it fine on fedora and ubuntu (via deadsnakes ppa, looks like)21:06
*** dansmith has joined #openstack-meeting21:06
timburkepyeclib needed an update, but it was tame enough that i went ahead and merged it when i confirmed the gate was happy with it21:07
timburke - Use Py_ssize_t when calling PyArg_Parse21:08
mattoliverauif we wont be able to use nose anymore, whats the alternative.. will be need to migrate to something like pytest.. or wait and hope someone fixes nose. or is {o,s}testr still a supported thing?21:09
timburkestestr seems to still be the "preferred" way in openstack, as best i can tell. it *also* doesn't work with py310 right now, though -- but i think the next testtools release should fix it21:10
timburkepersonally, i kinda like pytest -- and it's working *today*21:10
*** tdasilva_ has joined #openstack-meeting21:11
toskystestr is definitely alive (ostestr is totally deprecated and not used anymore)21:11
timburketl;dr: py310 is coming, and it's likely going to require some changes which we should be on the lookout for. if anyone's interested in staying on the bleeding edge, try it out and see what's broken ;-)21:11
*** tdasilva_ is now known as tdasilva21:11
timburkeany questions or comments?21:12
acolestimburke: thanks for the headsup21:12
mattoliverautosky: thanks!21:12
timburkeall right, on to updates!21:13
timburke#topic sharding21:13
*** openstack changes topic to "sharding (Meeting topic: swift)"21:13
timburkeso we've merged the tombstone counting21:14
timburkewhat are the next steps? acoles, mattoliverau21:14
acolesone last known piece of the shrinking jigsaw is getting a better handle on a shard being sharded enough before shrinking, which mattoliverau has been working on21:15
mattoliverauI've been playing more with the extra ACTIVE steps based on discussions from PTG.21:16
mattoliverauthe chain starts at
acolesI'm hopeful that the latest idea of looking at rows cleaved will work out21:17
mattoliverauit starts with adding rows_cleaved to the CleaveContexts so we can tell if a context is a smaller or larger21:17
mattoliverauyeah, thanks acoles me too :)21:18
*** zaitcev has joined #openstack-meeting21:18
mattoliverauI've attempting to see if adding an extra CLEAVED state will help with the edge case where fast cleaving shards become CLEAVED and therefore responsible for listings21:18
mattoliverauproducing holes until others shard.21:19
mattoliverauit's the last in the chain.. and it's still very much a WIP and still trying it out.21:19
timburkesounds good21:20
mattoliverauI also noticed acoles rebase a bunch of follow up sharding patches based around unification of the sharder and s-m-s-r tool21:20
acolesI have a couple of patches around sharder config, first is to share config loading between sharder and s-m-s-r21:20
mattoliverauso I want to take a look at those21:21
mattoliveraulol, same wave length :)21:21
acolesand based on that, a proposal to add absolute config values for shrink threshold etc
timburkeyeah, that seems like it ought to reduce confusion for ops -- good plan21:22
acolesthe percent based config is convenient but I am concerned that it is harder to explain to ops the consequences of config changes that have knock-on effects21:22
timburkenext up21:23
timburke#topic relinker21:23
*** openstack changes topic to "relinker (Meeting topic: swift)"21:23
acolesit keeps getting better!21:23
timburkei put up a change to clean up replication locks, which were preventing the old parts from getting cleaned up properly for EC policies21:24
timburkeand while investigating that, i realized there was a bug that would let two different processes hold partition locks if we're also trying to delete those lock files21:25
acoleswere we already deleting locks ? or is this a future-bug-waiting-to-happen?21:26
acolesI guess we already are in relinker cleanup?21:27
acolesits just the replication locks that we're adding21:27
acolesbut, any other places locks get deleted?21:27
timburkewe're already deleting locks -- but just during relinking21:28
timburkeafter that, i think i'd prioritize - relinker: Log and recon on SIGTERM signal21:30
mattoliveraureplication "should" be on hold on the policy that's being increased.. so it shouldn't be taking locks?21:31
acolesoh yeah, mattoliverau 's relinker recon patch merged which is really useful :)21:31
timburkethe partition lock is still taken during object writes, so we can update hashes.invalid21:32
mattoliverauthat ended up being more patchsets then I expected :P21:32
timburkei'm not as worried about the replication lock (for the reason mattoliverau pointed out), but i updated the cleanup patch to take that lock before deleting it as well21:33
*** dsariel has quit IRC21:33
acolesmattoliverau: replication locks may not be beng taken but the lock files were preventing the partition rmdir21:33
*** dsariel has joined #openstack-meeting21:33
timburkenext up21:34
timburke#topic stale EC frags21:34
*** openstack changes topic to "stale EC frags (Meeting topic: swift)"21:34
timburkewe landed - Quarantine stale EC fragments after checking handoffs21:34
timburkedid we have any noteworthy follow-ups that shook out of that?21:34
mattoliverau\o/ nice one acoles21:35
zaitcevI'm curious too.21:35
acolesI don't recall any follow up patches21:35
acolesah, did someone take a crack at the TODO re. finding the actual wanted frag?21:36
acolesbut that's more another topic than follow up21:36
timburkeoh yeah, mattoliverau did - WIP: Single frag iter hack21:37
timburkelooked fairly sane, and not as much code as i'd feared21:37
mattoliveraunot really related to stale EC frags, but I have been nerd sniped and playing with trying to use a fragment if we find it (ie duplication) and has lead to adding tracking to last primaries in the ring so rebalances in the reconstructor could be faster because we'd actually check the old primary for the fragment.21:37
mattoliverauring last primary tracking:
acoleswe'll be trying out the stale EC frag quarantine this week and hoping to see a bunch of lonely old frags (and annoying error logs) disappear21:38
timburkeand that previous-primary tracking seems likely to be useful in other contexts21:39
*** jmasud has joined #openstack-meeting21:39
timburkebe able to have single-replica policies where a rebalance doesn't automatically ensure availability issues ;-)21:40
timburkeall right; next up21:40
timburke#topic dark data watcher21:40
*** openstack changes topic to "dark data watcher (Meeting topic: swift)"21:40
timburkei've still not gotten around to looking at this problem yet. but i wanted to keep the topic as a reminder that i should21:41
zaitcevWell, we have 2 problems, AFAIK.21:42
zaitcev#1 is incompatibility with the sharded containers, which you tackled and then Alistair didn't like, and I don't understand what exactly he didn't like.21:42
zaitcev#2 is conflict with async pendings and that one I am fixing21:43
zaitcevOr, fixed. But I broke probe tests, so I tried to run baseline tests, and it turned out that they no longer work for me.21:43
zaitcevI dealt with that just today, and I'm getting to fixing up the tests.21:43
zaitcevOh, and thanks for landing my antisudo thing21:44
timburkecool! there was more progress on this than i'd thought :D21:44
timburkethanks zaitcev21:44
zaitcevBTW. In a couple of places we're running swift-manage-shard-ranges, which turned out to be impossible anymore for me, because I do not run an installed cluster. My code is in git repo, it's not in /usr/local. I use PYTHONPATH to make it work.21:45
zaitcevI scratched my head a little and threw a stub into ~/bin, moved on with my life.21:45
zaitcevI have a feeling you guys always run probe tests in some kind of VM or at least a container anymore.21:46
zaitcevMaybe I should do it too.21:46
zaitcevThat's it, thanks for listening.21:46
mattoliverauyeah I run it in an saio which installs swift (pip install -e .).. so maybe that's why I've never had the problem21:47
acoleszaitcev: I have always run probe tests in a VM, these days using vagrant-SAIO21:47
timburkeall right21:47
zaitcevMakes sense, thanks guys21:47
timburke#topic open discussion21:47
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:47
timburkeanything else we should bring up this week?21:47
zaitcevHave anyone heard of Minio?21:48
timburkeheard of it, though i haven't played with it21:48
zaitcevA friend of mine wanted an S3, but he installed that thing instead of Swift. Me sad.21:48
zaitcevApparently it's way easier to install in modern servers, it comes as a ready-to-go docker image.21:49
zaitcevIt may not be something NVIDIA cares about, but I think we ought to think about some kind of canned single-node.21:49
zaitcevBetter than SAIO21:49
zaitcevBTW, I made a post
zaitcevThen, a(nother) friend of mine wrote me and wagged his finger how NVIDIA is uppercase now tsk tsk.21:51
zaitcevI had to change all references.21:51
zaitcevSo, anyway, do we have some kind of ready-made Dockerfile? I would not be surprised if it's in the tree already, I'm just not aware.21:52
acoleswe do but I have never looked at it until right now !21:52
timburkeyeah, we should -- and
acolestdasilva gets the blame :D21:54
timburkeiirc it was useful as a test target for 1space21:55
mattoliveraudont we have
mattoliverauthat's being build by zuul21:55
mattoliverautdasilva: did that, he's pretty awesome21:56
timburkeyup! again, not something i've really played with, though :-(21:56
timburkei keep meaning to21:56
timburkeall right -- i'm gonna call it21:58
timburkethank you all for coming, and thank you for working on swift!21:58
*** openstack changes topic to "OpenStack Meetings ||"21:58
openstackMeeting ended Wed May 12 21:58:30 2021 UTC.  Information about MeetBot at . (v 0.1.4)21:58
openstackMinutes (text):
*** acoles has left #openstack-meeting21:58
*** zaitcev has left #openstack-meeting21:59
*** jmasud has quit IRC22:00
*** timburke has quit IRC22:08
*** timburke has joined #openstack-meeting22:09
*** jmasud has joined #openstack-meeting22:17
*** rakhmerov has joined #openstack-meeting22:22
*** rakhmerov has quit IRC22:26
*** e0ne has quit IRC22:51
*** tosky has quit IRC23:01
*** rcernin has joined #openstack-meeting23:07
*** macz_ has quit IRC23:08
*** manpreet has quit IRC23:26
*** mlavalle has quit IRC23:56

Generated by 2.17.2 by Marius Gedminas - find it at!