Wednesday, 2021-04-28

« Tuesday, 2021-04-27 Index Thursday, 2021-04-29 »
*** martial has quit IRC00:02
*** jmasud has joined #openstack-meeting00:14
*** rcernin has joined #openstack-meeting00:21
*** jmasud has quit IRC00:25
*** yasufum has joined #openstack-meeting00:30
*** b1airo has quit IRC01:07
*** priteau has quit IRC02:03
*** jmasud has joined #openstack-meeting02:04
*** rh-jelabarre has quit IRC02:21
*** armstrong has quit IRC02:24
*** evrardjp has quit IRC02:33
*** evrardjp has joined #openstack-meeting02:33
*** redrobot has quit IRC02:45
*** dsariel has joined #openstack-meeting03:01
*** jmasud has quit IRC03:19
*** jmasud has joined #openstack-meeting03:26
*** yasufum has quit IRC03:42
*** yasufum has joined #openstack-meeting03:47
*** redrobot has joined #openstack-meeting03:50
*** rbudden has quit IRC04:27
*** markmcclain has quit IRC04:27
*** markmcclain has joined #openstack-meeting04:28
*** vishalmanchanda has joined #openstack-meeting04:40
*** ralonsoh has joined #openstack-meeting05:27
*** udesale has joined #openstack-meeting05:41
*** slaweq has joined #openstack-meeting06:00
*** lpetrut has joined #openstack-meeting06:16
*** gyee has quit IRC06:35
*** dklyle has quit IRC06:59
*** rcernin has quit IRC07:25
*** rpittau|afk is now known as rpittau07:36
*** manpreet has joined #openstack-meeting07:39
*** e0ne has joined #openstack-meeting07:47
*** e0ne has quit IRC07:47
*** e0ne has joined #openstack-meeting07:58
*** e0ne has quit IRC07:58
*** e0ne has joined #openstack-meeting08:03
*** e0ne has quit IRC08:03
*** ociuhandu has joined #openstack-meeting08:15
*** rcernin has joined #openstack-meeting08:21
*** ociuhandu has quit IRC08:25
*** ociuhandu has joined #openstack-meeting08:26
*** ociuhandu has quit IRC08:32
*** rcernin has quit IRC08:32
*** e0ne has joined #openstack-meeting08:36
*** e0ne has quit IRC08:36
*** jmasud has quit IRC08:40
*** ociuhandu has joined #openstack-meeting08:41
*** bnemec has quit IRC08:48
*** bnemec has joined #openstack-meeting08:49
*** manubk has joined #openstack-meeting09:13
*** jmasud has joined #openstack-meeting09:22
*** jmasud has quit IRC09:22
*** priteau has joined #openstack-meeting09:22
*** Carlotronics has quit IRC09:24
*** Carlotronics has joined #openstack-meeting09:25
*** rcernin has joined #openstack-meeting09:39
*** tosky has joined #openstack-meeting09:45
*** manubk has quit IRC09:47
*** vishalmanchanda has quit IRC10:00
*** vishalmanchanda has joined #openstack-meeting10:11
*** whoami-rajat_ has joined #openstack-meeting10:22
*** dsariel has quit IRC10:23
*** ociuhandu has quit IRC10:39
*** ociuhandu has joined #openstack-meeting10:57
*** rubasov_ has quit IRC11:05
*** ociuhandu has quit IRC11:07
*** rubasov has joined #openstack-meeting11:08
*** ociuhandu has joined #openstack-meeting11:15
*** ociuhandu has quit IRC11:19
*** ociuhandu has joined #openstack-meeting11:20
*** yasufum has quit IRC11:22
*** ociuhandu has quit IRC11:25
*** dsariel has joined #openstack-meeting11:36
*** ociuhandu has joined #openstack-meeting11:36
*** ociuhandu has quit IRC11:41
*** udesale_ has joined #openstack-meeting11:43
*** udesale has quit IRC11:44
*** rh-jelabarre has joined #openstack-meeting11:49
*** udesale has joined #openstack-meeting11:52
*** ociuhandu has joined #openstack-meeting11:52
*** udesale_ has quit IRC11:53
*** udesale_ has joined #openstack-meeting11:55
*** udesale_ has quit IRC11:56
*** udesale_ has joined #openstack-meeting11:57
*** udesale has quit IRC11:57
*** yasufum has joined #openstack-meeting11:58
*** ociuhandu has quit IRC12:11
*** eharney has quit IRC12:23
*** ociuhandu has joined #openstack-meeting12:24
*** eharney has joined #openstack-meeting12:25
*** ociuhandu has quit IRC12:29
*** whoami-rajat_ is now known as whoami-rajat12:32
*** rbudden has joined #openstack-meeting12:36
*** ociuhandu has joined #openstack-meeting12:40
*** dsariel has quit IRC12:40
*** dsariel has joined #openstack-meeting12:41
*** lpetrut has quit IRC12:44
*** johanssone has quit IRC12:51
*** johanssone has joined #openstack-meeting12:54
*** macz_ has joined #openstack-meeting13:05
*** e0ne has joined #openstack-meeting13:11
*** eharney has quit IRC13:18
*** eharney has joined #openstack-meeting13:18
*** johanssone has quit IRC13:21
*** yasufum has quit IRC13:24
*** e0ne has quit IRC13:24
*** e0ne has joined #openstack-meeting13:24
*** johanssone has joined #openstack-meeting13:28
*** e0ne has quit IRC13:40
*** e0ne has joined #openstack-meeting13:40
*** yasufum has joined #openstack-meeting13:48
*** ociuhandu has quit IRC13:52
*** ociuhandu has joined #openstack-meeting13:53
*** ociuhandu has quit IRC13:58
*** ociuhandu has joined #openstack-meeting14:14
*** armstrong has joined #openstack-meeting14:15
*** ociuhandu has joined #openstack-meeting14:16
*** e0ne has quit IRC14:16
*** e0ne_ has joined #openstack-meeting14:16
*** shanuintouch has joined #openstack-meeting14:20
*** ociuhandu has quit IRC14:20
*** ociuhandu has joined #openstack-meeting14:21
*** e0ne has joined #openstack-meeting14:32
*** e0ne_ has quit IRC14:32
*** e0ne_ has joined #openstack-meeting14:54
*** e0ne has quit IRC14:55
*** rcernin has quit IRC15:01
*** ociuhandu has quit IRC15:07
*** ociuhandu has joined #openstack-meeting15:08
*** jamesmcarthur has joined #openstack-meeting15:10
*** rcernin has joined #openstack-meeting15:13
*** ociuhandu has quit IRC15:13
*** ociuhandu has joined #openstack-meeting15:15
*** rcernin has quit IRC15:17
*** e0ne_ has quit IRC15:20
*** e0ne has joined #openstack-meeting15:20
*** mlavalle has joined #openstack-meeting15:22
*** dklyle has joined #openstack-meeting15:38
*** ociuhandu has quit IRC15:42
*** ociuhandu has joined #openstack-meeting15:43
*** e0ne has quit IRC15:46
*** e0ne_ has joined #openstack-meeting15:46
*** ociuhandu has quit IRC15:48
*** jmasud has joined #openstack-meeting15:53
*** rcernin has joined #openstack-meeting15:59
*** QG has joined #openstack-meeting16:01
*** e0ne_ has quit IRC16:02
*** e0ne has joined #openstack-meeting16:02
*** udesale_ has quit IRC16:03
*** rcernin has quit IRC16:05
*** dklyle has quit IRC16:12
*** e0ne_ has joined #openstack-meeting16:18
*** e0ne has quit IRC16:19
*** dklyle has joined #openstack-meeting16:20
*** QG has quit IRC16:31
*** e0ne_ has quit IRC16:34
*** e0ne has joined #openstack-meeting16:34
*** rcernin has joined #openstack-meeting16:38
*** rpittau is now known as rpittau|afk16:39
*** rcernin has quit IRC16:43
*** timburke has quit IRC16:47
*** shanuintouch has quit IRC16:49
*** e0ne has quit IRC16:49
*** rcernin has joined #openstack-meeting16:57
*** rcernin has quit IRC17:02
*** e0ne has joined #openstack-meeting17:02
*** e0ne has quit IRC17:06
*** e0ne_ has joined #openstack-meeting17:06
*** timburke has joined #openstack-meeting17:11
*** bnemec has quit IRC17:16
*** bnemec has joined #openstack-meeting17:16
*** e0ne_ has quit IRC17:22
*** e0ne has joined #openstack-meeting17:22
*** lbragstad has quit IRC17:37
*** lbragstad has joined #openstack-meeting17:38
*** gyee has joined #openstack-meeting17:39
*** e0ne_ has joined #openstack-meeting17:43
*** e0ne has quit IRC17:44
*** ralonsoh has quit IRC17:53
*** e0ne has joined #openstack-meeting17:58
*** e0ne_ has quit IRC18:01
*** e0ne_ has joined #openstack-meeting18:04
*** e0ne has quit IRC18:06
*** e0ne_ has quit IRC18:15
*** e0ne has joined #openstack-meeting18:15
*** yasufum has quit IRC18:31
*** e0ne has quit IRC18:31
*** e0ne has joined #openstack-meeting18:31
*** e0ne_ has joined #openstack-meeting18:47
*** e0ne has quit IRC18:48
*** e0ne_ has quit IRC19:01
*** e0ne has joined #openstack-meeting19:02
*** e0ne has quit IRC19:04
*** e0ne has joined #openstack-meeting19:04
*** manpreet has quit IRC19:22
*** jamesmcarthur has quit IRC19:31
*** jamesmcarthur has joined #openstack-meeting19:32
*** e0ne has quit IRC19:33
*** jamesmcarthur_ has joined #openstack-meeting19:35
*** jamesmcarthur has quit IRC19:35
*** vishalmanchanda has quit IRC19:40
*** lbragstad_ has joined #openstack-meeting20:01
*** yasufum has joined #openstack-meeting20:03
*** lbragstad has quit IRC20:03
*** jmasud has quit IRC20:07
*** jmasud has joined #openstack-meeting20:15
*** _erlon_ has joined #openstack-meeting20:42
*** armstrong has quit IRC20:44
*** whoami-rajat has quit IRC20:52
*** njohnston has joined #openstack-meeting20:53
*** acoles has joined #openstack-meeting20:59
claygtimburke: i'm so ready!!!21:00
timburke#startmeeting swift21:00
openstackMeeting started Wed Apr 28 21:00:47 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
acoleso/21:01
mattoliverauo/21:01
claygo/21:02
*** zaitcev has joined #openstack-meeting21:03
timburkeas usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift21:04
timburkefirst up21:04
timburke#topic PTG21:04
*** openstack changes topic to "PTG (Meeting topic: swift)"21:04
timburkei wanted to thank everyone who came out to the PTG last week -- i feel like we had some good, productive discussions21:04
timburkeand that needing to explain some of our ideas to devs we don't necessarily get to talk to super-regularly helped firm up a lot of them21:05
mattoliverau+121:06
timburkei don't know that i've got a lot more to say, other than thanks again!21:06
timburke#topic rolling upgrade job failures21:06
*** openstack changes topic to "rolling upgrade job failures (Meeting topic: swift)"21:06
acolesI'm grateful to those who were up in their night time - thank you!21:06
timburkei don't know how much other people have noticed, but i've seen a fair few failures lately21:07
claygyes, I HAVE seen some rolling upgrade job failures - "grenade" too!21:08
acolesI noticed a few in last 24 hours21:08
acolesbut func-cors is rock solid :D21:08
timburkei suspect they've been flaky for a while (seem to be related to listing consistency issues), but we've had them disabled/non-voting a decent bit lately and hadn't noticed21:08
timburkei also think (but haven't yet verified) there's a chance they'll improve the next time we cut a tag, since i added the ability to retry failed func tests21:09
timburkejust wanted to keep people updated; nothing really you all need to do21:10
acolesat least one rolling upgrade fail was a timeout https://zuul.opendev.org/t/openstack/build/79a7ae5a3cc649d0a556a29e76dc080021:10
timburkeon to updates!21:11
timburkewe've got a lot of things in-flight these days; i think that was another nice benefit of the PTG :-)21:11
timburke#topic sharding21:11
*** openstack changes topic to "sharding (Meeting topic: swift)"21:12
timburkeso, current patches:21:12
timburkehttps://review.opendev.org/c/openstack/swift/+/784617 - Add sharding to swift-recon (already approved)21:12
timburkehttps://review.opendev.org/c/openstack/swift/+/785628 - swift-manage-shard-ranges: fix exit codes21:12
timburkehttps://review.opendev.org/c/openstack/swift/+/774002 - Fix shrinking making acceptors prematurely active21:12
timburkehttps://review.opendev.org/c/openstack/swift/+/777585 - stall cleaving at shard range gaps (already approved, but waiting on pre-req ^^^)21:12
timburkehttps://review.opendev.org/c/openstack/swift/+/782832 - Consider tombstone count before shrinking a shard21:12
timburkehttps://review.opendev.org/c/openstack/swift/+/787637 - Don't consider responses generated from cache as "already visited"21:12
*** yasufum has quit IRC21:13
timburkedo we have any upgrade concerns about the exit code changes? 'cause if not, i'm happy to +A :-)21:13
acolesre the exit codes (patch 785628) IIRC back 3 years, there was maybe a thought to differentiate warnings from errors using codes 1 and 2 (or vice-versa) but its slipped since then21:15
claygexit code changes on which patch?21:15
timburkesecond one21:15
acolesand  I discovered recently that argparse exists with 2 on invalid args21:15
acolesso my thinking with the patch is to try to line up all invalid cli to return 2 and any other non-success to be 1.21:17
timburkeseems reasonable, approving21:18
timburkei feel like we ought to prioritize the "prematurely active" patch since it's blocking the "stall cleaving" patch which is otherwise good to go21:18
acolesthanks21:18
timburkehow are we feeling about the tombstone counting? just waiting on review?21:19
acolesI think I attracted some interest in tombstones from clayg21:19
mattoliverauyup looks good (exit code). I'll look at prematurely active todat to unstick ut.21:19
mattoliverau*it21:19
timburkethanks!21:19
acolesthank mattoliverau21:19
acolesthanks*21:20
*** jmasud has quit IRC21:20
timburkei wouldn't mind talking through the "already visited" patch a bit, but maybe that'd be better next week21:21
timburkeany other sharding topics i'm forgetting?21:22
mattoliverauMaybe put the rest on priority review (if they aren't already) so I dont forget about them.. it's early here and my brain isn't working yet.21:22
acolesI need to be convinced on not including cached responses in the loop-detection21:22
mattoliverauI haven't really looped back around to actuve_age post PTG, so not much to say there yet. But want to get back too it soon.21:23
timburkemy main thought is that *we haven't gone to disk yet*21:23
mattoliverau*active_age /me can't type this morn21:23
timburkeooh -- yeah -- it'll be interesting to see if my idea pans out :-)21:24
acolestimburke: it may be that we need a way to provoke a backend request without mandating that is for objects only21:24
*** slaweq has quit IRC21:25
acolesbut also retain the break if that just results in the same loop, somehow21:25
timburke#topic relinker21:26
*** openstack changes topic to "relinker (Meeting topic: swift)"21:26
timburkeanother wall of patches:21:26
timburkehttps://review.opendev.org/c/openstack/swift/+/783731 - Rehash the parts actually touched when relinking21:26
timburkehttps://review.opendev.org/c/openstack/swift/+/788089 - Only mark partitions "done" if there were no (new) errors21:27
timburkehttps://review.opendev.org/c/openstack/swift/+/779655 - Add /recon/relinker endpoint and drop progress stats21:27
timburkehttps://review.opendev.org/c/openstack/swift/+/788413 - Log and recon on SIGTERM signal21:27
timburkehttps://review.opendev.org/c/openstack/swift/+/788177 - add aggregate data to recon drop21:27
*** jmasud has joined #openstack-meeting21:27
acolesre 788089 - when did we ignore errors?21:28
timburkeso the first two seem pretty useful for correctness and clear ops-signalling21:28
timburkethey weren't *ignored* exactly... i mean, we logged them and everything, and we'll exit non-zero21:28
timburkeit's just that we mark the partition as having been relinked21:29
acolesbut we set state to True?21:29
timburkeyup21:29
acoleseek21:29
timburkeso a subsequent relink either skips the partition that had errors, or ops need to manually go clear the state file21:30
acolesyeah, we should fix that21:30
mattoliverauyup21:30
mattoliverauthe last 3 are based around the new recon patch, split and one a worte up to something to trap signals and dump the error_code as appropriate to recon, cli return and log it.21:30
timburkethe "rehash parts touched" strikes my interest since we've had instances where we had hashes in partitions from part-power 17 when relinking into 19 (for instance)21:31
mattoliverau788414 took longer then expected when trying to write tests because of diferences in os.exit and os._exit.. which made the signals kill my test suit run.. fun times :P21:32
acoleshttps://review.opendev.org/c/openstack/swift/+/779655 (first in relinker recon chain) has been coming along - should we focus on getting that merged? I think there's still a few things to resolve like the option name, but hopefully it is close21:32
clayg@mattoliverau fun time!!! 🤣21:33
acolesi.e. should recon_interval actually be stats_interval like the replicator has?21:33
timburkeyeah, i definitely like the recon idea -- haven't had a chance to look at it since it was split up, unfortunately21:33
timburkei'll try to take another pass at it this afternoon21:33
mattoliverauyeah, if there is already a stats_interval elsewhere I'm all for using it. keeps things consistent.21:35
timburkeso mattoliverau -- do you have a preference on which fork people look at next after the first recon change?21:36
claygacoles: did you have other concerns that got dropped?  the name change is a good idea, and easy enough to fix 💪21:36
acolesI like that you've broken things out into some follow on patches21:37
mattoliverautimburke: no really. the trap just makes sure we write done and let people know things are done if the process is killed by something like (ahem ansible timeouts).21:38
mattoliverauthe aggregator we might need to discuss some more.21:38
timburkesounds like maybe i should look at signals next, then ;-)21:39
timburkei know acoles had some comments on the base patch -- did those ever get addressed?21:39
acolesclayg: mattoliverau I think the two non-nits were recon_interval to stats_interval and duplicated start_time, although the latter isn't a blocker. But we must straighten out the option name.21:40
mattoliverauI might go poke an op to take a look at the existing recon and the aggregator followup to see what they'd like to see, or rather if /what they can use.21:40
mattoliverauacoles: ahh yeah the start time, somehow I missed that again in yesterdays rework.21:40
*** rcernin has joined #openstack-meeting21:40
timburke👍21:41
timburke#topic stale EC frags21:42
*** openstack changes topic to "stale EC frags (Meeting topic: swift)"21:42
mattoliverauI haven't looked at the patch this morning so don't know what's there. but will push a new patchset today. maybe I'll wait until timburke has a look (if he get's to it this arvo his time). no pressure tho21:42
timburkewe've got a couple patches currently working their way through the gate (thanks clayg and acoles!)21:42
timburkehttps://review.opendev.org/c/openstack/swift/+/787279 - reconstructor: log more details when rebuild fails (already approved)21:42
timburkehttps://review.opendev.org/c/openstack/swift/+/788540 - reconstructor: extract closure for handle_response (already approved)21:42
timburkeso how are we feeling on21:42
timburkehttps://review.opendev.org/c/openstack/swift/+/786084 - Quarantine stale EC fragments21:42
mattoliverauI started a review on it last night.. but ran out of time. Planning on continueing it today. So don't have too much to say atm myself.21:43
*** bnemec has quit IRC21:44
*** rh-jelabarre has quit IRC21:45
timburkeacoles, any known rough edges to watch out for?21:45
timburkeor just waiting on review?21:45
*** bnemec has joined #openstack-meeting21:46
timburkeprobably will need a rebase once the other two land...21:46
timburkewe'll see what it looks like next week21:47
timburke#topic dark data watcher21:47
*** openstack changes topic to "dark data watcher (Meeting topic: swift)"21:47
timburkeso we've got a couple patches for some known deficiencies21:47
timburkehttps://review.opendev.org/c/openstack/swift/+/788398 - Make dark data watcher ignore the newly updated objects21:48
timburkehttps://review.opendev.org/c/openstack/swift/+/787656 - Work with sharded containers21:48
timburkei don't think either is quite ready yet (zaitcev's patch has a WIP in the commit message, and mine probably should, too)21:49
timburkebut i wanted to keep them on people's radars21:49
zaitcevMine needs tests.21:49
timburkei think those are the main major efforts in-flight right now21:50
acolestimburke: zaitcev thanks for those patches21:50
timburke#topic open discussion21:50
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:50
timburkeanyhting else we ought to bring up this week?21:50
mattoliveraunothing comes to mind21:52
timburkeso i had a thought that feels like a good idea, but idk if it presents some backwards compat issues21:53
timburkehttps://review.opendev.org/c/openstack/swift/+/787905 - proxy: Downgrade some client problems to info21:53
timburkebasically, stop logging client disconnects and timeouts at warning -- they're client behaviors, so it is (or can be) way too noisy at that level21:54
timburkewell, something to think about, anyway21:56
timburkethat's all i've got21:56
timburkethank you all for coming, and thank you for working on swift!21:57
timburkeand thanks for coming to the PTG :-)21:57
timburke#endmeeting21:57
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:57
openstackMeeting ended Wed Apr 28 21:57:20 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-04-28-21.00.html21:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-04-28-21.00.txt21:57
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-04-28-21.00.log.html21:57
mattoliverauthanks timburke, will think on the patch. I think it makes sense. esp when its client driven, swift has no control over it.21:58
*** zaitcev has left #openstack-meeting22:01
*** rcernin has quit IRC22:06
*** rcernin has joined #openstack-meeting22:07
*** macz_ has quit IRC22:42
*** tosky has quit IRC22:48
*** bcafarel has quit IRC23:11
*** bcafarel has joined #openstack-meeting23:11
*** mlavalle has quit IRC23:59
« Tuesday, 2021-04-27 Index Thursday, 2021-04-29 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!