| *** tosky has quit IRC | 00:00 | |
| *** slaweq has quit IRC | 00:05 | |
| *** slaweq has joined #openstack-meeting | 00:07 | |
| *** martial has quit IRC | 00:18 | |
| *** yasufum has quit IRC | 00:29 | |
| *** yasufum has joined #openstack-meeting | 00:29 | |
| *** macz_ has quit IRC | 00:39 | |
| *** benj_ has quit IRC | 00:54 | |
| *** zigo has quit IRC | 00:54 | |
| *** benj_ has joined #openstack-meeting | 00:54 | |
| *** mlavalle has quit IRC | 01:04 | |
| *** manpreet has quit IRC | 01:09 | |
| *** amotoki has quit IRC | 02:01 | |
| *** amotoki has joined #openstack-meeting | 02:02 | |
| *** lifeless has quit IRC | 02:25 | |
| *** lifeless has joined #openstack-meeting | 02:27 | |
| *** macz_ has joined #openstack-meeting | 02:31 | |
| *** rfolco has joined #openstack-meeting | 02:34 | |
| *** macz_ has quit IRC | 02:36 | |
| *** jamesdenton has quit IRC | 03:17 | |
| *** jamesdenton has joined #openstack-meeting | 03:18 | |
| *** psachin has joined #openstack-meeting | 03:20 | |
| *** rfolco has quit IRC | 03:24 | |
| *** armax has quit IRC | 03:38 | |
| *** manpreet has joined #openstack-meeting | 03:41 | |
| *** ricolin has quit IRC | 04:21 | |
| *** vishalmanchanda has joined #openstack-meeting | 04:55 | |
| *** arne_wiebalck has quit IRC | 05:27 | |
| *** yonglihe has quit IRC | 05:27 | |
| *** yonglihe has joined #openstack-meeting | 05:28 | |
| *** zeestrat has quit IRC | 05:28 | |
| *** arne_wiebalck has joined #openstack-meeting | 05:29 | |
| *** zeestrat has joined #openstack-meeting | 05:30 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:33 | |
| *** zbr has quit IRC | 06:06 | |
| *** manpreet has quit IRC | 06:10 | |
| *** yasufum has quit IRC | 06:12 | |
| *** gyee has quit IRC | 06:28 | |
| *** yasufum has joined #openstack-meeting | 06:38 | |
| *** mvalsecc has joined #openstack-meeting | 06:50 | |
| *** lpetrut has joined #openstack-meeting | 07:12 | |
| *** psachin has quit IRC | 07:17 | |
| *** ralonsoh has joined #openstack-meeting | 07:19 | |
| *** dklyle has quit IRC | 07:34 | |
| *** lifeless has quit IRC | 07:54 | |
| *** lifeless has joined #openstack-meeting | 07:56 | |
| *** tosky has joined #openstack-meeting | 08:02 | |
| *** rpittau|afk is now known as rpittau | 08:14 | |
| *** mvalsecc has quit IRC | 08:33 | |
| *** ociuhandu has joined #openstack-meeting | 08:42 | |
| *** geguileo has joined #openstack-meeting | 08:51 | |
| *** ociuhandu has quit IRC | 08:52 | |
| *** rcernin has quit IRC | 08:56 | |
| *** rcernin has joined #openstack-meeting | 08:56 | |
| *** ociuhandu has joined #openstack-meeting | 08:59 | |
| *** rfolco has joined #openstack-meeting | 09:00 | |
| *** rcernin has quit IRC | 09:23 | |
| *** rcernin has joined #openstack-meeting | 09:32 | |
| *** rcernin has quit IRC | 09:48 | |
| *** rcernin has joined #openstack-meeting | 09:49 | |
| *** macz_ has joined #openstack-meeting | 10:02 | |
| *** macz_ has quit IRC | 10:07 | |
| *** ssbarnea has joined #openstack-meeting | 10:18 | |
| *** yasufum has quit IRC | 10:18 | |
| *** zbr has joined #openstack-meeting | 10:24 | |
| *** rcernin has quit IRC | 10:31 | |
| *** ssbarnea has quit IRC | 10:36 | |
| *** ociuhandu has quit IRC | 10:47 | |
| *** ociuhandu has joined #openstack-meeting | 10:51 | |
| *** rcernin has joined #openstack-meeting | 10:53 | |
| *** zbr has quit IRC | 11:25 | |
| *** zbr has joined #openstack-meeting | 11:28 | |
| *** ociuhandu has quit IRC | 11:35 | |
| *** e0ne has joined #openstack-meeting | 11:37 | |
| *** zbr has quit IRC | 11:46 | |
| *** zbr has joined #openstack-meeting | 11:49 | |
| *** ociuhandu has joined #openstack-meeting | 11:49 | |
| *** ociuhandu_ has joined #openstack-meeting | 11:58 | |
| *** ociuhandu has quit IRC | 12:02 | |
| *** rcernin has quit IRC | 12:04 | |
| *** ociuhandu_ has quit IRC | 12:51 | |
| *** ociuhandu has joined #openstack-meeting | 12:51 | |
| *** ociuhandu has quit IRC | 13:06 | |
| *** ociuhandu has joined #openstack-meeting | 13:08 | |
| *** ociuhandu has quit IRC | 13:16 | |
| *** ociuhandu has joined #openstack-meeting | 13:24 | |
| *** zbr has quit IRC | 13:24 | |
| *** zbr has joined #openstack-meeting | 13:27 | |
| *** zbr has quit IRC | 13:45 | |
| *** zbr has joined #openstack-meeting | 13:47 | |
| *** ricolin has joined #openstack-meeting | 13:58 | |
| *** rfolco is now known as rfolco|brb | 13:59 | |
| *** ociuhandu has joined #openstack-meeting | 14:00 | |
| *** zbr has quit IRC | 14:03 | |
| *** zbr has joined #openstack-meeting | 14:04 | |
| *** ociuhandu has quit IRC | 14:07 | |
| *** ociuhandu has joined #openstack-meeting | 14:11 | |
| *** TrevorV has joined #openstack-meeting | 14:17 | |
| *** johanssone has quit IRC | 14:23 | |
| *** bbowen has quit IRC | 14:24 | |
| *** bbowen has joined #openstack-meeting | 14:24 | |
| *** johanssone has joined #openstack-meeting | 14:26 | |
| *** ociuhandu has quit IRC | 14:36 | |
| *** ociuhandu has joined #openstack-meeting | 14:43 | |
| *** ociuhandu has quit IRC | 15:11 | |
| *** ociuhandu has joined #openstack-meeting | 15:15 | |
| *** TrevorV has quit IRC | 15:18 | |
| *** ociuhandu has quit IRC | 15:20 | |
| *** ociuhandu has joined #openstack-meeting | 15:25 | |
| *** ralonsoh has quit IRC | 15:27 | |
| *** ralonsoh has joined #openstack-meeting | 15:27 | |
| *** zbr has quit IRC | 15:29 | |
| *** ociuhandu has quit IRC | 15:30 | |
| *** zbr has joined #openstack-meeting | 15:31 | |
| *** ralonsoh has quit IRC | 15:34 | |
| *** ociuhandu has joined #openstack-meeting | 15:35 | |
| *** dklyle has joined #openstack-meeting | 15:37 | |
| *** ralonsoh has joined #openstack-meeting | 15:38 | |
| *** ociuhandu has quit IRC | 15:39 | |
| *** ociuhandu has joined #openstack-meeting | 15:43 | |
| *** ralonsoh_ has joined #openstack-meeting | 15:45 | |
| *** ralonsoh has quit IRC | 15:46 | |
| *** ociuhandu has quit IRC | 15:47 | |
| *** zbr has quit IRC | 15:48 | |
| *** dsariel has joined #openstack-meeting | 15:49 | |
| *** zbr has joined #openstack-meeting | 15:51 | |
| *** macz_ has joined #openstack-meeting | 15:52 | |
| *** armax has joined #openstack-meeting | 16:00 | |
| *** rajinir has joined #openstack-meeting | 16:04 | |
| *** mlavalle has joined #openstack-meeting | 16:13 | |
| *** ociuhandu has joined #openstack-meeting | 16:29 | |
| *** rfolco|brb is now known as rfolco | 16:46 | |
| *** lpetrut has quit IRC | 16:46 | |
| *** rpittau is now known as rpittau|afk | 17:12 | |
| *** e0ne has quit IRC | 17:12 | |
| *** TrevorV has joined #openstack-meeting | 17:19 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:24 | |
| *** ociuhandu has quit IRC | 17:27 | |
| *** zbr has quit IRC | 17:27 | |
| *** ociuhandu_ has quit IRC | 17:28 | |
| *** zbr has joined #openstack-meeting | 17:30 | |
| *** ociuhandu has joined #openstack-meeting | 17:34 | |
| *** e0ne has joined #openstack-meeting | 17:34 | |
| *** ociuhandu has quit IRC | 17:39 | |
| *** zbr has quit IRC | 17:47 | |
| *** zbr has joined #openstack-meeting | 17:50 | |
| *** zbr has quit IRC | 17:56 | |
| *** gyee has joined #openstack-meeting | 17:57 | |
| *** zbr has joined #openstack-meeting | 17:58 | |
| *** yasufum has joined #openstack-meeting | 18:01 | |
| *** baojg has quit IRC | 18:37 | |
| *** e0ne has quit IRC | 18:51 | |
| *** armstrong has joined #openstack-meeting | 18:55 | |
| *** yasufum has quit IRC | 19:05 | |
| *** ralonsoh_ has quit IRC | 20:28 | |
| *** ociuhandu has joined #openstack-meeting | 20:42 | |
| *** ociuhandu has quit IRC | 20:46 | |
| *** ociuhandu has joined #openstack-meeting | 20:48 | |
| *** ociuhandu has quit IRC | 20:52 | |
| *** timburke has quit IRC | 20:53 | |
| *** vishalmanchanda has quit IRC | 20:54 | |
| *** zaitcev has joined #openstack-meeting | 20:56 | |
| *** ociuhandu has joined #openstack-meeting | 20:59 | |
| *** acoles has joined #openstack-meeting | 21:00 | |
| *** tdasilva has joined #openstack-meeting | 21:00 | |
| acoles | who's here for the swift meeting? | 21:03 |
|---|---|---|
| mattoliverau | o/ | 21:03 |
| kota_ | o/ | 21:03 |
| rledisez | o/ | 21:03 |
| mattoliverau | #startmeeting swift | 21:03 |
| openstack | Meeting started Wed Dec 9 21:03:42 2020 UTC and is due to finish in 60 minutes. The chair is mattoliverau. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:03 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:03 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:03 | |
| acoles | #startmeeting | 21:03 |
| openstack | The meeting name has been set to 'swift' | 21:03 |
| openstack | acoles: Error: Can't start another meeting, one is in progress. Use #endmeeting first. | 21:03 |
| mattoliverau | I beat ya | 21:03 |
| rledisez | :D | 21:03 |
| acoles | oh thanks mattoliverau for starting the meeting | 21:03 |
| *** ociuhandu has quit IRC | 21:04 | |
| mattoliverau | I'll start chairing i guess until tim comes online | 21:04 |
| acoles | does that mean you will chair as well hahah! | 21:04 |
| acoles | so, apologies form timburke, he has unexpected childcare duties | 21:04 |
| mattoliverau | #topic Audit watchers | 21:04 |
| *** openstack changes topic to "Audit watchers (Meeting topic: swift)" | 21:04 | |
| acoles | from* | 21:04 |
| acoles | thanks mattoliverau | 21:04 |
| mattoliverau | acoles: cool thanks for letting us know. | 21:05 |
| acoles | BTW the agenda is here https://wiki.openstack.org/wiki/Meetings/Swift | 21:05 |
| mattoliverau | oh thanks | 21:05 |
| *** armstrong has quit IRC | 21:05 | |
| zaitcev | But it's old | 21:05 |
| mattoliverau | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:05 |
| mattoliverau | Any updateing from audit watchers? | 21:05 |
| mattoliverau | I know I reviewed it again last night | 21:06 |
| mattoliverau | and it's looking really great | 21:06 |
| acoles | I know the final agenda topic is intended for today at least | 21:06 |
| mattoliverau | I think we need to get some documentation in place, but that is a follow up patch I feel. | 21:06 |
| zaitcev | I am going to write it. | 21:07 |
| zaitcev | By "it" I mean the doc for watchers. | 21:08 |
| mattoliverau | Cool, thanks zaitcev, I'll review it and land it when you're done. | 21:08 |
| *** rfolco has quit IRC | 21:08 | |
| mattoliverau | The PR in question is: https://review.opendev.org/c/openstack/swift/+/706653 | 21:09 |
| mattoliverau | I was temped to put a +A on it, but knew Tim said we planned to review it. | 21:09 |
| mattoliverau | *he | 21:09 |
| mattoliverau | If no more questions on audit watchers shall we move on? | 21:10 |
| zaitcev | Move on. | 21:10 |
| acoles | great work guys, thanks | 21:10 |
| zaitcev | It's dsariel's debut, BTW | 21:10 |
| *** ociuhandu has joined #openstack-meeting | 21:10 | |
| mattoliverau | \o/ | 21:10 |
| zaitcev | So I wasn't touching it on purpose, to let him get the lumps :-) | 21:11 |
| dsariel | with zaitcev's great help | 21:11 |
| mattoliverau | #topic s3api, +segments container, and ACLs | 21:11 |
| *** openstack changes topic to "s3api, +segments container, and ACLs (Meeting topic: swift)" | 21:11 | |
| mattoliverau | I know this is an old agenda, so do we have any update on this? | 21:11 |
| *** slaweq has quit IRC | 21:12 | |
| mattoliverau | #link https://review.opendev.org/763106 | 21:12 |
| zaitcev | I looked at it and it seemed fine | 21:12 |
| zaitcev | But I didn't +2 | 21:12 |
| zaitcev | Oh | 21:12 |
| zaitcev | I know. Clay sniped me on it. | 21:13 |
| mattoliverau | looks like it's +Aed | 21:13 |
| mattoliverau | tho not merged | 21:13 |
| mattoliverau | So I guess it just needs handholding through the gate. | 21:13 |
| zaitcev | I think we can move on from that particular thing onto s3api in general if anyone knows what's up with it. | 21:13 |
| *** slaweq has joined #openstack-meeting | 21:13 | |
| acoles | move on - IIRC last week it was just to nudge it for a +A | 21:14 |
| acoles | since then it has been in recheck-land | 21:14 |
| mattoliverau | kk | 21:14 |
| zaitcev | Yea. I knew it had no chance, so didn't recheck until Tim's "retry" patch. | 21:14 |
| mattoliverau | we can come back to s3api if anyone has anything at the end | 21:14 |
| mattoliverau | #topic what still has to be done in order to enable automatic sharding | 21:15 |
| *** openstack changes topic to "what still has to be done in order to enable automatic sharding (Meeting topic: swift)" | 21:15 | |
| acoles | that's a great question | 21:15 |
| mattoliverau | it is :) | 21:15 |
| zaitcev | before that, mattoliverau, are you working on that off the tip we have right now? | 21:15 |
| mattoliverau | thanks dsariel for adding it :) | 21:15 |
| dsariel | I will be happy to help with this. Anything I can do? | 21:16 |
| zaitcev | During PTG someone (Tim or Clay) mentioned that nVidia has some patches in production that are necessary for the current sharding. | 21:16 |
| zaitcev | So I was wondering where the development is occurring. | 21:16 |
| acoles | there's a few patches on gerrit around sharding, some of which we have shipped | 21:16 |
| zaitcev | Oh | 21:17 |
| zaitcev | I thought they weren't in gerrit. | 21:17 |
| mattoliverau | So I obivously can't speak for nvidia. | 21:17 |
| zaitcev | OK, I can find them. Well, David can find them heh. | 21:17 |
| mattoliverau | but I belive they're not using auto sharding. | 21:17 |
| zaitcev | yes yes, just the exisitng sharding | 21:17 |
| mattoliverau | they have some smarts in their controller that identify things that need shardsing and the sharding management tool is used. | 21:17 |
| acoles | but first, at a high level, my personal view is that we need to (a) put in place all we think we need to recover from split-brain autosharding and (b) convince ourselves that we have done the best we can to avoid split-brain auto-sharding | 21:18 |
| mattoliverau | Auto sharding, where I want to get too I believe are some upstream WIP patches I have. | 21:18 |
| mattoliverau | what acoles said ^ | 21:18 |
| acoles | we have a proprietary approach to avoiding split-brain sharding, and we do not enable autosharding | 21:18 |
| acoles | we use swift-manage-shard-ranges | 21:19 |
| zaitcev | Got it. | 21:19 |
| mattoliverau | Turns out the main problem with the current auto-sharding approach is there are ways theis split brain can occur. | 21:19 |
| acoles | oh, and one final piece, we need to have autoshrinking sorted too | 21:20 |
| mattoliverau | +1 | 21:20 |
| mattoliverau | I have one POC/WIP patch that improves the leader election, but after playing with it, it minimalises these edge cases, but doesn't remove them. | 21:20 |
| mattoliverau | So moved on to what acoles mentioned. If we have a way to recover from split brains and gaps then that needs to come first. | 21:21 |
| zaitcev | Guys | 21:22 |
| zaitcev | Our quorum is 1/2 or greater, right? | 21:22 |
| mattoliverau | them we might fine we're happy to have the simple "sam" leader election approach we have now.. or decide to improve leader election. | 21:23 |
| mattoliverau | we have 2 quorums | 21:23 |
| zaitcev | Can it be used productively, so that the minority always agrees with majority (which presumably has a leader selected)? | 21:23 |
| dsariel | can I ask a noob question: what is split-brain sharding? | 21:23 |
| mattoliverau | dsariel: when more then 1 thinks they are the leader and make a diffferent set of shard ranges | 21:24 |
| zaitcev | dsariel: it's a network partition, so now you have 20 nodes doing one thing and 15 nodes doing other thing. | 21:24 |
| dsariel | got it. thanks | 21:24 |
| mattoliverau | we have a ceil[replica/2] and a majority quorum ( replica / 2 + 1 ) | 21:24 |
| zaitcev | replica // 2 or | 21:25 |
| mattoliverau | so yeah we'd use a majority quorum for making leader election decisions if we went and asked. | 21:25 |
| zaitcev | py3 world is harsh | 21:25 |
| mattoliverau | lol | 21:25 |
| acoles | yes, so in the auto-sharding mode the node that thinks it is node index 0 in ring picks shard ranges and replicates them to other nodes. problem is if another node also thinks it is 0, but is likely to pick a different set of shards | 21:25 |
| mattoliverau | Yes, the WIP PR I have adds some majority quorum on who actaully is index 0 and what's the version of the ring, to get rid of old nodes who may not agree because they have an old ring | 21:26 |
| zaitcev | Oh, I see. | 21:27 |
| mattoliverau | but that's alot of expra requests. and does minimalise the split brain edge case window. but doesn't completely eradicate it. | 21:27 |
| zaitcev | Even if the network is split, the administrator is not split, the human maintains the rings, and that is the source of truth even if not used directly by sharding. | 21:28 |
| acoles | so mattoliverau 's work is towards my step (b) above - reduce the chance of a mistake in choosing the leader | 21:29 |
| mattoliverau | yup | 21:29 |
| acoles | I've been working on recovering from mistakes | 21:29 |
| mattoliverau | but turns out step (a) | 21:29 |
| mattoliverau | is what we need to solve. | 21:29 |
| acoles | so https://review.opendev.org/c/openstack/swift/+/765624 is a WIP, and i think mattoliverau may also have some ideas | 21:30 |
| acoles | ^^ building on some discussion at the PTG | 21:30 |
| mattoliverau | we apparently gerrit is not loading for me atm... | 21:31 |
| mattoliverau | *well | 21:31 |
| zaitcev | same here | 21:31 |
| acoles | I'm deliberately keeping it simple at first - the cases we have seen are 'simple' duplicate paths that you would expect if two nodes had acted as leaders but with different local sets of objects, so choosing slightly different shard ranges | 21:31 |
| mattoliverau | Anyway step (a) is what we want to solve first, when we do, leader election edge cases become less of an issue. | 21:31 |
| acoles | hmmm, I just pushed it gerrit but now also not loading for me | 21:32 |
| acoles | anyway, that patch adds a 'repair' command to swift-manage-shard-ranges that will find all paths, choose one and shrink all others into it | 21:32 |
| acoles | mattoliverau: did you have some graph visualisation stuff? IIRC you did some work before I returned to swift-land? it would be cool to see that too | 21:34 |
| mattoliverau | And I've been playing with a RangeScanner that can rebuild and/or choose best paths. It's latest addition is a testing out a new gap filler approach that uses the weighting algorithm to coose the best path (acoles spider suggestion). | 21:34 |
| acoles | cool. so mattoliverau checkout https://review.opendev.org/c/openstack/swift/+/765624 - we may have some overlaps :) | 21:35 |
| mattoliverau | acoles: yeah the patch includes some grpahvis to manage-shard-rangers show command that turns shardranges into a graph. | 21:35 |
| mattoliverau | nice :) | 21:35 |
| acoles | but I have dodged gap-filling for now | 21:35 |
| mattoliverau | I defintely will! | 21:36 |
| acoles | so one answer to dsariel's question - it would be great to have review of the patches we have in progress :) and review might include getting a container sharded with split-brain and checking out the new repair command etc | 21:36 |
| mattoliverau | and so should dsariel :) | 21:37 |
| mattoliverau | +1 | 21:37 |
| acoles | dsariel: the probe test in the patch may be a good starting point to understand the problem space | 21:37 |
| mattoliverau | the code isn't finished but reviewing and testing would be a huge help | 21:37 |
| acoles | (the probe test uses ridiculously small numbers of objects vs real life) | 21:38 |
| *** e0ne has joined #openstack-meeting | 21:38 | |
| acoles | I also have https://review.opendev.org/c/openstack/swift/+/765623/4 which adds a 'compact' command to swift-manage-shard-ranges, its a precursor to the other because it uses similar functionality i.e. shrinking unwanted shards) | 21:39 |
| acoles | sorry, I feel like this is shameless promotion of my patches, don't mean it to be | 21:39 |
| acoles | it's quite likely that reviewing those and mattoliverau's patches will generate further work to help move things along | 21:40 |
| mattoliverau | lol, not that's for all the work | 21:40 |
| mattoliverau | *thanks | 21:41 |
| mattoliverau | apparently I cant type this morning | 21:41 |
| acoles | dsariel: is that helpful? | 21:41 |
| mattoliverau | seeing as I can't access gerrit atm, I think this is my rangescanner (plus graphviz) POC/WIP: https://review.opendev.org/#/c/749614/ | 21:42 |
| dsariel | thanks, probe tests was the place I started to look at. I'll take a look on the patches. Guess will have many questions. Apologize in advance for that. | 21:42 |
| acoles | please ask questions | 21:42 |
| mattoliverau | you probably will, and thanks fine, great even. you'll see things a fresh which I think will be great! | 21:42 |
| mattoliverau | *that's fine | 21:43 |
| dsariel | Adding more objects to probe tests will increase the time they take. Is is possible to run them is a separate job? | 21:43 |
| acoles | BTW those patches I linked are on a chain that starts with a fix to shard audit that we found we needed in order to shrink some overlaps | 21:43 |
| mattoliverau | man, I need to read before I press <enter> :P | 21:43 |
| acoles | so dig down through the patch dependency | 21:43 |
| acoles | you can run an individual probetest with something like 'nosetests ./test/probe/test_sharder.py:TestManagedContainerSharding.test_manage_shard_ranges_repair_shard' | 21:44 |
| mattoliverau | Anything else on this topic? seems dsariel has a bunch of code to read and test :) | 21:45 |
| acoles | the small object count isn't necessarily a problem, I was just explaining that the tests aren't run at real world scale :) | 21:45 |
| dsariel | :-) | 21:45 |
| acoles | one other thing | 21:45 |
| acoles | I rediscovered this tool 'python swift/cli/shard-info.py' | 21:46 |
| acoles | it dumps all the root and shard container state after a probe test. its use is really limited to probe test analysis, it could definitely be improved, but it is a lot better that nothing | 21:47 |
| acoles | dsariel: reach out to us in #openstack-swift with any questions | 21:48 |
| mattoliverau | +100 | 21:48 |
| dsariel | awesome, thanks! will try it | 21:48 |
| mattoliverau | let's move on to open floor then. | 21:48 |
| mattoliverau | #topic open floor | 21:48 |
| *** openstack changes topic to "open floor (Meeting topic: swift)" | 21:48 | |
| *** timburke has joined #openstack-meeting | 21:49 | |
| mattoliverau | is there anything else anyone wants to bring up and discuss? | 21:49 |
| dsariel | thanks a lot for the directions | 21:49 |
| zaitcev | Yes | 21:49 |
| zaitcev | not on the topic of sharding though | 21:49 |
| * timburke sneaks in finally | 21:49 | |
| acoles | phew timburke will rescue us | 21:49 |
| mattoliverau | lol, hey timburke :) | 21:49 |
| zaitcev | timburke: 11 minutes left, come on | 21:50 |
| zaitcev | kid okay? | 21:50 |
| timburke | yup, just got overdue for his nap | 21:50 |
| zaitcev | so, I was looking at the failure of Romain's patch on py27 and so far I was unsuccessfull. | 21:51 |
| timburke | oh yeah -- the queuing patch, i think, is that right? | 21:52 |
| zaitcev | I pulled all the remotely relevant patches from eventlet 2.29 into the 2.25 that's locked in tox, but no dice. | 21:52 |
| zaitcev | I think I'll need to find just where the exceptions get stuck. | 21:53 |
| zaitcev | Most of the time it's ChunkWriteTimeout, although not always. | 21:53 |
| zaitcev | I'm going to dump every ChunkWriteTimeout as it's instantiated and trawl through them. | 21:54 |
| zaitcev | I'm not asking for help so far, but it looks grim | 21:54 |
| timburke | fwiw i suspect the ChunkWriteTimeout may be from an old watchdog for an already-passed test | 21:54 |
| zaitcev | Yeah, something like that. | 21:55 |
| timburke | it reminds me in some ways of the trouble i've seen in prod where a ChunkWriteTimeout pops and logs a path *but it has the wrong txn id* | 21:55 |
| timburke | i've grown worried about eventlet's (green)threadlocal behavior... | 21:56 |
| zaitcev | But it works fine on py3, right? | 21:56 |
| timburke | ...i guess? seems to be better, anyway | 21:57 |
| mattoliverau | sounds.. tedius, thanks zaitcev for going down this particular rabbit hole. | 21:57 |
| mattoliverau | we have 3 minutes before we reach time. Anything else or shall we move any discussions into #openstack-swift ? | 21:58 |
| zaitcev | I'm all set. | 21:58 |
| timburke | there are some py3 patches and some s3api patches i'd appreciate eyes on, but i can drop those in -swift | 21:59 |
| mattoliverau | kk thanks timburke | 21:59 |
| mattoliverau | timburke: maybe you could update priority reviews if you get the chance :) | 21:59 |
| mattoliverau | I'll call it | 21:59 |
| timburke | thank *you* mattoliverau! sorry i hadn't gotten to updating the agenda | 21:59 |
| timburke | that's a great idea! | 22:00 |
| acoles | thanks mattoliverau for jumping in to chair, great job! | 22:00 |
| mattoliverau | Thanks for all your hard work and thanks for working on swift! | 22:00 |
| mattoliverau | #endmeeting | 22:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:00 | |
| openstack | Meeting ended Wed Dec 9 22:00:15 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-12-09-21.03.html | 22:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-12-09-21.03.txt | 22:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-12-09-21.03.log.html | 22:00 |
| mattoliverau | my pleasure | 22:00 |
| *** zaitcev has left #openstack-meeting | 22:00 | |
| *** acoles has left #openstack-meeting | 22:00 | |
| *** rcernin has joined #openstack-meeting | 22:02 | |
| *** rcernin has quit IRC | 22:04 | |
| *** rcernin has joined #openstack-meeting | 22:05 | |
| *** ociuhandu has quit IRC | 22:09 | |
| *** raildo has quit IRC | 22:13 | |
| *** e0ne has quit IRC | 22:14 | |
| *** ociuhandu has joined #openstack-meeting | 22:15 | |
| *** timburke has quit IRC | 22:16 | |
| *** ociuhandu has quit IRC | 22:19 | |
| *** ociuhandu has joined #openstack-meeting | 22:20 | |
| *** tdasilva has quit IRC | 22:27 | |
| *** tdasilva has joined #openstack-meeting | 22:28 | |
| *** timburke has joined #openstack-meeting | 22:33 | |
| *** ociuhandu has quit IRC | 22:34 | |
| *** ociuhandu has joined #openstack-meeting | 22:41 | |
| *** baojg has joined #openstack-meeting | 22:43 | |
| *** dsariel has quit IRC | 22:51 | |
| *** baojg has quit IRC | 23:09 | |
| *** timburke has quit IRC | 23:09 | |
| *** baojg has joined #openstack-meeting | 23:09 | |
| *** TrevorV has quit IRC | 23:22 | |
| *** slaweq has quit IRC | 23:23 | |
| *** zbr has quit IRC | 23:46 | |
| *** zbr has joined #openstack-meeting | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!