| *** slaweq has quit IRC | 00:04 | |
| *** jiaopengju2 has joined #openstack-meeting | 00:06 | |
| *** jiaopengju1 has quit IRC | 00:06 | |
| *** slaweq has joined #openstack-meeting | 00:08 | |
| *** martial has quit IRC | 00:09 | |
| *** tetsuro has joined #openstack-meeting | 00:09 | |
| *** markmcclain has quit IRC | 00:13 | |
| *** slaweq has quit IRC | 00:13 | |
| *** ijw_ has quit IRC | 00:14 | |
| *** ijw has joined #openstack-meeting | 00:14 | |
| *** markmcclain has joined #openstack-meeting | 00:15 | |
| *** yamamoto has joined #openstack-meeting | 00:17 | |
| *** slaweq has joined #openstack-meeting | 00:18 | |
| *** slaweq has quit IRC | 00:22 | |
| *** markmcclain has quit IRC | 00:24 | |
| *** slaweq has joined #openstack-meeting | 00:29 | |
| *** slaweq has quit IRC | 00:34 | |
| *** slaweq has joined #openstack-meeting | 00:36 | |
| *** slaweq has quit IRC | 00:45 | |
| *** slaweq has joined #openstack-meeting | 00:46 | |
| *** yamamoto has quit IRC | 00:49 | |
| *** moguimar has quit IRC | 00:49 | |
| *** slaweq has quit IRC | 00:53 | |
| *** tetsuro_ has joined #openstack-meeting | 00:54 | |
| *** tetsuro has quit IRC | 00:58 | |
| *** slaweq has joined #openstack-meeting | 01:00 | |
| *** yamamoto has joined #openstack-meeting | 01:05 | |
| *** slaweq has quit IRC | 01:07 | |
| *** slaweq has joined #openstack-meeting | 01:12 | |
| *** slaweq has quit IRC | 01:20 | |
| *** slaweq has joined #openstack-meeting | 01:26 | |
| *** slaweq has quit IRC | 01:31 | |
| *** slaweq has joined #openstack-meeting | 01:35 | |
| *** slaweq has quit IRC | 01:39 | |
| *** slaweq has joined #openstack-meeting | 01:44 | |
| *** tetsuro_ has quit IRC | 01:47 | |
| *** tetsuro has joined #openstack-meeting | 01:48 | |
| *** tetsuro has quit IRC | 01:48 | |
| *** slaweq has quit IRC | 01:49 | |
| *** slaweq has joined #openstack-meeting | 01:52 | |
| *** slaweq has quit IRC | 01:56 | |
| *** slaweq has joined #openstack-meeting | 02:01 | |
| *** ociuhandu has joined #openstack-meeting | 02:01 | |
| *** bbowen__ has joined #openstack-meeting | 02:02 | |
| *** pots has quit IRC | 02:03 | |
| *** pots has joined #openstack-meeting | 02:03 | |
| *** slaweq has quit IRC | 02:05 | |
| *** bbowen_ has quit IRC | 02:05 | |
| *** smcginnis has quit IRC | 02:05 | |
| *** timburke has quit IRC | 02:05 | |
| *** rledisez has quit IRC | 02:05 | |
| *** jokke has quit IRC | 02:05 | |
| *** benj_ has quit IRC | 02:06 | |
| *** ociuhandu has quit IRC | 02:06 | |
| *** smcginnis has joined #openstack-meeting | 02:06 | |
| *** benj_ has joined #openstack-meeting | 02:07 | |
| *** ricolin has quit IRC | 02:09 | |
| *** slaweq has joined #openstack-meeting | 02:09 | |
| *** timburke has joined #openstack-meeting | 02:11 | |
| *** apetrich has quit IRC | 02:14 | |
| *** ricolin has joined #openstack-meeting | 02:19 | |
| *** slaweq has quit IRC | 02:19 | |
| *** rcernin has quit IRC | 02:21 | |
| *** masahito has joined #openstack-meeting | 02:25 | |
| *** jmasud has quit IRC | 02:33 | |
| *** jmasud has joined #openstack-meeting | 02:36 | |
| *** tetsuro has joined #openstack-meeting | 02:42 | |
| *** slaweq has joined #openstack-meeting | 02:45 | |
| *** ykatabam has quit IRC | 02:46 | |
| *** slaweq has quit IRC | 02:51 | |
| *** slaweq has joined #openstack-meeting | 02:54 | |
| *** rfolco has quit IRC | 02:54 | |
| *** slaweq has quit IRC | 02:58 | |
| *** ykatabam has joined #openstack-meeting | 02:59 | |
| *** andrebeltrami has quit IRC | 02:59 | |
| *** rcernin has joined #openstack-meeting | 02:59 | |
| *** slaweq has joined #openstack-meeting | 02:59 | |
| *** slaweq has quit IRC | 03:04 | |
| *** slaweq has joined #openstack-meeting | 03:09 | |
| *** slaweq has quit IRC | 03:13 | |
| *** armax has quit IRC | 03:19 | |
| *** slaweq has joined #openstack-meeting | 03:20 | |
| *** slaweq has quit IRC | 03:25 | |
| *** slaweq has joined #openstack-meeting | 03:30 | |
| *** vishalmanchanda has joined #openstack-meeting | 03:30 | |
| *** slaweq has quit IRC | 03:35 | |
| *** psachin has joined #openstack-meeting | 03:37 | |
| *** slaweq has joined #openstack-meeting | 03:39 | |
| *** manpreet has quit IRC | 03:40 | |
| *** slaweq has quit IRC | 03:43 | |
| *** yamamoto has quit IRC | 03:45 | |
| *** tetsuro_ has joined #openstack-meeting | 03:47 | |
| *** slaweq has joined #openstack-meeting | 03:50 | |
| *** tetsuro has quit IRC | 03:50 | |
| *** slaweq has quit IRC | 04:01 | |
| *** yamamoto has joined #openstack-meeting | 04:02 | |
| *** manpreet has joined #openstack-meeting | 04:03 | |
| *** slaweq has joined #openstack-meeting | 04:05 | |
| *** slaweq has quit IRC | 04:09 | |
| *** slaweq has joined #openstack-meeting | 04:12 | |
| *** rcernin has quit IRC | 04:14 | |
| *** slaweq has quit IRC | 04:16 | |
| *** rcernin has joined #openstack-meeting | 04:22 | |
| *** tetsuro has joined #openstack-meeting | 04:24 | |
| *** tetsuro_ has quit IRC | 04:27 | |
| *** rh-jelabarre has quit IRC | 04:29 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** rcernin has quit IRC | 04:37 | |
| *** rcernin has joined #openstack-meeting | 04:46 | |
| *** ykatabam has quit IRC | 04:51 | |
| *** ykatabam has joined #openstack-meeting | 04:51 | |
| *** masahito has quit IRC | 04:58 | |
| *** kaisers has quit IRC | 05:17 | |
| *** psahoo_ has joined #openstack-meeting | 05:19 | |
| *** gyee has quit IRC | 05:20 | |
| *** ianychoi_ has joined #openstack-meeting | 05:25 | |
| *** masahito has joined #openstack-meeting | 05:26 | |
| *** ianychoi has quit IRC | 05:28 | |
| *** kaisers has joined #openstack-meeting | 05:34 | |
| *** tetsuro has quit IRC | 05:57 | |
| *** manuvakery has joined #openstack-meeting | 06:01 | |
| *** dmellado has quit IRC | 06:11 | |
| *** apetrich has joined #openstack-meeting | 06:29 | |
| *** dmellado has joined #openstack-meeting | 06:41 | |
| *** dklyle has quit IRC | 06:46 | |
| *** lpetrut has joined #openstack-meeting | 06:51 | |
| *** dmellado has quit IRC | 07:07 | |
| *** slaweq has joined #openstack-meeting | 07:14 | |
| *** belmoreira has joined #openstack-meeting | 07:19 | |
| *** ijw has quit IRC | 07:45 | |
| *** e0ne has joined #openstack-meeting | 07:51 | |
| *** dmellado has joined #openstack-meeting | 07:53 | |
| *** moguimar has joined #openstack-meeting | 07:54 | |
| *** ijw has joined #openstack-meeting | 08:13 | |
| *** ijw has quit IRC | 08:18 | |
| *** belmoreira has quit IRC | 08:21 | |
| *** slaweq has quit IRC | 08:25 | |
| *** tosky has joined #openstack-meeting | 08:29 | |
| *** ociuhandu has joined #openstack-meeting | 08:34 | |
| *** ijw has joined #openstack-meeting | 08:44 | |
| *** ijw has quit IRC | 08:49 | |
| *** ijw has joined #openstack-meeting | 09:15 | |
| *** apetrich has quit IRC | 09:15 | |
| *** ijw has quit IRC | 09:20 | |
| *** slaweq has joined #openstack-meeting | 09:29 | |
| *** Lucas_Gray has joined #openstack-meeting | 09:42 | |
| *** rcernin has quit IRC | 09:43 | |
| *** ijw has joined #openstack-meeting | 09:46 | |
| *** rcernin has joined #openstack-meeting | 09:50 | |
| *** ijw has quit IRC | 09:51 | |
| *** Lucas_Gray has quit IRC | 09:58 | |
| *** Lucas_Gray has joined #openstack-meeting | 10:01 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** ijw has joined #openstack-meeting | 10:18 | |
| *** yamamoto has joined #openstack-meeting | 10:22 | |
| *** ijw has quit IRC | 10:23 | |
| *** yamamoto has quit IRC | 10:26 | |
| *** yamamoto has joined #openstack-meeting | 10:29 | |
| *** rcernin has quit IRC | 10:30 | |
| *** rcernin has joined #openstack-meeting | 10:35 | |
| *** yamamoto has quit IRC | 10:41 | |
| *** yamamoto has joined #openstack-meeting | 10:48 | |
| *** rcernin has quit IRC | 10:48 | |
| *** masahito has quit IRC | 10:48 | |
| *** ijw has joined #openstack-meeting | 10:50 | |
| *** rcernin has joined #openstack-meeting | 10:53 | |
| *** ijw has quit IRC | 10:54 | |
| *** psahoo_ has quit IRC | 11:03 | |
| *** rcernin has quit IRC | 11:07 | |
| *** armstrong has joined #openstack-meeting | 11:12 | |
| *** psahoo_ has joined #openstack-meeting | 11:15 | |
| *** yamamoto has quit IRC | 11:20 | |
| *** yamamoto has joined #openstack-meeting | 11:20 | |
| *** yamamoto has quit IRC | 11:21 | |
| *** ijw has joined #openstack-meeting | 11:21 | |
| *** rcernin has joined #openstack-meeting | 11:22 | |
| *** psahoo_ has quit IRC | 11:23 | |
| *** ijw has quit IRC | 11:26 | |
| *** belmoreira has joined #openstack-meeting | 11:27 | |
| *** yamamoto has joined #openstack-meeting | 11:31 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** psahoo_ has joined #openstack-meeting | 11:38 | |
| *** raildo has joined #openstack-meeting | 11:39 | |
| *** _erlon_ has joined #openstack-meeting | 11:44 | |
| *** apetrich has joined #openstack-meeting | 11:46 | |
| *** apetrich has quit IRC | 11:52 | |
| *** ijw has joined #openstack-meeting | 11:52 | |
| *** apetrich has joined #openstack-meeting | 11:52 | |
| *** rh-jelabarre has joined #openstack-meeting | 11:58 | |
| *** ijw has quit IRC | 11:59 | |
| *** rfolco has joined #openstack-meeting | 12:03 | |
| *** yamamoto has joined #openstack-meeting | 12:15 | |
| *** ijw has joined #openstack-meeting | 12:23 | |
| *** ijw has quit IRC | 12:28 | |
| *** yamamoto has quit IRC | 12:30 | |
| *** ijw has joined #openstack-meeting | 12:30 | |
| *** b1airo has quit IRC | 12:36 | |
| *** rh-jelabarre has quit IRC | 12:41 | |
| *** rcernin has quit IRC | 12:46 | |
| *** rcernin has joined #openstack-meeting | 12:49 | |
| *** psahoo_ has quit IRC | 12:59 | |
| *** psahoo_ has joined #openstack-meeting | 13:01 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:02 | |
| *** ijw has quit IRC | 13:06 | |
| *** psahoo_ has quit IRC | 13:08 | |
| *** psahoo_ has joined #openstack-meeting | 13:11 | |
| *** armstrong has quit IRC | 13:21 | |
| *** ijw has joined #openstack-meeting | 13:25 | |
| *** priteau has joined #openstack-meeting | 13:31 | |
| *** TrevorV has joined #openstack-meeting | 13:46 | |
| *** manuvakery has quit IRC | 13:48 | |
| *** dklyle has joined #openstack-meeting | 13:50 | |
| *** seba has quit IRC | 13:52 | |
| *** ijw has quit IRC | 13:56 | |
| *** ijw_ has joined #openstack-meeting | 13:56 | |
| *** Liang__ has joined #openstack-meeting | 13:59 | |
| *** Liang__ is now known as LiangFang | 14:00 | |
| *** ijw_ has quit IRC | 14:01 | |
| *** masahito has joined #openstack-meeting | 14:06 | |
| *** masahito has quit IRC | 14:25 | |
| *** masahito has joined #openstack-meeting | 14:26 | |
| *** Lucas_Gray has quit IRC | 14:27 | |
| *** ijw has joined #openstack-meeting | 14:27 | |
| *** yamamoto has joined #openstack-meeting | 14:28 | |
| *** ijw has quit IRC | 14:32 | |
| *** Lucas_Gray has joined #openstack-meeting | 14:34 | |
| *** jgriffith has joined #openstack-meeting | 14:35 | |
| *** rcernin has quit IRC | 14:37 | |
| *** slaweq has quit IRC | 14:38 | |
| *** Lucas_Gray has quit IRC | 14:39 | |
| *** slaweq has joined #openstack-meeting | 14:40 | |
| *** Lucas_Gray has joined #openstack-meeting | 14:40 | |
| *** ijw has joined #openstack-meeting | 14:41 | |
| *** mlavalle has joined #openstack-meeting | 14:44 | |
| *** slaweq has quit IRC | 14:48 | |
| *** armax has joined #openstack-meeting | 14:50 | |
| *** dklyle has quit IRC | 14:51 | |
| *** slaweq has joined #openstack-meeting | 14:51 | |
| *** belmoreira has quit IRC | 14:55 | |
| *** dklyle has joined #openstack-meeting | 15:00 | |
| *** larainema has joined #openstack-meeting | 15:04 | |
| *** psahoo_ has quit IRC | 15:11 | |
| *** yamamoto has quit IRC | 15:28 | |
| *** ykatabam has quit IRC | 15:29 | |
| *** LiangFang has quit IRC | 15:35 | |
| *** gyee has joined #openstack-meeting | 15:44 | |
| *** seba has joined #openstack-meeting | 15:45 | |
| *** apetrich has quit IRC | 15:46 | |
| *** yamamoto has joined #openstack-meeting | 15:58 | |
| *** ociuhandu_ has joined #openstack-meeting | 16:30 | |
| *** Lucas_Gray has quit IRC | 16:32 | |
| *** masahito has quit IRC | 16:33 | |
| *** ociuhandu has quit IRC | 16:34 | |
| *** ociuhandu_ has quit IRC | 16:34 | |
| *** yamamoto has quit IRC | 16:37 | |
| *** ociuhandu has joined #openstack-meeting | 16:48 | |
| *** lpetrut has quit IRC | 16:50 | |
| *** ociuhandu has quit IRC | 16:52 | |
| *** psachin has quit IRC | 16:53 | |
| *** e0ne has quit IRC | 17:44 | |
| *** maciejjozefczyk_ has joined #openstack-meeting | 17:47 | |
| *** maciejjozefczyk has quit IRC | 17:48 | |
| *** maciejjozefczyk has joined #openstack-meeting | 17:49 | |
| *** maciejjozefczyk_ has quit IRC | 17:52 | |
| *** lpetrut has joined #openstack-meeting | 18:10 | |
| *** lpetrut has quit IRC | 18:17 | |
| *** ijw has quit IRC | 18:18 | |
| *** apetrich has joined #openstack-meeting | 18:21 | |
| *** ijw has joined #openstack-meeting | 18:33 | |
| *** yamamoto has joined #openstack-meeting | 18:36 | |
| *** maciejjozefczyk has quit IRC | 18:36 | |
| *** maciejjozefczyk has joined #openstack-meeting | 18:37 | |
| *** ijw has quit IRC | 18:37 | |
| *** ijw has joined #openstack-meeting | 18:38 | |
| *** vishalmanchanda has quit IRC | 18:49 | |
| *** ociuhandu has joined #openstack-meeting | 19:04 | |
| *** yamamoto has quit IRC | 19:13 | |
| *** ociuhandu has quit IRC | 19:34 | |
| *** priteau has quit IRC | 19:44 | |
| *** rledisez has joined #openstack-meeting | 20:22 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:22 | |
| *** ociuhandu has joined #openstack-meeting | 20:22 | |
| *** ijw has quit IRC | 20:27 | |
| *** ijw has joined #openstack-meeting | 20:28 | |
| *** ijw has quit IRC | 20:31 | |
| *** ijw has joined #openstack-meeting | 20:31 | |
| *** dustinc has quit IRC | 20:49 | |
| timburke | #startmeeting swift | 21:00 |
|---|---|---|
| openstack | Meeting started Wed Aug 19 21:00:01 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| seongsoocho | o/ | 21:00 |
| *** patchbot has joined #openstack-meeting | 21:00 | |
| rledisez | o/ | 21:00 |
| clayg | o/ | 21:01 |
| *** raildo has quit IRC | 21:01 | |
| timburke | as always, agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke | #topic PTG | 21:01 |
| *** openstack changes topic to "PTG (Meeting topic: swift)" | 21:01 | |
| timburke | just wanted a quick reminder to register | 21:02 |
| timburke | #link https://www.eventbrite.com/e/project-teams-gathering-october-2020-tickets-116136313841 | 21:02 |
| timburke | i'll make an etherpad to start gathering topics, and send out a doodle to help us schedule meeting times | 21:03 |
| timburke | more details about the scheduling and reservation process at http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016497.html if anyone's curious | 21:04 |
| timburke | #topic s3api logging | 21:04 |
| *** openstack changes topic to "s3api logging (Meeting topic: swift)" | 21:04 | |
| *** rfolco has quit IRC | 21:05 | |
| timburke | clayg, i carried over the one still-open patch from last week's agenda -- is there much we need to discuss about it? | 21:05 |
| clayg | nope, all done - I approved https://review.opendev.org/#/c/735220/ | 21:05 |
| patchbot | patch 735220 - swift - proxy-logging: Be able to configure log_route - 3 patch sets | 21:05 |
| clayg | I think I asked a couple weeks ago if anyone else had noticed the duplication in things like transfer stats | 21:06 |
| *** ociuhandu has quit IRC | 21:06 | |
| timburke | excellent! and i see i accidentally took the wrong one off the agenda :-/ oops | 21:06 |
| clayg | but it's possible most clusters aren't dealing with such a large percentage of s3api requests 🤷♂️ | 21:06 |
| clayg | anyway, i put "update example configs" on my list - since it's a best practices issue we might discuss that as part of the review | 21:07 |
| timburke | given trends, i expect most clusters to see that percentage increasing | 21:07 |
| *** slaweq has quit IRC | 21:07 | |
| timburke | thanks for, er, following up with the follow-up work! | 21:07 |
| clayg | well then folks might want to be thinking about setting up a different route/prefix for their subrequets logging middleware! | 21:07 |
| *** maciejjozefczyk has quit IRC | 21:08 | |
| clayg | kill the topic for now; i'll re-raise later | 21:08 |
| timburke | #topic py3 crypto bug | 21:08 |
| *** openstack changes topic to "py3 crypto bug (Meeting topic: swift)" | 21:08 | |
| timburke | #link https://review.opendev.org/#/c/742033/ | 21:08 |
| patchbot | patch 742033 - swift - py3: Work with proper native string paths in crypt... - 4 patch sets | 21:08 |
| clayg | this one is still very much on my radar after a long delay - did anyone else look at it? | 21:08 |
| timburke | that was basically exactly my question :-) that, and "if so, do you have any questions about the approach or code?" | 21:09 |
| *** jmasud has quit IRC | 21:10 | |
| clayg | I think we discussed it last week and everyone was onboard with the weird glitch of trying to upgrade from py2-old-swift to py3-new-swift in one go where old py2-old-swift couldn't read the py3v2 format | 21:11 |
| timburke | i guess that's a no on looking at it. does anyone think they'll have time to look at it soon-ish? | 21:11 |
| *** yamamoto has joined #openstack-meeting | 21:12 | |
| timburke | clayg, that was my impression as well -- people were generally on board with requiring a py2-new-swift step in between, if only because there isn't really a way to do it safely otherwise | 21:12 |
| clayg | timburke: are you asking "besides clayg"? Sorry I haven't gotten back to it yet, hoping to do it this week! | 21:12 |
| clayg | or... at least "before next wednesday" (so you can REALLY fuss at me again next week) | 21:13 |
| timburke | no worries! we'll dig out from our review backlog some day! | 21:13 |
| timburke | all right, moving along then | 21:13 |
| timburke | #topic listen socket per worker | 21:13 |
| *** openstack changes topic to "listen socket per worker (Meeting topic: swift)" | 21:13 | |
| timburke | #link https://review.opendev.org/#/c/745603/ | 21:13 |
| patchbot | patch 745603 - swift - Bind a new socket per-worker - 3 patch sets | 21:13 |
| timburke | so i got around to testing this out in prod today | 21:13 |
| timburke | results look promising! | 21:14 |
| rledisez | timburke: did you observed improvement on bandwidth or ttfb for example? I could expect that the 99percentile ttfb decrease | 21:15 |
| timburke | went from generally 10-30 req/worker and one poor guy with 170 reqs to 15-35 req/worker | 21:15 |
| clayg | oh wow! | 21:15 |
| timburke | rledisez, not sure yet -- i'll definitely look for it now that you ask | 21:15 |
| *** kaisers1 has joined #openstack-meeting | 21:15 | |
| *** zaitcev has joined #openstack-meeting | 21:15 | |
| *** moguimar has quit IRC | 21:16 | |
| *** kaisers has quit IRC | 21:17 | |
| rledisez | at some point I was playing with uwsgi to work around that behavior and I was able to increase the bandwidth used by the proxy server (all the core was involved instead of just a few of them) | 21:17 |
| timburke | this one proxy has been running like that for two or three hours now, and it certainly doesn't seem to have made anything *worse* | 21:17 |
| rledisez | timburke: that's all I need to read :D | 21:17 |
| *** slaweq has joined #openstack-meeting | 21:18 | |
| timburke | as a heads-up, there's now an eventlet issue open about it, too | 21:19 |
| timburke | #link https://github.com/eventlet/eventlet/issues/635 | 21:19 |
| *** jmasud has joined #openstack-meeting | 21:20 | |
| timburke | oh, looks like it's closed now. i'm glad i replied :-) | 21:20 |
| timburke | #topic 404s during rebalance | 21:20 |
| *** openstack changes topic to "404s during rebalance (Meeting topic: swift)" | 21:20 | |
| timburke | #link https://review.opendev.org/#/c/744942/ | 21:21 |
| patchbot | patch 744942 - swift - Client should retry when there's just one 404 and ... - 5 patch sets | 21:21 |
| clayg | wow that's interesting! | 21:21 |
| timburke | so we've observed troubles where we'd return a 404 even though data definitely *did* exist in the cluster -- and we're pretty sure we traced it back to primaries generally getting overloaded and timing out coupled with a rebalance that meant *one* primary could respond quickly with a 404 | 21:22 |
| clayg | those new primaries just LOVE to hand out 404s - you want data? nope. you want data? nope, none for you either. This is easy 😎 | 21:23 |
| *** slaweq has quit IRC | 21:23 | |
| seongsoocho | that means, the first response from object node is 404 and the other is timeout ? | 21:24 |
| clayg | seongsoocho: yes [Timeout, Timeout, 404] should be 503 | 21:25 |
| seongsoocho | oh i see | 21:25 |
| *** ykatabam has joined #openstack-meeting | 21:25 | |
| clayg | or at least, in our experience the most useful thing a client can do with a response to that set of backend primary responses is exponential backoff and then retry | 21:25 |
| clayg | it's possible they'll get a 404 on [Timeout, 404, 404] - but that didn't seem to fit the pattern we observed most often | 21:26 |
| clayg | I think tim made it configurable - but the default is 1 - which is what we want | 21:27 |
| timburke | yeah, i added a new config option, ignore_rebalance_404s, that basically says, "if you get this many or fewer 404s from primaries, assume they're due to a rebalance and ignore them" | 21:27 |
| clayg | I want to try to see if ignore_rebalance_404s = 0 mostly still passes existing assertions?! | 21:27 |
| clayg | this patch and the worker socket patch are right there on the same list py3 crypto 😞 | 21:28 |
| timburke | which isn't great in terms of the logging you get out of best_response, but i'm realizing that logging isn't great regardless (*especially* for ec data...) | 21:28 |
| seongsoocho | then, operator should setting that value before the rebalance, and remove it after the rebalance? | 21:28 |
| *** ijw_ has joined #openstack-meeting | 21:29 | |
| timburke | you can -- but i think the default of 1 should be reasonable basically all the time (unless your rebalances are moving faster than replication can keep up) | 21:30 |
| *** jmasud has quit IRC | 21:30 | |
| *** andrebeltrami has joined #openstack-meeting | 21:31 | |
| clayg | seongsoocho: another neat thing is that [Timeout(), Timeout(), 404-with-tombstone-timestamp] will still return 404 😎 | 21:32 |
| kota_ | sorry i'm late | 21:32 |
| seongsoocho | aha. ok. it looks very good. 🙂 I also have experience there are some 503 response during rebalance. | 21:32 |
| timburke | unless you've got min_part_hours=0 (iirc), the rebalance should only move one replica's worth of data at a time. i suppose you could explicitly set ignore_rebalance_404s to 0 when you know your rings ought to be stable, but even then you may have an empty primary because the PUT went to handoffs or something | 21:33 |
| *** ijw has quit IRC | 21:33 | |
| *** slaweq has joined #openstack-meeting | 21:33 | |
| clayg | timburke: yeah I feel like this strategy of "keep responses out of best_response" isn't really extending very well | 21:34 |
| timburke | anyway, i mainly just wanted to raise awareness of this issue we've seen, check if anyone else has seen erroneous 404s during rebalances, and feel out whether the config tunable feels like the right sort of a knob | 21:34 |
| clayg | it feels kinda similar in spirit to something like write_affinity_handoff_delete_count | 21:36 |
| *** jmasud has joined #openstack-meeting | 21:36 | |
| timburke | fwiw, there are limits to how far you can push it: negative values get raised to 0, and values at replica_count or higher get reduced to replica_count - 1 | 21:37 |
| timburke | (so the default is actually "0 if replica_count == 1 else 1") | 21:37 |
| clayg | 💡 | 21:37 |
| clayg | i'll probably review this one first - what a great patch Tim! ❤️ | 21:38 |
| *** slaweq has quit IRC | 21:38 | |
| timburke | that was a bit of an issue for me with tests in the gate, since tempest tests (and i think dsvm?) run with replica_count == 1 | 21:38 |
| timburke | all right, that's all i've got | 21:38 |
| timburke | #topic open discussion | 21:38 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:38 | |
| timburke | what else should we talk about this week? | 21:38 |
| clayg | another thing to note is that patch is built on waterfall-ec | 21:39 |
| *** yamamoto has quit IRC | 21:39 | |
| kota_ | I added some comments to liberasurecode patch and waiting response from timburke | 21:39 |
| kota_ | please revisit that. | 21:40 |
| timburke | yes! thank you, kota_! sorry, i've been meaning to reply | 21:40 |
| kota_ | just notification ;) | 21:40 |
| clayg | Does anyone have any reservations they've been holding back on waterfall-ec; matt left a few comments I need to clean up and tim found a bug... but you know... otherwise it seems like it's almost ready! we're shipping it anyway 😁 | 21:42 |
| timburke | the long and short of it was that (1) i was lazy and didn't want to have to do string comparisons in C against "yes", "YES", "1", "true", "t", "y", etc. and (2) i wasn't sure how best to handle something like LIBERASURECODE_WRITE_LEGACY_CRC=asdf | 21:42 |
| clayg | LIBERASURECODE_WRITE_LEGACY_CRC=1 seems quite reasonable to me for C code, what does zaitcev think? | 21:43 |
| clayg | anyone have a link? | 21:43 |
| timburke | present/not present seemed to leave very little room for ambiguity :-) though i should probably at least make sure a blank value still counts as false | 21:43 |
| timburke | #link https://review.opendev.org/#/c/738959/ | 21:43 |
| patchbot | patch 738959 - liberasurecode - Be willing to write fragments with legacy crc - 2 patch sets | 21:43 |
| * zaitcev thinks just the existence of the variable was a flag to us | 21:44 | |
| timburke | and | 21:44 |
| timburke | #link | 21:44 |
| timburke | #link https://review.opendev.org/#/c/739164/ | 21:44 |
| patchbot | patch 739164 - swift - ec: Add an option to write fragments with legacy crc - 1 patch set | 21:44 |
| zaitcev | I'll check what Kota found over my +2, likely I missed something. | 21:45 |
| kota_ | thx zaitcev | 21:45 |
| kota_ | i might be worried too much. | 21:45 |
| *** tosky has quit IRC | 21:47 | |
| timburke | all right, seems like we're winding down | 21:47 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:48 |
| timburke | #endmeeting | 21:48 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:48 | |
| openstack | Meeting ended Wed Aug 19 21:48:13 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:48 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-08-19-21.00.html | 21:48 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-08-19-21.00.txt | 21:48 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-08-19-21.00.log.html | 21:48 |
| *** zaitcev has left #openstack-meeting | 21:48 | |
| *** patchbot has left #openstack-meeting | 21:51 | |
| *** yamamoto has joined #openstack-meeting | 22:06 | |
| *** jmasud has quit IRC | 22:14 | |
| *** jmasud has joined #openstack-meeting | 22:16 | |
| *** yamamoto has quit IRC | 22:25 | |
| *** TrevorV has quit IRC | 22:31 | |
| *** apetrich has quit IRC | 22:47 | |
| *** Lucas_Gray has quit IRC | 22:55 | |
| *** rcernin has joined #openstack-meeting | 22:59 | |
| *** Lucas_Gray has joined #openstack-meeting | 23:02 | |
| *** mlavalle has quit IRC | 23:03 | |
| *** ociuhandu has joined #openstack-meeting | 23:06 | |
| *** ociuhandu has quit IRC | 23:11 | |
| *** Lucas_Gray has quit IRC | 23:24 | |
| *** tetsuro has joined #openstack-meeting | 23:44 | |
| *** jmasud has quit IRC | 23:53 | |
| *** tetsuro has quit IRC | 23:57 | |
| *** jmasud has joined #openstack-meeting | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!