| *** tosky has quit IRC | 00:02 | |
| *** ociuhandu has quit IRC | 00:03 | |
| *** martial has quit IRC | 00:05 | |
| *** jmasud has joined #openstack-meeting | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:17 | |
| *** xarlos has joined #openstack-meeting | 00:18 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:18 | |
| *** rcernin_ has joined #openstack-meeting | 00:19 | |
| *** macz_ has quit IRC | 00:19 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** rcernin has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:21 | |
| *** yasufum has joined #openstack-meeting | 00:27 | |
| *** Adri2000 has quit IRC | 00:43 | |
| *** jamesmcarthur has quit IRC | 00:44 | |
| *** Adri2000 has joined #openstack-meeting | 00:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:45 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:49 | |
| *** jamesden_ has joined #openstack-meeting | 01:00 | |
| *** abhishekk has quit IRC | 01:02 | |
| *** jamesdenton has quit IRC | 01:03 | |
| *** abhishekk has joined #openstack-meeting | 01:04 | |
| *** mlavalle has quit IRC | 01:05 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:35 | |
| *** jamesmcarthur has quit IRC | 01:40 | |
| *** baojg has joined #openstack-meeting | 01:41 | |
| *** armax has quit IRC | 01:47 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:06 | |
| *** jamesmcarthur has quit IRC | 02:14 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:15 | |
| *** ricolin has joined #openstack-meeting | 02:45 | |
| *** jamesmcarthur has quit IRC | 02:47 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:49 | |
| *** jamesmcarthur has quit IRC | 02:50 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:50 | |
| *** jmasud has quit IRC | 02:55 | |
| *** jmasud has joined #openstack-meeting | 03:05 | |
| *** rcernin_ has quit IRC | 03:23 | |
| *** jmlowe has quit IRC | 03:39 | |
| *** armstrong has joined #openstack-meeting | 03:46 | |
| *** armstrong has quit IRC | 03:51 | |
| *** jmasud has quit IRC | 03:52 | |
| *** baojg has quit IRC | 03:57 | |
| *** rcernin_ has joined #openstack-meeting | 04:01 | |
| *** psachin has joined #openstack-meeting | 04:11 | |
| *** baojg has joined #openstack-meeting | 05:29 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-meeting | 05:33 | |
| *** ricolin_ has joined #openstack-meeting | 05:47 | |
| *** ricolin has quit IRC | 05:47 | |
| *** jamesmcarthur has quit IRC | 05:50 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:52 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:52 | |
| *** jamesmcarthur has quit IRC | 05:57 | |
| *** gyee has quit IRC | 05:59 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:00 | |
| *** zbr5 has joined #openstack-meeting | 06:04 | |
| *** jmasud has joined #openstack-meeting | 06:06 | |
| *** zbr has quit IRC | 06:06 | |
| *** zbr5 is now known as zbr | 06:06 | |
| *** jmasud has quit IRC | 06:21 | |
| *** jmasud has joined #openstack-meeting | 06:25 | |
| *** bbowen has quit IRC | 06:31 | |
| *** bbowen has joined #openstack-meeting | 06:31 | |
| *** jamesmcarthur has quit IRC | 06:37 | |
| *** slaweq has joined #openstack-meeting | 07:04 | |
| *** rcernin_ has quit IRC | 07:28 | |
| *** slaweq has quit IRC | 07:30 | |
| *** rcernin_ has joined #openstack-meeting | 07:32 | |
| *** jmasud has quit IRC | 07:45 | |
| *** slaweq has joined #openstack-meeting | 08:00 | |
| *** rbudden has quit IRC | 08:15 | |
| *** rbudden has joined #openstack-meeting | 08:15 | |
| *** rbudden has quit IRC | 08:15 | |
| *** rbudden has joined #openstack-meeting | 08:16 | |
| *** rbudden has quit IRC | 08:16 | |
| *** rbudden has joined #openstack-meeting | 08:17 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| *** rbudden has quit IRC | 08:17 | |
| *** rbudden has joined #openstack-meeting | 08:18 | |
| *** rbudden has quit IRC | 08:18 | |
| *** rbudden has joined #openstack-meeting | 08:18 | |
| *** rbudden has quit IRC | 08:19 | |
| *** rbudden has joined #openstack-meeting | 08:19 | |
| *** rbudden has quit IRC | 08:19 | |
| *** rbudden has joined #openstack-meeting | 08:20 | |
| *** rbudden has quit IRC | 08:20 | |
| *** rbudden has joined #openstack-meeting | 08:21 | |
| *** rbudden has quit IRC | 08:21 | |
| *** rbudden has joined #openstack-meeting | 08:22 | |
| *** rbudden has joined #openstack-meeting | 08:22 | |
| *** rbudden has quit IRC | 08:23 | |
| *** rbudden has joined #openstack-meeting | 08:23 | |
| *** rbudden has joined #openstack-meeting | 08:24 | |
| *** rbudden has quit IRC | 08:24 | |
| *** rbudden has joined #openstack-meeting | 08:25 | |
| *** rbudden has quit IRC | 08:25 | |
| *** rcernin_ has quit IRC | 08:35 | |
| *** jamesmcarthur has joined #openstack-meeting | 08:37 | |
| *** abhishekk has quit IRC | 08:40 | |
| *** psachin has quit IRC | 08:41 | |
| *** abhishekk has joined #openstack-meeting | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:42 | |
| *** tosky has joined #openstack-meeting | 08:47 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** e0ne has joined #openstack-meeting | 09:11 | |
| *** DinaBelova has quit IRC | 09:17 | |
| *** freerunner has quit IRC | 09:17 | |
| *** freerunner has joined #openstack-meeting | 09:19 | |
| *** DinaBelova has joined #openstack-meeting | 09:19 | |
| *** takahashi-tsc has joined #openstack-meeting | 09:20 | |
| *** yamamoto has joined #openstack-meeting | 09:21 | |
| *** yamamoto has quit IRC | 09:23 | |
| *** yamamoto has joined #openstack-meeting | 09:30 | |
| *** yamamoto has quit IRC | 09:37 | |
| *** baojg has quit IRC | 09:45 | |
| *** yamamoto has joined #openstack-meeting | 09:45 | |
| *** yamamoto has quit IRC | 09:45 | |
| *** baojg has joined #openstack-meeting | 09:46 | |
| *** baojg has quit IRC | 09:47 | |
| *** yamamoto has joined #openstack-meeting | 09:47 | |
| *** baojg has joined #openstack-meeting | 09:48 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** baojg has quit IRC | 10:00 | |
| *** baojg has joined #openstack-meeting | 10:00 | |
| *** yamamoto has joined #openstack-meeting | 10:03 | |
| *** ociuhandu has joined #openstack-meeting | 10:07 | |
| *** ociuhandu has quit IRC | 10:12 | |
| *** rpittau is now known as rpittau|bbl | 10:20 | |
| *** yamamoto has quit IRC | 10:20 | |
| *** e0ne has quit IRC | 10:21 | |
| *** ociuhandu has joined #openstack-meeting | 10:26 | |
| *** takahashi-tsc has quit IRC | 10:26 | |
| *** takahashi-tsc has joined #openstack-meeting | 10:27 | |
| *** priteau has joined #openstack-meeting | 10:32 | |
| *** takahashi-tsc has quit IRC | 10:32 | |
| *** e0ne has joined #openstack-meeting | 10:34 | |
| *** takahashi-tsc has joined #openstack-meeting | 10:34 | |
| *** e0ne has quit IRC | 10:34 | |
| *** rcernin_ has joined #openstack-meeting | 10:34 | |
| *** rcernin_ has quit IRC | 10:41 | |
| *** ociuhandu has quit IRC | 10:54 | |
| *** ociuhandu_ has joined #openstack-meeting | 10:54 | |
| *** ociuhandu_ has quit IRC | 10:54 | |
| *** ociuhandu has joined #openstack-meeting | 10:54 | |
| *** yamamoto has joined #openstack-meeting | 10:59 | |
| *** yamamoto has quit IRC | 11:07 | |
| *** ociuhandu has quit IRC | 11:14 | |
| *** ociuhandu_ has joined #openstack-meeting | 11:14 | |
| *** rpittau|bbl is now known as rpittau | 11:24 | |
| *** macz_ has joined #openstack-meeting | 11:34 | |
| *** baojg has quit IRC | 11:37 | |
| *** baojg has joined #openstack-meeting | 11:37 | |
| *** baojg has quit IRC | 11:37 | |
| *** baojg has joined #openstack-meeting | 11:38 | |
| *** baojg has quit IRC | 11:38 | |
| *** baojg has joined #openstack-meeting | 11:38 | |
| *** baojg has quit IRC | 11:39 | |
| *** macz_ has quit IRC | 11:39 | |
| *** baojg has joined #openstack-meeting | 11:39 | |
| *** baojg has quit IRC | 11:40 | |
| *** baojg has joined #openstack-meeting | 11:40 | |
| *** baojg has quit IRC | 11:40 | |
| *** baojg has joined #openstack-meeting | 11:41 | |
| *** baojg has quit IRC | 11:41 | |
| *** baojg has joined #openstack-meeting | 11:42 | |
| *** ociuhandu_ has quit IRC | 11:42 | |
| *** yamamoto has joined #openstack-meeting | 11:42 | |
| *** rcernin_ has joined #openstack-meeting | 11:42 | |
| *** baojg has quit IRC | 11:42 | |
| *** baojg has joined #openstack-meeting | 11:42 | |
| *** baojg has quit IRC | 11:43 | |
| *** baojg has joined #openstack-meeting | 11:43 | |
| *** baojg has quit IRC | 11:44 | |
| *** baojg has joined #openstack-meeting | 11:44 | |
| *** takahashi-tsc has quit IRC | 11:44 | |
| *** baojg has quit IRC | 11:44 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** baojg has joined #openstack-meeting | 11:46 | |
| *** takahashi-tsc has joined #openstack-meeting | 11:48 | |
| *** baojg has quit IRC | 11:50 | |
| *** bbowen has quit IRC | 11:57 | |
| *** bbowen has joined #openstack-meeting | 11:59 | |
| *** yasufum has quit IRC | 12:01 | |
| *** raildo has joined #openstack-meeting | 12:03 | |
| *** jkulik has joined #openstack-meeting | 12:03 | |
| *** yamamoto has joined #openstack-meeting | 12:20 | |
| *** yamamoto has quit IRC | 12:31 | |
| *** dmacpher_ has quit IRC | 12:31 | |
| *** dmacpher has joined #openstack-meeting | 12:35 | |
| *** e0ne has joined #openstack-meeting | 12:40 | |
| *** ociuhandu has joined #openstack-meeting | 12:42 | |
| *** ociuhandu has quit IRC | 12:44 | |
| *** ociuhandu has joined #openstack-meeting | 12:45 | |
| *** evrardjp_ has joined #openstack-meeting | 12:52 | |
| *** rh-jelabarre has joined #openstack-meeting | 12:54 | |
| *** evrardjp has quit IRC | 12:54 | |
| *** ociuhandu has quit IRC | 12:54 | |
| *** ociuhandu has joined #openstack-meeting | 12:56 | |
| *** rcernin_ has quit IRC | 12:58 | |
| *** ociuhandu has quit IRC | 13:00 | |
| *** rh-jelabarre has quit IRC | 13:01 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:04 | |
| *** rcernin_ has joined #openstack-meeting | 13:08 | |
| *** rh-jelabarre has quit IRC | 13:11 | |
| *** ociuhandu has joined #openstack-meeting | 13:19 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:22 | |
| *** rh-jelabarre has quit IRC | 13:22 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:22 | |
| *** ociuhandu has quit IRC | 13:26 | |
| *** dasp_ has joined #openstack-meeting | 13:31 | |
| *** dasp has quit IRC | 13:34 | |
| *** ociuhandu has joined #openstack-meeting | 13:38 | |
| *** ociuhandu has quit IRC | 13:45 | |
| *** ociuhandu has joined #openstack-meeting | 13:47 | |
| *** ociuhandu has quit IRC | 13:51 | |
| *** ociuhandu has joined #openstack-meeting | 13:54 | |
| *** takahashi-tsc has quit IRC | 14:04 | |
| *** yasufum has joined #openstack-meeting | 14:12 | |
| *** rcernin_ has quit IRC | 14:13 | |
| *** vishalmanchanda has quit IRC | 14:15 | |
| *** irclogbot_0 has quit IRC | 14:27 | |
| *** irclogbot_3 has joined #openstack-meeting | 14:31 | |
| *** rh-jelabarre has quit IRC | 14:38 | |
| *** rcernin_ has joined #openstack-meeting | 14:40 | |
| *** rcernin_ has quit IRC | 14:45 | |
| *** vishalmanchanda has joined #openstack-meeting | 14:52 | |
| *** rcernin_ has joined #openstack-meeting | 14:58 | |
| *** ricolin_ has quit IRC | 15:00 | |
| *** whoami-rajat has joined #openstack-meeting | 15:02 | |
| *** rcernin_ has quit IRC | 15:03 | |
| *** rcernin_ has joined #openstack-meeting | 15:07 | |
| *** TrevorV has joined #openstack-meeting | 15:08 | |
| *** rbudden has joined #openstack-meeting | 15:11 | |
| *** rcernin_ has quit IRC | 15:12 | |
| *** yasufum has quit IRC | 15:20 | |
| *** yasufum has joined #openstack-meeting | 15:20 | |
| *** slaweq has quit IRC | 15:21 | |
| *** slaweq has joined #openstack-meeting | 15:23 | |
| *** armax has joined #openstack-meeting | 15:41 | |
| *** yasufum has quit IRC | 15:57 | |
| *** e0ne has quit IRC | 15:58 | |
| *** imcsk8 has quit IRC | 16:01 | |
| *** jmasud has joined #openstack-meeting | 16:04 | |
| *** imcsk8 has joined #openstack-meeting | 16:04 | |
| *** macz_ has joined #openstack-meeting | 16:05 | |
| *** ricolin_ has joined #openstack-meeting | 16:22 | |
| *** Trevor_V has joined #openstack-meeting | 16:23 | |
| *** ociuhandu has quit IRC | 16:25 | |
| *** mlavalle has joined #openstack-meeting | 16:26 | |
| *** TrevorV has quit IRC | 16:26 | |
| *** yamamoto has joined #openstack-meeting | 16:30 | |
| *** yamamoto has quit IRC | 16:35 | |
| *** ociuhandu has joined #openstack-meeting | 16:36 | |
| *** gyee has joined #openstack-meeting | 16:50 | |
| *** evrardjp has joined #openstack-meeting | 17:03 | |
| *** evrardjp_ has quit IRC | 17:06 | |
| *** rcernin_ has joined #openstack-meeting | 17:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 17:10 | |
| *** rcernin_ has quit IRC | 17:12 | |
| *** evrardjp_ has joined #openstack-meeting | 17:25 | |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| *** evrardjp has quit IRC | 17:28 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:38 | |
| *** e0ne has joined #openstack-meeting | 17:38 | |
| *** ociuhandu has quit IRC | 17:42 | |
| *** ociuhandu_ has quit IRC | 17:42 | |
| *** evrardjp_ has quit IRC | 17:48 | |
| *** evrardjp has joined #openstack-meeting | 17:51 | |
| *** ralonsoh has quit IRC | 17:58 | |
| *** ricolin_ has quit IRC | 18:15 | |
| *** evrardjp_ has joined #openstack-meeting | 18:19 | |
| *** evrardjp has quit IRC | 18:22 | |
| *** evrardjp has joined #openstack-meeting | 18:32 | |
| *** evrardjp_ has quit IRC | 18:35 | |
| *** njohnston has quit IRC | 18:36 | |
| *** vishalmanchanda has quit IRC | 18:40 | |
| *** e0ne has quit IRC | 18:45 | |
| *** jmasud has quit IRC | 18:58 | |
| *** njohnston has joined #openstack-meeting | 19:01 | |
| *** jmasud has joined #openstack-meeting | 19:02 | |
| *** rcernin_ has joined #openstack-meeting | 19:09 | |
| *** jmasud has quit IRC | 19:12 | |
| *** rcernin_ has quit IRC | 19:14 | |
| *** jmasud has joined #openstack-meeting | 19:17 | |
| *** rcernin_ has joined #openstack-meeting | 19:25 | |
| *** rcernin_ has quit IRC | 19:30 | |
| *** whoami-rajat has quit IRC | 19:35 | |
| *** jamesmcarthur has quit IRC | 19:39 | |
| *** e0ne has joined #openstack-meeting | 19:51 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:53 | |
| *** jamesmcarthur has quit IRC | 20:05 | |
| *** rcernin_ has joined #openstack-meeting | 20:12 | |
| *** rcernin_ has quit IRC | 20:23 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:35 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 20:41 | |
| *** armax_ has joined #openstack-meeting | 20:41 | |
| *** tosky has quit IRC | 20:41 | |
| *** armax has quit IRC | 20:42 | |
| *** raildo has quit IRC | 20:42 | |
| *** armax_ is now known as armax | 20:42 | |
| *** raildo_ has joined #openstack-meeting | 20:42 | |
| *** tosky has joined #openstack-meeting | 20:42 | |
| *** jamesmcarthur has quit IRC | 20:42 | |
| *** rcernin_ has joined #openstack-meeting | 20:48 | |
| *** jmasud has quit IRC | 20:58 | |
| timburke_ | #startmeeting swift | 21:00 |
|---|---|---|
| openstack | Meeting started Wed Jan 20 21:00:13 2021 UTC and is due to finish in 60 minutes. The chair is timburke_. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| *** jamesmcarthur_ has quit IRC | 21:00 | |
| timburke_ | who's here for the swift meeting? | 21:00 |
| seongsoocho | o/ | 21:00 |
| kota_ | hi | 21:00 |
| seongsoocho | hi | 21:00 |
| rledisez | o/ | 21:00 |
| mattoliverau | o/ | 21:00 |
| *** acoles has joined #openstack-meeting | 21:01 | |
| timburke_ | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke_ | first up | 21:01 |
| timburke_ | #topic reconciler pipelines | 21:02 |
| *** openstack changes topic to "reconciler pipelines (Meeting topic: swift)" | 21:02 | |
| timburke_ | #link https://bugs.launchpad.net/swift/+bug/1910804 | 21:02 |
| openstack | Launchpad bug 1910804 in OpenStack Object Storage (swift) "Encryption doesn't play well with processes that copy cleartext data while preserving timestamps" [Undecided,New] | 21:02 |
| *** zaitcev has joined #openstack-meeting | 21:02 | |
| acoles | o/ | 21:03 |
| timburke_ | i just wanted to give an update on how we (nvidia) are addressing it, and get input on what we (as a community) want to do | 21:03 |
| timburke_ | nvidia is addressing it as an operations issue -- we wrote bad pipelines before, and now we'll have hardcoded reconciler pipelines that more-closely match what's in upstream docs | 21:04 |
| timburke_ | of course, there's still the upstream patch i wrote to forcibly remove some middlewares from the reconciler pipeline | 21:04 |
| timburke_ | #link https://review.opendev.org/c/openstack/swift/+/770522 | 21:05 |
| timburke_ | ...but i'm not sure whether that's necessarily the direction we should be going | 21:05 |
| acoles | we should probably add some doc/comment in reconciler sample conf to emphasise what not to do in the pipeline (and why) | 21:07 |
| timburke_ | does anyone have opinions on whether there should be new code to try to prevent the situation, or if it's purely a documentation issue? | 21:07 |
| zaitcev | I'm bothered by the fact that middlewares are no longer true middlewares if the core code knows about their intrinsic characteristics and modifies the pileline on the basis of the middleware names. | 21:08 |
| zaitcev | Why are they middlewares, then? Might as well re-format them as modules. | 21:08 |
| acoles | I guess there are (at least) 3 possible actions: 1. add doc 2. check pipeline and log warnings 3. modify bad piplines | 21:09 |
| zaitcev | I'd just dump it on docs, or better yet etc/object-server.conf-sample should have a working pipeline | 21:10 |
| zaitcev | And write a comment "do not insert symlink, dlo, slo" here. | 21:10 |
| rledisez | I would also add a note in the changelog to emphase that operator must fix their configuration (not sure anyone read documentation diff before an upgrade) | 21:11 |
| acoles | zaitcev: it's container-reconciler.conf-sample | 21:11 |
| *** jamesmcarthur has joined #openstack-meeting | 21:11 | |
| acoles | rledisez: +1 | 21:12 |
| timburke_ | all right, that souds like a plan then. i'll work on getting a docs/sample conf patch together, and plan on calling it out in the next release's changelog. thanks! | 21:13 |
| timburke_ | #topic ssync and non-durable frags | 21:13 |
| *** openstack changes topic to "ssync and non-durable frags (Meeting topic: swift)" | 21:13 | |
| timburke_ | acoles, how's it going? | 21:13 |
| acoles | we're deploying a fix this week: https://review.opendev.org/c/openstack/swift/+/770047 | 21:14 |
| acoles | and hoping to observe some 'stuck' handoff partitions get sync'd and cleaned up | 21:14 |
| timburke_ | anything else you need o that for right now? reviews, i suppose? | 21:15 |
| acoles | (the problem being non-durable EC fragments preventing handoff partitions being deleted) | 21:15 |
| acoles | reviews always welcome | 21:15 |
| zaitcev | maybe you could comment on the review with the outcome | 21:15 |
| *** jamesmcarthur has quit IRC | 21:16 | |
| acoles | yes, and/or here next week | 21:16 |
| *** Trevor_V has quit IRC | 21:16 | |
| *** sean-k-mooney has quit IRC | 21:16 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:17 | |
| timburke_ | sounds good | 21:17 |
| timburke_ | #topic shard cleanup | 21:17 |
| *** openstack changes topic to "shard cleanup (Meeting topic: swift)" | 21:17 | |
| timburke_ | mattoliverau, clayg, acoles i think you were all pushing on this a bit over the last week | 21:18 |
| mattoliverau | We have 2 patches going in 2 different directions. | 21:18 |
| timburke_ | how's it going? | 21:18 |
| mattoliverau | https://review.opendev.org/c/openstack/swift/+/770529 and https://review.opendev.org/c/openstack/swift/+/771086 | 21:19 |
| mattoliverau | the problem is this only really happens because autosharding + shrinking doesn't exist | 21:19 |
| *** jamesmcarthur has quit IRC | 21:20 | |
| mattoliverau | so the first potential fix is patch one, that inserts a kind of poison pill, or rather the delete ts of the root when the root 404s | 21:20 |
| *** jamesmcarthur has joined #openstack-meeting | 21:20 | |
| mattoliverau | so they can get cleaned up | 21:20 |
| mattoliverau | but the second path, simply makes reclaiming a deleted root container not possible until it has no shards | 21:20 |
| mattoliverau | the second means we could have shards hanging around, but gets us closer to the indended solution | 21:21 |
| acoles | the second is a pre-requisite for most other solutions IMHO | 21:21 |
| mattoliverau | +1 | 21:21 |
| acoles | i.e. avoid orphaned shards, then figure out how to delete them so the root can be deleted | 21:21 |
| mattoliverau | So I've been playing with acoles' compact shrinking patches | 21:21 |
| acoles | but dealing with orphans will be assisted by still having the root around | 21:22 |
| acoles | mattoliverau: yeah. that's like the third piece of the story :) | 21:22 |
| mattoliverau | So an op can do something about collapsing the empty shards back into the root so they can be reclaimed.. this shrinking code also is in the sharder for when we get to autosharding so it'll happen automagically at some point | 21:22 |
| timburke_ | i think i like that part the most ;-) | 21:23 |
| acoles | https://review.opendev.org/c/openstack/swift/+/765623 -> gives us a tool to get rid of empty shards | 21:23 |
| acoles | but again, relies on the root still existing! | 21:23 |
| timburke_ | will that work against a db that's been marked deleted? have we tested that yet? | 21:23 |
| mattoliverau | So I have a feeling maybe option 2 is best because it gets us further down to correct path. | 21:23 |
| mattoliverau | yup | 21:24 |
| timburke_ | cool! | 21:24 |
| acoles | I'm almost there on https://review.opendev.org/c/openstack/swift/+/765623 but have one piece left to write | 21:24 |
| mattoliverau | well, I think.. that's what I've been testing in my SAIO | 21:24 |
| acoles | timburke_: good question, need to add that test | 21:24 |
| timburke_ | sounds like good progress. anything else you guys need input on there? | 21:25 |
| zaitcev | Apropos sharding, I'm going to enable it on Train. So, I'm going to propose a few backports. | 21:25 |
| acoles | oh, cool, so mattoliverau did you try than manually? | 21:25 |
| acoles | that* | 21:25 |
| zaitcev | I don't have +2 on stable branches, so I'll have to ask to approve. | 21:25 |
| acoles | timburke_: reviews always welcome :) | 21:26 |
| zaitcev | Sorry, I thought you moved on from the orphan shards discussion. | 21:26 |
| mattoliverau | acoles: the compact worked to get it down to 1, it's the final collapse I need to test ;) | 21:26 |
| timburke_ | zaitcev, cool! let us know how it goes. how big are the containers you're looking at, out of curiosity? | 21:26 |
| mattoliverau | so your last peice as you will :) | 21:26 |
| zaitcev | timburke_: it's the 300GB container that I mentioned. | 21:26 |
| acoles | mattoliverau: the final collapse won't work yet, give me another 24 hours, but did you try it on a deleted root? | 21:26 |
| timburke_ | ah, right! that does sound familiar | 21:26 |
| mattoliverau | yeah | 21:27 |
| acoles | super | 21:27 |
| zaitcev | timburke_: A national phone provider has tens of millions of phones archive their text messages once a day. And all of the objects have expiration tags, so the expirers delete tens of millions of objects each day. | 21:27 |
| zaitcev | And all of that in 1 container | 21:27 |
| zaitcev | I cannot understand who thought it was a good idea, and how it works at all. They started in Queens. | 21:27 |
| acoles | zaitcev: do you know if the object names are monotonically increasing? | 21:28 |
| acoles | as in new objects append to end of sorted listing | 21:28 |
| zaitcev | acoles: I think the are "username.timestamp.gz" so they should be increasing. | 21:28 |
| zaitcev | acoles: wait, nm. User 'a' has all of its archives ahead of user 'z'. | 21:29 |
| timburke_ | well, at least there will be *some* load-spreading across the shards | 21:29 |
| *** jamesmcarthur has quit IRC | 21:29 | |
| zaitcev | I need to look for sure. | 21:29 |
| mattoliverau | I wonder if adding a vaccum facility to containers would be useful for cases like that too. | 21:29 |
| zaitcev | Oh it would. Bu | 21:29 |
| mattoliverau | rebalacne replicaiton would be a pain | 21:29 |
| zaitcev | t by the time I got there, sqlite vacuum would 100% fail with a segfault on that container. Not even running out of RAM, just segfault. | 21:30 |
| mattoliverau | lol, well thats useful :P | 21:30 |
| acoles | I had this idea once that we could use the fresh db with new epoch trick to, well, get a fresh root db (without any change to shards), but that was just a wild thought | 21:31 |
| acoles | just copy what we care about to a fresh db and ditch the old | 21:31 |
| zaitcev | In this particular case I calculated the number of objects and it was something like only 17 bytes per object in the .db, maybe 170. I don't think vacuuming was going to help my case. | 21:32 |
| timburke_ | all right, one last topic for updates | 21:33 |
| timburke_ | #topic relinker | 21:33 |
| *** openstack changes topic to "relinker (Meeting topic: swift)" | 21:33 | |
| timburke_ | rledisez, thanks for reviewing https://review.opendev.org/c/openstack/swift/+/769855 ! i pushed up a new patchset to roll in the test changes you recommended | 21:33 |
| *** e0ne has quit IRC | 21:33 | |
| rledisez | timburke_: yeah, i reviewed it just before the meeting. code looks good, just a concern that should be easy to fix I think. seems almost good to go | 21:34 |
| timburke_ | thanks! | 21:35 |
| *** jmasud has joined #openstack-meeting | 21:35 | |
| timburke_ | i haven't gotten back to https://review.opendev.org/c/openstack/swift/+/769632 (better-handle unmounted disks) yet, but hope to this week. i'm not so sure about the patch as written, since it'll log the warning but not change the exit code (which seems like it's still a bit dangerous) | 21:35 |
| timburke_ | that's all i've got | 21:35 |
| timburke_ | #topic open discussion | 21:35 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:35 | |
| timburke_ | what else should we bring up this week? | 21:35 |
| zaitcev | I said my piece about sharding, albeit out of turn. | 21:37 |
| rledisez | concerning p 769632 I think the solution is in the way progress is provided to the operator (either recon or a swift-object-relinker status) that would tell "on this device that is in the ring, there is no state file, it's suspect" | 21:38 |
| zaitcev | Also, I'm still cracking my head about why Romain's patch to delete Queue fails. It now fails with connect() returning EBADF which is just absurt. | 21:38 |
| timburke_ | seems to still point to some wires getting crossed down in eventlet :-/ | 21:39 |
| zaitcev | Oh, yeah. I started looking at S3 v4 signatures not working. | 21:40 |
| timburke_ | rledisez, good thought -- i'll see what i can do with the pre-device hook | 21:40 |
| zaitcev | No progress but I'm amazed that nobody noticed. | 21:40 |
| zaitcev | All users just silently back down to v2, I guess. | 21:40 |
| timburke_ | zaitcev, is it something involving unsigned payloads, aybe? i know there was a regression i accidetally introduced a bit ago... lemme find the patches... | 21:41 |
| zaitcev | No, or, I don't know. In my case I only ever did GET and they work for some names but not the others. | 21:41 |
| zaitcev | Almost like even-length name is okay | 21:42 |
| zaitcev | I need to dig deeper though. It certainly reproduces for me, using s3cmd. I need to re-try with Boto too. | 21:42 |
| timburke_ | huh. any non-ascii in the names, maybe? | 21:42 |
| zaitcev | Nope. Just names like s3cmd ls s3://test-1235163301 | 21:43 |
| timburke_ | (regression i was thinking of was caused by https://review.opendev.org/c/openstack/swift/+/767644, fixed by https://review.opendev.org/c/openstack/swift/+/770004) | 21:43 |
| zaitcev | I'll test, thanks a lot. | 21:44 |
| acoles | speaking of s3, I noticed some pipeline checks were not actually being executed in s3api middleware: https://review.opendev.org/c/openstack/swift/+/771467 | 21:44 |
| zaitcev | it was on my list to review | 21:45 |
| acoles | thanks | 21:45 |
| timburke_ | all right, looks like we can end a bit early | 21:47 |
| timburke_ | thak you all for coming, and thank you for working on swift! | 21:48 |
| timburke_ | #endmeeting | 21:48 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:48 | |
| openstack | Meeting ended Wed Jan 20 21:48:11 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:48 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-20-21.00.html | 21:48 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-20-21.00.txt | 21:48 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-20-21.00.log.html | 21:48 |
| *** acoles has left #openstack-meeting | 21:48 | |
| *** zaitcev has left #openstack-meeting | 21:49 | |
| *** rcernin_ has quit IRC | 21:49 | |
| *** rcernin has joined #openstack-meeting | 21:50 | |
| *** jamesden_ has quit IRC | 21:51 | |
| *** jmasud has quit IRC | 21:51 | |
| *** jamesdenton has joined #openstack-meeting | 21:51 | |
| *** jmasud has joined #openstack-meeting | 22:03 | |
| *** yamamoto has joined #openstack-meeting | 22:19 | |
| *** armax has quit IRC | 22:24 | |
| *** yamamoto has quit IRC | 22:30 | |
| *** yamamoto has joined #openstack-meeting | 22:30 | |
| *** slaweq has quit IRC | 22:32 | |
| *** raildo_ has quit IRC | 23:09 | |
| *** lbragstad has quit IRC | 23:13 | |
| *** lbragstad_ has joined #openstack-meeting | 23:13 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:29 | |
| *** jamesmcarthur has quit IRC | 23:34 | |
| *** tosky has quit IRC | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!