| *** jamesmcarthur has quit IRC | 00:01 | |
| *** jamesmcarthur has joined #openstack-infra | 00:01 | |
| *** jamesmcarthur has quit IRC | 00:06 | |
| *** dhill_ has joined #openstack-infra | 00:30 | |
| *** jamesmcarthur has joined #openstack-infra | 00:32 | |
| *** jamesmcarthur has quit IRC | 00:38 | |
| *** gary_perkins has joined #openstack-infra | 00:43 | |
| *** jamesmcarthur has joined #openstack-infra | 00:50 | |
| *** armax has joined #openstack-infra | 01:02 | |
| *** gagehugo has quit IRC | 01:13 | |
| *** armax has quit IRC | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| *** jamesmcarthur has joined #openstack-infra | 01:23 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #openstack-infra | 01:24 | |
| *** Adri2000 has quit IRC | 01:32 | |
| *** Adri2000 has joined #openstack-infra | 01:40 | |
| *** irdr has quit IRC | 01:45 | |
| *** irdr has joined #openstack-infra | 01:46 | |
| *** markvoelker has quit IRC | 01:50 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:30 | |
| *** bobh has joined #openstack-infra | 02:32 | |
| *** _alastor_ has joined #openstack-infra | 02:40 | |
| *** bhavikdbavishi has quit IRC | 02:41 | |
| *** wolverineav has joined #openstack-infra | 02:48 | |
| corvus | i wiped the drive for osd.1 and re-ran the rook operater; it created osd.3 on the drive with the new option, and the cluster is recovering. however, the recovery rate is 461 B/s, or 7 objects/s. at the current rate, it will take 132 days to recover. and then we'd need to do the other two. | 03:00 |
|---|---|---|
| *** wolverineav has quit IRC | 03:00 | |
| corvus | so i think we need to look into why the recovery data throughput is so slow. | 03:01 |
| *** bhavikdbavishi has joined #openstack-infra | 03:04 | |
| *** wolverineav has joined #openstack-infra | 03:27 | |
| *** bobh has quit IRC | 03:41 | |
| *** _alastor_ has quit IRC | 03:43 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** jamesmcarthur has joined #openstack-infra | 04:27 | |
| *** bhavikdbavishi has quit IRC | 04:30 | |
| *** wolverineav has quit IRC | 04:47 | |
| *** wolverineav has joined #openstack-infra | 04:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:51 | |
| *** wolverineav has quit IRC | 04:53 | |
| *** wolverineav has joined #openstack-infra | 04:54 | |
| *** jamesmcarthur has quit IRC | 04:57 | |
| *** wolverineav has quit IRC | 05:03 | |
| *** bhavikdbavishi has quit IRC | 05:16 | |
| *** wolverineav has joined #openstack-infra | 05:32 | |
| *** dhill_ has quit IRC | 06:09 | |
| *** wolverineav has quit IRC | 06:09 | |
| *** wolverineav has joined #openstack-infra | 06:42 | |
| *** bhavikdbavishi has joined #openstack-infra | 06:56 | |
| *** jamesmcarthur has joined #openstack-infra | 06:58 | |
| *** jamesmcarthur has quit IRC | 07:03 | |
| *** wolverineav has quit IRC | 07:24 | |
| *** whoami-rajat has joined #openstack-infra | 07:32 | |
| *** spa-87 has joined #openstack-infra | 07:50 | |
| *** gagehugo has joined #openstack-infra | 07:53 | |
| *** markvoelker has joined #openstack-infra | 07:56 | |
| *** wolverineav has joined #openstack-infra | 07:59 | |
| *** wolverineav has quit IRC | 08:07 | |
| *** spa-87 has quit IRC | 08:08 | |
| *** wolverineav has joined #openstack-infra | 08:09 | |
| *** jtomasek has joined #openstack-infra | 08:12 | |
| *** wolverineav has quit IRC | 08:14 | |
| *** wolverineav has joined #openstack-infra | 08:22 | |
| *** slaweq has joined #openstack-infra | 08:23 | |
| *** jtomasek has quit IRC | 08:25 | |
| *** wolverineav has quit IRC | 08:27 | |
| *** bhavikdbavishi has quit IRC | 08:37 | |
| *** slaweq has quit IRC | 08:42 | |
| *** yboaron_ has joined #openstack-infra | 08:50 | |
| *** slaweq has joined #openstack-infra | 08:58 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add spec for scale out scheduler https://review.openstack.org/621479 | 09:32 |
| *** slaweq has quit IRC | 09:35 | |
| *** wolverineav has joined #openstack-infra | 09:48 | |
| *** wolverineav has quit IRC | 09:56 | |
| *** xek has joined #openstack-infra | 10:31 | |
| *** xek has quit IRC | 10:36 | |
| *** pbourke has quit IRC | 10:43 | |
| *** egonzalez has quit IRC | 10:43 | |
| *** xek has joined #openstack-infra | 10:43 | |
| *** pbourke has joined #openstack-infra | 10:45 | |
| *** tosky has joined #openstack-infra | 10:52 | |
| *** e0ne has joined #openstack-infra | 11:09 | |
| *** tosky has quit IRC | 11:16 | |
| *** spa-87 has joined #openstack-infra | 11:16 | |
| *** egonzalez has joined #openstack-infra | 11:20 | |
| *** wolverineav has joined #openstack-infra | 11:39 | |
| *** wolverineav has quit IRC | 11:45 | |
| *** whoami-rajat has quit IRC | 11:50 | |
| *** dkehn has quit IRC | 12:08 | |
| *** e0ne has quit IRC | 12:10 | |
| *** e0ne has joined #openstack-infra | 12:12 | |
| *** kei-ichi has quit IRC | 12:30 | |
| *** e0ne has quit IRC | 12:47 | |
| *** e0ne has joined #openstack-infra | 12:51 | |
| *** e0ne has quit IRC | 13:18 | |
| *** wolverineav has joined #openstack-infra | 13:26 | |
| *** e0ne has joined #openstack-infra | 13:28 | |
| *** tosky has joined #openstack-infra | 13:29 | |
| *** e0ne has quit IRC | 13:30 | |
| *** wolverineav has quit IRC | 13:32 | |
| *** wolverineav has joined #openstack-infra | 13:53 | |
| *** wolverineav has quit IRC | 13:58 | |
| *** xek_ has joined #openstack-infra | 13:59 | |
| *** xek has quit IRC | 14:02 | |
| *** wolverineav has joined #openstack-infra | 14:07 | |
| *** wolverineav has quit IRC | 14:12 | |
| *** markvoelker has quit IRC | 14:25 | |
| *** spa-87 has quit IRC | 14:45 | |
| *** dkehn has joined #openstack-infra | 15:01 | |
| *** bobh has joined #openstack-infra | 15:49 | |
| *** ccamacho has quit IRC | 15:50 | |
| *** dklyle has quit IRC | 15:51 | |
| *** bobh has quit IRC | 15:53 | |
| *** bobh_ has joined #openstack-infra | 15:53 | |
| *** wolverineav has joined #openstack-infra | 16:07 | |
| *** wolverineav has quit IRC | 16:14 | |
| mnaser | corvus: if this is a ceph cluster there are a few settings you can tweak to speed up recovery | 16:36 |
| mnaser | there's a set limit of pg recovery/backfill operations per osd limit | 16:36 |
| corvus | mnaser: i ran "ceph tell 'osd.*' injectargs '--osd_max_backfills 10'" and it sped up a bit (picked 10 at random) | 16:38 |
| mnaser | corvus: up, but also theere is osd_max_recovery or something i think as welel | 16:38 |
| mnaser | ah but this wont be a recovery | 16:38 |
| mnaser | it's a backfill because it was recreated | 16:39 |
| *** _alastor_ has joined #openstack-infra | 16:39 | |
| corvus | mnaser: i ran "ceph tell 'osd.*' injectargs '--osd-recovery-max-active 10'" but it said "(not observed, change may require restart)" | 16:39 |
| mnaser | corvus: ceph does that, but most of the time it actually applies the change | 16:39 |
| mnaser | how many pgs are presently in backfill vs backfill_wait | 16:40 |
| corvus | mnaser: http://paste.openstack.org/show/738094/ | 16:40 |
| corvus | the recovery rate has increased (i've seen it get up to 10kB/s) but i'd still expect something a lot higher | 16:41 |
| corvus | sorry "KiB" to be precise :) | 16:41 |
| mnaser | ok so because there is only 3 osds, when you set the backfill value to 10, it's only doing 10 at a time because its probably just filling up that singular osd | 16:42 |
| mnaser | (the newly added one) | 16:42 |
| mnaser | because no data is being rebalanced anyways so | 16:42 |
| mnaser | so i'd ceph tell osd.4 '--osd_max_backfills 24' | 16:42 |
| mnaser | also, the KiB/s might be misleading because recovery happens on a per object basis and because im assumign this is hosting git, its a lot of small objects | 16:43 |
| corvus | still seems to be in the 8-12 objects/s range | 16:45 |
| corvus | (after changing to 24) | 16:45 |
| *** _alastor_ has quit IRC | 16:47 | |
| *** bobh_ has quit IRC | 16:53 | |
| *** bobh has joined #openstack-infra | 17:12 | |
| *** bobh has quit IRC | 17:16 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:16 | |
| *** bobh has joined #openstack-infra | 17:28 | |
| *** jamesmcarthur has joined #openstack-infra | 17:28 | |
| *** jamesmcarthur has quit IRC | 17:34 | |
| *** markvoelker has joined #openstack-infra | 17:43 | |
| *** markvoelker has quit IRC | 17:47 | |
| *** xek_ has quit IRC | 18:08 | |
| *** bobh has quit IRC | 18:13 | |
| *** tosky has quit IRC | 18:36 | |
| *** spa-87 has joined #openstack-infra | 18:38 | |
| *** e0ne has joined #openstack-infra | 18:56 | |
| *** e0ne has quit IRC | 18:57 | |
| *** hongbin has joined #openstack-infra | 19:12 | |
| *** whoami-rajat has joined #openstack-infra | 19:13 | |
| *** _alastor_ has joined #openstack-infra | 19:16 | |
| *** slaweq has joined #openstack-infra | 19:29 | |
| *** jamesmcarthur has joined #openstack-infra | 19:34 | |
| *** slaweq has quit IRC | 19:37 | |
| *** jamesmcarthur has quit IRC | 19:39 | |
| *** yboaron_ has quit IRC | 19:42 | |
| *** markvoelker has joined #openstack-infra | 19:43 | |
| *** _alastor_ has quit IRC | 19:44 | |
| *** bhavikdbavishi has quit IRC | 19:56 | |
| *** armax has joined #openstack-infra | 19:56 | |
| *** e0ne has joined #openstack-infra | 20:06 | |
| *** _alastor_ has joined #openstack-infra | 20:07 | |
| *** e0ne has quit IRC | 20:13 | |
| *** slaweq has joined #openstack-infra | 20:17 | |
| *** _alastor_ has quit IRC | 20:21 | |
| *** spa-87 has quit IRC | 20:23 | |
| *** armax has quit IRC | 20:27 | |
| *** slaweq has quit IRC | 20:31 | |
| *** dayou has quit IRC | 20:41 | |
| *** dayou has joined #openstack-infra | 20:42 | |
| *** slaweq has joined #openstack-infra | 20:44 | |
| *** jamesmcarthur has joined #openstack-infra | 20:45 | |
| *** e0ne has joined #openstack-infra | 20:51 | |
| *** e0ne has quit IRC | 20:51 | |
| *** pcrews has joined #openstack-infra | 21:03 | |
| *** jamesmcarthur has quit IRC | 21:07 | |
| *** _alastor_ has joined #openstack-infra | 21:40 | |
| *** jamesmcarthur has joined #openstack-infra | 22:33 | |
| *** wolverineav has joined #openstack-infra | 22:36 | |
| *** jamesmcarthur has quit IRC | 22:37 | |
| *** whoami-rajat has quit IRC | 22:42 | |
| *** hongbin has quit IRC | 22:57 | |
| *** wolverineav has quit IRC | 22:58 | |
| *** jamesmcarthur has joined #openstack-infra | 23:02 | |
| *** sshnaidm_ has joined #openstack-infra | 23:04 | |
| *** sshnaidm|off has quit IRC | 23:04 | |
| *** wolverineav has joined #openstack-infra | 23:05 | |
| *** wolverineav has quit IRC | 23:09 | |
| *** jamesmcarthur has quit IRC | 23:10 | |
| *** wolverineav has joined #openstack-infra | 23:13 | |
| *** wolverineav has quit IRC | 23:22 | |
| *** wolverineav has joined #openstack-infra | 23:42 | |
| *** wolverineav has quit IRC | 23:47 | |
| *** _alastor_ has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!