| *** lbragstad has joined #openstack-infra | 00:07 | |
| *** jkilpatr has joined #openstack-infra | 00:10 | |
| *** xarses has joined #openstack-infra | 00:23 | |
| *** tosky has quit IRC | 00:26 | |
| *** markvoelker has joined #openstack-infra | 00:36 | |
| *** spzala has joined #openstack-infra | 00:49 | |
| *** Apoorva has joined #openstack-infra | 00:53 | |
| *** spzala has quit IRC | 00:54 | |
| *** edmondsw has joined #openstack-infra | 00:59 | |
| *** edmondsw has quit IRC | 01:03 | |
| *** r-daneel has quit IRC | 01:03 | |
| *** hongbin_ has quit IRC | 01:04 | |
| *** hongbin has joined #openstack-infra | 01:05 | |
| *** hongbin has quit IRC | 01:05 | |
| *** hongbin has joined #openstack-infra | 01:06 | |
| *** Apoorva has quit IRC | 01:06 | |
| *** hongbin has quit IRC | 01:07 | |
| *** hongbin has joined #openstack-infra | 01:07 | |
| *** kmalloc has joined #openstack-infra | 01:07 | |
| *** hongbin has quit IRC | 01:07 | |
| *** hongbin has joined #openstack-infra | 01:08 | |
| mnaser | another spamwave | 01:08 |
|---|---|---|
| *** markvoelker has quit IRC | 01:10 | |
| *** hongbin has quit IRC | 01:13 | |
| *** sbra has quit IRC | 01:31 | |
| *** sbra has joined #openstack-infra | 01:32 | |
| *** kiennt26 has joined #openstack-infra | 01:35 | |
| *** kiennt26 has quit IRC | 01:35 | |
| *** kiennt26 has joined #openstack-infra | 01:36 | |
| *** r-daneel has joined #openstack-infra | 01:58 | |
| *** Goneri has joined #openstack-infra | 01:59 | |
| *** xinliang has joined #openstack-infra | 02:02 | |
| *** markvoelker has joined #openstack-infra | 02:02 | |
| *** r-daneel has quit IRC | 02:05 | |
| *** Goneri has quit IRC | 02:16 | |
| *** armax has joined #openstack-infra | 02:41 | |
| *** jascott1 has quit IRC | 02:43 | |
| *** armax has quit IRC | 02:44 | |
| *** armax has joined #openstack-infra | 02:45 | |
| *** armax has quit IRC | 02:45 | |
| *** armax has joined #openstack-infra | 02:45 | |
| *** armax has quit IRC | 02:46 | |
| *** armax has joined #openstack-infra | 02:46 | |
| *** armax has quit IRC | 02:46 | |
| *** armax has joined #openstack-infra | 02:47 | |
| *** armax has quit IRC | 02:47 | |
| *** armax has joined #openstack-infra | 02:48 | |
| *** armax has quit IRC | 02:48 | |
| *** olaph has joined #openstack-infra | 03:00 | |
| *** spzala has joined #openstack-infra | 03:01 | |
| *** spzala has quit IRC | 03:06 | |
| *** yamamoto has joined #openstack-infra | 03:08 | |
| *** yamamoto has quit IRC | 03:12 | |
| *** yamamoto has joined #openstack-infra | 03:12 | |
| *** yamamoto has quit IRC | 03:17 | |
| *** bobh has joined #openstack-infra | 03:17 | |
| *** yamamoto has joined #openstack-infra | 03:17 | |
| *** olaph1 has joined #openstack-infra | 03:18 | |
| *** olaph has quit IRC | 03:18 | |
| *** claudiub has quit IRC | 03:18 | |
| *** yamamoto has quit IRC | 03:20 | |
| *** ijw has quit IRC | 03:23 | |
| *** kmalloc has quit IRC | 03:36 | |
| *** jascott1 has joined #openstack-infra | 03:44 | |
| *** kiennt26 has quit IRC | 03:44 | |
| *** ramishra has joined #openstack-infra | 04:17 | |
| *** yamamoto has joined #openstack-infra | 04:21 | |
| *** ihrachys has quit IRC | 04:24 | |
| *** bobh has quit IRC | 04:29 | |
| *** yamamoto has quit IRC | 04:29 | |
| *** yamamoto has joined #openstack-infra | 04:30 | |
| *** rosmaita has quit IRC | 04:33 | |
| *** rosmaita has joined #openstack-infra | 04:34 | |
| *** edmondsw has joined #openstack-infra | 04:35 | |
| *** yamamoto has quit IRC | 04:38 | |
| *** edmondsw has quit IRC | 04:39 | |
| *** armax has joined #openstack-infra | 04:41 | |
| *** spzala has joined #openstack-infra | 05:00 | |
| *** spzala has quit IRC | 05:01 | |
| *** markvoelker has quit IRC | 05:03 | |
| *** kiennt26 has joined #openstack-infra | 05:06 | |
| *** yamamoto has joined #openstack-infra | 05:08 | |
| *** yamamoto has quit IRC | 05:09 | |
| *** yamamoto has joined #openstack-infra | 05:09 | |
| *** yamamoto has quit IRC | 05:11 | |
| *** yamamoto has joined #openstack-infra | 05:12 | |
| *** yamamoto has quit IRC | 05:17 | |
| *** olaph has joined #openstack-infra | 05:23 | |
| *** ijw has joined #openstack-infra | 05:24 | |
| *** olaph1 has quit IRC | 05:25 | |
| *** ijw has quit IRC | 05:30 | |
| *** olaph1 has joined #openstack-infra | 05:38 | |
| *** olaph has quit IRC | 05:39 | |
| *** kiennt26 has quit IRC | 05:53 | |
| *** armax has quit IRC | 06:01 | |
| *** olaph has joined #openstack-infra | 06:03 | |
| *** olaph1 has quit IRC | 06:05 | |
| *** armaan has joined #openstack-infra | 06:14 | |
| *** yamamoto has joined #openstack-infra | 06:17 | |
| *** edmondsw has joined #openstack-infra | 06:23 | |
| *** yamamoto has quit IRC | 06:25 | |
| *** dsariel has joined #openstack-infra | 06:26 | |
| *** edmondsw has quit IRC | 06:28 | |
| *** dbecker has quit IRC | 06:31 | |
| *** dbecker has joined #openstack-infra | 06:36 | |
| *** larainema has quit IRC | 06:40 | |
| *** sbra has quit IRC | 06:41 | |
| *** sbra has joined #openstack-infra | 06:41 | |
| *** vsaienk0 has joined #openstack-infra | 06:46 | |
| *** vsaienk0 has quit IRC | 06:56 | |
| *** vsaienk0 has joined #openstack-infra | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:03 | |
| *** olaph1 has joined #openstack-infra | 07:10 | |
| *** vsaienk0 has quit IRC | 07:11 | |
| *** olaph has quit IRC | 07:12 | |
| *** r-daneel has joined #openstack-infra | 07:16 | |
| *** cshastri has joined #openstack-infra | 07:16 | |
| *** olaph has joined #openstack-infra | 07:17 | |
| *** olaph1 has quit IRC | 07:18 | |
| *** ijw has joined #openstack-infra | 07:26 | |
| *** psachin has joined #openstack-infra | 07:28 | |
| *** psachin has quit IRC | 07:31 | |
| *** psachin has joined #openstack-infra | 07:33 | |
| *** markvoelker has quit IRC | 07:38 | |
| *** psachin has quit IRC | 07:41 | |
| *** edmondsw has joined #openstack-infra | 08:11 | |
| *** edmondsw has quit IRC | 08:16 | |
| *** frickler has quit IRC | 08:23 | |
| *** ijw has quit IRC | 08:36 | |
| *** frickler has joined #openstack-infra | 08:37 | |
| *** lathiat has quit IRC | 08:52 | |
| *** lathiat has joined #openstack-infra | 08:57 | |
| *** sshnaidm|afk has quit IRC | 08:58 | |
| *** e0ne has joined #openstack-infra | 09:16 | |
| *** ykarel|away has joined #openstack-infra | 09:20 | |
| *** markvoelker has joined #openstack-infra | 09:35 | |
| *** e0ne has quit IRC | 09:35 | |
| *** ijw has joined #openstack-infra | 09:37 | |
| *** olaph1 has joined #openstack-infra | 09:37 | |
| *** olaph has quit IRC | 09:38 | |
| *** ijw has quit IRC | 09:42 | |
| *** bkero-lounge has joined #openstack-infra | 09:58 | |
| *** edmondsw has joined #openstack-infra | 09:59 | |
| *** edmondsw has quit IRC | 10:03 | |
| *** markvoelker has quit IRC | 10:09 | |
| *** slaweq has joined #openstack-infra | 10:10 | |
| *** pbourke has quit IRC | 10:20 | |
| *** pbourke has joined #openstack-infra | 10:21 | |
| *** olaph has joined #openstack-infra | 10:23 | |
| *** olaph1 has quit IRC | 10:24 | |
| *** olaph1 has joined #openstack-infra | 10:28 | |
| *** olaph has quit IRC | 10:30 | |
| *** pcichy has quit IRC | 10:33 | |
| *** pcichy has joined #openstack-infra | 10:33 | |
| *** adriant has quit IRC | 10:39 | |
| *** adriant has joined #openstack-infra | 10:41 | |
| *** adriant has quit IRC | 10:43 | |
| *** slaweq has quit IRC | 10:44 | |
| *** adriant has joined #openstack-infra | 10:45 | |
| *** claudiub has joined #openstack-infra | 10:48 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** masber has quit IRC | 11:10 | |
| *** masber has joined #openstack-infra | 11:10 | |
| *** sshnaidm|afk has joined #openstack-infra | 11:13 | |
| *** yamamoto has quit IRC | 11:20 | |
| *** masber has quit IRC | 11:35 | |
| *** ijw has joined #openstack-infra | 11:37 | |
| *** olaph has joined #openstack-infra | 11:38 | |
| *** masuberu has joined #openstack-infra | 11:38 | |
| *** olaph1 has quit IRC | 11:39 | |
| *** edmondsw has joined #openstack-infra | 11:47 | |
| *** edmondsw has quit IRC | 11:52 | |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** olaph1 has joined #openstack-infra | 12:04 | |
| *** olaph has quit IRC | 12:05 | |
| *** markvoelker has joined #openstack-infra | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:07 | |
| *** yamamoto has quit IRC | 12:12 | |
| *** ijw has quit IRC | 12:22 | |
| *** slaweq has joined #openstack-infra | 12:23 | |
| *** dsariel has quit IRC | 12:29 | |
| *** dsariel has joined #openstack-infra | 12:31 | |
| *** markvoelker has quit IRC | 12:39 | |
| *** ramishra has quit IRC | 12:44 | |
| *** e0ne has joined #openstack-infra | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:57 | |
| *** dhill__ has quit IRC | 13:15 | |
| *** sshnaidm|afk has quit IRC | 13:18 | |
| *** e0ne has quit IRC | 13:19 | |
| *** tosky has joined #openstack-infra | 13:19 | |
| *** nicolasbock has joined #openstack-infra | 13:22 | |
| *** edmondsw has joined #openstack-infra | 13:35 | |
| *** e0ne has joined #openstack-infra | 13:36 | |
| *** dsariel has quit IRC | 13:36 | |
| *** e0ne has quit IRC | 13:37 | |
| *** edmondsw has quit IRC | 13:40 | |
| *** sshnaidm|afk has joined #openstack-infra | 13:48 | |
| *** ykarel|away has quit IRC | 13:54 | |
| *** psachin has joined #openstack-infra | 14:00 | |
| *** armaan has quit IRC | 14:00 | |
| *** armaan has joined #openstack-infra | 14:00 | |
| *** dsariel has joined #openstack-infra | 14:05 | |
| *** dhill_ has joined #openstack-infra | 14:22 | |
| *** ijw has joined #openstack-infra | 14:22 | |
| *** ijw has quit IRC | 14:27 | |
| *** dsariel has quit IRC | 14:30 | |
| *** markvoelker has joined #openstack-infra | 14:36 | |
| *** katkapilatova1 has joined #openstack-infra | 14:42 | |
| *** jascott1 has quit IRC | 14:57 | |
| *** markvoelker has quit IRC | 15:10 | |
| *** dsariel has joined #openstack-infra | 15:13 | |
| *** caphrim007_ has joined #openstack-infra | 15:18 | |
| *** pcichy has quit IRC | 15:19 | |
| *** pcichy has joined #openstack-infra | 15:19 | |
| *** freerunner has quit IRC | 15:19 | |
| *** gtmanfred has quit IRC | 15:20 | |
| *** tobiash has quit IRC | 15:20 | |
| *** caphrim007 has quit IRC | 15:21 | |
| *** freerunner has joined #openstack-infra | 15:21 | |
| *** gtmanfred has joined #openstack-infra | 15:22 | |
| *** tobiash has joined #openstack-infra | 15:22 | |
| *** swest has quit IRC | 15:23 | |
| *** swest has joined #openstack-infra | 15:24 | |
| *** dsariel has quit IRC | 15:34 | |
| *** dsariel has joined #openstack-infra | 15:43 | |
| *** armaan has quit IRC | 15:55 | |
| *** armaan has joined #openstack-infra | 15:55 | |
| *** psachin has quit IRC | 15:56 | |
| *** psachin has joined #openstack-infra | 15:58 | |
| *** armaan has quit IRC | 16:02 | |
| *** markvoelker has joined #openstack-infra | 16:07 | |
| *** psachin has quit IRC | 16:11 | |
| *** Goneri has joined #openstack-infra | 16:14 | |
| *** ociuhandu_ has joined #openstack-infra | 16:20 | |
| *** sbra has quit IRC | 16:20 | |
| *** sbra has joined #openstack-infra | 16:21 | |
| *** dsariel has quit IRC | 16:23 | |
| *** ijw has joined #openstack-infra | 16:23 | |
| *** dsariel has joined #openstack-infra | 16:29 | |
| *** dsariel has quit IRC | 16:34 | |
| *** psachin has joined #openstack-infra | 16:34 | |
| *** dsariel has joined #openstack-infra | 16:35 | |
| *** Goneri has quit IRC | 16:39 | |
| *** markvoelker has quit IRC | 16:40 | |
| *** ijw has quit IRC | 16:41 | |
| *** psachin has quit IRC | 16:43 | |
| fungi | corvus: i'm just starting to diagnose now, but rndc zonestatus says the signed serial is older than the actual serial: http://paste.openstack.org/show/644245/ | 16:44 |
| fungi | and syslog definitely shows the slaves doing an axfr of that older serial | 16:46 |
| fungi | also named on the master logs that it's loading the newer unsigned serial and the older signed serial, so that seems consistent | 16:47 |
| *** bobh has joined #openstack-infra | 16:48 | |
| *** psachin has joined #openstack-infra | 16:56 | |
| fungi | digging through the puppet modules, it looks like we actually do a service bind9 restart each time a zonefile is modified | 17:02 |
| fungi | i wonder if that's partly to blame, though if it is i'm not entirely sure how (service stop races the signature update maybe?) | 17:04 |
| fungi | i'd like to stop puppet for adns1, manually bump the serial on the zone, and see what happens (then catch the git repo up with that serial bump) | 17:05 |
| fungi | but first i have some chores i need to go do around here. will get back to this in a bit | 17:05 |
| corvus | fungi: oh, interesting theory. that sounds like a good plan. | 17:07 |
| *** cshastri has quit IRC | 17:08 | |
| *** dsariel has quit IRC | 17:08 | |
| *** edmondsw has joined #openstack-infra | 17:11 | |
| *** edmondsw has quit IRC | 17:16 | |
| *** psachin has quit IRC | 17:25 | |
| prometheanfire | did gate restart between the 12th and now? https://review.openstack.org/532874 | 17:32 |
| *** slaweq_ has joined #openstack-infra | 17:39 | |
| *** ijw has joined #openstack-infra | 17:42 | |
| *** ijw has quit IRC | 17:47 | |
| *** psachin has joined #openstack-infra | 17:53 | |
| *** slaweq_ has quit IRC | 17:59 | |
| *** nicolasbock has quit IRC | 18:08 | |
| *** nicolasbock has joined #openstack-infra | 18:09 | |
| *** psachin has quit IRC | 18:13 | |
| *** sshnaidm|afk is now known as sshnaidm | 18:27 | |
| *** nicolasbock has quit IRC | 18:28 | |
| *** bobh has quit IRC | 18:29 | |
| *** markvoelker has joined #openstack-infra | 18:37 | |
| *** ijw has joined #openstack-infra | 18:40 | |
| *** olaph1 is now known as olaph | 18:41 | |
| *** bobh has joined #openstack-infra | 18:41 | |
| *** yamamoto has quit IRC | 18:42 | |
| AJaeger | prometheanfire: best check https://wiki.openstack.org/wiki/Infrastructure_Status | 18:54 |
| *** slaweq has quit IRC | 18:54 | |
| AJaeger | prometheanfire: so, yes it did - and a recheck is needed on that one | 18:55 |
| prometheanfire | k | 18:55 |
| *** edmondsw has joined #openstack-infra | 19:00 | |
| *** dbecker has quit IRC | 19:00 | |
| *** sree has joined #openstack-infra | 19:01 | |
| *** edmondsw has quit IRC | 19:04 | |
| *** panda has quit IRC | 19:05 | |
| *** sree has quit IRC | 19:06 | |
| *** panda has joined #openstack-infra | 19:06 | |
| *** sree has joined #openstack-infra | 19:08 | |
| *** markvoelker has quit IRC | 19:11 | |
| *** sree has quit IRC | 19:13 | |
| *** ijw has quit IRC | 19:15 | |
| *** sree has joined #openstack-infra | 19:16 | |
| *** markvoelker has joined #openstack-infra | 19:19 | |
| *** dbecker has joined #openstack-infra | 19:19 | |
| *** sree has quit IRC | 19:21 | |
| *** slaweq has joined #openstack-infra | 19:21 | |
| *** e0ne has joined #openstack-infra | 19:23 | |
| *** slaweq has quit IRC | 19:26 | |
| *** bobh has quit IRC | 19:41 | |
| *** yamamoto has joined #openstack-infra | 19:42 | |
| *** jamesmcarthur has joined #openstack-infra | 19:47 | |
| fungi | corvus: not _conclusive_ but here's the zonestatus after editing /var/lib/bind/zones/zuul-ci.org/zone.db to increase the serial to 1515959169 and then running `sudo rndc reload zuul-ci.org`: http://paste.openstack.org/show/644247/ | 19:48 |
| fungi | i think we probably need to replace the "notify => Service[$::dns::namedservicename]," with an exec resource which does an rndc reload? | 19:49 |
| *** jamesmca_ has joined #openstack-infra | 19:49 | |
| fungi | in the /var/lib/bind/zones/${name} file resource definition | 19:49 |
| *** yamamoto has quit IRC | 19:49 | |
| *** bobh has joined #openstack-infra | 19:50 | |
| * pabelanger subscribes to new MLs | 19:51 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zone-zuul-ci.org master: Update zone serial after manual testing https://review.openstack.org/533429 | 19:51 |
| fungi | corvus: ^ | 19:51 |
| fungi | infra-root: ^ i'm self-approving that so i can reenable puppet on adns1.o.o | 19:52 |
| pabelanger | fungi: wfm | 19:52 |
| *** jamesmcarthur has quit IRC | 19:53 | |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Update zone serial after manual testing https://review.openstack.org/533429 | 19:53 |
| *** olaph has quit IRC | 19:53 | |
| *** olaph has joined #openstack-infra | 19:53 | |
| *** bobh has quit IRC | 19:54 | |
| fungi | okay, puppet reenabled for adns1 again | 19:56 |
| *** sree has joined #openstack-infra | 19:56 | |
| fungi | i'll see if i can get a few minutes to work on the puppet fix in a bit | 19:56 |
| *** bobh has joined #openstack-infra | 19:59 | |
| *** sree has quit IRC | 20:01 | |
| *** bobh has quit IRC | 20:04 | |
| *** bobh has joined #openstack-infra | 20:08 | |
| *** bobh has quit IRC | 20:10 | |
| *** bobh has joined #openstack-infra | 20:10 | |
| *** sree has joined #openstack-infra | 20:15 | |
| *** sree has quit IRC | 20:19 | |
| *** sree has joined #openstack-infra | 20:20 | |
| *** sree has quit IRC | 20:25 | |
| ssbarnea | is normal to see POST_FAILURE ? Got this a couple of minutes ago on https://review.openstack.org/#/c/533430/ --- did recheck, got same error. | 20:25 |
| *** dsariel has joined #openstack-infra | 20:27 | |
| prometheanfire | I've been getting a bunch of them | 20:28 |
| prometheanfire | https://review.openstack.org/532874 | 20:28 |
| clarkb | there is or was an issue with the tox jobs where proper failures result in post failure | 20:30 |
| clarkb | chwck ara to see if the run.yaml failes | 20:30 |
| *** olaph1 has joined #openstack-infra | 20:31 | |
| *** sree has joined #openstack-infra | 20:33 | |
| *** olaph has quit IRC | 20:33 | |
| *** jamesmca_ has quit IRC | 20:34 | |
| *** pcichy has quit IRC | 20:37 | |
| *** sree has quit IRC | 20:38 | |
| *** jamesmcarthur has joined #openstack-infra | 20:42 | |
| ssbarnea | somehow i see only some finger:// urls which are useless to me. i had the impression that this protocol was deprecated long time ago. | 20:45 |
| *** slaweq has joined #openstack-infra | 20:45 | |
| *** edmondsw has joined #openstack-infra | 20:48 | |
| *** slaweq has quit IRC | 20:49 | |
| *** edmondsw has quit IRC | 20:52 | |
| *** jamesmcarthur has quit IRC | 20:56 | |
| *** olaph has joined #openstack-infra | 21:13 | |
| *** Goneri has joined #openstack-infra | 21:14 | |
| *** olaph1 has quit IRC | 21:15 | |
| *** ijw has joined #openstack-infra | 21:15 | |
| *** armaan has joined #openstack-infra | 21:27 | |
| *** rhallisey has quit IRC | 21:29 | |
| *** bobh has quit IRC | 21:30 | |
| *** aeng has joined #openstack-infra | 21:34 | |
| *** threestrands has joined #openstack-infra | 21:34 | |
| *** ijw has quit IRC | 21:34 | |
| *** Goneri has quit IRC | 21:35 | |
| *** Goneri has joined #openstack-infra | 21:37 | |
| *** bobh has joined #openstack-infra | 21:41 | |
| *** bobh has quit IRC | 21:45 | |
| *** ralonsoh has joined #openstack-infra | 21:49 | |
| *** ralonsoh has quit IRC | 21:49 | |
| *** jascott1 has joined #openstack-infra | 21:49 | |
| hjensas | I see POST_FAILURES - There is a read only filesystem it is trying to place the logs onto it seems: http://paste.openstack.org/show/644296/ | 21:59 |
| *** pcaruana has quit IRC | 22:00 | |
| *** bobh has joined #openstack-infra | 22:01 | |
| *** armax has joined #openstack-infra | 22:03 | |
| *** bobh has quit IRC | 22:05 | |
| *** bobh has joined #openstack-infra | 22:10 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** bobh has quit IRC | 22:15 | |
| *** slaweq has quit IRC | 22:17 | |
| *** rlandy has joined #openstack-infra | 22:17 | |
| fungi | hjensas: good eye. looks like the server lost contact with one of the cinder volumes that filesystem spans | 22:17 |
| fungi | Jan 14 16:47:47 static kernel: [325191.787817] EXT4-fs (dm-2): Remounting filesystem read-only | 22:18 |
| *** rcernin has joined #openstack-infra | 22:18 | |
| fungi | so ~5.5 hours ago | 22:18 |
| fungi | i'll unmount it and get the fsck started under a screen session | 22:19 |
| *** markvoelker has quit IRC | 22:19 | |
| fungi | infra-root: ^ heads up, logical volume for logs.o.o is read-only due to write errirs on /dev/xvdg | 22:20 |
| *** bobh has joined #openstack-infra | 22:21 | |
| fungi | had to stop apache2 and kill the log maintenance cron script to be able to umount it | 22:22 |
| *** rlandy has quit IRC | 22:23 | |
| fungi | though in retrospect, i should probably reboot the instance to make sure /dev/xvdg reattaches sanely | 22:23 |
| fungi | i've commented out /srv/static/logs in /etc/fstab so it won't get mounted at boot | 22:24 |
| fungi | #status log rebooted static.openstack.org to make sure disconnected volume /dev/xvdg reattaches correctly | 22:25 |
| openstackstatus | fungi: finished logging | 22:25 |
| *** bobh has quit IRC | 22:25 | |
| fungi | #status log a `fsck -y` of /dev/mapper/main-logs is underway in a root screen session on static.openstack.org | 22:27 |
| openstackstatus | fungi: finished logging | 22:27 |
| *** bobh has joined #openstack-infra | 22:28 | |
| ianw | fungi: ok ... ping if i can help | 22:29 |
| fungi | "This message is to inform you that our monitoring systems have detected a problem with the server which hosts your Cloud Block Storage device 'static.openstack.org/main05' at 17:02 UTC." | 22:30 |
| ianw | heh, thanks, problem noted :) | 22:30 |
| fungi | according to subsequent updates, they got the device restored to normal | 22:30 |
| fungi | so in theory we should be fine once this fsck completes in... some... hours | 22:30 |
| *** ameliac has joined #openstack-infra | 22:32 | |
| *** bobh has quit IRC | 22:32 | |
| fungi | #status alert The filesystem for the logs.openstack.org site was marked read-only at 2018-01-14 16:47 UTC due to an outage incident at the service provider; a filesystem recovery is underway, but job logs uploaded between now and completion are unlikely to be retained so please refrain from rechecking due to POST_FAILURE results until this alert is rescinded. | 22:33 |
| openstackstatus | fungi: sending alert | 22:33 |
| *** sree has joined #openstack-infra | 22:34 | |
| *** jkilpatr has quit IRC | 22:34 | |
| ianw | fungi: if we moved /srv/static/logs somewhere, when we remount the cinder volume it might be easier to copy back these intermediate logs? | 22:34 |
| ianw | i mean, make it a symlink for now, say? | 22:34 |
| fungi | ianw: yes, we can in theory use rsync to copy the logs back in, but in the past when people have done that they've neglected to properly retain/fix up filesystem permissions | 22:34 |
| fungi | and there's ~100gb of space in the volume mounted at /srv/static anyway | 22:35 |
| *** ijw has joined #openstack-infra | 22:35 | |
| -openstackstatus- NOTICE: The filesystem for the logs.openstack.org site was marked read-only at 2018-01-14 16:47 UTC due to an outage incident at the service provider; a filesystem recovery is underway, but job logs uploaded between now and completion are unlikely to be retained so please refrain from rechecking due to POST_FAILURE results until this alert is rescinded. | 22:35 | |
| *** ChanServ changes topic to "The filesystem for the logs.openstack.org site was marked read-only at 2018-01-14 16:47 UTC due to an outage incident at the service provider; a filesystem recovery is underway, but job logs uploaded between now and completion are unlikely to be retained so please refrain from rechecking due to POST_FAILURE results until this alert is rescinded." | 22:35 | |
| *** edmondsw has joined #openstack-infra | 22:36 | |
| fungi | if whoever catches the tail end of the recovery wants to try and save the next few hours of logs, the logs volume can be mounted on a temporary mountpoint, rsync used to copy the newer logs into it, uploads temporarily disabled, a second rsync run to catch any stragglers, deletion of the contents of /srv/static/logs/, then unmount the logs volume and remount it there before reallowing uploads | 22:37 |
| *** sree has quit IRC | 22:38 | |
| fungi | also, unmounting or mounting /srv/static/logs has to be done with apache2 service stopped since it will hold an open file descriptor in the docroot of the vhost | 22:38 |
| openstackstatus | fungi: finished sending alert | 22:38 |
| ianw | i've got the screen open and will keep an eye | 22:39 |
| fungi | no promises that rsync'ing into that volume will be even remotely timely | 22:40 |
| *** ijw has quit IRC | 22:40 | |
| fungi | thanks for watching it ianw! | 22:40 |
| *** edmondsw has quit IRC | 22:41 | |
| *** bobh has joined #openstack-infra | 22:41 | |
| ianw | no ... i'm pretty sure i have symlinked out the logs dir during this happening before, then moved things back in slowly | 22:41 |
| ianw | but it's one more failure point. let's just kiss | 22:41 |
| fungi | might also not be a terrible idea to periodically check `df /srv/static` to make sure it doesn't fill up in the interim | 22:42 |
| ianw | ok, seems like enough headroom (inodes too) | 22:43 |
| *** armax has quit IRC | 22:45 | |
| *** bobh has quit IRC | 22:45 | |
| fungi | i bet after recovery we could just mount the logs volume back at /srv/static/logs and then do a second temporary read-only mount of the static volume somewhere other than /srv/static so we have an unshadowed copy of the filesystem to copy out of | 22:46 |
| fungi | but again, i'm not too concerned with preserving those | 22:46 |
| fungi | more interested in making sure we get the logs volume back into operation as soon as we can manage | 22:46 |
| *** sree has joined #openstack-infra | 22:46 | |
| ianw | ++ yep | 22:48 |
| *** bobh has joined #openstack-infra | 22:48 | |
| *** sree has quit IRC | 22:51 | |
| *** e0ne has quit IRC | 22:56 | |
| *** bobh has quit IRC | 22:58 | |
| *** bobh has joined #openstack-infra | 23:08 | |
| *** bobh has quit IRC | 23:12 | |
| *** bobh has joined #openstack-infra | 23:15 | |
| *** sree has joined #openstack-infra | 23:17 | |
| *** bobh has quit IRC | 23:20 | |
| *** sree has quit IRC | 23:21 | |
| *** sree has joined #openstack-infra | 23:39 | |
| *** sree has quit IRC | 23:44 | |
| *** dhajare has joined #openstack-infra | 23:48 | |
| *** bobh has joined #openstack-infra | 23:52 | |
| *** dingyichen has joined #openstack-infra | 23:55 | |
| *** bobh has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!