| *** hongbin has quit IRC | 00:31 | |
| *** shellclear has joined #openstack-infra | 00:35 | |
| *** shellclear has left #openstack-infra | 00:49 | |
| fungi | mnaser: done and done. i'll try to keep an eye on it and add your ssh keys once they're ready | 01:16 |
|---|---|---|
| mnaser | fungi: thanks. it's been hard to get to the bottom of this because it's a multinode job that takes ages to run | 01:17 |
| mnaser | i'm hoping this will be my last go at it | 01:17 |
| fungi | and what's the url to your keys again (so i don't have to dig it out of dusty scrollback) | 01:18 |
| mnaser | fungi: https://github.com/mnaser.keys :) | 01:21 |
| *** hwoarang has quit IRC | 01:35 | |
| fungi | thanks! | 01:36 |
| *** Adri2000 has quit IRC | 01:36 | |
| *** hwoarang has joined #openstack-infra | 01:37 | |
| *** Adri2000 has joined #openstack-infra | 01:47 | |
| *** xek_ has quit IRC | 02:04 | |
| mnaser | fungi: oddly enough seems like it’s queued 50 minutes later | 02:11 |
| *** gagehugo has quit IRC | 02:43 | |
| *** wolverineav has joined #openstack-infra | 02:55 | |
| *** bobh has joined #openstack-infra | 03:03 | |
| *** wolverineav has quit IRC | 03:03 | |
| *** bobh has quit IRC | 03:12 | |
| *** wolverineav has joined #openstack-infra | 03:12 | |
| *** hongbin has joined #openstack-infra | 03:21 | |
| *** wolverineav has quit IRC | 03:48 | |
| *** wolverineav has joined #openstack-infra | 03:56 | |
| *** hongbin_ has joined #openstack-infra | 04:06 | |
| *** hongbin has quit IRC | 04:09 | |
| *** wolverineav has quit IRC | 04:09 | |
| *** wolverineav has joined #openstack-infra | 04:12 | |
| *** wolverineav has quit IRC | 04:16 | |
| *** hongbin_ has quit IRC | 04:33 | |
| *** auristor has quit IRC | 05:02 | |
| *** ykarel has joined #openstack-infra | 05:12 | |
| *** wolverineav has joined #openstack-infra | 05:18 | |
| *** ykarel has quit IRC | 05:20 | |
| *** wolverineav has quit IRC | 05:23 | |
| *** ykarel has joined #openstack-infra | 05:25 | |
| *** auristor has joined #openstack-infra | 05:38 | |
| *** ykarel has quit IRC | 05:54 | |
| *** yboaron has joined #openstack-infra | 06:59 | |
| *** wolverineav has joined #openstack-infra | 07:10 | |
| *** wolverineav has quit IRC | 07:15 | |
| *** wolverineav has joined #openstack-infra | 07:36 | |
| *** wolverineav has quit IRC | 07:38 | |
| *** wolverineav has joined #openstack-infra | 07:38 | |
| *** wolverineav has quit IRC | 07:42 | |
| *** wolverineav has joined #openstack-infra | 07:43 | |
| *** spa-87 has joined #openstack-infra | 07:54 | |
| *** wolverineav has quit IRC | 08:10 | |
| *** spa-87_ has joined #openstack-infra | 08:33 | |
| *** spa-87 has quit IRC | 08:36 | |
| *** spa-87_ has quit IRC | 08:44 | |
| *** wolverineav has joined #openstack-infra | 09:26 | |
| *** slaweq has joined #openstack-infra | 09:58 | |
| *** tosky has joined #openstack-infra | 10:04 | |
| *** yamamoto has joined #openstack-infra | 10:23 | |
| *** yamamoto has quit IRC | 10:26 | |
| *** yamamoto has joined #openstack-infra | 10:26 | |
| *** jtomasek has joined #openstack-infra | 10:36 | |
| *** wolverineav has quit IRC | 10:41 | |
| *** pbourke has quit IRC | 10:54 | |
| *** pbourke has joined #openstack-infra | 10:56 | |
| *** pcaruana has joined #openstack-infra | 10:58 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** pcaruana has quit IRC | 11:00 | |
| *** yamamoto has joined #openstack-infra | 11:03 | |
| *** ykarel has joined #openstack-infra | 11:05 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:25 | |
| *** bhavikdbavishi has quit IRC | 12:02 | |
| *** dayou has quit IRC | 12:02 | |
| *** dayou has joined #openstack-infra | 12:04 | |
| *** jtomasek has quit IRC | 12:05 | |
| *** wolverineav has joined #openstack-infra | 12:15 | |
| *** wolverineav has quit IRC | 12:19 | |
| *** yamamoto has quit IRC | 12:30 | |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** slaweq has quit IRC | 12:45 | |
| *** ykarel has quit IRC | 12:53 | |
| *** spa-87 has joined #openstack-infra | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| fungi | mnaser: thanks, i sort of passed out last night, but will take a look now and see why it's not getting its node assignment fulfilled | 13:33 |
| *** yamamoto has quit IRC | 13:38 | |
| *** yamamoto has joined #openstack-infra | 13:38 | |
| fungi | 2018-12-30 01:17:24,388 INFO zuul.nodepool: Submitted node request <NodeRequest 300-0001006870 <NodeSet magnum-two-node-xenial [<Node None ('controller',):ubuntu-xenial-vexxhost>, <Node None ('compute1',):ubuntu-xenial-vexxhost>]>> | 13:39 |
| fungi | /var/log/nodepool/nodepool-launcher.log.2018-12-29_21:2018-12-30 01:24:43,931 ERROR nodepool.NodeLauncher-0001419406: Request 300-0001006870: Launch attempt 3/3 failed for node 0001419406: [...] nodepool.exceptions.ConnectionTimeoutException: Timeout waiting for connection on port 22 | 13:45 |
| fungi | the most recent ubuntu-xenial images uploaded to vexxhost regions are right at 24 hours old now, so i doubt it's related to a new image | 13:48 |
| fungi | especially since we have lots of other jobs running on nodes booted from the same image in vexxhost regions (just not any on the vexxhost-specific flavor) | 13:50 |
| fungi | could something have happened to break guest networking where those are getting scheduled? | 13:57 |
| *** spa-87_ has joined #openstack-infra | 14:01 | |
| fungi | actually, i see some similar launch failures for ubuntu-xenial-vexxhost nodes in vexxhost-sjc1 earlier in the day too | 14:03 |
| *** spa-87 has quit IRC | 14:04 | |
| fungi | some succeed on the second or third try | 14:04 |
| *** yamamoto has quit IRC | 14:06 | |
| *** yamamoto has joined #openstack-infra | 14:14 | |
| *** yamamoto has quit IRC | 14:20 | |
| *** spa-87 has joined #openstack-infra | 14:28 | |
| *** spa-87_ has quit IRC | 14:30 | |
| *** hamzy has joined #openstack-infra | 14:32 | |
| *** yboaron has quit IRC | 14:34 | |
| fungi | that might explain why other changes haven't gotten hung. they got lucky and didn't fail all three attempts | 14:34 |
| *** slaweq has joined #openstack-infra | 14:48 | |
| *** ykarel has joined #openstack-infra | 14:50 | |
| *** ykarel has quit IRC | 15:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Pin nodejs package to upstream repo https://review.openstack.org/627823 | 15:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add nodesets API route https://review.openstack.org/605877 | 15:39 |
| *** sshnaidm_ is now known as sshnaidm | 15:44 | |
| *** slaweq has quit IRC | 16:08 | |
| *** ykarel has joined #openstack-infra | 16:23 | |
| *** bobh has joined #openstack-infra | 16:27 | |
| *** bobh has quit IRC | 16:35 | |
| *** irclogbot_1 has quit IRC | 16:44 | |
| *** Vadmacs has joined #openstack-infra | 16:48 | |
| *** irclogbot_1 has joined #openstack-infra | 16:55 | |
| *** e0ne has joined #openstack-infra | 16:57 | |
| *** yboaron has joined #openstack-infra | 17:01 | |
| *** yboaron has quit IRC | 17:05 | |
| *** e0ne has quit IRC | 17:09 | |
| *** e0ne has joined #openstack-infra | 17:09 | |
| *** ykarel has quit IRC | 17:12 | |
| *** e0ne has quit IRC | 17:12 | |
| *** jamesmcarthur has joined #openstack-infra | 17:23 | |
| *** auristor has quit IRC | 17:40 | |
| *** auristor has joined #openstack-infra | 17:42 | |
| *** jamesmcarthur has quit IRC | 17:53 | |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| *** jamesmcarthur has quit IRC | 17:59 | |
| *** jamesmcarthur has joined #openstack-infra | 18:33 | |
| *** jtomasek has joined #openstack-infra | 18:37 | |
| mnaser | http://grafana.openstack.org/d/nuvIH5Imk/nodepool-vexxhost?orgId=1 looks like it seems.. ok? | 18:38 |
| *** jamesmcarthur has quit IRC | 18:41 | |
| *** spa-87 has quit IRC | 19:08 | |
| fungi | i think it's just the vexxhost-specific ubuntu-xenial-vexxhost launch attempts erroring. they aren't requested very often | 19:15 |
| fungi | http://grafana.openstack.org/d/nuvIH5Imk/nodepool-vexxhost?orgId=1&panelId=12&fullscreen&from=1546024510999&to=1546197311007 | 19:15 |
| *** hongbin has joined #openstack-infra | 19:17 | |
| *** jtomasek has quit IRC | 19:29 | |
| *** Vadmacs has quit IRC | 20:04 | |
| *** slaweq has joined #openstack-infra | 20:30 | |
| *** jamesmcarthur has joined #openstack-infra | 20:36 | |
| *** wolverineav has joined #openstack-infra | 20:41 | |
| *** jamesmcarthur has quit IRC | 20:42 | |
| *** wolverineav has quit IRC | 20:55 | |
| *** wolverineav has joined #openstack-infra | 21:00 | |
| *** wolverineav has quit IRC | 21:03 | |
| *** wolverin_ has joined #openstack-infra | 21:03 | |
| *** wolverin_ has quit IRC | 21:07 | |
| *** d0ugal has joined #openstack-infra | 21:17 | |
| *** jamesmcarthur has joined #openstack-infra | 21:20 | |
| *** jamesmcarthur has quit IRC | 21:20 | |
| *** wolverineav has joined #openstack-infra | 21:21 | |
| *** wolverineav has quit IRC | 21:23 | |
| fungi | mnaser: i'm at a bit of a loss as to why it would have repeatedly failed to ssh into the launched nodes there, but after manually dequeuing the change and rechecking again it seems to have gotten a node assignment now and started the job | 21:26 |
| *** jamesmcarthur has joined #openstack-infra | 21:26 | |
| *** wolverineav has joined #openstack-infra | 21:37 | |
| *** wolverineav has quit IRC | 21:40 | |
| *** slaweq has quit IRC | 21:53 | |
| *** wolverineav has joined #openstack-infra | 21:57 | |
| *** wolverineav has quit IRC | 22:01 | |
| *** wolverineav has joined #openstack-infra | 22:07 | |
| *** wolverineav has quit IRC | 22:20 | |
| *** wolverineav has joined #openstack-infra | 22:20 | |
| *** bobh has joined #openstack-infra | 22:21 | |
| *** bobh has quit IRC | 22:25 | |
| *** wolverineav has quit IRC | 22:29 | |
| *** jamesmcarthur has quit IRC | 22:41 | |
| *** hongbin has quit IRC | 22:43 | |
| *** jamesmcarthur has joined #openstack-infra | 22:44 | |
| *** wolverineav has joined #openstack-infra | 22:51 | |
| *** jamesmcarthur has quit IRC | 23:18 | |
| *** jamesmcarthur has joined #openstack-infra | 23:21 | |
| *** wolverineav has quit IRC | 23:25 | |
| *** jamesmcarthur has quit IRC | 23:26 | |
| *** jamesmcarthur has joined #openstack-infra | 23:29 | |
| *** jamesmcarthur has quit IRC | 23:34 | |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| *** jamesmcarthur has quit IRC | 23:47 | |
| *** jamesmcarthur has joined #openstack-infra | 23:51 | |
| *** jamesmcarthur has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!