| *** slaweq has joined #openstack-infra | 00:01 | |
| *** hongbin has quit IRC | 00:03 | |
| *** slaweq has quit IRC | 00:06 | |
| *** rakhmerov has quit IRC | 00:24 | |
| *** rakhmerov has joined #openstack-infra | 00:25 | |
| *** Thaelim has quit IRC | 00:26 | |
| *** jamesmcarthur has joined #openstack-infra | 00:28 | |
| *** dhill_ has quit IRC | 00:29 | |
| *** Thaelim has joined #openstack-infra | 00:31 | |
| *** jamesmcarthur has quit IRC | 00:34 | |
| *** salv-orlando has joined #openstack-infra | 00:41 | |
| *** olaph has quit IRC | 00:45 | |
| *** salv-orlando has quit IRC | 00:46 | |
| *** olaph has joined #openstack-infra | 00:49 | |
| mriedem | nova's functional job has been timing out a ton lately, | 00:49 |
|---|---|---|
| mriedem | and i can't really tell if it's the tests that are actually timing out, | 00:49 |
| mriedem | looking at http://logs.openstack.org/98/538498/2/check/nova-tox-functional/d0c14bc/job-output.txt.gz | 00:49 |
| mriedem | it looks like the tox -e functional run was only going 27 minutes before it was killed | 00:49 |
| mriedem | and i think the actual functional timeout is supposed to be 30 seconds | 00:50 |
| mriedem | i can't make heads or tails of the ansible output | 00:50 |
| mriedem | it looks like it eats up 3 minutes between the time TASK [tox : Run tox without tests] starts, and when the actual tests start running | 00:52 |
| mriedem | so i guess that's the extra 3 minutes | 00:53 |
| clarkb | mriedem: ya aiui pre-run, run, and post-run currebtly each get the timeout value time to run | 00:54 |
| clarkb | in this case run timed out after 30 minutes | 00:54 |
| mriedem | yeah, and i see what's going wrong in https://review.openstack.org/#/c/537933/ | 00:55 |
| mriedem | we're failing on the openstack-tox-functional job which is 30 minutes, but nova-tox-functional should be 45 minutes, we just need to stop running openstack-tox-functional on nova, and we have patches aligned to do that | 00:55 |
| mriedem | so nvm | 00:55 |
| mriedem | thanks | 00:55 |
| *** gongysh has joined #openstack-infra | 00:55 | |
| mriedem | clarkb: can you +W while you're here? https://review.openstack.org/#/c/533608/ | 00:57 |
| mriedem | i've fast approved the nova changes | 00:57 |
| mriedem | jgwentworth: ^ fyi | 00:57 |
| mriedem | giblet: ^ | 00:58 |
| clarkb | mriedem: there isnt much point as long as it is WIP'd | 00:59 |
| *** chem``` has joined #openstack-infra | 00:59 | |
| clarkb | I'm not sure why it is wip'd when it has the dependson though | 00:59 |
| mriedem | it was wip'ed before the backports were up | 01:00 |
| mriedem | are you able to override the -W? | 01:00 |
| mriedem | we've had stuff approved for days continually failing b/c the functional job timeouts (among other things) | 01:01 |
| clarkb | ya I think I can remove the vote but not going to sort that out on phone. | 01:02 |
| clarkb | I can poke it when next at computer | 01:03 |
| *** chem`` has quit IRC | 01:03 | |
| mriedem | ah sorry, np | 01:09 |
| *** gongysh has quit IRC | 01:18 | |
| *** jamesmcarthur has joined #openstack-infra | 01:19 | |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** dteselkin has quit IRC | 01:25 | |
| *** armax has joined #openstack-infra | 01:26 | |
| *** Thaelim has quit IRC | 01:28 | |
| *** Thaelim has joined #openstack-infra | 01:29 | |
| *** wolverineav has joined #openstack-infra | 01:31 | |
| *** dteselkin has joined #openstack-infra | 01:31 | |
| *** armax has quit IRC | 01:33 | |
| *** wolverineav has quit IRC | 01:35 | |
| *** salv-orlando has joined #openstack-infra | 01:42 | |
| *** salv-orlando has quit IRC | 01:47 | |
| *** jamesmcarthur has joined #openstack-infra | 01:51 | |
| *** wolverineav has joined #openstack-infra | 01:58 | |
| *** wolverineav has quit IRC | 02:02 | |
| *** wolverineav has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has quit IRC | 02:24 | |
| *** Thaelim has quit IRC | 02:26 | |
| *** wolverineav has quit IRC | 02:27 | |
| *** wolverineav has joined #openstack-infra | 02:33 | |
| *** Thaelim has joined #openstack-infra | 02:36 | |
| *** wolverineav has quit IRC | 02:37 | |
| *** salv-orlando has joined #openstack-infra | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** salv-orlando has quit IRC | 02:47 | |
| *** claudiub|2 has quit IRC | 02:58 | |
| *** jamesmcarthur has quit IRC | 03:04 | |
| *** jamesmcarthur has joined #openstack-infra | 03:06 | |
| *** wolverineav has joined #openstack-infra | 03:06 | |
| *** wolverineav has quit IRC | 03:11 | |
| *** jamesmcarthur has quit IRC | 03:11 | |
| *** Thaelim has quit IRC | 03:23 | |
| *** chem``` has quit IRC | 03:30 | |
| *** yamamoto has joined #openstack-infra | 03:34 | |
| *** salv-orlando has joined #openstack-infra | 03:44 | |
| *** olaph1 has joined #openstack-infra | 03:46 | |
| *** olaph has quit IRC | 03:47 | |
| *** salv-orlando has quit IRC | 03:48 | |
| *** rosmaita has quit IRC | 03:57 | |
| *** yamamoto has quit IRC | 03:59 | |
| *** gaurangt_ has joined #openstack-infra | 04:05 | |
| *** gaurangt has quit IRC | 04:06 | |
| *** gaurangt has joined #openstack-infra | 04:06 | |
| *** olaph has joined #openstack-infra | 04:06 | |
| *** olaph1 has quit IRC | 04:07 | |
| *** gaurangt_ has quit IRC | 04:09 | |
| openstackgerrit | Merged openstack-dev/hacking master: autospec check: Allow 'wraps' to be an alternative to autospec https://review.openstack.org/512902 | 04:16 |
| *** bhavik1 has joined #openstack-infra | 04:19 | |
| *** bhavik1 has quit IRC | 04:31 | |
| *** gaurangt has quit IRC | 04:34 | |
| *** agopi|out has joined #openstack-infra | 04:40 | |
| *** salv-orlando has joined #openstack-infra | 04:44 | |
| *** salv-orlando has quit IRC | 04:49 | |
| *** yamamoto has joined #openstack-infra | 05:00 | |
| *** yamamoto has quit IRC | 05:07 | |
| *** psachin has joined #openstack-infra | 05:39 | |
| *** salv-orlando has joined #openstack-infra | 05:45 | |
| *** salv-orlando has quit IRC | 05:50 | |
| *** gaurangt has joined #openstack-infra | 06:06 | |
| *** wolverineav has joined #openstack-infra | 06:20 | |
| *** wolverineav has quit IRC | 06:25 | |
| *** xarses_ has joined #openstack-infra | 06:45 | |
| *** salv-orlando has joined #openstack-infra | 06:46 | |
| *** andreww has joined #openstack-infra | 06:46 | |
| *** xarses_ has quit IRC | 06:49 | |
| *** salv-orlando has quit IRC | 06:51 | |
| *** andreww has quit IRC | 06:54 | |
| *** yamamoto has joined #openstack-infra | 06:56 | |
| *** afazekas_ has quit IRC | 06:58 | |
| *** afazekas has joined #openstack-infra | 06:58 | |
| *** salv-orlando has joined #openstack-infra | 07:04 | |
| *** jtomasek has joined #openstack-infra | 07:23 | |
| *** wolverineav has joined #openstack-infra | 07:25 | |
| *** oidgar has joined #openstack-infra | 07:30 | |
| *** jtomasek has quit IRC | 07:38 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: Make notification driver configurable https://review.openstack.org/538574 | 07:44 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: WIP: Add MQTT notification publisher https://review.openstack.org/538575 | 07:44 |
| *** yamamoto has quit IRC | 07:53 | |
| *** jamesmcarthur has joined #openstack-infra | 08:01 | |
| *** salv-orl_ has joined #openstack-infra | 08:02 | |
| *** salv-orlando has quit IRC | 08:03 | |
| *** jamesmcarthur has quit IRC | 08:05 | |
| *** salv-orl_ has quit IRC | 08:11 | |
| *** salv-orlando has joined #openstack-infra | 08:17 | |
| *** gongysh has joined #openstack-infra | 08:23 | |
| *** claudiub|2 has joined #openstack-infra | 08:44 | |
| *** yolanda has joined #openstack-infra | 08:44 | |
| *** wolverineav has quit IRC | 08:48 | |
| *** gongysh has quit IRC | 08:49 | |
| *** yee37933 has joined #openstack-infra | 08:52 | |
| *** yamamoto has joined #openstack-infra | 08:54 | |
| *** yee379 has quit IRC | 08:54 | |
| *** wolverineav has joined #openstack-infra | 08:58 | |
| *** slaweq has joined #openstack-infra | 08:58 | |
| *** armaan has joined #openstack-infra | 09:01 | |
| *** psachin has quit IRC | 09:01 | |
| *** wolverineav has quit IRC | 09:02 | |
| *** yolanda_ has joined #openstack-infra | 09:03 | |
| *** yamamoto has quit IRC | 09:03 | |
| *** slaweq has quit IRC | 09:05 | |
| *** slaweq has joined #openstack-infra | 09:09 | |
| *** jtomasek has joined #openstack-infra | 09:41 | |
| *** pcichy has joined #openstack-infra | 10:14 | |
| *** dsariel has joined #openstack-infra | 10:14 | |
| *** armaan has quit IRC | 10:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Moving nova functional test def to in tree https://review.openstack.org/533608 | 10:28 |
| AJaeger | mriedem: I rebased to get rid of the WIP from 533608. Now needs another config-core to +2A. | 10:28 |
| *** agopi|out has quit IRC | 10:35 | |
| *** salv-orlando has quit IRC | 10:37 | |
| *** slaweq has quit IRC | 10:38 | |
| *** slaweq has joined #openstack-infra | 10:38 | |
| *** pbourke has quit IRC | 10:41 | |
| *** jtomasek has quit IRC | 10:42 | |
| *** pbourke has joined #openstack-infra | 10:43 | |
| *** matbu has joined #openstack-infra | 10:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove oslo-cookiecutter legacy job https://review.openstack.org/538591 | 10:46 |
| *** e0ne has joined #openstack-infra | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:06 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** e0ne has quit IRC | 11:15 | |
| *** yamamoto has joined #openstack-infra | 11:20 | |
| *** armaan has joined #openstack-infra | 11:20 | |
| *** psachin has joined #openstack-infra | 11:21 | |
| *** yamamoto has quit IRC | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| *** salv-orlando has joined #openstack-infra | 11:37 | |
| *** armaan has quit IRC | 11:38 | |
| *** salv-orlando has quit IRC | 11:42 | |
| *** wolverineav has joined #openstack-infra | 11:56 | |
| *** erlon_ has joined #openstack-infra | 11:58 | |
| *** jamesmcarthur has joined #openstack-infra | 12:01 | |
| *** jamesmcarthur has quit IRC | 12:06 | |
| *** yamamoto has quit IRC | 12:16 | |
| *** slaweq has quit IRC | 12:18 | |
| *** psachin has quit IRC | 12:25 | |
| *** psachin has joined #openstack-infra | 12:30 | |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| *** salv-orlando has joined #openstack-infra | 12:38 | |
| *** salv-orlando has quit IRC | 12:43 | |
| *** olaph1 has joined #openstack-infra | 12:52 | |
| *** olaph has quit IRC | 12:53 | |
| *** gaurangt has quit IRC | 13:05 | |
| *** gaurangt has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:17 | |
| *** gaurangt has quit IRC | 13:17 | |
| *** jtomasek has joined #openstack-infra | 13:19 | |
| *** gaurangt has joined #openstack-infra | 13:20 | |
| *** salv-orlando has joined #openstack-infra | 13:39 | |
| *** jtomasek has quit IRC | 13:39 | |
| *** salv-orlando has quit IRC | 13:44 | |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| *** gaurangt has quit IRC | 13:57 | |
| *** gaurangt has joined #openstack-infra | 14:00 | |
| *** olaph has joined #openstack-infra | 14:08 | |
| *** olaph1 has quit IRC | 14:09 | |
| *** wolverineav has quit IRC | 14:12 | |
| *** psachin has quit IRC | 14:14 | |
| *** wolverineav has joined #openstack-infra | 14:18 | |
| *** quite has quit IRC | 14:23 | |
| *** dsariel has quit IRC | 14:24 | |
| *** slaweq has joined #openstack-infra | 14:24 | |
| *** quite has joined #openstack-infra | 14:25 | |
| *** dhill_ has joined #openstack-infra | 14:26 | |
| *** slaweq has quit IRC | 14:29 | |
| *** salv-orlando has joined #openstack-infra | 14:40 | |
| *** oidgar has quit IRC | 14:42 | |
| *** pcichy has quit IRC | 14:43 | |
| *** salv-orlando has quit IRC | 14:44 | |
| *** wolverineav has quit IRC | 14:46 | |
| pabelanger | infra-root: I think nodepool.o.o is down, meaning zookeeper is also down | 14:47 |
| dmsimard | pabelanger: I can't reach it either, what made you find out ? | 14:48 |
| pabelanger | queue for zuul growing and not launching nodes: grafana.openstack.org/dashboard/db/nodepool | 14:48 |
| pabelanger | I'm going to see if I can access console | 14:49 |
| dmsimard | pabelanger: the console is responsive | 14:49 |
| dmsimard | not sure if we have a password though | 14:49 |
| pabelanger | no, usually not | 14:49 |
| pabelanger | anything of importance on console? | 14:50 |
| dmsimard | just a login prompt | 14:50 |
| dmsimard | doesn't ping on local network either | 14:51 |
| dmsimard | let me check if we have a password just in case | 14:52 |
| *** jtomasek has joined #openstack-infra | 14:52 | |
| pabelanger | any open tickets? | 14:53 |
| dmsimard | there is, actually | 14:54 |
| dmsimard | reading | 14:54 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for stable branch cellsv1 job libvirt crash bug 1745838 https://review.openstack.org/538614 | 14:54 |
| openstack | bug 1745838 in OpenStack Compute (nova) "legacy-tempest-dsvm-cells constantly failing on stable pike and ocata due to libvirt connection reset" [Undecided,New] https://launchpad.net/bugs/1745838 | 14:54 |
| pabelanger | yah, see it now | 14:55 |
| dmsimard | pabelanger: er, they're replacing the chassis the server is on and they explicitely mention "Please do not attempt to access or modify 'nodepool.openstack.org' during this process." | 14:55 |
| pabelanger | http://paste.openstack.org/show/655937/ | 14:55 |
| pabelanger | yes | 14:55 |
| pabelanger | okay, so we'll have to ride it out | 14:55 |
| pabelanger | and work to migrate to zookeeper cluster this week | 14:56 |
| pabelanger | status alert due to server outage with zookeeper, zuul.o.o is currently offline. We are currently waiting for more information from our cloud provider when service will be restored. | 14:59 |
| dmsimard | pabelanger: let's send a notice ? #status notice Jobs are currently not running and are staying queued in Zuul pending the completion of a maintenance at our cloud provider. Jobs will resume running once this maintenance has been completed. | 14:59 |
| pabelanger | dmsimard: look good ^ | 14:59 |
| dmsimard | hah you beat me to it | 14:59 |
| dmsimard | I'm not sure I would do alert just because we're sunday, otherwise either message is good to me | 15:00 |
| pabelanger | your is actually better, zuul.o.o isn't offline, just not running jbos | 15:00 |
| pabelanger | will send your version | 15:01 |
| pabelanger | #status notice Jobs are currently not running and are staying queued in Zuul pending the completion of a maintenance at our cloud provider. Jobs will resume once this maintenance has been completed. | 15:02 |
| openstackstatus | pabelanger: sending notice | 15:02 |
| -openstackstatus- NOTICE: Jobs are currently not running and are staying queued in Zuul pending the completion of a maintenance at our cloud provider. Jobs will resume once this maintenance has been completed. | 15:03 | |
| pabelanger | dmsimard: there is also a ticket about logs.o.o failure this week too, | 15:04 |
| cmurphy | for jobs that already started running will they get to finish? there are some in the gate that have been finished for a few hours but are still in the queue | 15:04 |
| dmsimard | ^ pabelanger: I don't know ? I have no idea about the impact of ZK going away in terms of locks being lost | 15:05 |
| openstackstatus | pabelanger: finished sending notice | 15:05 |
| pabelanger | cmurphy: yah, think so. Data should be known to zuul.o.o, but is trying to write it into zookeeper. I think if host comes back online, and everything reconnects, finished jobs will start to merge. | 15:05 |
| cmurphy | okay cool | 15:05 |
| pabelanger | however, I think this is the first time we've had this happen | 15:05 |
| pabelanger | I think as long as zuul.o.o is online, data is safe | 15:06 |
| dmsimard | I'm out for now, I'll be back later to check on that and the logs.o.o rsync | 15:06 |
| *** bobh has joined #openstack-infra | 15:06 | |
| pabelanger | yah, I can watch for next hour or so | 15:07 |
| *** dsariel has joined #openstack-infra | 15:17 | |
| *** slaweq has joined #openstack-infra | 15:18 | |
| *** slaweq has quit IRC | 15:19 | |
| *** bobh has quit IRC | 15:26 | |
| pabelanger | ticket just updated, chassis swap done. And say nodepool.o.o should be back online | 15:34 |
| pabelanger | checking now | 15:34 |
| *** Goneri has joined #openstack-infra | 15:34 | |
| *** jtomasek has quit IRC | 15:35 | |
| pabelanger | okay, looks shutdown. Going to see about powering it back on | 15:37 |
| *** slaweq has joined #openstack-infra | 15:37 | |
| *** slaweq has quit IRC | 15:42 | |
| *** sshnaidm has quit IRC | 15:43 | |
| pabelanger | okay, for people playing at home 500: SR_HAS_NO_PBDS is the error currently | 15:47 |
| pabelanger | going to see if I can get somebody from rackspace on phone | 15:47 |
| pabelanger | webchat isn't working for me | 15:48 |
| corvus | that errors looks like missing physical block devices for the server; maybe they didn't remap something when they swapped. | 15:50 |
| corvus | (it's a xen error -- nice of them to pass it through to us) | 15:51 |
| pabelanger | okay, on phone with rax | 15:53 |
| pabelanger | on hold, waiting for an admin now | 15:55 |
| *** olaph1 has joined #openstack-infra | 15:55 | |
| *** olaph has quit IRC | 15:56 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Don't run legacy-tempest-dsvm-cells on newton/ocata/pike https://review.openstack.org/538619 | 15:58 |
| mordred | dmsimard, pabelanger, corvus: wow. fun week for us :( | 16:03 |
| *** pcichy has joined #openstack-infra | 16:04 | |
| pabelanger | yup, currently on hold for human. But have made contact | 16:04 |
| *** Goneri has quit IRC | 16:05 | |
| mordred | \o/ | 16:05 |
| pabelanger | yah, corvus looks to be right. Issue with block devices, becky pretty sure know how to fix it. Working on that now, needs another 10mins or so | 16:09 |
| corvus | kk | 16:10 |
| pabelanger | exchanged phone numbers and she'll be giving a call when things are better and updating support ticket | 16:10 |
| mordred | go go becky | 16:13 |
| pabelanger | yah, nodepool.o.o is back online | 16:17 |
| pabelanger | okay, nodepool list is working again | 16:19 |
| pabelanger | waiting to see if zuul.o.o reconnects | 16:19 |
| corvus | i don't see any zk activity from zuul yet... is zk up yet? | 16:19 |
| pabelanger | yes | 16:19 |
| corvus | nodepool hasn't connected to it yet either | 16:20 |
| pabelanger | 2018-01-28 15:43:35,448 WARNING kazoo.client: Connection dropped: socket connection error: None | 16:20 |
| corvus | they may have long connection retry intervals at this point | 16:20 |
| pabelanger | was last attempt from zookeeper | 16:20 |
| corvus | that's a really long time | 16:20 |
| *** dsariel has quit IRC | 16:20 | |
| pabelanger | http://paste.openstack.org/show/655947/ | 16:22 |
| pabelanger | maybe another 20mins? | 16:22 |
| corvus | wow. that certainly needs tuning. | 16:22 |
| pabelanger | yah | 16:22 |
| pabelanger | let me check nodepool | 16:22 |
| pabelanger | 2018-01-28 16:25:04,841 DEBUG nodepool.zk.ZooKeeper: ZooKeeper connection: CONNECTED | 16:25 |
| pabelanger | that is nl01 | 16:25 |
| corvus | pabelanger: nl04 is on too | 16:26 |
| pabelanger | corvus: yah, all launcher look to be back | 16:27 |
| pabelanger | so that is good | 16:27 |
| corvus | and nl03, and... er... one of the builders | 16:27 |
| pabelanger | do you want to see if zuul reconnects? | 16:28 |
| corvus | pabelanger: what did the connection intervals on nodepool look like? | 16:28 |
| *** olaph has joined #openstack-infra | 16:29 | |
| *** olaph1 has quit IRC | 16:29 | |
| pabelanger | it looked to be trying pretty aggressive: http://paste.openstack.org/show/655949/ | 16:30 |
| corvus | pabelanger: well, i don't know those messages reflect connection attempts -- i think those may be the nodepool threads themselves checking in on whether it's connected | 16:31 |
| corvus | pabelanger: i'm guessing we don't have kazoo logging enabled in nodepool | 16:31 |
| pabelanger | yah, possible. Let me check | 16:32 |
| pabelanger | yah, not kazoo handler in logging.conf for nodepool | 16:32 |
| pabelanger | but does exist in zuul | 16:32 |
| *** slaweq has joined #openstack-infra | 16:34 | |
| *** bobh has joined #openstack-infra | 16:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool master: Add kazoo logging to nodepool https://review.openstack.org/538623 | 16:38 |
| pabelanger | corvus: okay, ^ should help in the future | 16:38 |
| *** armax has joined #openstack-infra | 16:40 | |
| *** salv-orlando has joined #openstack-infra | 16:41 | |
| *** tosky has joined #openstack-infra | 16:42 | |
| corvus | there goes the connection retry | 16:44 |
| pabelanger | and connected | 16:44 |
| *** salv-orlando has quit IRC | 16:46 | |
| *** e0ne has joined #openstack-infra | 16:48 | |
| *** sshnaidm has joined #openstack-infra | 16:49 | |
| AJaeger | corvus: 675 changes with 20 mins difference means 9 days until everything is through. don't reboot your computer ;) | 16:49 |
| pabelanger | corvus: I'm not seeing zuul do anything new after connection, possible there is another issue? | 16:49 |
| corvus | pabelanger: yeah, i'm going to ask it for a thread dump | 16:50 |
| AJaeger | corvus: that's related to your project name stanza removal | 16:50 |
| corvus | AJaeger: i figured :) | 16:50 |
| pabelanger | corvus: okay, I have an existing appointment here in 10mins. Do you mind if I hand off to you now? | 16:50 |
| corvus | pabelanger: yep, thanks! | 16:50 |
| pabelanger | great! | 16:50 |
| corvus | i expect i will collect some debug info, save the queues, and restart | 16:51 |
| pabelanger | wfm | 16:51 |
| AJaeger | corvus: FYI https://review.openstack.org/538461 is a duplicate of https://review.openstack.org/538456 - you submitted two changes for that repo and one in merge-conflict | 16:51 |
| corvus | http://paste.openstack.org/show/655953/ | 16:52 |
| *** e0ne has quit IRC | 16:52 | |
| corvus | that's where we are | 16:53 |
| corvus | it's possible we're still recoverable | 16:53 |
| *** flwang has quit IRC | 16:54 | |
| pabelanger | oh, hmm | 16:55 |
| pabelanger | let me check something | 16:55 |
| pabelanger | corvus: http://paste.openstack.org/show/655954/ is the only error we got when we lost zookeeper | 16:56 |
| pabelanger | wonder if related | 16:56 |
| corvus | yes, it's waiting to unlock the next node | 16:57 |
| pabelanger | okay, I'll have to catch up when I return. | 16:58 |
| pabelanger | good luck | 16:58 |
| *** yamamoto has quit IRC | 16:58 | |
| *** flwang has joined #openstack-infra | 17:07 | |
| *** armax has quit IRC | 17:07 | |
| corvus | it shouldn't wait there more than 61 seconds. so it must be stuck in a loop where it's continuously getting retryable errors. we can probably synthetically debug this later. for now, i think it's best to restart. | 17:10 |
| corvus | #status notice Zuul has been restarted due to an outage in our cloud provider. Changes already in queues have been restored, but changes uploaded and approved since 12:30 UTC may need to be rechecked or reapproved. | 17:18 |
| openstackstatus | corvus: sending notice | 17:18 |
| *** spligak has joined #openstack-infra | 17:19 | |
| -openstackstatus- NOTICE: Zuul has been restarted due to an outage in our cloud provider. Changes already in queues have been restored, but changes uploaded and approved since 12:30 UTC may need to be rechecked or reapproved. | 17:20 | |
| openstackstatus | corvus: finished sending notice | 17:21 |
| *** slaweq has quit IRC | 17:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove sqlalchemy-migrate legacy tox job https://review.openstack.org/538631 | 17:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove jobs from ironic-inspector-tempest-plugin https://review.openstack.org/538632 | 17:36 |
| *** b_bezak has joined #openstack-infra | 17:37 | |
| AJaeger | config-core, could you review thse nova changes to help them with jobs, please? https://review.openstack.org/#/c/533608/ and https://review.openstack.org/#/c/538619 - the first one still waits for one dependent change to merge... | 17:39 |
| *** salv-orlando has joined #openstack-infra | 17:42 | |
| *** salv-orlando has quit IRC | 17:47 | |
| *** harlowja has quit IRC | 17:58 | |
| *** yamamoto has joined #openstack-infra | 17:59 | |
| *** yamamoto has quit IRC | 18:11 | |
| *** slaweq has joined #openstack-infra | 18:13 | |
| *** jamesmcarthur has joined #openstack-infra | 18:16 | |
| *** slaweq has quit IRC | 18:18 | |
| *** olaph1 has joined #openstack-infra | 18:21 | |
| *** olaph has quit IRC | 18:22 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 18:23 |
| openstackgerrit | Merged openstack-infra/nodepool master: Update docs to reflect newer Python requirement. https://review.openstack.org/538296 | 18:27 |
| *** pcichy has quit IRC | 18:32 | |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** salv-orlando has joined #openstack-infra | 18:43 | |
| *** salv-orlando has quit IRC | 18:47 | |
| *** olaph1 is now known as olaph | 18:58 | |
| *** slaweq has joined #openstack-infra | 18:58 | |
| *** agopi|out has joined #openstack-infra | 19:03 | |
| AJaeger | corvus: melissaml is also sending project name changes ;( I just sent her an email asking to abandon those or coordinate with you here... | 19:13 |
| *** slaweq has quit IRC | 19:29 | |
| *** rosmaita has joined #openstack-infra | 19:29 | |
| *** jamesmcarthur has joined #openstack-infra | 19:43 | |
| *** salv-orlando has joined #openstack-infra | 19:44 | |
| *** andreww has joined #openstack-infra | 19:48 | |
| *** jamesmcarthur has quit IRC | 19:48 | |
| *** salv-orlando has quit IRC | 19:49 | |
| *** andreww has quit IRC | 19:53 | |
| *** slaweq has joined #openstack-infra | 19:57 | |
| *** olaph1 has joined #openstack-infra | 19:58 | |
| *** olaph has quit IRC | 19:59 | |
| *** jamesmcarthur has joined #openstack-infra | 19:59 | |
| *** slaweq has quit IRC | 20:04 | |
| *** jamesmcarthur has quit IRC | 20:04 | |
| *** jamesmcarthur has joined #openstack-infra | 20:04 | |
| *** jamesmcarthur has quit IRC | 20:08 | |
| *** jamesmcarthur has joined #openstack-infra | 20:08 | |
| *** jamesmcarthur has quit IRC | 20:12 | |
| *** d0ugal has quit IRC | 20:17 | |
| *** d0ugal has joined #openstack-infra | 20:26 | |
| fungi | AJaeger: project name changes as in renaming git repositories? if so, we want those noted at https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames | 20:27 |
| cmurphy | i think he means like https://review.openstack.org/#/c/538397/ | 20:35 |
| fungi | cmurphy: oh! thanks. so not project REname changes. got it ;) | 20:38 |
| * fungi has been out of the loop for a week | 20:38 | |
| fungi | cmurphy: btw, that one needs its approval reapplied, looks like, before it'll enqueue into the gate | 20:40 |
| fungi | i'd add one myself, but since you seem to be around and you approved that one, it'd be more correct to let you remove and readd yours | 20:40 |
| cmurphy | fungi: i thought 'recheck' was all that was needed? i see it in the gate queue | 20:42 |
| fungi | cmurphy: oh, interesting, zuul didn't remove the verify +1 and comment that it was enqueuing into the gate | 20:44 |
| *** salv-orlando has joined #openstack-infra | 20:45 | |
| *** dsariel has joined #openstack-infra | 20:45 | |
| fungi | i s'pose we're all good in that case | 20:45 |
| *** olaph has joined #openstack-infra | 20:46 | |
| *** olaph1 has quit IRC | 20:48 | |
| *** salv-orlando has quit IRC | 20:50 | |
| *** dave-mccowan has joined #openstack-infra | 20:53 | |
| *** agopi|out has quit IRC | 20:54 | |
| *** slaweq has joined #openstack-infra | 20:57 | |
| *** dsariel has quit IRC | 21:03 | |
| *** bobh has quit IRC | 21:05 | |
| *** bobh has joined #openstack-infra | 21:08 | |
| *** bobh has quit IRC | 21:12 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Don't run legacy-tempest-dsvm-cells on newton/ocata/pike https://review.openstack.org/538619 | 21:15 |
| *** bobh has joined #openstack-infra | 21:25 | |
| *** bobh has quit IRC | 21:28 | |
| *** bobh has joined #openstack-infra | 21:28 | |
| *** bobh has quit IRC | 21:32 | |
| *** bobh has joined #openstack-infra | 21:39 | |
| *** bobh has quit IRC | 21:44 | |
| *** threestrands has joined #openstack-infra | 21:49 | |
| *** threestrands has quit IRC | 21:49 | |
| *** threestrands has joined #openstack-infra | 21:49 | |
| *** bobh has joined #openstack-infra | 21:49 | |
| cmurphy | i've been watching this job for a while https://review.openstack.org/#/c/524423/ and it was at the top of the gate queue a few minutes ago and was nearly finished but then was demoted in favor of some requirements jobs and had to restart its jobs and is now about to fail due to a timeout in the unit test job | 21:49 |
| cmurphy | is it normal for other jobs to overtake existing jobs like that? | 21:50 |
| *** slaweq has quit IRC | 21:51 | |
| *** bobh has quit IRC | 21:53 | |
| *** armax has joined #openstack-infra | 21:54 | |
| corvus | cmurphy: the only way something should be able to jump ahead in the queue is if an admin manually promotes a change | 21:56 |
| cmurphy | corvus: is that what happened? | 21:58 |
| corvus | cmurphy: when did you see it at the top of the gate queue? maybe 5 minutes before your question above? 21:44 ish? | 21:58 |
| cmurphy | corvus: yes just a few minutes before i posted that | 21:58 |
| *** bobh has joined #openstack-infra | 21:59 | |
| *** rcernin has joined #openstack-infra | 22:02 | |
| *** bobh has quit IRC | 22:03 | |
| *** armax has quit IRC | 22:04 | |
| *** armax has joined #openstack-infra | 22:04 | |
| *** salv-orlando has joined #openstack-infra | 22:04 | |
| *** armax has quit IRC | 22:04 | |
| *** armax has joined #openstack-infra | 22:05 | |
| *** armax has quit IRC | 22:05 | |
| *** armax has joined #openstack-infra | 22:06 | |
| *** armax has quit IRC | 22:06 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove architecture rules on lshw dependency in ironic-agent https://review.openstack.org/526212 | 22:07 |
| *** d0ugal has quit IRC | 22:08 | |
| cmurphy | corvus: it's possible i was looking at a different change | 22:08 |
| corvus | cmurphy: it looks like it's been behind 538093 and 538066 for a while, but a few mins before your message, the head of the queue and two dependents, also ahead of your change, failed and were removed | 22:11 |
| cmurphy | corvus: okay, I think I was just tracking the wrong change, sorry about that | 22:14 |
| *** slaweq has joined #openstack-infra | 22:20 | |
| *** bobh has joined #openstack-infra | 22:20 | |
| *** d0ugal has joined #openstack-infra | 22:20 | |
| *** bobh has quit IRC | 22:25 | |
| *** olaph1 has joined #openstack-infra | 22:29 | |
| *** olaph has quit IRC | 22:30 | |
| *** slaweq has quit IRC | 22:31 | |
| *** slaweq has joined #openstack-infra | 22:38 | |
| *** slaweq has quit IRC | 22:43 | |
| corvus | np | 22:46 |
| *** olaph1 is now known as olaph | 22:50 | |
| *** slaweq has joined #openstack-infra | 22:53 | |
| *** b_bezak has quit IRC | 22:58 | |
| *** slaweq has quit IRC | 22:58 | |
| *** bobh has joined #openstack-infra | 23:01 | |
| *** olaph has quit IRC | 23:03 | |
| *** quite has quit IRC | 23:05 | |
| *** bobh has quit IRC | 23:05 | |
| *** olaph has joined #openstack-infra | 23:06 | |
| *** quite has joined #openstack-infra | 23:07 | |
| *** olaph1 has joined #openstack-infra | 23:18 | |
| *** olaph has quit IRC | 23:18 | |
| *** bobh has joined #openstack-infra | 23:19 | |
| *** bobh has quit IRC | 23:24 | |
| *** agopi|out has joined #openstack-infra | 23:28 | |
| *** bobh has joined #openstack-infra | 23:29 | |
| *** d0ugal has quit IRC | 23:30 | |
| *** aviau has quit IRC | 23:30 | |
| *** aviau has joined #openstack-infra | 23:32 | |
| *** bobh has quit IRC | 23:33 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Export DIB_ROOT_LABEL from final state https://review.openstack.org/532279 | 23:35 |
| *** dingyichen has joined #openstack-infra | 23:44 | |
| *** lbragstad has quit IRC | 23:45 | |
| *** s-shiono has joined #openstack-infra | 23:46 | |
| *** d0ugal has joined #openstack-infra | 23:46 | |
| *** rcernin has quit IRC | 23:46 | |
| *** rcernin has joined #openstack-infra | 23:50 | |
| *** claudiub|2 has quit IRC | 23:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Don't install dmidecode on Fedora ppc64le https://review.openstack.org/536852 | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!