| *** ociuhandu has joined #openstack-infra | 00:00 | |
| *** ociuhandu has quit IRC | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:13 | |
| *** jamesmcarthur has joined #openstack-infra | 00:13 | |
| *** jamesmcarthur has quit IRC | 00:17 | |
| *** jamesmcarthur_ has joined #openstack-infra | 00:17 | |
| *** Lucas_Gray has quit IRC | 00:24 | |
| *** jamesmcarthur_ has quit IRC | 00:31 | |
| *** jamesmcarthur has joined #openstack-infra | 00:32 | |
| *** jamesmcarthur has quit IRC | 00:37 | |
| *** Lucas_Gray has joined #openstack-infra | 00:38 | |
| *** jamesmcarthur has joined #openstack-infra | 00:41 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** jamesmcarthur has joined #openstack-infra | 00:49 | |
| *** jamesmcarthur has quit IRC | 00:54 | |
| *** jamesmcarthur has joined #openstack-infra | 01:09 | |
| *** dave-mccowan has quit IRC | 01:11 | |
| *** jamesmcarthur has quit IRC | 01:38 | |
| *** xek has joined #openstack-infra | 01:38 | |
| *** rfolco has joined #openstack-infra | 01:45 | |
| *** xek has quit IRC | 01:47 | |
| *** rfolco has quit IRC | 02:04 | |
| *** Lucas_Gray has quit IRC | 02:05 | |
| *** jamesmcarthur has joined #openstack-infra | 02:08 | |
| *** jamesmcarthur has quit IRC | 02:15 | |
| *** rfolco has joined #openstack-infra | 02:26 | |
| *** rcernin has joined #openstack-infra | 02:32 | |
| *** rfolco has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #openstack-infra | 03:04 | |
| *** yamamoto has joined #openstack-infra | 03:05 | |
| *** yamamoto has quit IRC | 03:11 | |
| prometheanfire | getting the timeouts again | 03:21 |
|---|---|---|
| prometheanfire | see https://review.opendev.org/709279 just an fyi | 03:43 |
| *** jamesmcarthur has quit IRC | 04:34 | |
| *** yamamoto has joined #openstack-infra | 04:45 | |
| *** imacdonn has quit IRC | 04:46 | |
| *** imacdonn has joined #openstack-infra | 04:46 | |
| prometheanfire | ya, getting a ton of retry_limits | 05:08 |
| *** yamamoto has quit IRC | 05:18 | |
| *** yamamoto has joined #openstack-infra | 05:22 | |
| rm_work | yeah, i came here to say that ^^ | 05:24 |
| rm_work | anyone know what's up? | 05:24 |
| *** yamamoto has quit IRC | 05:32 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** auristor has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** ociuhandu has joined #openstack-infra | 05:36 | |
| *** auristor has joined #openstack-infra | 05:37 | |
| *** ociuhandu has quit IRC | 05:40 | |
| *** yamamoto has joined #openstack-infra | 05:42 | |
| *** yamamoto has quit IRC | 06:09 | |
| *** yamamoto has joined #openstack-infra | 06:24 | |
| *** admcleod has quit IRC | 06:25 | |
| *** yamamoto has quit IRC | 06:31 | |
| *** yamamoto has joined #openstack-infra | 06:32 | |
| *** diablo_rojo has quit IRC | 06:33 | |
| *** admcleod has joined #openstack-infra | 06:45 | |
| *** yamamoto has quit IRC | 07:15 | |
| *** yamamoto has joined #openstack-infra | 07:17 | |
| *** yamamoto has quit IRC | 07:26 | |
| *** yamamoto has joined #openstack-infra | 07:44 | |
| *** xek has joined #openstack-infra | 07:51 | |
| *** Lucas_Gray has joined #openstack-infra | 07:55 | |
| *** ociuhandu has joined #openstack-infra | 08:00 | |
| *** yamamoto has quit IRC | 08:04 | |
| *** ociuhandu has quit IRC | 08:05 | |
| AJaeger | infra-root, we have again lots of RETRY_LIMITs ;( Another OOM? | 08:26 |
| AJaeger | #status notice Zuul is unhappy, lots of jobs are failing with RETRY_LIMIT. Please wait until the issue is found and fixed with rechecks and approvals. | 08:29 |
| openstackstatus | AJaeger: sending notice | 08:29 |
| * AJaeger doubts that any infra-root is around, so let's tell people at least... | 08:30 | |
| -openstackstatus- NOTICE: Zuul is unhappy, lots of jobs are failing with RETRY_LIMIT. Please wait until the issue is found and fixed with rechecks and approvals. | 08:31 | |
| openstackstatus | AJaeger: finished sending notice | 08:33 |
| mnaser | AJaeger: ouch. Anything in specific you can catch from log streaming? | 08:58 |
| mnaser | AJaeger: maybe we can set max servers to 0 to stop new jobs from starting? | 08:59 |
| mnaser | at least we don’t just keep spinning VMs non stop on clouds | 09:00 |
| AJaeger | mnaser: I'm not root - and see discussion from yesterday where we had this already once. | 09:18 |
| AJaeger | mnaser: let's wait... | 09:19 |
| *** rcernin has quit IRC | 09:34 | |
| mnaser | AJaeger: ok so i think it might be one specific executor that's having that issue | 09:46 |
| mnaser | and so if you're unlucky enough all 3 retries go on it | 09:46 |
| frickler | mnaser: AJaeger: infra-root: it looks like ze04 is the culprit, restarting zuul-executor there now | 09:52 |
| frickler | #status log restarted zuul-executor on ze04 due to process pool failure | 09:58 |
| openstackstatus | frickler: finished logging | 09:58 |
| mnaser | thanks frickler | 09:59 |
| *** sshnaidm|afk has quit IRC | 10:10 | |
| *** sshnaidm|afk has joined #openstack-infra | 10:23 | |
| *** dpawlik has joined #openstack-infra | 10:39 | |
| *** ijw has joined #openstack-infra | 10:48 | |
| *** dpawlik has quit IRC | 10:51 | |
| *** ijw has quit IRC | 10:53 | |
| *** sshnaidm|afk has quit IRC | 11:02 | |
| *** sshnaidm|afk has joined #openstack-infra | 11:18 | |
| rm_work | so we good to go now hopefully? | 11:53 |
| rm_work | looks like yes, judging from zuul runs started in the last 1.5h | 11:55 |
| *** nicolasbock has joined #openstack-infra | 12:10 | |
| AJaeger | #status notice The failing Zuul process has been restarted, feel free to recheck. | 12:15 |
| openstackstatus | AJaeger: sending notice | 12:15 |
| AJaeger | thanks, frickler ! | 12:15 |
| -openstackstatus- NOTICE: The failing Zuul process has been restarted, feel free to recheck. | 12:17 | |
| openstackstatus | AJaeger: finished sending notice | 12:18 |
| *** Lucas_Gray has quit IRC | 12:26 | |
| *** Lucas_Gray has joined #openstack-infra | 12:32 | |
| *** adriant has quit IRC | 12:40 | |
| *** adriant has joined #openstack-infra | 12:40 | |
| *** ociuhandu has joined #openstack-infra | 12:52 | |
| *** ociuhandu has quit IRC | 13:01 | |
| *** ociuhandu has joined #openstack-infra | 13:01 | |
| *** Lucas_Gray has quit IRC | 13:03 | |
| *** ociuhandu has quit IRC | 13:12 | |
| *** ociuhandu has joined #openstack-infra | 13:13 | |
| *** Lucas_Gray has joined #openstack-infra | 13:13 | |
| *** ociuhandu has quit IRC | 13:18 | |
| *** ahosam has joined #openstack-infra | 13:48 | |
| *** nicolasbock has quit IRC | 13:51 | |
| *** tosky has joined #openstack-infra | 14:16 | |
| *** Lucas_Gray has quit IRC | 14:39 | |
| *** Lucas_Gray has joined #openstack-infra | 14:43 | |
| *** ociuhandu has joined #openstack-infra | 15:16 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add jobs to build gerrit 3.1 https://review.opendev.org/709295 | 15:19 |
| *** ociuhandu has quit IRC | 15:21 | |
| openstackgerrit | Merged zuul/zuul master: Recover from broken process pool https://review.opendev.org/709307 | 15:45 |
| openstackgerrit | Merged zuul/zuul master: Add foreground option https://review.opendev.org/635649 | 15:53 |
| *** tosky has quit IRC | 16:09 | |
| *** Lucas_Gray has quit IRC | 16:35 | |
| *** ociuhandu has joined #openstack-infra | 16:57 | |
| *** ociuhandu has quit IRC | 16:58 | |
| *** ociuhandu has joined #openstack-infra | 16:58 | |
| *** ociuhandu has quit IRC | 17:03 | |
| openstackgerrit | Andreas Jaeger proposed opendev/publications master: Fix links to doc-contrib-guide https://review.opendev.org/709368 | 17:26 |
| *** tosky has joined #openstack-infra | 17:30 | |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** rfolco has joined #openstack-infra | 17:39 | |
| *** ikla has joined #openstack-infra | 17:43 | |
| ikla | does neutron need to be running on a node that has all network devices? my controller node is a network node as well, but my hypervisors have different networks and interfaces. | 17:46 |
| *** jamesmcarthur has joined #openstack-infra | 17:50 | |
| *** jamesmcarthur has quit IRC | 17:54 | |
| *** jamesmcarthur has joined #openstack-infra | 17:55 | |
| *** jamesmcarthur has quit IRC | 17:56 | |
| *** jamesmcarthur has joined #openstack-infra | 17:57 | |
| fungi | ikla: you might be better off asking in #openstack or #openstack-neutron... what you've found here is the channel where we coordinate the infrastructure our community uses to collaborate and communicate with one another | 18:08 |
| *** jamesmcarthur has quit IRC | 18:12 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Increase zookeeper timeout during tests https://review.opendev.org/707945 | 18:13 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Improve error reporting when pr merge fails https://review.opendev.org/707637 | 18:15 |
| *** jamesmcarthur_ has joined #openstack-infra | 18:16 | |
| *** rfolco is now known as rfolco|afk | 18:21 | |
| *** jamesmcarthur_ has quit IRC | 18:22 | |
| *** jamesmcarthur has joined #openstack-infra | 18:25 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** ijw has joined #openstack-infra | 18:32 | |
| *** rfolco|afk has quit IRC | 18:45 | |
| *** jamesmcarthur has joined #openstack-infra | 19:17 | |
| *** stevebaker has quit IRC | 19:55 | |
| *** stevebaker has joined #openstack-infra | 19:58 | |
| *** jamesmcarthur has quit IRC | 20:09 | |
| *** ChanServ has quit IRC | 20:28 | |
| *** ChanServ has joined #openstack-infra | 20:42 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 20:42 | |
| *** ijw_ has joined #openstack-infra | 20:43 | |
| *** ijw has quit IRC | 20:45 | |
| *** ociuhandu has joined #openstack-infra | 20:49 | |
| ianw | frickler: thanks, did it look like https://storyboard.openstack.org/#!/story/2007290 ? | 21:02 |
| *** ChanServ has quit IRC | 21:10 | |
| *** ChanServ has joined #openstack-infra | 21:17 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 21:17 | |
| *** ociuhandu has quit IRC | 21:19 | |
| *** ociuhandu has joined #openstack-infra | 21:28 | |
| *** dpawlik has joined #openstack-infra | 21:34 | |
| *** ijw_ has quit IRC | 21:49 | |
| *** ijw has joined #openstack-infra | 21:51 | |
| *** ijw has quit IRC | 21:54 | |
| *** ijw has joined #openstack-infra | 21:54 | |
| *** ociuhandu has quit IRC | 22:00 | |
| *** rcernin has joined #openstack-infra | 22:01 | |
| *** dpawlik has quit IRC | 22:09 | |
| *** jamesmcarthur has joined #openstack-infra | 22:20 | |
| *** ociuhandu has joined #openstack-infra | 22:22 | |
| *** ociuhandu has quit IRC | 22:35 | |
| *** ociuhandu has joined #openstack-infra | 22:37 | |
| *** jamesmcarthur has quit IRC | 22:41 | |
| *** ociuhandu has quit IRC | 22:42 | |
| *** ijw_ has joined #openstack-infra | 22:46 | |
| *** ijw has quit IRC | 22:49 | |
| *** jamesmcarthur has joined #openstack-infra | 22:51 | |
| *** dklyle has joined #openstack-infra | 22:52 | |
| *** dklyle has quit IRC | 22:55 | |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** dchen has joined #openstack-infra | 23:30 | |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Uncap hacking https://review.opendev.org/709332 | 23:35 |
| *** ociuhandu has quit IRC | 23:35 | |
| *** ahosam has quit IRC | 23:40 | |
| *** ociuhandu has joined #openstack-infra | 23:45 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add _acme-challenge.docs.opendev.org https://review.opendev.org/709377 | 23:51 |
| *** dchen has quit IRC | 23:55 | |
| *** ociuhandu has quit IRC | 23:55 | |
| *** ociuhandu has joined #openstack-infra | 23:56 | |
| *** dchen has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!