| *** zzzeek has quit IRC | 00:01 | |
| *** zzzeek has joined #openstack-infra | 00:02 | |
| *** tosky has quit IRC | 00:17 | |
| *** dmellado has quit IRC | 01:03 | |
| *** dmellado has joined #openstack-infra | 01:12 | |
| *** dklyle has quit IRC | 01:18 | |
| *** dklyle has joined #openstack-infra | 01:18 | |
| *** yamamoto has joined #openstack-infra | 01:46 | |
| *** yamamoto has quit IRC | 01:50 | |
| *** dchen has quit IRC | 01:58 | |
| *** dchen has joined #openstack-infra | 01:59 | |
| *** ianychoi has joined #openstack-infra | 02:09 | |
| *** zzzeek has quit IRC | 02:20 | |
| *** zzzeek has joined #openstack-infra | 02:21 | |
| *** yamamoto has joined #openstack-infra | 02:27 | |
| *** dmellado has quit IRC | 02:36 | |
| *** ramishra has joined #openstack-infra | 02:37 | |
| *** yamamoto has quit IRC | 02:44 | |
| *** dklyle has quit IRC | 02:51 | |
| *** dmellado has joined #openstack-infra | 03:02 | |
| *** zzzeek has quit IRC | 03:02 | |
| *** zzzeek has joined #openstack-infra | 03:06 | |
| *** yamamoto has joined #openstack-infra | 03:12 | |
| *** yamamoto has quit IRC | 03:16 | |
| *** gyee has quit IRC | 03:33 | |
| *** psachin has joined #openstack-infra | 03:43 | |
| *** jamesmcarthur has quit IRC | 04:12 | |
| *** jamesmcarthur has joined #openstack-infra | 04:13 | |
| *** jamesmcarthur has quit IRC | 04:18 | |
| *** yamamoto has joined #openstack-infra | 04:22 | |
| *** armax has joined #openstack-infra | 04:25 | |
| *** armax has quit IRC | 04:26 | |
| *** dviroel has quit IRC | 04:28 | |
| *** jamesmcarthur has joined #openstack-infra | 04:37 | |
| *** ykarel|away has joined #openstack-infra | 04:40 | |
| *** ykarel|away is now known as ykarel | 04:41 | |
| *** yamamoto has quit IRC | 04:52 | |
| *** yamamoto has joined #openstack-infra | 05:05 | |
| *** yamamoto has quit IRC | 05:06 | |
| *** zzzeek has quit IRC | 05:08 | |
| *** zzzeek has joined #openstack-infra | 05:09 | |
| *** ysandeep|away is now known as ysandeep|rover | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:23 | |
| *** jamesmcarthur has joined #openstack-infra | 05:23 | |
| *** jamesmcarthur has quit IRC | 05:28 | |
| *** jamesmcarthur has joined #openstack-infra | 05:28 | |
| *** yamamoto has joined #openstack-infra | 05:28 | |
| *** yamamoto has quit IRC | 05:28 | |
| *** yamamoto has joined #openstack-infra | 05:28 | |
| *** yamamoto has quit IRC | 05:29 | |
| *** yamamoto has joined #openstack-infra | 05:29 | |
| *** yamamoto has quit IRC | 06:07 | |
| *** yamamoto has joined #openstack-infra | 06:08 | |
| *** yamamoto has quit IRC | 06:08 | |
| *** jamesmcarthur has quit IRC | 06:11 | |
| *** jamesmcarthur has joined #openstack-infra | 06:11 | |
| *** yamamoto has joined #openstack-infra | 06:14 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** yamamoto has joined #openstack-infra | 06:15 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** yamamoto has quit IRC | 06:20 | |
| *** vishalmanchanda has joined #openstack-infra | 06:22 | |
| *** yamamoto has joined #openstack-infra | 06:26 | |
| *** jamesmcarthur has joined #openstack-infra | 06:42 | |
| *** ralonsoh has joined #openstack-infra | 06:49 | |
| *** sboyron_ has joined #openstack-infra | 07:10 | |
| *** jcapitao has joined #openstack-infra | 07:22 | |
| *** jamesmcarthur has quit IRC | 07:29 | |
| *** eolivare has joined #openstack-infra | 07:39 | |
| *** jamesmcarthur has joined #openstack-infra | 07:43 | |
| *** slaweq has joined #openstack-infra | 07:44 | |
| *** ykarel has quit IRC | 07:44 | |
| *** ykarel has joined #openstack-infra | 07:46 | |
| *** jamesmcarthur has quit IRC | 07:52 | |
| *** hashar has joined #openstack-infra | 07:52 | |
| *** rpittau|afk is now known as rpittau | 07:53 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:01 | |
| *** jamesmcarthur has joined #openstack-infra | 08:08 | |
| *** gfidente has joined #openstack-infra | 08:10 | |
| *** nightmare_unreal has joined #openstack-infra | 08:11 | |
| *** andrewbonney has joined #openstack-infra | 08:15 | |
| *** xek has joined #openstack-infra | 08:15 | |
| *** amoralej|off is now known as amoralej | 08:17 | |
| *** jaicaa has quit IRC | 08:27 | |
| *** jpena|off is now known as jpena | 08:29 | |
| *** jaicaa has joined #openstack-infra | 08:30 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| *** ianychoi has quit IRC | 08:33 | |
| *** tosky has joined #openstack-infra | 08:36 | |
| *** ysandeep|lunch is now known as ysandeep|rover | 08:43 | |
| *** jamesmcarthur has joined #openstack-infra | 08:44 | |
| *** ianychoi has joined #openstack-infra | 08:48 | |
| *** dchen has quit IRC | 08:54 | |
| *** dchen has joined #openstack-infra | 08:54 | |
| *** hberaud has quit IRC | 08:58 | |
| *** hberaud has joined #openstack-infra | 08:58 | |
| *** jamesmcarthur has quit IRC | 09:03 | |
| *** yamamoto has quit IRC | 09:03 | |
| *** lucasagomes has joined #openstack-infra | 09:10 | |
| *** jamesmcarthur has joined #openstack-infra | 09:20 | |
| *** yamamoto has joined #openstack-infra | 09:22 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** ykarel is now known as ykarel|lunch | 09:33 | |
| *** ralonsoh has quit IRC | 09:34 | |
| *** ralonsoh has joined #openstack-infra | 09:34 | |
| *** yamamoto has joined #openstack-infra | 09:44 | |
| *** derekh has joined #openstack-infra | 09:45 | |
| *** dciabrin__ has quit IRC | 09:46 | |
| *** dtantsur|afk is now known as dtantsur | 10:00 | |
| *** dciabrin has joined #openstack-infra | 10:02 | |
| *** hashar has quit IRC | 10:07 | |
| *** hashar has joined #openstack-infra | 10:08 | |
| *** ykarel|lunch is now known as ykarel | 10:36 | |
| *** jamesmcarthur has joined #openstack-infra | 11:28 | |
| *** zzzeek has quit IRC | 11:28 | |
| *** zzzeek has joined #openstack-infra | 11:30 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** ociuhandu has joined #openstack-infra | 11:46 | |
| *** dviroel has joined #openstack-infra | 12:01 | |
| *** jcapitao has quit IRC | 12:02 | |
| *** ociuhandu has quit IRC | 12:27 | |
| *** ociuhandu has joined #openstack-infra | 12:27 | |
| *** ociuhandu has quit IRC | 12:28 | |
| *** ociuhandu has joined #openstack-infra | 12:28 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** ociuhandu has quit IRC | 12:41 | |
| *** ociuhandu has joined #openstack-infra | 12:41 | |
| *** ociuhandu has quit IRC | 12:46 | |
| *** yamamoto has quit IRC | 12:50 | |
| *** ociuhandu has joined #openstack-infra | 12:50 | |
| *** yamamoto has joined #openstack-infra | 13:00 | |
| *** yamamoto has quit IRC | 13:00 | |
| *** yamamoto has joined #openstack-infra | 13:00 | |
| *** yamamoto has quit IRC | 13:04 | |
| *** yamamoto has joined #openstack-infra | 13:05 | |
| *** yamamoto has quit IRC | 13:05 | |
| *** yamamoto has joined #openstack-infra | 13:05 | |
| *** yamamoto has quit IRC | 13:06 | |
| *** yamamoto has joined #openstack-infra | 13:07 | |
| *** yamamoto has quit IRC | 13:07 | |
| *** zzzeek has quit IRC | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:07 | |
| *** yamamoto has quit IRC | 13:07 | |
| *** zzzeek has joined #openstack-infra | 13:08 | |
| *** amoralej is now known as amoralej|lunch | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:09 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** zzzeek has quit IRC | 13:21 | |
| *** dmellado has quit IRC | 13:25 | |
| *** dmellado has joined #openstack-infra | 13:25 | |
| *** psachin has quit IRC | 13:26 | |
| *** jpena|lunch is now known as jpena | 13:26 | |
| *** zzzeek has joined #openstack-infra | 13:26 | |
| *** jamesmcarthur has joined #openstack-infra | 13:32 | |
| *** nweinber has joined #openstack-infra | 13:38 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/pbr master: Dropping lower constraints testing https://review.opendev.org/c/openstack/pbr/+/767724 | 13:39 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 13:39 | |
| *** ociuhandu has quit IRC | 13:40 | |
| *** ociuhandu has joined #openstack-infra | 13:41 | |
| *** ociuhandu has quit IRC | 13:52 | |
| *** ociuhandu has joined #openstack-infra | 13:54 | |
| *** amoralej|lunch is now known as amoralej | 13:59 | |
| *** dchen has quit IRC | 14:00 | |
| *** ociuhandu has quit IRC | 14:04 | |
| *** vishalmanchanda has quit IRC | 14:14 | |
| *** ociuhandu has joined #openstack-infra | 14:16 | |
| *** ysandeep|rover is now known as ysandeep|dinner | 14:20 | |
| *** ociuhandu has quit IRC | 14:20 | |
| *** ociuhandu has joined #openstack-infra | 14:22 | |
| *** ykarel_ has joined #openstack-infra | 14:26 | |
| *** ykarel has quit IRC | 14:29 | |
| *** ociuhandu has quit IRC | 14:37 | |
| *** ociuhandu has joined #openstack-infra | 14:40 | |
| *** ociuhandu has quit IRC | 14:47 | |
| *** ociuhandu has joined #openstack-infra | 14:48 | |
| *** ykarel_ has quit IRC | 14:49 | |
| *** hashar is now known as hasharAway | 14:51 | |
| *** dmellado has quit IRC | 14:53 | |
| *** dmellado has joined #openstack-infra | 14:54 | |
| *** ociuhandu has quit IRC | 14:55 | |
| arxcruz|ruck | weshay|ruck: o/ | 14:55 |
| weshay|ruck | zbr, fungi quick question... we have some jobs failing in the gate w/ post_failure.. the upload logs to swift is timing out. Is that something we can fix on our own by reducing the log size? | 14:56 |
| *** ociuhandu has joined #openstack-infra | 14:56 | |
| weshay|ruck | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ebc/774696/1/gate/tripleo-ci-centos-7-containers-multinode/ebc4ab1/job-output.txt | 14:56 |
| weshay|ruck | for example | 14:56 |
| weshay|ruck | or.. is there something else going on? | 14:56 |
| zbr | weshay|ruck: did the log size increase recently? last time when I seen this was caused by some switft issues, not log size. | 14:57 |
| fungi | weshay|ruck: probably, or by increasing the post-run timeout, but i'll see if the output gives me any other suspicions | 14:57 |
| weshay|ruck | zbr, checking | 14:57 |
| weshay|ruck | zbr I do track the log size.. but not a historical trend | 14:58 |
| weshay|ruck | but I should do that | 14:58 |
| weshay|ruck | ya.. noticing post_failures in a lot of non-tripleo jobs as well | 15:00 |
| zbr | weshay|ruck: last execution on collect logs looks like ~9.5min, which a lot but far below 30min limit i know for post. | 15:01 |
| fungi | there may be a problem with one of our swift donors, i'm trying to correlate | 15:01 |
| *** ociuhandu has quit IRC | 15:02 | |
| *** ociuhandu has joined #openstack-infra | 15:04 | |
| *** ysandeep|dinner is now known as ysandeep|rover | 15:04 | |
| weshay|ruck | tox example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_230/774072/1/gate/openstack-tox-linters/2304e12/job-output.txt | 15:05 |
| weshay|ruck | fungi++ | 15:05 |
| weshay|ruck | thanks | 15:05 |
| fungi | yeah, i'm digging in the executor debug log to see if there were any obvious errors emitted during that part of the log upload | 15:06 |
| fungi | that's actually where the console log normally ends because that's when the console log is uploaded | 15:08 |
| *** ricolin has joined #openstack-infra | 15:08 | |
| fungi | the failure was in TASK [get l2 networking] | 15:10 |
| *** yamamoto has joined #openstack-infra | 15:10 | |
| fungi | 2021-02-10 10:52:17,583 DEBUG zuul.AnsibleJob.output: [e: 40d4e56b902c4ce6a3ef1c181c0abf92] [build: ebc4ab1bc33b4d45907a20941a6e5cd9] Ansible output: b'fatal: [secondary]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 127, "stderr": "Shared connection to 149.202.183.167 closed.\\r\\n", "stderr_lines": ["Shared connection to 149.202.183.167 closed."], "stdout": "timeout: failed to | 15:10 |
| fungi | run command \xe2\x80\x98ip\xe2\x80\x99: No such file or directory\\r\\n", "stdout_lines": ["timeout: failed to run command \xe2\x80\x98ip\xe2\x80\x99: No such file or directory"]}' | 15:10 |
| *** dmellado has quit IRC | 15:10 | |
| fungi | so it determined that the node named "secondary" was unreachable | 15:11 |
| weshay|ruck | hrm | 15:11 |
| weshay|ruck | what about a single node case? like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_230/774072/1/gate/openstack-tox-linters/2304e12/job-output.txt | 15:12 |
| *** dmellado has joined #openstack-infra | 15:12 | |
| fungi | weshay|ruck: oh, that's an interesting one, see the Collect tox logs task output | 15:14 |
| weshay|ruck | aye | 15:14 |
| fungi | looks like we have a changed ssh host key on the node | 15:15 |
| fungi | we see that most often when there's ip address contention with other server instances on the network because the same addresses get assigned to multiple systems or deleted virtual machines end up not going away but their addresses are freed in nova/neutron | 15:16 |
| fungi | that one happened in rax-iad while your previous example was in ovh-gra1 | 15:17 |
| fungi | so completely different node providers | 15:17 |
| weshay|ruck | ya.. I see post_failures in rax IAD and ovh GRA1/BHS1 | 15:18 |
| *** hasharAway is now known as hashar | 15:22 | |
| fungi | your examples ran from different zuul executors as well | 15:23 |
| fungi | and several hours apart, and quite a few hours ago | 15:24 |
| fungi | weshay|ruck: discussion is also happening in #opendev but i found one where we got no logs uploaded and it seems the executor hit a socket timeout trying to communicate with storage.bhs.cloud.ovh.net to upload the logs to the swift there | 15:31 |
| *** yamamoto has quit IRC | 15:31 | |
| weshay|ruck | k.. | 15:33 |
| *** jamesmcarthur has joined #openstack-infra | 15:36 | |
| *** ociuhandu has quit IRC | 15:39 | |
| *** jamesmcarthur has quit IRC | 15:42 | |
| *** ociuhandu has joined #openstack-infra | 15:44 | |
| -openstackstatus- NOTICE: Recent POST_FAILURE results from Zuul for builds started prior to 15:47 UTC were due to network connectivity issues reaching one of our log storage providers, and can be safely rechecked | 15:51 | |
| *** dklyle has joined #openstack-infra | 15:52 | |
| *** david-lyle has joined #openstack-infra | 15:54 | |
| *** david-lyle has quit IRC | 15:54 | |
| *** ricolin has quit IRC | 15:56 | |
| *** xek has quit IRC | 15:57 | |
| *** ysandeep|rover is now known as ysandeep|out | 16:09 | |
| *** derekh has quit IRC | 16:10 | |
| *** derekh has joined #openstack-infra | 16:11 | |
| *** gyee has joined #openstack-infra | 16:37 | |
| *** jamesmcarthur has joined #openstack-infra | 16:45 | |
| *** ociuhandu has quit IRC | 16:54 | |
| *** ociuhandu has joined #openstack-infra | 16:55 | |
| *** ricolin has joined #openstack-infra | 16:56 | |
| *** lucasagomes has quit IRC | 16:58 | |
| *** ociuhandu has quit IRC | 17:01 | |
| *** bdodd has quit IRC | 17:01 | |
| *** ociuhandu has joined #openstack-infra | 17:03 | |
| *** ricolin has quit IRC | 17:06 | |
| *** ociuhandu_ has joined #openstack-infra | 17:12 | |
| *** xek has joined #openstack-infra | 17:14 | |
| *** ociuhandu has quit IRC | 17:15 | |
| *** ociuhandu_ has quit IRC | 17:16 | |
| *** jamesmcarthur has quit IRC | 17:20 | |
| *** jamesmcarthur has joined #openstack-infra | 17:20 | |
| *** jdwidari has joined #openstack-infra | 17:24 | |
| *** yamamoto has joined #openstack-infra | 17:29 | |
| *** rlandy is now known as rlandy|biab | 17:30 | |
| *** jdwidari has quit IRC | 17:30 | |
| *** ociuhandu has joined #openstack-infra | 17:38 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/project-config master: Add Backport-Candidate label to Ironic projects https://review.opendev.org/c/openstack/project-config/+/775244 | 17:39 |
| *** ociuhandu has quit IRC | 17:42 | |
| *** rpittau is now known as rpittau|afk | 17:43 | |
| *** yamamoto has quit IRC | 17:45 | |
| *** dtantsur is now known as dtantsur|afk | 17:45 | |
| *** jpena is now known as jpena|off | 17:47 | |
| *** amoralej is now known as amoralej|off | 17:50 | |
| *** jamesmcarthur has quit IRC | 17:53 | |
| *** bdodd has joined #openstack-infra | 17:54 | |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| *** ralonsoh has quit IRC | 17:54 | |
| *** derekh has quit IRC | 17:57 | |
| *** jamesmcarthur has quit IRC | 18:00 | |
| *** gfidente is now known as gfidente|afk | 18:02 | |
| *** dklyle has quit IRC | 18:09 | |
| *** jamesmcarthur has joined #openstack-infra | 18:17 | |
| *** eolivare has quit IRC | 18:19 | |
| *** rlandy|biab is now known as rlandy | 18:37 | |
| *** dklyle has joined #openstack-infra | 18:56 | |
| *** sboyron_ has quit IRC | 19:18 | |
| *** sshnaidm is now known as sshnaidm|afk | 19:19 | |
| *** bdodd has quit IRC | 19:29 | |
| *** yamamoto has joined #openstack-infra | 19:42 | |
| *** andrewbonney has quit IRC | 19:42 | |
| *** nightmare_unreal has quit IRC | 19:47 | |
| *** jamesmcarthur has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #openstack-infra | 19:53 | |
| *** gfidente|afk has quit IRC | 19:57 | |
| *** jamesmcarthur has quit IRC | 19:57 | |
| *** yamamoto has quit IRC | 19:59 | |
| *** jamesmcarthur has joined #openstack-infra | 20:04 | |
| *** ociuhandu has joined #openstack-infra | 21:17 | |
| *** hashar has quit IRC | 21:18 | |
| *** rcernin has joined #openstack-infra | 21:20 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| *** jamesmcarthur has joined #openstack-infra | 21:36 | |
| *** jamesmcarthur has quit IRC | 21:42 | |
| *** nweinber has quit IRC | 21:53 | |
| *** yamamoto has joined #openstack-infra | 21:57 | |
| *** jamesmcarthur has joined #openstack-infra | 21:57 | |
| *** gmann is now known as gmann_afk | 22:10 | |
| *** dklyle has quit IRC | 22:19 | |
| *** dklyle has joined #openstack-infra | 22:19 | |
| *** yamamoto has quit IRC | 22:25 | |
| *** ildikov has quit IRC | 22:40 | |
| *** ildikov has joined #openstack-infra | 22:40 | |
| *** PrinzElvis has quit IRC | 22:41 | |
| *** PrinzElvis has joined #openstack-infra | 22:41 | |
| *** yamamoto has joined #openstack-infra | 22:41 | |
| *** jamesmcarthur has quit IRC | 22:45 | |
| *** jamesmcarthur has joined #openstack-infra | 22:45 | |
| *** jamesmcarthur has quit IRC | 22:49 | |
| *** jamesmcarthur has joined #openstack-infra | 22:49 | |
| *** rlandy has quit IRC | 23:05 | |
| *** slaweq has quit IRC | 23:09 | |
| *** gmann_afk is now known as gmann | 23:11 | |
| *** ociuhandu has quit IRC | 23:19 | |
| dansmith | how do I logstash for only failed jobs? seems like status is now FAILURE,SUCCESS or SUCCESS,FAILURE and if I look for that exact string, I get nothing | 23:21 |
| *** jamesmcarthur has quit IRC | 23:41 | |
| *** jamesmcarthur has joined #openstack-infra | 23:42 | |
| clarkb | dansmith: thats a parsing issue | 23:43 |
| clarkb | somehow the multiline handling is merging together failed and successful logs | 23:43 |
| clarkb | dansmith: if you have an example log line from a job you'd like to find I can take a look and see if there is a way to express it to logstash that works | 23:44 |
| *** jamesmcarthur has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!