| jeblair | clarkb: looking at http://logs.openstack.org/05/64305/5/check/gate-zuul-python26/fb182f5/ do you have any suggestions about how i could find out which test failed? | 00:21 |
|---|---|---|
| jeblair | clarkb: some of this is coming back to me now -- i think if we use the gentle timeout fixture, it doesn't work, and if we use the non-gentle timeout, we lose info (like the accumulated logs) | 00:23 |
| clarkb | jeblair: looking, I need a break from writing slide stuff | 00:23 |
| clarkb | jeblair: ya, grab the subunit log. testr load that file | 00:25 |
| jeblair | FAIL: tests.test_scheduler.TestScheduler.test_zuul_url_return | 00:25 |
| jeblair | clarkb: is there any chance that's a false reading? i'm surprised by that, because it's such a simple test. | 00:26 |
| clarkb | once you have the log file loaded you can use testr to inspect it | 00:26 |
| clarkb | jeblair: it also see a process return code error (one of the test runners exited with non zero status) | 00:27 |
| *** DennyZhang has joined #openstack-infra | 00:27 | |
| clarkb | jeblair: is it possible taht test was a bystander in a larger fail? | 00:27 |
| jeblair | clarkb: since the non-gentle timeout was set, i'm guessing that's the process return code error; i'm assuming it killed the test running process? | 00:27 |
| clarkb | oh ya it iwll do that | 00:28 |
| jeblair | so does it write "im starting test_zuul_url_return" to the log file before it starts running the test? | 00:29 |
| jeblair | or does it wait till afterwords to write anything? | 00:29 |
| *** rakhmerov has joined #openstack-infra | 00:29 | |
| jeblair | clarkb: can i ask testr to show me all the info for that test run? | 00:30 |
| *** UtahDave has quit IRC | 00:30 | |
| clarkb | jeblair: you can do `testr last` and `testr failing` | 00:31 |
| clarkb | there is a `testr stats` but it doesn't do much | 00:32 |
| jeblair | clarkb: so 'testr last --subunit' looks helpful; though it seems to print info about all tests... | 00:32 |
| jeblair | clarkb: how do i interpret the worker tags? | 00:33 |
| clarkb | jeblair: there is one tag per process eg worker-0 worker-1 worker-2 and so on. All tests run with teh same worker tag ran in the same process | 00:34 |
| *** rakhmerov has quit IRC | 00:34 | |
| jeblair | the possible values seem to be '-worker-0', 'worker-0', '-worker-0 -worker-2', 'worker-0 worker-2', '-worker-0 -worker-3', 'worker-0 worker-3' | 00:34 |
| jeblair | so what does it mean for something to be tagged 'worker0 worker-3' as opposed to just 'worker-0'? what does a '-' at the beginning signify? and why is there no 'worker-1'? | 00:35 |
| clarkb | I do not know. /me grabs the file too to take a look | 00:35 |
| clarkb | jeblair: the -worker seems to be a closing tag for that particular job | 00:36 |
| clarkb | jeblair: I don't see doubled up workers (I am looking at subunit v1 directly, `subunit-2to1 < source_file_from_jenkins > output_file`) | 00:37 |
| jeblair | clarkb: huh. 'testr last --subunit' does the doubled-up. i'll look at it your way. | 00:38 |
| jeblair | clarkb: still no worker-1. | 00:39 |
| clarkb | ya, so that worker must've gotten stuck? | 00:40 |
| jeblair | clarkb: so should i read the first 7 lines of that output as "worker2 started test_layous; test_layouts was successful; worker2 is finished;" ? | 00:40 |
| clarkb | eg it didn't write back any subunit | 00:40 |
| clarkb | jeblair: yes | 00:40 |
| jeblair | clarkb: so shouldn't we see a similar start stanza for worker-1? | 00:40 |
| *** banix has joined #openstack-infra | 00:42 | |
| clarkb | we should, which makes me think it never managed to write one | 00:44 |
| *** xchu has joined #openstack-infra | 00:46 | |
| jeblair | clarkb: switching back to the end of the stream: worker-2's process-returncode error is timestamped 30 seconds after the start and end markers for test_zuul_url_return (despite the fact that they appear subsequently in the log) | 00:48 |
| jeblair | clarkb: the timeout fixture is configured for 30 seconds | 00:48 |
| jeblair | though test_zuul_url_return's start and end markers have the same timestamp. | 00:48 |
| *** dstanek has joined #openstack-infra | 00:50 | |
| *** adalbas has quit IRC | 00:51 | |
| *** Ryan_Lane has quit IRC | 00:51 | |
| clarkb | jeblair: possibly the next test caused the problem then? | 00:53 |
| jeblair | clarkb: but we don't know what that is, right? zuul_url is the last one in the log | 00:54 |
| *** dstanek has quit IRC | 00:55 | |
| jeblair | (i'm running it in a loop locally, fwiw) | 00:55 |
| clarkb | jeblair: right we would have to process of elimination it since subunit didn't manage to output anything before it hang | 00:55 |
| jeblair | clarkb: how? check the subunit log from a successful run since they should be in the same order? | 00:56 |
| *** banix has quit IRC | 00:57 | |
| clarkb | jeblair: they will be the same order on the same host iirc | 00:57 |
| clarkb | so yes | 00:57 |
| clarkb | (there is a feature request to have the test lists get archived too so that this check is easier) | 00:57 |
| jeblair | clarkb: well, these might be on different hosts | 00:58 |
| *** banix has joined #openstack-infra | 01:01 | |
| jeblair | clarkb: hrm, so i'm looking at the log from http://logs.openstack.org/05/64305/2/check/gate-zuul-python26/aa09da1/ | 01:02 |
| jeblair | clarkb: which is an earlier failed run, but it failed because of a test failure, not because of a timeout | 01:02 |
| jeblair | clarkb: and the log looks very similar; same number of tests run, and zuul_url_return is the final one, except that it has output in that log | 01:03 |
| jeblair | also, still no worker-1. | 01:03 |
| *** pcrews_ has quit IRC | 01:05 | |
| openstackgerrit | A change was merged to openstack-infra/config: zuul status: allow changes to be collapsed https://review.openstack.org/63018 | 01:14 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Clean up the copyright header https://review.openstack.org/56783 | 01:15 |
| *** loq_mac has joined #openstack-infra | 01:21 | |
| openstackgerrit | A change was merged to openstack/requirements: Add oauthlib library to replace oauth2 https://review.openstack.org/61198 | 01:21 |
| *** loq_mac has quit IRC | 01:28 | |
| *** rakhmerov has joined #openstack-infra | 01:30 | |
| *** rakhmerov has quit IRC | 01:35 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: zuul status: Collapse changes by default https://review.openstack.org/63019 | 01:50 |
| *** loq_mac has joined #openstack-infra | 01:53 | |
| clarkb | jeblair: fungi: pleia2: zaro: and whoever else might be interested https://review.openstack.org/64321 first draft of my LCA talk | 02:01 |
| clarkb | I am going to go grab dinner now | 02:01 |
| *** loq_mac has quit IRC | 02:05 | |
| *** DennyZha` has joined #openstack-infra | 02:08 | |
| *** DennyZhang has quit IRC | 02:09 | |
| *** praneshp has quit IRC | 02:12 | |
| *** DennyZha` has quit IRC | 02:13 | |
| *** loq_mac has joined #openstack-infra | 02:18 | |
| *** dims has quit IRC | 02:22 | |
| *** loquacities has joined #openstack-infra | 02:23 | |
| *** loq_mac has quit IRC | 02:23 | |
| *** harlowja has quit IRC | 02:24 | |
| *** xchu has quit IRC | 02:24 | |
| *** banix has quit IRC | 02:25 | |
| *** banix has joined #openstack-infra | 02:30 | |
| *** rakhmerov has joined #openstack-infra | 02:31 | |
| *** DennyZha` has joined #openstack-infra | 02:35 | |
| *** rakhmerov has quit IRC | 02:35 | |
| *** dims has joined #openstack-infra | 02:37 | |
| *** DennyZha` has quit IRC | 02:39 | |
| *** krotscheck has quit IRC | 02:40 | |
| *** nati_ueno has quit IRC | 02:44 | |
| *** yamahata has quit IRC | 03:00 | |
| *** tma996 has joined #openstack-infra | 03:04 | |
| *** krotscheck has joined #openstack-infra | 03:06 | |
| *** loquacities has quit IRC | 03:09 | |
| *** senk has joined #openstack-infra | 03:12 | |
| *** harlowja has joined #openstack-infra | 03:15 | |
| *** senk has joined #openstack-infra | 03:15 | |
| *** SergeyLukjanov has joined #openstack-infra | 03:18 | |
| *** loq_mac has joined #openstack-infra | 03:23 | |
| *** reed has quit IRC | 03:24 | |
| *** banix has quit IRC | 03:26 | |
| *** rakhmerov has joined #openstack-infra | 03:32 | |
| *** zhiyan has joined #openstack-infra | 03:36 | |
| *** rakhmerov has quit IRC | 03:36 | |
| *** senk has quit IRC | 03:42 | |
| *** elasticio has joined #openstack-infra | 03:43 | |
| hub_cap | so um, on a new test/dev machine im not getting change-id's after i commit code locally... anyone know _why_ off the top of their head? i have the latest git review and such.. | 03:47 |
| clarkb | you need to setup git review in the repo | 03:54 |
| clarkb | git review -s | 03:54 |
| *** SergeyLukjanov has quit IRC | 03:59 | |
| *** dguitarbite has joined #openstack-infra | 04:06 | |
| *** krotscheck has quit IRC | 04:09 | |
| *** dstanek has joined #openstack-infra | 04:20 | |
| *** jhesketh_ has quit IRC | 04:20 | |
| *** DennyZhang has joined #openstack-infra | 04:31 | |
| *** rakhmerov has joined #openstack-infra | 04:32 | |
| *** jhesketh_ has joined #openstack-infra | 04:33 | |
| lifeless | jeblair: what was teh question you had for me about locking? | 04:34 |
| *** rakhmerov has quit IRC | 04:37 | |
| *** rongze has joined #openstack-infra | 04:45 | |
| *** DennyZhang has quit IRC | 04:46 | |
| *** DennyZhang has joined #openstack-infra | 04:48 | |
| *** boris-42 has joined #openstack-infra | 05:01 | |
| *** DennyZhang has quit IRC | 05:03 | |
| *** DennyZhang has joined #openstack-infra | 05:07 | |
| *** dstanek has quit IRC | 05:15 | |
| *** rakhmerov has joined #openstack-infra | 05:33 | |
| *** rakhmerov has quit IRC | 05:37 | |
| *** DennyZhang has quit IRC | 05:42 | |
| *** tma996 has quit IRC | 05:45 | |
| *** harlowja has quit IRC | 05:50 | |
| *** harlowja has joined #openstack-infra | 05:53 | |
| *** harlowja has quit IRC | 05:53 | |
| *** harlowja has joined #openstack-infra | 05:53 | |
| *** loq_mac has quit IRC | 05:58 | |
| *** rongze has quit IRC | 05:59 | |
| *** tma996 has joined #openstack-infra | 06:16 | |
| *** rakhmerov has joined #openstack-infra | 06:34 | |
| *** rakhmerov has quit IRC | 06:38 | |
| *** chandankumar has quit IRC | 06:46 | |
| *** tma996 has quit IRC | 06:49 | |
| *** zhiyan has left #openstack-infra | 06:51 | |
| *** DennyZhang has joined #openstack-infra | 06:51 | |
| *** harlowja is now known as harlowja_away | 06:53 | |
| *** harlowja_away is now known as harlowja | 07:04 | |
| *** pcrews_ has joined #openstack-infra | 07:09 | |
| *** DennyZhang has quit IRC | 07:10 | |
| *** DennyZhang has joined #openstack-infra | 07:10 | |
| *** praneshp has joined #openstack-infra | 07:12 | |
| *** DennyZhang has quit IRC | 07:21 | |
| *** praneshp has quit IRC | 07:29 | |
| *** rakhmerov has joined #openstack-infra | 07:35 | |
| *** praneshp has joined #openstack-infra | 07:38 | |
| *** rakhmerov has quit IRC | 07:39 | |
| *** tma996 has joined #openstack-infra | 07:46 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:48 | |
| *** praneshp has quit IRC | 07:55 | |
| *** SergeyLukjanov has quit IRC | 07:57 | |
| *** harlowja is now known as harlowja_away | 08:04 | |
| *** alexpilotti has quit IRC | 08:17 | |
| *** tma996 has quit IRC | 08:20 | |
| *** rakhmerov has joined #openstack-infra | 08:35 | |
| *** rakhmerov has quit IRC | 08:40 | |
| *** loq_mac has joined #openstack-infra | 09:10 | |
| *** tma996 has joined #openstack-infra | 09:16 | |
| *** tma996 has quit IRC | 09:20 | |
| *** rossella_s has joined #openstack-infra | 09:26 | |
| *** xchu has joined #openstack-infra | 09:29 | |
| *** rakhmerov has joined #openstack-infra | 09:36 | |
| *** rakhmerov has quit IRC | 09:41 | |
| *** jhesketh_ has quit IRC | 10:05 | |
| *** tma996 has joined #openstack-infra | 10:17 | |
| *** elasticio has quit IRC | 10:26 | |
| *** rakhmerov has joined #openstack-infra | 10:37 | |
| *** rakhmerov has quit IRC | 10:42 | |
| *** loq_mac has quit IRC | 10:42 | |
| *** loq_mac has joined #openstack-infra | 10:42 | |
| *** loq_mac has quit IRC | 10:43 | |
| *** tma996 has quit IRC | 10:51 | |
| *** michchap has quit IRC | 10:52 | |
| *** michchap has joined #openstack-infra | 10:53 | |
| *** rossella_s has quit IRC | 11:00 | |
| *** xchu has quit IRC | 11:17 | |
| *** rakhmerov has joined #openstack-infra | 11:38 | |
| *** rakhmerov has quit IRC | 11:43 | |
| *** tma996 has joined #openstack-infra | 11:47 | |
| *** tma996 has quit IRC | 11:52 | |
| openstackgerrit | Tim Landscheidt proposed a change to openstack-infra/git-review: Retrieve remote pushurl independently of user's locale https://review.openstack.org/64307 | 11:59 |
| *** tma996 has joined #openstack-infra | 12:48 | |
| *** rakhmerov has joined #openstack-infra | 12:57 | |
| *** rakhmerov has quit IRC | 13:01 | |
| *** rakhmerov has joined #openstack-infra | 13:02 | |
| *** tma996 has quit IRC | 13:02 | |
| *** nicedice has quit IRC | 13:02 | |
| *** tma996 has joined #openstack-infra | 13:02 | |
| *** adalbas has joined #openstack-infra | 13:03 | |
| *** nicedice has joined #openstack-infra | 13:03 | |
| *** rakhmerov has quit IRC | 13:06 | |
| *** rakhmerov has joined #openstack-infra | 14:02 | |
| *** CaptTofu has joined #openstack-infra | 14:20 | |
| *** dstanek has joined #openstack-infra | 14:24 | |
| *** paul-- has quit IRC | 14:38 | |
| *** banix has joined #openstack-infra | 14:40 | |
| *** CaptTofu has quit IRC | 14:48 | |
| jeblair | restarting zuul | 15:15 |
| *** CaptTofu has joined #openstack-infra | 15:19 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:24 | |
| *** salv-orlando has quit IRC | 15:24 | |
| *** dstanek has quit IRC | 15:25 | |
| *** pcrews_ has quit IRC | 15:25 | |
| *** dstanek has joined #openstack-infra | 15:29 | |
| *** CaptTofu has quit IRC | 15:36 | |
| * fungi will be internetting from the car again today, so probably disappearing for extended periods as we pass through swampland low population densities | 15:37 | |
| fungi | er, of low | 15:37 |
| *** dguitarbite has quit IRC | 15:43 | |
| *** harlowja_away is now known as harlowja | 15:44 | |
| *** ryanpetrello has quit IRC | 15:53 | |
| jeblair | zuul has restarted | 15:55 |
| *** banix has quit IRC | 15:57 | |
| *** banix has joined #openstack-infra | 15:58 | |
| *** banix has quit IRC | 16:11 | |
| *** ryanpetrello has joined #openstack-infra | 16:16 | |
| *** salv-orlando has joined #openstack-infra | 16:27 | |
| *** dstanek has quit IRC | 16:30 | |
| *** salv-orlando_ has joined #openstack-infra | 16:39 | |
| *** nati_ueno has joined #openstack-infra | 16:40 | |
| *** pcrews_ has joined #openstack-infra | 16:41 | |
| *** salv-orlando has quit IRC | 16:42 | |
| *** salv-orlando_ is now known as salv-orlando | 16:42 | |
| *** nati_ueno has quit IRC | 16:42 | |
| *** jaypipes has quit IRC | 16:47 | |
| *** harlowja is now known as harlowja_away | 16:49 | |
| *** SergeyLukjanov has quit IRC | 16:50 | |
| *** ryanpetrello has quit IRC | 16:57 | |
| *** dstanek has joined #openstack-infra | 17:02 | |
| *** salv-orlando has quit IRC | 17:03 | |
| *** salv-orlando has joined #openstack-infra | 17:05 | |
| *** praneshp has joined #openstack-infra | 17:06 | |
| *** salv-orlando has quit IRC | 17:10 | |
| *** ryanpetrello has joined #openstack-infra | 17:11 | |
| *** salv-orlando has joined #openstack-infra | 17:16 | |
| *** salv-orlando has quit IRC | 17:17 | |
| fungi | clarkb: i like "for great good" (all over yuo screen) | 17:20 |
| *** ryanpetrello has quit IRC | 17:29 | |
| *** banix has joined #openstack-infra | 17:30 | |
| clarkb | fungi: titles are hard and I shamelessly took that from the learn you a haskell book | 17:32 |
| fungi | yeah, i recognized it | 17:34 |
| fungi | also, "...skynet" | 17:34 |
| clarkb | >_> | 17:34 |
| *** tma996 has quit IRC | 17:37 | |
| *** banix has quit IRC | 17:43 | |
| *** rakhmerov has quit IRC | 17:43 | |
| *** che-arne has quit IRC | 17:50 | |
| *** che-arne has joined #openstack-infra | 17:52 | |
| *** SergeyLukjanov has joined #openstack-infra | 18:01 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 18:03 | |
| *** _SergeyLukjanov has quit IRC | 18:07 | |
| *** salv-orlando has joined #openstack-infra | 18:15 | |
| *** SergeyLukjanov has joined #openstack-infra | 18:16 | |
| *** banix has joined #openstack-infra | 18:16 | |
| *** jaypipes has joined #openstack-infra | 18:21 | |
| *** adalbas has quit IRC | 18:32 | |
| *** rakhmerov has joined #openstack-infra | 18:50 | |
| *** banix has quit IRC | 18:50 | |
| *** rakhmerov has quit IRC | 19:23 | |
| *** praneshp_ has joined #openstack-infra | 19:24 | |
| *** praneshp has quit IRC | 19:24 | |
| *** praneshp_ is now known as praneshp | 19:24 | |
| *** pcrews_ has quit IRC | 19:44 | |
| *** Ryan_Lane has joined #openstack-infra | 19:49 | |
| clarkb | zaro: the second scp plugin patch lgtm, we should test it | 19:51 |
| clarkb | on jenkins-dev | 19:51 |
| *** che-arne has quit IRC | 19:54 | |
| *** che-arne has joined #openstack-infra | 19:55 | |
| *** salv-orlando has quit IRC | 20:06 | |
| *** Ryan_Lane has quit IRC | 20:14 | |
| *** rakhmerov has joined #openstack-infra | 20:20 | |
| *** Ryan_Lane has joined #openstack-infra | 20:24 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:37 | |
| *** NikitaKonovalov has quit IRC | 20:37 | |
| *** Ryan_Lane has quit IRC | 20:38 | |
| *** DennyZhang has joined #openstack-infra | 20:46 | |
| *** rakhmerov has quit IRC | 20:54 | |
| *** dripton__ has quit IRC | 21:07 | |
| *** senk has joined #openstack-infra | 21:38 | |
| *** rakhmerov has joined #openstack-infra | 21:50 | |
| *** rakhmerov has quit IRC | 21:59 | |
| *** zehicle_at_dell has quit IRC | 22:00 | |
| *** Ryan_Lane has joined #openstack-infra | 22:10 | |
| *** senk has quit IRC | 22:15 | |
| *** dripton__ has joined #openstack-infra | 22:20 | |
| *** loq_mac has joined #openstack-infra | 22:21 | |
| *** SergeyLukjanov has quit IRC | 22:47 | |
| *** ryanpetrello has joined #openstack-infra | 22:51 | |
| *** banix has joined #openstack-infra | 22:53 | |
| *** DennyZha` has joined #openstack-infra | 22:55 | |
| *** rakhmerov has joined #openstack-infra | 22:56 | |
| *** DennyZhang has quit IRC | 22:57 | |
| *** DennyZha` has quit IRC | 22:59 | |
| *** rakhmerov has quit IRC | 23:09 | |
| *** zehicle_at_dell has joined #openstack-infra | 23:26 | |
| *** loq_mac has quit IRC | 23:29 | |
| *** elasticio has joined #openstack-infra | 23:47 | |
| *** ryanpetrello has quit IRC | 23:52 | |
| *** ryanpetrello has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!