| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add support for requiring github pr head status https://review.openstack.org/449390 | 00:01 |
|---|---|---|
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Adds github triggering from status updates https://review.openstack.org/453844 | 00:01 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement pipeline requirement on github reviews https://review.openstack.org/453845 | 00:01 |
| jlk | jeblair: mordred: ^^ Three changes on top of the refactor. You've already seen one of them, but now it's rebased. More to come, but I wanted to get these out there today. | 00:02 |
| mordred | jlk: \o/ | 00:02 |
| mordred | jlk: tis open in my browser | 00:03 |
| * jlk shakes fist at tox-py35 | 00:15 | |
| *** adam_g has quit IRC | 02:00 | |
| *** adam_g has joined #zuul | 02:04 | |
| *** adam_g has quit IRC | 02:25 | |
| *** adam_g has joined #zuul | 02:26 | |
| *** adam_g has quit IRC | 02:33 | |
| *** adam_g has joined #zuul | 02:34 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Ensure PRs arent rejected for stale negative reviews https://review.openstack.org/460700 | 03:07 |
| jlk | That one was a slog :/ | 03:07 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Represent github change ID in status page by PR number https://review.openstack.org/460716 | 03:09 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Comment on PRs if a remote call to merge a change failed https://review.openstack.org/460762 | 03:10 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include exc_info in reporter failure https://review.openstack.org/460765 | 03:11 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add cachecontrol to requests to github https://review.openstack.org/461587 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Include exc_info in reporter failure https://review.openstack.org/460765 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Adds github triggering from status updates https://review.openstack.org/453844 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement github trigger requirement status https://review.openstack.org/463386 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Implement pipeline requirement on github reviews https://review.openstack.org/453845 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Ensure PRs arent rejected for stale negative reviews https://review.openstack.org/460700 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Represent github change ID in status page by PR number https://review.openstack.org/460716 | 04:48 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Comment on PRs if a remote call to merge a change failed https://review.openstack.org/460762 | 04:48 |
| jlk | That's enough for tonight. | 04:49 |
| *** isaacb has joined #zuul | 05:33 | |
| *** isaacb has quit IRC | 06:05 | |
| *** adam_g has quit IRC | 06:52 | |
| *** adam_g has joined #zuul | 06:53 | |
| *** DangerousDaren has joined #zuul | 06:56 | |
| *** jroll has quit IRC | 07:12 | |
| *** adam_g has quit IRC | 07:33 | |
| *** adam_g has joined #zuul | 07:34 | |
| *** isaacb has joined #zuul | 08:10 | |
| *** isaacb has quit IRC | 08:27 | |
| *** Cibo_ has quit IRC | 09:00 | |
| *** hashar has joined #zuul | 09:47 | |
| *** jroll has joined #zuul | 10:25 | |
| *** jkilpatr has quit IRC | 10:38 | |
| *** adam_g has quit IRC | 10:47 | |
| *** adam_g has joined #zuul | 10:49 | |
| *** jkilpatr has joined #zuul | 10:56 | |
| *** hashar has quit IRC | 10:59 | |
| *** adam_g has quit IRC | 11:41 | |
| *** hashar has joined #zuul | 12:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Display command/shell results in a banner https://review.openstack.org/467603 | 12:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 12:47 |
| *** dkranz has joined #zuul | 12:51 | |
| *** Cibo_ has joined #zuul | 12:51 | |
| *** Cibo_ has quit IRC | 13:07 | |
| *** Cibo_ has joined #zuul | 13:25 | |
| SpamapS | jeblair: in my SSH agent patch I'm getting leaked git.Repo's in py27 btw | 13:25 |
| SpamapS | 2017-05-24 06:24:13,582 zuul.test DEBUG Leaked git repo object: <git.Repo "/home/clint/tmp/tmpFUnTtu/zuul-test/executor-git/github.com/org/project/.git"> | 13:25 |
| SpamapS | it adds a small delay as it waits for the ssh-agent to start.. like, 2-3ms delay.. but maybe that's enough to change the race that causes leaky git.Repo's | 13:27 |
| * SpamapS will ponder whilst shuttling children | 13:30 | |
| *** Cibo_ has quit IRC | 13:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 14:18 |
| jeblair | SpamapS: is that with a successful test or a failing one? (leaked git repos on failing tests is fairly normal as parts of zuul are still running) | 14:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 14:37 |
| *** pabelanger has quit IRC | 14:52 | |
| *** pabelanger has joined #zuul | 14:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Put variables into the inventory https://review.openstack.org/467635 | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Write inventory as yaml not ini https://review.openstack.org/467634 | 15:30 |
| *** DangerousDaren has quit IRC | 15:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't spam logs with tracebacks on missing image https://review.openstack.org/467664 | 15:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't spam logs with tracebacks on missing image https://review.openstack.org/467664 | 16:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for defining groups in nodesets https://review.openstack.org/467611 | 16:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 16:26 |
| *** hashar has quit IRC | 16:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't log tracebacks for server-side missing image https://review.openstack.org/467664 | 16:28 |
| SpamapS | jeblair: oh failing, ok | 16:31 |
| pabelanger | SpamapS: when you have time: https://review.openstack.org/#/q/status:open+topic:py3-nodepool :) | 16:42 |
| SpamapS | pabelanger: ACK, I'll hit that soon. | 16:45 |
| *** rcarrillocruz has quit IRC | 17:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Don't log tracebacks for server-side missing image https://review.openstack.org/467664 | 17:25 |
| *** Cibo_ has joined #zuul | 17:46 | |
| *** rcarrillocruz has joined #zuul | 17:47 | |
| jeblair | woohoo! i have the first zuul-cloner test applied to the executor working locally | 17:48 |
| jeblair | i should be able to clean that up and push it this afternoon | 17:49 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Python3: RuntimeError: dictionary changed size during iteration https://review.openstack.org/466049 | 18:01 |
| SpamapS | pabelanger: reviewed | 18:03 |
| pabelanger | thanks, will look shortly | 18:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 18:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Wrap map() in list() for python3 https://review.openstack.org/466069 | 18:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Python3: encode / decode data as utf8 https://review.openstack.org/466065 | 18:17 |
| pabelanger | SpamapS: thanks, I missed that time | 18:18 |
| *** rcarrillocruz has quit IRC | 18:27 | |
| SpamapS | pabelanger: np, will re-review soon | 18:34 |
| *** rcarrillocruz has joined #zuul | 18:36 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Python3: encode / decode data as utf8 https://review.openstack.org/466065 | 19:06 |
| * SpamapS is going in circles on this ssh agent stuff :-P | 19:10 | |
| SpamapS | somehow starting an SSH agent breaks everything :-P | 19:10 |
| *** hashar has joined #zuul | 19:16 | |
| *** hashar has quit IRC | 19:21 | |
| mordred | SpamapS: you know you're going to find it and it's going to be a comma | 19:24 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Wrap map() in list() for python3 https://review.openstack.org/466069 | 19:24 |
| *** hashar has joined #zuul | 19:25 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Support both python2 and python3 virtualenv https://review.openstack.org/467640 | 19:26 |
| *** openstackgerrit has quit IRC | 19:48 | |
| SpamapS | mordred: >:| | 20:00 |
| SpamapS | my bet is on () | 20:01 |
| mordred | SpamapS: or some spaces | 20:01 |
| SpamapS | the annoying thing is there doesn't seem to be any error. Things just stop working right. :-P | 20:05 |
| SpamapS | and when tests are run one-off, they don't fail | 20:06 |
| SpamapS | :-P | 20:06 |
| SpamapS | oh looks like I did find one bug that was making alarm clock fails look weird.. but now it's just that all the tests are timing out :-P | 20:16 |
| SpamapS | deadlocking in threading stuff | 20:24 |
| SpamapS | futex(0x2a3fe60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff^Cstrace: Process 26671 detached | 20:24 |
| *** rcarrillocruz has quit IRC | 20:27 | |
| SpamapS | aha | 20:29 |
| SpamapS | I think it's the __del__ method | 20:29 |
| *** rcarrillocruz has joined #zuul | 20:35 | |
| SpamapS | oh also I think we have some tests that call zuul.cmd code that does basicConfig | 20:39 |
| *** openstackgerrit has joined #zuul | 20:44 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/462712 | 20:44 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 20:44 |
| *** Cibo_ has quit IRC | 20:47 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add SSH Agent Primitives and usage https://review.openstack.org/462712 | 20:57 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add support for bwrap https://review.openstack.org/453851 | 20:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to job repos https://review.openstack.org/467375 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Wait for merger to complete before item is ready https://review.openstack.org/463054 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add default-branch property to projects https://review.openstack.org/467334 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger https://review.openstack.org/461176 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor https://review.openstack.org/461177 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename 'repos' job attribute to 'required-projects' https://review.openstack.org/467376 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add override-branch property to jobs https://review.openstack.org/467775 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Return resulting commits from merger https://review.openstack.org/467776 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use the executor cached repos more often https://review.openstack.org/467777 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Check out the appropriate branch in executor https://review.openstack.org/467778 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Enable test_one_branch cloner test https://review.openstack.org/467779 | 21:25 |
| jeblair | my eyes are bleeding from the 'git commit -p' i just did | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Disable leaked repo check https://review.openstack.org/466810 | 21:29 |
| jeblair | SpamapS, Shrews, clarkb, jlk: ^ i think we're at that point. | 21:29 |
| Shrews | :( | 21:30 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Cleanup failed upload records https://review.openstack.org/467783 | 21:39 |
| clarkb | there is the patch I said I would write earler | 21:39 |
| clarkb | if it looks good in v3 branch I will likely port to master | 21:39 |
| SpamapS | jeblair: oh my! | 21:45 |
| jeblair | SpamapS: nice george takei impression | 21:46 |
| SpamapS | jeblair: impulse power to 50% captain. | 21:48 |
| SpamapS | so .. there's an evil thing happening because of the ssh-agent | 21:49 |
| SpamapS | when tests timeout hard and die.. the ssh-agent isn't getting killed | 21:50 |
| SpamapS | I have about 20 errant ssh-agents sitting around on my laptop :-P | 21:50 |
| SpamapS | (and they're foreground) | 21:50 |
| mordred | SpamapS: ewwww | 21:51 |
| SpamapS | I think process grouping is the answer | 21:51 |
| SpamapS | Indeed, they're all part of the process group of the desktop session manager which is upstart in Ubuntu 16.04 | 21:53 |
| clarkb | systemd | 21:54 |
| SpamapS | yeah that's the answer right? | 21:54 |
| SpamapS | just systemd it up | 21:54 |
| SpamapS | All systemd for justice | 21:54 |
| * SpamapS shakes deadlocked python process violently | 22:02 | |
| SpamapS | TELL ME YOUR SECRETS | 22:02 |
| *** jkilpatr has quit IRC | 22:05 | |
| jeblair | SpamapS: the executor sets up each ansible run in its own process group; i'm surprised ssh-agent isn't automatically part of that | 22:12 |
| *** hashar has quit IRC | 22:13 | |
| jeblair | SpamapS: oh, right, because we want it to span multiple playbooks | 22:13 |
| SpamapS | jeblair: right | 22:13 |
| SpamapS | we're up one level from that | 22:13 |
| jeblair | SpamapS: so yeah, you can probably do something similar | 22:14 |
| SpamapS | jeblair: indeed that's what I think I'll chase | 22:14 |
| jeblair | SpamapS: the os.setsid() call is the magic i believe | 22:14 |
| jeblair | SpamapS: then os.getpgid()/os.killpg() on the kill side | 22:14 |
| jeblair | (both of those are used in the executor for ansible) | 22:15 |
| SpamapS | jeblair: thanks, was looking for os.setpgid | 22:15 |
| SpamapS | jeblair: I'm not sure that will work the same. By putting it in a different pgrp I think it will get adopted by init (though it will get reaped if it exits). Testing now. | 22:19 |
| SpamapS | indeed.. | 22:21 |
| SpamapS | jeblair: what I'm suggesting is that when running tests we need the test runner processes to detach from the controlling terminal, and then ssh-agent to be in the same process group as the executor, so if executor dies, they get HUP'd | 22:22 |
| SpamapS | well not detach | 22:23 |
| SpamapS | they need to be in their own pgrp | 22:23 |
| SpamapS | but then pwn the terminal, except that's going to be.. ugly | 22:23 |
| * SpamapS isn't really sure what to do now | 22:23 | |
| jeblair | SpamapS: oh, it seems that you *don't* want them in their own group then | 22:23 |
| SpamapS | this may only be a concern while testing | 22:24 |
| SpamapS | and there may be better ways to figure this out. | 22:24 |
| jeblair | SpamapS: i wonder what ssh-agent is doing so that it doesn't get killed when the test process dies | 22:24 |
| SpamapS | it doesn't ignore HUP or anything like that | 22:25 |
| SpamapS | just tested that | 22:25 |
| jeblair | SpamapS: i'll also throw this out: it may not be worth engineering clean shutdown for hard test timeouts; it may be better to fix whatever is causing the hard timeout so you're back to zuul shutting down the test cleanly on its own | 22:25 |
| *** jkilpatr has joined #zuul | 22:26 | |
| SpamapS | jeblair: I agree. | 22:26 |
| SpamapS | http://paste.ubuntu.com/24648427/ <-- demonstrates the problem | 22:27 |
| SpamapS | the ssh-agent will live forever | 22:27 |
| SpamapS | or maybe, as long as the desktop session manager lives ;) | 22:27 |
| SpamapS | jeblair: this may just be a minor yak shave since I'm losing my mind trying to figure out why we're hard timing out just because I started a process. | 22:27 |
| * SpamapS tries with --analyze-isolation | 22:29 | |
| * SpamapS has now determined it wasn't his __del__ method.. I think... probably. | 22:33 | |
| *** adam_g has joined #zuul | 22:39 | |
| * SpamapS is now manually killing off subprocesses when they get deadlocked in hopes that testr will consider that a fail and find the bad interaction | 23:09 | |
| SpamapS | damnit | 23:09 |
| SpamapS | tests.unit.test_scheduler.TestScheduler.test_failed_change_at_head_with_queue unknown - no conflicts | 23:09 |
| mordred | SpamapS: I would like to say that you have an excellent skill at uncovering the fascinating race conditions and deadlocks | 23:23 |
| SpamapS | mordred: it's my super-anti-power | 23:26 |
| SpamapS | I'm at the phase of commenting all the code out and uncommenting one logical block at a time. :-P | 23:31 |
| SpamapS | which honestly doesn't help.. it pretty much only breaks if I run ssh-agent :-P | 23:32 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!