| *** saneax is now known as saneax-_-|AFK | 00:34 | |
| *** jamielennox is now known as jamielennox|away | 00:58 | |
| *** jamielennox|away is now known as jamielennox | 01:12 | |
| *** abregman has joined #zuul | 05:52 | |
| *** saneax-_-|AFK is now known as saneax | 06:36 | |
| *** Cibo_ has quit IRC | 08:16 | |
| *** Cibo_ has joined #zuul | 08:17 | |
| *** hashar has joined #zuul | 09:29 | |
| *** Cibo_ has quit IRC | 10:34 | |
| *** Cibo_ has joined #zuul | 10:38 | |
| mordred | SpamapS: there is never enough distracting | 11:59 |
|---|---|---|
| *** Cibo_ has quit IRC | 12:21 | |
| openstackgerrit | Merged openstack-infra/nodepool: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/424974 | 12:21 |
| openstackgerrit | Merged openstack-infra/nodepool: Create mandatory /etc/nodepool for later use https://review.openstack.org/424000 | 12:21 |
| openstackgerrit | Merged openstack-infra/nodepool: Removed redundant 'the' https://review.openstack.org/420525 | 12:21 |
| openstackgerrit | Merged openstack-infra/nodepool: Ensure env-vars are strings in config validate https://review.openstack.org/416427 | 12:24 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable test_dependent_behind_dequeue https://review.openstack.org/413768 | 12:43 |
| openstackgerrit | Merged openstack-infra/zuul: Correct comments in requirement test cases https://review.openstack.org/422900 | 12:54 |
| *** saneax is now known as saneax-_-|AFK | 13:30 | |
| openstackgerrit | Merged openstack-infra/zuul: Add zuul->np test for invalid node type https://review.openstack.org/425686 | 13:36 |
| Shrews | jeblair: mordred: i wish there were some way to "share" that experimental integration test. it pretty much one-way right now | 13:43 |
| *** abregman has quit IRC | 13:47 | |
| *** saneax-_-|AFK is now known as saneax | 14:19 | |
| mordred | Shrews: what do you mean by share it? we should be able to add it to the experimental pipeline of both projects? | 14:20 |
| Shrews | mordred: right now, i don't know if i break any of the tests with a nodepool change until after it merges. would be nice to know beforehand | 14:21 |
| mordred | Shrews: in that case, it might be worthwhile to make the test a nonvoting check job and add it to both zuul and nodepool | 14:23 |
| *** saneax is now known as saneax-_-|AFK | 14:52 | |
| *** hashar has quit IRC | 15:11 | |
| jeblair | Shrews, mordred: it is on both zuul and nodepool | 15:33 |
| jeblair | (and it wouldn't be much of an integration test if it didn't test both things) | 15:34 |
| Shrews | jeblair: oh? so i can 'check experimental' on a nodepool change? | 15:35 |
| jeblair | Shrews: yes | 15:35 |
| Shrews | well then... ignore me | 15:35 |
| morgan | jeblair: wooo i feel good i had an understanding of where we are/were ;) | 16:29 |
| mordred | jeblair: https://review.openstack.org/#/c/426578 from pabelanger if you have a sec - it fixes a bug which is currently causing nodepool dsvm jobs to run long and flirt with the job timeout | 17:25 |
| jeblair | mordred, pabelanger: it would be neat if we could make that safe by default. i forsee this happening a lot otherwise. :) | 17:26 |
| mordred | jeblair: although I'm a little concerned about the unittest failure | 17:26 |
| mordred | jeblair: ++ | 17:26 |
| jeblair | mordred, pabelanger: maybe an env var that lists things that should be unpaused, and if the image is not in that variable, consider it paused. that would be fun bash to write. | 17:27 |
| openstackgerrit | Merged openstack-infra/zuul: Cloner: use zuul_url always when project set https://review.openstack.org/389851 | 17:56 |
| *** jamielennox is now known as jamielennox|away | 17:58 | |
| *** Cibo_ has joined #zuul | 18:12 | |
| *** timrc has quit IRC | 18:25 | |
| *** jlk has quit IRC | 18:26 | |
| *** Shuo has joined #zuul | 18:27 | |
| openstackgerrit | Merged openstack-infra/nodepool: Fix fedora 25 pause bug with devstack https://review.openstack.org/426578 | 18:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add storeNode ZK API method https://review.openstack.org/427353 | 18:44 |
| Shrews | new API method i'll need for an upcoming change ^^^^ | 18:46 |
| Shrews | current body of work is getting pretty large, so trying to break it up | 18:46 |
| mordred | Shrews: I see what I think is an error - although it's not actually an error in that patch | 18:48 |
| *** jlk has joined #zuul | 18:48 | |
| Shrews | mordred: \o/ | 18:48 |
| Shrews | mordred: responded! thx | 18:50 |
| Shrews | mordred: and hopefully my rambling made some sort of sense | 18:52 |
| mordred | Shrews: AH. makes sense - it read to me like a type/mistake, but your rambling makes sense | 18:53 |
| Shrews | mordred: the setter is easy to miss | 18:53 |
| mordred | well - no, I'm just dumb - I saw the setter - but still thought that self.id in the ctr was a typo :) | 18:54 |
| Shrews | it was fixing a mistake where i never set the uninitialized value to None. should have included that in the commit msg i guess | 18:55 |
| Shrews | added another inline comment for others | 18:56 |
| *** Cibo_ has quit IRC | 19:05 | |
| *** Cibo_ has joined #zuul | 19:13 | |
| *** SpamapS has quit IRC | 19:39 | |
| *** SpamapS has joined #zuul | 19:41 | |
| *** Cibo_ has quit IRC | 19:48 | |
| SpamapS | hmm | 20:06 |
| SpamapS | when I try to run a single unit test today, it just fails on zookeeper | 20:06 |
| SpamapS | http://paste.openstack.org/show/597083/ | 20:07 |
| SpamapS | Shrews: ^ jeblair ? | 20:07 |
| Shrews | SpamapS: did you start a zookeeper? | 20:07 |
| SpamapS | uh, no? | 20:07 |
| SpamapS | tox needs a zk running now? | 20:07 |
| Shrews | yeah | 20:08 |
| SpamapS | oh that's silly. We can just start one in the test suite as a test resource. | 20:08 |
| jeblair | either locally or set NODEPOOL_ZK_HOST | 20:08 |
| SpamapS | on a random port | 20:08 |
| Shrews | SpamapS: we used to do that | 20:08 |
| SpamapS | but? | 20:09 |
| Shrews | SpamapS: this docker image works lovely for me: https://hub.docker.com/r/jplock/zookeeper/ | 20:09 |
| Shrews | iirc, turned out to be a pain. and it wasn't how we (mostly I, at the time) was testing it | 20:10 |
| jeblair | yeah, ensuring it was started correctly was finicky, and new zk versions made it worse | 20:11 |
| SpamapS | bummer | 20:11 |
| SpamapS | we had very few problems back in the juju days doing it that way | 20:11 |
| SpamapS | (the main problem was packaging zookeeper, which we did. ;) | 20:11 |
| jeblair | i just started running one locally on my workstation (which is what the unit test nodes do too). Shrews is happy with the docker solution. and it does use chroots, so it doesn't muck with anything else in the zookeeper. | 20:12 |
| Shrews | docker start zk ; tox -epy27 FTW | 20:13 |
| SpamapS | I'm having trouble getting it running on my WS | 20:13 |
| SpamapS | docker is borked because I'm using an rc build of the kernel | 20:13 |
| SpamapS | and apt-get install zookeeper didn't seem to start a zookeeper | 20:14 |
| Shrews | eww | 20:14 |
| jeblair | SpamapS: there's a second package for the init script | 20:14 |
| jeblair | SpamapS: zookeeperd | 20:14 |
| SpamapS | zookeeperd | 20:14 |
| SpamapS | haha this is funny | 20:14 |
| SpamapS | I believe I argued for that split. :) | 20:14 |
| jeblair | SpamapS: yeah, i kinda like it :) | 20:14 |
| SpamapS | I argued for it because juju ran zookeeper on its own, and I didn't want zk always running on my laptop. ;) | 20:15 |
| Shrews | yep. exactly why i use the docker solution | 20:15 |
| SpamapS | I can always just apt-get remove it :) | 20:15 |
| Shrews | i have sparse amounts of memory as it is | 20:15 |
| SpamapS | Shrews: yeah normally I'd prefer that, but I have blocked myself by trying to track down intel graphics driver bugs for the kernel team. ;) | 20:15 |
| SpamapS | Oh I have plenty of memory, but zk will trash your SSD | 20:16 |
| SpamapS | it's always writing checkpoints | 20:16 |
| SpamapS | and keeping your disk from sleeping | 20:16 |
| SpamapS | not exactly something they designed with laptops in mind :) | 20:16 |
| jeblair | SpamapS: maybe it can be configured to write checkpoints to a tmpfs? | 20:17 |
| jeblair | SpamapS: related: export ZUUL_TEST_ROOT=/tmpfs | 20:17 |
| jeblair | SpamapS: is in my .bashrc | 20:17 |
| SpamapS | that would be ideal yeah | 20:17 |
| jeblair | SpamapS: (so the git repo thrashing happens there) | 20:17 |
| SpamapS | yeah I took your advice on that one and mounted a tmpfs for just that reason | 20:17 |
| SpamapS | shaved about 10s off the run | 20:17 |
| jeblair | oh good i'm repeating myself :) | 20:17 |
| jeblair | gotta run grab lunch now | 20:18 |
| * Shrews images jeblair running down a gazelle for lunch like a cheetah | 20:19 | |
| Shrews | imagines* | 20:19 |
| Shrews | imagins*? gah | 20:19 |
| Shrews | me no spelly goodly | 20:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Start nodepool in the test-config phase https://review.openstack.org/427390 | 20:21 |
| mordred | jeblair: ^^ | 20:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Replace NodeRequestWorker with NodeRequestHandler https://review.openstack.org/427393 | 20:24 |
| Shrews | ah, wunderbar! check experimental does work for ^^^. bonus that i didn't break the one passing test | 20:31 |
| mordred | Shrews: \o/ | 20:33 |
| *** Shuo has quit IRC | 21:02 | |
| mordred | Shrews, jeblair: nodepool test still failing - but I don't see any errors in the logs :( http://logs.openstack.org/90/427390/1/check/gate-dsvm-nodepool/9bdc346/logs/screen-nodepool.txt.gz | 21:22 |
| Shrews | mordred: uh, hrm | 21:25 |
| Shrews | TASK [copy console log] <--- why is that the last line of ansible_log.txt ? | 21:25 |
| mordred | Shrews: because it can't log anything else to the file after it has copied the log that it logs to? | 21:26 |
| Shrews | i don't understand why that errored. weird | 21:28 |
| mordred | me either. the nodepool log shows 2017-01-31 21:19:39,573 DEBUG nodepool.NodePool: Deficit: ubuntu-trusty: 0 (start: 0 min-ready: 1 ready: 1 capacity: 4) | 21:34 |
| mordred | but then nodepool list is showing the node in building state | 21:35 |
| jeblair | http://logs.openstack.org/90/427390/1/check/gate-dsvm-nodepool/9bdc346/logs/screen-nodepool.txt.gz#_2017-01-31_20_55_46_186 | 21:38 |
| jeblair | that ssh warning seems to be the last interaction with the node | 21:38 |
| jeblair | i'm guessing the ssh connectivity test never finished? | 21:38 |
| mordred | jeblair: well, at least it was able to connect to the node | 21:40 |
| mordred | :) | 21:40 |
| jeblair | mordred, Shrews: i would have expected some sort of timeout message afterwords though; i think we only wait a few mins? | 21:42 |
| jeblair | though maybe it got caught it some unbreakable paramiko thing | 21:42 |
| mordred | jeblair: I'm quite tempted to make a patch that adds in a ton more logging just to see where it's going south | 21:43 |
| jeblair | ++ | 21:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: DNM - Print debugging ftw https://review.openstack.org/427471 | 21:47 |
| mordred | jeblair: the test-config patch does seem to have fixed the nova exception we were getting earlire | 21:48 |
| *** jamielennox|away is now known as jamielennox | 22:15 | |
| *** Shuo has joined #zuul | 22:15 | |
| *** jeblair has quit IRC | 22:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Revert "Create mandatory /etc/nodepool for later use" https://review.openstack.org/427486 | 22:19 |
| *** jeblair has joined #zuul | 22:22 | |
| mordred | jeblair: welcome back! I have a patch up to revert a patch that was recently landed in nodepool ... as well as a held node: 2001:4800:1ae1:18:f816:3eff:fef9:9b8f in case you care to look | 22:23 |
| mordred | jeblair: Revert "Create mandatory /etc/nodepool for later use" https://review.openstack.org/427486 | 22:23 |
| jeblair | yeah, there's a reason why we don't do that | 22:33 |
| jeblair | nodepool user doesn't have perms for that | 22:33 |
| jeblair | and who says the nodepool-base element is optional? | 22:33 |
| jeblair | nodepool doesn't care about what's on the image except: (a) it can log into the server. (b) it can write to /etc/nodepool | 22:35 |
| jeblair | that's definitely a requirement | 22:35 |
| jeblair | pabelanger, mordred: ^ fyi | 22:35 |
| mordred | jeblair: nod. | 22:35 |
| jeblair | mordred: thanks for finding that and the fix. can we add a comment there so that people won't be tempted to do that again | 22:37 |
| mordred | jeblair: yah | 22:38 |
| mordred | jeblair: I'm going to do that as follow up since I already have a couple of depends-on things pointed at that patch and there's no real need for more churn | 22:38 |
| jeblair | mordred: ok | 22:39 |
| jeblair | mordred: are you still waiting on test results to confirm that? | 22:40 |
| mordred | jeblair: I'm not. I'm fairly darned confident | 22:40 |
| jeblair | mordred: er, to rephrase, tests that would confirm that haven't returned yet, right? :) | 22:40 |
| mordred | jeblair: I am tailing the nodepool log on the bad node - last print was 2017-01-31 22:13:06,026 INFO nodepool.NodeLauncher: Creating config dir | 22:41 |
| mordred | jeblair: and have not gotten "Created config dir" - and it's been there for about 15 minutes | 22:41 |
| jeblair | mordred: yeah, i agree, i expect the sudo to hand because sudo over ssh is hard :) | 22:41 |
| jeblair | sudo to *hang* | 22:41 |
| mordred | jeblair: but yes - the tests that confirm the revert are still running :) | 22:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Add comment explaining test-not-create behavior https://review.openstack.org/427491 | 22:43 |
| mordred | jeblair: thereyago | 22:43 |
| jeblair | mordred: cool, should at least make us think twice | 22:43 |
| mordred | jeblair: so that one and the two before it should make everything happy again | 22:43 |
| jeblair | mordred: i also added some thoughts to the original change: https://review.openstack.org/424000 | 22:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Run pre and post playbooks https://review.openstack.org/427494 | 22:50 |
| jeblair | mordred: ^ let me know what you think of that. i think that's about ready to work on adding the secure ansible module stuff from v2.5 back in | 22:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Run pre and post playbooks https://review.openstack.org/427494 | 22:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Run pre and post playbooks https://review.openstack.org/427494 | 22:59 |
| *** saneax-_-|AFK is now known as saneax | 23:01 | |
| jeblair | mordred: the main nodepool dsvm job passed, but not the -src one... is that expected? | 23:12 |
| jeblair | http://logs.openstack.org/86/427486/1/check/gate-dsvm-nodepool-src-nv/e0e63b6/logs/devstacklog.txt.gz#_2017-01-31_22_36_48_893 | 23:12 |
| jeblair | mordred: oh, i bet the -src job needs the same service enablement change as the main one | 23:16 |
| openstackgerrit | Merged openstack-infra/nodepool: Start nodepool in the test-config phase https://review.openstack.org/427390 | 23:17 |
| jeblair | remote: https://review.openstack.org/427503 Update nodepool-src-dsvm to disable services | 23:19 |
| jeblair | mordred: ^ | 23:19 |
| jeblair | mordred: i've approved the other changes based on that explanation :) | 23:19 |
| openstackgerrit | Merged openstack-infra/nodepool: Revert "Create mandatory /etc/nodepool for later use" https://review.openstack.org/427486 | 23:22 |
| openstackgerrit | Merged openstack-infra/nodepool: Add comment explaining test-not-create behavior https://review.openstack.org/427491 | 23:22 |
| mordred | jeblair: oh bother. thank you | 23:25 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!