| *** AndyWojo has joined #openstack-powervm | 00:48 | |
| *** svenkat has quit IRC | 01:07 | |
| *** thorst has joined #openstack-powervm | 01:11 | |
| *** arunmani_ has joined #openstack-powervm | 01:58 | |
| *** thorst has quit IRC | 02:02 | |
| *** thorst has joined #openstack-powervm | 02:29 | |
| *** thorst has quit IRC | 02:30 | |
| *** thorst has joined #openstack-powervm | 02:41 | |
| *** thorst has quit IRC | 02:41 | |
| *** thorst has joined #openstack-powervm | 02:51 | |
| *** thorst has quit IRC | 02:51 | |
| *** edmondsw has joined #openstack-powervm | 03:00 | |
| *** arunmani_ has quit IRC | 03:01 | |
| *** edmondsw has quit IRC | 03:04 | |
| *** thorst has joined #openstack-powervm | 03:22 | |
| *** thorst has quit IRC | 03:40 | |
| *** arunmani_ has joined #openstack-powervm | 03:50 | |
| *** chhavi has joined #openstack-powervm | 04:32 | |
| *** thorst has joined #openstack-powervm | 04:37 | |
| *** thorst has quit IRC | 04:42 | |
| *** arunman has joined #openstack-powervm | 05:32 | |
| *** thorst has joined #openstack-powervm | 05:38 | |
| *** thorst has quit IRC | 05:43 | |
| *** tjakobs has joined #openstack-powervm | 05:56 | |
| *** tjakobs has quit IRC | 06:02 | |
| *** edmondsw has joined #openstack-powervm | 06:36 | |
| *** edmondsw has quit IRC | 06:41 | |
| *** chas has joined #openstack-powervm | 07:26 | |
| *** thorst has joined #openstack-powervm | 07:39 | |
| *** thorst has quit IRC | 07:44 | |
| *** thorst has joined #openstack-powervm | 08:40 | |
| *** thorst has quit IRC | 08:59 | |
| *** chas_ has joined #openstack-powervm | 09:15 | |
| *** k0da has joined #openstack-powervm | 09:17 | |
| *** chas has quit IRC | 09:19 | |
| *** thorst has joined #openstack-powervm | 09:56 | |
| *** thorst has quit IRC | 10:01 | |
| *** thorst has joined #openstack-powervm | 10:57 | |
| *** thorst has quit IRC | 11:02 | |
| *** edmondsw has joined #openstack-powervm | 11:19 | |
| *** thorst has joined #openstack-powervm | 11:41 | |
| *** thorst has quit IRC | 11:45 | |
| *** thorst has joined #openstack-powervm | 12:01 | |
| *** arunman has quit IRC | 12:05 | |
| *** svenkat has joined #openstack-powervm | 12:07 | |
| *** chas_ has quit IRC | 12:25 | |
| *** chas has joined #openstack-powervm | 12:25 | |
| *** chas has quit IRC | 12:31 | |
| *** jwcroppe has quit IRC | 12:36 | |
| openstackgerrit | Merged openstack/networking-powervm master: Fix docstrings for neutron-lib constants https://review.openstack.org/469536 | 12:36 |
|---|---|---|
| *** kylek3h has joined #openstack-powervm | 12:52 | |
| *** jpasqualetto has joined #openstack-powervm | 12:53 | |
| *** jwcroppe has joined #openstack-powervm | 12:54 | |
| *** mdrabe has joined #openstack-powervm | 13:10 | |
| *** chas has joined #openstack-powervm | 13:12 | |
| *** chas has quit IRC | 13:16 | |
| *** chas has joined #openstack-powervm | 13:18 | |
| *** chas has quit IRC | 13:18 | |
| *** chas has joined #openstack-powervm | 13:18 | |
| efried | thorst What actual good do LifecycleEvent notifications do us? | 13:21 |
| efried | From what I can tell, if the state of an instance is changed out of band, nova does things that are at best pointless, but often invasive and just wrong. | 13:23 |
| esberglu | efried: thorst: edmondsw: Okay CI is looking much better since the new novalink packages and the updated drmgr | 13:23 |
| efried | esberglu Cool deal. | 13:24 |
| esberglu | Back to "normal" failure rates. Next we just need to figure out those timeout tests | 13:24 |
| efried | thorst If I shut down the LPAR out of band, nova runs power-off (which turns out to be a no-op, but generates traffic). But if I boot a shut-down LPAR out of band, nova shuts it back down - which seems kind of awful. | 13:28 |
| *** tjakobs has joined #openstack-powervm | 13:29 | |
| thorst | efried: nope | 13:33 |
| thorst | that's the contract with nova | 13:33 |
| thorst | we stick with it even if we dislike it | 13:33 |
| thorst | if we dislike it, we try to change it in core nova | 13:33 |
| thorst | :-) | 13:33 |
| thorst | but the reason for that is really how nova handles libvirt | 13:33 |
| efried | thorst Well, there's a conf option to disable the events entirely. | 13:33 |
| thorst | they destroy and rebuild a VM whenever you power it on. | 13:33 |
| efried | And... we could also just not send them. | 13:33 |
| thorst | I'd rather send them and let the user decide to turn them off. | 13:34 |
| efried | thorst In the case of pvc, does the user control their nova.conf, or does pvc take responsibility for that? | 13:38 |
| thorst | pvc is turnkey...user doesn't go into config files | 13:38 |
| efried | thorst Okay, so pvc could decide to set that conf option. Problem solved. | 13:38 |
| thorst | any deployer could, yep | 13:39 |
| thorst | but that doesn't mean we shouldn't have that in our driver... | 13:39 |
| thorst | :-) | 13:39 |
| edmondsw | esberglu do we set OS_TEST_TIMEOUT and if so where and to what? | 13:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/networking-powervm master: Updated from global requirements https://review.openstack.org/465572 | 13:58 |
| *** arunman has joined #openstack-powervm | 14:03 | |
| esberglu | edmondsw: Not that I know of. I think that would just be something we throw into the local.conf files | 14:08 |
| esberglu | Actually maybe not. Why do you ask? | 14:09 |
| edmondsw | esberglu that controls how long tempest will wait for a test to complete before it raises a timeout | 14:10 |
| edmondsw | esberglu maybe we should try bumping that | 14:10 |
| esberglu | edmondsw: Yeah sure | 14:10 |
| edmondsw | long term I would like to figure out why things are taking so long, of course, and see what we can do to address that | 14:10 |
| edmondsw | what storage driver are we using for something like test_admin_delete_servers_of_others OOT? | 14:11 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events https://review.openstack.org/469982 | 14:11 |
| edmondsw | esberglu https://github.com/openstack/tempest/blob/23aa354f178878ee4d5385db7ce51999726ef8b4/tempest/scenario/test_volume_boot_pattern.py#L29-L31 | 14:14 |
| edmondsw | esberglu we don't control the tests, so we can't add things like that, but we do control what storage driver we use, right? | 14:14 |
| edmondsw | efried your thoughts on this? | 14:16 |
| *** smatzek has joined #openstack-powervm | 14:16 | |
| *** smatzek_ has joined #openstack-powervm | 14:17 | |
| *** smatzek__ has joined #openstack-powervm | 14:18 | |
| efried | edmondsw That would be a stopgap, of course. My fear is that it'll sweep the problem under the rug and we'll never go oil that wheel cause we've muffled the squeaking. | 14:18 |
| edmondsw | efried yeah, totally agree | 14:19 |
| efried | It will also cause me to mix metaphors horribly. | 14:19 |
| edmondsw | :) | 14:19 |
| edmondsw | efried what about the storage driver question? | 14:19 |
| efried | edmondsw I'm not sure what the question was. | 14:21 |
| *** smatzek has quit IRC | 14:21 | |
| edmondsw | what kind of storage are we using OOT for things like test_admin_delete_servers_of_others? | 14:21 |
| edmondsw | is there a faster storage option that we could configure to use instead? | 14:22 |
| *** smatzek_ has quit IRC | 14:22 | |
| edmondsw | efried one of the reasons I was thinking of trying OS_TEST_TIMEOUT bump was to see if it would help at all... i.e., determine whether the tests would still fail or if they're just being too slow for the current timeout | 14:23 |
| *** kriskend has joined #openstack-powervm | 14:23 | |
| edmondsw | or do we already have a sense of that, whether it's a speed vs. hung/blocked issue? | 14:24 |
| efried | edmondsw A fair point. As for which storage driver, pretty sure we only have SSP capability in the CI right now. But that's the fastest one anyway, for pretty much any operation you'd care to name. | 14:26 |
| edmondsw | k | 14:26 |
| edmondsw | I thought that might be the case, but had to ask :) | 14:26 |
| esberglu | efried: edmondsw: Just catching back up. Yeah we just have SSP. As far as speed vs. blocked I don't think we've determined anything | 14:27 |
| esberglu | Unless efried unconvered something I'm not aware of | 14:27 |
| efried | I have not uncovered anything. | 14:28 |
| edmondsw | esberglu so let's try bumping OS_TEST_TIMEOUT and see if that gets rid of, or even impacts, the number of timeouts we're seeing | 14:28 |
| edmondsw | just as a temporary measure | 14:29 |
| edmondsw | efried agreed? | 14:29 |
| esberglu | edmondsw: Sure | 14:29 |
| efried | esberglu Can you do that in a sandboxy way, versus committing a change in the repo? | 14:29 |
| efried | Just want to make sure we don't forget about it. | 14:30 |
| esberglu | efried: Yeah I should be able to | 14:36 |
| efried | cool. | 14:36 |
| *** smatzek__ has quit IRC | 14:37 | |
| esberglu | efried: https://github.com/openstack/tempest/blob/master/tox.ini#L23 | 14:46 |
| esberglu | That line means I can just set OS_TEST_TIMEOUT in /etc/environment and it will then get passed in? | 14:46 |
| efried | esberglu Perhaps? | 14:47 |
| efried | Pretty sure the testr command starts off by outputting the env vars it's using, so you should be able to tell pretty quick. | 14:47 |
| esberglu | efried: Yeah it does. I doubled the timeout from 500s to 1000s for now | 14:47 |
| efried | ight | 14:47 |
| edmondsw | esberglu the default was 500 not 1200? | 14:49 |
| edmondsw | e.g. https://github.com/openstack/tempest/blob/master/tox.ini#L47 | 14:49 |
| esberglu | edmondsw: Yeah I saw that as well. But if you look at the test it fails in 8 min 20 sec (500s) | 14:51 |
| edmondsw | k | 14:51 |
| edmondsw | esberglu maybe from this: https://github.com/openstack/tempest/blob/422ded886232067a393ec8fc4971580525e41ffe/.testr.conf#L4 | 14:52 |
| edmondsw | esberglu or this: https://github.com/openstack/tempest/blob/108ace85d38210d214e0413c76106324f82a809a/tempest/cmd/init.py#L32 | 14:52 |
| esberglu | edmondsw: Yeah I was trying to track down where it was getting set but gave up when I realized it was 500. Overriding in /etc/environment should work | 14:52 |
| thorst | edmondsw: I still owe you a box right? | 14:53 |
| edmondsw | thorst yep | 14:53 |
| efried | And then I owe you a setup tutorial | 14:53 |
| thorst | k. Will PM you the details. It'll need a fresh install. | 14:53 |
| edmondsw | tx | 14:53 |
| efried | thorst Send him the install wiki page too. I can never find the damn thing. | 14:55 |
| thorst | efried: yes sir mr. efried sir | 14:55 |
| efried | thorst (edmondsw) please take a look at https://review.openstack.org/#/c/469982/ and let me know if this is reasonable so I can proceed with UT fixup. | 15:02 |
| thorst | -1 doesn't pass pep8 | 15:02 |
| efried | thorst Yeah, I know, I have made no attempt to do anything other than functional testing (which looks good). | 15:02 |
| thorst | efried: I can't look at anything unless its all green | 15:03 |
| efried | thorst Just looking for buy-in on the high points of the approach. | 15:03 |
| thorst | (I'm kidding) | 15:03 |
| edmondsw | pulling it up | 15:03 |
| efried | thorst edmondsw In live testing, the result of the change is that we don't emit any lifecycle events at all during spawn/destroy (which is good); but we still see a couple of get_instance()s per VM (which I think is better than before, but it would be nice to figure out a way to reduce that). | 15:08 |
| efried | The next step to improve things would be to maintain a cache of LPAR states, which would allow us to skip out before get_instance if it hasn't changed. | 15:09 |
| efried | That's getting real close to the Pareto point, though... | 15:09 |
| efried | Cause I would need to figure out a way to expire entries from the cache. | 15:11 |
| efried | Couldn't find a simple "expiring cache" impl in a standard python lib. | 15:12 |
| efried | Could expire entries by handling DELETED events from powervm; and/or by polling the whole LPAR list periodically. | 15:13 |
| efried | Again, not sure whether the gain is worth all that extra logic. | 15:13 |
| thorst | I have beef with your cache | 15:18 |
| thorst | noted in the review. | 15:19 |
| efried | thorst MY cache? MY cache?? It's YOUR cache! | 15:19 |
| thorst | beef is had | 15:21 |
| *** k0da has quit IRC | 15:43 | |
| efried | thorst Responded to your comments. Verbosely. Pending further discussion, I stand by things like the cache and the task_state stuff. Let me know what you think. | 16:00 |
| *** jwcroppe has quit IRC | 16:01 | |
| *** jwcroppe has joined #openstack-powervm | 16:02 | |
| thorst | efried: I think the part I was missing was that this is a short lived cache. | 16:03 |
| edmondsw | efried didn't get through it all yet, but gave you some comments to start looking at | 16:03 |
| thorst | I assumed it was built when the driver started. | 16:03 |
| efried | thorst Yeah, that would be... bad. | 16:03 |
| efried | esberglu Is this where you were seeing the permission problem? http://184.172.12.213/81/470281/1/check/nova-out-of-tree-pvm/07817a4/console.html | 16:04 |
| *** chas has quit IRC | 16:09 | |
| *** chas has joined #openstack-powervm | 16:10 | |
| efried | edmondsw Thanks, responded. | 16:14 |
| *** chas has quit IRC | 16:14 | |
| *** mdrabe has quit IRC | 16:30 | |
| *** mdrabe has joined #openstack-powervm | 16:44 | |
| *** jwcroppe has quit IRC | 16:58 | |
| *** jwcroppe has joined #openstack-powervm | 17:02 | |
| *** chhavi has quit IRC | 17:04 | |
| esberglu | efried: Yeah it is | 17:19 |
| *** esberglu has quit IRC | 17:24 | |
| *** esberglu has joined #openstack-powervm | 17:24 | |
| *** esberglu has quit IRC | 17:29 | |
| *** esberglu has joined #openstack-powervm | 17:37 | |
| *** dwayne has quit IRC | 17:43 | |
| *** jwcroppe has quit IRC | 17:58 | |
| *** arunman has quit IRC | 18:01 | |
| *** dwayne has joined #openstack-powervm | 18:04 | |
| *** jwcroppe has joined #openstack-powervm | 18:11 | |
| *** chas has joined #openstack-powervm | 18:11 | |
| *** chas has quit IRC | 18:15 | |
| esberglu | efried: edmondsw: thorst: Still hitting the timeouts with the value doubled | 18:32 |
| esberglu | http://184.172.12.213/27/469327/2/silent/neutron-out-of-tree-pvm/47f4fba/powervm_os_ci.html | 18:33 |
| efried | Cool, so most likely points to a real bug. | 18:33 |
| esberglu | What's the longest reasonable time that those tests should take? The timeout is now between 16 and 17 minutes | 18:34 |
| edmondsw | esberglu what efried said. I would think 16-17 min is plenty | 18:38 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events https://review.openstack.org/469982 | 18:41 |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events https://review.openstack.org/469982 | 18:47 |
| *** k0da has joined #openstack-powervm | 18:54 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events https://review.openstack.org/469982 | 19:09 |
| *** kylek3h has quit IRC | 19:11 | |
| *** jwcroppe has quit IRC | 19:12 | |
| *** chas has joined #openstack-powervm | 20:12 | |
| *** chas has quit IRC | 20:16 | |
| openstackgerrit | Eric Fried proposed openstack/nova-powervm master: WIP: Performance improvements for Lifecycle events https://review.openstack.org/469982 | 20:43 |
| *** dwayne has quit IRC | 20:44 | |
| *** kriskend has quit IRC | 20:52 | |
| *** jpasqualetto has quit IRC | 21:10 | |
| *** thorst has quit IRC | 21:11 | |
| *** svenkat has quit IRC | 21:11 | |
| *** thorst has joined #openstack-powervm | 21:15 | |
| *** thorst has quit IRC | 21:19 | |
| *** k0da has quit IRC | 21:30 | |
| *** thorst has joined #openstack-powervm | 21:39 | |
| *** mdrabe has quit IRC | 21:47 | |
| *** thorst has quit IRC | 21:58 | |
| *** edmondsw has quit IRC | 22:03 | |
| *** svenkat has joined #openstack-powervm | 22:04 | |
| *** tjakobs has quit IRC | 22:11 | |
| *** chas has joined #openstack-powervm | 22:12 | |
| *** chas has quit IRC | 22:17 | |
| *** toan has joined #openstack-powervm | 22:33 | |
| *** toanster has quit IRC | 22:40 | |
| *** jwcroppe has joined #openstack-powervm | 22:47 | |
| *** thorst has joined #openstack-powervm | 22:49 | |
| *** thorst has quit IRC | 22:49 | |
| *** thorst has joined #openstack-powervm | 22:55 | |
| *** esberglu has quit IRC | 22:59 | |
| *** esberglu has joined #openstack-powervm | 23:00 | |
| *** esberglu has quit IRC | 23:04 | |
| *** edmondsw has joined #openstack-powervm | 23:11 | |
| *** thorst has quit IRC | 23:14 | |
| *** edmondsw has quit IRC | 23:16 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!