| *** matsuhashi has joined #openstack-qa | 00:27 | |
| *** mkollaro has quit IRC | 00:31 | |
| *** nosnos has joined #openstack-qa | 00:34 | |
| *** Lingxian has joined #openstack-qa | 00:48 | |
| *** jamielennox is now known as jamielennox|away | 01:06 | |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use nova.conf for auth_token configs. https://review.openstack.org/52258 | 01:07 |
|---|---|---|
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use nova.conf for auth_token configs. https://review.openstack.org/52258 | 01:10 |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use cinder.conf for auth_token configs. https://review.openstack.org/52259 | 01:11 |
| *** openstackrecheck has quit IRC | 01:19 | |
| *** openstackrecheck has joined #openstack-qa | 01:19 | |
| *** yfujioka has joined #openstack-qa | 01:25 | |
| *** mriedem has joined #openstack-qa | 01:33 | |
| openstackgerrit | wanghao proposed a change to openstack/tempest: add negative test cases for create snapshots: https://review.openstack.org/50217 | 01:36 |
| *** dims has joined #openstack-qa | 01:39 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Add missing comma to REQUIRED_FILES list. https://review.openstack.org/52264 | 01:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add missing comma to REQUIRED_FILES list. https://review.openstack.org/52264 | 01:45 |
| *** openstackrecheck has quit IRC | 01:48 | |
| *** openstackrecheck has joined #openstack-qa | 01:48 | |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use nova.conf for auth_token configs. https://review.openstack.org/52258 | 01:48 |
| *** dims has quit IRC | 01:52 | |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use cinder.conf for auth_token configs. https://review.openstack.org/52259 | 01:53 |
| *** SergeyLukjanov has joined #openstack-qa | 01:58 | |
| *** SergeyLukjanov has quit IRC | 02:35 | |
| *** kong has joined #openstack-qa | 02:52 | |
| *** Lingxian has quit IRC | 02:52 | |
| *** mriedem has quit IRC | 03:23 | |
| *** Madan has joined #openstack-qa | 03:24 | |
| *** mriedem has joined #openstack-qa | 03:24 | |
| *** raies has quit IRC | 03:37 | |
| *** Lingxian has joined #openstack-qa | 03:38 | |
| *** kong has quit IRC | 03:38 | |
| *** kong has joined #openstack-qa | 03:39 | |
| *** Lingxian has quit IRC | 03:39 | |
| *** Lingxian has joined #openstack-qa | 03:45 | |
| *** kong has quit IRC | 03:45 | |
| *** Lingxian has quit IRC | 03:46 | |
| *** Lingxian has joined #openstack-qa | 03:46 | |
| *** kong has joined #openstack-qa | 03:49 | |
| *** Lingxian has quit IRC | 03:49 | |
| *** kong has quit IRC | 03:50 | |
| *** kong has joined #openstack-qa | 03:53 | |
| *** kong has quit IRC | 03:53 | |
| *** kong has joined #openstack-qa | 03:54 | |
| *** HenryG has quit IRC | 03:54 | |
| openstackgerrit | wanghao proposed a change to openstack/tempest: add positive tests for volume https://review.openstack.org/50813 | 04:00 |
| openstackgerrit | huangtianhua proposed a change to openstack/tempest: add role negative tests https://review.openstack.org/51107 | 04:04 |
| *** yfujioka has quit IRC | 04:05 | |
| *** kong has quit IRC | 04:18 | |
| *** Lingxian has joined #openstack-qa | 04:20 | |
| *** yfried_mob has joined #openstack-qa | 04:24 | |
| *** kong has joined #openstack-qa | 04:26 | |
| *** Lingxian has quit IRC | 04:26 | |
| *** giulivo has quit IRC | 04:28 | |
| *** yfried_mob has quit IRC | 04:29 | |
| *** kong has quit IRC | 04:31 | |
| *** yfried has joined #openstack-qa | 04:31 | |
| *** Lingxian has joined #openstack-qa | 04:31 | |
| *** mriedem has quit IRC | 04:35 | |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Separate negative tests api/network/security_groups https://review.openstack.org/51457 | 04:44 |
| *** Lingxian has quit IRC | 04:44 | |
| *** Lingxian has joined #openstack-qa | 04:47 | |
| *** prekarat has joined #openstack-qa | 04:53 | |
| *** Lingxian has quit IRC | 05:00 | |
| *** Lingxian has joined #openstack-qa | 05:03 | |
| *** kong has joined #openstack-qa | 05:10 | |
| *** Lingxian has quit IRC | 05:10 | |
| *** kong has quit IRC | 05:14 | |
| *** Lingxian has joined #openstack-qa | 05:15 | |
| *** kong has joined #openstack-qa | 05:19 | |
| *** Lingxian has quit IRC | 05:19 | |
| *** DennyZhang has joined #openstack-qa | 05:20 | |
| *** yfried_mob has joined #openstack-qa | 05:21 | |
| *** kong has quit IRC | 05:21 | |
| *** Lingxian has joined #openstack-qa | 05:22 | |
| *** kong has joined #openstack-qa | 05:23 | |
| *** sth has joined #openstack-qa | 05:25 | |
| *** sth has joined #openstack-qa | 05:25 | |
| *** sthaha has quit IRC | 05:26 | |
| *** Lingxian has quit IRC | 05:27 | |
| *** Lingxian has joined #openstack-qa | 05:29 | |
| *** kong has quit IRC | 05:29 | |
| *** Lingxian has quit IRC | 05:30 | |
| *** DennyZhang has quit IRC | 05:31 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Enable run_ssh and add floatingip + keypair auth https://review.openstack.org/50337 | 05:31 |
| *** Lingxian has joined #openstack-qa | 05:31 | |
| *** DennyZhang has joined #openstack-qa | 05:31 | |
| *** kong has joined #openstack-qa | 05:36 | |
| *** Lingxian has quit IRC | 05:36 | |
| *** DennyZhang has quit IRC | 05:38 | |
| *** DennyZhang has joined #openstack-qa | 05:38 | |
| *** yfried_mob has quit IRC | 05:41 | |
| *** Lingxian has joined #openstack-qa | 05:41 | |
| *** kong has quit IRC | 05:41 | |
| *** kong has joined #openstack-qa | 05:47 | |
| *** Lingxian has quit IRC | 05:47 | |
| *** kong has quit IRC | 05:53 | |
| *** kong has joined #openstack-qa | 05:54 | |
| *** DennyZhang has quit IRC | 05:57 | |
| *** Lingxian has joined #openstack-qa | 05:58 | |
| *** kong has quit IRC | 06:00 | |
| *** kong has joined #openstack-qa | 06:05 | |
| *** Lingxian has quit IRC | 06:05 | |
| *** Lingxian has joined #openstack-qa | 06:07 | |
| *** kong has quit IRC | 06:07 | |
| *** kong has joined #openstack-qa | 06:12 | |
| *** Lingxian has quit IRC | 06:12 | |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: VPNaas vpnservice test cases https://review.openstack.org/47339 | 06:12 |
| *** kong has quit IRC | 06:15 | |
| *** raies has joined #openstack-qa | 06:20 | |
| *** afazekas has joined #openstack-qa | 06:21 | |
| *** sth has quit IRC | 06:26 | |
| *** rdekel has joined #openstack-qa | 06:43 | |
| *** rdekel has left #openstack-qa | 06:43 | |
| *** Dharmit has joined #openstack-qa | 07:03 | |
| *** jlibosva has joined #openstack-qa | 07:08 | |
| kashyap | afazekas, Heya, have you tried snapshots (nova image create) lately? | 07:23 |
| kashyap | A quick question - from an investigation few months ago, I found this how it works - http://kashyapc.wordpress.com/2013/03/11/openstack-nova-image-create-under-the-hood/ | 07:23 |
| kashyap | afazekas, What I'm wondering is, at *what* point the guest is restarted (from the 'virsh managedsave' state - performed as the first step from the above post) | 07:24 |
| kashyap | Maybe I should ask on #openstack-nova too :) | 07:25 |
| afazekas | http://www.fpaste.org/47416/38199518/ | 07:33 |
| afazekas | kashyap: you can ask the strace | 07:33 |
| afazekas | you can trace the libvirtd in a similar way | 07:36 |
| openstackgerrit | ChenZheng proposed a change to openstack/tempest: Test for the agent management extension API https://review.openstack.org/48421 | 07:52 |
| raies | afazekas: hi please check why it is getting failed ? http://logs.openstack.org/39/47339/17/check/check-tempest-devstack-vm-neutron/5a4de39/testr_results.html.gz | 07:58 |
| *** ivanzhu has joined #openstack-qa | 07:59 | |
| *** jhenner has joined #openstack-qa | 08:00 | |
| afazekas | http://logs.openstack.org/39/47339/17/check/check-tempest-devstack-vm-neutron/5a4de39/logs/tempest.txt.gz#_2013-10-17_06_25_53_427 | 08:01 |
| raies | ok got it | 08:02 |
| raies | thanks :) | 08:02 |
| kashyap | afazekas, Hey, I was away for lunch. Looking... | 08:04 |
| *** ivanzhu has quit IRC | 08:05 | |
| *** yassine has joined #openstack-qa | 08:05 | |
| *** ivanzhu has joined #openstack-qa | 08:06 | |
| *** dguitarbite has quit IRC | 08:09 | |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: VPNaas vpnservice test cases https://review.openstack.org/47339 | 08:12 |
| *** fbo_away is now known as fbo | 08:14 | |
| *** jhenner has quit IRC | 08:17 | |
| kashyap | afazekas, Ok, thanks. I'm buried in a couple of other things, and have to travel in 2 days, so I'm in a super rush mode :) | 08:20 |
| *** jlibosva has quit IRC | 08:24 | |
| *** jlibosva has joined #openstack-qa | 08:25 | |
| *** giulivo has joined #openstack-qa | 08:33 | |
| *** gszasz has joined #openstack-qa | 08:34 | |
| *** marun has joined #openstack-qa | 08:36 | |
| *** yfujioka has joined #openstack-qa | 08:45 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Enable run_ssh and add floatingip + keypair auth https://review.openstack.org/50337 | 08:46 |
| afazekas | mkoderer: https://review.openstack.org/#/c/52145/ | 08:48 |
| afazekas | https://bugs.launchpad.net/tempest/+bug/1218391 | 08:48 |
| afazekas | http://status.openstack.org/rechecks/ 121 recheck | 08:48 |
| afazekas | giulivo: ^^ | 08:49 |
| giulivo | thanks afazekas | 08:54 |
| giulivo | on a side node, yesterday I went trough a few additions of tests requiring admin privileges | 08:55 |
| giulivo | as a general rule, do we want those to be in the admin/ subdirectory? | 08:55 |
| *** jlibosva has quit IRC | 09:09 | |
| *** jlibosva has joined #openstack-qa | 09:10 | |
| afazekas | giulivo: I do not like the admin directory :), but generally the tests which requires admin privileges (per the default policy) | 09:14 |
| giulivo | afazekas, I stepped into a case where a guy was just switching the existing test class from base to baseAdmin and adding a few admin tests | 09:15 |
| giulivo | but didn't know if that was okay to do or if we should instead suggest to create a new file in the admin subdir | 09:15 |
| *** jlibosva1 has joined #openstack-qa | 09:20 | |
| mkoderer | afazekas: ok already merged, any other important reviews? | 09:20 |
| * mkoderer is currently quite busy | 09:20 | |
| *** jlibosva has quit IRC | 09:21 | |
| afazekas | mkoderer: For me it is important: https://review.openstack.org/#/c/50337/ | 09:26 |
| mkoderer | afazekas: ok cool I will have a look | 09:27 |
| *** yfried_mob has joined #openstack-qa | 09:29 | |
| afazekas | giulivo: I am not sure, may be better to have it in the admin dir. but IMHO it is not a critical issue | 09:31 |
| afazekas | giulivo: may be you can ask it on the today meeting | 09:31 |
| giulivo | okay not really worth a topic per se I think, but I'll try to ask | 09:32 |
| *** safchain has joined #openstack-qa | 09:35 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Enable run_ssh and add floatingip + keypair auth https://review.openstack.org/50337 | 09:36 |
| *** jhenner has joined #openstack-qa | 09:37 | |
| *** yfried_mob has quit IRC | 09:41 | |
| openstackgerrit | A change was merged to openstack/tempest: Fixing ImagesOneServerTestXml issue https://review.openstack.org/52145 | 09:42 |
| *** prekarat has quit IRC | 10:00 | |
| giulivo | anyone expert about what volume 'roll detaching' is supposed to achieve? | 10:02 |
| *** yfujioka has quit IRC | 10:03 | |
| giulivo | looks like it just brings back the volume in in-use is it for rollback? | 10:06 |
| *** ivanzhu has quit IRC | 10:10 | |
| *** yfried_mob has joined #openstack-qa | 10:25 | |
| afazekas | giulivo: https://review.openstack.org/#/c/11713/ | 10:31 |
| giulivo | afazekas, thanks! | 10:33 |
| giulivo | wait_for_volume only seem to raise when in error state so that shouldn't affect tests ... but let's see | 10:33 |
| *** prekarat has joined #openstack-qa | 10:37 | |
| *** weshay has quit IRC | 10:42 | |
| *** weshay has joined #openstack-qa | 10:43 | |
| *** matsuhashi has quit IRC | 10:44 | |
| *** matsuhashi has joined #openstack-qa | 10:45 | |
| *** matsuhashi has quit IRC | 10:45 | |
| *** matsuhashi has joined #openstack-qa | 10:46 | |
| *** matsuhashi has quit IRC | 10:46 | |
| *** matsuhashi has joined #openstack-qa | 10:47 | |
| Anju | giulivo: please have a look https://review.openstack.org/#/c/51835/ | 10:50 |
| giulivo | hi Anju :) | 10:51 |
| *** matsuhashi has quit IRC | 10:51 | |
| *** yfried_mob has quit IRC | 10:53 | |
| *** thomasbiege1 has joined #openstack-qa | 10:55 | |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Separate negative tests api/network/security_groups https://review.openstack.org/51457 | 11:14 |
| *** mkollaro has joined #openstack-qa | 11:23 | |
| *** prekarat has quit IRC | 11:27 | |
| *** yfried_mob has joined #openstack-qa | 11:28 | |
| afazekas | sdague: ping | 11:34 |
| *** dkorn has joined #openstack-qa | 11:35 | |
| sdague | what's up? | 11:36 |
| afazekas | I try to enable the ssh tests , but looks like the reboot test is failing randomly | 11:39 |
| afazekas | https://review.openstack.org/#/c/50337/8 | 11:39 |
| raies | afazekas: please check https://review.openstack.org/#/c/47339/ | 11:40 |
| sdague | afazekas: ok, so we'll need to figure out how to get that to more reliably work, otherwise we can't bring it in | 11:41 |
| afazekas | sdague: tempest.api.compute.servers.test_server_actions.ServerActionsTestXML.test_reboot_server_hard the ssh connection sometimes not working after reboot, looks like with neutron the issue is more frequent (or it happens always) | 11:41 |
| sdague | can you file a bug on ? | 11:41 |
| *** shilpa_ has joined #openstack-qa | 11:41 | |
| sdague | I agree that it's probably a race on network connecting | 11:41 |
| sdague | but would be a good neutron / nova bug to get fix | 11:42 |
| afazekas | sdague: https://bugs.launchpad.net/nova/+bug/1240849 | 11:42 |
| sdague | fixed | 11:42 |
| sdague | afazekas: great | 11:42 |
| afazekas | in the current patch set the whole reboot test case is skipped when we have neutron | 11:42 |
| openstackgerrit | Chmouel Boudjnah proposed a change to openstack-dev/devstack: Just reset the repo instead of trying to co files https://review.openstack.org/48411 | 11:43 |
| afazekas | should I create a skipped and a not skipped version of the reboot test case ? | 11:43 |
| sdague | so reboot is the only test that doesn't work? | 11:46 |
| afazekas | it is the only one with random failure | 11:46 |
| sdague | yeh, I say we could make that a skip on that bug | 11:46 |
| *** nosnos has quit IRC | 11:48 | |
| afazekas | Ok, I'll add unconditional skip instead of just skipping with neutron | 11:48 |
| *** nosnos has joined #openstack-qa | 11:49 | |
| sdague | yeh, I think that's the best bet | 11:49 |
| *** mkollaro has quit IRC | 11:51 | |
| *** nosnos has quit IRC | 11:54 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Enable run_ssh and add floatingip + keypair auth https://review.openstack.org/50337 | 11:57 |
| *** mkollaro has joined #openstack-qa | 12:10 | |
| raies | afazekas: hi are u there :) | 12:10 |
| kashyap | giulivo, Hi, did you ever manage to create a compute node in a 15G guest? | 12:13 |
| giulivo | hi kashyap :) | 12:15 |
| giulivo | you mean a 15G guest or openstack-in-openstack? | 12:16 |
| kashyap | Nah, OpenStack compute node in a VM :) | 12:16 |
| kashyap | Inside, obviously, I'll boot Cirros | 12:16 |
| giulivo | ok so yes but maybe not in a 15G guest | 12:17 |
| giulivo | 15G of disk or of ram_ | 12:17 |
| giulivo | ? | 12:17 |
| kashyap | giulivo, Disk. | 12:17 |
| kashyap | giulivo, This on my laptop. I took up the challenge to configure/demo Controller+ Compute Node on my laptop. | 12:18 |
| kashyap | Have to see how far I can go :-) | 12:18 |
| giulivo | so maybe not 15 but the m1.small is 20G and there it works | 12:18 |
| afazekas | raies: @attr(type='smoke') is missing | 12:18 |
| raies | ok I will amend it also :) | 12:18 |
| *** marun has quit IRC | 12:19 | |
| *** yfried_mob has quit IRC | 12:20 | |
| *** yfried_mob has joined #openstack-qa | 12:20 | |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: VPNaas vpnservice test cases https://review.openstack.org/47339 | 12:21 |
| raies | afazekas: ^^ | 12:21 |
| *** prekarat has joined #openstack-qa | 12:24 | |
| afazekas | kashyap: if your goal is just booting one cirros vm, even 6GiB can be enough. | 12:25 |
| kashyap | giulivo, afazekas, Yeah, just found out. I can do controller node (10G) and compute node (7G) just fine with a couple of cirros images | 12:26 |
| afazekas | If you forget to install the debug packagages, may be even less :) | 12:26 |
| *** prekarat has quit IRC | 12:27 | |
| *** prekarat has joined #openstack-qa | 12:27 | |
| afazekas | kashyap: 8 month before 16-20Gib was enough for tempest runs (aio), now I do not know the minimum | 12:30 |
| kashyap | No worries, I'll let you know how it goes. | 12:30 |
| *** shilpa_ has quit IRC | 12:37 | |
| openstackgerrit | Anju Tiwari proposed a change to openstack/tempest: Changed the exception name https://review.openstack.org/51835 | 12:41 |
| *** yfried_mob has quit IRC | 12:41 | |
| *** HenryG has joined #openstack-qa | 12:48 | |
| *** adalbas has joined #openstack-qa | 13:02 | |
| openstackgerrit | Arata Notsu proposed a change to openstack-dev/devstack: Do not install pip when OFFLINE=True https://review.openstack.org/52380 | 13:03 |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: Adds test to test_server_actions https://review.openstack.org/52381 | 13:06 |
| *** thomasbiege1 has quit IRC | 13:06 | |
| *** thomasbiege1 has joined #openstack-qa | 13:07 | |
| *** mriedem has joined #openstack-qa | 13:11 | |
| *** Madan has quit IRC | 13:17 | |
| *** yfried_mob has joined #openstack-qa | 13:18 | |
| *** raies_ has joined #openstack-qa | 13:24 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Forced isolation for the nova quota test https://review.openstack.org/52387 | 13:24 |
| raies_ | afazekas: https://review.openstack.org/#/c/47339/ | 13:24 |
| raies_ | comment incorporated | 13:24 |
| openstackgerrit | Denis Egorenko proposed a change to openstack-dev/devstack: Added Savanna Project https://review.openstack.org/50601 | 13:26 |
| *** jlibosva1 has quit IRC | 13:27 | |
| *** jlibosva has joined #openstack-qa | 13:27 | |
| raies_ | giulivo: https://review.openstack.org/#/c/47339/ needs approval :) | 13:29 |
| *** SergeyLukjanov has joined #openstack-qa | 13:32 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 13:33 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 13:33 | |
| giulivo | afazekas, I wanted to approach this https://review.openstack.org/#/c/39621/ | 13:38 |
| giulivo | but can't find the owner around over irc | 13:38 |
| giulivo | any idea what is his nick? | 13:39 |
| *** mkollaro has quit IRC | 13:41 | |
| *** thomasbiege1 has quit IRC | 13:44 | |
| afazekas | giulivo: ivanzhu | 13:45 |
| afazekas | or ivanzhu1 | 13:46 |
| *** dims has joined #openstack-qa | 13:51 | |
| dansmith | sdague: ping | 13:56 |
| dansmith | sdague: http://paste.openstack.org/show/48587/ | 13:56 |
| sdague | dansmith: yo, what's up | 13:56 |
| dansmith | sdague: that hacky thing gets us a devstack running on master, with a compute running on havana | 13:57 |
| dansmith | and shows that we're already broken for rolling upgrades :/ | 13:57 |
| dansmith | I'd like to get this massaged into something that could be a gate test as soon as possible so that we can start seeing these issues on patches | 13:57 |
| sdague | ok, sure | 13:57 |
| sdague | so the model is going to be control plane first, old compute? | 13:58 |
| dansmith | with a couple of tweaks to fix things we're going to backport, it runs tempest smoke | 13:58 |
| dansmith | yes | 13:58 |
| dansmith | well, really "conductor first" and we're going to have to do some work on api and scheduler, but.. compute last, most likely | 13:58 |
| dansmith | but, this at least gives us some visibility into obviously broken things right now | 13:59 |
| sdague | ok, I'm trying to think how we'd address this in a generic way | 13:59 |
| sdague | grenade has the infrastructure to have 2 versions of openstack checked out | 13:59 |
| *** jlibosva has quit IRC | 13:59 | |
| dansmith | meaning make devstack itself support his weird mode? | 13:59 |
| sdague | yeh | 13:59 |
| *** jlibosva has joined #openstack-qa | 13:59 | |
| sdague | devstack would be hard, because it only thinks about 1 branch for a project | 14:00 |
| dansmith | well, devstack or grenade, whatever | 14:00 |
| sdague | but we might actually be able to make grenade do this sort of thing | 14:00 |
| dansmith | if grenade is close, then that's fine, but if not, could we even just slap this into an experimental job nearly as-is? | 14:00 |
| afazekas | Is neutron expected to be able to enforce the '0' quota for rules and sec groups ? Does it means unlimited for neutron ? | 14:01 |
| dansmith | the hacktastic technique, I mean | 14:01 |
| sdague | dansmith: I'm a little concerned that the hacktasticness would make it pretty hard to integrate into devstack-gate | 14:02 |
| sdague | mordred: you active enough to have a pov here? | 14:02 |
| dansmith | sdague: because? does devstack run in screen? | 14:02 |
| sdague | it doesn't in the gate | 14:02 |
| sdague | we direct exec | 14:03 |
| sdague | because of screen races | 14:03 |
| dansmith | okay | 14:03 |
| dansmith | I started trying to lay down another tree instead of doing this switcheroo, | 14:04 |
| dansmith | but it seemed easier to bounce it after, since devstack won't build any nova stuff unless n-cpu is enabled | 14:04 |
| sdague | yeh, the gate is extra sensitive on that front as well, as we don't let you clone your own things, otherwise 10% of the jobs would fail from network issues. So the gate pre-clones for you | 14:05 |
| dansmith | yeah | 14:05 |
| sdague | dtroyer: you up? | 14:06 |
| dansmith | is the switcheroo hack actually a problem for you, if it were better integrated? | 14:06 |
| dansmith | because if we had a way in devstack to check out a branch before running n-cpu, we could make it more integrated | 14:06 |
| yfried | sdague: ping | 14:07 |
| sdague | but if someone ^C conductor and restarts, it would be on the old branch | 14:07 |
| sdague | which would seem unexpected | 14:07 |
| *** raies_ has quit IRC | 14:07 | |
| dansmith | yep, indeed | 14:07 |
| sdague | so it feels like n-cpu has to be starting out of another directory | 14:08 |
| sdague | so that restack would do something moderately sane | 14:08 |
| dansmith | I guess the other argument is that once we diverge enough we'll have other things that switch (like reqs) | 14:08 |
| dansmith | well, obviously two parallel stacks is the cleaner way to do it, | 14:09 |
| dansmith | I'm just trying to weight the relative gain :) | 14:09 |
| dansmith | er, weigh | 14:09 |
| sdague | yep, I think though with other services, grenade already has the infrastructure to manage 2 branchs | 14:09 |
| dansmith | okay, well, if so, then cool | 14:09 |
| dansmith | we can wait for folks to wake up | 14:09 |
| sdague | yeh | 14:10 |
| dansmith | I was just pretty horrified yesterday when I realized how broken we *already* are from havana | 14:10 |
| sdague | yep, that's fair | 14:10 |
| yfried | sdague: could you please take a look at these patches? they are a chain of commits and I would like it to be merged as soon as possible so that all comments and fixes be done to the latest patch were it's a simpler operation, with no dependencies. | 14:11 |
| yfried | https://review.openstack.org/#/c/51457/ | 14:11 |
| yfried | https://review.openstack.org/#/c/51459/ | 14:11 |
| yfried | https://review.openstack.org/#/c/51590/ | 14:11 |
| sdague | so I think that if we had a generic variable that when we hit the TARGET environment, it kept n-cpu running out of the old path - https://github.com/openstack-dev/grenade/blob/master/grenade.sh#L225 that would be the cleanest approach today | 14:11 |
| openstackgerrit | A change was merged to openstack/tempest: Skip negative compute quota tests that don't work with Neutron https://review.openstack.org/52207 | 14:13 |
| dansmith | sdague: but does it upgrade to a new target directory, or in-place? I thought the latter | 14:13 |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Use lower case true/false in the sample config file https://review.openstack.org/52395 | 14:13 |
| sdague | no, new directory | 14:13 |
| dansmith | okay | 14:13 |
| sdague | https://github.com/openstack-dev/grenade/blob/master/grenaderc#L22 | 14:14 |
| dansmith | so we just don't kill n-cpu at the beginning of upgrade-nova and don't start it after? | 14:14 |
| sdague | yeh, that might work | 14:14 |
| dansmith | okay | 14:14 |
| sdague | I think the actual n-cpu start is fully broken out as a function | 14:15 |
| dansmith | https://github.com/openstack-dev/grenade/blob/master/upgrade-nova#L53 | 14:15 |
| dansmith | not that I see | 14:15 |
| dansmith | oh, start_nova, you meant? | 14:16 |
| sdague | oh, yeh, so start_nova isn't quite small enough | 14:16 |
| *** yfried_mob has quit IRC | 14:16 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: Enable run_ssh and add floatingip + keypair auth https://review.openstack.org/50337 | 14:16 |
| *** yfried_mob has joined #openstack-qa | 14:16 | |
| dansmith | so, I assume grenade will start basing on havana at some point and bunch of this will get retooled, right? | 14:16 |
| sdague | yeh, in the gate the grenaderc is set | 14:16 |
| sdague | so right now it's actually grizzly -> master | 14:17 |
| dansmith | right | 14:17 |
| dansmith | but there's a bunch of stuff in, say, upgrade-nova that we don't need for havana->master right? | 14:17 |
| sdague | right | 14:17 |
| dansmith | okay | 14:17 |
| sdague | so in the master branch we'll delete the work arounds | 14:17 |
| dansmith | right | 14:17 |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add some negative tests for attach_interfaces https://review.openstack.org/49231 | 14:18 |
| sdague | but I don't think the stable branch for grenade is planned until next week, to make sure everyone else is set first | 14:18 |
| *** dansmith is now known as russellbyproxy | 14:18 | |
| *** russellbyproxy is now known as dansmith | 14:18 | |
| openstackgerrit | A change was merged to openstack/tempest: Move negative action tests to right place https://review.openstack.org/49982 | 14:18 |
| openstackgerrit | A change was merged to openstack/tempest: VPNaas vpnservice test cases https://review.openstack.org/47339 | 14:18 |
| dansmith | sdague: yeah | 14:19 |
| sdague | so I'd say first pass, get a devstack change landed that slices off starting nova_compute on it's own | 14:20 |
| sdague | then we can probably adapt grenade to only upgrade parts, which will make a good test here | 14:20 |
| sdague | and have it be a 2nd grenade job | 14:20 |
| dansmith | where is start_nova, is that in devstack? | 14:21 |
| *** Lingxian has joined #openstack-qa | 14:25 | |
| mkoderer | sdague: I won't make it to the QA meeting today | 14:27 |
| *** jlibosva has quit IRC | 14:35 | |
| *** jlibosva has joined #openstack-qa | 14:35 | |
| *** bill_az_ has joined #openstack-qa | 14:36 | |
| sdague | dansmith: yeh, in devstack | 14:37 |
| sdague | lib/nova | 14:37 |
| yfried | sdague: ping | 14:40 |
| sdague | yfried: I looked at one of the changes, I'll get to code reviews later, plenty else on my plate right now | 14:41 |
| yfried | sdague: tnx. just making sure you saw my request. much appreciated | 14:42 |
| *** raies_ has joined #openstack-qa | 14:44 | |
| openstackgerrit | Anju Tiwari proposed a change to openstack/tempest: Changed the exception name https://review.openstack.org/51835 | 14:44 |
| *** jhenner has quit IRC | 14:45 | |
| Anju | afazekas: I have rebased two times | 14:46 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Add a skip for meter-label cli tests https://review.openstack.org/52211 | 14:46 |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use nova.conf for auth_token configs. https://review.openstack.org/52258 | 14:48 |
| sdague | mkoderer: can you submit your stress tests summit proposal to summit.openstack.org ? | 14:50 |
| sdague | I'd like to get that locked in the schedule there | 14:50 |
| mkoderer | sdague: I booked my flights and my hotel | 14:51 |
| openstackgerrit | A change was merged to openstack/tempest: Separate negative tests api/network/security_groups https://review.openstack.org/51457 | 14:51 |
| mkoderer | sdague: but I am still waiting for final approval | 14:51 |
| mordred | sdague: reading | 14:51 |
| sdague | mkoderer: ok, well let's put your session in, and if you can't make it, we'll turn over the floor to someone else. But maybe it will help on your approval if you are officially on the schedule :) | 14:52 |
| mordred | sdague, dansmith: can you tl;dr catch me up on the question above? | 14:52 |
| sdague | mordred: yes | 14:52 |
| mkoderer | sdague: ok | 14:52 |
| sdague | mordred: so, dansmith would like to be able to test an environment which is current nova control plane w/ old nova-compute | 14:53 |
| sdague | to ensure all the message versioning that nova is doing, actually, you know, works | 14:53 |
| mordred | sdague: he would like to test rolling upgrades in other words | 14:54 |
| sdague | yes | 14:54 |
| sdague | or at least the first steps | 14:54 |
| mordred | so, just so you know, that's what lifeless is currently working on, although he's probably a few steps from having it be gatable | 14:54 |
| mordred | at the very least it's probably worth syncing up so that people don't duplicate effort - OR | 14:54 |
| sdague | nova has put huge investments into versioned nova-compute | 14:54 |
| dansmith | mordred: I don't think it's the same, | 14:55 |
| sdague | mordred: ok, well, that's sort of first I'd heard of lifeless doing that :) | 14:55 |
| mordred | so that they do duplicate it due to timing issues, but do so as part of a plan | 14:55 |
| dansmith | mordred: he's talking about CD'ing the whole stack together, AFAIK | 14:55 |
| mordred | dansmith: the end goal is the same | 14:55 |
| mordred | you can't CD the stack without performing rolling upgrades | 14:55 |
| mordred | and you can't really do it without testing that it can be done :) | 14:55 |
| mordred | I'm just saying - syncing is prolly good idea - it's entirely possible that the stuff from above is a great idea | 14:56 |
| dansmith | yes, end goal is the same, but I think the work to be done from either end is going to be distinct for quite a while :) | 14:56 |
| mordred | most likely | 14:56 |
| sdague | yeh, I'd agree with that | 14:56 |
| sdague | because I think this really turns into a 2nd grenade job | 14:56 |
| sdague | where we just don't upgrade nova-compute | 14:56 |
| * mordred just wants to make sure we eventually meet in the middle, and not one of us in north dakota and the other in thailand | 14:56 | |
| mordred | sdague: ++ | 14:57 |
| mordred | that sounds like a good job | 14:57 |
| dansmith | sdague: can or do we have a summit session set aside for upgrade testing ? | 14:57 |
| dansmith | sdague: we should make sure I'm not double-booked for that slot | 14:57 |
| sdague | dansmith: yes, I was going to hold a slot for that | 14:57 |
| dansmith | okay, cool | 14:57 |
| sdague | dansmith: you want to propose it? | 14:57 |
| dansmith | sure | 14:58 |
| sdague | we don't yet have text, and I'll play with the schedule to try to not double book you | 14:58 |
| * sdague thinks scheduling is going to be *fun* | 14:58 | |
| dansmith | well, we haven't really done nova yet either, so I'll just try to keep this in mind | 14:58 |
| sdague | yep | 14:58 |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: Revert "VPNaas vpnservice test cases" https://review.openstack.org/52405 | 14:59 |
| dansmith | sdague: suggestified | 15:00 |
| mtreinish | sdague: we have a stable/havanna tempest branch now https://git.openstack.org/cgit/openstack/tempest/log/?h=stable/havana | 15:00 |
| sdague | mtreinish: great | 15:01 |
| sdague | mordred: so you think one QA upgrade session is good enough to discuss these multiple approaches? | 15:02 |
| sdague | or should we carve out special time to talk about the approach lifeless is taking, especially if it means more/different gating jobs? | 15:03 |
| dansmith | sdague: assuming lifeless is going to have other sessions for his grander stuff, I think that we can probably talk about the general idea of this in one session | 15:03 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: api/network/security_groups_negative add testcases https://review.openstack.org/51459 | 15:03 |
| sdague | dansmith: ok, cool | 15:03 |
| *** yfried_mob has quit IRC | 15:03 | |
| dansmith | sdague: I say that because, I don't think it will really take much time to discuss what I've got/proposed here | 15:03 |
| dansmith | unless someone is religiously opposed to upgrade testing :) | 15:04 |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part1 https://review.openstack.org/39621 | 15:05 |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part2 https://review.openstack.org/39609 | 15:05 |
| sdague | dansmith: yeh, I was more thinking that we need to also discuss things like grenade refactoring to let the upgrade scripts be used by packages as well | 15:05 |
| sdague | I'll reword the proposal a little bit to include a couple other things we need in that block which I think you'd be good to be at | 15:06 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Set tempest version for icehouse https://review.openstack.org/52412 | 15:06 |
| raies_ | sdague: hi , in https://review.openstack.org/#/c/47339/ Auther and commiter names were reflecting different | 15:07 |
| dansmith | sdague: well, okay, that might make it longer, but if you want that stuffed in the same thing, that's fine | 15:07 |
| raies_ | sdague: due to my mistake | 15:07 |
| *** mkollaro has joined #openstack-qa | 15:07 | |
| sdague | dansmith: ok, before I do that, do you think you'll need the full block for just rolling upgrade testing? | 15:07 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/tempest: Add shelve/unshelve test of nova APIs https://review.openstack.org/49892 | 15:07 |
| * sdague hits cancel on the edit | 15:08 | |
| raies_ | sdague: please suggest how to make Author name to "raiesmh08" | 15:08 |
| dansmith | sdague: well, if we get lifeless in there to talk about his stuff and my proposed stuff, I would think that would be a good whole session on live upgrades | 15:08 |
| sdague | raies: you can't change an old commit | 15:08 |
| sdague | dansmith: ok, I think that's fair | 15:08 |
| sdague | I'll take the session as is | 15:08 |
| dansmith | sdague: and then maybe a separate one for general upgrade testing improvements? | 15:08 |
| sdague | and have another grenade futures session | 15:08 |
| dansmith | yeah | 15:08 |
| sdague | ok, done | 15:10 |
| raies_ | sdague: So Author name will not change even in any case ? | 15:10 |
| mordred | sdague: I agree with dansmith - I think we can talk about it in one session | 15:11 |
| sdague | mordred: ok, http://summit.openstack.org/cfp/details/278 is preapproved, if there is other language you want in there, let me know | 15:11 |
| sdague | I also specifically commented that we want lifeless in that conversation as well | 15:11 |
| mordred | raies_: once it's merged, that's how it is | 15:12 |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use nova.conf for auth_token configs. https://review.openstack.org/52258 | 15:12 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: edit inheritence tree api/network/security_groups https://review.openstack.org/51590 | 15:12 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: api/network/security_groups_negative add testcases https://review.openstack.org/51459 | 15:12 |
| raies_ | mordred: ok, I have reverted the merge, now wht should I do ? | 15:13 |
| mkoderer | sdague: http://summit.openstack.org/cfp/details/279 | 15:13 |
| sdague | mkoderer: thanks | 15:13 |
| *** yfried_mob has joined #openstack-qa | 15:13 | |
| mordred | raies_: wait - why did you revert the change? is it a bad chage? I'm confused as to what you're trying to accomplish | 15:13 |
| mordred | raies_: do you want to resubmit a new version of that change now? | 15:14 |
| raies_ | mordred: I wanted that I will recommit it with same Author and Committr name | 15:15 |
| raies_ | mordred: any suggestion ? | 15:16 |
| mordred | raies_: why? that seems like a lot of extra work for a lot of people | 15:16 |
| *** shilpa_ has joined #openstack-qa | 15:17 | |
| raies_ | mordred: then I should abandon revert ? | 15:17 |
| mordred | raies_: yes. at this point I think it's just good to make sure that next time if you need the author to be different, make sure you get that done before the patch lands | 15:18 |
| mordred | raies_: for now, I don't think it's hurting anything | 15:18 |
| raies_ | ok, it will be better to abandon the revert :) | 15:19 |
| afazekas | Can we increase the full tempest/devstack jobs time limit to 90 minute ? | 15:25 |
| mtreinish | afazekas: are we above an hour? if so then something is wrong | 15:26 |
| afazekas | https://review.openstack.org/#/c/52395/ | 15:26 |
| afazekas | 55 minute and 60 minute | 15:26 |
| *** shilpa_ has quit IRC | 15:27 | |
| mtreinish | wow the runtime is really weird | 15:27 |
| mtreinish | did you look at the list of slowest tests? | 15:27 |
| mtreinish | negative tests taking >400sec | 15:28 |
| mtreinish | something isn't right there | 15:28 |
| afazekas | mtreinish: you are correct, it is abnormal | 15:29 |
| *** jlibosva has quit IRC | 15:30 | |
| mtreinish | afazekas: I think it was https://review.openstack.org/#/c/50374/ I think that it's getting stuck in the setUp for a long time | 15:31 |
| afazekas | I see the issue | 15:32 |
| afazekas | test_rebuild_deleted_server | 15:33 |
| afazekas | test_suspend_server_invalid_state | 15:33 |
| *** mlavalle has joined #openstack-qa | 15:35 | |
| mtreinish | afazekas: ohhh, the single server gets deleted | 15:37 |
| afazekas | no this is not an issue | 15:37 |
| afazekas | the pause is the other | 15:37 |
| afazekas | suspend and pause | 15:37 |
| *** yassine has quit IRC | 15:39 | |
| giulivo | I knew that tracking of the slower tests turned to be useful :) | 15:39 |
| openstackgerrit | Lingxian Kong proposed a change to openstack/tempest: add some negative tests for attach_interfaces https://review.openstack.org/49231 | 15:42 |
| openstackgerrit | afazekas proposed a change to openstack/tempest: server status remained in unexpected state https://review.openstack.org/52429 | 15:43 |
| openstackgerrit | Anju Tiwari proposed a change to openstack/tempest: Changed the exception name https://review.openstack.org/51835 | 15:45 |
| afazekas | mtreinish: https://review.openstack.org/52429 it makes it faster, but the deletion will cause an exception in the log | 15:48 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Add stable/havana to devstack-gate https://review.openstack.org/52437 | 15:50 |
| *** yfried_mob has quit IRC | 15:55 | |
| afazekas | jaypipes: it resumed and unpaused where it paused and suspended | 15:55 |
| *** gszasz has quit IRC | 15:57 | |
| *** yfried_mob has joined #openstack-qa | 16:01 | |
| jaypipes | afazekas: no it doesn't. | 16:02 |
| jaypipes | afazekas: http://paste.openstack.org/show/48646/ | 16:02 |
| jaypipes | afazekas: that's where it pauses... and does not unpause. | 16:02 |
| jaypipes | afazekas: the unpause should be done in that test method, not the one on your patch... | 16:03 |
| *** thomasbiege1 has joined #openstack-qa | 16:07 | |
| *** SnowDust has joined #openstack-qa | 16:07 | |
| *** thomasbiege1 has quit IRC | 16:09 | |
| SnowDust | is there any tempest 101 | 16:10 |
| SnowDust | ? | 16:10 |
| *** yfried_mob has quit IRC | 16:10 | |
| *** yfried_mob has joined #openstack-qa | 16:11 | |
| SnowDust | anyone can give me a walkthrough on tempest code organisation .. very highlevel ? | 16:11 |
| afazekas | jaypipes: I have modified that method, I will submit better patch | 16:12 |
| openstackgerrit | afazekas proposed a change to openstack/tempest: server status remained in unexpected state https://review.openstack.org/52429 | 16:12 |
| SnowDust | :) no one interested seems .. ! | 16:12 |
| SnowDust | a highlevel .. template code walkthrough needed .. anyone ? | 16:13 |
| SnowDust | oops i meant .. a highlevel .. tempest code walkthrough needed .. anyone | 16:13 |
| *** dkorn has quit IRC | 16:13 | |
| *** Lingxian has quit IRC | 16:18 | |
| SnowDust | a highlevel .. tempest code walkthrough needed .. anyone ? | 16:18 |
| *** yfried_mob has quit IRC | 16:18 | |
| SnowDust | a highlevel .. tempest code walkthrough needed .. anyone ? | 16:18 |
| *** yfried|mob has joined #openstack-qa | 16:18 | |
| SnowDust | oopz i found the developer doc | 16:19 |
| SnowDust | thanks all | 16:19 |
| SnowDust | done :) | 16:19 |
| *** SnowDust has left #openstack-qa | 16:19 | |
| *** afazekas has quit IRC | 16:24 | |
| *** marun has joined #openstack-qa | 16:26 | |
| *** fbo is now known as fbo_away | 16:26 | |
| *** openstackrecheck has quit IRC | 16:30 | |
| *** openstackrecheck has joined #openstack-qa | 16:30 | |
| openstackgerrit | Dan Prince proposed a change to openstack-dev/devstack: Use cinder.conf for auth_token configs. https://review.openstack.org/52259 | 16:33 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/52422 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 16:34 |
| clarkb | jog0: ^ doesn't seem to be a consistent problem. I will add more debugging lines to elasticRecheck.py | 16:35 |
| openstackrecheck | openstack/heat change: https://review.openstack.org/52038 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 16:35 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: edit inheritence tree api/network/security_groups https://review.openstack.org/51590 | 16:36 |
| openstackrecheck | openstack/keystone change: https://review.openstack.org/51568 failed tempest with an unrecognized error | 16:40 |
| *** afazekas has joined #openstack-qa | 16:41 | |
| *** VinodGupta has joined #openstack-qa | 16:43 | |
| *** DennyZhang has joined #openstack-qa | 16:46 | |
| *** andreaf has joined #openstack-qa | 16:50 | |
| *** VinodGupta has quit IRC | 16:52 | |
| dansmith | dkranz: around? | 16:52 |
| Anju | afazekas: ping | 16:55 |
| *** DennyZhang has quit IRC | 16:57 | |
| *** DennyZhang has joined #openstack-qa | 16:57 | |
| dansmith | all: sdague and mtreinish are running a little behind schedule, so they'll be here to start the meeting, but just a tad late | 16:58 |
| *** safchain has quit IRC | 17:00 | |
| mtreinish | meeting time | 17:02 |
| *** Dharmit has quit IRC | 17:05 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Log uncaught exceptions in bot's run loop. https://review.openstack.org/52458 | 17:12 |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/52429 failed tempest because of: https://bugs.launchpad.net/bugs/1239856 | 17:16 |
| *** yfried has quit IRC | 17:18 | |
| openstackgerrit | Mehdi Abaakouk proposed a change to openstack-dev/devstack: Add a comment about how to daemonize process https://review.openstack.org/51621 | 17:20 |
| openstackgerrit | Mehdi Abaakouk proposed a change to openstack-dev/devstack: Don't kill ceilometer prematurally in devstackgate https://review.openstack.org/51623 | 17:20 |
| openstackgerrit | Mehdi Abaakouk proposed a change to openstack-dev/devstack: Don't kill sar prematurally in devstack-gate https://review.openstack.org/51622 | 17:20 |
| openstackgerrit | Mehdi Abaakouk proposed a change to openstack-dev/devstack: Don't kill nova-bm-deploy-helper prematurally https://review.openstack.org/51624 | 17:20 |
| jog0 | clarkb: are all the elasticsearch servers the same version? | 17:21 |
| jog0 | or are you in the middle of an upgrade | 17:21 |
| clarkb | jog0: yes, they are currently all the same version. When I do the upgrade I will do it in one giant manuever | 17:21 |
| clarkb | jog0: I am currently aiming for monday | 17:22 |
| jog0 | clarkb: cool | 17:22 |
| jog0 | I will be in Edinburgh on Monday | 17:22 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Log uncaught exceptions in bot's run loop. https://review.openstack.org/52458 | 17:23 |
| *** bill_az_ has quit IRC | 17:27 | |
| *** raies_ has quit IRC | 17:30 | |
| *** rockygrober has joined #openstack-qa | 17:31 | |
| *** openstackrecheck has quit IRC | 17:31 | |
| *** openstackrecheck has joined #openstack-qa | 17:31 | |
| clarkb | jog0: mtreinish: recheck bot restarted to pick up the exception catching and logging | 17:32 |
| *** rockyg has quit IRC | 17:33 | |
| sdague | clarkb: I take it that it's not auto restarting on git changes? | 17:34 |
| *** rockygrober has quit IRC | 17:34 | |
| *** rockygrober has joined #openstack-qa | 17:34 | |
| clarkb | sdague: it *should* be but I have disabled puppet because there is an oustanding puppet change that needs review for that host... | 17:35 |
| clarkb | so I kicked it manually | 17:35 |
| sdague | ok | 17:36 |
| jog0 | clarkb: we may have a swift / infra / gate issue https://bugs.launchpad.net/openstack-ci/+bug/1240256 | 17:37 |
| jog0 | sdague: ^ | 17:37 |
| jog0 | swift is hungry for moar VM | 17:37 |
| clarkb | sdague: I did kick it by running puppet though, so the puppet bits seem to be working | 17:37 |
| *** SergeyLukjanov has quit IRC | 17:37 | |
| clarkb | jog0: there isn't much we can do about that | 17:37 |
| * jog0 cries a little | 17:38 | |
| openstackgerrit | afazekas proposed a change to openstack/tempest: server status remained in unexpected state https://review.openstack.org/52429 | 17:38 |
| jog0 | clarkb: we are running tempest on RAX still? | 17:38 |
| jog0 | this may be related to that | 17:39 |
| clarkb | I mean we could try doing performance testing of some sort before putting slaves into the ready pool, but nothing prevents a bad neighbor from showing up after entering the pool | 17:39 |
| clarkb | jog0: we are using RAX in the check queue, but not the gate | 17:39 |
| sdague | jog0: it 503s under load? | 17:39 |
| sdague | that seems like a problem with the stack | 17:39 |
| jog0 | sdague: correct 503s | 17:39 |
| jog0 | the issue happens in both check and gate queues | 17:40 |
| sdague | we are only doing 4 simultaneous test threads | 17:40 |
| sdague | throw more VM at it seems weird | 17:40 |
| jog0 | sdague: this is what swift folks said https://bugs.launchpad.net/openstack-ci/+bug/1240256/comments/3 | 17:40 |
| sdague | I would think swift could handle a max of 4 simul requests even with small vm | 17:40 |
| * afazekas just comment changed https://review.openstack.org/#/c/52429/ | 17:40 | |
| Anju | afazekas: ping | 17:41 |
| *** yfried|mob has quit IRC | 17:42 | |
| clarkb | jog0: these aren't dinky VMs either. They are 8GB with 4 vcpu instances | 17:42 |
| clarkb | which is middle of the pack for our providers | 17:43 |
| sdague | jog0: I just put questions on the list | 17:43 |
| sdague | jog0: sorry on the bug | 17:43 |
| sdague | dtroyer: when you get a chance - https://review.openstack.org/#/c/51623/ - will unstick the ceilo guys, and is minimally impacting change | 17:43 |
| jog0 | sdague: awesome thanks | 17:44 |
| jog0 | when I saw swift failing I was a little surprised | 17:44 |
| dtroyer | sdague: +A | 17:46 |
| afazekas | Anju: I am fully exhausted, I'll go to sleep | 17:46 |
| *** mlavalle has quit IRC | 17:49 | |
| *** SergeyLukjanov has joined #openstack-qa | 17:50 | |
| *** yfried has joined #openstack-qa | 17:51 | |
| jog0 | dtroyer: have a moment for a 10 line devstack review https://review.openstack.org/#/c/44349/ | 17:53 |
| Anju | afazekas: sure afazekas | 17:53 |
| Anju | afazekas: gn | 17:53 |
| *** DennyZhang has quit IRC | 17:54 | |
| dtroyer | jog0: I'm OK with the code but garyk's −1 concerns me. If gary or someone else from Neutron sez it's good to go then I'll be on board. I just don't know enough about those drivers to say either way. | 17:56 |
| jog0 | dtroyer: yeah I haven't been able to get ahold of him | 17:56 |
| jog0 | but the tempest tests pass which is why i think he may have outdated info | 17:57 |
| dtroyer | jog0: is there anyone else on Neutron that could confirm? | 17:57 |
| *** marun has quit IRC | 17:58 | |
| *** afazekas has quit IRC | 17:58 | |
| jog0 | dtroyer: not sure we had a similar patch in nova let me dig that up | 17:59 |
| dtroyer | jog0: kk….I'm heading to lunch in a minute | 17:59 |
| jog0 | dtroyer: https://review.openstack.org/#/c/49660/ | 17:59 |
| jog0 | see the response to garyk's comment | 17:59 |
| jog0 | plus the tempest quota tests pass which either means garyk is wrong or the tempest tests stink | 18:00 |
| jog0 | both are possible | 18:00 |
| dtroyer | jog0: :) that review hasn't merged yet…can we trust it to be accurate? | 18:01 |
| jog0 | dtroyer: not sure | 18:02 |
| jog0 | dtroyer: I will harass the neutron folks until I get a clear answer | 18:03 |
| jog0 | thanks dtroyer | 18:03 |
| *** yfried_mob has joined #openstack-qa | 18:07 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/50389 failed tempest because of: https://bugs.launchpad.net/bugs/1239637 | 18:12 |
| *** prekarat has quit IRC | 18:15 | |
| *** mlavalle has joined #openstack-qa | 18:15 | |
| *** yfried_mob has quit IRC | 18:22 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/51623 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 and https://bugs.launchpad.net/bugs/1240256 and | 18:32 |
| *** rdekel has joined #openstack-qa | 18:40 | |
| *** rdekel has quit IRC | 18:46 | |
| clarkb | I am going to fix that extra 'and' too | 18:53 |
| clarkb | it annoys me | 18:53 |
| mtreinish | clarkb: heh sorry about that | 18:56 |
| mtreinish | sdague already fixed it for the gerrit messages | 18:56 |
| clarkb | mtreinish: no problem :) | 18:56 |
| sdague | clarkb: oh, I assumed it was common code :) | 18:56 |
| sdague | maybe we should fix that | 18:57 |
| clarkb | sdague: I will look at it. I need to chainsaw elastic-recheck today for the logstash work. I am going to see if I can make it forward and backward compatible so that we don't lose info without migrating data | 18:57 |
| jog0 | clarkb: sweet | 18:59 |
| sdague | clarkb: ok, cool, I was going to librarize the results stuff, but if you get there first, more power to you | 18:59 |
| lifeless | sdague: dansmith: mordred: - hi, you invoked me? | 19:01 |
| mordred | lifeless: yes. long ago in a galaxy far away | 19:01 |
| mordred | we're talking about testing to ensure different sides of internal protocols don't derp each other | 19:01 |
| mordred | lifeless: specifically, that a rolling upgrade could be in progress and nova compute and nova control plane could talk across versions | 19:02 |
| lifeless | right | 19:02 |
| dansmith | lifeless: at the micro level, I want to run two devstacks on two different git revisions and make sure they still pass tests | 19:02 |
| mordred | lifeless: I mentioned you because obviously rolling upgradability is on your roadmap | 19:02 |
| dansmith | lifeless: sounds like eventually you'll be able to CD us to that state for an even better scenario | 19:02 |
| lifeless | dansmith: ok, so our handwave plan is 'deploy a cloud at 'last known good', then put serious load on it, then live upgrade it to the new revision, of whatever change is being tested. | 19:03 |
| dansmith | lifeless: sounds awesome.. nova will fall over real hard right now, so we've got work to do of course | 19:03 |
| lifeless | yeah | 19:03 |
| *** yfried_mob has joined #openstack-qa | 19:03 | |
| * mordred mainly just wanted dansmith and lifeless to chat - so, job done | 19:03 | |
| lifeless | we should be filing those bugs in decemberish | 19:03 |
| dansmith | lifeless: you're planning to make this work fast enough to gate project changes or what? | 19:04 |
| lifeless | dansmith: totally | 19:04 |
| lifeless | dansmith: https://etherpad.openstack.org/p/tripleo-test-cluster | 19:04 |
| dansmith | lifeless: by december? | 19:04 |
| lifeless | dansmith: One hopes | 19:05 |
| dansmith | lifeless: well, you won't be able to gate on that behavior in december, because nova is already broken from havana :) | 19:05 |
| dansmith | lifeless: and just to be clear, you mean doing things like shooting the nova-api node in the head, upgrading, and then moving on to the nova-scheduler, etc, right? | 19:06 |
| lifeless | dansmith: we weren't planning on having a per-service granularity, rather the granularity that our test cluster can afford | 19:07 |
| lifeless | dansmith: so if we say that we want a 6 node cloud | 19:07 |
| dansmith | lifeless: right, see, I want to specifically test per-nova-service | 19:07 |
| lifeless | dansmith: yeah, I can see why. | 19:08 |
| lifeless | dansmith: thing is, we deploy entire nodes at a time | 19:08 |
| lifeless | dansmith: how many nova services are there? It might be feasible to run a totally exploded set of services | 19:08 |
| lifeless | api, scheduler, compute, conductor | 19:08 |
| sdague | lifeless: I guess a question I have on that approach is what kind of artifact capture on fail? | 19:08 |
| dansmith | lifeless: right now, we really only need two: compute and (everything else) | 19:09 |
| dansmith | lifeless: but eventually it will be probably four or so | 19:09 |
| lifeless | dansmith: ok, so our initial thing will deliver that | 19:09 |
| lifeless | dansmith: as we have (today) 'notcompute' + 'compute' :) | 19:09 |
| dansmith | okay | 19:09 |
| sdague | definitely thinking about how we effectively have enough first fail capture so that people can resolve the issue through just combing over the artifacts | 19:09 |
| lifeless | sdague: so the plan is, to get an experimental queue going, then silent, then when the failure are not due to the test environment, move that up to check | 19:10 |
| sdague | lifeless: anyway, I set asside a QA session for the topic as well, wanted to make sure you were heads up on it to be there | 19:10 |
| lifeless | and when the check is something folk are comfortable with move it to gate | 19:10 |
| lifeless | sdague: thank you | 19:10 |
| dansmith | lifeless, mordred, sdague: I still think that a smaller case of two parallel devstacks to help us iterate on the small problem is suitable in the short-term, and it sounds like we'll be able to deprecate it at some point in favor of lifeless' stuff | 19:10 |
| lifeless | sdague: what day ? | 19:10 |
| sdague | lifeless: I don't know yet | 19:10 |
| lifeless | sdague: ack | 19:10 |
| sdague | I accepted it, haven't scheduled | 19:10 |
| lifeless | sdague: tuesday is my hell-day :) | 19:11 |
| sdague | if there are bad blocks for you, let me know | 19:11 |
| sdague | ok, I'll try not to do tuesday | 19:11 |
| sdague | I think we have qa in 3 blocks over 3 days | 19:11 |
| mordred | dansmith: I agree | 19:11 |
| mordred | dansmith: ALSO - it might be possible that even once lifeless stuff is ready, that some per-service level stuff might still be useful - kinda like unit tests | 19:12 |
| mordred | but I think we probably need to learn more | 19:12 |
| dansmith | mordred: yeah | 19:12 |
| lifeless | oh, so here's a point | 19:12 |
| lifeless | we don't have the hardware resources to deploy to the CD cloud on check, nor even on gate | 19:13 |
| lifeless | because zuul needs both check and gate to be parallel | 19:13 |
| lifeless | this is why the test cluster for check and gate is virtualised | 19:13 |
| *** mlavalle has quit IRC | 19:13 | |
| lifeless | so we'll also be getting some signal post gate | 19:13 |
| lifeless | about where we missed things, which we'll feed back in as needed tests | 19:13 |
| sdague | yeh, it will be interesting to see this evolve | 19:14 |
| sdague | lifeless: so any thoughts on virtual CD? | 19:14 |
| sdague | to try to get closer to the ability to deploy on the check queue? | 19:15 |
| lifeless | sdague: see the etherpad I linked | 19:15 |
| sdague | ok, gotcha | 19:15 |
| lifeless | sdague: we're aiming straight at that with a cluster of directly-libvirt provisioned VM's | 19:15 |
| lifeless | each on their own ovs bridge | 19:15 |
| lifeless | 6 such clusters on a physical machine (deployed via nova-baremetal) | 19:16 |
| sdague | ok, cool | 19:16 |
| sdague | snow fortress... here we come | 19:16 |
| lifeless | 10 machines will get us 60 concurrent test environments | 19:16 |
| lifeless | which will be used to check a) nova baremetal functional tests, b) tripleo bootstrap story, c) tripleo overcloud story, d) openstack upgrade story | 19:17 |
| lifeless | we | 19:17 |
| lifeless | we're in crawl mode at the moment, but unlike previous attempts I'm pretty confident this is going to come together | 19:17 |
| sdague | cool | 19:18 |
| sdague | look forward to hearing more of this in summit | 19:18 |
| sdague | clarkb: so, how close are you to needing to chainsaw e-r? I can pivot over to working on libraryizing the results stuff if that would help you | 19:19 |
| *** dkranz has quit IRC | 19:20 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/52478 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 19:47 |
| clarkb | sdague: I was going to try and do code review this afternoon and chainsaw e-r this evening | 19:58 |
| clarkb | sdague: if you want to take a stab at libraryizing it first I say go for it | 19:58 |
| sdague | yeh, I've got one meeting coming up, and I'll see if I can dive a bit after | 19:59 |
| *** yfried_mob has quit IRC | 20:01 | |
| clarkb | sdague: great. So I realized a thing about elasticsearch when reading about the new logstash schema. If you search for build_queue:"check" it will look in @fields.build_queue too, so I just need to update the queries to use the newer flatter schema and the queries should all work | 20:01 |
| *** yfried_mob has joined #openstack-qa | 20:01 | |
| clarkb | sdague: much less chainsawing than I originally thought | 20:01 |
| *** yfried_mob has quit IRC | 20:11 | |
| *** SergeyLukjanov has quit IRC | 20:15 | |
| *** raies has quit IRC | 20:15 | |
| *** mlavalle has joined #openstack-qa | 20:24 | |
| *** dkranz has joined #openstack-qa | 20:24 | |
| sdague | clarkb: cool, other than that, has the results structure changed at all? | 20:42 |
| clarkb | sdague: not really. the '@' is removed from all fields but timestamp (message and tags etc) so any queries that use those will need to have @tags:"" or tags:"" | 20:43 |
| sdague | clarkb: is the new stuff out there? | 20:43 |
| clarkb | and @fields is gone with all of the fields in that dict moved to the top of scope | 20:43 |
| sdague | clarkb: yeh, the fields struct is the issue | 20:43 |
| clarkb | sdague: out there meaning deployed? not yet. I am planning to do it all on monday so that I have lots of time to fix problems :) | 20:43 |
| clarkb | but I have much of the pipeline sorted out and tested locally | 20:44 |
| clarkb | so I don't expect too many problems (I have just doomed this venture by saying that) | 20:44 |
| sdague | ok, any chance in bring up the server somewhere to poke at? | 20:45 |
| sdague | my concern is the return json is super deep structurally | 20:46 |
| sdague | so we often have 6 level nested dictionaries to get to answers | 20:46 |
| clarkb | sdague: not really without spinning up a cluster side by side which I have been avoiding due to the size of the things | 20:47 |
| clarkb | the API shouldn't change | 20:47 |
| sdague | ok | 20:47 |
| clarkb | elasticsearch seems to be pretty good about that | 20:47 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: don't explose if a query no longer has failures https://review.openstack.org/52509 | 20:51 |
| sdague | ok, great | 20:51 |
| sdague | you'll like that commit message | 20:51 |
| sdague | I almost wrote it like a letter home to mom | 20:51 |
| sdague | but I decided that would be over the top | 20:51 |
| sdague | ok, so if we aren't affraid of es breaking it's API, I'll just focus on the library stuff in the morning in er | 20:52 |
| sdague | I need a code day | 20:52 |
| clarkb | sdague: I am not worried about es breaking the api. I am told thattools like kibana2 keep on trucking because APIs aren't horribly changed | 20:52 |
| *** yfried_mob has joined #openstack-qa | 20:59 | |
| *** mlavalle has quit IRC | 21:01 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/52497 failed tempest with an unrecognized error | 21:02 |
| *** rockyg has joined #openstack-qa | 21:03 | |
| *** rockygrober has quit IRC | 21:03 | |
| clarkb | sdague: for more info on what I plan on doing there is https://review.openstack.org/#/c/52223/ to isntall kibana3 along side kibana2, https://review.openstack.org/#/c/52452/ to upgrade elasticsearch to 0.90.3, and https://review.openstack.org/#/c/52453/ to upgrade logstash to 1.2.1 | 21:05 |
| clarkb | the last piece of the puzzle is e-r | 21:05 |
| sdague | so, if it's just the queries file, that should be pretty simple | 21:06 |
| clarkb | sdague: queries file then some of the internal templates used by e-r | 21:06 |
| clarkb | won't be bad | 21:06 |
| sdague | yeh | 21:06 |
| sdague | I am getting some really weird results right now running queries though... we might have some funny bugs in here that no one noticed yest | 21:07 |
| clarkb | I wouldn't be surprised | 21:07 |
| *** dkranz has quit IRC | 21:08 | |
| *** weshay has quit IRC | 21:10 | |
| *** mlavalle has joined #openstack-qa | 21:12 | |
| mkollaro | I'm getting this error https://paste.fedoraproject.org/47643/82044327/ with the test "list_servers_filtered_by_ip_regex" | 21:12 |
| mkollaro | this test is usually skipped, since there is a bug in neutron, but I'm using novanetwork | 21:12 |
| *** SergeyLukjanov has joined #openstack-qa | 21:13 | |
| mkollaro | however, this looks like a problem in the code of tempest and not a bug in novanetwork | 21:13 |
| mkollaro | any ideas? | 21:13 |
| clarkb | mkollaro: most of our tempest runs run against novanetwork. Is the test skipped tehre as well? | 21:14 |
| sdague | hmmmm.... pyelasticsearch might be doing some bad caching | 21:14 |
| mkollaro | no, @skip_because(bug="1182883", condition=config.TempestConfig().service_available.neutron) | 21:14 |
| clarkb | mkollaro: right but neutron won't be available on most of our tests so in theory that test is beng run | 21:15 |
| *** dkranz has joined #openstack-qa | 21:15 | |
| mkollaro | hm | 21:15 |
| sdague | mkollaro: link into the code for the test? | 21:16 |
| sdague | clarkb: yeh, pyelasticsearch is definitely wonkyland on caching.... I need to sort this one out | 21:16 |
| sdague | as this could be an issue for some of our query tools | 21:16 |
| mkollaro | sdague: https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_list_server_filters.py#L219 | 21:17 |
| clarkb | sdague: ok, I am curious to know what you find | 21:19 |
| mkollaro | hm, I should probably set the fixed_network_name in the configuration | 21:19 |
| mkollaro | that should do it | 21:19 |
| openstackgerrit | Dean Troyer proposed a change to openstack-dev/devstack: Fix typos and thinkos in docs https://review.openstack.org/52514 | 21:19 |
| sdague | thinkos... I like it | 21:21 |
| sdague | clarkb: yeh, so it looks like if we reuse the pyelasticsearch object for queries, it's not entirely stable. We really need to build a new one each time | 21:23 |
| *** yfried_mob has quit IRC | 21:24 | |
| clarkb | sdague: :( | 21:25 |
| clarkb | sdague: we have to do that with GitPython, new repo object for each git operation. Caches are fun | 21:25 |
| clarkb | sdague: is that because the connections or dying or because it is caching results info? | 21:25 |
| openstackrecheck | openstack/requirements change: https://review.openstack.org/52137 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 and https://bugs.launchpad.net/bugs/1218391 and | 21:26 |
| sdague | clarkb: I don't know yet | 21:26 |
| sdague | I'm trying to figure it out right now | 21:26 |
| *** yfried has quit IRC | 21:30 | |
| sdague | hmmm... this really is weird, I'm going to need to play in the morning. As my brain is not wrapping around this well. | 21:31 |
| dkranz | dansmith: Sorry, I've been socked in at a meeting all week. Saw your ping. | 21:35 |
| dansmith | dkranz: t'was on behalf of sdague | 21:35 |
| sdague | dkranz: no worries | 21:35 |
| dkranz | sdague: ok Sorry I missed the meeting | 21:35 |
| dkranz | sdague: Thought had a free slot but things changed :( | 21:36 |
| clarkb | sdague: np, I will hack up what I think I need for e-r to be happy with new logstash schema and rebase as necessary | 21:37 |
| clarkb | sdague: I don't expect it to be horrible | 21:37 |
| sdague | yep, cool, let me know when you need it landed, and I'll be on trigger for it | 21:37 |
| *** mriedem has quit IRC | 21:46 | |
| mtreinish | sdague: https://review.openstack.org/52509 | 21:47 |
| mtreinish | I like how it exploses | 21:47 |
| mtreinish | why do we want to change that | 21:47 |
| sdague | explosing is fun | 21:49 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: don't explode if a query no longer has failures https://review.openstack.org/52509 | 21:50 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Make e-r compatible with old/new logstash schemas. https://review.openstack.org/52521 | 21:55 |
| clarkb | sdague: mtreinish jog0 ^ that *should* be forward and backward compatible, I haven't tested it yet as I think jenkins will do that for me | 21:56 |
| mtreinish | clarkb: the jenkins tests are pretty synthetic I just wrote a fake response to test the classify and is_ready methods | 21:58 |
| clarkb | mtreinish: I see. | 21:59 |
| clarkb | jog0: I know you have it setup to run locally. Any chance I could bug you to give that change a try? | 21:59 |
| jog0 | clarkb: I don't have it set up at the moment but i can in about an hour | 22:00 |
| * jog0 hasn't had lunch today | 22:00 | |
| clarkb | jog0: no rush and thanks | 22:01 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Reorganize project feature config options https://review.openstack.org/52524 | 22:01 |
| *** adalbas has quit IRC | 22:32 | |
| *** HenryG has quit IRC | 22:32 | |
| *** salv-orlando has quit IRC | 22:34 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/52524 failed tempest with an unrecognized error | 22:43 |
| *** mkollaro has quit IRC | 22:46 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/52287 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 and https://bugs.launchpad.net/bugs/1235435 and | 22:58 |
| *** jaypipes has quit IRC | 22:59 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Reorganize project feature config options https://review.openstack.org/52524 | 23:00 |
| *** jamielennox|away is now known as jamielennox | 23:13 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/52253 failed tempest with an unrecognized error | 23:21 |
| *** mriedem has joined #openstack-qa | 23:25 | |
| *** mkollaro has joined #openstack-qa | 23:32 | |
| *** dims has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!