*** harlowja is now known as harlowja_away | 00:01 | |
*** melwitt has quit IRC | 00:02 | |
*** rockyg has quit IRC | 00:02 | |
dtroyer | adam_g: this one? https://etherpad.openstack.org/p/devstack-service-redux | 00:05 |
---|---|---|
adam_g | dtroyer, yeah! | 00:05 |
*** yamahata has joined #openstack-qa | 00:10 | |
*** harlowja_away is now known as harlowja | 00:11 | |
andreaf | mtreinish: ok | 00:15 |
openstackgerrit | A change was merged to openstack-dev/devstack: Correct swift service name in docs https://review.openstack.org/126220 | 00:17 |
openstackrecheck | openstack/python-openstackclient change: https://review.openstack.org/125714 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1286818 | 00:20 |
*** aimon has quit IRC | 00:21 | |
*** yfujioka has joined #openstack-qa | 00:22 | |
*** openstackstatus has quit IRC | 00:23 | |
*** openstackstatus has joined #openstack-qa | 00:24 | |
*** ChanServ sets mode: +v openstackstatus | 00:24 | |
*** bnemec has quit IRC | 00:28 | |
*** arosen has quit IRC | 00:28 | |
jogo | mtreinish: ping | 00:31 |
*** dmorita has joined #openstack-qa | 00:34 | |
*** yjiang5 has joined #openstack-qa | 00:37 | |
openstackgerrit | Joe Gordon proposed a change to openstack/tempest: TEST: Stop suppressing errors during wait_for_server_termination https://review.openstack.org/127085 | 00:38 |
openstackgerrit | Daisuke Morita proposed a change to openstack-dev/devstack: Enable Swift's newer feature of container-sync https://review.openstack.org/126824 | 00:41 |
openstackgerrit | Sergey Shnaidman proposed a change to openstack/tempest: DHCPv6 network tests https://review.openstack.org/117458 | 00:41 |
*** sdmonov has quit IRC | 00:42 | |
*** markmcclain has quit IRC | 00:43 | |
bknudson | tempest runs are seeming sickly -- http://logs.openstack.org/62/127062/1/check/check-tempest-dsvm-full/bc82fee/console.html#_2014-10-08_23_38_48_083 | 00:44 |
*** aimon has joined #openstack-qa | 00:48 | |
mtreinish | jogo: pong | 00:48 |
jogo | mtreinish: so the nova delete thing | 00:49 |
jogo | in tempest | 00:49 |
jogo | the current model of doing try: wait_for_termination except Exception: | 00:50 |
jogo | is hiding bugs | 00:50 |
jogo | at least one libvirt issue | 00:50 |
mtreinish | bknudson: I'd suspect your patch there, because all the jobs are failing with the same auth errors | 00:50 |
jogo | I know you are not a fan of raising errors during class cleanup and whatnot | 00:51 |
mtreinish | jogo: so that might be, but if we can't reliably delete, we can't test because everything will fail | 00:51 |
bknudson | mtreinish: I could very well have broken something. | 00:51 |
jogo | mtreinish: I posted https://review.openstack.org/#/c/127085/ to see what it looks like | 00:51 |
*** JayJ has quit IRC | 00:52 | |
jogo | mtreinish: long term I think we need an approach for how to test this | 00:52 |
jogo | well medium term | 00:52 |
openstackgerrit | Ghanshyam Mann proposed a change to openstack/tempest: Adds test for list/get volume attachments V2 APIs https://review.openstack.org/87191 | 00:52 |
jogo | before nova has proper functional testing and after we make sure we can delete reliably enough to gate on it | 00:52 |
mtreinish | jogo: the other thing is I feel like if the delete was important it'd be directly tested in the test case | 00:52 |
jogo | mtreinish: I think they should be done that way IMHO | 00:53 |
mtreinish | the cleanup stage is really an after thought to prevent leaks | 00:53 |
bknudson | I didn't think that change made any functional changes... should just be docs | 00:53 |
jogo | mtreinish: maybe that is the answer, move tests over to do the delete in test more often | 00:53 |
*** markmcclain has joined #openstack-qa | 00:53 | |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Update README for post config split https://review.openstack.org/125470 | 00:53 |
*** salv-orlando has quit IRC | 00:55 | |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Better check for pypi mirror host connectivity https://review.openstack.org/126290 | 00:55 |
*** mmedvede has joined #openstack-qa | 00:55 | |
mtreinish | bknudson: yeah it looks like just string changes, but the consistent auth failures in multiple jobs seems like it would be more than a coincidence | 00:55 |
mtreinish | jogo: yeah, I think that would be fine | 00:55 |
mtreinish | then we'd be failing as part of the test case and it makes the debug simpler | 00:56 |
bknudson | heat shut down : http://logs.openstack.org/62/127062/1/check/check-tempest-dsvm-full/bc82fee/logs/screen-h-api.txt.gz#_2014-10-08_23_08_53_264 | 00:56 |
bknudson | DuplicateOptError: duplicate option: admin_user | 00:56 |
mtreinish | bknudson: well that would do it :) | 00:57 |
mtreinish | but if that is an issue, why isn't it failing everywhere? | 00:58 |
mtreinish | you're not changing the opt name there | 00:58 |
jogo | mtreinish: cool, I'll start pushing some patches up to do that tomorrow | 00:58 |
mtreinish | just the help strings | 00:58 |
openstackgerrit | A change was merged to openstack-dev/devstack: Stop n-cpu by correct process name /w fake https://review.openstack.org/126420 | 00:59 |
openstackgerrit | A change was merged to openstack/tempest: Fwaas API Test Enhancement https://review.openstack.org/102820 | 00:59 |
mtreinish | jogo: heh, well if gate failures go up, I'm going to point the angry mob at you :) | 00:59 |
bknudson | for some reason it's using keystoneclient.middleware.auth_token rather than keystonemiddleware.auth_token | 01:00 |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Remove TESTR_FULL option for running tempest https://review.openstack.org/124190 | 01:01 |
bknudson | so that might be it... a mismatch because keystoneclient auth_token help text is different than keystonemiddleware auth_token help text. | 01:01 |
bknudson | I think that would explain it. | 01:01 |
jogo | mtreinish: fair enough | 01:01 |
*** JayJ has joined #openstack-qa | 01:05 | |
*** JayJ has quit IRC | 01:06 | |
*** JayJ has joined #openstack-qa | 01:07 | |
*** JayJ has quit IRC | 01:07 | |
mtreinish | bknudson: yeah I think that could do it | 01:08 |
bknudson | mtreinish: the heat code still has refs to keystoneclient.middleware... I should have caught those. :( | 01:09 |
*** topshare has joined #openstack-qa | 01:23 | |
*** edwa6643 has joined #openstack-qa | 01:24 | |
*** edwa66431 has quit IRC | 01:26 | |
*** suneelb has quit IRC | 01:39 | |
*** oomichi_ has joined #openstack-qa | 01:44 | |
*** edwa6643 has left #openstack-qa | 01:46 | |
openstackrecheck | openstack/keystone change: https://review.openstack.org/119946 failed gate-keystone-python26, gate-keystone-python27 in the gate queue with an unrecognized error | 01:47 |
*** mmedvede has quit IRC | 01:48 | |
bknudson | unbelievable | 01:54 |
*** markmcclain has quit IRC | 01:55 | |
*** dimsum_ has joined #openstack-qa | 02:06 | |
*** nosnos has joined #openstack-qa | 02:06 | |
openstackrecheck | openstack/python-glanceclient change: https://review.openstack.org/125965 failed because of: gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/1370496 https://bugs.launchpad.net/bugs/1353962 | 02:08 |
*** weshay has quit IRC | 02:25 | |
*** Longgeek has joined #openstack-qa | 02:26 | |
*** dimsum_ has quit IRC | 02:28 | |
*** dimsum_ has joined #openstack-qa | 02:28 | |
*** zz_dimtruck is now known as dimtruck | 02:31 | |
*** dimsum_ has quit IRC | 02:33 | |
*** toure has joined #openstack-qa | 02:45 | |
*** openstackgerrit has quit IRC | 02:45 | |
*** yfujioka has quit IRC | 02:52 | |
*** mjturek has quit IRC | 02:54 | |
*** coolsvap|afk is now known as coolsvap | 02:55 | |
*** edwa66431 has joined #openstack-qa | 02:58 | |
*** pcrews has quit IRC | 03:03 | |
*** ramishra has joined #openstack-qa | 03:03 | |
*** rushiagr_away is now known as rushiagr | 03:12 | |
*** toure has quit IRC | 03:21 | |
*** harlowja is now known as harlowja_away | 03:29 | |
*** harlowja_away is now known as harlowja | 03:31 | |
*** bknudson has quit IRC | 03:33 | |
*** nosnos has quit IRC | 03:35 | |
*** ramishra has quit IRC | 03:37 | |
*** ramishra has joined #openstack-qa | 03:37 | |
*** coolsvap is now known as coolsvap|afk | 03:37 | |
*** coolsvap|afk is now known as coolsvap | 03:45 | |
*** ramishra has quit IRC | 03:48 | |
*** gmann has quit IRC | 04:03 | |
*** gmann has joined #openstack-qa | 04:03 | |
*** yfujioka has joined #openstack-qa | 04:04 | |
*** rushiagr is now known as rushiagr_away | 04:08 | |
*** edwa66431 has quit IRC | 04:10 | |
*** ramishra has joined #openstack-qa | 04:13 | |
*** chandankumar has joined #openstack-qa | 04:17 | |
*** ramishra has quit IRC | 04:24 | |
*** e0ne has joined #openstack-qa | 04:34 | |
*** amotoki has joined #openstack-qa | 04:34 | |
*** nosnos has joined #openstack-qa | 04:35 | |
*** rushiagr_away is now known as rushiagr | 04:37 | |
*** Poornima has joined #openstack-qa | 04:38 | |
*** e0ne has quit IRC | 04:39 | |
*** e0ne has joined #openstack-qa | 04:52 | |
*** chandankumar has quit IRC | 04:52 | |
*** SridharG has joined #openstack-qa | 04:52 | |
*** e0ne has quit IRC | 04:56 | |
*** Madan has joined #openstack-qa | 05:02 | |
*** jamielennox has quit IRC | 05:04 | |
*** jamielennox has joined #openstack-qa | 05:08 | |
*** swamireddy has joined #openstack-qa | 05:18 | |
*** harlowja is now known as harlowja_away | 05:35 | |
*** neeti has joined #openstack-qa | 05:37 | |
*** dimtruck is now known as zz_dimtruck | 05:42 | |
*** afazekas has joined #openstack-qa | 05:51 | |
*** yamahata has quit IRC | 05:51 | |
*** yamahata has joined #openstack-qa | 05:51 | |
*** coolsvap is now known as coolsvap|afk | 06:00 | |
*** ramishra has joined #openstack-qa | 06:04 | |
*** chandankumar has joined #openstack-qa | 06:04 | |
*** neeti has quit IRC | 06:06 | |
*** ramishra has quit IRC | 06:09 | |
*** coolsvap|afk is now known as coolsvap | 06:12 | |
*** pkoniszewski has joined #openstack-qa | 06:13 | |
*** jlibosva has joined #openstack-qa | 06:22 | |
*** chandankumar has quit IRC | 06:27 | |
*** chandankumar has joined #openstack-qa | 06:28 | |
*** pcrews has joined #openstack-qa | 06:39 | |
*** vigneshvar has joined #openstack-qa | 06:42 | |
*** pcrews has quit IRC | 06:47 | |
*** topshare has quit IRC | 06:48 | |
*** topshare has joined #openstack-qa | 06:51 | |
*** markus_z has joined #openstack-qa | 06:53 | |
*** topshare has quit IRC | 06:54 | |
*** gszasz has joined #openstack-qa | 06:56 | |
*** topshare has joined #openstack-qa | 06:56 | |
*** k4n0 has joined #openstack-qa | 06:56 | |
*** flaper87|afk is now known as flaper87 | 06:58 | |
*** k4n0 has quit IRC | 07:02 | |
*** topshare has quit IRC | 07:03 | |
*** topshare has joined #openstack-qa | 07:07 | |
*** topshare has quit IRC | 07:09 | |
*** topshare has joined #openstack-qa | 07:10 | |
*** topshare has quit IRC | 07:13 | |
*** k4n0 has joined #openstack-qa | 07:14 | |
*** topshare has joined #openstack-qa | 07:18 | |
*** andreaf has quit IRC | 07:26 | |
*** topshare has quit IRC | 07:29 | |
*** topshare has joined #openstack-qa | 07:29 | |
*** Albert__13 has joined #openstack-qa | 07:33 | |
*** Albert_13 has joined #openstack-qa | 07:34 | |
*** topshare has quit IRC | 07:35 | |
*** topshare has joined #openstack-qa | 07:35 | |
*** luqas has joined #openstack-qa | 07:35 | |
*** topshare has quit IRC | 07:36 | |
*** topshare has joined #openstack-qa | 07:37 | |
*** k4n0 has quit IRC | 07:42 | |
*** yjiang5 has quit IRC | 07:45 | |
*** ramishra has joined #openstack-qa | 07:45 | |
*** Albert_13 has left #openstack-qa | 07:46 | |
*** jordanP has joined #openstack-qa | 07:47 | |
*** topshare has quit IRC | 07:47 | |
*** jlanoux has joined #openstack-qa | 07:48 | |
*** ramishra has quit IRC | 07:50 | |
*** kashyap has joined #openstack-qa | 07:51 | |
*** gfidente has quit IRC | 07:53 | |
openstackrecheck | openstack/cinder change: https://review.openstack.org/120306 failed gate-cinder-pep8 in the gate queue with an unrecognized error | 07:54 |
*** k4n0 has joined #openstack-qa | 07:54 | |
*** dmellado_ has joined #openstack-qa | 07:56 | |
*** dmellado_ has quit IRC | 07:57 | |
*** mpaolino has joined #openstack-qa | 08:00 | |
*** safchain has joined #openstack-qa | 08:00 | |
*** gfidente has joined #openstack-qa | 08:01 | |
*** jpich has joined #openstack-qa | 08:05 | |
*** qba73 has joined #openstack-qa | 08:08 | |
*** yamamoto has quit IRC | 08:08 | |
*** Matt1 has joined #openstack-qa | 08:08 | |
*** shardy_z is now known as shardy | 08:09 | |
*** swamireddy has quit IRC | 08:09 | |
*** topshare has joined #openstack-qa | 08:10 | |
*** topshare has quit IRC | 08:10 | |
*** topshare has joined #openstack-qa | 08:10 | |
*** tkammer has quit IRC | 08:10 | |
*** salv-orlando has joined #openstack-qa | 08:11 | |
*** swamireddy has joined #openstack-qa | 08:12 | |
*** topshare has quit IRC | 08:13 | |
*** tkammer has joined #openstack-qa | 08:14 | |
*** tkammer has quit IRC | 08:18 | |
*** gfidente has quit IRC | 08:24 | |
*** oomichi_ has quit IRC | 08:24 | |
*** tkammer has joined #openstack-qa | 08:34 | |
*** gszasz has quit IRC | 08:37 | |
*** gszasz has joined #openstack-qa | 08:42 | |
*** tkammer has quit IRC | 08:42 | |
*** vigneshvar has quit IRC | 08:45 | |
*** ramishra has joined #openstack-qa | 08:46 | |
*** SridharG has left #openstack-qa | 08:47 | |
*** ramishra has quit IRC | 08:51 | |
*** tkammer has joined #openstack-qa | 08:57 | |
*** tkammer has quit IRC | 09:02 | |
*** SridharG has joined #openstack-qa | 09:09 | |
*** tkammer has joined #openstack-qa | 09:17 | |
*** zz_johnthetubagu is now known as johnthetubaguy | 09:17 | |
*** gfidente has joined #openstack-qa | 09:19 | |
*** qba73 has quit IRC | 09:26 | |
chandankumar | gmann, hello | 09:30 |
*** SridharG has left #openstack-qa | 09:31 | |
chandankumar | gmann, sent the new patch set with modification : https://review.openstack.org/#/c/121055/ | 09:31 |
chandankumar | gmann, please review it when you are free | 09:31 |
*** SridharG has joined #openstack-qa | 09:38 | |
*** coolsvap is now known as coolsvap|afk | 09:56 | |
*** jhenner has quit IRC | 09:56 | |
*** ramishra has joined #openstack-qa | 09:56 | |
*** jlanoux has quit IRC | 09:57 | |
*** tkammer has quit IRC | 10:00 | |
*** ramishra has quit IRC | 10:01 | |
*** dmorita has quit IRC | 10:05 | |
*** jhenner has joined #openstack-qa | 10:06 | |
*** ramishra has joined #openstack-qa | 10:06 | |
*** cdent has joined #openstack-qa | 10:08 | |
*** tnurlygayanov has joined #openstack-qa | 10:14 | |
*** ramishra has quit IRC | 10:14 | |
*** yamahata has quit IRC | 10:19 | |
*** tosky has joined #openstack-qa | 10:21 | |
*** tkammer has joined #openstack-qa | 10:22 | |
*** ramishra has joined #openstack-qa | 10:24 | |
*** psedlak_ has joined #openstack-qa | 10:25 | |
*** aix_ has quit IRC | 10:38 | |
*** dimsum_ has joined #openstack-qa | 10:51 | |
*** vigneshvar has joined #openstack-qa | 10:59 | |
*** tkammer has quit IRC | 11:01 | |
*** ramishra has quit IRC | 11:06 | |
*** aix_ has joined #openstack-qa | 11:07 | |
*** dimsum_ has quit IRC | 11:13 | |
*** dimsum_ has joined #openstack-qa | 11:13 | |
*** dims_ has joined #openstack-qa | 11:14 | |
*** tkammer has joined #openstack-qa | 11:14 | |
*** dimsum_ has quit IRC | 11:17 | |
*** dmellado_ has joined #openstack-qa | 11:17 | |
*** dmellado_ has quit IRC | 11:18 | |
*** Madan has quit IRC | 11:23 | |
*** dmellado_ has joined #openstack-qa | 11:25 | |
*** dmellado_ has quit IRC | 11:25 | |
*** Madan has joined #openstack-qa | 11:26 | |
*** amotoki_ has joined #openstack-qa | 11:34 | |
*** amotoki_ has quit IRC | 11:34 | |
*** ramishra has joined #openstack-qa | 11:37 | |
*** gmann_ has joined #openstack-qa | 11:37 | |
gmann_ | mtreinish: I have updated bug numbers for meeting- https://etherpad.openstack.org/p/Tempest-bug-report | 11:39 |
*** kashyap has quit IRC | 11:40 | |
*** ramishra has quit IRC | 11:41 | |
gmann_ | mtreinish: I re-spin Nova V2.1 tests spec - https://review.openstack.org/#/c/96661/ Please have a look. | 11:42 |
masayukig | gmann: Thank you for updating :) I'll try to triage some bugs later.. | 11:43 |
*** amotoki has quit IRC | 11:45 | |
*** coolsvap|afk is now known as coolsvap | 11:46 | |
*** hockeynut has quit IRC | 11:46 | |
*** swamireddy has quit IRC | 11:46 | |
gmann_ | masayukig: np :) | 11:46 |
*** yfujioka has quit IRC | 11:48 | |
*** hockeynut has joined #openstack-qa | 11:48 | |
*** mgagne has quit IRC | 11:48 | |
*** SridharG has quit IRC | 11:49 | |
*** edwa6643 has joined #openstack-qa | 11:50 | |
*** StevenK has quit IRC | 11:50 | |
*** mgagne has joined #openstack-qa | 11:51 | |
*** mgagne is now known as Guest92054 | 11:51 | |
*** StevenK has joined #openstack-qa | 11:51 | |
*** SridharG has joined #openstack-qa | 11:52 | |
*** gmann_ has quit IRC | 11:54 | |
*** edwa6643 has quit IRC | 11:54 | |
*** edwa6643 has joined #openstack-qa | 11:58 | |
*** gszasz has quit IRC | 11:58 | |
*** chandankumar has quit IRC | 11:59 | |
*** aix_ has quit IRC | 11:59 | |
*** tkammer has quit IRC | 12:01 | |
*** coolsvap is now known as coolsvap|afk | 12:01 | |
*** jhenner has quit IRC | 12:01 | |
*** jhenner has joined #openstack-qa | 12:02 | |
*** mmedvede has joined #openstack-qa | 12:04 | |
*** afazekas has quit IRC | 12:04 | |
*** tosky has quit IRC | 12:06 | |
*** weshay has joined #openstack-qa | 12:10 | |
*** mpaolino has quit IRC | 12:11 | |
*** chandankumar has joined #openstack-qa | 12:13 | |
*** cnesa16 has joined #openstack-qa | 12:13 | |
*** Madan has quit IRC | 12:13 | |
*** markmcclain has joined #openstack-qa | 12:15 | |
*** mkollaro has joined #openstack-qa | 12:15 | |
*** tkammer has joined #openstack-qa | 12:16 | |
*** rushiagr is now known as rushiagr_away | 12:20 | |
*** openstackgerrit has joined #openstack-qa | 12:23 | |
*** dims_ has quit IRC | 12:23 | |
*** mpaolino has joined #openstack-qa | 12:24 | |
*** dimsum_ has joined #openstack-qa | 12:24 | |
*** ramishra has joined #openstack-qa | 12:25 | |
*** salv-orlando has quit IRC | 12:27 | |
openstackgerrit | A change was merged to openstack-dev/devstack: Install latest EPEL release https://review.openstack.org/127080 | 12:33 |
*** nosnos has quit IRC | 12:35 | |
*** nosnos has joined #openstack-qa | 12:36 | |
*** nosnos has quit IRC | 12:40 | |
*** mpaolino has quit IRC | 12:42 | |
cdent | sdad (that's awesome, congrats!), jogo, mtreinish: when you get a chance can you put some eyes on https://review.openstack.org/#/c/115212/ it will probably be a good exerciser of the run_process changes | 12:44 |
*** adalbas has joined #openstack-qa | 12:48 | |
*** JayJ has joined #openstack-qa | 12:49 | |
*** rushiagr_away is now known as rushiagr | 12:49 | |
*** aix has joined #openstack-qa | 12:50 | |
*** dkranz has quit IRC | 12:54 | |
*** jhenner1 has joined #openstack-qa | 12:54 | |
*** tosky has joined #openstack-qa | 12:54 | |
*** jhenner1 has quit IRC | 12:54 | |
*** jhenner1 has joined #openstack-qa | 12:55 | |
*** tkammer has quit IRC | 12:55 | |
*** aysyd has joined #openstack-qa | 12:55 | |
*** zz_dimtruck is now known as dimtruck | 12:55 | |
*** mpaolino has joined #openstack-qa | 12:57 | |
*** mpaolino has quit IRC | 12:57 | |
*** jhenner has quit IRC | 12:57 | |
*** amotoki has joined #openstack-qa | 13:01 | |
*** ramishra has quit IRC | 13:03 | |
*** gszasz has joined #openstack-qa | 13:03 | |
*** dimtruck is now known as zz_dimtruck | 13:05 | |
*** tkammer has joined #openstack-qa | 13:08 | |
*** zz_dimtruck is now known as dimtruck | 13:10 | |
*** jaypipes has joined #openstack-qa | 13:12 | |
*** ChanServ sets mode: +o jaypipes | 13:12 | |
*** zz_jgrimm is now known as jgrimm | 13:13 | |
*** mriedem has joined #openstack-qa | 13:18 | |
*** vigneshvar has quit IRC | 13:19 | |
*** gszasz has quit IRC | 13:24 | |
*** dpaterson has joined #openstack-qa | 13:24 | |
*** dimtruck is now known as zz_dimtruck | 13:24 | |
*** dustins has joined #openstack-qa | 13:26 | |
EmilienM | jlibosva: I can rebase the patch of javelin/neutron if you don't mind | 13:29 |
jlibosva | EmilienM: go ahead, I just did recheck cause I have a feeling that -1 from Jenkins scares reviewers | 13:30 |
openstackgerrit | Emilien Macchi proposed a change to openstack/tempest: javelin: add network and secgroup resources https://review.openstack.org/100169 | 13:31 |
EmilienM | jlibosva: done | 13:31 |
jlibosva | EmilienM++ | 13:31 |
*** markmcclain has quit IRC | 13:33 | |
openstackgerrit | Zu Qiang proposed a change to openstack-dev/devstack: Reset no_proxy when the node is installed behind a proxy https://review.openstack.org/94988 | 13:37 |
*** aysyd has quit IRC | 13:40 | |
EmilienM | jlibosva: I started some work on unit testing, but no time to continue this week. I'll see this WE a bit. | 13:40 |
jlibosva | EmilienM: ok | 13:41 |
*** gszasz has joined #openstack-qa | 13:42 | |
*** afazekas has joined #openstack-qa | 13:45 | |
mtreinish | cdent: I put a +2 on the unskip, I think now is a good time to try running that again | 13:46 |
cdent | cool thank you mtreinish | 13:46 |
*** chandankumar has quit IRC | 13:48 | |
*** gordc has joined #openstack-qa | 13:48 | |
*** jecarey has quit IRC | 13:50 | |
*** jgrimm is now known as zz_jgrimm | 13:59 | |
openstackgerrit | Ashish Kumar Gupta proposed a change to openstack/tempest: Port API Tests Enhancements https://review.openstack.org/107277 | 14:01 |
*** toure_ has joined #openstack-qa | 14:02 | |
*** chandankumar has joined #openstack-qa | 14:02 | |
*** kashyap has joined #openstack-qa | 14:04 | |
*** jecarey has joined #openstack-qa | 14:04 | |
*** dmellado has quit IRC | 14:07 | |
*** zz_dimtruck is now known as dimtruck | 14:09 | |
*** dmellado has joined #openstack-qa | 14:10 | |
*** salv-orlando has joined #openstack-qa | 14:10 | |
*** pkoniszewski has quit IRC | 14:11 | |
*** dkranz has joined #openstack-qa | 14:13 | |
*** chandankumar has quit IRC | 14:18 | |
*** mwagner_lap has joined #openstack-qa | 14:19 | |
*** bnemec has joined #openstack-qa | 14:26 | |
*** bknudson has joined #openstack-qa | 14:26 | |
*** dimtruck is now known as zz_dimtruck | 14:29 | |
*** chandankumar has joined #openstack-qa | 14:32 | |
*** ramishra has joined #openstack-qa | 14:37 | |
*** dustins has quit IRC | 14:37 | |
*** zz_dimtruck is now known as dimtruck | 14:37 | |
*** rbak has joined #openstack-qa | 14:38 | |
*** dustins has joined #openstack-qa | 14:40 | |
*** markmcclain has joined #openstack-qa | 14:41 | |
*** markmcclain has quit IRC | 14:41 | |
*** k4n0 has quit IRC | 14:42 | |
*** ramishra has quit IRC | 14:42 | |
*** pkoniszewski has joined #openstack-qa | 14:42 | |
*** markmcclain has joined #openstack-qa | 14:43 | |
*** zz_jgrimm is now known as jgrimm | 14:46 | |
*** chandankumar has quit IRC | 14:48 | |
*** Poornima has quit IRC | 14:49 | |
*** pcrews has joined #openstack-qa | 14:52 | |
*** JayJ has quit IRC | 14:53 | |
*** markmcclain has quit IRC | 14:53 | |
*** JayJ has joined #openstack-qa | 14:54 | |
*** yamahata has joined #openstack-qa | 14:58 | |
*** atiwari has joined #openstack-qa | 15:03 | |
*** davpat2112 has joined #openstack-qa | 15:13 | |
*** dpaterson has quit IRC | 15:15 | |
*** ashishg has joined #openstack-qa | 15:15 | |
*** salv-orlando has quit IRC | 15:15 | |
openstackrecheck | Console logs not available after 13:26s for gate-horizon-python27 126247,6,6481115 | 15:19 |
*** amotoki has quit IRC | 15:20 | |
*** gszasz has quit IRC | 15:22 | |
*** kashyap has quit IRC | 15:23 | |
*** mlavalle has joined #openstack-qa | 15:23 | |
*** corvus is now known as jeblair | 15:30 | |
*** ashishg has quit IRC | 15:32 | |
*** dustins has quit IRC | 15:34 | |
*** dustins has joined #openstack-qa | 15:37 | |
*** ramishra has joined #openstack-qa | 15:37 | |
*** pkoniszewski has quit IRC | 15:38 | |
*** chandankumar has joined #openstack-qa | 15:39 | |
*** chandankumar has quit IRC | 15:41 | |
*** chandankumar has joined #openstack-qa | 15:42 | |
*** ramishra has quit IRC | 15:43 | |
*** kashyap has joined #openstack-qa | 15:52 | |
*** markmcclain has joined #openstack-qa | 15:56 | |
*** markus_z has quit IRC | 15:56 | |
dkranz | jogo: about https://review.openstack.org/#/c/123562/ (server termination) | 15:58 |
dkranz | jogo: I can approve your patch | 15:58 |
*** markmcclain1 has joined #openstack-qa | 15:58 | |
dkranz | jogo: But my concern is that most tests do not follow the path you are adding/fixing | 15:58 |
dkranz | jogo: they use create_test_server | 15:58 |
dkranz | jogo: so why not just make wait_for_termination fail if it does not succeed as in your other proposed patch? | 15:59 |
dkranz | jogo: that is, there is nothing wrong with the change, but it only impacts a tiny set of actual server deletions | 15:59 |
*** markmcclain has quit IRC | 16:00 | |
*** mkollaro has quit IRC | 16:02 | |
*** Guest92054 is now known as mgagne | 16:03 | |
*** markmcclain1 has quit IRC | 16:03 | |
*** mkollaro has joined #openstack-qa | 16:03 | |
*** mgagne is now known as mgagne_ | 16:03 | |
*** aimon has quit IRC | 16:03 | |
*** markmcclain has joined #openstack-qa | 16:04 | |
*** mgagne_ is now known as Guest31178 | 16:04 | |
*** Guest31178 has quit IRC | 16:05 | |
*** Guest31178 has joined #openstack-qa | 16:05 | |
*** gszasz has joined #openstack-qa | 16:05 | |
*** yamahata has quit IRC | 16:06 | |
*** safchain has quit IRC | 16:07 | |
*** Guest31178 is now known as mgagne | 16:07 | |
openstackrecheck | openstack/glance change: https://review.openstack.org/127248 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error | 16:08 |
*** dimtruck is now known as zz_dimtruck | 16:12 | |
*** cbader has joined #openstack-qa | 16:15 | |
*** cdent has quit IRC | 16:23 | |
*** yjiang5 has joined #openstack-qa | 16:26 | |
*** zz_dimtruck is now known as dimtruck | 16:29 | |
*** swamireddy has joined #openstack-qa | 16:32 | |
*** aimon has joined #openstack-qa | 16:32 | |
*** melwitt has joined #openstack-qa | 16:34 | |
*** gszasz has quit IRC | 16:35 | |
*** mkollaro has quit IRC | 16:37 | |
*** ramishra has joined #openstack-qa | 16:38 | |
*** jpich has quit IRC | 16:39 | |
*** luqas has quit IRC | 16:40 | |
*** Albert__13 has quit IRC | 16:41 | |
*** melwitt has quit IRC | 16:41 | |
*** melwitt has joined #openstack-qa | 16:41 | |
*** ramishra has quit IRC | 16:42 | |
mtreinish | dkranz: if you've got a sec can you review: https://review.openstack.org/#/c/121988/ | 16:50 |
mtreinish | after that lands I'm going to push out the first tempest-lib release | 16:50 |
*** vrovachev has quit IRC | 16:52 | |
*** johnthetubaguy is now known as zz_johnthetubagu | 16:52 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: DO NOT MERGE: Test tempest-lib stuff https://review.openstack.org/127291 | 16:53 |
dkranz | mtreinish: ok | 16:55 |
mtreinish | dkranz: thanks | 16:56 |
*** jordanP has quit IRC | 16:56 | |
*** andreaf has joined #openstack-qa | 17:05 | |
*** jlibosva has quit IRC | 17:05 | |
*** SridharG has quit IRC | 17:06 | |
openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/126490 failed because of: gate-tempest-dsvm-neutron-full: https://bugs.launchpad.net/bugs/1352498 https://bugs.launchpad.net/bugs/1348204 https://bugs.launchpad.net/bugs/1374458, gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/1370496 | 17:06 |
*** salv-orlando has joined #openstack-qa | 17:08 | |
*** gfidente has quit IRC | 17:08 | |
*** swamireddy1 has joined #openstack-qa | 17:16 | |
*** swamireddy has quit IRC | 17:17 | |
*** harlowja_away is now known as harlowja | 17:19 | |
*** tosky has quit IRC | 17:24 | |
openstackgerrit | Sébastien Han proposed a change to openstack-dev/devstack: Add the ability to activate Nova ceph without Cinder https://review.openstack.org/126204 | 17:26 |
*** yfried has joined #openstack-qa | 17:34 | |
*** dimtruck is now known as zz_dimtruck | 17:35 | |
*** suneelb has joined #openstack-qa | 17:43 | |
*** tosky has joined #openstack-qa | 17:47 | |
*** chandankumar has quit IRC | 17:49 | |
*** davpat2112 has quit IRC | 17:50 | |
openstackrecheck | openstack/tempest change: https://review.openstack.org/115212 failed gate-tempest-dsvm-neutron-full in the gate queue with an unrecognized error | 17:51 |
*** boden has joined #openstack-qa | 17:51 | |
*** swamireddy1 has quit IRC | 17:53 | |
*** aix has quit IRC | 17:53 | |
*** SridharG has joined #openstack-qa | 17:55 | |
dkranz | jogo: I have to go to another meeting and will be back in an hour | 17:56 |
dkranz | jogo: If I am just wrong I can just approve your patch, or mtreinish can | 17:56 |
mtreinish | dkranz: it will cause a failure, just in the cleanup phase which is not always an obvious location for debug | 17:57 |
jogo | dkranz: right, I want a failed delete to cause tests to fail. but not during the cleanup phase | 17:58 |
dkranz | jogo: you mean not during the class cleanup phase, but fail during addCleanUp phase? | 18:00 |
jogo | dkranz: I think so yes | 18:00 |
mtreinish | dkranz: no, make delete and wait on delete part of the test | 18:00 |
*** SridharG has left #openstack-qa | 18:00 | |
jogo | dkranz: tests should explicitly clean up the server | 18:00 |
*** mattoliverau has quit IRC | 18:00 | |
*** jhesketh has quit IRC | 18:00 | |
*** mattoliverau has joined #openstack-qa | 18:01 | |
dkranz | jogo: Then we should not have class cleanup at all | 18:01 |
jogo | dkranz: well class cleanup is a fall back | 18:01 |
jogo | of sorts | 18:01 |
jogo | dkranz: but maybe not | 18:01 |
dkranz | jogo: wouldn't it be better to make create_test_server just do the addCleanUp then? | 18:01 |
*** jhesketh has joined #openstack-qa | 18:02 | |
dkranz | jogo: And fix the tests that are not using create_test_server to use it? | 18:02 |
jogo | dkranz: as long as that causes the test to fail not during the cleanup stage | 18:02 |
mtreinish | dkranz: addCleanup would be the same problem | 18:02 |
jogo | as in if you can see this job failed versus this cleanup failed | 18:02 |
dkranz | jogo: well you're change is just putting in an addCleanUp | 18:02 |
*** zz_dimtruck is now known as dimtruck | 18:02 | |
dkranz | jogo: which actually confused me a little because I thought addCleanUp kind of ignored failures | 18:02 |
mtreinish | dkranz: that patch just adds a wait to make sure we're not just issuing an async delete and moving on | 18:03 |
jogo | mtreinish: I do use addcleanup right now https://review.openstack.org/#/c/123562/2/tempest/api/compute/admin/test_servers.py | 18:03 |
dkranz | mtreinish: yes, but the failure will still happen during cleanup, right? | 18:03 |
mtreinish | the fixes jogo is talking about will have to come later and will require changing some tests | 18:03 |
mtreinish | dkranz: yes, but failures in addCleanup can be very confusing | 18:04 |
dkranz | mtreinish: I know | 18:04 |
jogo | so maybe I should revise my patch to not use addcleanup | 18:04 |
dkranz | jogo: that would better achieve what you want in these cases but leave untouched the majority of server creates which use create_test_server | 18:05 |
jogo | dkranz: so I need to revise those next | 18:05 |
jogo | should these tests use create_test_server ? | 18:05 |
dkranz | jogo: seems like you want a try/finally for deletion that does not use addCleanUp | 18:05 |
jogo | dkranz: good idea | 18:06 |
dkranz | jogo: But the implementation of that would look exactly like what addCleanUp does I think | 18:06 |
dkranz | jogo: this is not so easy | 18:07 |
jogo | dkranz: yeah, none of the answers we have are ideal | 18:07 |
mtreinish | dkranz: the other option would be to just wrap cleanup to detect a 404 and ignore it | 18:07 |
mtreinish | s/cleanup/delete/ | 18:08 |
mtreinish | that way you can add the cleanup for protection | 18:08 |
jogo | what about https://docs.python.org/2/library/unittest.html#unittest.TestCase.doCleanups | 18:08 |
*** andreaf has quit IRC | 18:08 | |
dkranz | jogo: yeah, we could hook in there | 18:09 |
jogo | try: finally: do cleanups | 18:09 |
dkranz | jogo: got to run, back at next hour | 18:10 |
jogo | dkranz: kk | 18:10 |
*** danfinch has quit IRC | 18:13 | |
*** e0ne has joined #openstack-qa | 18:28 | |
*** dustins has quit IRC | 18:31 | |
openstackgerrit | Sergey Shnaidman proposed a change to openstack/tempest: Create subnet without gateway and explicit IP ver https://review.openstack.org/118424 | 18:32 |
*** suneelb has quit IRC | 18:32 | |
*** suneelb has joined #openstack-qa | 18:32 | |
*** dustins has joined #openstack-qa | 18:33 | |
*** suneelb has quit IRC | 18:33 | |
*** suneelb has joined #openstack-qa | 18:34 | |
*** kashyap has quit IRC | 18:34 | |
*** ramishra has joined #openstack-qa | 18:40 | |
*** chandankumar has joined #openstack-qa | 18:44 | |
*** ramishra has quit IRC | 18:44 | |
openstackgerrit | A change was merged to openstack/tempest-lib: Don't edit commit history with migrate from tempest tool https://review.openstack.org/121988 | 18:47 |
*** andreaf has joined #openstack-qa | 18:47 | |
*** e0ne has quit IRC | 18:48 | |
*** rushiagr is now known as rushiagr_away | 19:00 | |
dkranz | jogo: saw your comment in the patch we were discussing. The only downside to the suggested approach is that we will be serializing execution around the synchronous deletion time which is now done in the background. afazekas noted this in his review comment. | 19:02 |
dkranz | jogo: but obviously we can't both do cleanup in the background and do it before the test completes | 19:03 |
dkranz | jogo: One possible compromise would be to keep the class level cleanup but record more information so that the log message tells you which test "failed" | 19:04 |
dkranz | jogo: I almost think we would be better off not trying to track these failures but running explicit stress tests around deletion | 19:06 |
*** dustins has quit IRC | 19:06 | |
dkranz | jogo: parallel tempest in the constrained dsg environment is a kind of stress test anyway, but not a good one. | 19:06 |
dkranz | jogo: there already is such a test in the tempest stress framewoek | 19:07 |
*** toure_ has quit IRC | 19:07 | |
*** mwagner_lap has quit IRC | 19:08 | |
openstackrecheck | Console logs not available after 13:33s for check-tempest-dsvm-neutron-heat-slow-icehouse 107654,4,c52f90d | 19:10 |
*** jordanP has joined #openstack-qa | 19:11 | |
*** dustins has joined #openstack-qa | 19:11 | |
afazekas | dkranz: AFAIK now we do not have the logs from class level operations, just one exception | 19:19 |
dkranz | afazekas: you mean from BaseComputeTest.clear_servers, I think that is right | 19:20 |
dkranz | afazekas: But we could include, for example, the test in which the test was created and print that out too | 19:21 |
dkranz | afazekas: But I don't know if that would satisfy the need. | 19:21 |
dkranz | afazekas: Honestly, I think we would do better to run a lot of stress jobs and debug these issues there | 19:21 |
afazekas | dkranz: BTW at tempest/test.py at L239 we might want to remove the try block | 19:22 |
*** cbader has quit IRC | 19:22 | |
dkranz | afazekas: I don't see a try block there ?? | 19:22 |
afazekas | dkranz: It is late here and I am hungry , lets continue it tomorrow | 19:22 |
dkranz | afazekas: github url? | 19:22 |
dkranz | afazekas: ok :) | 19:23 |
afazekas | L293 | 19:23 |
*** chandankumar has quit IRC | 19:23 | |
dkranz | mtreinish, andreaf : I question whether we want the try block here https://github.com/openstack/tempest/blob/master/tempest/test.py#L292 | 19:25 |
dkranz | Why should we delete the tenant if the call to clean its resources fails? | 19:25 |
dkranz | What is the benefit? | 19:25 |
dkranz | I thought the point was to make source resource clean was always called, not to delete tenants if it fails. | 19:26 |
*** atiwari has quit IRC | 19:26 | |
dkranz | If we delete the tenant then we strand the resources and things like the cleanup script cannot run | 19:26 |
dkranz | afazekas: I think you are right ^^^ | 19:28 |
*** afazekas has quit IRC | 19:28 | |
mtreinish | dkranz: it's needed for the test-accounts work, if we don't ensure we properly release the credentials at the end of a test class we'll exhaust the set | 19:30 |
dkranz | mtreinish: But if resource_cleanup fails, there may be objects still in the tenant, and re-using it may fail in strange ways | 19:30 |
adam_g | javelin question.. i'd like to make creation of certain resources conditional on the services that are deployed. would it be preferred to modify the javelin code in tempest to skip creation of $foo if $bar, or to update grenade to generate a resources.yaml based on ENABLED_SERVICES or similar? | 19:31 |
adam_g | jogo, ^ | 19:31 |
mtreinish | dkranz: it depends on the failure really. But, in the case of resource cred list exhaustion the failures would be opaque, (InvalidConfig error in setupclasss) but I feel like an unexpected resource causing a failure would at least show the source of the issue | 19:33 |
mtreinish | the other thing is we were doing a really poor job of handling this before that was added | 19:34 |
mtreinish | dkranz: if you've got a suggestion on how to improve that logic push up a patch. | 19:34 |
mtreinish | but things there will change a bit when andreaf starts working on the split for the second part of that bp | 19:35 |
dkranz | mtreinish: I am thinking about it but need to study how exactly the test account flow works. I reviewed some of the patches but it was a gwhile ago. | 19:35 |
dkranz | mtreinish: I thought what we were doing a poor job of before was making sure that the "resource deleter stuff" was always called regardless of what happened during setUpClass or test execution. | 19:36 |
mtreinish | adam_g: I think they're trying to decouple javelin as much as possible from the tempest config | 19:36 |
dkranz | mtreinish: anyway, I'll think about it. | 19:36 |
mtreinish | dkranz: yeah, but then when we added that (or had it with safe_setup) the issue became skips and attr errors | 19:37 |
*** tkammer has quit IRC | 19:37 | |
adam_g | mtreinish, makes sense | 19:37 |
dkranz | mtreinish: Yes, but those are different uses. Do me, the tenant deletion even on error came along with that but is not desired. | 19:37 |
mtreinish | the tearDownClasses often depended on things being set in the setupclass which wasn't true when we raised an exception and called tearDownclass | 19:37 |
mtreinish | dkranz: as a first pass you can switch that except to just be attr error | 19:38 |
dkranz | mtreinish: yes, I don't deny that | 19:38 |
mtreinish | and see what happens | 19:38 |
dkranz | mtreinish: I think that is on the right track. We should only catch exceptions and not fail if they are exceptions that we are anticipating | 19:39 |
*** e0ne has joined #openstack-qa | 19:39 | |
mtreinish | adam_g: so I think handling the logic in the grenade side would probably be the right approach for that | 19:39 |
mtreinish | dkranz: the problem with that, which is why I think andreaf did a blanket exception was that code gets run by ever test class, and at this point the set of known exceptions isn't always consistent | 19:40 |
mtreinish | I really don't like blanket exceptions either | 19:40 |
*** tosky has quit IRC | 19:40 | |
*** tosky has joined #openstack-qa | 19:41 | |
dkranz | mtreinish: the right thing here is to actually use the code in the resource cleanup script for resource cleanup if it gets called due to a raisd exception | 19:41 |
dkranz | mtreinish: so we make sure to get them all | 19:42 |
dkranz | mtreinish: right now we might lose things if cleaning a particular resource raises an exception before others can be cleaned. | 19:42 |
dkranz | mtreinish: not so easy | 19:42 |
mtreinish | yeah it would probably require refactoring every test class. That is the basic approach andreaf is taking with his staged setup/teardown | 19:43 |
mtreinish | to at least group things, it's not at the individual call level | 19:43 |
mtreinish | but this way we should have a better idea of what to catch and ignore | 19:44 |
mtreinish | dkranz: https://review.openstack.org/#/c/115353/ | 19:44 |
mtreinish | although it looks like that patch is missing the second half which pairs the setup stage to the teardowns | 19:45 |
dkranz | mtreinish: If we do this right, we could get rid of all the service-specific stuff in dpatersons script since that logic should really be in the base classes of tempest proper | 19:47 |
*** dimsum_ has quit IRC | 19:48 | |
*** Longgeek has quit IRC | 19:55 | |
andreaf | mtreinish, dkranz: I think for now it's up to the test classes to handle failures on cleanup properly, e.g. if a cleanup fails continue with the next one | 19:58 |
dkranz | andreaf: I agree. What I want to avoid is deletion of the user/tenant if something failed unexpectedly. | 20:00 |
andreaf | mtreinish, dkranz: to be able to actually handle the cleanups and retries on cleanups in the base class we would need to have a common type of resources (e.g. test resources) and handle whatever try/except or retry logic we want to have in the base class | 20:00 |
dkranz | andreaf: It should really be on OpenStack to free resources used by a tenant when the tenant is deleted. | 20:00 |
*** toure_ has joined #openstack-qa | 20:00 | |
andreaf | dkranz: so we could specialize things here a bit to solve this in the tearDownClass implementation | 20:01 |
andreaf | dkranz, what we need to do for test accounts is to cleanup resources which actually means only releasing locks | 20:01 |
jogo | dkranz: yeah I could build a separate stress test for this | 20:02 |
dkranz | andreaf: so we don't have to delete the user/tenant? | 20:02 |
jogo | dkranz: I already added a patch to log the failure to delete | 20:02 |
andreaf | dkranz: while in case of dynamically provisioned credentials we actually delete the credentials | 20:02 |
mtreinish | andreaf: I just left you a review comment with a super hacky code snippet of what you could try to do on the cleanup side for the setup split | 20:02 |
dkranz | jogo: That would be a big plus. The fact that we are using the normal gate jobs as stress tests is not good IMO | 20:02 |
mtreinish | I don't think we should land the framework without having the cleanup side mapped out too | 20:02 |
dkranz | jogo: It causes goal conflict | 20:02 |
dkranz | jogo: like fast vs easier to debug races | 20:03 |
andreaf | dkranz, we could differentiate the behaviour depending on the type of credentials, so in case we use pre-provisioned ones do the cleanup, else not | 20:03 |
andreaf | mtreinish: ok let me have a look | 20:03 |
dkranz | andreaf: I was going to suggest another flag, but that is probably good enough | 20:03 |
mtreinish | dkranz: we already have the config options for that | 20:04 |
dkranz | mtreinish: yes | 20:04 |
jogo | dkranz: yeah I'll turn that into a stress test | 20:05 |
jogo | dkranz mtreinish: speaking of stress tests | 20:05 |
jogo | this is the kind of thing we should run as a nova 'functional' test or at least not gate anything besides nova on | 20:05 |
andreaf | mtreinish: yes but there's only two things that need teardown, credentials and resources | 20:05 |
andreaf | setting up the clients wouldn't need a teardown I think | 20:06 |
mtreinish | andreaf: ok, then that makes it easier | 20:06 |
mtreinish | I was just being thorough and adding them for each stage | 20:06 |
mtreinish | jogo: we run the stress tests as a periodic only at this point | 20:06 |
dkranz | jogo: I don't see stress tests as gating. That is part of our current problem. | 20:07 |
dkranz | jogo: but it could run continuously | 20:07 |
jogo | could we make some stress tests run as non-voting in the check queues or something? | 20:08 |
dkranz | jogo: absolutely, but for the create/delete stress test it would only make sense on nova as you sais | 20:08 |
dkranz | said | 20:08 |
mtreinish | jogo: yeah that would be easy, just setup a json file with the tests you want to run and add a tox target for it (or one that takes in the json file as a posarg) | 20:08 |
andreaf | mtreinish, so my approach was, for skip_checks and other code before credentials are setup we don't need any teardown | 20:08 |
jogo | dkranz: right, so if we could have a nova-stress test job | 20:08 |
mtreinish | then we can add that job to the nova queue easily | 20:09 |
jogo | mtreinish: is this something that should become a tempest-lib based thing and be called a functional test? | 20:09 |
jogo | or save that for another interation? | 20:09 |
jogo | iteration | 20:09 |
andreaf | mtreinish, then we start doing things that might break, but I agree we need to track what was done that needs teardown - I guess my patch was hacked together a bit to quickly | 20:09 |
mtreinish | jogo: that would probably be the next iteration, splitting the stress test framework is on the list for migration | 20:10 |
mtreinish | but it's not clear whether it's tempest-lib or something else at this point | 20:10 |
dkranz | jogo: https://github.com/openstack/tempest/blob/master/tempest/stress/actions/server_create_destroy.py | 20:11 |
mtreinish | because really all it is a custom test runner | 20:11 |
mtreinish | jogo: if you're serious about setting this up, there was a test case that jeblair wanted to add. Something they were hitting in the wild | 20:11 |
mtreinish | with snapshotting | 20:11 |
mtreinish | I don't remember exactly what it was though | 20:12 |
jogo | mtreinish: writing some tempest tests sounds like fun so I'm in | 20:12 |
jogo | mtreinish: are the stress tests at least self gating? | 20:13 |
dkranz | jogo: what do you mean? | 20:13 |
mtreinish | jogo: you wouldn't necessarily have to write new ones, you can use any existing test | 20:13 |
mtreinish | jogo: no the tests in tempest/stress aren't self gating | 20:13 |
jogo | dkranz: do they get run on tempest changes to stress tests | 20:13 |
jogo | mtreinish: shouldn't they be? | 20:13 |
mtreinish | probably, but we normally encourage just writing a regular api or scenario test and then they can be used with the stress framework | 20:14 |
jogo | mtreinish: so I have been seeing failures in delete and in rebuild and a few other operations | 20:14 |
jogo | mtreinish: shouldn't there at least be a stress sanity job | 20:14 |
mtreinish | jogo: there is a nightly | 20:14 |
jogo | why not gating? | 20:15 |
jogo | something that runs things just once | 20:15 |
dkranz | jogo: that makes sense | 20:15 |
dkranz | jogo: I will check what we do | 20:15 |
mtreinish | jogo: I think it was because the stress specific tests are basically duplicates of existing test cases | 20:16 |
mtreinish | we probably should sanity check them to avoid bit rot etc | 20:16 |
mtreinish | but that's why we did it as a nightly | 20:16 |
jogo | mtreinish: nightly are silly | 20:16 |
mtreinish | no argument from me | 20:16 |
mtreinish | it's also been like >1yr since we did anything with those jobs I think | 20:16 |
openstackgerrit | Fergal Mc Carthy proposed a change to openstack-dev/devstack: local.conf processing doesn't handle '=' in values https://review.openstack.org/124467 | 20:17 |
*** toure_ has quit IRC | 20:17 | |
jogo | mtreinish dkranz: I'll look into moving the nightly job into a sanity job that runs in check queue (non-voting for starters) | 20:17 |
mtreinish | jogo: I think it runs for an hour by default... | 20:17 |
mtreinish | jogo: yep: http://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n85 | 20:18 |
*** edwa6643 has quit IRC | 20:19 | |
jogo | are tempest logs in logstash? | 20:20 |
andreaf | mtreinish: in your example, if the 2nd stage fails with an unexpected exception, resources from the first step are not cleaned up | 20:20 |
jogo | never mind they are | 20:20 |
andreaf | mtreinish: so regardless the type of the exception being expected or not, we should try and cleanup whatever resources were setup | 20:20 |
mtreinish | andreaf: oops, yeah we probably should put it in a giant try blcok then to handle that | 20:21 |
*** e0ne has quit IRC | 20:21 | |
jogo | mtreinish: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIGRlbGV0ZSBzZXJ2ZXJcIiBBTkQgdGFnczpcInRlbXBlc3QudHh0XCIgQU5EIHRhZ3M6XCJtdWx0aWxpbmVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQxMjg4NjA3MjMyNywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ== | 20:21 |
jogo | message:"Failed to delete server" AND tags:"tempest.txt" AND tags:"multiline" | 20:21 |
mtreinish | the pseudo lifo logic would take care of handling the staged approach | 20:21 |
jogo | hmm never mind that | 20:21 |
*** e0ne has joined #openstack-qa | 20:21 | |
mtreinish | andreaf: we should still keep the individual try blocks after each stage I think | 20:22 |
jogo | mtreinish: so message:"Waiting for deletion of server" AND tags:"tempest.txt" AND build_status:"SUCCESS" isn't as frequent as I thought | 20:26 |
jogo | 26 in last 7days | 20:26 |
mtreinish | jogo: well that's probably a good thing | 20:27 |
jogo | versus over 500 for success and faiure | 20:27 |
jogo | mtreinish: yes | 20:27 |
openstackrecheck | openstack/glance change: https://review.openstack.org/127248 failed gate-tempest-dsvm-postgres-full in the gate queue with an unrecognized error | 20:27 |
mtreinish | jogo: and it looks like a good chunk are on check jobs so they could just be bad patches | 20:28 |
mtreinish | or they might all be check | 20:28 |
*** e0ne has quit IRC | 20:28 | |
jogo | oh interesting yeah | 20:30 |
mtreinish | although if they're on sucesses that's probably not a good thing if they are bad patches... | 20:30 |
jogo | its a bunch of different changes | 20:30 |
mtreinish | bbiab, gonna work my way home | 20:33 |
jogo | mtreinish: you aren't at home? | 20:33 |
*** jhenner1 has quit IRC | 20:42 | |
openstackgerrit | Joe Gordon proposed a change to openstack/tempest: Add list servers to large ops test https://review.openstack.org/78480 | 20:45 |
*** edwa6643 has joined #openstack-qa | 20:52 | |
mtreinish | jogo: I was at a stabucks | 20:52 |
openstackgerrit | Christian Berendt proposed a change to openstack/tempest: Remove @author tags from copyright statements https://review.openstack.org/127344 | 20:53 |
mtreinish | jogo: ^^^ is there a hacking rule for that? | 20:54 |
jogo | mtreinish: he posted a hacking patch this morning | 20:55 |
jogo | I don't really think a hacking pach is needed for that though | 20:56 |
jogo | same as vim modelines | 20:56 |
jogo | once we nix all the cases of them | 20:56 |
jogo | they shouldn't be coming back | 20:56 |
jogo | because if they aren't there at all, its easy to see they are silly | 20:56 |
mtreinish | jogo: you'd think, but I the ones being removed there were brand new. Plus I feel if it's the kind of thing to go to the effort of removing everywhere then it should probably be blocked | 20:57 |
openstackgerrit | Joe Gordon proposed a change to openstack/tempest: Make test_list_servers_filter_exist_host use create_test_server https://review.openstack.org/123562 | 20:58 |
* jogo looks | 20:58 | |
mtreinish | fwiw, I think tempest has a local rule about the vim modelines | 20:59 |
jogo | nova does too I think | 20:59 |
jogo | so the answer is, when I get around to it is make a directory for off by default rules | 20:59 |
jogo | that projects can consume | 20:59 |
mtreinish | jogo: yeah that would owrk | 20:59 |
jogo | although author tags is a good candidate for a on by default rule | 21:00 |
jogo | i have to review his patch, it had evil regex in it | 21:00 |
openstackgerrit | Steve Martinelli proposed a change to openstack-infra/elastic-recheck: Add query for bug 1379201 https://review.openstack.org/127346 | 21:00 |
mtreinish | andreaf: I just saw your review on that author tags patch. See ^^^ | 21:01 |
*** dustins has quit IRC | 21:01 | |
*** markmcclain has quit IRC | 21:05 | |
andreaf | mtreinish: ok ,so the proposal is to have an importable hacking rule ? | 21:10 |
mtreinish | andreaf: there wouldn't be any import it would just be included in the next hacking release | 21:10 |
mtreinish | the first tempest-lib release has been pushed: https://pypi.python.org/pypi/tempest-lib/0.0.1 | 21:12 |
mtreinish | we probably should update some of the metadata for the next one | 21:12 |
andreaf | mtreinish, yay :) | 21:13 |
*** HenryG has quit IRC | 21:14 | |
andreaf | mtreinish: I'll try and propose an update on resource cleanup tomorrow | 21:14 |
mtreinish | andreaf: ok cool | 21:14 |
*** jgrimm is now known as zz_jgrimm | 21:17 | |
*** gordc has quit IRC | 21:22 | |
openstackgerrit | Joe Gordon proposed a change to openstack/tempest: Move default stress test to run things 3 times instead of time based https://review.openstack.org/127351 | 21:23 |
ianw | dtroyer: do you mind if I self approve https://review.openstack.org/#/c/124227/ (Single quote iniset argument in merge_config_file) -- it is holding up Baodong | 21:23 |
openstackgerrit | Adam Gandelman proposed a change to openstack-dev/grenade: Generate javelin config dynamically https://review.openstack.org/127352 | 21:23 |
jogo | mtreinish: ^ | 21:23 |
dtroyer | ianw: I +A it…just a tiny bit better than self-approve ;) | 21:24 |
ianw | dtroyer: great, thanks. i need to make time to think about the ini_add stuff, if it can be done multi-line with an array maybe | 21:25 |
dtroyer | ianw: that sounds like it should work, without thinking about it too hard. I think it should at least remove one layer of quote-stripping | 21:26 |
ianw | dtroyer: context switching into that code is about an half-an-hour exercise in itself :) | 21:26 |
openstackgerrit | Adam Gandelman proposed a change to openstack-dev/grenade: Generate javelin config dynamically https://review.openstack.org/127352 | 21:26 |
dtroyer | ianw: I hear you, one reason I hadn't dug in too far yet | 21:27 |
*** mmedvede has quit IRC | 21:27 | |
*** rbak has quit IRC | 21:30 | |
*** rbak has joined #openstack-qa | 21:30 | |
jogo | dkranz: https://review.openstack.org/#/c/127351/ | 21:37 |
jogo | does anyone track the nightly jobs? | 21:37 |
dkranz | jogo: I commented on that patch | 21:38 |
dkranz | jogo: I do, with some frequency | 21:38 |
jogo | I saw, and follow up is ^ | 21:38 |
jogo | dkranz: oh neat! | 21:38 |
jogo | kk let me revise the patch then | 21:38 |
dkranz | jogo: This is one of the biggest holes we have from a process standpoint though | 21:38 |
jogo | dkranz: which? | 21:39 |
dkranz | jogo: the fact that the only process we have that works is blocking failures at the gate | 21:39 |
dkranz | jogo: any sane organization would tier tests | 21:39 |
dkranz | jogo: so that some run post-merge, but frequently | 21:40 |
jogo | dkranz: well in this case I just want these tests to be self gated | 21:40 |
dkranz | jogo: but that only works if some one is known to be looking | 21:40 |
dkranz | jogo: I know and am not objecting to that | 21:40 |
jogo | dkranz: so this is exactly the outcome of the qa midcycle in germany | 21:40 |
*** adalbas has quit IRC | 21:40 | |
jogo | dkranz: I think the difference between gating and post-merge tests is where the burden to maintain lands | 21:41 |
jogo | gating, the entire team has to maintain | 21:41 |
dkranz | jogo: exactly | 21:41 |
*** mriedem has quit IRC | 21:41 | |
jogo | post-merge only the people who care about X have to maintain | 21:41 |
jogo | because entire team can just kill the code if it stops working | 21:41 |
jogo | so AFAIK infra needs help getting post-merge jobs to work | 21:41 |
jogo | dkranz: if you want to help | 21:41 |
dkranz | jogo: I'm not sure how. I watch the tempest nightlies. | 21:42 |
*** ramishra has joined #openstack-qa | 21:42 | |
dkranz | jogo: This is more of a process problem than technical. | 21:42 |
jogo | dkranz: there is a doc on this somewhere | 21:42 |
dkranz | jogo: what kind of help does infra need? | 21:42 |
mtreinish | dkranz: well to be fair the stress nightlies haven't failed in a long time | 21:42 |
jogo | mtreinish: do you remember where that doc is? | 21:43 |
dkranz | mtreinish: Yes, but they are not very stressful :) | 21:43 |
dkranz | mtreinish: let's crank them up and see what happens | 21:43 |
mtreinish | jogo: doc for what? | 21:43 |
jogo | mtreinish: roadmap for getting to post-merge testing | 21:43 |
mtreinish | nah, I didn't even know there was a doc published for it | 21:44 |
mtreinish | dkranz: go for it, I've been thinking we need to run all the scenario tests as part of that nightly stress job | 21:44 |
jogo | mtreinish: AFAIK the email sean and jim sent out | 21:44 |
mtreinish | oh, the email. Yeah I can pull up that link | 21:45 |
jogo | dkranz: ++ to cranking the stress up | 21:45 |
dkranz | jogo, mtreinish : ok... | 21:45 |
mtreinish | http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html | 21:45 |
jogo | dkranz: one of the steps for getting to post-merge was to make it easy to discover when those jobs begin failing etc | 21:46 |
openstackgerrit | Joe Gordon proposed a change to openstack/tempest: Add new tox target to sanity check stress tests https://review.openstack.org/127351 | 21:46 |
*** ramishra has quit IRC | 21:46 | |
jogo | mtreinish: what do you think of the new target name? | 21:46 |
dkranz | jogo: when you get to the point that the watchers care, they just need to look! | 21:47 |
dkranz | jogo: an email to the watchers is all you need | 21:47 |
jogo | dkranz: well we want to make it easy for the people who don't care (the core team) to know when the job hasn't worked in a long time | 21:47 |
dkranz | jogo: I get an email every day about tempest | 21:47 |
jogo | dkranz: what about of the jobs fail 1% of the time | 21:47 |
mtreinish | jogo: yeah that's fine, I would have gone stress-check, but I'm fine with sanity | 21:47 |
mtreinish | dkranz: well only if it fails | 21:48 |
jogo | mtreinish: I liked the name | 21:48 |
dkranz | jogo: Sure, it would be better if we kept a history, nice graphs, etc. | 21:48 |
mtreinish | jogo: heh, stress our sanity :) | 21:48 |
jogo | dkranz: anyway that email includes a bunch of steps | 21:48 |
jogo | dkranz: and infra should know where we are in it | 21:48 |
mtreinish | dkranz: there was a thread started by jhesketh about how to handle visualizing this stuff (specifically post jobs, but its the same for periodics) | 21:49 |
dkranz | mtreinish: sure | 21:49 |
*** suneelb has quit IRC | 21:49 | |
*** edwa66431 has joined #openstack-qa | 21:49 | |
jogo | mtreinish: oh neat I must have missed that | 21:50 |
mtreinish | jogo: well he was proposing just sending them to the -qa list as a solution | 21:50 |
jogo | oh that is still a thing? | 21:51 |
* jogo wants a dashboard | 21:51 | |
mtreinish | yeah the list is there just for the nightlies | 21:51 |
mtreinish | yeah I want a dashboard too | 21:51 |
mtreinish | that's the only thing I miss from the days of a single jenkins | 21:51 |
*** suneelb1 has joined #openstack-qa | 21:51 | |
jogo | haha yeah | 21:51 |
jogo | well actually a dash is easy | 21:52 |
jogo | graphite | 21:52 |
jogo | jhesketh: ping | 21:52 |
jogo | I think with graphite + logstash we can build a decent dashboard quickly | 21:52 |
jogo | graphite shows the pass fail rates, and logstash to find the specific jobs that failed | 21:53 |
jogo | mtreinish: what ML was that thread on? | 21:53 |
*** edwa6643 has quit IRC | 21:53 | |
openstackrecheck | openstack/keystone change: https://review.openstack.org/119804 failed because of: gate-tempest-dsvm-neutron-full: https://bugs.launchpad.net/bugs/1357055 https://bugs.launchpad.net/bugs/1311066 | 21:54 |
*** boden has quit IRC | 21:54 | |
mtreinish | jogo: http://lists.openstack.org/pipermail/openstack-qa/2014-October/001258.html | 21:54 |
mtreinish | I moved it to the dev list but it died there except for one reply from fungi that reporting back to gerrit post merge is hard | 21:55 |
mtreinish | let me dig up the dev list side | 21:55 |
mtreinish | jogo: http://lists.openstack.org/pipermail/openstack-dev/2014-October/047560.html | 21:55 |
*** mmedvede has joined #openstack-qa | 21:56 | |
*** edwa6643 has joined #openstack-qa | 22:02 | |
*** edwa66431 has quit IRC | 22:05 | |
jogo | mtreinish: thanks, responded | 22:05 |
jogo | I think this is a very solvable thing | 22:05 |
jogo | fungi: ^ | 22:05 |
fungi | cool | 22:07 |
fungi | i'll let you work out the map-merge-to-triggering-change logic | 22:08 |
jogo | fungi: which logic is that? | 22:08 |
*** jhenner has joined #openstack-qa | 22:08 | |
fungi | the logic to figure out how a commit in the post pipeline maps back to a change to report on | 22:08 |
*** jordanP has quit IRC | 22:09 | |
jogo | fungi: oh it wouldn't | 22:09 |
fungi | zuul handles them differently because they are different things | 22:09 |
mtreinish | jogo: once we get subunit2sql up and running we probably could add that to the equation also | 22:09 |
*** Sukhdev has joined #openstack-qa | 22:09 | |
* fungi will read the reply before commenting further then | 22:09 | |
dkranz | mtreinish, jogo : I've got to go now but the current nightly stress doesn't really do much. It only runs the few things that are marked as 'stress'. This is more appropriate for the new thing Joe introduced. | 22:09 |
mtreinish | get per test data for those jobs | 22:09 |
dkranz | I am going to make it do something more real, and targeted. We should probably have multiple jobs for different stress areas. | 22:10 |
jogo | mtreinish: ahh right, it will be another tool for analysis | 22:10 |
jogo | dkranz: ++ that sounds like a great idea | 22:11 |
dkranz | jogo: or I could just make tox run a sequence of stress runs | 22:11 |
jogo | dkranz: long run I would like to make the nova targeted stress stuff be a nova only functional test or something | 22:11 |
dkranz | jogo: right. Should do both of job-per-service and running a sequence of targeted tests | 22:11 |
mtreinish | dkranz: as I was saying before it should just be a job that takes one of the json files in from posargs then we can handle the config per job in d-g | 22:12 |
jogo | dkranz: ? | 22:12 |
jogo | all I am trying to do at the moment is make stress jobs self gating | 22:12 |
dkranz | jogo: right, that is fine. I'm not talking about touching the one you added. | 22:12 |
dkranz | mtreinish: ok, but I think we may want more than that. I'll come up with something tomorrow. | 22:13 |
jogo | does the @stresstest thing still work? | 22:14 |
mtreinish | jogo: heh, that's what I was just leaving a review comment on your patch for | 22:16 |
mtreinish | it does, that's what -a is used for on run_stress, it runs all the tests that have the decorator set | 22:16 |
mtreinish | jogo: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/run_stress.py#n35 | 22:17 |
fungi | jogo: so on your reply, yes that should work just fine. i think trying to report post job results back to changes in gerrit is a non-starter for multiple reasons but a status dashboard of some sort (assuming people will actually look at it) would fit well | 22:18 |
jogo | fungi: yeah, and it shouldn't be too hard to build one I think | 22:18 |
fungi | agreed... something on status.o.o site would make sense | 22:19 |
jogo | fungi: yeah | 22:19 |
openstackgerrit | Adam Gandelman proposed a change to openstack-dev/grenade: Generate javelin config dynamically https://review.openstack.org/127352 | 22:19 |
fungi | it's been the solution we've held up in multiple past conversations as probably the best option, just nobody's done it yet | 22:20 |
* mtreinish is not looking forward to writing the dashboard view for subunit2sql, sucks at anything on the web... | 22:20 | |
jogo | mtreinish: so this is silly | 22:20 |
jogo | mtreinish: why do I need a json file ? | 22:20 |
mtreinish | jogo: this is what I was saying earlier those tests in tempest.stress.actions are really just duplicates of other test cases | 22:21 |
jogo | mtreinish: now I get it | 22:21 |
jogo | I thought you meant the tempest/stress folder | 22:21 |
jogo | err directory | 22:22 |
jogo | fungi: yeah I don't really see a better way | 22:22 |
mtreinish | the json file is just the import format to the stress runner to tell it which test cases to run and the conditions on how to run them (like how to handle setupclass, how to loop, etc) | 22:22 |
openstackgerrit | Andrea Frittoli proposed a change to openstack/tempest: Framework for staged setup https://review.openstack.org/115353 | 22:22 |
andreaf | mtreinish, still WIP ^^^ but you may want to look if you like it better | 22:22 |
mtreinish | andreaf: ok I'll take a look | 22:23 |
jogo | fungi: so what other steps are still TODOs for getting post-gate jobs | 22:23 |
*** dkranz has quit IRC | 22:23 | |
jogo | or whatever we are calling them, the post-merge pool of jobs that we run in a loop | 22:24 |
*** jecarey has quit IRC | 22:24 | |
jogo | fungi: also if you could respond to http://lists.openstack.org/pipermail/openstack-dev/2014-October/048203.html with your thoughts so its on the record | 22:25 |
jogo | fungi: as this actually sounds like something we can get a POC up before Paris | 22:28 |
*** ramishra has joined #openstack-qa | 22:43 | |
*** ramishra has quit IRC | 22:47 | |
*** suneelb has joined #openstack-qa | 22:49 | |
*** suneelb1 has quit IRC | 22:50 | |
*** JayJ has quit IRC | 22:53 | |
*** JayJ has joined #openstack-qa | 22:54 | |
*** andreaf has quit IRC | 23:01 | |
*** rbak has quit IRC | 23:07 | |
*** HenryG has joined #openstack-qa | 23:07 | |
openstackgerrit | A change was merged to openstack-dev/devstack: Single quote iniset argument in merge_config_file https://review.openstack.org/124227 | 23:08 |
*** suneelb has quit IRC | 23:09 | |
*** suneelb has joined #openstack-qa | 23:10 | |
*** jhenner has quit IRC | 23:12 | |
*** dimtruck is now known as zz_dimtruck | 23:14 | |
*** aimon has quit IRC | 23:16 | |
*** bknudson has quit IRC | 23:17 | |
*** yfujioka has joined #openstack-qa | 23:22 | |
*** yfujioka_ has joined #openstack-qa | 23:23 | |
*** flaper87 is now known as flaper87|afk | 23:25 | |
openstackgerrit | Michael Hudson-Doyle proposed a change to openstack/tempest: fix AttributeError when test_function_help fails https://review.openstack.org/127384 | 23:36 |
*** JayJ has quit IRC | 23:37 | |
*** zz_jgrimm is now known as jgrimm | 23:43 | |
*** ramishra has joined #openstack-qa | 23:44 | |
*** edwa66431 has joined #openstack-qa | 23:48 | |
*** ramishra has quit IRC | 23:48 | |
*** edwa6643 has quit IRC | 23:50 | |
*** mlavalle has quit IRC | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!