Thursday, 2014-10-09

*** harlowja is now known as harlowja_away00:01
*** melwitt has quit IRC00:02
*** rockyg has quit IRC00:02
dtroyeradam_g: this one?  https://etherpad.openstack.org/p/devstack-service-redux00:05
adam_gdtroyer, yeah!00:05
*** yamahata has joined #openstack-qa00:10
*** harlowja_away is now known as harlowja00:11
andreafmtreinish: ok00:15
openstackgerritA change was merged to openstack-dev/devstack: Correct swift service name in docs  https://review.openstack.org/12622000:17
openstackrecheckopenstack/python-openstackclient change: https://review.openstack.org/125714 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/128681800:20
*** aimon has quit IRC00:21
*** yfujioka has joined #openstack-qa00:22
*** openstackstatus has quit IRC00:23
*** openstackstatus has joined #openstack-qa00:24
*** ChanServ sets mode: +v openstackstatus00:24
*** bnemec has quit IRC00:28
*** arosen has quit IRC00:28
jogomtreinish: ping00:31
*** dmorita has joined #openstack-qa00:34
*** yjiang5 has joined #openstack-qa00:37
openstackgerritJoe Gordon proposed a change to openstack/tempest: TEST: Stop suppressing errors during wait_for_server_termination  https://review.openstack.org/12708500:38
openstackgerritDaisuke Morita proposed a change to openstack-dev/devstack: Enable Swift's newer feature of container-sync  https://review.openstack.org/12682400:41
openstackgerritSergey Shnaidman proposed a change to openstack/tempest: DHCPv6 network tests  https://review.openstack.org/11745800:41
*** sdmonov has quit IRC00:42
*** markmcclain has quit IRC00:43
bknudsontempest runs are seeming sickly -- http://logs.openstack.org/62/127062/1/check/check-tempest-dsvm-full/bc82fee/console.html#_2014-10-08_23_38_48_08300:44
*** aimon has joined #openstack-qa00:48
mtreinishjogo: pong00:48
jogomtreinish: so the nova delete thing00:49
jogoin tempest00:49
jogothe current model of doing try: wait_for_termination except Exception:00:50
jogois hiding bugs00:50
jogoat least one libvirt issue00:50
mtreinishbknudson: I'd suspect your patch there, because all the jobs are failing with the same auth errors00:50
jogoI know you are not a fan of raising errors during class cleanup and whatnot00:51
mtreinishjogo: so that might be, but if we can't reliably delete, we can't test because everything will fail00:51
bknudsonmtreinish: I could very well have broken something.00:51
jogomtreinish: I posted https://review.openstack.org/#/c/127085/ to see what it looks like00:51
*** JayJ has quit IRC00:52
jogomtreinish: long term I think we need an approach for how to test this00:52
jogowell medium term00:52
openstackgerritGhanshyam Mann proposed a change to openstack/tempest: Adds test for list/get volume attachments V2 APIs  https://review.openstack.org/8719100:52
jogobefore nova has proper functional testing and after we make sure we can delete reliably enough to gate on it00:52
mtreinishjogo: the other thing is I feel like if the delete was important it'd be directly tested in the test case00:52
jogomtreinish: I think they should be done that way IMHO00:53
mtreinishthe cleanup stage is really an after thought to prevent leaks00:53
bknudsonI didn't think that change made any functional changes... should just be docs00:53
jogomtreinish: maybe that is the answer, move tests over to do the delete in test more often00:53
*** markmcclain has joined #openstack-qa00:53
openstackgerritA change was merged to openstack-infra/devstack-gate: Update README for post config split  https://review.openstack.org/12547000:53
*** salv-orlando has quit IRC00:55
openstackgerritA change was merged to openstack-infra/devstack-gate: Better check for pypi mirror host connectivity  https://review.openstack.org/12629000:55
*** mmedvede has joined #openstack-qa00:55
mtreinishbknudson: yeah it looks like just string changes, but the consistent auth failures in multiple jobs seems like it would be more than a coincidence00:55
mtreinishjogo: yeah, I think that would be fine00:55
mtreinishthen we'd be failing as part of the test case and it makes the debug simpler00:56
bknudsonheat shut down : http://logs.openstack.org/62/127062/1/check/check-tempest-dsvm-full/bc82fee/logs/screen-h-api.txt.gz#_2014-10-08_23_08_53_26400:56
bknudsonDuplicateOptError: duplicate option: admin_user00:56
mtreinishbknudson: well that would do it :)00:57
mtreinishbut if that is an issue, why isn't it failing everywhere?00:58
mtreinishyou're not changing the opt name there00:58
jogomtreinish: cool, I'll start pushing some patches up to do that tomorrow00:58
mtreinishjust the help strings00:58
openstackgerritA change was merged to openstack-dev/devstack: Stop n-cpu by correct process name /w fake  https://review.openstack.org/12642000:59
openstackgerritA change was merged to openstack/tempest: Fwaas API Test Enhancement  https://review.openstack.org/10282000:59
mtreinishjogo: heh, well if gate failures go up, I'm going to point the angry mob at you :)00:59
bknudsonfor some reason it's using keystoneclient.middleware.auth_token rather than keystonemiddleware.auth_token01:00
openstackgerritA change was merged to openstack-infra/devstack-gate: Remove TESTR_FULL option for running tempest  https://review.openstack.org/12419001:01
bknudsonso that might be it... a mismatch because keystoneclient auth_token help text is different than keystonemiddleware auth_token help text.01:01
bknudsonI think that would explain it.01:01
jogomtreinish: fair enough01:01
*** JayJ has joined #openstack-qa01:05
*** JayJ has quit IRC01:06
*** JayJ has joined #openstack-qa01:07
*** JayJ has quit IRC01:07
mtreinishbknudson: yeah I think that could do it01:08
bknudsonmtreinish: the heat code still has refs to keystoneclient.middleware... I should have caught those. :(01:09
*** topshare has joined #openstack-qa01:23
*** edwa6643 has joined #openstack-qa01:24
*** edwa66431 has quit IRC01:26
*** suneelb has quit IRC01:39
*** oomichi_ has joined #openstack-qa01:44
*** edwa6643 has left #openstack-qa01:46
openstackrecheckopenstack/keystone change: https://review.openstack.org/119946 failed gate-keystone-python26, gate-keystone-python27 in the gate queue with an unrecognized error01:47
*** mmedvede has quit IRC01:48
bknudsonunbelievable01:54
*** markmcclain has quit IRC01:55
*** dimsum_ has joined #openstack-qa02:06
*** nosnos has joined #openstack-qa02:06
openstackrecheckopenstack/python-glanceclient change: https://review.openstack.org/125965 failed because of: gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/1370496 https://bugs.launchpad.net/bugs/135396202:08
*** weshay has quit IRC02:25
*** Longgeek has joined #openstack-qa02:26
*** dimsum_ has quit IRC02:28
*** dimsum_ has joined #openstack-qa02:28
*** zz_dimtruck is now known as dimtruck02:31
*** dimsum_ has quit IRC02:33
*** toure has joined #openstack-qa02:45
*** openstackgerrit has quit IRC02:45
*** yfujioka has quit IRC02:52
*** mjturek has quit IRC02:54
*** coolsvap|afk is now known as coolsvap02:55
*** edwa66431 has joined #openstack-qa02:58
*** pcrews has quit IRC03:03
*** ramishra has joined #openstack-qa03:03
*** rushiagr_away is now known as rushiagr03:12
*** toure has quit IRC03:21
*** harlowja is now known as harlowja_away03:29
*** harlowja_away is now known as harlowja03:31
*** bknudson has quit IRC03:33
*** nosnos has quit IRC03:35
*** ramishra has quit IRC03:37
*** ramishra has joined #openstack-qa03:37
*** coolsvap is now known as coolsvap|afk03:37
*** coolsvap|afk is now known as coolsvap03:45
*** ramishra has quit IRC03:48
*** gmann has quit IRC04:03
*** gmann has joined #openstack-qa04:03
*** yfujioka has joined #openstack-qa04:04
*** rushiagr is now known as rushiagr_away04:08
*** edwa66431 has quit IRC04:10
*** ramishra has joined #openstack-qa04:13
*** chandankumar has joined #openstack-qa04:17
*** ramishra has quit IRC04:24
*** e0ne has joined #openstack-qa04:34
*** amotoki has joined #openstack-qa04:34
*** nosnos has joined #openstack-qa04:35
*** rushiagr_away is now known as rushiagr04:37
*** Poornima has joined #openstack-qa04:38
*** e0ne has quit IRC04:39
*** e0ne has joined #openstack-qa04:52
*** chandankumar has quit IRC04:52
*** SridharG has joined #openstack-qa04:52
*** e0ne has quit IRC04:56
*** Madan has joined #openstack-qa05:02
*** jamielennox has quit IRC05:04
*** jamielennox has joined #openstack-qa05:08
*** swamireddy has joined #openstack-qa05:18
*** harlowja is now known as harlowja_away05:35
*** neeti has joined #openstack-qa05:37
*** dimtruck is now known as zz_dimtruck05:42
*** afazekas has joined #openstack-qa05:51
*** yamahata has quit IRC05:51
*** yamahata has joined #openstack-qa05:51
*** coolsvap is now known as coolsvap|afk06:00
*** ramishra has joined #openstack-qa06:04
*** chandankumar has joined #openstack-qa06:04
*** neeti has quit IRC06:06
*** ramishra has quit IRC06:09
*** coolsvap|afk is now known as coolsvap06:12
*** pkoniszewski has joined #openstack-qa06:13
*** jlibosva has joined #openstack-qa06:22
*** chandankumar has quit IRC06:27
*** chandankumar has joined #openstack-qa06:28
*** pcrews has joined #openstack-qa06:39
*** vigneshvar has joined #openstack-qa06:42
*** pcrews has quit IRC06:47
*** topshare has quit IRC06:48
*** topshare has joined #openstack-qa06:51
*** markus_z has joined #openstack-qa06:53
*** topshare has quit IRC06:54
*** gszasz has joined #openstack-qa06:56
*** topshare has joined #openstack-qa06:56
*** k4n0 has joined #openstack-qa06:56
*** flaper87|afk is now known as flaper8706:58
*** k4n0 has quit IRC07:02
*** topshare has quit IRC07:03
*** topshare has joined #openstack-qa07:07
*** topshare has quit IRC07:09
*** topshare has joined #openstack-qa07:10
*** topshare has quit IRC07:13
*** k4n0 has joined #openstack-qa07:14
*** topshare has joined #openstack-qa07:18
*** andreaf has quit IRC07:26
*** topshare has quit IRC07:29
*** topshare has joined #openstack-qa07:29
*** Albert__13 has joined #openstack-qa07:33
*** Albert_13 has joined #openstack-qa07:34
*** topshare has quit IRC07:35
*** topshare has joined #openstack-qa07:35
*** luqas has joined #openstack-qa07:35
*** topshare has quit IRC07:36
*** topshare has joined #openstack-qa07:37
*** k4n0 has quit IRC07:42
*** yjiang5 has quit IRC07:45
*** ramishra has joined #openstack-qa07:45
*** Albert_13 has left #openstack-qa07:46
*** jordanP has joined #openstack-qa07:47
*** topshare has quit IRC07:47
*** jlanoux has joined #openstack-qa07:48
*** ramishra has quit IRC07:50
*** kashyap has joined #openstack-qa07:51
*** gfidente has quit IRC07:53
openstackrecheckopenstack/cinder change: https://review.openstack.org/120306 failed gate-cinder-pep8 in the gate queue with an unrecognized error07:54
*** k4n0 has joined #openstack-qa07:54
*** dmellado_ has joined #openstack-qa07:56
*** dmellado_ has quit IRC07:57
*** mpaolino has joined #openstack-qa08:00
*** safchain has joined #openstack-qa08:00
*** gfidente has joined #openstack-qa08:01
*** jpich has joined #openstack-qa08:05
*** qba73 has joined #openstack-qa08:08
*** yamamoto has quit IRC08:08
*** Matt1 has joined #openstack-qa08:08
*** shardy_z is now known as shardy08:09
*** swamireddy has quit IRC08:09
*** topshare has joined #openstack-qa08:10
*** topshare has quit IRC08:10
*** topshare has joined #openstack-qa08:10
*** tkammer has quit IRC08:10
*** salv-orlando has joined #openstack-qa08:11
*** swamireddy has joined #openstack-qa08:12
*** topshare has quit IRC08:13
*** tkammer has joined #openstack-qa08:14
*** tkammer has quit IRC08:18
*** gfidente has quit IRC08:24
*** oomichi_ has quit IRC08:24
*** tkammer has joined #openstack-qa08:34
*** gszasz has quit IRC08:37
*** gszasz has joined #openstack-qa08:42
*** tkammer has quit IRC08:42
*** vigneshvar has quit IRC08:45
*** ramishra has joined #openstack-qa08:46
*** SridharG has left #openstack-qa08:47
*** ramishra has quit IRC08:51
*** tkammer has joined #openstack-qa08:57
*** tkammer has quit IRC09:02
*** SridharG has joined #openstack-qa09:09
*** tkammer has joined #openstack-qa09:17
*** zz_johnthetubagu is now known as johnthetubaguy09:17
*** gfidente has joined #openstack-qa09:19
*** qba73 has quit IRC09:26
chandankumargmann, hello09:30
*** SridharG has left #openstack-qa09:31
chandankumargmann, sent the new patch set with modification : https://review.openstack.org/#/c/121055/09:31
chandankumargmann, please review it when you are free09:31
*** SridharG has joined #openstack-qa09:38
*** coolsvap is now known as coolsvap|afk09:56
*** jhenner has quit IRC09:56
*** ramishra has joined #openstack-qa09:56
*** jlanoux has quit IRC09:57
*** tkammer has quit IRC10:00
*** ramishra has quit IRC10:01
*** dmorita has quit IRC10:05
*** jhenner has joined #openstack-qa10:06
*** ramishra has joined #openstack-qa10:06
*** cdent has joined #openstack-qa10:08
*** tnurlygayanov has joined #openstack-qa10:14
*** ramishra has quit IRC10:14
*** yamahata has quit IRC10:19
*** tosky has joined #openstack-qa10:21
*** tkammer has joined #openstack-qa10:22
*** ramishra has joined #openstack-qa10:24
*** psedlak_ has joined #openstack-qa10:25
*** aix_ has quit IRC10:38
*** dimsum_ has joined #openstack-qa10:51
*** vigneshvar has joined #openstack-qa10:59
*** tkammer has quit IRC11:01
*** ramishra has quit IRC11:06
*** aix_ has joined #openstack-qa11:07
*** dimsum_ has quit IRC11:13
*** dimsum_ has joined #openstack-qa11:13
*** dims_ has joined #openstack-qa11:14
*** tkammer has joined #openstack-qa11:14
*** dimsum_ has quit IRC11:17
*** dmellado_ has joined #openstack-qa11:17
*** dmellado_ has quit IRC11:18
*** Madan has quit IRC11:23
*** dmellado_ has joined #openstack-qa11:25
*** dmellado_ has quit IRC11:25
*** Madan has joined #openstack-qa11:26
*** amotoki_ has joined #openstack-qa11:34
*** amotoki_ has quit IRC11:34
*** ramishra has joined #openstack-qa11:37
*** gmann_ has joined #openstack-qa11:37
gmann_mtreinish: I have updated bug numbers for meeting-  https://etherpad.openstack.org/p/Tempest-bug-report11:39
*** kashyap has quit IRC11:40
*** ramishra has quit IRC11:41
gmann_mtreinish: I re-spin Nova V2.1 tests spec - https://review.openstack.org/#/c/96661/  Please have a look.11:42
masayukiggmann: Thank you for updating :) I'll try to triage some bugs later..11:43
*** amotoki has quit IRC11:45
*** coolsvap|afk is now known as coolsvap11:46
*** hockeynut has quit IRC11:46
*** swamireddy has quit IRC11:46
gmann_masayukig: np :)11:46
*** yfujioka has quit IRC11:48
*** hockeynut has joined #openstack-qa11:48
*** mgagne has quit IRC11:48
*** SridharG has quit IRC11:49
*** edwa6643 has joined #openstack-qa11:50
*** StevenK has quit IRC11:50
*** mgagne has joined #openstack-qa11:51
*** mgagne is now known as Guest9205411:51
*** StevenK has joined #openstack-qa11:51
*** SridharG has joined #openstack-qa11:52
*** gmann_ has quit IRC11:54
*** edwa6643 has quit IRC11:54
*** edwa6643 has joined #openstack-qa11:58
*** gszasz has quit IRC11:58
*** chandankumar has quit IRC11:59
*** aix_ has quit IRC11:59
*** tkammer has quit IRC12:01
*** coolsvap is now known as coolsvap|afk12:01
*** jhenner has quit IRC12:01
*** jhenner has joined #openstack-qa12:02
*** mmedvede has joined #openstack-qa12:04
*** afazekas has quit IRC12:04
*** tosky has quit IRC12:06
*** weshay has joined #openstack-qa12:10
*** mpaolino has quit IRC12:11
*** chandankumar has joined #openstack-qa12:13
*** cnesa16 has joined #openstack-qa12:13
*** Madan has quit IRC12:13
*** markmcclain has joined #openstack-qa12:15
*** mkollaro has joined #openstack-qa12:15
*** tkammer has joined #openstack-qa12:16
*** rushiagr is now known as rushiagr_away12:20
*** openstackgerrit has joined #openstack-qa12:23
*** dims_ has quit IRC12:23
*** mpaolino has joined #openstack-qa12:24
*** dimsum_ has joined #openstack-qa12:24
*** ramishra has joined #openstack-qa12:25
*** salv-orlando has quit IRC12:27
openstackgerritA change was merged to openstack-dev/devstack: Install latest EPEL release  https://review.openstack.org/12708012:33
*** nosnos has quit IRC12:35
*** nosnos has joined #openstack-qa12:36
*** nosnos has quit IRC12:40
*** mpaolino has quit IRC12:42
cdentsdad (that's awesome, congrats!), jogo, mtreinish: when you get a chance can you put some eyes on https://review.openstack.org/#/c/115212/ it will probably be a good exerciser of the run_process changes12:44
*** adalbas has joined #openstack-qa12:48
*** JayJ has joined #openstack-qa12:49
*** rushiagr_away is now known as rushiagr12:49
*** aix has joined #openstack-qa12:50
*** dkranz has quit IRC12:54
*** jhenner1 has joined #openstack-qa12:54
*** tosky has joined #openstack-qa12:54
*** jhenner1 has quit IRC12:54
*** jhenner1 has joined #openstack-qa12:55
*** tkammer has quit IRC12:55
*** aysyd has joined #openstack-qa12:55
*** zz_dimtruck is now known as dimtruck12:55
*** mpaolino has joined #openstack-qa12:57
*** mpaolino has quit IRC12:57
*** jhenner has quit IRC12:57
*** amotoki has joined #openstack-qa13:01
*** ramishra has quit IRC13:03
*** gszasz has joined #openstack-qa13:03
*** dimtruck is now known as zz_dimtruck13:05
*** tkammer has joined #openstack-qa13:08
*** zz_dimtruck is now known as dimtruck13:10
*** jaypipes has joined #openstack-qa13:12
*** ChanServ sets mode: +o jaypipes13:12
*** zz_jgrimm is now known as jgrimm13:13
*** mriedem has joined #openstack-qa13:18
*** vigneshvar has quit IRC13:19
*** gszasz has quit IRC13:24
*** dpaterson has joined #openstack-qa13:24
*** dimtruck is now known as zz_dimtruck13:24
*** dustins has joined #openstack-qa13:26
EmilienMjlibosva: I can rebase the patch of javelin/neutron if you don't mind13:29
jlibosvaEmilienM: go ahead, I just did recheck cause I have a feeling that -1 from Jenkins scares reviewers13:30
openstackgerritEmilien Macchi proposed a change to openstack/tempest: javelin: add network and secgroup resources  https://review.openstack.org/10016913:31
EmilienMjlibosva: done13:31
jlibosvaEmilienM++13:31
*** markmcclain has quit IRC13:33
openstackgerritZu Qiang proposed a change to openstack-dev/devstack: Reset no_proxy when the node is installed behind a proxy  https://review.openstack.org/9498813:37
*** aysyd has quit IRC13:40
EmilienMjlibosva: I started some work on unit testing, but no time to continue this week. I'll see this WE a bit.13:40
jlibosvaEmilienM: ok13:41
*** gszasz has joined #openstack-qa13:42
*** afazekas has joined #openstack-qa13:45
mtreinishcdent: I put a +2 on the unskip, I think now is a good time to try running that again13:46
cdentcool thank you mtreinish13:46
*** chandankumar has quit IRC13:48
*** gordc has joined #openstack-qa13:48
*** jecarey has quit IRC13:50
*** jgrimm is now known as zz_jgrimm13:59
openstackgerritAshish Kumar Gupta proposed a change to openstack/tempest: Port  API Tests Enhancements  https://review.openstack.org/10727714:01
*** toure_ has joined #openstack-qa14:02
*** chandankumar has joined #openstack-qa14:02
*** kashyap has joined #openstack-qa14:04
*** jecarey has joined #openstack-qa14:04
*** dmellado has quit IRC14:07
*** zz_dimtruck is now known as dimtruck14:09
*** dmellado has joined #openstack-qa14:10
*** salv-orlando has joined #openstack-qa14:10
*** pkoniszewski has quit IRC14:11
*** dkranz has joined #openstack-qa14:13
*** chandankumar has quit IRC14:18
*** mwagner_lap has joined #openstack-qa14:19
*** bnemec has joined #openstack-qa14:26
*** bknudson has joined #openstack-qa14:26
*** dimtruck is now known as zz_dimtruck14:29
*** chandankumar has joined #openstack-qa14:32
*** ramishra has joined #openstack-qa14:37
*** dustins has quit IRC14:37
*** zz_dimtruck is now known as dimtruck14:37
*** rbak has joined #openstack-qa14:38
*** dustins has joined #openstack-qa14:40
*** markmcclain has joined #openstack-qa14:41
*** markmcclain has quit IRC14:41
*** k4n0 has quit IRC14:42
*** ramishra has quit IRC14:42
*** pkoniszewski has joined #openstack-qa14:42
*** markmcclain has joined #openstack-qa14:43
*** zz_jgrimm is now known as jgrimm14:46
*** chandankumar has quit IRC14:48
*** Poornima has quit IRC14:49
*** pcrews has joined #openstack-qa14:52
*** JayJ has quit IRC14:53
*** markmcclain has quit IRC14:53
*** JayJ has joined #openstack-qa14:54
*** yamahata has joined #openstack-qa14:58
*** atiwari has joined #openstack-qa15:03
*** davpat2112 has joined #openstack-qa15:13
*** dpaterson has quit IRC15:15
*** ashishg has joined #openstack-qa15:15
*** salv-orlando has quit IRC15:15
openstackrecheckConsole logs not available after 13:26s for gate-horizon-python27 126247,6,648111515:19
*** amotoki has quit IRC15:20
*** gszasz has quit IRC15:22
*** kashyap has quit IRC15:23
*** mlavalle has joined #openstack-qa15:23
*** corvus is now known as jeblair15:30
*** ashishg has quit IRC15:32
*** dustins has quit IRC15:34
*** dustins has joined #openstack-qa15:37
*** ramishra has joined #openstack-qa15:37
*** pkoniszewski has quit IRC15:38
*** chandankumar has joined #openstack-qa15:39
*** chandankumar has quit IRC15:41
*** chandankumar has joined #openstack-qa15:42
*** ramishra has quit IRC15:43
*** kashyap has joined #openstack-qa15:52
*** markmcclain has joined #openstack-qa15:56
*** markus_z has quit IRC15:56
dkranzjogo: about https://review.openstack.org/#/c/123562/ (server termination)15:58
dkranzjogo: I can approve your patch15:58
*** markmcclain1 has joined #openstack-qa15:58
dkranzjogo: But my concern is that most tests do not follow the path you are adding/fixing15:58
dkranzjogo: they use create_test_server15:58
dkranzjogo: so why not just make wait_for_termination fail if it does not succeed as in your other proposed patch?15:59
dkranzjogo: that is, there is nothing wrong with the change, but it only impacts a tiny set of actual server deletions15:59
*** markmcclain has quit IRC16:00
*** mkollaro has quit IRC16:02
*** Guest92054 is now known as mgagne16:03
*** markmcclain1 has quit IRC16:03
*** mkollaro has joined #openstack-qa16:03
*** mgagne is now known as mgagne_16:03
*** aimon has quit IRC16:03
*** markmcclain has joined #openstack-qa16:04
*** mgagne_ is now known as Guest3117816:04
*** Guest31178 has quit IRC16:05
*** Guest31178 has joined #openstack-qa16:05
*** gszasz has joined #openstack-qa16:05
*** yamahata has quit IRC16:06
*** safchain has quit IRC16:07
*** Guest31178 is now known as mgagne16:07
openstackrecheckopenstack/glance change: https://review.openstack.org/127248 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error16:08
*** dimtruck is now known as zz_dimtruck16:12
*** cbader has joined #openstack-qa16:15
*** cdent has quit IRC16:23
*** yjiang5 has joined #openstack-qa16:26
*** zz_dimtruck is now known as dimtruck16:29
*** swamireddy has joined #openstack-qa16:32
*** aimon has joined #openstack-qa16:32
*** melwitt has joined #openstack-qa16:34
*** gszasz has quit IRC16:35
*** mkollaro has quit IRC16:37
*** ramishra has joined #openstack-qa16:38
*** jpich has quit IRC16:39
*** luqas has quit IRC16:40
*** Albert__13 has quit IRC16:41
*** melwitt has quit IRC16:41
*** melwitt has joined #openstack-qa16:41
*** ramishra has quit IRC16:42
mtreinishdkranz: if you've got a sec can you review: https://review.openstack.org/#/c/121988/16:50
mtreinishafter that lands I'm going to push out the first tempest-lib release16:50
*** vrovachev has quit IRC16:52
*** johnthetubaguy is now known as zz_johnthetubagu16:52
openstackgerritMatthew Treinish proposed a change to openstack/tempest: DO NOT MERGE: Test tempest-lib stuff  https://review.openstack.org/12729116:53
dkranzmtreinish: ok16:55
mtreinishdkranz: thanks16:56
*** jordanP has quit IRC16:56
*** andreaf has joined #openstack-qa17:05
*** jlibosva has quit IRC17:05
*** SridharG has quit IRC17:06
openstackrecheckopenstack-dev/devstack change: https://review.openstack.org/126490 failed because of: gate-tempest-dsvm-neutron-full: https://bugs.launchpad.net/bugs/1352498 https://bugs.launchpad.net/bugs/1348204 https://bugs.launchpad.net/bugs/1374458, gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/137049617:06
*** salv-orlando has joined #openstack-qa17:08
*** gfidente has quit IRC17:08
*** swamireddy1 has joined #openstack-qa17:16
*** swamireddy has quit IRC17:17
*** harlowja_away is now known as harlowja17:19
*** tosky has quit IRC17:24
openstackgerritSébastien Han proposed a change to openstack-dev/devstack: Add the ability to activate Nova ceph without Cinder  https://review.openstack.org/12620417:26
*** yfried has joined #openstack-qa17:34
*** dimtruck is now known as zz_dimtruck17:35
*** suneelb has joined #openstack-qa17:43
*** tosky has joined #openstack-qa17:47
*** chandankumar has quit IRC17:49
*** davpat2112 has quit IRC17:50
openstackrecheckopenstack/tempest change: https://review.openstack.org/115212 failed gate-tempest-dsvm-neutron-full in the gate queue with an unrecognized error17:51
*** boden has joined #openstack-qa17:51
*** swamireddy1 has quit IRC17:53
*** aix has quit IRC17:53
*** SridharG has joined #openstack-qa17:55
dkranzjogo: I have to go to another meeting and will be back in an hour17:56
dkranzjogo: If I am just wrong I can just approve your patch, or mtreinish can17:56
mtreinishdkranz: it will cause a failure, just in the cleanup phase which is not always an obvious location for debug17:57
jogodkranz: right, I want a failed delete to cause tests to fail. but not during the cleanup phase17:58
dkranzjogo: you mean not during the class cleanup phase, but fail during addCleanUp phase?18:00
jogodkranz: I think so yes18:00
mtreinishdkranz: no, make delete and wait on delete part of the test18:00
*** SridharG has left #openstack-qa18:00
jogodkranz: tests should explicitly clean up the server18:00
*** mattoliverau has quit IRC18:00
*** jhesketh has quit IRC18:00
*** mattoliverau has joined #openstack-qa18:01
dkranzjogo: Then we should not have class cleanup at all18:01
jogodkranz: well class cleanup is a fall back18:01
jogoof sorts18:01
jogodkranz: but maybe not18:01
dkranzjogo: wouldn't it be better to make create_test_server just do the addCleanUp then?18:01
*** jhesketh has joined #openstack-qa18:02
dkranzjogo: And fix the tests that are not using create_test_server to use it?18:02
jogodkranz: as long as that causes the test to fail not during the cleanup stage18:02
mtreinishdkranz: addCleanup would be the same problem18:02
jogoas in if you can see this job failed versus this cleanup failed18:02
dkranzjogo: well you're change is just putting in an addCleanUp18:02
*** zz_dimtruck is now known as dimtruck18:02
dkranzjogo: which actually confused me a little because I thought addCleanUp kind of ignored failures18:02
mtreinishdkranz: that patch just adds a wait to make sure we're not just issuing an async delete and moving on18:03
jogomtreinish: I do use addcleanup right now https://review.openstack.org/#/c/123562/2/tempest/api/compute/admin/test_servers.py18:03
dkranzmtreinish: yes, but the failure will still happen during cleanup, right?18:03
mtreinishthe fixes jogo is talking about will have to come later and will require changing some tests18:03
mtreinishdkranz: yes, but failures in addCleanup can be very confusing18:04
dkranzmtreinish: I know18:04
jogoso maybe I should revise my patch to not use addcleanup18:04
dkranzjogo: that would better achieve what you want in these cases but leave untouched the majority of server creates which use create_test_server18:05
jogodkranz: so I need to revise those next18:05
jogoshould these tests use create_test_server ?18:05
dkranzjogo: seems like you want a try/finally for deletion that does not use addCleanUp18:05
jogodkranz: good idea18:06
dkranzjogo: But the implementation of that would look exactly like what addCleanUp does I think18:06
dkranzjogo: this is not so easy18:07
jogodkranz: yeah, none of the answers we have are ideal18:07
mtreinishdkranz: the other option would be to just wrap cleanup to detect a 404 and ignore it18:07
mtreinishs/cleanup/delete/18:08
mtreinishthat way you can add the cleanup for protection18:08
jogowhat about https://docs.python.org/2/library/unittest.html#unittest.TestCase.doCleanups18:08
*** andreaf has quit IRC18:08
dkranzjogo: yeah, we could hook in there18:09
jogotry: finally: do cleanups18:09
dkranzjogo: got to run, back at next hour18:10
jogodkranz: kk18:10
*** danfinch has quit IRC18:13
*** e0ne has joined #openstack-qa18:28
*** dustins has quit IRC18:31
openstackgerritSergey Shnaidman proposed a change to openstack/tempest: Create subnet without gateway and explicit IP ver  https://review.openstack.org/11842418:32
*** suneelb has quit IRC18:32
*** suneelb has joined #openstack-qa18:32
*** dustins has joined #openstack-qa18:33
*** suneelb has quit IRC18:33
*** suneelb has joined #openstack-qa18:34
*** kashyap has quit IRC18:34
*** ramishra has joined #openstack-qa18:40
*** chandankumar has joined #openstack-qa18:44
*** ramishra has quit IRC18:44
openstackgerritA change was merged to openstack/tempest-lib: Don't edit commit history with migrate from tempest tool  https://review.openstack.org/12198818:47
*** andreaf has joined #openstack-qa18:47
*** e0ne has quit IRC18:48
*** rushiagr is now known as rushiagr_away19:00
dkranzjogo: saw your comment in the patch we were discussing. The only downside to the suggested approach is that we will be serializing execution around the synchronous deletion time which is now done in the background. afazekas noted this in his review comment.19:02
dkranzjogo: but obviously we can't both do cleanup in the background and do it before the test completes19:03
dkranzjogo: One possible compromise would be to keep the class level cleanup but record more information so that the log message tells you which test "failed"19:04
dkranzjogo: I almost think we would be better off not trying to track these failures but running explicit stress tests around deletion19:06
*** dustins has quit IRC19:06
dkranzjogo: parallel tempest in the constrained dsg environment is a kind of stress test anyway, but not a good one.19:06
dkranzjogo: there already is such a test in the tempest stress framewoek19:07
*** toure_ has quit IRC19:07
*** mwagner_lap has quit IRC19:08
openstackrecheckConsole logs not available after 13:33s for check-tempest-dsvm-neutron-heat-slow-icehouse 107654,4,c52f90d19:10
*** jordanP has joined #openstack-qa19:11
*** dustins has joined #openstack-qa19:11
afazekasdkranz: AFAIK now we do not have the logs from class level operations, just one exception19:19
dkranzafazekas: you mean from BaseComputeTest.clear_servers, I think that is right19:20
dkranzafazekas: But we could include, for example, the test in which the test was created and print that out too19:21
dkranzafazekas: But I don't know if that would satisfy the need.19:21
dkranzafazekas: Honestly, I think we would do better to run a lot of stress jobs and debug these issues there19:21
afazekasdkranz:  BTW at tempest/test.py at L239 we might want to remove the try block19:22
*** cbader has quit IRC19:22
dkranzafazekas: I don't see a try block there ??19:22
afazekasdkranz: It is late here and I am hungry , lets continue it tomorrow19:22
dkranzafazekas: github url?19:22
dkranzafazekas: ok :)19:23
afazekasL29319:23
*** chandankumar has quit IRC19:23
dkranzmtreinish, andreaf : I question whether we want the try block here https://github.com/openstack/tempest/blob/master/tempest/test.py#L29219:25
dkranzWhy should we delete the tenant if the call to clean its resources fails?19:25
dkranzWhat is the benefit?19:25
dkranzI thought the point was to  make source resource clean was always called, not to delete tenants if it fails.19:26
*** atiwari has quit IRC19:26
dkranzIf we delete the tenant then we strand the resources and things like the cleanup script cannot run19:26
dkranzafazekas: I think you are right ^^^19:28
*** afazekas has quit IRC19:28
mtreinishdkranz: it's needed for the test-accounts work, if we don't ensure we properly release the credentials at the end of a test class we'll exhaust the set19:30
dkranzmtreinish: But if resource_cleanup fails, there may be objects still in the tenant, and re-using it may fail in strange ways19:30
adam_gjavelin question.. i'd like to make creation of certain resources conditional on the services that are deployed. would it be preferred to modify the javelin code in tempest to skip creation of $foo if $bar, or to update grenade to generate a resources.yaml based on ENABLED_SERVICES or similar?19:31
adam_gjogo, ^19:31
mtreinishdkranz: it depends on the failure really. But, in the case of resource cred list exhaustion the failures would be opaque, (InvalidConfig error in setupclasss) but I feel like an unexpected resource causing a failure would at least show the source of the issue19:33
mtreinishthe other thing is we were doing a really poor job of handling this before that was added19:34
mtreinishdkranz: if you've got a suggestion on how to improve that logic push up a patch.19:34
mtreinishbut things there will change a bit when andreaf starts working on the split for the second part of that bp19:35
dkranzmtreinish: I am thinking about it but need to study how exactly the test account flow works. I reviewed some of the patches but it was a gwhile ago.19:35
dkranzmtreinish: I thought what we were doing a poor job of before was making sure that the "resource deleter stuff" was always called regardless of what happened during setUpClass or test execution.19:36
mtreinishadam_g: I think they're trying to decouple javelin as much as possible from the tempest config19:36
dkranzmtreinish: anyway, I'll think about it.19:36
mtreinishdkranz: yeah, but then when we added that (or had it with safe_setup) the issue became skips and attr errors19:37
*** tkammer has quit IRC19:37
adam_gmtreinish, makes sense19:37
dkranzmtreinish: Yes, but those are different uses. Do me, the tenant deletion even on error came along with that but is not desired.19:37
mtreinishthe tearDownClasses often depended on things being set in the setupclass which wasn't true when we raised an exception and called tearDownclass19:37
mtreinishdkranz: as a first pass you can switch that except to just be attr error19:38
dkranzmtreinish: yes, I don't deny that19:38
mtreinishand see what happens19:38
dkranzmtreinish: I think that is on the right track. We should only catch exceptions and not fail if they are exceptions that we are anticipating19:39
*** e0ne has joined #openstack-qa19:39
mtreinishadam_g: so I think handling the logic in the grenade side would probably be the right approach for that19:39
mtreinishdkranz: the problem with that, which is why I think andreaf did a blanket exception was that code gets run by ever test class, and at this point the set of known exceptions isn't always consistent19:40
mtreinishI really don't like blanket exceptions either19:40
*** tosky has quit IRC19:40
*** tosky has joined #openstack-qa19:41
dkranzmtreinish: the right thing here is to actually use the code in the resource cleanup script for resource cleanup if it gets called due to a raisd exception19:41
dkranzmtreinish: so we make sure to get them all19:42
dkranzmtreinish: right now we might lose things if cleaning a particular resource raises an exception before others can be cleaned.19:42
dkranzmtreinish: not so easy19:42
mtreinishyeah it would probably require refactoring every test class. That is the basic approach andreaf is taking with his staged setup/teardown19:43
mtreinishto at least group things, it's not at the individual call level19:43
mtreinishbut this way we should have a better idea of what to catch and ignore19:44
mtreinishdkranz: https://review.openstack.org/#/c/115353/19:44
mtreinishalthough it looks like that patch is missing the second half which pairs the setup stage to the teardowns19:45
dkranzmtreinish: If we do this right, we could get rid of all the service-specific stuff in dpatersons script since that logic should really be in the base classes of tempest proper19:47
*** dimsum_ has quit IRC19:48
*** Longgeek has quit IRC19:55
andreafmtreinish, dkranz: I think for now it's up to the test classes to handle failures on cleanup properly, e.g. if a cleanup fails continue with the next one19:58
dkranzandreaf: I agree. What I want to avoid is deletion of the user/tenant if something failed unexpectedly.20:00
andreafmtreinish, dkranz: to be able to actually handle the cleanups and retries on cleanups in the base class we would need to have a common type of resources (e.g. test resources) and handle whatever try/except or retry logic we want to have in the base class20:00
dkranzandreaf: It should really be on OpenStack to free resources used by a tenant when the tenant is deleted.20:00
*** toure_ has joined #openstack-qa20:00
andreafdkranz: so we could specialize things here a bit to solve this in the tearDownClass implementation20:01
andreafdkranz, what we need to do for test accounts is to cleanup resources which actually means only releasing locks20:01
jogodkranz: yeah I could build a separate stress test for this20:02
dkranzandreaf: so we don't have to delete the user/tenant?20:02
jogodkranz: I already added a patch to log the failure to delete20:02
andreafdkranz: while in case of dynamically provisioned credentials we actually delete the credentials20:02
mtreinishandreaf: I just left you a review comment with a super hacky code snippet of what you could try to do on the cleanup side for the setup split20:02
dkranzjogo: That would be a big plus. The fact that we are using the normal gate jobs as stress tests is not good IMO20:02
mtreinishI don't think we should land the framework without having the cleanup side mapped out too20:02
dkranzjogo: It causes goal conflict20:02
dkranzjogo: like fast vs easier to debug races20:03
andreafdkranz, we could differentiate the behaviour depending on the type of credentials, so in case we use pre-provisioned ones do the cleanup, else not20:03
andreafmtreinish: ok let me have a look20:03
dkranzandreaf: I was going to suggest another flag, but that is probably good enough20:03
mtreinishdkranz: we already have the config options for that20:04
dkranzmtreinish: yes20:04
jogodkranz: yeah I'll turn that into a stress test20:05
jogodkranz mtreinish: speaking of stress tests20:05
jogothis is the kind of thing we should run as a nova 'functional' test or at least not gate anything besides nova on20:05
andreafmtreinish: yes but there's only two things that need teardown, credentials and resources20:05
andreafsetting up the clients wouldn't need a teardown I think20:06
mtreinishandreaf: ok, then that makes it easier20:06
mtreinishI was just being thorough and adding them for each stage20:06
mtreinishjogo: we run the stress tests as a periodic only at this point20:06
dkranzjogo: I don't see stress tests as gating. That is part of our current problem.20:07
dkranzjogo: but it could run continuously20:07
jogocould we make some stress tests run as non-voting in the check queues or something?20:08
dkranzjogo: absolutely, but for the create/delete stress test it would only make sense on nova as you sais20:08
dkranzsaid20:08
mtreinishjogo: yeah that would be easy, just setup a json file with the tests you want to run and add a tox target for it (or one that takes in the json file as a posarg)20:08
andreafmtreinish, so my approach was, for skip_checks and other code before credentials are setup we don't need any teardown20:08
jogodkranz: right, so if we could have a nova-stress test job20:08
mtreinishthen we can add that job to the nova queue easily20:09
jogomtreinish: is this something that should become a tempest-lib based thing and be called a functional test?20:09
jogoor save that for another interation?20:09
jogoiteration20:09
andreafmtreinish, then we start doing things that might break, but I agree we need to track what was done that needs teardown - I guess my patch was hacked together a bit to quickly20:09
mtreinishjogo: that would probably be the next iteration, splitting the stress test framework is on the list for migration20:10
mtreinishbut it's not clear whether it's tempest-lib or something else at this point20:10
dkranzjogo: https://github.com/openstack/tempest/blob/master/tempest/stress/actions/server_create_destroy.py20:11
mtreinishbecause really all it is a custom test runner20:11
mtreinishjogo: if you're serious about setting this up, there was a test case that jeblair wanted to add. Something they were hitting in the wild20:11
mtreinishwith snapshotting20:11
mtreinishI don't remember exactly what it was though20:12
jogomtreinish: writing some tempest tests sounds like fun so I'm in20:12
jogomtreinish: are the stress tests at least self gating?20:13
dkranzjogo: what do you mean?20:13
mtreinishjogo: you wouldn't necessarily have to write new ones, you can use any existing test20:13
mtreinishjogo: no the tests in tempest/stress aren't self gating20:13
jogodkranz: do they get run on tempest changes to stress tests20:13
jogomtreinish: shouldn't they be?20:13
mtreinishprobably, but we normally encourage just writing a regular api or scenario test and then they can be used with the stress framework20:14
jogomtreinish: so I have been seeing failures in delete and in rebuild and a few other operations20:14
jogomtreinish: shouldn't there at least be a stress sanity job20:14
mtreinishjogo: there is a nightly20:14
jogowhy not gating?20:15
jogosomething that runs things just once20:15
dkranzjogo: that makes sense20:15
dkranzjogo: I will check what we do20:15
mtreinishjogo: I think it was because the stress specific tests are basically duplicates of existing test cases20:16
mtreinishwe probably should sanity check them to avoid bit rot etc20:16
mtreinishbut that's why we did it as a nightly20:16
jogomtreinish: nightly are silly20:16
mtreinishno argument from me20:16
mtreinishit's also been like >1yr since we did anything with those jobs I think20:16
openstackgerritFergal Mc Carthy proposed a change to openstack-dev/devstack: local.conf processing doesn't handle '=' in values  https://review.openstack.org/12446720:17
*** toure_ has quit IRC20:17
jogomtreinish dkranz: I'll look into moving the nightly job into a sanity job that runs in check queue (non-voting for starters)20:17
mtreinishjogo: I think it runs for an hour by default...20:17
mtreinishjogo: yep: http://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n8520:18
*** edwa6643 has quit IRC20:19
jogoare tempest logs in logstash?20:20
andreafmtreinish: in your example, if the 2nd stage fails with an unexpected exception, resources from the first step are not cleaned up20:20
jogonever mind they are20:20
andreafmtreinish: so regardless the type of the exception being expected or not, we should try and cleanup whatever resources were setup20:20
mtreinishandreaf: oops, yeah we probably should put it in a giant try blcok then to handle that20:21
*** e0ne has quit IRC20:21
jogomtreinish: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIGRlbGV0ZSBzZXJ2ZXJcIiBBTkQgdGFnczpcInRlbXBlc3QudHh0XCIgQU5EIHRhZ3M6XCJtdWx0aWxpbmVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQxMjg4NjA3MjMyNywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==20:21
jogomessage:"Failed to delete server" AND tags:"tempest.txt" AND tags:"multiline"20:21
mtreinishthe pseudo lifo logic would take care of handling the staged approach20:21
jogohmm never mind that20:21
*** e0ne has joined #openstack-qa20:21
mtreinishandreaf: we should still keep the individual try blocks after each stage I think20:22
jogomtreinish: so message:"Waiting for deletion of server" AND tags:"tempest.txt" AND build_status:"SUCCESS" isn't as frequent as I thought20:26
jogo26 in last 7days20:26
mtreinishjogo: well that's probably a good thing20:27
jogoversus over 500 for success and faiure20:27
jogomtreinish: yes20:27
openstackrecheckopenstack/glance change: https://review.openstack.org/127248 failed gate-tempest-dsvm-postgres-full in the gate queue with an unrecognized error20:27
mtreinishjogo: and it looks like a good chunk are on check jobs so they could just be bad patches20:28
mtreinishor they might all be check20:28
*** e0ne has quit IRC20:28
jogooh interesting yeah20:30
mtreinishalthough if they're on sucesses that's probably not a good thing if they are bad patches...20:30
jogoits a bunch of different changes20:30
mtreinishbbiab, gonna work my way home20:33
jogomtreinish: you aren't at home?20:33
*** jhenner1 has quit IRC20:42
openstackgerritJoe Gordon proposed a change to openstack/tempest: Add list servers to large ops test  https://review.openstack.org/7848020:45
*** edwa6643 has joined #openstack-qa20:52
mtreinishjogo: I was at a stabucks20:52
openstackgerritChristian Berendt proposed a change to openstack/tempest: Remove @author tags from copyright statements  https://review.openstack.org/12734420:53
mtreinishjogo: ^^^ is there a hacking rule for that?20:54
jogomtreinish: he posted a hacking patch this morning20:55
jogoI don't really think a hacking pach is needed for that though20:56
jogosame as vim modelines20:56
jogoonce we nix all the cases of them20:56
jogothey shouldn't be coming back20:56
jogobecause if they aren't there at all, its easy to see they are silly20:56
mtreinishjogo: you'd think, but I the ones being removed there were brand new. Plus I feel if it's the kind of thing to go to the effort of removing everywhere then it should probably be blocked20:57
openstackgerritJoe Gordon proposed a change to openstack/tempest: Make test_list_servers_filter_exist_host use create_test_server  https://review.openstack.org/12356220:58
* jogo looks20:58
mtreinishfwiw, I think tempest has a local rule about the vim modelines20:59
jogonova does too I think20:59
jogoso the answer is, when I get around to it  is make a directory for off by default rules20:59
jogothat projects can consume20:59
mtreinishjogo: yeah that would owrk20:59
jogoalthough author tags is a good candidate for a on by default rule21:00
jogoi have to review his patch, it had evil regex in it21:00
openstackgerritSteve Martinelli proposed a change to openstack-infra/elastic-recheck: Add query for bug 1379201  https://review.openstack.org/12734621:00
mtreinishandreaf: I just saw your review on that author tags patch. See ^^^21:01
*** dustins has quit IRC21:01
*** markmcclain has quit IRC21:05
andreafmtreinish: ok ,so the proposal is to have an importable hacking rule ?21:10
mtreinishandreaf: there wouldn't be any import it would just be included in the next hacking release21:10
mtreinishthe first tempest-lib release has been pushed: https://pypi.python.org/pypi/tempest-lib/0.0.121:12
mtreinishwe probably should update some of the metadata for the next one21:12
andreafmtreinish, yay :)21:13
*** HenryG has quit IRC21:14
andreafmtreinish: I'll try and propose an update on resource cleanup tomorrow21:14
mtreinishandreaf: ok cool21:14
*** jgrimm is now known as zz_jgrimm21:17
*** gordc has quit IRC21:22
openstackgerritJoe Gordon proposed a change to openstack/tempest: Move default stress test to run things 3 times instead of time based  https://review.openstack.org/12735121:23
ianwdtroyer: do you mind if I self approve https://review.openstack.org/#/c/124227/ (Single quote iniset argument in merge_config_file) -- it is holding up Baodong21:23
openstackgerritAdam Gandelman proposed a change to openstack-dev/grenade: Generate javelin config dynamically  https://review.openstack.org/12735221:23
jogomtreinish: ^21:23
dtroyerianw: I +A it…just a tiny bit better than self-approve ;)21:24
ianwdtroyer: great, thanks.  i need to make time to think about the ini_add stuff, if it can be done multi-line with an array maybe21:25
dtroyerianw: that sounds like it should work, without thinking about it too hard.  I think it should at least remove one layer of quote-stripping21:26
ianwdtroyer: context switching into that code is about an half-an-hour exercise in itself :)21:26
openstackgerritAdam Gandelman proposed a change to openstack-dev/grenade: Generate javelin config dynamically  https://review.openstack.org/12735221:26
dtroyerianw: I hear you, one reason I hadn't dug in too far yet21:27
*** mmedvede has quit IRC21:27
*** rbak has quit IRC21:30
*** rbak has joined #openstack-qa21:30
jogodkranz: https://review.openstack.org/#/c/127351/21:37
jogodoes anyone track the nightly jobs?21:37
dkranzjogo: I commented on that patch21:38
dkranzjogo: I do, with some frequency21:38
jogoI saw, and follow up is ^21:38
jogodkranz: oh neat!21:38
jogokk let me revise the patch then21:38
dkranzjogo: This is one of the biggest holes we have from a process standpoint though21:38
jogodkranz: which?21:39
dkranzjogo: the fact that the only process we have that works is blocking failures at the gate21:39
dkranzjogo: any sane organization would tier tests21:39
dkranzjogo: so that some run post-merge, but frequently21:40
jogodkranz: well in this case I just want these tests to be self gated21:40
dkranzjogo: but that only works if some one is known to be looking21:40
dkranzjogo: I know and am not objecting to that21:40
jogodkranz: so this is exactly the outcome of the  qa midcycle in germany21:40
*** adalbas has quit IRC21:40
jogodkranz: I think the difference between  gating and post-merge tests is where the burden to maintain  lands21:41
jogogating, the entire team has to maintain21:41
dkranzjogo: exactly21:41
*** mriedem has quit IRC21:41
jogopost-merge only the people who care about X have to maintain21:41
jogobecause entire team can just kill the code if it stops working21:41
jogoso AFAIK infra needs help getting post-merge jobs to work21:41
jogodkranz: if you want to help21:41
dkranzjogo: I'm not sure how. I watch the tempest nightlies.21:42
*** ramishra has joined #openstack-qa21:42
dkranzjogo: This is more of a process problem than technical.21:42
jogodkranz: there is a doc on this somewhere21:42
dkranzjogo: what kind of help does infra need?21:42
mtreinishdkranz: well to be fair the stress nightlies haven't failed in a long time21:42
jogomtreinish: do you remember where that doc is?21:43
dkranzmtreinish: Yes, but they are not very stressful :)21:43
dkranzmtreinish: let's crank them up and see what happens21:43
mtreinishjogo: doc for what?21:43
jogomtreinish: roadmap for getting to post-merge testing21:43
mtreinishnah, I didn't even know there was a doc published for it21:44
mtreinishdkranz: go for it, I've been thinking we need to run all the scenario tests as part of that nightly stress job21:44
jogomtreinish: AFAIK the email sean and jim sent out21:44
mtreinishoh, the email. Yeah I can pull up that link21:45
jogodkranz: ++ to cranking the stress up21:45
dkranzjogo, mtreinish : ok...21:45
mtreinishhttp://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html21:45
jogodkranz: one of the steps for getting to post-merge was to make it easy to discover when those jobs begin failing etc21:46
openstackgerritJoe Gordon proposed a change to openstack/tempest: Add new tox target to sanity check stress tests  https://review.openstack.org/12735121:46
*** ramishra has quit IRC21:46
jogomtreinish: what do you think of the new target name?21:46
dkranzjogo: when you get to the point that the watchers care, they just need to look!21:47
dkranzjogo: an email to the watchers is all you need21:47
jogodkranz: well we want to make it easy for the people who don't care (the core team) to know when the job hasn't worked in a long time21:47
dkranzjogo: I get an email every day about tempest21:47
jogodkranz: what about of the jobs fail 1% of the time21:47
mtreinishjogo: yeah that's fine, I would have gone stress-check, but I'm fine with sanity21:47
mtreinishdkranz: well only if it fails21:48
jogomtreinish: I liked the name21:48
dkranzjogo: Sure, it would be better if we kept a history, nice graphs, etc.21:48
mtreinishjogo: heh,  stress our sanity :)21:48
jogodkranz: anyway that email includes a bunch of steps21:48
jogodkranz: and infra should know where we are in it21:48
mtreinishdkranz: there was a thread started by jhesketh about how to handle visualizing this stuff (specifically post jobs, but its the same for periodics)21:49
dkranzmtreinish: sure21:49
*** suneelb has quit IRC21:49
*** edwa66431 has joined #openstack-qa21:49
jogomtreinish: oh neat I must have missed that21:50
mtreinishjogo: well he was proposing just sending them to the -qa list as a solution21:50
jogooh that is still a thing?21:51
* jogo wants a dashboard21:51
mtreinishyeah the list is there just for the nightlies21:51
mtreinishyeah I want a dashboard too21:51
mtreinishthat's the only thing I miss from the days of a single jenkins21:51
*** suneelb1 has joined #openstack-qa21:51
jogohaha yeah21:51
jogowell actually a dash is easy21:52
jogographite21:52
jogojhesketh: ping21:52
jogoI think with graphite + logstash we can build a decent dashboard quickly21:52
jogographite shows the pass fail rates, and logstash to find the specific jobs that failed21:53
jogomtreinish: what ML was that thread on?21:53
*** edwa6643 has quit IRC21:53
openstackrecheckopenstack/keystone change: https://review.openstack.org/119804 failed because of: gate-tempest-dsvm-neutron-full: https://bugs.launchpad.net/bugs/1357055 https://bugs.launchpad.net/bugs/131106621:54
*** boden has quit IRC21:54
mtreinishjogo: http://lists.openstack.org/pipermail/openstack-qa/2014-October/001258.html21:54
mtreinishI moved it to the dev list but it died there except for one reply from fungi that reporting back to gerrit post merge is hard21:55
mtreinishlet me dig up the dev list side21:55
mtreinishjogo: http://lists.openstack.org/pipermail/openstack-dev/2014-October/047560.html21:55
*** mmedvede has joined #openstack-qa21:56
*** edwa6643 has joined #openstack-qa22:02
*** edwa66431 has quit IRC22:05
jogomtreinish: thanks, responded22:05
jogoI think this is a very solvable thing22:05
jogofungi: ^22:05
fungicool22:07
fungii'll let you work out the map-merge-to-triggering-change logic22:08
jogofungi: which logic is that?22:08
*** jhenner has joined #openstack-qa22:08
fungithe logic to figure out how a commit in the post pipeline maps back to a change to report on22:08
*** jordanP has quit IRC22:09
jogofungi: oh it wouldn't22:09
fungizuul handles them differently because they are different things22:09
mtreinishjogo: once we get subunit2sql up and running we probably could add that to the equation also22:09
*** Sukhdev has joined #openstack-qa22:09
* fungi will read the reply before commenting further then22:09
dkranzmtreinish, jogo : I've got to go now but the current nightly stress doesn't really do much. It only runs the few things that are marked as 'stress'. This is more appropriate for the new thing Joe introduced.22:09
mtreinishget per test data for those jobs22:09
dkranzI am going to make it do something more real, and targeted. We should probably have multiple jobs for different stress areas.22:10
jogomtreinish: ahh right, it will be another tool for analysis22:10
jogodkranz: ++ that sounds like a great idea22:11
dkranzjogo: or I could just make tox run a sequence of stress runs22:11
jogodkranz: long run I would like to make the nova targeted stress stuff be a nova only functional test or something22:11
dkranzjogo: right. Should do both of job-per-service and running a sequence of targeted tests22:11
mtreinishdkranz: as I was saying before it should just be a job that takes one of the json files in from posargs then we can handle the config per job in d-g22:12
jogodkranz: ?22:12
jogoall I am trying to do at the moment is make stress jobs self gating22:12
dkranzjogo: right, that is fine. I'm not talking about touching the one you added.22:12
dkranzmtreinish: ok, but I think we may want more than that. I'll come up with something tomorrow.22:13
jogodoes the @stresstest thing still work?22:14
mtreinishjogo: heh, that's what I was just leaving a review comment on your patch for22:16
mtreinishit does, that's what -a is used for on run_stress, it runs all the tests that have the decorator set22:16
mtreinishjogo: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/cmd/run_stress.py#n3522:17
fungijogo: so on your reply, yes that should work just fine. i think trying to report post job results back to changes in gerrit is a non-starter for multiple reasons but a status dashboard of some sort (assuming people will actually look at it) would fit well22:18
jogofungi: yeah, and it shouldn't be too hard to build one I think22:18
fungiagreed... something on status.o.o site would make sense22:19
jogofungi: yeah22:19
openstackgerritAdam Gandelman proposed a change to openstack-dev/grenade: Generate javelin config dynamically  https://review.openstack.org/12735222:19
fungiit's been the solution we've held up in multiple past conversations as probably the best option, just nobody's done it yet22:20
* mtreinish is not looking forward to writing the dashboard view for subunit2sql, sucks at anything on the web...22:20
jogomtreinish: so this is silly22:20
jogomtreinish: why do I need a json file ?22:20
mtreinishjogo: this is what I was saying earlier those tests in tempest.stress.actions are really just duplicates of other test cases22:21
jogomtreinish: now I get it22:21
jogoI thought you meant the tempest/stress folder22:21
jogoerr directory22:22
jogofungi: yeah I don't really see a better way22:22
mtreinishthe json file is just the import format to the stress runner to tell it which test cases to run and the conditions on how to run them (like how to handle setupclass, how to loop, etc)22:22
openstackgerritAndrea Frittoli  proposed a change to openstack/tempest: Framework for staged setup  https://review.openstack.org/11535322:22
andreafmtreinish, still WIP ^^^ but you may want to look if you like it better22:22
mtreinishandreaf: ok I'll take a look22:23
jogofungi: so what other steps are still TODOs for getting post-gate jobs22:23
*** dkranz has quit IRC22:23
jogoor whatever we are calling them, the post-merge pool of jobs that we run in a loop22:24
*** jecarey has quit IRC22:24
jogofungi: also if you could respond to http://lists.openstack.org/pipermail/openstack-dev/2014-October/048203.html with your thoughts so its on the record22:25
jogofungi: as this actually sounds like something we can get a POC up before Paris22:28
*** ramishra has joined #openstack-qa22:43
*** ramishra has quit IRC22:47
*** suneelb has joined #openstack-qa22:49
*** suneelb1 has quit IRC22:50
*** JayJ has quit IRC22:53
*** JayJ has joined #openstack-qa22:54
*** andreaf has quit IRC23:01
*** rbak has quit IRC23:07
*** HenryG has joined #openstack-qa23:07
openstackgerritA change was merged to openstack-dev/devstack: Single quote iniset argument in merge_config_file  https://review.openstack.org/12422723:08
*** suneelb has quit IRC23:09
*** suneelb has joined #openstack-qa23:10
*** jhenner has quit IRC23:12
*** dimtruck is now known as zz_dimtruck23:14
*** aimon has quit IRC23:16
*** bknudson has quit IRC23:17
*** yfujioka has joined #openstack-qa23:22
*** yfujioka_ has joined #openstack-qa23:23
*** flaper87 is now known as flaper87|afk23:25
openstackgerritMichael Hudson-Doyle proposed a change to openstack/tempest: fix AttributeError when test_function_help fails  https://review.openstack.org/12738423:36
*** JayJ has quit IRC23:37
*** zz_jgrimm is now known as jgrimm23:43
*** ramishra has joined #openstack-qa23:44
*** edwa66431 has joined #openstack-qa23:48
*** ramishra has quit IRC23:48
*** edwa6643 has quit IRC23:50
*** mlavalle has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!