Monday, 2014-06-09

*** yamahata has quit IRC00:04
openstackrecheckopenstack/python-saharaclient change: https://review.openstack.org/97255 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/129847200:12
*** markmcclain1 has quit IRC00:14
*** matsuhashi has joined #openstack-qa00:21
*** bearhands is now known as comstud00:29
*** dmorita has joined #openstack-qa00:29
*** zhikunliu has joined #openstack-qa00:31
mriedemmtreinish: sdague: flashgordon: doing a 5th recheck on this now https://review.openstack.org/#/c/98632/ - so far in 4 runs it hasn't hit the nova-network ssh timeout failure00:36
mriedemnot ideal to revert that but if it gets things moving we might consider it00:36
*** nati_ueno has quit IRC00:53
oomichimriedem: nice investigation01:01
oomichimriedem: I guess server creation of each test would cause the heavy load and ssh setup could not make it before ssh connecting, right?01:01
mriedemoomichi: that's something we were guessing on friday morning,01:03
oomichimriedem: by reverting the patch, we can reduce the server creations, and tempest can try connecting the same server which is already set up at the first setUp.01:03
mriedemthe problem started on 6/501:03
mriedemwhich is when that change merged01:03
mriedemthere wasn't much else that merged in nova around that time01:04
oomichimriedem: nice catch:)01:04
mriedemi think we wanted to try and fix the actual ssh issue and keep the change to use independent instances,01:04
mriedembut unless we have a fix in the next day or so i'd think we'd want to get the gate flowing again - anyway, just wanted to see if that revert hit the bug01:04
mriedemso far it hasn't01:04
mtreinishmriedem: yeah it's really bizarre to me though why that would be causing issues01:05
mtreinishit doesn't add that many server creates01:05
mriedemmtreinish: how many are we talking? 3?01:05
mriedemoh one per test case01:06
mriedemsince it doesn't reuse them01:06
mtreinishyeah 1 per test01:06
mtreinishwhich is actually more than I thought01:06
mriedemtest cases run in serial too right?01:07
mtreinishthey do per class01:07
mtreinishbut there is xml and json01:07
mtreinishand v301:07
mriedemso create a test instance for each test case that runs and that instance should be deleted for each test case01:07
mriedemok so 3 versions of that test class running01:07
mriedemthey were sharing a common instance before01:07
mriedemassuming it was going to active and not rebuilt01:08
mriedemanyway, i don't like the revert, just wanted to see if it hit the bug so we know it's an option01:08
oomichiA server status becomes "active", but Tempest cannot connect the server with ssh connecting.01:13
oomichimtreinish: how about https://review.openstack.org/#/c/97812 for debugging?01:13
*** nati_ueno has joined #openstack-qa01:19
*** nati_ueno has quit IRC01:20
*** yamahata has joined #openstack-qa01:23
openstackrecheckopenstack/horizon change: https://review.openstack.org/94579 failed because of: gate-grenade-dsvm: unrecognized error, gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/132061701:34
*** amotoki has joined #openstack-qa01:36
*** nosnos has joined #openstack-qa01:43
openstackrecheckopenstack/nova change: https://review.openstack.org/98559 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1298472 https://bugs.launchpad.net/bugs/1285908, gate-grenade-dsvm-partial-ncpu: https://bugs.launchpad.net/bugs/1273908, gate-tempest-dsvm-full: unrecognized error01:49
*** GMann has joined #openstack-qa01:51
*** catohornet has joined #openstack-qa01:54
*** yamahata has quit IRC01:54
*** yamahata has joined #openstack-qa01:54
*** mriedem has quit IRC02:03
openstackgerritGhanshyam Mann proposed a change to openstack/tempest: Read template's expected value orchest test Part-1  https://review.openstack.org/9867602:13
openstackrecheckopenstack/taskflow change: https://review.openstack.org/98031 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1285908, gate-tempest-dsvm-full: unrecognized error02:21
*** catohornet has quit IRC02:23
*** mgagne1 is now known as mgagne02:30
*** lbragstad has joined #openstack-qa03:07
*** Abhijeet has joined #openstack-qa03:08
*** nosnos has quit IRC03:18
*** Madan has joined #openstack-qa03:28
*** Poornima has joined #openstack-qa03:32
openstackrecheckopenstack/nova change: https://review.openstack.org/79863 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/128590803:46
*** GMann has quit IRC03:56
openstackgerritAnthony Woods proposed a change to openstack-dev/devstack: Adds neutron linuxbridge-agent support to ironic deployment.  https://review.openstack.org/9831704:00
openstackgerritGhanshyam Mann proposed a change to openstack/tempest: Read template's expected value orchest test Part-1  https://review.openstack.org/9867604:01
*** matsuhashi has quit IRC04:04
*** nosnos has joined #openstack-qa04:06
*** Madan has quit IRC04:08
*** Madan has joined #openstack-qa04:11
*** matsuhashi has joined #openstack-qa04:13
*** yfried has quit IRC04:15
*** k4n0 has joined #openstack-qa04:16
*** SridharG has joined #openstack-qa04:29
openstackrecheckopenstack/heat change: https://review.openstack.org/95178 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/128590804:30
*** SridharG has quit IRC04:33
*** SridharG has joined #openstack-qa04:35
*** k4n0 has quit IRC04:39
*** k4n0 has joined #openstack-qa04:44
*** Madan has quit IRC04:47
*** yfried has joined #openstack-qa04:48
*** SridharG has quit IRC04:49
*** Madan has joined #openstack-qa04:49
*** rdekel has joined #openstack-qa04:49
*** psharma has joined #openstack-qa04:53
*** SridharG has joined #openstack-qa04:59
*** lbragstad has quit IRC05:01
*** Longgeek has joined #openstack-qa05:04
*** che-arne has joined #openstack-qa05:36
*** flashgordon is now known as jogo05:46
*** debu__ has joined #openstack-qa05:53
*** raies has joined #openstack-qa06:00
raieshi06:01
raiesclarkb: hi06:07
openstackrecheckopenstack/sahara change: https://review.openstack.org/97451 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1285908 https://bugs.launchpad.net/bugs/125487206:17
openstackgerritRyan Hsu proposed a change to openstack/tempest: Handle backup not found in test_create_backup  https://review.openstack.org/9869306:21
raiescyeoh: hi06:24
raiescyeoh: to check if cell extension is enabled or not, I have used test.is_extension_enabled("cell", 'compute') , when I do not enable n-cell in localrc then in spite of skip code lead to error showing amessage that Celss is not enabled06:29
raieswhat correct value should here -- test.is_extension_enabled("cell", 'compute')06:29
raiescyeoh: ^^06:29
*** coolsvap|afk is now known as coolsvap06:37
*** debu__ has quit IRC06:39
*** [1]evgenyf has joined #openstack-qa06:42
*** SridharG has quit IRC06:43
*** SridharG has joined #openstack-qa06:43
*** Longgeek has quit IRC06:46
*** bkopilov has joined #openstack-qa06:49
*** yrabl has joined #openstack-qa06:49
*** jlibosva has joined #openstack-qa06:56
*** jpich has joined #openstack-qa06:56
*** salv-orlando has joined #openstack-qa06:57
*** nosnos has quit IRC06:57
*** salv-orlando has quit IRC06:59
*** che-arne_afk has joined #openstack-qa06:59
*** salv-orlando has joined #openstack-qa07:00
*** salv-orlando has quit IRC07:00
*** raies has quit IRC07:01
*** raies has joined #openstack-qa07:02
raiescyeoh: you there ??07:02
*** GMann has joined #openstack-qa07:02
*** che-arne has quit IRC07:03
openstackgerritAndrey Pavlov proposed a change to openstack/tempest: add block_device_mapping to create_server params for nova clients  https://review.openstack.org/9869907:06
*** zhikunliu has quit IRC07:13
*** Longgeek has joined #openstack-qa07:16
openstackrecheckopenstack/cliff change: https://review.openstack.org/97405 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/129847207:20
*** Longgeek has quit IRC07:22
*** jhenner has quit IRC07:22
*** qba73 has joined #openstack-qa07:26
raiesmkoderer: hi07:27
*** che-arne has joined #openstack-qa07:30
*** che-arne_afk has quit IRC07:31
*** GMann has quit IRC07:34
*** jordanP has joined #openstack-qa07:45
openstackgerritAndrey Pavlov proposed a change to openstack/tempest: fix usage of allow_tenant_isolation  https://review.openstack.org/9713907:50
openstackgerritAndrey Pavlov proposed a change to openstack/tempest: Add 'Member' role to all created users  https://review.openstack.org/9646007:50
*** che-arne_afk has joined #openstack-qa07:52
*** che-arne_afk_ has joined #openstack-qa07:56
*** che-arne has quit IRC07:56
*** che-arne_afk has quit IRC07:57
*** jhenner has joined #openstack-qa08:01
*** openstackgerrit has quit IRC08:02
*** vrovachev has joined #openstack-qa08:03
*** GMann has joined #openstack-qa08:03
*** masayukig has quit IRC08:05
*** masayukig_ has joined #openstack-qa08:06
*** openstackgerrit has joined #openstack-qa08:07
*** plomakin has quit IRC08:08
*** anteaya has quit IRC08:10
*** ewindisch has quit IRC08:10
*** _ewindisch has joined #openstack-qa08:11
*** maurosr has quit IRC08:11
*** _ewindisch is now known as ewindisch08:11
openstackrecheckopenstack/taskflow change: https://review.openstack.org/97964 failed gate-grenade-dsvm in the gate queue with an unrecognized error08:13
*** plomakin has joined #openstack-qa08:13
*** maurosr has joined #openstack-qa08:14
*** andreaf has joined #openstack-qa08:16
*** smurashov has quit IRC08:20
*** smurashov has joined #openstack-qa08:21
*** tristanC has quit IRC08:23
*** tristanC has joined #openstack-qa08:24
*** coolsvap has quit IRC08:31
*** coolsvap has joined #openstack-qa08:32
openstackgerritMark McLoughlin proposed a change to openstack-infra/elastic-recheck: Match #1298472 fingerprint against console tag  https://review.openstack.org/9871608:38
*** yamahata has quit IRC08:41
*** Poornima has quit IRC08:44
*** gszasz has joined #openstack-qa08:45
*** tkammer_ has joined #openstack-qa08:49
*** Abhijeet has quit IRC08:50
*** coolsvap is now known as coolsvap|afk08:50
*** GMann has quit IRC08:51
*** GMann has joined #openstack-qa08:51
*** rdekel has quit IRC08:52
*** rdekel has joined #openstack-qa08:52
*** GMann has quit IRC08:53
*** che-arne_afk_ is now known as che-arne08:55
*** mkollaro has joined #openstack-qa08:59
*** catohornet has joined #openstack-qa09:00
openstackrecheckopenstack/ceilometer change: https://review.openstack.org/95666 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/129847209:00
*** coolsvap|afk is now known as coolsvap09:01
openstackgerritAnthony Woods proposed a change to openstack-dev/devstack: Adds neutron linuxbridge-agent support to ironic deployment.  https://review.openstack.org/9831709:03
*** Longgeek has joined #openstack-qa09:07
*** che-arne has quit IRC09:09
*** Longgeek has quit IRC09:12
*** che-arne has joined #openstack-qa09:12
*** salv-orlando has joined #openstack-qa09:18
*** salv-orlando has quit IRC09:18
*** roeyc has joined #openstack-qa09:19
*** [1]evgenyf has quit IRC09:21
*** salv-orlando has joined #openstack-qa09:27
*** dosaboy_ is now known as disaboy09:29
*** disaboy is now known as dosaboy09:30
*** matsuhashi has quit IRC09:31
*** matsuhashi has joined #openstack-qa09:31
*** salv-orlando has quit IRC09:32
openstackgerritMh Raies proposed a change to openstack/tempest: Compute "cells" basic API test  https://review.openstack.org/9033709:33
openstackgerritPranali Deore proposed a change to openstack/tempest: Added test case for nova keypair  https://review.openstack.org/7954909:34
*** [1]evgenyf has joined #openstack-qa09:37
openstackgerritVadim Rovachev proposed a change to openstack/tempest: Added cinder and neutron notifications tests for ceilometer  https://review.openstack.org/6716409:37
*** matsuhashi has quit IRC09:39
*** matsuhashi has joined #openstack-qa09:40
*** tosky has joined #openstack-qa09:41
*** catohornet has quit IRC09:42
*** fbo_away is now known as fbo09:44
openstackgerritVadim Rovachev proposed a change to openstack/tempest: Added cinder and neutron notifications tests for ceilometer  https://review.openstack.org/6716409:54
openstackrecheckopenstack/ceilometer change: https://review.openstack.org/96819 failed because of: gate-ceilometer-python27: https://bugs.launchpad.net/bugs/132682009:56
*** che-arne has quit IRC10:02
*** che-arne has joined #openstack-qa10:03
*** che-arne_afk has joined #openstack-qa10:06
*** che-arne has quit IRC10:09
*** anteaya has joined #openstack-qa10:10
*** SridharGaddam has joined #openstack-qa10:17
*** coolsvap is now known as coolsvap|afk10:17
*** SridharG has quit IRC10:18
*** coolsvap|afk is now known as coolsvap10:25
*** dmorita has quit IRC10:28
*** qba73 has quit IRC10:30
*** Longgeek has joined #openstack-qa10:31
openstackgerritMh Raies proposed a change to openstack/tempest: Missing Nova CLI tests  https://review.openstack.org/9873510:33
*** che-arne has joined #openstack-qa10:40
*** malini_afk is now known as malini10:40
*** malini has left #openstack-qa10:41
*** che-arne_afk has quit IRC10:43
*** rdekel has quit IRC10:47
openstackgerritMh Raies proposed a change to openstack/tempest: Missing Nova CLI tests  https://review.openstack.org/9873510:47
*** matsuhashi has quit IRC10:51
openstackrecheckopenstack/glance change: https://review.openstack.org/92739 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1298472 https://bugs.launchpad.net/bugs/128590810:53
openstackgerritVadim Rovachev proposed a change to openstack/tempest: Add glance notifications tests for ceilometer  https://review.openstack.org/7125910:53
openstackgerritAnthony Woods proposed a change to openstack-dev/devstack: Adds neutron linuxbridge-agent support to ironic deployment.  https://review.openstack.org/9831710:54
*** che-arne_afk has joined #openstack-qa10:55
*** matsuhashi has joined #openstack-qa10:58
*** che-arne has quit IRC10:58
*** dkorn has joined #openstack-qa11:13
vrovachevfolks, o/11:16
vrovachevI see several strange issues with devstack+tempest now (at least speaking about ceilometer testing)11:16
vrovachevhttps://review.openstack.org/#/c/98344/ and https://review.openstack.org/#/c/95640/ seem to have one root for failings11:17
vrovachevso tempest job and ceilometerclient gate are failing because of one reason11:17
vrovachevalthough tests are different11:18
vrovachevin console.log it's seens that "wrong nova credentials" are provided11:18
vrovachevis that known issue?11:18
*** dkorn1 has joined #openstack-qa11:18
openstackrecheckopenstack/taskflow change: https://review.openstack.org/98623 failed because of: gate-tempest-dsvm-neutron: https://bugs.launchpad.net/bugs/125489011:18
vrovachevsdague, mtreinish ^^11:18
*** matsuhashi has quit IRC11:18
*** dkorn has quit IRC11:18
*** matsuhashi has joined #openstack-qa11:19
sdaguevrovachev: right now there are a ton of failures in the tree. I'd suggest trying to create an elastic recheck pattern for ones you are seeing if they don't have one yet11:20
*** che-arne has joined #openstack-qa11:23
*** che-arne_afk has quit IRC11:23
vrovachevsdague: okay :) thank so much :)11:23
*** matsuhashi has quit IRC11:24
openstackgerritYair Fried proposed a change to openstack/tempest: Increases debug data in case of ssh key failure  https://review.openstack.org/8558811:32
openstackrecheckConsole logs not available after 13:27s for check-tempest-dsvm-full 98499,1,6b6784c11:34
*** che-arne_afk has joined #openstack-qa11:35
openstackgerritMh Raies proposed a change to openstack/tempest: Mising CLI test for Keystone  https://review.openstack.org/9874911:35
*** weshay has joined #openstack-qa11:37
*** Madan has quit IRC11:37
*** che-arne has quit IRC11:37
*** jordanP has quit IRC11:44
*** jordanP has joined #openstack-qa11:45
openstackgerritVadim Rovachev proposed a change to openstack/tempest: Add glance notifications tests for ceilometer  https://review.openstack.org/7125911:46
openstackgerritPooja Tiwari proposed a change to openstack/tempest: Added unit test for bug#1325436  https://review.openstack.org/9875111:49
*** eezhova has quit IRC11:52
*** eezhova has joined #openstack-qa11:59
openstackrecheckConsole logs not available after 14:31s for check-tempest-dsvm-full 98470,2,c77f91112:00
*** dkranz has joined #openstack-qa12:01
*** giulivo has joined #openstack-qa12:07
*** mwagner_lap has quit IRC12:08
*** adalbas has joined #openstack-qa12:11
*** dkorn1 has quit IRC12:12
openstackrecheckConsole logs not available after 14:06s for check-tempest-dsvm-full 95823,6,5cdbec212:15
openstackrecheckopenstack/heat change: https://review.openstack.org/95178 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/128590812:16
*** che-arne has joined #openstack-qa12:19
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: update 1275500 to match more broadly  https://review.openstack.org/9876712:19
*** che-arne_afk has quit IRC12:20
*** aysyd has joined #openstack-qa12:24
openstackgerritVadim Rovachev proposed a change to openstack/tempest: Add swift notifications tests for ceilometer  https://review.openstack.org/7099812:25
openstackgerritA change was merged to openstack-infra/elastic-recheck: Match #1298472 fingerprint against console tag  https://review.openstack.org/9871612:27
*** coolsvap is now known as coolsvap|afk12:27
*** levi_b has joined #openstack-qa12:27
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: remove 1285908 query  https://review.openstack.org/9877412:32
openstackgerritA change was merged to openstack-infra/elastic-recheck: update 1275500 to match more broadly  https://review.openstack.org/9876712:33
*** che-arne_afk has joined #openstack-qa12:34
*** early has quit IRC12:35
sdaguedkranz you about?12:36
*** openstackrecheck has quit IRC12:37
*** openstackrecheck has joined #openstack-qa12:37
*** che-arne has quit IRC12:38
*** ildikov_ has quit IRC12:38
*** early has joined #openstack-qa12:39
*** che-arne has joined #openstack-qa12:40
*** SridharGaddam has quit IRC12:42
*** che-arne_afk has quit IRC12:43
openstackgerritA change was merged to openstack-infra/elastic-recheck: remove 1285908 query  https://review.openstack.org/9877412:47
*** levi_b has quit IRC12:49
*** annegentle has joined #openstack-qa12:49
*** che-arne_afk has joined #openstack-qa12:50
*** SridharG has joined #openstack-qa12:50
*** jchai has joined #openstack-qa12:52
openstackrecheckConsole logs not available after 14:38s for check-tempest-dsvm-full 91554,10,cf9f41512:52
*** k4n0 has quit IRC12:52
*** che-arne has quit IRC12:53
*** openstackrecheck has quit IRC12:58
*** openstackrecheck has joined #openstack-qa12:58
mtreinishsdague: did you see mriedem's revert which hasn't hit the ssh timeout in 6 or 7 checks?13:13
sdaguemtreinish: url?13:13
mtreinishhttps://review.openstack.org/#/c/98632/13:13
*** che-arne has joined #openstack-qa13:13
*** mriedem has joined #openstack-qa13:14
sdaguewell, at this point I'm up for anything13:15
sdaguemriedem: you want to un WIP that?13:15
andreafmtreinish, sdague: i just triggered another recheck - I think even if it doesn't necessarily fix the issue it's worth a try13:15
sdagueI thought as core reviewers we could do it13:15
mriedemdid someone say 'up for anything'?13:15
sdaguehttps://review.openstack.org/#/c/98632/13:15
*** rbak has quit IRC13:15
mriedemdone13:16
openstackrecheckConsole logs not available after 13:41s for check-grenade-dsvm-icehouse 95789,1,fa3ff4913:16
sdaguemtreinish: you want to +A it, and we can pop it to the top of gate13:17
mtreinishsdague: yep I did13:17
*** che-arne_afk has quit IRC13:17
*** jaypipes has joined #openstack-qa13:19
*** ChanServ sets mode: +o jaypipes13:19
mtreinishI do want to figure out what that change broke though, it's not obvious to me13:19
*** che-arne_afk has joined #openstack-qa13:20
*** che-arne has quit IRC13:21
sdaguemtreinish: honestly, I think nova-network is racing on itself13:21
sdagueand the net effect of that change is we build more computes13:22
sdagueespecially more computes at the same time13:22
sdaguebecause the Actions classes have a habit of running at aprox the same time13:22
*** bkopilov has quit IRC13:22
*** dustins has joined #openstack-qa13:23
sdaguemriedem: I also deleted the fingerprint that was top of the list but was 80% success13:23
sdaguebecause it was definitely hiding issues13:23
mriedemsdague: the libvirt one?13:24
sdagueyeh13:24
mriedemcool13:24
mriedemwas wondering about that13:24
sdagueas far as can tell even in failing runs, the libvirt error wasn't actually related13:24
sdagueat least in the one I stepped through13:24
sdaguewhich had 2 fails13:24
sdagueone was 1275500 (though the query there was too narrow, so not actually hitting any more)13:25
sdagueand one was this phantom nova-network just doesn't do the right thing bit13:25
*** jaypipes has quit IRC13:28
*** jaypipes has joined #openstack-qa13:29
*** ChanServ sets mode: +o jaypipes13:29
openstackrecheckConsole logs not available after 13:28s for check-tempest-dsvm-full 71259,4,340034a13:29
*** Obulpathi has joined #openstack-qa13:31
*** Obulpathi has quit IRC13:31
*** malini has joined #openstack-qa13:32
*** Obulpathi has joined #openstack-qa13:32
*** ekhugen has joined #openstack-qa13:33
*** jaypipes has quit IRC13:34
*** jaypipes has joined #openstack-qa13:35
*** ChanServ sets mode: +o jaypipes13:35
*** jaypipes has quit IRC13:36
*** jaypipes has joined #openstack-qa13:38
*** ChanServ sets mode: +o jaypipes13:38
*** vrovachev has quit IRC13:39
*** yrabl has quit IRC13:39
mriedemmtreinish: sdague: do we have a grenade bug yet for the case that every service fails to start?13:40
mriedemi can't find root cause yet13:40
mtreinishmriedem: I thought there was a catch all bug for service fails to start13:40
*** cbader has joined #openstack-qa13:40
mtreinishwe should be capturing the failure files now too13:40
mriedemoh nvm i guess it was bug 1298472 and markmc fixed the e-r query for that to catch grenade fails13:42
mriedemi was just surprised by the number of 'service failed to start' messages in this log13:42
mriedemhttp://logs.openstack.org/93/98493/1/check/check-grenade-dsvm/bfb5945/console.html.gz13:42
*** salv-orlando has joined #openstack-qa13:42
openstackrecheckConsole logs not available after 13:26s for check-grenade-dsvm 98263,1,1f4f1c913:43
*** che-arne has joined #openstack-qa13:44
*** jaypipes has quit IRC13:45
*** jaypipes has joined #openstack-qa13:46
*** ChanServ sets mode: +o jaypipes13:46
*** che-arne_afk has quit IRC13:47
*** jaypipes has quit IRC13:49
*** jaypipes has joined #openstack-qa13:49
*** ChanServ sets mode: +o jaypipes13:49
*** Longgeek has quit IRC13:55
*** Longgeek has joined #openstack-qa13:55
openstackrecheckConsole logs not available after 14:13s for check-grenade-dsvm 75365,31,bf7c18413:57
openstackgerritMalini Kamalambal proposed a change to openstack/tempest: Add Tests for Message & Claim APIs  https://review.openstack.org/9494413:57
*** mwagner_lap has joined #openstack-qa14:00
*** salv-orlando has quit IRC14:01
*** beekneemech is now known as bnemec14:02
*** ekhugen has quit IRC14:04
*** rwsu has joined #openstack-qa14:04
*** rbak has joined #openstack-qa14:06
*** jgrimm has joined #openstack-qa14:06
*** salv-orlando has joined #openstack-qa14:08
*** ildikov has joined #openstack-qa14:11
*** dustins1 has joined #openstack-qa14:13
*** dustins has quit IRC14:16
openstackrecheckConsole logs not available after 13:36s for check-tempest-dsvm-full 98721,3,81be61114:18
*** ekhugen has joined #openstack-qa14:19
*** mlavalle has joined #openstack-qa14:22
*** rbak has quit IRC14:25
openstackrecheckopenstack/python-keystoneclient change: https://review.openstack.org/98534 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/129847214:26
*** SlickN1k has joined #openstack-qa14:27
*** rbak has joined #openstack-qa14:27
*** kgriffs|afk is now known as kgriffs14:28
sdaguemriedem: the actions patch did pass through check and gate when it landed. So it's really suspicious to me that we got that lucky and that's the real issue.14:36
sdaguebut like I said in email and on channel, nova-network is a blackhole14:36
sdagueso we really have no idea what's going on14:36
*** dims has joined #openstack-qa14:36
mriedemsdague: did we still have some outstanding debug patches to land to help diagnose issues?14:37
mriedemheard something about floating IPs maybe?14:37
sdaguethere is some stuff tempest side14:37
mriedemwere there any infra network changes around 6/5?14:37
mtreinishmriedem: they were on the tempest side, and they didn't show anything14:37
mriedemthat could impact how it serves IPs?14:37
sdaguebut honestly, I don't think they are going to help14:37
sdaguemriedem: well we're on HP cloud 1.1 now14:38
sdagueso ... maybe14:38
mriedemwhen did that move happen?14:38
sdaguelast week14:38
mriedembefore 6/5?14:38
sdaguethat was why everything was on fire mon / tues14:38
sdagueyes14:38
mriedemok14:38
mriedemand that's using neutron?14:38
sdagueyes14:38
mriedemblack holes....14:38
mriedemindeed14:38
sdaguebut the point being, go look at an n-net log14:38
sdagueany one14:39
sdagueit's all at debug, and pretty haphazard14:39
*** salv-orlando has quit IRC14:39
mriedemdo we need some more trace in n-net logs?14:40
openstackrecheckConsole logs not available after 13:32s for check-tempest-dsvm-postgres-full 98715,2,e855dd014:40
* mriedem is reading the email now14:40
mriedemok, so more trace logging needed in n-net14:41
sdaguemriedem: well, honestly less trace logging14:41
sdaguemore work unit logging14:42
sdagueas well as actually checking the work that it attempted to do happened14:42
dimsmriedem, to figure out why we are seeing the ssh timeout? (Bug 1298472 - SSHTimeout in tempest scenario tests using nova-network)14:42
* dims catching up14:42
mriedemyes14:42
mriedemdims: yes14:42
mriedemsdague: ok, got some charts i have to work on today (not looking forward to it), then i can help with this if no one else has jumped on by then14:43
sdaguemriedem: cool, thanks14:43
*** salv-orlando has joined #openstack-qa14:46
*** roeyc has quit IRC14:47
*** psharma has quit IRC14:48
*** coolsvap|afk is now known as coolsvap14:50
openstackrecheckConsole logs not available after 13:25s for check-grenade-dsvm 98776,1,531dd4814:53
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: do not override ceilometer default polling interval  https://review.openstack.org/9881514:54
*** lbragstad has joined #openstack-qa14:54
sdaguemtreinish: how you feeling about this - https://review.openstack.org/#/c/9781214:57
sdagueit apparently got lost somewhere14:57
mtreinishsdague: does it actually provide useful debug info?14:58
*** bkopilov has joined #openstack-qa14:59
openstackgerritA change was merged to openstack/tempest: Revert "don't reuse servers in test_server_actions"  https://review.openstack.org/9863215:00
openstackgerritA change was merged to openstack/tempest: provide ability to tweak oslo log defaults  https://review.openstack.org/9851515:00
sdaguegetting the console? I'm not sure we're going to know until we hit an error case15:00
*** yfried has quit IRC15:01
mtreinishsdague: the code looks fine to me, but I'm just trying to weigh dumping more info into the failure logs vs the potential benefit15:01
mtreinishbecause it'll go on the console output which just makes figuring out what failed harder15:02
mtreinishsdague: maybe if we paired it with something like this: https://review.openstack.org/#/c/97894/15:03
openstackgerritSridhar Gaddam proposed a change to openstack/tempest: Improve IPV6 parity in Security Group testcases  https://review.openstack.org/9413015:03
sdaguemtreinish: I actually think that's going to take us backwards15:05
mtreinishmoving the failure logs to a separate file?15:06
sdagueyep15:06
sdagueif you want to put them at the end of the output run again, I'm fine with it, but I don't actually want them shuffled off to another file15:06
*** mriedem has quit IRC15:07
sdagueif we think the logs are confusing, we should get to the bottom of the lines that are horribly confusing and fix them15:08
dkranzmtreinish: I think I agree with Sean. I was looking at some failures and missed having failures show up at the end.15:08
sdaguenot take them out of the console15:08
*** jaypipes has quit IRC15:09
mtreinishmy problem is when trying to debug a failure, that having huge fail logs with console dumps and other things makes it difficult to figure out exactly what failed from the console output15:09
mtreinishmoving the failures to the end doesn't actually help with that15:09
mtreinishor maybe it would15:10
mtreinishlet me respin that to put it at the end15:10
*** cdent has joined #openstack-qa15:10
openstackrecheckConsole logs not available after 13:23s for check-grenade-dsvm 94360,1,e4110af15:10
*** che-arne_afk has joined #openstack-qa15:10
*** mriedem has joined #openstack-qa15:11
sdaguealso, one of the issues is the subunit log format isn't our log format15:13
sdagueI wonder if there is a good way to reset that15:13
*** che-arne has quit IRC15:13
*** kgriffs is now known as kgriffs|afk15:14
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Change how tempest debug logs are displayed  https://review.openstack.org/9789415:14
mriedemmtreinish: just for context, the patch to dump console logs on timeout/failure are for debugging the various timed out waiting for state failures, not the n-net issue15:17
mriedembut i'm all for dumping those to some other log file if possible15:18
*** morganfainberg_Z is now known as morganfainberg15:18
mtreinishmriedem: no I got that, I'm just concerned with figuring out what actually failed in those case.15:19
mtreinishThe console logs, at least for me, gunk up the console log and make it hard to figure out what happened15:19
*** che-arne has joined #openstack-qa15:19
mtreinishsdague: wait we log the the console at debug level right? I thought we only captured at info and above in the fixture15:20
mtreinishor did that change with your log refactoring?15:20
sdaguemtreinish: good question, I don't know15:20
sdaguehonestly I've got like 25 patches outstanding right now, I need to start sifting back through them to figure out what's viable15:21
*** jaypipes has joined #openstack-qa15:21
*** ChanServ sets mode: +o jaypipes15:21
*** che-arne_afk has quit IRC15:22
sdaguemtreinish: we must be logging debug because of all the glanceclient spam that's in there15:24
sdaguemtreinish: any idea how to actually adjust the log format line that's in subunit so that it matches tempest?15:24
sdaguebecause that's part of what makes thing confusing as well, the request ids and log levels aren't displayed15:25
openstackrecheckConsole logs not available after 14:49s for check-grenade-dsvm-partial-ncpu 98603,2,bd3cf2715:25
mtreinishsdague: oh it looks like we capture at the config log_level because of fixture bug. because we use nuke_handlers=False the fixture would override the configured log_level15:27
mtreinishsdague: I don't know about the subunit thing, I'd have to dig into it15:27
*** Obulpath_ has joined #openstack-qa15:28
*** tkammer_ has quit IRC15:29
*** jgrimm has quit IRC15:29
*** kgriffs|afk is now known as kgriffs15:29
sdaguemtreinish: oh, I see it15:29
*** Obulpath_ has quit IRC15:31
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Make run_stress.py script an entry point  https://review.openstack.org/9251915:31
*** Obulpath_ has joined #openstack-qa15:31
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Add bug skip for test_container_synchronization()  https://review.openstack.org/9260615:32
*** Obulpathi has quit IRC15:32
openstackgerritSean Dague proposed a change to openstack/tempest: make logformat match oslo  https://review.openstack.org/9881915:33
sdaguemtreinish: ^^^15:33
*** jpich has quit IRC15:34
*** samuelbercovici has joined #openstack-qa15:35
openstackrecheckConsole logs not available after 13:26s for check-tempest-dsvm-full 97337,1,1b04ad615:38
*** Obulpath_ has quit IRC15:41
*** Obulpathi has joined #openstack-qa15:41
*** mlavalle_ has joined #openstack-qa15:42
*** mlavalle has quit IRC15:42
*** mlavalle_ is now known as mlavalle15:42
mtreinishsdague: shouldn't you pull that from the CONF object? (I think the format is a config field)15:43
sdaguewe could, that's a pretty close approximation15:43
openstackgerritSean Dague proposed a change to openstack/tempest: make logformat match oslo  https://review.openstack.org/9881915:46
*** Obulpath_ has joined #openstack-qa15:46
*** mlavalle_ has joined #openstack-qa15:47
*** mlavalle has quit IRC15:49
*** mlavalle_ is now known as mlavalle15:49
*** Obulpathi has quit IRC15:49
*** jlibosva has quit IRC15:51
*** bauzas has joined #openstack-qa15:52
openstackrecheckConsole logs not available after 13:24s for check-tempest-dsvm-full 95370,3,ab062d815:53
*** mrodden has quit IRC15:54
*** jchai has quit IRC15:54
*** samuelbercovici1 has joined #openstack-qa15:56
*** malini is now known as malini_afk15:57
*** andreaf has quit IRC16:00
*** samuelbercovici has quit IRC16:00
*** samuelbercovici1 is now known as samuelbercovici16:00
*** aimon_ has quit IRC16:02
*** beisner-weekend has quit IRC16:03
*** ekhugen has quit IRC16:06
*** mrodden has joined #openstack-qa16:07
openstackgerritGreg Lucas proposed a change to openstack-dev/devstack: Use correct path when extracting tar.gz image  https://review.openstack.org/9882516:07
*** ildikov_ has joined #openstack-qa16:08
*** aimon_ has joined #openstack-qa16:10
*** yamahata has joined #openstack-qa16:10
openstackrecheckConsole logs not available after 13:34s for check-tempest-dsvm-postgres-full-icehouse 98505,4,3b8341316:10
*** ildikov has quit IRC16:10
*** gszasz has quit IRC16:11
*** yfried has joined #openstack-qa16:13
*** yfried has quit IRC16:13
*** yfried has joined #openstack-qa16:13
*** beisner has joined #openstack-qa16:14
*** jgrimm has joined #openstack-qa16:16
*** che-arne has quit IRC16:16
*** jordanP has quit IRC16:17
*** che-arne has joined #openstack-qa16:17
*** Longgeek has quit IRC16:19
*** ekhugen has joined #openstack-qa16:19
*** markmcclain has joined #openstack-qa16:21
*** jaypipes has quit IRC16:21
*** che-arne has quit IRC16:21
*** yamahata has quit IRC16:22
*** yamahata has joined #openstack-qa16:22
openstackrecheckConsole logs not available after 13:33s for check-tempest-dsvm-full 75346,8,c6660fb16:26
*** aimon_ has quit IRC16:31
*** yamahata has quit IRC16:35
*** yamahata has joined #openstack-qa16:35
*** [1]evgenyf has quit IRC16:37
openstackrecheckConsole logs not available after 13:29s for check-grenade-dsvm 88067,5,4d36a6216:40
*** catohornet has joined #openstack-qa16:51
*** dkorn has joined #openstack-qa16:54
openstackrecheckConsole logs not available after 13:25s for check-grenade-dsvm 97782,3,6a90f5b16:57
*** aimon_ has joined #openstack-qa17:01
*** dkorn has quit IRC17:03
openstackrecheckopenstack/heat change: https://review.openstack.org/94755 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/125487217:04
*** marun has joined #openstack-qa17:07
*** mmaglana has joined #openstack-qa17:09
*** kgriffs is now known as kgriffs|afk17:09
*** harlowja_away is now known as harlowja17:09
*** marun has quit IRC17:11
openstackrecheckConsole logs not available after 13:30s for check-grenade-dsvm-partial-ncpu-icehouse 98497,2,803dcfa17:18
*** samuelbercovici has quit IRC17:23
*** nati_ueno has joined #openstack-qa17:27
openstackgerritmeera-belur proposed a change to openstack/tempest: Adding tests for the barbican - key management service  https://review.openstack.org/9884217:29
*** markmcclain has quit IRC17:30
*** markmcclain has joined #openstack-qa17:33
*** Manishanker has joined #openstack-qa17:36
*** jay_ has joined #openstack-qa17:43
openstackrecheckConsole logs not available after 13:24s for check-grenade-dsvm 89897,28,4ba168c17:46
openstackgerritChuck Short proposed a change to openstack-dev/devstack: Add utopic support  https://review.openstack.org/9884417:47
*** malini_afk is now known as malini17:48
*** Manishanker has quit IRC17:52
*** cdent has quit IRC17:54
*** malini is now known as malini_afk17:59
openstackrecheckConsole logs not available after 13:33s for check-tempest-dsvm-postgres-full 97339,4,ac9838418:00
*** mkoderer has quit IRC18:02
*** coolsvap is now known as coolsvap|afk18:03
*** devananda has joined #openstack-qa18:08
devanandahi! I'm wondering why we're not seeing any E-R posts on ironic changes18:08
devanandaeg, https://review.openstack.org/#/c/86536/ totally matched the recheck for 1326754 and it's showing up in the graphs on status.oo.o/elastic-recheck/18:09
devanandabut we dont see any posts on the reviews when there's a failure18:09
*** kgriffs|afk is now known as kgriffs18:09
clarkbdevananda: because its behind18:11
clarkbcurrently unable to index quickly enough18:11
devanandaclarkb: so I think it's never posted a single comment on any ironic patch, actually18:13
devanandaclarkb: at least none this month (as far back as my local inbox goes)18:14
clarkbwell you only just started classifying things18:14
clarkband we have been in this hole since before you did that18:14
*** jaypipes has joined #openstack-qa18:15
*** ChanServ sets mode: +o jaypipes18:15
*** mrodden has quit IRC18:17
openstackrecheckConsole logs not available after 13:27s for check-tempest-dsvm-full-icehouse 92519,4,812221518:17
devanandaclarkb: we've had E-R checks up since thursday, and E-R has commented on plenty of /other/ things over the weekend, and those bugs have been matched by errors that ironic hit over the weekend, but E-R didn't comment18:18
*** kgriffs is now known as kgriffs|afk18:19
devanandaclarkb: E-R is clearly running against Nova changes today. So, when should I expect E-R to be commenting on Ironic for queries we added last week?18:21
*** mrodden has joined #openstack-qa18:21
*** kgriffs|afk is now known as kgriffs18:21
clarkbdevananda: it should be doing it now18:22
clarkbbut it is behind18:22
devanandaclarkb: days behind?18:22
clarkbthe logstash job queue graph on the zuul status page describes it18:22
clarkbdevananda: no18:22
clarkbdevananda: but E-R gives up after 15 minutes18:22
devanandaah18:23
clarkbwe are working on fixing the backlog18:23
clarkbgoing to stop indexing some data that hasn't been useful and add more workers18:23
*** atiwari has joined #openstack-qa18:24
*** dims has quit IRC18:25
*** dims has joined #openstack-qa18:25
*** ildikov has joined #openstack-qa18:27
dkranzboris-42: Are you intending to provide more detail in https://review.openstack.org/#/c/94473/ ?18:28
boris-42devananda sure18:29
boris-42dkranz sure*18:29
boris-42devananda sorry*18:29
*** ildikov has quit IRC18:29
boris-42dkranz there are 2 more devs in rally that are interested in supporting better integration of rally & tempest18:29
boris-42dkranz so I hope they will help me as well with this part18:29
dkranzboris-42: ok, I am asking because it is untouched since May 2318:30
boris-42dkranz sorry for that, have a lot of work18:30
*** markmcclain has quit IRC18:30
dkranzboris-42: understood18:30
boris-42dkranz would like to get osprofiler in openstack finally18:30
*** ildikov_ has quit IRC18:30
*** markmcclain has joined #openstack-qa18:30
boris-42dkranz seems like it's close to done https://github.com/stackforge/osprofiler18:30
openstackrecheckConsole logs not available after 13:27s for check-grenade-dsvm 92606,2,2d732d418:30
dkranzboris-42: sometimes this sort of thing happens due to too much work, other times due to being dropped.18:31
dkranzboris-42: I just was asking which.18:31
*** jay_ has quit IRC18:31
*** markmcclain has quit IRC18:31
boris-42dkranz heh I will continue work on that stuff as well=)18:31
boris-42dkranz no worries18:31
*** markmcclain has joined #openstack-qa18:31
dkranzboris-42: ok, great. I might make some concrete suggestion as well.18:31
boris-42dkranz good thanks18:31
*** nati_ueno has quit IRC18:31
boris-42dkranz https://github.com/pboris/logaas btw maybe will be interesting for you guys project18:32
*** jhenner has quit IRC18:34
*** nati_uen_ has joined #openstack-qa18:34
*** jhenner has joined #openstack-qa18:35
openstackgerritSean Dague proposed a change to openstack/tempest: make logformat match oslo  https://review.openstack.org/9881918:39
*** Obulpath_ has quit IRC18:40
*** dkranz has quit IRC18:42
*** ildikov|2 has joined #openstack-qa18:45
*** afazekas has joined #openstack-qa18:45
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Validate ssh client auth before we use it  https://review.openstack.org/9844618:46
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Add extra debug info to ssh failure  https://review.openstack.org/9850718:46
*** Obulpathi has joined #openstack-qa18:47
*** Obulpath_ has joined #openstack-qa18:48
openstackgerritRob Crittenden proposed a change to openstack-dev/devstack: Configure endpoints to use SSL natively or via proxy  https://review.openstack.org/9885418:51
*** Obulpathi has quit IRC18:51
*** weshay has quit IRC18:54
*** ildikov|2 is now known as ildikov18:54
*** dkranz has joined #openstack-qa18:58
*** markmcclain has quit IRC19:00
*** nati_ueno has joined #openstack-qa19:02
*** Obulpath_ has quit IRC19:04
*** dkranz has quit IRC19:06
*** nati_uen_ has quit IRC19:06
*** markmcclain has joined #openstack-qa19:07
*** weshay has joined #openstack-qa19:09
openstackgerritMorgan Fainberg proposed a change to openstack-dev/devstack: Use apache2ctl after apache is installed.  https://review.openstack.org/9885919:10
openstackrecheckConsole logs not available after 13:23s for check-tempest-dsvm-full 91727,3,0f0e5bf19:11
*** nati_ueno has quit IRC19:15
*** Obulpathi has joined #openstack-qa19:16
*** markmcclain1 has joined #openstack-qa19:19
*** dkranz has joined #openstack-qa19:19
*** markmcclain1 has quit IRC19:20
*** markmcclain1 has joined #openstack-qa19:20
*** markmcclain has quit IRC19:23
openstackrecheckConsole logs not available after 14:37s for check-tempest-dsvm-postgres-full 90143,7,248d50d19:31
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Bump hacking to 0.9.x series  https://review.openstack.org/9886819:41
sdaguemtreinish: I actually think on nova api v3 we should set the default in tempest to false19:42
sdaguebecause it's experimental19:42
sdaguenot via devstack19:42
*** markmcclain1 has quit IRC19:42
mtreinishsdague: ok, that's fine. It was just a patch to test it anyway19:43
sdagueyep19:43
sdagueI just did a recheck, now that it has a chance19:43
mtreinishhonestly it was hard to see much of a time savings without the v3 tests though19:43
openstackrecheckConsole logs not available after 13:23s for check-grenade-dsvm 97814,1,1aeca5019:44
*** mmaglana_ has joined #openstack-qa19:50
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: Revert "Set CEILOMETER_PIPELINE_INTERVAL to 15"  https://review.openstack.org/9887119:52
*** mmaglana has quit IRC19:52
*** marun has joined #openstack-qa19:55
*** mmaglana has joined #openstack-qa19:55
*** mmaglana_ has quit IRC19:58
openstackrecheckConsole logs not available after 13:33s for check-grenade-dsvm 97944,1,631847819:58
*** morganfainberg is now known as morganfainberg_Z20:06
*** boris-42 has quit IRC20:08
*** boris-42 has joined #openstack-qa20:11
openstackrecheckConsole logs not available after 13:24s for check-grenade-dsvm 97013,1,831831e20:12
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Bump hacking to 0.9.x series  https://review.openstack.org/9886820:18
mtreinishjogo: ^^^ man the new hacking killed tempest...20:19
mtreinishit needed 17 ignores20:19
mriedemmtreinish: where are the .failure files stored for grenade?20:26
openstackrecheckConsole logs not available after 14:27s for check-grenade-dsvm-partial-ncpu-icehouse 94130,7,28589e220:27
mtreinishmriedem: on the log server or in general?20:28
openstackgerritA change was merged to openstack-dev/devstack: Update setuptools to latest for .dist-info support  https://review.openstack.org/9826320:28
mriedemmtreinish: like here http://logs.openstack.org/79/96679/3/check/check-grenade-dsvm-partial-ncpu/7bc0290/logs/20:28
mriedemi've been seeing grenade failing since last week a lot with lots of services failing to start20:28
mriedembut not sure why they aren't starting20:28
mtreinishmriedem: it's supposed to make a status dir and put the failure files in there20:28
mriedemthat must be busted...20:29
mtreinishmriedem: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n49620:29
*** Obulpathi has quit IRC20:29
*** kgriffs is now known as kgriffs|afk20:29
*** Obulpathi has joined #openstack-qa20:30
mtreinishmriedem: I probably screwed something up20:31
mriedemmtreinish: looking20:33
mriedemmtreinish: methinks you did20:33
mriedemsudo rm $BASE/logs/grenade.sh.log.*20:33
*** kgriffs|afk is now known as kgriffs20:33
mriedemgrenade.sh log /opt/stack/logs/grenade.sh.log.2014-06-09-16375020:33
mriedemtherefore BASE=/opt/stack20:33
mriedemyet: if [ $(ls $BASE/stack/status/*.failure | wc -l) -gt 0 ]; then20:33
afazekassdague: https://review.openstack.org/#/c/98212/20:33
*** mwagner_lap has quit IRC20:34
*** markmcclain has joined #openstack-qa20:34
mriedemactually: tee "/opt/stack/status/stack/horizon.failure"20:35
mriedemso should be looking in: $BASE/stack/status/stack/*.failure20:36
mriedemi can push a patch20:36
mtreinishmriedem: why the extra 'stack', shouldn't it be without the stack?20:37
mtreinishor am I misreading?20:37
mriedemhttp://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n9120:38
mriedemright?20:38
mriedemmtreinish: http://logs.openstack.org/79/96679/3/check/check-grenade-dsvm-partial-ncpu/7bc0290/logs/devstack-gate-cleanup-host.txt20:39
mriedem2014-06-09 17:00:25.848 | ++ ls '/opt/stack/stack/status/*.failure' 2014-06-09 17:00:25.849 | ls: cannot access /opt/stack/stack/status/*.failure: No such file or directory20:39
mriedemme fixy20:39
*** nati_ueno has joined #openstack-qa20:40
openstackrecheckConsole logs not available after 13:24s for check-grenade-dsvm 92479,2,75da9a620:40
mtreinishmriedem: oh I missed the extra stack in the tee command20:40
mriedemnp, fixing20:40
mriedemit's even right 2 lines down20:41
mriedembut now in the ls command20:41
mriedem*not20:41
mriedemsudo cp $BASE/status/stack/*.failure $BASE/logs/status/20:41
*** jhenner has quit IRC20:41
mtreinishmeh, details20:41
openstackgerritMatt Riedemann proposed a change to openstack-infra/devstack-gate: Fix collection of *.failure files  https://review.openstack.org/9888520:43
mriedemmtreinish: sdague: ^20:43
*** Sukhdev has joined #openstack-qa20:49
mtreinishmriedem: can you rebase your network connectivity tempest patches?20:51
mtreinishI think it'll have trouble getting past check unless it's on top of the revert from this morning20:51
mriedemmtreinish: this revert? https://review.openstack.org/#/c/98632/20:51
mtreinishyeah the rebase button should work fine in this case20:52
mriedemhmmm20:52
mriedemi don't know if i trust that20:52
mtreinishit worked for me earlier today :)20:53
mriedemok well if all of my scores are gone i'm going to hunt you down20:53
openstackgerritMatt Riedemann proposed a change to openstack/tempest: Consolidate _check_tenant_network_connectivity in NetworkScenarioTest  https://review.openstack.org/9584420:54
openstackgerritMatt Riedemann proposed a change to openstack/tempest: Consolidate _check_public_network_connectivity into NetworkScenarioTest  https://review.openstack.org/9583120:54
mtreinishI'll just push it through20:54
mriedemthanks20:54
openstackrecheckConsole logs not available after 13:52s for gate-keystone-pep8 42967,5,36b3dba20:54
mtreinishmriedem: why did the scores disappear from the first one but not the second?20:55
mriedemmtreinish: idk20:55
mriedemrebase black magic20:55
mriedemi just rebased locally, no changes20:55
mriedemmtreinish: oh i see20:56
mriedemcommit message on the first must have been edited in gerrit20:56
mriedemwas a nit20:56
mriedemhttps://review.openstack.org/#/c/95831/4..5//COMMIT_MSG20:56
mtreinishok, that explains it20:56
*** jlibosva has joined #openstack-qa21:00
*** jlibosva has quit IRC21:00
*** jlibosva has joined #openstack-qa21:01
*** giulivo has quit IRC21:03
jogomtreinish: thats a lot of skips :)21:04
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Enable E113,E111,H302,F812,E713 rules  https://review.openstack.org/9889921:04
jogomtreinish: I should have mentioned that hacking isn't in our pypi mirror yet http://pypi.openstack.org/openstack/hacking/21:04
mtreinishjogo: that's fine I'll just recheck when it is21:05
jogo98824,121:06
jogois the patch21:06
openstackrecheckConsole logs not available after 13:33s for check-grenade-dsvm 84645,5,d2901ff21:08
*** morganfainberg_Z is now known as morganfainberg21:10
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Enable H402,E251,E265 rules  https://review.openstack.org/9890321:16
*** mdenny has quit IRC21:17
*** SridharG has quit IRC21:18
*** flwang has quit IRC21:18
openstackrecheckConsole logs not available after 13:40s for gate-nova-python26 98201,2,0d7db4e21:21
*** mdenny has joined #openstack-qa21:26
*** kgriffs is now known as kgriffs|afk21:28
*** ekhugen has quit IRC21:30
openstackgerritAdam Gandelman proposed a change to openstack-infra/elastic-recheck: Add query for Ironic bug 1328274  https://review.openstack.org/9890521:32
*** Obulpathi has quit IRC21:33
*** dkranz has quit IRC21:34
openstackrecheckConsole logs not available after 13:29s for check-grenade-dsvm 98021,1,e7d8a3621:35
openstackgerritmeera-belur proposed a change to openstack/tempest: Adding tests for the barbican - key management service  https://review.openstack.org/9884221:36
*** cbader has quit IRC21:37
*** bauzas has quit IRC21:37
*** mriedem has quit IRC21:45
*** mrodden has quit IRC21:46
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-postgres-full 98098,2,bab195421:50
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Enable E129,E128 rules  https://review.openstack.org/9890921:52
*** kgriffs|afk has quit IRC21:53
*** malini_afk has quit IRC21:53
jogomtreinish: E25121:56
*** andreaf has joined #openstack-qa21:57
jogohttps://github.com/jcrocholl/pep8/issues/30121:57
*** jlibosva has quit IRC21:59
mtreinishjogo: I think I noticed something weird about that one, but I think I worked around it: https://review.openstack.org/#/c/9890321:59
mtreinishbut, I can't remember which changes go with which rule anymore21:59
*** dims_ has joined #openstack-qa22:00
jogoahh I see what you did, you changed the indent format22:01
jogo'22:01
jogohttps://review.openstack.org/#/c/98903/1/tempest/api/volume/test_volumes_get.py22:01
*** catohornet1 has joined #openstack-qa22:01
*** markmcclain has quit IRC22:01
*** adalbas has quit IRC22:02
*** dims has quit IRC22:02
sdagueso really? hacking changes :P22:05
* jogo hides22:06
sdaguewith the current state of the failures, hacking changes seems pretty gratuitous22:07
*** rdekel has joined #openstack-qa22:08
mtreinishsdague: I was pretty burnt out today, but this was something to spin my extra cycles on without much load22:08
*** nati_ueno has quit IRC22:08
jogosdague: agreed, but I have been holding off doing this for a few months22:08
openstackgerritA change was merged to openstack-infra/devstack-gate: Fix collection of *.failure files  https://review.openstack.org/9888522:08
jogomtreinish: hehe22:09
sdagueso, I'll mildy accept it from mtreinish as he was firedrilling last week as well22:09
sdaguebut o_O to jogo :P22:09
jogohttp://logs.openstack.org/98/97598/1/check/check-grenade-dsvm-partial-ncpu/2125f27/console.html#_2014-06-09_19_16_59_23122:09
jogois that a new failure mode?22:10
openstackrecheckConsole logs not available after 13:23s for check-tempest-dsvm-full 98819,2,6e4b65922:10
*** salv-orlando has quit IRC22:10
sdaguemtreinish: tomorrow, I think we should work on an etherpad to sort out the host validitor bits, then see if we can split it up in a way to tag team22:10
mtreinishsdague: ok, makes sense. I have bits and pieces that I started today22:11
*** malini_afk has joined #openstack-qa22:11
*** malini_afk is now known as malini22:11
*** rdekel has quit IRC22:12
*** kgriffs has joined #openstack-qa22:12
sdaguemtreinish: yeh, it just seems like it's probably a bunch of work. So if we can tag team it, it might be more energizing than either one of us just getting bogged down in it22:13
*** tosky has quit IRC22:13
sdaguejogo: honestly, I'm not looking at any more failures today, because otherwise I'll get stabby22:13
*** salv-orlando has joined #openstack-qa22:14
*** salv-orlando has quit IRC22:15
jogohaha22:15
jogosdague: maybe you should do some hacking patches to calm your nerves like mtreinish22:16
*** salv-orlando has joined #openstack-qa22:16
*** salv-orlando has quit IRC22:17
openstackgerritSean Dague proposed a change to openstack/tempest: make logformat match oslo  https://review.openstack.org/9881922:17
mtreinishwell I'm almost through all 17 ignores :)22:17
mtreinishbut I guess the other projects are fair game22:17
*** eglynn has joined #openstack-qa22:17
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint fur bug 1268725  https://review.openstack.org/9891322:19
*** salv-orlando has joined #openstack-qa22:19
mtreinishjogo: quick question how does H305 handle different python versions? Like if something was made stdlib in py3 but was third party in py222:19
jogooh guh22:19
jogonoqa?22:20
jogomtreinish: I don't think we can do anything else about that22:20
mtreinishhahaha, ok22:20
jogoin py2 mode we don't know about p3 and vice versa22:20
mtreinishbecause I just realized I'm running it with py3 on my box22:20
*** salv-orlando has quit IRC22:20
mtreinishand I think there are a few of those22:20
sdagueyeh, I'd just say ignore the rule then22:21
sdagueit seems like it will generate more work22:21
eglynn... now I know it's bad form to just hop on the channel, agitating for reviewer love to be bestowed upon one's favourite patches22:21
sdagueand not really save us anything22:21
eglynn... but if any of the grenade cores are around and have a minute to take another pass on the ceilo grenade patch22:21
eglynn... that would be most appreciated! :)22:22
eglynnhttps://review.openstack.org/9446822:22
mtreinisheglynn: well there are only 2 grenade cores right now sdague and dtroyer :)22:22
*** salv-orlando has joined #openstack-qa22:22
*** salv-orlando has quit IRC22:23
eglynnmtreinish: a-ha, ok, thanks for the heads-up22:23
eglynnmtreinish: ... since I've intruded already, a quick question on the QA/Infra meetup?22:24
eglynnmtreinish: ... are you still planning a ceilometer-specific sub-sprint?22:24
*** mwagner_lap has joined #openstack-qa22:24
openstackrecheckConsole logs not available after 13:25s for check-tempest-dsvm-full 97569,3,c3e850e22:24
* eglynn asks as internal travel-budget planning is afoot ...22:25
*** salv-orlando has joined #openstack-qa22:25
mtreinisheglynn: we can, it really depends if ceilo people want to show up22:26
eglynnmtreinish: OK, so it *may* be possible for me to attend22:27
eglynnmtreinish: ... and of course I would welcome such concentrated attention on ceilo testing from the QA ninjas22:28
eglynnmtreinish: ... but would depend on whether other Red Hatters are gonna be covered the mid-cycle already22:28
*** weshay has quit IRC22:29
eglynns/covered/covering/22:30
eglynnmtreinish: ... but can I take it that if there's ceilo folk in attendance, then there's space on the agenda for such a "sub-sprint"?22:31
mtreinisheglynn: well the agenda for the 3 discussion/code sprint days is up in the air at this point, and the attendance list is pretty light at this point22:31
mtreinishso we can play it by ear for now22:32
mtreinishis there anyone else who's working on ceilo testing who you think should come if we have a day for it?22:32
eglynn... one of the Mirantis folks may make sense also22:34
eglynn... I'll it run them in their TZ tmrw22:35
eglynn... v. late now in MSK22:35
eglynnmtreinish: ... I'll get back to you to confirm, thanks!22:35
eglynn... run it *by them22:36
* eglynn loses the ability to type coherently, blames the lateness of the hour22:37
mtreinisheglynn: ok cool22:40
mtreinishjogo: I really like H904, but I am going to hate fixing it in tempest...22:40
openstackrecheckConsole logs not available after 13:25s for check-grenade-dsvm 97728,7,befb02722:41
jogohaha yeah22:41
*** dustins1 has quit IRC22:42
mtreinishboth that and H405 generate more lines than can fit in my terminal buffer...22:43
*** hemna is now known as hemna_22:45
*** jgrimm has quit IRC22:45
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Enable H407,H305,E123,H307,E122 rules  https://review.openstack.org/9891822:45
*** eglynn has quit IRC22:46
*** nati_ueno has joined #openstack-qa22:47
*** jamielennox|away is now known as jamielennox22:47
*** Sukhdev has quit IRC22:49
*** aysyd has quit IRC22:51
openstackrecheckConsole logs not available after 13:23s for check-tempest-dsvm-full-icehouse 90462,9,c88011c22:54
*** locke105 has quit IRC22:56
*** rbak has quit IRC22:58
openstackrecheckConsole logs not available after 13:28s for check-tempest-dsvm-neutron 98779,1,bed811723:07
*** kgriffs is now known as kgriffs|afk23:13
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Remove duplicate query for 1326820  https://review.openstack.org/9891923:13
*** locke105 has joined #openstack-qa23:19
openstackrecheckConsole logs not available after 13:39s for check-tempest-dsvm-postgres-full 98546,1,4a0e95223:22
*** nati_ueno has quit IRC23:22
*** atiwari has quit IRC23:24
*** morganfainberg is now known as morganfainberg_Z23:25
*** nati_ueno has joined #openstack-qa23:28
*** nati_uen_ has joined #openstack-qa23:31
*** nati_ueno has quit IRC23:32
openstackrecheckConsole logs not available after 13:57s for check-tempest-dsvm-full 97059,6,b5d6f2e23:36
*** aimon_ has quit IRC23:39
*** morganfainberg_Z is now known as morganfainberg23:42
*** masayukig_ has quit IRC23:43
*** masayukig has joined #openstack-qa23:43
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 and 1326855  https://review.openstack.org/9891923:48
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-neutron 95666,4,1bd716723:49
*** catohornet1 has quit IRC23:53
*** aimon has joined #openstack-qa23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!