*** yamahata has quit IRC | 00:04 | |
openstackrecheck | openstack/python-saharaclient change: https://review.openstack.org/97255 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1298472 | 00:12 |
---|---|---|
*** markmcclain1 has quit IRC | 00:14 | |
*** matsuhashi has joined #openstack-qa | 00:21 | |
*** bearhands is now known as comstud | 00:29 | |
*** dmorita has joined #openstack-qa | 00:29 | |
*** zhikunliu has joined #openstack-qa | 00:31 | |
mriedem | mtreinish: sdague: flashgordon: doing a 5th recheck on this now https://review.openstack.org/#/c/98632/ - so far in 4 runs it hasn't hit the nova-network ssh timeout failure | 00:36 |
mriedem | not ideal to revert that but if it gets things moving we might consider it | 00:36 |
*** nati_ueno has quit IRC | 00:53 | |
oomichi | mriedem: nice investigation | 01:01 |
oomichi | mriedem: I guess server creation of each test would cause the heavy load and ssh setup could not make it before ssh connecting, right? | 01:01 |
mriedem | oomichi: that's something we were guessing on friday morning, | 01:03 |
oomichi | mriedem: by reverting the patch, we can reduce the server creations, and tempest can try connecting the same server which is already set up at the first setUp. | 01:03 |
mriedem | the problem started on 6/5 | 01:03 |
mriedem | which is when that change merged | 01:03 |
mriedem | there wasn't much else that merged in nova around that time | 01:04 |
oomichi | mriedem: nice catch:) | 01:04 |
mriedem | i think we wanted to try and fix the actual ssh issue and keep the change to use independent instances, | 01:04 |
mriedem | but unless we have a fix in the next day or so i'd think we'd want to get the gate flowing again - anyway, just wanted to see if that revert hit the bug | 01:04 |
mriedem | so far it hasn't | 01:04 |
mtreinish | mriedem: yeah it's really bizarre to me though why that would be causing issues | 01:05 |
mtreinish | it doesn't add that many server creates | 01:05 |
mriedem | mtreinish: how many are we talking? 3? | 01:05 |
mriedem | oh one per test case | 01:06 |
mriedem | since it doesn't reuse them | 01:06 |
mtreinish | yeah 1 per test | 01:06 |
mtreinish | which is actually more than I thought | 01:06 |
mriedem | test cases run in serial too right? | 01:07 |
mtreinish | they do per class | 01:07 |
mtreinish | but there is xml and json | 01:07 |
mtreinish | and v3 | 01:07 |
mriedem | so create a test instance for each test case that runs and that instance should be deleted for each test case | 01:07 |
mriedem | ok so 3 versions of that test class running | 01:07 |
mriedem | they were sharing a common instance before | 01:07 |
mriedem | assuming it was going to active and not rebuilt | 01:08 |
mriedem | anyway, i don't like the revert, just wanted to see if it hit the bug so we know it's an option | 01:08 |
oomichi | A server status becomes "active", but Tempest cannot connect the server with ssh connecting. | 01:13 |
oomichi | mtreinish: how about https://review.openstack.org/#/c/97812 for debugging? | 01:13 |
*** nati_ueno has joined #openstack-qa | 01:19 | |
*** nati_ueno has quit IRC | 01:20 | |
*** yamahata has joined #openstack-qa | 01:23 | |
openstackrecheck | openstack/horizon change: https://review.openstack.org/94579 failed because of: gate-grenade-dsvm: unrecognized error, gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1320617 | 01:34 |
*** amotoki has joined #openstack-qa | 01:36 | |
*** nosnos has joined #openstack-qa | 01:43 | |
openstackrecheck | openstack/nova change: https://review.openstack.org/98559 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1298472 https://bugs.launchpad.net/bugs/1285908, gate-grenade-dsvm-partial-ncpu: https://bugs.launchpad.net/bugs/1273908, gate-tempest-dsvm-full: unrecognized error | 01:49 |
*** GMann has joined #openstack-qa | 01:51 | |
*** catohornet has joined #openstack-qa | 01:54 | |
*** yamahata has quit IRC | 01:54 | |
*** yamahata has joined #openstack-qa | 01:54 | |
*** mriedem has quit IRC | 02:03 | |
openstackgerrit | Ghanshyam Mann proposed a change to openstack/tempest: Read template's expected value orchest test Part-1 https://review.openstack.org/98676 | 02:13 |
openstackrecheck | openstack/taskflow change: https://review.openstack.org/98031 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1285908, gate-tempest-dsvm-full: unrecognized error | 02:21 |
*** catohornet has quit IRC | 02:23 | |
*** mgagne1 is now known as mgagne | 02:30 | |
*** lbragstad has joined #openstack-qa | 03:07 | |
*** Abhijeet has joined #openstack-qa | 03:08 | |
*** nosnos has quit IRC | 03:18 | |
*** Madan has joined #openstack-qa | 03:28 | |
*** Poornima has joined #openstack-qa | 03:32 | |
openstackrecheck | openstack/nova change: https://review.openstack.org/79863 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1285908 | 03:46 |
*** GMann has quit IRC | 03:56 | |
openstackgerrit | Anthony Woods proposed a change to openstack-dev/devstack: Adds neutron linuxbridge-agent support to ironic deployment. https://review.openstack.org/98317 | 04:00 |
openstackgerrit | Ghanshyam Mann proposed a change to openstack/tempest: Read template's expected value orchest test Part-1 https://review.openstack.org/98676 | 04:01 |
*** matsuhashi has quit IRC | 04:04 | |
*** nosnos has joined #openstack-qa | 04:06 | |
*** Madan has quit IRC | 04:08 | |
*** Madan has joined #openstack-qa | 04:11 | |
*** matsuhashi has joined #openstack-qa | 04:13 | |
*** yfried has quit IRC | 04:15 | |
*** k4n0 has joined #openstack-qa | 04:16 | |
*** SridharG has joined #openstack-qa | 04:29 | |
openstackrecheck | openstack/heat change: https://review.openstack.org/95178 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1285908 | 04:30 |
*** SridharG has quit IRC | 04:33 | |
*** SridharG has joined #openstack-qa | 04:35 | |
*** k4n0 has quit IRC | 04:39 | |
*** k4n0 has joined #openstack-qa | 04:44 | |
*** Madan has quit IRC | 04:47 | |
*** yfried has joined #openstack-qa | 04:48 | |
*** SridharG has quit IRC | 04:49 | |
*** Madan has joined #openstack-qa | 04:49 | |
*** rdekel has joined #openstack-qa | 04:49 | |
*** psharma has joined #openstack-qa | 04:53 | |
*** SridharG has joined #openstack-qa | 04:59 | |
*** lbragstad has quit IRC | 05:01 | |
*** Longgeek has joined #openstack-qa | 05:04 | |
*** che-arne has joined #openstack-qa | 05:36 | |
*** flashgordon is now known as jogo | 05:46 | |
*** debu__ has joined #openstack-qa | 05:53 | |
*** raies has joined #openstack-qa | 06:00 | |
raies | hi | 06:01 |
raies | clarkb: hi | 06:07 |
openstackrecheck | openstack/sahara change: https://review.openstack.org/97451 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1285908 https://bugs.launchpad.net/bugs/1254872 | 06:17 |
openstackgerrit | Ryan Hsu proposed a change to openstack/tempest: Handle backup not found in test_create_backup https://review.openstack.org/98693 | 06:21 |
raies | cyeoh: hi | 06:24 |
raies | cyeoh: to check if cell extension is enabled or not, I have used test.is_extension_enabled("cell", 'compute') , when I do not enable n-cell in localrc then in spite of skip code lead to error showing amessage that Celss is not enabled | 06:29 |
raies | what correct value should here -- test.is_extension_enabled("cell", 'compute') | 06:29 |
raies | cyeoh: ^^ | 06:29 |
*** coolsvap|afk is now known as coolsvap | 06:37 | |
*** debu__ has quit IRC | 06:39 | |
*** [1]evgenyf has joined #openstack-qa | 06:42 | |
*** SridharG has quit IRC | 06:43 | |
*** SridharG has joined #openstack-qa | 06:43 | |
*** Longgeek has quit IRC | 06:46 | |
*** bkopilov has joined #openstack-qa | 06:49 | |
*** yrabl has joined #openstack-qa | 06:49 | |
*** jlibosva has joined #openstack-qa | 06:56 | |
*** jpich has joined #openstack-qa | 06:56 | |
*** salv-orlando has joined #openstack-qa | 06:57 | |
*** nosnos has quit IRC | 06:57 | |
*** salv-orlando has quit IRC | 06:59 | |
*** che-arne_afk has joined #openstack-qa | 06:59 | |
*** salv-orlando has joined #openstack-qa | 07:00 | |
*** salv-orlando has quit IRC | 07:00 | |
*** raies has quit IRC | 07:01 | |
*** raies has joined #openstack-qa | 07:02 | |
raies | cyeoh: you there ?? | 07:02 |
*** GMann has joined #openstack-qa | 07:02 | |
*** che-arne has quit IRC | 07:03 | |
openstackgerrit | Andrey Pavlov proposed a change to openstack/tempest: add block_device_mapping to create_server params for nova clients https://review.openstack.org/98699 | 07:06 |
*** zhikunliu has quit IRC | 07:13 | |
*** Longgeek has joined #openstack-qa | 07:16 | |
openstackrecheck | openstack/cliff change: https://review.openstack.org/97405 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1298472 | 07:20 |
*** Longgeek has quit IRC | 07:22 | |
*** jhenner has quit IRC | 07:22 | |
*** qba73 has joined #openstack-qa | 07:26 | |
raies | mkoderer: hi | 07:27 |
*** che-arne has joined #openstack-qa | 07:30 | |
*** che-arne_afk has quit IRC | 07:31 | |
*** GMann has quit IRC | 07:34 | |
*** jordanP has joined #openstack-qa | 07:45 | |
openstackgerrit | Andrey Pavlov proposed a change to openstack/tempest: fix usage of allow_tenant_isolation https://review.openstack.org/97139 | 07:50 |
openstackgerrit | Andrey Pavlov proposed a change to openstack/tempest: Add 'Member' role to all created users https://review.openstack.org/96460 | 07:50 |
*** che-arne_afk has joined #openstack-qa | 07:52 | |
*** che-arne_afk_ has joined #openstack-qa | 07:56 | |
*** che-arne has quit IRC | 07:56 | |
*** che-arne_afk has quit IRC | 07:57 | |
*** jhenner has joined #openstack-qa | 08:01 | |
*** openstackgerrit has quit IRC | 08:02 | |
*** vrovachev has joined #openstack-qa | 08:03 | |
*** GMann has joined #openstack-qa | 08:03 | |
*** masayukig has quit IRC | 08:05 | |
*** masayukig_ has joined #openstack-qa | 08:06 | |
*** openstackgerrit has joined #openstack-qa | 08:07 | |
*** plomakin has quit IRC | 08:08 | |
*** anteaya has quit IRC | 08:10 | |
*** ewindisch has quit IRC | 08:10 | |
*** _ewindisch has joined #openstack-qa | 08:11 | |
*** maurosr has quit IRC | 08:11 | |
*** _ewindisch is now known as ewindisch | 08:11 | |
openstackrecheck | openstack/taskflow change: https://review.openstack.org/97964 failed gate-grenade-dsvm in the gate queue with an unrecognized error | 08:13 |
*** plomakin has joined #openstack-qa | 08:13 | |
*** maurosr has joined #openstack-qa | 08:14 | |
*** andreaf has joined #openstack-qa | 08:16 | |
*** smurashov has quit IRC | 08:20 | |
*** smurashov has joined #openstack-qa | 08:21 | |
*** tristanC has quit IRC | 08:23 | |
*** tristanC has joined #openstack-qa | 08:24 | |
*** coolsvap has quit IRC | 08:31 | |
*** coolsvap has joined #openstack-qa | 08:32 | |
openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/elastic-recheck: Match #1298472 fingerprint against console tag https://review.openstack.org/98716 | 08:38 |
*** yamahata has quit IRC | 08:41 | |
*** Poornima has quit IRC | 08:44 | |
*** gszasz has joined #openstack-qa | 08:45 | |
*** tkammer_ has joined #openstack-qa | 08:49 | |
*** Abhijeet has quit IRC | 08:50 | |
*** coolsvap is now known as coolsvap|afk | 08:50 | |
*** GMann has quit IRC | 08:51 | |
*** GMann has joined #openstack-qa | 08:51 | |
*** rdekel has quit IRC | 08:52 | |
*** rdekel has joined #openstack-qa | 08:52 | |
*** GMann has quit IRC | 08:53 | |
*** che-arne_afk_ is now known as che-arne | 08:55 | |
*** mkollaro has joined #openstack-qa | 08:59 | |
*** catohornet has joined #openstack-qa | 09:00 | |
openstackrecheck | openstack/ceilometer change: https://review.openstack.org/95666 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1298472 | 09:00 |
*** coolsvap|afk is now known as coolsvap | 09:01 | |
openstackgerrit | Anthony Woods proposed a change to openstack-dev/devstack: Adds neutron linuxbridge-agent support to ironic deployment. https://review.openstack.org/98317 | 09:03 |
*** Longgeek has joined #openstack-qa | 09:07 | |
*** che-arne has quit IRC | 09:09 | |
*** Longgeek has quit IRC | 09:12 | |
*** che-arne has joined #openstack-qa | 09:12 | |
*** salv-orlando has joined #openstack-qa | 09:18 | |
*** salv-orlando has quit IRC | 09:18 | |
*** roeyc has joined #openstack-qa | 09:19 | |
*** [1]evgenyf has quit IRC | 09:21 | |
*** salv-orlando has joined #openstack-qa | 09:27 | |
*** dosaboy_ is now known as disaboy | 09:29 | |
*** disaboy is now known as dosaboy | 09:30 | |
*** matsuhashi has quit IRC | 09:31 | |
*** matsuhashi has joined #openstack-qa | 09:31 | |
*** salv-orlando has quit IRC | 09:32 | |
openstackgerrit | Mh Raies proposed a change to openstack/tempest: Compute "cells" basic API test https://review.openstack.org/90337 | 09:33 |
openstackgerrit | Pranali Deore proposed a change to openstack/tempest: Added test case for nova keypair https://review.openstack.org/79549 | 09:34 |
*** [1]evgenyf has joined #openstack-qa | 09:37 | |
openstackgerrit | Vadim Rovachev proposed a change to openstack/tempest: Added cinder and neutron notifications tests for ceilometer https://review.openstack.org/67164 | 09:37 |
*** matsuhashi has quit IRC | 09:39 | |
*** matsuhashi has joined #openstack-qa | 09:40 | |
*** tosky has joined #openstack-qa | 09:41 | |
*** catohornet has quit IRC | 09:42 | |
*** fbo_away is now known as fbo | 09:44 | |
openstackgerrit | Vadim Rovachev proposed a change to openstack/tempest: Added cinder and neutron notifications tests for ceilometer https://review.openstack.org/67164 | 09:54 |
openstackrecheck | openstack/ceilometer change: https://review.openstack.org/96819 failed because of: gate-ceilometer-python27: https://bugs.launchpad.net/bugs/1326820 | 09:56 |
*** che-arne has quit IRC | 10:02 | |
*** che-arne has joined #openstack-qa | 10:03 | |
*** che-arne_afk has joined #openstack-qa | 10:06 | |
*** che-arne has quit IRC | 10:09 | |
*** anteaya has joined #openstack-qa | 10:10 | |
*** SridharGaddam has joined #openstack-qa | 10:17 | |
*** coolsvap is now known as coolsvap|afk | 10:17 | |
*** SridharG has quit IRC | 10:18 | |
*** coolsvap|afk is now known as coolsvap | 10:25 | |
*** dmorita has quit IRC | 10:28 | |
*** qba73 has quit IRC | 10:30 | |
*** Longgeek has joined #openstack-qa | 10:31 | |
openstackgerrit | Mh Raies proposed a change to openstack/tempest: Missing Nova CLI tests https://review.openstack.org/98735 | 10:33 |
*** che-arne has joined #openstack-qa | 10:40 | |
*** malini_afk is now known as malini | 10:40 | |
*** malini has left #openstack-qa | 10:41 | |
*** che-arne_afk has quit IRC | 10:43 | |
*** rdekel has quit IRC | 10:47 | |
openstackgerrit | Mh Raies proposed a change to openstack/tempest: Missing Nova CLI tests https://review.openstack.org/98735 | 10:47 |
*** matsuhashi has quit IRC | 10:51 | |
openstackrecheck | openstack/glance change: https://review.openstack.org/92739 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1298472 https://bugs.launchpad.net/bugs/1285908 | 10:53 |
openstackgerrit | Vadim Rovachev proposed a change to openstack/tempest: Add glance notifications tests for ceilometer https://review.openstack.org/71259 | 10:53 |
openstackgerrit | Anthony Woods proposed a change to openstack-dev/devstack: Adds neutron linuxbridge-agent support to ironic deployment. https://review.openstack.org/98317 | 10:54 |
*** che-arne_afk has joined #openstack-qa | 10:55 | |
*** matsuhashi has joined #openstack-qa | 10:58 | |
*** che-arne has quit IRC | 10:58 | |
*** dkorn has joined #openstack-qa | 11:13 | |
vrovachev | folks, o/ | 11:16 |
vrovachev | I see several strange issues with devstack+tempest now (at least speaking about ceilometer testing) | 11:16 |
vrovachev | https://review.openstack.org/#/c/98344/ and https://review.openstack.org/#/c/95640/ seem to have one root for failings | 11:17 |
vrovachev | so tempest job and ceilometerclient gate are failing because of one reason | 11:17 |
vrovachev | although tests are different | 11:18 |
vrovachev | in console.log it's seens that "wrong nova credentials" are provided | 11:18 |
vrovachev | is that known issue? | 11:18 |
*** dkorn1 has joined #openstack-qa | 11:18 | |
openstackrecheck | openstack/taskflow change: https://review.openstack.org/98623 failed because of: gate-tempest-dsvm-neutron: https://bugs.launchpad.net/bugs/1254890 | 11:18 |
vrovachev | sdague, mtreinish ^^ | 11:18 |
*** matsuhashi has quit IRC | 11:18 | |
*** dkorn has quit IRC | 11:18 | |
*** matsuhashi has joined #openstack-qa | 11:19 | |
sdague | vrovachev: right now there are a ton of failures in the tree. I'd suggest trying to create an elastic recheck pattern for ones you are seeing if they don't have one yet | 11:20 |
*** che-arne has joined #openstack-qa | 11:23 | |
*** che-arne_afk has quit IRC | 11:23 | |
vrovachev | sdague: okay :) thank so much :) | 11:23 |
*** matsuhashi has quit IRC | 11:24 | |
openstackgerrit | Yair Fried proposed a change to openstack/tempest: Increases debug data in case of ssh key failure https://review.openstack.org/85588 | 11:32 |
openstackrecheck | Console logs not available after 13:27s for check-tempest-dsvm-full 98499,1,6b6784c | 11:34 |
*** che-arne_afk has joined #openstack-qa | 11:35 | |
openstackgerrit | Mh Raies proposed a change to openstack/tempest: Mising CLI test for Keystone https://review.openstack.org/98749 | 11:35 |
*** weshay has joined #openstack-qa | 11:37 | |
*** Madan has quit IRC | 11:37 | |
*** che-arne has quit IRC | 11:37 | |
*** jordanP has quit IRC | 11:44 | |
*** jordanP has joined #openstack-qa | 11:45 | |
openstackgerrit | Vadim Rovachev proposed a change to openstack/tempest: Add glance notifications tests for ceilometer https://review.openstack.org/71259 | 11:46 |
openstackgerrit | Pooja Tiwari proposed a change to openstack/tempest: Added unit test for bug#1325436 https://review.openstack.org/98751 | 11:49 |
*** eezhova has quit IRC | 11:52 | |
*** eezhova has joined #openstack-qa | 11:59 | |
openstackrecheck | Console logs not available after 14:31s for check-tempest-dsvm-full 98470,2,c77f911 | 12:00 |
*** dkranz has joined #openstack-qa | 12:01 | |
*** giulivo has joined #openstack-qa | 12:07 | |
*** mwagner_lap has quit IRC | 12:08 | |
*** adalbas has joined #openstack-qa | 12:11 | |
*** dkorn1 has quit IRC | 12:12 | |
openstackrecheck | Console logs not available after 14:06s for check-tempest-dsvm-full 95823,6,5cdbec2 | 12:15 |
openstackrecheck | openstack/heat change: https://review.openstack.org/95178 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1285908 | 12:16 |
*** che-arne has joined #openstack-qa | 12:19 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update 1275500 to match more broadly https://review.openstack.org/98767 | 12:19 |
*** che-arne_afk has quit IRC | 12:20 | |
*** aysyd has joined #openstack-qa | 12:24 | |
openstackgerrit | Vadim Rovachev proposed a change to openstack/tempest: Add swift notifications tests for ceilometer https://review.openstack.org/70998 | 12:25 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Match #1298472 fingerprint against console tag https://review.openstack.org/98716 | 12:27 |
*** coolsvap is now known as coolsvap|afk | 12:27 | |
*** levi_b has joined #openstack-qa | 12:27 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove 1285908 query https://review.openstack.org/98774 | 12:32 |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update 1275500 to match more broadly https://review.openstack.org/98767 | 12:33 |
*** che-arne_afk has joined #openstack-qa | 12:34 | |
*** early has quit IRC | 12:35 | |
sdague | dkranz you about? | 12:36 |
*** openstackrecheck has quit IRC | 12:37 | |
*** openstackrecheck has joined #openstack-qa | 12:37 | |
*** che-arne has quit IRC | 12:38 | |
*** ildikov_ has quit IRC | 12:38 | |
*** early has joined #openstack-qa | 12:39 | |
*** che-arne has joined #openstack-qa | 12:40 | |
*** SridharGaddam has quit IRC | 12:42 | |
*** che-arne_afk has quit IRC | 12:43 | |
openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove 1285908 query https://review.openstack.org/98774 | 12:47 |
*** levi_b has quit IRC | 12:49 | |
*** annegentle has joined #openstack-qa | 12:49 | |
*** che-arne_afk has joined #openstack-qa | 12:50 | |
*** SridharG has joined #openstack-qa | 12:50 | |
*** jchai has joined #openstack-qa | 12:52 | |
openstackrecheck | Console logs not available after 14:38s for check-tempest-dsvm-full 91554,10,cf9f415 | 12:52 |
*** k4n0 has quit IRC | 12:52 | |
*** che-arne has quit IRC | 12:53 | |
*** openstackrecheck has quit IRC | 12:58 | |
*** openstackrecheck has joined #openstack-qa | 12:58 | |
mtreinish | sdague: did you see mriedem's revert which hasn't hit the ssh timeout in 6 or 7 checks? | 13:13 |
sdague | mtreinish: url? | 13:13 |
mtreinish | https://review.openstack.org/#/c/98632/ | 13:13 |
*** che-arne has joined #openstack-qa | 13:13 | |
*** mriedem has joined #openstack-qa | 13:14 | |
sdague | well, at this point I'm up for anything | 13:15 |
sdague | mriedem: you want to un WIP that? | 13:15 |
andreaf | mtreinish, sdague: i just triggered another recheck - I think even if it doesn't necessarily fix the issue it's worth a try | 13:15 |
sdague | I thought as core reviewers we could do it | 13:15 |
mriedem | did someone say 'up for anything'? | 13:15 |
sdague | https://review.openstack.org/#/c/98632/ | 13:15 |
*** rbak has quit IRC | 13:15 | |
mriedem | done | 13:16 |
openstackrecheck | Console logs not available after 13:41s for check-grenade-dsvm-icehouse 95789,1,fa3ff49 | 13:16 |
sdague | mtreinish: you want to +A it, and we can pop it to the top of gate | 13:17 |
mtreinish | sdague: yep I did | 13:17 |
*** che-arne_afk has quit IRC | 13:17 | |
*** jaypipes has joined #openstack-qa | 13:19 | |
*** ChanServ sets mode: +o jaypipes | 13:19 | |
mtreinish | I do want to figure out what that change broke though, it's not obvious to me | 13:19 |
*** che-arne_afk has joined #openstack-qa | 13:20 | |
*** che-arne has quit IRC | 13:21 | |
sdague | mtreinish: honestly, I think nova-network is racing on itself | 13:21 |
sdague | and the net effect of that change is we build more computes | 13:22 |
sdague | especially more computes at the same time | 13:22 |
sdague | because the Actions classes have a habit of running at aprox the same time | 13:22 |
*** bkopilov has quit IRC | 13:22 | |
*** dustins has joined #openstack-qa | 13:23 | |
sdague | mriedem: I also deleted the fingerprint that was top of the list but was 80% success | 13:23 |
sdague | because it was definitely hiding issues | 13:23 |
mriedem | sdague: the libvirt one? | 13:24 |
sdague | yeh | 13:24 |
mriedem | cool | 13:24 |
mriedem | was wondering about that | 13:24 |
sdague | as far as can tell even in failing runs, the libvirt error wasn't actually related | 13:24 |
sdague | at least in the one I stepped through | 13:24 |
sdague | which had 2 fails | 13:24 |
sdague | one was 1275500 (though the query there was too narrow, so not actually hitting any more) | 13:25 |
sdague | and one was this phantom nova-network just doesn't do the right thing bit | 13:25 |
*** jaypipes has quit IRC | 13:28 | |
*** jaypipes has joined #openstack-qa | 13:29 | |
*** ChanServ sets mode: +o jaypipes | 13:29 | |
openstackrecheck | Console logs not available after 13:28s for check-tempest-dsvm-full 71259,4,340034a | 13:29 |
*** Obulpathi has joined #openstack-qa | 13:31 | |
*** Obulpathi has quit IRC | 13:31 | |
*** malini has joined #openstack-qa | 13:32 | |
*** Obulpathi has joined #openstack-qa | 13:32 | |
*** ekhugen has joined #openstack-qa | 13:33 | |
*** jaypipes has quit IRC | 13:34 | |
*** jaypipes has joined #openstack-qa | 13:35 | |
*** ChanServ sets mode: +o jaypipes | 13:35 | |
*** jaypipes has quit IRC | 13:36 | |
*** jaypipes has joined #openstack-qa | 13:38 | |
*** ChanServ sets mode: +o jaypipes | 13:38 | |
*** vrovachev has quit IRC | 13:39 | |
*** yrabl has quit IRC | 13:39 | |
mriedem | mtreinish: sdague: do we have a grenade bug yet for the case that every service fails to start? | 13:40 |
mriedem | i can't find root cause yet | 13:40 |
mtreinish | mriedem: I thought there was a catch all bug for service fails to start | 13:40 |
*** cbader has joined #openstack-qa | 13:40 | |
mtreinish | we should be capturing the failure files now too | 13:40 |
mriedem | oh nvm i guess it was bug 1298472 and markmc fixed the e-r query for that to catch grenade fails | 13:42 |
mriedem | i was just surprised by the number of 'service failed to start' messages in this log | 13:42 |
mriedem | http://logs.openstack.org/93/98493/1/check/check-grenade-dsvm/bfb5945/console.html.gz | 13:42 |
*** salv-orlando has joined #openstack-qa | 13:42 | |
openstackrecheck | Console logs not available after 13:26s for check-grenade-dsvm 98263,1,1f4f1c9 | 13:43 |
*** che-arne has joined #openstack-qa | 13:44 | |
*** jaypipes has quit IRC | 13:45 | |
*** jaypipes has joined #openstack-qa | 13:46 | |
*** ChanServ sets mode: +o jaypipes | 13:46 | |
*** che-arne_afk has quit IRC | 13:47 | |
*** jaypipes has quit IRC | 13:49 | |
*** jaypipes has joined #openstack-qa | 13:49 | |
*** ChanServ sets mode: +o jaypipes | 13:49 | |
*** Longgeek has quit IRC | 13:55 | |
*** Longgeek has joined #openstack-qa | 13:55 | |
openstackrecheck | Console logs not available after 14:13s for check-grenade-dsvm 75365,31,bf7c184 | 13:57 |
openstackgerrit | Malini Kamalambal proposed a change to openstack/tempest: Add Tests for Message & Claim APIs https://review.openstack.org/94944 | 13:57 |
*** mwagner_lap has joined #openstack-qa | 14:00 | |
*** salv-orlando has quit IRC | 14:01 | |
*** beekneemech is now known as bnemec | 14:02 | |
*** ekhugen has quit IRC | 14:04 | |
*** rwsu has joined #openstack-qa | 14:04 | |
*** rbak has joined #openstack-qa | 14:06 | |
*** jgrimm has joined #openstack-qa | 14:06 | |
*** salv-orlando has joined #openstack-qa | 14:08 | |
*** ildikov has joined #openstack-qa | 14:11 | |
*** dustins1 has joined #openstack-qa | 14:13 | |
*** dustins has quit IRC | 14:16 | |
openstackrecheck | Console logs not available after 13:36s for check-tempest-dsvm-full 98721,3,81be611 | 14:18 |
*** ekhugen has joined #openstack-qa | 14:19 | |
*** mlavalle has joined #openstack-qa | 14:22 | |
*** rbak has quit IRC | 14:25 | |
openstackrecheck | openstack/python-keystoneclient change: https://review.openstack.org/98534 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1298472 | 14:26 |
*** SlickN1k has joined #openstack-qa | 14:27 | |
*** rbak has joined #openstack-qa | 14:27 | |
*** kgriffs|afk is now known as kgriffs | 14:28 | |
sdague | mriedem: the actions patch did pass through check and gate when it landed. So it's really suspicious to me that we got that lucky and that's the real issue. | 14:36 |
sdague | but like I said in email and on channel, nova-network is a blackhole | 14:36 |
sdague | so we really have no idea what's going on | 14:36 |
*** dims has joined #openstack-qa | 14:36 | |
mriedem | sdague: did we still have some outstanding debug patches to land to help diagnose issues? | 14:37 |
mriedem | heard something about floating IPs maybe? | 14:37 |
sdague | there is some stuff tempest side | 14:37 |
mriedem | were there any infra network changes around 6/5? | 14:37 |
mtreinish | mriedem: they were on the tempest side, and they didn't show anything | 14:37 |
mriedem | that could impact how it serves IPs? | 14:37 |
sdague | but honestly, I don't think they are going to help | 14:37 |
sdague | mriedem: well we're on HP cloud 1.1 now | 14:38 |
sdague | so ... maybe | 14:38 |
mriedem | when did that move happen? | 14:38 |
sdague | last week | 14:38 |
mriedem | before 6/5? | 14:38 |
sdague | that was why everything was on fire mon / tues | 14:38 |
sdague | yes | 14:38 |
mriedem | ok | 14:38 |
mriedem | and that's using neutron? | 14:38 |
sdague | yes | 14:38 |
mriedem | black holes.... | 14:38 |
mriedem | indeed | 14:38 |
sdague | but the point being, go look at an n-net log | 14:38 |
sdague | any one | 14:39 |
sdague | it's all at debug, and pretty haphazard | 14:39 |
*** salv-orlando has quit IRC | 14:39 | |
mriedem | do we need some more trace in n-net logs? | 14:40 |
openstackrecheck | Console logs not available after 13:32s for check-tempest-dsvm-postgres-full 98715,2,e855dd0 | 14:40 |
* mriedem is reading the email now | 14:40 | |
mriedem | ok, so more trace logging needed in n-net | 14:41 |
sdague | mriedem: well, honestly less trace logging | 14:41 |
sdague | more work unit logging | 14:42 |
sdague | as well as actually checking the work that it attempted to do happened | 14:42 |
dims | mriedem, to figure out why we are seeing the ssh timeout? (Bug 1298472 - SSHTimeout in tempest scenario tests using nova-network) | 14:42 |
* dims catching up | 14:42 | |
mriedem | yes | 14:42 |
mriedem | dims: yes | 14:42 |
mriedem | sdague: ok, got some charts i have to work on today (not looking forward to it), then i can help with this if no one else has jumped on by then | 14:43 |
sdague | mriedem: cool, thanks | 14:43 |
*** salv-orlando has joined #openstack-qa | 14:46 | |
*** roeyc has quit IRC | 14:47 | |
*** psharma has quit IRC | 14:48 | |
*** coolsvap|afk is now known as coolsvap | 14:50 | |
openstackrecheck | Console logs not available after 13:25s for check-grenade-dsvm 98776,1,531dd48 | 14:53 |
openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: do not override ceilometer default polling interval https://review.openstack.org/98815 | 14:54 |
*** lbragstad has joined #openstack-qa | 14:54 | |
sdague | mtreinish: how you feeling about this - https://review.openstack.org/#/c/97812 | 14:57 |
sdague | it apparently got lost somewhere | 14:57 |
mtreinish | sdague: does it actually provide useful debug info? | 14:58 |
*** bkopilov has joined #openstack-qa | 14:59 | |
openstackgerrit | A change was merged to openstack/tempest: Revert "don't reuse servers in test_server_actions" https://review.openstack.org/98632 | 15:00 |
openstackgerrit | A change was merged to openstack/tempest: provide ability to tweak oslo log defaults https://review.openstack.org/98515 | 15:00 |
sdague | getting the console? I'm not sure we're going to know until we hit an error case | 15:00 |
*** yfried has quit IRC | 15:01 | |
mtreinish | sdague: the code looks fine to me, but I'm just trying to weigh dumping more info into the failure logs vs the potential benefit | 15:01 |
mtreinish | because it'll go on the console output which just makes figuring out what failed harder | 15:02 |
mtreinish | sdague: maybe if we paired it with something like this: https://review.openstack.org/#/c/97894/ | 15:03 |
openstackgerrit | Sridhar Gaddam proposed a change to openstack/tempest: Improve IPV6 parity in Security Group testcases https://review.openstack.org/94130 | 15:03 |
sdague | mtreinish: I actually think that's going to take us backwards | 15:05 |
mtreinish | moving the failure logs to a separate file? | 15:06 |
sdague | yep | 15:06 |
sdague | if you want to put them at the end of the output run again, I'm fine with it, but I don't actually want them shuffled off to another file | 15:06 |
*** mriedem has quit IRC | 15:07 | |
sdague | if we think the logs are confusing, we should get to the bottom of the lines that are horribly confusing and fix them | 15:08 |
dkranz | mtreinish: I think I agree with Sean. I was looking at some failures and missed having failures show up at the end. | 15:08 |
sdague | not take them out of the console | 15:08 |
*** jaypipes has quit IRC | 15:09 | |
mtreinish | my problem is when trying to debug a failure, that having huge fail logs with console dumps and other things makes it difficult to figure out exactly what failed from the console output | 15:09 |
mtreinish | moving the failures to the end doesn't actually help with that | 15:09 |
mtreinish | or maybe it would | 15:10 |
mtreinish | let me respin that to put it at the end | 15:10 |
*** cdent has joined #openstack-qa | 15:10 | |
openstackrecheck | Console logs not available after 13:23s for check-grenade-dsvm 94360,1,e4110af | 15:10 |
*** che-arne_afk has joined #openstack-qa | 15:10 | |
*** mriedem has joined #openstack-qa | 15:11 | |
sdague | also, one of the issues is the subunit log format isn't our log format | 15:13 |
sdague | I wonder if there is a good way to reset that | 15:13 |
*** che-arne has quit IRC | 15:13 | |
*** kgriffs is now known as kgriffs|afk | 15:14 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Change how tempest debug logs are displayed https://review.openstack.org/97894 | 15:14 |
mriedem | mtreinish: just for context, the patch to dump console logs on timeout/failure are for debugging the various timed out waiting for state failures, not the n-net issue | 15:17 |
mriedem | but i'm all for dumping those to some other log file if possible | 15:18 |
*** morganfainberg_Z is now known as morganfainberg | 15:18 | |
mtreinish | mriedem: no I got that, I'm just concerned with figuring out what actually failed in those case. | 15:19 |
mtreinish | The console logs, at least for me, gunk up the console log and make it hard to figure out what happened | 15:19 |
*** che-arne has joined #openstack-qa | 15:19 | |
mtreinish | sdague: wait we log the the console at debug level right? I thought we only captured at info and above in the fixture | 15:20 |
mtreinish | or did that change with your log refactoring? | 15:20 |
sdague | mtreinish: good question, I don't know | 15:20 |
sdague | honestly I've got like 25 patches outstanding right now, I need to start sifting back through them to figure out what's viable | 15:21 |
*** jaypipes has joined #openstack-qa | 15:21 | |
*** ChanServ sets mode: +o jaypipes | 15:21 | |
*** che-arne_afk has quit IRC | 15:22 | |
sdague | mtreinish: we must be logging debug because of all the glanceclient spam that's in there | 15:24 |
sdague | mtreinish: any idea how to actually adjust the log format line that's in subunit so that it matches tempest? | 15:24 |
sdague | because that's part of what makes thing confusing as well, the request ids and log levels aren't displayed | 15:25 |
openstackrecheck | Console logs not available after 14:49s for check-grenade-dsvm-partial-ncpu 98603,2,bd3cf27 | 15:25 |
mtreinish | sdague: oh it looks like we capture at the config log_level because of fixture bug. because we use nuke_handlers=False the fixture would override the configured log_level | 15:27 |
mtreinish | sdague: I don't know about the subunit thing, I'd have to dig into it | 15:27 |
*** Obulpath_ has joined #openstack-qa | 15:28 | |
*** tkammer_ has quit IRC | 15:29 | |
*** jgrimm has quit IRC | 15:29 | |
*** kgriffs|afk is now known as kgriffs | 15:29 | |
sdague | mtreinish: oh, I see it | 15:29 |
*** Obulpath_ has quit IRC | 15:31 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Make run_stress.py script an entry point https://review.openstack.org/92519 | 15:31 |
*** Obulpath_ has joined #openstack-qa | 15:31 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Add bug skip for test_container_synchronization() https://review.openstack.org/92606 | 15:32 |
*** Obulpathi has quit IRC | 15:32 | |
openstackgerrit | Sean Dague proposed a change to openstack/tempest: make logformat match oslo https://review.openstack.org/98819 | 15:33 |
sdague | mtreinish: ^^^ | 15:33 |
*** jpich has quit IRC | 15:34 | |
*** samuelbercovici has joined #openstack-qa | 15:35 | |
openstackrecheck | Console logs not available after 13:26s for check-tempest-dsvm-full 97337,1,1b04ad6 | 15:38 |
*** Obulpath_ has quit IRC | 15:41 | |
*** Obulpathi has joined #openstack-qa | 15:41 | |
*** mlavalle_ has joined #openstack-qa | 15:42 | |
*** mlavalle has quit IRC | 15:42 | |
*** mlavalle_ is now known as mlavalle | 15:42 | |
mtreinish | sdague: shouldn't you pull that from the CONF object? (I think the format is a config field) | 15:43 |
sdague | we could, that's a pretty close approximation | 15:43 |
openstackgerrit | Sean Dague proposed a change to openstack/tempest: make logformat match oslo https://review.openstack.org/98819 | 15:46 |
*** Obulpath_ has joined #openstack-qa | 15:46 | |
*** mlavalle_ has joined #openstack-qa | 15:47 | |
*** mlavalle has quit IRC | 15:49 | |
*** mlavalle_ is now known as mlavalle | 15:49 | |
*** Obulpathi has quit IRC | 15:49 | |
*** jlibosva has quit IRC | 15:51 | |
*** bauzas has joined #openstack-qa | 15:52 | |
openstackrecheck | Console logs not available after 13:24s for check-tempest-dsvm-full 95370,3,ab062d8 | 15:53 |
*** mrodden has quit IRC | 15:54 | |
*** jchai has quit IRC | 15:54 | |
*** samuelbercovici1 has joined #openstack-qa | 15:56 | |
*** malini is now known as malini_afk | 15:57 | |
*** andreaf has quit IRC | 16:00 | |
*** samuelbercovici has quit IRC | 16:00 | |
*** samuelbercovici1 is now known as samuelbercovici | 16:00 | |
*** aimon_ has quit IRC | 16:02 | |
*** beisner-weekend has quit IRC | 16:03 | |
*** ekhugen has quit IRC | 16:06 | |
*** mrodden has joined #openstack-qa | 16:07 | |
openstackgerrit | Greg Lucas proposed a change to openstack-dev/devstack: Use correct path when extracting tar.gz image https://review.openstack.org/98825 | 16:07 |
*** ildikov_ has joined #openstack-qa | 16:08 | |
*** aimon_ has joined #openstack-qa | 16:10 | |
*** yamahata has joined #openstack-qa | 16:10 | |
openstackrecheck | Console logs not available after 13:34s for check-tempest-dsvm-postgres-full-icehouse 98505,4,3b83413 | 16:10 |
*** ildikov has quit IRC | 16:10 | |
*** gszasz has quit IRC | 16:11 | |
*** yfried has joined #openstack-qa | 16:13 | |
*** yfried has quit IRC | 16:13 | |
*** yfried has joined #openstack-qa | 16:13 | |
*** beisner has joined #openstack-qa | 16:14 | |
*** jgrimm has joined #openstack-qa | 16:16 | |
*** che-arne has quit IRC | 16:16 | |
*** jordanP has quit IRC | 16:17 | |
*** che-arne has joined #openstack-qa | 16:17 | |
*** Longgeek has quit IRC | 16:19 | |
*** ekhugen has joined #openstack-qa | 16:19 | |
*** markmcclain has joined #openstack-qa | 16:21 | |
*** jaypipes has quit IRC | 16:21 | |
*** che-arne has quit IRC | 16:21 | |
*** yamahata has quit IRC | 16:22 | |
*** yamahata has joined #openstack-qa | 16:22 | |
openstackrecheck | Console logs not available after 13:33s for check-tempest-dsvm-full 75346,8,c6660fb | 16:26 |
*** aimon_ has quit IRC | 16:31 | |
*** yamahata has quit IRC | 16:35 | |
*** yamahata has joined #openstack-qa | 16:35 | |
*** [1]evgenyf has quit IRC | 16:37 | |
openstackrecheck | Console logs not available after 13:29s for check-grenade-dsvm 88067,5,4d36a62 | 16:40 |
*** catohornet has joined #openstack-qa | 16:51 | |
*** dkorn has joined #openstack-qa | 16:54 | |
openstackrecheck | Console logs not available after 13:25s for check-grenade-dsvm 97782,3,6a90f5b | 16:57 |
*** aimon_ has joined #openstack-qa | 17:01 | |
*** dkorn has quit IRC | 17:03 | |
openstackrecheck | openstack/heat change: https://review.openstack.org/94755 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1254872 | 17:04 |
*** marun has joined #openstack-qa | 17:07 | |
*** mmaglana has joined #openstack-qa | 17:09 | |
*** kgriffs is now known as kgriffs|afk | 17:09 | |
*** harlowja_away is now known as harlowja | 17:09 | |
*** marun has quit IRC | 17:11 | |
openstackrecheck | Console logs not available after 13:30s for check-grenade-dsvm-partial-ncpu-icehouse 98497,2,803dcfa | 17:18 |
*** samuelbercovici has quit IRC | 17:23 | |
*** nati_ueno has joined #openstack-qa | 17:27 | |
openstackgerrit | meera-belur proposed a change to openstack/tempest: Adding tests for the barbican - key management service https://review.openstack.org/98842 | 17:29 |
*** markmcclain has quit IRC | 17:30 | |
*** markmcclain has joined #openstack-qa | 17:33 | |
*** Manishanker has joined #openstack-qa | 17:36 | |
*** jay_ has joined #openstack-qa | 17:43 | |
openstackrecheck | Console logs not available after 13:24s for check-grenade-dsvm 89897,28,4ba168c | 17:46 |
openstackgerrit | Chuck Short proposed a change to openstack-dev/devstack: Add utopic support https://review.openstack.org/98844 | 17:47 |
*** malini_afk is now known as malini | 17:48 | |
*** Manishanker has quit IRC | 17:52 | |
*** cdent has quit IRC | 17:54 | |
*** malini is now known as malini_afk | 17:59 | |
openstackrecheck | Console logs not available after 13:33s for check-tempest-dsvm-postgres-full 97339,4,ac98384 | 18:00 |
*** mkoderer has quit IRC | 18:02 | |
*** coolsvap is now known as coolsvap|afk | 18:03 | |
*** devananda has joined #openstack-qa | 18:08 | |
devananda | hi! I'm wondering why we're not seeing any E-R posts on ironic changes | 18:08 |
devananda | eg, https://review.openstack.org/#/c/86536/ totally matched the recheck for 1326754 and it's showing up in the graphs on status.oo.o/elastic-recheck/ | 18:09 |
devananda | but we dont see any posts on the reviews when there's a failure | 18:09 |
*** kgriffs|afk is now known as kgriffs | 18:09 | |
clarkb | devananda: because its behind | 18:11 |
clarkb | currently unable to index quickly enough | 18:11 |
devananda | clarkb: so I think it's never posted a single comment on any ironic patch, actually | 18:13 |
devananda | clarkb: at least none this month (as far back as my local inbox goes) | 18:14 |
clarkb | well you only just started classifying things | 18:14 |
clarkb | and we have been in this hole since before you did that | 18:14 |
*** jaypipes has joined #openstack-qa | 18:15 | |
*** ChanServ sets mode: +o jaypipes | 18:15 | |
*** mrodden has quit IRC | 18:17 | |
openstackrecheck | Console logs not available after 13:27s for check-tempest-dsvm-full-icehouse 92519,4,8122215 | 18:17 |
devananda | clarkb: we've had E-R checks up since thursday, and E-R has commented on plenty of /other/ things over the weekend, and those bugs have been matched by errors that ironic hit over the weekend, but E-R didn't comment | 18:18 |
*** kgriffs is now known as kgriffs|afk | 18:19 | |
devananda | clarkb: E-R is clearly running against Nova changes today. So, when should I expect E-R to be commenting on Ironic for queries we added last week? | 18:21 |
*** mrodden has joined #openstack-qa | 18:21 | |
*** kgriffs|afk is now known as kgriffs | 18:21 | |
clarkb | devananda: it should be doing it now | 18:22 |
clarkb | but it is behind | 18:22 |
devananda | clarkb: days behind? | 18:22 |
clarkb | the logstash job queue graph on the zuul status page describes it | 18:22 |
clarkb | devananda: no | 18:22 |
clarkb | devananda: but E-R gives up after 15 minutes | 18:22 |
devananda | ah | 18:23 |
clarkb | we are working on fixing the backlog | 18:23 |
clarkb | going to stop indexing some data that hasn't been useful and add more workers | 18:23 |
*** atiwari has joined #openstack-qa | 18:24 | |
*** dims has quit IRC | 18:25 | |
*** dims has joined #openstack-qa | 18:25 | |
*** ildikov has joined #openstack-qa | 18:27 | |
dkranz | boris-42: Are you intending to provide more detail in https://review.openstack.org/#/c/94473/ ? | 18:28 |
boris-42 | devananda sure | 18:29 |
boris-42 | dkranz sure* | 18:29 |
boris-42 | devananda sorry* | 18:29 |
*** ildikov has quit IRC | 18:29 | |
boris-42 | dkranz there are 2 more devs in rally that are interested in supporting better integration of rally & tempest | 18:29 |
boris-42 | dkranz so I hope they will help me as well with this part | 18:29 |
dkranz | boris-42: ok, I am asking because it is untouched since May 23 | 18:30 |
boris-42 | dkranz sorry for that, have a lot of work | 18:30 |
*** markmcclain has quit IRC | 18:30 | |
dkranz | boris-42: understood | 18:30 |
boris-42 | dkranz would like to get osprofiler in openstack finally | 18:30 |
*** ildikov_ has quit IRC | 18:30 | |
*** markmcclain has joined #openstack-qa | 18:30 | |
boris-42 | dkranz seems like it's close to done https://github.com/stackforge/osprofiler | 18:30 |
openstackrecheck | Console logs not available after 13:27s for check-grenade-dsvm 92606,2,2d732d4 | 18:30 |
dkranz | boris-42: sometimes this sort of thing happens due to too much work, other times due to being dropped. | 18:31 |
dkranz | boris-42: I just was asking which. | 18:31 |
*** jay_ has quit IRC | 18:31 | |
*** markmcclain has quit IRC | 18:31 | |
boris-42 | dkranz heh I will continue work on that stuff as well=) | 18:31 |
boris-42 | dkranz no worries | 18:31 |
*** markmcclain has joined #openstack-qa | 18:31 | |
dkranz | boris-42: ok, great. I might make some concrete suggestion as well. | 18:31 |
boris-42 | dkranz good thanks | 18:31 |
*** nati_ueno has quit IRC | 18:31 | |
boris-42 | dkranz https://github.com/pboris/logaas btw maybe will be interesting for you guys project | 18:32 |
*** jhenner has quit IRC | 18:34 | |
*** nati_uen_ has joined #openstack-qa | 18:34 | |
*** jhenner has joined #openstack-qa | 18:35 | |
openstackgerrit | Sean Dague proposed a change to openstack/tempest: make logformat match oslo https://review.openstack.org/98819 | 18:39 |
*** Obulpath_ has quit IRC | 18:40 | |
*** dkranz has quit IRC | 18:42 | |
*** ildikov|2 has joined #openstack-qa | 18:45 | |
*** afazekas has joined #openstack-qa | 18:45 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Validate ssh client auth before we use it https://review.openstack.org/98446 | 18:46 |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Add extra debug info to ssh failure https://review.openstack.org/98507 | 18:46 |
*** Obulpathi has joined #openstack-qa | 18:47 | |
*** Obulpath_ has joined #openstack-qa | 18:48 | |
openstackgerrit | Rob Crittenden proposed a change to openstack-dev/devstack: Configure endpoints to use SSL natively or via proxy https://review.openstack.org/98854 | 18:51 |
*** Obulpathi has quit IRC | 18:51 | |
*** weshay has quit IRC | 18:54 | |
*** ildikov|2 is now known as ildikov | 18:54 | |
*** dkranz has joined #openstack-qa | 18:58 | |
*** markmcclain has quit IRC | 19:00 | |
*** nati_ueno has joined #openstack-qa | 19:02 | |
*** Obulpath_ has quit IRC | 19:04 | |
*** dkranz has quit IRC | 19:06 | |
*** nati_uen_ has quit IRC | 19:06 | |
*** markmcclain has joined #openstack-qa | 19:07 | |
*** weshay has joined #openstack-qa | 19:09 | |
openstackgerrit | Morgan Fainberg proposed a change to openstack-dev/devstack: Use apache2ctl after apache is installed. https://review.openstack.org/98859 | 19:10 |
openstackrecheck | Console logs not available after 13:23s for check-tempest-dsvm-full 91727,3,0f0e5bf | 19:11 |
*** nati_ueno has quit IRC | 19:15 | |
*** Obulpathi has joined #openstack-qa | 19:16 | |
*** markmcclain1 has joined #openstack-qa | 19:19 | |
*** dkranz has joined #openstack-qa | 19:19 | |
*** markmcclain1 has quit IRC | 19:20 | |
*** markmcclain1 has joined #openstack-qa | 19:20 | |
*** markmcclain has quit IRC | 19:23 | |
openstackrecheck | Console logs not available after 14:37s for check-tempest-dsvm-postgres-full 90143,7,248d50d | 19:31 |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Bump hacking to 0.9.x series https://review.openstack.org/98868 | 19:41 |
sdague | mtreinish: I actually think on nova api v3 we should set the default in tempest to false | 19:42 |
sdague | because it's experimental | 19:42 |
sdague | not via devstack | 19:42 |
*** markmcclain1 has quit IRC | 19:42 | |
mtreinish | sdague: ok, that's fine. It was just a patch to test it anyway | 19:43 |
sdague | yep | 19:43 |
sdague | I just did a recheck, now that it has a chance | 19:43 |
mtreinish | honestly it was hard to see much of a time savings without the v3 tests though | 19:43 |
openstackrecheck | Console logs not available after 13:23s for check-grenade-dsvm 97814,1,1aeca50 | 19:44 |
*** mmaglana_ has joined #openstack-qa | 19:50 | |
openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: Revert "Set CEILOMETER_PIPELINE_INTERVAL to 15" https://review.openstack.org/98871 | 19:52 |
*** mmaglana has quit IRC | 19:52 | |
*** marun has joined #openstack-qa | 19:55 | |
*** mmaglana has joined #openstack-qa | 19:55 | |
*** mmaglana_ has quit IRC | 19:58 | |
openstackrecheck | Console logs not available after 13:33s for check-grenade-dsvm 97944,1,6318478 | 19:58 |
*** morganfainberg is now known as morganfainberg_Z | 20:06 | |
*** boris-42 has quit IRC | 20:08 | |
*** boris-42 has joined #openstack-qa | 20:11 | |
openstackrecheck | Console logs not available after 13:24s for check-grenade-dsvm 97013,1,831831e | 20:12 |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Bump hacking to 0.9.x series https://review.openstack.org/98868 | 20:18 |
mtreinish | jogo: ^^^ man the new hacking killed tempest... | 20:19 |
mtreinish | it needed 17 ignores | 20:19 |
mriedem | mtreinish: where are the .failure files stored for grenade? | 20:26 |
openstackrecheck | Console logs not available after 14:27s for check-grenade-dsvm-partial-ncpu-icehouse 94130,7,28589e2 | 20:27 |
mtreinish | mriedem: on the log server or in general? | 20:28 |
openstackgerrit | A change was merged to openstack-dev/devstack: Update setuptools to latest for .dist-info support https://review.openstack.org/98263 | 20:28 |
mriedem | mtreinish: like here http://logs.openstack.org/79/96679/3/check/check-grenade-dsvm-partial-ncpu/7bc0290/logs/ | 20:28 |
mriedem | i've been seeing grenade failing since last week a lot with lots of services failing to start | 20:28 |
mriedem | but not sure why they aren't starting | 20:28 |
mtreinish | mriedem: it's supposed to make a status dir and put the failure files in there | 20:28 |
mriedem | that must be busted... | 20:29 |
mtreinish | mriedem: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n496 | 20:29 |
*** Obulpathi has quit IRC | 20:29 | |
*** kgriffs is now known as kgriffs|afk | 20:29 | |
*** Obulpathi has joined #openstack-qa | 20:30 | |
mtreinish | mriedem: I probably screwed something up | 20:31 |
mriedem | mtreinish: looking | 20:33 |
mriedem | mtreinish: methinks you did | 20:33 |
mriedem | sudo rm $BASE/logs/grenade.sh.log.* | 20:33 |
*** kgriffs|afk is now known as kgriffs | 20:33 | |
mriedem | grenade.sh log /opt/stack/logs/grenade.sh.log.2014-06-09-163750 | 20:33 |
mriedem | therefore BASE=/opt/stack | 20:33 |
mriedem | yet: if [ $(ls $BASE/stack/status/*.failure | wc -l) -gt 0 ]; then | 20:33 |
afazekas | sdague: https://review.openstack.org/#/c/98212/ | 20:33 |
*** mwagner_lap has quit IRC | 20:34 | |
*** markmcclain has joined #openstack-qa | 20:34 | |
mriedem | actually: tee "/opt/stack/status/stack/horizon.failure" | 20:35 |
mriedem | so should be looking in: $BASE/stack/status/stack/*.failure | 20:36 |
mriedem | i can push a patch | 20:36 |
mtreinish | mriedem: why the extra 'stack', shouldn't it be without the stack? | 20:37 |
mtreinish | or am I misreading? | 20:37 |
mriedem | http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n91 | 20:38 |
mriedem | right? | 20:38 |
mriedem | mtreinish: http://logs.openstack.org/79/96679/3/check/check-grenade-dsvm-partial-ncpu/7bc0290/logs/devstack-gate-cleanup-host.txt | 20:39 |
mriedem | 2014-06-09 17:00:25.848 | ++ ls '/opt/stack/stack/status/*.failure' 2014-06-09 17:00:25.849 | ls: cannot access /opt/stack/stack/status/*.failure: No such file or directory | 20:39 |
mriedem | me fixy | 20:39 |
*** nati_ueno has joined #openstack-qa | 20:40 | |
openstackrecheck | Console logs not available after 13:24s for check-grenade-dsvm 92479,2,75da9a6 | 20:40 |
mtreinish | mriedem: oh I missed the extra stack in the tee command | 20:40 |
mriedem | np, fixing | 20:40 |
mriedem | it's even right 2 lines down | 20:41 |
mriedem | but now in the ls command | 20:41 |
mriedem | *not | 20:41 |
mriedem | sudo cp $BASE/status/stack/*.failure $BASE/logs/status/ | 20:41 |
*** jhenner has quit IRC | 20:41 | |
mtreinish | meh, details | 20:41 |
openstackgerrit | Matt Riedemann proposed a change to openstack-infra/devstack-gate: Fix collection of *.failure files https://review.openstack.org/98885 | 20:43 |
mriedem | mtreinish: sdague: ^ | 20:43 |
*** Sukhdev has joined #openstack-qa | 20:49 | |
mtreinish | mriedem: can you rebase your network connectivity tempest patches? | 20:51 |
mtreinish | I think it'll have trouble getting past check unless it's on top of the revert from this morning | 20:51 |
mriedem | mtreinish: this revert? https://review.openstack.org/#/c/98632/ | 20:51 |
mtreinish | yeah the rebase button should work fine in this case | 20:52 |
mriedem | hmmm | 20:52 |
mriedem | i don't know if i trust that | 20:52 |
mtreinish | it worked for me earlier today :) | 20:53 |
mriedem | ok well if all of my scores are gone i'm going to hunt you down | 20:53 |
openstackgerrit | Matt Riedemann proposed a change to openstack/tempest: Consolidate _check_tenant_network_connectivity in NetworkScenarioTest https://review.openstack.org/95844 | 20:54 |
openstackgerrit | Matt Riedemann proposed a change to openstack/tempest: Consolidate _check_public_network_connectivity into NetworkScenarioTest https://review.openstack.org/95831 | 20:54 |
mtreinish | I'll just push it through | 20:54 |
mriedem | thanks | 20:54 |
openstackrecheck | Console logs not available after 13:52s for gate-keystone-pep8 42967,5,36b3dba | 20:54 |
mtreinish | mriedem: why did the scores disappear from the first one but not the second? | 20:55 |
mriedem | mtreinish: idk | 20:55 |
mriedem | rebase black magic | 20:55 |
mriedem | i just rebased locally, no changes | 20:55 |
mriedem | mtreinish: oh i see | 20:56 |
mriedem | commit message on the first must have been edited in gerrit | 20:56 |
mriedem | was a nit | 20:56 |
mriedem | https://review.openstack.org/#/c/95831/4..5//COMMIT_MSG | 20:56 |
mtreinish | ok, that explains it | 20:56 |
*** jlibosva has joined #openstack-qa | 21:00 | |
*** jlibosva has quit IRC | 21:00 | |
*** jlibosva has joined #openstack-qa | 21:01 | |
*** giulivo has quit IRC | 21:03 | |
jogo | mtreinish: thats a lot of skips :) | 21:04 |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Enable E113,E111,H302,F812,E713 rules https://review.openstack.org/98899 | 21:04 |
jogo | mtreinish: I should have mentioned that hacking isn't in our pypi mirror yet http://pypi.openstack.org/openstack/hacking/ | 21:04 |
mtreinish | jogo: that's fine I'll just recheck when it is | 21:05 |
jogo | 98824,1 | 21:06 |
jogo | is the patch | 21:06 |
openstackrecheck | Console logs not available after 13:33s for check-grenade-dsvm 84645,5,d2901ff | 21:08 |
*** morganfainberg_Z is now known as morganfainberg | 21:10 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Enable H402,E251,E265 rules https://review.openstack.org/98903 | 21:16 |
*** mdenny has quit IRC | 21:17 | |
*** SridharG has quit IRC | 21:18 | |
*** flwang has quit IRC | 21:18 | |
openstackrecheck | Console logs not available after 13:40s for gate-nova-python26 98201,2,0d7db4e | 21:21 |
*** mdenny has joined #openstack-qa | 21:26 | |
*** kgriffs is now known as kgriffs|afk | 21:28 | |
*** ekhugen has quit IRC | 21:30 | |
openstackgerrit | Adam Gandelman proposed a change to openstack-infra/elastic-recheck: Add query for Ironic bug 1328274 https://review.openstack.org/98905 | 21:32 |
*** Obulpathi has quit IRC | 21:33 | |
*** dkranz has quit IRC | 21:34 | |
openstackrecheck | Console logs not available after 13:29s for check-grenade-dsvm 98021,1,e7d8a36 | 21:35 |
openstackgerrit | meera-belur proposed a change to openstack/tempest: Adding tests for the barbican - key management service https://review.openstack.org/98842 | 21:36 |
*** cbader has quit IRC | 21:37 | |
*** bauzas has quit IRC | 21:37 | |
*** mriedem has quit IRC | 21:45 | |
*** mrodden has quit IRC | 21:46 | |
openstackrecheck | Console logs not available after 13:25s for gate-tempest-dsvm-postgres-full 98098,2,bab1954 | 21:50 |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Enable E129,E128 rules https://review.openstack.org/98909 | 21:52 |
*** kgriffs|afk has quit IRC | 21:53 | |
*** malini_afk has quit IRC | 21:53 | |
jogo | mtreinish: E251 | 21:56 |
*** andreaf has joined #openstack-qa | 21:57 | |
jogo | https://github.com/jcrocholl/pep8/issues/301 | 21:57 |
*** jlibosva has quit IRC | 21:59 | |
mtreinish | jogo: I think I noticed something weird about that one, but I think I worked around it: https://review.openstack.org/#/c/98903 | 21:59 |
mtreinish | but, I can't remember which changes go with which rule anymore | 21:59 |
*** dims_ has joined #openstack-qa | 22:00 | |
jogo | ahh I see what you did, you changed the indent format | 22:01 |
jogo | ' | 22:01 |
jogo | https://review.openstack.org/#/c/98903/1/tempest/api/volume/test_volumes_get.py | 22:01 |
*** catohornet1 has joined #openstack-qa | 22:01 | |
*** markmcclain has quit IRC | 22:01 | |
*** adalbas has quit IRC | 22:02 | |
*** dims has quit IRC | 22:02 | |
sdague | so really? hacking changes :P | 22:05 |
* jogo hides | 22:06 | |
sdague | with the current state of the failures, hacking changes seems pretty gratuitous | 22:07 |
*** rdekel has joined #openstack-qa | 22:08 | |
mtreinish | sdague: I was pretty burnt out today, but this was something to spin my extra cycles on without much load | 22:08 |
*** nati_ueno has quit IRC | 22:08 | |
jogo | sdague: agreed, but I have been holding off doing this for a few months | 22:08 |
openstackgerrit | A change was merged to openstack-infra/devstack-gate: Fix collection of *.failure files https://review.openstack.org/98885 | 22:08 |
jogo | mtreinish: hehe | 22:09 |
sdague | so, I'll mildy accept it from mtreinish as he was firedrilling last week as well | 22:09 |
sdague | but o_O to jogo :P | 22:09 |
jogo | http://logs.openstack.org/98/97598/1/check/check-grenade-dsvm-partial-ncpu/2125f27/console.html#_2014-06-09_19_16_59_231 | 22:09 |
jogo | is that a new failure mode? | 22:10 |
openstackrecheck | Console logs not available after 13:23s for check-tempest-dsvm-full 98819,2,6e4b659 | 22:10 |
*** salv-orlando has quit IRC | 22:10 | |
sdague | mtreinish: tomorrow, I think we should work on an etherpad to sort out the host validitor bits, then see if we can split it up in a way to tag team | 22:10 |
mtreinish | sdague: ok, makes sense. I have bits and pieces that I started today | 22:11 |
*** malini_afk has joined #openstack-qa | 22:11 | |
*** malini_afk is now known as malini | 22:11 | |
*** rdekel has quit IRC | 22:12 | |
*** kgriffs has joined #openstack-qa | 22:12 | |
sdague | mtreinish: yeh, it just seems like it's probably a bunch of work. So if we can tag team it, it might be more energizing than either one of us just getting bogged down in it | 22:13 |
*** tosky has quit IRC | 22:13 | |
sdague | jogo: honestly, I'm not looking at any more failures today, because otherwise I'll get stabby | 22:13 |
*** salv-orlando has joined #openstack-qa | 22:14 | |
*** salv-orlando has quit IRC | 22:15 | |
jogo | haha | 22:15 |
jogo | sdague: maybe you should do some hacking patches to calm your nerves like mtreinish | 22:16 |
*** salv-orlando has joined #openstack-qa | 22:16 | |
*** salv-orlando has quit IRC | 22:17 | |
openstackgerrit | Sean Dague proposed a change to openstack/tempest: make logformat match oslo https://review.openstack.org/98819 | 22:17 |
mtreinish | well I'm almost through all 17 ignores :) | 22:17 |
mtreinish | but I guess the other projects are fair game | 22:17 |
*** eglynn has joined #openstack-qa | 22:17 | |
openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint fur bug 1268725 https://review.openstack.org/98913 | 22:19 |
*** salv-orlando has joined #openstack-qa | 22:19 | |
mtreinish | jogo: quick question how does H305 handle different python versions? Like if something was made stdlib in py3 but was third party in py2 | 22:19 |
jogo | oh guh | 22:19 |
jogo | noqa? | 22:20 |
jogo | mtreinish: I don't think we can do anything else about that | 22:20 |
mtreinish | hahaha, ok | 22:20 |
jogo | in py2 mode we don't know about p3 and vice versa | 22:20 |
mtreinish | because I just realized I'm running it with py3 on my box | 22:20 |
*** salv-orlando has quit IRC | 22:20 | |
mtreinish | and I think there are a few of those | 22:20 |
sdague | yeh, I'd just say ignore the rule then | 22:21 |
sdague | it seems like it will generate more work | 22:21 |
eglynn | ... now I know it's bad form to just hop on the channel, agitating for reviewer love to be bestowed upon one's favourite patches | 22:21 |
sdague | and not really save us anything | 22:21 |
eglynn | ... but if any of the grenade cores are around and have a minute to take another pass on the ceilo grenade patch | 22:21 |
eglynn | ... that would be most appreciated! :) | 22:22 |
eglynn | https://review.openstack.org/94468 | 22:22 |
mtreinish | eglynn: well there are only 2 grenade cores right now sdague and dtroyer :) | 22:22 |
*** salv-orlando has joined #openstack-qa | 22:22 | |
*** salv-orlando has quit IRC | 22:23 | |
eglynn | mtreinish: a-ha, ok, thanks for the heads-up | 22:23 |
eglynn | mtreinish: ... since I've intruded already, a quick question on the QA/Infra meetup? | 22:24 |
eglynn | mtreinish: ... are you still planning a ceilometer-specific sub-sprint? | 22:24 |
*** mwagner_lap has joined #openstack-qa | 22:24 | |
openstackrecheck | Console logs not available after 13:25s for check-tempest-dsvm-full 97569,3,c3e850e | 22:24 |
* eglynn asks as internal travel-budget planning is afoot ... | 22:25 | |
*** salv-orlando has joined #openstack-qa | 22:25 | |
mtreinish | eglynn: we can, it really depends if ceilo people want to show up | 22:26 |
eglynn | mtreinish: OK, so it *may* be possible for me to attend | 22:27 |
eglynn | mtreinish: ... and of course I would welcome such concentrated attention on ceilo testing from the QA ninjas | 22:28 |
eglynn | mtreinish: ... but would depend on whether other Red Hatters are gonna be covered the mid-cycle already | 22:28 |
*** weshay has quit IRC | 22:29 | |
eglynn | s/covered/covering/ | 22:30 |
eglynn | mtreinish: ... but can I take it that if there's ceilo folk in attendance, then there's space on the agenda for such a "sub-sprint"? | 22:31 |
mtreinish | eglynn: well the agenda for the 3 discussion/code sprint days is up in the air at this point, and the attendance list is pretty light at this point | 22:31 |
mtreinish | so we can play it by ear for now | 22:32 |
mtreinish | is there anyone else who's working on ceilo testing who you think should come if we have a day for it? | 22:32 |
eglynn | ... one of the Mirantis folks may make sense also | 22:34 |
eglynn | ... I'll it run them in their TZ tmrw | 22:35 |
eglynn | ... v. late now in MSK | 22:35 |
eglynn | mtreinish: ... I'll get back to you to confirm, thanks! | 22:35 |
eglynn | ... run it *by them | 22:36 |
* eglynn loses the ability to type coherently, blames the lateness of the hour | 22:37 | |
mtreinish | eglynn: ok cool | 22:40 |
mtreinish | jogo: I really like H904, but I am going to hate fixing it in tempest... | 22:40 |
openstackrecheck | Console logs not available after 13:25s for check-grenade-dsvm 97728,7,befb027 | 22:41 |
jogo | haha yeah | 22:41 |
*** dustins1 has quit IRC | 22:42 | |
mtreinish | both that and H405 generate more lines than can fit in my terminal buffer... | 22:43 |
*** hemna is now known as hemna_ | 22:45 | |
*** jgrimm has quit IRC | 22:45 | |
openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Enable H407,H305,E123,H307,E122 rules https://review.openstack.org/98918 | 22:45 |
*** eglynn has quit IRC | 22:46 | |
*** nati_ueno has joined #openstack-qa | 22:47 | |
*** jamielennox|away is now known as jamielennox | 22:47 | |
*** Sukhdev has quit IRC | 22:49 | |
*** aysyd has quit IRC | 22:51 | |
openstackrecheck | Console logs not available after 13:23s for check-tempest-dsvm-full-icehouse 90462,9,c88011c | 22:54 |
*** locke105 has quit IRC | 22:56 | |
*** rbak has quit IRC | 22:58 | |
openstackrecheck | Console logs not available after 13:28s for check-tempest-dsvm-neutron 98779,1,bed8117 | 23:07 |
*** kgriffs is now known as kgriffs|afk | 23:13 | |
openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 https://review.openstack.org/98919 | 23:13 |
*** locke105 has joined #openstack-qa | 23:19 | |
openstackrecheck | Console logs not available after 13:39s for check-tempest-dsvm-postgres-full 98546,1,4a0e952 | 23:22 |
*** nati_ueno has quit IRC | 23:22 | |
*** atiwari has quit IRC | 23:24 | |
*** morganfainberg is now known as morganfainberg_Z | 23:25 | |
*** nati_ueno has joined #openstack-qa | 23:28 | |
*** nati_uen_ has joined #openstack-qa | 23:31 | |
*** nati_ueno has quit IRC | 23:32 | |
openstackrecheck | Console logs not available after 13:57s for check-tempest-dsvm-full 97059,6,b5d6f2e | 23:36 |
*** aimon_ has quit IRC | 23:39 | |
*** morganfainberg_Z is now known as morganfainberg | 23:42 | |
*** masayukig_ has quit IRC | 23:43 | |
*** masayukig has joined #openstack-qa | 23:43 | |
openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 and 1326855 https://review.openstack.org/98919 | 23:48 |
openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-neutron 95666,4,1bd7167 | 23:49 |
*** catohornet1 has quit IRC | 23:53 | |
*** aimon has joined #openstack-qa | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!