Tuesday, 2014-09-02

*** salv-orlando has quit IRC00:02
openstackgerritMaru Newby proposed a change to openstack-infra/elastic-recheck: Add query for neutron func fail without logs  https://review.openstack.org/11823100:16
openstackgerritMaru Newby proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1364171  https://review.openstack.org/11823100:19
*** dmorita has joined #openstack-qa00:26
*** marun is now known as marun_afk00:30
openstackgerritMiguel Lavalle proposed a change to openstack/tempest: Migrate test_load_balancer_basic to tempest client  https://review.openstack.org/11823200:30
*** marun_afk is now known as marun00:42
*** dims has joined #openstack-qa00:47
*** yamahata has quit IRC00:55
*** ianw has quit IRC00:57
*** ianw has joined #openstack-qa00:57
*** jamielen- has joined #openstack-qa00:58
*** jamielennox has quit IRC01:00
*** jamielen- is now known as jamielennox01:01
*** jyuso has joined #openstack-qa01:04
*** ianw has quit IRC01:09
*** jamielennox_ has joined #openstack-qa01:10
*** ianw has joined #openstack-qa01:10
*** jamielen- has joined #openstack-qa01:11
*** ianw has quit IRC01:11
*** ianw has joined #openstack-qa01:11
*** jamielen| has joined #openstack-qa01:12
*** jamielennox has quit IRC01:13
*** jamielennox_ has quit IRC01:15
*** jamielen- has quit IRC01:15
*** jamielen| is now known as jamielennox01:18
*** ajo__ has quit IRC01:18
*** amotoki has joined #openstack-qa01:26
*** marun is now known as marun_afk01:26
*** ianw has quit IRC01:28
*** jamielennox_ has joined #openstack-qa01:28
*** dims has quit IRC01:28
*** ianw has joined #openstack-qa01:29
*** amotoki has quit IRC01:31
*** jamielennox has quit IRC01:31
*** jamielennox_ is now known as jamielennox01:40
*** mestery has quit IRC01:48
*** nosnos has joined #openstack-qa01:48
*** marun_afk has quit IRC01:50
openstackgerritMasayuki Igawa proposed a change to openstack-dev/devstack: [WIP] Change output of worlddump to stdout instead of file by default  https://review.openstack.org/11823801:52
*** yamahata has joined #openstack-qa01:53
*** beisner- has joined #openstack-qa01:58
*** beisner has quit IRC01:58
*** beisner has joined #openstack-qa01:58
*** amotoki has joined #openstack-qa02:00
*** mlavalle_ has quit IRC02:07
*** ianw has quit IRC02:09
*** jamielen- has joined #openstack-qa02:10
*** ianw has joined #openstack-qa02:11
*** jamielennox has quit IRC02:13
*** Poornima_ has joined #openstack-qa02:17
*** mlavalle_ has joined #openstack-qa02:21
*** mlavalle_ has quit IRC02:32
*** mlavalle_ has joined #openstack-qa02:32
*** mlavalle_ has quit IRC02:33
*** jamielennox has joined #openstack-qa02:47
*** ianw has quit IRC02:48
*** jamielennox_ has joined #openstack-qa02:48
*** ianw has joined #openstack-qa02:49
openstackgerritMasayuki Igawa proposed a change to openstack/tempest: Migrate test_server_cfn_init to tempest clients  https://review.openstack.org/11771402:49
*** mspreitz has joined #openstack-qa02:51
*** jamielen- has quit IRC02:52
*** jamielennox has quit IRC02:52
*** jyuso has quit IRC02:58
*** ianw has quit IRC03:01
*** jamielen- has joined #openstack-qa03:01
*** ianw has joined #openstack-qa03:01
*** jyuso has joined #openstack-qa03:02
*** jamielennox_ has quit IRC03:05
*** amotoki_ has joined #openstack-qa03:27
*** amotoki_ has quit IRC03:31
*** Madan has joined #openstack-qa03:31
*** xianghuihui has joined #openstack-qa03:39
*** mestery has joined #openstack-qa03:41
*** xianghui has quit IRC03:42
*** nosnos has quit IRC03:44
*** nosnos has joined #openstack-qa03:45
*** jamielennox has joined #openstack-qa03:47
*** jamielennox_ has joined #openstack-qa03:48
*** ianw has quit IRC03:48
*** ianw has joined #openstack-qa03:48
*** nosnos has quit IRC03:49
*** jamielen- has quit IRC03:50
*** jamielennox has quit IRC03:52
*** mestery has quit IRC04:08
*** chandankumar has joined #openstack-qa04:11
*** yfried_ has quit IRC04:11
*** jamielennox has joined #openstack-qa04:14
*** jamielen- has joined #openstack-qa04:15
*** ianw has quit IRC04:16
*** ianw has joined #openstack-qa04:16
*** nosnos has joined #openstack-qa04:17
*** jamielennox_ has quit IRC04:18
*** xianghuihui has quit IRC04:19
*** jamielennox has quit IRC04:19
*** ildikov_ has quit IRC04:33
*** chandankumar has quit IRC04:38
*** chandankumar has joined #openstack-qa04:51
*** Santosh_ has quit IRC04:54
*** yfried_ has joined #openstack-qa05:16
openstackgerritajay proposed a change to openstack/tempest: Improve nova CLI existing tests  https://review.openstack.org/10176705:25
*** ildikov has joined #openstack-qa05:25
*** amotoki_ has joined #openstack-qa05:28
*** amotoki_ has quit IRC05:33
*** Madan has quit IRC05:36
yfried_masayukig: if I want to retrigger the gate check on an approved patch, I need to do "reverify" right?05:37
*** Madan has joined #openstack-qa05:39
openstackgerritYair Fried proposed a change to openstack/tempest: Adds status check for FloatingIP in scenarios  https://review.openstack.org/10270005:47
*** ianw has quit IRC05:50
*** ianw has joined #openstack-qa05:51
*** jamielennox has joined #openstack-qa05:52
*** mspreitz has quit IRC05:53
*** jamielen- has quit IRC05:54
coolsvapyfried_, you need to do recheck only05:55
*** aimon has joined #openstack-qa05:57
*** jyuso has quit IRC06:04
*** jyuso has joined #openstack-qa06:04
*** k4n0 has joined #openstack-qa06:06
openstackgerritA change was merged to openstack/tempest: remove default=None for config options  https://review.openstack.org/9647606:19
*** flaper87|afk is now known as flaper8706:26
*** neeti has joined #openstack-qa06:30
yfried_gmann_: here? are you still working on https://review.openstack.org/#/c/113807/1 ?06:34
*** SridharG has joined #openstack-qa06:34
gmann_yfried: yes, i was waiting for your patch to land. I will rebase this now.06:35
openstackgerritGhanshyam Mann proposed a change to openstack/tempest: Migrate test_large_ops to tempest client  https://review.openstack.org/11380706:36
openstackgerritGhanshyam Mann proposed a change to openstack/tempest: Migrate test_large_ops to tempest client  https://review.openstack.org/11380706:39
gmann_yfried_: done06:40
yfried_gmann_: tnx. I didn't know about this test. gonna add connectivity check to it to make it great06:40
*** cnesa9 has quit IRC06:42
*** nmagnezi has joined #openstack-qa06:44
gmann_yfried_: that present in TestSecurityGroupsBasicOps. do you really think that needs to be added in test_large_ops too ?06:44
yfried_gmann_: secgroup test only does 2 vms. large_ops allows for checking unlimited number of VMs based on conf file, right? this is just a small change in the test and would benefit non-gate testers greatly with large scale tests06:47
*** tkammer has joined #openstack-qa06:47
yfried_gmann_: why does test_large_ops have 3 identical tests?06:48
*** salv-orlando has joined #openstack-qa06:48
gmann_yfried_: that was split out because of timeout issue. gate has per test timeout OS_TEST_TIMEOUT (500 sec)06:50
gmann_yfried_: this tests was written to boot VMs(as per min count) 3 times06:51
*** jamielennox_ has joined #openstack-qa06:52
*** ianw has quit IRC06:52
*** ianw has joined #openstack-qa06:52
*** jamielen- has joined #openstack-qa06:53
gmann_yfried_: regarding adding connectivity check in this- Agree, that can enhance the tests but need to verify it would not cause timeout issue as it has min_count of VM06:53
gmann_based on config06:53
*** ildikov has quit IRC06:54
*** jamielennox has quit IRC06:56
*** jamielennox_ has quit IRC06:57
*** chandankumar has quit IRC07:01
*** openstackgerrit has quit IRC07:02
*** ildikov has joined #openstack-qa07:04
gmann_yfried_: in case CONF.scenario.large_ops_number is very large and we add connectivity check also for each VM then, definitely there can be timeout possibility as those will be running in one test.07:05
gmann_yfried_: what you think?07:05
*** mpaolino has joined #openstack-qa07:09
*** jamielen- is now known as jamielennox07:11
yfried_gmann_: are you talking about the gate timeout?07:12
*** gszasz has joined #openstack-qa07:13
yfried_gmann_: not sure I understand you. do you mean that adding connectivity check to the gate would cause it to timeout on large numbers?07:13
gmann_yfried_: yes07:13
gmann_yfried_: Yes, thats my concern but you can try that if that does not have any timeout issue (but with large number of CONF.scenario.large_ops_number)07:14
gmann_jogo: ^^ what you think on adding connectivity check in test_large_ops ?07:16
raieshi07:16
raiescmyster: hi07:16
cmysterhi raies07:17
raiesmy nova-api start is all time failed07:17
raiescmyster: logs shows follwing07:17
raies-07:17
cmysterjust - ?07:18
raiescmyster: http://pastebin.com/UW7y7b8s07:18
raiesplease check above logs07:18
raiescmyster: did u get that07:19
raies??07:19
cmysterraies: something is very wrong with your setup, see the logs at line 2807:19
*** Longgeek has joined #openstack-qa07:20
raiesI checked that but unable to find out wht is wrong there07:20
*** Longgeek has quit IRC07:20
raiesI tried to find that on google still not sollution not found07:21
cmysteroh ?07:21
cmysterfirst hit on google :) http://stackoverflow.com/questions/21659637/sudo-no-tty-present-and-no-askpass-program-specified-netbeans07:21
cmysterits netbeans, but that is the same symptoms07:22
*** jlanoux has joined #openstack-qa07:22
raiescmyster: any suggestion ??07:22
cmysterso its a system issue. if this sudo is code related, then you hit a bug. the bug being that this specific code cannot work on your system07:22
*** Longgeek has joined #openstack-qa07:23
cmysterI am not sure how you can work around it, but see in that link I sent you if there are any workarounds07:23
cmysterI'd also open a bug here07:23
*** afazekas has joined #openstack-qa07:28
*** amotoki_ has joined #openstack-qa07:28
*** luqas has joined #openstack-qa07:29
raiescmyster: I um unable to solve this problem :(07:33
raiesany help ??07:33
*** amotoki_ has quit IRC07:33
*** catohornet has joined #openstack-qa07:34
raiesafazekas: hi I am facing nova api start failure issue because of  http://pastebin.com/UW7y7b8s can you please help me on this ??07:35
*** raies_ has joined #openstack-qa07:36
yfried_gmann_: large ops only runs in gate gate-tempest-dsvm-neutron-large-ops, right?07:39
*** raies has quit IRC07:39
*** markus_z has joined #openstack-qa07:39
yfried_gmann_: I believe that, assuming the tests are fine, the gate shouldn't have any problem since by the time the 1st connectivity check is done, all other machines SHOULD achieve connectivity.07:42
*** flaper87 is now known as flaper87|afk07:43
*** bkopilov has joined #openstack-qa07:44
cmysterraies_: I am not sure if I can help you with this issue. as I said and added a link to it above, it something that might need to be changed in the code itself. the code from my understanding is running a command that fails. that command if probably not robust enough to be run on your environment, hence its failure. I suggest that you open a bug here with description. If you want to you can try a solution from the link I pasted above07:44
cmysterand change the hard coded command07:44
gmann_yfried_: not sure which all job it runs but might be that only.07:44
gmann_yfried_: hope so,  you can give try to that.07:45
*** rdekel has joined #openstack-qa07:46
*** flaper87|afk is now known as flaper8707:48
*** bkopilov has quit IRC07:49
*** safchain has joined #openstack-qa07:52
*** salv-orlando has quit IRC07:52
*** zz_johnthetubagu is now known as johnthetubaguy07:56
*** dims has joined #openstack-qa07:56
yfried_gmann_: unrelated - are you familiar with scenario "minimum basic ops"?07:59
*** gszasz has quit IRC08:00
*** dims has quit IRC08:00
coolsvapgmann_, hi08:01
*** mestery has joined #openstack-qa08:03
gmann_yfried_: not much idea.08:04
gmann_coolsvap: hi08:04
*** afazekas has quit IRC08:05
*** afazekas has joined #openstack-qa08:06
*** jamielennox is now known as jamielennox|away08:09
*** cnesa9 has joined #openstack-qa08:10
coolsvapgmann_, I just updated the cinder qos patch08:10
*** mkollaro has joined #openstack-qa08:11
coolsvapthe teardown was getting failures due to creds, I think the tearDown involving admin creds need to be in admin tests class08:11
cmysterraies_: now that I think about it, did you make sure that what ever user you are using has full and password-less sudo permissions?08:12
*** morganfainberg is now known as morganfainberg_Z08:12
gmann_coolsvap: ok, I will check that but later :)08:13
coolsvapgmann_, sure08:13
*** cnesa10 has joined #openstack-qa08:20
*** cnesa9 has quit IRC08:22
*** bkopilov has joined #openstack-qa08:22
*** mestery has quit IRC08:22
*** mestery has joined #openstack-qa08:23
*** flaper87 is now known as flaper87|afk08:37
*** jlibosva has joined #openstack-qa08:38
*** gszasz has joined #openstack-qa08:41
*** ZZelle has joined #openstack-qa08:43
*** ildikov_ has joined #openstack-qa08:44
*** andreaf has joined #openstack-qa08:47
*** ildikov has quit IRC08:47
*** flaper87|afk is now known as flaper8708:50
*** mestery has quit IRC08:52
*** mestery has joined #openstack-qa08:52
ZZellecyeoh, hi08:59
*** mwagner_lap has joined #openstack-qa09:04
*** e0ne has joined #openstack-qa09:06
*** luqas has quit IRC09:09
*** qba73 has joined #openstack-qa09:23
*** salv-orlando has joined #openstack-qa09:28
raies_cmyster: yes09:28
*** jlanoux_ has joined #openstack-qa09:29
*** aix has joined #openstack-qa09:29
*** amotoki_ has joined #openstack-qa09:29
*** jlanoux has quit IRC09:30
*** amotoki_ has quit IRC09:34
raies_cmyster: yes what did you get from it ??09:35
*** wuhg has joined #openstack-qa09:38
*** jlanoux has joined #openstack-qa09:39
*** luqas has joined #openstack-qa09:40
raies_afazekas: hi u there ??09:41
*** jlanoux_ has quit IRC09:42
*** tosky has joined #openstack-qa09:58
*** tkammer has quit IRC10:01
*** cdent has joined #openstack-qa10:05
*** e0ne has quit IRC10:12
*** tkammer has joined #openstack-qa10:14
*** e0ne has joined #openstack-qa10:15
*** mestery has quit IRC10:15
*** mestery has joined #openstack-qa10:15
*** Poornima_ has quit IRC10:16
yfried_mtreinish: afazekas: sdague: andreaf: masayukig: https://review.openstack.org/#/c/111719/ this fails to merge. keeps failing on gate-grenade-dsvm-partial-ncpu with sshtimeout10:17
yfried_any ideas? it passed the test fine previously10:17
*** mestery has quit IRC10:21
*** Madan has quit IRC10:21
*** mestery has joined #openstack-qa10:22
*** Madan has joined #openstack-qa10:28
*** jyuso has quit IRC10:28
*** jlanoux_ has joined #openstack-qa10:32
*** evgenyf has joined #openstack-qa10:33
*** jlanoux has quit IRC10:35
*** yamahata has quit IRC10:36
*** tkammer has quit IRC10:40
*** dims has joined #openstack-qa10:46
*** jlanoux has joined #openstack-qa10:49
*** dims has quit IRC10:50
*** jlanoux_ has quit IRC10:52
*** tkammer has joined #openstack-qa10:53
*** marun_afk has joined #openstack-qa11:14
*** amotoki_ has joined #openstack-qa11:15
*** marun_afk is now known as marun11:16
sdaguelifeless: http://logs.openstack.org/09/117609/1/check/gate-requirements-pypy/5bf7145/console.html.gz11:18
sdagueseems like a testtools bug that's biting us a lot11:18
sdagueis that on your radar?11:18
*** e0ne has quit IRC11:19
*** sergeysh has joined #openstack-qa11:19
lifelesssdague: fixed it a few days ago11:19
sdagueis there a release out with it?11:20
lifelesssdague: two releases since11:20
sdagueok, cool11:20
sdaguecan you close the bug? I was just trying to figure out if it was worth rechecking a patch11:20
lifeless0.9.38 broke things, 0.9.39 fixed it about 30m later11:20
lifelesssdague: whats the bug #?11:21
sdague136290911:21
sdaguedoes that not show up on your testtools list?11:21
*** amotoki has quit IRC11:22
lifelesswas filed after the fix I think11:22
*** amotoki_ is now known as amotoki11:22
sdagueok, cool11:25
*** mwagner_lap has quit IRC11:25
*** dims has joined #openstack-qa11:42
*** neeti has quit IRC11:43
*** zz_dimtruck is now known as dimtruck11:44
*** mestery has quit IRC11:46
*** k4n0 has quit IRC11:46
*** neeti has joined #openstack-qa11:48
cdentsdague: when you have a moment can I get your eyes on my ceilo+javelin changes: https://review.openstack.org/#/c/102354/11:52
cdentAfter a lot of discussion with jogo we came up with a compromise that seems to make everyone somewhat happy, and now we'd like to get it out the door.11:52
*** wuhg has quit IRC11:52
sdaguecdent: +211:55
cdentcool, thanks11:55
*** rdekel has quit IRC11:55
cdentit's been a long road, who else can I ping to move things along11:55
*** weshay has joined #openstack-qa11:56
*** mmedvede has joined #openstack-qa11:58
*** SridharG has left #openstack-qa12:00
*** tkammer has quit IRC12:02
*** dpaterson has joined #openstack-qa12:05
*** mestery has joined #openstack-qa12:06
*** mpaolino has quit IRC12:06
*** dims has quit IRC12:08
*** dims has joined #openstack-qa12:08
*** Madan has quit IRC12:08
*** dmorita has quit IRC12:12
*** mestery_ has joined #openstack-qa12:12
*** mestery has quit IRC12:13
*** rdekel has joined #openstack-qa12:14
*** tkammer has joined #openstack-qa12:17
*** yfried_ has quit IRC12:17
*** mmedvede has quit IRC12:20
*** mpaolino has joined #openstack-qa12:21
*** aysyd has joined #openstack-qa12:24
ZZellesdague, hi12:25
*** adalbas has joined #openstack-qa12:25
ZZellesdague,  could you have a look to https://review.openstack.org/103498 and https://review.openstack.org/104526 ? They are neutron tempest tests12:26
*** mwagner_lap has joined #openstack-qa12:29
*** mmedvede has joined #openstack-qa12:33
*** mestery has joined #openstack-qa12:35
*** mestery_ has quit IRC12:35
*** e0ne has joined #openstack-qa12:39
*** masayukig has quit IRC12:44
*** neeti has quit IRC12:46
*** masayukig has joined #openstack-qa12:51
*** dustins has joined #openstack-qa13:04
*** pcrews has joined #openstack-qa13:04
*** ildikov_ has quit IRC13:04
*** tkammer has quit IRC13:06
*** flaper87 is now known as flaper87|afk13:06
*** ildikov has joined #openstack-qa13:07
*** yfried_ has joined #openstack-qa13:14
dimsAny devstack folks around? seeing a weird failure trying to add some new oslo.libs - https://review.openstack.org/#/c/117304/13:18
*** tkammer has joined #openstack-qa13:18
*** dimtruck is now known as zz_dimtruck13:19
mtreinishdims: http://logs.openstack.org/04/117304/2/check/check-tempest-dsvm-full/915f806/logs/devstacklog.txt.gz#_2014-09-01_14_45_25_645 doesn't that mean oslo.concurrency isn't on the nodepool image?13:21
*** jecarey_ has quit IRC13:21
andreafmtreinish, ping - did you have a change to look at testresources for test-accounts bp yet?13:22
*** zz_jgrimm is now known as jgrimm13:22
andreafs/change/chance13:22
dimsmtreinish: ah i see let me hunt where to add stuff to nodepool images13:22
mtreinishandreaf: yeah I took a brief look, I think it would end up being too much extra work13:22
mtreinishit doesn't exactly suit our needs so we'd have to hack around it a bit13:23
mtreinishI think going your approach might be the best13:23
andreafmtreinish: the PoC with splitting setupClass in parts you mean?#13:25
andreafmtreinish, so I'm not sure if we have to hold off the test accounts bp on that13:26
mtreinishandreaf: yeah13:26
andreafmtreinish: not all tests will work fine but that's because of the tests not because of the test-accounts code13:26
mtreinishandreaf: I guess we could push forward with landing the test-accounts code13:27
andreafmtreinish: what I'd do is remove the huge change adding safe_setup everywhere13:27
andreafmtreinish, and push forward the rest13:28
mtreinishbut if it's not useable because of bugs around setupclass, I'm not sure of the value13:28
andreafmtreinish, the only concern is that people running without tenant isolation would have a problem then13:28
andreafmtreinish: it will be usable on a subset of the tests at least - we can have an experimental job and fix all the issues in tests in small chunks13:29
mtreinishandreaf: sure I guess that's fine, I just don't think running on a subset of tests is particularly useful. Especially because I doubt we'll be able to map out that set.13:30
andreafmtreinish, what I mean it run on all tests in an experimental job that will fail - and fix issues later13:31
andreafmtreinish, until all tests passes13:31
mtreinishyeah I think that's probably fine, but I just don't think it'll be something useful for other people until we fix issues13:32
andreafmtreinish, yes I agree13:32
andreafmtreinish, the other way would be to start fixing tests to make sure cleanup of credentials is always called13:32
andreafmtreinish, and only when all are done merge the test-accounts code13:33
andreafmtreinish, that would be a separate bp I guess13:33
mtreinishandreaf: yeah, probably an essential prio one actually13:34
mtreinishI think if we land the test-accounts code now, we just have to make it clear that it probably won't work13:34
*** mriedem has joined #openstack-qa13:35
andreafmtreinish, and we break the no-tenant-isolation case, don't we?13:35
mtreinishandreaf: oh, yeah we can't do that13:36
andreafmtreinish: I'll put down a spec about ensuring resource / credentials cleanup - I think that's the right place for people to review our ideas around this13:36
andreafmtreinish: once it's approved we could make a small task force like for scenario migration to get the bp done13:37
*** dkranz has quit IRC13:37
*** malini1 has joined #openstack-qa13:39
mtreinishandreaf: yeah that sounds good13:39
mtreinishhopefully the timing will work out for Juno13:39
gmann_mtreinish: Can you please have look into - https://review.openstack.org/#/c/100132/ .  As long as its there it need rebase  every time:)13:46
*** timello_ has joined #openstack-qa13:48
*** toure has joined #openstack-qa13:49
*** sc68cal_ is now known as sc68cal13:55
*** markus_z has quit IRC13:55
*** marun has quit IRC13:59
mtreinishgmann_: ok +A13:59
gmann_mtreinish: thanks14:01
*** marun has joined #openstack-qa14:01
mtreinishgmann_: I think after that lands you should probably propose that as a new rule for hacking itself14:02
gmann_mtreinish: hacking rule has been added in this patch only. or you mean some other rule?14:04
mtreinishgmann_: I mean add the rule to: http://git.openstack.org/cgit/openstack-dev/hacking/14:05
gmann_mtreinish: oh i see. yes i will14:05
*** zz_dimtruck is now known as dimtruck14:05
*** openstackgerrit has joined #openstack-qa14:06
*** rbak has joined #openstack-qa14:06
*** amotoki has quit IRC14:07
*** tosky has quit IRC14:13
*** bkopilov has quit IRC14:16
*** raies_ has quit IRC14:18
openstackgerritSwami Reddy proposed a change to openstack/tempest: Add "reboot_instance" EC2 API test case  https://review.openstack.org/11594414:20
*** mrodden has quit IRC14:26
*** mrodden has joined #openstack-qa14:27
openstackgerritAndrea Frittoli  proposed a change to openstack/qa-specs: add bp:resource-cleanup  https://review.openstack.org/11835214:27
*** malini1 has quit IRC14:27
*** malini1 has joined #openstack-qa14:27
andreafmtreinish, ^^^14:28
andreafhttps://blueprints.launchpad.net/tempest/+spec/resource-cleanup / https://review.openstack.org/11835214:28
*** bkopilov has joined #openstack-qa14:29
mtreinishandreaf: at a quick glance it looks good14:29
mtreinishI'll do a more detailed review this afternoon14:29
andreafmtreinish, ok thank you14:29
*** andreaf has quit IRC14:31
*** yamahata has joined #openstack-qa14:31
openstackgerritCedric Brandily proposed a change to openstack/tempest: Add multi-provider extension positive test cases  https://review.openstack.org/10452614:31
*** mestery_ has joined #openstack-qa14:31
openstackgerritCedric Brandily proposed a change to openstack/tempest: Add provider extension positive test cases  https://review.openstack.org/10349814:31
*** SridharG has joined #openstack-qa14:32
openstackgerritCedric Brandily proposed a change to openstack/tempest: Add partial specs (multi-)provider extension positive test cases  https://review.openstack.org/10523414:32
*** mestery has quit IRC14:33
*** sballe has joined #openstack-qa14:35
*** mestery_ is now known as mestery14:35
*** nmagnezi has quit IRC14:37
*** toure has quit IRC14:39
ZZellemtreinish, hi14:39
*** toure has joined #openstack-qa14:40
*** dkranz has joined #openstack-qa14:41
*** mspreitz has joined #openstack-qa14:41
Shrewsdkranz: do you have some time today to see if https://review.openstack.org/112660 is worthy of a second +2?14:42
dkranzShrews: let me look14:42
Shrewsdkranz: thanks much!14:42
mtreinishZZelle: hi14:43
ZZellemtreinish,  could you have a look to https://review.openstack.org/103498 and https://review.openstack.org/104526 ? They are neutron tempest tests14:44
mtreinishZZelle: sure, I'll put them on my list for reviews today14:44
ZZellemtreinish, and finally https://review.openstack.org/105234 which is less important (only supported by neutron with ML2 plugin)14:44
ZZellemtreinish, thanks14:45
*** wuhg has joined #openstack-qa14:45
*** ildikov_ has joined #openstack-qa14:46
*** jecarey has joined #openstack-qa14:48
*** ildikov has quit IRC14:49
*** mpaolino has quit IRC14:56
*** bkopilov has quit IRC14:59
*** atiwari has joined #openstack-qa15:00
*** markus_z has joined #openstack-qa15:03
*** yamahata has quit IRC15:03
*** yamahata has joined #openstack-qa15:03
*** nosnos has quit IRC15:06
*** nosnos has joined #openstack-qa15:07
*** sballe_ has joined #openstack-qa15:08
*** sballe has quit IRC15:08
*** yamahata has quit IRC15:11
openstackgerritZhu Zhu proposed a change to openstack/tempest: Allow out of quota failure status code to be 413 or 403  https://review.openstack.org/11809915:11
*** nosnos has quit IRC15:11
*** mspreitz_ has joined #openstack-qa15:12
*** markmcclain has joined #openstack-qa15:15
*** markmcclain has quit IRC15:16
*** mspreitz has quit IRC15:16
*** markmcclain has joined #openstack-qa15:18
*** jecarey_ has joined #openstack-qa15:23
openstackgerritBryan Havenstein proposed a change to openstack-dev/devstack: Added simple autodetection for 32/64 bit Cirros image arch  https://review.openstack.org/11787715:26
openstackgerritJim Rollenhagen proposed a change to openstack-dev/devstack: Fix ironic-python-agent tarball unpack  https://review.openstack.org/11738415:28
*** aimon has quit IRC15:29
*** kmartin has joined #openstack-qa15:30
*** SridharG has quit IRC15:30
jrollsdague: responded to your comments on https://review.openstack.org/#/c/117384/15:31
*** annegentle has quit IRC15:32
*** mspreitz has joined #openstack-qa15:33
*** mspreitz_ has quit IRC15:34
*** evgenyf has quit IRC15:45
mkollaroI'm experiencing some very weird behavior with the tempest.config module - when I run this code https://gist.github.com/mkollaro/c1ee2aacd7352a1b655f , I don't get the exception - the program just prints the keys and ends15:45
mkollaroyet if I comment out line #4, the exception traceback is printed normally15:46
mkollaro(the file is in the tools dir)15:47
*** tkammer has quit IRC15:48
*** cdent_ has joined #openstack-qa15:51
*** markus_z has quit IRC15:51
mtreinishmkollaro: check the log file, the default log settings capture exceptions and print them in the logfile instead of stdout or stderr15:51
mtreinishby running a getattr on the config object you're triggering the config file to be parsed if it already wasn't15:52
mtreinishwhich would load those logging settings15:52
mtreinishalthough I thought we fixed that default behavior15:52
mkollaromtreinish: can I disable that behavior? I just want to find out what are some options in the config...15:52
*** Longgeek has quit IRC15:54
mtreinishmkollaro: those config options are defined here: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/config.py#n4715:54
*** cdent has quit IRC15:54
*** cdent_ is now known as cdent15:54
*** sleipneir012 has joined #openstack-qa15:54
*** marun_ has joined #openstack-qa15:55
mkollaromtreinish: ok, I'll try it with that...thanks :)15:55
mtreinishmkollaro: but if you need to get the log messages on the console just set: use_stderr True in the default section15:56
*** gszasz has quit IRC15:56
*** andreaf has joined #openstack-qa15:57
*** marun has quit IRC15:57
*** sballe_ has quit IRC15:58
*** rdekel has quit IRC15:58
*** eglynn_ has joined #openstack-qa15:59
*** afazekas has quit IRC15:59
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Fix use of nonexistent class variable in accounts  https://review.openstack.org/11463616:01
openstackgerritMatthew Treinish proposed a change to openstack/tempest: Fix use of nonexistent class variable in accounts  https://review.openstack.org/11463616:01
*** cbader has joined #openstack-qa16:02
eglynn_jogo: good morning sir!16:02
*** andreaf has quit IRC16:04
eglynn_jogo: I'd guess cdent has already been holding your feet over hot-coals to get a vote one-way-or-the-other on his now simplified grenade patch?16:04
eglynn_jogo: ... since Sean has given it his blessings, t'would be great to have your input also on https://review.openstack.org/10235416:04
*** sballe has joined #openstack-qa16:04
*** sleipneir012 has quit IRC16:08
*** davidlenwell_ is now known as davidlenwell16:08
*** morganfainberg_Z is now known as morganfainberg16:11
*** davidlenwell is now known as meatpopcicle16:15
*** meatpopcicle is now known as davidlenwell16:15
*** mestery_ has joined #openstack-qa16:16
*** sballe has quit IRC16:17
*** andreaf has joined #openstack-qa16:17
*** andreaf has quit IRC16:18
*** andreaf has joined #openstack-qa16:18
*** mestery has quit IRC16:19
*** jlanoux has quit IRC16:25
*** yjiang5 has joined #openstack-qa16:25
*** mestery has joined #openstack-qa16:26
*** mestery_ has quit IRC16:30
*** marun_ has quit IRC16:30
*** tkammer has joined #openstack-qa16:31
*** safchain has quit IRC16:34
*** gema_ has quit IRC16:34
*** gema has joined #openstack-qa16:35
*** qba73 has quit IRC16:37
*** ildikov_ has quit IRC16:38
*** gema has quit IRC16:38
*** jlibosva has quit IRC16:39
*** e0ne has quit IRC16:42
*** suneelb has joined #openstack-qa16:52
*** luqas has quit IRC16:55
*** sballe has joined #openstack-qa16:55
*** sballe_ has joined #openstack-qa16:59
*** sballe has quit IRC17:03
yfried_andreaf: sdague: dkranz: masayukig: mtreinish: https://review.openstack.org/#/c/111719/ needs Approved. was approved but failed and needed rebase17:03
sdagueyfried_: done17:04
*** harlowja has joined #openstack-qa17:04
*** sballe has joined #openstack-qa17:04
*** dencaval has joined #openstack-qa17:05
yfried_sdague: tnx. hope this one would go through17:05
*** sballe_ has quit IRC17:07
*** SridharG has joined #openstack-qa17:11
jogoeglynn_: done, although I am not tempest core17:11
*** SridharG has quit IRC17:11
eglynn_jogo: thanks! I see, I was thinking of your grenade core-ness17:13
* eglynn_ is puzzled why javelin isn't under the grenade repo17:14
mtreinisheglynn_: mostly because it depends on the tempest rest clients, which don't expose a stable interface yet17:16
mtreinishonce the tempest lib work is done I expect we'll actually make it a separate project17:16
eglynn_mtreinish: a-ha, ok, a 3rd project separate from both tempest and grenade?17:17
*** bnemec has quit IRC17:17
mtreinishprobably, besides the packaging concerns of putting it in grenade. I'm thinking it might be a generally useful tool by itself17:17
eglynn_mtreinish: cool thanks for the clarification17:18
*** bnemec has joined #openstack-qa17:18
eglynn_mtreinish: ... in the meantime can I take the liberty of adding you as a reviewer of that javelin patch?17:19
eglynn_mtreinish: https://review.openstack.org/10235417:19
*** torandu has quit IRC17:19
mtreinisheglynn_: I already +A'd it17:19
eglynn_mtreinish: thank you sir! :)17:20
*** torandu has joined #openstack-qa17:20
*** sballe has quit IRC17:20
*** sballe has joined #openstack-qa17:24
yfried_mtreinish: sdague: http://status.openstack.org/zuul/ Seems like the gate is overloaded/stuck. am I right? just wondering...17:25
dencavalhey guys, i'm new to tempest and I have a question:17:27
mtreinishyfried_: it's Juno 3 this week things will be slow17:27
dencavalIs it enough to run "run_tempest.sh tempest.api.object_storage"? Or should I also run stress tests as example to cover Swift deployment?17:28
yfried_mtreinish: sdague: https://review.openstack.org/#/c/112660/ this is going to damage this https://review.openstack.org/#/c/111719/ from merging. since the later is crucial part of a bp that's due this week, any chance of stopping it from merging today?17:29
*** eglynn_ has quit IRC17:29
yfried_mtreinish: sdague: the later is blocking all network scenario from client migration17:30
mtreinishdencaval: it depends on what exactly you're trying to test. The stress tests just loop over N workers tempest test cases to generate load for a set period of time17:31
mtreinishdencaval: but if you just want to make sure your new swift deployment is working. Just running the api tests to verify that the api functions work correctly should be fine17:32
*** sballe_ has joined #openstack-qa17:32
*** sballe_ has quit IRC17:32
*** aix has quit IRC17:35
*** e0ne has joined #openstack-qa17:37
*** SridharG has joined #openstack-qa17:41
*** wuhg has quit IRC17:42
*** e0ne has quit IRC17:48
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add order column to run table  https://review.openstack.org/11786117:49
*** bkopilov has joined #openstack-qa17:49
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add subunit2sql-db-manage utility  https://review.openstack.org/11841817:49
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add session objects for get_all* db api calls  https://review.openstack.org/11841917:49
*** mikal_ is now known as mikal17:50
openstackgerritMatthew Treinish proposed a change to openstack-infra/subunit2sql: Add db api function to get the latest run  https://review.openstack.org/11788217:51
*** gema has joined #openstack-qa17:51
*** e0ne has joined #openstack-qa17:52
*** aimon has joined #openstack-qa17:54
openstackgerritMatt Riedemann proposed a change to openstack/tempest: POC: Add list-networks test for os-networks API  https://review.openstack.org/11294417:57
*** coolsvap has quit IRC18:00
*** SridharG has left #openstack-qa18:00
openstackgerritBryan Havenstein proposed a change to openstack-dev/devstack: Added simple autodetection for 32/64 bit Cirros image arch  https://review.openstack.org/11787718:03
*** e0ne has quit IRC18:04
yfried_mtreinish: sdague: dkranz: andreaf: asking again if we can stop https://review.openstack.org/#/c/112660/ from merging18:05
yfried_^^^18:05
*** mspreitz_ has joined #openstack-qa18:07
dkranzyfried_: You made a comment three weeks ago. Why is it so important to stop it?18:07
*** mspreitz has quit IRC18:08
*** jamielennox|away is now known as jamielennox_18:09
yfried_dkranz: regardless of my comment, it's gonna stop  https://review.openstack.org/#/c/111719/  from merging.18:09
*** coolsvap has joined #openstack-qa18:09
openstackgerritSergey Shnaidman proposed a change to openstack/tempest: Add possibility to create subnet without gateway  https://review.openstack.org/11842418:10
yfried_dkranz:  https://review.openstack.org/#/c/111719/  is crucial for scenario-client bp and blocking all network scenario from migrating. deadline for that bp is this week18:10
yfried_dkranz: https://review.openstack.org/#/c/112660/ can wait until all scenarios migrate18:11
*** e0ne has joined #openstack-qa18:12
*** harlowja has quit IRC18:12
*** harlowja has joined #openstack-qa18:13
*** mspreitz_ has quit IRC18:13
openstackgerritEmily Hugenbruch proposed a change to openstack/tempest: Add services tags to tempest/cli  https://review.openstack.org/11106818:14
openstackgerritEmily Hugenbruch proposed a change to openstack/tempest: Add services tags to tempest/cli  https://review.openstack.org/11106818:16
*** e0ne has quit IRC18:17
dkranzShrews: ping18:18
openstackgerritRob Crittenden proposed a change to openstack-dev/devstack: Configure endpoints to use SSL natively or via proxy  https://review.openstack.org/9885418:19
yfried_dkranz: is my reson legit?18:21
dkranzyfried_: I am not sure but I blocked it18:21
yfried_dkranz: tnx18:21
dkranzmtreinish, sdague : I am not sure what to do about the conflicting changes we approved as a group . I put a -2 on https://review.openstack.org/#/c/112660/ until we can sort it out.18:22
dkranzShrews: ^^^18:23
sdagueif you are going to kick something out, please update the commit message18:23
sdagueso it drops out of the gate18:23
sdagueotherwise it runs to the end, resets, and adds an hour to everyone's run time18:23
dkranzsdague: Didn't know that. Is there a specific string to put in the message?18:23
sdagueno, I just usually explain what I did and why18:23
sdaguea commit message change counts as a new upload18:24
dkranzsdague: ok, thanks18:24
sdagueso it pulls from the system, goes back to the check queue18:24
*** cdent has quit IRC18:24
*** johnthetubaguy is now known as zz_johnthetubagu18:26
*** dmellado has quit IRC18:28
yfried_dkranz: this as well: https://review.openstack.org/#/c/100132/18:28
yfried_dkranz: a matter of priorities.18:28
*** dmellado has joined #openstack-qa18:29
yfried_dkranz: prettier code vs bp deadline. not my call, though18:29
*** gema has quit IRC18:29
*** gema has joined #openstack-qa18:31
*** mkollaro has quit IRC18:32
*** suneelb has quit IRC18:33
*** suneelb has joined #openstack-qa18:34
*** davidlenwell has quit IRC18:36
*** davidlenwell has joined #openstack-qa18:36
*** suneelb has quit IRC18:39
*** suneelb has joined #openstack-qa18:39
*** afazekas has joined #openstack-qa18:40
*** dimtruck is now known as zz_dimtruck18:43
mtreinishdkranz: I think the deadline for the ironic fix is harder than the tempest client migration...18:44
*** adam_g` is now known as adaM_g18:44
*** adaM_g is now known as adam_g18:44
adam_gyea, was just going to note. ironic has some QA requirements to meet with deadlines as well18:45
dkranzmtreinish: So you think I should put back my +2?18:45
sdaguedtroyer: you about?18:45
Shrewsdkranz: back now... catching up18:45
mtreinishdkranz: yeah, mostly because the ironc ci requirement is hard on thurs IIUC, but the client migration stuff we can merge after thurs18:45
dtroyersdague: just back from lunch18:45
dkranzShrews: So we approved multiple conflicting patches based on bp deadlines18:46
Shrewsfun!18:46
sdaguedtroyer: where is the patch for handling the service stop bits correctly?18:46
dkranzmtreinish: I think one issue is that the non-ironic changes are much harder to rebase. Or so yfried_ said.18:46
sdagueI'm going to dig in on that18:47
adam_gsdague, any chance of a +A @ https://review.openstack.org/#/c/107814/ ? needed a rebase after it was approved last week but has been languishing since18:47
sdagueadam_g: you realize the bug # in the code is wrong, right?18:48
*** zz_dimtruck is now known as dimtruck18:48
sdaguecan we get that fixed?18:48
adam_gdoh18:48
dtroyersdague: the last review is https://review.openstack.org/#/c/117339/ but I've come to the conclusion that we're running things wrong and am working on a parallel set of functions that do things differentlt18:48
sdaguethat was why masayuki -1ed18:48
yfried_dkranz: mtreinish: Shrews: they are. since they touch a lot of code all around, so it's almost never a trivial rebase18:49
sdaguedtroyer: ok, that's fine18:49
dtroyersdague: the basic problem is that with the current fashion of requiring a shell to process the command arg to screen_it we'll never get the right pid18:49
openstackgerritAdam Gandelman proposed a change to openstack-dev/devstack: Set flavor id for Ironic's baremetal flavor  https://review.openstack.org/10781418:49
*** flaper87|afk is now known as flaper8718:49
sdaguedtroyer: right, agreed18:49
sdaguedtroyer: so what's your new approach?18:49
adam_gdkranz, mtreinish: theres another network scenario that needs a change for ironic https://review.openstack.org/#/c/113707/ that will probably cause the same pain if it races with the client migration https://review.openstack.org/#/c/113707/18:50
dtroyersdague: its similar but removing all of the cd commands, but also splitting out the screen bits so we can call that directly for things that really don't have a purpose without screen, such as tailing a log file.18:51
*** cnesa10 has quit IRC18:51
dkranzmtreinish: I'm not really sure what to do here. It would have been better to coordinate this up front but here we are.18:51
sdagueok, cool. Well if you need help, let me know. Got to go pick up veggies right now, back in a bit18:51
mtreinishadam_g, Shrews, yfried_: can someone try a rebase locally to see if we can build a working tree of everything18:52
mtreinishthen we can just fast path them all through18:52
dkranzmtreinish: Why does tempest need hard deadlines based on releases to merge something?18:52
mtreinishdkranz: we don't, that was my argument for holding up the client refactor patches18:53
mtreinishbut ironic needs to be able to run tempest for it's ci requirements18:53
adam_gmtreinish, ill give it a poke now18:53
dkranzadam_g: Thanks!18:53
yfried_Shrews: this is kinda late in the game, but could you just add to setUp: "if CONF.ironic: return"18:54
yfried_Shrews: that way you prevent setup from running on ironic skipped tests and the change is much much smaller18:54
dkranzadam_g: So if you are going to do this I should remove the -2 from the ironic. Basically you are going to do the rebase of the other two patches, right?18:55
adam_gdkranz, i would start with yfried's client migration and rebase the other two on ehtat18:55
yfried_adam_g: tnx18:55
dkranzadam_g: So we need to prevent the two ironic patches from merging at the moment18:56
adam_gdkranz, 113707 is unapproved, 112660 has your -218:57
dkranzadam_g: ok, I'll upload a new commit message to kick it out as Sean suggested18:57
*** rockyg has joined #openstack-qa18:57
yfried_Shrews: ?18:58
openstackgerritDavid Kranz proposed a change to openstack/tempest: Fix network/secgroup scenario tests for ironic  https://review.openstack.org/11266018:59
Shrewsadam_g: are you rebasing 112660 then?18:59
adam_gShrews, yes18:59
Shrewsack18:59
adam_gShrews, assuming its okay with you?18:59
Shrewsadam_g: go for it18:59
*** sweston_ is now known as sweston18:59
yfried_adam_g: if my patch is the 1st, then you don't need to re-submit it, am I right?19:00
Shrewsadam_g: i was switching computers so you beat me to it19:00
dkranzadam_g, Shrews , yfried_ : ok, https://review.openstack.org/#/c/111719/ is not out of the gate queue and clean. Thanks guys.19:00
adam_gyfried_, resubmit yours? no, shouldn't19:00
dkranzI mean "now out of the gate queue"19:00
yfried_adam_g: dkranz: tnx.19:00
yfried_dkranz: we guessed19:00
yfried_Shrews: did you see my question (since you were on and off line)19:01
Shrewsyfried_: no19:01
yfried_[21:54] <yfried_> Shrews: this is kinda late in the game, but could you just add to setUp: "if CONF.ironic: return"19:01
yfried_[21:54] <yfried_> Shrews: that way you prevent setup from running on ironic skipped tests and the change is much much smaller19:01
yfried_Shrews: what do you think?19:02
adam_gyfried_, we do not want to skip the entire class, only some subset of the tests19:02
*** aimon has quit IRC19:02
*** suneelb has quit IRC19:02
*** suneelb has joined #openstack-qa19:03
Shrewsright, what adam_g said. we can skip the entire sec group tests, but not the others19:04
yfried_adam_g: Shrews: then change the condition to suite. I'd still like to keep the setUp method, as this is basic part of the python unitttest19:04
adam_gyfried_, which test class are you talking about?19:04
adam_gthe changes here? https://review.openstack.org/#/c/112660/13/tempest/scenario/test_network_basic_ops.py19:05
yfried_Shrews: if I'm not mistaken, your concern was that the setUp is executed even when the test is skipped and at great cost, so you moved it to be explicitly executed at the beginning of both tests in network_basic_ops19:05
yfried_Shrews: adam_g: hang on - I realize I'm wrong19:06
Shrewstbf, that was adam's change, but that's the reason, yes.19:06
*** yjiang5 has quit IRC19:07
yfried_Shrews: adam_g: dammit! I can't find an elegant solution to the problem and the only solution we have feels like nails on the chalkboard of my soul19:08
*** aimon has joined #openstack-qa19:08
*** marun_ has joined #openstack-qa19:08
adam_gyfried_, only creating servers when we actually need them?19:08
mtreinishadam_g: this might be relevant: https://github.com/testing-cabal/testtools/commit/9953dbd6d9f0de0d51b6354e33091689063b177c19:09
mtreinishI saw it go through the other day19:09
mtreinishbut it doesn't solve any issues in the meantime19:09
mtreinishbecause we'd still need a new release to pick it up19:09
mtreinishoh wait 1.0.0 was pushed not too long ago19:10
yfried_mtreinish: so it solves the issue, or not? I'm hanging on a thread here man :)19:10
*** jecarey has quit IRC19:12
Shrewsmtreinish: you'd need to update tempest requirements.txt first to make that the minimum (and hope 1.0.0 doesn't break other things)  :)19:12
mtreinishyfried_: I'm not sure what which issue you're discussing, I don't have any issues with that patch19:12
mtreinishyfried_: I just saw the comment about setUp being run on skip and I remembered seeing a patch to fix that19:12
mtreinishShrews: yeah, we'd have to get the min version pin into g-r first too, which will cause other issues too19:13
mtreinishbut we'll at least be running with it by tomorrow (assuming it didn't already get picked up by the mirrors)19:13
Shrewsi'm glad to see that fixed though19:14
yfried_Shrews: are you ok with trying this out first?19:14
Shrewstrying what out?19:14
yfried_Shrews: new testtools19:15
*** Ch00k has joined #openstack-qa19:15
Shrewsthat seems excessive as a first step. let's see what the rebases do19:16
mtreinishyfried_: that also can't really happen until kilo, this week is dep freeze so we couldn't land the requirements pin until that opens up again19:17
*** aimon has quit IRC19:22
jogomriedem: do you know why we have a spike in https://bugs.launchpad.net/grenade/+bug/134961719:23
mriedemjogo: neutron full jobs?19:24
mriedemdidn't some neutron jobs get out of the experimental queue recently?19:25
*** malini1 has quit IRC19:25
mtreinishmriedem: yeah like 2 weeks ago it became voting everywhere19:25
*** dmellado has quit IRC19:26
*** aimon has joined #openstack-qa19:26
jogomriedem: hmm good guess, I'll see what jobs its hitting19:26
*** yjiang5 has joined #openstack-qa19:26
jogomriedem: yup neutron-full19:26
mriedemjogo: you'll see salv-orlando has some notes starting in the bug report on 8/2419:26
jogosalv-orlando: ^19:26
mriedemjogo: yeah, sounds like metadata service19:27
jogomriedem: for https://bugs.launchpad.net/nova/+bug/132365819:27
jogocan we mark that as confirmed critical for nova19:27
jogoits the #2 bug19:28
jogoohh also neutron19:28
jogowow neutron full is failing all over19:28
* jogo goes over to neutron room to complain19:29
mtreinishjogo: heh, well we kind of knew that was going to happen19:29
mtreinishit's a repeat of last year's feature freeze when we turned on parallel everywhere else19:30
*** dmellado has joined #openstack-qa19:30
jogomtreinish: heh yeah19:30
jogowe did but maybe we mark as non-voting for now?19:30
jogoto reduce the number of gate resets etc19:30
jogowell at least general rechecks19:30
salv-orlandomriedem, jogo, mtreinish: I’m afraid that bug is not neutron specific. I have not been able to reproduce it yet, but it has to do with user data not being pushed into the instance.19:30
jogosalv-orlando: https://bugs.launchpad.net/nova/+bug/1323658 you mean?19:31
salv-orlandojogo: nope sorry 134961719:31
mriedemi've always wondered if the ec2 tests running around the same time are causing issues with the scenario tests19:32
jogosalv-orlando: so marking this as a confirmed critical nova bug then19:32
salv-orlandofor 1323658 it’s different. there is no auth failure - it’s a connection refused… I’ve noticed however that in some cases cirros boots in readonly mode19:32
salv-orlandothe interface gets correctly the IP address19:32
salv-orlandobut then sshd does not start. I opened a different bug for that case but I wonder if we’re not witnessing the same thing here.19:33
mtreinishmriedem: yeah there could be an interaction there, I wouldn't be that surprised19:34
jogosalv-orlando: going back to 134961719:34
jogoit sounds like it isn't just a neutron thing so turning off the neutron-full jobs would just hide the issue19:34
salv-orlandojogo: correct.It happens in our 3rd party CI as well.19:35
salv-orlandoalso grenade-partial-ncpu fails as well19:35
salv-orlandowe merged something on aug 21st/22nd which made this bug much worse19:35
salv-orlandobut I could not trace it to any specific commit.19:35
salv-orlandoSo I think that one of those commit altered things enough to make the bug surface. It could have been something as simple as a timing change in tempest.19:36
jogosdague: how hard would it be to add a server side string grep to os-loganalyze19:36
jogosalv-orlando: with neutron, the nova-metadata service is proxies via neutron right?19:38
mtreinishjogo: can't use just use a browser search tool19:38
jogomtreinish: when its a 18MB file over coffee shop wifi19:39
mtreinishjogo: ahh, ok that makes sense...19:40
salv-orlandojogo: correct. But before we go there I actually checked the failure rate with configdrive is a lot higher19:40
salv-orlandoindeed pg jobs fail less frequently than mysql jobs.19:40
salv-orlandoI think pg==metadata mysql==configdrive, is that correct?19:41
salv-orlandoso I’m trying to root cause the issue on a race in the instance metadata cache.19:41
salv-orlandoBut have found nothing so far.19:41
mtreinishsalv-orlando: yep: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#n43519:41
jogomtreinish salv-orlando: hmm19:42
jogosalv-orlando: if this is happening on partial-ncpu that is because something on the compute side probablly (which is 'old')19:44
salv-orlandowhat do you mean old?19:45
mtreinishsalv-orlando: icehouse19:45
mtreinishthat job keeps n-cpu on icehouse for the whole run19:45
jogosalv-orlando: grenade goes from 'old' to 'new' and old here is icehouse19:45
salv-orlandojogo, mtreinish: thanks got it.19:45
adam_gmtreinish, dkranz, yfried_ Shrews i have our patches rebased on the client migration. will push up in a bit19:46
salv-orlandoso could it be that we’re seeing the same failure mode but the grenade-partial-ncpu failue has a different root cause?19:46
jogosalv-orlando: correct19:46
salv-orlandojogo: also melwitt suspects there is actually no metadata issue whatsoever. Did you read her comment on the bug report?19:52
jogosalv-orlando: hmm just read it19:54
jogothat is possible19:54
salv-orlandothe MTU thing?19:54
jogoand explains why this is happening in neutron only19:54
jogodifferent network stack19:54
sdaguejogo: re: server side string grep.... I was actually headed down that path19:56
sdagueit actually should be quite easy in the current architecture, I think we have a pretty good set of layer abstractions in the filter chain19:56
*** morganfainberg is now known as needs19:57
*** needs is now known as needscoffee19:57
openstackgerritMikhail S Medvedev proposed a change to openstack/tempest: Skip orchestration tests if Neutron is not available  https://review.openstack.org/11845419:58
jogois there a way to run cloud-init with even more logging?19:59
salv-orlandojogo: melwitt also pointed to a bug for this MTU issue in cirros. Fix has been committed but afaict there has no been any new cirros build since it merged.20:00
jogosalv-orlando: hmm20:00
salv-orlandoalso, it would be great to understand why this problem which was sporafic until aug 21st suddenly became a big trouble.20:00
jogosalv-orlando: is that when neutron-full was turned on?20:01
salv-orlandonope - over a week after that20:01
salv-orlando(almost a week) https://review.openstack.org/#/c/88289/20:02
jogosalv-orlando: I am seeing a spike on Sept 1st20:02
jogohttp://status.openstack.org/elastic-recheck/20:02
jogodid we update the fingerprint?20:02
salv-orlandoprobably… I am sure this has been hitting the gate since aug 21st20:03
salv-orlandoor early morning 22nd GMT20:03
salv-orlandojogo: running logstash query. very slow.20:04
jogosalv-orlando: does anyone know scott moser or whoever is the cirros maint20:04
*** Ch00k has quit IRC20:04
salv-orlandomeh the devstack people probably? dtroyer maybe?20:05
*** smoser has joined #openstack-qa20:06
smosero/20:07
jogosalv-orlando: just pinged him and asked him to join the room20:07
jogosmoser meet salv-orlando one of our resident network debugging gururs20:07
jogoguru's20:07
jogogurus*20:07
jogosalv-orlando: want to catch smoser up on the issue?20:08
salv-orlandosure why not20:08
salv-orlandosmoser: we are talking about bug 134961720:08
salv-orlandoand one of the hypothesis made is that SSH connections might fails in some cases because of an already committed cirros bug20:09
*** afazekas has quit IRC20:09
* salv-orlando finding it20:09
salv-orlandosmoser: https://bugs.launchpad.net/cirros/+bug/130195820:09
salv-orlandoour questions are… 1) do you find it reasonable that in some instances cirros’ guest MTU might have a mismatch with that of the the sender?20:10
salv-orlandoI’d say that at SSH level there should be SAR because it’s TCP so MTU mismatches should be masked, but I’m not that networking guru you might expect20:11
salv-orlando2) as the bug has been now committed, what’s the process for building a new image we can use in devstack?20:12
salv-orlandosmoser: thanks in advance for your time.20:12
*** malini1 has joined #openstack-qa20:12
salv-orlandojogo: the trend is really worrying (http://goo.gl/yqzfdq) but it might be due to increased upstream activity because of feature freeze20:14
jogosalv-orlando:  yeah things are pretty busy rightnow20:15
jogohttp://jogo.github.io/gate/20:15
jogofailure rates aren't catestrophic yet20:15
salv-orlandojogo: but you haven’t updated your script connecting to graphite to fetch neutron’s full job hit count20:16
mtreinishjogo: the dsvm numbers look pretty tame20:16
*** toure has quit IRC20:16
*** sballe has quit IRC20:16
mtreinishjogo: although 50% trove py27 failure is no good20:16
*** aimon has quit IRC20:16
mtreinishsalv-orlando: oh, yeah that's a good point I didn't even notice it was missing there20:17
*** aimon has joined #openstack-qa20:17
*** malini1 has quit IRC20:17
jogosalv-orlando: oh your  right let me fix that20:17
*** malini1 has joined #openstack-qa20:17
openstackgerritpatrick-crews proposed a change to openstack-infra/elastic-recheck: Adding query namespace to elastic-recheck  https://review.openstack.org/10195820:19
*** NuclearW0rm has joined #openstack-qa20:21
smoserreading up20:21
smosersalv-orlando, incorrect mtu causes completely arbitrary failures20:22
smoserits amazing how many different places that will pop up.20:22
smoserso ssh connections failing definitely could be that.20:23
smoserregarding new cirros , we just have to get a new build.20:23
smoseri could look to cherry pick that set of changes and release a 0.3.3. which i do agree would be useful.20:23
salv-orlandosmoser: I think it would be great to have that20:24
smoserthe general plan for cirros is that 0.3 is not being developed any more.20:24
jogosalv-orlando: page udated20:24
smoserand 0.4 will have a slew of features.20:24
salv-orlandosmoser: we can update devstack to point to the new build with a simple patch and see if the failure goes away20:24
smoserthe mtu fix can probably be cherry picked though.20:25
salv-orlandosmoser: the diff looked pretty small20:25
smoserthe larger problem of openstack providing information about the MTU of its networks to the guests is much more severe.20:25
smoseras only dhcp is easily solved20:25
*** sballe has joined #openstack-qa20:26
salv-orlandomy opinion is that the network service should keep its reliance on guest configuration to a minimum20:27
smoserthe goal would be to solve that more general issue in https://blueprints.launchpad.net/nova/+spec/metadata-service-network-info20:27
smoseryour opinion is valid. and that would be wonderful.20:27
smoserand you'd think that "mtu discovery" would be a solved problem in 2014.20:27
smoserbut it is not.20:28
jogosmoser salv-orlando: can we turn stop sending MTU info to the guests in the gate?20:28
smoseryour problem is that you're using nested networks.20:28
smoserand the default 1500 mtu doesn't work.20:28
smoserat least i suspect thats the issue.20:28
salv-orlandojogo: the problem is in my opinion that neutron sets a MTU different from the default which cirros expects20:28
smoserright.20:28
jogosalv-orlando: so can we cahnge the value neutron uses?20:29
salv-orlandojogo: yes that was one of the options I discussed with smoser20:29
smoserwell, it does what it does because of the MTU of the network *it* is using.20:29
smoserwhich is probagly 150020:29
salv-orlandowe might do that only for gate tests in single host most20:29
salv-orlandomeh20:29
salv-orlandosingle host mode20:29
smoserso you can't shove nested networking with 1500 MTU into a network with 1500 MTU.20:29
smoseryou could possibly just bump the mtu of the host20:29
*** mrodden has quit IRC20:30
smosersalv-orlando, but i can try to get you a cirros with that patched dhcp that hpays attention to that also.20:30
smoserit would in general be a nice thing to have.20:30
dkranzekhugen: In https://review.openstack.org/#/c/111068/5, why did you add the imports of test while moving the files?20:30
jogosalv-orlando: changing the MTU setting in the gate to match cirros is probablly the quickest way forward right?20:30
*** mrodden has joined #openstack-qa20:30
salv-orlandosmoser: sure that would be good. and jogo you’re right I agree that’s the quickest way to validate this hypothesis20:30
jogosmoser: yeah that would be great, we can do both concurrently20:31
jogosalv-orlando: want to push up a patch to devstack-gate for this?20:31
salv-orlandostill, we have to go through the gate queue20:31
ekhugendkranz, oh probably because I had needed those imports for adding the tags, I should've removed them when I pulled the tags20:31
smoseri'm virtuallly certain that neutron isn't *choosing* to set the mtu less than 150020:31
ekhugensorry, will remove those now20:31
smoserbut is being forced into it.20:31
salv-orlandojogo: sure. I need a few minutes for that.20:31
smoserbecause of nested networking.20:31
salv-orlandosmoser: we had infinite discussions about that in the past. Never been a fan of pushing MTU selection through DHCP20:32
salv-orlandobut it was the only viable solution at the time.20:32
smoserthere is no other solution.20:32
salv-orlandoI think this proposal around enhanced metadata would be better.20:32
salv-orlandoand will work only for those who don’t like dhcp20:32
salv-orlandos/only/also20:33
*** rfolco has quit IRC20:33
*** mlavalle has joined #openstack-qa20:33
*** rfolco has joined #openstack-qa20:35
openstackgerritEmily Hugenbruch proposed a change to openstack/tempest: Add services tags to tempest/cli  https://review.openstack.org/11106820:36
*** aimon has quit IRC20:41
salv-orlandojogo, smoser: so in gate tests all interfaces on the ovs bridges have MTU==1500 and no option for MTU size is stored in dnsmasq’s OPTS file.20:43
salv-orlandothis is from a failing log: http://logs.openstack.org/95/117695/5/gate/gate-tempest-dsvm-neutron-full/d28fe19/console.html#_2014-09-02_16_44_29_43320:44
*** atiwari has quit IRC20:45
smosersalv-orlando, so cirros will just pick the default mtu for ehternet which is 150020:47
smoserwhich would be fine.20:47
smoserbut the problem is that your ovs network can't actually do 1500 mtu20:47
smoseri think.20:47
salv-orlandosmoser: the thing is that in gate tests we’re not encapsulating anything in tunnels because they’re single host tests20:47
salv-orlandosmoser: which makes me things that recently I’ve seen the network binding change to vxlan in devstack. hmmm20:48
smoseri dont know. as i said, its amazing how random mtu issues show themselves.20:49
*** mestery has quit IRC20:53
*** mwagner_lap has quit IRC20:53
*** andreaf has quit IRC20:54
*** mrodden has quit IRC20:55
*** mestery has joined #openstack-qa20:55
*** aimon has joined #openstack-qa20:56
*** mrodden has joined #openstack-qa20:56
*** needscoffee is now known as morganfainberg20:56
*** mlavalle has quit IRC20:56
*** mlavalle has joined #openstack-qa20:56
salv-orlandojogo, smoser: so this could clearly explain what’s going on -> http://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=8feaf6c9516094df58df84479d73779e87a7926421:00
*** mlavalle has quit IRC21:02
*** mlavalle has joined #openstack-qa21:02
*** suneelb has quit IRC21:03
*** suneelb has joined #openstack-qa21:03
*** aimon has quit IRC21:04
*** mlavalle has quit IRC21:04
*** mlavalle has joined #openstack-qa21:04
jogosalv-orlando: so we just revert?21:07
salv-orlandojogo: let me first have a positive on that. Also we might just change devstack-gate to not do vxlan in single node tests21:08
*** malini1 has left #openstack-qa21:09
*** flaper87 is now known as flaper87|afk21:09
jogosalv-orlando: awesome21:09
mtreinishjogo, salv-orlando: yeah I think changing it on the d-g side might make more sense here21:10
*** aysyd has quit IRC21:10
mtreinishsdague: ^^^ any thoughts?21:10
*** timello_ has quit IRC21:10
sdaguesalv-orlando: so given that single node is the default case maybe we need to revert and instead make doing vxlan optional21:13
sdaguebecause this is going to break for people locally as well, right?21:13
salv-orlandosdague: that is what I’m checking now21:15
*** melwitt has joined #openstack-qa21:15
openstackgerritEmily Hugenbruch proposed a change to openstack/tempest: Reduce boilerplate code using get_client_manager  https://review.openstack.org/11846921:16
*** mlavalle has quit IRC21:17
*** mlavalle has joined #openstack-qa21:18
*** atiwari has joined #openstack-qa21:18
*** weshay has quit IRC21:21
*** cnesa10 has joined #openstack-qa21:21
*** atiwari has quit IRC21:22
*** atiwari has joined #openstack-qa21:23
*** cbader has quit IRC21:25
*** dustins has quit IRC21:26
*** mmedvede has quit IRC21:31
salv-orlandosdague, jogo, mtreinish: the vxlan change does not have whatsoever impact on MTU. There is no header appended anywhere. MTU size on the devstack host and on the cirros guest are identical. No traffic reaches the br-tun which the vxlan patch added.21:32
*** morganfainberg is now known as steve_notmorgan21:32
salv-orlandoWe can go ahead with resetting network type to local for single-node tests, but I think we’re after a red herring21:32
*** steve_notmorgan is now known as morganfainberg21:32
openstackgerritAdam Gandelman proposed a change to openstack/tempest: Fix load balancer scenario if no tenant network  https://review.openstack.org/11370721:32
openstackgerritAdam Gandelman proposed a change to openstack/tempest: Fix network/secgroup scenario tests for ironic  https://review.openstack.org/11266021:33
jogosalv-orlando: too bad21:33
*** Sukhdev has joined #openstack-qa21:33
adam_gdkranz, mtreinish Shrews okay. https://review.openstack.org/#/c/113707/ is now the tip of the stack containing the last 2 remaining ironic tempest patches, rebased on yfried_'s client migration21:36
*** dkranz has quit IRC21:37
*** jgrimm is now known as zz_jgrimm21:42
openstackgerritBryan Havenstein proposed a change to openstack-dev/devstack: Added simple autodetection for 32/64 bit Cirros image arch  https://review.openstack.org/11787721:43
mtreinishadam_g: ok re-reviewed, I fast +Ad the first one, but the second one still needed a second pre-rebase +A21:43
adam_gmtreinish, righto. thanks!21:43
openstackgerritBryan Havenstein proposed a change to openstack-dev/devstack: Added simple autodetection for 32/64 bit Cirros image arch  https://review.openstack.org/11787721:44
adam_gmtreinish, AFAICS the unrelated server name change was just a string fix. currently the server ends up named 'server-smoke-%d-foo'21:48
*** sballe_ has joined #openstack-qa21:49
*** masayukig has quit IRC21:52
*** sballe has quit IRC21:53
*** masayukig has joined #openstack-qa21:57
openstackgerritA change was merged to openstack-dev/devstack: Faster nova fixed key generation  https://review.openstack.org/9609822:06
*** dims has quit IRC22:07
*** dims has joined #openstack-qa22:08
mtreinishadam_g: yeah, I wasn't saying the fix was incorrect, just that it wasn't part of the logical commit22:11
*** dpaterson has quit IRC22:11
*** dimtruck is now known as zz_dimtruck22:12
*** dims has quit IRC22:12
*** atiwari has quit IRC22:14
*** annegentle has joined #openstack-qa22:15
*** aimon has joined #openstack-qa22:15
*** rockyg has quit IRC22:16
*** jecarey_ has quit IRC22:21
*** shardy is now known as shardy_afk22:22
*** tkammer has quit IRC22:30
*** aimon_ has joined #openstack-qa22:32
*** aimon has quit IRC22:34
*** aimon_ is now known as aimon22:34
*** aimon has quit IRC22:52
*** aimon has joined #openstack-qa22:53
*** dmellado has quit IRC22:56
*** dmellado has joined #openstack-qa22:57
*** rbak has quit IRC22:58
jogomtreinish: so barbican23:13
jogodoes it have tempest tests?23:13
jogoand is it run in the gate?23:13
*** ianw has quit IRC23:17
*** jamielen^ has joined #openstack-qa23:18
*** ianw has joined #openstack-qa23:18
*** ZZelle_ has joined #openstack-qa23:19
*** ZZelle_ has quit IRC23:20
*** jamielennox_ has quit IRC23:20
*** cnesa10 has quit IRC23:27
*** cnesa10 has joined #openstack-qa23:28
*** jamielennox has joined #openstack-qa23:32
*** jamielen^ has left #openstack-qa23:33
*** Sukhdev has quit IRC23:33
*** oomichi has quit IRC23:35
mtreinishjogo: it's still incubated right? then it was run in the gate it would have a non-voting job on tempest23:35
mtreinishand a gating job on there own project23:35
jogomtreinish: ahh23:36
mtreinishI don't think it has any tests merged yet, but I remember seeing a patch at some point for something23:36
jogomtreinish: so how does that work with integration with things23:36
jogolike nova23:36
jogonova would have a a non-voting barbican+nova job23:36
mtreinishjogo: https://review.openstack.org/#/c/117032/23:36
mtreinishjogo: yeah if you want to test integration the job should be added on the experimental queue23:37
jogomtreinish: thanks23:37
jogothat makes sense to me23:37
mtreinishjogo: we normally start with the experimental pipeline on tempest too and upgrade it to non-voting23:37
jogore: https://review.openstack.org/104001'23:37
*** markmcclain has quit IRC23:37
*** oomichi has joined #openstack-qa23:45
*** mlavalle has quit IRC23:50
openstackgerritDean Troyer proposed a change to openstack-dev/devstack: Run processes without screen  https://review.openstack.org/11733923:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!