Thursday, 2016-01-21

mriedemmtreinish: anything come out of the multinode n-net testing discussion?00:02
mtreinishmriedem: https://review.openstack.org/#/c/270528/100:03
*** rbak has joined #openstack-qa00:03
mtreinishbut I suspect it'll fail on neutron00:03
mriedemnice00:04
mtreinishoh, yep it really does: https://jenkins01.openstack.org/job/gate-tempest-dsvm-neutron-full/19558/console00:05
mriedem:/00:07
mriedemcan't win00:07
mriedemlet's just remove nova-net00:07
mtreinishif we make devstack work well for everyone with neutron I'm totally fine with that00:08
mtreinishalthough I understand n-net, neutron still confuses me once you get past a basic setup00:09
*** jlwhite has joined #openstack-qa00:09
*** rbak has quit IRC00:12
*** ccneill_ has joined #openstack-qa00:14
*** ccneill_ has quit IRC00:18
*** weshay has quit IRC00:19
openstackrecheckConsole logs not available after 13:27s for gate-keystone-linters 269824,6,d2d380000:20
*** jlwhite has quit IRC00:22
*** mriedem is now known as mriedem_dinner00:28
clarkbmtreinish: hrm so neutron didn't like the change to use public?00:29
*** dims has quit IRC00:29
*** pvaneck has quit IRC00:30
*** itzikb has quit IRC00:30
mtreinishclarkb: I kinda expected that, because networks are tenant scoped in neutron, we'd have to have tempest create a network name public on every tenant when it's created00:32
clarkbhuh00:32
clarkbmtreinish: and you already create one called private?00:32
mtreinishno, but I think it's a weird artifact of how the ssh code works in tempest00:33
mtreinishit's all a mess00:33
*** itzikb has joined #openstack-qa00:34
mtreinishclarkb: oh: http://logs.openstack.org/28/270528/1/check/gate-tempest-dsvm-neutron-large-ops/c42313c//console.html#_2016-01-21_00_01_08_87100:34
openstackrecheckConsole logs not available after 13:25s for gate-swift-python27 270523,1,1d3910d00:35
mtreinishits different than I thought the public network exists, but you can't attach directly to it00:35
* mtreinish wonders what "It" is?00:36
*** ccneill_ has joined #openstack-qa00:37
clarkbnot allowed to create ya what it isn't allowed00:38
openstackrecheckopenstack-dev/devstack change: https://review.openstack.org/267538 failed because of: gate-tempest-dsvm-large-ops: https://bugs.launchpad.net/bugs/1491949 https://bugs.launchpad.net/bugs/150061500:38
openstackLaunchpad bug 1491949 in OpenStack Compute (nova) "gate-tempest-dsvm-large-ops fails to deallocate instance network due to rpc timeout" [Medium,Confirmed]00:38
openstackLaunchpad bug 1500615 in OpenStack Compute (nova) "Large Ops scenario is taking too long" [High,Confirmed]00:38
oomichimtreinish: hi, are you still there?00:38
mtreinishoomichi: sort of00:38
oomichimtreinish: cool :)00:38
oomichimtreinish: sometimes, people propose negative tests to tempest still now00:39
oomichimtreinish: these tests are not integration tests in many cases, and they are not good to merge them into tempest00:39
*** rbak has joined #openstack-qa00:40
oomichimtreinish: how about implementing them in each project own repo with external plugin?00:40
oomichimtreinish: even if core projects00:40
*** markvoelker_ has quit IRC00:41
mtreinishoomichi: you honestly don't even need to use tempest to test it. Most of them are very shallow and just need the db and the api service00:41
mtreinishyou can test it much faster as a functional test that doesn't require a full cloud00:41
*** Swami has quit IRC00:42
oomichimtreinish: yeah, most(all) cases can be covered with functional tests of each own repo.00:42
mtreinishthere are a few exceptions, where the negative check is a real integration point between projects and I think those would be ok to have in tempest00:42
mtreinishwe came up with that rule back in HK because of all those 1 line shallow negative fuzz tests00:43
*** hitalia has quit IRC00:43
oomichibut sometimes I aslo want to check the validation works fine on a full cloud which is delpoyed with devstack00:43
mtreinishand those still don't belong in tempest00:43
mtreinishoomichi: for example?00:43
oomichimtreinish: when changing validation framework on project side, as a sample00:44
oomichilike nova v2.0 -> nova v2.100:44
mtreinishoomichi: oh, to compare that things work the same before and after when you adopt jsonschema00:45
oomichibut these cases are fine if implementing them in each own repo and testing them with tempest external plugin00:45
oomichimtreinish: yeah, right.00:46
mtreinishoomichi: so plugins enable anyone to do whatever they want, that's kinda the point :)00:46
mtreinishso if a project wanted to do that we can't really stop them00:46
oomichimtreinish: and we have a plan to implement swagger which is also a kind of framework, so I still want to test negative cases on nova00:47
mtreinishI just don't think its worth it when you weigh it against the maint. cost of having all those tests00:47
mtreinishoomichi: well, I don't think we'll remove the negative tests that are already in tree00:47
oomichimtreinish: "tree" means "tempest tree" ?00:48
mtreinishright, if there are tests already no one has proposed removing them00:48
oomichiI can agree with that because refstack also uses some negative tests.00:49
mtreinishI just don't want to repeat the pre-HK summit situation where all those negative tests for requests with random fuzz input00:49
oomichimtreinish: big +1 :)00:49
oomichithis question cames from https://review.openstack.org/#/c/257255/00:49
mtreinishI think targetted negative tests that go deeper than the db or cross a service boundary are also probably fine too00:50
mtreinishoomichi: that's a weird case, it's a fuzz test on neutron through nova :)00:51
*** ajmiller has joined #openstack-qa00:51
mtreinishwell fuzz isn't the right word00:51
mtreinishit's neglecting to use a required arg00:51
mtreinishoomichi: what is your opinion on it?00:52
mtreinishthe docs on the topic are kinda out of date: http://docs.openstack.org/developer/tempest/HACKING.html#negative-tests00:53
oomichimtreinish: that just checks rest api layer, so +1 for jordan's opinion00:53
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-neutron-full 267004,7,a9d303f00:53
mtreinishI don't think anyone has pushed patch using that framework in a very long time :)00:53
oomichimtreinish: yeah, right ;)00:53
oomichimtreinish: so I'd like to clarify *new* negative tests need to be implemented with tempest external plugins for all projects by keeping the existing negative ones in tempest tree.00:54
oomichimtreinish: is this fine for you?00:55
mtreinishoomichi: I think you should push a patch to update the docs and explain things clearly. (especially why) and then link that in to 25725500:55
boris-42mtreinish: hi00:55
oomichimtreinish: yeah, I will do that :)00:55
boris-42mtreinish: are you going to change this line https://github.com/openstack/tempest/blob/master/tools/install_venv_common.py#L106 in future?00:55
boris-42mtreinish: to force version of pip <8?00:56
boris-42mtreinish: currently rally verify is broken because of that00:56
*** rbak has quit IRC00:57
mtreinishoomichi: well I question the need to specifically say external plugins, because I think for most cases using a tempest plugin to do negative testing is the wrong approach00:57
mtreinishboris-42: it's not that function, it's your virtualenv version00:57
mtreinishboris-42: you must be pulling in virtualenv 14 which will put pip8 in the venv00:57
boris-42mtreinish: so basically you can take at our logs00:58
boris-42mtreinish: http://logs.openstack.org/06/270506/4/check/gate-rally-dsvm-verify-light/6f052b4/rally-verify/6_verify_start_--set_smoke_--xfails-file_expected_failures.yaml.txt.gz00:58
mtreinishboris-42: why do you even use that script? just do 'virtualenv tempest-env && tempest-env/bin/pip install -U -r requirements' it's basically the same thing but then you can control the install story00:59
mtreinishoh, the issue there is actually -U00:59
oomichimtreinish: are you saying "we need to implement negative tests in each project repo *without* external plugin interface as possible" ?00:59
oomichibecause most negative test cases are just checking api layer and they don't need devstack env, right?01:00
mtreinishoomichi: well I think whether its a plugin or not isn't the issue. I think doing negative testing at the black box level is inefficient01:01
*** hitalia has joined #openstack-qa01:01
mtreinishand by saying you should do it in a plugin is saying you should do it via black box testing01:01
mtreinishoomichi: I think if we just say, do it outside of tempest (whether as an external plugin or a more targetted functional test)01:03
*** safchain has quit IRC01:03
oomichimtreinish: I guess I understand your idea. so the black box testing should not be static. ideally it should be implemented with fuzz framework as black box testing, right?01:04
boris-42mtreinish: yep -U is the thing that breaks stuff01:04
boris-42mtreinish: if we just remove it it will work well for us01:04
boris-42mtreinish: so basically we run this script https://github.com/openstack/tempest/blob/master/tools/install_venv.py that runs that one01:04
mriedem_dinnermtreinish: what was the fix to keep pip 8 from installing in venvs?01:05
boris-42mtreinish: in any case it's broken now and I believe it will be nice to fix it01:05
oomichimtreinish: the best way is the combination of external plugin + fuzz framework.01:05
*** mriedem_dinner is now known as mriedem01:05
*** jacky-zhang has joined #openstack-qa01:05
mtreinishmriedem_dinner: constraints protects the regular jobs from pulling in virtualenv 1401:05
mriedemoh that's right my venv is too new01:05
mtreinishmriedem_dinner: constraints boris-42 so that whole code path is kinda crusty and there isn't a reason to use it. install_venv_common was from oslo-incubator01:06
mtreinishit was the common abstraction of what every project was doing for run_tests.sh back in the grizzly days01:07
boris-42mtreinish: ok we will remove usage of it from rally verify01:07
boris-42mtreinish: maybe it makes sense to remove those scripts then?01:07
boris-42mtreinish: or at least add some deprecation notes01:08
mtreinishboris-42: I'm kinda surprised we manually run pip install -U pip>1.4 in there01:08
mtreinishboris-42: we can straight up remove it, we just need to update run_tests.sh to do the venv creation01:08
mtreinisherr run_tests.sh and run_tempest.sh01:08
mtreinishremoving those would break people01:08
*** salv-orlando has quit IRC01:09
boris-42mtreinish: yep01:10
boris-42mtreinish: so it should be deprecated first01:10
*** hitalia has quit IRC01:11
mtreinishboris-42: you can just remove tools/install_venv* but in the same patch you just need to update the calls in run_(tests|tempest).sh to call virtualenv directly instead01:11
openstackrecheckopenstack/nova change: https://review.openstack.org/213094 failed gate-grenade-dsvm-multinode in the gate queue with an unrecognized error01:11
mriedemmtreinish: ah i see https://github.com/openstack/nova/blob/master/tools/install_venv_common.py#L11601:12
mriedemyeah we should cap that to pip<801:12
mriedembecause running py27 locally is busted01:12
mtreinishmriedem: well if you use run_tests.sh01:13
mriedemmtreinish: we dont01:13
mtreinishbut who does that01:13
mriedemnova removed that01:13
mriedemoh snap no we didn't01:13
mtreinishthen why is install_venv_common still there01:13
mtreinishthat's the only path that would get called from01:13
mriedemor yes we did https://github.com/openstack/nova/commit/2dc5d9f8632c83c9bc7ee6871c6631edf69a557001:14
mriedemok yeah, my problem was blindly pip install upgrading pip and virtualenv01:14
openstackgerritSean M. Collins proposed openstack-dev/devstack: Begin new lib/neutron  https://review.openstack.org/16843801:15
boris-42mtreinish: actually in case of rally01:15
boris-42mtreinish: if we remove that script01:15
boris-42mtreinish: we will break a lot of old releases01:16
*** zhangjn has joined #openstack-qa01:16
boris-42mtreinish: so maybe we can just fix it?01:16
boris-42mtreinish: and deprecate for now01:16
openstackgerritMatthew Treinish proposed openstack/tempest: Remove virtualenv management scripts from oslo-incubator  https://review.openstack.org/27056401:21
mtreinishboris-42: the tools dir were just helper utilities we never had any guarantees on them. They aren't even packaged in the sdist01:22
mtreinishyou have to check out the repo to get them01:22
boris-42mtreinish: so your patch will break *all* rally releases01:22
boris-42mtreinish: please let's depracate them01:23
boris-42mtreinish: we will remove from rally01:23
boris-42mtreinish: usage of it01:23
boris-42mtreinish: and after we get at least 1 working release in Rally we will remove them from tempest01:23
mtreinishboris-42: TBH, I don't understand how or why they're being used in rally01:23
boris-42mtreinish: this line https://github.com/openstack/rally/blob/master/rally/verification/tempest/tempest.py#L20001:24
boris-42mtreinish: so "rally verify" installs tempest via this command01:25
openstackrecheckConsole logs not available after 13:23s for gate-nova-python27 203751,17,3781d1b01:26
*** j20012 has quit IRC01:28
*** jpas has quit IRC01:29
mtreinishboris-42: so you clone tempest from source and then run that command. Why? I mean that just seems like a bad idea to depend on a git version of any dependency. I want to say we won't break but things slip up01:29
mtreinishyou don't install lib dependencies from git, it's the same thing01:29
mtreinishboris-42: but anyway I've got to go now we can pick this up later (or in the review)01:31
boris-42mtreinish: ok01:35
*** jecarey has joined #openstack-qa01:38
openstackrecheckConsole logs not available after 13:21s for gate-horizon-dsvm-integration 270542,1,0232e7201:40
*** jpas has joined #openstack-qa01:41
*** jecarey has quit IRC01:43
*** zhangjn has quit IRC01:45
*** zhangjn has joined #openstack-qa01:51
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-postgres-full 270528,1,0ff994801:53
*** dims has joined #openstack-qa01:54
*** erlon has quit IRC01:56
*** hoangcx has joined #openstack-qa01:57
*** zhangjn has quit IRC01:58
*** ccneill_ has quit IRC02:05
*** ylobankov has joined #openstack-qa02:06
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 267152,3,3c1c60702:07
gmannmtreinish: oomichi : request for review - https://review.openstack.org/#/c/265051/02:15
gmannmtreinish: oomichi i really want this before migration to lib otherwise i have to do all compute clients  changes twice02:16
*** zhangjn has joined #openstack-qa02:16
*** bochi-michael has joined #openstack-qa02:20
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm 269291,2,058a96902:21
openstackgerritDavid Paterson proposed openstack/tempest: Add unit tests for account generator  https://review.openstack.org/27057602:22
*** yamamoto has joined #openstack-qa02:23
*** woodster_ has quit IRC02:26
*** zackf has joined #openstack-qa02:29
phschwartzI am gettig an error when doing a run_tespest.sh and I don't have much experience with running tempest. Anyone able to take a look at the error and point me in the right direction? http://paste.openstack.org/show/484489/02:30
*** zhangjn has quit IRC02:32
*** akshai has joined #openstack-qa02:35
*** yamamoto has quit IRC02:37
*** yamamoto has joined #openstack-qa02:37
*** akshai has quit IRC02:40
openstackrecheckConsole logs not available after 13:23s for gate-grenade-dsvm 190273,25,10360de02:41
*** nicholasgoracke has joined #openstack-qa02:43
*** ylobankov has quit IRC02:44
*** yamamoto has quit IRC02:46
*** itzikb has quit IRC02:47
clarkbphschwartz: no module named subunit make sure you have installed tempest properly02:48
*** jasonsb has joined #openstack-qa02:48
*** vilobhmm111 has left #openstack-qa02:53
openstackrecheckConsole logs not available after 13:26s for gate-grenade-dsvm 270565,1,26a961102:54
*** yamahata has quit IRC03:00
openstackgerritMerged openstack-dev/devstack: Remove vme and ssse3 from gate64 cpu mapping  https://review.openstack.org/26974203:00
*** dgonzalez has quit IRC03:03
*** bkopilov has quit IRC03:04
oomichigmann: thanks for doing that, nice implementation. just comments about docstring thing.03:06
phschwartzclarkb: I follows the docs on the o.o doc site the venv that was built by run_tempest also looks to have had python-subunit installed as a dep03:06
gmannoomichi: Thanks a lot. I will check03:07
openstackrecheckConsole logs not available after 13:24s for gate-horizon-npm-run-test 270553,2,c1e7b5703:08
*** binbincong has joined #openstack-qa03:11
*** agireud has quit IRC03:11
*** dgonzalez has joined #openstack-qa03:13
*** shuquan has joined #openstack-qa03:14
*** agireud has joined #openstack-qa03:15
openstackrecheckConsole logs not available after 13:22s for gate-horizon-npm-run-lint 259236,17,e2cfc0e03:21
*** cody-somerville_ has joined #openstack-qa03:22
*** oomichi is now known as oomichi_away03:23
openstackgerritfengzhr proposed openstack/tempest: There is no api for searching images with size  https://review.openstack.org/27058503:25
*** agireud has quit IRC03:27
*** shuquan has quit IRC03:30
*** shuquan has joined #openstack-qa03:31
*** zhangjn has joined #openstack-qa03:34
mriedemphschwartz: run_tempest.sh with -V eventually calls this https://github.com/openstack/tempest/blob/master/tools/install_venv_common.py#L11603:35
mriedemphschwartz: and pip 8 is busted03:35
openstackrecheckConsole logs not available after 13:25s for gate-grenade-dsvm 270073,2,724a47303:35
mriedemphschwartz: i'm not sure how much people are actually supposed to be using run_tempest.sh, I would think you should just run it via tox03:36
*** shausy has joined #openstack-qa03:36
phschwartzmriedem: I am open to suggestions. I have not run it much beyond run_tempest.sh03:36
phschwartzmriedem: and yeah still in a pip <8.003:37
mriedemd-g runs it via tox https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L73303:37
mriedemphschwartz: but did you see if you had pip 8 in the ven?03:37
mriedem*venv03:37
*** shuquan has quit IRC03:38
*** yamamoto_ has joined #openstack-qa03:39
phschwartzmriedem: no pip 7.1.3 in venv. Do I just run tox from the tempest dir. Never tried running it with tox before03:39
mriedemyeah i think so03:39
*** dgonzalez has quit IRC03:39
*** dgonzalez has joined #openstack-qa03:40
phschwartzi will have to try it with both. The -V might be a quick fix as I figure out tox for it03:40
mriedemrun_tempest.sh might just be buggy since it doubt it's maintained given it's now how tempest is run in the gate03:40
mriedem*i doubt03:40
mriedembut mtreinish was talking about something related in here earlier03:40
mriedemphschwartz: what is your virtualenv version?03:42
mriedemthat needs to be <1403:42
*** dims has quit IRC03:42
*** armax has quit IRC03:42
openstackgerritleoyyoung proposed openstack/tempest: The admin infos are in auth section, correcct in configuration.rst  https://review.openstack.org/27058703:43
phschwartzmriedem: it is <1403:43
openstackrecheckConsole logs not available after 13:23s for gate-nova-python27 270344,2,f4aa83903:49
*** zackf has quit IRC03:50
*** zackf has joined #openstack-qa03:51
*** agireud has joined #openstack-qa03:52
*** tochi has joined #openstack-qa03:52
openstackgerritXav Paice proposed openstack/tempest: Cap Pip<8 due to pip bug  https://review.openstack.org/27058803:53
*** shausy has quit IRC03:57
openstackrecheckConsole logs not available after 13:25s for gate-nova-python34 248989,74,ee66acc04:02
*** vgridnev has joined #openstack-qa04:05
openstackgerritMerged openstack-dev/devstack: Support for logging the nova-dhcpbridge command line  https://review.openstack.org/26970604:08
openstackgerritMerged openstack-dev/devstack: Remove vme and ssse3 from gate64 cpu mapping  https://review.openstack.org/26974304:09
*** Poornima has joined #openstack-qa04:09
*** mriedem has quit IRC04:12
openstackgerritMerged openstack-dev/devstack: Make OSprofiler repo and branch configurable  https://review.openstack.org/26788804:13
*** zackf has quit IRC04:14
*** zackf has joined #openstack-qa04:14
openstackrecheckConsole logs not available after 13:22s for experimental-tempest-dsvm-multinode-full-dibtest 254063,3,0d2237004:16
*** armax has joined #openstack-qa04:20
*** bkopilov has joined #openstack-qa04:26
openstackrecheckConsole logs not available after 13:26s for gate-tempest-dsvm-neutron-full 264510,1,5c2221404:29
*** vgridnev has quit IRC04:35
*** zhangjn has quit IRC04:36
openstackrecheckConsole logs not available after 13:24s for gate-grenade-dsvm 270417,1,6e03d0204:42
*** coolsvap|away is now known as coolsvap04:49
*** nicholasgoracke has quit IRC04:50
openstackrecheckConsole logs not available after 13:27s for gate-tempest-dsvm-postgres-full 269965,1,895747404:56
openstackrecheckopenstack/nova change: https://review.openstack.org/263430 failed because of: gate-grenade-dsvm-multinode: unrecognized error, gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1526737 https://bugs.launchpad.net/bugs/1532809 https://bugs.launchpad.net/bugs/135557304:56
openstackLaunchpad bug 1526737 in OpenStack Compute (nova) "Race between nova-network and dnsmasq allocating ips" [High,In progress] - Assigned to Dan Smith (danms)04:56
openstackLaunchpad bug 1532809 in OpenStack Compute (nova) "Gate failures when DHCP lease cannot be acquired" [Undecided,New]04:56
openstackLaunchpad bug 1355573 in OpenStack Compute (nova) "tempest volume scenarios periodically fail /w SSHTimeout" [Undecided,New]04:56
openstackrecheckopenstack/requirements change: https://review.openstack.org/267725 failed because of: gate-tempest-dsvm-neutron-full: https://bugs.launchpad.net/bugs/150900404:57
openstackLaunchpad bug 1509004 in neutron ""test_dualnet_dhcp6_stateless_from_os" failures seen in the gate" [Medium,Confirmed]04:57
*** vgridnev has joined #openstack-qa04:58
*** david-lyle has quit IRC05:02
*** david-lyle has joined #openstack-qa05:03
*** zhangjn has joined #openstack-qa05:05
*** markvoelker has joined #openstack-qa05:07
openstackrecheckConsole logs not available after 13:24s for gate-grenade-dsvm-multinode 264014,11,ac53a2805:10
*** markvoelker has quit IRC05:12
*** SridharG has joined #openstack-qa05:12
*** markvoelker has joined #openstack-qa05:12
openstackrecheckConsole logs not available after 13:29s for gate-tempest-dsvm-neutron-full 270591,1,4f3f2b705:24
*** SridharG has quit IRC05:28
openstackgerritPiyush Pathak proposed openstack/tempest: Make 2 functions in network_client use **kwargs  https://review.openstack.org/25884805:32
*** cp16net has joined #openstack-qa05:34
openstackrecheckConsole logs not available after 13:27s for gate-tempest-dsvm-full-ceph 206923,37,9d0e77705:37
*** SridharG has joined #openstack-qa05:40
*** vgridnev has quit IRC05:45
*** jasonsb has quit IRC05:47
*** shausy has joined #openstack-qa05:48
openstackrecheckConsole logs not available after 13:38s for gate-nova-pep8 248989,75,b3960ed05:51
*** SridharG has quit IRC05:54
*** Poornima_K has joined #openstack-qa05:56
*** Poornima has quit IRC05:57
cp16netall trove gate jobs are failing building the image needed for the tests06:00
*** aopincar has joined #openstack-qa06:00
*** markvoelker_ has joined #openstack-qa06:01
*** hichihara has joined #openstack-qa06:02
cp16netlooks like its something with pip version (i have a hunch its related to the pip 8.0 release since it started failing about a day ago)06:02
eliqiaohi gmann , I get issue when run tempest case, can you tell how to use auth from /etc/openstack/cloud.yml instead of ./etc/tempest.conf06:03
*** Poornima_K has quit IRC06:03
eliqiaoor anyone else here can give a hand on how to run tempest with `Dynamic Credentials`06:05
*** markvoelker has quit IRC06:05
eliqiaoor, goacha, To enable dynamic credentials in the auth section with the use_dynamic_credentials option.06:05
*** markvoelker has joined #openstack-qa06:05
*** Poornima has joined #openstack-qa06:06
*** markvoelker_ has quit IRC06:06
*** vgridnev has joined #openstack-qa06:08
openstackgerritYusuke Hayashi proposed openstack-dev/devstack: Rdo-release install failure under proxy  https://review.openstack.org/27061206:09
*** jasonsb has joined #openstack-qa06:10
gmanneliqiao: yea making use_dynamic_credentials true will do all thing, you just need admin cred in conf06:10
eliqiaogmann: thx, I think I have some misleading with cloud.yaml, if using cloud.yaml file, I need to set use_dynamic_credentials=false.06:11
*** oomichi_away is now known as oomichi06:11
eliqiaogmann: I will try as you said.06:11
gmanneliqiao: yea that is another way to run tempest with pre provisioned cred06:12
gmanneliqiao: but use_dynamic_credentials=true will make things easy for you06:12
openstackrecheckConsole logs not available after 13:23s for gate-grenade-dsvm-multinode 270585,1,f32723f06:14
openstackrecheckopenstack/nova change: https://review.openstack.org/254434 failed gate-grenade-dsvm-multinode in the gate queue with an unrecognized error06:14
eliqiaogmann: okay, thanks. one more question: I use oslo-config-generator to generate a oslo.conf file, but the conf file only contains oslo_concurrency and DEFAULT section. Is this an issue?06:15
*** gamado has joined #openstack-qa06:15
eliqiaogmann: I need to copy the tempest.conf example file from #link http://docs.openstack.org/developer/tempest/sampleconf.html06:16
*** jasonsb has quit IRC06:16
eliqiaoPS: I am using latest master branch and did nothing changes.06:16
gmanneliqiao: yea,06:16
gmanneliqiao: should work06:16
gmanneliqiao: by default  use_dynamic_credentials is true in tempest so should work as it is06:17
*** yfried has quit IRC06:17
cp16netlooks like its related to pip source changing and tripleo-image-elements needs an update https://review.openstack.org/#/c/270494/06:18
*** zhangjn has quit IRC06:20
*** neeti has joined #openstack-qa06:21
eliqiaogmann: I added admin creds, but still not work, can you help to check http://paste.openstack.org/show/484498/06:22
gmanneliqiao: just 1 min. ll check06:22
*** zhangjn has joined #openstack-qa06:23
*** _nadya_ has joined #openstack-qa06:24
*** vgridnev has quit IRC06:26
*** SridharG has joined #openstack-qa06:29
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 270588,1,2afaa9f06:31
gmanneliqiao: you conf for admin cred06:32
*** piyush_ has joined #openstack-qa06:32
eliqiaogmann: ya..06:32
eliqiaoI shoud miss something ?06:33
gmanneliqiao: admin_username, admin_tenant_name admin_password06:33
gmanneliqiao: do you have these in tempest.conf06:33
eliqiaogmann: yes, I have them06:34
gmanneliqiao: not commented right?06:34
gmanneliqiao: can you please paste those on paste.org06:34
eliqiaogmann: see the first several lines of http://paste.openstack.org/show/484498/06:34
eliqiaogmann: okay06:34
eliqiaogmann: http://paste.openstack.org/show/484504/06:35
eliqiaogmann: seems I need [identity] section06:39
gmanneliqiao: auth_url is None there06:39
piyush_Hi Gmann:06:40
gmanneliqiao: yea in identity section you need     cfg.StrOpt('uri',06:40
gmann               help="Full URI of the OpenStack Identity API (Keystone), v2"),06:40
gmann    cfg.StrOpt('uri_v3',06:40
gmann               help='Full URI of the OpenStack Identity API (Keystone), v3'),06:40
gmanneliqiao: set those which is used to get token from keystone and then tempest start creating cred using that token06:40
gmannpiyush_: hi06:41
piyush_gmann: can you please have a look on he https://jenkins04.openstack.org/job/gate-tempest-python34/1414/console and help me what are the changes needed in patch06:41
piyush_gmann:06:41
piyush_https://review.openstack.org/#/c/258848/12/tempest/tests/common/test_dynamic_creds.py06:41
gmannpiyush_: ok, I will check06:41
piyush_gmann: thanks06:41
*** yamahata has joined #openstack-qa06:41
gmannpiyush_: np06:41
*** zhangjn has quit IRC06:44
openstackrecheckConsole logs not available after 13:21s for gate-tempest-dsvm-neutron-src-os-brick 269867,4,7617a9f06:44
eliqiaogmann: ah, bingo. thanks, it works after set uri and uri_v306:45
eliqiaogmann: BTW, what's uri and url mean?06:45
gmanneliqiao: ok. those are used for getting token from keystone06:45
eliqiaogmann: Thank you very much :) I google them by http://stackoverflow.com/questions/4913343/what-is-the-difference-between-uri-url-and-urn06:47
*** vgridnev has joined #openstack-qa06:47
gmanneliqiao: oh you mean difference between uri and url06:48
*** hoangcx has quit IRC06:48
gmanneliqiao: yea url mainly locator06:48
gmanneliqiao: yea nice explanation in mentioned link :)06:48
eliqiaogmann: ah, thanks for your help, you are nice. thanks again.06:49
gmanneliqiao: np! :)06:49
*** hoangcx has joined #openstack-qa06:50
eliqiaooh goacha, uri is 'a' resource , url is a set of rescources. :)06:51
*** anshul has joined #openstack-qa06:51
*** Poornima has quit IRC06:52
*** zhangjn has joined #openstack-qa06:58
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm 264133,3,1bb365f06:58
*** zhangjn has quit IRC06:58
openstackgerritAndreas Jaeger proposed openstack-infra/elastic-recheck: Remove argparse from requirements  https://review.openstack.org/27041406:58
*** zhangjn has joined #openstack-qa06:59
*** Poornima has joined #openstack-qa07:04
*** agireud has quit IRC07:04
*** achanda has quit IRC07:06
openstackrecheckConsole logs not available after 13:21s for gate-grenade-dsvm-multinode 189843,30,f4cb8b107:11
*** yfried has joined #openstack-qa07:13
*** agireud has joined #openstack-qa07:16
*** neeti has quit IRC07:17
*** rcernin has joined #openstack-qa07:20
openstackrecheckConsole logs not available after 13:21s for gate-grenade-dsvm 269380,2,211ce5707:25
*** vgridnev has quit IRC07:26
*** sshnaidm has quit IRC07:27
*** neeti has joined #openstack-qa07:36
*** itzikb has joined #openstack-qa07:36
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-neutron-full 263166,2,542740d07:38
openstackgerritMasayuki Igawa proposed openstack/openstack-health: [WIP] Add parameter for options in URL  https://review.openstack.org/27064307:42
*** safchain has joined #openstack-qa07:48
openstackgerritGhanshyam Mann proposed openstack/tempest: Separate base microversion client and compute client  https://review.openstack.org/26505107:53
openstackrecheckConsole logs not available after 13:24s for gate-horizon-npm-run-lint 259236,18,27b644207:53
*** hoangcx_ has joined #openstack-qa08:00
*** hoangcx has quit IRC08:00
*** ihrachys has joined #openstack-qa08:01
*** mkoderer_ is now known as mkoderer08:03
*** achanda has joined #openstack-qa08:06
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 257638,3,60c1f0408:07
*** zhangjn has quit IRC08:09
*** zhangjn_ has joined #openstack-qa08:09
*** tosky has joined #openstack-qa08:09
*** zhangjn has joined #openstack-qa08:11
*** Serlex has joined #openstack-qa08:11
*** achanda has quit IRC08:13
*** zhangjn_ has quit IRC08:14
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-full 267962,8,c5b085f08:21
*** ajo_ is now known as ajo08:24
toabctl_is it somehow possible (via tempest.conf) to disable tempest tests from external plugins (in my case I want to disable the manila tempest tests) ?08:25
gmanntoabctl_: you can only disable by uninstalling the plugin or do not run specific plugin tests via regex08:27
toabctl_hm. that's bad08:27
*** vgridnev has joined #openstack-qa08:27
*** hoangcx_ is now known as hoangcx08:28
toabctl_the tempest tests are not separated from the manila core code. so regex is the only option I guess08:28
openstackgerritItzik Brown proposed openstack/tempest: qos basic scenario  https://review.openstack.org/21717708:29
openstackgerritMasayuki Igawa proposed openstack/openstack-health: [WIP] Add parameter for options in URL  https://review.openstack.org/27064308:32
*** gfidente has joined #openstack-qa08:32
*** gfidente has quit IRC08:32
*** gfidente has joined #openstack-qa08:32
openstackrecheckConsole logs not available after 13:26s for gate-grenade-dsvm 252809,5,6bc4dce08:34
*** agireud has quit IRC08:35
*** yfried has quit IRC08:35
*** yfried has joined #openstack-qa08:35
*** hichihara has quit IRC08:35
*** armax has quit IRC08:36
*** shardy has joined #openstack-qa08:39
openstackgerritAlex Petrov proposed openstack/tempest: Add a test for Get Service Catalog  https://review.openstack.org/26590508:39
openstackgerritMaho Koshiya proposed openstack/tempest: Add tests of port_id and fixed_ips in the attach interfaces nova API  https://review.openstack.org/25725508:44
*** agireud has joined #openstack-qa08:47
*** jlanoux has joined #openstack-qa08:47
openstackrecheckConsole logs not available after 13:23s for gate-grenade-dsvm 265610,2,644738108:47
toabctl_gmann: what about the ServiceAvailableGroup config option for manila?08:48
gmanntoabctl_: yea that will useful if they use that in their tests08:49
*** yfried_ has joined #openstack-qa08:50
toabctl_gmann: there is at least a config option and the config option is registered in the plugin. anything else that needs to be done?08:50
gmanntoabctl_: no, that is enough. add config option and skip tests based on that08:51
gmanntoabctl_: but make it true by default to not break devstack etc08:51
toabctl_gmann: so tempest is checking that and automatically skip the tests?08:52
gmanntoabctl_: tempest will not check, that should be checked in manila tempest plugin08:52
toabctl_gmann: it's true by default. it's just for local runs where manila is installed but not fully configured.08:52
gmanntoabctl_: config option should be added in manila tempest pluginc config option and skip tests in manila itself08:52
gmanntoabctl_: checking08:53
toabctl_gmann: http://git.openstack.org/cgit/openstack/manila/tree/manila_tempest_tests/plugin.py#n3308:53
gmanntoabctl_: https://github.com/openstack/manila/blob/fe29873faf4ffd3275f1dc85d32c5e8104b8249c/manila_tempest_tests/tests/api/base.py#L185-L18908:54
gmanntoabctl_: it seems they are skipping the tests08:54
*** yfried has quit IRC08:55
toabctl_gmann: great. thanks!08:55
gmanntoabctl_: try by making that false in config08:55
*** dtantsur|afk is now known as dtantsur08:58
openstackrecheckConsole logs not available after 13:21s for experimental-tempest-dsvm-multinode-full-dibtest 254063,3,886427b09:01
*** agireud has quit IRC09:05
*** apevec has joined #openstack-qa09:08
*** apevec has joined #openstack-qa09:08
*** shausy has quit IRC09:08
openstackgerritMasayuki Igawa proposed openstack/openstack-health: [WIP] Add parameter for options in URL  https://review.openstack.org/27064309:08
*** agireud has joined #openstack-qa09:09
*** shausy has joined #openstack-qa09:09
*** markvoelker has quit IRC09:12
*** Liuqing has joined #openstack-qa09:12
openstackrecheckConsole logs not available after 13:21s for gate-grenade-dsvm-multinode 270612,1,553ffeb09:15
*** rossella_s has joined #openstack-qa09:18
openstackgerritJoseph Lanoux proposed openstack/tempest: Add a PINGABLE and a SSHABLE waiters  https://review.openstack.org/25951509:19
*** piyush_ has quit IRC09:20
*** rossella_s has quit IRC09:21
*** rossella_s has joined #openstack-qa09:21
*** Guest45232 is now known as BobBall09:22
*** drptbl has joined #openstack-qa09:24
*** jordanP has joined #openstack-qa09:27
openstackrecheckConsole logs not available after 13:24s for gate-tempest-dsvm-neutron-full 213112,19,04e26e409:28
openstackrecheckopenstack/nova change: https://review.openstack.org/233076 failed because of: gate-grenade-dsvm-multinode: unrecognized error, gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1463631, gate-tempest-dsvm-large-ops: https://bugs.launchpad.net/bugs/1500615, gate-tempest-dsvm-neutron-full: unrecognized error09:29
openstackLaunchpad bug 1463631 in grenade "60_nova/resources.sh:106:ping_check_public fails intermittently" [Undecided,Confirmed]09:29
openstackLaunchpad bug 1500615 in OpenStack Compute (nova) "Large Ops scenario is taking too long" [High,Confirmed]09:29
*** vgridnev has quit IRC09:32
*** vgridnev has joined #openstack-qa09:33
*** tochi has quit IRC09:36
*** markvoelker has joined #openstack-qa09:42
*** jacky-zhang has quit IRC09:42
openstackrecheckConsole logs not available after 13:26s for gate-tempest-dsvm-postgres-full 248989,76,7e83ffc09:43
openstackrecheckopenstack/keystone change: https://review.openstack.org/266396 failed because of: gate-grenade-dsvm: https://bugs.launchpad.net/bugs/1526737 https://bugs.launchpad.net/bugs/1532809 https://bugs.launchpad.net/bugs/135557309:44
openstackLaunchpad bug 1526737 in OpenStack Compute (nova) "Race between nova-network and dnsmasq allocating ips" [High,In progress] - Assigned to Dan Smith (danms)09:44
openstackLaunchpad bug 1532809 in OpenStack Compute (nova) "Gate failures when DHCP lease cannot be acquired" [Undecided,New]09:44
openstackLaunchpad bug 1355573 in OpenStack Compute (nova) "tempest volume scenarios periodically fail /w SSHTimeout" [Undecided,New]09:44
*** sshnaidm has joined #openstack-qa09:44
*** cp16net has quit IRC09:46
*** cp16net has joined #openstack-qa09:47
*** jordanP has quit IRC09:47
*** markvoelker has quit IRC09:48
*** zhangjn has quit IRC09:48
*** jordanP has joined #openstack-qa09:51
*** zhangjn has joined #openstack-qa09:52
*** Liuqing has quit IRC09:52
*** zhangjn has quit IRC09:53
openstackrecheckConsole logs not available after 13:28s for gate-grenade-dsvm-multinode 270482,5,fd4c14c09:57
openstackgerritIgor Duarte Cardoso proposed openstack-dev/devstack: Automate OVS bridge creation for multiple mappings  https://review.openstack.org/26979709:58
*** tosky has quit IRC09:59
*** hoangcx has quit IRC10:01
openstackgerritIgor Duarte Cardoso proposed openstack-dev/devstack: Automate OVS bridge creation for multiple mappings  https://review.openstack.org/26979710:02
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-qa10:02
*** yamahata has quit IRC10:04
openstackgerritMasayuki Igawa proposed openstack/openstack-health: Add parameters for options in URL  https://review.openstack.org/27064310:05
*** zhangjn has joined #openstack-qa10:05
openstackrecheckConsole logs not available after 13:22s for gate-nova-python27 270658,1,e9330bc10:11
*** achanda has joined #openstack-qa10:11
*** dtantsur is now known as dtantsur|brb10:15
*** achanda has quit IRC10:17
openstackgerritJoseph Lanoux proposed openstack/tempest: Add a PINGABLE and a SSHABLE waiters  https://review.openstack.org/25951510:19
*** Liuqing has joined #openstack-qa10:24
*** aix has joined #openstack-qa10:24
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-neutron-dvr 259746,12,2e4827110:24
masayukigmtreinish: oomichi: If you have the time, could you have a look this? https://review.openstack.org/#/c/268328/10:27
*** StevenK has quit IRC10:29
*** rm_work has quit IRC10:29
oomichimasayukig: ps3 which you updated seems good for me.10:30
oomichipretty readable10:31
oomichimasayukig: +210:32
*** rm_work has joined #openstack-qa10:32
masayukigthanks :)10:32
oomichimasayukig: can you take a look at https://review.openstack.org/#/c/263982 ? not so difficult ;)10:33
*** zhangjn has quit IRC10:33
masayukigoomichi: sure10:33
oomichimasayukig: 3q10:33
*** gszasz has joined #openstack-qa10:36
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm 213094,14,6a6f2ff10:38
*** markvoelker has joined #openstack-qa10:39
*** amotoki has quit IRC10:40
*** drptbl has quit IRC10:40
masayukigoomichi: Test method's names are out of scope, right? such as test_add_list_remove_router_on_l3_agent(), _remove_network_from_dhcp_agent(), etc.10:41
oomichimasayukig: oops, nice point. please -1 on that because the patch doesn't get +2 at all at this time10:41
masayukigoomichi: oh, really? sure.10:42
oomichimasayukig: sorry, waite10:42
oomichimasayukig: sorry, matte10:42
oomichimasayukig: is that api tests or scenario tests, or unit tests?10:43
masayukigoomichi: api tests10:43
*** markvoelker has quit IRC10:44
oomichimasayukig: I am thinking it is difficult to change api/scenario test names because tempest consumers were specifying the tests by test names before10:44
oomichimasayukig: refstack has been changed  for specifying tests  with uuid IIUC, but I am not sure the other consumers also do that already10:44
oomichimasayukig: so in this case, I'd like to avoid changing the test names10:45
*** drptbl has joined #openstack-qa10:45
masayukigoomichi: hmm, sure. I've already put +2 on it.10:45
oomichimasayukig: thanks :)10:45
openstackrecheckConsole logs not available after 13:24s for gate-nova-python27 250757,14,398abf910:51
*** cdent has joined #openstack-qa10:51
*** StevenK has joined #openstack-qa10:53
*** tosky has joined #openstack-qa10:57
*** e0ne has joined #openstack-qa10:58
*** dims has joined #openstack-qa11:04
openstackrecheckConsole logs not available after 13:24s for gate-tempest-pep8 217177,23,440c3c911:05
*** aix has quit IRC11:07
*** aysyd has joined #openstack-qa11:07
*** _nadya_ has quit IRC11:12
jordanPoomichi, masayukig while you are on a review spree. Could you have a quick look at https://review.openstack.org/#/c/265343 it's short11:14
oomichijordanP: sure, looking11:16
jordanPty11:16
jordanPthe general idea is that there should be one way, and only one way to create a test serve11:16
jordanPr11:16
jordanPimo we should make this a general rule, -1 patches that add yet another wrapper around common method11:17
openstackrecheckConsole logs not available after 13:23s for gate-grenade-dsvm-multinode 223953,6,2e7ff0411:18
oomichijordanP: nice patch :)11:20
oomichijordanP: I am not so strict for blocking patches which use bare client methods11:21
oomichijordanP: sometimes, wrapper methods make problems difficult to be investigated11:21
jordanPoomichi, yeah okay if the bare client is used inside the test; not inside another wrapper/utility method11:22
oomichijordanP: case by case11:22
jordanPyep, you're right11:22
*** _nadya_ has joined #openstack-qa11:22
oomichijordanP: btw, thanks for https://review.openstack.org/#/c/263982/ :)11:23
jordanPyou're welcome11:23
*** ajo has quit IRC11:24
*** ajo has joined #openstack-qa11:25
openstackgerritGhanshyam Mann proposed openstack/qa-specs: Sync requirements from global requirements  https://review.openstack.org/27074411:28
*** Poornima has quit IRC11:28
*** vgridnev has quit IRC11:31
*** vgridnev has joined #openstack-qa11:31
*** yamamoto_ has quit IRC11:31
openstackrecheckConsole logs not available after 13:24s for gate-grenade-dsvm-multinode 259931,7,54c9f8211:31
*** rfolco has joined #openstack-qa11:32
*** markvoelker has joined #openstack-qa11:34
gmannoomichi: https://review.openstack.org/270744, https://review.openstack.org/#/c/270745/1 https://review.openstack.org/#/c/270720/11:35
gmannoomichi: lets see how it goes11:35
*** ccarmack has joined #openstack-qa11:36
*** markvoelker has quit IRC11:39
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-full-liberty 267642,5,12252c811:45
openstackgerritGhanshyam Mann proposed openstack/tempest: Separate base microversion client and compute client  https://review.openstack.org/26505111:47
*** drptbl has quit IRC11:47
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-neutron-full 233622,6,84d40d111:59
*** drptbl has joined #openstack-qa11:59
*** aix has joined #openstack-qa12:03
*** boris-42 has quit IRC12:03
*** drptbl has quit IRC12:03
*** cdent has quit IRC12:05
*** drptbl has joined #openstack-qa12:05
*** cdent has joined #openstack-qa12:06
*** coolsvap is now known as coolsvap|away12:07
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-cells 270686,1,342e2a012:12
*** bkopilov has quit IRC12:13
*** dtantsur|brb is now known as dtantsur12:13
*** achanda has joined #openstack-qa12:14
*** vgridnev has quit IRC12:16
*** vgridnev has joined #openstack-qa12:18
*** raildo-afk is now known as raildo12:20
*** achanda has quit IRC12:20
*** erlon has joined #openstack-qa12:24
*** ylobankov has joined #openstack-qa12:24
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm 245369,6,c4bda7812:25
*** oomichi is now known as oomichi_away12:26
*** markvoelker has joined #openstack-qa12:29
*** oomichi_away has quit IRC12:31
*** ylobankov has quit IRC12:33
*** markvoelker has quit IRC12:35
*** jordanP has quit IRC12:36
openstackrecheckConsole logs not available after 13:37s for gate-tempest-dsvm-neutron-full 267725,2,816b6df12:39
*** ylobankov has joined #openstack-qa12:41
openstackgerritItzik Brown proposed openstack/tempest: qos basic scenario  https://review.openstack.org/21717712:41
*** ccarmack has left #openstack-qa12:45
*** ylobankov has quit IRC12:50
openstackrecheckConsole logs not available after 13:28s for gate-tempest-dsvm-neutron-full-kilo 264323,8,422e68e12:53
*** tochi has joined #openstack-qa12:56
openstackgerritAlex Petrov proposed openstack/tempest: Add a test for checking a cascade (enable-disable) operation for a project  https://review.openstack.org/27017712:57
*** tochi has quit IRC12:57
openstackgerritAlex Petrov proposed openstack/tempest: Add a test for 'Delete subtree' Identity API V3 method  https://review.openstack.org/26970113:01
openstackgerritAlex Petrov proposed openstack/tempest: Add a test for checking a cascade (enable-disable) operation for a project  https://review.openstack.org/27017713:03
*** ihrachys has quit IRC13:04
*** ihrachys has joined #openstack-qa13:05
openstackrecheckConsole logs not available after 13:26s for gate-grenade-dsvm 265601,12,72d853713:06
*** jamielennox is now known as jamielennox|away13:07
*** amotoki has joined #openstack-qa13:12
openstackgerritYusuke Hayashi proposed openstack-dev/devstack: Rdo-release install failure under proxy  https://review.openstack.org/27061213:12
*** nicholasgoracke has joined #openstack-qa13:15
*** weshay_xchat has joined #openstack-qa13:18
*** weshay_xchat is now known as weshay13:18
*** nicholasgoracke has quit IRC13:20
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 233076,27,4d023ec13:20
*** markvoelker has joined #openstack-qa13:24
*** markvoelker has quit IRC13:29
*** nicholasgoracke has joined #openstack-qa13:32
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-neutron-full 246600,17,cfdc5fb13:34
*** vgridnev has quit IRC13:34
openstackgerritAlex Petrov proposed openstack/tempest: Add a test for checking a cascade (enable-disable) operation for a project  https://review.openstack.org/27017713:36
*** nicholasgoracke has quit IRC13:36
*** vgridnev has joined #openstack-qa13:36
*** annemccormick has joined #openstack-qa13:46
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-neutron-full 252458,13,5e1f14613:47
openstackgerritAlex Petrov proposed openstack/tempest: Add a test for 'Delete subtree' Identity API V3 method  https://review.openstack.org/26970113:47
*** yamamoto has joined #openstack-qa13:48
*** yamamoto has quit IRC13:49
*** bkopilov has joined #openstack-qa13:49
cdentI have really mixed feelings on this "make jobs non-voting" when they regularly fail stuff. I get that we need to not stop the whole machine. But on the other hand unless it hurts, nothing will get fixed. So my head spins.13:50
*** markvoelker has joined #openstack-qa13:50
*** yamamoto has joined #openstack-qa13:52
*** markvoelker has quit IRC13:53
*** markvoelker has joined #openstack-qa13:53
yfried_what is the "Expected Fail" and how can I mark tests that are expected to fail?13:53
*** Liuqing has quit IRC13:56
*** markvoelker has quit IRC13:57
*** markvoelker has joined #openstack-qa13:59
*** ansiwen is now known as ansiwen|afk13:59
sdaguecdent: make it hurt so that it gets fixed assumes that the people getting hurt can fix things14:00
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-neutron-full 253033,13,a4ee1b414:00
sdagueotherwise it's just torture14:00
sdagueor vengence14:01
cdentone could argue that a tortured people's will eventually revolt against the powers that torture them14:01
cdentbut history doesn't really bear that out14:01
cdentthus the head spin14:01
*** weshay has quit IRC14:01
cdentbut my inner tester is really weirded out by the state of testing14:02
cdentthat part of me thinks it should be impossible to recheck14:02
*** shardy has quit IRC14:03
sdaguewhen the tests are deterministic, I'm with you14:03
cdentexactly, they aren't, that's a bug14:04
sdagueand when the test results can be overridden by humans, because humans have determined that this is not an issue14:04
cdent(we've had this talk before and agreed that my idealism is nice, but not a part of the current world)14:04
sdagueyeh. We could prune everything back to deterministic only tests, it would mean to drop all the devstack ones14:05
sdaguewe'd merge code a lot faster14:05
*** shardy has joined #openstack-qa14:05
*** pc_m has quit IRC14:07
*** lezbar has quit IRC14:07
*** lezbar__ has joined #openstack-qa14:07
*** pc_m has joined #openstack-qa14:07
cdentI know that at least some of the lack of determinism is the nature of the system, but some of it seems to be because of bad test design and flaky infrastructure. Given that our project is devoted to making better infrastructure, that seems...weird14:07
sdaguecdent: no real argument, but there are very few people working on that14:08
*** edmondsw has joined #openstack-qa14:08
cdentyeah, we've covered that ground before too ... :(14:08
cdentI think the economics of the situation need adjustment, but I'm unable to identify the variables, nor the fulcrums14:09
*** bochi-michael has quit IRC14:11
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 267382,1,98ec8dd14:14
openstackgerritIhar Hrachyshka proposed openstack-dev/devstack: neutron: enable ml2 mtu calculation mechanism  https://review.openstack.org/26760414:15
*** sabeen1 has joined #openstack-qa14:17
*** achanda has joined #openstack-qa14:17
*** weshay_xchat has joined #openstack-qa14:18
*** jlanoux has quit IRC14:19
*** nicholasgoracke has joined #openstack-qa14:20
*** dkranz has joined #openstack-qa14:20
*** sabeen3 has joined #openstack-qa14:21
*** achanda has quit IRC14:23
*** sabeen1 has quit IRC14:24
*** raildo is now known as raildo-afk14:24
*** mriedem has joined #openstack-qa14:28
openstackrecheckConsole logs not available after 13:24s for gate-tempest-dsvm-neutron-full 270695,1,3d0ccca14:28
*** yamamoto has quit IRC14:28
*** drptbl has quit IRC14:28
*** ihrachys has quit IRC14:28
*** nicholasgoracke has quit IRC14:36
*** nicholasgoracke has joined #openstack-qa14:36
*** ihrachys has joined #openstack-qa14:39
*** drptbl has joined #openstack-qa14:40
*** aopincar has quit IRC14:41
*** regXboi has joined #openstack-qa14:41
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-postgres-full 269624,10,ae95b8514:41
*** jffische has joined #openstack-qa14:44
*** itzikb has quit IRC14:44
*** jordanP has joined #openstack-qa14:48
*** binbincong has quit IRC14:51
*** binbincong has joined #openstack-qa14:52
*** jecarey has joined #openstack-qa14:53
*** mtanino has joined #openstack-qa14:54
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-neutron-src-glance_store 270701,1,ad28fb914:55
*** malini has joined #openstack-qa14:59
*** jlanoux has joined #openstack-qa14:59
*** vgridnev has quit IRC15:01
*** ylobankov has joined #openstack-qa15:02
*** sigmavirus24_awa is now known as sigmavirus2415:03
*** yamamoto has joined #openstack-qa15:03
*** nicholas_ has joined #openstack-qa15:04
*** nicholasgoracke has quit IRC15:05
*** boris-42 has joined #openstack-qa15:07
*** nicholasgoracke has joined #openstack-qa15:07
*** neeti has quit IRC15:08
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-full-ceph 269135,2,bb9b1bf15:08
*** jasonsb has joined #openstack-qa15:09
*** nicholas_ has quit IRC15:10
*** gamado has quit IRC15:12
*** rbak has joined #openstack-qa15:13
*** ylobankov has quit IRC15:15
openstackgerritMatthew Treinish proposed openstack-dev/devstack: Revert "set the validation path to fixed for n-net"  https://review.openstack.org/27085615:16
*** ylobankov has joined #openstack-qa15:16
openstackgerritMatthew Treinish proposed openstack-dev/devstack: Revert "set the validation path to fixed for n-net"  https://review.openstack.org/27085715:16
mtreinishmriedem, clarkb: https://review.openstack.org/#/c/270857/15:17
*** ylobankov has quit IRC15:17
*** achanda has joined #openstack-qa15:20
*** achanda has quit IRC15:20
*** ylobankov has joined #openstack-qa15:20
*** achanda has joined #openstack-qa15:20
*** ccarmack has joined #openstack-qa15:20
*** jaypipes has joined #openstack-qa15:21
*** ChanServ sets mode: +o jaypipes15:21
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-large-ops 270686,1,6cd9ff615:22
openstackgerritMonty Taylor proposed openstack-dev/devstack: Remove leftover homedir clouds.yaml files  https://review.openstack.org/27086015:22
*** jasonsb has quit IRC15:22
*** ylobankov has quit IRC15:23
*** zz_dimtruck is now known as dimtruck15:26
*** jecarey has quit IRC15:26
*** jecarey has joined #openstack-qa15:27
*** jecarey has quit IRC15:28
*** jecarey_ has joined #openstack-qa15:28
*** mragupat has joined #openstack-qa15:28
*** jlwhite has joined #openstack-qa15:29
*** pglass has joined #openstack-qa15:29
*** ansiwen|afk has quit IRC15:30
*** ylobankov has joined #openstack-qa15:32
*** nicholas_ has joined #openstack-qa15:32
*** mfisher_ora has joined #openstack-qa15:32
*** achanda has quit IRC15:33
*** anshul has quit IRC15:35
openstackrecheckConsole logs not available after 13:33s for gate-tempest-dsvm-postgres-full 254703,22,298b9f115:35
*** nicholas_ has quit IRC15:35
*** nicholasgoracke has quit IRC15:36
*** nicholasgoracke has joined #openstack-qa15:36
*** nicholasgoracke has quit IRC15:36
*** drptbl has quit IRC15:36
*** nicholasgoracke has joined #openstack-qa15:36
*** asselin__ has quit IRC15:37
*** markvoelker has quit IRC15:38
*** drptbl has joined #openstack-qa15:38
*** asselin_ has joined #openstack-qa15:39
*** jecarey_ has quit IRC15:43
*** jecarey has joined #openstack-qa15:43
*** amotoki has quit IRC15:44
*** mragupat has quit IRC15:44
*** mragupat has joined #openstack-qa15:45
*** jecarey has quit IRC15:48
mtreinishmriedem, clarkb, dims: https://jenkins03.openstack.org/job/gate-grenade-dsvm-multinode/1897/consoleFull the revert caught bluebox on the first go around :)15:48
mriedemfailed already?15:49
openstackrecheckConsole logs not available after 13:23s for gate-grenade-dsvm-multinode 269764,2,3a7088f15:49
mriedemmtreinish: oh it passed15:49
mriedemnice!15:49
mriedemoh wait, no15:49
mriedemsetup was successful, test is still running15:49
mtreinishwell it hasn't gotten there yet15:49
mriedemso we play the waiting game15:49
mriedemwhich sucks15:49
mtreinishbut, if it works, then we'll know15:49
mriedemwho has hungry hungry hippos15:49
*** ylobankov has quit IRC15:55
*** achanda has joined #openstack-qa15:55
*** shardy has quit IRC15:57
openstackgerritSean Dague proposed openstack-dev/devstack: Always cache tokens in a shared memcache  https://review.openstack.org/27047415:58
*** shardy has joined #openstack-qa15:59
*** jecarey has joined #openstack-qa16:00
sdaguemtreinish: so this is the thing that uses floating ips all the time for grenade again?16:00
sdaguethat basically makes debugging those things an order of magnitude harder16:00
clarkbsdague: you cannot use the private IPs to ssh from tempest16:01
mtreinishsure, I'm not saying we land it as a straight revert. This is just a test to confirm for right now16:01
*** markvoelker has joined #openstack-qa16:01
*** achanda has quit IRC16:02
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-neutron-full 268701,2,8f7584c16:02
*** sshnaidm has quit IRC16:04
*** jasonsb has joined #openstack-qa16:04
*** SridharG has quit IRC16:07
*** pcrews has joined #openstack-qa16:07
mtreinishclarkb: btw, did you see http://logs.openstack.org/28/270528/1/check/gate-grenade-dsvm-multinode/4c3bd9b/logs/grenade.sh.txt.gz#_2016-01-21_00_28_35_316 (that was with the patch which did s/PRIVATE/PUBLIC/)16:09
*** ylobankov has joined #openstack-qa16:09
mtreinishbesides breaking every other job, it doesn't look like it fixed the issue16:09
*** rcernin has quit IRC16:09
clarkbmtreinish: I hadn't16:10
*** yamahata has joined #openstack-qa16:10
*** yamamoto has quit IRC16:11
*** yamamoto has joined #openstack-qa16:11
*** yamamoto has quit IRC16:11
clarkbdoes that not use the 172.24.4.0/23 network?16:12
*** Serlex has quit IRC16:12
*** yfried_ has quit IRC16:12
*** amotoki has joined #openstack-qa16:13
*** ylobankov has quit IRC16:14
mtreinishclarkb: d'oh, it's grenade: http://logs.openstack.org/28/270528/1/check/gate-grenade-dsvm-multinode/4c3bd9b/logs/old/tempest_conf.txt.gz16:14
*** jasonsb has quit IRC16:14
mtreinishso it's using liberty devstack to configure tempest :/16:14
*** shausy has quit IRC16:14
sdaguecan someone summarize the perceived issue?16:14
clarkbsdague: tempest cannot ssh to the VM private networks if the VM is on the other compute node16:15
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 259515,15,75ae53716:15
sdaguewhy did this just pop up?16:15
sdaguewhat changed?16:16
clarkbsdague: tempest stoppped ssh ing to the public net16:16
clarkbI summarized in a comment on 27052816:16
mtreinishsdague: its only really showing up on bluebox and internap, which I think accounts for the recentish spike16:16
clarkbit won't be a 100% fail because ssh to the local VMs will work16:17
sdagueclarkb: when did that change?16:17
sdaguebecause I though that was months ago16:17
mtreinishclarkb: well, it is 100% fail on those providers16:17
clarkbI have no idea16:17
clarkbI didn't make the change16:17
mtreinishsdague: that change landed in dec16:18
*** jpas has quit IRC16:18
sdagueok16:18
clarkball I know is that that is not supported by the networking16:18
clarkband the networking is failing16:18
sdagueok, but it was working in december16:18
mtreinisherr, no the liberty one landed more recently: https://review.openstack.org/#/c/263685/16:18
mtreinishwhich would be the cause of the issue because it's grenade16:18
*** bnemec has quit IRC16:18
sdagueok, so that's in the time window16:19
sdaguethat's a what changed16:19
*** bnemec has joined #openstack-qa16:20
sdaguealthough that's only synchronizing a value which is listed other places16:20
sdagueso there is probably a bigger unwind16:20
mtreinishwell the compute ssh_connect_method is a deprecated group for the validation option16:21
mtreinishhow does oslo.config handle that?16:21
*** vgridnev has joined #openstack-qa16:21
sdaguemtreinish: before it was set, it was falling back to some default which was the other way16:21
sdagueeven though we've told the config to go with fixed ips16:22
Sam-I-Amsdague: nevermind, you're talking about the thing in here :)16:22
Sam-I-Amthe thing holding up 27041716:22
Sam-I-Amtoomanyircchannels16:22
*** krtaylor has quit IRC16:23
*** raildo-afk is now known as raildo16:24
*** aix has quit IRC16:26
mtreinishsdague: oh, it's not a deprecated option: https://github.com/openstack/tempest/blob/master/tempest/config.py#L626-L63316:27
mtreinishI don't know if compute.ssh_connect_method was ever actually an option16:28
*** jvrbanac_ has joined #openstack-qa16:29
openstackrecheckConsole logs not available after 13:24s for gate-tempest-dsvm-neutron-full 247396,4,b26621416:29
mtreinishgoing back to grizzly-eol it didn't exist there either... :/16:29
sdagueyeh, I don't know, anyway, there is clean up there needed16:29
sdagueI approved the revert16:29
*** jvrbanac_ has quit IRC16:30
mtreinishclarkb: well, it still failed, but it's using the public ip now: http://logs.openstack.org/57/270857/1/check/gate-grenade-dsvm-multinode/9a36e7d//logs/grenade.sh.txt.gz#_2016-01-21_16_23_18_52616:31
*** jvrbanac has joined #openstack-qa16:32
clarkbthe console shows it did seem to get networking, next I guess check if the floating ip attach went well according to nova16:34
clarkbya I can see it on the subnode n-net log16:35
*** krtaylor has joined #openstack-qa16:35
clarkbnow to check that we install the bridges correctly16:36
sdagueright, now I think you are just inside the ssh doesn't work bug, that has yet to be solved.16:38
clarkbhttp://logs.openstack.org/57/270857/1/check/gate-grenade-dsvm-multinode/9a36e7d/console.html#_2016-01-21_15_46_57_048 and http://logs.openstack.org/57/270857/1/check/gate-grenade-dsvm-multinode/9a36e7d/console.html#_2016-01-21_15_47_18_020 look good to me16:38
*** jlwhite has quit IRC16:38
mtreinishclarkb: http://logs.openstack.org/57/270857/1/check/gate-grenade-dsvm-multinode/9a36e7d//logs/old/screen-n-api.txt.gz#_2016-01-21_16_16_27_822 is the floating ip attach request16:39
sdaguehonestly, the api path has never been an issue with these, jump straight to the world dump and look at iptables16:40
mtreinishalthough I don't see anything in the api log after the 202 for that request id16:40
sdaguemtreinish: the info line at the completion of the request16:41
sdagueso it should be the last thing with req-id16:41
mtreinishwhich is a 202, which means it should have done a cast somewhere16:41
mtreinishand the req id should be in the n-net or n-cpu logs16:42
openstackrecheckConsole logs not available after 13:24s for gate-tempest-dsvm-cells 253353,7,eba1ccd16:42
*** ccneill_ has joined #openstack-qa16:42
sdaguesure, and it is, on the subnode16:47
sdaguehttp://logs.openstack.org/57/270857/1/check/gate-grenade-dsvm-multinode/9a36e7d//logs/subnode-2/old/screen-n-net.txt.gz#_2016-01-21_16_16_25_19416:47
mtreinishugh, forgot it's multinode. I'm still not used to debuging with more than one set of logs16:48
*** grafuls has quit IRC16:49
*** rcernin has joined #openstack-qa16:49
*** ccneill_ is now known as ccneill16:49
sdagueyep, which is part of why this is super hard16:49
sdaguebut, honestly, I've not found anything in the call path that would be problematic to date16:50
*** yfried_ has joined #openstack-qa16:51
*** vgridnev has quit IRC16:51
*** bnemec has quit IRC16:52
clarkbit could be mtu issues similar to what neutron was running into16:52
clarkbsince nova net doesn't have an option to set those via dhcp iirc16:52
clarkbif it does we can just set that instead16:52
*** dwalleck has joined #openstack-qa16:55
openstackrecheckConsole logs not available after 13:22s for gate-glance_store-python27 270734,1,bb3702916:56
mtreinishclarkb: google shows some mtu stuff was added to n-net in juno by vish: https://review.openstack.org/#/q/topic:bp/better-support-for-multiple-networks,n,z16:56
clarkbwe can try setting that then16:56
*** vgridnev has joined #openstack-qa16:57
*** davidlenwell has quit IRC16:58
mtreinishclarkb: it looks like it adds an api arg to set the mtu when the network is created: https://review.openstack.org/#/c/93759/30/doc/api_samples/os-extended-networks/network-create-req.json16:59
clarkbmtreinish: that doesn't look like it is setting the mtu in the VMs via dhcp though (dhcp is set to false)16:59
*** davidlenwell has joined #openstack-qa16:59
mtreinishyeah, I didn't read closely enough17:00
clarkbwe need to tell cirros that its mtu is smaller than 150017:01
*** vgridnev has quit IRC17:01
*** bnemec has joined #openstack-qa17:02
mtreinishah, there is a network_device_mtu option in nova, or there was17:02
mtreinishoh, qa meeting time, I'll dig up a link after17:02
jordanPmtreinish, yes :)17:02
*** vgridnev has joined #openstack-qa17:02
mtreinishclarkb: https://github.com/openstack/nova/blob/a1151c5529fb48dfe0ba2971118394e73c27a982/nova/objects/network.py#L3417:03
clarkbmtreinish: will that affect how dhcp leases are hadned out? I suppose we can try it and see17:04
*** blogan is now known as patient-zer0-bl017:06
*** patient-zer0-bl0 is now known as patient-0-bl0gan17:06
*** dtantsur is now known as dtantsur|afk17:06
*** coolsvap|away has quit IRC17:07
*** hitalia has joined #openstack-qa17:08
clarkbI think we naively set the mtu to 1450 on the tunnel side17:08
clarkbso we would want to tell cirros that or less17:08
clarkbhttp://logs.openstack.org/57/270857/1/check/gate-grenade-dsvm-multinode/9a36e7d/console.html#_2016-01-21_15_46_41_59117:08
openstackrecheckConsole logs not available after 13:23s for gate-horizon-pep8 244991,13,c9bc01917:09
clarkbanother option is to supply that via config drive and a user script thing17:10
*** salv-orlando has joined #openstack-qa17:10
clarkbor I guess metadata service for multinode since config drive isn't migrateable17:10
*** jlwhite has joined #openstack-qa17:11
openstackgerritSean Dague proposed openstack-dev/devstack: Always cache tokens in a shared memcache  https://review.openstack.org/27047417:12
sdagueclarkb: config drive is now migratable I believe17:12
*** hitalia has quit IRC17:13
*** ccarmack has quit IRC17:14
*** ccarmack has joined #openstack-qa17:14
clarkbworth noting that if it is an MTU issue that bridging the private address space to the hypervisors will have the same problem.17:18
*** Swami has joined #openstack-qa17:21
*** jogo has quit IRC17:22
openstackgerritAlex Petrov proposed openstack/tempest: Identity V3: missing api coverage for 'Get Endpoint' method  https://review.openstack.org/27092817:22
openstackrecheckConsole logs not available after 13:23s for gate-cinder-python34 270730,1,2c85ab717:22
*** mragupat_ has joined #openstack-qa17:23
*** cjmartinez is now known as castulo17:23
*** jogo has joined #openstack-qa17:24
*** mragupat has quit IRC17:26
*** sigmavirus24 is now known as sigmavirus24_awa17:27
*** armax has joined #openstack-qa17:28
*** nicholas_ has joined #openstack-qa17:29
*** nicholasgoracke has quit IRC17:31
*** drptbl has quit IRC17:32
openstackrecheckConsole logs not available after 13:25s for gate-nova-python34 270658,2,d5c0be317:36
*** e0ne has quit IRC17:37
*** safchain has quit IRC17:37
*** hemna has quit IRC17:38
openstackgerritClint Adams proposed openstack-dev/devstack: Add script to autogenerate doc/source/plugin-registry.rst  https://review.openstack.org/26992017:40
*** stevebaker has quit IRC17:43
*** _nadya_ has quit IRC17:44
*** ccarmack has quit IRC17:47
openstackrecheckConsole logs not available after 13:23s for gate-grenade-dsvm-multinode 248989,80,474938a17:49
*** davidlenwell has quit IRC17:50
*** jasonsb has joined #openstack-qa17:52
*** gszasz has quit IRC17:54
*** davidlenwell has joined #openstack-qa17:55
*** fesp has joined #openstack-qa17:56
*** _nadya_ has joined #openstack-qa17:58
*** yfried_ has quit IRC17:58
*** ihrachys has quit IRC18:00
*** jlanoux has quit IRC18:00
*** aix has joined #openstack-qa18:01
openstackrecheckConsole logs not available after 13:26s for gate-tempest-dsvm-neutron-full 265051,7,7c88ee018:03
*** sabeen3 has quit IRC18:04
*** mragupat_ has quit IRC18:09
*** gema_ is now known as gema18:09
*** mragupat has joined #openstack-qa18:09
*** nicholas_ has quit IRC18:10
*** jordanP has quit IRC18:12
*** mragupat has quit IRC18:13
*** malini has quit IRC18:15
openstackrecheckConsole logs not available after 13:21s for gate-grenade-dsvm-multinode 270077,1,f854f8918:17
*** ccarmack has joined #openstack-qa18:18
*** itzikb has joined #openstack-qa18:19
*** nicholasgoracke has joined #openstack-qa18:20
*** ccarmack has quit IRC18:22
*** ccarmack has joined #openstack-qa18:27
*** stevebaker has joined #openstack-qa18:28
openstackrecheckConsole logs not available after 13:28s for gate-tempest-dsvm-neutron-full 246600,17,c7836d918:30
*** rossella_s has quit IRC18:31
*** tosky_ has joined #openstack-qa18:31
*** rossella_s has joined #openstack-qa18:31
*** woodster_ has joined #openstack-qa18:32
*** fesp has quit IRC18:32
*** tosky has quit IRC18:32
*** tosky_ is now known as tosky18:32
*** ccarmack has quit IRC18:36
*** dtroyer has quit IRC18:38
*** edmondsw has quit IRC18:41
*** dtroyer has joined #openstack-qa18:41
*** malini has joined #openstack-qa18:42
openstackrecheckConsole logs not available after 13:28s for gate-horizon-npm-run-lint 259022,8,88062a518:44
*** gfidente is now known as gfidente|afk18:45
*** sabeen has joined #openstack-qa18:46
*** mriedem has quit IRC18:48
*** ylobankov has joined #openstack-qa18:48
*** sabeen3 has joined #openstack-qa18:49
*** ccneill has quit IRC18:50
*** mriedem has joined #openstack-qa18:51
*** sabeen has quit IRC18:51
*** hemna has joined #openstack-qa18:55
*** e0ne has joined #openstack-qa18:57
*** pvaneck has joined #openstack-qa18:57
openstackrecheckConsole logs not available after 13:27s for gate-grenade-dsvm-multinode 233076,27,c62658e18:57
*** nicholas_ has joined #openstack-qa18:58
*** nicholasgoracke has quit IRC18:58
*** ccneill has joined #openstack-qa18:58
*** santoshk has joined #openstack-qa18:58
*** tosky has quit IRC19:01
*** sigmavirus24_awa is now known as sigmavirus2419:02
*** mragupat has joined #openstack-qa19:03
*** jasonsb has quit IRC19:08
*** jhenner has joined #openstack-qa19:11
openstackrecheckConsole logs not available after 13:27s for gate-nova-docs 253556,10,65b593619:11
*** yamahata has quit IRC19:12
*** fesp has joined #openstack-qa19:13
*** sshnaidm has joined #openstack-qa19:14
*** vhosakot has joined #openstack-qa19:14
*** rbak_ has joined #openstack-qa19:16
*** dwalleck has quit IRC19:17
*** hitalia has joined #openstack-qa19:17
*** aix has quit IRC19:19
*** rbak has quit IRC19:20
*** ylobankov has quit IRC19:21
*** cdent has quit IRC19:22
*** ylobankov has joined #openstack-qa19:22
*** malini has quit IRC19:23
openstackrecheckConsole logs not available after 13:24s for gate-grenade-dsvm-multinode 254950,13,0daa37819:24
*** ylobankov has quit IRC19:25
*** ylobankov has joined #openstack-qa19:28
sigmavirus24jecarey: did you see https://review.openstack.org/270544 ?19:28
*** edmondsw has joined #openstack-qa19:29
*** jffische has quit IRC19:30
santoshkHello everyone, does anyone do any testing on system with 2 devstack setups with each having its own keystone services...19:32
santoshk  Testcase we are trying to test is basically we will have 2 (or more) such devstack setup where tacker server runs only on one devstack and will be able to spawn vnfs on both devstacks (local and also remote)19:33
santoshkAre there any projects which do work like this..19:33
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm-multinode 202617,43,e3c1ca019:38
sdagueclarkb: I've been poking off / on with this issue for ever. Is there any signal we can send from a node that will hold it? Because people staring at things and getting lucky is too tedious. We need to see one of these envs live.19:39
*** gfidente|afk has quit IRC19:39
anteayasdague: I think it would have to be a message to nodepool to not delete the node, however there would need to be some place for nodepool to say, hey I've got a node someone needs to look at19:40
anteayaso something needs to tell nodepool to hold the node19:40
sdagueright, we know what the fails look like inside the node.19:41
*** yamahata has joined #openstack-qa19:41
clarkbthere isnt currently anything like thst19:41
sdaguethey are good enough to pattern match on19:41
sdaguebut today the meatware is one person watch a bunch of job streams carefully19:42
sdaguesend a message to #infra19:42
sdaguehope someone sees it fast enough to get the hold19:42
*** salv-orlando has quit IRC19:42
sdagueprobably write off a day looking for the failure you care about.19:42
sdaguewhich is why no one is looking any further at this issue19:43
clarkbbased on neutrons testing I bet it is an mtu issue19:44
sdaguecould be19:45
clarkbthe problem with trying to do this automatically is we have little insight to job logs before the job completes19:45
clarkbyou cant poll for all logs on a host as the job runs19:46
sdagueno, we do, it's a running system19:46
sdaguewe can watch those logs19:46
clarkbwhere?19:46
sdagueon the control node19:46
clarkbright you cant easily do thst19:46
sdaguewhy?19:47
clarkbbecause the only connection from slave to zuul/nodepool is gearman19:47
sdagueok19:48
sdagueand we can't make a topic which is "hold me" ?19:48
clarkbso you have to build the logic into the job and return other results or completely change how we run jobs19:48
clarkbsdague topic where?19:49
*** e0ne has quit IRC19:49
sdagueI don't know the side back to nodepool19:50
sdaguebut yes, we put a watcher in devstack-gate that tails the tempest log19:50
clarkbits also a ddos vector19:50
sdagueyes, so is letting anyone upload a gerrit change19:51
openstackrecheckConsole logs not available after 13:21s for gate-grenade-dsvm 269173,3,0568cc319:51
clarkbsdague but current system times you out19:52
sdaguehowever, we're at thousands of hours looking into this, because we have no way to programatically hold and inspect a node when it actually happens19:52
clarkbholds are the anti time out19:52
sdagueso timeout holds after 6 hours19:52
clarkbok19:53
sdaguethe important thing is that we don't put all our debuggers into busy waits getting frustrated looking for exactly the right fail and catching it. We know how to describe that that failure is. We could even flag the hold on / off on days when people say they have time to look at them.19:54
openstackgerritIan Cordasco proposed openstack-dev/hacking: Update flake8 requirement for 0.11.0  https://review.openstack.org/27054419:54
sdagueanyway, just thinking out loud how an ideal system would look here19:54
sdagueto figure out if there are any bits of it we could actually have19:54
clarkbsdague: I think the prefered method is to not busy wait and reproduce directly instead19:55
sdagueclarkb: and exactly how does that happen with multinode jobs?19:55
sdaguethat are timing based19:55
clarkbthe same as single node19:55
clarkbits not timing19:55
clarkbnone of these issues have been timing19:55
clarkbusing unroutable IPs is deterministic19:56
clarkbkeystone not caching is deterministic19:56
sdagueI'm not talking about that issue19:56
sdagueall the easy to debug issues are easy19:56
sdagueI think we can agree on that19:56
sdaguethe "I can't ssh to this thing" which has been around for 3 years19:57
sdaguenot easy19:57
clarkbthen what are we talking about?19:57
clarkbI think it is19:57
sdagueright, so have a dozen people19:57
clarkbI think we have just ignored the answer for so long19:57
clarkbmtus are wonky19:57
sdagueevery 3 months, for the last 3 years19:57
*** ylobankov has quit IRC19:57
*** jecarey_ has joined #openstack-qa19:57
sdagueif you've got it solved for good and have patches, that's great. But we've been down this road before where "oh, clearly it's A"19:58
*** ccarmack has joined #openstack-qa19:58
sdagueand it wasn't A19:58
*** vgridnev has quit IRC19:58
clarkbI dont have patches because I dont think nova supports the fix19:58
*** amotoki has quit IRC19:58
*** amotoki has joined #openstack-qa19:58
clarkbwhich us tell cirros via dhcp what the mtu is19:58
*** jecarey_ has quit IRC19:59
clarkband in fairness there has been >1 issue with this20:00
clarkbdhcp fails sometimes, metadata fails sometimes, NAT fails sometimes etc20:01
*** jecarey__ has joined #openstack-qa20:01
clarkbso we likely have corrected specific problems20:01
*** jecarey has quit IRC20:01
*** nicholas_ has quit IRC20:02
*** austin81 has joined #openstack-qa20:03
*** jecarey__ has quit IRC20:03
sdagueright, but if we had hold mechanisms we could actually connect to the guest over the console, change the mtu on the guest console, and boom it's back. then we would know.20:03
*** jecarey has joined #openstack-qa20:03
clarkbyou can also do thst deterministically on a local install20:04
sdagueok, then write up a reproduce scenario20:04
clarkbsingle node should even be sufficient I think20:04
sdaguegot to run now, family outing.20:04
*** fesp has quit IRC20:04
openstackrecheckConsole logs not available after 13:23s for gate-nova-docs 252389,12,f5a670720:05
*** ylobankov has joined #openstack-qa20:05
*** nicholasgoracke has joined #openstack-qa20:07
*** aysyd has quit IRC20:09
openstackgerritMerged openstack/tempest: [sahara] adding new plugin versions to sahara tempest tests  https://review.openstack.org/25596820:11
*** vhosakot has quit IRC20:11
openstackgerritTim Buckley proposed openstack/openstack-health: Enable debounce in filter fields.  https://review.openstack.org/27101620:11
*** markvoelker has quit IRC20:11
*** rfolco has quit IRC20:11
*** jbernard has quit IRC20:15
*** jbernard has joined #openstack-qa20:16
openstackrecheckConsole logs not available after 13:24s for gate-tempest-dsvm-neutron-full 259746,12,1cb2f4120:18
openstackgerritTim Buckley proposed openstack/openstack-health: Enable caching of run metadata keys.  https://review.openstack.org/27102320:20
*** amotoki has quit IRC20:21
*** dimtruck is now known as zz_dimtruck20:24
*** vgridnev has joined #openstack-qa20:26
*** ylobankov has quit IRC20:29
*** krtaylor has quit IRC20:31
*** markvoelker has joined #openstack-qa20:31
openstackrecheckConsole logs not available after 13:26s for gate-tempest-dsvm-postgres-full 270790,1,32b2d5620:32
*** sabeen3 has quit IRC20:32
*** sabeen has joined #openstack-qa20:33
*** zz_dimtruck is now known as dimtruck20:35
*** jecarey has quit IRC20:36
*** itzikb has quit IRC20:37
*** _nadya_ has quit IRC20:40
*** ylobankov has joined #openstack-qa20:44
*** yarkot has joined #openstack-qa20:45
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-full-ceph 245678,37,d07974b20:45
*** cdent has joined #openstack-qa20:47
*** jamielennox|away is now known as jamielennox20:47
*** mragupat has quit IRC20:47
*** mragupat has joined #openstack-qa20:47
mtreinishtimothyb89: working on responsiveness? :)20:48
*** ylobankov has quit IRC20:50
*** yamahata has quit IRC20:52
*** ylobankov has joined #openstack-qa20:53
*** ccarmack has quit IRC20:54
*** ccarmack has joined #openstack-qa20:58
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-neutron-large-ops 248989,80,f3e8f6020:59
*** nicholasgoracke has quit IRC21:02
*** ccarmack has quit IRC21:03
*** pglass has quit IRC21:06
*** yamahata has joined #openstack-qa21:06
*** erlon has quit IRC21:06
*** pglbutt has joined #openstack-qa21:07
*** sabeen2 has joined #openstack-qa21:07
*** sabeen has quit IRC21:10
*** dimtruck is now known as zz_dimtruck21:11
*** salv-orlando has joined #openstack-qa21:11
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm 269135,2,fde5d6721:12
*** jlwhite has quit IRC21:14
*** jecarey has joined #openstack-qa21:15
*** yarkot has quit IRC21:18
*** annemccormick has quit IRC21:19
*** jlwhite has joined #openstack-qa21:20
*** ylobankov has quit IRC21:21
*** jhenner has quit IRC21:22
*** jlwhite has quit IRC21:23
*** jlwhite has joined #openstack-qa21:23
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-neutron-full 247396,4,3f9ae5421:26
*** nicholasgoracke has joined #openstack-qa21:27
*** zz_dimtruck is now known as dimtruck21:29
*** nicholasgoracke has quit IRC21:33
timothyb89mtreinish: a bit, yes :) still a lot to do, but those were some nice quick fixes21:34
*** jdandrea has joined #openstack-qa21:39
*** markvoelker has quit IRC21:39
openstackrecheckConsole logs not available after 13:26s for gate-tempest-dsvm-full 270441,1,14248ee21:40
*** markvoelker has joined #openstack-qa21:41
*** cdent has quit IRC21:42
*** shardy has quit IRC21:43
*** ajmiller_ has quit IRC21:49
*** krtaylor has joined #openstack-qa21:53
openstackrecheckConsole logs not available after 13:23s for gate-tempest-dsvm-full 206923,40,17713a821:53
*** Swami has quit IRC21:55
*** Swami has joined #openstack-qa21:55
*** dims_ has joined #openstack-qa21:56
*** dims has quit IRC21:57
*** markvoelker has quit IRC22:04
*** nicholasgoracke has joined #openstack-qa22:06
openstackrecheckConsole logs not available after 13:22s for gate-grenade-dsvm 193135,93,1f1f5b022:08
openstackgerritJamie Lennox proposed openstack-dev/devstack: Put service users in a seperate domain  https://review.openstack.org/27105122:09
*** mfisher_ora has quit IRC22:12
sigmavirus24jecarey: thoughts on adding more integration jobs to hacking's gate?22:17
sigmavirus24It's super easy to do22:17
*** regXboi has quit IRC22:19
openstackrecheckConsole logs not available after 13:25s for gate-tempest-dsvm-neutron-dvr 270177,4,656381622:21
*** markvoelker has joined #openstack-qa22:24
*** vgridnev has quit IRC22:25
*** dkranz has quit IRC22:26
jlvillaldtroyer: do you recall if there have been any patches proposed in the past for devstack to set a PS4 variable?22:28
jlvillalTo help with debugging.  For example:  export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'22:28
*** rossella_s has quit IRC22:31
*** rossella_s has joined #openstack-qa22:31
*** ylobankov has joined #openstack-qa22:32
openstackgerritMerged openstack-dev/devstack: Update the vnc config options group 'DEFAULT' to 'vnc'  https://review.openstack.org/21180822:33
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-full 153038,103,f3919af22:35
*** vgridnev has joined #openstack-qa22:36
*** pglbutt has quit IRC22:37
*** weshay_xchat has quit IRC22:40
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-neutron-full 267587,15,2f8608922:48
vgridnevHi, could someone take a look on this fix to unlock saharaclient gate: https://review.openstack.org/#/c/270077/. Thanks!22:48
*** ajmiller has quit IRC22:50
*** ajmiller has joined #openstack-qa22:50
*** vgridnev has quit IRC22:50
openstackgerritMatt Riedemann proposed openstack/tempest: Add a test for cinder multiattach  https://review.openstack.org/26660522:52
*** mriedem has quit IRC22:53
*** pglass has joined #openstack-qa22:53
*** harlowja has quit IRC22:56
*** harlowja has joined #openstack-qa22:56
*** mragupat has quit IRC22:56
openstackrecheckConsole logs not available after 13:25s for experimental-tempest-dsvm-multinode-full-dibtest 267724,1,339236823:01
*** dimtruck is now known as zz_dimtruck23:06
*** jecarey has quit IRC23:06
dtroyerjlvillal: I do not recall such a ting23:12
*** ksantoshk has joined #openstack-qa23:12
jlvillaldtroyer: Thanks. I might propose something. Doing some debugging on other issues and trying it out to help out my debugging.23:13
*** pglass has quit IRC23:15
*** santoshk has quit IRC23:16
openstackrecheckConsole logs not available after 13:24s for gate-tempest-dsvm-full 270865,1,fd92c2423:16
openstackrecheckopenstack/horizon change: https://review.openstack.org/268087 failed gate-horizon-npm-run-test in the gate queue with an unrecognized error23:16
*** ChuckC has joined #openstack-qa23:24
*** nicholasgoracke has quit IRC23:26
*** jpas has joined #openstack-qa23:29
openstackrecheckConsole logs not available after 13:34s for gate-tempest-dsvm-neutron-full 263257,9,a97501423:30
*** apevec has quit IRC23:31
*** sigmavirus24 is now known as sigmavirus24_awa23:34
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: WIP: Add initial optional influxdb support for test_runs  https://review.openstack.org/27108823:39
mtreinishSpamapS: ^^^ the initial ideas of what I was thinking for that23:40
mtreinishI have no idea if that does anything yet still need to setup influx on my machine23:41
*** tochi has joined #openstack-qa23:41
mtreinishbut that'll probably be an exercise for later tonight or tomorrow23:41
openstackgerritDavid Paterson proposed openstack/tempest: Fix logic in dump_accounts  https://review.openstack.org/27109023:43
openstackgerritTim Buckley proposed openstack/openstack-health: Enable debounce in filter fields.  https://review.openstack.org/27101623:43
openstackrecheckConsole logs not available after 13:22s for gate-tempest-dsvm-neutron-full 259470,5,e8ab27423:43
*** rcernin has quit IRC23:47
*** isq_ has quit IRC23:48
SpamapSmtreinish: nice23:52
SpamapSmtreinish: reading23:52
*** edmondsw has quit IRC23:55
*** isq has joined #openstack-qa23:56
openstackrecheckConsole logs not available after 13:26s for gate-tempest-dsvm-neutron-full 270792,1,3bfba4c23:57
*** sabeen2 has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!