| *** woodster_ has quit IRC | 00:10 | |
| *** akshai has joined #openstack-qa | 00:18 | |
| *** akshai has quit IRC | 00:21 | |
| *** armax has joined #openstack-qa | 00:22 | |
| *** mpaolino has joined #openstack-qa | 00:34 | |
| *** mpaolino has quit IRC | 00:36 | |
| *** salv-orlando has joined #openstack-qa | 00:47 | |
| *** salv-orlando has quit IRC | 00:50 | |
| *** cody-somerville has quit IRC | 00:54 | |
| *** marcusvrn has joined #openstack-qa | 00:57 | |
| *** salv-orlando has joined #openstack-qa | 01:05 | |
| *** armax has quit IRC | 01:08 | |
| *** salv-orlando has quit IRC | 01:12 | |
| *** marcusvrn has quit IRC | 01:19 | |
| *** sputnik13 has joined #openstack-qa | 01:41 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-dev/devstack: [WIP][EXPERIMENT] can nova-api be run under apache? https://review.openstack.org/179316 | 01:41 |
|---|---|---|
| *** sputnik13 has quit IRC | 01:43 | |
| *** sputnik13 has joined #openstack-qa | 01:43 | |
| *** sputnik13 has quit IRC | 01:45 | |
| *** sputnik13 has joined #openstack-qa | 01:49 | |
| *** patrickeast has joined #openstack-qa | 01:49 | |
| *** zz_dimtruck is now known as dimtruck | 01:54 | |
| *** patrickeast has quit IRC | 02:00 | |
| *** Kennan has quit IRC | 02:01 | |
| *** dimsum__ has quit IRC | 02:04 | |
| *** Kennan has joined #openstack-qa | 02:06 | |
| *** Kennan has quit IRC | 02:08 | |
| *** Kennan has joined #openstack-qa | 02:11 | |
| *** dimtruck is now known as zz_dimtruck | 02:15 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/devstack: clean up logging around run_process https://review.openstack.org/178649 | 02:16 |
| *** sabeen1 has quit IRC | 02:17 | |
| *** achanda has quit IRC | 02:25 | |
| *** sputnik13 has quit IRC | 02:28 | |
| *** patrickeast has joined #openstack-qa | 02:41 | |
| *** patrickeast has quit IRC | 02:42 | |
| openstackrecheck | Console logs not available after 13:23s for check-tempest-dsvm-neutron-full 179316,10,2e9ef00 | 02:43 |
| *** salv-orlando has joined #openstack-qa | 02:56 | |
| *** sergsh has quit IRC | 02:56 | |
| *** sergsh has joined #openstack-qa | 02:56 | |
| *** salv-orlando has quit IRC | 03:01 | |
| *** dimsum__ has joined #openstack-qa | 03:05 | |
| *** dimsum__ has quit IRC | 03:10 | |
| *** salv-orlando has joined #openstack-qa | 03:14 | |
| *** sputnik13 has joined #openstack-qa | 03:16 | |
| *** yfried|afk is now known as yfried | 03:19 | |
| *** salv-orlando has quit IRC | 03:19 | |
| *** coolsvapl is now known as coolsvap | 03:22 | |
| *** achanda has joined #openstack-qa | 03:26 | |
| openstackgerrit | Merged openstack-dev/devstack: Add /usr/local/bin to exec_dirs in rootwrap.conf https://review.openstack.org/176041 | 03:28 |
| *** achanda has quit IRC | 03:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 03:34 |
| *** itzikb has joined #openstack-qa | 03:38 | |
| *** fedexo has joined #openstack-qa | 04:04 | |
| *** sputnik13 has quit IRC | 04:12 | |
| *** AlecTaylor has joined #openstack-qa | 04:21 | |
| AlecTaylor | hi | 04:21 |
| *** sputnik13 has joined #openstack-qa | 04:23 | |
| *** heyongli has quit IRC | 04:30 | |
| *** heyongli has joined #openstack-qa | 04:31 | |
| *** sks has joined #openstack-qa | 04:31 | |
| *** Poornima has joined #openstack-qa | 04:35 | |
| *** yfried has quit IRC | 04:39 | |
| *** itzikb has quit IRC | 04:44 | |
| openstackgerrit | Ramakrishnan G proposed openstack-dev/devstack: Add support for ilo drivers in lib/ironic https://review.openstack.org/174733 | 04:44 |
| *** bkopilov has joined #openstack-qa | 04:46 | |
| AlecTaylor | PaaS cloud, considering: libcloud+consul+etcd+mesos+deis+drone, thoughts? | 04:57 |
| *** woodster_ has joined #openstack-qa | 04:57 | |
| *** itzikb has joined #openstack-qa | 05:02 | |
| *** achanda has joined #openstack-qa | 05:02 | |
| *** Madan has joined #openstack-qa | 05:10 | |
| *** k4n0 has joined #openstack-qa | 05:14 | |
| *** anshul has joined #openstack-qa | 05:31 | |
| *** jyuso has joined #openstack-qa | 05:35 | |
| *** gamado has joined #openstack-qa | 05:40 | |
| *** BharatK has joined #openstack-qa | 05:43 | |
| *** rdekel has joined #openstack-qa | 05:49 | |
| *** ibiris_away is now known as ibiris | 05:50 | |
| *** salv-orlando has joined #openstack-qa | 05:53 | |
| *** salv-orlando has quit IRC | 05:56 | |
| *** salv-orlando has joined #openstack-qa | 05:59 | |
| *** salv-orlando has quit IRC | 05:59 | |
| *** AlecTaylor has quit IRC | 06:05 | |
| *** salv-orlando has joined #openstack-qa | 06:11 | |
| *** pkoniszewski has joined #openstack-qa | 06:15 | |
| *** salv-orlando has quit IRC | 06:15 | |
| *** pkoniszewski has quit IRC | 06:16 | |
| *** pkoniszewski has joined #openstack-qa | 06:17 | |
| *** fedexo has quit IRC | 06:22 | |
| *** sergsh has quit IRC | 06:25 | |
| *** sergsh_ has quit IRC | 06:25 | |
| *** yfried has joined #openstack-qa | 06:25 | |
| *** gfidente has joined #openstack-qa | 06:34 | |
| *** gfidente has quit IRC | 06:34 | |
| *** gfidente has joined #openstack-qa | 06:34 | |
| *** yfried is now known as yfried|afk | 06:36 | |
| *** yfried|afk is now known as yfried | 06:37 | |
| *** itzikb has quit IRC | 06:39 | |
| *** dimsum__ has joined #openstack-qa | 06:42 | |
| *** ibiris is now known as ibiris_away | 06:43 | |
| openstackgerrit | Ravikumar Venkatesan proposed openstack/tempest: Adds testcase for volume backup-export and import https://review.openstack.org/165371 | 06:45 |
| *** salv-orl_ has joined #openstack-qa | 06:45 | |
| *** ibiris_away is now known as ibiris | 06:47 | |
| *** dimsum__ has quit IRC | 06:47 | |
| *** neeti has joined #openstack-qa | 06:51 | |
| *** SridharG has joined #openstack-qa | 07:01 | |
| *** jose_lausuch has joined #openstack-qa | 07:09 | |
| jose_lausuch | Hi, I have a question about tempest, can anyone give me a hand? | 07:09 |
| *** safchain has joined #openstack-qa | 07:10 | |
| *** yamahata has quit IRC | 07:14 | |
| *** itzikb has joined #openstack-qa | 07:15 | |
| *** gamado has quit IRC | 07:16 | |
| *** yog__ has joined #openstack-qa | 07:23 | |
| *** achanda has quit IRC | 07:23 | |
| *** achanda has joined #openstack-qa | 07:26 | |
| *** apevec has joined #openstack-qa | 07:32 | |
| *** apevec has quit IRC | 07:32 | |
| *** apevec has joined #openstack-qa | 07:32 | |
| *** Kennan has quit IRC | 07:33 | |
| *** Kennan has joined #openstack-qa | 07:34 | |
| *** markus_z has joined #openstack-qa | 07:36 | |
| *** achanda has quit IRC | 07:37 | |
| *** safchain has quit IRC | 07:37 | |
| *** safchain has joined #openstack-qa | 07:39 | |
| *** woodster_ has quit IRC | 07:40 | |
| *** itzikb has quit IRC | 07:58 | |
| *** itzikb has joined #openstack-qa | 07:59 | |
| *** jhenner has joined #openstack-qa | 08:00 | |
| *** ajo has joined #openstack-qa | 08:26 | |
| *** sergsh_ has joined #openstack-qa | 08:27 | |
| *** itzikb_ has joined #openstack-qa | 08:30 | |
| *** sergsh has joined #openstack-qa | 08:31 | |
| *** itzikb has quit IRC | 08:34 | |
| *** yog__ has quit IRC | 08:42 | |
| *** raies has joined #openstack-qa | 08:45 | |
| andreaf | jose_lausuch: Hi | 09:01 |
| openstackgerrit | Sirushti Murugesan proposed openstack-dev/grenade: Add initial support for Heat https://review.openstack.org/86978 | 09:04 |
| *** e0ne has joined #openstack-qa | 09:04 | |
| jose_lausuch | andreaf: hi | 09:05 |
| andreaf | jose_lausuch: you had a question on tempest? | 09:06 |
| jose_lausuch | any way to output a list of all the tempest test (or per subunits) cases in this manner: | 09:06 |
| jose_lausuch | tempest.api.image.v2.test_images.ListImagesTest.test_index_no_params | 09:06 |
| jose_lausuch | tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_status | 09:06 |
| jose_lausuch | ...... | 09:06 |
| jose_lausuch | I tried testr list-tests | 09:07 |
| andreaf | testr list-tests | 09:07 |
| jose_lausuch | but it outputs a lot of things, errors and so on | 09:07 |
| andreaf | jose_lausuch: well something may be wrong in your setup - what I find useful is to run some tests using tox - e.g. tox -e py27 | 09:09 |
| andreaf | jose_lausuch: then activate the created virtual env and from there testr list-tests | 09:09 |
| jose_lausuch | I did a testr init | 09:09 |
| *** yfried is now known as yfried|afk | 09:10 | |
| *** e0ne has quit IRC | 09:10 | |
| jose_lausuch | the output is for example : from tempest_lib.common.utils import data_utils\nImportError: No module named tempest_lib.common.utils\nN\x10\x9b\xc8', stderr=None | 09:10 |
| raies | anyone around on mirantis Fule ? | 09:10 |
| *** itzikb_ has quit IRC | 09:12 | |
| *** salv-orlando has joined #openstack-qa | 09:13 | |
| *** salv-orl_ has quit IRC | 09:13 | |
| *** jordanP has joined #openstack-qa | 09:14 | |
| *** yog__ has joined #openstack-qa | 09:18 | |
| *** sergsh has quit IRC | 09:22 | |
| *** sergsh_ has quit IRC | 09:22 | |
| *** jyuso has quit IRC | 09:27 | |
| *** mpaolino has joined #openstack-qa | 09:29 | |
| andreaf | jose_lausuch: right it looks like you're missing some dependency - tempest_lib in this case | 09:29 |
| andreaf | jose_lausuch: make sure you have everything defined in requirements.txt and test-requirements.txt installed in your system / virtual environment | 09:30 |
| *** itzikb_ has joined #openstack-qa | 09:31 | |
| jose_lausuch | andreaf: I have the line "tempest-lib>=0.4.0" in requirements.txt | 09:32 |
| andreaf | jose_lausuch: right - but is it installed? | 09:33 |
| jose_lausuch | how do I check? | 09:34 |
| jose_lausuch | (sorry, I'm newbie in tempest) | 09:34 |
| *** sergsh has joined #openstack-qa | 09:35 | |
| *** sergsh_ has joined #openstack-qa | 09:35 | |
| *** mpaolino has quit IRC | 09:35 | |
| jose_lausuch | andreaf: it seems so, ./.venv/lib/python2.7/site-packages/tempest_lib | 09:36 |
| andreaf | jose_lausuch: just to make sure, you can pip freeze | grep tempest_lib | 09:37 |
| jose_lausuch | ok, then no output | 09:38 |
| andreaf | jose_lausuch: so did you activate your virtual environment? | 09:39 |
| andreaf | oh sorry my typo | 09:39 |
| andreaf | it's tempest-lib as opposed to tempest_lib | 09:40 |
| *** rdekel has quit IRC | 09:40 | |
| jose_lausuch | activate with testr init? | 09:40 |
| jose_lausuch | pip freeze | grep tempest outputs now tempest-lib==0.5.0 | 09:42 |
| jose_lausuch | but still I get a lot of other errors | 09:42 |
| jose_lausuch | dependencies | 09:42 |
| *** salv-orlando has quit IRC | 09:43 | |
| jose_lausuch | No module named oslo_concurrency | 09:44 |
| *** salv-orl_ has joined #openstack-qa | 09:46 | |
| jose_lausuch | andreaf: ok now it works | 09:47 |
| jose_lausuch | it was just the missing oslo.concurrency | 09:47 |
| jose_lausuch | dont know why it was not installed | 09:47 |
| *** gamado has joined #openstack-qa | 09:47 | |
| jose_lausuch | andreaf: next question, how do I get a subunit test list? for example for glance? | 09:48 |
| jose_lausuch | I got 1731 test cases, I guess that is the complete suite | 09:48 |
| *** luqas has joined #openstack-qa | 09:49 | |
| *** salv-orlando has joined #openstack-qa | 09:50 | |
| *** gamado has quit IRC | 09:52 | |
| *** salv-orl_ has quit IRC | 09:53 | |
| *** dimsum__ has joined #openstack-qa | 09:54 | |
| *** gamado has joined #openstack-qa | 10:04 | |
| *** gamado has quit IRC | 10:11 | |
| *** armax has joined #openstack-qa | 10:15 | |
| *** yfried|afk is now known as yfried | 10:16 | |
| *** rdekel has joined #openstack-qa | 10:17 | |
| *** pc_m has joined #openstack-qa | 10:17 | |
| *** yog__ has quit IRC | 10:21 | |
| openstackgerrit | Sirushti Murugesan proposed openstack-dev/grenade: Add initial support for Heat https://review.openstack.org/86978 | 10:24 |
| andreaf | jose_lausuch: you can filter your tests in may ways, you can use OS_TEST_PATH environment variable to define the root folder for test discovery | 10:26 |
| andreaf | jose_lausuch: so you could use OS_TEST_PATH=./tempest/api/image to get all the glance API tests | 10:27 |
| andreaf | jose_lausuch: that will not include scenario tests though, but those are only a few | 10:27 |
| andreaf | jose_lausuch: or alternatively you can grep the resulting list | 10:28 |
| *** kiran-r has joined #openstack-qa | 10:28 | |
| andreaf | jose_lausuch: if you only want to run glance tests you can also change your list of available services in your configuration | 10:28 |
| *** yog__ has joined #openstack-qa | 10:29 | |
| *** salv-orl_ has joined #openstack-qa | 10:31 | |
| *** e0ne has joined #openstack-qa | 10:31 | |
| *** salv-orlando has quit IRC | 10:34 | |
| *** kiran-r is now known as _kiran_ | 10:35 | |
| *** weshay has joined #openstack-qa | 10:36 | |
| *** itzikb_ has quit IRC | 10:38 | |
| *** itzikb_ has joined #openstack-qa | 10:39 | |
| *** samueldmq has joined #openstack-qa | 10:40 | |
| *** armax has quit IRC | 10:44 | |
| *** salv-orl_ has quit IRC | 10:46 | |
| *** salv-orlando has joined #openstack-qa | 10:47 | |
| *** SridharG has quit IRC | 10:47 | |
| *** luqas has quit IRC | 10:57 | |
| openstackgerrit | Sirushti Murugesan proposed openstack-dev/grenade: Add initial support for Heat https://review.openstack.org/86978 | 11:01 |
| *** Poornima has quit IRC | 11:11 | |
| *** Longgeek has joined #openstack-qa | 11:15 | |
| *** luqas has joined #openstack-qa | 11:19 | |
| *** salv-orlando has quit IRC | 11:23 | |
| *** Madan has quit IRC | 11:24 | |
| *** luqas has quit IRC | 11:29 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-dev/devstack: Add toggle to run Nova API and EC2-API under Apache2 https://review.openstack.org/179316 | 11:31 |
| *** luqas has joined #openstack-qa | 11:31 | |
| kashyap | sdague: Hi, assuming you're keeping track of it, do you recall if there was a conclusion as to what to use for Neutron default in DevStack? LinuxBridge or OVS? | 11:33 |
| sdague | kashyap: I would like to see a LinuxBridge default to get us started, because otherwise I think a ton of folks are going to just revert to n-net manually to get work done when ovs dies on them in a way they don't understand. | 11:34 |
| kashyap | sdague: Yep, agreed. And, that'd (reverting to n-net) be counter productive. | 11:35 |
| kashyap | Counter productive in long term, i.e. | 11:35 |
| *** k4n0 has quit IRC | 11:37 | |
| *** dimsum__ is now known as dims | 11:41 | |
| sdague | right | 11:43 |
| *** luqas has quit IRC | 11:51 | |
| *** dkranz has joined #openstack-qa | 11:52 | |
| *** rfolco has joined #openstack-qa | 11:58 | |
| *** aix has joined #openstack-qa | 12:05 | |
| *** luqas has joined #openstack-qa | 12:07 | |
| samueldmq | ianw, hi, thanks for your review at 'Honor the flag for Identity v3 API only jobs' | 12:14 |
| samueldmq | ianw, https://review.openstack.org/#/c/179663/ | 12:14 |
| samueldmq | ianw, just replied | 12:14 |
| *** luqas has quit IRC | 12:15 | |
| *** aysyd has joined #openstack-qa | 12:19 | |
| *** salv-orlando has joined #openstack-qa | 12:24 | |
| *** luqas has joined #openstack-qa | 12:29 | |
| *** luqas has quit IRC | 12:34 | |
| *** woodster_ has joined #openstack-qa | 12:36 | |
| *** salv-orlando has quit IRC | 12:37 | |
| *** SridharG has joined #openstack-qa | 12:47 | |
| *** cbader has joined #openstack-qa | 12:49 | |
| *** pc_m has quit IRC | 12:57 | |
| *** jaypipes has joined #openstack-qa | 12:58 | |
| *** ChanServ sets mode: +o jaypipes | 12:58 | |
| *** bknudson has quit IRC | 13:01 | |
| *** yfried is now known as yfried|afk | 13:02 | |
| openstackgerrit | Sirushti Murugesan proposed openstack-dev/grenade: Add initial support for Heat https://review.openstack.org/86978 | 13:08 |
| *** miqui has joined #openstack-qa | 13:09 | |
| *** dims has quit IRC | 13:14 | |
| *** dustins has joined #openstack-qa | 13:14 | |
| *** dims has joined #openstack-qa | 13:15 | |
| *** erlon_awaY has joined #openstack-qa | 13:15 | |
| openstackgerrit | yuxingw proposed openstack/tempest-lib: Password is not strong enough failures https://review.openstack.org/178546 | 13:17 |
| *** yfried|afk is now known as yfried | 13:18 | |
| *** _kiran_ is now known as kiran-r | 13:18 | |
| *** jhenner has quit IRC | 13:20 | |
| *** SridharG has left #openstack-qa | 13:21 | |
| *** mestery_ is now known as mestery | 13:25 | |
| *** erlon_awaY is now known as erlon | 13:27 | |
| *** bknudson has joined #openstack-qa | 13:33 | |
| *** Yogi has joined #openstack-qa | 13:34 | |
| *** kiran-r has quit IRC | 13:37 | |
| *** coolsvap is now known as coolsvap|afk | 13:37 | |
| *** neeti has quit IRC | 13:38 | |
| *** luqas has joined #openstack-qa | 13:40 | |
| *** jeblair_ is now known as jeblair | 13:44 | |
| *** superdan is now known as dansmith | 13:44 | |
| openstackgerrit | yuxingw proposed openstack/tempest: Add a method to generate password with https://review.openstack.org/178561 | 13:45 |
| *** mriedem has joined #openstack-qa | 13:46 | |
| *** dboik has joined #openstack-qa | 13:46 | |
| *** luqas has quit IRC | 13:48 | |
| *** luqas has joined #openstack-qa | 13:51 | |
| *** itzikb_ has quit IRC | 13:52 | |
| *** luqas has quit IRC | 13:55 | |
| openstackgerrit | Sirushti Murugesan proposed openstack-dev/grenade: Add initial support for Heat https://review.openstack.org/86978 | 13:57 |
| *** yfried is now known as yfried|afk | 14:02 | |
| *** rbak has joined #openstack-qa | 14:08 | |
| openstackgerrit | Sridhar Gaddam proposed openstack-dev/devstack: Set local_ip only when TENANT_TUNNELS are enabled https://review.openstack.org/179787 | 14:08 |
| *** Yogi has quit IRC | 14:12 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for nova unit test bug 1451389 https://review.openstack.org/179788 | 14:13 |
| openstack | bug 1451389 in OpenStack Compute (nova) "Nova gate broke due to failed unit test" [Critical,In progress] https://launchpad.net/bugs/1451389 - Assigned to Sean Dague (sdague) | 14:13 |
| mriedem | sdague: ^ | 14:13 |
| sdague | thanks | 14:13 |
| *** e0ne has quit IRC | 14:14 | |
| mriedem | seems odd that requests isn't capped on stable/kilo | 14:15 |
| mriedem | https://github.com/openstack/requirements/blob/stable/kilo/global-requirements.txt#L139 | 14:15 |
| mriedem | this 2.7.0 thing will hit in stable/kilo for nova too | 14:15 |
| *** pkoniszewski has quit IRC | 14:15 | |
| *** salv-orlando has joined #openstack-qa | 14:16 | |
| *** rossella_s has quit IRC | 14:16 | |
| *** rossella_s has joined #openstack-qa | 14:16 | |
| *** pc_m has joined #openstack-qa | 14:17 | |
| *** pglass has joined #openstack-qa | 14:17 | |
| mriedem | flashgordon: shouldn't we cap all libraries on stable/kilo, not just the openstack ones? like requests, alembic, anyjson, etc etc? | 14:17 |
| sdague | mriedem: this is up for discussion | 14:18 |
| sdague | mriedem: honestly, mocking and testing the signature of a method outside of your codebase is just terrible | 14:19 |
| mriedem | alright, well i'll backport dansmith's change anyway since we're uncapped and the test is there also | 14:19 |
| mriedem | *sdague's change :) | 14:19 |
| *** Yogi has joined #openstack-qa | 14:21 | |
| *** yfried|afk is now known as yfried | 14:22 | |
| *** BharatK has quit IRC | 14:23 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for nova unit test bug 1451389 https://review.openstack.org/179788 | 14:27 |
| openstack | bug 1451389 in OpenStack Compute (nova) "Nova gate broke due to failed unit test" [Critical,In progress] https://launchpad.net/bugs/1451389 - Assigned to Sean Dague (sdague) | 14:27 |
| *** dboik_ has joined #openstack-qa | 14:30 | |
| *** salv-orlando has quit IRC | 14:32 | |
| *** armax has joined #openstack-qa | 14:33 | |
| *** dboik has quit IRC | 14:34 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Temporarily add trove plugin configuration https://review.openstack.org/167412 | 14:34 |
| *** zz_jgrimm is now known as jgrimm | 14:35 | |
| *** ajmiller has joined #openstack-qa | 14:35 | |
| *** jgrimm is now known as zz_jgrimm | 14:38 | |
| *** mtanino has joined #openstack-qa | 14:40 | |
| *** edleafe has left #openstack-qa | 14:51 | |
| openstackgerrit | yuxingw proposed openstack/tempest-lib: Password is not strong enough failures https://review.openstack.org/178546 | 14:54 |
| *** openstackrecheck has quit IRC | 15:04 | |
| *** openstackrecheck has joined #openstack-qa | 15:04 | |
| *** aix has quit IRC | 15:13 | |
| *** aix has joined #openstack-qa | 15:15 | |
| *** anshul has quit IRC | 15:23 | |
| openstackgerrit | yuxingw proposed openstack/tempest: Add a method to generate password with https://review.openstack.org/178561 | 15:24 |
| *** e0ne has joined #openstack-qa | 15:28 | |
| *** sks has quit IRC | 15:28 | |
| *** sabeen has joined #openstack-qa | 15:29 | |
| *** mtanino has quit IRC | 15:33 | |
| *** yamahata has joined #openstack-qa | 15:35 | |
| *** rdekel has quit IRC | 15:37 | |
| *** sputnik13 has quit IRC | 15:38 | |
| *** Longgeek_ has joined #openstack-qa | 15:40 | |
| *** salv-orlando has joined #openstack-qa | 15:41 | |
| *** Longgeek has quit IRC | 15:42 | |
| *** mwagner_lap has quit IRC | 15:47 | |
| *** mlavalle has joined #openstack-qa | 15:51 | |
| *** mtanino has joined #openstack-qa | 15:54 | |
| *** armax has quit IRC | 15:56 | |
| *** sabeen2 has joined #openstack-qa | 15:58 | |
| *** whayutin_ has joined #openstack-qa | 15:59 | |
| *** catohornet has joined #openstack-qa | 15:59 | |
| openstackgerrit | Joe D'Andrea proposed openstack-dev/devstack: cinder setup now refers to CINDER_VOLUME_CLEAR and volume_clear https://review.openstack.org/178819 | 16:01 |
| *** sabeen has quit IRC | 16:01 | |
| *** whayutin_ has quit IRC | 16:02 | |
| *** whayutin_ has joined #openstack-qa | 16:02 | |
| *** weshay has quit IRC | 16:02 | |
| *** artom has joined #openstack-qa | 16:03 | |
| *** catohornet has quit IRC | 16:07 | |
| *** safchain has quit IRC | 16:09 | |
| *** yamahata has quit IRC | 16:09 | |
| *** sergsh_ has quit IRC | 16:10 | |
| *** sergsh has quit IRC | 16:10 | |
| *** jose_lausuch has quit IRC | 16:10 | |
| *** markus_z has quit IRC | 16:11 | |
| *** salv-orlando has quit IRC | 16:12 | |
| *** Nic has joined #openstack-qa | 16:12 | |
| *** yfried has quit IRC | 16:14 | |
| *** yog__ has quit IRC | 16:16 | |
| *** zz_jgrimm is now known as jgrimm | 16:18 | |
| mriedem | mestery: remember the network namespace bug where nbd was crashing the kernel? how was that resolved? | 16:33 |
| *** jordanP has quit IRC | 16:33 | |
| *** yamahata has joined #openstack-qa | 16:33 | |
| mtreinish | mriedem: we turned off nbd | 16:34 |
| mriedem | mtreinish: in devstack? | 16:34 |
| mtreinish | yep | 16:34 |
| mtreinish | iirc | 16:34 |
| mriedem | ok, so like the lvm race/lock fix, which was also in devstack | 16:35 |
| mtreinish | mriedem: well it was a kernel bug which we were triggering by using network namespaces and nbd at the same time. | 16:36 |
| mriedem | so that's interesting b/c in these charts i was going to say for lessons learned, push fixes into the code rather than devstack b/c production deployments shouldn't be using devstack... :) | 16:36 |
| mriedem | production deployments are most likely using something else like puppet, chef, ansible, right? | 16:37 |
| mtreinish | it could have happened to anyone using "precise" I think | 16:37 |
| mriedem | so we fix a thing in devstack and it might take awhile to duplicate in those tools | 16:37 |
| mriedem | mtreinish: was it only turned off for ubuntu precise or all distros and all versions? | 16:37 |
| mtreinish | mriedem: I don't remember the full details, salv-orlando might he was the one who found it | 16:38 |
| mtreinish | mriedem: but I think we either changed the default in devstack or nova for every env | 16:38 |
| sdague | mriedem: we installed guestfs actually | 16:38 |
| sdague | nbd was always a terrible idea | 16:38 |
| mestery | ++ | 16:39 |
| mtreinish | mriedem: https://bugs.launchpad.net/neutron/+bug/1273386 | 16:39 |
| openstack | Launchpad bug 1273386 in devstack "Neutron namespace metadata proxy triggers kernel crash on Ubuntu 12.04/3.2 kernel" [Undecided,In progress] - Assigned to Alan Pevec (apevec) | 16:39 |
| sdague | no, the fact that the kernel pooped the bed with net ns + nbd ... that's a real issue, but we did punt around it | 16:40 |
| *** flashgordon is now known as jogo | 16:40 | |
| jogo | mriedem: I think dhellmann had a ML post about this the other week -- saying hold off we will sort this out at the summit or something | 16:42 |
| mriedem | i found the guestfs change in nova https://review.openstack.org/#/c/120467/ | 16:42 |
| *** suneelb has joined #openstack-qa | 16:44 | |
| *** cody-somerville has joined #openstack-qa | 16:45 | |
| mriedem | don't see anything mentioned in the juno release notes about nbd/guestfs/namespace issues https://wiki.openstack.org/wiki/ReleaseNotes/Juno | 16:46 |
| *** hitalia has joined #openstack-qa | 16:47 | |
| sdague | mriedem: yeh, I don't think it made release notes | 16:51 |
| *** Nic has quit IRC | 16:51 | |
| *** Nic has joined #openstack-qa | 16:52 | |
| *** Pixellina has joined #openstack-qa | 16:55 | |
| mriedem | jgriffith: for this https://review.openstack.org/#/c/165281/ - were you going to put something in the kilo releaste notes about that per the DocImpact? | 16:56 |
| mriedem | i didn't see anything for lvm here https://wiki.openstack.org/wiki/ReleaseNotes/Kilo | 16:56 |
| jgriffith | mriedem: indeed | 16:58 |
| jgriffith | mriedem: I'll do it now, thanks | 16:58 |
| mriedem | thanks | 16:58 |
| mriedem | i'm working on some charts that says for these big hairy gate issues, what are the lessons learned and i wanted to say 'we push changes into the code rather than devstack since production != devstack' but i don't see that sometimes - maybe not possible, but at least release notes should get the message out | 16:59 |
| jgriffith | mriedem: great idea, this one is interesting since it's "both" :) | 17:00 |
| mriedem | jgriffith: i.e. https://review.openstack.org/#/c/149046/ ? | 17:00 |
| jgriffith | mriedem: yep | 17:00 |
| mriedem | i mean https://review.openstack.org/#/c/148747/ | 17:01 |
| *** Yogi2 has joined #openstack-qa | 17:01 | |
| jgriffith | mriedem: same thing, it's referenced in the commit I think... | 17:01 |
| mriedem | yeah | 17:01 |
| jgriffith | mriedem: yeah, it's there | 17:01 |
| jgriffith | two parter | 17:01 |
| jgriffith | :) | 17:01 |
| mriedem | just missing the docs telling people, hey, if you're running cinder with lvm, you should use lvmetad and/or this filter thing | 17:02 |
| *** Yogi has quit IRC | 17:03 | |
| *** dboik_ has quit IRC | 17:04 | |
| *** dboik has joined #openstack-qa | 17:05 | |
| *** Longgeek_ has quit IRC | 17:10 | |
| *** achanda has joined #openstack-qa | 17:11 | |
| *** hitalia has quit IRC | 17:12 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/170780 failed gate-tempest-dsvm-postgres-full in the gate queue with an unrecognized error | 17:13 |
| *** sputnik13 has joined #openstack-qa | 17:16 | |
| *** patrickeast has joined #openstack-qa | 17:18 | |
| *** hitalia has joined #openstack-qa | 17:19 | |
| *** Sukhdev has joined #openstack-qa | 17:22 | |
| *** luqas has joined #openstack-qa | 17:22 | |
| *** armax has joined #openstack-qa | 17:23 | |
| *** Yogi has joined #openstack-qa | 17:24 | |
| *** Yogi2 has quit IRC | 17:25 | |
| *** dboik_ has joined #openstack-qa | 17:29 | |
| *** salv-orlando has joined #openstack-qa | 17:30 | |
| *** armax has quit IRC | 17:31 | |
| *** luqas has quit IRC | 17:33 | |
| *** dboik has quit IRC | 17:34 | |
| *** Pixellina has quit IRC | 17:37 | |
| *** bkopilov has quit IRC | 17:38 | |
| *** salv-orlando has quit IRC | 17:39 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for libvirt socket hangup bug 1451506 https://review.openstack.org/179851 | 17:42 |
| openstack | bug 1451506 in OpenStack Compute (nova) "spawn failed with "libvirtError: internal error: received hangup / error event on socket" in the gate" [Undecided,New] https://launchpad.net/bugs/1451506 | 17:42 |
| *** jogo has quit IRC | 17:42 | |
| *** jogo has joined #openstack-qa | 17:43 | |
| mriedem | mtreinish: ^ | 17:44 |
| mtreinish | mriedem: heh, take many screen shots | 17:44 |
| mriedem | i could just copy/paste the irc log, but that would be ugly to read in chart form | 17:45 |
| *** mwagner_lap has joined #openstack-qa | 17:45 | |
| mtreinish | heh, yeah I think it wouldn't be the easiest slide to read | 17:46 |
| *** luqas has joined #openstack-qa | 17:49 | |
| *** luqas has quit IRC | 17:51 | |
| *** aix has quit IRC | 17:58 | |
| *** sabeen2 has quit IRC | 17:58 | |
| *** luqas has joined #openstack-qa | 17:59 | |
| *** armax has joined #openstack-qa | 18:00 | |
| *** luqas has quit IRC | 18:01 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Respect configured cred provider in scenario utils https://review.openstack.org/179852 | 18:01 |
| mtreinish | andreaf: ^^^ should be a simple review | 18:02 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Respect configured cred provider in scenario utils https://review.openstack.org/179852 | 18:04 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for libvirt socket hangup bug 1451506 https://review.openstack.org/179851 | 18:04 |
| openstack | bug 1451506 in OpenStack Compute (nova) "spawn failed with "libvirtError: internal error: received hangup / error event on socket" in the gate" [Undecided,Confirmed] https://launchpad.net/bugs/1451506 | 18:04 |
| *** EmilienM is now known as EmilienM|mtg | 18:05 | |
| *** Sukhdev has quit IRC | 18:09 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/178059 failed because of: gate-nova-python27: https://bugs.launchpad.net/bugs/1451389 | 18:15 |
| openstack | Launchpad bug 1451389 in OpenStack Compute (nova) "Nova gate broke due to failed unit test" [Critical,Fix committed] - Assigned to Sean Dague (sdague) | 18:15 |
| openstackgerrit | Joe D'Andrea proposed openstack-dev/devstack: cinder setup now refers to CINDER_VOLUME_CLEAR and volume_clear https://review.openstack.org/178819 | 18:15 |
| sdague | mriedem: oh, do you have that backport? | 18:16 |
| mriedem | sdague: yes | 18:16 |
| sdague | url? | 18:16 |
| mriedem | https://review.openstack.org/#/c/179805/ | 18:16 |
| mriedem | pep8 of course | 18:16 |
| mriedem | gdi | 18:16 |
| mriedem | sec | 18:16 |
| sdague | lol | 18:17 |
| sdague | man, I need an mriedem gdi counter | 18:17 |
| sdague | we need that in graphite actually | 18:17 |
| mriedem | swear jar? | 18:17 |
| sdague | no, just more as a state of world | 18:17 |
| sdague | it's a 5 gdi day... look out | 18:17 |
| *** openstackrecheck has quit IRC | 18:18 | |
| *** openstackrecheck has joined #openstack-qa | 18:18 | |
| *** sabeen has joined #openstack-qa | 18:18 | |
| mtreinish | sdague: heh, I think there's probably a bit of a correlation to that and days when he has to deal with his internal ci | 18:19 |
| mriedem | compared to last week, i've been surprised as to how much i haven't been getting bugged internally | 18:19 |
| mriedem | although as noted in nova, my manager messed up my irc nick so i'm ignoring that one | 18:19 |
| mriedem | https://review.openstack.org/#/c/179805/ refreshed | 18:20 |
| sdague | mriedem: that sounds like some damn fine management there | 18:20 |
| *** EmilienM|mtg is now known as EmilienM | 18:21 | |
| *** suneelb has quit IRC | 18:26 | |
| *** itzikb_ has joined #openstack-qa | 18:26 | |
| *** apevec has quit IRC | 18:28 | |
| *** Sukhdev has joined #openstack-qa | 18:29 | |
| *** dboik_ has quit IRC | 18:31 | |
| *** samueldmq_ has joined #openstack-qa | 18:37 | |
| *** boris-42 has joined #openstack-qa | 18:37 | |
| openstackrecheck | openstack/requirements change: https://review.openstack.org/179706 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error | 18:42 |
| mtreinish | sdague: so it looks like grenade with tempest is enforcing the base g-r on the tempest being installed (at least for kilo->master) | 18:49 |
| mtreinish | do you have a suggestion for disabling the devstack g-r sync for tempest only? | 18:49 |
| *** subscope_ has joined #openstack-qa | 18:49 | |
| mtreinish | jlk: did you see my comments on: gate-tempest-dsvm-neutron-large-opsSUCCESS in 25m 23s | 18:51 |
| jlk | I did, haven't gotten to them yet | 18:51 |
| mtreinish | d'oh wrong paste buffer :) | 18:51 |
| mriedem | sdague: mtreinish: ever put any thought into logging instance faults when we hit an unexpected error state in tempest? | 18:51 |
| mtreinish | heh, I'm glad you got what I meant though | 18:51 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tempest: Updated from global requirements https://review.openstack.org/176851 | 18:52 |
| mtreinish | mriedem: logging what exactly? | 18:52 |
| mriedem | mtreinish: e.g. https://review.openstack.org/#/c/164738/2/nova/compute/manager.py | 18:52 |
| mriedem | like we're waiting for a status change in tempest and the vm_state goes to ERROR | 18:52 |
| mriedem | sometimes the console log just says 'while waiting for x we got ERROR' | 18:52 |
| mriedem | we could get the instance faults for the instance before teardown and log those in the tempest log | 18:52 |
| sdague | mtreinish: it's always forced base | 18:53 |
| mtreinish | sdague: sure but there is no base branch for tempest and we're venv isolated via tox | 18:54 |
| sdague | so, probably you need to change the code that does g-r syncs in those cases, and also remove tempest from projects.txt, because we're saying that it matching the requirements of the system isn't required | 18:55 |
| mtreinish | sdague: well we still want it to be installable on master without a venv which is why it tracks projects.txt | 18:56 |
| mtreinish | but for the stable branches there is a conflict because of different reqs | 18:57 |
| *** sslypush1nko is now known as sslypushenko | 18:57 | |
| mtreinish | I'd be fine with removing it from projects.txt on stable | 18:58 |
| *** rockyg has joined #openstack-qa | 18:59 | |
| *** itzikb_ has quit IRC | 19:00 | |
| *** whayutin_ is now known as weshay | 19:03 | |
| mriedem | mtreinish: nvm, i guess someone already added it: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/waiters.py#n72 | 19:03 |
| mtreinish | mriedem: oh, that's what you meant. I wasn't sure what you were asking to add | 19:04 |
| *** hitalia has quit IRC | 19:05 | |
| *** yfried has joined #openstack-qa | 19:07 | |
| *** dboik has joined #openstack-qa | 19:09 | |
| *** artom has quit IRC | 19:13 | |
| *** samueldmq_ has quit IRC | 19:14 | |
| *** artom has joined #openstack-qa | 19:16 | |
| *** artom has quit IRC | 19:17 | |
| *** artom has joined #openstack-qa | 19:17 | |
| openstackgerrit | Merged openstack-dev/devstack: clean up logging around run_process https://review.openstack.org/178649 | 19:18 |
| *** yfried is now known as yfried|afk | 19:20 | |
| *** salv-orlando has joined #openstack-qa | 19:22 | |
| andreaf | mtreinish: well, yes, I guess it's a good fix to have... I didn't use the credential provider there before because it's not a test, so I though at the time it would be good to use the configured credentials - but it the configured credentials are in accounts.yaml it won't work | 19:26 |
| *** e0ne has quit IRC | 19:26 | |
| mtreinish | andreaf: yeah exactly a bug got filed on that this morning | 19:26 |
| andreaf | mtreinish: and it's a singleton, so even if we create new credentials on the fly it will only be once | 19:26 |
| mtreinish | oh, heh I didn't even notice that | 19:27 |
| mtreinish | well, it'll be once per worker (assuming that can even be split to multiple workers) | 19:27 |
| andreaf | mtreinish: my only question is whether at test list time without a config file (the py27 case) it will fail with a InvalidConfiguration as it does now | 19:27 |
| andreaf | mtreinish: so I guess if tests pass it's ok | 19:28 |
| mtreinish | andreaf: hmm https://jenkins05.openstack.org/job/gate-tempest-python27/3484/console | 19:28 |
| mtreinish | the tests fail with InvalidCredentials | 19:29 |
| mtreinish | oh because it's trying to pull root creds for tenant isolation | 19:29 |
| *** e0ne has joined #openstack-qa | 19:30 | |
| andreaf | mtreinish: I'm happy to see that auth removal was merged :) | 19:30 |
| andreaf | mtreinish: btw if you have some review time: https://review.openstack.org/#/c/168367/ - the next patch on the cleanup post resource-cleanup bp | 19:31 |
| mtreinish | andreaf: yeah, although it only passes grenade because 0.4.0 included most of the needed bits :) | 19:31 |
| mtreinish | the ssh removal patch is stuck on that because grenade with kilo is using 0.4.0 | 19:32 |
| mtreinish | andreaf: sure I'll take a look later today | 19:32 |
| mtreinish | andreaf: did the patch doing the class var for cred allocation land? | 19:32 |
| andreaf | mtreinish: yes it did :) | 19:32 |
| andreaf | mtreinish: next one is extending the get_client_manager to roles | 19:33 |
| *** salv-orlando has quit IRC | 19:33 | |
| andreaf | mtreinish: and next one is to add roles to the class level credentials | 19:33 |
| mtreinish | ok, cool | 19:33 |
| andreaf | mtreinish: but this last one needs some rebasing | 19:33 |
| *** e0ne has quit IRC | 19:35 | |
| mtreinish | andreaf: heh, I'll trade you for some subunit2sql reviews, I've got a couple of open patches there | 19:35 |
| andreaf | mtreinish: sure | 19:35 |
| mtreinish | cool, thanks | 19:36 |
| *** e0ne has joined #openstack-qa | 19:37 | |
| *** jgrimm is now known as zz_jgrimm | 19:37 | |
| *** e0ne has quit IRC | 19:39 | |
| *** e0ne has joined #openstack-qa | 19:39 | |
| *** luqas has joined #openstack-qa | 19:40 | |
| *** yfried|afk is now known as yfried | 19:40 | |
| *** Yogi has quit IRC | 19:41 | |
| *** e0ne has quit IRC | 19:41 | |
| *** e0ne has joined #openstack-qa | 19:42 | |
| *** e0ne has quit IRC | 19:42 | |
| *** e0ne has joined #openstack-qa | 19:43 | |
| *** e0ne has quit IRC | 19:43 | |
| *** e0ne has joined #openstack-qa | 19:44 | |
| *** luqas has quit IRC | 19:45 | |
| *** e0ne has quit IRC | 19:45 | |
| *** yfried is now known as yfried|afk | 19:50 | |
| *** e0ne has joined #openstack-qa | 19:57 | |
| *** dboik has quit IRC | 19:59 | |
| *** mriedem has quit IRC | 19:59 | |
| *** dboik has joined #openstack-qa | 19:59 | |
| *** dboik_ has joined #openstack-qa | 20:02 | |
| *** hitalia has joined #openstack-qa | 20:03 | |
| *** dboik has quit IRC | 20:05 | |
| *** hitalia has quit IRC | 20:08 | |
| *** subscope_ has quit IRC | 20:08 | |
| *** gfidente has quit IRC | 20:13 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/170780 failed because of: gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/1451506 | 20:16 |
| openstack | Launchpad bug 1451506 in OpenStack Compute (nova) "spawn failed with "libvirtError: internal error: received hangup / error event on socket" in the gate" [Undecided,Confirmed] | 20:16 |
| *** rockyg has quit IRC | 20:17 | |
| *** rcarrill` is now known as rcarrillocruz | 20:17 | |
| jogo | jgriffith: https://bugs.launchpad.net/cinder/+bug/1429151 is back | 20:23 |
| openstack | Launchpad bug 1429151 in Cinder "SimpleReadOnlyCinderClientTest.test_cinder_encryption_type_list fails randomly" [High,Confirmed] | 20:23 |
| jogo | mtreinish: how far we are from removing cinder CLI tests from tempest? | 20:23 |
| *** hockeynut has quit IRC | 20:24 | |
| *** hockeynut has joined #openstack-qa | 20:24 | |
| jogo | mtreinish: because just hit https://bugs.launchpad.net/cinder/+bug/1429151 which is no fun | 20:24 |
| openstack | Launchpad bug 1429151 in Cinder "SimpleReadOnlyCinderClientTest.test_cinder_encryption_type_list fails randomly" [High,Confirmed] | 20:24 |
| *** e0ne has quit IRC | 20:26 | |
| *** dustins has quit IRC | 20:29 | |
| *** dustins has joined #openstack-qa | 20:29 | |
| *** Sukhdev has quit IRC | 20:29 | |
| *** e0ne has joined #openstack-qa | 20:30 | |
| *** boris-42 has quit IRC | 20:30 | |
| *** cyeoh has quit IRC | 20:31 | |
| *** Trozz has quit IRC | 20:31 | |
| *** dboik_ has quit IRC | 20:31 | |
| *** Trozz has joined #openstack-qa | 20:31 | |
| *** hockeynut has quit IRC | 20:33 | |
| *** fungi has quit IRC | 20:33 | |
| *** jvrbanac has quit IRC | 20:34 | |
| *** hockeynut has joined #openstack-qa | 20:34 | |
| *** tchaypo has quit IRC | 20:34 | |
| *** cyeoh has joined #openstack-qa | 20:34 | |
| *** dboik has joined #openstack-qa | 20:34 | |
| *** jvrbanac has joined #openstack-qa | 20:35 | |
| *** fungi has joined #openstack-qa | 20:35 | |
| *** boris-42 has joined #openstack-qa | 20:36 | |
| mtreinish | jogo: thurs or friday: http://lists.openstack.org/pipermail/openstack-dev/2015-April/063018.html | 20:36 |
| mtreinish | or whatever you define as the end of this week I guess | 20:36 |
| *** tchaypo has joined #openstack-qa | 20:37 | |
| jogo | mtreinish: excellent | 20:41 |
| *** zz_jgrimm has quit IRC | 20:43 | |
| *** maurosr has quit IRC | 20:44 | |
| *** e0ne has quit IRC | 20:45 | |
| *** Yogi has joined #openstack-qa | 20:45 | |
| *** maurosr has joined #openstack-qa | 20:46 | |
| *** zz_jgrimm has joined #openstack-qa | 20:46 | |
| *** samueldmq_ has joined #openstack-qa | 20:48 | |
| *** hitalia has joined #openstack-qa | 20:49 | |
| mtreinish | jogo: although jgriffith said he was going to work extra hard on getting the cinder stuff migrated before I land the removal patch so it might be sooner | 20:50 |
| jogo | mtreinish: that is preferable but either way that issue goes away | 20:51 |
| jogo | clarkb: ping re: multinode | 20:51 |
| jogo | clarkb: in past 24 hours there have been 7 failures of which about 4 look to be valid | 20:51 |
| clarkb | ok? | 20:52 |
| jogo | versus 61 successful | 20:52 |
| clarkb | jogo: just lookingat the nova net job? | 20:52 |
| jogo | clarkb: yes | 20:52 |
| clarkb | cool, thought those numbers were low if including dvr | 20:52 |
| jogo | err 3 are valid | 20:54 |
| jogo | clarkb: I am not tracking DVR at all focusing on getting nova-net only at the moment | 20:54 |
| clarkb | ok | 20:54 |
| *** miqui has quit IRC | 20:55 | |
| *** salv-orlando has joined #openstack-qa | 20:55 | |
| jogo | so the question is at what point to can we make this voting | 20:55 |
| jogo | I personally think its ready now | 20:55 |
| jogo | dansmith: ^ | 20:56 |
| dansmith | talking about multinode? | 20:56 |
| clarkb | I think its mostly up to you all, we have hpcloud providing test nodes again so not super worried about node availability right this moment | 20:56 |
| clarkb | might be good to ask infra as a whole though | 20:56 |
| jogo | dansmith: yes | 20:57 |
| dansmith | I thought there was still a 5% failure rate? | 20:57 |
| jogo | clarkb: kk, well if dansmith says yes I will go to infra and ask | 20:57 |
| jogo | dansmith: its 3 or so for bad failures in 61 successful jobs and 4 valid failures | 20:58 |
| dansmith | when i was looking last week, it seemed like the ones that weren't valid were DOA subnodes | 20:59 |
| jogo | dansmith: its close to 5% still yeah, if we turn off the shelve test for now we can get that rate higher | 20:59 |
| jogo | dansmith: there are a few that simply ran on bad patches | 20:59 |
| dansmith | I want it voting ASAP, but also don't want to piss people off | 20:59 |
| clarkb | it may help on that front to collect the list of nodes that exploded and pass that data off to rax | 21:00 |
| clarkb | since I am fairly convinced at this point it would have to be a cloud error especially since we will see ssh work a few times then stop working | 21:00 |
| jogo | clarkb: I think that only happened once | 21:00 |
| jogo | so there a few possible fixes for the test_shelve_instance patch but ??? | 21:00 |
| *** Sukhdev has joined #openstack-qa | 21:00 | |
| jogo | dansmith: this one is a maybe: https://review.openstack.org/#/c/85249/18 | 21:01 |
| dansmith | TBH, I'm in the middle of trying to unfuck something so I'm not much good for discussion | 21:01 |
| *** luqas has joined #openstack-qa | 21:02 | |
| jogo | dansmith: kk, its just we have been sitting on this job for what seems to be weeks and haven't done anything | 21:02 |
| jogo | still seeing the same bugs, and at an impass | 21:02 |
| jogo | sdague: ^ | 21:03 |
| jogo | I am going to dust off my skip test_shelve_instance in multihost just in case | 21:03 |
| jogo | and will make sure the idea of voting multinode is ok with infra | 21:03 |
| *** artom has quit IRC | 21:05 | |
| *** artom has joined #openstack-qa | 21:05 | |
| *** apevec has joined #openstack-qa | 21:06 | |
| *** luqas has quit IRC | 21:07 | |
| *** artom has quit IRC | 21:07 | |
| *** artom has joined #openstack-qa | 21:08 | |
| clarkb | jogo: I will -1 if you skip test_shelve :) | 21:09 |
| clarkb | jogo: seriously though, I spent way too much time debugging that to have nova's response be meh | 21:09 |
| clarkb | either we have the test because its valuable or we remove the test | 21:09 |
| *** Yogi has quit IRC | 21:10 | |
| jogo | clarkb: so we wouldn't be skipping in all cases. And frankly I have no idea what the issue is and don't want to dig into the networking muck | 21:12 |
| clarkb | jogo: I already wrote a bug with what I thought was very detailed analysis | 21:12 |
| clarkb | jogo: nova net does not restore leases to dnsmasq on unshelve | 21:12 |
| clarkb | so node boots and never gets a lease, then test fails | 21:12 |
| jogo | clarkb: but the test doesn't always fail when migrating between nodes right? | 21:13 |
| clarkb | no I think it does | 21:13 |
| clarkb | it does not fail when it unshlves on the same node | 21:13 |
| jogo | clarkb: hmm I thought we found a counter example | 21:14 |
| * jogo double checks | 21:14 | |
| clarkb | oh right, maybe we did | 21:14 |
| clarkb | my bug should have details though /me goes to find it | 21:15 |
| *** dustins has quit IRC | 21:15 | |
| clarkb | https://bugs.launchpad.net/nova/+bug/1445569 I did not mention if it was consistent or not, just pointed out the problem in the one job | 21:16 |
| openstack | Launchpad bug 1445569 in OpenStack Compute (nova) "No dhcp lease after shelve unshelve" [Undecided,Confirmed] | 21:16 |
| clarkb | however https://bugs.launchpad.net/nova/+bug/1439355 looks like a dup | 21:16 |
| openstack | Launchpad bug 1439355 in OpenStack Compute (nova) "tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance periodically failing on multinode job" [High,Confirmed] | 21:16 |
| clarkb | so we should probably merge them with the infos preserved | 21:17 |
| jogo | clarkb: feel free to merge them, I am digging through logs to see if I can find a successful shelve unshelve accross nodes | 21:17 |
| *** mmedvede has quit IRC | 21:18 | |
| samueldmq | hi guys, I am creating an experimental job for devstack to work towards having all the services working with keystone v3 | 21:18 |
| samueldmq | I would appreciate your review | 21:18 |
| samueldmq | https://review.openstack.org/#/c/179663/ | 21:18 |
| samueldmq | mtreinish, ^ | 21:18 |
| jogo | clarkb: yup first example I tried http://logs.openstack.org/97/179697/4/check/check-tempest-dsvm-multinode-full/195d859/logs/screen-n-cpu.txt.gz | 21:20 |
| jogo | a0d4b3ff-5925-489c-ab47-71c9929db34c | 21:20 |
| jogo | that instane is logged in both n-cpu logs | 21:20 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-dev/devstack: [WIP] Just checking contents of qemu.conf https://review.openstack.org/179943 | 21:22 |
| jgriffith | mtreinish: patch has been up for a week or so | 21:22 |
| jgriffith | mtreinish: https://review.openstack.org/#/c/179280/ | 21:22 |
| jgriffith | mtreinish: actually just feels like a week :) | 21:22 |
| mtreinish | thingee: ^^^ +A away | 21:26 |
| jogo | clarkb: so where do we stand with this bug? | 21:28 |
| clarkb | wherever nova gets debygging it? | 21:28 |
| clarkb | I think I nailed down the behavior | 21:29 |
| jogo | clarkb: so I don't even know to reproduce this issue, set up multinode and run it 20 times? | 21:31 |
| clarkb | just shelve unshelve in a loop and ping after unshelve | 21:32 |
| jogo | that may work | 21:32 |
| jogo | I don't think my hp cloud quota is big enough for multi node though :( | 21:32 |
| jogo | either way this is holding a job hostage for one bug | 21:33 |
| jlk | mtreinish: so, reading through this code, I think we can get rid of _detect_server_image_flavor, and instead make it _reset_server_flavor and just explicitly set it back to primary each time. It'll mean rebuilds for every resize test, but guarantees we to from flavor_ref -> flaver_ref_alt every time. | 21:33 |
| clarkb | its not holding anything hostage... | 21:33 |
| clarkb | the job is running | 21:33 |
| clarkb | its reporting valid resulys | 21:34 |
| jogo | clarkb: but not voting, so no one looks at it | 21:34 |
| mtreinish | jlk: sure that works fine. My other thought is we could just break the resize tests out into a separate file and have each test explicitly create a server (and then clean it up after) | 21:34 |
| mtreinish | but either approach is fine | 21:34 |
| jlk | preference on using self.client.rebuild vs self.rebuild_server ? | 21:35 |
| mtreinish | jlk: it should be self.rebuild_server (it's not actually a rebuild it deletes the server and makes a new one) | 21:35 |
| *** mwagner_lap has quit IRC | 21:35 | |
| jlk | nod | 21:35 |
| clarkb | jogo: we can make it voting | 21:36 |
| clarkb | or at least I think we can, but I don't think we skip that test | 21:36 |
| jogo | clarkb: that works for me as well | 21:37 |
| clarkb | one of the major values to this test has been that we have fixed problems in shelve, if nova is going to support shelve we should test it | 21:37 |
| * jlk moves more code around. | 21:37 | |
| jogo | clarkb: so I agree, skipping would not mean skip this in all cases just the multinode case | 21:37 |
| jogo | anyway in a moment I will go over to infra and sort out if there is an issue with making this voting | 21:38 |
| clarkb | jogo: the only place we catch bugs in it is on mulitnode | 21:39 |
| *** dkranz has quit IRC | 21:39 | |
| *** samueldmq has quit IRC | 21:40 | |
| *** samueldmq_ is now known as samueldmq | 21:40 | |
| clarkb | jogo: basically the single node shelve unshelve is very trivial because you are restoring state to the place where it previousl existed. This means that if you do not clean up that state properly and/or don't write it back out things continue to work just fine | 21:41 |
| *** pc_m has quit IRC | 21:41 | |
| clarkb | and since tempest is only doing blackbox testing it does not assert intermediate state on each compute node to know if things get borked | 21:41 |
| mtreinish | clarkb: sigh, although I think there might be a couple of tests which do check for an intermediate state | 21:42 |
| mtreinish | it's been hard to track them all down... | 21:43 |
| clarkb | mtreinish: you would need to inspect dnsmasq mac addr mappings and list interface ip addr assignments | 21:44 |
| clarkb | not sure if tempest does any of that at the cloud layer | 21:44 |
| mtreinish | clarkb: oh I meant like intermediate state on a server bring up. Like there are tests which try to do things why a server is listed as building from nova | 21:45 |
| mtreinish | s/why/while | 21:45 |
| openstackgerrit | Joe Gordon proposed openstack/tempest: Skip test_shelve_instance in multinode case https://review.openstack.org/174652 | 21:45 |
| mtreinish | clarkb: oh, I hope we don't have things like that anymore. There used to be debugging stuff which ran local iptables and ovs commands, which was wrong on several level. | 21:46 |
| *** aix has joined #openstack-qa | 21:46 | |
| jlk | mtreinish: what's the decision point for booting up multiple servers for multiple tests? | 21:49 |
| clarkb | in fact that would probably make a good start, why not add a test like that to nova's functional tests? | 21:49 |
| clarkb | jogo: ^ | 21:49 |
| jlk | where does the parallelism get sorted out? | 21:49 |
| *** artom has quit IRC | 21:49 | |
| clarkb | jogo: and hopefully you can iterate on that to find the bug | 21:49 |
| mtreinish | jlk: parallelism happens at the test class level, so everything run inside of the class is run serially (although with no guarantee on ordering) | 21:50 |
| jlk | and what decides to run multiple classes at once? | 21:51 |
| mtreinish | jlk: normally we just have each test create it's own server if you need multiples. Single server reuse thing can get hairy sometimes (as you've experienced firsthand) | 21:51 |
| *** rfolco has quit IRC | 21:51 | |
| mtreinish | jlk: the test runner, so in the default case testr | 21:51 |
| jlk | yeah I'm just thinking of ways to express that each of these resize tests could be run in parallel | 21:51 |
| jlk | since doing it the way I'm about to do it makes it slower | 21:52 |
| mtreinish | jlk: if they're in the same class then they won't be run in parallel | 21:52 |
| jlk | okay | 21:52 |
| *** ekhugen has quit IRC | 21:52 | |
| *** zz_jgrimm has quit IRC | 21:54 | |
| *** timello has quit IRC | 21:54 | |
| mtreinish | jlk: I mean you could try and get clever and in setupClass create 3 servers by issuing 3 create calls and then do a wait loop after all three are created | 21:55 |
| *** ekhugen has joined #openstack-qa | 21:55 | |
| jlk | nah | 21:55 |
| jlk | just making a token effort to not introduce a performance decrease | 21:56 |
| mtreinish | heh, no one will really notice because the scenario tests with real volume ops are still much slower :) | 21:56 |
| *** zz_jgrimm has joined #openstack-qa | 21:56 | |
| jlk | hah alright | 21:56 |
| *** timello has joined #openstack-qa | 21:57 | |
| *** samueldmq has quit IRC | 21:57 | |
| *** dboik has quit IRC | 22:02 | |
| openstackgerrit | Jesse Keating proposed openstack/tempest: Delete server after resize test https://review.openstack.org/179272 | 22:05 |
| jlk | mtreinish: ^^ take 2 :) | 22:05 |
| openstackrecheck | openstack/keystone change: https://review.openstack.org/179796 failed because of: gate-tempest-dsvm-full: https://bugs.launchpad.net/bugs/1451506 | 22:07 |
| openstack | Launchpad bug 1451506 in OpenStack Compute (nova) "spawn failed with "libvirtError: internal error: received hangup / error event on socket" in the gate" [Undecided,Confirmed] | 22:07 |
| *** ekhugen has quit IRC | 22:08 | |
| *** zz_jgrimm has quit IRC | 22:09 | |
| *** mmedvede has joined #openstack-qa | 22:09 | |
| jogo | clarkb: so interesting data point | 22:10 |
| jogo | clarkb: according to alaski the network bits should be very similar to resize | 22:10 |
| jogo | clarkb: in tempest resize tests don't try to ssh at the end | 22:11 |
| *** maurosr has quit IRC | 22:11 | |
| jogo | mtreinish: ^ | 22:11 |
| *** ekhugen has joined #openstack-qa | 22:11 | |
| jogo | and only the scenario test that does an ssh check at the end fails | 22:11 |
| mtreinish | jogo: yeah the ssh bits on the resize check are broken | 22:12 |
| mtreinish | there's a bp in progress to fix all of that | 22:12 |
| *** zz_jgrimm has joined #openstack-qa | 22:12 | |
| jogo | mtreinish: do they run? | 22:12 |
| mtreinish | (which I probably should review again soon) | 22:12 |
| jlk | I don't recall seeing ssh bits in the resize tests | 22:12 |
| mtreinish | jlk: oh I assumed jogo looked and those were tests with run_ssh flags, I don't actually know | 22:13 |
| jlk | they aren't. | 22:13 |
| jlk | but if they were, they'd probably be broken unless there is a healthy timeout for getting an address on the rebuilt server :) | 22:13 |
| jogo | mtreinish: yup jlk is right | 22:13 |
| jogo | jlk: go on | 22:13 |
| jlk | could be our setup, but we see a minute or so of delay before ARP picks up the new location of the IP address | 22:14 |
| jogo | jlk: the scenario test test_shelve_instance is failing with that | 22:14 |
| jlk | when resizing. | 22:14 |
| jogo | jlk: oh very interesting | 22:14 |
| jlk | reverting the resize and the ARP picks up nearly instantly | 22:14 |
| jlk | I think I filed a bug on this | 22:14 |
| *** maurosr has joined #openstack-qa | 22:14 | |
| jogo | jlk: link? | 22:14 |
| jlk | still looking | 22:14 |
| jogo | jlk: this may be what we are seing with shelve unshelve | 22:14 |
| jogo | which can end up moving an instance | 22:15 |
| jlk | https://bugs.launchpad.net/neutron/+bug/1450874 | 22:15 |
| openstack | Launchpad bug 1450874 in neutron "Delay in network access after instance resize/migration using linuxbridge and vlan" [Undecided,New] | 22:15 |
| jogo | jlk: the issue I am seeing is with nova-network but I wonder if it is related | 22:16 |
| jlk | could be | 22:16 |
| *** zz_jgrimm has quit IRC | 22:16 | |
| jogo | jlk: any idea of what the underlying issue is / how to fix it? | 22:17 |
| jlk | I have the benefit of being able to manually run the test in my environment, so I could wait around for arp to finally work | 22:17 |
| jlk | I don't. I'm mostly guessing that it's an ARP cache issue | 22:17 |
| clarkb | jogo: arp is not the cause of our problems | 22:17 |
| clarkb | jogo: dnsmasq is | 22:17 |
| jlk | I only guess that because it eventually works. | 22:17 |
| clarkb | in our case the node boots, does 3 dhcp requests whcih fail because dnsmasq has no IP addr mapped for it | 22:18 |
| clarkb | the VM console logs and syslog for dnsmasq seem to confirm this | 22:18 |
| *** maurosr has quit IRC | 22:19 | |
| *** ekhugen has quit IRC | 22:19 | |
| *** zz_jgrimm has joined #openstack-qa | 22:19 | |
| clarkb | also if you are using neutron with l2pop then I expect all sorts of arp problems <_< | 22:20 |
| clarkb | jlk: are you using l2pop? | 22:20 |
| *** maurosr has joined #openstack-qa | 22:20 | |
| jlk | clarkb: I don't think so. | 22:21 |
| jlk | ml2, vlan | 22:21 |
| *** ekhugen has joined #openstack-qa | 22:21 | |
| jogo | clarkb: so DHCPDISCOVER(br100) fa:16:3e:fb:3e:3e no address available is what you found right? | 22:21 |
| jlk | unless it's a default? | 22:21 |
| clarkb | jlk: ya that sounds right | 22:21 |
| clarkb | er jogo^ | 22:21 |
| jogo | clarkb: because I see that in successful tempest runs a lot as well no address available | 22:21 |
| clarkb | jlk: I am not sure if its the default, but with l2pop regular arp doesn't work and you have neutron fill arp tables aiui | 22:21 |
| jlk | oh interesting. | 22:22 |
| *** mwagner_lap has joined #openstack-qa | 22:22 | |
| clarkb | jlk: if neutron is slow writing out arp tables for you I would expect that sort of behavior | 22:22 |
| clarkb | jogo: its probably racy | 22:22 |
| clarkb | jogo: and if the addr is available before the third requst its fine, if it isn't then you lose | 22:22 |
| jogo | clarkb: possibly what makes this harder to figure out is we don't test connectivity very often in tempest | 22:22 |
| *** pglass has quit IRC | 22:23 | |
| clarkb | jogo: oh ya that would also be a source of passing even if dhcp failed | 22:23 |
| jogo | clarkb: http://logs.openstack.org/80/170780/11/check/check-tempest-dsvm-multinode-full/8bff988//logs/syslog.txt.gz | 22:23 |
| jogo | successful run | 22:23 |
| jogo | with a bunch of warnings | 22:23 |
| jogo | checking single node | 22:24 |
| *** hitalia has quit IRC | 22:24 | |
| clarkb | jogo: but I have seen it where dhcp will fail twice then succeed on the third attempt | 22:24 |
| jlk | clarkb: just confirmed, l2_population is turned off | 22:24 |
| clarkb | just in my local `nova console-log` against devstack with cirros nodes | 22:24 |
| jogo | clarkb: I don't see any ' no address available' for single node devstack | 22:25 |
| jogo | so that is good at least | 22:25 |
| jogo | clarkb: I wonder if this is a general issue with all migration or multihost | 22:26 |
| clarkb | ya, I am not sure | 22:26 |
| jogo | good news is it looks like it isn't shelve specific anymore | 22:27 |
| clarkb | ya I was pretty sure it wasn't I just never had enough data to prove it | 22:28 |
| jogo | makes sense | 22:28 |
| jogo | bbiab, relocating | 22:28 |
| jogo | updated the bug https://bugs.launchpad.net/nova/+bug/1445569 | 22:28 |
| openstack | Launchpad bug 1445569 in OpenStack Compute (nova) "No dhcp lease after shelve unshelve" [High,Confirmed] | 22:28 |
| clarkb | but tempest not ssh'ing all the time would explain that | 22:28 |
| *** hitalia has joined #openstack-qa | 22:29 | |
| *** mmedvede has quit IRC | 22:32 | |
| *** aysyd has quit IRC | 22:33 | |
| *** bknudson has quit IRC | 22:44 | |
| *** cody-somerville has quit IRC | 22:44 | |
| *** shashankhegde has joined #openstack-qa | 22:46 | |
| openstackgerrit | Merged openstack/tempest: Fix logging.conf sample https://review.openstack.org/178723 | 22:51 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to convert append only tables to MyISAM https://review.openstack.org/179964 | 22:52 |
| openstackgerrit | Merged openstack/tempest: Honour ssh_connect_method in test_baremetal_server_ops https://review.openstack.org/174828 | 22:55 |
| *** armax has quit IRC | 22:57 | |
| openstackrecheck | openstack/keystone change: https://review.openstack.org/179846 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error | 22:59 |
| *** mlavalle has quit IRC | 23:00 | |
| *** rbak has quit IRC | 23:00 | |
| *** Sukhdev has quit IRC | 23:00 | |
| *** zz_jgrimm has quit IRC | 23:03 | |
| *** zz_jgrimm has joined #openstack-qa | 23:06 | |
| jogo | clarkb: going to check to see if I spot the dnsmasq error on instances that haven't been migrated | 23:11 |
| *** shashankhegde has quit IRC | 23:12 | |
| *** ajo has quit IRC | 23:13 | |
| *** cody-somerville has joined #openstack-qa | 23:14 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to convert append only tables to MyISAM https://review.openstack.org/179964 | 23:17 |
| *** shashankhegde has joined #openstack-qa | 23:22 | |
| *** zz_jgrimm has quit IRC | 23:27 | |
| openstackgerrit | yuxingw proposed openstack/tempest-lib: Password is not strong enough failures https://review.openstack.org/178546 | 23:27 |
| *** samueldmq has joined #openstack-qa | 23:28 | |
| *** maurosr has quit IRC | 23:28 | |
| jogo | clarkb: so I am seeing no address available on instances that nothing has been done to | 23:28 |
| *** maurosr has joined #openstack-qa | 23:29 | |
| *** zz_jgrimm has joined #openstack-qa | 23:29 | |
| jogo | clarkb: so we are seeing a nova multihost bug? | 23:30 |
| clarkb | jogo: my hunch has been its a race in nova network setting up dnsmasq | 23:31 |
| openstackgerrit | Jamie Lennox proposed openstack-dev/devstack: Remove admin auth from glance https://review.openstack.org/179970 | 23:34 |
| openstackgerrit | David Paterson proposed openstack/tempest: Added cleanup for tenant quotas https://review.openstack.org/179971 | 23:37 |
| *** hemna is now known as hemnafk | 23:37 | |
| jogo | clarkb: do we need to set /etc/dnsmasq? | 23:37 |
| clarkb | jogo: aiui either nova or devstack does all of that configuration | 23:37 |
| mtreinish | clarkb: nova should I think | 23:38 |
| jogo | by default it sets it to '' | 23:39 |
| jogo | see nova/network/linux_net | 23:39 |
| jogo | CONFIG_FILE=["/etc/nova/nova.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/opt/stack/data/nova/networks/nova-br100.pid --dhcp-optsfile=/opt/stack/data/nova/networks/nova-br100.opts --listen-address=10.1.0.3 --except-interface=lo --dhcp-range=set:private,10.1.0.2,static,255.255.240.0,86400s --dhcp-lease-max=4096 | 23:39 |
| jogo | --dhcp-hostsfile=/opt/stack/data/nova/networks/nova-br100.conf --dhcp-script=/usr/local/bin/nova-dhcpbridge --no-hosts --leasefile-ro --domain=novalocal --addn-hosts=/opt/stack/data/nova/networks/nova-br100.hosts | 23:40 |
| clarkb | jogo: I think your nova-br100* files are the ones that get updated | 23:40 |
| *** ajmiller has quit IRC | 23:41 | |
| *** dims_ has joined #openstack-qa | 23:42 | |
| jogo | clarkb: not sure | 23:44 |
| *** dims has quit IRC | 23:44 | |
| jogo | where are the dhcpbridge logs? | 23:44 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-dev/devstack: [WIP] Just checking contents of qemu.conf https://review.openstack.org/179943 | 23:46 |
| clarkb | jogo: no idea, that would be a devstack thing | 23:46 |
| *** cody-somerville has quit IRC | 23:49 | |
| *** greenstatic has joined #openstack-qa | 23:53 | |
| *** greenstatic has left #openstack-qa | 23:55 | |
| jogo | clarkb: so when no address is on one node | 23:58 |
| jogo | dnsmasq works on the other | 23:58 |
| jogo | fa:16:3e:d0:50:9b | 23:58 |
| *** BadCub is now known as BadCub_Hiding | 23:58 | |
| jogo | in http://logs.openstack.org/52/179952/1/check/check-tempest-dsvm-multinode-full/95537fd//logs/syslog.txt.gz and http://logs.openstack.org/52/179952/1/check/check-tempest-dsvm-multinode-full/95537fd//logs/subnode-1/syslog.txt.gz | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!