| *** mtanino has quit IRC | 00:00 | |
| *** tsekiyam_ has quit IRC | 00:01 | |
| *** suneelb has quit IRC | 00:02 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/175739 failed gate-tempest-dsvm-full, gate-tempest-dsvm-postgres-full, gate-tempest-dsvm-neutron-full, gate-devstack-dsvm-cells, gate-tempest-dsvm-nova-v21-full, gate-tempest-dsvm-full-ceph in the gate queue with an unrecognized error | 00:10 |
|---|---|---|
| *** davideagnello has quit IRC | 00:14 | |
| *** mlavalle has quit IRC | 00:15 | |
| *** yamamoto has quit IRC | 00:16 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Add unit test to ensure sane test class subclassing https://review.openstack.org/176539 | 00:18 |
| openstackrecheck | openstack/glance change: https://review.openstack.org/174405 failed gate-tempest-dsvm-full, gate-tempest-dsvm-postgres-full, gate-tempest-dsvm-neutron-full, gate-devstack-dsvm-cells, gate-tempest-dsvm-full-ceph in the gate queue with an unrecognized error | 00:23 |
| *** marun has quit IRC | 00:26 | |
| *** zz_dimtruck is now known as dimtruck | 00:27 | |
| openstackrecheck | openstack/glance change: https://review.openstack.org/171022 failed gate-tempest-dsvm-full, gate-tempest-dsvm-postgres-full, gate-tempest-dsvm-neutron-full, gate-devstack-dsvm-cells, gate-tempest-dsvm-full-ceph in the gate queue with an unrecognized error | 00:40 |
| *** bknudson has quit IRC | 00:45 | |
| *** annashen has joined #openstack-qa | 00:45 | |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/175122 failed gate-horizon-python27, gate-horizon-python27-django14, gate-horizon-selenium, gate-tempest-dsvm-full in the gate queue with an unrecognized error | 00:50 |
| *** pc_m has quit IRC | 00:53 | |
| *** amotoki has joined #openstack-qa | 00:55 | |
| openstackrecheck | openstack/glance change: https://review.openstack.org/169813 failed gate-tempest-dsvm-full, gate-tempest-dsvm-postgres-full, gate-tempest-dsvm-neutron-full, gate-devstack-dsvm-cells, gate-tempest-dsvm-full-ceph in the gate queue with an unrecognized error | 00:57 |
| *** mtanino has joined #openstack-qa | 00:58 | |
| *** annashen has quit IRC | 01:01 | |
| *** amotoki has quit IRC | 01:04 | |
| *** annashen has joined #openstack-qa | 01:06 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/devstack: Select Libreswan driver for openswan in Fedora https://review.openstack.org/176192 | 01:06 |
| *** annashen has quit IRC | 01:06 | |
| *** amotoki has joined #openstack-qa | 01:09 | |
| openstackrecheck | Console logs not available after 13:21s for check-devstack-dsvm-cells 176518,1,1e4f255 | 01:14 |
| *** dboik has joined #openstack-qa | 01:15 | |
| *** dboik has quit IRC | 01:19 | |
| *** bknudson has joined #openstack-qa | 01:20 | |
| *** shashankhegde has quit IRC | 01:21 | |
| *** wuhg has joined #openstack-qa | 01:21 | |
| *** achanda has quit IRC | 01:22 | |
| *** marun has joined #openstack-qa | 01:26 | |
| *** mlavalle has joined #openstack-qa | 01:28 | |
| *** haigang has joined #openstack-qa | 01:41 | |
| *** Longgeek has joined #openstack-qa | 01:46 | |
| openstackrecheck | Console logs not available after 13:22s for gate-horizon-python27 139026,5,af61ad7 | 01:49 |
| *** shashankhegde has joined #openstack-qa | 01:51 | |
| *** Longgeek has quit IRC | 01:56 | |
| *** Longgeek has joined #openstack-qa | 01:56 | |
| *** david-lyle has quit IRC | 01:59 | |
| openstackgerrit | Sam Wan proposed openstack-dev/devstack: Fixes multiple global_filter created in /etc/lvm/lvm.conf by set_lvm_filter. https://review.openstack.org/176222 | 02:00 |
| openstackrecheck | Console logs not available after 13:23s for gate-nova-python26 149371,3,4b27a81 | 02:02 |
| *** davideagnello has joined #openstack-qa | 02:02 | |
| *** davideagnello has quit IRC | 02:07 | |
| *** annashen has joined #openstack-qa | 02:07 | |
| *** suneelb has joined #openstack-qa | 02:10 | |
| *** annashen has quit IRC | 02:12 | |
| *** haigang has quit IRC | 02:12 | |
| *** harlowja is now known as harlowja_away | 02:12 | |
| *** ajmiller has joined #openstack-qa | 02:13 | |
| *** ajo has joined #openstack-qa | 02:13 | |
| openstackrecheck | Console logs not available after 13:22s for gate-horizon-python27-django14 173552,1,a527119 | 02:15 |
| *** patrickeast has quit IRC | 02:18 | |
| *** yamamoto has joined #openstack-qa | 02:22 | |
| *** marun has quit IRC | 02:23 | |
| *** rfolco has quit IRC | 02:28 | |
| openstackrecheck | Console logs not available after 13:22s for check-tempest-dsvm-full 176526,2,f4de365 | 02:29 |
| *** suneelb has quit IRC | 02:33 | |
| *** haigang has joined #openstack-qa | 02:39 | |
| openstackrecheck | Console logs not available after 13:22s for check-tempest-dsvm-full 175567,3,d7de63d | 02:42 |
| *** achanda has joined #openstack-qa | 02:53 | |
| openstackrecheck | Console logs not available after 13:22s for check-tempest-dsvm-full 176457,1,606290a | 02:56 |
| *** achanda has quit IRC | 02:59 | |
| *** yamamoto has quit IRC | 02:59 | |
| *** yamamoto has joined #openstack-qa | 03:05 | |
| *** dimtruck is now known as zz_dimtruck | 03:08 | |
| openstackrecheck | Console logs not available after 13:24s for gate-horizon-python27-django14 176532,1,4c6a872 | 03:09 |
| *** mmedvede has quit IRC | 03:10 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Make aiopcpu subnode directory more deterministic https://review.openstack.org/169449 | 03:15 |
| *** mmedvede has joined #openstack-qa | 03:18 | |
| *** annashen has joined #openstack-qa | 03:30 | |
| *** yamamoto has quit IRC | 03:31 | |
| *** shashankhegde has quit IRC | 03:42 | |
| *** Madan has joined #openstack-qa | 03:43 | |
| openstackgerrit | Merged openstack-dev/grenade: support apache based upgrades for keystone https://review.openstack.org/175831 | 03:46 |
| openstackgerrit | Merged openstack-dev/grenade: add process debugging when things fail https://review.openstack.org/175935 | 03:47 |
| openstackgerrit | Merged openstack-infra/devstack-gate: add support for stable/kilo https://review.openstack.org/175391 | 03:49 |
| openstackgerrit | Merged openstack/tempest: remove swift cli tests https://review.openstack.org/175447 | 03:49 |
| *** oomichi has joined #openstack-qa | 03:52 | |
| *** yamamoto has joined #openstack-qa | 03:56 | |
| *** annashen has quit IRC | 04:04 | |
| openstackgerrit | Merged openstack-dev/devstack: made several changes to guides to comply to doc conventions https://review.openstack.org/176029 | 04:05 |
| *** mlavalle_ has joined #openstack-qa | 04:05 | |
| *** mlavalle has quit IRC | 04:05 | |
| *** davideagnello has joined #openstack-qa | 04:07 | |
| *** krtaylor has quit IRC | 04:08 | |
| *** mlavalle_ has quit IRC | 04:11 | |
| *** krtaylor has joined #openstack-qa | 04:11 | |
| *** Sukhdev has joined #openstack-qa | 04:17 | |
| *** coolsvap|afk is now known as coolsvap | 04:31 | |
| *** coolsvap is now known as coolsvap|afk | 04:32 | |
| *** annashen has joined #openstack-qa | 04:35 | |
| *** annashen has quit IRC | 04:48 | |
| *** ajmiller has quit IRC | 04:54 | |
| *** david-lyle has joined #openstack-qa | 04:59 | |
| *** mtanino has quit IRC | 05:01 | |
| *** sks has joined #openstack-qa | 05:05 | |
| *** Poornima has joined #openstack-qa | 05:08 | |
| *** mmedvede has quit IRC | 05:08 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/devstack: Fix negated services with common prefix https://review.openstack.org/173593 | 05:10 |
| *** krtaylor has quit IRC | 05:19 | |
| *** Triveni has joined #openstack-qa | 05:19 | |
| *** annashen has joined #openstack-qa | 05:20 | |
| *** marrusl has quit IRC | 05:21 | |
| *** mmedvede has joined #openstack-qa | 05:21 | |
| *** rwsu has quit IRC | 05:21 | |
| openstackgerrit | Sam Wan proposed openstack-dev/devstack: Fixes multiple global_filter created in /etc/lvm/lvm.conf by set_lvm_filter. https://review.openstack.org/176222 | 05:26 |
| *** shashankhegde has joined #openstack-qa | 05:26 | |
| *** davideagnello has quit IRC | 05:30 | |
| *** armax has quit IRC | 05:32 | |
| *** achanda has joined #openstack-qa | 05:33 | |
| *** krtaylor has joined #openstack-qa | 05:33 | |
| *** marrusl has joined #openstack-qa | 05:34 | |
| *** k4n0 has joined #openstack-qa | 05:35 | |
| *** Longgeek has quit IRC | 05:44 | |
| *** mmedvede has quit IRC | 05:46 | |
| *** luqas has joined #openstack-qa | 05:51 | |
| *** luqas has quit IRC | 05:55 | |
| *** Longgeek has joined #openstack-qa | 05:57 | |
| *** mmedvede has joined #openstack-qa | 06:04 | |
| *** BharatK has joined #openstack-qa | 06:04 | |
| *** pkoniszewski has joined #openstack-qa | 06:04 | |
| *** ibiris_away is now known as ibiris | 06:06 | |
| *** afazekas_ has joined #openstack-qa | 06:11 | |
| *** yog__ has joined #openstack-qa | 06:14 | |
| *** Longgeek has quit IRC | 06:16 | |
| *** Sukhdev has quit IRC | 06:17 | |
| *** anshul has quit IRC | 06:20 | |
| *** Longgeek has joined #openstack-qa | 06:26 | |
| *** anshul has joined #openstack-qa | 06:33 | |
| *** woodster_ has quit IRC | 06:40 | |
| *** Longgeek_ has joined #openstack-qa | 06:43 | |
| *** Longgeek has quit IRC | 06:46 | |
| *** annashen has quit IRC | 06:46 | |
| *** sileht has quit IRC | 06:48 | |
| *** yfried|afk has quit IRC | 06:56 | |
| *** mpaolino has joined #openstack-qa | 06:57 | |
| *** neeti has joined #openstack-qa | 07:00 | |
| *** sputnik13 has joined #openstack-qa | 07:02 | |
| openstackgerrit | Ramakrishnan G proposed openstack-dev/devstack: Add support for ilo drivers in lib/ironic https://review.openstack.org/174733 | 07:03 |
| openstackgerrit | Ramakrishnan G proposed openstack-dev/devstack: Add support for ilo drivers in lib/ironic https://review.openstack.org/174733 | 07:05 |
| openstackgerrit | Sam Wan proposed openstack-dev/devstack: Fixes multiple global_filter created in /etc/lvm/lvm.conf by set_lvm_filter. https://review.openstack.org/176222 | 07:07 |
| *** achanda has quit IRC | 07:07 | |
| *** pkoniszewski has quit IRC | 07:07 | |
| *** pkoniszewski has joined #openstack-qa | 07:07 | |
| *** Longgeek_ has quit IRC | 07:09 | |
| *** shashankhegde has quit IRC | 07:09 | |
| *** shashankhegde has joined #openstack-qa | 07:10 | |
| *** sileht has joined #openstack-qa | 07:10 | |
| *** Longgeek has joined #openstack-qa | 07:11 | |
| *** luqas has joined #openstack-qa | 07:11 | |
| *** pkoniszewski has quit IRC | 07:12 | |
| *** rushiagr_away is now known as rushiagr | 07:12 | |
| *** jyuso has joined #openstack-qa | 07:13 | |
| *** heyongli has quit IRC | 07:15 | |
| *** alex_xu has quit IRC | 07:15 | |
| *** alex_xu_ has joined #openstack-qa | 07:15 | |
| *** jyuso1 has quit IRC | 07:15 | |
| *** weshay has joined #openstack-qa | 07:17 | |
| *** heyongli has joined #openstack-qa | 07:17 | |
| *** shashankhegde has quit IRC | 07:18 | |
| *** davideagnello has joined #openstack-qa | 07:18 | |
| *** tkammer|not_here has joined #openstack-qa | 07:23 | |
| *** davideagnello has quit IRC | 07:23 | |
| *** bkopilov has quit IRC | 07:26 | |
| *** yfried|afk has joined #openstack-qa | 07:26 | |
| *** SridharG has joined #openstack-qa | 07:29 | |
| *** jlanoux has joined #openstack-qa | 07:40 | |
| *** e0ne has joined #openstack-qa | 07:41 | |
| *** bkopilov has joined #openstack-qa | 07:44 | |
| *** coolsvap|afk is now known as coolsvap | 07:52 | |
| *** coolsvap is now known as coolsvap|afk | 07:53 | |
| *** e0ne is now known as e0ne_ | 07:53 | |
| *** safchain has joined #openstack-qa | 07:56 | |
| *** yfried|afk has quit IRC | 08:01 | |
| *** gfidente has joined #openstack-qa | 08:03 | |
| *** gfidente has quit IRC | 08:03 | |
| *** gfidente has joined #openstack-qa | 08:03 | |
| *** apevec has joined #openstack-qa | 08:08 | |
| *** apevec has quit IRC | 08:08 | |
| *** apevec has joined #openstack-qa | 08:08 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:08 | |
| *** jyuso has quit IRC | 08:10 | |
| *** jordanP has joined #openstack-qa | 08:11 | |
| *** e0ne_ is now known as e0ne | 08:14 | |
| *** liusheng has left #openstack-qa | 08:15 | |
| *** bkopilov has quit IRC | 08:19 | |
| *** jordanP has quit IRC | 08:19 | |
| *** e0ne is now known as e0ne_ | 08:25 | |
| *** yfried|afk has joined #openstack-qa | 08:25 | |
| *** e0ne_ is now known as e0ne | 08:26 | |
| *** andreykurilin__ has joined #openstack-qa | 08:27 | |
| *** ihrachyshka has joined #openstack-qa | 08:27 | |
| *** Longgeek has quit IRC | 08:31 | |
| *** Longgeek has joined #openstack-qa | 08:31 | |
| *** e0ne has quit IRC | 08:32 | |
| *** jordanP has joined #openstack-qa | 08:43 | |
| *** hichihara has quit IRC | 08:46 | |
| *** sputnik1_ has joined #openstack-qa | 08:55 | |
| *** sputnik13 has quit IRC | 08:55 | |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/175869 failed gate-horizon-python27, gate-horizon-python27-django14, gate-horizon-selenium in the gate queue with an unrecognized error | 08:55 |
| *** Matt2 has quit IRC | 08:56 | |
| *** sputnik13 has joined #openstack-qa | 08:58 | |
| *** sputnik1_ has quit IRC | 09:01 | |
| *** davideagnello has joined #openstack-qa | 09:08 | |
| *** davideagnello has quit IRC | 09:12 | |
| *** e0ne has joined #openstack-qa | 09:13 | |
| *** shardy has joined #openstack-qa | 09:18 | |
| *** e0ne is now known as e0ne_ | 09:31 | |
| *** haigang has quit IRC | 09:31 | |
| *** anshul has quit IRC | 09:32 | |
| *** aix has joined #openstack-qa | 09:35 | |
| *** e0ne_ is now known as e0ne | 09:35 | |
| *** mmedvede has quit IRC | 09:43 | |
| *** tosky has joined #openstack-qa | 09:43 | |
| *** anshul has joined #openstack-qa | 09:44 | |
| *** andreykurilin__ has quit IRC | 09:44 | |
| *** yamamoto has quit IRC | 09:45 | |
| *** annashen has joined #openstack-qa | 09:47 | |
| *** annashen has quit IRC | 09:51 | |
| *** e0ne is now known as e0ne_ | 09:53 | |
| *** Longgeek has quit IRC | 09:57 | |
| *** Longgeek has joined #openstack-qa | 09:59 | |
| *** pcrews has quit IRC | 09:59 | |
| *** cody-somerville has joined #openstack-qa | 09:59 | |
| *** cody-somerville has quit IRC | 09:59 | |
| *** cody-somerville has joined #openstack-qa | 09:59 | |
| *** mmedvede has joined #openstack-qa | 10:00 | |
| *** e0ne_ has quit IRC | 10:03 | |
| *** yamamoto has joined #openstack-qa | 10:07 | |
| *** e0ne has joined #openstack-qa | 10:09 | |
| openstackgerrit | nayna-patel proposed openstack/tempest: CRUD operations for Glance Namespaces https://review.openstack.org/176686 | 10:09 |
| openstackrecheck | openstack/ceilometer change: https://review.openstack.org/174320 failed gate-tempest-dsvm-neutron-full in the gate queue with an unrecognized error | 10:12 |
| *** afazekas_ has quit IRC | 10:14 | |
| *** amotoki_ has joined #openstack-qa | 10:15 | |
| *** samueldmq has joined #openstack-qa | 10:21 | |
| *** mpaolino has quit IRC | 10:26 | |
| *** k4n0 has quit IRC | 10:28 | |
| *** cdent has joined #openstack-qa | 10:32 | |
| *** afazekas has joined #openstack-qa | 10:32 | |
| *** Triveni has quit IRC | 10:32 | |
| *** k4n0 has joined #openstack-qa | 10:34 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** rushiagr is now known as rushiagr_away | 10:40 | |
| openstackgerrit | Abhishek G M proposed openstack/tempest: Test Creation of server in a shared_network https://review.openstack.org/168910 | 10:45 |
| cdent | Is there a preferred way to turn on USE_VENV in a devstack-based gate functional test? gnocchi doesn't follow global requirements and is ahead of keystone on oslo.db but keystone-manage barfs, thinking perhaps venv will ork around that | 10:47 |
| *** sputnik13 has quit IRC | 10:48 | |
| *** pkoniszewski has joined #openstack-qa | 10:48 | |
| openstackgerrit | Abhishek G M proposed openstack/tempest: Test Create of VM with multiple Nics https://review.openstack.org/168040 | 10:50 |
| *** davideagnello has joined #openstack-qa | 10:56 | |
| *** SridharG1 has joined #openstack-qa | 11:00 | |
| *** davideagnello has quit IRC | 11:01 | |
| *** SridharG has quit IRC | 11:03 | |
| openstackrecheck | openstack/ceilometer change: https://review.openstack.org/176572 failed because of: gate-tempest-dsvm-postgres-full: https://bugs.launchpad.net/bugs/1429151 | 11:04 |
| openstack | Launchpad bug 1429151 in Cinder "SimpleReadOnlyCinderClientTest.test_cinder_encryption_type_list fails randomly" [High,Confirmed] | 11:04 |
| *** yamamoto has joined #openstack-qa | 11:07 | |
| nithyag_ | ping @afazekas | 11:08 |
| *** aix has quit IRC | 11:22 | |
| *** yamamoto has quit IRC | 11:23 | |
| *** david-lyle has quit IRC | 11:29 | |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: add convenience tools https://review.openstack.org/174111 | 11:33 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: allow DEVSTACK_DIRS to be overridden https://review.openstack.org/176700 | 11:33 |
| *** alex_xu_ has quit IRC | 11:38 | |
| *** weshay has quit IRC | 11:41 | |
| *** alex_xu has joined #openstack-qa | 11:41 | |
| *** weshay has joined #openstack-qa | 11:46 | |
| *** yog__ has quit IRC | 11:51 | |
| *** bknudson has quit IRC | 11:57 | |
| *** BharatK has quit IRC | 11:58 | |
| *** aix has joined #openstack-qa | 12:00 | |
| *** aix has quit IRC | 12:00 | |
| *** aix has joined #openstack-qa | 12:00 | |
| *** jhenner has quit IRC | 12:03 | |
| *** Triveni has joined #openstack-qa | 12:03 | |
| *** e0ne is now known as e0ne_ | 12:04 | |
| *** david-lyle has joined #openstack-qa | 12:05 | |
| *** cdent has quit IRC | 12:13 | |
| *** cdent has joined #openstack-qa | 12:14 | |
| *** jgrimm is now known as zz_jgrimm | 12:15 | |
| afazekas | nithyag_, pong | 12:16 |
| *** luqas has quit IRC | 12:24 | |
| *** e0ne_ is now known as e0ne | 12:24 | |
| *** k4n0 has quit IRC | 12:27 | |
| *** woodster_ has joined #openstack-qa | 12:29 | |
| *** david-lyle has quit IRC | 12:31 | |
| openstackgerrit | Ramakrishnan G proposed openstack-dev/devstack: Add support for ilo drivers in lib/ironic https://review.openstack.org/174733 | 12:35 |
| *** jhenner has joined #openstack-qa | 12:35 | |
| *** SridharG1 has quit IRC | 12:37 | |
| *** neeti_ has joined #openstack-qa | 12:37 | |
| *** neeti has quit IRC | 12:38 | |
| *** mpaolino has joined #openstack-qa | 12:40 | |
| mtreinish | cdent: we just use tox to run tests for tempest in a dsvm env. It takes care of the venv separation for running the tests | 12:41 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 12:41 |
| mtreinish | cdent: or are you trying to install gnocchi in a venv and run tests from a different place? | 12:41 |
| cdent | mtreinish: I'm not entirely certain what the problem or scope really is | 12:42 |
| cdent | Basically when keystone-manage runs (during the devstack build) it conflicts with an oslo_db that gnocchi wants to install. | 12:42 |
| sdague | cdent: correct, gnocchi needs to stay within global-requirements to be runnable in a devstack | 12:43 |
| cdent | If I add USE_VENV to the localrc that is getting generated via the gate_hook things work | 12:43 |
| cdent | sdague: we're thinking about ways to work around that because it is has proved too limited | 12:43 |
| sdague | USE_VENV is all kind of experimental at this point | 12:43 |
| cdent | what we will probably do is making gnocchi _always_ run in a venv | 12:44 |
| *** jhenner has quit IRC | 12:44 | |
| * cdent finds some links | 12:44 | |
| cdent | https://review.openstack.org/#/c/176503/ | 12:44 |
| sdague | cdent: so, basically, that probably will mean the ops community rejects it entirely | 12:44 |
| *** davideagnello has joined #openstack-qa | 12:45 | |
| cdent | I think the hope is that it is a temporary patch while waiting for the rest of the world to catch up, but I don't actually know | 12:45 |
| cdent | I was just trying to see what was possible | 12:45 |
| cdent | the original review that introduced the problem was: https://review.openstack.org/#/c/175872/ | 12:46 |
| *** amotoki_ has quit IRC | 12:46 | |
| cdent | s/introduced/uncovered/ | 12:46 |
| sdague | it's not actually a problem, it's the way the system is designed to work | 12:47 |
| *** Longgeek has quit IRC | 12:47 | |
| cdent | I'm increasingly feeling like the purpose of the gate in my mind is not the same as the purpose of the gate in the general problem. | 12:47 |
| *** erlon has joined #openstack-qa | 12:47 | |
| cdent | In my mind it's point should be to discover and fix breakage as fast as possible. | 12:47 |
| cdent | Whereas it seems like it's point is to keep things stable. | 12:47 |
| cdent | Which has the knock on effect of slowing down the fixing of breakage. | 12:48 |
| sdague | right, it's pre-commit ci | 12:48 |
| *** Longgeek has joined #openstack-qa | 12:48 | |
| sdague | so if you put things into pre-commit ci the point is the tree is never broken | 12:48 |
| sdague | so that all 1000 developers get to start from known good points | 12:48 |
| sdague | and aren't constantly questioning if their code is broken because of someone else's commit | 12:49 |
| sdague | you can also do post-commit ci in the system, but people don't very much | 12:49 |
| sdague | which would be it's ok to break things, and we'll fix them up later | 12:50 |
| *** davideagnello has quit IRC | 12:50 | |
| cdent | Okay, so going back to the practical issue: | 12:50 |
| cdent | If gnocchi wants to be decoupled from the the release cycle (which has requirements freezes and the like on the cycle) but still wants to be play well with others and make use of the ci, what are some options for accomplishing that? | 12:52 |
| cdent | The two that I can think of at this point are: a) turn on venvs throughout devstack b) make gnocchi use its own venv | 12:52 |
| cdent | sdague: are there others? | 12:52 |
| sdague | those are probably it, however it doesn't actually "play well with others" in either of those cases | 12:53 |
| *** Longgeek_ has joined #openstack-qa | 12:55 | |
| cdent | How are you definiing "play well with others"? Uses the same requirements? | 12:56 |
| *** SridharG has joined #openstack-qa | 12:56 | |
| sdague | works as a system level install | 12:56 |
| sdague | with the other components | 12:56 |
| cdent | K, just wanted to clarify | 12:56 |
| sdague | it basically makes it unpackagable by distros | 12:56 |
| *** Longgeek has quit IRC | 12:56 | |
| cdent | I don't have a particular iron in this fire, I'm just trying to gather as much data as possible or having a conversation with jd and mehdi as we decide what to do | 12:57 |
| sdague | sure | 12:57 |
| sdague | so a current standing tenant is the system level services should all be coinstallable at a system level | 12:57 |
| cdent | there's a next issue which is that keystone-manage and keystone-all are easy_install script and thus use pkg_resources to create a WorkingSet (which is the thing that blows up when the wrong oslo_db is present). Is that a common style or is the general pattern these days to use console_script entry points? | 12:59 |
| *** bknudson has joined #openstack-qa | 13:00 | |
| sdague | cdent: that's a style that exists, it's be hit by openstack client after that point if you get past that one | 13:00 |
| sdague | or on the swift / ceilometermiddleware start | 13:01 |
| * cdent nods | 13:01 | |
| cdent | hmmm | 13:01 |
| *** dkranz has joined #openstack-qa | 13:02 | |
| *** zz_dimtruck is now known as dimtruck | 13:03 | |
| *** jhenner has joined #openstack-qa | 13:07 | |
| *** rahmu has quit IRC | 13:08 | |
| *** rahmu has joined #openstack-qa | 13:09 | |
| *** Triveni has quit IRC | 13:09 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/168778 failed gate-tempest-dsvm-nova-v21-full in the gate queue with an unrecognized error | 13:10 |
| *** yamamoto has joined #openstack-qa | 13:11 | |
| *** jecarey has joined #openstack-qa | 13:13 | |
| *** yamamoto has quit IRC | 13:16 | |
| *** mriedem_away has quit IRC | 13:21 | |
| *** dustins has joined #openstack-qa | 13:21 | |
| *** luqas has joined #openstack-qa | 13:25 | |
| *** e0ne has quit IRC | 13:25 | |
| *** sks has quit IRC | 13:25 | |
| cdent | thanks for the info sdague that gives me enough to chew on and think about how to move things along | 13:27 |
| *** neeti_ has quit IRC | 13:27 | |
| *** e0ne has joined #openstack-qa | 13:27 | |
| *** dimtruck is now known as zz_dimtruck | 13:28 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest-lib: Switch from using custom urlparse import logic to six https://review.openstack.org/176731 | 13:28 |
| *** rfolco has joined #openstack-qa | 13:28 | |
| sdague | coolio | 13:29 |
| *** mpaolino has quit IRC | 13:29 | |
| *** luqas has quit IRC | 13:30 | |
| openstackgerrit | Baodong (Robert) Li proposed openstack-dev/devstack: Add /usr/local/bin to exec_dirs in rootwrap.conf https://review.openstack.org/176041 | 13:35 |
| *** dboik has joined #openstack-qa | 13:39 | |
| *** wuhg has quit IRC | 13:39 | |
| *** SridharGaddam has quit IRC | 13:42 | |
| *** mriedem has joined #openstack-qa | 13:47 | |
| *** shardy_ has joined #openstack-qa | 13:57 | |
| *** shardy has quit IRC | 13:59 | |
| *** tochi has quit IRC | 13:59 | |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 14:01 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: make the ss debug include all services https://review.openstack.org/176748 | 14:01 |
| sdague | dansmith: so.... I think nova might have the same stop problem that keystone had | 14:01 |
| sdague | there is an oslo fix for that | 14:02 |
| dansmith | sdague: the same "stop problem" ? | 14:02 |
| *** shardy_ has quit IRC | 14:02 | |
| sdague | the old oslo service code ignores signals if there are open connections | 14:02 |
| dansmith | hmm, really? | 14:03 |
| sdague | apparently | 14:03 |
| sdague | bknudson figured out that was the root issue | 14:03 |
| *** shardy has joined #openstack-qa | 14:03 | |
| sdague | it ignores the term signal at least | 14:03 |
| dansmith | I just did something to nova the other day that was only happening when we received the term signal | 14:03 |
| dansmith | related to deleting all events | 14:03 |
| bknudson | doesn't ignore it... allows the term signal to shut down the process | 14:03 |
| dansmith | so that we don't hang forever | 14:03 |
| *** yamamoto has joined #openstack-qa | 14:04 | |
| bknudson | to recreate, all I did was open a connection to the process using nc | 14:04 |
| sdague | bknudson: ok, you explain the issue then, because I don't have it fully in my head | 14:04 |
| dansmith | sdague: https://review.openstack.org/#/c/169056/ | 14:04 |
| dansmith | sdague: so roman at least seems to be able to make it do the term thing: https://bugs.launchpad.net/nova/+bug/1438183 | 14:05 |
| openstack | Launchpad bug 1438183 in OpenStack Compute (nova) "Graceful shutdown of nova-compute service fails" [Medium,Fix released] - Assigned to Dan Smith (danms) | 14:05 |
| sdague | dansmith: did that end up in stable/kilo? | 14:05 |
| dansmith | sdague: targeted for rc1 | 14:05 |
| sdague | https://bugs.launchpad.net/oslo-incubator/+bug/1446583 | 14:05 |
| openstack | Launchpad bug 1446583 in oslo-incubator "services no longer reliably stop in stable/kilo" [Critical,Fix committed] - Assigned to Julien Danjou (jdanjou) | 14:05 |
| bknudson | the server would fork off a bunch of child processes... these child processes were handling the SIGTERM, by setting a flag to exit... | 14:05 |
| dansmith | sdague: without that fix, and instances mid-boot, you might perceive it as ignoring the term signal | 14:05 |
| bknudson | but for some reason, maybe related to eventlet, the child wouldn't actually process the SIGTERM if it was reading from the socket. | 14:05 |
| bknudson | by default, a SIGTERM will just kill the process anyways, so there was no reason for the child to actually try to catch it and handle it if all it was going to do was shut down anyway | 14:06 |
| *** paul_glass has joined #openstack-qa | 14:06 | |
| bknudson | so the fix was to change the child process from trying to handle SIGTERM and just let SIGTERM kill the process. | 14:06 |
| dansmith | that likely doesn't apply to nova | 14:07 |
| sdague | dansmith: because? | 14:07 |
| *** luqas has joined #openstack-qa | 14:08 | |
| dansmith | sdague: well, I meant nova-compute, because it has no child processes | 14:08 |
| dansmith | and because we can't ignore it and achieve the same goal | 14:08 |
| dansmith | we need to handle it | 14:08 |
| bknudson | this was definitely related to how the oslo-incubator code handled the multi-process model. | 14:08 |
| sdague | sure, the reason this failed - http://logs.openstack.org/23/176723/1/check/check-grenade-dsvm-neutron/6307eb0/ however was because nova-api didn't shut down | 14:08 |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/176012 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error | 14:08 |
| *** jhenner has quit IRC | 14:09 | |
| *** Poornima has quit IRC | 14:09 | |
| bknudson | I don't understand why eventlet seems to have an issue with handling a signal while it's reading. | 14:09 |
| dansmith | sdague: okay, nova-api would apply I guess because (a) child processes and (b) we don't need to do the same thing that nova-compute does on term | 14:09 |
| bknudson | typically if you get a signal, even if it's handled, the read gets a ESIG or something. | 14:10 |
| dansmith | bknudson: http://logs.openstack.org/23/176723/1/check/check-grenade-dsvm-neutron/6307eb0/logs/old/screen-n-api.txt.gz | 14:10 |
| dansmith | bknudson: "child caught sigterm" -- is that the symptom/ | 14:10 |
| bknudson | EINTR | 14:10 |
| sdague | dansmith: actually, yes, I think that's typically the issue | 14:10 |
| dansmith | okay | 14:10 |
| sdague | the symptom | 14:10 |
| sdague | we can look at the working run | 14:11 |
| dansmith | I *think* nova-api would be the only server affected in nova, based on your description | 14:11 |
| bknudson | http://git.openstack.org/cgit/openstack/oslo-incubator/tree/openstack/common/service.py#n350 | 14:11 |
| bknudson | -- that's the service code... so it's like "Child %(pid)d killed by signal %(sig)d" or "'Child %(pid)s exited with status %(code)d'" | 14:11 |
| sdague | dansmith: so... I thought I saw it with nova-conductor in other runs | 14:12 |
| dansmith | bknudson: yeah, see the bottom of that log | 14:12 |
| *** rbak has joined #openstack-qa | 14:12 | |
| dansmith | sdague: ah, true, conductor does use the same worker model as api, | 14:12 |
| bknudson | y, that's exactly what it looked like. | 14:12 |
| dansmith | sdague: but it needs to handle it like compute does | 14:12 |
| dansmith | sdague: it can't just punt like api could | 14:12 |
| sdague | dansmith: ok, so... that's a different patch? | 14:13 |
| *** ChuckC has quit IRC | 14:13 | |
| bknudson | oh, so you don't want the children to just get killed? because then my patch breaks nova. | 14:13 |
| dansmith | I dunno, let me look | 14:13 |
| dansmith | bknudson: not for conductor, ideally | 14:13 |
| sdague | bknudson: nova-compute doesn't have workers | 14:13 |
| sdague | so it shouldn't affect that | 14:13 |
| bknudson | y, this is nova-api log I'm looking at. | 14:13 |
| dansmith | right | 14:14 |
| sdague | yep | 14:14 |
| dansmith | there are lots of patches in that bug, which one should I be looking at? | 14:14 |
| sdague | the oslo incubator one probably | 14:14 |
| dansmith | 176151/ | 14:14 |
| dansmith | right, yeah, we don't want to do that in nova-conductor if we can help it | 14:15 |
| bknudson | you want nova-conductor to keep running after signaled? | 14:15 |
| dansmith | it needs to finish things up, yeah | 14:15 |
| bknudson | of course, if it gets a SIGKILL it's going away anyways. | 14:15 |
| dansmith | it could be instrumenting a migration between two computes | 14:16 |
| dansmith | bknudson: of course, but term is supposed to be "finish your work, get to a stopping point, and shut down" | 14:16 |
| bknudson | from what I've seen of RHEL is that it gives your process a few seconds. | 14:16 |
| dansmith | eh? | 14:16 |
| bknudson | it'll do a SIGTERM and then a few seconds later it tries another signal. | 14:17 |
| dansmith | on system shutdown you mean | 14:17 |
| bknudson | y | 14:17 |
| dansmith | right, but that's different | 14:17 |
| openstackgerrit | Martin Pavlásek proposed openstack/tempest: Added test of default domain in Keystone https://review.openstack.org/31782 | 14:17 |
| bknudson | than service stop? | 14:17 |
| dansmith | sure | 14:17 |
| dansmith | and also, not everyone is on rhel, systemd, etc | 14:17 |
| dansmith | if you're rolling out new code, | 14:18 |
| *** shashankhegde has joined #openstack-qa | 14:18 | |
| dansmith | you need to be able to sigterm the process, let it land gracefully, and then restart it with new code | 14:18 |
| dansmith | for minimal disruption to the running deployment | 14:18 |
| dansmith | that's the point of this stuff | 14:18 |
| bknudson | how long will it take? | 14:18 |
| dansmith | on compute it could be quite a while | 14:18 |
| dansmith | we go to great pains to stop listening for new rpc requests, and let all running threads finish before shutting down | 14:19 |
| dansmith | else instances and db state end up in all manner of broken when you restart | 14:19 |
| bknudson | oh, good. | 14:19 |
| bknudson | at least you thought about it and it's not just random behavior... unlike keystone not stopping when there's a connection open. | 14:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: allow DEVSTACK_DIRS to be overridden https://review.openstack.org/176700 | 14:20 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: add convenience tools https://review.openstack.org/174111 | 14:20 |
| dansmith | bknudson: no, this is very calculated behavior | 14:20 |
| sdague | dansmith: so .... why might API be hanging | 14:20 |
| bknudson | ok, so the question for the bug was only about nova-api. | 14:20 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 14:20 | |
| dansmith | sdague: well, it sounds like bknudson suspects weird eventlet stuff? | 14:20 |
| dansmith | it certainly seems like it's doing the right thing in that log, with things tearing down, etc | 14:21 |
| bknudson | the nova-api log looks the same as the keystone log, so I expect it's the same problem... easy to recreate by just opening a connection to nova and trying to stop it. | 14:21 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 14:21 | |
| dansmith | bknudson: but, both api and conductor have connections open to the mq all the time | 14:23 |
| dansmith | bknudson: maybe this is a bug in the wsgi stuff? | 14:23 |
| sdague | so - http://logs.openstack.org/23/176723/1/check/check-grenade-dsvm-neutron/6307eb0/logs/old/screen-n-api.txt.gz#_2015-04-23_13_18_35_857 was an inbound connection 15 seconds prior to the kill | 14:23 |
| bknudson | I didn't look at outbound connections at all. | 14:23 |
| *** rwsu has joined #openstack-qa | 14:24 | |
| dansmith | sdague: which seems like a lot of padding before kill time | 14:24 |
| *** anshul has quit IRC | 14:24 | |
| sdague | it http keepalive on? | 14:24 |
| bknudson | some web clients use a persistent connections. | 14:24 |
| dansmith | right, but all the clients here are commandline bits or tempest, right? | 14:25 |
| bknudson | the keystoneclient session code uses connection pools | 14:25 |
| dansmith | and those should be gone immediately I think | 14:25 |
| dansmith | I'm spinning up a devstack to poke at | 14:25 |
| sdague | all the clients will be the python-*client libs or tempest | 14:26 |
| dansmith | right | 14:26 |
| sdague | python-*clients use requests | 14:26 |
| dansmith | none of which should still be living by the time we run the kill, yes? | 14:26 |
| dansmith | I mean, none of the processes making those calls through the clients | 14:26 |
| openstackgerrit | Jens Rosenboom proposed openstack-dev/devstack: DO NOT MERGE - Update default cirros version https://review.openstack.org/166778 | 14:27 |
| sdague | well, in the neutron case, neutron is still up | 14:27 |
| sdague | nova comes down first | 14:28 |
| dansmith | ah, right | 14:28 |
| dansmith | I'm fine if we can blame this on neutron, yet again :P | 14:28 |
| *** mtanino has joined #openstack-qa | 14:28 | |
| dansmith | sdague: you said you saw this on conductor once? | 14:28 |
| bknudson | I started devstack, was able to CTRL-C n-api, then I restarted it, in another window did 'nc localhost 8775', then tried to CTRL-C n-api again... it didn't shut down. | 14:29 |
| bknudson | this is the same problem keystone-all had. | 14:29 |
| dansmith | bknudson: and just to be clear, when you kill the nc, it doesn't finish shutting down, right? | 14:30 |
| bknudson | dansmith: correct, 2015-04-23 09:28:45.606 INFO nova.openstack.common.service [-] Child 32127 exited with status 1 | 14:30 |
| bknudson | that was the last line. | 14:30 |
| sdague | dansmith: yeh, there is an ER query in the bug | 14:30 |
| bknudson | then I CTRL-C again and it goes away | 14:30 |
| dansmith | oh, hang on | 14:30 |
| bknudson | or if I disconnect the client it goes away | 14:31 |
| dansmith | the first batch of caffeine is just starting to metabolize... | 14:31 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVGhlIGZvbGxvd2luZyBzZXJ2aWNlcyBhcmUgc3RpbGwgcnVubmluZ1wiIEFORCBtZXNzYWdlOlwiZGllXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0Mjk3OTk0NTY0OTB9 | 14:31 |
| dansmith | the change to incubator doesn't look like it would break anything in nova, because it's not like it's avoiding calling the thing we use in compute to finish processing | 14:31 |
| dansmith | so we must make another call to signal() ourselves | 14:31 |
| bknudson | sdague: are those all failures? I wonder if sometimes they do eventually stop. | 14:32 |
| sdague | bknudson: those are all failures | 14:33 |
| sdague | that's after the 10 second wait | 14:33 |
| sdague | I can increase the wait time if you think they'll eventually exit | 14:33 |
| bknudson | 10 seconds should be enough | 14:33 |
| bknudson | at least for keystone-all... I don't know about the other nova- processes. | 14:33 |
| sdague | that's the message you get when it goes fatal (note the die) | 14:33 |
| sdague | based on what's in the request logs, I feel like at least for nova-api there shouldn't be any actual outstanding work | 14:34 |
| bknudson | if you're saying that nova-conductor needs all the time it can get to eventually shut down, then 10 seconds might not be enough. | 14:34 |
| *** achanda has joined #openstack-qa | 14:34 | |
| sdague | sure, agree, dansmith can weigh in on that | 14:34 |
| dansmith | well, | 14:35 |
| dansmith | conductor and compute need all the time they can get to shutdown | 14:35 |
| dansmith | however, | 14:35 |
| dansmith | if this is a -infra timeout, and 10 seconds should be enough based on the workload, then that's fine | 14:35 |
| sdague | we're in a pretty constrained env here, so they should be idle | 14:35 |
| sdague | yeh, it's a grenade timeout | 14:35 |
| dansmith | however, it doesn't sound like nova-api is going to eventually shut down | 14:35 |
| bknudson | might be interesting to see what those processes are up to and waiting on, if that's possible to dump out. | 14:35 |
| *** BadCub_Hiding is now known as BadCub | 14:36 | |
| sdague | https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/shutdown.sh#L30 | 14:36 |
| bknudson | I assumed for keystone-all that admin doesn't care what users might be up to. | 14:36 |
| dansmith | okay, so, I don't think this makes it any worse for nova | 14:36 |
| *** artom has joined #openstack-qa | 14:36 | |
| dansmith | and it probably means that conductor is not doing what we think it is/should today | 14:36 |
| dansmith | compute works differently and it's the important one | 14:37 |
| dansmith | so if this fixes the problem, we should just sync it to nova | 14:37 |
| sdague | yeh, let me find a conductor fail | 14:37 |
| sdague | so we can look in more detail | 14:37 |
| dansmith | well, my point is, | 14:37 |
| *** anshul has joined #openstack-qa | 14:37 | |
| dansmith | conductor using the process worker model isn't calling the term handler(s) now I don't think | 14:37 |
| dansmith | and this won't change it | 14:37 |
| sdague | ok | 14:37 |
| dansmith | conductor hasn't had long-running processes for long, so we probably just made an assumption that we were good, when we're not | 14:38 |
| bknudson | is there a test for -- conductor is busy and got a shutdown so it waits? | 14:38 |
| dansmith | bknudson: no | 14:38 |
| bknudson | that would not be an easy test to write | 14:38 |
| dansmith | yeah | 14:38 |
| dansmith | the actual mechanics of what happens is really wrapped up in the messaging code, etc | 14:39 |
| dansmith | which is why it has regressed in compute a few times | 14:39 |
| *** jhenner has joined #openstack-qa | 14:39 | |
| *** zz_dimtruck is now known as dimtruck | 14:40 | |
| nithyag_ | afazekas Could you please have a look at the new patch https://review.openstack.org/#/c/149229 | 14:40 |
| *** ajmiller has joined #openstack-qa | 14:40 | |
| nithyag_ | afazekas I hope this patch addresses your comments. Please let me know if you see any gap | 14:41 |
| nithyag_ | afazekas leaving now, will work on your suggestions if any on the new patch tomorrow. thanks | 14:44 |
| *** achanda has quit IRC | 14:44 | |
| *** jhenner has quit IRC | 14:46 | |
| *** SridharGaddam has joined #openstack-qa | 14:48 | |
| *** ChuckC has joined #openstack-qa | 14:48 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of StringIO to use it from six https://review.openstack.org/176783 | 14:49 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of urlparse to import from six https://review.openstack.org/176784 | 14:49 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of ConfigParser to import from six https://review.openstack.org/176785 | 14:49 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of httplib to import from six https://review.openstack.org/176786 | 14:49 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Fix raise syntax in test.py for python3 compat https://review.openstack.org/176787 | 14:49 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of urllib and HTMLParser to import from six https://review.openstack.org/176788 | 14:49 |
| *** nithyag_ has quit IRC | 14:50 | |
| *** Madan has quit IRC | 14:50 | |
| *** nithyag_ has joined #openstack-qa | 14:50 | |
| *** SridharGaddam has quit IRC | 14:51 | |
| *** SridharGaddam has joined #openstack-qa | 14:53 | |
| *** gamado has joined #openstack-qa | 14:54 | |
| *** jhenner has joined #openstack-qa | 14:55 | |
| sdague | dansmith: oh... mf | 14:55 |
| sdague | http://logs.openstack.org/23/176723/2/check/check-grenade-dsvm-neutron/1011118/logs/grenade.sh.txt.gz | 14:56 |
| sdague | so the entire socket dump is there | 14:56 |
| sdague | I think it's ceilometer-collector | 14:56 |
| sdague | that's holding open things | 14:56 |
| dansmith | to nova-api? | 14:57 |
| sdague | no, never mind, I think I'm wrong | 14:57 |
| mtreinish | sdague: I did just see a random commit fly by somewhere which said a new requests thing is hanging things | 14:57 |
| mtreinish | dug it up: https://review.openstack.org/#/c/176771/ | 14:58 |
| mtreinish | doubt that's really anything though | 14:59 |
| sdague | yeh, unrelated | 14:59 |
| *** mlavalle has joined #openstack-qa | 14:59 | |
| *** sabeen1 has joined #openstack-qa | 15:00 | |
| *** pcrews has joined #openstack-qa | 15:02 | |
| *** tkammer|not_here has quit IRC | 15:03 | |
| *** ihrachyshka has quit IRC | 15:03 | |
| *** SridharGaddam has quit IRC | 15:04 | |
| *** sabeen2 has joined #openstack-qa | 15:04 | |
| *** marun has joined #openstack-qa | 15:04 | |
| *** SridharGaddam has joined #openstack-qa | 15:05 | |
| *** sabeen1 has quit IRC | 15:07 | |
| *** apevec has quit IRC | 15:08 | |
| *** rdekel has joined #openstack-qa | 15:11 | |
| *** jlanoux has quit IRC | 15:16 | |
| *** tkammer|not_here has joined #openstack-qa | 15:17 | |
| *** dboik has quit IRC | 15:18 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: add liberty to feature grid https://review.openstack.org/175493 | 15:21 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: only force off KEYSTONE_USE_WSGI up until kilo https://review.openstack.org/175531 | 15:21 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: make the ss debug include all services https://review.openstack.org/176748 | 15:22 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of urllib and HTMLParser to import from six https://review.openstack.org/176788 | 15:23 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Fix comprehension syntax error with python3 https://review.openstack.org/176814 | 15:23 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of iteritems to use six instead https://review.openstack.org/176815 | 15:23 |
| mtreinish | man I really hate python3... | 15:23 |
| mtreinish | after all those patches you can only do a testr list-tests and run the unit tests with 30 failures | 15:24 |
| sdague | it's not python3, it's the 2 versions of python at once | 15:24 |
| mtreinish | yeah, that's a fair point | 15:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/176820 | 15:27 |
| *** dboik has joined #openstack-qa | 15:29 | |
| *** david-lyle has joined #openstack-qa | 15:29 | |
| *** armax has joined #openstack-qa | 15:30 | |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 15:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tempest: Updated from global requirements https://review.openstack.org/176851 | 15:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tempest-lib: Updated from global requirements https://review.openstack.org/176852 | 15:36 |
| *** davideagnello has joined #openstack-qa | 15:41 | |
| *** bkopilov has joined #openstack-qa | 15:46 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch use of hashing strings to unicode in accounts https://review.openstack.org/176862 | 15:46 |
| *** tsekiyama has joined #openstack-qa | 15:52 | |
| *** shashankhegde has quit IRC | 15:53 | |
| dansmith | sdague: bknudson: I can't repro locally | 15:53 |
| dansmith | if I establish a connection to nova-api, and then "pkill nova-api" I see all the shutdown messages as expected, but everything dies and the connection terminates | 15:53 |
| dansmith | no nova-api processes left afterwards | 15:53 |
| bknudson | is that what grenade does? pkill? | 15:54 |
| *** jlanoux has joined #openstack-qa | 15:55 | |
| sdague | dansmith: pkill -g process leader pid | 15:55 |
| *** dboik has quit IRC | 15:56 | |
| *** david-lyle has quit IRC | 15:56 | |
| dansmith | ah, there we go | 15:57 |
| dansmith | leaves me with one child and the parent | 15:57 |
| bknudson | `pkill -g 4384` killed it, where bknudson 4384 28633 11 10:56 pts/10 00:00:02 /usr/bin/python /usr/local/bin/nova-api | 15:57 |
| dansmith | sdague: one more of those commands cleans it up, no need to disconnect | 15:57 |
| *** mlavalle has quit IRC | 15:58 | |
| dansmith | let me try again to confirm | 15:58 |
| sdague | dansmith: ok, cool | 15:58 |
| bknudson | if you send another signal while it's waiting it shuts down. | 15:58 |
| bknudson | the oslo-incubator code disables its signal handler after the first signal | 15:58 |
| dansmith | hmm, now I can't get it to happen :) | 15:59 |
| sdague | bknudson: so we could work around the tests with a double kill? | 15:59 |
| sdague | dansmith: it's not all the time | 15:59 |
| bknudson | sdague: yes, that's another workaround | 15:59 |
| *** jlanoux is now known as jlanoux_away | 15:59 | |
| bknudson | keep signalling it | 15:59 |
| sdague | and I don't know why we get stretches where it happens a lot | 15:59 |
| *** jlanoux_away is now known as jlanoux | 15:59 | |
| bknudson | I didn't think that was a fix though... I didn't want keystone to not shut down just because a client was connected | 15:59 |
| *** amotoki_ has joined #openstack-qa | 15:59 | |
| bknudson | someone might consider that a DoS | 15:59 |
| dansmith | sdague: yeah, but like, worked first time, and hasn't ten in a row :) | 16:00 |
| *** dboik has joined #openstack-qa | 16:00 | |
| sdague | bknudson: sure, I agree, having the fix is better | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of urllib and HTMLParser to import from six https://review.openstack.org/176788 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of ConfigParser to import from six https://review.openstack.org/176785 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of urlparse to import from six https://review.openstack.org/176784 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Fix raise syntax in test.py for python3 compat https://review.openstack.org/176787 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of httplib to import from six https://review.openstack.org/176786 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Fix comprehension syntax error with python3 https://review.openstack.org/176814 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of iteritems to use six instead https://review.openstack.org/176815 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch all uses of StringIO to use it from six https://review.openstack.org/176783 | 16:00 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Switch use of hashing strings to unicode in accounts https://review.openstack.org/176862 | 16:01 |
| dansmith | sdague: seriously, I wonder if it's something to do with the first time it's started from either devstack or screen or whatever | 16:01 |
| dansmith | sdague: let me restack and try again | 16:02 |
| bknudson | it's very easily reproducible for me... I always just use the devstack screen & restart it there. | 16:02 |
| dansmith | bknudson: yeah, that's what I'm doing | 16:03 |
| bknudson | so, just wanted to mention that there weren't times where it sometimes worked and sometimes didn't. | 16:03 |
| bknudson | (for me) | 16:03 |
| dansmith | yeah, that's what has me very curious about root cause | 16:03 |
| *** ihrachyshka has joined #openstack-qa | 16:04 | |
| sdague | dansmith: honestly, the speed of the machine might matter | 16:05 |
| dansmith | sdague: mine is very fast | 16:05 |
| dansmith | but | 16:05 |
| dansmith | I hit it the FIRST time I tried | 16:05 |
| dansmith | but we'll see in a minute if that's related | 16:05 |
| dansmith | but, a second kill seemed to fix it, which is excellent if so, I just want to see it again | 16:06 |
| dansmith | ...like the money I sent to the IRS | 16:06 |
| dansmith | damn, I could have earned respect from mriedem if the joke had been "like that money I sent to that TV preacher" | 16:07 |
| *** hemnafk is now known as hemna | 16:07 | |
| mriedem | i'm lost | 16:07 |
| mriedem | the baker's put your money to good use | 16:08 |
| mriedem | you don't need it back, you were paid in karma | 16:08 |
| dansmith | mriedem: https://www.phactual.com/16-christmas-vacation-quotes-we-all-know-and-love/ | 16:09 |
| mriedem | dansmith: i know the summer vacation quotes better | 16:09 |
| mriedem | nothing to be proud of russ....50 yards... | 16:10 |
| dansmith | sdague: happened again on the first run from devstack | 16:10 |
| mriedem | https://www.youtube.com/watch?v=c1wcnvDpdVU | 16:10 |
| dansmith | sdague: second kill took it out | 16:10 |
| sdague | ok good to know | 16:10 |
| dansmith | sdague: second attempt at restarting in the screen doesn't hit it | 16:10 |
| dansmith | weird. | 16:10 |
| *** jlanoux has quit IRC | 16:10 | |
| *** pkoniszewski has quit IRC | 16:11 | |
| openstackgerrit | Dan Smith proposed openstack-dev/devstack: Remove the lvm.conf filter during cleanup https://review.openstack.org/176889 | 16:14 |
| dansmith | sdague: that's why my clean.sh wasn't working ^ | 16:14 |
| dansmith | sdague: because of switching between ceph and lvm I think | 16:14 |
| sdague | oh, yeh, good catch | 16:14 |
| sdague | the lvm filter isn't idempotent in it's add either, right? | 16:15 |
| dansmith | I don't think it is, because I had a few in there | 16:16 |
| openstackgerrit | Roman Podoliaka proposed openstack/tempest: Add a test case for rebuild of instances with volumes https://review.openstack.org/175949 | 16:16 |
| dansmith | so we could run this clean at the top of that function if you want | 16:16 |
| sdague | any chance you want to throw a 2nd patch up to fix that | 16:16 |
| sdague | no, I think the clean fix is fine | 16:16 |
| sdague | it would also be nice to make the add idempotent | 16:16 |
| dansmith | I mean, we run clean before we do the filter | 16:16 |
| dansmith | which just nukes the line if it's there, before we try to regenerate/add it | 16:16 |
| *** aix has quit IRC | 16:17 | |
| sdague | oh, hmmm... it seems safer to not constantly delete/add, but that would be a thing we could do | 16:17 |
| sdague | yeh, either way | 16:17 |
| dansmith | well, | 16:17 |
| dansmith | we have to make sure we could match anything anyway, so this seems easier | 16:18 |
| sdague | yeh, ok, that would work for me | 16:18 |
| openstackgerrit | Dan Smith proposed openstack-dev/devstack: Remove the lvm.conf filter during cleanup https://review.openstack.org/176889 | 16:18 |
| dansmith | that ^ | 16:18 |
| dansmith | the problem for me was that it was different based on the ceph or lvm config, | 16:19 |
| dansmith | so just nuking any previous devstack lines avoids us guessing what it might look like | 16:19 |
| *** tkammer|not_here has quit IRC | 16:19 | |
| *** jhenner has quit IRC | 16:21 | |
| openstackgerrit | Sridhar Gaddam proposed openstack-dev/devstack: Devstack to support an IPv6 infra installation https://review.openstack.org/176898 | 16:22 |
| *** weshay has quit IRC | 16:23 | |
| dansmith | sdague: so should I put this service kill thing in the release notes under known issues with the workaround? | 16:23 |
| *** ajo has quit IRC | 16:24 | |
| sdague | dansmith: so I think we're landing this for an rc3 because of the currently blocking neutron thing | 16:24 |
| dansmith | sdague: hmm, okay, where is that being discussed? | 16:24 |
| *** SridharG has left #openstack-qa | 16:25 | |
| sdague | #openstack-relmgr-office | 16:25 |
| sdague | that's where ttx is making calls on such things | 16:25 |
| dansmith | ah, okay | 16:25 |
| *** mlavalle has joined #openstack-qa | 16:26 | |
| sdague | here is the relevant conversation paste - http://paste.openstack.org/show/205334/ | 16:26 |
| *** Triveni has joined #openstack-qa | 16:26 | |
| dansmith | I'm in there, so I have scrollback | 16:27 |
| dansmith | I just didn't realize | 16:27 |
| sdague | ok, sorry, yeh, there has been a lot of channel hopping | 16:27 |
| dansmith | sdague: so just to be clear, we *could* put a simple hack of "kill; sleep 1; kill" into our code to unblock this, yes? | 16:27 |
| sdague | that sounds possible | 16:28 |
| dansmith | if it were my call, which it isn't, I'd not cause another rc3 for nova for this just because our scripts hit it | 16:28 |
| sdague | now that you confirmed that it's always working on second kill | 16:28 |
| sdague | feel free to counter propose to ttx about it | 16:29 |
| *** patrickeast has joined #openstack-qa | 16:31 | |
| *** mwagner_lap has quit IRC | 16:31 | |
| dansmith | sdague: where is the kill happening? I don't see pkill in all of grenade | 16:32 |
| sdague | devstack | 16:32 |
| sdague | we call into devstack functions for most of the work | 16:32 |
| dansmith | okay, so grenade calls unstack? | 16:32 |
| sdague | no, it calls devstack functions | 16:32 |
| sdague | projects/60_nova/shutdown.sh | 16:32 |
| sdague | stop_nova_rest | 16:33 |
| dansmith | gotcha | 16:33 |
| sdague | the shutdown functions source BASE_DEVSTACK... | 16:33 |
| sdague | that eventually gets you into functions-common:stop_process | 16:33 |
| dansmith | yeah | 16:33 |
| *** Swami has joined #openstack-qa | 16:34 | |
| sdague | https://github.com/openstack-dev/devstack/blob/master/functions-common#L1346 | 16:34 |
| dansmith | sdague: so we could do this either in devstack or grenade I think | 16:34 |
| sdague | what's your grenade proposed fix? | 16:35 |
| dansmith | sdague: devstack would make it possible to just hit nova-api, but just calling stop_nova_rest twice in grenade would be more likely to just work I think | 16:35 |
| sdague | so stop_nova_rest twice isn't going to work | 16:35 |
| sdague | because we rm the pid file | 16:35 |
| sdague | so that we don't have stale pids | 16:36 |
| *** e0ne has quit IRC | 16:36 | |
| dansmith | ah | 16:36 |
| sdague | https://github.com/openstack-dev/devstack/blob/master/functions-common#L1347 | 16:36 |
| sdague | that's what I was starting to write... then discovered that | 16:36 |
| *** zz_jgrimm is now known as jgrimm | 16:36 | |
| dansmith | sdague: so the other grenade-only change could be just really hacky after stop_nova_rest: pkill nova-api | 16:37 |
| sdague | we could capture the pid files before the kill so we could rekill. | 16:37 |
| sdague | dansmith: yeh, we'll still have the conductor racing every once in a while | 16:37 |
| dansmith | well, we could do the same for conductor | 16:38 |
| *** Triveni has quit IRC | 16:38 | |
| *** luqas has quit IRC | 16:38 | |
| *** Triveni has joined #openstack-qa | 16:38 | |
| dansmith | I guess you want to avoid the obligatory double-kill of everything in devstack | 16:38 |
| sdague | we could do the double kill blindly | 16:39 |
| sdague | honestly, if we need to work around in test code, that would be my thought on the way to do | 16:39 |
| sdague | scheduler fails some times as well - http://logs.openstack.org/41/176041/2/check/check-grenade-dsvm/763fe2c/logs/grenade.sh.txt.gz | 16:39 |
| dansmith | hmm | 16:39 |
| dansmith | even more suspect :/ | 16:39 |
| sdague | I think this is a race we've had for a long time | 16:40 |
| sdague | the first oslo-incubator change also removed an eventlet sleep, which might make it more likely to expose | 16:40 |
| sdague | dansmith: you want to propose the double kill into stable/kilo devstack? | 16:42 |
| openstackgerrit | Dan Smith proposed openstack-dev/devstack: Workaround potential failure to shutdown services https://review.openstack.org/176909 | 16:42 |
| dansmith | sdague: ^ | 16:42 |
| dansmith | that what you had in mind? | 16:42 |
| sdague | yeh, but stable/kilo only | 16:42 |
| dansmith | okay | 16:42 |
| dansmith | probably want a comment above it too | 16:43 |
| sdague | and probably should have a comment referencing the bug | 16:43 |
| sdague | yep | 16:43 |
| *** ajmiller_ has joined #openstack-qa | 16:43 | |
| sdague | hopefully we can remove this after backports | 16:43 |
| sdague | after release | 16:43 |
| sdague | once you get that up, I'll adjust the grenade stack to be on top of it | 16:44 |
| dansmith | /review.openstack.org/176910 | 16:44 |
| dansmith | https://review.openstack.org/176910 | 16:44 |
| *** cody-somerville has quit IRC | 16:45 | |
| dansmith | I'm going to use this bug to justify a third cup of coffee, just FYI | 16:46 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: allow DEVSTACK_DIRS to be overridden https://review.openstack.org/176700 | 16:46 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: add convenience tools https://review.openstack.org/174111 | 16:46 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: make the ss debug include all services https://review.openstack.org/176748 | 16:46 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 16:46 |
| sdague | dansmith: will you put wisky in it? | 16:46 |
| dansmith | no, waaay too early for that | 16:46 |
| dansmith | and also, I don't like whisky | 16:47 |
| *** ajmiller has quit IRC | 16:47 | |
| *** shardy is now known as shardy_afk | 16:48 | |
| sdague | :) | 16:48 |
| sdague | mtreinish: ... I'm seeing a lot more failures on test_volume_boot_pattern in master | 16:48 |
| sdague | is that a thing? | 16:48 |
| dansmith | sdague: failures with lvm? | 16:49 |
| sdague | dansmith: yep | 16:49 |
| *** dboik has quit IRC | 16:51 | |
| sdague | also, seriously, wth keystone - http://logs.openstack.org/12/176012/2/gate/gate-tempest-dsvm-full/4f6c11f/logs/screen-n-api.txt.gz?level=TRACE#_2015-04-23_13_41_17_500 | 16:51 |
| *** apevec has joined #openstack-qa | 16:51 | |
| *** apevec has joined #openstack-qa | 16:51 | |
| mtreinish | sdague: is there a pattern? There're have been a few generic non-descript ssh failures on that test | 16:52 |
| mtreinish | but I haven't seen a major spike | 16:52 |
| *** mpaolino has joined #openstack-qa | 16:52 | |
| sdague | mtreinish: it's killed a few of my patches today | 16:52 |
| *** jordanP has quit IRC | 16:52 | |
| sdague | I'm looking deeper | 16:52 |
| sdague | though I need to stop for lunch soon | 16:52 |
| *** miqui has joined #openstack-qa | 16:53 | |
| sdague | https://review.openstack.org/#/c/176012/ which is a noop patch effectively got whacked by it | 16:53 |
| mtreinish | sdague: yeah that was an ssh timeout, mriedem has hit a couple of those too | 16:54 |
| *** dustins_ has joined #openstack-qa | 16:56 | |
| *** dboik has joined #openstack-qa | 16:57 | |
| openstackgerrit | Jens Rosenboom proposed openstack-dev/devstack: DO NOT MERGE - Update default cirros version https://review.openstack.org/166778 | 16:58 |
| *** safchain has quit IRC | 16:59 | |
| *** dustins has quit IRC | 16:59 | |
| *** nfedotov has joined #openstack-qa | 16:59 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:01 | |
| mtreinish | sdague: so I used my fancy new subunit2sql tool for failure analysis and got: 99.1123% and 0.1024% over the past week for boot volume v2 | 17:01 |
| mtreinish | with this nice graph: http://imgur.com/Bx8DYMp | 17:02 |
| mtreinish | where green is daily success count, red is skips and blue are failures | 17:02 |
| mtreinish | mriedem: ^^^ | 17:02 |
| mtreinish | obviously the graphing side still needs some work :) | 17:02 |
| *** cbader has quit IRC | 17:03 | |
| *** cbader has joined #openstack-qa | 17:03 | |
| mriedem | heh | 17:03 |
| mtreinish | also only gate | 17:04 |
| *** suneelb has joined #openstack-qa | 17:05 | |
| *** rushiagr_away is now known as rushiagr | 17:07 | |
| *** Longgeek_ has quit IRC | 17:07 | |
| *** yamamoto has quit IRC | 17:08 | |
| *** harlowja_away is now known as harlowja | 17:13 | |
| *** marcusvrn has quit IRC | 17:20 | |
| *** marcusvrn has joined #openstack-qa | 17:20 | |
| *** achanda has joined #openstack-qa | 17:21 | |
| jogo | mriedem: we have a lot of failures http://status.openstack.org//elastic-recheck/data/uncategorized.html | 17:21 |
| *** Triveni has quit IRC | 17:21 | |
| jogo | mriedem: based on http://jogo.github.io/gate/ it looks like it was fixed | 17:22 |
| jogo | is there a bug opened for this? so I can add a e-r bug | 17:22 |
| mriedem | jogo: for this? http://logs.openstack.org/20/174320/3/gate/gate-tempest-dsvm-neutron-full/6ca2c9a/console.html#_2015-04-23_10_11_17_002 | 17:26 |
| *** marcusvrn1 has joined #openstack-qa | 17:26 | |
| jogo | yeah | 17:27 |
| *** marcusvrn has quit IRC | 17:27 | |
| jogo | or was it http://logs.openstack.org/49/85249/18/check/check-tempest-dsvm-full/77170c5/logs/devstacklog.txt.gz#_2015-04-23_01_38_39_158 ? | 17:28 |
| *** ihrachyshka has quit IRC | 17:29 | |
| jogo | ahh https://bugs.launchpad.net/devstack/+bug/1447476 | 17:30 |
| openstack | Launchpad bug 1447476 in OpenStack Dashboard (Horizon) "No module named pathlib" [Critical,Fix released] | 17:30 |
| *** marun has quit IRC | 17:34 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/elastic-recheck: Add fingerprint for bug 1447476 https://review.openstack.org/176919 | 17:35 |
| openstack | bug 1447476 in OpenStack Dashboard (Horizon) "No module named pathlib" [Critical,Fix released] https://launchpad.net/bugs/1447476 | 17:35 |
| jogo | mriedem: ^ | 17:35 |
| *** patrickeast_ has joined #openstack-qa | 17:36 | |
| *** patrickeast has quit IRC | 17:37 | |
| *** patrickeast_ is now known as patrickeast | 17:37 | |
| mriedem | jogo: k sec | 17:37 |
| *** dboik has quit IRC | 17:38 | |
| *** dboik has joined #openstack-qa | 17:39 | |
| *** gamado has quit IRC | 17:41 | |
| *** sileht has quit IRC | 17:44 | |
| *** shashankhegde has joined #openstack-qa | 17:45 | |
| openstackgerrit | Merged openstack-dev/devstack: Fix issue with ml2 plugin on using provider network https://review.openstack.org/174731 | 17:45 |
| *** e0ne has joined #openstack-qa | 17:53 | |
| *** sileht has joined #openstack-qa | 17:55 | |
| *** e0ne is now known as e0ne_ | 18:01 | |
| openstackgerrit | Sean M. Collins proposed openstack-dev/devstack: [WIP] Test VXLAN tunnel type with Linux Bridge https://review.openstack.org/176927 | 18:02 |
| *** e0ne_ has quit IRC | 18:06 | |
| *** david-lyle has joined #openstack-qa | 18:07 | |
| *** tosky has quit IRC | 18:08 | |
| *** tosky_ has joined #openstack-qa | 18:08 | |
| *** tosky_ has quit IRC | 18:09 | |
| *** melwitt has joined #openstack-qa | 18:11 | |
| *** e0ne has joined #openstack-qa | 18:13 | |
| *** dboik_ has joined #openstack-qa | 18:15 | |
| *** dboik has quit IRC | 18:18 | |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: allow DEVSTACK_DIRS to be overridden https://review.openstack.org/176700 | 18:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: add convenience tools https://review.openstack.org/174111 | 18:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: make the ss debug include all services https://review.openstack.org/176748 | 18:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 18:19 |
| dansmith | sdague: what happened? | 18:20 |
| sdague | so... the whole stack was going to be blocked on the master patch you pushed | 18:20 |
| sdague | which we didn't want to land | 18:20 |
| dansmith | why? | 18:21 |
| sdague | so I had to push your stable/kilo with a different idempotent id, so this would only block on that | 18:21 |
| sdague | why what? | 18:21 |
| dansmith | oh, you were using depends-on somewhere? | 18:21 |
| sdague | yes | 18:21 |
| dansmith | I gotcha | 18:21 |
| sdague | so that we could test it | 18:21 |
| dansmith | the why was why blocked | 18:21 |
| sdague | yeh | 18:21 |
| sdague | so even if I abandoned the master patch, the depends-on blocks on abandoned things | 18:21 |
| dansmith | I didn't -x the cherry pick, but didn't think it'd matter about the change id | 18:22 |
| dansmith | yeah | 18:22 |
| dansmith | got it now | 18:22 |
| sdague | I only just figured that out, now things should be good to go | 18:22 |
| dansmith | okay | 18:22 |
| sdague | but still have to wait for test results | 18:22 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Fix tenant isolation and unit tests with py3 https://review.openstack.org/176934 | 18:23 |
| dansmith | sdague: so this needs abandoning as well? https://review.openstack.org/#/c/176910/ | 18:23 |
| *** mpaolino has quit IRC | 18:23 | |
| sdague | yeh | 18:23 |
| sdague | I was going to make sure to explain to you what was going on before I did that | 18:24 |
| *** morganfainberg is now known as grebniafnagrom | 18:24 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add fingerprint for bug 1447476 https://review.openstack.org/176919 | 18:24 |
| openstack | bug 1447476 in OpenStack Dashboard (Horizon) "No module named pathlib" [Critical,Fix released] https://launchpad.net/bugs/1447476 | 18:24 |
| dansmith | heh, it's cool | 18:24 |
| *** grebniafnagrom is now known as morganfainberg | 18:24 | |
| *** cdent has quit IRC | 18:27 | |
| *** jlk has joined #openstack-qa | 18:27 | |
| jlk | hogepodge: ping; Did you author https://etherpad.openstack.org/p/havana-capabilities-test-procedure ? | 18:28 |
| *** e0ne is now known as e0ne_ | 18:28 | |
| hogepodge | jlk yes | 18:28 |
| jlk | quick question then, line 60, should that be --load-list rather than --load-tests ? | 18:29 |
| hogepodge | jlk probably | 18:29 |
| *** e0ne_ is now known as e0ne | 18:29 | |
| hogepodge | jlk yes, it's in error | 18:30 |
| jlk | alright cool. Just making sure I'm doing it right :) | 18:30 |
| hogepodge | jlk feel free to fix it in the etherpad. more official docs (which are also in error) are in the defcore repository now | 18:31 |
| jlk | gotcha | 18:31 |
| jlk | want a PR for those? | 18:31 |
| hogepodge | Sure, always happy for that. | 18:31 |
| hogepodge | This is the file in question. https://github.com/openstack/defcore/blob/master/2015.04/procedure.rst | 18:32 |
| hogepodge | jlk: I can do it too. Don't want to be the "submit a pr" guy | 18:32 |
| jlk | heh, s'ok | 18:32 |
| jlk | change up in Gerrit | 18:34 |
| *** ArxCruz_ has joined #openstack-qa | 18:35 | |
| *** openstackrecheck has quit IRC | 18:36 | |
| *** openstackrecheck has joined #openstack-qa | 18:36 | |
| *** ArxCruz_ has quit IRC | 18:36 | |
| *** e0ne is now known as e0ne_ | 18:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Make multinode first class with ansible https://review.openstack.org/172614 | 18:43 |
| *** e0ne_ is now known as e0ne | 18:44 | |
| *** gfidente has quit IRC | 18:46 | |
| *** Swami has quit IRC | 18:49 | |
| *** dkranz has quit IRC | 18:50 | |
| *** yamamoto has joined #openstack-qa | 18:57 | |
| *** amotoki_ has quit IRC | 18:57 | |
| *** yamamoto has quit IRC | 19:01 | |
| *** mwagner_lap has joined #openstack-qa | 19:03 | |
| *** dkranz has joined #openstack-qa | 19:06 | |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/176865 failed gate-grenade-dsvm in the gate queue with an unrecognized error | 19:07 |
| *** shashankhegde has quit IRC | 19:08 | |
| mtreinish | sdague: if you get a sec: https://review.openstack.org/#/c/172507/ should be pretty quick | 19:09 |
| *** patrickeast has quit IRC | 19:10 | |
| *** rushiagr is now known as rushiagr_away | 19:14 | |
| dansmith | sdague: are we set -e here? | 19:15 |
| *** marun has joined #openstack-qa | 19:15 | |
| sdague | yes | 19:15 |
| sdague | we try to set -e everywhere | 19:16 |
| dansmith | okay, I didn't realize, sorry | 19:16 |
| sdague | no problem, I just was looking at zuul and saw the failure in progress, so reved it | 19:16 |
| dansmith | cool | 19:16 |
| *** jhenner has joined #openstack-qa | 19:23 | |
| *** dkranz has quit IRC | 19:24 | |
| *** dkranz has joined #openstack-qa | 19:37 | |
| SergeyLukjanov | mtreinish, sdague, I just saw a highlight from meeting - sahara definitely interested in grenade support | 19:39 |
| *** jecarey has quit IRC | 19:42 | |
| *** Swami has joined #openstack-qa | 19:43 | |
| *** nfedotov has quit IRC | 19:47 | |
| mtreinish | SergeyLukjanov: ok, good to know | 19:48 |
| mtreinish | I think we were discussing how we'd use sahara or ironic to highlight how to use the new modular grenade with an out of tree plugin | 19:48 |
| SergeyLukjanov | mtreinish, I think we could try to use it for sahara | 19:51 |
| SergeyLukjanov | mtreinish, I'm unfortunately not really aware of how updated grenade works... | 19:51 |
| mtreinish | SergeyLukjanov: fortunately it's much simpler to understand now | 19:52 |
| mtreinish | SergeyLukjanov: to add a project you just need to tell grenade about one of these dirs: http://git.openstack.org/cgit/openstack-dev/grenade/tree/projects/60_nova | 19:52 |
| mtreinish | and the files there all have a defined interface to work within | 19:53 |
| *** amotoki_ has joined #openstack-qa | 19:57 | |
| *** annashen has joined #openstack-qa | 19:58 | |
| *** mtanino has quit IRC | 19:58 | |
| *** patrickeast has joined #openstack-qa | 19:59 | |
| *** Sukhdev has joined #openstack-qa | 19:59 | |
| *** ajo has joined #openstack-qa | 20:00 | |
| mtreinish | dkranz: if you get a sec my open review list is getting a bit unwieldy. If you get a sec could take a look at: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:accounts_v3,n,z , https://review.openstack.org/174671 , and https://review.openstack.org/#/c/176731/ | 20:03 |
| mtreinish | they should all be pretty quick | 20:03 |
| dkranz | mtreinish: ok, working from home tomorrow so should definitely be able to then if not later today | 20:04 |
| mtreinish | dkranz: cool, thanks a lot | 20:04 |
| *** annashen has quit IRC | 20:04 | |
| mtreinish | I just took a look at my open review page at it was sitting at ~50 so figured I should push to get the simple ones landed at least :) | 20:06 |
| *** annashen has joined #openstack-qa | 20:06 | |
| mtreinish | granted it's my own fault for trying to tackle py3 this morning | 20:06 |
| *** annashen has quit IRC | 20:12 | |
| *** mtanino has joined #openstack-qa | 20:12 | |
| mtreinish | sdague: thanks for the +A, a good chunk of my docs patches should be in the gate now | 20:13 |
| sdague | yep | 20:14 |
| *** tkammer|not_here has joined #openstack-qa | 20:15 | |
| jlk | hrm, I seem to still be running into the problem with https://review.openstack.org/#/c/163549/ in my tempest setup, even though I"m using master | 20:15 |
| jlk | namely I'm getting oslo_config.cfg.RequiredOptError: value required for option: lock_path | 20:16 |
| mtreinish | jlk: is you're venv stale? | 20:16 |
| jlk | I did a pip install -U -r requirements.txt in the .venv | 20:16 |
| mtreinish | jlk: hmm, that patch did change tempest to use the oslo lib for the lockutils instead of a local copy from the incubator | 20:16 |
| mtreinish | it did change the config file semantics a bit, but it should have been backwards compat | 20:17 |
| mtreinish | jlk: which group is your lock_path option in? | 20:18 |
| *** cdent has joined #openstack-qa | 20:18 | |
| jlk | let me find it. | 20:19 |
| jlk | [DEFAULT] | 20:19 |
| jlk | and it's commented out actually | 20:19 |
| jlk | so I have no lock_path set | 20:19 |
| jlk | hah, that would explain it | 20:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: allow DEVSTACK_DIRS to be overridden https://review.openstack.org/176700 | 20:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: add convenience tools https://review.openstack.org/174111 | 20:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: make the ss debug include all services https://review.openstack.org/176748 | 20:19 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 20:19 |
| mtreinish | jlk: heh, yep :) | 20:19 |
| sdague | damn, I missed one of the lib/neutron references | 20:19 |
| mtreinish | jlk: oh, you know what, I could see prior to that patch we weren't enforcing that as a requirement and in certain configs it wasn't used | 20:19 |
| sdague | but it seemed to pass the shutdown part | 20:20 |
| mtreinish | jlk: but since we switched to using the lib it actually enforces the requirement now | 20:20 |
| *** shashankhegde has joined #openstack-qa | 20:20 | |
| sdague | mtreinish: you want to +A https://review.openstack.org/#/c/176932 ? | 20:20 |
| mtreinish | jlk: also fwiw, the new location is in the oslo_concurrency group. It'll probably emit a deprecation warning if you use it in default | 20:21 |
| jlk | mtreinish: what's the lock path used? | 20:21 |
| jlk | my old file doesn't even have that section :/ | 20:21 |
| *** marun has quit IRC | 20:21 | |
| * jlk looks in the new sample config | 20:21 | |
| mtreinish | sdague: done | 20:22 |
| jlk | oh there is no default listed, so... what's a good reasonbale choice for this? I don't really understand what this path will be used for | 20:22 |
| mtreinish | jlk: just something in /tmp is fine | 20:23 |
| SergeyLukjanov | mtreinish, I'll take a look on it tomorrow, it's near to midnight now | 20:23 |
| mtreinish | jlk: it's only used for writing files for external locks | 20:23 |
| mtreinish | jlk: tempest needs it because there are certain tests which can't run at the same time | 20:23 |
| sdague | mtreinish: so, tempest could really overwrite that and give it a default | 20:23 |
| mtreinish | jlk: http://docs.openstack.org/developer/tempest/configuration.html#lock-path | 20:23 |
| sdague | all the security concerns with services are pretty much irrelevant in the tempest case | 20:23 |
| mtreinish | sdague: I don't think there is an api for doing that in oslo_concurrency | 20:24 |
| mtreinish | and dhellmann will yell at me if I start set_default on config opts from a library | 20:24 |
| sdague | you can't just conf.set_override? | 20:24 |
| mtreinish | sdague: according to dhellmann opt names and groups aren't part of the stable api | 20:24 |
| mtreinish | err public api | 20:24 |
| mtreinish | they should hopefully be stable | 20:25 |
| mtreinish | sdague: we've hit this several times already in tempest after moving things to libs | 20:25 |
| mtreinish | sdague: the most recent being: https://review.openstack.org/#/c/173581/ | 20:25 |
| sdague | so, honestly, I think that's a good reason to look at removing oslo pieces in the process. | 20:26 |
| *** jecarey has joined #openstack-qa | 20:26 | |
| *** e0ne is now known as e0ne_ | 20:26 | |
| sdague | because if the answer is "you can't make reasonable application defaults", that's ... really problematic | 20:27 |
| *** e0ne has joined #openstack-qa | 20:27 | |
| *** andreykurilin__ has joined #openstack-qa | 20:28 | |
| mtreinish | sdague: oh, I just checked there is an api available to set the default lock_path | 20:29 |
| sdague | I also don't actually understand why it's problematic to set options like that, there has to be a deprecation cycle in oslo, they can't just remove things | 20:30 |
| sdague | and if you do that on a deprecated option, it should just warn | 20:30 |
| mtreinish | sdague: I've done that many times before, but dhellmann says I shouldn't | 20:31 |
| sdague | right, I don't find that a reasonable position | 20:32 |
| sdague | because you can still deprecate things | 20:33 |
| sdague | you just now have to warn when they are going away | 20:33 |
| sdague | and applications adjust to them | 20:33 |
| *** dboik_ has quit IRC | 20:36 | |
| *** artom has quit IRC | 20:36 | |
| *** apevec has quit IRC | 20:37 | |
| mtreinish | sdague: actually how should I handle the multiprocess aspect of it? I can't just call tempfile like this: http://paste.openstack.org/show/205397/ | 20:38 |
| mtreinish | because that would result in a different lock path for each worker | 20:38 |
| *** marun has joined #openstack-qa | 20:39 | |
| sdague | so... if tempest was actually properly installed, it would be set to /var/lock/tempest/ | 20:39 |
| sdague | barring that I'd force it to $TMP_DIR/tempest | 20:39 |
| sdague | there is a python call to get the system definition of where tmp is, right? | 20:40 |
| mtreinish | sdague: yeah tempfile.tempdir | 20:40 |
| *** dustins_ has quit IRC | 20:40 | |
| *** achanda has quit IRC | 20:41 | |
| sdague | actually I think you want tempfile.gettempdir() | 20:41 |
| *** annashen has joined #openstack-qa | 20:42 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Set a tempest lock_path default to a tmpdir https://review.openstack.org/176975 | 20:42 |
| mtreinish | sdague: ^^^ | 20:42 |
| mtreinish | oh, maybe | 20:42 |
| sdague | in reading the docs, I think you want the function | 20:42 |
| mtreinish | yep just tested it manually, you're right | 20:43 |
| *** jgrimm is now known as zz_jgrimm | 20:43 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Set a tempest lock_path default to a tmpdir https://review.openstack.org/176975 | 20:43 |
| mtreinish | sdague: ^^^ | 20:44 |
| sdague | do you have to make sure it exists? | 20:44 |
| sdague | or will it make it for you? | 20:44 |
| mtreinish | sdague: that's a good question | 20:45 |
| mtreinish | I probably have to create it | 20:45 |
| sdague | you could write a unit test for it | 20:45 |
| sdague | and find out | 20:45 |
| openstackgerrit | Merged openstack/tempest: Decouple Sahara templates test from vanilla plugin https://review.openstack.org/161370 | 20:46 |
| sdague | ok, calling it a night | 20:47 |
| *** mlavalle has quit IRC | 20:47 | |
| *** e0ne has quit IRC | 20:47 | |
| *** samueldmq has quit IRC | 20:51 | |
| *** dboik has joined #openstack-qa | 20:55 | |
| openstackgerrit | Andrea Frittoli proposed openstack/tempest: Add OS_TOP_LEVEL to testr conf https://review.openstack.org/176979 | 20:56 |
| andreaf | mtreinish: let me know if have any concern about this: https://review.openstack.org/176979 | 20:57 |
| andreaf | oomichi, gmann, masayukig: I addressed your comments on https://review.openstack.org/#/c/153681 (I hope) :) | 21:01 |
| mtreinish | andreaf: the lock_path patch doesn't work :) | 21:02 |
| andreaf | mtreinish: oh, too bad :( | 21:02 |
| mtreinish | I need to play with it a bit more, I'll add a unit test on the next rev to show it working | 21:02 |
| andreaf | mtreinish: ok cool | 21:03 |
| mtreinish | andreaf: for the os_top_level thing I think we need to think about how we'll do that in the future | 21:03 |
| mtreinish | I think that interface will work, but when we have a real entry point/cli we'll want to just globally install tempest | 21:04 |
| *** shardy_afk is now known as shardy | 21:05 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 21:05 | |
| andreaf | mtreinish: the problem is that at the moment .testr.conf is lost when tempest is installed - so what I do is install tempest and then pull .testr.conf from source | 21:06 |
| openstackgerrit | Merged openstack/tempest: Start a tempest config guide networking section https://review.openstack.org/172507 | 21:06 |
| openstackgerrit | Merged openstack/tempest: Add configuration guide sections for more required options https://review.openstack.org/174047 | 21:06 |
| openstackgerrit | Merged openstack/tempest: Add config guide section on service feature enabled sections https://review.openstack.org/174086 | 21:07 |
| andreaf | mtreinish: I guess if we want to have a proper CLI end point, the configuration needed for it will become part of the install | 21:07 |
| openstackgerrit | Merged openstack/tempest: Add section to config guide for setting up services https://review.openstack.org/174149 | 21:07 |
| mtreinish | andreaf: yeah, that's what I was thinking | 21:07 |
| mtreinish | but I'm not sure how to do that :) | 21:07 |
| andreaf | mtreinish: neither am I - but that's something we should do in liberty I think | 21:09 |
| mtreinish | andreaf: yep | 21:10 |
| mtreinish | but I guess in the short term adding another OS env var is fine | 21:10 |
| mtreinish | it's not like it hurts anything | 21:10 |
| jlk | Can any of you help me figure out what tempest is doing during setUpClass (tempest.api.compute.servers.test_create_server.ServersTestJSON) ? It's getting a 503 back, but I'm having a hard time finding the matching error on the openstack side | 21:11 |
| jlk | I think it's in the credentials setup | 21:11 |
| jlk | or not actually , etm e post the trace | 21:12 |
| mtreinish | jlk: it should have more details in the tempest log file about the request and response | 21:12 |
| jlk | lsoh where does the log go? | 21:12 |
| jlk | ah, log_file setting | 21:13 |
| mtreinish | jlk: yeah by default it tries stdout, but I think it gets lost because of the test runner. I have a patch up to switch that default value to tempest.log in your cwd | 21:14 |
| *** tkammer|not_here has quit IRC | 21:15 | |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Set a tempest lock_path default to a tmpdir https://review.openstack.org/176975 | 21:16 |
| mtreinish | andreaf, sdague: ^^^ | 21:16 |
| *** ajo has quit IRC | 21:17 | |
| jlk | hah, oops | 21:21 |
| jlk | so log_path isn't enough to get good logging, stupid file is empty | 21:21 |
| *** cdent has quit IRC | 21:22 | |
| *** dboik has quit IRC | 21:22 | |
| mtreinish | hmm, that's odd. normally all I need to do is set log_file and it should start logging as soon as the config file is loaded | 21:22 |
| mtreinish | well if you have the traceback I can take a look at that and see if anything stands out | 21:23 |
| mtreinish | andreaf: heh, I ran unit tests locally but forgot pep8 :) | 21:24 |
| jlk | https://gist.github.com/j2sol/1d8ce608ad4baa397d88 | 21:24 |
| jlk | so it's failing in create_network | 21:24 |
| mtreinish | jlk: yeah, that's all I can tell is that the 503 is being returned by neutron | 21:25 |
| jlk | so if I crank up debug and verbose, would I be able to get the actual details of the call, like what network it was trying to create with which details? | 21:25 |
| *** cbader has quit IRC | 21:26 | |
| andreaf | mtreinish: oh, and if you have a sec to re-review this one: https://review.openstack.org/#/c/153681/ it has been around for 40 patchsets, I really hope it will be enough :D | 21:26 |
| jlk | actually | 21:26 |
| mtreinish | andreaf: heh, sure. But 40 is nothing | 21:26 |
| jlk | why is tempest trying to create a network? | 21:26 |
| jlk | and can I avoid doing that? | 21:27 |
| mtreinish | jlk: if you have tenant isolation enabled and neutron enabled tempest will try to make a network for each of the new tenants it creates | 21:27 |
| mtreinish | so it has an isolated network for testing with that user | 21:27 |
| jlk | hrm. | 21:27 |
| mtreinish | jlk: I just landed some docs around those config options | 21:28 |
| jlk | so maybe I have a bad setting in here for what CIDR to use for this network | 21:28 |
| mtreinish | jlk: but neutron shouldn't 503 if you give it a bad cidr | 21:28 |
| jlk | oh okay | 21:28 |
| mtreinish | or at least I would hope not, that should be a 400 or something | 21:28 |
| *** marcusvrn1 has quit IRC | 21:29 | |
| *** marcusvrn has joined #openstack-qa | 21:29 | |
| mtreinish | jlk: unfortunately it doesn't look like the doc changes have been pushed to the docs site yet | 21:29 |
| mtreinish | jlk: is there anything in the neutron logs? | 21:29 |
| jlk | that's what I'm trying to find | 21:30 |
| jlk | I was hoping for some unique identifier from the tempest side I could use to search | 21:30 |
| mtreinish | jlk: that gets logged in the tempest log file :) | 21:30 |
| mtreinish | all the request ids | 21:30 |
| jlk | 2015-04-23 21:26:55.927 20785 INFO tempest_lib.common.rest_client [req-4ec0f920-43eb-49fd-addd-4e286b36ec2b ] Request (ServersTestJSON:setUpClass): 503 POST https://bbg-staging-01.openstack.blueboxgrid.com:9797/v2.0/networks 0.390s | 21:30 |
| jlk | yeah | 21:30 |
| openstackgerrit | Matthew Treinish proposed openstack/tempest: Set a tempest lock_path default to a tmpdir https://review.openstack.org/176975 | 21:31 |
| andreaf | mtreinish: do you plan on adding a py34 unittest job to tempest? | 21:31 |
| mtreinish | andreaf: yep: https://review.openstack.org/176869 | 21:32 |
| mtreinish | experimental to start | 21:32 |
| jlk | so neutron says: | 21:32 |
| jlk | 2015-04-23 21:23:25.610 18016 TRACE neutron.api.v2.resource NoNetworkAvailable: Unable to create the network. No tenant network is available for allocation. | 21:32 |
| mtreinish | and that returns a 503 | 21:32 |
| mtreinish | mestery: ^^^ you're stuff is weird again | 21:32 |
| * mestery looks | 21:32 | |
| jlk | what does that even mean? This is a pretty empty cluster, what resource isn't available? | 21:33 |
| mtreinish | I have no idea, I've not seen that before | 21:33 |
| mtreinish | jlk: heh, when the ptl is looking ask google: https://ask.openstack.org/en/question/29214/unable-to-create-the-network-no-tenant-network-is-available-for-allocation/ :) | 21:34 |
| jlk | maybe I should just make a tenant and not use isolation | 21:34 |
| mtreinish | not sure that works or not | 21:34 |
| mtreinish | jlk: sure that's allow_tenant_isolation in the auth section | 21:34 |
| mtreinish | but make sure you run serially | 21:35 |
| mtreinish | otherwise the tests will fight with each other in parallel | 21:35 |
| mtreinish | jlk: while I'm waiting for the docs to get uploaded, from the review: http://docs-draft.openstack.org/26/176026/2/check/gate-tempest-docs/7e06adc//doc/build/html/configuration.html#networking | 21:36 |
| jlk | thanks | 21:37 |
| mtreinish | still missing some details, but it was what I started with | 21:37 |
| mtreinish | jlk: the other thing is if network creations aren't working, you might have some issue running the neutron tests | 21:40 |
| jlk | yeah, | 21:40 |
| jlk | I'm going to try and create one via horizon and see what happens | 21:40 |
| *** dkranz has quit IRC | 21:40 | |
| *** achanda has joined #openstack-qa | 21:41 | |
| jlk | oh hrm | 21:41 |
| jlk | this is a vlan setup | 21:41 |
| jlk | so maybe we can't make new vlan networks | 21:41 |
| mtreinish | oh, I could see that | 21:42 |
| jlk | so tenant_isolation always requires a tenant network? | 21:42 |
| *** shashankhegde has quit IRC | 21:42 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/149371 failed gate-tempest-dsvm-neutron-full in the gate queue with an unrecognized error | 21:42 |
| mtreinish | jlk: that's how it's written now | 21:43 |
| jlk | alright I'll go without isolation | 21:43 |
| jlk | and figure out all the accounts I need to make | 21:43 |
| mtreinish | jlk: you could try using an accounts file if you want to run in parallel. That gives you more flexability with setup | 21:43 |
| mtreinish | there's a script up for review right now to automate creating one: https://review.openstack.org/#/c/172605/ | 21:45 |
| *** mpaolino has joined #openstack-qa | 21:45 | |
| jlk | looks like using accounts has interesting limitations on the network stuff | 21:45 |
| mtreinish | although I haven't tried using it yet or reviewed the lastest rev either | 21:45 |
| mtreinish | jlk: well if you use the static config options for providing accounts to tempest, they have to be on a shared network or have only 1 network available | 21:46 |
| jlk | nod | 21:46 |
| mtreinish | using an accounts.yaml file will let you tell tempest which network to use with which account | 21:46 |
| mtreinish | but that code path is only like a week old | 21:48 |
| mtreinish | specifying networks with accounts.yaml that is | 21:48 |
| *** mpaolino has quit IRC | 21:49 | |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: allow DEVSTACK_DIRS to be overridden https://review.openstack.org/176700 | 21:49 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: add convenience tools https://review.openstack.org/174111 | 21:49 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: make the ss debug include all services https://review.openstack.org/176748 | 21:49 |
| openstackgerrit | Sean Dague proposed openstack-dev/grenade: fix neutron grenade due to kilo rename of lib/neutron https://review.openstack.org/176723 | 21:49 |
| mtreinish | jlk: anyway I've got to run for now. I'll probably be back online later tonight at some point. | 21:50 |
| jlk | thanks | 21:51 |
| mtreinish | jlk: if you could file a bug for the tenant isolation w/ vlan provider networks that would be awesome | 21:52 |
| jlk | thanks | 21:53 |
| jlk | er yeah | 21:53 |
| mtreinish | I think we can easily add a config option to disable the network autocreation in tenant isolation | 21:53 |
| jlk | oh that'd be sweet. | 21:53 |
| mtreinish | but I don't want to forget about it :) | 21:53 |
| *** Sukhdev has quit IRC | 21:59 | |
| jlk | mtreinish: https://bugs.launchpad.net/tempest/+bug/1447829 | 22:01 |
| openstack | Launchpad bug 1447829 in tempest "allow_tenant_isolation network creation fails with vlan networks" [Undecided,New] | 22:01 |
| *** Sukhdev has joined #openstack-qa | 22:01 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 22:01 | |
| *** suneelb has quit IRC | 22:01 | |
| *** suneelb has joined #openstack-qa | 22:02 | |
| *** suneelb1 has joined #openstack-qa | 22:02 | |
| *** suneelb1 has quit IRC | 22:02 | |
| *** suneelb1 has joined #openstack-qa | 22:02 | |
| jlk | grr next up, something keeps looking for 'public' as a key in server['addresses'] | 22:03 |
| jlk | something is defaulting to 'public' as a network name | 22:03 |
| *** Kennan has quit IRC | 22:03 | |
| *** Kennan has joined #openstack-qa | 22:03 | |
| *** mriedem is now known as mriedem_away | 22:04 | |
| *** andreykurilin__ has quit IRC | 22:04 | |
| *** suneelb2 has joined #openstack-qa | 22:05 | |
| *** suneelb3 has joined #openstack-qa | 22:06 | |
| *** suneelb2 has quit IRC | 22:06 | |
| *** suneelb1 has quit IRC | 22:06 | |
| *** dboik has joined #openstack-qa | 22:06 | |
| *** suneelb has quit IRC | 22:06 | |
| *** Kennan2 has joined #openstack-qa | 22:07 | |
| *** dboik_ has joined #openstack-qa | 22:08 | |
| *** Kennan has quit IRC | 22:08 | |
| *** openstackstatus has quit IRC | 22:09 | |
| *** dboik has quit IRC | 22:10 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/devstack: Remove the lvm.conf filter during cleanup https://review.openstack.org/176889 | 22:12 |
| *** paul_glass has quit IRC | 22:15 | |
| *** greghaynes has quit IRC | 22:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Fix swapiness sysctl set https://review.openstack.org/177006 | 22:22 |
| *** mlavalle has joined #openstack-qa | 22:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Add -e to gate wrap script https://review.openstack.org/177017 | 22:26 |
| jlk | okay, seems tempest doesn't like A) fixed_network definition, B) provided accounts, so not isolated, and C) needing floating IPs from a shared network to ssh in | 22:33 |
| *** bknudson has quit IRC | 22:33 | |
| *** mlavalle has quit IRC | 22:39 | |
| *** mlavalle has joined #openstack-qa | 22:39 | |
| mtreinish | jlk: thanks for filing the bug | 22:41 |
| mtreinish | jlk: all 3 of those are config options in tempest iirc | 22:41 |
| mtreinish | A should be fixed_network_name in the compute section | 22:41 |
| jlk | yeah | 22:41 |
| jlk | they are | 22:41 |
| mtreinish | I thought hogepodge was actually running with that config in his testing | 22:42 |
| jlk | but it seems that in some test setups, even if the use_floatingip_for_ssh is set to true, the tests still try to find the network_for_ssh network in the server info | 22:42 |
| jlk | which that defaults to 'public' | 22:42 |
| jlk | so it's trying to find a public network | 22:42 |
| jlk | even if it doesn't use it, which cases a backtrace | 22:43 |
| jlk | I think I just managed to fool it by putting the fixed network name value in for network_for_ssh | 22:43 |
| mtreinish | jlk: there is a 'ssh_connect_method' option in the compute group | 22:44 |
| mtreinish | I think you can set that to fixed to stop it from trying to use floating ip | 22:44 |
| jlk | no, I need to use floating IP | 22:44 |
| mtreinish | oh | 22:44 |
| mtreinish | so it's just a case of a stupid default again... | 22:45 |
| jlk | well | 22:45 |
| jlk | it's a case of trying to get that data when it's not needed | 22:45 |
| jlk | le tme see if I can find the code | 22:45 |
| mtreinish | fwiw there is a patch series up to try and make all the ssh options a bit cleaner | 22:45 |
| *** shashankhegde has joined #openstack-qa | 22:45 | |
| jlk | https://github.com/openstack/tempest/blob/master/tempest/common/utils/linux/remote_client.py#L37 | 22:46 |
| jlk | that code block in there doesn't care if use_floatingip_for_ssh is set | 22:47 |
| jlk | so it always tries to find the address for network_for_ssh | 22:47 |
| mtreinish | ah, ok yeah that's a bug | 22:47 |
| mtreinish | also maybe a duplicate option too | 22:47 |
| jlk | I could start a fix for this, but I wouldn't be able to quickly create unit tests around it | 22:48 |
| openstackrecheck | openstack/glance change: https://review.openstack.org/176956 failed gate-tempest-dsvm-full in the gate queue with an unrecognized error | 22:48 |
| *** mvandijk has quit IRC | 22:48 | |
| mtreinish | jlk: well there is a refactor up for a bunch of this here: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/ssh-auth-strategy,n,z | 22:48 |
| mtreinish | I need to prioritize reviews on that series | 22:49 |
| jlk | ah let me see if any of that touches this code | 22:49 |
| jlk | oh yes, yes it does | 22:49 |
| *** mvandijk has joined #openstack-qa | 22:50 | |
| *** erlon has quit IRC | 22:51 | |
| jlk | hrm, maybe it's that RemoteClient is supposed to be called with the IP address string when use_floatingip_for_ssh is set | 22:55 |
| jlk | that makes more sense | 22:55 |
| jlk | and maybe some tests aren't doing that right | 22:55 |
| mtreinish | jlk: do you have run_ssh option enabled? There are some tests which don't use it right | 22:56 |
| jlk | yes | 22:57 |
| mtreinish | we don't gate on it so there are probably bugs with it | 22:57 |
| jlk | well | 22:57 |
| mtreinish | we probably should add that to the help for that option | 22:57 |
| mtreinish | that flag isn't globally respected so some tests will still ssh in | 22:57 |
| mtreinish | just not as many | 22:57 |
| mtreinish | this whole section of tempest is a mess | 22:57 |
| mtreinish | it was all added ad hoc over time and no one really thought about it | 22:58 |
| jlk | okay I think I was overzealous in trying to get ssh working | 22:58 |
| jlk | let me remove that and see where we get | 22:58 |
| jlk | seems like every place that would do run_ssh needs to check for floating_ip stuff and create it if required | 22:59 |
| jlk | or move that code into the RemoteClient | 23:00 |
| *** annashen has quit IRC | 23:00 | |
| *** annashen has joined #openstack-qa | 23:00 | |
| mtreinish | jlk: yeah I'm pretty sure that's what the bp is addressing. It's going to break out all of those options into a separate config section and centrally check them | 23:01 |
| *** suneelb3 has quit IRC | 23:01 | |
| mtreinish | then refactor all the create server calls to use them | 23:01 |
| *** suneelb has joined #openstack-qa | 23:01 | |
| jlk | alright | 23:01 |
| *** suneelb has quit IRC | 23:01 | |
| *** suneelb has joined #openstack-qa | 23:02 | |
| jlk | I've got run-ssh turned off this time, we'll see which ones fail this time around. | 23:02 |
| *** melwitt has quit IRC | 23:02 | |
| *** rbak has quit IRC | 23:03 | |
| *** suneelb1 has joined #openstack-qa | 23:03 | |
| *** annashen has quit IRC | 23:05 | |
| jlk | oooh ooh! I think it's working! | 23:06 |
| *** suneelb has quit IRC | 23:06 | |
| *** hemna is now known as hemnafk | 23:07 | |
| mtreinish | jlk: awesome | 23:07 |
| mtreinish | yeah, I think you were just hitting a broken test when run_ssh is enabled | 23:07 |
| jlk | yeah, really seems like it | 23:08 |
| jlk | I thought I needed ssh for one of these tests to work | 23:08 |
| jlk | Ran 113 tests in 466.223s | 23:09 |
| jlk | FAILED (failures=1) | 23:09 |
| jlk | HUZZAH | 23:09 |
| jlk | that one failure is a policy fix | 23:09 |
| *** mlavalle has quit IRC | 23:09 | |
| mtreinish | awesome | 23:11 |
| hogepodge | jlk ooh, send me your configuration | 23:11 |
| jlk | well, a number of SKIPPED tests, which may mean not passing the defcore assertions. Gotta review those | 23:11 |
| hogepodge | I'd like to see if I can reproduce ssh passing on my own cloud | 23:12 |
| hogepodge | (of course, redact any private stuff like passwords) | 23:12 |
| jlk | hogepodge: I just turned off run_ssh | 23:12 |
| hogepodge | Ah | 23:12 |
| *** ajmiller_ is now known as ajmiller | 23:12 | |
| mtreinish | hogepodge: well run_ssh is kinda in a weird state right now because it's not gated and not globally honored | 23:12 |
| mtreinish | so some tests do ssh, regardless of what that config flag is | 23:12 |
| mtreinish | and most of the ones which listen to it don't work when it's enabled | 23:13 |
| jlk | yeah, particularly when floatingips | 23:13 |
| mtreinish | that bp is trying to fix it, I think the last time they ran a test patch on top of it with run_ssh enabled it worked | 23:13 |
| hogepodge | mtreinish: do you think this problem will be fixed in liberty? If not, we're going to have to pull tests that rely on it from defcore. It's not reasonable to expect tests we can't test. | 23:14 |
| jlk | hogepodge: so if I'm using the 2015.04.required.txt test list, do all of them have to be OK, non SKIP ? | 23:14 |
| mtreinish | hogepodge: yeah it'll definitely be fixed in liberty | 23:14 |
| mtreinish | the reviews are up already | 23:14 |
| hogepodge | jlk everything has to be ok, except for those marked as flagged in the json file. We leave them in there because they're still capabilities we want. It's weird | 23:14 |
| hogepodge | So if tempest fixed them today, it's likely we would move our testing hash to a working version of tempest so we could remove flagged status. | 23:15 |
| jlk | okay looks like just 3 flagged | 23:16 |
| jlk | Ran 113 tests in 299.370s | 23:16 |
| jlk | OK | 23:16 |
| hogepodge | jlk wow, impressive. there's one flagged swift test too | 23:16 |
| hogepodge | (maybe two) | 23:16 |
| hogepodge | for swift it's best to use the accounts.yaml because you need to have roles assigned | 23:17 |
| hogepodge | It's documented in tempest.conf | 23:17 |
| jlk | okay | 23:17 |
| mtreinish | hogepodge: well you can use tenant isolation too :) | 23:17 |
| jlk | I'll be tackling swift next week | 23:17 |
| openstackrecheck | Console logs not available after 13:26s for check-grenade-dsvm-neutron 176012,2,4deca5e | 23:17 |
| mtreinish | hogepodge: have you played with network support in accounts.yaml? I'm curious if that works for you guys | 23:18 |
| hogepodge | mtreinish: tenant isolation assumed admin credentials, though. So yes if you have them. | 23:18 |
| mtreinish | because my test for that code path was kinda synthetic | 23:18 |
| mtreinish | hogepodge: yeah it does. Eventually we want to be able to have domain admin support in a v3 only env | 23:18 |
| mtreinish | but that's probably a long way off | 23:18 |
| hogepodge | mtreinish: not yet. I'm prepping a kilo demo right now, on top of handling incoming test requests. | 23:19 |
| openstackgerrit | Adolfo Duarte proposed openstack/tempest: Tests to cover FIP in dvr multinode setup https://review.openstack.org/177033 | 23:20 |
| mtreinish | hogepodge: ok, no worries I was just curious | 23:23 |
| *** wuhg has joined #openstack-qa | 23:28 | |
| *** sabeen2 has quit IRC | 23:28 | |
| *** marun has quit IRC | 23:34 | |
| *** dimtruck is now known as zz_dimtruck | 23:34 | |
| *** Swami has quit IRC | 23:35 | |
| *** hichihara has joined #openstack-qa | 23:36 | |
| openstackgerrit | Adolfo Duarte proposed openstack/tempest: Tests to cover FIP in dvr multinode setup https://review.openstack.org/177033 | 23:38 |
| *** ajmiller has quit IRC | 23:41 | |
| *** tochi has joined #openstack-qa | 23:42 | |
| jlk | hogepodge: this test, verify_created_server_vcpus relies on run_ssh, which is broken: https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_create_server.py#L98-L107 | 23:48 |
| jlk | hogepodge: but this test isn't flagged. | 23:48 |
| jlk | hogepodge: same with test_attach_detach_volume https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L87-L90 | 23:50 |
| mtreinish | hogepodge, jlk: fwiw I feel like that's a weird test to require. It's mostly testing libvirt (or whatever hypervisor) boots with the right number of cpus. That doesn't feel very interopy (for lack of a better word) | 23:51 |
| jlk | mtreinish: also it seems to rely upon flavor metadata for VCPU count | 23:51 |
| jlk | which isn't a standard thing? | 23:52 |
| jlk | oh wait | 23:52 |
| jlk | derp. sorry I was thinking about image metadata | 23:52 |
| jlk | not flavor, disregard me | 23:52 |
| *** suneelb1 has quit IRC | 23:53 | |
| mtreinish | jlk: there is at least a similar test which doesn't ssh to validate the attachment: https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L62 | 23:54 |
| jlk | nod | 23:54 |
| mtreinish | jlk: also I don't think that test with ssh will be reliable at all, we have a similar test in scenario and it's been skipped for forever, because the device name is rarely what you expect it to be | 23:58 |
| mtreinish | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_stamp_pattern.py | 23:58 |
| jlk | oh man | 23:58 |
| jlk | so when using a static user list | 23:58 |
| jlk | there are some tests that seem to fail upon repeated runs | 23:59 |
| jlk | things such as ssh key creation | 23:59 |
| jlk | and metadata setting | 23:59 |
| mtreinish | jlk: using accounts.yaml or the config options? | 23:59 |
| jlk | accounts.yaml | 23:59 |
| mtreinish | but either way its clearly a bug | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!