| openstackgerrit | Monty Taylor proposed openstack-dev/devstack master: Use the proper keystone endpoints in clouds.yaml https://review.openstack.org/460786 | 00:07 |
|---|---|---|
| openstackgerrit | Monty Taylor proposed openstack-dev/devstack stable/newton: Use the proper keystone endpoints in clouds.yaml https://review.openstack.org/460775 | 00:08 |
| *** ihrachys has joined #openstack-qa | 00:12 | |
| openstackrecheck | Console logs not available after 20:11s for gate-gnocchi-tox-py35-mysql-ubuntu-xenial 458238,4,d4b0d70 | 00:12 |
| *** ihrachys has quit IRC | 00:25 | |
| openstackrecheck | Console logs not available after 20:03s for gate-gnocchi-tox-py27-mysql-ubuntu-xenial 460622,1,ada11f6 | 00:33 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Redirect stderr to stdout for some ansible calls https://review.openstack.org/458689 | 00:33 |
| openstackrecheck | openstack/trove change: https://review.openstack.org/443919 failed gate-trove-scenario-dsvm-mysql-single-ubuntu-xenial in the gate queue with an unrecognized error | 00:33 |
| openstackgerrit | Monty Taylor proposed openstack-dev/devstack master: Use the proper keystone endpoints in clouds.yaml https://review.openstack.org/460786 | 00:34 |
| openstackgerrit | Monty Taylor proposed openstack-dev/devstack stable/newton: Use the proper keystone endpoints in clouds.yaml https://review.openstack.org/460775 | 00:36 |
| openstackgerrit | Monty Taylor proposed openstack-dev/devstack stable/ocata: Use the proper keystone endpoints in clouds.yaml https://review.openstack.org/460788 | 00:37 |
| *** xyang1 has quit IRC | 00:37 | |
| *** xyang1 has joined #openstack-qa | 00:38 | |
| *** salv-orlando has joined #openstack-qa | 00:39 | |
| *** pvaneck has quit IRC | 00:40 | |
| *** markvoelker has joined #openstack-qa | 00:41 | |
| *** rfolco has joined #openstack-qa | 00:41 | |
| *** rfolco has quit IRC | 00:41 | |
| *** salv-orlando has quit IRC | 00:44 | |
| *** markvoelker has quit IRC | 00:46 | |
| openstackrecheck | Console logs not available after 20:03s for gate-nova-python27-ubuntu-xenial 460661,1,40ddc8d | 00:54 |
| *** ihrachys has joined #openstack-qa | 00:55 | |
| *** david_5 has quit IRC | 00:58 | |
| *** sapcc-bot2 has quit IRC | 00:58 | |
| *** carthaca_2 has quit IRC | 00:58 | |
| *** carthaca_ has joined #openstack-qa | 00:59 | |
| *** david_ has joined #openstack-qa | 00:59 | |
| *** sapcc-bot has joined #openstack-qa | 00:59 | |
| *** cloud-liang has joined #openstack-qa | 01:00 | |
| *** dkranz has joined #openstack-qa | 01:00 | |
| *** ihrachys has quit IRC | 01:05 | |
| *** fnaval has joined #openstack-qa | 01:05 | |
| *** fnaval has quit IRC | 01:07 | |
| *** fnaval has joined #openstack-qa | 01:08 | |
| openstackrecheck | Console logs not available after 20:03s for gate-tempest-dsvm-neutron-src-tooz-ubuntu-xenial 460615,1,0106d2f | 01:15 |
| *** Apoorva_ has joined #openstack-qa | 01:16 | |
| *** armax has quit IRC | 01:17 | |
| *** erlon has joined #openstack-qa | 01:19 | |
| *** Apoorva has quit IRC | 01:20 | |
| *** yolkfull has joined #openstack-qa | 01:20 | |
| *** Apoorva_ has quit IRC | 01:21 | |
| *** yangyapeng has joined #openstack-qa | 01:22 | |
| *** spzala has quit IRC | 01:22 | |
| *** spzala has joined #openstack-qa | 01:23 | |
| *** rbak has joined #openstack-qa | 01:26 | |
| *** spzala has quit IRC | 01:27 | |
| openstackgerrit | Ian Wienand proposed openstack-dev/devstack master: uwsgi: don't enable python plugin https://review.openstack.org/458912 | 01:29 |
| *** zz_dimtruck is now known as dimtruck | 01:29 | |
| *** tuanluong has joined #openstack-qa | 01:31 | |
| *** hongbin has joined #openstack-qa | 01:31 | |
| *** yangyapeng has quit IRC | 01:33 | |
| openstackrecheck | Console logs not available after 20:04s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 459110,5,5600e94 | 01:35 |
| *** yamahata_ has quit IRC | 01:35 | |
| *** markvoelker has joined #openstack-qa | 01:38 | |
| *** yangyapeng has joined #openstack-qa | 01:40 | |
| Kevin_Zheng | ping gmann | 01:43 |
| Kevin_Zheng | ping oomichi | 01:43 |
| *** spzala has joined #openstack-qa | 01:44 | |
| *** gouthamr has quit IRC | 01:44 | |
| *** gouthamr has joined #openstack-qa | 01:45 | |
| *** salv-orlando has joined #openstack-qa | 01:45 | |
| *** spzala has quit IRC | 01:49 | |
| *** ramishra has joined #openstack-qa | 01:49 | |
| *** salv-orlando has quit IRC | 01:50 | |
| *** spzala has joined #openstack-qa | 01:50 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/tempest master: wip https://review.openstack.org/460484 | 01:50 |
| *** spzala has quit IRC | 01:55 | |
| openstackrecheck | Console logs not available after 20:04s for gate-octavia-v1-dsvm-scenario-multinode 440934,14,c86010f | 01:55 |
| *** dsariel has joined #openstack-qa | 02:00 | |
| *** Nakato has quit IRC | 02:06 | |
| *** ramishra has quit IRC | 02:07 | |
| *** ramishra has joined #openstack-qa | 02:08 | |
| *** Nakato has joined #openstack-qa | 02:09 | |
| *** rbak has quit IRC | 02:09 | |
| *** spzala has joined #openstack-qa | 02:11 | |
| *** rbak has joined #openstack-qa | 02:12 | |
| openstackrecheck | Console logs not available after 20:04s for gate-networking-ovn-python27-ubuntu-xenial 460693,2,89af0c1 | 02:15 |
| *** spzala has quit IRC | 02:16 | |
| *** dave-mccowan has quit IRC | 02:27 | |
| *** rbak has quit IRC | 02:30 | |
| *** rbak has joined #openstack-qa | 02:31 | |
| *** spzala has joined #openstack-qa | 02:32 | |
| *** Nakato has quit IRC | 02:32 | |
| *** Nakato has joined #openstack-qa | 02:33 | |
| *** jhakimra has quit IRC | 02:34 | |
| *** rbak has quit IRC | 02:35 | |
| openstackrecheck | Console logs not available after 20:04s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 460676,1,fa451b7 | 02:35 |
| *** spzala has quit IRC | 02:36 | |
| *** yolkfull has quit IRC | 02:44 | |
| *** yolkfull has joined #openstack-qa | 02:44 | |
| *** armax has joined #openstack-qa | 02:45 | |
| *** salv-orlando has joined #openstack-qa | 02:46 | |
| *** spzala has joined #openstack-qa | 02:48 | |
| *** salv-orlando has quit IRC | 02:51 | |
| *** yolkfull_ has joined #openstack-qa | 02:51 | |
| *** spzala has quit IRC | 02:53 | |
| *** yolkfull__ has joined #openstack-qa | 02:53 | |
| *** yolkfull has quit IRC | 02:54 | |
| openstackrecheck | Console logs not available after 20:07s for gate-patrole-pep8-ubuntu-xenial 459683,5,e3acbf2 | 02:55 |
| *** yolkfull_ has quit IRC | 02:56 | |
| *** yolkfull has joined #openstack-qa | 02:57 | |
| *** yolkfull__ has quit IRC | 02:59 | |
| *** spzala has joined #openstack-qa | 03:01 | |
| *** yolkfull_ has joined #openstack-qa | 03:09 | |
| *** yolkfull__ has joined #openstack-qa | 03:10 | |
| *** yolkfull has quit IRC | 03:12 | |
| *** hongbin_ has joined #openstack-qa | 03:13 | |
| *** hongbin has quit IRC | 03:13 | |
| *** yolkfull_ has quit IRC | 03:14 | |
| openstackrecheck | Console logs not available after 20:06s for gate-gnocchi-tox-py27-mysql-ubuntu-xenial 460623,2,be3efb9 | 03:15 |
| *** armax has quit IRC | 03:18 | |
| *** david-lyle is now known as dklyle | 03:19 | |
| *** dklyle is now known as david-lyle | 03:20 | |
| *** david-lyle is now known as dklyle | 03:21 | |
| *** dklyle is now known as david-lyle | 03:23 | |
| openstackgerrit | Liang Hao proposed openstack/tempest master: Add test about server attach a volume which is in use https://review.openstack.org/459623 | 03:25 |
| *** yolkfull__ is now known as yolkfull | 03:25 | |
| *** yamahata_ has joined #openstack-qa | 03:35 | |
| openstackgerrit | zhufl proposed openstack/tempest master: Correct skip condition for migrate_server https://review.openstack.org/460816 | 03:35 |
| openstackrecheck | Console logs not available after 20:08s for gate-functional-dsvm-magnum-swarm-ubuntu-xenial 449073,5,cafac10 | 03:36 |
| openstackgerrit | Zhenyu Zheng proposed openstack/tempest master: wip https://review.openstack.org/460484 | 03:38 |
| *** aojea has joined #openstack-qa | 03:38 | |
| *** mattoliverau has quit IRC | 03:42 | |
| *** mattoliverau_ has joined #openstack-qa | 03:44 | |
| *** mattoliverau_ has quit IRC | 03:45 | |
| *** aojea has quit IRC | 03:45 | |
| *** gouthamr has quit IRC | 03:50 | |
| *** gkadam has joined #openstack-qa | 03:51 | |
| *** mriedem has quit IRC | 03:52 | |
| *** gouthamr has joined #openstack-qa | 03:52 | |
| openstackrecheck | Console logs not available after 20:04s for gate-tempest-dsvm-py35-ubuntu-xenial 460515,1,e85b98c | 03:56 |
| *** gouthamr has quit IRC | 03:56 | |
| *** zhufl has quit IRC | 03:59 | |
| *** zhufl has joined #openstack-qa | 04:00 | |
| openstackgerrit | Merged openstack-dev/devstack master: Set OS_AUTH_TYPE to password https://review.openstack.org/437912 | 04:00 |
| *** fnaval has quit IRC | 04:02 | |
| *** yolkfull_ has joined #openstack-qa | 04:07 | |
| *** yolkfull has quit IRC | 04:10 | |
| *** yolkfull__ has joined #openstack-qa | 04:10 | |
| *** yolkfull_ has quit IRC | 04:12 | |
| *** amotoki has joined #openstack-qa | 04:14 | |
| openstackrecheck | Console logs not available after 20:04s for gate-patrole-pep8-ubuntu-xenial 460687,1,0b8f56c | 04:16 |
| *** hongbin_ has quit IRC | 04:29 | |
| openstackrecheck | Console logs not available after 20:04s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 460675,1,76e7cab | 04:36 |
| *** udesale has joined #openstack-qa | 04:41 | |
| *** salv-orlando has joined #openstack-qa | 04:47 | |
| *** armax has joined #openstack-qa | 04:48 | |
| *** gkadam__ has joined #openstack-qa | 04:48 | |
| *** gkadam has quit IRC | 04:49 | |
| *** gkadam__ is now known as regain | 04:49 | |
| *** regain is now known as gkadam | 04:49 | |
| *** gkadam has quit IRC | 04:50 | |
| *** gkadam has joined #openstack-qa | 04:51 | |
| *** salv-orlando has quit IRC | 04:52 | |
| *** itzikb has quit IRC | 04:54 | |
| openstackrecheck | Console logs not available after 20:07s for gate-k8s-cloud-provider-golang-dsvm-conformance-ubuntu-xenial 456561,32,2592386 | 04:56 |
| *** yangyape_ has joined #openstack-qa | 04:58 | |
| *** xyang1 has quit IRC | 05:02 | |
| *** dimtruck is now known as zz_dimtruck | 05:03 | |
| *** zz_dimtruck is now known as dimtruck | 05:03 | |
| *** gabor_antal has quit IRC | 05:05 | |
| *** dimtruck is now known as zz_dimtruck | 05:12 | |
| *** erlon has quit IRC | 05:13 | |
| openstackgerrit | zhufl proposed openstack/tempest master: Create a server in resource_setup in ImagesOneServerTestJSON https://review.openstack.org/460830 | 05:16 |
| openstackrecheck | Console logs not available after 20:03s for gate-group-based-policy-pep8-ubuntu-xenial 460685,1,655da0d | 05:16 |
| *** shausy has joined #openstack-qa | 05:22 | |
| *** prateek has joined #openstack-qa | 05:24 | |
| openstackrecheck | Console logs not available after 20:02s for gate-neutron-python27-ubuntu-xenial 423382,8,f38ed90 | 05:36 |
| *** salv-orlando has joined #openstack-qa | 05:48 | |
| *** salv-orlando has quit IRC | 05:53 | |
| openstackrecheck | Console logs not available after 20:04s for gate-functional-dsvm-magnum-k8s-ubuntu-xenial 455353,7,b68b9d2 | 05:56 |
| *** cloud-liang has quit IRC | 06:04 | |
| *** cloud-liang has joined #openstack-qa | 06:04 | |
| openstackrecheck | Console logs not available after 20:02s for gate-grenade-dsvm-neutron-ubuntu-xenial 460062,6,4e18666 | 06:16 |
| openstackgerrit | zhufl proposed openstack/tempest master: Use min_count to create servers in ListServersNegativeTestJSON https://review.openstack.org/460844 | 06:22 |
| *** cloud-liang has quit IRC | 06:28 | |
| *** cloud-liang has joined #openstack-qa | 06:29 | |
| *** pcaruana has joined #openstack-qa | 06:36 | |
| openstackrecheck | Console logs not available after 20:07s for gate-manila-ui-pep8-ubuntu-xenial 460698,3,aa01277 | 06:36 |
| *** yolkfull_ has joined #openstack-qa | 06:37 | |
| *** gcheresh_ has joined #openstack-qa | 06:38 | |
| *** yolkfull__ has quit IRC | 06:40 | |
| *** yolkfull has joined #openstack-qa | 06:40 | |
| *** yolkfull_ has quit IRC | 06:43 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/tempest master: Fix volume attach tests failing when using FIP as ssh method https://review.openstack.org/460484 | 06:43 |
| openstackgerrit | Merged openstack-dev/devstack stable/ocata: Enable EPEL mirror by default https://review.openstack.org/458940 | 06:43 |
| openstackgerrit | Merged openstack-dev/devstack stable/newton: Enable EPEL mirror by default https://review.openstack.org/458941 | 06:44 |
| *** e0ne has joined #openstack-qa | 06:46 | |
| *** salv-orlando has joined #openstack-qa | 06:49 | |
| *** anshul has joined #openstack-qa | 06:53 | |
| *** salv-orlando has quit IRC | 06:54 | |
| *** shardy has joined #openstack-qa | 06:56 | |
| openstackrecheck | Console logs not available after 20:23s for gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial 444036,12,b203055 | 06:57 |
| *** salv-orlando has joined #openstack-qa | 06:57 | |
| *** nkorabli has joined #openstack-qa | 07:04 | |
| *** tesseract has joined #openstack-qa | 07:05 | |
| openstackgerrit | Merged openstack/coverage2sql master: Update requirements https://review.openstack.org/460432 | 07:09 |
| *** pgadiya has joined #openstack-qa | 07:11 | |
| openstackrecheck | Console logs not available after 20:06s for gate-keystone-pep8-ubuntu-xenial 387161,13,0c8f966 | 07:17 |
| *** cloud-liang has quit IRC | 07:18 | |
| *** aojea has joined #openstack-qa | 07:18 | |
| *** cloud-liang has joined #openstack-qa | 07:19 | |
| *** e0ne has quit IRC | 07:19 | |
| *** Serlex has joined #openstack-qa | 07:23 | |
| *** salv-orlando has quit IRC | 07:23 | |
| *** udesale has quit IRC | 07:30 | |
| *** markus_z has joined #openstack-qa | 07:30 | |
| *** tuanluong has quit IRC | 07:34 | |
| *** sridharg has joined #openstack-qa | 07:35 | |
| openstackrecheck | Console logs not available after 20:05s for gate-networking-ovn-python27-ubuntu-xenial 460688,2,c073bf5 | 07:37 |
| *** yamamoto has quit IRC | 07:40 | |
| *** gszasz has joined #openstack-qa | 07:51 | |
| *** makowals has joined #openstack-qa | 07:54 | |
| *** nkorabli has quit IRC | 07:54 | |
| *** nkorabli has joined #openstack-qa | 07:55 | |
| *** armax has quit IRC | 07:55 | |
| *** gszasz has quit IRC | 07:55 | |
| *** shausy has quit IRC | 07:55 | |
| openstackrecheck | Console logs not available after 20:06s for gate-python-neutronclient-python27-ubuntu-xenial 460263,1,dae0e52 | 07:57 |
| *** gfidente has joined #openstack-qa | 07:58 | |
| *** nkorabli has quit IRC | 07:59 | |
| *** andreaf has quit IRC | 08:02 | |
| openstackgerrit | zhufl proposed openstack/tempest master: Add show_server when validatable=True in create_test_server https://review.openstack.org/438829 | 08:02 |
| *** gszasz has joined #openstack-qa | 08:02 | |
| *** andreaf has joined #openstack-qa | 08:04 | |
| *** aarefiev_afk is now known as aarefiev | 08:07 | |
| *** jdandrea has quit IRC | 08:07 | |
| *** asselin__ has joined #openstack-qa | 08:08 | |
| *** aojea has quit IRC | 08:08 | |
| *** asselin has joined #openstack-qa | 08:10 | |
| *** jdandrea has joined #openstack-qa | 08:10 | |
| *** asselin_ has quit IRC | 08:11 | |
| *** asselin__ has quit IRC | 08:13 | |
| *** dtantsur|afk is now known as dtantsur | 08:17 | |
| openstackrecheck | Console logs not available after 20:04s for gate-grenade-dsvm-neutron-ubuntu-xenial 372374,24,7d2859e | 08:17 |
| *** lucas-afk is now known as lucasagomes | 08:17 | |
| *** nkorabli has joined #openstack-qa | 08:25 | |
| *** shausy has joined #openstack-qa | 08:34 | |
| *** rossella_s has joined #openstack-qa | 08:36 | |
| openstackrecheck | Console logs not available after 20:05s for gate-heat-dsvm-functional-convg-mysql-lbaasv2-non-apache-ubuntu-xenial 458499,2,4eeb0c3 | 08:37 |
| *** shausy has quit IRC | 08:41 | |
| *** makowals has quit IRC | 08:42 | |
| *** tuanluong has joined #openstack-qa | 08:43 | |
| *** yangyape_ is now known as yangyapepeng | 08:47 | |
| *** shausy has joined #openstack-qa | 08:48 | |
| *** shausy has quit IRC | 08:51 | |
| *** e0ne has joined #openstack-qa | 08:52 | |
| *** shausy has joined #openstack-qa | 08:52 | |
| openstackrecheck | Console logs not available after 20:03s for gate-neutron-dsvm-api-ubuntu-xenial 460300,3,7856460 | 08:57 |
| *** aojea has joined #openstack-qa | 08:59 | |
| *** yolkfull_ has joined #openstack-qa | 09:03 | |
| *** shausy has quit IRC | 09:04 | |
| *** yolkfull has quit IRC | 09:06 | |
| *** yamahata_ has quit IRC | 09:08 | |
| *** gszasz has quit IRC | 09:13 | |
| *** cdent has joined #openstack-qa | 09:14 | |
| *** markvoelker has quit IRC | 09:16 | |
| *** gcheresh has joined #openstack-qa | 09:17 | |
| *** gcheresh_ has quit IRC | 09:18 | |
| openstackrecheck | Console logs not available after 20:11s for gate-neutron-python27-ubuntu-xenial 460524,3,9ff12d5 | 09:18 |
| *** odyssey4me_ is now known as odyssey4me | 09:20 | |
| *** tosky has joined #openstack-qa | 09:37 | |
| openstackrecheck | Console logs not available after 20:08s for gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial 458238,4,d702d6e | 09:38 |
| *** wznoinsk has joined #openstack-qa | 09:46 | |
| *** udesale has joined #openstack-qa | 09:58 | |
| openstackrecheck | Console logs not available after 20:10s for gate-keystone-pep8-ubuntu-xenial 387710,15,17a8a2c | 09:58 |
| *** sdague has joined #openstack-qa | 10:00 | |
| *** lyarwood has quit IRC | 10:03 | |
| *** tesseract has quit IRC | 10:04 | |
| *** tesseract has joined #openstack-qa | 10:08 | |
| *** markvoelker has joined #openstack-qa | 10:17 | |
| *** yangyapepeng has quit IRC | 10:18 | |
| openstackrecheck | Console logs not available after 20:03s for gate-murano-pep8-ubuntu-xenial 459772,8,81be5ce | 10:18 |
| *** yangyapepeng has joined #openstack-qa | 10:18 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** yangyapepeng has quit IRC | 10:23 | |
| *** yangyapeng has quit IRC | 10:32 | |
| *** gszasz has joined #openstack-qa | 10:34 | |
| openstackrecheck | Console logs not available after 20:04s for gate-tempest-dsvm-networking-odl-carbon-snapshot-v2driver 460694,2,161e3a4 | 10:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Use compat names for journal files https://review.openstack.org/460947 | 10:44 |
| *** tuanluong has quit IRC | 10:52 | |
| *** BobBall_AWOL is now known as BobBall | 10:54 | |
| *** udesale__ has joined #openstack-qa | 10:54 | |
| *** udesale has quit IRC | 10:56 | |
| *** cdent has quit IRC | 10:58 | |
| openstackrecheck | Console logs not available after 20:09s for gate-rally-dsvm-keystone-v2api-rally 460214,6,20b3505 | 10:58 |
| *** cloud-liang has quit IRC | 11:02 | |
| *** rossella_s has quit IRC | 11:04 | |
| *** rossella_s has joined #openstack-qa | 11:05 | |
| *** yamamoto has joined #openstack-qa | 11:05 | |
| *** yamamoto_ has joined #openstack-qa | 11:06 | |
| *** yolkfull_ has quit IRC | 11:08 | |
| *** makowals has joined #openstack-qa | 11:09 | |
| *** gcheresh has quit IRC | 11:10 | |
| *** yamamoto has quit IRC | 11:10 | |
| *** makowals has quit IRC | 11:10 | |
| *** makowals has joined #openstack-qa | 11:17 | |
| *** yamamoto_ has quit IRC | 11:18 | |
| openstackrecheck | Console logs not available after 20:08s for gate-k8s-cloud-provider-golang-dsvm-conformance-ubuntu-xenial 460504,5,c24895f | 11:18 |
| *** shardy is now known as shardy_lunch | 11:22 | |
| *** cloud-liang has joined #openstack-qa | 11:28 | |
| *** cloud-liang has quit IRC | 11:29 | |
| *** gkadam has quit IRC | 11:29 | |
| *** cdent has joined #openstack-qa | 11:35 | |
| openstackrecheck | Console logs not available after 20:17s for gate-gnocchi-tox-py35-postgresql-ubuntu-xenial 458098,7,b6a821d | 11:39 |
| *** csomerville has joined #openstack-qa | 11:40 | |
| *** cody-somerville has quit IRC | 11:40 | |
| *** Serlex has quit IRC | 11:45 | |
| *** tesseract has quit IRC | 11:50 | |
| *** shardy_lunch is now known as shardy | 11:55 | |
| *** dave-mccowan has joined #openstack-qa | 11:56 | |
| openstackrecheck | Console logs not available after 20:12s for gate-telemetry-dsvm-integration-gnocchi-ubuntu-xenial 453239,15,4890203 | 11:59 |
| *** aojea has quit IRC | 12:01 | |
| *** tesseract has joined #openstack-qa | 12:05 | |
| *** lucasagomes is now known as lucas-hungry | 12:11 | |
| *** yamamoto has joined #openstack-qa | 12:15 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** yamamoto has joined #openstack-qa | 12:18 | |
| openstackrecheck | Console logs not available after 20:04s for gate-gnocchi-tox-py35-mysql-ubuntu-xenial 459873,5,17534b7 | 12:19 |
| *** artom has quit IRC | 12:20 | |
| *** artom has joined #openstack-qa | 12:20 | |
| *** artom has quit IRC | 12:22 | |
| *** artom has joined #openstack-qa | 12:22 | |
| *** gszasz has quit IRC | 12:22 | |
| *** pgadiya has quit IRC | 12:24 | |
| *** Kevin_Zheng has quit IRC | 12:27 | |
| *** artom has quit IRC | 12:30 | |
| *** artom has joined #openstack-qa | 12:30 | |
| *** artom has quit IRC | 12:32 | |
| *** artom has joined #openstack-qa | 12:32 | |
| *** artom has quit IRC | 12:38 | |
| *** markvoelker has joined #openstack-qa | 12:38 | |
| *** artom has joined #openstack-qa | 12:38 | |
| openstackrecheck | Console logs not available after 20:05s for gate-k8s-cloud-provider-golang-dsvm-conformance-ubuntu-xenial 456561,34,97a2bb1 | 12:39 |
| *** dkranz_ has joined #openstack-qa | 12:40 | |
| *** dkranz has quit IRC | 12:40 | |
| *** mriedem has joined #openstack-qa | 12:41 | |
| *** artom has quit IRC | 12:42 | |
| *** artom has joined #openstack-qa | 12:42 | |
| *** dustins has joined #openstack-qa | 12:46 | |
| *** yamamoto has quit IRC | 12:47 | |
| *** yangyapepeng has joined #openstack-qa | 12:49 | |
| *** yangyapepeng has quit IRC | 12:53 | |
| *** yangyapepeng has joined #openstack-qa | 12:53 | |
| openstackrecheck | Console logs not available after 20:14s for gate-functional-dsvm-magnum-k8s-ubuntu-xenial 404253,41,9de4fbe | 12:59 |
| *** catintheroof has joined #openstack-qa | 13:01 | |
| *** gszasz has joined #openstack-qa | 13:01 | |
| *** ihrachys has joined #openstack-qa | 13:01 | |
| *** catintheroof has quit IRC | 13:04 | |
| *** catintheroof has joined #openstack-qa | 13:04 | |
| *** porrua has joined #openstack-qa | 13:05 | |
| *** udesale__ has quit IRC | 13:07 | |
| *** lucas-hungry is now known as lucasagomes | 13:08 | |
| *** catinthe_ has joined #openstack-qa | 13:18 | |
| *** dpaterson_ has joined #openstack-qa | 13:19 | |
| openstackrecheck | Console logs not available after 20:12s for gate-tempest-dsvm-neutron-dvr-ubuntu-xenial 460695,1,b6c9604 | 13:20 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Use compat names for journal files https://review.openstack.org/460947 | 13:20 |
| *** anshul has quit IRC | 13:21 | |
| openstackgerrit | Sean Dague proposed openstack-dev/devstack master: make USE_SCREEN=False imply USE_SYSTEMD=True https://review.openstack.org/460062 | 13:21 |
| *** catintheroof has quit IRC | 13:21 | |
| *** blancos has joined #openstack-qa | 13:22 | |
| *** arturb has quit IRC | 13:23 | |
| *** dsariel has quit IRC | 13:27 | |
| *** awaugama has joined #openstack-qa | 13:32 | |
| *** aojea has joined #openstack-qa | 13:36 | |
| openstackrecheck | Console logs not available after 20:08s for gate-patrole-pep8-ubuntu-xenial 460687,1,df1b774 | 13:40 |
| *** dpaterson_ has quit IRC | 13:45 | |
| *** gcheresh has joined #openstack-qa | 13:46 | |
| *** yamamoto has joined #openstack-qa | 13:48 | |
| openstackrecheck | Console logs not available after 20:07s for gate-octavia-v1-dsvm-scenario-multinode 440934,16,4734507 | 14:00 |
| *** makowals has quit IRC | 14:01 | |
| *** cdent has quit IRC | 14:05 | |
| *** felipemonteiro has joined #openstack-qa | 14:07 | |
| *** felipemonteiro_ has joined #openstack-qa | 14:09 | |
| *** rbak has joined #openstack-qa | 14:10 | |
| *** Serlex has joined #openstack-qa | 14:10 | |
| *** felipemonteiro has quit IRC | 14:12 | |
| *** prateek has quit IRC | 14:14 | |
| *** yamamoto has quit IRC | 14:16 | |
| *** aarefiev is now known as aarefiev_afk | 14:19 | |
| *** witek has quit IRC | 14:20 | |
| openstackrecheck | Console logs not available after 20:05s for gate-osc-dsvm-functional-ubuntu-xenial 460356,3,567444b | 14:20 |
| *** yamahata_ has joined #openstack-qa | 14:25 | |
| *** hongbin has joined #openstack-qa | 14:26 | |
| *** armax has joined #openstack-qa | 14:33 | |
| mriedem | mtreinish: andreaf: would you agree that http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/volume/test_volumes_actions.py#n36 is not a useful test in tempest? it's testing the os-attach/os-detach APIs in cinder which users shouldn't need to use, all they do is set state on the volume in the cinder db and are for nova's usage during actual volume attach/detach, tested in http://git.openstack.org/cgit/openstack/tem | 14:36 |
| mriedem | /tree/tempest/api/compute/volumes/test_attach_volume.py#n61 | 14:36 |
| openstackgerrit | Pradeep Kumar proposed openstack/patrole master: Nova test for Volume client https://review.openstack.org/459683 | 14:37 |
| mtreinish | mriedem: yeah that's not super useful | 14:37 |
| smcginnis | We'll know pretty quick if those APIs break anyway. I think they are well covered implictly. | 14:37 |
| openstackgerrit | Matthew Treinish proposed openstack-dev/devstack master: WIP: Use uwsgi for glance-api https://review.openstack.org/459451 | 14:37 |
| openstackgerrit | Matthew Treinish proposed openstack-dev/devstack master: Add external lock dir to glance api config https://review.openstack.org/459513 | 14:37 |
| mtreinish | smcginnis: yeah if we broke cinder os-attach any of the nova volume attach tests would fail | 14:38 |
| mriedem | mtreinish: first step is i have to get that cinder test deprecated from defcore... | 14:38 |
| mtreinish | wait it's in defcore? | 14:38 |
| mriedem | so this is probably something we can't actually remove for like what, a year or more? | 14:38 |
| mriedem | mtreinish: of course | 14:38 |
| mriedem | mtreinish: https://github.com/openstack/interop/blob/master/2017.01.json#L2676 | 14:39 |
| mtreinish | I think I might have mentioned this to them a while ago actually. I'm having vague memories of explaining that api to them | 14:39 |
| openstackrecheck | Console logs not available after 20:12s for gate-tempest-dsvm-py35-ubuntu-xenial 459110,5,715ba5c | 14:40 |
| *** gouthamr has joined #openstack-qa | 14:44 | |
| *** ramishra has quit IRC | 14:46 | |
| openstackgerrit | Pradeep Kumar proposed openstack/patrole master: Nova test for Volume client https://review.openstack.org/459683 | 14:48 |
| *** cdent has joined #openstack-qa | 14:52 | |
| *** aarefiev_afk has quit IRC | 14:59 | |
| openstackrecheck | Console logs not available after 20:22s for gate-heat-pep8-ubuntu-xenial 437810,5,f57407d | 15:01 |
| *** gcheresh has quit IRC | 15:03 | |
| *** rossella_s has quit IRC | 15:04 | |
| *** zz_dimtruck is now known as dimtruck | 15:04 | |
| *** rossella_s has joined #openstack-qa | 15:04 | |
| *** openstackgerrit has quit IRC | 15:17 | |
| openstackrecheck | Console logs not available after 20:07s for gate-nova-pep8-ubuntu-xenial 456877,3,2a4a460 | 15:21 |
| *** nkorabli has quit IRC | 15:22 | |
| *** nkorabli has joined #openstack-qa | 15:22 | |
| *** mkrai has joined #openstack-qa | 15:28 | |
| *** pcaruana has quit IRC | 15:28 | |
| mkrai | Hi I am facing issue in keystone while installing devstack | 15:29 |
| mkrai | http://paste.openstack.org/show/608341/ | 15:29 |
| *** openstackgerrit has joined #openstack-qa | 15:29 | |
| openstackgerrit | Matthew Treinish proposed openstack-dev/devstack master: WIP: Use uwsgi for glance-api https://review.openstack.org/459451 | 15:29 |
| mkrai | Keystone logs http://paste.openstack.org/show/608342/ | 15:29 |
| mkrai | Hi mattt Are you aware of the above error? | 15:29 |
| mkrai | devstack master seems to be broken | 15:29 |
| mtreinish | mkrai: I assume you meant to ping me. I don't think master devstack is broken, we're running it constantly | 15:34 |
| mtreinish | mkrai: where are you hitting this failure? | 15:35 |
| mtreinish | mkrai: in a gate job or are you running it locally? | 15:35 |
| mkrai | mtreinish: Hi yes | 15:35 |
| mkrai | Running locally | 15:35 |
| mtreinish | ok, and what does your local.conf look like? | 15:35 |
| *** sdague has quit IRC | 15:36 | |
| mkrai | mtreinish: https://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack | 15:36 |
| mkrai | I have used the same conf file | 15:36 |
| mtreinish | well nothing looks too crazy there | 15:38 |
| mtreinish | can you paste more of the log? I suspect the real failure is earlier (but it's not fatal) and what you're seeing is just what's bubbling up | 15:39 |
| *** fnaval has joined #openstack-qa | 15:40 | |
| mkrai | Ok please give me a moment | 15:40 |
| openstackrecheck | Console logs not available after 20:05s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 460512,1,98e4ace | 15:41 |
| mkrai | mtreinish: http://paste.openstack.org/show/608346/ | 15:42 |
| lbragstad | open("./python_plugin.so"): No such file or directory [core/utils.c line 3686] | 15:43 |
| lbragstad | that looks suspicious | 15:43 |
| mtreinish | mkrai: and what's your host operating system? | 15:43 |
| mkrai | I check on keystone channel and they told me that | 15:43 |
| *** Apoorva has joined #openstack-qa | 15:43 | |
| mkrai | mtreinish: mkrai the devstack project recently switched keystone to uwsgi instead of apache | 15:44 |
| mtreinish | lbragstad: yeah, I think that's the issue on centos7 that a bunch of people are hitting | 15:44 |
| mkrai | https://github.com/openstack-dev/devstack/commit/6ed53156b6198e69d59d1cf3a3497e96f5b7a870 | 15:44 |
| mkrai | Ubuntu 16.04 | 15:44 |
| lbragstad | mkrai have you updated the system package index? | 15:45 |
| mkrai | yes | 15:45 |
| lbragstad | mkrai have you tried installing uwsgi-plugin-python3 ? | 15:45 |
| mkrai | not explicitly | 15:45 |
| mtreinish | lbragstad: it builds it from source off of pip iirc | 15:45 |
| lbragstad | mriedem ah - the uwsgi plugin? | 15:46 |
| lbragstad | mtreinish * | 15:46 |
| lbragstad | sorry | 15:46 |
| lbragstad | rogue tab | 15:46 |
| mtreinish | yeah. I think there were issues with the packaged version | 15:46 |
| mtreinish | but I might be confusing that with something else | 15:46 |
| lbragstad | huh - intersting | 15:47 |
| mtreinish | lbragstad: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L69-L109 | 15:47 |
| *** Yangyapeng has joined #openstack-qa | 15:48 | |
| mtreinish | lbragstad: also why python3? | 15:48 |
| lbragstad | mtreinish not entirely sure, i was doing some googling | 15:48 |
| lbragstad | that could be wrong | 15:49 |
| lbragstad | just looking to see if anyone else has hit this before | 15:49 |
| mtreinish | mkrai: can I see the whole devstack log? I know it's gonna be big | 15:49 |
| mkrai | mtreinish: ok | 15:50 |
| mkrai | mtreinish: http://paste.openstack.org/show/608351/ | 15:59 |
| *** david-lyle has quit IRC | 16:00 | |
| mtreinish | heh, yeah that's probably too big for paste, it's truncating pretty early | 16:00 |
| openstackrecheck | Console logs not available after 20:06s for gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial 456780,2,e244033 | 16:01 |
| mkrai | mtreinish: :( | 16:01 |
| mkrai | mtreinish: I will paste the last log that would be more relevant | 16:02 |
| mtreinish | mkrai, lbragstad: also I just checked a gate run and the missing binary message is normal: http://logs.openstack.org/74/459474/10/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/edf488e/logs/screen-key-p.txt.gz | 16:02 |
| mtreinish | mkrai: sure, I'm just trying to see where thigns start to go bad, in the first paste there are warnings, so maybe a bit more above that too | 16:03 |
| lbragstad | mtreinish aha nice, that's good to know | 16:03 |
| *** sridharg has quit IRC | 16:04 | |
| mtreinish | lbragstad: according to: http://uwsgi.unbit.narkive.com/QpuIAJNX/unable-to-load-uwsgi-plugin-python-plugin-so-cannot-open-shared-object-file-no-such-file-or it gets embedded in the binary when you install from pip | 16:04 |
| mkrai | mtreinish: http://paste.openstack.org/show/608353/ | 16:05 |
| *** spzala has quit IRC | 16:06 | |
| *** spzala has joined #openstack-qa | 16:06 | |
| *** aojea has quit IRC | 16:06 | |
| *** aojea has joined #openstack-qa | 16:07 | |
| mtreinish | mkrai: just curious what's the ip address of the machine you're running this on? | 16:09 |
| mkrai | 10.223.197.220 | 16:10 |
| mtreinish | ok, well at least it's trying to connect to the proper adddress | 16:11 |
| *** spzala has quit IRC | 16:11 | |
| mtreinish | the only other thing I can think of is whether apache is actually running | 16:11 |
| *** aojea has quit IRC | 16:11 | |
| mkrai | mtreinish: http://paste.openstack.org/show/608356/ | 16:14 |
| *** shardy has quit IRC | 16:14 | |
| mtreinish | mkrai: well that would explain it. Although I have no idea why apache wouldn't be running | 16:15 |
| mtreinish | mkrai: actually isn't it httpd or apache2? (I know the package is apache2 but I can never remember what ubuntu does for the service name) | 16:16 |
| mkrai | mtreinish: Ok its apache2 and its running | 16:17 |
| mtreinish | mkrai: can you check the apache logs then | 16:18 |
| mkrai | yes | 16:18 |
| mkrai | mtreinish: http://paste.openstack.org/show/608357/ | 16:20 |
| mkrai | That's apache error log | 16:20 |
| mtreinish | mkrai: there should be separate log files setup for the different services | 16:21 |
| mtreinish | is there a keystone admin one? | 16:21 |
| mtreinish | (or a keystone one at all) | 16:21 |
| openstackgerrit | Matthew Treinish proposed openstack-dev/devstack master: WIP: Use uwsgi for glance-api https://review.openstack.org/459451 | 16:21 |
| openstackrecheck | Console logs not available after 20:19s for gate-tempest-dsvm-networking-odl-carbon-snapshot-v2driver 460704,1,bb6edfb | 16:21 |
| *** nkorabli has quit IRC | 16:22 | |
| *** yangyapepeng has quit IRC | 16:22 | |
| *** Yangyapeng has quit IRC | 16:22 | |
| *** nkorabli has joined #openstack-qa | 16:22 | |
| mkrai | mtreinish: I have already shared the keystone admin logs | 16:23 |
| mkrai | mtreinish: Shall I share it again? | 16:23 |
| mtreinish | no you shared the uwsgi log, apache is what actually handles the actually http | 16:24 |
| mkrai | mtreinish: Can you please tell me the log location if possible? | 16:25 |
| mtreinish | the requests are forwarded to uwsgi with mod_proxy_uwsgi | 16:25 |
| mkrai | I have checked /opt/stack/logs and /var/log | 16:25 |
| mtreinish | it should be in the apache logs dir /var/log/apache2 | 16:25 |
| mtreinish | if its not there I would check the config to see if the site is enabled in /etc/apache2/site-enabled | 16:25 |
| mtreinish | because the issue you're hitting is that the endpoint isn't found. So apache either isn't configured to setup the proxy properly. Or something went wrong and it's crashing (which seems unlikely because you said it's still running) | 16:26 |
| mkrai | mtreinish: No there only this error log | 16:26 |
| *** aojea has joined #openstack-qa | 16:27 | |
| mkrai | and 2 more file which is access.log and other_vhosts_access.log | 16:27 |
| mtreinish | ok, then that likely means it's not being configured | 16:27 |
| mtreinish | or not started if the config files are being run | 16:27 |
| mkrai | One thing I am running this setup behind proxy | 16:28 |
| *** nkorabli has quit IRC | 16:28 | |
| *** nkorabli has joined #openstack-qa | 16:28 | |
| *** cdent has quit IRC | 16:29 | |
| mtreinish | well that might have something to do with it | 16:32 |
| mtreinish | but, I've never run devstack behind a proxy, so I'm not sure how that would effect this | 16:33 |
| mtreinish | anyway I need to step away for a bit | 16:33 |
| mtreinish | sry I couldnt get to the bottom of your issue, if you're still around when Im back feel free to ping me | 16:33 |
| mkrai | mtreinish: sure. Thank you | 16:34 |
| *** dave-mccowan has quit IRC | 16:36 | |
| *** rossella_s has quit IRC | 16:38 | |
| openstackrecheck | Console logs not available after 20:05s for gate-tempest-dsvm-cells-ubuntu-xenial 460515,1,0c8340b | 16:41 |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| *** tesseract has quit IRC | 16:45 | |
| *** markus_z has quit IRC | 16:46 | |
| *** gcheresh has joined #openstack-qa | 16:48 | |
| *** david-lyle has joined #openstack-qa | 16:50 | |
| *** tosky has quit IRC | 16:52 | |
| *** lucasagomes is now known as lucas-afk | 16:54 | |
| *** gcheresh has quit IRC | 16:59 | |
| *** dave-mccowan has joined #openstack-qa | 16:59 | |
| openstackrecheck | Console logs not available after 20:05s for gate-aodh-dsvm-functional-mysql-ubuntu-xenial 460486,1,4cce2c3 | 17:01 |
| *** harlowja has quit IRC | 17:06 | |
| *** gfidente has quit IRC | 17:08 | |
| *** e0ne has quit IRC | 17:08 | |
| *** aojea has quit IRC | 17:08 | |
| *** asselin_ has joined #openstack-qa | 17:09 | |
| *** asselin has quit IRC | 17:11 | |
| *** yamahata_ has quit IRC | 17:11 | |
| felipemonteiro_ | mtreinish: andreaf: If a current client is supported in keystone should it also be supported in tempest? I'm wondering whether I should abandon: https://review.openstack.org/#/c/452739/ | 17:14 |
| openstackrecheck | Console logs not available after 20:08s for gate-k8s-cloud-provider-golang-dsvm-conformance-ubuntu-xenial 460504,6,1e5a4bc | 17:22 |
| *** asselin__ has joined #openstack-qa | 17:25 | |
| *** amotoki has quit IRC | 17:25 | |
| *** asselin_ has quit IRC | 17:28 | |
| *** aojea has joined #openstack-qa | 17:36 | |
| *** aojea_ has joined #openstack-qa | 17:38 | |
| *** aojea has quit IRC | 17:41 | |
| openstackrecheck | Console logs not available after 20:07s for gate-rally-dsvm-neutron-neutron-ubuntu-xenial 458531,5,42a5ed1 | 17:42 |
| *** aojea_ has quit IRC | 17:43 | |
| *** yamahata_ has joined #openstack-qa | 17:49 | |
| *** e0ne has joined #openstack-qa | 17:57 | |
| openstackrecheck | Console logs not available after 20:24s for gate-tempest-dsvm-py35-ubuntu-xenial 457715,13,b607f69 | 18:02 |
| *** gszasz has quit IRC | 18:03 | |
| *** catintheroof has joined #openstack-qa | 18:04 | |
| *** catinthe_ has quit IRC | 18:08 | |
| *** BobBall has quit IRC | 18:14 | |
| *** gcheresh has joined #openstack-qa | 18:15 | |
| openstackgerrit | Artom Lifshitz proposed openstack/tempest master: Integration tests for tagged attach https://review.openstack.org/391947 | 18:18 |
| openstackrecheck | Console logs not available after 20:07s for gate-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet 454411,6,2f01a9f | 18:22 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: Added query for bug 1687044 https://review.openstack.org/461121 | 18:24 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 18:24 |
| *** david-lyle_ has joined #openstack-qa | 18:24 | |
| clarkb | smcginnis: mtreinish http://logs.openstack.org/27/459727/5/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/c58eb92/console.html seems to be a common issue? | 18:25 |
| smcginnis | clarkb: I've been seeing it a lot lately. | 18:25 |
| smcginnis | clarkb: But no idea what changed. There was actually a previous bug (and fix) quite a while back that added explicit waiting for volume deactivation. | 18:26 |
| clarkb | smcginnis: is http://logs.openstack.org/27/459727/5/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/c58eb92/logs/screen-c-vol.txt.gz?level=TRACE#_2017-04-28_04_24_52_553 the underlying issue that we can write an e-r query against? | 18:26 |
| smcginnis | clarkb: I confirmed that code is still there. | 18:26 |
| *** david-lyle has quit IRC | 18:26 | |
| smcginnis | I've seen the "not deactivated in time" message, but also cases where it apparently did deactivate, or at least returned from the _wait_for_volume_deactivation successfully, but still resulted in that error message. | 18:27 |
| clarkb | smcginnis: ya I think in the cases where it did deactivate if you look at timestamps it does so after the tempest timeout | 18:29 |
| clarkb | mtreinish felt the timeout was already incredibly generous though so didn't want to make it longer | 18:30 |
| smcginnis | clarkb: What is the timeout now? | 18:31 |
| clarkb | 196seconds | 18:31 |
| smcginnis | Should really be plenty of time. | 18:32 |
| smcginnis | Are these nodes getting extra bogged down? Noisy neighbors maybe? | 18:32 |
| smcginnis | It really shouldn't take that long to deactivate an LVM volume. | 18:32 |
| clarkb | the example at http://logs.openstack.org/14/460814/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/44d46f9/console.html ran on infracloud which has 1:1 cpu allocation iirc | 18:33 |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Replace generic api_extensions checks https://review.openstack.org/460687 | 18:33 |
| clarkb | http://logs.openstack.org/27/459727/5/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/c58eb92/console.html ran on rax-ord public cloud so possible there | 18:33 |
| clarkb | smcginnis: you can also review the dstat logs | 18:33 |
| *** ihrachys_ has joined #openstack-qa | 18:36 | |
| clarkb | it doesn't look to overloaded when that test failed | 18:36 |
| clarkb | system load is ~4 on 8 core host | 18:36 |
| smcginnis | I see a few CPU spikes in the 80-90 range, but nothing too bad. | 18:36 |
| clarkb | looks like tempest is timing out ~3 minutes after the error too | 18:37 |
| clarkb | so the timeout in tempest is quite generous | 18:37 |
| smcginnis | Seems like it should be plenty of time. | 18:37 |
| *** ihrachys has quit IRC | 18:38 | |
| *** harlowja has joined #openstack-qa | 18:38 | |
| smcginnis | Only thing I can find sounding similar for LVM deactivation also involved encryption: https://bugzilla.redhat.com/show_bug.cgi?id=1097322 | 18:39 |
| openstack | bugzilla.redhat.com bug 1097322 in lvm2 "laptop with LUKS + LVM shutdowns too slow." [Unspecified,Closed: eol] - Assigned to lvm-team | 18:39 |
| clarkb | smcginnis: looks like cinder only waits about 3 seconds | 18:42 |
| openstackrecheck | Console logs not available after 20:03s for gate-neutron-fwaas-python27-ubuntu-xenial 323971,23,5c2194a | 18:42 |
| clarkb | smcginnis: http://logs.openstack.org/14/460814/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/44d46f9/logs/screen-c-vol.txt.gz?level=DEBUG#_2017-04-28_15_48_04_033 is first check then third and last is at http://logs.openstack.org/14/460814/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/44d46f9/logs/screen-c-vol.txt.gz?level=DEBUG#_2017-04-28_15_48_07_271 then less than a second later its | 18:43 |
| clarkb | decided to have errored | 18:43 |
| clarkb | so we probably can reduce tempest timeout to like 10 seconds safely | 18:43 |
| clarkb | smcginnis: is 3 seconds just too little? | 18:44 |
| *** odyssey4me has quit IRC | 18:44 | |
| smcginnis | clarkb: Hmm, it could be in this case. | 18:44 |
| smcginnis | clarkb: Still seems like excessively long, especially for the amount of IO actually done here. But maybe if we bump that up we will be better off. | 18:45 |
| *** dpaterson_ has joined #openstack-qa | 18:45 | |
| smcginnis | clarkb: Certainly could make it better for customer environments with much higher load. | 18:45 |
| clarkb | I'm not sure how strong our iops garuntees can be in this sort of setup | 18:45 |
| clarkb | 3 seconds seems like a long time but then we are running this in VMs on top of diverse hardware in different clouds and continents. Basically nothing surprises me anymore | 18:46 |
| clarkb | possibly something in kernel changed to make it less quick? | 18:46 |
| *** odyssey4me has joined #openstack-qa | 18:46 | |
| clarkb | smcginnis: or maybe the initial request is failing so no matter how long you wait ti will never happen? | 18:46 |
| smcginnis | clarkb: I would have thought we would get an error back then, but I'm not super knowledgable on that bit. | 18:49 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/tempest master: Prevent error in _parse_resp when nullable list https://review.openstack.org/461124 | 18:49 |
| clarkb | smcginnis: ya I don't know much about it either | 18:49 |
| clarkb | smcginnis: http://logs.openstack.org/14/460814/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/44d46f9/logs/screen-c-vol.txt.gz?level=DEBUG#_2017-04-28_15_48_04_032 is the underlying command right? | 18:49 |
| clarkb | ah yup -a n means activate no | 18:50 |
| smcginnis | yep | 18:50 |
| clarkb | smcginnis: will cinder capture the stdout/stderr of the command? maybe try setting -d or -v? | 18:51 |
| * clarkb checks syslog as apparently it can log there | 18:51 | |
| smcginnis | clarkb: If an error occurs immediately on calling lvchange -an it logs the stderr and stdout. | 18:54 |
| smcginnis | clarkb: If it outright couldn't deactivate, I would expect that call to fail immediately. | 18:54 |
| clarkb | smcginnis: in http://logs.openstack.org/14/460814/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/44d46f9/logs/syslog.txt.gz if you grep for c95e8d64-a1c9-4214-9de1-195a4aaebc8e the lvchange is not logged | 18:54 |
| clarkb | smcginnis: almost like rootwrap didn't actually run the command | 18:54 |
| smcginnis | Ever see anything like that before? I'm completely lost if that's the case. | 18:55 |
| clarkb | nope | 18:55 |
| clarkb | oh wait there it s my grep foo must've been bad | 18:56 |
| * smcginnis sighs a big sigh of releif | 18:56 | |
| clarkb | http://logs.openstack.org/14/460814/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/44d46f9/logs/syslog.txt.gz#_Apr_28_15_48_03 | 18:56 |
| clarkb | smcginnis: it is interesting that the lvremove that corresponds to the delete has retry_activation = 1 | 18:57 |
| *** e0ne has quit IRC | 18:58 | |
| clarkb | * retry_deactivation | 18:59 |
| *** ihrachys_ is now known as ihrachys | 19:02 | |
| *** artom has quit IRC | 19:02 | |
| *** artom has joined #openstack-qa | 19:02 | |
| openstackrecheck | Console logs not available after 20:05s for gate-tempest-dsvm-py35-ubuntu-xenial 460616,2,0c9ca73 | 19:02 |
| *** artom has quit IRC | 19:04 | |
| *** artom has joined #openstack-qa | 19:04 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: Added query for bug 1687044 https://review.openstack.org/461121 | 19:05 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 19:05 |
| *** david-lyle_ is now known as david-lyle | 19:09 | |
| *** xyang1 has joined #openstack-qa | 19:15 | |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Adds create metadef resource test https://review.openstack.org/461130 | 19:18 |
| mtreinish | clarkb: are we spiking on volume test timeouts? | 19:20 |
| clarkb | mtreinish: I think so. They keep showing up in uncategorized list | 19:21 |
| clarkb | initially started trying to figure out how to query for it then went down the hole of debugging it | 19:21 |
| clarkb | I think the cinder traceback is the query | 19:21 |
| openstackrecheck | Console logs not available after 20:03s for gate-neutron-fwaas-python27-ubuntu-xenial 425769,15,2bbf82e | 19:23 |
| mtreinish | clarkb: ok, feel free to push the query up to e-r. I'll review it quickly | 19:24 |
| clarkb | smcginnis: mtreinish do you know if there is an open bug for this issue? | 19:24 |
| mtreinish | clarkb: is it this: https://review.openstack.org/#/c/461121/ | 19:25 |
| *** fandi has joined #openstack-qa | 19:25 | |
| clarkb | mtreinish: yes | 19:25 |
| clarkb | thanks ihrachys | 19:25 |
| ihrachys | clarkb, howdy | 19:26 |
| clarkb | ihrachys: just saw change 461121 thanks for pushing it | 19:26 |
| smcginnis | clarkb, mtreinish: Attempt at getting at least a work around for this: https://review.openstack.org/#/c/461129/ | 19:27 |
| clarkb | mtreinish: separately we can reduce the tempest timeout quite a bit I think since cinder only tries for ~3 seconds | 19:27 |
| clarkb | and ^ will make it go up to ~10s | 19:27 |
| smcginnis | clarkb: Yep, still plenty of a window between there and the tempest timeout. | 19:28 |
| mtreinish | clarkb: do you want to decrease the tempest timeout? | 19:32 |
| mtreinish | clarkb: I'd rather keep the 196 as the default (for outside the gate cases) but we can tune it in devstack | 19:32 |
| openstackgerrit | Ihar Hrachyshka proposed openstack/tempest master: Dump ARP entry for the failed ip address on connectivity failure https://review.openstack.org/454892 | 19:32 |
| openstackgerrit | Ihar Hrachyshka proposed openstack/tempest master: DNM: check that dump ARP entries works https://review.openstack.org/461131 | 19:32 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Added query for bug 1687044 https://review.openstack.org/461121 | 19:33 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 19:33 |
| clarkb | mtreinish: it looks hardcoded in cinder so all clouds affected? | 19:34 |
| mtreinish | clarkb: for lvm, but tempest is run against clouds with random driver X | 19:36 |
| clarkb | ah | 19:36 |
| mtreinish | but we can easily tune the timeout for the gate case if you think it's worthwhile | 19:36 |
| mtreinish | I haven't thought too much about it | 19:37 |
| clarkb | it could save a few minutes of job runtime not ure how much wall time though | 19:37 |
| *** Serlex has quit IRC | 19:41 | |
| openstackrecheck | Console logs not available after 20:48s for nodepool-coverage-ubuntu-xenial 455464,3,cf237c6 | 19:43 |
| openstackgerrit | Matthew Treinish proposed openstack-dev/devstack master: WIP: Use uwsgi for glance-api https://review.openstack.org/459451 | 19:53 |
| openstackgerrit | Felipe Monteiro proposed openstack/qa-specs master: RBAC testing multiple policies https://review.openstack.org/461140 | 19:58 |
| *** openstackrecheck has quit IRC | 19:58 | |
| *** openstackrecheck has joined #openstack-qa | 19:58 | |
| openstackgerrit | Felipe Monteiro proposed openstack/qa-specs master: RBAC testing multiple policies https://review.openstack.org/461140 | 19:59 |
| openstackgerrit | Felipe Monteiro proposed openstack/qa-specs master: RBAC testing multiple policies https://review.openstack.org/461140 | 19:59 |
| openstackgerrit | Felipe Monteiro proposed openstack/qa-specs master: RBAC testing multiple policies https://review.openstack.org/461140 | 20:00 |
| *** Apoorva has quit IRC | 20:02 | |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Adds create metadef resource test https://review.openstack.org/461130 | 20:05 |
| openstackgerrit | Merged openstack/patrole master: Stop using aliases for creds manager https://review.openstack.org/460328 | 20:05 |
| *** Apoorva has joined #openstack-qa | 20:09 | |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Replace generic api_extensions checks https://review.openstack.org/460687 | 20:10 |
| *** dave-mccowan has quit IRC | 20:10 | |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Adds missing volumes client tests https://review.openstack.org/459855 | 20:13 |
| openstackgerrit | Felipe Monteiro proposed openstack/patrole master: Identity V3 Tests - Domain Configurations https://review.openstack.org/457328 | 20:15 |
| openstackgerrit | Felipe Monteiro proposed openstack/patrole master: Identity V3 Tests - Domain Configurations https://review.openstack.org/457328 | 20:16 |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Refactor identity tests https://review.openstack.org/460312 | 20:18 |
| openstackgerrit | Samantha Blanco proposed openstack/patrole master: Adds missing volumes client tests https://review.openstack.org/459855 | 20:20 |
| openstackrecheck | Console logs not available after 20:30s for gate-keystone-python27-ubuntu-xenial 257636,20,aaafa22 | 20:21 |
| openstackgerrit | Artom Lifshitz proposed openstack/tempest master: Integration tests for tagged attach https://review.openstack.org/391947 | 20:29 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/452623 failed gate-tempest-dsvm-neutron-dvr-ubuntu-xenial, gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial in the gate queue with an unrecognized error | 20:29 |
| *** blancos has quit IRC | 20:30 | |
| clarkb | smcginnis: mtreinish ihrachys ok that e-r chagne is in and at least for all pipelines we have the volume fail rendered. Its a big problem :) | 20:31 |
| clarkb | and it just started. We might want to check if it only affects xenail and if so did the kernel/kvm/etc just update | 20:32 |
| ihrachys | clarkb, it affects newton too I think, and we don't have UCA there | 20:32 |
| smcginnis | clarkb: That's my assumption. I don't see anything that changed in cinder itself that should have triggered this. | 20:32 |
| smcginnis | Something is causing LVM deactivation to take longer than normal. | 20:33 |
| *** asselin has joined #openstack-qa | 20:48 | |
| *** vhosakot has joined #openstack-qa | 20:48 | |
| *** asselin__ has quit IRC | 20:50 | |
| openstackrecheck | Console logs not available after 20:06s for gate-gnocchi-tox-py27-mysql-ubuntu-xenial 461139,1,c77223b | 20:52 |
| *** dustins has quit IRC | 20:55 | |
| *** Apoorva_ has joined #openstack-qa | 21:01 | |
| *** Apoorva has quit IRC | 21:04 | |
| *** dimtruck is now known as zz_dimtruck | 21:06 | |
| *** dpaterson_ has quit IRC | 21:07 | |
| *** catintheroof has quit IRC | 21:14 | |
| *** esberglu has quit IRC | 21:17 | |
| *** harlowja has quit IRC | 21:17 | |
| *** harlowja has joined #openstack-qa | 21:17 | |
| *** aojea has joined #openstack-qa | 21:21 | |
| openstackrecheck | Console logs not available after 20:06s for gate-nova-python27-ubuntu-xenial 457711,3,e198feb | 21:24 |
| *** cody-somerville has joined #openstack-qa | 21:27 | |
| *** cody-somerville has quit IRC | 21:27 | |
| *** cody-somerville has joined #openstack-qa | 21:27 | |
| *** csomerville has quit IRC | 21:29 | |
| *** gcheresh has quit IRC | 21:29 | |
| *** nkorabli has quit IRC | 21:30 | |
| *** nkorabli has joined #openstack-qa | 21:30 | |
| *** nkorabli has quit IRC | 21:33 | |
| *** nkorabli has joined #openstack-qa | 21:34 | |
| *** asselin has quit IRC | 21:34 | |
| *** zz_dimtruck is now known as dimtruck | 21:35 | |
| *** aojea has quit IRC | 21:35 | |
| *** harlowja has quit IRC | 21:36 | |
| *** Apoorva_ has quit IRC | 21:36 | |
| *** mriedem has quit IRC | 21:37 | |
| *** Apoorva has joined #openstack-qa | 21:37 | |
| openstackrecheck | Console logs not available after 20:06s for gate-tempest-dsvm-py35-ubuntu-xenial 461129,1,4f37ab3 | 21:44 |
| clarkb | smcginnis: mtreinish issue also seems to manifest as http://logs.openstack.org/15/456915/6/gate/gate-tempest-dsvm-py35-ubuntu-xenial/17c6289/logs/screen-c-vol.txt.gz?level=WARNING#_2017-04-28_14_42_57_037 | 21:46 |
| clarkb | mtreinish: ihrachys ^ I'm gonna push an update to the e-r query to catch those too | 21:47 |
| mtreinish | clarkb: ok | 21:47 |
| smcginnis | clarkb: Yeah, looks like a different message for the same root cause. | 21:47 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck master: Extend query for bug 1687044 https://review.openstack.org/461166 | 21:52 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 21:52 |
| clarkb | I tried to make the query string readable | 21:52 |
| mtreinish | clarkb: I just noticed the wildcard in the filename, I should have caught that before | 21:53 |
| mtreinish | shouldn't we just use the c-vol tag | 21:53 |
| *** felipemonteiro_ has quit IRC | 21:54 | |
| clarkb | mtreinish: its there to catch grenade | 21:54 |
| clarkb | so logs/new/c-vol /logs/old/cvol and logs/cvol | 21:55 |
| clarkb | I think its fine | 21:55 |
| mtreinish | the tag will do that | 21:55 |
| clarkb | ah | 21:55 |
| mtreinish | that's why we have the tags field on logstash | 21:55 |
| clarkb | right I remmber that now | 21:55 |
| mtreinish | because the wild card makes the query more expensive | 21:55 |
| clarkb | so what is it tags:cvol? | 21:55 |
| * clarkb looks at logstash | 21:55 | |
| clarkb | I'll push a new patchset once I have that figured out | 21:56 |
| mtreinish | ok | 21:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck master: Extend query for bug 1687044 https://review.openstack.org/461166 | 21:57 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 21:57 |
| clarkb | I think thats right, I tested it in kibana | 21:57 |
| clarkb | smcginnis: mtreinish diffing pacakges from today and the 24th we did get a new kernel | 21:58 |
| clarkb | linux-image-4.4.0-75-generic was 72-generic | 21:58 |
| *** cmurphy has quit IRC | 21:59 | |
| clarkb | http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_4.4.0-75.96/changelog its actually a fairly large changelog, reading through it now | 21:59 |
| clarkb | it happens on xenial only and across all the clouds so pretty sure its not a cloud specific performance issue (though doesn't rule out performance entirely) | 22:04 |
| openstackrecheck | Console logs not available after 20:11s for gate-tempest-dsvm-py35-ubuntu-xenial 459451,9,09a7f1f | 22:04 |
| *** Apoorva has quit IRC | 22:05 | |
| *** cmurphy has joined #openstack-qa | 22:05 | |
| *** Apoorva has joined #openstack-qa | 22:05 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Extend query for bug 1687044 https://review.openstack.org/461166 | 22:08 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 22:08 |
| clarkb | smcginnis: do you think it would be possible to have cinder run lvmdump if it gets into an error state like that? | 22:12 |
| clarkb | smcginnis: or is it better to have that happen externally? | 22:12 |
| clarkb | mtreinish: ^ we could have tempest do that instead maybe? it breaks the rules about black box testing though | 22:12 |
| mtreinish | clarkb: it does, I'll -2 a change like that | 22:14 |
| *** gouthamr has quit IRC | 22:14 | |
| mtreinish | but we can use it for debugging in the gate | 22:14 |
| mtreinish | even if it never merges | 22:14 |
| clarkb | ya might be better to just have devstack-gate do an lvmdump then | 22:15 |
| clarkb | that way we can merge it and get better data. it appears to keep records for volumes over time even though you run it at point in time | 22:15 |
| *** catintheroof has joined #openstack-qa | 22:17 | |
| *** xyang1 has quit IRC | 22:18 | |
| *** asselin has joined #openstack-qa | 22:19 | |
| mtreinish | ok | 22:22 |
| clarkb | I'm going to put an auto hold rule in nodepool and see if we can catch one there | 22:23 |
| *** yangyapepeng has joined #openstack-qa | 22:25 | |
| openstackrecheck | Console logs not available after 20:11s for keystone-coverage-ubuntu-xenial 257636,21,d0e513c | 22:25 |
| *** vhosakot has quit IRC | 22:27 | |
| *** openstackrecheck has quit IRC | 22:28 | |
| *** openstackrecheck has joined #openstack-qa | 22:28 | |
| *** yangyapepeng has quit IRC | 22:30 | |
| *** Apoorva_ has joined #openstack-qa | 22:32 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/460928 failed because of: gate-tempest-dsvm-neutron-full-ubuntu-xenial: https://bugs.launchpad.net/bugs/1687044 | 22:34 |
| openstack | Launchpad bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] | 22:34 |
| clarkb | it looks like new kernel would've been pulled in on the 24th | 22:35 |
| clarkb | which is earlier than the problems we are seeing | 22:35 |
| *** Apoorva has quit IRC | 22:36 | |
| clarkb | oh though I don't know when we got a successfully built image with that into places which could be delayed | 22:36 |
| clarkb | based on dib build logs we would've seen this around teh 23rd so ya I don't think that is the issue | 22:38 |
| *** porrua has quit IRC | 22:40 | |
| mtreinish | clarkb: hmm, maybe the lvm packages then? | 22:46 |
| clarkb | those haven't changed. Udev updated though | 22:47 |
| clarkb | I'm hoping to catch a host that failed using nodepool then go from there | 22:48 |
| clarkb | depending on what I find there maybe we add lvmdump to d-g | 22:48 |
| *** awaugama has quit IRC | 22:55 | |
| *** fnaval has quit IRC | 22:55 | |
| *** asselin_ has joined #openstack-qa | 22:58 | |
| *** asselin__ has joined #openstack-qa | 22:59 | |
| *** asselin has quit IRC | 23:01 | |
| *** asselin_ has quit IRC | 23:03 | |
| openstackgerrit | Matthew Treinish proposed openstack-dev/devstack master: WIP: Use uwsgi for glance-api https://review.openstack.org/459451 | 23:05 |
| *** asselin has joined #openstack-qa | 23:10 | |
| *** catintheroof has quit IRC | 23:11 | |
| *** asselin__ has quit IRC | 23:12 | |
| *** asselin_ has joined #openstack-qa | 23:13 | |
| *** asselin has quit IRC | 23:17 | |
| *** esberglu has joined #openstack-qa | 23:24 | |
| *** esberglu has quit IRC | 23:28 | |
| *** harlowja has joined #openstack-qa | 23:30 | |
| *** tosky has joined #openstack-qa | 23:44 | |
| *** Apoorva_ has quit IRC | 23:48 | |
| *** harlowja has quit IRC | 23:55 | |
| *** harlowja has joined #openstack-qa | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!