| *** markvoelker has joined #openstack-qa | 00:00 | |
| *** spzala has joined #openstack-qa | 00:03 | |
| *** josdotso has quit IRC | 00:03 | |
| *** yangyapeng has quit IRC | 00:04 | |
| *** tqtran has quit IRC | 00:09 | |
| openstackrecheck | Console logs not available after 13:24s for gate-cinder-python34-db 322459,54,e4d04d4 | 00:09 |
|---|---|---|
| *** spzala has quit IRC | 00:12 | |
| openstackrecheck | Console logs not available after 13:29s for gate-nova-python34-db 352610,4,beda9d3 | 00:23 |
| *** Swami has quit IRC | 00:24 | |
| *** woodster_ has quit IRC | 00:29 | |
| *** Suyi has quit IRC | 00:35 | |
| openstackgerrit | Joshua Hesketh proposed openstack-dev/grenade: Check if c-vol is enabled https://review.openstack.org/355678 | 00:36 |
| openstackrecheck | Console logs not available after 13:25s for gate-nova-pep8-ubuntu-xenial 338553,11,fba2c46 | 00:36 |
| *** hieulq has joined #openstack-qa | 00:36 | |
| *** amotoki has joined #openstack-qa | 00:47 | |
| *** jamielennox is now known as jamielennox|away | 00:48 | |
| *** spzala has joined #openstack-qa | 00:49 | |
| *** jamielennox|away is now known as jamielennox | 00:49 | |
| openstackrecheck | Console logs not available after 13:35s for gate-tempest-dsvm-cells 317046,16,c66ec29 | 00:50 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Enable to add sudo permission to tempest user https://review.openstack.org/355682 | 00:57 |
| *** gyee has quit IRC | 00:59 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full-ceph-plugin-src-glance_store 329788,7,a5b3c07 | 01:03 |
| openstackgerrit | Merged openstack-dev/devstack: Revert "Setup cellsv2 for Nova" https://review.openstack.org/355599 | 01:04 |
| mtreinish | jroll, dansmith: ^^^ | 01:06 |
| dansmith | mtreinish: woot, thanks | 01:06 |
| *** tqtran has joined #openstack-qa | 01:09 | |
| *** tqtran has quit IRC | 01:13 | |
| *** weshay has quit IRC | 01:15 | |
| openstackrecheck | Console logs not available after 13:21s for gate-tempest-dsvm-full-ubuntu-trusty-mitaka 327191,23,8708f3a | 01:17 |
| *** Apoorva_ has joined #openstack-qa | 01:19 | |
| *** Apoorva has quit IRC | 01:23 | |
| *** Apoorva_ has quit IRC | 01:24 | |
| *** rajinir has quit IRC | 01:25 | |
| *** yangyapeng has joined #openstack-qa | 01:26 | |
| *** spzala has quit IRC | 01:29 | |
| *** spzala has joined #openstack-qa | 01:30 | |
| openstackrecheck | Console logs not available after 13:22s for gate-grenade-dsvm-neutron-ubuntu-trusty 355586,1,3d321cd | 01:30 |
| *** spzala has quit IRC | 01:34 | |
| *** EinstCrazy has joined #openstack-qa | 01:34 | |
| *** rfolco has quit IRC | 01:39 | |
| openstackgerrit | Merged openstack/tempest: Add missing config file read to tempest init https://review.openstack.org/353064 | 01:41 |
| *** nwkarsten has joined #openstack-qa | 01:43 | |
| *** amotoki has quit IRC | 01:43 | |
| openstackrecheck | Console logs not available after 13:21s for gate-cinder-python27-db-ubuntu-xenial 328052,43,e43ac7e | 01:43 |
| *** larainema has quit IRC | 01:49 | |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add optional H203 to check that assertIs(Not)None is used https://review.openstack.org/276517 | 01:50 |
| *** gyx has joined #openstack-qa | 01:51 | |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-full-devstack-plugin-ceph 355608,5,1014fee | 01:57 |
| *** yangyape_ has joined #openstack-qa | 01:59 | |
| *** EinstCrazy has quit IRC | 01:59 | |
| *** rbak has joined #openstack-qa | 02:02 | |
| *** yangyapeng has quit IRC | 02:03 | |
| *** EinstCrazy has joined #openstack-qa | 02:03 | |
| *** dimtruck is now known as zz_dimtruck | 02:05 | |
| *** zz_dimtruck is now known as dimtruck | 02:05 | |
| *** jamielennox is now known as jamielennox|away | 02:07 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-cells 317109,8,b482477 | 02:10 |
| *** EinstCra_ has joined #openstack-qa | 02:11 | |
| *** EinstCrazy has quit IRC | 02:14 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-cells 355659,1,5691e25 | 02:24 |
| *** jamielennox|away is now known as jamielennox | 02:26 | |
| *** mriedem is now known as mriedem_away | 02:27 | |
| *** fnaval has quit IRC | 02:30 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-cells 354336,1,b831fcb | 02:37 |
| *** fnaval has joined #openstack-qa | 02:44 | |
| *** rbak has quit IRC | 02:47 | |
| openstackrecheck | Console logs not available after 13:25s for gate-tempest-dsvm-cells 348317,3,4b5f901 | 02:50 |
| *** yamahata has quit IRC | 02:54 | |
| *** krtaylor has joined #openstack-qa | 03:02 | |
| *** yamahata has joined #openstack-qa | 03:03 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full-devstack-plugin-ceph 354293,6,4a002d8 | 03:04 |
| *** gema has quit IRC | 03:05 | |
| *** dimtruck is now known as zz_dimtruck | 03:07 | |
| *** nwkarste_ has joined #openstack-qa | 03:11 | |
| *** nwkarsten has quit IRC | 03:13 | |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-cells 355561,1,5ebb50d | 03:17 |
| *** nwkarste_ has quit IRC | 03:18 | |
| *** gema has joined #openstack-qa | 03:19 | |
| openstackgerrit | Masanori Itoh proposed openstack-dev/devstack: Add 'hitch' support for 'tls-proxy' option. https://review.openstack.org/355253 | 03:30 |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-cells 316398,37,b8177aa | 03:31 |
| openstackgerrit | zhufl proposed openstack/tempest: Add existing volumes when using pre-provisioned credentials https://review.openstack.org/353875 | 03:36 |
| *** yamahata has quit IRC | 03:43 | |
| openstackrecheck | Console logs not available after 13:24s for gate-grenade-dsvm-neutron-ubuntu-trusty 355666,1,0cc5d42 | 03:45 |
| *** ramishra has quit IRC | 03:45 | |
| *** rajinir has joined #openstack-qa | 03:45 | |
| *** nwkarsten has joined #openstack-qa | 03:46 | |
| *** ramishra has joined #openstack-qa | 03:51 | |
| *** nwkarsten has quit IRC | 03:52 | |
| *** nwkarsten has joined #openstack-qa | 03:53 | |
| *** yangyape_ has quit IRC | 03:56 | |
| *** nwkarsten has quit IRC | 03:57 | |
| *** yangyapeng has joined #openstack-qa | 03:58 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-full-devstack-plugin-ceph 322459,55,be97e2b | 03:58 |
| *** fnaval has quit IRC | 04:09 | |
| *** fnaval has joined #openstack-qa | 04:09 | |
| *** tqtran has joined #openstack-qa | 04:10 | |
| *** hichihara has joined #openstack-qa | 04:13 | |
| *** tqtran has quit IRC | 04:14 | |
| *** bkopilov has joined #openstack-qa | 04:25 | |
| *** bkopilov is now known as bkopilov_wfh | 04:25 | |
| *** tochi has quit IRC | 04:27 | |
| *** gcheresh has joined #openstack-qa | 04:28 | |
| *** chandankumar has joined #openstack-qa | 04:30 | |
| *** yfried has joined #openstack-qa | 04:32 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-cells 324784,13,a49195c | 04:37 |
| *** hieulq has quit IRC | 04:39 | |
| *** hieulq has joined #openstack-qa | 04:45 | |
| *** pgadiya has joined #openstack-qa | 04:48 | |
| *** tochi has joined #openstack-qa | 05:03 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-full-devstack-plugin-ceph 316398,37,e6662c6 | 05:03 |
| openstackrecheck | Console logs not available after 13:24s for gate-tempest-dsvm-cells 353807,5,b1a28a7 | 05:17 |
| *** _nadya_ has joined #openstack-qa | 05:19 | |
| *** _nadya_ has quit IRC | 05:24 | |
| openstackrecheck | Console logs not available after 13:22s for gate-tempest-dsvm-cells 354662,5,2a99c76 | 05:30 |
| *** hichihara has quit IRC | 05:36 | |
| *** sridharg has joined #openstack-qa | 05:46 | |
| *** r-mibu has quit IRC | 05:46 | |
| openstackrecheck | Console logs not available after 13:27s for gate-tempest-dsvm-full-devstack-plugin-ceph 354293,6,aa0f0f7 | 05:48 |
| openstackgerrit | guo yunxian proposed openstack/os-testr: Add support for Python versions https://review.openstack.org/355730 | 05:48 |
| openstackgerrit | guo yunxian proposed openstack/os-testr: Add support for Python versions https://review.openstack.org/355730 | 05:51 |
| gyx | We are now support python 3.5 in community, yes? | 05:53 |
| gyx | so I commit this patch | 05:53 |
| *** rajinir has quit IRC | 05:55 | |
| *** yangyape_ has joined #openstack-qa | 05:55 | |
| *** yangyapeng has quit IRC | 05:58 | |
| *** markvoelker has quit IRC | 05:58 | |
| *** gyx has quit IRC | 06:00 | |
| *** gyx has joined #openstack-qa | 06:00 | |
| openstackrecheck | Console logs not available after 13:23s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 353875,2,4d42805 | 06:01 |
| *** jpena|off has quit IRC | 06:02 | |
| *** jpena|off has joined #openstack-qa | 06:04 | |
| *** gvrangan has quit IRC | 06:06 | |
| *** r-mibu has joined #openstack-qa | 06:06 | |
| *** paul-carlton2 has joined #openstack-qa | 06:07 | |
| *** jpena|off has quit IRC | 06:10 | |
| *** jpena|off has joined #openstack-qa | 06:11 | |
| *** tqtran has joined #openstack-qa | 06:11 | |
| *** rcernin has joined #openstack-qa | 06:14 | |
| openstackrecheck | Console logs not available after 13:32s for gate-tempest-dsvm-cells 349666,6,377df44 | 06:15 |
| *** tqtran has quit IRC | 06:15 | |
| *** mewald has joined #openstack-qa | 06:22 | |
| *** edannon has joined #openstack-qa | 06:26 | |
| *** csomerville has quit IRC | 06:29 | |
| *** cody-somerville has joined #openstack-qa | 06:30 | |
| *** cody-somerville has quit IRC | 06:30 | |
| *** cody-somerville has joined #openstack-qa | 06:30 | |
| openstackgerrit | zhufl proposed openstack/tempest: Add existing volumes when using pre-provisioned credentials https://review.openstack.org/353875 | 06:30 |
| *** spzala has joined #openstack-qa | 06:31 | |
| openstackgerrit | garyk proposed openstack-dev/devstack: Enable neutron to work in a multi node setup https://review.openstack.org/355244 | 06:31 |
| *** spzala has quit IRC | 06:35 | |
| *** yangyape_ has quit IRC | 06:38 | |
| *** EinstCra_ has quit IRC | 06:38 | |
| *** bkopilov_wfh is now known as bkopilov | 06:39 | |
| *** yangyapeng has joined #openstack-qa | 06:40 | |
| *** EinstCrazy has joined #openstack-qa | 06:42 | |
| *** yangyape_ has joined #openstack-qa | 06:42 | |
| *** yangyapeng has quit IRC | 06:46 | |
| *** ihrachys has joined #openstack-qa | 06:47 | |
| *** gvrangan has joined #openstack-qa | 06:49 | |
| *** gyx has quit IRC | 06:50 | |
| *** martinkopec has joined #openstack-qa | 06:50 | |
| *** markvoelker has joined #openstack-qa | 06:51 | |
| *** nradojevic has joined #openstack-qa | 06:52 | |
| *** yamahata has joined #openstack-qa | 06:53 | |
| *** Lord_Phoenix has joined #openstack-qa | 06:54 | |
| *** yamahata has quit IRC | 07:02 | |
| *** jpena|off is now known as jpena | 07:07 | |
| *** gyx has joined #openstack-qa | 07:11 | |
| *** gyx has quit IRC | 07:18 | |
| *** gyx has joined #openstack-qa | 07:19 | |
| openstackrecheck | openstack/glance change: https://review.openstack.org/355534 failed because of: gate-tempest-dsvm-full-devstack-plugin-ceph: https://bugs.launchpad.net/bugs/1449136 | 07:26 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 07:26 |
| *** e0ne has joined #openstack-qa | 07:28 | |
| *** hichihara has joined #openstack-qa | 07:28 | |
| *** Poornima has joined #openstack-qa | 07:28 | |
| *** bauzas_off is now known as bauzas | 07:38 | |
| *** ifarkas_afk is now known as ifarkas | 07:40 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:55 | |
| *** pilgrimstack has joined #openstack-qa | 08:01 | |
| *** markvoelker has quit IRC | 08:01 | |
| *** paul-carlton2 has quit IRC | 08:04 | |
| *** pilgrimstack has quit IRC | 08:05 | |
| *** pilgrimstack has joined #openstack-qa | 08:07 | |
| *** Serlex has joined #openstack-qa | 08:18 | |
| *** sshnaidm has quit IRC | 08:21 | |
| *** lucas-dinner is now known as lucasagomes | 08:21 | |
| *** sshnaidm has joined #openstack-qa | 08:21 | |
| *** paul-carlton2 has joined #openstack-qa | 08:24 | |
| *** gvrangan has quit IRC | 08:26 | |
| *** mvk has quit IRC | 08:29 | |
| *** gcheresh_ has joined #openstack-qa | 08:32 | |
| *** gcheresh has quit IRC | 08:33 | |
| *** tochi has quit IRC | 08:35 | |
| *** cdent has joined #openstack-qa | 08:39 | |
| *** yaume has joined #openstack-qa | 08:40 | |
| *** yamamoto has quit IRC | 08:44 | |
| openstackrecheck | openstack/glance change: https://review.openstack.org/354659 failed because of: gate-tempest-dsvm-full-devstack-plugin-ceph: https://bugs.launchpad.net/bugs/1449136 | 08:47 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 08:47 |
| *** abehl has joined #openstack-qa | 08:54 | |
| *** mvk has joined #openstack-qa | 08:55 | |
| openstackgerrit | zhufl proposed openstack/tempest: Fix OverLimit's message https://review.openstack.org/355798 | 09:01 |
| *** markvoelker has joined #openstack-qa | 09:02 | |
| *** gfidente has joined #openstack-qa | 09:03 | |
| *** markvoelker has quit IRC | 09:07 | |
| lennyb | afazekas, pls review trivial fix #link https://review.openstack.org/#/c/355469/ .thanks | 09:10 |
| *** _nadya_ has joined #openstack-qa | 09:22 | |
| *** _nadya_ has quit IRC | 09:22 | |
| *** _nadya_ has joined #openstack-qa | 09:22 | |
| openstackgerrit | Vasyl Saienko proposed openstack-dev/grenade: Revert "Verify cellsv2 setup after upgrade" https://review.openstack.org/355813 | 09:30 |
| *** yamamoto has joined #openstack-qa | 09:31 | |
| *** kbaikov has quit IRC | 09:35 | |
| openstackgerrit | Benny Kopilov proposed openstack/tempest: Volume encrypted actions https://review.openstack.org/343571 | 09:38 |
| *** nwkarsten has joined #openstack-qa | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 09:40 | |
| *** yamamoto has quit IRC | 09:41 | |
| *** ramishra has quit IRC | 09:42 | |
| openstackrecheck | Console logs not available after 13:26s for periodic-tempest-dsvm-stress-master 353885,2,97c53ca | 09:43 |
| jroll | can I get some eyes on a devstack/cells patch that helps unbreak ironic's grenade job? https://review.openstack.org/#/c/355602/ | 09:44 |
| *** ramishra has joined #openstack-qa | 09:44 | |
| *** nwkarsten has quit IRC | 09:44 | |
| jroll | actually, I feel like that should have a merge conflict with https://github.com/openstack-dev/devstack/commit/9301e8875b8c50e486a3751b3d894d4ccdc94f84 | 09:45 |
| jroll | so maybe not urgent, ignore me :) | 09:45 |
| openstackgerrit | zhufl proposed openstack/tempest: Add InvalidConfiguration's error message https://review.openstack.org/353885 | 09:45 |
| *** yamamoto has joined #openstack-qa | 09:46 | |
| *** yamamoto has quit IRC | 09:46 | |
| *** jpena has quit IRC | 09:46 | |
| *** dmellado has quit IRC | 09:46 | |
| *** jpena has joined #openstack-qa | 09:47 | |
| *** yamamoto has joined #openstack-qa | 09:48 | |
| *** tosky has joined #openstack-qa | 09:49 | |
| *** hichihara has quit IRC | 09:54 | |
| *** dmellado has joined #openstack-qa | 09:56 | |
| *** raies has joined #openstack-qa | 09:57 | |
| *** ihrachys has quit IRC | 09:58 | |
| raies | gmann: Hi | 09:58 |
| openstackrecheck | Console logs not available after 13:26s for gate-tempest-dsvm-cells 355781,1,bdd2075 | 09:58 |
| raies | gmann: review comment incorporated https://review.openstack.org/#/c/353438/ | 09:59 |
| raies | please have a look | 09:59 |
| *** yangyape_ has quit IRC | 10:01 | |
| *** EinstCrazy has quit IRC | 10:01 | |
| *** markvoelker has joined #openstack-qa | 10:03 | |
| openstackgerrit | Benny Kopilov proposed openstack/tempest: Cinder multibackend migration tests https://review.openstack.org/318569 | 10:03 |
| *** EinstCrazy has joined #openstack-qa | 10:04 | |
| bkopilov | Hi afazekas , The patch for configuring tempest.conf cinder backends is commited to devstack , https://review.openstack.org/#/c/325895/ | 10:05 |
| bkopilov | afazekas, Before enabling multibackend support in tempest.conf , we have to add another cinder backend to upstream tests , where can i enable it ? today upstream uses a single lvm , i would like to add 2 lvm backends | 10:07 |
| *** markvoelker has quit IRC | 10:08 | |
| bkopilov | afazekas, in devstack (on local run on my laptop) there is a conf file and it should be like : CINDER_ENABLED_BACKENDS=lvm:lvm_1,lvm:lvm_2 | 10:08 |
| *** tqtran has joined #openstack-qa | 10:12 | |
| openstackrecheck | Console logs not available after 13:24s for gate-grenade-dsvm-neutron-ubuntu-trusty 318436,87,583240b | 10:13 |
| *** ihrachys has joined #openstack-qa | 10:14 | |
| *** yangyapeng has joined #openstack-qa | 10:16 | |
| *** tqtran has quit IRC | 10:17 | |
| *** yangyapeng has quit IRC | 10:22 | |
| *** sdague has joined #openstack-qa | 10:23 | |
| *** ihrachys has quit IRC | 10:23 | |
| afazekas | bkopilov, is it enough to change just the CINDER_ENABLED_BACKENDS ? | 10:24 |
| *** kbaikov has joined #openstack-qa | 10:24 | |
| bkopilov | afazekas, yes , it works locally on my laptop , by default its a single lvm | 10:24 |
| *** mvk has quit IRC | 10:25 | |
| *** yamamoto has quit IRC | 10:25 | |
| afazekas | bkopilov, https://github.com/openstack-infra/devstack-gate | 10:25 |
| afazekas | bkopilov, devstack-gate just have master branch, you might want to backport the devstack change | 10:26 |
| *** mvk has joined #openstack-qa | 10:26 | |
| afazekas | IMHO it can be default in all master job | 10:27 |
| *** spzala has joined #openstack-qa | 10:31 | |
| bkopilov | afazekas, ok , can i add here : https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh ? line 239 | 10:32 |
| afazekas | gcheresh_, IMHO the test is not too much different, but neutron full tempest run is already 10 minute longer than nova-network if we do not want to extend it to double infinite, considering the neutron repo is reasonable . What I really miss I do not see the neutron.tests.tempest.scenario.* running anywhere. | 10:33 |
| afazekas | bkopilov, IMHO yes, but we will see on the review | 10:34 |
| bkopilov | afazekas, ok , Thanks ! | 10:35 |
| *** spzala has quit IRC | 10:36 | |
| afazekas | gcheresh_ : IMHO you can propose the udp + security group + ipv6 to tempest now, but I would not be surprised if we would start moving things to neutron side in the near future | 10:38 |
| afazekas | andreaf, ^ | 10:39 |
| gcheresh_ | afazekas: andreaf Actually I will prefer to write as much as possible to neutron and not tempest, I just understood that the tests for the features that are already written in tempest should go to tempest, but if you say that such a test will better suit to neutron, I would like to write it in neutron git | 10:43 |
| *** EinstCrazy has quit IRC | 10:44 | |
| *** EinstCrazy has joined #openstack-qa | 10:45 | |
| *** EinstCrazy has quit IRC | 10:50 | |
| afazekas | gcheresh_, ok, I do not know the full history of the neutron-tempest tests, but IMHO we miss a dedicated job for 'neutron.tests.tempest.scenario.* ' or I am just blind .. | 10:50 |
| * afazekas will try to ask someone from the neutron team | 10:51 | |
| gcheresh_ | afazekas: Indeed, there is no such job at the moment, To test out new patches we run with " check experimental", but I think Kuba is working on making gate run scenario jobs in neutron, it will take time I guess | 10:56 |
| gcheresh_ | afazekas: andreaf another issue I raised here and didn't get an answer: https://review.openstack.org/#/c/344193/ fails on verification, is it a bug, should I rebase again or there is something else? | 10:59 |
| openstackgerrit | Benny Kopilov proposed openstack-infra/devstack-gate: Enable support for cinder multi-backend in tempest https://review.openstack.org/355846 | 11:01 |
| openstackgerrit | Benny Kopilov proposed openstack-infra/devstack-gate: Enable support for cinder multi-backend in tempest https://review.openstack.org/355846 | 11:03 |
| *** yamamoto has joined #openstack-qa | 11:03 | |
| afazekas | gcheresh_: you get reply from Elastic Recheck | 11:03 |
| *** markvoelker has joined #openstack-qa | 11:04 | |
| afazekas | gcheresh_, does it explains it ? | 11:04 |
| gcheresh_ | afazekas: yes, so it will be merged only when this bugs will be fixed? | 11:06 |
| afazekas | gcheresh_, Hopefully it was a temporary issue | 11:07 |
| gcheresh_ | afazekas: I did reverify 3 times and it always fails, so sadly it's not | 11:08 |
| *** markvoelker has quit IRC | 11:08 | |
| afazekas | http://status.openstack.org/elastic-recheck/#1449136 | 11:10 |
| afazekas | 12 failures in 24 hour | 11:11 |
| afazekas | Here is a recent working one : http://logs.openstack.org/91/353791/3/check/gate-tempest-dsvm-neutron-full-ubuntu-trusty-mitaka/5c6cb40/console.html | 11:12 |
| afazekas | Maybe worth checking are there any common thing in the failing slaves / cases | 11:13 |
| gcheresh_ | ok, I see you rechecked again, didn't refresh the page, so sent reverify too, let's see the result | 11:14 |
| *** gyx has quit IRC | 11:20 | |
| *** gyx has joined #openstack-qa | 11:20 | |
| openstackrecheck | Console logs not available after 13:27s for gate-tempest-dsvm-full-lio-src-os-brick 354576,3,7fa9cf1 | 11:24 |
| *** ramishra has quit IRC | 11:30 | |
| *** yangyapeng has joined #openstack-qa | 11:32 | |
| *** ramishra has joined #openstack-qa | 11:32 | |
| *** yangyapeng has quit IRC | 11:36 | |
| openstackgerrit | Igor Milovanovic proposed openstack-dev/devstack: Fixed language. https://review.openstack.org/352392 | 11:39 |
| openstackrecheck | Console logs not available after 13:37s for gate-swift-tox-func-ubuntu-xenial 318441,87,bac591a | 11:41 |
| *** rfolco has joined #openstack-qa | 11:41 | |
| *** rodrigods has quit IRC | 11:51 | |
| *** rodrigods has joined #openstack-qa | 11:51 | |
| *** sshnaidm is now known as sshnaidm|lnch | 11:52 | |
| *** jpena is now known as jpena|lunch | 11:57 | |
| openstackrecheck | Console logs not available after 13:43s for gate-tempest-dsvm-full-devstack-plugin-ceph 290793,6,ccd7613 | 11:59 |
| openstackrecheck | openstack/glance change: https://review.openstack.org/353243 failed because of: gate-tempest-dsvm-full-devstack-plugin-ceph: https://bugs.launchpad.net/bugs/1449136 | 12:00 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 12:00 |
| openstackgerrit | Benny Kopilov proposed openstack/tempest: Volume encrypted actions https://review.openstack.org/343571 | 12:04 |
| *** markvoelker has joined #openstack-qa | 12:05 | |
| *** sigmavirus|away is now known as sigmavirus | 12:05 | |
| *** lucasagomes is now known as lucas-hungry | 12:06 | |
| *** mriedem_away has quit IRC | 12:07 | |
| *** markvoelker has quit IRC | 12:09 | |
| *** yaume has quit IRC | 12:13 | |
| openstackgerrit | Masanori Itoh proposed openstack-dev/devstack: Add 'hitch' support for 'tls-proxy' option. https://review.openstack.org/355253 | 12:14 |
| *** weshay has joined #openstack-qa | 12:17 | |
| openstackrecheck | Console logs not available after 13:37s for gate-tempest-dsvm-full-devstack-plugin-ceph 334860,8,de715b8 | 12:18 |
| *** yfried has quit IRC | 12:20 | |
| *** markvoelker has joined #openstack-qa | 12:26 | |
| *** yangyapeng has joined #openstack-qa | 12:30 | |
| *** gouthamr has joined #openstack-qa | 12:32 | |
| *** yamamoto has quit IRC | 12:33 | |
| openstackgerrit | Liron Kuchlani proposed openstack/tempest: Update volume type details https://review.openstack.org/355898 | 12:34 |
| openstackgerrit | Benny Kopilov proposed openstack-infra/devstack-gate: Enable support for cinder multi-backend in tempest https://review.openstack.org/355846 | 12:36 |
| openstackrecheck | Console logs not available after 13:36s for gate-tempest-dsvm-cells 355805,2,b7291d0 | 12:39 |
| *** yamamoto has joined #openstack-qa | 12:39 | |
| *** gyx has quit IRC | 12:41 | |
| *** rlrossit has joined #openstack-qa | 12:43 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** rbak has joined #openstack-qa | 12:44 | |
| *** raildo has joined #openstack-qa | 12:45 | |
| *** rbak has quit IRC | 12:48 | |
| *** bswartz has joined #openstack-qa | 12:50 | |
| openstackrecheck | Console logs not available after 13:33s for gate-tempest-dsvm-cells 346966,6,5a03e7f | 12:52 |
| *** jpena|lunch is now known as jpena | 12:56 | |
| *** woodster_ has joined #openstack-qa | 13:04 | |
| *** yamamoto has joined #openstack-qa | 13:07 | |
| *** yamamoto has quit IRC | 13:07 | |
| openstackrecheck | Console logs not available after 13:27s for gate-tempest-dsvm-full-devstack-plugin-ceph 355798,1,24fb131 | 13:08 |
| *** sshnaidm|lnch is now known as sshnaidm | 13:09 | |
| *** lucas-hungry is now known as lucasagomes | 13:10 | |
| *** edmondsw has joined #openstack-qa | 13:11 | |
| *** lifeless has quit IRC | 13:11 | |
| *** mriedem has joined #openstack-qa | 13:12 | |
| *** porrua has joined #openstack-qa | 13:13 | |
| openstackgerrit | Liron Kuchlani proposed openstack/tempest: Update volume type details https://review.openstack.org/355898 | 13:13 |
| *** spzala_ has joined #openstack-qa | 13:19 | |
| openstackgerrit | Yuriy Nesenenko proposed openstack/tempest: Only admin can get actions with quotas https://review.openstack.org/353936 | 13:21 |
| *** dustins has joined #openstack-qa | 13:22 | |
| openstackrecheck | Console logs not available after 13:26s for gate-cinder-pep8-ubuntu-xenial 355836,1,3dfb2f2 | 13:23 |
| mriedem | gmann: not sure if you saw but i had to make a 2 line change in https://review.openstack.org/#/c/327191/ for the new request schema format | 13:25 |
| mriedem | gmann: we pass networks='auto' and networks='none' now rather than a list of dicts | 13:25 |
| *** Poornima has quit IRC | 13:26 | |
| *** lifeless has joined #openstack-qa | 13:29 | |
| *** jpena is now known as jpena|brb | 13:33 | |
| *** jecarey_ has quit IRC | 13:34 | |
| sdague | mtreinish: you up yet? | 13:34 |
| sdague | the image http tests in tempest seem to be high failure rate, because git.openstack.org connectivity issues | 13:36 |
| openstackrecheck | Console logs not available after 13:28s for gate-grenade-dsvm-neutron-ubuntu-trusty 318436,87,ab22f7b | 13:36 |
| *** dkranz has joined #openstack-qa | 13:36 | |
| *** nwkarsten has joined #openstack-qa | 13:40 | |
| *** hichihara has joined #openstack-qa | 13:41 | |
| *** ramishra has quit IRC | 13:44 | |
| *** ramishra has joined #openstack-qa | 13:44 | |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-full-lio-src-os-brick 354576,4,17ee218 | 13:50 |
| *** yamahata has joined #openstack-qa | 13:55 | |
| *** jpena|brb is now known as jpena | 13:58 | |
| *** pgadiya has quit IRC | 14:00 | |
| *** Lord_Phoenix has quit IRC | 14:02 | |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-neutron-src-glance_store 355850,1,c679c65 | 14:04 |
| *** amuller has joined #openstack-qa | 14:06 | |
| amuller | afazekas: ping, re: your conversation with gcheresh_ | 14:06 |
| *** yamamoto has joined #openstack-qa | 14:07 | |
| *** gvrangan has joined #openstack-qa | 14:08 | |
| *** jecarey has joined #openstack-qa | 14:08 | |
| *** hichihara has quit IRC | 14:08 | |
| *** armax has joined #openstack-qa | 14:10 | |
| mtreinish | sdague: I am now | 14:10 |
| mtreinish | sdague: we can change that to another address, but we picked that originally because it was supposed to be more reliable than others | 14:11 |
| sdague | yeh, it seems really explody | 14:11 |
| sdague | honestly, it's a glance v1 only call right? | 14:11 |
| *** mfisher_ora has joined #openstack-qa | 14:11 | |
| *** jecarey has quit IRC | 14:12 | |
| mtreinish | sdague: I think so, not by design though, more because no one ever bothered to write a v2 test for it | 14:13 |
| *** yamamoto has quit IRC | 14:13 | |
| *** tqtran has joined #openstack-qa | 14:14 | |
| afazekas | amuller, hi | 14:15 |
| sdague | well, earlier today I could get to the fail rates for that on openstack health, but now http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.ListImagesTest is blank | 14:15 |
| sdague | anyway, if you spot check the gate, there are 3 images tests are are borked and they look like connectivity to git.o.o is part of it | 14:16 |
| *** jecarey has joined #openstack-qa | 14:16 | |
| amuller | afazekas: quick note, neutron.tests.scenario are very new, there is an experimental job that runs those tests | 14:16 |
| *** pgadiya has joined #openstack-qa | 14:16 | |
| amuller | afazekas: but in any case, about writing new tests in the tempest or neutron repo | 14:16 |
| amuller | afazekas: mtreinish, armax and I came to an understanding, and that was new tests will be written to either of the two repos according to the API the tests target | 14:16 |
| amuller | afazekas: If it's a base resource: port, network, subnet, floating IP, router or security group, then the test will be written in the Tempest repo | 14:17 |
| amuller | otherwise, to Neutron | 14:17 |
| amuller | afazekas: mtreinish: After reading your conversation with gcheresh_ I sense that things may have changed on your side? | 14:17 |
| *** _AmitGalitz_ has quit IRC | 14:18 | |
| amuller | afazekas: BTW these guidelines are documented here http://docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests | 14:18 |
| *** tqtran has quit IRC | 14:18 | |
| mtreinish | sdague: you need the test method too: http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image | 14:19 |
| afazekas | amuller: The topic was scenario test for actual ipv6 udp filtering | 14:19 |
| amuller | right | 14:19 |
| mtreinish | sdague: but yeah you can see the spike in failures, and the run time too | 14:19 |
| sdague | mriedem: ok, so that doesn't help if the test dies in setUp | 14:19 |
| mriedem | NO | 14:19 |
| mtreinish | sdague: you use setUpClass instead of the test method | 14:19 |
| mtreinish | sdague: have you mentioned this in -infra? Because if the nodes are having trouble hitting git.o.o that makes everything bad | 14:20 |
| *** asselin has joined #openstack-qa | 14:21 | |
| sdague | mtreinish: I did not, it was brought up there | 14:21 |
| amuller | afazekas: Scenario tests should be split according to the same guidelines | 14:21 |
| mtreinish | fungi, pleia2: ^^^ | 14:21 |
| sdague | but, this isn't something I can chase today | 14:21 |
| pleia2 | I can't look today, stepping out to meet up with some folks soon | 14:22 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Make everything plural https://review.openstack.org/355967 | 14:23 |
| mtreinish | mriedem: ^^^ should be fast | 14:23 |
| fungi | mtreinish: sdague: i'm probably misparsing scrollback, but you say there are three tests in tempest failing because they perform remote git operations? i didn't think individual tests reached out to our git mirrors | 14:24 |
| fungi | or am i conflating several discussions when the ^^^ only referred to some specific thing | 14:24 |
| fungi | ? | 14:24 |
| mtreinish | fungi: one of the glance tests needs an http target we use the git index page as something that's supposed to be reliable | 14:24 |
| fungi | ahh | 14:25 |
| *** hongbin has joined #openstack-qa | 14:25 | |
| afazekas | amuller: So you prefer tempest for the test we discussed with andreaf and gcheresh_ , and according to the doc it is the good place | 14:25 |
| mtreinish | fungi: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n417 | 14:25 |
| fungi | mtreinish: sdague: okay, so when we perform git operations we retry on failure because network operations outside the job node are unreliable | 14:25 |
| mtreinish | fungi: err, not index page the openstack logo :) | 14:26 |
| mriedem | mtreinish: whack that mole | 14:26 |
| mtreinish | I guess it really is an image | 14:26 |
| mtreinish | mriedem: I think I got it this time | 14:26 |
| fungi | mtreinish: can those tests hit a local service on the job node instead? | 14:26 |
| * mtreinish crosses fingers | 14:26 | |
| mtreinish | fungi: there were issues with that in the past (it's been long enough I don't remember the details) | 14:26 |
| mtreinish | which is why we switched | 14:26 |
| mtreinish | fungi: fwiw, it's only gotten bad in the past couple days: http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image | 14:28 |
| fungi | mtreinish: from any particular provider/region, or consistently distributed http(s?) failures from everywhere? | 14:29 |
| fungi | one particular provider having network issues for the past few days could certainly account for something like that, and isn't at all unheard of | 14:29 |
| *** tosky has quit IRC | 14:30 | |
| afazekas | amuller: IMHO in 6-12 month the neutron-full job will take too long time, and we might need to figure out something. | 14:30 |
| mtreinish | fungi: err, give me a sec. The data is there, but we don't graph that today. I'll check es for the failures (that'll probably be faster) | 14:31 |
| amuller | afazekas: To be clear, my preferance would be not to have *any* neutron tests in the tempest repo and have everything in neutron | 14:31 |
| *** zz_dimtruck is now known as dimtruck | 14:31 | |
| mtreinish | fungi: it looks like all bluebox | 14:32 |
| fungi | mtreinish: ooh, that helps. it only got brought back online very recently | 14:32 |
| amuller | afazekas: but I wanted to see if our previous understanding changed | 14:33 |
| fungi | it's possible our uplink from that environment (it's basically a small public cloud deployment) could be very constrained or something | 14:33 |
| mtreinish | fungi: at least that's what: | 14:33 |
| mtreinish | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22The%20HTTP%20URL%20is%20invalid%3A%20HTTPConnectionPool(host%3D'git.openstack.org'%2C%20port%3D80)%3A%20Max%20retries%20exceeded%20with%20url%3A%20%2Fstatic%2Fopenstack.png%20(Caused%20by%20NewConnectionError('%3A%20Failed%20to%20establish%20a%20new%20connection%3A%20%5BErrno%20-2%5D%20Name%20or%20service%20not%20know | 14:33 |
| mtreinish | n'%2C))%5C%22 | 14:33 |
| mtreinish | ugh, paste fail | 14:33 |
| *** tosky has joined #openstack-qa | 14:33 | |
| mtreinish | shows | 14:33 |
| *** rajinir has joined #openstack-qa | 14:34 | |
| afazekas | amuller: AFAIK no, but it would be good topic for mid cycle or summit | 14:34 |
| mtreinish | fungi: http://goo.gl/z8hTff | 14:34 |
| mtreinish | that's better :) | 14:34 |
| *** pgadiya has quit IRC | 14:36 | |
| afazekas | amuller: The linked doc expected to be valid, Just I did not read it before | 14:36 |
| sdague | mtreinish: so, follow up question, has this test ever legit caught a fail? Because I feel like we end up with a big burst of false negatives once a year. | 14:37 |
| fungi | mtreinish: looking at the trending for that query, looks like only ~7 runs were impacted in the last week (it's matching multiple lines in each) | 14:38 |
| fungi | most instances reported the same error ~4 times in rapid succession | 14:39 |
| mtreinish | fungi: yeah it's twice per test, there are a couple that do the http thing | 14:41 |
| mtreinish | fungi: http://logs.openstack.org/03/355303/1/gate/gate-tempest-dsvm-neutron-dvr/a2cff45/console.html#_2016-08-16_12_14_06_942044 | 14:41 |
| *** xinli has joined #openstack-qa | 14:44 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1613749 https://review.openstack.org/355988 | 14:44 |
| openstack | bug 1613749 in OpenStack-Gate "Git timeouts from bluebox" [Undecided,New] https://launchpad.net/bugs/1613749 | 14:44 |
| mtreinish | mriedem: ^^^ that might be too specific | 14:44 |
| fungi | it definitely seems like something is going on in bluebox, though if it's happening on average once a day we're likely to have trouble reproducing it reliably | 14:44 |
| mtreinish | fungi: sdague said it's hitting some things in the gate right now | 14:45 |
| mtreinish | fungi: oh it looks like there is another failure mode on osic: http://logs.openstack.org/55/352455/3/gate/gate-tempest-dsvm-cells/7881266/console.html#_2016-08-16_13_38_04_783682 | 14:47 |
| amuller | afazekas: OK, should gcheresh_ send the test to tempest then from your perspective? | 14:48 |
| mtreinish | fungi: that looks like the majority of the failures (I had to pull my pending patch to list failures by test on o-h to see it) | 14:48 |
| mtreinish | well except that's localhost | 14:48 |
| fungi | osic is recently unique in that it's switched to ipv6-only for global addressing | 14:50 |
| mtreinish | fungi: well that would explain all the failures for tests using 127.0.0.1 | 14:50 |
| mtreinish | but I wonder where that address is coming from | 14:50 |
| fungi | hrm, 127.0.0.1 isn't global so shouldn't be impacted | 14:51 |
| mtreinish | fungi: anyway it looks like there are 2 failure modes here, the bluebox can't reach git, and the osic one trying to hit 127.0.0.1 | 14:51 |
| fungi | i wonder if whatever is supposed to be binding on 9292/tcp is thrown off by lack of non-rfc1918 addresses? | 14:52 |
| fungi | that looks like a made-up port number. it's not in the ephemeral ports range but it's also not registered with iana | 14:52 |
| mtreinish | oh, that's the connection to glance... | 14:53 |
| fungi | glance listens on 9292/tcp? | 14:53 |
| fungi | weird | 14:53 |
| Jokke_ | fungi: and 9191 | 14:54 |
| fungi | i guess that's a devstackism to be able to run all these independent rest apis in an aio install | 14:54 |
| *** gcheresh_ has quit IRC | 14:55 | |
| fungi | so is glance possibly failing to start in these cases? failing to listen on ipv4? could it be only binding on ::1 instead of 127.0.0.1? | 14:55 |
| fungi | or is it supposed to listen on all configured addresses? | 14:55 |
| mtreinish | fungi: the bind_host is set to 0.0.0.0 http://logs.openstack.org/55/352455/3/gate/gate-tempest-dsvm-cells/7881266/logs/screen-g-api.txt.gz#_2016-08-16_12_52_15_208 | 14:58 |
| fungi | mtreinish: oh, i wonder if that shouldn't be :: instead of 0.0.0.0 | 14:58 |
| fungi | regardless, it should have bound to all available ipv4 addresses which should include (at a minimum) 127.0.0.1 | 14:59 |
| *** yaume has joined #openstack-qa | 15:00 | |
| mtreinish | fungi: right, and the log shows it accepting connections from 127.0.0.1 | 15:02 |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-full-devstack-plugin-ceph 323661,3,df1eaaf | 15:02 |
| fungi | mtreinish: up at the top of that console log, we do have some network diagnostics | 15:02 |
| *** jlwhite has quit IRC | 15:03 | |
| *** antwash has quit IRC | 15:03 | |
| *** castulo has quit IRC | 15:03 | |
| fungi | you can see eth0 has 10.0.60.80/8, 2001:4800:1ae1:18:3501:cd25:145a:926/64, 2001:4800:1ae1:18:f816:3eff:feb1:4989/64 and fe80::f816:3eff:feb1:4989/64 bound at the time the job started | 15:03 |
| *** luzC has quit IRC | 15:03 | |
| fungi | and it does have global routes on eth0 through 10.0.0.1 and fe80::def | 15:04 |
| *** luzC has joined #openstack-qa | 15:04 | |
| *** ifarkas is now known as ifarkas_afk | 15:04 | |
| *** abehl has quit IRC | 15:04 | |
| fungi | and the arp/neighbor table does show both of those gateways resolving to reachable hardware addresses | 15:05 |
| fungi | though traceroute6 to the outside doesn't appear to work there | 15:05 |
| *** martinkopec has quit IRC | 15:06 | |
| *** pglass has joined #openstack-qa | 15:08 | |
| *** nradojevic has quit IRC | 15:08 | |
| *** raies has quit IRC | 15:08 | |
| *** hockeynut has joined #openstack-qa | 15:10 | |
| mtreinish | fungi: hmm, I'm looking at the glance logs and I see a connection established from 127.0.0.1 around the exact time of the first timeout | 15:10 |
| mtreinish | fungi: http://logs.openstack.org/55/352455/3/gate/gate-tempest-dsvm-cells/7881266/logs/screen-g-api.txt.gz#_2016-08-16_13_07_19_907 (which is 1 min before the first 60 second timeout is logged on the same url) | 15:11 |
| mtreinish | but, I can't verify it's the actual request since tempest does run in parallel | 15:11 |
| *** Poornima has joined #openstack-qa | 15:11 | |
| *** edannon has quit IRC | 15:13 | |
| *** castulo has joined #openstack-qa | 15:13 | |
| *** vhosakot has joined #openstack-qa | 15:13 | |
| *** antwash has joined #openstack-qa | 15:13 | |
| *** jlwhite has joined #openstack-qa | 15:14 | |
| *** gvrangan has quit IRC | 15:14 | |
| *** hockeynu_ has joined #openstack-qa | 15:15 | |
| openstackgerrit | Merged openstack/openstack-health: Add failed runs list to test page https://review.openstack.org/352569 | 15:16 |
| afazekas | amuller: based on the doc, yes ; andreaf do you agree ? | 15:16 |
| amuller | afazekas: I don't want Genadi to do double work | 15:17 |
| amuller | afazekas: if you think there's going to be a move of tests to the neutron repo in the future... | 15:17 |
| *** xinli has quit IRC | 15:17 | |
| *** hockeynut has quit IRC | 15:18 | |
| *** fnaval has quit IRC | 15:20 | |
| *** mtanino has joined #openstack-qa | 15:20 | |
| afazekas | amuller, Personally I would be ok with both repo for _new_ scenario test. We can solve timing things in an another way (For Ex. create first half and second half tempest job) . we will see what happens in the future, but for now we have a written doc | 15:24 |
| cdent | Is there a way in openstack-health to visualize the time test runs are taking? Not indvididual tests, but a whole job. What I want to see is if there's been a change from A to lots-more-than-A | 15:24 |
| timothyb89 | cdent: for most tempest jobs you can see timing details in the stackviz output | 15:25 |
| mtreinish | timothyb89: I think it was more an aggregate tracking to see spikes | 15:26 |
| mtreinish | cdent: not today, I've been fighting that problem a bit in my spare time. The biggest issue is the db doesn't track the actual run time, just the sum of the tests in a run | 15:26 |
| mtreinish | so to graph the actual wall time of the run I need to find the first and last tests in a run and subtract the times | 15:26 |
| mtreinish | which is expensive to query for a lot of runs | 15:27 |
| mtreinish | I think the fastest path is to change the db to store the wall time when we populate it | 15:27 |
| mtreinish | which I haven't had time to do yet | 15:27 |
| mtreinish | cdent: heh, so the short answer is not yet :) | 15:27 |
| cdent | thanks for the detailed response mtreinish | 15:27 |
| cdent | I agree that recording the explicit time makes sense | 15:28 |
| * cdent looks around for clones, yet again | 15:28 | |
| mtreinish | cdent: actually the patch should be pretty easy: https://github.com/openstack-infra/subunit2sql/blob/master/subunit2sql/read_subunit.py#L138-L142 | 15:30 |
| mtreinish | that function just needs to be changed to not do a sum and instead do the delta between the start time on the first test run and the end time of the last test | 15:30 |
| mtreinish | there might be some backwards incompat concerns (if people are depending on the current behavior) so maybe make it an opt in thing | 15:31 |
| mtreinish | and we can just always use that in the infra db | 15:31 |
| * cdent adds a non-commital but hopeful not to his todo list | 15:35 | |
| *** paul-carlton2 has quit IRC | 15:35 | |
| *** gvrangan has joined #openstack-qa | 15:36 | |
| *** tosky_ has joined #openstack-qa | 15:37 | |
| *** _nadya_ has quit IRC | 15:38 | |
| chandankumar | https://github.com/openstack/tempest/blob/master/tox.ini#L134 | 15:39 |
| *** xinli has joined #openstack-qa | 15:39 | |
| chandankumar | i have a small query here what is {posargs} ? | 15:39 |
| *** tosky has quit IRC | 15:40 | |
| mtreinish | chandankumar: positional args: http://tox.readthedocs.io/en/1.0/example/general.html#interactively-passing-positional-arguments | 15:40 |
| *** amotoki has joined #openstack-qa | 15:40 | |
| *** amotoki has quit IRC | 15:43 | |
| amuller | afazekas: OK, thank you | 15:43 |
| *** amotoki has joined #openstack-qa | 15:43 | |
| *** fnaval has joined #openstack-qa | 15:43 | |
| *** porrua has quit IRC | 15:44 | |
| openstackrecheck | Console logs not available after 14:01s for gate-cinder-python34-db 337061,8,94e3f6a | 15:44 |
| *** rcernin has quit IRC | 15:46 | |
| afazekas | amuller, thank you | 15:46 |
| chandankumar | mtreinish, Thanks :-) | 15:48 |
| mugsie | is there anything custom about the setup of the default grenade gate, that plugins should be setting in theirs? | 15:48 |
| openstackgerrit | Matthew Van Dijk proposed openstack-dev/devstack: Make the Neutron l3 plugin use the subnetpools https://review.openstack.org/356026 | 15:49 |
| *** e0ne has quit IRC | 15:49 | |
| mugsie | we are seeing failures in our gate, that are not being caught by the main one | 15:49 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Fix typo in test_attr_list handling https://review.openstack.org/355385 | 15:53 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting https://review.openstack.org/355393 | 15:53 |
| *** hieulq_ has joined #openstack-qa | 15:55 | |
| mtreinish | andreaf: ^^^ that 1 char patch got much bigger... | 15:56 |
| andreaf | mtreinish: heh yes so the entire process_results function has no unittest coverage | 15:56 |
| andreaf | mtreinish: I still have a couple of unit tests to write | 15:57 |
| openstackgerrit | Karthik Prabhu Vinod proposed openstack-dev/grenade: Check if c-vol is enabled https://review.openstack.org/355678 | 15:57 |
| mtreinish | andreaf: heh, yeah I wasn't so good about writing tests when I started subunit2sql | 15:57 |
| mtreinish | and that code hasn't really been touched much since | 15:58 |
| andreaf | I also filed https://storyboard.openstack.org/?#!/story/2000702 and https://storyboard.openstack.org/?#!/story/2000701 | 15:58 |
| openstackrecheck | Console logs not available after 13:35s for gate-tempest-dsvm-full-ceph-plugin-src-os-brick 355765,1,6601c26 | 15:58 |
| *** kbaikov has quit IRC | 15:59 | |
| andreaf | mtreinish: unit testing process_results requires a lot of mocking, so the tests are quite long in terms code, but I think they are valuable | 16:00 |
| andreaf | mtreinish: let me know what you think | 16:00 |
| *** yaume has quit IRC | 16:00 | |
| mtreinish | andreaf: yeah it lgtm, I'm leaving some quick comments inline. But I agree it's good to do | 16:01 |
| andreaf | mtreinish: cool thanks | 16:02 |
| *** venkat has joined #openstack-qa | 16:02 | |
| mriedem | oomichi: mtreinish: i'm going to propose that we make gate-tempest-dsvm-multinode-live-migration voting for nova, check and gate, | 16:05 |
| mriedem | i see it's in the experimental queue for tempest | 16:05 |
| mriedem | i'm wondering if we can add it to the tempest runs, it's skipped on stable and only runs the live migration tests, so it takes ~30 minutes | 16:05 |
| mtreinish | mriedem: we probably want it to be symmetric right? | 16:05 |
| mtreinish | we have a little room now because we're down to 2 stable branches running instead of 3 so an extra job on tempest shouldn't add too much stress | 16:06 |
| mriedem | ok, check only or both? | 16:06 |
| mriedem | i think project-config will require both check and gate if it's voting | 16:06 |
| mugsie | is there anyway for us to remove a project from being tested in a grenade test run? | 16:07 |
| *** amotoki has quit IRC | 16:07 | |
| mtreinish | well we have a lot of voting check only jobs on tempest | 16:07 |
| mriedem | mugsie: project or test? | 16:08 |
| mtreinish | but if it has to be both that's fine I guess | 16:08 |
| mriedem | mtreinish: oh yeah i guess that's true | 16:08 |
| mugsie | project preferably, but I suppose I could just blockl thier tempest tests | 16:08 |
| *** sridharg has quit IRC | 16:09 | |
| mriedem | mugsie: service_available.project = False in tempest.conf | 16:09 |
| mriedem | $project | 16:09 |
| mtreinish | mriedem: it just feels like we'd be overloading an already overworked gate | 16:09 |
| mriedem | i'll just omit from tempest for now | 16:10 |
| *** hockeynu_ has quit IRC | 16:10 | |
| mugsie | mriedem: is there a devstack var I can set to do this? it needs to be blocked in our gate | 16:10 |
| *** sshnaidm is now known as sshnaidm|afk | 16:11 | |
| mriedem | mtreinish: speaking of overworked, have you noticed that gate-tempest-dsvm-neutron-multinode-full takes about 90 minutes? | 16:11 |
| mriedem | mugsie: which project/service are you trying to disable? | 16:11 |
| mugsie | glance | 16:11 |
| mriedem | heh, so you don't plan on using nova either? | 16:12 |
| mtreinish | mriedem: I had not, but I could see that. At a min the node setup takes longer (not saying that's all it is) | 16:12 |
| mriedem | or any scenario tests | 16:12 |
| mugsie | mriedem: I want to unblock the gate :) | 16:12 |
| *** gvrangan has quit IRC | 16:12 | |
| mugsie | seen as glances upgrade is not hugely relevent to how well designate has upgraded, I am OK wiht that | 16:13 |
| *** paul-carlton2 has joined #openstack-qa | 16:14 | |
| mriedem | well, i'm not aware of a devstack variable for this, and you can't pass local conf through from the project-config job def to devstack yet | 16:14 |
| mriedem | b/c devstack-gate doesn't support that yet | 16:14 |
| mriedem | did glance break upgrades? | 16:14 |
| openstackrecheck | Console logs not available after 14:51s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 349952,3,1e400f7 | 16:14 |
| mriedem | mugsie: maybe a revert on the glance side, or a blacklist of a bad glance_store version in g-r? | 16:14 |
| mugsie | well, glance broke grenade for designate anyway | 16:15 |
| mriedem | do you know what the breaking change is? | 16:15 |
| mugsie | not yet | 16:15 |
| mugsie | glance are looking at it | 16:15 |
| *** xinli has quit IRC | 16:15 | |
| mriedem | have a link to a failure? | 16:15 |
| mriedem | logstash could help pin point when it started failing | 16:15 |
| mugsie | http://logs.openstack.org/35/348535/5/check/gate-grenade-dsvm-designate-powerdns/e7746e5/console.html#_2016-08-15_22_29_38_505947 | 16:15 |
| mriedem | hmm, i saw that on stable branches last night too | 16:16 |
| mtreinish | that's affecting everyone right now | 16:16 |
| mugsie | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Max%20retries%20exceeded%20with%20url%3A%20%2Fv1%2Fimages%5C%22%20AND%20build_status%3AFAILURE%20AND%20build_name%3A*grenade* | 16:16 |
| mtreinish | https://bugs.launchpad.net/openstack-gate/+bug/1613749 | 16:16 |
| openstack | Launchpad bug 1613749 in OpenStack-Gate "Git timeouts from bluebox" [Undecided,New] | 16:16 |
| mtreinish | I need to update it, because it's also osic (it manifests in a slightly different way on osic) | 16:17 |
| mugsie | mtreinish: different error message | 16:17 |
| mugsie | or is that the osic difference? | 16:17 |
| mtreinish | it is | 16:17 |
| mtreinish | it's timing out waiting on glance on osic, while on bluebox glance returns a failure | 16:18 |
| mtreinish | but in both cases its when we ask glance to fetch somethign from git.o.o | 16:18 |
| sdague | mtreinish: we no longer have dstat in human readable format - http://logs.openstack.org/10/352610/4/gate/gate-tempest-dsvm-cells/a663b0a/logs/dstat-csv.txt.gz ? | 16:18 |
| *** dtantsur is now known as dtantsur|afk | 16:18 | |
| mtreinish | sdague: we do: http://logs.openstack.org/10/352610/4/gate/gate-tempest-dsvm-cells/a663b0a/logs/screen-dstat.txt.gz | 16:18 |
| *** chandankumar has quit IRC | 16:19 | |
| sdague | ah, great | 16:19 |
| mugsie | mtreinish: the only projects with failres from the grenade tests is designate | 16:19 |
| mtreinish | mugsie: it's not limited to grenade | 16:19 |
| mtreinish | although I think the path sdague is going down right now is that the failures are load related? | 16:20 |
| sdague | mugsie: here is the suspicious thing | 16:21 |
| sdague | those 3 test causes seem coupled | 16:21 |
| sdague | in failing all together | 16:21 |
| sdague | and they all have one of these - http://logs.openstack.org/35/348535/5/check/gate-grenade-dsvm-designate-powerdns/e7746e5/logs/new/screen-g-api.txt.gz?level=WARNING#_2016-08-15_22_15_07_925 | 16:22 |
| sdague | which... also has a keystone entry that a token can't be found | 16:22 |
| mtreinish | mugsie: see: http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image and you'll see that test is failing lots of different jobs | 16:23 |
| *** pilgrimstack has quit IRC | 16:23 | |
| *** yamahata has quit IRC | 16:23 | |
| *** venkat has quit IRC | 16:23 | |
| mugsie | mtreinish: yeah, I saw that. but no other grenade job is failing with them :/ | 16:23 |
| sdague | and 36 hours ago keystone merged a really giant logging patch, which ends up making keystone be under a higher load processing logs | 16:23 |
| mugsie | like, at all | 16:23 |
| mugsie | oh, i saw you talking about that in -dev i think | 16:24 |
| sdague | mtreinish: .... also notable, mugsie | 16:24 |
| *** porrua has joined #openstack-qa | 16:24 | |
| sdague | mugsie's run, and my failed run (same way) are osic | 16:24 |
| mtreinish | sdague: right it only is failing on osic and bluebox | 16:25 |
| sdague | I wonder if there is a provider speed issue | 16:25 |
| sdague | so bluebox was on the slow side | 16:25 |
| mtreinish | we assumed it was a networking issue this morning, but it could just be a speed thing | 16:25 |
| mugsie | back to the old HP vs RAX rate limiting? | 16:25 |
| sdague | especially because keystone is now often burning a whole core | 16:25 |
| sdague | in the dstat you can see it at 10% cpu | 16:25 |
| sdague | and often the highest cpu using process | 16:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1613749 https://review.openstack.org/355988 | 16:28 |
| openstack | bug 1613749 in OpenStack-Gate "Git timeouts from bluebox" [Undecided,New] https://launchpad.net/bugs/1613749 | 16:28 |
| *** Suyi has joined #openstack-qa | 16:28 | |
| mtreinish | mriedem: ^^^ | 16:28 |
| openstackrecheck | Console logs not available after 13:35s for gate-grenade-dsvm-neutron-ubuntu-trusty 355586,1,2f805d6 | 16:29 |
| mtreinish | sdague: well we'll know for sure if that's it when the revert lands | 16:30 |
| mriedem | what revert? | 16:30 |
| sdague | mriedem: https://review.openstack.org/#/c/356010/ | 16:31 |
| mriedem | dstanek: https://github.com/openstack/oslo.log/blob/master/oslo_log/handlers.py#L33 | 16:33 |
| mriedem | mtreinish: i thought the messages with the OR need to be in () | 16:33 |
| mriedem | but i was trying to do the same thing in logstash and it was just spinning | 16:34 |
| mtreinish | mriedem: tbh I neglected to test it, so I probably got it wrong | 16:35 |
| mtreinish | I knew you'd run it for me either way (if I left the numbers out of the commit msg) | 16:35 |
| mriedem | ha | 16:35 |
| *** xinli has joined #openstack-qa | 16:36 | |
| mriedem | yup | 16:37 |
| mtreinish | mriedem: the query might be too big, those messages are kinda long | 16:37 |
| mriedem | and you need to escape the double quotes | 16:37 |
| mriedem | i think i have it | 16:37 |
| mriedem | (message:"The HTTP URL is invalid: HTTPConnectionPool(host='git.openstack.org', port=80): Max retries exceeded with url: /static/openstack.png (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))" OR message:"urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=9292): Max retries exceeded with url: /v1/images (Caused by ReadTimeoutError(\"HTTPConnectionPool | 16:37 |
| mriedem | yup | 16:38 |
| mriedem | that's the money query | 16:38 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for bug 1613749 https://review.openstack.org/355988 | 16:39 |
| openstack | bug 1613749 in OpenStack-Gate "Timeouts when requesting a glance image created with a remote image from git.o.o" [Undecided,New] https://launchpad.net/bugs/1613749 | 16:39 |
| mriedem | http://goo.gl/uvD4Gu | 16:39 |
| mriedem | mostly bluebox | 16:39 |
| mriedem | 1% osic | 16:40 |
| *** hockeynut has joined #openstack-qa | 16:40 | |
| mtreinish | mriedem: I'm seeing the opposite, it says 315 osic 6 bluebox | 16:41 |
| mriedem | oh yeah you're right | 16:41 |
| openstackrecheck | Console logs not available after 14:00s for gate-cinder-python34-db 355231,3,673df27 | 16:43 |
| *** tosky_ has quit IRC | 16:44 | |
| *** kcobb has quit IRC | 16:47 | |
| *** kcobb has joined #openstack-qa | 16:48 | |
| sdague | so, I don't know osic performance off the top of my head | 16:49 |
| sdague | I know that bluebox was on the slower side | 16:49 |
| sdague | I'm still guessing this has to do with keystone load | 16:49 |
| *** amuller has left #openstack-qa | 16:50 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Add link to node_provider grouped run page from test page https://review.openstack.org/356060 | 16:50 |
| mtreinish | timothyb89: ^^^ I think I did that right | 16:50 |
| *** nwkarsten has quit IRC | 16:54 | |
| *** nwkarste_ has joined #openstack-qa | 16:58 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Make everything plural https://review.openstack.org/355967 | 16:58 |
| *** sabeen has joined #openstack-qa | 17:02 | |
| *** lucasagomes is now known as lucas-dinner | 17:02 | |
| *** nwkarste_ has quit IRC | 17:03 | |
| *** Poornima has quit IRC | 17:03 | |
| *** hockeynut has quit IRC | 17:03 | |
| openstackrecheck | Console logs not available after 13:55s for gate-tempest-dsvm-neutron-full-ubuntu-xenial 355961,1,9ea3c39 | 17:04 |
| *** cdent has quit IRC | 17:05 | |
| *** yamahata has joined #openstack-qa | 17:05 | |
| *** _nadya_ has joined #openstack-qa | 17:09 | |
| *** david-lyle_ has joined #openstack-qa | 17:09 | |
| *** Suyi has quit IRC | 17:10 | |
| *** e0ne has joined #openstack-qa | 17:11 | |
| *** david-lyle has quit IRC | 17:13 | |
| *** david-lyle_ is now known as david-lyle | 17:13 | |
| *** Suyi has joined #openstack-qa | 17:14 | |
| *** gvrangan has joined #openstack-qa | 17:14 | |
| *** openstackrecheck has quit IRC | 17:15 | |
| *** openstackrecheck has joined #openstack-qa | 17:15 | |
| *** _nadya_ has quit IRC | 17:16 | |
| *** tqtran has joined #openstack-qa | 17:19 | |
| *** _nadya_ has joined #openstack-qa | 17:21 | |
| *** Serlex has quit IRC | 17:23 | |
| *** hieulq_ has quit IRC | 17:29 | |
| *** chandankumar has joined #openstack-qa | 17:32 | |
| *** pglass has quit IRC | 17:33 | |
| *** hieulq_ has joined #openstack-qa | 17:36 | |
| *** cdent has joined #openstack-qa | 17:36 | |
| *** paul-carlton2 has quit IRC | 17:38 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Add link to node_provider grouped run page from test page https://review.openstack.org/356060 | 17:40 |
| *** nwkarste_ has joined #openstack-qa | 17:45 | |
| *** nwkarst__ has joined #openstack-qa | 17:46 | |
| *** nwkarste_ has quit IRC | 17:50 | |
| *** hieulq_ has quit IRC | 17:50 | |
| *** tqtran has quit IRC | 17:55 | |
| *** tqtran has joined #openstack-qa | 17:58 | |
| openstackgerrit | Tim Buckley proposed openstack/stackviz: Add webpack build configuration and new boilerplate https://review.openstack.org/353085 | 18:00 |
| *** rcernin has joined #openstack-qa | 18:01 | |
| *** dwalleck has joined #openstack-qa | 18:03 | |
| *** tqtran has quit IRC | 18:03 | |
| oomichi | castulo: hi, will you continue working for https://bugs.launchpad.net/tempest/+bug/1086590 ? | 18:03 |
| openstack | Launchpad bug 1086590 in tempest "Add tests that use parameters on GET functions" [Medium,In progress] - Assigned to Castulo J. Martinez (castulo-martinez) | 18:03 |
| openstackgerrit | Dan Smith proposed openstack-dev/devstack: Move create_cell to late in the setup process https://review.openstack.org/355602 | 18:03 |
| oomichi | castulo: that seems hard to define closing the bug | 18:03 |
| openstackgerrit | Dan Smith proposed openstack-dev/devstack: Move create_cell to late in the setup process https://review.openstack.org/355602 | 18:06 |
| *** pglass has joined #openstack-qa | 18:08 | |
| *** dwalleck has quit IRC | 18:09 | |
| *** tqtran has joined #openstack-qa | 18:12 | |
| *** _nadya_ has quit IRC | 18:15 | |
| *** cdent has quit IRC | 18:16 | |
| *** tqtran has quit IRC | 18:19 | |
| *** pvaneck has joined #openstack-qa | 18:19 | |
| castulo | oomichi: yes actually I still have two patches on review, those are the last ones, once that is done the bug can be closed. These are the patches: https://review.openstack.org/#/c/317088/ and https://review.openstack.org/#/c/325516/ and looks like the second one most likely will have to be moved to a plugin. If that is the case only the first patch would need to merge to be able to close the bug. | 18:19 |
| oomichi | castulo: ah, I missed them. Thanks for doing that. | 18:22 |
| castulo | happy to help :) | 18:22 |
| oomichi | castulo: yeah, nice help :-) | 18:23 |
| *** chandankumar has quit IRC | 18:24 | |
| *** rbak has joined #openstack-qa | 18:25 | |
| *** cody-somerville has quit IRC | 18:29 | |
| *** cody-somerville has joined #openstack-qa | 18:29 | |
| *** csomerville has joined #openstack-qa | 18:30 | |
| *** tqtran has joined #openstack-qa | 18:32 | |
| *** gvrangan has quit IRC | 18:33 | |
| openstackgerrit | Maxim Nestratov proposed openstack-dev/devstack: libvirt virt_type=parallels support enchancement https://review.openstack.org/356091 | 18:33 |
| *** cody-somerville has quit IRC | 18:34 | |
| *** mvk has quit IRC | 18:38 | |
| *** _nadya_ has joined #openstack-qa | 18:40 | |
| *** rlrossit has quit IRC | 18:40 | |
| *** amotoki has joined #openstack-qa | 18:45 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 18:46 |
| *** _nadya_ has quit IRC | 18:48 | |
| openstackgerrit | Matt Riedemann proposed openstack/tempest: Validate floating IP in server['addresses'] scenario https://review.openstack.org/356095 | 18:48 |
| mriedem | oomichi: ^ | 18:48 |
| *** amotoki has quit IRC | 18:48 | |
| sdague | mtreinish: are you still debugging these tests, or should we skip them? | 18:50 |
| mtreinish | sdague: which tests? | 18:50 |
| sdague | the 3 glance tests | 18:51 |
| mtreinish | oh, the glance thing | 18:51 |
| sdague | that keep failing in a cluster | 18:51 |
| mtreinish | I just got back from lunch, so I was lsot for context | 18:51 |
| sdague | they are always failing in a cluster, which includes a keystone auth issue | 18:51 |
| sdague | even after the keystone debug revert | 18:51 |
| mtreinish | sdague: the keystone auth issue is still there after the revert? | 18:51 |
| sdague | yep | 18:52 |
| mtreinish | hmm | 18:52 |
| *** cdent has joined #openstack-qa | 18:53 | |
| oomichi | mriedem: the last part(line 155-) seems good for me, but I guess the first one(line 130-) is not mandatory. That seems too dependent on the bug | 18:53 |
| fungi | which glance failure is coupled with that? the timeout reading from git.o.o or the timeout reading from 127.0.0.1:9292 | 18:53 |
| fungi | or have you found more evidence to connect those two behaviors now? | 18:53 |
| mriedem | oomichi: associating the floating IP refreshes the instance info cache | 18:53 |
| mriedem | so it should be in server.addresses when that comes back | 18:54 |
| mriedem | as should deleting/disassociating the floating ip should refresh the cache and remove it from server.addresses | 18:54 |
| mriedem | i do the pre check to make sure the post delete check is valid | 18:54 |
| mriedem | oomichi: anyway, if i'm correct about this, it should pass with nova-network and fail with neutron | 18:55 |
| oomichi | mriedem: yeah, I know. but the test is a minimum basic test as the name. the pre-check seems a little overkill | 18:55 |
| mtreinish | sdague: looking at the log that keystone auth traceback comes after all the 3 tests fail | 18:55 |
| mtreinish | fungi: this is the osic one with 127.0.0.1 and the timeout | 18:55 |
| mtreinish | the bluebox failure is much more uncommon | 18:55 |
| mriedem | oomichi: ok, i should maybe move this to the compute api test FloatingIPsTestJSON then | 18:56 |
| mriedem | oomichi: if it works in the scenario test, i'll just move it to FloatingIPsTestJSON later | 18:56 |
| mtreinish | sdague: like the last failure was ~5min before the keystone auth traceback in the glance log | 18:57 |
| sdague | mtreinish: there is only 1 glance auth issue | 18:58 |
| sdague | at least, typically | 18:59 |
| mtreinish | sdague: http://logs.openstack.org/10/352610/4/gate/gate-tempest-dsvm-cells/1d2248f/logs/screen-g-api.txt.gz?level=TRACE#_2016-08-16_17_11_03_725 is the traceback I was looking at | 18:59 |
| sdague | oh, right, odd | 18:59 |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-full-devstack-plugin-ceph 353069,3,651bd23 | 19:00 |
| sdague | anyway, I think the question is, is anyone going to own actively debugging these tests? Or do we skip them to not impact the release. | 19:00 |
| oomichi | mriedem: ah, ok. The existing tests in FloatingIPsTestJSON don't create both server and floating-ip. so we need to add a new one | 19:00 |
| *** e0ne has quit IRC | 19:01 | |
| mriedem | yup | 19:01 |
| mriedem | but i'd like to see this confirm the bug first | 19:01 |
| oomichi | mriedem: that should be a small/easy one | 19:01 |
| oomichi | mriedem: yeah, that is nice to debug that | 19:01 |
| mriedem | don't jinx it :) | 19:01 |
| oomichi | mriedem: jinx? https://www.google.com/search?q=jinx&safe=off&client=firefox-b-ab&source=lnms&tbm=isch&sa=X&ved=0ahUKEwi8ga3zz8bOAhVP3GMKHeA4DRMQ_AUICCgB&biw=1536&bih=770 ? | 19:03 |
| oomichi | I am not sure the charactor ;) | 19:03 |
| sdague | I do guess it's everything that uses http_image | 19:05 |
| sdague | I'm sort of surprised we use that for the list_image test | 19:05 |
| mtreinish | sdague: it's part of the filtering test path. We want to create a couple different type of images in the setupclass | 19:07 |
| mtreinish | sdague: I'm looking through the logs again and I think that unauthorization error is glance coming back and updating the registry with the create call from the failed test several minutes earlier | 19:07 |
| sdague | right, but we've always known this was dicey | 19:07 |
| mtreinish | and the token has expired | 19:07 |
| sdague | mtreinish: ok, so that's just noise | 19:07 |
| *** nikhil has joined #openstack-qa | 19:07 | |
| mriedem | oomichi: no not that | 19:07 |
| nikhil | sdague: o/ | 19:07 |
| sdague | ok, well http_image has always been so dicey | 19:08 |
| sdague | I'm really surprised it was allowed to leak into other tests | 19:08 |
| mriedem | https://www.google.com/search?q=jinx&ie=utf-8&oe=utf-8#safe=off&q=jinx+meaning | 19:08 |
| oomichi | mriedem: ah, I see. We need a bad luck for debugging bugs ;) | 19:10 |
| mriedem | no, you just can't say fixing it will be easy | 19:10 |
| mriedem | that's bad luck | 19:10 |
| *** gvrangan has joined #openstack-qa | 19:11 | |
| mtreinish | sdague: the other weird thing is it seems isolated to a few different jobs: http://status.openstack.org/openstack-health/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image | 19:13 |
| openstackrecheck | Console logs not available after 13:45s for gate-grenade-dsvm-multinode 347471,20,1038f14 | 19:14 |
| oomichi | heh, we can make fixing easier if reproducing it on the gate anyway | 19:14 |
| mtreinish | fungi: I think you're right and the bluebox issue is unrelated to the osic one | 19:15 |
| sdague | mtreinish: there is nothing useful on that page for me about jobs | 19:16 |
| mtreinish | sdague: it's only on cells, ceph, pika src, zmq src, and designate grenade | 19:17 |
| mtreinish | that's an odd mix | 19:18 |
| openstackgerrit | Sean Dague proposed openstack/tempest: reduce exposure to http image issues https://review.openstack.org/356103 | 19:19 |
| sdague | mtreinish: do they run limitted tests? | 19:20 |
| sdague | are they doing something to the network? | 19:20 |
| *** _nadya_ has joined #openstack-qa | 19:20 | |
| *** martinkopec has joined #openstack-qa | 19:20 | |
| sdague | so, I think that https://review.openstack.org/356103 is probably the way we should limit our exposure here | 19:20 |
| *** martinkopec has quit IRC | 19:21 | |
| mtreinish | sdague: I don't think that'll get us past this (ignoring the failure inline) because we set that option in the gate | 19:24 |
| sdague | mtreinish: correct, we'd also have to unset that | 19:25 |
| oomichi | mkoderer: hi, are you around? | 19:25 |
| sdague | but it at least gives us that knob | 19:25 |
| sdague | it's a little weird that our list image test requires that your glance has outward http connectivity, for instance. | 19:25 |
| *** gvrangan has quit IRC | 19:27 | |
| openstackrecheck | Console logs not available after 13:33s for gate-tempest-dsvm-full-devstack-plugin-ceph 240649,7,133951d | 19:27 |
| mtreinish | sdague: probably, but it's been that way forever: https://review.openstack.org/#/c/2884/5 | 19:29 |
| mtreinish | sdague: lets just blame jaypipes :) | 19:30 |
| mtreinish | sdague: I think fungi was onto something with the osic ipv6 move being related. http://logs.openstack.org/38/350438/3/gate/gate-oslo.messaging-src-dsvm-full-pika/29c0c3c/console.html#_2016-08-16_02_19_49_713182 seems like it would definitely be related | 19:33 |
| fungi | i hate being onto something | 19:33 |
| *** tqtran has quit IRC | 19:34 | |
| fungi | i'm probably just on something | 19:34 |
| mtreinish | fungi: heh, it can be both | 19:34 |
| sdague | oh, yeh, I would totally believe this as an ipv6 issue | 19:35 |
| sdague | so... they are all trusty right? | 19:35 |
| sdague | that's the other cluster | 19:35 |
| mtreinish | sdague: oh yeah, that could be the common thread there | 19:36 |
| sdague | so trusty + ipv6 + osic | 19:36 |
| sdague | I could believe that's our fail cluster | 19:36 |
| mtreinish | fungi: ^^^ is that the angle you were looking into? | 19:36 |
| mtreinish | mriedem: I think everything is good now on: http://status.openstack.org/elastic-recheck/data/others.html | 19:37 |
| mtreinish | all the links are right, and things are updating again | 19:37 |
| mriedem | ncie http://status.openstack.org/elastic-recheck/data/integrated_gate.html | 19:37 |
| mriedem | *nice | 19:37 |
| sdague | mtreinish: right, and these tests aren't in smoke | 19:39 |
| mtreinish | ah, yep that would explain the pattern I was seeing in the jobs hitting it then | 19:40 |
| fungi | mtreinish: sdague: cloudnull was looking into it since he has root access to the environment and routers and whatnot | 19:42 |
| *** _nadya_ has quit IRC | 19:42 | |
| fungi | one possibility is we're overtaxing the neutron router there | 19:42 |
| sdague | fungi: sure, though it doesn't show up on xenial hosts | 19:42 |
| openstackrecheck | Console logs not available after 13:39s for gate-grenade-dsvm-multinode 355898,2,2fa0b15 | 19:43 |
| openstackgerrit | Merged openstack-dev/devstack: Remove lib/cinder_backends/xiv https://review.openstack.org/355236 | 19:44 |
| *** xinli has quit IRC | 19:44 | |
| openstackgerrit | Merged openstack-dev/devstack: Configure project_network_cidr for tempest https://review.openstack.org/353670 | 19:45 |
| *** tqtran has joined #openstack-qa | 19:47 | |
| *** hockeynut has joined #openstack-qa | 19:48 | |
| *** gvrangan has joined #openstack-qa | 19:50 | |
| *** hockeynut has quit IRC | 19:53 | |
| *** hockeynut has joined #openstack-qa | 19:54 | |
| *** Apoorva has joined #openstack-qa | 19:55 | |
| *** nwkarst__ has quit IRC | 19:57 | |
| *** nwkarsten has joined #openstack-qa | 19:57 | |
| openstackrecheck | Console logs not available after 14:04s for gate-tempest-dsvm-full-devstack-plugin-ceph 352610,4,97cb666 | 19:57 |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/355469 failed because of: gate-tempest-dsvm-full-devstack-plugin-ceph: https://bugs.launchpad.net/bugs/1449136 | 19:59 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 19:59 |
| *** tqtran has quit IRC | 19:59 | |
| *** nwkarste_ has joined #openstack-qa | 20:00 | |
| *** nwkarsten has quit IRC | 20:02 | |
| *** e0ne has joined #openstack-qa | 20:05 | |
| *** tqtran has joined #openstack-qa | 20:05 | |
| openstackgerrit | Augustina Ragwitz proposed openstack-dev/devstack: WIP: add tempest run support to devstack https://review.openstack.org/351367 | 20:07 |
| *** vhosakot has quit IRC | 20:10 | |
| *** porrua has quit IRC | 20:10 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for bug 1613749 https://review.openstack.org/355988 | 20:11 |
| openstack | bug 1613749 in OpenStack-Gate "Timeouts when requesting a glance image created with a remote image from git.o.o" [Undecided,New] https://launchpad.net/bugs/1613749 | 20:11 |
| *** vhosakot has joined #openstack-qa | 20:11 | |
| openstackgerrit | Thiago Paiva Brito proposed openstack/tempest: [WIP] Removing baremetal tests from tempest tree https://review.openstack.org/355586 | 20:11 |
| openstackrecheck | Console logs not available after 13:35s for gate-tempest-dsvm-full-devstack-plugin-ceph 353875,3,05868cc | 20:12 |
| openstackgerrit | Thiago Paiva Brito proposed openstack/tempest: [WIP] Removing baremetal tests from tempest tree https://review.openstack.org/355586 | 20:13 |
| *** _AmitGalitz_ has joined #openstack-qa | 20:14 | |
| *** openstackrecheck has quit IRC | 20:14 | |
| *** openstackrecheck has joined #openstack-qa | 20:15 | |
| *** tqtran has quit IRC | 20:16 | |
| *** e0ne has quit IRC | 20:18 | |
| *** yaume has joined #openstack-qa | 20:18 | |
| *** gvrangan has quit IRC | 20:18 | |
| *** mfisher_ora has quit IRC | 20:19 | |
| *** rbak has quit IRC | 20:21 | |
| *** rbak has joined #openstack-qa | 20:21 | |
| *** rbak has quit IRC | 20:22 | |
| *** rbak has joined #openstack-qa | 20:24 | |
| openstackgerrit | Thiago Paiva Brito proposed openstack/tempest: [WIP] Removing baremetal tests from tempest tree https://review.openstack.org/355586 | 20:25 |
| *** jordanP has joined #openstack-qa | 20:27 | |
| openstackrecheck | Console logs not available after 13:27s for gate-tempest-dsvm-full-devstack-plugin-ceph 356036,1,b000efb | 20:29 |
| *** tqtran has joined #openstack-qa | 20:31 | |
| *** gouthamr has quit IRC | 20:33 | |
| *** tqtran has quit IRC | 20:34 | |
| *** gvrangan has joined #openstack-qa | 20:36 | |
| *** gvrangan has quit IRC | 20:42 | |
| *** tqtran has joined #openstack-qa | 20:42 | |
| openstackrecheck | Console logs not available after 13:44s for gate-tempest-dsvm-cells 346278,5,526d217 | 20:43 |
| *** yfried has joined #openstack-qa | 20:45 | |
| *** armax has quit IRC | 20:45 | |
| *** e0ne has joined #openstack-qa | 20:45 | |
| openstackgerrit | Thiago Paiva Brito proposed openstack/tempest: [WIP] Removing baremetal tests from tempest tree https://review.openstack.org/355586 | 20:48 |
| *** tqtran has quit IRC | 20:48 | |
| *** jordanP has quit IRC | 20:52 | |
| *** yaume has quit IRC | 20:54 | |
| *** Apoorva_ has joined #openstack-qa | 20:54 | |
| openstackrecheck | Console logs not available after 13:35s for gate-tempest-dsvm-postgres-full-ubuntu-trusty-mitaka 355441,1,afadde8 | 20:57 |
| *** Apoorva has quit IRC | 20:57 | |
| mtreinish | timothyb89: hmm: http://logs.openstack.org/60/356060/2/check/gate-openstack-health-nodejs4-npm-run-test/824afb0/reports/build/#/test/tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image | 20:59 |
| mtreinish | the links don't work | 20:59 |
| *** gvrangan has joined #openstack-qa | 21:00 | |
| *** raildo has quit IRC | 21:01 | |
| *** yfried has quit IRC | 21:02 | |
| *** sdague has quit IRC | 21:04 | |
| *** dkranz has quit IRC | 21:05 | |
| *** tqtran has joined #openstack-qa | 21:07 | |
| *** yamamoto has joined #openstack-qa | 21:08 | |
| openstackgerrit | Thiago Paiva Brito proposed openstack/tempest: [WIP] Removing baremetal tests from tempest tree https://review.openstack.org/355586 | 21:11 |
| *** Apoorva_ has quit IRC | 21:11 | |
| *** Apoorva has joined #openstack-qa | 21:12 | |
| *** yamamoto has quit IRC | 21:15 | |
| openstackgerrit | Thiago Paiva Brito proposed openstack/tempest: [WIP] Removing baremetal tests from tempest tree https://review.openstack.org/355586 | 21:16 |
| openstackrecheck | Console logs not available after 13:45s for gate-tempest-dsvm-full-devstack-plugin-ceph 355923,2,b5d13f9 | 21:17 |
| *** jecarey has quit IRC | 21:20 | |
| *** e0ne has quit IRC | 21:20 | |
| *** cdent has quit IRC | 21:21 | |
| *** jecarey has joined #openstack-qa | 21:23 | |
| *** mvk has joined #openstack-qa | 21:23 | |
| *** jordanP has joined #openstack-qa | 21:23 | |
| *** spzala_ has quit IRC | 21:23 | |
| *** spzala has joined #openstack-qa | 21:23 | |
| *** rcernin has quit IRC | 21:27 | |
| *** dustins has quit IRC | 21:27 | |
| *** jecarey has quit IRC | 21:28 | |
| *** spzala has quit IRC | 21:28 | |
| *** jordanP has quit IRC | 21:32 | |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-full-devstack-plugin-ceph 343571,7,e528a86 | 21:32 |
| *** tqtran has quit IRC | 21:37 | |
| *** tqtran has joined #openstack-qa | 21:37 | |
| *** tqtran has quit IRC | 21:42 | |
| *** weshay has quit IRC | 21:47 | |
| *** nwkarste_ has quit IRC | 21:48 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Add link to node_provider grouped run page from test page https://review.openstack.org/356060 | 21:48 |
| *** nwkarsten has joined #openstack-qa | 21:48 | |
| mtreinish | timothyb89: ^^^ ok this time | 21:48 |
| openstackrecheck | Console logs not available after 13:43s for gate-nova-python27-db-ubuntu-xenial 332861,11,490e7da | 21:51 |
| *** nwkarsten has quit IRC | 21:52 | |
| *** tqtran has joined #openstack-qa | 21:54 | |
| openstackgerrit | Merged openstack-dev/devstack: add networking write up for devstack https://review.openstack.org/354658 | 21:54 |
| *** edmondsw has quit IRC | 21:58 | |
| *** tqtran has quit IRC | 21:59 | |
| *** sabeen has quit IRC | 21:59 | |
| *** spzala has joined #openstack-qa | 22:00 | |
| *** nwkarsten has joined #openstack-qa | 22:01 | |
| *** mriedem has quit IRC | 22:04 | |
| *** nwkarsten has quit IRC | 22:06 | |
| openstackrecheck | Console logs not available after 13:41s for gate-tempest-dsvm-full-devstack-plugin-ceph 251503,20,621de00 | 22:10 |
| *** tqtran has joined #openstack-qa | 22:10 | |
| *** tqtran has quit IRC | 22:20 | |
| *** pglass has quit IRC | 22:20 | |
| openstackrecheck | Console logs not available after 13:36s for gate-tempest-dsvm-cells 290793,6,79c3d4e | 22:25 |
| *** tqtran has joined #openstack-qa | 22:27 | |
| openstackgerrit | Matthew Treinish proposed openstack/openstack-health: Add link to node_provider grouped run page from test page https://review.openstack.org/356060 | 22:27 |
| *** dimtruck is now known as zz_dimtruck | 22:29 | |
| *** zz_dimtruck is now known as dimtruck | 22:29 | |
| *** yamahata has quit IRC | 22:31 | |
| openstackrecheck | openstack/os-brick change: https://review.openstack.org/355765 failed gate-tempest-dsvm-full-ceph-plugin-src-os-brick in the gate queue with an unrecognized error | 22:32 |
| *** tqtran has quit IRC | 22:36 | |
| *** tqtran has joined #openstack-qa | 22:39 | |
| *** dimtruck is now known as zz_dimtruck | 22:39 | |
| *** gouthamr has joined #openstack-qa | 22:39 | |
| *** gouthamr_ has joined #openstack-qa | 22:40 | |
| *** tqtran has quit IRC | 22:43 | |
| *** gouthamr has quit IRC | 22:44 | |
| *** gouthamr_ is now known as gouthamr | 22:44 | |
| *** yamamoto has joined #openstack-qa | 22:46 | |
| *** yamahata has joined #openstack-qa | 22:51 | |
| openstackrecheck | Console logs not available after 14:14s for gate-grenade-dsvm-neutron-multinode 356094,1,94f5528 | 22:52 |
| *** jecarey has joined #openstack-qa | 22:53 | |
| *** hockeynut has quit IRC | 22:55 | |
| *** vhosakot has quit IRC | 22:56 | |
| auggy | mtreinish: i am running a recheck on your d-g patch since i pushed the devstack change | 23:00 |
| mtreinish | auggy: oh, heh I forgot to look at the results from that | 23:00 |
| auggy | i saw an error on the original run - /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 757: -evenv: command not found | 23:01 |
| *** spzala has quit IRC | 23:01 | |
| auggy | it looks like it's not creating the TEMPEST_COMMAND | 23:01 |
| auggy | but i could be wrong and misunderstanding the error | 23:01 |
| *** spzala has joined #openstack-qa | 23:01 | |
| auggy | so i thought well, if anything lets get a new run with the updated devstack | 23:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 23:05 |
| mtreinish | auggy: ^^^ maybe? | 23:05 |
| *** hongbin has quit IRC | 23:05 | |
| *** tqtran has joined #openstack-qa | 23:05 | |
| auggy | mtreinish: SURE WHY NOT | 23:06 |
| *** spzala has quit IRC | 23:06 | |
| auggy | haha no really, i have no idea, that's the only thing that popped into my head | 23:06 |
| *** rbak has quit IRC | 23:06 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/352573 failed because of: gate-tempest-dsvm-cells: https://bugs.launchpad.net/bugs/1449136 https://bugs.launchpad.net/bugs/1613749 | 23:11 |
| openstack | Launchpad bug 1449136 in OpenStack-Gate "OpenStack pypi mirrors disconnecting connections" [Undecided,New] | 23:11 |
| openstack | Launchpad bug 1613749 in OpenStack-Gate "Timeouts when requesting a glance image created with a remote image from git.o.o" [Undecided,New] | 23:11 |
| *** tqtran has quit IRC | 23:19 | |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-full-devstack-plugin-ceph 355924,2,763f3a5 | 23:26 |
| *** tochi has joined #openstack-qa | 23:39 | |
| *** zz_dimtruck is now known as dimtruck | 23:40 | |
| *** csomerville has quit IRC | 23:42 | |
| *** moravec has quit IRC | 23:44 | |
| *** cody-somerville has joined #openstack-qa | 23:45 | |
| *** tqtran has joined #openstack-qa | 23:46 | |
| *** moravec has joined #openstack-qa | 23:49 | |
| *** bswartz has quit IRC | 23:49 | |
| *** ihrachys has joined #openstack-qa | 23:50 | |
| *** tqtran has quit IRC | 23:51 | |
| *** tqtran has joined #openstack-qa | 23:54 | |
| openstackrecheck | Console logs not available after 13:29s for gate-tempest-dsvm-full-devstack-plugin-ceph 355925,2,d72a37c | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!