| *** adarazs has joined #openstack-qa | 00:02 | |
| *** hemna is now known as hemnafk | 00:09 | |
| openstackgerrit | Christopher Yeoh proposed a change to openstack/tempest: Enable the Nova V3 API Tests https://review.openstack.org/57589 | 00:12 |
|---|---|---|
| openstackrecheck | openstack/swift change: https://review.openstack.org/58400 failed tempest because of: https://bugs.launchpad.net/bugs/1249065 | 00:13 |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/56691 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 00:14 |
| *** mkollaro1 has quit IRC | 00:29 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58398 failed tempest because of: https://bugs.launchpad.net/bugs/1210483 | 00:30 |
| *** matsuhashi has joined #openstack-qa | 00:32 | |
| *** sthaha has joined #openstack-qa | 00:42 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/56893 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 00:44 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Just a quick sanity check Do not merge https://review.openstack.org/58417 | 00:50 |
| *** atiwari has quit IRC | 01:01 | |
| *** nosnos has joined #openstack-qa | 01:09 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58267 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 01:10 |
| openstackgerrit | Joris Roovers proposed a change to openstack/tempest: Adds full agent mgmt API support in network_client https://review.openstack.org/58419 | 01:12 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Run df after gate https://review.openstack.org/56298 | 01:13 |
| openstackgerrit | A change was merged to openstack/tempest: Add test for update/reset_snapshot_status API https://review.openstack.org/55331 | 01:13 |
| openstackgerrit | A change was merged to openstack/tempest: Enhance tests for flavor_extra_spec API https://review.openstack.org/50523 | 01:13 |
| *** jorisroovers has quit IRC | 01:14 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/52704 failed tempest because of: https://bugs.launchpad.net/bugs/1254872 | 01:15 |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/57589 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 01:15 |
| *** jorisroovers has joined #openstack-qa | 01:17 | |
| *** mriedem has joined #openstack-qa | 01:18 | |
| openstackgerrit | Christopher Yeoh proposed a change to openstack-infra/devstack-gate: TEMPEST_LARGE_OPS_NUMBER tuning https://review.openstack.org/58422 | 01:19 |
| *** jorisroovers has quit IRC | 01:31 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/58284 failed tempest because of: https://bugs.launchpad.net/bugs/1210483 and https://bugs.launchpad.net/bugs/1253896 | 01:34 |
| openstackrecheck | openstack/keystone change: https://review.openstack.org/55900 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 01:41 |
| openstackgerrit | huangtianhua proposed a change to openstack/tempest: Add migrate negative tests https://review.openstack.org/51089 | 01:44 |
| openstackrecheck | openstack/glance change: https://review.openstack.org/54198 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 01:49 |
| openstackrecheck | openstack/python-novaclient change: https://review.openstack.org/58418 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 01:57 |
| *** ivanzhu has joined #openstack-qa | 02:02 | |
| *** nosnos_ has joined #openstack-qa | 02:02 | |
| *** nosnos has quit IRC | 02:05 | |
| *** coolsvap has quit IRC | 02:06 | |
| *** rhsu has quit IRC | 02:08 | |
| *** mriedem has quit IRC | 02:09 | |
| openstackgerrit | LiShaokai proposed a change to openstack/tempest: Add test for volume when creating a volume without display name https://review.openstack.org/58197 | 02:17 |
| *** zhikunliu has joined #openstack-qa | 02:19 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/53805 failed tempest because of: https://bugs.launchpad.net/bugs/1249065 | 02:21 |
| openstackrecheck | openstack/python-neutronclient change: https://review.openstack.org/55190 failed tempest because of: https://bugs.launchpad.net/bugs/1235435 and https://bugs.launchpad.net/bugs/1249065 and https://bugs.launchpad.net/bugs/1251448 and https://bugs.launchpad.net/bugs/1253896 | 02:27 |
| *** jhenner|afk has quit IRC | 02:37 | |
| *** dims has quit IRC | 02:38 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/58423 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 02:39 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58213 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 02:41 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/56893 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 02:49 |
| openstackgerrit | huangtianhua proposed a change to openstack/tempest: Add migrate negative tests https://review.openstack.org/51089 | 03:01 |
| ivanzhu | cyeoh: hi, could you please review this https://review.openstack.org/#/c/43070/ again? just rebase, no other change, and the part1 has been merged. | 03:01 |
| openstackgerrit | huangtianhua proposed a change to openstack/tempest: Add migrate negative tests https://review.openstack.org/51089 | 03:03 |
| cyeoh | ivanzhu: sure, will have a look this afternoon | 03:03 |
| ivanzhu | cyeoh: thanks | 03:03 |
| cyeoh | ivanzhu: btw for the cases where are there minimal changes to the test itself (not the client xml/json), I think we need to have another look at how much we can share rather than copy/paste | 03:04 |
| cyeoh | ivanzhu: for example with https://review.openstack.org/#/c/43070/3/tempest/api/compute/v3/admin/test_availability_zone.py there is very little change at all and I'd hope there was some way we could somehow avoid the cut/paste for that file | 03:04 |
| ivanzhu | cyeoh: ok, got it | 03:05 |
| openstackgerrit | huangtianhua proposed a change to openstack/tempest: Add migrate negative tests https://review.openstack.org/51089 | 03:06 |
| giulivo | guys as far as I can remember bugs mentioned by merged submissions are now moved in "released" state and not left in "committed" state | 03:07 |
| giulivo | is that correct? | 03:07 |
| openstackgerrit | wanghao proposed a change to openstack/tempest: add some test for force delete: https://review.openstack.org/55977 | 03:08 |
| openstackgerrit | huangtianhua proposed a change to openstack/tempest: Add migrate negative tests https://review.openstack.org/51089 | 03:08 |
| *** matsuhashi has quit IRC | 03:11 | |
| openstackrecheck | openstack/glance change: https://review.openstack.org/54198 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 03:12 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/53805 failed tempest because of: https://bugs.launchpad.net/bugs/1235435 and https://bugs.launchpad.net/bugs/1249065 | 03:27 |
| *** coolsvap has joined #openstack-qa | 03:50 | |
| openstackgerrit | Mauro S. M. Rodrigues proposed a change to openstack-dev/grenade: Refactor new options into a new config file. https://review.openstack.org/57744 | 03:51 |
| openstackgerrit | Mauro S. M. Rodrigues proposed a change to openstack-dev/grenade: Add clean configuration files to upgrade from H to I https://review.openstack.org/53874 | 03:51 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/35760 failed tempest with an unrecognized error | 03:52 |
| *** jorisroovers has joined #openstack-qa | 03:59 | |
| *** jroovers has joined #openstack-qa | 04:01 | |
| *** jorisroovers has quit IRC | 04:03 | |
| *** rhsu has joined #openstack-qa | 04:19 | |
| *** matsuhashi has joined #openstack-qa | 04:19 | |
| *** giulivo has quit IRC | 04:22 | |
| *** coolsvap has quit IRC | 04:31 | |
| *** nosnos_ has quit IRC | 04:32 | |
| *** nosnos has joined #openstack-qa | 04:32 | |
| *** SushilKM has joined #openstack-qa | 05:04 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/53789 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 05:04 |
| *** coolsvap_ has joined #openstack-qa | 05:05 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58432 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 05:07 |
| *** SushilKM has quit IRC | 05:08 | |
| *** jasondotstar has quit IRC | 05:10 | |
| *** zigo_ has quit IRC | 05:10 | |
| *** zigo has joined #openstack-qa | 05:11 | |
| openstackgerrit | A change was merged to openstack/tempest: Add some tests for security_group_rules api https://review.openstack.org/54228 | 05:11 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/55496 failed tempest with an unrecognized error | 05:12 |
| *** ivanzhu1 has joined #openstack-qa | 05:14 | |
| *** ivanzhu has quit IRC | 05:15 | |
| openstackrecheck | openstack-infra/devstack-gate change: https://review.openstack.org/58422 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 05:20 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/52557 failed tempest with an unrecognized error | 05:21 |
| openstackrecheck | openstack-dev/pbr change: https://review.openstack.org/57892 failed tempest with an unrecognized error | 05:23 |
| *** coolsvap_ has quit IRC | 05:24 | |
| *** coolsvap has joined #openstack-qa | 05:29 | |
| *** mkollaro has joined #openstack-qa | 05:30 | |
| *** zhikunliu has quit IRC | 05:31 | |
| *** matsuhashi has quit IRC | 05:50 | |
| *** matsuhashi has joined #openstack-qa | 05:51 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58435 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 05:51 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/54234 failed tempest with an unrecognized error | 05:51 |
| *** matsuhas_ has joined #openstack-qa | 05:53 | |
| *** matsuhashi has quit IRC | 05:53 | |
| *** Madan has joined #openstack-qa | 05:55 | |
| *** rhsu1 has joined #openstack-qa | 06:00 | |
| *** rhsu has quit IRC | 06:00 | |
| *** yfried has joined #openstack-qa | 06:14 | |
| *** matsuhas_ has quit IRC | 06:20 | |
| yfried | marun: ping | 06:21 |
| *** coolsvap has quit IRC | 06:26 | |
| *** matsuhashi has joined #openstack-qa | 06:29 | |
| *** coolsvap has joined #openstack-qa | 06:33 | |
| *** coolsvap has quit IRC | 06:36 | |
| openstackgerrit | Daisuke Morita proposed a change to openstack/tempest: Check HTTP response headers of Swift API in detail https://review.openstack.org/55770 | 06:46 |
| *** tkammer has joined #openstack-qa | 06:49 | |
| *** matsuhashi has quit IRC | 06:53 | |
| *** matsuhashi has joined #openstack-qa | 06:54 | |
| *** tkammer has quit IRC | 06:56 | |
| *** tkammer has joined #openstack-qa | 06:56 | |
| *** matsuhashi has quit IRC | 06:58 | |
| *** matsuhashi has joined #openstack-qa | 06:59 | |
| *** ivanzhu1 has quit IRC | 07:10 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-dev/devstack: Move neutron cache dir into a function https://review.openstack.org/58394 | 07:13 |
| *** afazekas has joined #openstack-qa | 07:13 | |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: Adds delete api test to glance https://review.openstack.org/58451 | 07:18 |
| *** matsuhashi has quit IRC | 07:26 | |
| *** matsuhashi has joined #openstack-qa | 07:27 | |
| *** SergeyLukjanov has joined #openstack-qa | 07:38 | |
| *** matsuhas_ has joined #openstack-qa | 07:39 | |
| *** matsuhashi has quit IRC | 07:42 | |
| *** dkorn has joined #openstack-qa | 07:42 | |
| *** Grishkin has quit IRC | 07:43 | |
| *** Grishkin has joined #openstack-qa | 07:45 | |
| *** che-arne has quit IRC | 07:45 | |
| *** rhsu1 has quit IRC | 07:46 | |
| *** jlibosva has joined #openstack-qa | 07:59 | |
| *** matsuhas_ has quit IRC | 08:05 | |
| *** matsuhashi has joined #openstack-qa | 08:05 | |
| *** SergeyLukjanov has quit IRC | 08:06 | |
| *** SushilKM has joined #openstack-qa | 08:08 | |
| openstackrecheck | openstack/ceilometer change: https://review.openstack.org/58211 failed tempest with an unrecognized error | 08:13 |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/57083 failed tempest with an unrecognized error | 08:13 |
| *** sgran has joined #openstack-qa | 08:21 | |
| *** rushiagr has joined #openstack-qa | 08:21 | |
| *** psharma has joined #openstack-qa | 08:22 | |
| *** ken1ohmichi has joined #openstack-qa | 08:28 | |
| *** dkorn has quit IRC | 08:33 | |
| *** SushilKM has quit IRC | 08:35 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58190 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 08:39 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58439 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 08:52 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58383 failed tempest with an unrecognized error | 09:04 |
| *** safchain has joined #openstack-qa | 09:06 | |
| openstackgerrit | jeffrey proposed a change to openstack/tempest: Adds client and CRUD tests for Ceilometer alarms https://review.openstack.org/39237 | 09:06 |
| *** fbo_away is now known as fbo | 09:06 | |
| *** yassine has joined #openstack-qa | 09:10 | |
| rushiagr | Hi all | 09:10 |
| *** jpich has joined #openstack-qa | 09:11 | |
| *** giulivo has joined #openstack-qa | 09:16 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/53805 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 09:20 |
| *** yfried has quit IRC | 09:37 | |
| openstackgerrit | jeffrey proposed a change to openstack/tempest: Add vcpu & disk testcases This change adds some vcpu and disk testcases to check whether they are recorded in Ceilometer. https://review.openstack.org/58470 | 09:37 |
| openstackgerrit | jeffrey proposed a change to openstack/tempest: Add client and tests for Ceilometer compute metering https://review.openstack.org/43481 | 09:37 |
| openstackgerrit | jeffrey proposed a change to openstack/tempest: Add vcpu & disk testcases https://review.openstack.org/58470 | 09:47 |
| *** jhenner has joined #openstack-qa | 09:50 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/tempest: Add hard reboot for outputting console log https://review.openstack.org/58478 | 09:51 |
| giulivo | ping ken1ohmichi | 09:55 |
| ken1ohmichi | giulivo: hi! | 09:55 |
| giulivo | :) | 09:55 |
| *** matsuhas_ has joined #openstack-qa | 09:58 | |
| *** matsuhas_ has quit IRC | 09:58 | |
| *** matsuhashi has quit IRC | 10:02 | |
| *** SergeyLukjanov has joined #openstack-qa | 10:02 | |
| *** gszasz has joined #openstack-qa | 10:03 | |
| *** ken1ohmichi has left #openstack-qa | 10:12 | |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: port servers_negative tests into v3 part1 https://review.openstack.org/40637 | 10:15 |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: prot servers negative tests into v3 Part2 https://review.openstack.org/40638 | 10:16 |
| openstackrecheck | openstack/python-keystoneclient change: https://review.openstack.org/57110 failed tempest with an unrecognized error | 10:17 |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/58459 failed tempest with an unrecognized error | 10:17 |
| *** ken1ohmichi has joined #openstack-qa | 10:21 | |
| *** YorikSar has quit IRC | 10:24 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Add Savanna testing support https://review.openstack.org/57325 | 10:24 |
| *** YorikSar has joined #openstack-qa | 10:25 | |
| openstackgerrit | A change was merged to openstack/tempest: Avoid deleting ports assigned to router interfaces https://review.openstack.org/57805 | 10:53 |
| openstackgerrit | A change was merged to openstack/tempest: Cleanup using about the data_utils module and functions again https://review.openstack.org/58180 | 10:53 |
| openstackrecheck | openstack/python-keystoneclient change: https://review.openstack.org/56033 failed tempest because of: https://bugs.launchpad.net/bugs/1249065 | 10:53 |
| *** jroovers has quit IRC | 10:54 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/58234 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 10:58 |
| *** rossella_s has joined #openstack-qa | 11:03 | |
| *** yfried has joined #openstack-qa | 11:06 | |
| *** rfolco has joined #openstack-qa | 11:11 | |
| openstackrecheck | openstack/heat change: https://review.openstack.org/58479 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 11:19 |
| *** ken1ohmichi has quit IRC | 11:19 | |
| afazekas | cyeoh:IMHO we should not add new v3 tests when they are disabled https://review.openstack.org/#/c/57589/ | 11:21 |
| afazekas | Does anyone has any objection against approving the https://review.openstack.org/#/c/57589/ ? | 11:23 |
| rushiagr | When I ran tests from tempest/thirdparty/boto directory, is the call to the openstack deployment made through boto? | 11:25 |
| rushiagr | To be more specific, can one directly make calls to openstack cluster using AWS EC2 API, or is there some abstraction layer in tempest? | 11:26 |
| *** jorisroovers has joined #openstack-qa | 11:26 | |
| rushiagr | I am in the process of understanding Tempest. I'll soon be testing EC2 api compatibility between cinder and EBS, and fill in the gaps if there exists | 11:27 |
| *** jorisroovers has quit IRC | 11:27 | |
| *** jorisroovers has joined #openstack-qa | 11:27 | |
| openstackrecheck | openstack-infra/devstack-gate change: https://review.openstack.org/57325 failed tempest with an unrecognized error | 11:31 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/56893 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 11:31 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/tempest: Enable a uuid flavor https://review.openstack.org/56326 | 11:34 |
| openstackgerrit | Mauro S. M. Rodrigues proposed a change to openstack-dev/grenade: Add clean configuration files to upgrade from H to I https://review.openstack.org/53874 | 11:43 |
| *** jroovers has joined #openstack-qa | 11:44 | |
| *** jorisroovers has quit IRC | 11:46 | |
| openstackrecheck | openstack/ceilometer change: https://review.openstack.org/58488 failed tempest because of: https://bugs.launchpad.net/bugs/1210483 | 11:51 |
| openstackgerrit | A change was merged to openstack/tempest: port test_availability_zone into nova v3 part2 https://review.openstack.org/43070 | 11:52 |
| openstackgerrit | A change was merged to openstack-dev/devstack: edit-glance-manage-command-for-recreate-db https://review.openstack.org/47552 | 11:53 |
| *** gszasz has quit IRC | 11:58 | |
| *** gszasz has joined #openstack-qa | 12:00 | |
| cyeoh | afazekas, yea I agree. In retrospect that was a mistake on my part | 12:00 |
| cyeoh | afazekas: hopefully it won't come back to bite me. | 12:01 |
| afazekas | :) | 12:01 |
| afazekas | try to approve the https://review.openstack.org/#/c/57589/ and we will see :) | 12:01 |
| cyeoh | heh, I can - is the gate still broken? | 12:02 |
| cyeoh | or at least I shouldn't approve it, but you can... | 12:02 |
| cyeoh | (there is the recent email on openstack-dev saying its broken atm) | 12:04 |
| afazekas | it is broken , but not tempest related: https://bugs.launchpad.net/openstack-ci/+bug/1255041 | 12:04 |
| *** Madan has quit IRC | 12:05 | |
| cyeoh | afazekas: ah I see. Well I think you should approve 57589 - was just waiting for jog0 to say he was happy with it really. We've got a fair number of V3 API Nova changesets moving through at the moment and I'd really prefer we were gating on them :-) | 12:09 |
| cyeoh | afazekas: s/should/could/ - not trying to tell you what to do :-) | 12:10 |
| *** nosnos has quit IRC | 12:11 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/40187 failed tempest with an unrecognized error | 12:11 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: Support Neutron security groups in scenario testing https://review.openstack.org/50892 | 12:32 |
| openstackgerrit | Joe Hakim Rahme proposed a change to openstack/tempest: Adds improvements to the Swift TempURL test https://review.openstack.org/57287 | 12:41 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/55411 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 12:54 |
| *** dims has joined #openstack-qa | 12:55 | |
| *** markmcclain has joined #openstack-qa | 12:56 | |
| *** SushilKM has joined #openstack-qa | 12:58 | |
| *** dkranz has joined #openstack-qa | 13:01 | |
| *** rushiagr has quit IRC | 13:02 | |
| *** jhenner1 has joined #openstack-qa | 13:03 | |
| *** jhenner has quit IRC | 13:04 | |
| *** sthaha has quit IRC | 13:04 | |
| *** sthaha has joined #openstack-qa | 13:05 | |
| *** sthaha has quit IRC | 13:05 | |
| *** sthaha has joined #openstack-qa | 13:05 | |
| *** sthaha has quit IRC | 13:07 | |
| *** SushilKM has quit IRC | 13:09 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/55057 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 13:16 |
| openstackrecheck | openstack/python-glanceclient change: https://review.openstack.org/43689 failed tempest with an unrecognized error | 13:17 |
| *** mkollaro has quit IRC | 13:18 | |
| *** mkollaro1 has joined #openstack-qa | 13:18 | |
| *** enikanorov_ has joined #openstack-qa | 13:19 | |
| *** obondarev has joined #openstack-qa | 13:19 | |
| *** weshay has joined #openstack-qa | 13:21 | |
| *** qianlin has joined #openstack-qa | 13:26 | |
| *** rahmu has quit IRC | 13:26 | |
| *** rahmu has joined #openstack-qa | 13:28 | |
| *** SushilKM has joined #openstack-qa | 13:28 | |
| *** amotoki has joined #openstack-qa | 13:37 | |
| *** rahmu has quit IRC | 13:47 | |
| *** rahmu has joined #openstack-qa | 13:49 | |
| *** markmcclain has quit IRC | 13:50 | |
| dkranz | mtreinish: ping | 13:54 |
| dkranz | mtreinish: Can you send an email to the list explaining the impact of the conf auto-generation and what the new workflow is? | 13:55 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/58234 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 13:58 |
| *** yfried has quit IRC | 14:01 | |
| *** mkollaro has joined #openstack-qa | 14:03 | |
| *** SushilKM has quit IRC | 14:05 | |
| dkranz | mtreinish: Also, see https://bugs.launchpad.net/tempest/+bug/1255105 I just filed. | 14:06 |
| *** mkollaro1 has quit IRC | 14:06 | |
| dkranz | mtreinish: generate_sample does not work | 14:06 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/58415 failed tempest because of: https://bugs.launchpad.net/bugs/1224001 and https://bugs.launchpad.net/bugs/1253896 | 14:07 |
| *** jhenner1 has quit IRC | 14:09 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/58286 failed tempest with an unrecognized error | 14:09 |
| *** jd__` has joined #openstack-qa | 14:09 | |
| *** jd__ has quit IRC | 14:10 | |
| *** jd__` is now known as jd__ | 14:10 | |
| *** mriedem has joined #openstack-qa | 14:11 | |
| *** tkammer has quit IRC | 14:12 | |
| *** tkammer has joined #openstack-qa | 14:13 | |
| *** salv-orlando has quit IRC | 14:15 | |
| *** psharma has quit IRC | 14:16 | |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/58286 failed tempest because of: https://bugs.launchpad.net/bugs/1239637 and https://bugs.launchpad.net/bugs/1253896 | 14:16 |
| *** salv-orlando has joined #openstack-qa | 14:21 | |
| *** prekarat has joined #openstack-qa | 14:22 | |
| *** jhenner has joined #openstack-qa | 14:22 | |
| openstackrecheck | openstack-dev/pbr change: https://review.openstack.org/57892 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 14:24 |
| *** dkranz has quit IRC | 14:26 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/54941 failed tempest with an unrecognized error | 14:34 |
| *** qianlin has quit IRC | 14:35 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Add Savanna testing support https://review.openstack.org/57325 | 14:41 |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/57083 failed tempest with an unrecognized error | 14:43 |
| openstackgerrit | A change was merged to openstack-dev/devstack: support memcache for keystone token backend https://review.openstack.org/56691 | 14:46 |
| openstackgerrit | Abhijeet Malawade proposed a change to openstack/tempest: Add testcases for security groups and images https://review.openstack.org/47079 | 14:47 |
| *** masayukig has joined #openstack-qa | 14:52 | |
| mtreinish | dkranz: you need to set an env variable to make that work with tempest. | 14:53 |
| mtreinish | MODULEPATH=tempest.common.generate_sample_tempest | 14:53 |
| openstackgerrit | Abhijeet Malawade proposed a change to openstack/tempest: Add testcases for security groups and images https://review.openstack.org/47079 | 14:53 |
| mtreinish | dkranz: I'll send a reply to the list about it being live now | 14:55 |
| mtreinish | and how to use it | 14:55 |
| *** jroovers has quit IRC | 14:56 | |
| sdague | mtreinish: seems like there should be a wrapper for that, as well as docs in the README | 15:01 |
| mtreinish | sdague: I thought about a wrapper script but it would honestly be a one line script setting the env variable and calling the script with all the args | 15:02 |
| mtreinish | that seemed silly to me | 15:02 |
| mtreinish | yeah I'll update the docs and the readme | 15:02 |
| sdague | mtreinish: it's UX, because discovering how to do it isn't obvious | 15:02 |
| sdague | I'm all for one line scripts that make things discoverable | 15:03 |
| mtreinish | sdague: If you run tox -epep8 and it fails on the config check it does describe the proper procedure for running the script (with the env variable) | 15:05 |
| mtreinish | which I thought was sufficient | 15:05 |
| mtreinish | but I can add a wrapper and fix up the readme | 15:05 |
| *** dkranz has joined #openstack-qa | 15:15 | |
| *** tkammer has quit IRC | 15:29 | |
| *** DennyZhang has joined #openstack-qa | 15:32 | |
| *** DennyZhang has quit IRC | 15:34 | |
| *** DennyZhang has joined #openstack-qa | 15:36 | |
| afazekas | Is it safe to approve https://review.openstack.org/#/c/57589/ ? | 15:37 |
| mtreinish | afazekas: yes I believe it is | 15:38 |
| afazekas | https://review.openstack.org/#/c/50337/15 rebased, requesting review | 15:41 |
| openstackrecheck | openstack/oslo.messaging change: https://review.openstack.org/58531 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 15:42 |
| afazekas | https://review.openstack.org/#/c/57402/ another ssh related change, IMHO if we dont want to use the implicit connection validation, the __init__ should not do at all | 15:42 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Improve the UX around sample config generation https://review.openstack.org/58549 | 15:43 |
| mtreinish | sdague: ^^^ | 15:43 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/57479 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 15:45 |
| afazekas | mtreinish: the MODULEPATH is not exported and it is on a different line | 15:46 |
| mtreinish | afazekas: oops | 15:47 |
| * afazekas suspect another issue .. | 15:48 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Improve the UX around sample config generation https://review.openstack.org/58549 | 15:48 |
| mtreinish | afazekas: on 57589 you have to rebase again because of the sample config generation | 15:50 |
| afazekas | :) | 15:50 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/57480 failed tempest because of: https://bugs.launchpad.net/bugs/1224001 | 15:52 |
| *** jgrimm has joined #openstack-qa | 15:56 | |
| *** yfried has joined #openstack-qa | 15:57 | |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/58284 failed tempest because of: https://bugs.launchpad.net/bugs/1249065 | 15:58 |
| *** afazekas has quit IRC | 16:02 | |
| *** jgrimm has quit IRC | 16:02 | |
| openstackgerrit | Mauro S. M. Rodrigues proposed a change to openstack-dev/grenade: Add clean configuration files to upgrade from H to I https://review.openstack.org/53874 | 16:03 |
| *** SergeyLukjanov has quit IRC | 16:08 | |
| openstackrecheck | openstack-infra/devstack-gate change: https://review.openstack.org/57325 failed tempest with an unrecognized error | 16:10 |
| *** jgrimm has joined #openstack-qa | 16:13 | |
| sdague | dtroyer: https://review.openstack.org/#/c/57744/ got a chance to look into that today? | 16:13 |
| openstackgerrit | Armando Migliaccio proposed a change to openstack/tempest: Ensure no dangling resources are left if tests are skipped https://review.openstack.org/58054 | 16:18 |
| *** masayukig has quit IRC | 16:20 | |
| *** masayukig has joined #openstack-qa | 16:21 | |
| dtroyer | sdague: sure | 16:22 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58548 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 16:22 |
| *** masayukig has quit IRC | 16:25 | |
| sdague | because I think that change will let me repropose the d-g changes | 16:25 |
| *** rhsu has joined #openstack-qa | 16:26 | |
| *** rhsu1 has joined #openstack-qa | 16:30 | |
| *** rhsu has quit IRC | 16:30 | |
| openstackrecheck | openstack/heat change: https://review.openstack.org/58552 failed tempest with an unrecognized error | 16:30 |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/58550 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 16:33 |
| openstackrecheck | openstack/swift change: https://review.openstack.org/56692 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 16:34 |
| *** rhsu1 has quit IRC | 16:34 | |
| openstackrecheck | openstack/swift change: https://review.openstack.org/58400 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 16:35 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/30988 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 16:36 |
| portante | Hello folks, any review love for https://review.openstack.org/#/c/57544/ and https://review.openstack.org/#/c/57526/ ? | 16:38 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/50749 failed tempest because of: https://bugs.launchpad.net/bugs/1254872 | 16:38 |
| portante | I'm beggin' and pleadin' | 16:38 |
| sdague | portante: both those look straight forward to me, I'll let dtroyer weigh in as well | 16:40 |
| dtroyer | portante: queue depth is hurting more than just the gate tests…I think I need a personal zuul | 16:41 |
| openstackgerrit | Mauro S. M. Rodrigues proposed a change to openstack-dev/grenade: Add clean configuration files to upgrade from H to I https://review.openstack.org/53874 | 16:42 |
| *** rhsu has joined #openstack-qa | 16:42 | |
| dtroyer | portante: I did have a question on 57526: was 12 chosen for the length for a specific reason? Since it is set for both PKI and UUID tokens, maybe len(UUID)? | 16:43 |
| portante | dtroyer: just to keep it short, 12 characters of resolution on a PKI or UUID token would be enough to search for it across the logs to correlate things | 16:45 |
| dtroyer | portante: OK, I wasn't sure if that was long enough for UUID, having never actually thought about it before. thanks | 16:46 |
| *** SergeyLukjanov has joined #openstack-qa | 16:47 | |
| *** masayukig has joined #openstack-qa | 16:51 | |
| portante | dtroyer: welcome | 16:52 |
| openstackgerrit | Peter Portante proposed a change to openstack-dev/devstack: Shorten PKI token logging https://review.openstack.org/57526 | 16:54 |
| openstackgerrit | Peter Portante proposed a change to openstack-dev/devstack: Use the swift logging adapter for txn IDs https://review.openstack.org/57544 | 16:54 |
| dkranz | sdague, mtreinish ; Can you guys approve https://review.openstack.org/#/c/58547/ ? I need it to debug the whitelist flaky issue. | 16:54 |
| *** masayukig has quit IRC | 16:55 | |
| mtreinish | dkranz: +2 | 16:56 |
| dkranz | mtreinish: Thanks. | 16:56 |
| sdague | dkranz: done | 17:01 |
| dkranz | sdague: Thanks. On to the hard part... | 17:02 |
| *** jgrimm has quit IRC | 17:08 | |
| *** markmcclain has joined #openstack-qa | 17:09 | |
| *** rhsu has quit IRC | 17:09 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58538 failed tempest with an unrecognized error | 17:15 |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/58054 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 17:15 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/57479 failed tempest with an unrecognized error | 17:15 |
| *** SushilKM has joined #openstack-qa | 17:15 | |
| openstackrecheck | openstack/horizon change: https://review.openstack.org/58529 failed tempest because of: https://bugs.launchpad.net/bugs/1225024 | 17:15 |
| openstackrecheck | openstack/python-keystoneclient change: https://review.openstack.org/57541 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 17:16 |
| *** SergeyLukjanov has quit IRC | 17:23 | |
| *** SergeyLukjanov has joined #openstack-qa | 17:23 | |
| openstackgerrit | Dean Troyer proposed a change to openstack-dev/devstack: Eradicate last of underscores in options https://review.openstack.org/58588 | 17:24 |
| openstackgerrit | Dean Troyer proposed a change to openstack-dev/devstack: freshen the LDAP support https://review.openstack.org/58590 | 17:25 |
| *** reaper has joined #openstack-qa | 17:27 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58352 failed tempest because of: https://bugs.launchpad.net/bugs/1250168 | 17:28 |
| openstackgerrit | Dean Troyer proposed a change to openstack-dev/devstack: freshen the LDAP support https://review.openstack.org/58590 | 17:29 |
| *** BobBall has joined #openstack-qa | 17:30 | |
| *** safchain has quit IRC | 17:31 | |
| BobBall | Probably a question that's been answered to death... But why do we test everything twice, once for XML and once for JSON? | 17:34 |
| *** rhsu has joined #openstack-qa | 17:37 | |
| *** markmcclain has quit IRC | 17:39 | |
| openstackgerrit | A change was merged to openstack/tempest: Add hard reboot for outputting console log https://review.openstack.org/58478 | 17:47 |
| *** rossella_s has quit IRC | 17:49 | |
| openstackgerrit | A change was merged to openstack/tempest: Enable the Nova V3 API Tests https://review.openstack.org/57589 | 17:51 |
| *** masayukig has joined #openstack-qa | 17:52 | |
| *** masayukig has quit IRC | 17:53 | |
| *** masayukig has joined #openstack-qa | 17:54 | |
| *** SushilKM has quit IRC | 17:55 | |
| *** hemnafk is now known as hemna | 17:56 | |
| *** jlibosva has quit IRC | 17:58 | |
| *** jlibosva has joined #openstack-qa | 17:58 | |
| *** masayukig has quit IRC | 17:58 | |
| *** gszasz has quit IRC | 18:14 | |
| jog0 | cyeoh: I am happy with it | 18:15 |
| *** atiwari has joined #openstack-qa | 18:16 | |
| *** DennyZhang has quit IRC | 18:17 | |
| dkranz | BobBall: I don't think it has been answered to death. | 18:18 |
| dkranz | sdague: Is there a reason we can't, say, run only json tests in the mysql run and only xml in the postgres run? | 18:19 |
| openstackrecheck | openstack-dev/devstack change: https://review.openstack.org/58588 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 18:21 |
| openstackrecheck | openstack/ceilometer change: https://review.openstack.org/58587 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 18:24 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: scenario/network_basic_ops: reassociate floating-ip https://review.openstack.org/56889 | 18:25 |
| openstackgerrit | Yair Fried proposed a change to openstack/tempest: scenario/network_basic_ops: detach floating-ip https://review.openstack.org/55810 | 18:25 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/57812 failed tempest because of: https://bugs.launchpad.net/bugs/1224001 and https://bugs.launchpad.net/bugs/1253896 | 18:29 |
| openstackgerrit | gordon chung proposed a change to openstack-dev/devstack: split collector service https://review.openstack.org/58600 | 18:32 |
| *** prekarat has quit IRC | 18:34 | |
| *** SergeyLukjanov_ has joined #openstack-qa | 18:37 | |
| *** SergeyLukjanov has quit IRC | 18:37 | |
| *** mkollaro has quit IRC | 18:37 | |
| marun | sdague: ping | 18:39 |
| mtreinish | dkranz: we've talked about doing that before | 18:40 |
| mtreinish | but with parallel the time to complete isn't that bad so there really isn't a need anymore | 18:41 |
| dkranz | mtreinish: Did we decide not to do it, or there just wasn't a volunteer | 18:41 |
| *** jlibosva has quit IRC | 18:41 | |
| *** SergeyLukjanov has joined #openstack-qa | 18:41 | |
| marun | -1 to the the idea that more is always better | 18:41 |
| dkranz | mtreinish: I don't agree | 18:41 |
| mtreinish | dkranz: parallel testing was the priority and I think we decided to revist it after that was done | 18:41 |
| dkranz | mtreinish: The most valuable thing we could do right now is run a lot more jobs before allowing a merge. | 18:41 |
| mtreinish | marun: I agree | 18:41 |
| dkranz | mtreinish: I think it would be pretty easy to do. | 18:42 |
| marun | dkranz: isn't that still arguing 'more is better'? ;) | 18:42 |
| *** fbo is now known as fbo_away | 18:42 | |
| dkranz | marun: No | 18:42 |
| dkranz | marun: More as in "reduce chance a flaky bug gets in" is different | 18:42 |
| *** SergeyLukjanov_ has quit IRC | 18:42 | |
| dkranz | marun: But we are in a catch 22 right now | 18:43 |
| marun | dkranz: that's assuming that the best way to find these bugs is using the current approach | 18:43 |
| marun | dkranz: although, no good alternative in the short term :( | 18:43 |
| marun | dkranz: agreed on catch 22 | 18:43 |
| dkranz | marun: If we new we had no flaky bugs we should run tempest 100 times before we let it through | 18:43 |
| dkranz | marun: But we can't do that now because nothing would actually get through | 18:44 |
| marun | dkranz: right. so cutting down execution time for a given run is very much desirable even with parallel testing | 18:44 |
| dkranz | mtreinish: I will make a proposal. I think we just need to rename the non-XML classes to JSON.. for those not already doing that | 18:44 |
| dkranz | mtreinish: Then filter one run to exclude "XML" and the other to exclude "JSON" | 18:45 |
| dkranz | mtreinish: They would both run tests that didn't have xml and json | 18:45 |
| mtreinish | dkranz: I actually think having them together is better it helps us catch race conditions. It's the only way we can run the same test at the same time. | 18:45 |
| dkranz | mtreinish: But they usually do not run at the same time. | 18:46 |
| mtreinish | dkranz: you wouldn't need to update the JSON class names. Just do one XML and one not XML that should be enough | 18:46 |
| dkranz | mtreinish: I suppose so. | 18:47 |
| dkranz | mtreinish: But I always thought it a little odd that we have Foo and XMLFoo | 18:47 |
| dkranz | mtreinish: I'll put this on the adenda for the next meeting. | 18:48 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Allow overriding USE_GET_PIP via env vars https://review.openstack.org/58104 | 18:48 |
| mtreinish | dkranz: but that's the only way they can run at the same. (and the earlier run tests sometimes do run at the same time, I've caught bugs from it) | 18:48 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Move neutron cache dir into a function https://review.openstack.org/58394 | 18:49 |
| mtreinish | dkranz: yeah it is weird, I think it was mostly laziness/not wanting to change an existing class when we were adding the XML testing | 18:49 |
| dkranz | mtreinish: No big deal. I'm just suggesting we be more purposeful about what is getting run together to find bugs vs for some other reason | 18:50 |
| dkranz | mtreinish: ANd that I'm sure solutions to the flaky issue will require more resources | 18:50 |
| sdague | BobBall: because, the moment we started testing xml in addition to json, we found it was full of bugs. Tests that we turn on, and they find issues, are tests we tend to leave on, because we know review didn't remove those issues. | 18:54 |
| *** masayukig has joined #openstack-qa | 18:54 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58598 failed tempest with an unrecognized error | 18:55 |
| sdague | so, I'm confused, why is this coming up again? | 18:55 |
| sdague | honestly, until someone comes up with tests that ensure we hit aggregates (and other global things) in parallel, I'm -2 on splitting out the json/xml jobs | 18:57 |
| *** jlibosva has joined #openstack-qa | 18:58 | |
| *** masayukig has quit IRC | 18:59 | |
| jog0 | sdague: https://review.openstack.org/#/c/57367/1 | 19:03 |
| sdague | jog0: lgtm | 19:04 |
| jog0 | sdague: thanks | 19:04 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove queries for dead bugs https://review.openstack.org/57367 | 19:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add doc on queries.yaml https://review.openstack.org/57368 | 19:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add launchpad support to check_success https://review.openstack.org/57374 | 19:07 |
| openstackrecheck | openstack/cinder change: https://review.openstack.org/50917 failed tempest with an unrecognized error | 19:13 |
| *** openstackrecheck has quit IRC | 19:13 | |
| *** openstackrecheck has joined #openstack-qa | 19:13 | |
| *** mlavalle has joined #openstack-qa | 19:15 | |
| marun | sdague: so, modest proposal | 19:20 |
| marun | sdague: we're working on getting request ids logged by neutron services and returned to clients. then we need to log those request ids in the tempest clients, and correlate failures that appear in the logs with the originating request | 19:21 |
| mtreinish | marun: tempest already logs request ids for nova, cinder, and glance | 19:22 |
| marun | mtreinish: so we just have to add the same for neutron. i think that's the easy part, in any case | 19:22 |
| mtreinish | marun: yeah it's super simple: https://git.openstack.org/cgit/openstack/tempest/tree/tempest/common/rest_client.py#n352 | 19:23 |
| marun | mtreinish: any thoughts as to how to correlate log errors tagged with a request id with the corresponding client request? | 19:23 |
| marun | mtreinish: I'm imagining being able to have it equivalent to an assertion failure | 19:24 |
| marun | mtreinish: line xyz caused the following error(s): | 19:24 |
| marun | mtreinish: thoughts? | 19:24 |
| mtreinish | marun: the way we do it now is that the request ids get printed on the console output during a test failure | 19:24 |
| mtreinish | so then we can use that when parsing logs | 19:24 |
| marun | mtreinish: so, done manually for now? | 19:24 |
| marun | mtreinish: or is there some magic to the parsing you are talking about? | 19:25 |
| *** jhenner has quit IRC | 19:25 | |
| mtreinish | no the log parsing is done manually right now | 19:25 |
| mtreinish | having a tool do it would be nice | 19:25 |
| mtreinish | sdague: could you extend os-log-analyzer to just return log messages from a certain request-id? | 19:26 |
| sdague | mtreinish: possibly | 19:26 |
| sdague | though realistically logstash is a better option, as then you get to see all the logs | 19:26 |
| mtreinish | sdague: oh yeah that's true | 19:27 |
| mtreinish | marun: I forgot about logstash, that makes the parsing pretty easy | 19:27 |
| marun | mtreinish, sdague: so it is in the realm of possibility to output correlation between failures logged in services and tempest module/class/test/line as part of a gate job? | 19:29 |
| mtreinish | marun: yeah you could just have a post run script that will look at the subunit log for request id messages then queries logstash for those request ids and dumps them in a seperate file(or some other reporting mechanism) | 19:32 |
| marun | mtreinish: ok. I'll look into it and will probably have more questions. I think getting this capability could be helpful in diagnosing the current neutron issues. | 19:33 |
| mtreinish | clarkb: what is the delay for populating logstash after a run like these days? | 19:33 |
| sdague | marun: honestly, instead of writing that, I'd get familiar with logstash | 19:33 |
| sdague | you get it all for free with that | 19:33 |
| *** fbo_away is now known as fbo | 19:34 | |
| marun | sdague: so you don't think having direct correlation is valuable? | 19:34 |
| dkranz | This is very similar to what the whitelist checker does now. | 19:34 |
| marun | sdague: I mean, we have correlation for non-async errors now. | 19:34 |
| sdague | marun: I think that you actually get it today | 19:34 |
| sdague | with logstash | 19:35 |
| marun | sdague: the errors print right in the log | 19:35 |
| sdague | without having to write new things | 19:35 |
| marun | sdague: I | 19:35 |
| sdague | anyway, otp | 19:35 |
| mtreinish | marun: there would be some limitations with this approach too. Most tests have several request-ids generated. (they make several api calls) Also logs from setUpClass don't get injected into the subunit stream so you wouldn't get the ids from those api calls | 19:35 |
| marun | mtreinish: that is a problem | 19:36 |
| marun | mtreinish: or maybe just another reason not to use setupClass ? | 19:36 |
| marun | sdague: instructions on logstash? | 19:36 |
| marun | sdague: I think the ideal case would be direct correlation so not everyone had to know how to use logstash, but I should certainly know how to use it in the interim | 19:37 |
| marun | dkranz: whitelist checker? | 19:37 |
| dkranz | marun: It scans the log files for errors and prints them to the console log | 19:38 |
| dkranz | marun: I think Sean was suggesting using logstash | 19:38 |
| mtreinish | marun: the setupClass ids get logged, but we use a fixture during the base class setUp() to turn on the log injection which obviously setUpClass() happens before | 19:38 |
| marun | oh, https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors | 19:38 |
| clarkb | mtreinish: it is quite fast now that we don't index debug logs, under 5 minutes consistently and usually under a minute | 19:39 |
| mtreinish | clarkb: ok cool | 19:40 |
| openstackrecheck | openstack/ceilometer change: https://review.openstack.org/57603 failed tempest with an unrecognized error | 19:40 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Add config options for enabled extensions https://review.openstack.org/58604 | 19:41 |
| marun | dkranz: how is progress with that blueprint? I was talking to markmcclain about this very issue yesterday. | 19:42 |
| *** yfried has quit IRC | 19:42 | |
| dkranz | I am pushing the final entries into the whitelist and then we are going to turn it on. | 19:42 |
| dkranz | marun: I was actually working on that today. | 19:43 |
| dkranz | marun: THe trouble has been that every few days new ones show up. | 19:43 |
| EmilienM | sdague: i'm comparing clean.sh from devstack & grenade and i'm wondering why clean.sh from grenade does not use cleanup_$PROJECT | 19:43 |
| EmilienM | sdague: i'm working on https://review.openstack.org/#/c/58401/ by the way | 19:44 |
| marun | dkranz: We have too many errors in the neutron logs at present :( | 19:44 |
| dkranz | marun: Yes, I was going to ignore neutron for now. | 19:45 |
| mtreinish | dkranz: careful that's a slippery slope... | 19:45 |
| marun | mtreinish: I agree in principle, but I'm not sure we need to add more gate churn right now. | 19:46 |
| marun | mtreinish: I'm hoping we can get the races under control and then move on to erroring on things that might not actually be causes for concern. | 19:47 |
| dkranz | mtreinish: I'm putting in mechanism, not policy :) | 19:50 |
| dkranz | mtreinish: I'm happy to slam the gate shut as much as will be tolerated. | 19:51 |
| dkranz | mtreinish: The problem is that it is a lot of work to add a zillion failures to the white list and I don't think it is worth it for neutron now. | 19:51 |
| mtreinish | dkranz: I know, it's just another exception for neutron gating | 19:52 |
| mtreinish | I want to see that list shrinking not growing | 19:52 |
| dkranz | mtreinish: Do you have a better idea? | 19:53 |
| mtreinish | dkranz: no I think it's what we need to do | 19:53 |
| dkranz | mtreinish: Sadly, yes. | 19:53 |
| openstackrecheck | openstack/python-glanceclient change: https://review.openstack.org/58405 failed tempest with an unrecognized error | 19:55 |
| *** masayukig has joined #openstack-qa | 19:55 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Up the default concurrency on tempest runs https://review.openstack.org/58605 | 19:56 |
| *** torgomatic has left #openstack-qa | 19:58 | |
| *** masayukig has quit IRC | 20:00 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/55745 failed tempest with an unrecognized error | 20:01 |
| adalbas | dkranz, about the whitelist, those are the errors that are expected to happen in normal tempest execution. But could they also be an unexpected error? How reliable are we expecting that? | 20:03 |
| *** rfolco has quit IRC | 20:07 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/55747 failed tempest because of: https://bugs.launchpad.net/bugs/1249065 | 20:09 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-dev/grenade: Add neutron upgrade process https://review.openstack.org/58401 | 20:09 |
| *** jlibosva has quit IRC | 20:11 | |
| *** Nayana has joined #openstack-qa | 20:15 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/55818 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 20:26 |
| openstackrecheck | openstack-infra/devstack-gate change: https://review.openstack.org/57066 failed tempest with an unrecognized error | 20:27 |
| openstackrecheck | openstack/requirements change: https://review.openstack.org/58362 failed tempest with an unrecognized error | 20:27 |
| *** Nayana has quit IRC | 20:29 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/55814 failed tempest with an unrecognized error | 20:30 |
| dkranz | adalbas: The goal is for the whitelist to be empty and any new errors will fail the merge. | 20:34 |
| dkranz | adalbas: But there are a ton of errors now. They are all bugs but I don't know when they will get fixed. | 20:34 |
| dkranz | adalbas: When the white list is capturing all current errors we will start failing jobs on errors in the logs. | 20:35 |
| dkranz | adalbas: As the bugs are fixed, we remove their strings from the whitelist. | 20:35 |
| adalbas | dkranz, ok, i got the idea now! tks | 20:38 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Add config options for enabled extensions https://review.openstack.org/58604 | 20:40 |
| *** DennyZhang has joined #openstack-qa | 20:45 | |
| openstackrecheck | openstack/nova change: https://review.openstack.org/54493 failed tempest with an unrecognized error | 20:46 |
| *** jamielennox is now known as jamielennox|away | 20:52 | |
| *** masayukig has joined #openstack-qa | 20:56 | |
| *** masayukig has quit IRC | 20:57 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-dev/grenade: Add neutron upgrade process https://review.openstack.org/58401 | 20:57 |
| *** masayukig has joined #openstack-qa | 20:57 | |
| *** jamielennox|away is now known as jamielennox | 21:00 | |
| openstackrecheck | openstack-infra/devstack-gate change: https://review.openstack.org/58605 failed tempest because of: https://bugs.launchpad.net/bugs/1210483 and https://bugs.launchpad.net/bugs/1250168 | 21:01 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/tempest: Enable a uuid flavor https://review.openstack.org/56326 | 21:02 |
| *** masayukig has quit IRC | 21:03 | |
| sdague | marun: sorry, other meetings | 21:03 |
| sdague | http://logstash.openstack.org | 21:03 |
| openstackrecheck | openstack/neutron change: https://review.openstack.org/57812 failed tempest because of: https://bugs.launchpad.net/bugs/1224001 and https://bugs.launchpad.net/bugs/1253896 | 21:04 |
| openstackgerrit | David Kranz proposed a change to openstack/tempest: Add hopefully last batch to the whitelist https://review.openstack.org/58617 | 21:05 |
| marun | sdague: Not to discount the utility of logstash, but it sure would be helpful if developers could correlate tempest failures during local runs | 21:05 |
| marun | sorry, by correlate I mean be able to trace actions taken by tempest via the clients and operations on the backend | 21:06 |
| marun | having to do it manually sucks. | 21:06 |
| sdague | marun: so sure, I'm just saying that we're building a ton of infrastructure around logstash, so the fastest path there might be adding a mini logstash to devstack | 21:06 |
| marun | I'm not saying this is anything like an easy fix, just something to shoot for eventually. | 21:06 |
| marun | sdague: ah, fair enough | 21:06 |
| sdague | which is something that's been talked about | 21:06 |
| sdague | just because you can then just dump "req-XXXXXXXXXXXXXXXXXXXXXXXXXXXX" into the search field, and yuo get the entire request path | 21:07 |
| sdague | regardless of logs | 21:07 |
| sdague | I guess the other option would be pushing all the logs to syslog, so you'd get unification there, then you could use grep | 21:08 |
| sdague | that would be relatively light weight | 21:08 |
| marun | do you have docs to suggest for logstash? | 21:08 |
| marun | sdague: I'm afraid I'm a bit slow | 21:09 |
| marun | It still strikes me to bea bit of a hammer looking for a nail when it comes to tracking down failures in an individual run. | 21:10 |
| sdague | marun: it might be. it's just there are a lot of nails to strike on things like this, so having a big hammer than handles a lot of cases is sort of nice. | 21:11 |
| sdague | but I'd agree that my focus is trying to get the gate more debugable so that we can nail down the races | 21:12 |
| marun | No argument from me as to the utility of using logstash. | 21:12 |
| *** goqu has joined #openstack-qa | 21:13 | |
| marun | sdague: I've asked a few times, and I apologize if its a stupid question, but are they are docs that you know of to use logstash effectively? | 21:13 |
| marun | sdague: or is it just something to discover by playing with it? | 21:15 |
| sdague | marun: honestly, I mostly just used their main website. And then read some of the elastic search API docs as that's what we are using with elastic-recheck | 21:15 |
| sdague | marun: a bunch of it is trial and error playing around | 21:15 |
| marun | sdague: so all the results are in there and i just filter from 'all'? | 21:16 |
| sdague | marun: how much do yuo know about lucene? | 21:16 |
| marun | sdague: nil, would that be a place to start? | 21:16 |
| sdague | it's probably easiest to build up the model of what's going on from that | 21:16 |
| sdague | ok, let me give you the overview of what this is | 21:16 |
| sdague | lucene is an open source search engine, basically indexes documents, and makes them full text searchable, smartly. Also documents can have metadata, which you can use to adjust the search. | 21:17 |
| sdague | elastic search is a clustered lucene, that horizontally scales and shards it so you can handle big indexes | 21:18 |
| marun | ok | 21:18 |
| sdague | openstack-infra runs a 6 node es cluster for this | 21:18 |
| sdague | logstash is a system that takes log files, and makes them search engine documents, 1 per line of log file | 21:18 |
| sdague | which you can metadata tag however you like | 21:18 |
| openstackrecheck | openstack/keystone change: https://review.openstack.org/53964 failed tempest with an unrecognized error | 21:19 |
| sdague | in openstack-infra we run all logs that come out of test runs into logstash (at INFO level and above) | 21:19 |
| sdague | and tag each line with the build id, job name, success status, and about 10 other useful metadata things | 21:20 |
| *** yassine has quit IRC | 21:20 | |
| sdague | and keep the last 10ish days of it (that's as much as our cluster currently handles) | 21:20 |
| openstackgerrit | David Kranz proposed a change to openstack/tempest: Add hopefully last batch to the whitelist https://review.openstack.org/58617 | 21:20 |
| sdague | kibana is the nice web front end to all of this, which you can access via http://logstash.openstack.org | 21:20 |
| sdague | you can also hit the elastic-search api directly, which we do to do things like this - http://status.openstack.org/elastic-recheck/ | 21:21 |
| marun | ah, so that's how that works | 21:21 |
| marun | cool | 21:21 |
| sdague | yeh, I should blog this up with pictures. :) (/me adds to list) | 21:22 |
| marun | :) | 21:22 |
| marun | sdague: so, the kibana interface. I'm having trouble figuring out how to drill down to a specific neutron job | 21:23 |
| marun | the column filtering on the left looks cool but it seems to display too limited a set of results to be useful | 21:24 |
| *** mlavalle has quit IRC | 21:24 | |
| jog0 | sdague: just had fun reading the TC meeting notes | 21:24 |
| marun | sdague: so if I wanted to filter by, say 'check-tempest-devstack-vm-neutron', what do I do? | 21:24 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/58577 failed tempest because of: https://bugs.launchpad.net/bugs/1251448 | 21:24 |
| *** safchain has joined #openstack-qa | 21:25 | |
| sdague | marun: build_name:"check-tempest-devstack-vm-neutron" | 21:25 |
| sdague | AND and OR do things too | 21:26 |
| marun | ah, ok. so the column filtering only takes one so far and then manual query building comes into play | 21:26 |
| sdague | build_name:"check-tempest-devstack-vm-neutron" AND message:"something" | 21:26 |
| jog0 | marun: for all neutron you can do ' build_name:*neutron*' | 21:26 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Add Savanna testing support https://review.openstack.org/57325 | 21:26 |
| sdague | message == text of the log line, the rest of the fields you might want to use are the things on the left | 21:27 |
| marun | sdague, jog0: thank you, most helpful. | 21:27 |
| sdague | cool, no problem | 21:27 |
| dkranz | mtreinish, sdague : Getting closer... | 21:28 |
| dkranz | https://review.openstack.org/#/c/58617/ | 21:28 |
| dkranz | mtreinish, sdague : I'm just *ing the neutron files when I see something. Tired of that. | 21:29 |
| *** ken1ohmichi has joined #openstack-qa | 21:31 | |
| *** masayukig has joined #openstack-qa | 21:33 | |
| marun | dkranz: will the check for errors be run after every check/gate job? | 21:34 |
| marun | dkranz: any way to report results for neutron without failing the job? | 21:34 |
| dkranz | marun: Yes | 21:34 |
| marun | dkranz: to both questions? | 21:34 |
| sdague | marun: I think that's basically what the wildcard exclude will do | 21:34 |
| dkranz | marun: Yes | 21:34 |
| marun | dkranz: ok, cool. | 21:35 |
| sdague | because everything it finds it prints to stdout at the end | 21:35 |
| dkranz | marun: It would be easy to just dump all neutron errors. Do you want me to do that? | 21:35 |
| dkranz | marun: I can remove the partial whitelist for neutron that is there. | 21:35 |
| marun | dkranz: yes | 21:35 |
| marun | dkranz: I think making it noisy will be a good way to encourage people to start caring | 21:35 |
| *** devlaps has joined #openstack-qa | 21:36 | |
| dkranz | marun: So I will need to change the script to not fail neutron. I think there is a shell variable that will tell if we are neutron. | 21:36 |
| marun | dkranz: awesome! | 21:36 |
| dkranz | sdague: Any shell var set in devstack-gate script is available to our code, right?> | 21:36 |
| sdague | it should be, but we also have a tempest.conf var | 21:37 |
| sdague | does the whitelist code read our tempest.conf? | 21:37 |
| *** masayukig has quit IRC | 21:37 | |
| dkranz | sdague: Oh yeah. I"ll use that. | 21:37 |
| dkranz | sdague: No, but it could. | 21:38 |
| dkranz | sdague: Maybe env var is actually easier. | 21:38 |
| *** atiwari has quit IRC | 21:39 | |
| dkranz | Once https://review.openstack.org/#/c/58617/ goes in I will make the neutron change. | 21:39 |
| *** dims has quit IRC | 21:40 | |
| openstackgerrit | Walter A. Boring IV (hemna) proposed a change to openstack/tempest: Pass the max_runs option correctly https://review.openstack.org/58628 | 21:40 |
| *** mlavalle has joined #openstack-qa | 21:40 | |
| EmilienM | sdague: i'm confused about my patch on grenade to bring neutron support > https://review.openstack.org/#/c/58401/ | 21:41 |
| EmilienM | sdague: jenkins fails because of localrc is using nova-network, while i made some modifications in my patchsets. Logs: https://jenkins01.openstack.org/job/check-grenade-devstack-vm/6646/consoleText | 21:42 |
| EmilienM | I think i miss something | 21:42 |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/58604 failed tempest with an unrecognized error | 21:42 |
| dkranz | sdague: Perhaps I should make all checks spit out what was there and only use the whitelist to determine pass/fail? | 21:42 |
| dkranz | sdague: Not just neutron. | 21:42 |
| *** ken1ohmichi has quit IRC | 21:44 | |
| dkranz | mtreinish: Can you approve https://review.openstack.org/#/c/58617/ ? (new whitelist) | 21:48 |
| mtreinish | dkranz: done | 21:49 |
| dkranz | mtreinish: Thanks | 21:49 |
| openstackrecheck | openstack/nova change: https://review.openstack.org/25358 failed tempest because of: https://bugs.launchpad.net/bugs/1253896 | 21:53 |
| jog0 | well that went no where | 21:53 |
| *** dims has joined #openstack-qa | 21:54 | |
| sdague | EmilienM: we're still going grizzly -> master | 21:54 |
| sdague | you might be caught in that | 21:55 |
| jog0 | sdague: I think you hit the underlying issue at the end -- a small team can make sure there are no unknown bugs in the gate (or at least try). but a small team *cannot* fix the bugs | 21:57 |
| jog0 | QA shouldn't have to hunt down teams ot fix bugs | 21:57 |
| sdague | right, I think we'll get there | 21:58 |
| *** DennyZhang has quit IRC | 22:00 | |
| dkranz | sdague: Are you in favor of dumping all log errors for all projects, or just neutron? | 22:02 |
| dkranz | sdague: I am | 22:02 |
| sdague | can I see how bad it will look in the console first before I say my opinion? | 22:03 |
| marun | i think it is a great idea | 22:05 |
| marun | ERROR would ideally mean something rather than indicate a non-exceptional state | 22:05 |
| portante | Hello folks, any other folks able to review https://review.openstack.org/#/c/57544/ and https://review.openstack.org/#/c/57526/ ? | 22:06 |
| sdague | portante: +Aed | 22:06 |
| portante | great, thanks! | 22:07 |
| dkranz | sdague: Yes. I will propose the change and you can look at the console after the check run. | 22:08 |
| sdague | ok, it's time for me to head out for some beer | 22:10 |
| *** sdague has left #openstack-qa | 22:12 | |
| *** sdague has joined #openstack-qa | 22:12 | |
| mtreinish | jog0: really, I don't have to fix bugs anymore? | 22:24 |
| jog0 | mtreinish: just teh tempest ones :) | 22:24 |
| *** SergeyLukjanov has quit IRC | 22:25 | |
| clarkb | mtreinish: I heard a rumor you were going to backport that nova oslo local change | 22:25 |
| clarkb | mtreinish: is that something that is on your radar? (want to make sure it actually gets backported to havana) | 22:25 |
| openstackrecheck | openstack/python-novaclient change: https://review.openstack.org/58418 failed tempest with an unrecognized error | 22:26 |
| mtreinish | clarkb: I'm doing what? I've only been back one day I don't think I signed up for anything | 22:26 |
| clarkb | mtreinish: thats what I thought | 22:26 |
| clarkb | maybe someones tab completion failed and they meant mriedem ? | 22:26 |
| mtreinish | that happens a lot | 22:27 |
| mtreinish | way too often actually | 22:27 |
| clarkb | mtreinish: there was a really great bug found in the old version of oslo synced to nova specifically ni local.py | 22:27 |
| mtreinish | especially because we both work for IBM | 22:27 |
| clarkb | and needs backporting to havana | 22:27 |
| mriedem | clarkb: i backported that one | 22:28 |
| mriedem | so maybe they did mean me? | 22:28 |
| clarkb | mriedem: probably | 22:28 |
| mriedem | clarkb: mtreinish: https://review.openstack.org/#/c/57537/ | 22:28 |
| clarkb | thank you | 22:29 |
| *** mriedem has quit IRC | 22:35 | |
| openstackgerrit | David Kranz proposed a change to openstack/tempest: Dump all log errors and special-case neutron https://review.openstack.org/58638 | 22:37 |
| openstackgerrit | David Kranz proposed a change to openstack/tempest: Dump all errors to console and special-case neutron https://review.openstack.org/58640 | 22:42 |
| *** dkranz has quit IRC | 22:48 | |
| openstackgerrit | A change was merged to openstack/tempest: Add hopefully last batch to the whitelist https://review.openstack.org/58617 | 22:57 |
| *** hemna is now known as hemnafk | 23:04 | |
| *** jpich has quit IRC | 23:04 | |
| openstackgerrit | Bob Ball proposed a change to openstack/tempest: Ensure create_test_server is skipped https://review.openstack.org/58645 | 23:08 |
| openstackrecheck | openstack/python-keystoneclient change: https://review.openstack.org/53038 failed tempest because of: https://bugs.launchpad.net/bugs/1254872 | 23:08 |
| *** amotoki has quit IRC | 23:09 | |
| *** che-arne has joined #openstack-qa | 23:10 | |
| openstackrecheck | openstack/tempest change: https://review.openstack.org/58640 failed tempest with an unrecognized error | 23:19 |
| *** reaper has quit IRC | 23:26 | |
| *** mriedem has joined #openstack-qa | 23:32 | |
| openstackgerrit | A change was merged to openstack-dev/devstack: Shorten PKI token logging https://review.openstack.org/57526 | 23:35 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Use the swift logging adapter for txn IDs https://review.openstack.org/57544 | 23:35 |
| *** giulivo has quit IRC | 23:50 | |
| *** weshay has quit IRC | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!