| *** dims has joined #openstack-qa | 00:11 | |
| *** adalbas has quit IRC | 00:17 | |
| *** matsuhashi has joined #openstack-qa | 00:28 | |
| *** rockyg has quit IRC | 00:38 | |
| *** nosnos has joined #openstack-qa | 00:38 | |
| openstackgerrit | Sean Dague proposed a change to openstack/tempest: remove test_list_servers_by_changes_since https://review.openstack.org/48548 | 00:38 |
|---|---|---|
| *** weshay has quit IRC | 00:41 | |
| *** sthaha has joined #openstack-qa | 00:45 | |
| *** sthaha has joined #openstack-qa | 00:45 | |
| *** malini_afk is now known as malini | 00:52 | |
| dtroyer | jamielennox: I'm here but don't have time tonight…tomorrow? what TZ are you in? I'm US Central | 01:14 |
| *** mriedem has quit IRC | 01:23 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Switch neutron smoke job over to parallel https://review.openstack.org/48551 | 01:28 |
| *** jerryz has joined #openstack-qa | 01:29 | |
| jerryz | dtroyer: ping | 01:29 |
| jamielennox | dtroyer: that's ok i might catch you next week then - Australia +10 | 01:36 |
| *** rfolco has quit IRC | 01:39 | |
| jerryz | hi , i got an oslo.config version conflict error while starting nova-api on devstack , can anybody help? | 01:49 |
| *** malini has left #openstack-qa | 01:51 | |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part2 https://review.openstack.org/39609 | 01:58 |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part1 https://review.openstack.org/39621 | 02:07 |
| openstackgerrit | Ivan-Zhu proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part2 https://review.openstack.org/39609 | 02:08 |
| openstackgerrit | A change was merged to openstack-dev/devstack: Enable tenant isolation to tempest for neutron https://review.openstack.org/46314 | 02:16 |
| openstackgerrit | DennyZhang proposed a change to openstack/tempest: _get_tenant_by_name doesn't return correctly https://review.openstack.org/48516 | 02:18 |
| *** anteaya has quit IRC | 02:50 | |
| *** dims has quit IRC | 02:59 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/tempest: Bump oslo.config version https://review.openstack.org/48560 | 02:59 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Further shuffling to get this to fit the standard code layout https://review.openstack.org/48372 | 03:03 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1230407 https://review.openstack.org/48374 | 03:03 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Update future work section in README https://review.openstack.org/48472 | 03:03 |
| *** matsuhashi has quit IRC | 03:15 | |
| *** dkranz has joined #openstack-qa | 03:29 | |
| *** marun has quit IRC | 03:37 | |
| *** marun has joined #openstack-qa | 03:38 | |
| *** marun has quit IRC | 03:50 | |
| *** marun has joined #openstack-qa | 03:51 | |
| openstackgerrit | ChenZheng proposed a change to openstack/tempest: Test for the agent management extension API https://review.openstack.org/48421 | 03:53 |
| *** marun has quit IRC | 03:56 | |
| *** marun has joined #openstack-qa | 03:56 | |
| *** rhsu has joined #openstack-qa | 03:56 | |
| *** matsuhashi has joined #openstack-qa | 03:56 | |
| *** jerryz has quit IRC | 04:14 | |
| *** jhenner has joined #openstack-qa | 04:17 | |
| *** SergeyLukjanov has joined #openstack-qa | 04:31 | |
| *** jhenner has quit IRC | 04:37 | |
| openstackgerrit | DennyZhang proposed a change to openstack/tempest: _error_in_logs function of driver.py shall check all nodes https://review.openstack.org/48147 | 04:42 |
| *** jhenner has joined #openstack-qa | 04:46 | |
| *** DennyZhang has joined #openstack-qa | 04:56 | |
| *** madan has joined #openstack-qa | 05:03 | |
| *** afazekas_zz has quit IRC | 05:20 | |
| *** prekarat has joined #openstack-qa | 05:22 | |
| *** madan has quit IRC | 05:25 | |
| *** SergeyLukjanov has quit IRC | 05:29 | |
| *** madan has joined #openstack-qa | 05:38 | |
| *** afazekas has joined #openstack-qa | 05:41 | |
| *** SergeyLukjanov has joined #openstack-qa | 05:42 | |
| *** SergeyLukjanov has quit IRC | 05:44 | |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: port test_flavors into v3 part1 https://review.openstack.org/48419 | 05:47 |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part1 https://review.openstack.org/39621 | 05:47 |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: port test_flavors into v3 part2 https://review.openstack.org/48440 | 05:47 |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: port test_images and test_server_actions into v3 part2 https://review.openstack.org/39609 | 05:47 |
| openstackgerrit | Yang Yu proposed a change to openstack/tempest: Adding glance API update /v1/images/<image_id> https://review.openstack.org/48578 | 05:53 |
| openstackgerrit | Yang Yu proposed a change to openstack/tempest: Adding glance API update /v1/images/<image_id> https://review.openstack.org/48578 | 05:55 |
| openstackgerrit | Hoisaleshwara Madan V S proposed a change to openstack/tempest: port test_flavors into v3 part2 https://review.openstack.org/48580 | 06:01 |
| *** DennyZhang has quit IRC | 06:03 | |
| openstackgerrit | afazekas proposed a change to openstack-dev/devstack: Use the rdo havana repo with the RHEL family https://review.openstack.org/48581 | 06:04 |
| *** marun has quit IRC | 06:06 | |
| *** afazekas_ has joined #openstack-qa | 06:16 | |
| *** afazekas_ has quit IRC | 06:17 | |
| openstackgerrit | Yaguang Tang proposed a change to openstack/tempest: Fix TempestConfig instance has no attribute 'debug' https://review.openstack.org/48592 | 06:17 |
| openstackgerrit | Zhu Zhu proposed a change to openstack/tempest: Test for the nova diagnostics API https://review.openstack.org/48249 | 06:19 |
| *** Madan1 has joined #openstack-qa | 06:26 | |
| *** rhsu has quit IRC | 06:30 | |
| openstackgerrit | A change was merged to openstack/tempest: Adds more test to cover Swift tempURL middleware https://review.openstack.org/44287 | 06:41 |
| *** jhenner has quit IRC | 06:52 | |
| *** jlibosva has joined #openstack-qa | 07:08 | |
| *** jamielennox is now known as jamielennox|away | 07:11 | |
| *** prekarat has quit IRC | 07:24 | |
| *** fbo_away is now known as fbo | 07:25 | |
| *** jhenner has joined #openstack-qa | 07:32 | |
| *** giulivo has joined #openstack-qa | 07:39 | |
| openstackgerrit | Xiang Hui proposed a change to openstack/tempest: Add security groups updating test for ports https://review.openstack.org/46894 | 07:43 |
| giulivo | sad thing is I noted a few new comers abusing of | 07:46 |
| giulivo | no bug | 07:46 |
| giulivo | :( | 07:46 |
| *** yassine has joined #openstack-qa | 07:47 | |
| kashyap | abusing of 'no bug'? | 07:48 |
| *** SergeyLukjanov has joined #openstack-qa | 07:53 | |
| giulivo | using recheck no bug or reverify no bug | 07:57 |
| giulivo | to rerun gate | 07:58 |
| *** kong has joined #openstack-qa | 08:00 | |
| kong | anyone here? I found a url:https://docs.google.com/spreadsheet/ccc?key=0AmYuZ6T4IJETdEVNTWlYVUVOWURmOERSZ0VGc1BBQWc#gid=0, and I want to help to add tempest test cases. But I don't know how, any hints are appreciated. | 08:02 |
| *** SergeyLukjanov has quit IRC | 08:09 | |
| giulivo | kong, to contribute to tempest you follow the same rules which apply to other core projects | 08:09 |
| giulivo | https://wiki.openstack.org/wiki/How_To_Contribute | 08:09 |
| kong | as bug or blueprint? | 08:09 |
| kong | giulivo, I used to contribute other projects by fixing bug, but I don't know wether adding test cases is a bugfix or I should summit a bp for it? | 08:12 |
| giulivo | so you can add tests without filing blueprints or bugs | 08:12 |
| giulivo | we tend to consider the addition of tests a bug not a bp and use the bp more for the actual framework enhancements | 08:13 |
| *** SergeyLukjanov has joined #openstack-qa | 08:13 | |
| kong | giulivo, thanks, I'll try! | 08:15 |
| giulivo | kong, thank you for helping, I didn't even know about that spreadsheet! :) | 08:16 |
| kong | :) | 08:16 |
| *** thomasbiege1 has joined #openstack-qa | 08:16 | |
| *** thomasbiege1 has quit IRC | 08:19 | |
| giulivo | ping mtreinish | 08:34 |
| *** samalba has quit IRC | 08:52 | |
| *** Madan1 has quit IRC | 08:52 | |
| *** samalba has joined #openstack-qa | 08:53 | |
| raies | giulivo: hi please have a look on https://review.openstack.org/#/c/47339/ | 08:56 |
| *** shilpa_ has joined #openstack-qa | 09:04 | |
| Anju | afazekas : ping | 09:04 |
| afazekas | pong | 09:05 |
| Anju | afazekas: https://github.com/openstack/tempest/blob/master/tempest/api/image/v1/test_image_members.py#L45 | 09:05 |
| Anju | is this important to check the status here | 09:05 |
| afazekas | it never harms to check | 09:06 |
| Anju | but afazekas : https://review.openstack.org/#/c/47167/ | 09:07 |
| Anju | i have implemented the the setup class for image | 09:07 |
| Anju | but | 09:07 |
| afazekas | giulivo: ping | 09:08 |
| giulivo | pong afazekas | 09:08 |
| Anju | giulivo : hii | 09:08 |
| Anju | https://review.openstack.org/#/c/47167/ | 09:08 |
| Anju | giulivo : i have done what you said | 09:09 |
| afazekas | Are you saying the negative tests creating images ? | 09:09 |
| Anju | no they not | 09:09 |
| Anju | but for other tests we can do this | 09:10 |
| giulivo | yeah not the Anju's tests | 09:10 |
| afazekas | giulivo: I do not fully follow your comment here https://review.openstack.org/#/c/47167/ | 09:11 |
| giulivo | afazekas, not Anju's tests | 09:11 |
| giulivo | should be a separate change? | 09:11 |
| Anju | yes afazekas , I understand giulivo does not want this for my tests | 09:11 |
| afazekas | they are just several byte images, does not have a huge performance impact, especially on parallel | 09:12 |
| afazekas | they are not snaphosts | 09:12 |
| giulivo | also it should eventually be done in a separate change | 09:13 |
| giulivo | which this isn't about | 09:13 |
| giulivo | I'm still of the idea we should share the image though, parallel is only for classes not for tests right? | 09:14 |
| afazekas | yes | 09:14 |
| afazekas | 1 kib image upload is not huge time | 09:14 |
| afazekas | and afaik we are going into the less shearing direction | 09:15 |
| giulivo | into the less sharing direction? | 09:15 |
| afazekas | Usually worth to share volumes, volume-snapshots and servers | 09:15 |
| giulivo | I see , so firstly Anju's changes should be merged because they do not relate to the topic | 09:16 |
| afazekas | I am not sure the setupclasses will live forever in the code.. | 09:16 |
| Anju | giulivo , afazekas : I have done that changes also | 09:16 |
| afazekas | giulivo: yes it would be an unrelated change | 09:17 |
| giulivo | why that should be? like, what would be the advantage of dropping setupclass ? | 09:17 |
| giulivo | sorry about that Anju | 09:17 |
| afazekas | the testtools and fixtures not really designed to work together with setupclasses | 09:17 |
| Anju | No giulivo : if you want that | 09:17 |
| Anju | I have a patch for that also | 09:17 |
| giulivo | Anju, I might not have longer term vision | 09:18 |
| giulivo | you can try to push that though, in a separate change, I won't vote that | 09:20 |
| Anju | afazekas are u agree | 09:20 |
| Anju | ? | 09:20 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-dev/devstack: Add Neutron Metering Agent support https://review.openstack.org/48042 | 09:26 |
| *** madan has quit IRC | 09:27 | |
| *** kong has quit IRC | 09:35 | |
| *** madan has joined #openstack-qa | 09:38 | |
| *** madan has quit IRC | 09:42 | |
| *** Madan1 has joined #openstack-qa | 09:43 | |
| giulivo | Anju, so I think I was suggesting you the wrong route with that change | 09:45 |
| giulivo | the images sharing probably isn't worth as much as servers or volumes, as per afazekas suggestion | 09:46 |
| Anju | giulivo: but it your suggestion is looking good | 09:47 |
| giulivo | so the thing is that for whatever reason, glance accepts images with no content | 09:47 |
| giulivo | being this good or not I don't know, but it does | 09:48 |
| Anju | hmm , ok giulivo | 09:48 |
| giulivo | and at this point, creating an image isn't very demanding because what we're creating are zero data images, which don't take up much time/resources | 09:48 |
| Anju | hmm that is | 09:49 |
| giulivo | for that reason, afazekas suggests sharing the image creation isn't really worth it or at least not as much as with, say, servers | 09:49 |
| *** Madan1 has quit IRC | 09:50 | |
| Anju | giulivo: https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_server_rescue.py | 09:51 |
| afazekas | giulivo: I have no objection about sharing it. | 09:51 |
| giulivo | afazekas, I get that, only it isn't that important as with other type of resources | 09:52 |
| afazekas | If it can be done in safe way | 09:52 |
| Anju | afazekas , giulivo : thanks | 09:53 |
| afazekas | giulivo: now we do not have good enough reason to share it, we would gain maximum sever sec even on single thread | 09:53 |
| *** Madan has joined #openstack-qa | 09:54 | |
| afazekas | However I would like to share the servers even outside to the class boundaries.. the testresources way LGTM | 09:55 |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: Base class for Network Admin API tests https://review.openstack.org/48613 | 09:55 |
| giulivo | afazekas, at the same time, I've the feeling we could reorganize the classes and files a bit | 10:01 |
| *** Madan1 has joined #openstack-qa | 10:02 | |
| openstackgerrit | Sean Dague proposed a change to openstack/tempest: actually register the debug_opts https://review.openstack.org/48615 | 10:07 |
| afazekas | sdague: https://review.openstack.org/#/c/48271/1/tempest/config.py | 10:08 |
| sdague | afazekas: ok, cool | 10:09 |
| afazekas | you missed the self.debug :) | 10:09 |
| sdague | yep :) | 10:10 |
| sdague | well, that's why it's a review | 10:10 |
| afazekas | sdague: I am planning to add some iscsiadm and tgtd related debug code to the volume boot test case | 10:10 |
| sdague | ok, great | 10:11 |
| afazekas | Do you think an sql lock monitoring script (service) for devstack would be useful ? | 10:11 |
| sdague | afazekas: you think that's the issue? | 10:11 |
| afazekas | no, it is related to the neutron max connection issue | 10:11 |
| afazekas | with the volume attachment issue, I am not sure is it libvirt , nova, tgtd , or cinder issue yet | 10:12 |
| afazekas | A basic lock monitoring just querying the information_schema from the mysql/mariadb , and if it founds a lock wait above n=2, sec it logs a lot of other information schema queries | 10:14 |
| afazekas | If we are lucky it will print out a the query from a an another transaction which holds the lock | 10:15 |
| afazekas | If we are unlucky, we will see null query, but we might now which process (follow lsof) , was the other | 10:16 |
| giulivo | afazekas, can you point me to a failed job? I'd like to check the trace. do they all freeze on the same step? | 10:16 |
| afazekas | http://status.openstack.org/rechecks/ | 10:16 |
| afazekas | 2th issue is the volume attachment | 10:17 |
| afazekas | check the syslog | 10:17 |
| afazekas | this is one lock issue related: https://bugs.launchpad.net/neutron/+bug/1211915 | 10:17 |
| afazekas | as I remember several nova issue also related to lock waits.. | 10:18 |
| afazekas | sdague: do you think a 'dbmon' devstack service would be useful ? | 10:18 |
| openstackgerrit | Vincent Hou proposed a change to openstack-dev/devstack: Add the creation of /var/run/openstack when zeromq is chosen https://review.openstack.org/48616 | 10:19 |
| openstackgerrit | Zhu Zhu proposed a change to openstack/tempest: Test for the nova diagnostics API https://review.openstack.org/48249 | 10:20 |
| sdague | so the boto one is failing because of this - http://logs.openstack.org/87/47487/4/check/gate-tempest-devstack-vm-neutron/4128a28/logs/screen-n-cpu.txt.gz#_2013-09-27_02_06_06_457 | 10:21 |
| afazekas | sdague: there are more reasons, one is that | 10:23 |
| afazekas | does not nova api do any additional locking which is missing when the ec2 used ? | 10:24 |
| sdague | afazekas: not that I know of | 10:26 |
| sdague | afazekas: what do you think is going on? | 10:26 |
| afazekas | http://logs.openstack.org/87/47487/4/check/gate-tempest-devstack-vm-neutron/4128a28/console.html has an another issue | 10:29 |
| afazekas | https://bugs.launchpad.net/neutron/+bug/1223956 , http://logs.openstack.org/87/47487/4/check/gate-tempest-devstack-vm-neutron/4128a28/logs/screen-q-svc.txt.gz?level=ERROR | 10:31 |
| afazekas | 'Lock wait timeout exceeded;' | 10:31 |
| afazekas | sdague: the n-cpu exception happens after 400 sec timeout when the teardown tries to delete the machine which remained in the pending (BUILD) state | 10:33 |
| sdague | ok, so what's the crux of the issue? | 10:34 |
| afazekas | it is possible the nova's neutron client at one task is waiting for a neutron http response | 10:34 |
| sdague | boto can't launch guests? | 10:34 |
| giulivo | afazekas, it seems to be failing always at the 3rd launch, which is when it expects cinder to create a new volume from the snap | 10:35 |
| afazekas | may we see this issue, because it is after router related test case ? | 10:35 |
| giulivo | I noticed it says the iscsi target isn't available yet | 10:35 |
| giulivo | (in nov) | 10:35 |
| giulivo | I wonder, how does tgtd gets noticed of the changes to the volumes? kill -HUP? | 10:37 |
| *** thomasbiege1 has joined #openstack-qa | 10:40 | |
| openstackgerrit | A change was merged to openstack/tempest: Add positive tests for os-floating-ip-pools https://review.openstack.org/47794 | 10:41 |
| giulivo | afazekas, the volume starts with 4020e0dd looking at the cinder logs, looks like tgtd-admin is executed before returning the volume as available | 10:43 |
| afazekas | giulivo: as I remember there is tgtd command for that, probably siggup (or it has domain socket) | 10:43 |
| giulivo | afazekas, still, nova will log that the iscsi is not available | 10:43 |
| giulivo | http://logs.openstack.org/64/47264/2/gate/gate-tempest-devstack-vm-full/dced339/logs/screen-c-vol.txt.gz | 10:43 |
| giulivo | volume id is 4020e0dd-24a0-453b-985d-e50cb2dd0de1 | 10:45 |
| afazekas | http://logs.openstack.org/64/47264/2/gate/gate-tempest-devstack-vm-full/dced339/logs/syslog.txt.gz '[sdb] Attached SCSI disk' '[sda] Attached SCSI disk' | 10:45 |
| afazekas | can you search for them, as I remember the disk should be ready for use. | 10:46 |
| giulivo | if you search for the same volume id in http://logs.openstack.org/64/47264/2/gate/gate-tempest-devstack-vm-full/dced339/logs/screen-n-cpu.txt.gz#_2013-09-24_04_44_31_806 | 10:47 |
| giulivo | you'll notice iscsiadm is saying the volume isn't showing up | 10:47 |
| giulivo | and to my surprise it tried that 3 times | 10:47 |
| * afazekas still do not understand why not to ask qemu to directly use the iscsi... | 10:47 | |
| giulivo | it tries 3 times and after 5 seconds the volume still isn't available | 10:48 |
| giulivo | it neve pops up as attached scsi disk in syslog, because it never is | 10:50 |
| afazekas | Sep 24 04:44:32 devstack-precise-hpcloud-az3-378359 tgtd: conn_close(101) connection closed, 0x22f8f28 1 | 10:50 |
| afazekas | http://logs.openstack.org/64/47264/2/gate/gate-tempest-devstack-vm-full/dced339/logs/syslog.txt.gz | 10:50 |
| afazekas | Sep 24 04:43:14 devstack-precise-hpcloud-az3-378359 kernel: [ 4935.277354] sd 6:0:0:1: [sdb] Attached SCSI disk | 10:51 |
| sdague | giulivo: so I'm chasing this version of that fail - http://logs.openstack.org/87/47487/4/gate/gate-tempest-devstack-vm-postgres-full/247d81e/logs/ | 10:52 |
| sdague | so timestamps are different | 10:52 |
| sdague | however, as best I can tell cinder things everything is fine | 10:52 |
| *** matsuhashi has quit IRC | 10:52 | |
| afazekas | Sep 24 04:44:17 devstack-precise-hpcloud-az3-378359 kernel: [ 4998.500015] scsi 7:0:0:0: Attached scsi generic sg4 type 12 | 10:52 |
| afazekas | it does not continues | 10:52 |
| sdague | but the iscsi volume isn't actually provisioned | 10:52 |
| giulivo | sdague, well, apparently it returns after tgt-admin --update returns okay yes but on the initiator side the volume isn't found | 10:53 |
| sdague | giulivo: does tgt create logs? | 10:53 |
| sdague | maybe we can capture them in the gate | 10:53 |
| sdague | afazekas: ... oh, it occurs to me on the neutron sql side, we've got the mysql slow logs, which are currently registering every sql statement | 10:55 |
| sdague | so you should have enough there if you think db contention is the issue | 10:55 |
| afazekas | sdague: I saw them, but I do not know how to distinguish the sessions and transaction | 10:57 |
| giulivo | sdague, if in debug state should write to messages | 10:58 |
| giulivo | so into syslog | 10:59 |
| sdague | giulivo: we probably aren't running in debug state right? | 11:03 |
| giulivo | I think we should start that with debug mode enabled, but it's something to change on the target ubuntu not in devstack, not sure where we can do that | 11:03 |
| giulivo | the log says debug:0 | 11:03 |
| sdague | afazekas: ok ... so here is a crazy thought | 11:04 |
| sdague | what are the chances that neutron is actually running myisam | 11:04 |
| giulivo | but we can also enable debug at runtime with './tgtadm --mode system --op update --name debug --value on' apparently | 11:05 |
| afazekas | lock issue with myisam ? | 11:05 |
| afazekas | transaction isolation with myisam ? | 11:05 |
| giulivo | is it worth adding that to devstack or asking for a change to the config on the VM? | 11:05 |
| afazekas | sdague: I think it is has low chance | 11:05 |
| sdague | afazekas: no I think it's 100% chance | 11:06 |
| sdague | giulivo: yeh, getting tgt into debug mode would be a devstack change | 11:07 |
| afazekas | SHOW TABLE STATUS; | 11:08 |
| afazekas | Engine filed | 11:08 |
| *** thomasbiege1 has quit IRC | 11:09 | |
| afazekas | http://www.fpaste.org/42654/28022213/ | 11:10 |
| sdague | ok, never mind, we're setting the default in devstack | 11:10 |
| sdague | I know in unit tests we actually default the other way, to make people set their creates correctly | 11:10 |
| afazekas | git grep -i Innodb | 11:12 |
| afazekas | neutron/db/model_base.py: __table_args__ = {'mysql_engine': 'InnoDB'} | 11:12 |
| * afazekas afaik the myisam not enough smart to have too many kind of locking issues :) | 11:13 | |
| sdague | it locks the tables | 11:15 |
| openstackgerrit | Giulio Fidente proposed a change to openstack-dev/devstack: ensure tgtd is running in debug mode https://review.openstack.org/48626 | 11:18 |
| giulivo | sdague, ^^ hopefully doesn't break anything | 11:18 |
| afazekas | we should separate the swift logs a different log file | 11:21 |
| openstackgerrit | A change was merged to openstack/tempest: Add test for nova API of VM lock/unlock https://review.openstack.org/47846 | 11:22 |
| giulivo | afazekas, agreed that would also be nice | 11:23 |
| *** thomasbiege1 has joined #openstack-qa | 11:24 | |
| giulivo | have any idea how? shall we change the log facility to some localX and configure syslog accordingly? | 11:24 |
| *** shanks has joined #openstack-qa | 11:25 | |
| afazekas | giulivo: is ubuntu and fedora/rhel using the same syslogd implementation ? | 11:25 |
| giulivo | huh I don't think so, not newer fedora and older ubuntu for sure | 11:25 |
| giulivo | but to be honest, I'm not even sure how to configure all the swift services on the correct log facility | 11:26 |
| giulivo | that's a task I can take though | 11:26 |
| afazekas | http://docs.openstack.org/developer/swift/development_saio.html#optional-setting-up-rsyslog-for-individual-logging | 11:26 |
| *** thomasbiege3 has joined #openstack-qa | 11:27 | |
| giulivo | so they do use different localX already only we don't catch those in the syslog | 11:27 |
| *** thomasbiege1 has quit IRC | 11:27 | |
| afazekas | yes | 11:29 |
| *** thomasbiege3 has quit IRC | 11:33 | |
| *** thomasbiege has joined #openstack-qa | 11:37 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-dev/devstack: xenapi: set dhcp timeout on VM installation https://review.openstack.org/48630 | 11:38 |
| *** matsuhashi has joined #openstack-qa | 11:44 | |
| *** thomasbiege has quit IRC | 11:44 | |
| *** mkollaro has joined #openstack-qa | 11:48 | |
| *** dims has joined #openstack-qa | 11:51 | |
| *** SergeyLukjanov has quit IRC | 11:55 | |
| openstackgerrit | Mh Raies proposed a change to openstack/tempest: Adding testcase for dhcp agent scheduler https://review.openstack.org/48632 | 11:58 |
| *** matsuhashi has quit IRC | 12:01 | |
| *** afazekas is now known as afazekas_food | 12:01 | |
| *** Madan has quit IRC | 12:03 | |
| *** SergeyLukjanov has joined #openstack-qa | 12:05 | |
| *** adalbas has joined #openstack-qa | 12:05 | |
| *** matsuhashi has joined #openstack-qa | 12:10 | |
| *** matsuhashi has quit IRC | 12:11 | |
| *** shilpa_ has quit IRC | 12:20 | |
| *** thomasbiege has joined #openstack-qa | 12:20 | |
| *** thomasbiege has quit IRC | 12:22 | |
| *** dims has quit IRC | 12:22 | |
| *** dims has joined #openstack-qa | 12:23 | |
| *** weshay has joined #openstack-qa | 12:28 | |
| *** matsuhashi has joined #openstack-qa | 12:35 | |
| *** afazekas_food has quit IRC | 12:45 | |
| *** jasondotstar has joined #openstack-qa | 12:50 | |
| *** rfolco has joined #openstack-qa | 12:55 | |
| openstackgerrit | DennyZhang proposed a change to openstack/tempest: _error_in_logs function of driver.py shall check all nodes https://review.openstack.org/48147 | 12:59 |
| *** HenryG has joined #openstack-qa | 13:11 | |
| *** nosnos has quit IRC | 13:15 | |
| *** mriedem has joined #openstack-qa | 13:18 | |
| *** salv-orlando has joined #openstack-qa | 13:18 | |
| *** HenryG has quit IRC | 13:19 | |
| *** HenryG has joined #openstack-qa | 13:19 | |
| openstackgerrit | A change was merged to openstack/tempest: add __str__ function to RestClient class https://review.openstack.org/47350 | 13:24 |
| *** afazekas has joined #openstack-qa | 13:26 | |
| *** yassine has quit IRC | 13:31 | |
| *** yassine has joined #openstack-qa | 13:31 | |
| openstackgerrit | Zhu Zhu proposed a change to openstack/tempest: Test for the nova diagnostics API https://review.openstack.org/48249 | 13:39 |
| *** yassine has quit IRC | 13:53 | |
| *** yassine has joined #openstack-qa | 13:53 | |
| *** jasondotstar has quit IRC | 13:54 | |
| *** anteaya has joined #openstack-qa | 14:04 | |
| *** sthaha has quit IRC | 14:26 | |
| *** adalbas has quit IRC | 14:28 | |
| *** jlibosva has quit IRC | 14:30 | |
| *** adalbas has joined #openstack-qa | 14:44 | |
| openstackgerrit | Chmouel Boudjnah proposed a change to openstack-dev/devstack: Just reset the repo instead of trying to co files https://review.openstack.org/48411 | 14:45 |
| *** matsuhashi has quit IRC | 14:51 | |
| *** marun has joined #openstack-qa | 14:55 | |
| openstackgerrit | Cyril Roelandt proposed a change to openstack-dev/devstack: Revert "Revert "Swift: configure Ceilometer when it is enabled"" https://review.openstack.org/48678 | 15:22 |
| openstackgerrit | Zhi Kun Liu proposed a change to openstack/tempest: add volume list tests https://review.openstack.org/47508 | 15:32 |
| *** rhsu has joined #openstack-qa | 15:33 | |
| sdague | jgriffith I'm headed out for a ride and lunch, giulivo was looking into this issue earlier, if he's still on maybe he's made more progress | 15:35 |
| openstackgerrit | DennyZhang proposed a change to openstack/tempest: _error_in_logs function of driver.py shall check all nodes https://review.openstack.org/48147 | 15:35 |
| jgriffith | sdague: giulivo I'd propose we add a debug: iscsi get targets in the connect | 15:39 |
| jgriffith | where you're looking here is after libvirt tries to do the attach | 15:40 |
| jgriffith | it's NOT that the target isn't found necessarily | 15:40 |
| jgriffith | but actually that the iscsi attach to /dev/disk/by-path is not there | 15:40 |
| jgriffith | sdague: giulivo I'll looook at submitting a patch | 15:40 |
| *** DennyZhang has joined #openstack-qa | 15:43 | |
| *** rhsu has quit IRC | 15:45 | |
| *** yassine has quit IRC | 15:47 | |
| *** DennyZhang has quit IRC | 15:48 | |
| *** DennyZhang has joined #openstack-qa | 15:49 | |
| openstackgerrit | Zhi Kun Liu proposed a change to openstack/tempest: add admin server rebuild tests https://review.openstack.org/46065 | 15:51 |
| *** mlavalle has joined #openstack-qa | 15:52 | |
| openstackgerrit | Zhi Kun Liu proposed a change to openstack/tempest: add admin server rebuild tests https://review.openstack.org/46065 | 15:53 |
| openstackgerrit | Zhi Kun Liu proposed a change to openstack/tempest: add admin server rebuild tests https://review.openstack.org/46065 | 15:57 |
| *** thomasbiege has joined #openstack-qa | 15:59 | |
| *** thomasbiege has quit IRC | 16:01 | |
| *** rhsu has joined #openstack-qa | 16:17 | |
| *** mlavalle has quit IRC | 16:32 | |
| *** mlavalle has joined #openstack-qa | 16:33 | |
| *** jasondotstar has joined #openstack-qa | 16:44 | |
| *** afazekas is now known as afazekas_zz | 16:47 | |
| *** giulivo has quit IRC | 16:48 | |
| *** jerryz has joined #openstack-qa | 16:53 | |
| *** SergeyLukjanov has quit IRC | 17:03 | |
| openstackgerrit | Dean Troyer proposed a change to openstack-dev/devstack: Add meta-config via local.conf https://review.openstack.org/46768 | 17:05 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove screen-key.txt from required files https://review.openstack.org/48702 | 17:09 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add note to remove while True used for waiting https://review.openstack.org/48703 | 17:11 |
| *** DennyZhang has quit IRC | 17:11 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove screen-key.txt from required files https://review.openstack.org/48702 | 17:17 |
| *** mlavalle has quit IRC | 17:20 | |
| *** RecheckWatchBot_ has joined #openstack-qa | 17:31 | |
| *** RecheckWatchBot_ has quit IRC | 17:31 | |
| *** RecheckWatchBotI has joined #openstack-qa | 17:33 | |
| *** hemnafk is now known as hemna_ | 17:36 | |
| *** RecheckWatchBotI has quit IRC | 17:37 | |
| *** RecheckWatchBotI has joined #openstack-qa | 17:38 | |
| *** RecheckWatchBotI has quit IRC | 17:38 | |
| *** RecheckWatchBotI has joined #openstack-qa | 17:39 | |
| RecheckWatchBotI | openstack/cinder change: https://review.openstack.org/48460 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 17:43 |
| jog0 | mtreinish: got the bot back up | 17:43 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/46735 failed tempest with an unrecognized error | 17:45 |
| jog0 | http://logs.openstack.org/35/46735/3/check/gate-tempest-devstack-vm-neutron/97848aa/testr_results.html.gz | 17:47 |
| jog0 | looks like a tempest bug in there too | 17:47 |
| *** mkollaro has quit IRC | 17:50 | |
| *** SergeyLukjanov has joined #openstack-qa | 17:50 | |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48647 failed tempest with an unrecognized error | 17:51 |
| RecheckWatchBotI | openstack/horizon change: https://review.openstack.org/45901 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 17:53 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48325 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 17:54 |
| RecheckWatchBotI | openstack/python-ceilometerclient change: https://review.openstack.org/47331 failed tempest with an unrecognized error | 17:54 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/22278 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 17:57 |
| RecheckWatchBotI | openstack-dev/devstack change: https://review.openstack.org/48351 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 17:57 |
| *** pabelanger has joined #openstack-qa | 18:06 | |
| RecheckWatchBotI | openstack/ceilometer change: https://review.openstack.org/48663 failed tempest because of: https://bugs.launchpad.net/bugs/1211915 | 18:08 |
| RecheckWatchBotI | openstack-dev/devstack change: https://review.openstack.org/48411 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 18:09 |
| RecheckWatchBotI | openstack/glance change: https://review.openstack.org/45774 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 18:10 |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/47508 failed tempest because of: https://bugs.launchpad.net/bugs/1227915 | 18:12 |
| RecheckWatchBotI | openstack/ceilometer change: https://review.openstack.org/48667 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 18:12 |
| *** pabelanger has left #openstack-qa | 18:13 | |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/42667 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 18:13 |
| *** mlavalle has joined #openstack-qa | 18:15 | |
| * jog0 is sorry for spamming this room with the recheckwatchbot | 18:15 | |
| sdague | jog0: the tempest issue is in the gate now | 18:18 |
| sdague | jog0: https://review.openstack.org/#/c/48271/ should fix the tempest piece of that bug (which is secondary) | 18:18 |
| jog0 | sdague: cool | 18:19 |
| jog0 | figured it was secondary | 18:19 |
| sdague | it's funny, we knew it was enough of an issue to put a tear down routing to try to dump a bunch of debug info | 18:20 |
| sdague | but apparently that merged wrong | 18:21 |
| sdague | so the debug flag wasn't in the config correctly :) | 18:21 |
| *** Mann has joined #openstack-qa | 18:21 | |
| *** Mann has left #openstack-qa | 18:22 | |
| *** jasondotstar has quit IRC | 18:23 | |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48666 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 18:26 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48622 failed tempest because of: https://bugs.launchpad.net/bugs/1211915 | 18:27 |
| RecheckWatchBotI | openstack-dev/devstack change: https://review.openstack.org/48678 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 18:27 |
| RecheckWatchBotI | openstack/glance change: https://review.openstack.org/42817 failed tempest because of: https://bugs.launchpad.net/bugs/1218391 | 18:27 |
| RecheckWatchBotI | openstack/keystone change: https://review.openstack.org/48569 failed tempest with an unrecognized error | 18:28 |
| RecheckWatchBotI | openstack/keystone change: https://review.openstack.org/48558 failed tempest with an unrecognized error | 18:28 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48326 failed tempest because of: https://bugs.launchpad.net/bugs/1223161 | 18:29 |
| RecheckWatchBotI | openstack/glance change: https://review.openstack.org/46986 failed tempest with an unrecognized error | 18:29 |
| *** rockyg has joined #openstack-qa | 18:30 | |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/47162 failed tempest with an unrecognized error | 18:30 |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/41345 failed tempest with an unrecognized error | 18:30 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/46824 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 18:35 |
| RecheckWatchBotI | openstack/cinder change: https://review.openstack.org/48350 failed tempest with an unrecognized error | 18:36 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/46231 failed tempest with an unrecognized error | 18:36 |
| RecheckWatchBotI | openstack/glance change: https://review.openstack.org/48401 failed tempest with an unrecognized error | 18:41 |
| RecheckWatchBotI | openstack/python-keystoneclient change: https://review.openstack.org/40703 failed tempest with an unrecognized error | 18:42 |
| jog0 | sdague: for the attribute debug error | 18:43 |
| jog0 | whatever the root bug is its happening a lot | 18:43 |
| jog0 | hmm http://logs.openstack.org/03/40703/7/check/gate-tempest-devstack-vm-neutron/d5fe57c/logs/screen-n-cpu.txt.gz?level=TRACE | 18:44 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/44670 failed tempest with an unrecognized error | 18:50 |
| RecheckWatchBotI | openstack/horizon change: https://review.openstack.org/48513 failed tempest because of: https://bugs.launchpad.net/bugs/1218391 | 18:53 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48544 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 19:00 |
| sdague | jog0: interesting | 19:00 |
| jog0 | sdague: see my latest comment in 1230407 | 19:00 |
| sdague | not yet | 19:00 |
| sdague | man, I didn't realize how useful that level param would become :) | 19:02 |
| jog0 | yeah its pretty boss | 19:02 |
| sdague | plus it's discoverable now, which I think helps | 19:02 |
| jog0 | yeah | 19:02 |
| jog0 | helps a lot | 19:02 |
| openstackgerrit | DennyZhang proposed a change to openstack/tempest: _error_in_logs function of driver.py shall check all nodes https://review.openstack.org/48147 | 19:03 |
| jog0 | sdague: anyway heading out to find some food. Now that elastic-recheck is up and running (and fast thanks to you) we should have less recheck-no-bugs | 19:04 |
| sdague | jog0: awesome | 19:04 |
| jog0 | too bad thats the easy part, now we need to fix em | 19:04 |
| RecheckWatchBotI | openstack/cinder change: https://review.openstack.org/48718 failed tempest with an unrecognized error | 19:22 |
| dkranz | sdague: I have been assuming that getting a NotFound exception from deletes that happen during tearDown is "expected" and not a sign of a bug. | 19:29 |
| dkranz | sdague: Do you agree with that? | 19:29 |
| dkranz | sdague: My interpretation of tearDown is "make sure nothing in the list is still allocated/running" | 19:30 |
| dkranz | mtreinish: ^^^ | 19:41 |
| openstackgerrit | A change was merged to openstack/tempest: _create_creds function shouldn't call rand_name twice https://review.openstack.org/48518 | 19:42 |
| jgriffith | sdague: I'd like to propose either: 1. Upping the wait factor in the virt connect iscsi from 2 --> 3 | 19:46 |
| jgriffith | sdague: Or increasing the conf file settings in the gate test (iscsi retry count) | 19:46 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/47943 failed tempest with an unrecognized error | 19:46 |
| jgriffith | sdague: I *think* the retry count is likely preferrable until we have more info | 19:46 |
| jgriffith | sdague: but the fact here is the portal shows up indicating that the target is at least visible, the iscsiadm connect however doesn't complete in the 9 seconds alloted | 19:47 |
| jgriffith | hmm... actually | 19:48 |
| jgriffith | it's one cycle more so it's like 27 seconds | 19:49 |
| *** rfolco has quit IRC | 19:56 | |
| *** SergeyLukjanov has quit IRC | 19:58 | |
| *** jasondotstar has joined #openstack-qa | 20:11 | |
| *** rockyg has quit IRC | 20:18 | |
| *** rockyg has joined #openstack-qa | 20:18 | |
| *** mkollaro has joined #openstack-qa | 20:19 | |
| *** mkollaro has quit IRC | 20:19 | |
| openstackgerrit | A change was merged to openstack/tempest: Added Negative tests for image member https://review.openstack.org/47167 | 20:20 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48488 failed tempest with an unrecognized error | 20:20 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48720 failed tempest with an unrecognized error | 20:27 |
| salv-orlando | hi - while investigating bug 1230407 I found that in some cases VMs are stuck because qemu-nbd hangs while injecting data into the instance, which might be a different bug which manifests in the same way as 1230407. | 20:37 |
| salv-orlando | Is that something potential credible or it must be just something messed up in my env, in your opinion? | 20:37 |
| RecheckWatchBotI | openstack-dev/devstack change: https://review.openstack.org/48411 failed tempest with an unrecognized error | 20:38 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48721 failed tempest with an unrecognized error | 20:42 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/42736 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 20:42 |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/48147 failed tempest with an unrecognized error | 20:43 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48666 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 20:43 |
| RecheckWatchBotI | openstack/requirements change: https://review.openstack.org/48722 failed tempest because of: https://bugs.launchpad.net/bugs/1218391 | 20:46 |
| RecheckWatchBotI | openstack/heat change: https://review.openstack.org/48727 failed tempest with an unrecognized error | 20:47 |
| jog0 | jgriffith: can we mark 1226337 as critical for cr1? | 20:48 |
| *** jasondotstar has quit IRC | 20:48 | |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/48271 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 20:49 |
| jgriffith | jog0: cr1? | 20:56 |
| jgriffith | jog0: The problem is I still don't believe anybody really has any good idea of what's going on here | 20:56 |
| jog0 | anyway rc1* | 20:57 |
| jgriffith | jog0: and I'm also trying to figure out if we EVER had a version of this test that actually didn't cause havoc | 20:57 |
| jog0 | jgriffith: yeah but we all should agree its bad right? | 20:57 |
| jgriffith | jog0: I think so yes | 20:57 |
| jog0 | jgriffith: :( thats scary | 20:57 |
| jgriffith | jog0: So the only proposals I have at this point: | 20:58 |
| jgriffith | 1. TGT it completely screwed on ubuntu | 20:58 |
| jgriffith | Which means we're screwed, based on pas experiences with bugs in ubuntu | 20:58 |
| jgriffith | 2. We're too busy on the system and just not waiting long enough | 20:58 |
| jgriffith | Either answer is bad | 20:58 |
| jgriffith | I'm bothered by the fact that the tgt login is succesful | 20:58 |
| jgriffith | The "cinder failed and didn't notice" is kinda crap IMO | 20:59 |
| jgriffith | The target's there, reported succesful creation | 20:59 |
| jgriffith | and Nova sees the target and logs in to it | 20:59 |
| jgriffith | but... the connect never completes | 20:59 |
| jgriffith | That could be a problem with target still of course | 20:59 |
| jgriffith | One option if we get the neutron bug fixed is to up the iscsi retry count in the default config and do some data gathering | 21:00 |
| jgriffith | It's crap, but I'm not sure what else to try (except maybe the tgt debug options | 21:00 |
| jgriffith | jog0: thoughts? | 21:02 |
| mlavalle | dkranz: I am trying to get a fresh run ofhttps://jenkins.openstack.org/job/gate-tempest-devstack-vm-neutron-full/ where do I go to accomplish that? | 21:02 |
| jgriffith | jog0: I kinda got carried a way there ;) | 21:02 |
| dkranz | mlavalle: I think you should be able to do "check experimental" on any tempest patch. | 21:02 |
| mlavalle | dkranz: in gerrit? | 21:03 |
| dkranz | mlavalle: Yes | 21:03 |
| mlavalle | let me see | 21:03 |
| dkranz | mlavalle: It is a review comment | 21:03 |
| dims | jgriffith, when the connect does complete, does it log anything in say syslog? | 21:03 |
| dkranz | mlavalle: I will paste the link to the file where you can see how this works | 21:04 |
| jgriffith | dims: that's the issue, it never completes | 21:04 |
| jgriffith | dims: we timeout and fail before the attach is made | 21:04 |
| mlavalle | dkranz: thanks | 21:04 |
| jgriffith | dims: and no, I'm not seeing anything in the logs. | 21:04 |
| jgriffith | dims: but I haven't looked closely in syslog either | 21:04 |
| jog0 | jgriffith: I think your on the right track | 21:05 |
| dims | jgriffith, gotcha | 21:05 |
| dkranz | mlavalle: Sorry, my dog has to go out. I'll paste it in a little bit :) | 21:06 |
| mlavalle | dkranz: np | 21:06 |
| mlavalle | dkranz: first things first ;-) | 21:06 |
| jgriffith | dims: interesting iscsid: connection6:0 is operational now | 21:06 |
| dims | how long after we gave up did it get operational? | 21:08 |
| jgriffith | dims: actually that seems to coorelate to the iscsiadm command that said wasn't succesful | 21:12 |
| RecheckWatchBotI | openstack/python-keystoneclient change: https://review.openstack.org/40703 failed tempest with an unrecognized error | 21:12 |
| jgriffith | http://logs.openstack.org/64/47264/2/gate/gate-tempest-devstack-vm-full/dced339/logs/screen-n-cpu.txt.gz#_2013-09-24_04_44_17_515 | 21:12 |
| dims | jgriffith, sigh! | 21:12 |
| RecheckWatchBotI | openstack/heat change: https://review.openstack.org/44534 failed tempest with an unrecognized error | 21:14 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48544 failed tempest with an unrecognized error | 21:15 |
| jgriffith | dims: I can push a change that bumps the retries up to at least disprove that | 21:15 |
| jgriffith | dims: other than that I don't have much else to go on with this | 21:15 |
| RecheckWatchBotI | openstack/requirements change: https://review.openstack.org/48739 failed tempest with an unrecognized error | 21:16 |
| jgriffith | dims: I'd like to add a log output there that shows a few other things | 21:16 |
| jgriffith | dims: but the other problem is given the neutron bug I don't know that it would EVER get in anyway | 21:16 |
| dims | jgriffith, see this? "Sep 24 04:34:44 devstack-precise-hpcloud-az3-378359 kernel: [ 4425.701784] connection1:0: detected conn error (1020)" | 21:19 |
| dims | in http://logs.openstack.org/64/47264/2/gate/gate-tempest-devstack-vm-full/dced339/logs/kern_log.txt.gz | 21:19 |
| dkranz | mlavalle: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml | 21:21 |
| RecheckWatchBotI | openstack/python-keystoneclient change: https://review.openstack.org/48673 failed tempest with an unrecognized error | 21:21 |
| dims | jgriffith, guessing that is because of "iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'manual')" in screen-n-cpu - same time stamp | 21:21 |
| mlavalle | dkranz: thanks | 21:21 |
| dkranz | mlavalle: You can see that the full neutron run is in the experimental queue for : tempest, neutron, nova | 21:21 |
| dkranz | mlavalle: So you can check experimental on any patch in any of those projects | 21:22 |
| jgriffith | dims: interesting | 21:22 |
| dims | which means if we had let it go on and waited, we may have succeeded which was your option #2 ("just not waiting long enough") | 21:23 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48325 failed tempest with an unrecognized error | 21:23 |
| jgriffith | dims: but that's the serverrescue test | 21:24 |
| jgriffith | dims: which is attempting a delete | 21:24 |
| dims | jgriffith, k that makes sense then | 21:25 |
| jgriffith | I'd have been suspicious if the next connect failed (ie the iscsi sub was jacked) but it doesn't appear to be | 21:25 |
| dims | i thought salv-orlando pointed to a review where he enabled debugging in tgt right? | 21:26 |
| jgriffith | dims: yeah | 21:27 |
| jgriffith | looking | 21:27 |
| RecheckWatchBotI | openstack/heat change: https://review.openstack.org/48240 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 21:27 |
| mlavalle | dkranz: thanks. It is clear now. I was looking for a button…. It was driving me crazy… have a nice weekend | 21:28 |
| dkranz | mlavalle: Same to you. I guess it is time to start it :) | 21:28 |
| *** anteaya has quit IRC | 21:28 | |
| dkranz | mlavalle: Can you paste the link to the etherpad tracking the full neutron issue? | 21:29 |
| mlavalle | dkranz: sure | 21:29 |
| *** mriedem has quit IRC | 21:30 | |
| RecheckWatchBotI | openstack/keystone change: https://review.openstack.org/48746 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 21:30 |
| RecheckWatchBotI | openstack/horizon change: https://review.openstack.org/48513 failed tempest with an unrecognized error | 21:32 |
| RecheckWatchBotI | openstack/cinder change: https://review.openstack.org/48742 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 21:33 |
| mlavalle | dkranz; done | 21:33 |
| dkranz | mlavalle: I don't see it... | 21:33 |
| RecheckWatchBotI | openstack/glance change: https://review.openstack.org/48076 failed tempest with an unrecognized error | 21:34 |
| dkranz | jog0: I have not seen this failure, have you? http://logs.openstack.org/45/41345/8/check/gate-tempest-devstack-vm-neutron/e91142b/console.html | 21:36 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/28313 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 21:38 |
| jog0 | dkranz: no sounds like a infra bug though | 21:39 |
| dkranz | jog0: OK, I'll ask them | 21:40 |
| jgriffith | dims: https://review.openstack.org/#/c/48752/ | 21:40 |
| jgriffith | jog0: ^^ | 21:40 |
| mlavalle | dkranz: Hopefully you will see it here https://etherpad.openstack.org/gate-tempest-devstack-vm-quantum-full | 21:42 |
| jog0 | jgriffith: ther are a few typos in the commit msg and you can add the related-bug tag or something | 21:43 |
| jog0 | https://wiki.openstack.org/wiki/GitCommitMessages#Including_external_references | 21:44 |
| dkranz | mlavalle: GOt it | 21:44 |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/48147 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 21:44 |
| jog0 | jgriffith:do you want to merge or just run checks on it a bunch? | 21:46 |
| jgriffith | jog0: well... depends on if we can trust it via running a bunch of checks on it | 21:46 |
| jgriffith | jog0: I'm fine with just doing that as a start | 21:47 |
| RecheckWatchBotI | openstack/requirements change: https://review.openstack.org/48745 failed tempest with an unrecognized error | 21:47 |
| jgriffith | jog0: see if we get more info, or hit the failure | 21:47 |
| jog0 | using rechecks will be faster I think | 21:50 |
| jgriffith | jog0: cool by me | 21:50 |
| jog0 | can you -1 your patch and if you want to merge I will be happy to +2 it | 21:50 |
| jgriffith | works for me | 21:51 |
| jgriffith | jog0: -1 w/message "test, do not merge" | 21:51 |
| jgriffith | jog0: I'll run rechecks on it and see where we get | 21:51 |
| jog0 | cool | 21:52 |
| jgriffith | jog0: if it actually helps after the week-end, I'l fix up the commit mesg | 21:52 |
| jog0 | sounds good | 21:52 |
| jog0 | I am hoping it will help | 21:53 |
| *** sfineberg has quit IRC | 21:53 | |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48666 failed tempest with an unrecognized error | 21:54 |
| *** sfineberg has joined #openstack-qa | 21:56 | |
| RecheckWatchBotI | openstack/heat change: https://review.openstack.org/47860 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 21:58 |
| RecheckWatchBotI | openstack/python-novaclient change: https://review.openstack.org/48406 failed tempest with an unrecognized error | 21:59 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make unit tests pass again with new code layout https://review.openstack.org/48758 | 22:24 |
| sdague | dkranz: sorry, was in meetings. NotFound in tear down is probably fine. | 22:26 |
| RecheckWatchBotI | openstack/heat change: https://review.openstack.org/48155 failed tempest with an unrecognized error | 22:30 |
| *** mlavalle has quit IRC | 22:42 | |
| RecheckWatchBotI | openstack-dev/devstack change: https://review.openstack.org/48351 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 22:49 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48697 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 22:49 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48686 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 22:49 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48720 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 22:50 |
| RecheckWatchBotI | openstack/keystone change: https://review.openstack.org/48746 failed tempest with an unrecognized error | 22:50 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48752 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 22:51 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Enforce E501 max-line-length at 100 chars https://review.openstack.org/48763 | 22:51 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Cleanup tests https://review.openstack.org/48764 | 22:51 |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/48168 failed tempest because of: https://bugs.launchpad.net/bugs/1226337 | 23:04 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48591 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 23:09 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Stop using While True to wait for ElasticSearch https://review.openstack.org/48766 | 23:12 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Revert "Enable q-vpn service" https://review.openstack.org/48767 | 23:16 |
| jeblair | http://logs.openstack.org/46/47546/2/gate/gate-grenade-devstack-vm/b3d8610/console.html | 23:40 |
| jeblair | sdague, dtroyer: ^ i've seen a couple of jobs fail the horizon grenade test | 23:40 |
| sdague | yeh so it's a quantum symbol colision | 23:40 |
| sdague | http://logs.openstack.org/46/47546/2/gate/gate-grenade-devstack-vm/b3d8610/logs/old/screen-horizon.txt.gz | 23:41 |
| sdague | which... is weird | 23:41 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/45676 failed tempest with an unrecognized error | 23:42 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/45678 failed tempest because of: https://bugs.launchpad.net/bugs/1229475 | 23:42 |
| jeblair | sdague: maybe something is missing from the quantumclient package released? | 23:42 |
| jeblair | today | 23:42 |
| RecheckWatchBotI | openstack/neutron change: https://review.openstack.org/48765 failed tempest with an unrecognized error | 23:43 |
| jeblair | mordred: ^ | 23:43 |
| sdague | oh... that could be it | 23:43 |
| mordred | yes. sorry. crappit | 23:43 |
| sdague | sorry in all the confusion I didn't realize that happened | 23:43 |
| mordred | I need to do another release | 23:43 |
| mordred | I got confused and forgot to | 23:43 |
| sdague | heh | 23:43 |
| jeblair | i'm guessing the mirror recently picked that up | 23:44 |
| jeblair | i think that's going to block everything now | 23:44 |
| sdague | jeblair: yeh, looks like | 23:45 |
| sdague | I know it hit jgriffith's change earlier | 23:45 |
| sdague | and I don't see a single pass on the board right now on that job | 23:45 |
| jeblair | so how about we remove it from the mirror now | 23:45 |
| jeblair | mordred: ^ | 23:45 |
| mordred | I just uploaded the fix to pypi | 23:46 |
| jeblair | oh ok, then maybe let's run the mirror scripts? | 23:46 |
| mordred | how about I put that on the mirror too? | 23:46 |
| jeblair | that's faster. :) | 23:46 |
| mordred | done | 23:47 |
| mordred | wow. it's truly amazing that we ninja approved one thing without tests and it did, in fact, break something | 23:47 |
| jeblair | ok. in 20 minutes or less, change #2 in gate will hit it, but then we should be good afterwords | 23:47 |
| jeblair | mordred: heh :) | 23:48 |
| mordred | putting jenkins01 into shutdown mode | 23:48 |
| jeblair | we should switch back to -infra for that | 23:48 |
| mordred | oh. yeah | 23:48 |
| RecheckWatchBotI | openstack/nova change: https://review.openstack.org/48647 failed tempest with an unrecognized error | 23:50 |
| RecheckWatchBotI | openstack/tempest change: https://review.openstack.org/48516 failed tempest because of: https://bugs.launchpad.net/bugs/1230407 | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!