| *** kumarmn has quit IRC | 00:08 | |
| *** kumarmn has joined #openstack-trove | 00:09 | |
| *** itlinux has joined #openstack-trove | 00:10 | |
| *** kumarmn has quit IRC | 00:13 | |
| *** kumarmn has joined #openstack-trove | 00:15 | |
| *** kumarmn has quit IRC | 00:18 | |
| *** kumarmn has joined #openstack-trove | 00:18 | |
| *** rcernin has joined #openstack-trove | 00:21 | |
| *** kumarmn has quit IRC | 00:23 | |
| *** kumarmn has joined #openstack-trove | 00:24 | |
| *** chhavi has joined #openstack-trove | 00:34 | |
| *** chhavi has quit IRC | 00:38 | |
| *** kumarmn has quit IRC | 00:39 | |
| *** kumarmn has joined #openstack-trove | 00:39 | |
| *** kumarmn has quit IRC | 00:43 | |
| *** gouthamr has joined #openstack-trove | 00:49 | |
| *** smatzek has joined #openstack-trove | 01:02 | |
| *** Kevin_Zheng has joined #openstack-trove | 01:21 | |
| *** fanzhang has quit IRC | 01:24 | |
| *** fanzhang_ has joined #openstack-trove | 01:24 | |
| smatzek | hi fanzhang_. I have noted some of your reviews but haven't had a chance to give them a good review yet. I'm still very focused on getting the master, stable and troveclient gates working so we can deliver code in general. | 01:25 |
|---|---|---|
| smatzek | would you mind watching https://review.openstack.org/#/c/514710 during your day and doing rechecks if/when it fails? I just did a recheck so it will be another 3 hours or so until it succeeds or fails again. | 01:26 |
| *** kumarmn has joined #openstack-trove | 01:26 | |
| smatzek | I've found one case where some nodes that the jobs get scheduled on have older versions of qemu and libvirt and when that happens the nested DB VM fails to boot, nova sits forever in the build state and the tests fail. I have a fix in mind which I'll try to write and test out tomorrow. | 01:27 |
| fanzhang_ | morning smatzek | 01:29 |
| fanzhang_ | I'll do some rechecks if necessary, just go to sleep and relax :) | 01:31 |
| *** kumarmn has quit IRC | 01:36 | |
| *** kumarmn has joined #openstack-trove | 01:37 | |
| *** kumarmn has quit IRC | 01:41 | |
| *** rcernin has quit IRC | 01:43 | |
| *** rcernin has joined #openstack-trove | 01:44 | |
| *** fanzhang_ is now known as fanzhang | 01:44 | |
| *** smatzek has quit IRC | 01:50 | |
| openstackgerrit | jian.song proposed openstack/trove master: Fix requirepass problem with redis https://review.openstack.org/514955 | 02:13 |
| *** zhaochao has quit IRC | 02:15 | |
| *** zhaochao has joined #openstack-trove | 02:16 | |
| *** gcb has joined #openstack-trove | 02:16 | |
| *** wong has joined #openstack-trove | 02:17 | |
| wong | morning | 02:17 |
| fanzhang | morning:) | 02:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-troveclient master: Updated from global requirements https://review.openstack.org/500033 | 03:18 |
| *** chhavi has joined #openstack-trove | 04:11 | |
| *** kumarmn has joined #openstack-trove | 04:40 | |
| *** kumarmn has quit IRC | 04:51 | |
| *** tianhui has quit IRC | 06:10 | |
| openstackgerrit | David Rabel proposed openstack/trove stable/pike: Remove Mitaka reference in install/dashboard.rst https://review.openstack.org/515609 | 06:13 |
| *** gouthamr has quit IRC | 06:21 | |
| *** spectr has joined #openstack-trove | 06:22 | |
| *** magicboiz has joined #openstack-trove | 06:27 | |
| *** magicboiz has quit IRC | 06:32 | |
| *** magicboiz has joined #openstack-trove | 06:32 | |
| *** kumarmn has joined #openstack-trove | 06:51 | |
| *** magicboiz has quit IRC | 06:53 | |
| *** magicboiz has joined #openstack-trove | 06:55 | |
| *** kumarmn has quit IRC | 06:55 | |
| *** magicboiz has quit IRC | 07:00 | |
| *** magicboiz has joined #openstack-trove | 07:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/trove-dashboard stable/pike: Imported Translations from Zanata https://review.openstack.org/493761 | 07:07 |
| *** tesseract has joined #openstack-trove | 07:22 | |
| *** itlinux has quit IRC | 07:43 | |
| *** rcernin has quit IRC | 07:58 | |
| *** gcb has quit IRC | 08:00 | |
| *** magicboiz has quit IRC | 08:00 | |
| *** gcb has joined #openstack-trove | 08:03 | |
| *** magicboiz has joined #openstack-trove | 08:03 | |
| *** magicboiz has quit IRC | 08:08 | |
| *** magicboiz has joined #openstack-trove | 08:15 | |
| *** dasTor has joined #openstack-trove | 08:37 | |
| *** tianhui has joined #openstack-trove | 09:10 | |
| *** tosky has joined #openstack-trove | 09:17 | |
| *** gcb has quit IRC | 09:37 | |
| *** tianhui has quit IRC | 09:38 | |
| *** tianhui_ has joined #openstack-trove | 09:38 | |
| *** links has quit IRC | 09:42 | |
| *** kumarmn has joined #openstack-trove | 09:51 | |
| *** maciejjozefczyk has joined #openstack-trove | 09:52 | |
| *** maciejjo1 has quit IRC | 09:55 | |
| *** kumarmn has quit IRC | 09:56 | |
| *** lijinhui has left #openstack-trove | 10:07 | |
| *** daidv has quit IRC | 10:12 | |
| *** robcresswell has quit IRC | 11:03 | |
| *** smatzek has joined #openstack-trove | 11:15 | |
| *** kumarmn has joined #openstack-trove | 11:52 | |
| *** kumarmn has quit IRC | 11:57 | |
| *** magicboiz has quit IRC | 12:16 | |
| openstackgerrit | Samuel Matzek proposed openstack/python-troveclient master: WIP: Fix gate / add tempest job https://review.openstack.org/515393 | 12:42 |
| *** kei_yama has quit IRC | 13:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-troveclient stable/pike: Updated from global requirements https://review.openstack.org/505886 | 13:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/trove stable/pike: Updated from global requirements https://review.openstack.org/493203 | 13:49 |
| *** spectr has quit IRC | 13:49 | |
| *** zhaochao has quit IRC | 13:53 | |
| *** kumarmn has joined #openstack-trove | 13:57 | |
| *** kumarmn has quit IRC | 14:00 | |
| *** kumarmn has joined #openstack-trove | 14:00 | |
| *** tianhui_ has quit IRC | 14:00 | |
| *** tianhui has joined #openstack-trove | 14:01 | |
| *** gouthamr has joined #openstack-trove | 14:01 | |
| *** Dinesh_Bhor has quit IRC | 14:04 | |
| *** tosky has quit IRC | 14:06 | |
| *** tosky has joined #openstack-trove | 14:08 | |
| *** spectr has joined #openstack-trove | 14:09 | |
| *** McClymontS has joined #openstack-trove | 14:16 | |
| *** McClymontS has quit IRC | 14:26 | |
| *** kumarmn has quit IRC | 14:38 | |
| *** kumarmn has joined #openstack-trove | 14:38 | |
| *** spectr has quit IRC | 14:39 | |
| *** kumarmn has quit IRC | 14:45 | |
| *** tosky has quit IRC | 15:12 | |
| *** links has joined #openstack-trove | 15:19 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP: Scope the use of kvm virt type in gate https://review.openstack.org/515741 | 15:26 |
| *** McClymontS has joined #openstack-trove | 15:45 | |
| *** McClymontS has quit IRC | 15:47 | |
| *** dasTor has quit IRC | 15:49 | |
| *** itlinux has joined #openstack-trove | 15:57 | |
| *** robcresswell has joined #openstack-trove | 16:19 | |
| *** tosky has joined #openstack-trove | 16:19 | |
| *** links has quit IRC | 16:38 | |
| openstackgerrit | Merged openstack/trove master: Fix requirepass problem with redis https://review.openstack.org/514955 | 17:09 |
| openstackgerrit | Samuel Matzek proposed openstack/python-troveclient master: Fix gate / add tempest job https://review.openstack.org/515393 | 17:18 |
| *** itlinux has quit IRC | 17:28 | |
| smatzek | gate status update: master is working fairly well, we're able to merge do and do a few rechecks. | 17:40 |
| smatzek | once https://review.openstack.org/#/c/514769/ and https://review.openstack.org/#/c/515393/ the python-troveclient gate should be unblocked | 17:41 |
| smatzek | stable/pike remains blocked waiting for https://review.openstack.org/#/c/514710/ to pass rechecks | 17:42 |
| smatzek | I've been investigating a timeout failure where the instance fails with a KVM error like this http://logs.openstack.org/10/514710/3/gate/legacy-trove-scenario-dsvm-mysql-multi/b05115e/logs/libvirt/qemu/instance-00000001.txt.gz | 17:43 |
| smatzek | This only happens when Trove's devstack plugin reconfigures Nova to use kvm virt_type. However, I don't want to disable that reconfigure as it greatly increases our gate job success ratio. | 17:44 |
| smatzek | In my initial investigation, all the successes I saw were using later versions of libvirt and qemu. This morning I've found successes with those older versions of libvirt and qemu. | 17:45 |
| smatzek | This failure is similar to this bug https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1682077 | 17:46 |
| openstack | Launchpad bug 1682077 in linux-lts-xenial (Ubuntu) "nested KVM fails - KVM: entry failed, hardware error 0x0 " [High,Confirmed] | 17:46 |
| smatzek | that bug points at the kernel of the libvirt host or the host of that VM if it is a VM. | 17:46 |
| smatzek | it may also have something to do with the processor model. I have seen failures only with (Haswell, no TSX) processors but we also have one success. This is leading me to believe the issue lies in the host of the VM the jobs are running in, probably in its kernel. | 17:48 |
| smatzek | The next round of investigation is to gather the hosts from all the successes and fails I can find where we've set virt_type kvm to see if there is a pattern. From the data set I have so far, the hostnames of the devstack VMs that have failed are *-ovh-gra1 and I have no successes with that hostname. | 17:49 |
| smatzek | I do have a success on ubuntu-xenial-ovh-bhs1 | 17:50 |
| smatzek | one thing I missed above, is that the kernel level between success and failures of the devstack VM are the same. | 17:52 |
| *** itlinux has joined #openstack-trove | 17:53 | |
| *** harlowja has quit IRC | 18:05 | |
| *** harlowja has joined #openstack-trove | 18:05 | |
| *** smatzek has quit IRC | 18:52 | |
| *** chhavi has quit IRC | 19:02 | |
| *** smatzek has joined #openstack-trove | 19:27 | |
| *** tesseract has quit IRC | 19:33 | |
| *** smatzek has quit IRC | 19:47 | |
| *** smatzek has joined #openstack-trove | 19:55 | |
| *** rcernin has joined #openstack-trove | 19:57 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: Do not configure kvm virt_type in devstack https://review.openstack.org/515814 | 19:59 |
| *** itlinux has quit IRC | 20:01 | |
| *** miqui has joined #openstack-trove | 20:04 | |
| *** smatzek has quit IRC | 20:05 | |
| *** McClymontS has joined #openstack-trove | 20:39 | |
| *** itlinux has joined #openstack-trove | 20:41 | |
| *** itlinux has quit IRC | 21:07 | |
| *** smatzek has joined #openstack-trove | 21:07 | |
| *** itlinux has joined #openstack-trove | 21:15 | |
| smatzek | amrith, given the above ^, I've looked into seeing if it is host specific and if there bread crumbs in the logs to help. What I've found so far is that all the failures I've found happen in the ovh cloud, but its not tied to a specific region. These are two runs one, succeeded and one failed and both were in the bhs1 region of ovh cloud. | 21:16 |
| smatzek | I'm not sure where to go from here. I could check host name and avoid setting virt_type kvm ovh cloud instances, but putting nodepool specific info in the trovestack devstack plugin seems wrong. | 21:17 |
| smatzek | I could try to fire up a VM of some sort in the devstack plugin and see if it fails with kvm virt, but that seems like a lot of work and overkill. The patch to fix the stable/pike gate has hit this particular problem several times in its multiple rechecks and given that we may actually have higher overall success rate if we run with qemu all the time. | 21:19 |
| smatzek | I have a patch up against master to test this. I have seen the tests succeed often enough with qemu. | 21:19 |
| smatzek | I don't really want to throw out the kvm virt_type runs since they do go faster, but I'm on the fence here. Thoughts? | 21:20 |
| smatzek | the two runs on the same region in ovh: | 21:25 |
| smatzek | http://logs.openstack.org/10/514710/3/gate/legacy-trove-scenario-dsvm-mysql-multi/7fbfa70 | 21:25 |
| smatzek | http://logs.openstack.org/10/514710/3/gate/legacy-trove-scenario-dsvm-mysql-single/9ca522e | 21:26 |
| *** McClymontS has quit IRC | 21:27 | |
| *** rcernin has quit IRC | 21:28 | |
| *** itlinux has quit IRC | 21:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-troveclient stable/pike: Updated from global requirements https://review.openstack.org/505886 | 22:03 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-troveclient master: Updated from global requirements https://review.openstack.org/500033 | 22:05 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/trove stable/pike: Updated from global requirements https://review.openstack.org/493203 | 22:06 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-troveclient master: Updated from global requirements https://review.openstack.org/500033 | 22:07 |
| *** smatzek has quit IRC | 22:09 | |
| *** smatzek has joined #openstack-trove | 22:10 | |
| *** smatzek has quit IRC | 22:11 | |
| *** smatzek has joined #openstack-trove | 22:11 | |
| *** smatzek has quit IRC | 22:16 | |
| *** itlinux has joined #openstack-trove | 22:33 | |
| *** itlinux has quit IRC | 22:46 | |
| *** kumarmn has joined #openstack-trove | 22:54 | |
| *** tosky has quit IRC | 23:00 | |
| *** kumarmn has quit IRC | 23:03 | |
| *** kumarmn has joined #openstack-trove | 23:04 | |
| *** kumarmn has quit IRC | 23:08 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!