| *** itlinux has quit IRC | 00:12 | |
| *** itlinux has joined #openstack-trove | 00:15 | |
| *** itlinux has quit IRC | 00:18 | |
| *** harlowja has quit IRC | 01:10 | |
| *** gouthamr has quit IRC | 01:33 | |
| *** georgelorch has joined #openstack-trove | 01:34 | |
| openstackgerrit | Fan Zhang proposed openstack/trove master: Fix missing package and remove inexistent option. https://review.openstack.org/487716 | 01:38 |
|---|---|---|
| *** fanzhang__ has quit IRC | 01:39 | |
| *** fanzhang has joined #openstack-trove | 01:40 | |
| fanzhang | amrith, are you there ? | 01:46 |
| fanzhang | amrith, the bug number is right. I have updated the bug description. As for .gitignore, is there anything wrong for ignoring .idea directory while do commiting? | 01:50 |
| *** itlinux has joined #openstack-trove | 02:01 | |
| *** itlinux has quit IRC | 02:03 | |
| openstackgerrit | Fan Zhang proposed openstack/trove master: Remove inexistent option in install guide. https://review.openstack.org/487716 | 02:08 |
| openstackgerrit | Merged openstack/trove-dashboard master: Imported Translations from Zanata https://review.openstack.org/486899 | 02:08 |
| *** itlinux has joined #openstack-trove | 02:26 | |
| *** gouthamr has joined #openstack-trove | 03:22 | |
| *** gouthamr has quit IRC | 03:46 | |
| openstackgerrit | Merged openstack/python-troveclient master: Updated from global requirements https://review.openstack.org/485964 | 03:48 |
| *** gouthamr has joined #openstack-trove | 03:58 | |
| *** gouthamr has quit IRC | 03:58 | |
| *** gouthamr has joined #openstack-trove | 03:59 | |
| *** itlinux has quit IRC | 04:07 | |
| openstackgerrit | Merged openstack/python-troveclient master: Drop pycrypto from tests dependencies https://review.openstack.org/484819 | 04:11 |
| *** links has joined #openstack-trove | 04:20 | |
| *** gouthamr has quit IRC | 04:30 | |
| *** harlowja has joined #openstack-trove | 04:32 | |
| *** harlowja has quit IRC | 05:28 | |
| *** harlowja has joined #openstack-trove | 05:41 | |
| *** fanzhang has quit IRC | 05:57 | |
| *** magicboiz has joined #openstack-trove | 06:15 | |
| *** magicboiz has quit IRC | 06:15 | |
| *** magicboiz has joined #openstack-trove | 06:15 | |
| *** rcernin has joined #openstack-trove | 06:15 | |
| *** fanzhang has joined #openstack-trove | 06:20 | |
| *** magicboiz has quit IRC | 06:24 | |
| *** magicboiz has joined #openstack-trove | 06:27 | |
| *** magicboiz has quit IRC | 06:31 | |
| *** magicboiz has joined #openstack-trove | 06:32 | |
| *** harlowja has quit IRC | 06:48 | |
| *** zhaochao has quit IRC | 06:53 | |
| *** zhaochao has joined #openstack-trove | 06:54 | |
| *** fanzhang has quit IRC | 07:33 | |
| *** fanzhang has joined #openstack-trove | 07:33 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** fanzhang has quit IRC | 09:00 | |
| *** fanzhang has joined #openstack-trove | 09:05 | |
| *** tosky has joined #openstack-trove | 09:06 | |
| *** tosky has quit IRC | 09:09 | |
| *** tosky has joined #openstack-trove | 09:33 | |
| *** gcb has quit IRC | 09:53 | |
| *** amoralej has joined #openstack-trove | 10:09 | |
| amoralej | amrith, would it be possible to release a new version of troveclient with https://review.openstack.org/#/c/487417/ before freeze? | 10:10 |
| amoralej | unit tests are failing with 2.11.0 | 10:11 |
| amrith | amoralej sure thing | 10:51 |
| amrith | will do it right now | 10:51 |
| amoralej | amrith, thanks! | 10:51 |
| amrith | amoralej I just checked the queue and there's one more change I'd squeeze in if I could | 10:54 |
| amrith | that is this one | 10:54 |
| amrith | https://review.openstack.org/482018 | 10:54 |
| amrith | it will be a little over an hour before the change merges | 10:54 |
| amrith | Instead of releasing two today, I'll just wait for that one if it is OK with you | 10:55 |
| amoralej | amrith, sure, i can wait | 10:55 |
| amrith | thx amoralej I don't think the release team is awake yet :) | 10:57 |
| amoralej | yeah :) | 10:57 |
| *** AlexeyAbashkin has joined #openstack-trove | 11:40 | |
| *** AlexeyAbashkin has quit IRC | 11:47 | |
| *** rcernin has quit IRC | 11:55 | |
| *** Dinesh_Bhor is now known as Dinesh_Bhor|afk | 12:01 | |
| *** Dinesh_Bhor|afk is now known as Dinesh_Bhor | 12:02 | |
| *** openstackgerrit has joined #openstack-trove | 12:11 | |
| openstackgerrit | Merged openstack/python-troveclient master: Remove log translations https://review.openstack.org/482018 | 12:11 |
| *** rcernin has joined #openstack-trove | 12:16 | |
| *** magicboiz has quit IRC | 12:27 | |
| *** smatzek has joined #openstack-trove | 12:28 | |
| smatzek | amrith, can you take a look at https://review.openstack.org/#/c/487963/ ? It should unwedge the gate. | 12:28 |
| amrith | amoralej one client patch coming up | 12:34 |
| *** magicboiz has joined #openstack-trove | 12:55 | |
| *** magicboiz has quit IRC | 12:55 | |
| *** links has quit IRC | 13:16 | |
| *** zhaochao has quit IRC | 13:20 | |
| *** rcernin has quit IRC | 13:38 | |
| *** miqui has joined #openstack-trove | 13:49 | |
| *** gouthamr has joined #openstack-trove | 13:54 | |
| *** rcernin has joined #openstack-trove | 14:01 | |
| *** itlinux has joined #openstack-trove | 14:24 | |
| *** itlinux has quit IRC | 14:24 | |
| *** trevormc has joined #openstack-trove | 14:28 | |
| *** miqui has quit IRC | 14:33 | |
| *** Dinesh_Bhor has quit IRC | 14:46 | |
| *** harlowja has joined #openstack-trove | 14:51 | |
| *** cp16net has quit IRC | 14:59 | |
| *** trevormc has quit IRC | 15:06 | |
| *** rcernin has quit IRC | 15:13 | |
| *** links has joined #openstack-trove | 15:23 | |
| smatzek | amrith, the gate-trove-scenario-dsvm-mysql-multi-ubuntu-xenial gate has been failing "Wait for non-affinity replica to fail." a lot in the past few days. Is this a common sporadic failure point for the gate? | 15:38 |
| smatzek | I spent hours digging into this yesterday, to the point of looking at the implementation of the Nova anti-affinity filter to see exactly how its checking for anti-affinity. | 15:40 |
| *** itlinux has joined #openstack-trove | 15:40 | |
| smatzek | despite the test case attempting to make a replica of an anti-affinity master with only 1 host, Nova is allowing the replica to be scheduled and become active. The nova scheduler filter is getting an empty list for the list of hosts in the anti-affinity group. | 15:41 |
| amrith | it is periodic, happens more when the CI is under stress | 15:42 |
| *** itlinux has quit IRC | 15:48 | |
| smatzek | have you looked into it much, are you thinking it's some Nova race condition, or something in Trove that could be touched up? i.e. Trove sporadically not passing the Nova server group info correctly? | 15:48 |
| smatzek | if I can get time today between my teleconferences I'm thinking of tossing up a DNM/WIP review to add some more logging in this area to get more hints to root cause | 15:50 |
| amrith | smatzek, I have. the issue is that trove's gate job spawns many VM's simultaneously and is not deterministic. Under load some tests, and some cleanups take a long time and therefore not having any hosts is a possible outcome. Under no load conditions (say Saturday morning eastern time) your job will pass fine but close to a release milestone, this is always a problem. | 15:50 |
| amrith | smatzek go for it, have fun | 15:50 |
| *** harlowja has quit IRC | 15:51 | |
| smatzek | since you're here atm, there is a fix up for review to unwedge one of the gates that Trevor and I worked on yesterday. ^ It's currently failing verify on the anti-affinity failure though. | 15:52 |
| *** amoralej is now known as amoralej|off | 16:04 | |
| *** openstackstatus has quit IRC | 16:41 | |
| *** openstack has joined #openstack-trove | 16:42 | |
| *** links has quit IRC | 16:44 | |
| *** openstackstatus has joined #openstack-trove | 16:44 | |
| *** ChanServ sets mode: +v openstackstatus | 16:44 | |
| *** harlowja has joined #openstack-trove | 16:54 | |
| *** harlowja_ has joined #openstack-trove | 16:55 | |
| *** harlowja has quit IRC | 16:59 | |
| *** trevormc has joined #openstack-trove | 17:04 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being taken offline for roughly 5 minutes to perform a database backup and reconfiguration | 17:14 | |
| *** itlinux has joined #openstack-trove | 17:14 | |
| *** trevormc has quit IRC | 17:16 | |
| *** tosky has quit IRC | 17:31 | |
| *** openstackgerrit has quit IRC | 17:33 | |
| *** openstackstatus has quit IRC | 18:11 | |
| *** openstack has joined #openstack-trove | 18:11 | |
| *** openstackstatus has joined #openstack-trove | 18:14 | |
| *** ChanServ sets mode: +v openstackstatus | 18:14 | |
| *** tosky has joined #openstack-trove | 18:16 | |
| smatzek | amrith, for what its worth, the anti-affinity gate test that keeps failing at load appears to be an issue in Nova, but there is insufficient logging to prove it at this point. In the Trove logs I can see that Trove is creating the anti-affinity master VM with a servergroup, the server group exists. It then waits for it to finish. | 18:32 |
| amrith | yes :) | 18:33 |
| smatzek | about 2 minutes later the replica is created and the servergroup is correctly passed (based on novaclient's curl debug output in trove log). During scheduling of the replica, Nova's filter does a call to get the hosts of all instances in the server group to ensure it doesn't schedule on those, and gets an empty list. | 18:33 |
| amrith | been there, done that, have the t-shirt ... have also turned up logging and and debugged much further. | 18:33 |
| smatzek | not Trove's problem. | 18:33 |
| amrith | no, it is Trove's problem | 18:34 |
| amrith | but, ... if you think it isn't please do escalate it as a problem with Nova. | 18:34 |
| smatzek | for the record, these types of issues bug me, which gives me the drive to keep going on them.... | 18:34 |
| amrith | Like I said earlier, go for it, have fun. | 18:35 |
| amrith | well, for the record, have fun | 18:35 |
| smatzek | if Trove is correctly passing the servergroup on all cases and there is a 2 minute gap between master and slave/replica creation and Nova still doesn't have record of the master's membership in the group, seems to be some problem in Nova. | 18:35 |
| smatzek | any tips on how to inject more logging into Nova while in Trove's gate ? :) | 18:36 |
| smatzek | btw, can I get one of those t-shirts too ??? | 18:37 |
| smatzek | or get a dump of DB table contents? | 18:39 |
| *** openstackstatus has quit IRC | 18:42 | |
| *** openstack has joined #openstack-trove | 18:42 | |
| *** openstackstatus has joined #openstack-trove | 18:44 | |
| *** ChanServ sets mode: +v openstackstatus | 18:44 | |
| *** openstackgerrit has joined #openstack-trove | 19:11 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Add logging to investigate gate failure https://review.openstack.org/488575 | 19:11 |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Add logging to investigate gate failure https://review.openstack.org/488575 | 19:37 |
| *** chenying__ has quit IRC | 19:45 | |
| *** chenying__ has joined #openstack-trove | 19:47 | |
| *** tosky has quit IRC | 19:54 | |
| *** tosky has joined #openstack-trove | 20:01 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Add logging to investigate gate failure https://review.openstack.org/488575 | 20:08 |
| *** smatzek has quit IRC | 20:30 | |
| *** georgelorch has quit IRC | 21:08 | |
| *** gouthamr has quit IRC | 21:24 | |
| *** gouthamr has joined #openstack-trove | 21:37 | |
| *** itlinux has quit IRC | 22:05 | |
| *** georgelorch has joined #openstack-trove | 22:46 | |
| *** tosky has quit IRC | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!