| *** smatzek has quit IRC | 00:04 | |
| *** harlowja has quit IRC | 00:38 | |
| *** gcb has joined #openstack-trove | 01:30 | |
| *** gcb has quit IRC | 01:37 | |
| *** gcb has joined #openstack-trove | 01:39 | |
| *** zhaochao has quit IRC | 01:51 | |
| *** zhaochao has joined #openstack-trove | 01:53 | |
| *** itlinux has joined #openstack-trove | 01:54 | |
| *** itlinux has quit IRC | 02:01 | |
| *** https_GK1wmSU has joined #openstack-trove | 02:05 | |
| *** https_GK1wmSU has left #openstack-trove | 02:05 | |
| *** harlowja has joined #openstack-trove | 02:22 | |
| *** itlinux has joined #openstack-trove | 02:27 | |
| *** harlowja has quit IRC | 02:44 | |
| *** itlinux has quit IRC | 03:01 | |
| *** chenying has joined #openstack-trove | 03:17 | |
| *** gouthamr has quit IRC | 03:28 | |
| *** harlowja has joined #openstack-trove | 03:38 | |
| *** links has joined #openstack-trove | 04:07 | |
| *** itlinux has joined #openstack-trove | 05:05 | |
| *** harlowja has quit IRC | 05:10 | |
| *** hoonetorg has quit IRC | 05:27 | |
| *** gcb has quit IRC | 05:29 | |
| *** gcb_ has joined #openstack-trove | 05:29 | |
| *** itlinux has quit IRC | 05:35 | |
| *** hoonetorg has joined #openstack-trove | 05:44 | |
| *** sudhirag098 has joined #openstack-trove | 05:51 | |
| *** harlowja has joined #openstack-trove | 05:51 | |
| *** harlowja has quit IRC | 06:51 | |
| *** rcernin has joined #openstack-trove | 06:55 | |
| *** pcaruana has joined #openstack-trove | 07:06 | |
| *** rcernin has quit IRC | 07:48 | |
| *** rcernin has joined #openstack-trove | 07:50 | |
| *** tosky has joined #openstack-trove | 09:24 | |
| *** smatzek has joined #openstack-trove | 11:04 | |
| openstackgerrit | Gábor Antal proposed openstack/trove master: Remove unnecessary setUp() calls in tests https://review.openstack.org/489567 | 11:22 |
|---|---|---|
| *** hoonetorg has quit IRC | 11:54 | |
| *** hoonetorg has joined #openstack-trove | 12:06 | |
| *** links has quit IRC | 12:40 | |
| *** sudhirag098 has quit IRC | 12:43 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Add logging to investigate gate failure https://review.openstack.org/488575 | 12:45 |
| *** kylek3h has joined #openstack-trove | 13:01 | |
| *** gouthamr has joined #openstack-trove | 13:03 | |
| *** tosky has quit IRC | 13:31 | |
| *** tosky has joined #openstack-trove | 13:31 | |
| *** rcernin has quit IRC | 15:14 | |
| *** pcaruana has quit IRC | 15:22 | |
| *** pcaruana has joined #openstack-trove | 15:40 | |
| *** rcernin has joined #openstack-trove | 15:49 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Add logging to investigate gate failure https://review.openstack.org/488575 | 15:50 |
| *** links has joined #openstack-trove | 16:12 | |
| *** itlinux has joined #openstack-trove | 16:25 | |
| *** pcaruana has quit IRC | 16:58 | |
| smatzek | amrith, SlickNik, do either of you know how to specify nova.conf settings for Trove gate jobs? or if it's even possible? | 17:01 |
| smatzek | gate-trove-scenario-dsvm-mysql-multi-ubuntu-xenial is consistently failing on the anti-affinity replica test and all signs point to Nova scheduler violating the anti-affinity policy. With the lack of logging in the scheduler its tough to get a smoking gun, so I am slowly adding more logging on the Trove and Trove test side to get the smoking gun. | 17:05 |
| smatzek | The reason I ask about nova.conf settings is that in the gate this conf, which has a default of 'True' is being set to 'False'. If this were enabled, nova compute would likely catch the scheduler's violation of the policy, Trove's test would pass and our gate would be unwedged. https://github.com/openstack/nova/blob/master/nova/conf/workarounds.py#L128-L143 | 17:08 |
| *** kumarmn has joined #openstack-trove | 17:13 | |
| smatzek | correction on the defaults. It's defaulted to 'False' which would mean the nova compute double check of the policy would happen, but in the gate the value is 'True', disabling the re-check of the policy by nova compute and letting nova scheduler violations of the policy fly in the gate. | 17:20 |
| *** tosky has quit IRC | 17:39 | |
| *** links has quit IRC | 17:57 | |
| *** tosky has joined #openstack-trove | 18:03 | |
| *** itlinux has quit IRC | 18:06 | |
| *** itlinux has joined #openstack-trove | 18:34 | |
| *** itlinux has quit IRC | 18:36 | |
| *** kumarmn has quit IRC | 18:40 | |
| *** kumarmn has joined #openstack-trove | 18:41 | |
| *** kumarmn_ has joined #openstack-trove | 18:41 | |
| *** kumarmn has quit IRC | 18:45 | |
| *** kumarmn_ has quit IRC | 18:51 | |
| smatzek | amrith, for the current gate issue with replicas/clusters, we now have a proof that Nova is violating the policy. This link shows a nova client response showing 2 VMs in an anti-affinity server group, in a 1 compute host environment. http://logs.openstack.org/75/488575/8/check/gate-trove-scenario-dsvm-mysql-multi-ubuntu-xenial/886ca21/console.html#_2017-08-01_17_10_08_307727 | 18:54 |
| smatzek | both VMs are ACTIVE as well, which can be seen in the nova compute log. | 18:56 |
| smatzek | it also appears that in the failing cases Nova scheduler is not emitting INFO messages like this Successfully synced instances from host 'ubuntu-xenial-rax-ord-10003022'., so it is not getting the latest instance-host info which would explain why it's allowing the violation to happen | 18:59 |
| *** kumarmn has joined #openstack-trove | 19:02 | |
| *** itlinux has joined #openstack-trove | 19:35 | |
| *** rcernin has quit IRC | 19:36 | |
| *** kumarmn has quit IRC | 19:37 | |
| amrith | sure you can set nova options; just change then in our devstack plugin :) | 19:39 |
| amrith | it is a kludge | 19:39 |
| amrith | but we can do it. | 19:39 |
| amrith | we already do it to tweak the kvm options | 19:39 |
| amrith | and the nested hypervisor settings | 19:39 |
| *** kumarmn has joined #openstack-trove | 19:50 | |
| smatzek | can you point me at where that code lives? It would be a kludge, since the post above I've continued to look into nova scheduler logs for succeeding and failing runs. In all failing cases Nova compute to Nova scheudler rpc/notifications are not happening. I'm not sure why yet, infra setup failure?? | 19:50 |
| smatzek | setting that conf would at least catch this one case, but if those notifications/rpcs aren't working between nova components it may be moot. | 19:51 |
| *** itlinux has quit IRC | 19:54 | |
| *** kumarmn has quit IRC | 19:56 | |
| smatzek | I've also diffed the confs dumped out by the nova scheduler and nova compute processes for successful and failing gate runs. No differences there. | 19:56 |
| smatzek | amrith, nevermind on the question about where the code is to set nova conf settings, I found it. | 20:17 |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Test enabling Nova compute policy check https://review.openstack.org/489731 | 20:25 |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: Test enabling nova policy check in nova-cpu.conf https://review.openstack.org/489733 | 20:28 |
| *** itlinux has joined #openstack-trove | 20:35 | |
| smatzek | I think this change to devstack, related to other nova changes is responsible. https://github.com/openstack-dev/devstack/commit/f3d533157267a1568eeb36a349aee637195e9c8a | 21:23 |
| smatzek | Nova's own gate is currently not seeing RPC between compute and scheduler. | 21:23 |
| smatzek | The change adds transport_url under the [oslo_messaging_notifications] | 21:24 |
| smatzek | section. In nova.conf used by the scheduler values are the same, in nova-cpu.conf used by nova compute, the values are different. | 21:24 |
| *** smatzek has quit IRC | 21:42 | |
| *** eandersson has quit IRC | 21:50 | |
| *** smatzek has joined #openstack-trove | 22:01 | |
| *** smatzek_ has joined #openstack-trove | 22:02 | |
| *** smatzek has quit IRC | 22:06 | |
| itlinux | hello all.. is there a yaml to build trove under Tripleo? TY | 22:08 |
| *** kylek3h has quit IRC | 22:12 | |
| *** kylek3h has joined #openstack-trove | 22:13 | |
| *** kylek3h has quit IRC | 22:13 | |
| *** smatzek_ has quit IRC | 22:20 | |
| *** smatzek_ has joined #openstack-trove | 22:36 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: WIP/DNM Turn off track_instance_changes https://review.openstack.org/489773 | 22:42 |
| *** itlinux has quit IRC | 22:49 | |
| *** smatzek_ has quit IRC | 22:49 | |
| *** gouthamr has quit IRC | 23:43 | |
| -openstackstatus- NOTICE: osic nodes have been removed from nodepool due to a problem with the mirror host beginning around 22:20 UTC. please recheck any jobs with failures installing packages. | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!