15:00:27 #startmeeting neutron_ci 15:00:28 Meeting started Wed Jul 1 15:00:27 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:30 hi 15:00:31 The meeting name has been set to 'neutron_ci' 15:00:32 hi 15:02:30 lets wait few more minutes for others 15:02:35 \p 15:02:37 \o 15:03:56 bcafarel: njohnston: ping :) 15:04:09 Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:04:25 o/ 15:04:37 sorry I was listening to Edge session 15:04:42 ahh 15:04:43 ok 15:04:48 I forgot about it 15:04:58 ok, lets start 15:05:00 #topic Actions from previous meetings 15:05:05 bcafarel to check gate status on rocky and queens (uwsgi problem) 15:05:39 ok so patch in neutron itself is not enough, we need multiple fixes in devstack 15:05:58 latest iteration in devstack itself looks good to go https://review.opendev.org/#/c/735615/ 15:06:43 then we will need https://review.opendev.org/#/c/738851/ or something like that on top - with Depends-On I still see failures that should be fixed by the devstack one 15:06:52 hopefully recheck once it is merged should be greener 15:07:14 and once rocky is finally back on track, similar backports for older branches :) 15:07:24 ok 15:07:39 so it seems like we are still far from green gate for queens and rocky 15:08:11 o/ 15:08:47 yup :/ but there is progress at least 15:09:00 thx bcafarel for taking care of it 15:09:20 ok, next one 15:09:25 maciejjozefczyk will check test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindings failiure in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:09:40 fixed 15:09:50 thx maciejjozefczyk :) 15:09:55 #link https://review.opendev.org/#/c/738415/ 15:10:06 so next one 15:10:09 ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:10:41 maciejjozefczyk: one question, should we backport it to ussuri? 15:10:42 slaweq, sorry, I didn't start debbuging 15:11:23 slaweq, hmm we can 15:11:45 maciejjozefczyk: ok, can You propose backport? 15:11:49 slaweq, clicked :D 15:11:52 thx 15:12:05 ralonsoh: sure, I know You were busy with other things 15:12:12 will You try to check that next week? 15:12:15 sure 15:12:28 #action ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:12:29 thx 15:12:37 ok, next one 15:12:42 slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:12:51 I didn't have too much time to check this one 15:13:57 but from what I looked today it seems for me like maybe some tests are "overlapping" and other test cleanded some port/bridge 15:14:26 I will probably add some additional debug logging to be able to maybe investigate it more when it will happen again 15:14:45 and I will try to continue work on it this week 15:14:49 #action slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html 15:15:01 and the last one 15:15:02 maciejjozefczyk to check failing neutron-ovn-tempest-ovs-master-fedora periodic job 15:15:11 yeah 15:15:15 #link https://review.opendev.org/#/c/737984/ 15:15:47 I saw that ovn jobs were failing on this patch today morning 15:15:54 the issue is trivial, ovs compilation code was duplicated in both ovn and ovs modules 15:15:56 so lets wait how it will be now :) 15:16:06 yeah, slaweq, this should be fine now 15:16:14 (I hope so) :D 15:16:39 and I think that one may be interesting for focal transition too 15:16:43 This is about cleaning and unifying the way we compile ovs/ovn, so next time we'll need to fix one function instead two 15:16:48 bcafarel, indeed 15:17:25 ++ 15:17:27 thx 15:18:03 ok 15:18:17 that are all actions from last week 15:18:23 lets move on to the next topic 15:18:24 #topic Stadium projects 15:18:59 with migration to zuul v3 stadium projects are actually good now 15:19:07 as only neutron-ovn-grenade job is still missing 15:19:22 \o/ 15:19:35 anything else You want to discuss about stadium projects today? 15:21:12 ok, so lets move on 15:21:18 #topic Stable branches 15:21:23 Ussuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 15:21:25 Train dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 15:22:05 from what I was seeing this week, branches which are not EM are running pretty good now 15:22:09 I mostly looked at rocky that week, but I think ussuri to stein were OK 15:22:14 and EM are red due to known reasons 15:23:14 I thought about one small improvement to safe some gate resources: wdyt about moving all non-voting jobs to experimental queue in the EM branches? 15:23:55 :) I was thinking about that when filling https://review.opendev.org/#/c/738851/ 15:24:20 :) 15:24:28 so there is at least 2 of us 15:24:42 that would safe about 4-6 jobs, mostly multinode 15:24:55 so pretty many vms spawned to test each patch 15:25:16 probability is low that someone will work on fixing them in EM, and I don't think anyone checks their results in backports 15:25:26 exactly 15:25:37 almost noboday is checking non-voting jobs in master 15:27:31 ralonsoh: njohnston any thoughts? 15:27:43 if You are ok with this I will propose such patch this week 15:27:44 agree with removing them 15:28:18 I absolutely agree 15:28:27 There is no reason to be using those resources 15:28:47 ok 15:28:49 thx 15:28:54 so I will propose such change 15:29:12 #action slaweq to move non-voting jobs to experimental queue in EM branches 15:29:32 anything else regarding stable branches? or can we move on? 15:29:50 nothing from me 15:30:44 nothing from me 15:30:54 ok, lets move on then 15:30:56 #topic Grafana 15:31:02 #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:32:09 I don't see any serious problems there 15:33:16 I pushed small patch to update dashboard https://review.opendev.org/738784 15:34:32 ok, lets talk about some specific issues 15:34:34 #topic Tempest/Scenario 15:34:48 I found few failures for which I opened LPs 15:34:56 first one is 15:35:04 tempest.api.network.admin.test_routers.RoutersAdminTest.test_create_router_set_gateway_with_fixed_ip 15:35:09 in job neutron-tempest-dvr-ha-multinode-full 15:35:19 it happens very often 15:35:28 like e.g.: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c4/734876/1/check/neutron-tempest-dvr-ha-multinode-full/2c466a0/testr_results.html 15:35:35 Bug reported https://bugs.launchpad.net/neutron/+bug/1885897 15:35:35 Launchpad bug 1885897 in neutron "Tempest test_create_router_set_gateway_with_fixed_ip test is failing often in dvr scenario job" [High,Confirmed] 15:35:51 in fact this is main reason of often failures of this job 15:36:34 any volunteer to check that one? 15:36:43 if not, I will try to check it this week 15:36:44 sorry, not this week 15:37:10 ok, I will check it 15:37:29 #action slaweq to investigate failures in test_create_router_set_gateway_with_fixed_ip test 15:37:41 I think that this may be some tempest cleaning issue maybe 15:37:45 or something like that 15:37:50 as it happens very often 15:38:11 next one is related to qos test 15:38:25 I found it in neutron-tempest-plugin-scenario-openvswitch job but I think it may happen also in other jobs 15:38:30 neutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update 15:38:35 https://20f4a85411442f4e3555-9f5a5e2736e26bdd8715596753fafe10.ssl.cf2.rackcdn.com/734876/1/check/neutron-tempest-plugin-scenario-openvswitch/a31f86b/testr_results.html 15:38:42 Bug reported https://bugs.launchpad.net/neutron/+bug/1885899 15:38:42 Launchpad bug 1885899 in neutron "test_qos_basic_and_update test is failing" [Critical,Confirmed] 15:38:55 seems like nc wasn't spawned properly - maybe we should add additional logging in https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/common/utils.py#L122 ? 15:40:14 anyone wants to check that? 15:40:27 but I remember you changed the way to spawn nc 15:40:31 making it more reliable 15:40:45 yes 15:40:57 sorry, but next week 15:41:01 not this one 15:41:18 ok, lets keep it unassigned, maybe someone will want to check it 15:41:29 I marked it as critical because it impacts voting jobs 15:41:47 ok, next one 15:42:07 this is related only to the ovn based jobs where test neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers is failing 15:42:14 like e.g. https://4ec598fcefc6b0367120-6910015cdc6b96c34eca0ab65a68e7f2.ssl.cf5.rackcdn.com/696926/18/check/neutron-ovn-tempest-full-multinode-ovs-master/c1c51ca/testr_results.html 15:42:25 Bug reported: https://bugs.launchpad.net/neutron/+bug/1885898 15:42:25 Launchpad bug 1885898 in neutron "test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job" [High,Confirmed] 15:42:39 maciejjozefczyk: will You have time to take a look into this? 15:43:10 slaweq, You have my sword 15:43:20 maciejjozefczyk: thx a lot 15:43:33 :) 15:43:34 #action maciejjozefczyk to check neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failure in ovn jobs 15:44:43 maciejjozefczyk: there is also another failure in ovn based jobs 15:44:44 neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle 15:44:48 https://7bea12b2d1429b68c6c8-10caedded388001c6bbc38619ca4b324.ssl.cf2.rackcdn.com/737047/8/check/neutron-ovn-tempest-full-multinode-ovs-master/592e31b/testr_results.html 15:44:56 Bug reported: https://bugs.launchpad.net/neutron/+bug/1885900 15:44:56 Launchpad bug 1885900 in neutron "test_trunk_subport_lifecycle is failing in ovn based jobs" [Critical,Confirmed] 15:44:57 slaweq, yeah notiecd that bug :( 15:45:08 that fails pretty often now? 15:45:14 You will probably don't have time to work on both this week but please keep it in mind :) 15:45:54 I saw couple of times at lease in last week 15:46:12 see http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22line%20240%2C%20in%20test_trunk_subport_lifecycle%5C%22 15:46:47 and that's all what I had for today 15:47:03 I opened LP for each of failures so we can track them there 15:47:07 slaweq, thanks for the link 15:47:16 maciejjozefczyk: yw :) 15:47:26 anything else You want to talk about today? 15:48:42 * bcafarel looks for correct tab to copy link 15:49:31 https://review.opendev.org/#/c/738163/ I started to change a few bits for focal 15:49:46 still early WIP but if you want to add stuff, please do so 15:50:18 thx bcafarel 15:50:21 sure bcafarel thanks 15:50:30 ovn fails in all the cases :D 15:50:41 I added myself to the reviewers to be up to date with this :) 15:50:50 damm ovn :P 15:50:51 +1 15:50:58 :) yes it needs more work on the skip ovs/ovn compilation bits 15:51:02 we should move it to the stadium project ;P 15:51:12 ahah 15:51:19 maciejjozefczyk: wdyt? 15:51:21 :D 15:51:50 ROFL 15:53:00 ok, I think we can finish this meeting now 15:53:04 thx for attending 15:53:08 and see You next week :) 15:53:12 #endmeeting