Wednesday, 2020-07-01

*** yamamoto has joined #openstack-meeting-300:46
*** yamamoto has quit IRC00:49
*** yamamoto has joined #openstack-meeting-300:58
*** markvoelker has joined #openstack-meeting-301:20
*** markvoelker has quit IRC01:24
*** ricolin_ has joined #openstack-meeting-301:43
*** ricolin_ is now known as ricolin03:30
*** psachin has joined #openstack-meeting-303:33
*** markvoelker has joined #openstack-meeting-304:03
*** markvoelker has quit IRC04:08
*** apetrich has quit IRC04:43
*** mugsie has quit IRC04:53
*** mugsie has joined #openstack-meeting-304:57
*** markvoelker has joined #openstack-meeting-305:03
*** markvoelker has quit IRC05:07
*** links has joined #openstack-meeting-305:34
*** njohnston has quit IRC06:18
*** markvoelker has joined #openstack-meeting-307:04
*** markvoelker has quit IRC07:09
*** slaweq has quit IRC07:12
*** slaweq has joined #openstack-meeting-307:13
*** markvoelker has joined #openstack-meeting-307:18
*** markvoelker has quit IRC07:23
*** ralonsoh has joined #openstack-meeting-307:33
*** e0ne has joined #openstack-meeting-307:58
*** apetrich has joined #openstack-meeting-308:48
*** yamamoto has quit IRC09:37
*** ricolin has quit IRC09:52
*** yamamoto has joined #openstack-meeting-309:55
*** yamamoto has quit IRC10:06
*** yamamoto has joined #openstack-meeting-310:07
*** apetrich has quit IRC10:12
*** njohnston has joined #openstack-meeting-311:01
*** markvoelker has joined #openstack-meeting-311:15
*** markvoelker has quit IRC11:19
*** raildo has joined #openstack-meeting-311:28
*** apetrich has joined #openstack-meeting-311:45
*** csatari has quit IRC11:53
*** csatari has joined #openstack-meeting-311:56
*** irclogbot_3 has quit IRC12:16
*** irclogbot_0 has joined #openstack-meeting-312:17
*** trident has quit IRC12:46
*** trident has joined #openstack-meeting-312:49
*** ricolin has joined #openstack-meeting-312:54
*** psachin has quit IRC13:31
*** liuyulong has joined #openstack-meeting-313:50
*** yamamoto has quit IRC13:50
*** yamamoto has joined #openstack-meeting-314:11
*** yamamoto has quit IRC14:11
*** yamamoto has joined #openstack-meeting-314:12
*** yamamoto has quit IRC14:16
*** maciejjozefczyk has joined #openstack-meeting-314:28
*** links has quit IRC14:55
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Wed Jul  1 15:00:27 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
slaweqhi15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
ralonsohhi15:00
slaweqlets wait few more minutes for others15:02
maciejjozefczyk\p15:02
maciejjozefczyk\o15:02
slaweqbcafarel: njohnston: ping :)15:03
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:04
bcafarelo/15:04
bcafarelsorry I was listening to Edge session15:04
slaweqahh15:04
slaweqok15:04
slaweqI forgot about it15:04
slaweqok, lets start15:04
slaweq#topic Actions from previous meetings15:05
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:05
slaweqbcafarel to check gate status on rocky and queens (uwsgi problem)15:05
*** suryasingh has joined #openstack-meeting-315:05
bcafarelok so patch in neutron itself is not enough, we need multiple fixes in devstack15:05
bcafarellatest iteration in devstack itself looks good to go https://review.opendev.org/#/c/735615/15:05
bcafarelthen we will need https://review.opendev.org/#/c/738851/ or something like that on top - with Depends-On I still see failures that should be fixed by the devstack one15:06
bcafarelhopefully recheck once it is merged should be greener15:06
bcafareland once rocky is finally back on track, similar backports for older branches :)15:07
slaweqok15:07
slaweqso it seems like we are still far from green gate for queens and rocky15:07
njohnstono/15:08
bcafarelyup :/ but there is progress at least15:08
slaweqthx bcafarel for taking care of it15:09
slaweqok, next one15:09
slaweqmaciejjozefczyk will check test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindings failiure in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html15:09
maciejjozefczykfixed15:09
slaweqthx maciejjozefczyk :)15:09
maciejjozefczyk#link https://review.opendev.org/#/c/738415/15:09
slaweqso next one15:10
slaweqralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html15:10
slaweqmaciejjozefczyk: one question, should we backport it to ussuri?15:10
ralonsohslaweq, sorry, I didn't start debbuging15:10
maciejjozefczykslaweq, hmm we can15:11
slaweqmaciejjozefczyk: ok, can You propose backport?15:11
maciejjozefczykslaweq, clicked :D15:11
slaweqthx15:11
slaweqralonsoh: sure, I know You were busy with other things15:12
slaweqwill You try to check that next week?15:12
ralonsohsure15:12
slaweq#action ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html15:12
slaweqthx15:12
slaweqok, next one15:12
slaweqslaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html15:12
slaweqI didn't have too much time to check this one15:12
slaweqbut from what I looked today it seems for me like maybe some tests are "overlapping" and other test cleanded some port/bridge15:13
slaweqI will probably add some additional debug logging to be able to maybe investigate it more when it will happen again15:14
slaweqand I will try to continue work on it this week15:14
slaweq#action slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html15:14
slaweqand the last one15:15
slaweqmaciejjozefczyk to check failing neutron-ovn-tempest-ovs-master-fedora periodic job15:15
maciejjozefczykyeah15:15
maciejjozefczyk#link https://review.opendev.org/#/c/737984/15:15
slaweqI saw that ovn jobs were failing on this patch today morning15:15
maciejjozefczykthe issue is trivial, ovs compilation code was duplicated in both ovn and ovs modules15:15
slaweqso lets wait how it will be now :)15:15
maciejjozefczykyeah, slaweq, this should be fine now15:16
maciejjozefczyk(I hope so) :D15:16
bcafareland I think that one may be interesting for focal transition too15:16
maciejjozefczykThis is about cleaning and unifying the way we compile ovs/ovn, so next time we'll need to fix one function instead two15:16
maciejjozefczykbcafarel, indeed15:16
slaweq++15:17
slaweqthx15:17
slaweqok15:18
slaweqthat are all actions from last week15:18
slaweqlets move on to the next topic15:18
slaweq#topic Stadium projects15:18
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:18
slaweqwith migration to zuul v3 stadium projects are actually good now15:18
slaweqas only neutron-ovn-grenade job is still missing15:19
njohnston\o/15:19
slaweqanything else You want to discuss about stadium projects today?15:19
slaweqok, so lets move on15:21
slaweq#topic Stable branches15:21
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:21
slaweqUssuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=115:21
slaweqTrain dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=115:21
slaweqfrom what I was seeing this week, branches which are not EM are running pretty good now15:22
bcafarelI mostly looked at rocky that week, but I think ussuri to stein were OK15:22
slaweqand EM are red due to known reasons15:22
slaweqI thought about one small improvement to safe some gate resources: wdyt about moving all non-voting jobs to experimental queue in the EM branches?15:23
bcafarel:) I was thinking about that when filling https://review.opendev.org/#/c/738851/15:23
slaweq:)15:24
slaweqso there is at least 2 of us15:24
slaweqthat would safe about 4-6 jobs, mostly multinode15:24
slaweqso pretty many vms spawned to test each patch15:24
bcafarelprobability is low that someone will work on fixing them in EM, and I don't think anyone checks their results in backports15:25
slaweqexactly15:25
slaweqalmost noboday is checking non-voting jobs in master15:25
slaweqralonsoh: njohnston any thoughts?15:27
slaweqif You are ok with this I will propose such patch this week15:27
ralonsohagree with removing them15:27
njohnstonI absolutely agree15:28
njohnstonThere is no reason to be using those resources15:28
slaweqok15:28
slaweqthx15:28
slaweqso I will propose such change15:28
slaweq#action slaweq to move non-voting jobs to experimental queue in EM branches15:29
slaweqanything else regarding stable branches? or can we move on?15:29
bcafarelnothing from me15:29
njohnstonnothing from me15:30
slaweqok, lets move on then15:30
slaweq#topic Grafana15:30
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:30
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:31
slaweqI don't see any serious problems there15:32
slaweqI pushed small patch to update dashboard https://review.opendev.org/73878415:33
slaweqok, lets talk about some specific issues15:34
slaweq#topic Tempest/Scenario15:34
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:34
slaweqI found few failures for which I opened LPs15:34
slaweqfirst one is15:34
slaweqtempest.api.network.admin.test_routers.RoutersAdminTest.test_create_router_set_gateway_with_fixed_ip15:35
slaweqin job neutron-tempest-dvr-ha-multinode-full15:35
slaweqit happens very often15:35
*** factor has joined #openstack-meeting-315:35
slaweqlike e.g.: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c4/734876/1/check/neutron-tempest-dvr-ha-multinode-full/2c466a0/testr_results.html15:35
slaweq    Bug reported https://bugs.launchpad.net/neutron/+bug/188589715:35
openstackLaunchpad bug 1885897 in neutron "Tempest test_create_router_set_gateway_with_fixed_ip test is failing often in dvr scenario job" [High,Confirmed]15:35
slaweqin fact this is main reason of often failures of this job15:35
slaweqany volunteer to check that one?15:36
slaweqif not, I will try to check it this week15:36
ralonsohsorry, not this week15:36
slaweqok, I will check it15:37
slaweq#action slaweq to investigate failures in test_create_router_set_gateway_with_fixed_ip test15:37
slaweqI think that this may be some tempest cleaning issue maybe15:37
slaweqor something like that15:37
slaweqas it happens very often15:37
slaweqnext one is related to qos test15:38
slaweqI found it in neutron-tempest-plugin-scenario-openvswitch job but I think it may happen also in other jobs15:38
slaweqneutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update15:38
slaweqhttps://20f4a85411442f4e3555-9f5a5e2736e26bdd8715596753fafe10.ssl.cf2.rackcdn.com/734876/1/check/neutron-tempest-plugin-scenario-openvswitch/a31f86b/testr_results.html15:38
slaweq    Bug reported https://bugs.launchpad.net/neutron/+bug/188589915:38
openstackLaunchpad bug 1885899 in neutron "test_qos_basic_and_update test is failing" [Critical,Confirmed]15:38
slaweqseems like nc wasn't spawned properly - maybe we should add additional logging in https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/common/utils.py#L122 ?15:38
slaweqanyone wants to check that?15:40
ralonsohbut I remember you changed the way to spawn nc15:40
ralonsohmaking it more reliable15:40
slaweqyes15:40
ralonsohsorry, but next week15:40
ralonsohnot this one15:41
slaweqok, lets keep it unassigned, maybe someone will want to check it15:41
slaweqI marked it as critical because it impacts voting jobs15:41
slaweqok, next one15:41
slaweqthis is related only to the ovn based jobs where test     neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers is failing15:42
slaweqlike e.g. https://4ec598fcefc6b0367120-6910015cdc6b96c34eca0ab65a68e7f2.ssl.cf5.rackcdn.com/696926/18/check/neutron-ovn-tempest-full-multinode-ovs-master/c1c51ca/testr_results.html15:42
slaweqBug reported: https://bugs.launchpad.net/neutron/+bug/188589815:42
openstackLaunchpad bug 1885898 in neutron "test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job" [High,Confirmed]15:42
slaweqmaciejjozefczyk: will You have time to take a look into this?15:42
maciejjozefczykslaweq, You have my sword15:43
slaweqmaciejjozefczyk: thx a lot15:43
bcafarel:)15:43
slaweq#action maciejjozefczyk to check neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failure in ovn jobs15:43
slaweqmaciejjozefczyk: there is also another failure in ovn based jobs15:44
slaweqneutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle15:44
slaweqhttps://7bea12b2d1429b68c6c8-10caedded388001c6bbc38619ca4b324.ssl.cf2.rackcdn.com/737047/8/check/neutron-ovn-tempest-full-multinode-ovs-master/592e31b/testr_results.html15:44
slaweq    Bug reported: https://bugs.launchpad.net/neutron/+bug/188590015:44
openstackLaunchpad bug 1885900 in neutron "test_trunk_subport_lifecycle is failing in ovn based jobs" [Critical,Confirmed]15:44
maciejjozefczykslaweq, yeah notiecd that bug :(15:44
maciejjozefczykthat fails pretty often now?15:45
slaweqYou will probably don't have time to work on both this week but please keep it in mind :)15:45
slaweqI saw couple of times at lease in last week15:45
slaweqsee http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22line%20240%2C%20in%20test_trunk_subport_lifecycle%5C%2215:46
slaweqand that's all what I had for today15:46
slaweqI opened LP for each of failures so we can track them there15:47
maciejjozefczykslaweq, thanks for the link15:47
slaweqmaciejjozefczyk: yw :)15:47
slaweqanything else You want to talk about today?15:47
* bcafarel looks for correct tab to copy link15:48
bcafarelhttps://review.opendev.org/#/c/738163/ I started to change a few bits for focal15:49
bcafarelstill early WIP but if you want to add stuff, please do so15:49
slaweqthx bcafarel15:50
maciejjozefczyksure bcafarel thanks15:50
maciejjozefczykovn fails in all the cases :D15:50
slaweqI added myself to the reviewers to be up to date with this :)15:50
slaweqdamm ovn :P15:50
njohnston+115:50
bcafarel:) yes it needs more work on the skip ovs/ovn compilation bits15:50
slaweqwe should move it to the stadium project ;P15:51
bcafarelahah15:51
slaweqmaciejjozefczyk: wdyt?15:51
slaweq:D15:51
njohnstonROFL15:51
slaweqok, I think we can finish this meeting now15:53
slaweqthx for attending15:53
slaweqand see You next week :)15:53
slaweq#endmeeting15:53
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:53
openstackMeeting ended Wed Jul  1 15:53:12 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:53
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.html15:53
ralonsohbye15:53
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.txt15:53
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.log.html15:53
bcafarelo/15:53
maciejjozefczyklol :P15:54
maciejjozefczykbb!15:54
*** ricolin has quit IRC16:01
*** liuyulong has quit IRC16:05
*** etp has quit IRC16:16
*** etp has joined #openstack-meeting-316:19
*** etp has quit IRC16:24
*** etp has joined #openstack-meeting-316:44
*** factor has quit IRC16:51
*** icarusfactor has joined #openstack-meeting-316:51
*** ralonsoh has quit IRC17:10
*** ralonsoh has joined #openstack-meeting-317:10
*** bnemec has quit IRC17:15
*** factor has joined #openstack-meeting-317:16
*** icarusfactor has quit IRC17:16
*** bnemec has joined #openstack-meeting-317:21
*** tsmith2 has quit IRC17:26
*** freerunner has quit IRC17:26
*** hongbin has joined #openstack-meeting-317:28
*** irclogbot_0 has quit IRC17:28
*** tsmith2 has joined #openstack-meeting-317:29
*** freerunner has joined #openstack-meeting-317:29
*** irclogbot_2 has joined #openstack-meeting-317:30
*** njohnston_ has joined #openstack-meeting-317:43
*** njohnston has quit IRC17:43
*** yamamoto has joined #openstack-meeting-317:44
*** njohnston_ is now known as njohnston17:45
*** yamamoto has quit IRC17:53
*** e0ne has quit IRC17:59
*** slaweq has quit IRC18:25
*** hongbin has quit IRC18:44
*** factor has quit IRC18:46
*** factor has joined #openstack-meeting-318:46
*** factor has quit IRC19:04
*** factor has joined #openstack-meeting-319:04
*** factor has quit IRC19:10
*** hongbin has joined #openstack-meeting-319:18
*** ralonsoh has quit IRC19:50
*** hongbin has quit IRC20:06
*** e0ne has joined #openstack-meeting-320:20
*** slaweq has joined #openstack-meeting-320:25
*** maciejjozefczyk has quit IRC20:36
*** hongbin has joined #openstack-meeting-320:54
*** e0ne has quit IRC21:07
*** slaweq has quit IRC21:19
*** factor has joined #openstack-meeting-321:29
*** factor has quit IRC21:33
*** factor has joined #openstack-meeting-321:33
*** factor has quit IRC21:49
*** factor has joined #openstack-meeting-321:50
*** icarusfactor has joined #openstack-meeting-322:33
*** factor has quit IRC22:36
*** hongbin has quit IRC23:11
*** hongbin has joined #openstack-meeting-323:16
*** yamamoto has joined #openstack-meeting-323:52
*** yamamoto has quit IRC23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!