15:00:24 #startmeeting neutron_ci 15:00:25 Meeting started Tue Nov 10 15:00:24 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:26 hi 15:00:28 The meeting name has been set to 'neutron_ci' 15:00:34 hi 15:00:59 o/ 15:01:05 Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:01:12 please open now and we can move on :) 15:01:17 Hi 15:01:52 hey again 15:02:17 #topic Actions from previous meetings 15:02:27 slaweq to propose patch to check console log before ssh to instance in tempest 15:02:32 Patch https://review.opendev.org/#/c/761964/ 15:02:38 I need to address gmann's comments there 15:03:32 next one 15:03:34 slaweq to check failing neutron-grenade-ovn job 15:03:40 I still didn't had time for that one 15:03:47 #action slaweq to check failing neutron-grenade-ovn job 15:03:54 ralonsoh to check fullstack issue https://bugs.launchpad.net/neutron/+bug/1902678 15:03:57 Launchpad bug 1902678 in neutron "[Fullstack] Wrong output of the cmdline causes tests timeouts" [Critical,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) 15:04:18 #link https://review.opendev.org/#/c/761202/ 15:05:39 oops I know the author of the linked change 15:05:57 hehehe that was a side effect 15:07:12 thx, approved 15:07:20 ok, next one 15:07:22 slaweq to blacklist some cinder related tests in the neutron-tempest-* jobs 15:07:30 I started proposing something else, first in grenade jobs: 15:07:31 https://review.opendev.org/#/c/761518/ 15:07:41 basically I think that we can disable cinder services in our jobs 15:08:05 we don't need to test cinder in our gate so if we will disable cinder services we will have 2 adventages: 15:08:14 - no cinder related tests run in the jobs 15:08:23 - less resources used by job 15:08:34 +1 to this 15:08:44 I first want to try with grenade but later will do the same for scenario jobs also 15:08:49 +1 15:10:24 and that's all regarding actions from last week 15:10:30 I think we can move on 15:10:32 #topic Stadium projects 15:11:02 lajoskatona: anything regarding stadium projects' ci for today? 15:11:16 nothing in my mind 15:11:24 one stadium/stable topic https://bugs.launchpad.net/neutron/+bug/1903689 15:11:24 Launchpad bug 1903689 in neutron "[stable/ussuri] Functional job fails - AttributeError: module 'neutron_lib.constants' has no attribute 'DEVICE_OWNER_DISTRIBUTED'" [Medium,New] 15:11:49 sfc and fwaas functional are broken on ussuri, not sure for other stadium projects - look like they do not cap the neutron-lib version used? 15:11:52 I need to go for some help ti infra or qa team regardning some hard to fix ODL job failures, but that;s a small thing 15:14:27 bcafarel: will You take care of it with? 15:14:33 is it reported somewhere on LP? 15:15:06 if I find some time for it yes, but most probably not this week :) 15:15:48 bcafarel: ok, please at least report it on LP so we will not lost it 15:16:21 slaweq: it is, bug 1903689 (a few lines above) 15:16:22 bug 1903689 in neutron "[stable/ussuri] Functional job fails - AttributeError: module 'neutron_lib.constants' has no attribute 'DEVICE_OWNER_DISTRIBUTED'" [Medium,New] https://launchpad.net/bugs/1903689 15:18:08 ok, thx bcafarel 15:18:43 ok, lets move on then 15:18:45 #topic Stable branches 15:18:51 bcafarel: any updates here? 15:19:31 we disabled rally job in rocky, details in https://review.opendev.org/#/c/761391/ 15:20:14 full support branches I saw patches requiring a few rechecks, but no big issue at least (timeouts, volume test, etc) 15:21:26 thx 15:21:39 yes, rocky should be finally unblocked with that Your patch 15:22:21 sorry, I forgot about dashboards 15:22:24 Victoria dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 15:22:25 Ussuri dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 15:23:12 one thing we may need for stable is https://review.opendev.org/#/c/761178/ devstack backport, but I did not have time to check if we hit this issue too in stable 15:23:33 though first we need it in master :) 15:23:40 I saw this only on master branch 15:25:11 ok, anything else regarding stable branches' ci? 15:25:29 not from me 15:25:46 ok, lets move on then 15:25:53 #topic Grafana 15:26:36 #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate 15:27:05 in overall it looks as usuall 15:27:54 but functional and fullstack tests are failing pretty often recently 15:28:40 lets discuss about them now 15:28:42 #topic fullstack/functional 15:29:04 I found today few cases where some ovn functional tests are failing like: 15:29:15 https://cd9fca83cb14b422f67b-df162b2d8bf471312ec213b580870b4a.ssl.cf5.rackcdn.com/752795/20/check/neutron-functional-with-uwsgi/1243600/controller/logs/dsvm-functional-logs/neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_global_events/testrun.txt 15:29:25 so it's some error 500 from the server 15:29:31 did You saw it also maybe? 15:30:19 not familiar with this one 15:30:32 a couple of times, yes 15:30:41 I think this is related to the DB transaction 15:30:45 I can check it 15:30:55 ralonsoh: thx 15:31:01 I will open LP for that also 15:31:06 and will send You link to it 15:31:16 thanks 15:31:37 #action slaweq to report bug regarding error 500 in ovn functional tests 15:31:51 #action ralonsoh to check error 500 in ovn functional tests 15:31:54 :) 15:31:56 thx 15:32:25 other issue which I saw (again) recently was timeout in namespace operations: 15:32:27 https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ef3/759908/3/check/neutron-functional-with-uwsgi/ef3de15/testr_results.html 15:33:40 ralonsoh: I was pretty sure You were fixing such bug some time ago, no? 15:33:44 this is on "set_link_attribute" method 15:36:10 so is it different one? 15:36:23 I think so 15:37:02 ok, I will open LP for that one too 15:37:28 #action slaweq to report LP regarding functional test timeout on set_link_attribute method 15:37:57 ok, I think we can move on to the next topic now 15:38:04 #topic Tempest/Scenario 15:39:26 first of all there are some patches which can hopefully improve a bit our ci: https://review.opendev.org/#/q/topic:improve-neutron-ci+(status:open+OR+status:merged) 15:39:39 please review them if You will have few minutes :) 15:39:57 and tbh I didn 15:40:42 I didn't found any new issues there - in most cases those jobs are failing due to bug in devstack (fixed by Nate), kernel panic in guest vms, or some timeouts 15:41:42 I think this is the fix from Nate: https://review.opendev.org/761178 15:42:05 lajoskatona: yes 15:42:57 do You have maybe any examples of the failures which You would like to discuss here? 15:43:07 or maybe anything else regarding our CI? 15:43:50 * mlavalle will review those patches 15:43:52 all good from me (busy catching up on these improve-neutron-ci reviews) 15:44:25 thx 15:44:51 if there is nothing else, I think we can finish earlier today 15:45:05 thx for attending this meeting and have a great day 15:45:07 o/ 15:45:09 o/ 15:45:12 bye 15:45:12 #endmeeting