15:00:24 <slaweq> #startmeeting neutron_ci
15:00:25 <openstack> Meeting started Tue Nov 10 15:00:24 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:26 <slaweq> hi
15:00:28 <openstack> The meeting name has been set to 'neutron_ci'
15:00:34 <ralonsoh> hi
15:00:59 <obondarev> o/
15:01:05 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:01:12 <slaweq> please open now and we can move on :)
15:01:17 <lajoskatona> Hi
15:01:52 <bcafarel> hey again
15:02:17 <slaweq> #topic Actions from previous meetings
15:02:27 <slaweq> slaweq to propose patch to check console log before ssh to instance in tempest
15:02:32 <slaweq> Patch https://review.opendev.org/#/c/761964/
15:02:38 <slaweq> I need to address gmann's comments there
15:03:32 <slaweq> next one
15:03:34 <slaweq> slaweq to check failing neutron-grenade-ovn job
15:03:40 <slaweq> I still didn't had time for that one
15:03:47 <slaweq> #action slaweq to check failing neutron-grenade-ovn job
15:03:54 <slaweq> ralonsoh to check fullstack issue https://bugs.launchpad.net/neutron/+bug/1902678
15:03:57 <openstack> Launchpad bug 1902678 in neutron "[Fullstack] Wrong output of the cmdline causes tests timeouts" [Critical,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)
15:04:18 <ralonsoh> #link https://review.opendev.org/#/c/761202/
15:05:39 <bcafarel> oops I know the author of the linked change
15:05:57 <ralonsoh> hehehe that was a side effect
15:07:12 <slaweq> thx, approved
15:07:20 <slaweq> ok, next one
15:07:22 <slaweq> slaweq to blacklist some cinder related tests in the neutron-tempest-* jobs
15:07:30 <slaweq> I started proposing something else, first in grenade jobs:
15:07:31 <slaweq> https://review.opendev.org/#/c/761518/
15:07:41 <slaweq> basically I think that we can disable cinder services in our jobs
15:08:05 <slaweq> we don't need to test cinder in our gate so if we will disable cinder services we will have 2 adventages:
15:08:14 <slaweq> - no cinder related tests run in the jobs
15:08:23 <slaweq> - less resources used by job
15:08:34 <ralonsoh> +1 to this
15:08:44 <slaweq> I first want to try with grenade but later will do the same for scenario jobs also
15:08:49 <lajoskatona> +1
15:10:24 <slaweq> and that's all regarding actions from last week
15:10:30 <slaweq> I think we can move on
15:10:32 <slaweq> #topic Stadium projects
15:11:02 <slaweq> lajoskatona: anything regarding stadium projects' ci for today?
15:11:16 <lajoskatona> nothing  in my mind
15:11:24 <bcafarel> one stadium/stable topic https://bugs.launchpad.net/neutron/+bug/1903689
15:11:24 <openstack> Launchpad bug 1903689 in neutron "[stable/ussuri] Functional job fails - AttributeError: module 'neutron_lib.constants' has no attribute 'DEVICE_OWNER_DISTRIBUTED'" [Medium,New]
15:11:49 <bcafarel> sfc and fwaas functional are broken on ussuri, not sure for other stadium projects - look like they do not cap the neutron-lib version used?
15:11:52 <lajoskatona> I need to go for some help ti infra or qa team regardning some hard to fix ODL job failures, but that;s a small thing
15:14:27 <slaweq> bcafarel: will You take care of it with?
15:14:33 <slaweq> is it reported somewhere on LP?
15:15:06 <bcafarel> if I find some time for it yes, but most probably not this week :)
15:15:48 <slaweq> bcafarel: ok, please at least report it on LP so we will not lost it
15:16:21 <bcafarel> slaweq: it is, bug 1903689 (a few lines above)
15:16:22 <openstack> bug 1903689 in neutron "[stable/ussuri] Functional job fails - AttributeError: module 'neutron_lib.constants' has no attribute 'DEVICE_OWNER_DISTRIBUTED'" [Medium,New] https://launchpad.net/bugs/1903689
15:18:08 <slaweq> ok, thx bcafarel
15:18:43 <slaweq> ok, lets move on then
15:18:45 <slaweq> #topic Stable branches
15:18:51 <slaweq> bcafarel: any updates here?
15:19:31 <bcafarel> we disabled rally job in rocky, details in https://review.opendev.org/#/c/761391/
15:20:14 <bcafarel> full support branches I saw patches requiring a few rechecks, but no big issue at least (timeouts, volume test, etc)
15:21:26 <slaweq> thx
15:21:39 <slaweq> yes, rocky should be finally unblocked with that Your patch
15:22:21 <slaweq> sorry, I forgot about dashboards
15:22:24 <slaweq> Victoria dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1
15:22:25 <slaweq> Ussuri dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1
15:23:12 <bcafarel> one thing we may need for stable is https://review.opendev.org/#/c/761178/ devstack backport, but I did not have time to check if we hit this issue too in stable
15:23:33 <bcafarel> though first we need it in master :)
15:23:40 <slaweq> I saw this only on master branch
15:25:11 <slaweq> ok, anything else regarding stable branches' ci?
15:25:29 <bcafarel> not from me
15:25:46 <slaweq> ok, lets move on then
15:25:53 <slaweq> #topic Grafana
15:26:36 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:27:05 <slaweq> in overall it looks as usuall
15:27:54 <slaweq> but functional and fullstack tests are failing pretty often recently
15:28:40 <slaweq> lets discuss about them now
15:28:42 <slaweq> #topic fullstack/functional
15:29:04 <slaweq> I found today few cases where some ovn functional tests are failing like:
15:29:15 <slaweq> https://cd9fca83cb14b422f67b-df162b2d8bf471312ec213b580870b4a.ssl.cf5.rackcdn.com/752795/20/check/neutron-functional-with-uwsgi/1243600/controller/logs/dsvm-functional-logs/neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_global_events/testrun.txt
15:29:25 <slaweq> so it's some error 500 from the server
15:29:31 <slaweq> did You saw it also maybe?
15:30:19 <bcafarel> not familiar with this one
15:30:32 <ralonsoh> a couple of times, yes
15:30:41 <ralonsoh> I think this is related to the DB transaction
15:30:45 <ralonsoh> I can check it
15:30:55 <slaweq> ralonsoh: thx
15:31:01 <slaweq> I will open LP for that also
15:31:06 <slaweq> and will send You link to it
15:31:16 <ralonsoh> thanks
15:31:37 <slaweq> #action slaweq to report bug regarding error 500 in ovn functional tests
15:31:51 <slaweq> #action ralonsoh to check error 500 in ovn functional tests
15:31:54 <slaweq> :)
15:31:56 <slaweq> thx
15:32:25 <slaweq> other issue which I saw (again) recently was timeout in namespace operations:
15:32:27 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ef3/759908/3/check/neutron-functional-with-uwsgi/ef3de15/testr_results.html
15:33:40 <slaweq> ralonsoh: I was pretty sure You were fixing such bug some time ago, no?
15:33:44 <ralonsoh> this is on "set_link_attribute" method
15:36:10 <slaweq> so is it different one?
15:36:23 <ralonsoh> I think so
15:37:02 <slaweq> ok, I will open LP for that one too
15:37:28 <slaweq> #action slaweq to report LP regarding functional test timeout on set_link_attribute method
15:37:57 <slaweq> ok, I think we can move on to the next topic now
15:38:04 <slaweq> #topic Tempest/Scenario
15:39:26 <slaweq> first of all there are some patches which can hopefully improve a bit our ci: https://review.opendev.org/#/q/topic:improve-neutron-ci+(status:open+OR+status:merged)
15:39:39 <slaweq> please review them if You will have few minutes :)
15:39:57 <slaweq> and tbh I didn
15:40:42 <slaweq> I didn't found any new issues there - in most cases those jobs are failing due to bug in devstack (fixed by Nate), kernel panic in guest vms, or some timeouts
15:41:42 <lajoskatona> I think this is the fix from Nate: https://review.opendev.org/761178
15:42:05 <slaweq> lajoskatona: yes
15:42:57 <slaweq> do You have maybe any examples of the failures which You would like to discuss here?
15:43:07 <slaweq> or maybe anything else regarding our CI?
15:43:50 * mlavalle will review those patches
15:43:52 <bcafarel> all good from me (busy catching up on these improve-neutron-ci reviews)
15:44:25 <slaweq> thx
15:44:51 <slaweq> if there is nothing else, I think we can finish earlier today
15:45:05 <slaweq> thx for attending this meeting and have a great day
15:45:07 <slaweq> o/
15:45:09 <bcafarel> o/
15:45:12 <ralonsoh> bye
15:45:12 <slaweq> #endmeeting