15:01:26 <ykarel> #startmeeting neutron_ci
15:01:26 <opendevmeet> Meeting started Mon Dec  2 15:01:26 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:26 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:01:32 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:01:37 <ralonsoh> hi
15:01:47 <slaweq> o/
15:01:49 <ralonsoh> irc, right? no video
15:02:16 <ykarel> yes irc
15:02:26 <opendevreview> Bodo Petermann proposed openstack/neutron-lib master: VPNaaS-related device owner definitions  https://review.opendev.org/c/openstack/neutron-lib/+/936842
15:02:33 <haleyb> o/
15:02:55 <opendevreview> Rodolfo Alonso proposed openstack/neutron stable/2024.2: Fix the tagging policy engine  https://review.opendev.org/c/openstack/neutron/+/936843
15:03:02 <mlavalle1> \o
15:04:10 <ykarel> let's start with topics
15:04:12 <ykarel> #topic Actions from previous meetings
15:04:17 <opendevreview> Rodolfo Alonso proposed openstack/neutron stable/2024.2: Fix the tagging policy engine  https://review.opendev.org/c/openstack/neutron/+/936843
15:04:21 <ykarel> ykarel to check/report issue for stadiums
15:04:28 <ykarel> https://bugs.launchpad.net/neutron/+bug/2089673
15:04:28 <ykarel> https://review.opendev.org/c/openstack/releases/+/936238
15:04:28 <ykarel> https://review.opendev.org/q/topic:%22stadium-stable-jobs%22
15:04:28 <ykarel> https://review.opendev.org/c/openstack/neutron-fwaas/+/936739
15:05:19 <ykarel> most of the fixes already merged, 2 pending, please check those
15:05:41 <ykarel> #topic Stable branches
15:06:10 <ykarel> Bernard not around today, from what i see all good in stable/unmaintained branches
15:06:19 <ykarel> #topic Stadium projects
15:06:41 <ykarel> lajos seems not around today
15:06:49 <ykarel> from weekly runs
15:06:50 <ykarel> dynamic-routing,fwaas,vpnaas,sfc,bgpvpn functional job failed with noble switch, https://review.opendev.org/c/openstack/neutron/+/932920 should clear it
15:07:06 <ykarel> the fix merged, so next run should be good
15:07:22 <ykarel> #topic Rechecks
15:07:38 <ykarel> we still have random issues requiring rechecks
15:07:55 <ykarel> and had 4/56 bare rechecks
15:07:57 <ykarel> not bad
15:08:05 <ykarel> #topic fullstack/functional
15:08:10 <ykarel> test_virtual_port_host_update_upon_failover
15:08:16 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_14e/935040/2/gate/neutron-functional-with-uwsgi/14e888f/testr_results.html
15:08:17 <ykarel> - proposed backport https://review.opendev.org/c/openstack/neutron/+/936815
15:08:25 <ykarel> please check the backport ^
15:08:33 <ykarel> test_ovn_nb_sync_repair
15:08:39 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9ef/936340/1/check/neutron-functional-with-uwsgi/9ef9cb1/testr_results.html
15:08:39 <ykarel> - proposed backports https://review.opendev.org/q/If1da6c5992aa4635da5a4b5c6eaa06db56d693b4
15:08:50 <ykarel> please check ^ backports too
15:08:59 <ykarel> test_ha_router_failover_host_failure
15:09:13 <ykarel> Fails like AssertionError: destination ip 42.0.0.248 is replying to ping from namespace test-28f80714-08a4-4d39-90f2-3ce92ea6217e, but it shouldn't
15:09:23 <ykarel> seen couple of times
15:09:25 <ykarel> - https://5c003e9488bb945dd53b-63e63f4901d8c8037ad85f11ee8978c0.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-fullstack-with-pyroute2-master/06dd02e/testr_results.html
15:09:25 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e5c/periodic/opendev.org/openstack/neutron/master/neutron-fullstack/e5cb4b0/testr_results.html
15:09:25 <ykarel> - https://92076981c91368a87195-3da5177dffc08bd93069f7a2b6d5d8cd.ssl.cf2.rackcdn.com/936428/2/check/neutron-fullstack/6f8e782/testr_results.html
15:09:40 <ykarel> Seems same issue https://bugs.launchpad.net/neutron/+bug/2083609 just fails differently
15:10:02 <ykarel> slaweq as you worked on that issue, if you can confirm ^
15:10:53 <slaweq> I will take a look
15:11:07 <ykarel> thx slaweq
15:11:28 <ykarel> #action slawek to check/confirm fullstack failures with lp #2083609
15:11:37 <ykarel> #topic Tempest/Scenario
15:11:48 <ykarel> test_list_security_groups_list_all_tenants_filter
15:11:57 <ykarel> seen once
15:12:00 <ykarel> Fails like Details: HTTP response body is invalid (None is not of type 'string'
15:12:06 <ykarel> https://1a679a15ede886d68369-dfb11b8e45238c790fb951827c502144.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-ovn-tempest-postgres-full/c92c3d6/testr_results.html
15:12:37 <ykarel> seems triggered by a concurrent test where description was None
15:14:27 <ykarel> can keep an eye for now, will report lp if see it again
15:14:31 <ralonsoh> let me check that
15:14:39 <ralonsoh> this could be as easy as always adding a description
15:14:43 <ralonsoh> we do that in n-t-p
15:15:27 <ykarel> that test does does add the description, some other likely not
15:15:36 <ykarel> but yes adding explicit should clear it
15:15:56 <ralonsoh> I'll open a LP at least to document it
15:16:09 <ykarel> #action ralonsoh to open bug for test test_list_security_groups_list_all_tenants_filter
15:16:11 <ykarel> thx ralonsoh
15:16:47 <ykarel> we still have couple of random failures related to wsgi switch
15:17:22 <ykarel> we have couple of fixes and test patch in flight
15:17:27 <ykarel> thx ralonsoh for working on those
15:17:29 <ralonsoh> working on this now: https://review.opendev.org/c/openstack/neutron/+/936429
15:17:34 <ralonsoh> (check the lower patches)
15:18:07 <ykarel> meanwhile we can also consider a CI workaround https://review.opendev.org/c/openstack/neutron/+/936829 to avoid such issues impacting gate
15:18:17 <ykarel> basically reducing api workers to 1
15:18:51 <ralonsoh> I know this is hitting the CI frequently but not so much to be a problem right now
15:19:02 <ralonsoh> with the current config, we find more issues every day
15:19:12 <ralonsoh> I would, for now, keep the current number of workers
15:20:04 <ykarel> ok if we can live with these failures, will keep it open if we want to go that way some time
15:20:31 <ykarel> #topic Periodic
15:20:33 <opendevreview> Bodo Petermann proposed openstack/neutron-vpnaas master: Switch to neutron-lib device owner constants  https://review.opendev.org/c/openstack/neutron-vpnaas/+/936844
15:20:48 <ykarel> with noble switch mariadb job is broken
15:20:50 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-mariadb-full&project=openstack%2Fneutron&branch=master&skip=0
15:21:02 <ykarel> reported https://bugs.launchpad.net/devstack/+bug/2090835
15:21:18 <ykarel> also pushed a fix for that, will do run test job for it
15:21:32 <ykarel> #topic Grafana
15:21:37 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:21:45 <ykarel> let's also have quick look here
15:23:11 <ykarel> we can see some spikes in ovn jobs, and those are mostly wsgi switch related issues
15:23:22 <ykarel> apart from that it looks normal
15:23:28 <ykarel> anything to add?
15:23:38 <ralonsoh> no
15:24:46 <slaweq> nope
15:24:52 <mlavalle1> no
15:24:59 <ykarel> #topic On Demand
15:25:11 <ykarel> just a small one
15:25:13 <ykarel> Updating cirros to 0.6.3 https://review.opendev.org/q/topic:%22bug/2045549%22
15:25:29 <ykarel> anything else you would like to raise?
15:26:09 <ralonsoh> no thanks
15:26:13 <mlavalle1> no
15:27:12 <ykarel> thx everyone for joining
15:27:16 <ykarel> #endmeeting