15:01:26 <ykarel> #startmeeting neutron_ci 15:01:26 <opendevmeet> Meeting started Mon Dec 2 15:01:26 2024 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:26 <opendevmeet> The meeting name has been set to 'neutron_ci' 15:01:32 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 15:01:37 <ralonsoh> hi 15:01:47 <slaweq> o/ 15:01:49 <ralonsoh> irc, right? no video 15:02:16 <ykarel> yes irc 15:02:26 <opendevreview> Bodo Petermann proposed openstack/neutron-lib master: VPNaaS-related device owner definitions https://review.opendev.org/c/openstack/neutron-lib/+/936842 15:02:33 <haleyb> o/ 15:02:55 <opendevreview> Rodolfo Alonso proposed openstack/neutron stable/2024.2: Fix the tagging policy engine https://review.opendev.org/c/openstack/neutron/+/936843 15:03:02 <mlavalle1> \o 15:04:10 <ykarel> let's start with topics 15:04:12 <ykarel> #topic Actions from previous meetings 15:04:17 <opendevreview> Rodolfo Alonso proposed openstack/neutron stable/2024.2: Fix the tagging policy engine https://review.opendev.org/c/openstack/neutron/+/936843 15:04:21 <ykarel> ykarel to check/report issue for stadiums 15:04:28 <ykarel> https://bugs.launchpad.net/neutron/+bug/2089673 15:04:28 <ykarel> https://review.opendev.org/c/openstack/releases/+/936238 15:04:28 <ykarel> https://review.opendev.org/q/topic:%22stadium-stable-jobs%22 15:04:28 <ykarel> https://review.opendev.org/c/openstack/neutron-fwaas/+/936739 15:05:19 <ykarel> most of the fixes already merged, 2 pending, please check those 15:05:41 <ykarel> #topic Stable branches 15:06:10 <ykarel> Bernard not around today, from what i see all good in stable/unmaintained branches 15:06:19 <ykarel> #topic Stadium projects 15:06:41 <ykarel> lajos seems not around today 15:06:49 <ykarel> from weekly runs 15:06:50 <ykarel> dynamic-routing,fwaas,vpnaas,sfc,bgpvpn functional job failed with noble switch, https://review.opendev.org/c/openstack/neutron/+/932920 should clear it 15:07:06 <ykarel> the fix merged, so next run should be good 15:07:22 <ykarel> #topic Rechecks 15:07:38 <ykarel> we still have random issues requiring rechecks 15:07:55 <ykarel> and had 4/56 bare rechecks 15:07:57 <ykarel> not bad 15:08:05 <ykarel> #topic fullstack/functional 15:08:10 <ykarel> test_virtual_port_host_update_upon_failover 15:08:16 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_14e/935040/2/gate/neutron-functional-with-uwsgi/14e888f/testr_results.html 15:08:17 <ykarel> - proposed backport https://review.opendev.org/c/openstack/neutron/+/936815 15:08:25 <ykarel> please check the backport ^ 15:08:33 <ykarel> test_ovn_nb_sync_repair 15:08:39 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9ef/936340/1/check/neutron-functional-with-uwsgi/9ef9cb1/testr_results.html 15:08:39 <ykarel> - proposed backports https://review.opendev.org/q/If1da6c5992aa4635da5a4b5c6eaa06db56d693b4 15:08:50 <ykarel> please check ^ backports too 15:08:59 <ykarel> test_ha_router_failover_host_failure 15:09:13 <ykarel> Fails like AssertionError: destination ip 42.0.0.248 is replying to ping from namespace test-28f80714-08a4-4d39-90f2-3ce92ea6217e, but it shouldn't 15:09:23 <ykarel> seen couple of times 15:09:25 <ykarel> - https://5c003e9488bb945dd53b-63e63f4901d8c8037ad85f11ee8978c0.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-fullstack-with-pyroute2-master/06dd02e/testr_results.html 15:09:25 <ykarel> - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e5c/periodic/opendev.org/openstack/neutron/master/neutron-fullstack/e5cb4b0/testr_results.html 15:09:25 <ykarel> - https://92076981c91368a87195-3da5177dffc08bd93069f7a2b6d5d8cd.ssl.cf2.rackcdn.com/936428/2/check/neutron-fullstack/6f8e782/testr_results.html 15:09:40 <ykarel> Seems same issue https://bugs.launchpad.net/neutron/+bug/2083609 just fails differently 15:10:02 <ykarel> slaweq as you worked on that issue, if you can confirm ^ 15:10:53 <slaweq> I will take a look 15:11:07 <ykarel> thx slaweq 15:11:28 <ykarel> #action slawek to check/confirm fullstack failures with lp #2083609 15:11:37 <ykarel> #topic Tempest/Scenario 15:11:48 <ykarel> test_list_security_groups_list_all_tenants_filter 15:11:57 <ykarel> seen once 15:12:00 <ykarel> Fails like Details: HTTP response body is invalid (None is not of type 'string' 15:12:06 <ykarel> https://1a679a15ede886d68369-dfb11b8e45238c790fb951827c502144.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-ovn-tempest-postgres-full/c92c3d6/testr_results.html 15:12:37 <ykarel> seems triggered by a concurrent test where description was None 15:14:27 <ykarel> can keep an eye for now, will report lp if see it again 15:14:31 <ralonsoh> let me check that 15:14:39 <ralonsoh> this could be as easy as always adding a description 15:14:43 <ralonsoh> we do that in n-t-p 15:15:27 <ykarel> that test does does add the description, some other likely not 15:15:36 <ykarel> but yes adding explicit should clear it 15:15:56 <ralonsoh> I'll open a LP at least to document it 15:16:09 <ykarel> #action ralonsoh to open bug for test test_list_security_groups_list_all_tenants_filter 15:16:11 <ykarel> thx ralonsoh 15:16:47 <ykarel> we still have couple of random failures related to wsgi switch 15:17:22 <ykarel> we have couple of fixes and test patch in flight 15:17:27 <ykarel> thx ralonsoh for working on those 15:17:29 <ralonsoh> working on this now: https://review.opendev.org/c/openstack/neutron/+/936429 15:17:34 <ralonsoh> (check the lower patches) 15:18:07 <ykarel> meanwhile we can also consider a CI workaround https://review.opendev.org/c/openstack/neutron/+/936829 to avoid such issues impacting gate 15:18:17 <ykarel> basically reducing api workers to 1 15:18:51 <ralonsoh> I know this is hitting the CI frequently but not so much to be a problem right now 15:19:02 <ralonsoh> with the current config, we find more issues every day 15:19:12 <ralonsoh> I would, for now, keep the current number of workers 15:20:04 <ykarel> ok if we can live with these failures, will keep it open if we want to go that way some time 15:20:31 <ykarel> #topic Periodic 15:20:33 <opendevreview> Bodo Petermann proposed openstack/neutron-vpnaas master: Switch to neutron-lib device owner constants https://review.opendev.org/c/openstack/neutron-vpnaas/+/936844 15:20:48 <ykarel> with noble switch mariadb job is broken 15:20:50 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-mariadb-full&project=openstack%2Fneutron&branch=master&skip=0 15:21:02 <ykarel> reported https://bugs.launchpad.net/devstack/+bug/2090835 15:21:18 <ykarel> also pushed a fix for that, will do run test job for it 15:21:32 <ykarel> #topic Grafana 15:21:37 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate 15:21:45 <ykarel> let's also have quick look here 15:23:11 <ykarel> we can see some spikes in ovn jobs, and those are mostly wsgi switch related issues 15:23:22 <ykarel> apart from that it looks normal 15:23:28 <ykarel> anything to add? 15:23:38 <ralonsoh> no 15:24:46 <slaweq> nope 15:24:52 <mlavalle1> no 15:24:59 <ykarel> #topic On Demand 15:25:11 <ykarel> just a small one 15:25:13 <ykarel> Updating cirros to 0.6.3 https://review.opendev.org/q/topic:%22bug/2045549%22 15:25:29 <ykarel> anything else you would like to raise? 15:26:09 <ralonsoh> no thanks 15:26:13 <mlavalle1> no 15:27:12 <ykarel> thx everyone for joining 15:27:16 <ykarel> #endmeeting