15:01:56 <ykarel> #startmeeting neutron_ci
15:01:56 <opendevmeet> Meeting started Mon Dec 16 15:01:56 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:56 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:56 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:02:03 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:02:26 <haleyb> o/
15:02:47 <bcafarel> o/
15:04:22 <slaweq> o/
15:05:35 <ykarel> ok let's start with the topics
15:05:52 <ykarel> #topic Actions from previous meetings
15:06:02 <ykarel> ralonsoh to check lp #2088423
15:06:16 <ykarel> seems rodolfo not around today
15:07:12 <ykarel> but i saw he pushed a wip patch for this
15:07:26 <ykarel> https://review.opendev.org/c/openstack/neutron/+/937715
15:07:53 <ykarel> which seems to work based on test result
15:08:06 <opendevreview> yatin proposed openstack/neutron master: [DNM] repro functional failure  https://review.opendev.org/c/openstack/neutron/+/937757
15:08:12 <ykarel> next one
15:08:15 <ykarel> ralonsoh to open a bug related to error in test_list_security_groups_list_all_tenants_filter
15:08:23 <ykarel> he opened https://bugs.launchpad.net/neutron/+bug/2091711
15:08:36 <ykarel> next one on me
15:08:38 <ykarel> ykarel to check for oom-kill in functional failures
15:08:50 <ykarel> for this i reported https://bugs.launchpad.net/neutron/+bug/2091855
15:09:07 <ykarel> and proposed CI fix https://review.opendev.org/c/openstack/neutron/+/937759
15:09:16 <ykarel> please review this
15:09:35 <ykarel> in my tests this is helping
15:09:48 <ykarel> when job runs on raxflex provider
15:10:30 <ykarel> neutron-functional-1 https://zuul.opendev.org/t/openstack/build/d76eb751a1814284bcf1f195fbb1de2a : SUCCESS in 1h 01m 25s
15:11:03 <ykarel> #topic Stable branches
15:11:13 <ykarel> bcafarel, anything to share for stable?
15:11:37 <bcafarel> overall good, I saw a few functional failures (not on the same tests), so I think https://bugs.launchpad.net/neutron/+bug/2091855 backport will be useful :)
15:11:52 <bcafarel> (at least I found a few stable jobs in the opensearch query)
15:14:11 <ykarel> bcafarel, that above issue i have seen only in master
15:14:27 <ykarel> but the failures in opensearch in stable also known one
15:14:57 <ykarel> basically https://bugs.launchpad.net/neutron/+bug/2088423
15:15:05 <ykarel> so can backport that once it fixed in master
15:15:31 <bcafarel> ack sounds good
15:16:33 <ykarel> so with this it seems issue looks noble specific
15:16:51 <ykarel> as not observed in stable branches yet, but can watch out that
15:17:01 <ykarel> #topic Stadium projects
15:17:15 <ykarel> vpnaas jobs failed at some ubuntu mirror sync issues
15:17:48 <ykarel> seems some temporary mirror issue, seen in other neutron periodic jobs for last few days, latest run not hit it
15:17:59 <ykarel> neutron-dynamic-router tempest test failure
15:18:05 <ykarel> https://96efaede4ab69a28dc72-7291d6e818a7c847826cc66aee2194e8.ssl.cf1.rackcdn.com/periodic-weekly/opendev.org/openstack/neutron-dynamic-routing/master/neutron-tempest-plugin-dynamic-routing/b3b4458/testr_results.html
15:18:19 <ykarel> 1 test failed and this looks related to other issues we saw with wsgi switch
15:18:24 <ykarel> https://bugs.launchpad.net/neutron/+bug/2085421
15:18:42 <ykarel> #topic Rechecks
15:19:05 <ykarel> with the random issues across functional/fullstack/tempest we have quite good number of rechecks
15:19:19 <ykarel> bare recheck wise we had 8/65
15:19:29 <ykarel> most of those bare in 2 patches from Liu
15:20:04 <ykarel> let's take a look at failures
15:20:05 <ykarel> #topic fullstack/functional
15:20:20 <ykarel> functional tests oom failures https://bugs.launchpad.net/neutron/+bug/2091855
15:20:43 <ykarel> this we already discussed above and impacting CI quite a lot
15:21:05 <ykarel> fullstack ha router test failures https://bugs.launchpad.net/neutron/+bug/2083609
15:21:28 <ykarel> for this i did some checks and updated the bug with details
15:21:33 <ykarel> and proposed a revert https://review.opendev.org/c/openstack/neutron/+/937758
15:21:49 <ykarel> locally couldn't reproduce the issue with the revert
15:22:06 <ykarel> please review this too
15:22:23 <ykarel> with these we should reduce these issues
15:22:40 <ykarel> test_agent_metadata_port_dhcp_reenable_event failed once
15:22:47 <ykarel> https://b98571c7009a427b691e-4e71828f84900af50a9a26357b84a827.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional/9622f70/testr_results.html
15:23:16 <ykarel> only seen once, if anyone wants to take a look?
15:23:47 <ykarel> ok for now will just have a watch on this
15:23:58 <ykarel> #topic Tempest/Scenario
15:24:14 <ykarel> we still have wsgi switch related issues, like the one where LSPUP event is missed
15:24:25 <ykarel> rodolfo looking into that already
15:25:11 <ykarel> test_rebuild_server
15:25:19 <ykarel> seen this twice
15:25:21 <ykarel> - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_701/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-lts/7013470/testr_results.html
15:25:21 <ykarel> - https://8a147734cd8e4123a164-4f751c91a5a63e04cb6adb4d4cd771f4.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovs-tempest-with-os-ken-master/b4df811/testr_results.html
15:27:34 <ykarel> from opensearch atleast i see the test failing in other project gates
15:27:49 <ykarel> so will do check if all failing similarily and report an issue
15:28:02 <ykarel> #action ykarel to check and report an issue with test_rebuild_server
15:28:33 <ykarel> i recall we reported an issue in past for this test, but likely issue was different, but will cross check that
15:28:43 <ykarel> #topic Periodic
15:29:02 <ykarel> linuxbridge stable 2023.2/2024.1 broken as running with wsgi since https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935627
15:29:16 <ykarel> need to fix job definitions for these
15:29:31 <ykarel> anyone want to send the fixes?
15:30:14 <ykarel> kk i will do send that tomorrow
15:30:26 <ykarel> #action ykarel to send job fixes for linuxbridge stable jobs
15:30:37 <ykarel> stable/2024.1 sdk job broken https://zuul.openstack.org/builds?job_name=openstacksdk-functional-devstack-networking&project=openstack%2Fneutron&branch=stable%2F2024.1&skip=0
15:30:47 <ykarel> triggered with https://review.opendev.org/c/openstack/openstacksdk/+/927779
15:31:17 <ykarel> for this also likely need a job fixes to not use openstacksdk master in stable job variants
15:31:31 <ykarel> since that sdk patch
15:31:53 <ykarel> will also send job fix for that
15:32:04 <ykarel> #action ykarel to send job fix for sdk stable
15:32:11 <ykarel> #topic Grafana
15:32:18 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:32:30 <ykarel> let's have a quick look at dashboard
15:34:11 <ykarel> we can see some spikes on 12-15 on tempest jobs, and those were related to mirros issue
15:34:48 <ykarel> rest looks the known issues, or patch specifics
15:34:53 <ykarel> anything to add?
15:35:26 <bcafarel> nothing from me :)
15:37:30 <ykarel> k let's move then :)
15:37:35 <ykarel> #topic On Demand
15:37:44 <ykarel> a small one from me
15:37:48 <ykarel> Ok to Cancel next week CI meeting, will anyone be around ?
15:38:09 <bcafarel> I was thinking of same topic, I don't think many will be around
15:40:16 <ykarel> yes haleyb already pointed he will be not around next few weeks
15:40:29 <ykarel> i guess same for slaweq atleast next week :)
15:41:08 <ykarel> with the limited attendance today i think we agree on this so let's close next one and resume on new year :)
15:41:41 <ykarel> i mean both next ones, 23rd and 30th Dec
15:42:05 <haleyb> sorry, in two meetings at once, but yes, out from thursday until january 9th
15:44:01 <ykarel> okk
15:44:16 <ykarel> added review priority to both ci fixes
15:44:19 <ykarel> https://review.opendev.org/c/openstack/neutron/+/937759
15:44:26 <ykarel> https://review.opendev.org/c/openstack/neutron/+/937758
15:44:32 <ykarel> please check those when get a chance
15:44:35 <ykarel> that's it for today
15:44:40 <ykarel> thx everyone for joining
15:44:44 <ykarel> #endmeeting