14:09:28 <ykarel> #startmeeting neutron_ci
14:09:28 <opendevmeet> Meeting started Mon May  5 14:09:28 2025 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:09:28 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:09:28 <opendevmeet> The meeting name has been set to 'neutron_ci'
14:09:33 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
14:09:37 <mlavalle> \o
14:09:37 <ykarel> IRC meeting today
14:09:40 <mlavalle> lol
14:09:40 <ralonsoh> hello
14:09:51 <mlavalle> irc?
14:10:18 <ykarel> yes irc today
14:10:47 <ykarel> let's give few minutes for others if someone else around to join
14:11:08 <bcafarel> o/
14:13:33 <lajoskatona> o/
14:14:35 <haleyb> o/
14:15:07 <ykarel> thx everyone for joining, let's start with topic
14:15:17 <ykarel> #topic Actions from previous meetings
14:15:27 <ykarel> mlavalle to continue with https://bugs.launchpad.net/neutron/+bug/2104019
14:16:04 <mlavalle> it seems the situation has improved since the change I proposed to the test: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack&skip=0
14:16:54 <ykarel> but wasn't that test fixing part of failures?
14:17:11 <ykarel> the failures like  AssertionError: 119 not less than 90 : Sent 218 packets, received 99 packets, lost 119 packets
14:17:11 <mlavalle> there are still two cases that I'm investigating. One is the case when after the failover, pings and tcp cinnectivity succeed
14:17:17 <mlavalle> but udp still fail
14:17:39 <mlavalle> for that I've been testing with https://review.opendev.org/c/openstack/neutron/+/948280
14:18:00 <mlavalle> more difficuklt to reproduce, so I'll continue rechecking
14:18:22 <ykarel> k thx may be you can add more jobs in same patch replicating neutron-fullstack job
14:18:30 <ykarel> that would be better then multiple rechecks
14:18:50 <mlavalle> ok
14:19:19 <ykarel> like done for another job in https://review.opendev.org/c/openstack/neutron/+/942615/5/zuul.d/project.yaml
14:19:32 <mlavalle> will try that
14:19:58 <ykarel> thx mlavalle for checking this
14:20:08 <ykarel> ykarel to check bagpipe failure is reproducible or not
14:20:16 <mlavalle> ;-)
14:20:24 <ykarel> i did couple of rechecks last weeks and that didn't reproduced
14:20:32 <ykarel> ykarel to report issue for the unit test failure
14:20:44 <ykarel> reported https://bugs.launchpad.net/neutron/+bug/2109326
14:20:46 <lajoskatona> didnt it was reported by ralonsoh?
14:20:57 <ykarel> and also pushed the fix for that
14:21:07 <lajoskatona> ahho, ok another one
14:21:34 <ykarel> lajoskatona, which issue?
14:22:14 <opendevreview> Merged openstack/neutron-fwaas master: Add neutron-tempest-plugin-fwaas-ovn job to the check and gate queues  https://review.opendev.org/c/openstack/neutron-fwaas/+/948062
14:22:23 <ykarel> this is from 2 weeks back, as last week we didn't had any meeting
14:22:33 <lajoskatona> https://bugs.launchpad.net/networking-bagpipe/+bug/2109316
14:23:09 <lajoskatona> ykarel: yes this one also quite old, but I found this morning :-)
14:23:22 <ykarel> k that's something different
14:23:31 <lajoskatona> ohh, I see, and this one is in networking-bagpipe not in neutron
14:23:48 <ykarel> thx ralonsoh lajoskatona for taking care of https://bugs.launchpad.net/networking-bagpipe/+bug/2109316
14:24:02 <ykarel> k let's move to next
14:24:06 <ykarel> ykarel to report issue for skip grenade job
14:24:26 <ykarel> didn't reported but not seen again, will report if i observe that again
14:24:33 <ykarel> #topic Stable branches
14:24:45 <ykarel> not much activity on stable, few issues in periodic jobs, will check in next topics
14:24:53 <ykarel> bcafarel, anything to add for stable?
14:25:45 <bcafarel> good summmary indeed, only that 2023.2 is now EOL (one less to track)
14:26:25 <ykarel> thx bcafarel for the update
14:26:53 <ykarel> not seeing 2023.2 under https://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic-weekly
14:27:46 <lajoskatona> 2023.2 was deleted I think
14:28:47 <lajoskatona> https://review.opendev.org/c/openstack/releases/+/948214
14:29:34 <ykarel> ok then we should be good
14:29:41 <bcafarel> and drop-grenade for 2024.1 https://review.opendev.org/c/openstack/neutron/+/948708 (just reviewed it)
14:29:44 <ykarel> thx lajoskatona for the link
14:30:30 <ykarel> k let's move to next
14:30:31 <ykarel> #topic Stadium projects
14:30:39 <ykarel> bagpipe test_tagged_attachment test failed  https://zuul.openstack.org/buildset/223a8e14c19e420898927560f664b872
14:30:50 <lajoskatona> all green, except bagpipe
14:30:58 <lajoskatona> yes exactly
14:31:46 <lajoskatona> and the uni ttest failure ralonsoh reported and we mentioned few minutes back
14:31:59 <ykarel> lajoskatona, can you check if this is not consistent failure?
14:32:09 <ykarel> or we need a bug report to track this
14:32:36 <lajoskatona> ykarel: sure, I'll check it
14:32:40 <ykarel> thx lajoskatona
14:32:48 <lajoskatona> and open bugreport if necessary
14:32:51 <ykarel> #action lajoskatona to check bagpipe tempest failure
14:33:01 <ykarel> #topic Rechecks
14:33:31 <ykarel> not much activity this week, and 0/6 bare rechecks
14:33:43 <ykarel> #topic fullstack/functional
14:33:52 <ykarel> random job timeouts https://zuul.openstack.org/builds?job_name=neutron-functional&project=openstack%2Fneutron&branch=master&result=TIMED_OUT&skip=0
14:34:48 <ykarel> i checked few of the failures, 1 was specific to node with 1 cpu and tempest concurrency 1
14:35:01 <ykarel> but some just seems stuck at tests run and jobs timeout
14:35:49 <ykarel> or there are more issues, so something to be checked
14:37:18 <ykarel> will check and create a bug for this, compiling available timeout failures
14:37:31 <ykarel> #action ykarel to check and report bug for functional timeouts
14:37:49 <ykarel> i recall we had some related bugs in past so also have to check those if these duplicate
14:37:53 <ykarel> or something new
14:38:31 <ykarel> #topic Periodic
14:38:37 <ykarel> centos 9-stream jobs broken https://bugs.launchpad.net/neutron/+bug/2109591 - being fixed with https://review.opendev.org/c/openstack/devstack/+/948511
14:38:45 <ykarel> oslo-master jobs broken https://bugs.launchpad.net/neutron/+bug/2109592 - fixed with https://review.opendev.org/c/openstack/neutron/+/948561
14:38:51 <ykarel> openstacksdk jobs broken - fixed with https://review.opendev.org/c/openstack/designate/+/902846 and https://review.opendev.org/c/openstack/octavia/+/902812
14:38:57 <ykarel> stable/2025.1 sdk broken, proposed backports for designate, octavia fixes above ^
14:39:08 <ykarel> stable centos 9 stream jobs broken as tempest runs with master constraints, can drop these jobs from stable or can try fixing using TEMPEST_VENV_UPPER_CONSTRAINTS=stable constraints ?
14:39:38 <ykarel> for ^ pushed https://review.opendev.org/c/openstack/neutron/+/948796
14:39:51 <ykarel> wdyt ?
14:41:37 <lajoskatona> +1 for stable constraints
14:42:56 <bcafarel> let's see results but looks good to me
14:43:02 <ykarel> results green
14:43:28 <ykarel> ok then please review, and will also backport to other stable branches
14:44:01 <opendevreview> yatin proposed openstack/neutron stable/2024.2: [Stable Only][CI][fips jobs] Use stable constraints for tempest  https://review.opendev.org/c/openstack/neutron/+/948805
14:44:20 <lajoskatona> ykarel: will review, thanks
14:44:29 <opendevreview> yatin proposed openstack/neutron stable/2024.1: [Stable Only][CI][fips jobs] Use stable constraints for tempest  https://review.opendev.org/c/openstack/neutron/+/948807
14:45:02 <bcafarel> oh true I needed refresh - added to my review list
14:45:08 <ykarel> for centos stream one required couple of devstack fixes https://review.opendev.org/c/openstack/devstack/+/948511
14:45:21 <ykarel> let's see if the temporary patch get's accepted
14:45:54 <ykarel> centos and rocky jobs green with these set of patches
14:46:00 <ykarel> https://zuul.opendev.org/t/openstack/buildset/8aadd8a0b73849d5a223ea27b1ba3921
14:46:51 <ykarel> #topic Grafana
14:47:01 <ykarel> let's have quick look at https://grafana.opendev.org/d/f913631585/neutron-failure-rate
14:49:51 <ykarel> looks normal, some functional failures in gate, job timeout one which we already discussed
14:50:06 <ykarel> check failures some known random issues, others likely specific to patches
14:50:13 <ykarel> #topic On Demand
14:50:18 <ykarel> anything else you would like to raise?
14:50:26 <ralonsoh> just a heads-up
14:50:35 <ralonsoh> pyroute 0.9.11 bump was reverted
14:50:42 <ralonsoh> because of an issue in os-vif
14:50:46 <ralonsoh> that's all
14:51:19 <ralonsoh> *0.9.1 not 0.9.11
14:51:26 <lajoskatona> thanks, so perhaps this patch is also useless:
14:51:29 <lajoskatona> https://review.opendev.org/c/openstack/networking-bagpipe/+/948789 ?
14:51:41 <ralonsoh> well, eventually we'll have it back again
14:51:51 <ralonsoh> so it could be useful to have code working with all versions
14:52:10 <ralonsoh> (if possible)
14:52:32 <lajoskatona> ack, thanks, I will check with older pyroute2 than
14:53:17 <ykarel> thx ralonsoh for the update
14:53:36 <ykarel> if nothing else let's close early and have everyone ~ 7 minutes back
14:53:50 <ykarel> #endmeeting