14:09:28 <ykarel> #startmeeting neutron_ci 14:09:28 <opendevmeet> Meeting started Mon May 5 14:09:28 2025 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:09:28 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:09:28 <opendevmeet> The meeting name has been set to 'neutron_ci' 14:09:33 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 14:09:37 <mlavalle> \o 14:09:37 <ykarel> IRC meeting today 14:09:40 <mlavalle> lol 14:09:40 <ralonsoh> hello 14:09:51 <mlavalle> irc? 14:10:18 <ykarel> yes irc today 14:10:47 <ykarel> let's give few minutes for others if someone else around to join 14:11:08 <bcafarel> o/ 14:13:33 <lajoskatona> o/ 14:14:35 <haleyb> o/ 14:15:07 <ykarel> thx everyone for joining, let's start with topic 14:15:17 <ykarel> #topic Actions from previous meetings 14:15:27 <ykarel> mlavalle to continue with https://bugs.launchpad.net/neutron/+bug/2104019 14:16:04 <mlavalle> it seems the situation has improved since the change I proposed to the test: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack&skip=0 14:16:54 <ykarel> but wasn't that test fixing part of failures? 14:17:11 <ykarel> the failures like AssertionError: 119 not less than 90 : Sent 218 packets, received 99 packets, lost 119 packets 14:17:11 <mlavalle> there are still two cases that I'm investigating. One is the case when after the failover, pings and tcp cinnectivity succeed 14:17:17 <mlavalle> but udp still fail 14:17:39 <mlavalle> for that I've been testing with https://review.opendev.org/c/openstack/neutron/+/948280 14:18:00 <mlavalle> more difficuklt to reproduce, so I'll continue rechecking 14:18:22 <ykarel> k thx may be you can add more jobs in same patch replicating neutron-fullstack job 14:18:30 <ykarel> that would be better then multiple rechecks 14:18:50 <mlavalle> ok 14:19:19 <ykarel> like done for another job in https://review.opendev.org/c/openstack/neutron/+/942615/5/zuul.d/project.yaml 14:19:32 <mlavalle> will try that 14:19:58 <ykarel> thx mlavalle for checking this 14:20:08 <ykarel> ykarel to check bagpipe failure is reproducible or not 14:20:16 <mlavalle> ;-) 14:20:24 <ykarel> i did couple of rechecks last weeks and that didn't reproduced 14:20:32 <ykarel> ykarel to report issue for the unit test failure 14:20:44 <ykarel> reported https://bugs.launchpad.net/neutron/+bug/2109326 14:20:46 <lajoskatona> didnt it was reported by ralonsoh? 14:20:57 <ykarel> and also pushed the fix for that 14:21:07 <lajoskatona> ahho, ok another one 14:21:34 <ykarel> lajoskatona, which issue? 14:22:14 <opendevreview> Merged openstack/neutron-fwaas master: Add neutron-tempest-plugin-fwaas-ovn job to the check and gate queues https://review.opendev.org/c/openstack/neutron-fwaas/+/948062 14:22:23 <ykarel> this is from 2 weeks back, as last week we didn't had any meeting 14:22:33 <lajoskatona> https://bugs.launchpad.net/networking-bagpipe/+bug/2109316 14:23:09 <lajoskatona> ykarel: yes this one also quite old, but I found this morning :-) 14:23:22 <ykarel> k that's something different 14:23:31 <lajoskatona> ohh, I see, and this one is in networking-bagpipe not in neutron 14:23:48 <ykarel> thx ralonsoh lajoskatona for taking care of https://bugs.launchpad.net/networking-bagpipe/+bug/2109316 14:24:02 <ykarel> k let's move to next 14:24:06 <ykarel> ykarel to report issue for skip grenade job 14:24:26 <ykarel> didn't reported but not seen again, will report if i observe that again 14:24:33 <ykarel> #topic Stable branches 14:24:45 <ykarel> not much activity on stable, few issues in periodic jobs, will check in next topics 14:24:53 <ykarel> bcafarel, anything to add for stable? 14:25:45 <bcafarel> good summmary indeed, only that 2023.2 is now EOL (one less to track) 14:26:25 <ykarel> thx bcafarel for the update 14:26:53 <ykarel> not seeing 2023.2 under https://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic-weekly 14:27:46 <lajoskatona> 2023.2 was deleted I think 14:28:47 <lajoskatona> https://review.opendev.org/c/openstack/releases/+/948214 14:29:34 <ykarel> ok then we should be good 14:29:41 <bcafarel> and drop-grenade for 2024.1 https://review.opendev.org/c/openstack/neutron/+/948708 (just reviewed it) 14:29:44 <ykarel> thx lajoskatona for the link 14:30:30 <ykarel> k let's move to next 14:30:31 <ykarel> #topic Stadium projects 14:30:39 <ykarel> bagpipe test_tagged_attachment test failed https://zuul.openstack.org/buildset/223a8e14c19e420898927560f664b872 14:30:50 <lajoskatona> all green, except bagpipe 14:30:58 <lajoskatona> yes exactly 14:31:46 <lajoskatona> and the uni ttest failure ralonsoh reported and we mentioned few minutes back 14:31:59 <ykarel> lajoskatona, can you check if this is not consistent failure? 14:32:09 <ykarel> or we need a bug report to track this 14:32:36 <lajoskatona> ykarel: sure, I'll check it 14:32:40 <ykarel> thx lajoskatona 14:32:48 <lajoskatona> and open bugreport if necessary 14:32:51 <ykarel> #action lajoskatona to check bagpipe tempest failure 14:33:01 <ykarel> #topic Rechecks 14:33:31 <ykarel> not much activity this week, and 0/6 bare rechecks 14:33:43 <ykarel> #topic fullstack/functional 14:33:52 <ykarel> random job timeouts https://zuul.openstack.org/builds?job_name=neutron-functional&project=openstack%2Fneutron&branch=master&result=TIMED_OUT&skip=0 14:34:48 <ykarel> i checked few of the failures, 1 was specific to node with 1 cpu and tempest concurrency 1 14:35:01 <ykarel> but some just seems stuck at tests run and jobs timeout 14:35:49 <ykarel> or there are more issues, so something to be checked 14:37:18 <ykarel> will check and create a bug for this, compiling available timeout failures 14:37:31 <ykarel> #action ykarel to check and report bug for functional timeouts 14:37:49 <ykarel> i recall we had some related bugs in past so also have to check those if these duplicate 14:37:53 <ykarel> or something new 14:38:31 <ykarel> #topic Periodic 14:38:37 <ykarel> centos 9-stream jobs broken https://bugs.launchpad.net/neutron/+bug/2109591 - being fixed with https://review.opendev.org/c/openstack/devstack/+/948511 14:38:45 <ykarel> oslo-master jobs broken https://bugs.launchpad.net/neutron/+bug/2109592 - fixed with https://review.opendev.org/c/openstack/neutron/+/948561 14:38:51 <ykarel> openstacksdk jobs broken - fixed with https://review.opendev.org/c/openstack/designate/+/902846 and https://review.opendev.org/c/openstack/octavia/+/902812 14:38:57 <ykarel> stable/2025.1 sdk broken, proposed backports for designate, octavia fixes above ^ 14:39:08 <ykarel> stable centos 9 stream jobs broken as tempest runs with master constraints, can drop these jobs from stable or can try fixing using TEMPEST_VENV_UPPER_CONSTRAINTS=stable constraints ? 14:39:38 <ykarel> for ^ pushed https://review.opendev.org/c/openstack/neutron/+/948796 14:39:51 <ykarel> wdyt ? 14:41:37 <lajoskatona> +1 for stable constraints 14:42:56 <bcafarel> let's see results but looks good to me 14:43:02 <ykarel> results green 14:43:28 <ykarel> ok then please review, and will also backport to other stable branches 14:44:01 <opendevreview> yatin proposed openstack/neutron stable/2024.2: [Stable Only][CI][fips jobs] Use stable constraints for tempest https://review.opendev.org/c/openstack/neutron/+/948805 14:44:20 <lajoskatona> ykarel: will review, thanks 14:44:29 <opendevreview> yatin proposed openstack/neutron stable/2024.1: [Stable Only][CI][fips jobs] Use stable constraints for tempest https://review.opendev.org/c/openstack/neutron/+/948807 14:45:02 <bcafarel> oh true I needed refresh - added to my review list 14:45:08 <ykarel> for centos stream one required couple of devstack fixes https://review.opendev.org/c/openstack/devstack/+/948511 14:45:21 <ykarel> let's see if the temporary patch get's accepted 14:45:54 <ykarel> centos and rocky jobs green with these set of patches 14:46:00 <ykarel> https://zuul.opendev.org/t/openstack/buildset/8aadd8a0b73849d5a223ea27b1ba3921 14:46:51 <ykarel> #topic Grafana 14:47:01 <ykarel> let's have quick look at https://grafana.opendev.org/d/f913631585/neutron-failure-rate 14:49:51 <ykarel> looks normal, some functional failures in gate, job timeout one which we already discussed 14:50:06 <ykarel> check failures some known random issues, others likely specific to patches 14:50:13 <ykarel> #topic On Demand 14:50:18 <ykarel> anything else you would like to raise? 14:50:26 <ralonsoh> just a heads-up 14:50:35 <ralonsoh> pyroute 0.9.11 bump was reverted 14:50:42 <ralonsoh> because of an issue in os-vif 14:50:46 <ralonsoh> that's all 14:51:19 <ralonsoh> *0.9.1 not 0.9.11 14:51:26 <lajoskatona> thanks, so perhaps this patch is also useless: 14:51:29 <lajoskatona> https://review.opendev.org/c/openstack/networking-bagpipe/+/948789 ? 14:51:41 <ralonsoh> well, eventually we'll have it back again 14:51:51 <ralonsoh> so it could be useful to have code working with all versions 14:52:10 <ralonsoh> (if possible) 14:52:32 <lajoskatona> ack, thanks, I will check with older pyroute2 than 14:53:17 <ykarel> thx ralonsoh for the update 14:53:36 <ykarel> if nothing else let's close early and have everyone ~ 7 minutes back 14:53:50 <ykarel> #endmeeting