14:09:28 #startmeeting neutron_ci 14:09:28 Meeting started Mon May 5 14:09:28 2025 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:09:28 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:09:28 The meeting name has been set to 'neutron_ci' 14:09:33 Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 14:09:37 \o 14:09:37 IRC meeting today 14:09:40 lol 14:09:40 hello 14:09:51 irc? 14:10:18 yes irc today 14:10:47 let's give few minutes for others if someone else around to join 14:11:08 o/ 14:13:33 o/ 14:14:35 o/ 14:15:07 thx everyone for joining, let's start with topic 14:15:17 #topic Actions from previous meetings 14:15:27 mlavalle to continue with https://bugs.launchpad.net/neutron/+bug/2104019 14:16:04 it seems the situation has improved since the change I proposed to the test: https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack&skip=0 14:16:54 but wasn't that test fixing part of failures? 14:17:11 the failures like AssertionError: 119 not less than 90 : Sent 218 packets, received 99 packets, lost 119 packets 14:17:11 there are still two cases that I'm investigating. One is the case when after the failover, pings and tcp cinnectivity succeed 14:17:17 but udp still fail 14:17:39 for that I've been testing with https://review.opendev.org/c/openstack/neutron/+/948280 14:18:00 more difficuklt to reproduce, so I'll continue rechecking 14:18:22 k thx may be you can add more jobs in same patch replicating neutron-fullstack job 14:18:30 that would be better then multiple rechecks 14:18:50 ok 14:19:19 like done for another job in https://review.opendev.org/c/openstack/neutron/+/942615/5/zuul.d/project.yaml 14:19:32 will try that 14:19:58 thx mlavalle for checking this 14:20:08 ykarel to check bagpipe failure is reproducible or not 14:20:16 ;-) 14:20:24 i did couple of rechecks last weeks and that didn't reproduced 14:20:32 ykarel to report issue for the unit test failure 14:20:44 reported https://bugs.launchpad.net/neutron/+bug/2109326 14:20:46 didnt it was reported by ralonsoh? 14:20:57 and also pushed the fix for that 14:21:07 ahho, ok another one 14:21:34 lajoskatona, which issue? 14:22:14 Merged openstack/neutron-fwaas master: Add neutron-tempest-plugin-fwaas-ovn job to the check and gate queues https://review.opendev.org/c/openstack/neutron-fwaas/+/948062 14:22:23 this is from 2 weeks back, as last week we didn't had any meeting 14:22:33 https://bugs.launchpad.net/networking-bagpipe/+bug/2109316 14:23:09 ykarel: yes this one also quite old, but I found this morning :-) 14:23:22 k that's something different 14:23:31 ohh, I see, and this one is in networking-bagpipe not in neutron 14:23:48 thx ralonsoh lajoskatona for taking care of https://bugs.launchpad.net/networking-bagpipe/+bug/2109316 14:24:02 k let's move to next 14:24:06 ykarel to report issue for skip grenade job 14:24:26 didn't reported but not seen again, will report if i observe that again 14:24:33 #topic Stable branches 14:24:45 not much activity on stable, few issues in periodic jobs, will check in next topics 14:24:53 bcafarel, anything to add for stable? 14:25:45 good summmary indeed, only that 2023.2 is now EOL (one less to track) 14:26:25 thx bcafarel for the update 14:26:53 not seeing 2023.2 under https://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic-weekly 14:27:46 2023.2 was deleted I think 14:28:47 https://review.opendev.org/c/openstack/releases/+/948214 14:29:34 ok then we should be good 14:29:41 and drop-grenade for 2024.1 https://review.opendev.org/c/openstack/neutron/+/948708 (just reviewed it) 14:29:44 thx lajoskatona for the link 14:30:30 k let's move to next 14:30:31 #topic Stadium projects 14:30:39 bagpipe test_tagged_attachment test failed https://zuul.openstack.org/buildset/223a8e14c19e420898927560f664b872 14:30:50 all green, except bagpipe 14:30:58 yes exactly 14:31:46 and the uni ttest failure ralonsoh reported and we mentioned few minutes back 14:31:59 lajoskatona, can you check if this is not consistent failure? 14:32:09 or we need a bug report to track this 14:32:36 ykarel: sure, I'll check it 14:32:40 thx lajoskatona 14:32:48 and open bugreport if necessary 14:32:51 #action lajoskatona to check bagpipe tempest failure 14:33:01 #topic Rechecks 14:33:31 not much activity this week, and 0/6 bare rechecks 14:33:43 #topic fullstack/functional 14:33:52 random job timeouts https://zuul.openstack.org/builds?job_name=neutron-functional&project=openstack%2Fneutron&branch=master&result=TIMED_OUT&skip=0 14:34:48 i checked few of the failures, 1 was specific to node with 1 cpu and tempest concurrency 1 14:35:01 but some just seems stuck at tests run and jobs timeout 14:35:49 or there are more issues, so something to be checked 14:37:18 will check and create a bug for this, compiling available timeout failures 14:37:31 #action ykarel to check and report bug for functional timeouts 14:37:49 i recall we had some related bugs in past so also have to check those if these duplicate 14:37:53 or something new 14:38:31 #topic Periodic 14:38:37 centos 9-stream jobs broken https://bugs.launchpad.net/neutron/+bug/2109591 - being fixed with https://review.opendev.org/c/openstack/devstack/+/948511 14:38:45 oslo-master jobs broken https://bugs.launchpad.net/neutron/+bug/2109592 - fixed with https://review.opendev.org/c/openstack/neutron/+/948561 14:38:51 openstacksdk jobs broken - fixed with https://review.opendev.org/c/openstack/designate/+/902846 and https://review.opendev.org/c/openstack/octavia/+/902812 14:38:57 stable/2025.1 sdk broken, proposed backports for designate, octavia fixes above ^ 14:39:08 stable centos 9 stream jobs broken as tempest runs with master constraints, can drop these jobs from stable or can try fixing using TEMPEST_VENV_UPPER_CONSTRAINTS=stable constraints ? 14:39:38 for ^ pushed https://review.opendev.org/c/openstack/neutron/+/948796 14:39:51 wdyt ? 14:41:37 +1 for stable constraints 14:42:56 let's see results but looks good to me 14:43:02 results green 14:43:28 ok then please review, and will also backport to other stable branches 14:44:01 yatin proposed openstack/neutron stable/2024.2: [Stable Only][CI][fips jobs] Use stable constraints for tempest https://review.opendev.org/c/openstack/neutron/+/948805 14:44:20 ykarel: will review, thanks 14:44:29 yatin proposed openstack/neutron stable/2024.1: [Stable Only][CI][fips jobs] Use stable constraints for tempest https://review.opendev.org/c/openstack/neutron/+/948807 14:45:02 oh true I needed refresh - added to my review list 14:45:08 for centos stream one required couple of devstack fixes https://review.opendev.org/c/openstack/devstack/+/948511 14:45:21 let's see if the temporary patch get's accepted 14:45:54 centos and rocky jobs green with these set of patches 14:46:00 https://zuul.opendev.org/t/openstack/buildset/8aadd8a0b73849d5a223ea27b1ba3921 14:46:51 #topic Grafana 14:47:01 let's have quick look at https://grafana.opendev.org/d/f913631585/neutron-failure-rate 14:49:51 looks normal, some functional failures in gate, job timeout one which we already discussed 14:50:06 check failures some known random issues, others likely specific to patches 14:50:13 #topic On Demand 14:50:18 anything else you would like to raise? 14:50:26 just a heads-up 14:50:35 pyroute 0.9.11 bump was reverted 14:50:42 because of an issue in os-vif 14:50:46 that's all 14:51:19 *0.9.1 not 0.9.11 14:51:26 thanks, so perhaps this patch is also useless: 14:51:29 https://review.opendev.org/c/openstack/networking-bagpipe/+/948789 ? 14:51:41 well, eventually we'll have it back again 14:51:51 so it could be useful to have code working with all versions 14:52:10 (if possible) 14:52:32 ack, thanks, I will check with older pyroute2 than 14:53:17 thx ralonsoh for the update 14:53:36 if nothing else let's close early and have everyone ~ 7 minutes back 14:53:50 #endmeeting