14:01:31 <ykarel> #startmeeting neutron_ci 14:01:31 <opendevmeet> Meeting started Mon Sep 23 14:01:31 2024 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:31 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:31 <opendevmeet> The meeting name has been set to 'neutron_ci' 14:01:36 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 14:01:46 <ralonsoh> hello 14:01:49 <ralonsoh> video? 14:02:33 <ralonsoh> no, not today, I think 14:02:36 <mlavalle> \o 14:02:41 <bcafarel> o/ 14:02:46 <slaweq> o/ 14:03:19 <ykarel> IRC today 14:04:32 <ykarel> hello everyone, let's start with topics 14:04:33 <ykarel> #topic Actions from previous meetings 14:04:41 <ykarel> ykarel to report bug for sdk failure and remove this job from periodic queue 14:05:06 <ykarel> reported https://bugs.launchpad.net/cinder/+bug/2081177 14:05:13 <ykarel> and proposed https://review.opendev.org/q/topic:%22bug/2081177%22 14:05:24 <ykarel> with this jobs green, so not removing these 14:05:50 <ykarel> #topic Stable branches 14:06:04 <ykarel> bcafarel, anything to add? 14:06:23 <ykarel> couple of patches merged in stable, no consistent failures seen in periodic stables 14:06:37 <bcafarel> yep, a bit behind on my backlog but branches look good (including new 2024.2) 14:06:43 <ykarel> apart from the wallaby one 14:06:50 <ykarel> k thx for the update 14:06:58 <ykarel> #topic Stadium projects 14:06:59 <bcafarel> thanks lajoskatona for filling the changes to switch to 2024.2 tempest jobs in stadium projects 14:07:20 <ykarel> vpnaas failure https://zuul.openstack.org/buildset/1810f2aa0ecb492db96fd6e3606c823a 14:07:28 <ykarel> Related to https://bugs.launchpad.net/neutron/+bug/2080072 ? 14:07:47 <ykarel> lajoskatona, anything to add? you checked the vpnaas failure? 14:08:38 <ykarel> seems lajoskatona is not around, let's move 14:08:47 <ykarel> #topic Rechecks 14:09:15 <ykarel> it was better this week, just 1/29 bare recheck, and rechecks were also less 14:09:28 <ykarel> #topic fullstack/functional 14:09:38 <ykarel> Too many open files https://launchpad.net/bugs/2080199 14:09:54 <ykarel> we still hitting this with the proposed ulimit bump patch 14:10:12 <ykarel> not merged, but in test patch we can see that 14:10:35 <ykarel> so need to narrow it down further 14:10:48 <lajoskatona> o/ 14:11:16 <lajoskatona> Sorry I was late (lost my time sensing waiting for a doctors appointment with my sons....) 14:11:58 <ykarel> lajoskatona, is this reproducible locally? or not tried that? just to see what we can do to further narrow it down 14:12:12 <ykarel> currently this is the main issue we have 14:12:29 <ralonsoh> do you remember since when we have this issue? I don't 14:12:50 <ralonsoh> could be a FT patch introduced during the last weeks 14:12:58 <lajoskatona> ykarel,ralonsoh: I have to check it 14:13:16 <ykarel> i think we started seeing in first week of september 14:13:28 <ykarel> and it's seen in all the branches 14:14:26 <lajoskatona> some patches are green today also for vpnaas so, not sure if it is ageneral issue or just a glitch 14:14:46 <lajoskatona> hat was fwaas as I remember 14:15:28 <ykarel> lajoskatona, the vpnaas one related to https://bugs.launchpad.net/neutron/+bug/2080072 ? 14:16:47 <lajoskatona> possible, but it was merged more than 1 week ago, let me check 14:18:53 <lajoskatona> I check it offline and check if #2080072 and the patch for that can be related 14:19:12 <ykarel> thx lajoskatona 14:19:22 <ykarel> #action lajoskatona to check vpnaas failure 14:19:48 <ykarel> test_metadata_proxy_rate_limiting_ipv6 14:19:59 <ykarel> https://402540ecdd63d1ecb8cc-46682989e655431f9d8a4a56f6d0c4c2.ssl.cf1.rackcdn.com/929793/3/gate/neutron-functional-with-uwsgi/5a7d403/testr_results.html 14:20:09 <ykarel> happened again after new logs added by slaweq 14:20:22 <ykarel> slaweq can you check it further with new logs available now? 14:20:39 <slaweq> yes, I will try to find some time and look into that this week 14:20:46 <ykarel> thx slaweq 14:21:04 <ykarel> #action slaweq to check https://bugs.launchpad.net/neutron/+bug/2079048/comments/7 14:21:16 <ykarel> #topic Tempest/Scenario 14:21:23 <ykarel> designate scenario random failures https://bugs.launchpad.net/neutron/+bug/2081252 14:21:33 <ykarel> jobs green now since revert merged https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/930022 14:21:44 <ykarel> #topic Periodic 14:22:02 <ykarel> postgres job fails with AttributeError: 'NoneType' object has no attribute 'id' 14:22:06 <ralonsoh> I'm checking this 14:22:08 <ykarel> https://e22ff80d6617e0aebdbb-fab3512704550cb902a3eea3d4491f2b.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-postgres-full/6b355b7/controller/logs/screen-q-svc.txt 14:22:11 <ralonsoh> I'll open a new bug 14:22:15 <ykarel> Another version of https://bugs.launchpad.net/neutron/+bug/2072567 14:22:21 <ykarel> thx ralonsoh 14:22:30 <ykarel> #action ralonsoh to check postgres issue 14:22:43 <ykarel> #topic Grafana 14:22:49 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate 14:23:03 <ykarel> let's also have quick look to the dashboard 14:25:50 <ykarel> there are spikes in functional failure, and that's known issue 14:27:17 <ykarel> other jobs also have spikes across jobs in check queue, haven't checked the failures myself so not sure patch specific 14:27:24 <ykarel> but will watch it out this week 14:27:26 <ykarel> anything to add? 14:27:46 <ralonsoh> no thanks 14:28:27 <bcafarel> week was probably also a bit quiet wit the branching and close release 14:28:30 <bcafarel> and nothing to add either 14:28:37 <ykarel> k thx 14:28:39 <ykarel> #topic On Demand 14:28:47 <lajoskatona> I have one topic 14:28:49 <ykarel> anything else you would like to raise? 14:28:53 <ykarel> yes please 14:29:20 <lajoskatona> we have periodic jobs running for unmaintained branches, I think we can remove those 14:29:38 <lajoskatona> for stable we can start them but not for unmaintained 14:30:21 <ralonsoh> did other projects remove them? 14:30:32 <ykarel> but those are the only way to detect failures(when they actually start) when there are no activity on those branches? am i missing something? 14:30:34 <lajoskatona> good question 14:30:51 <lajoskatona> I don't know how other projects handle periodic jobs on older branches 14:31:18 <lajoskatona> ykarel: yes , that's true 14:31:30 <ralonsoh> we can have periodic-weekly for these branches 14:31:49 <lajoskatona> Another way is to keep them and I create zuul dashboard to follow them 14:31:53 <ykarel> yes periodic-weekly should be enough, but we need to have something 14:32:21 <lajoskatona> ok, let's keep then periodic-weekly 14:32:27 <ralonsoh> +1 14:32:35 <ykarel> +1 14:32:54 <bcafarel> +1 (at least unless we get general guidelines for all projects, good to keep an eye on these branches) 14:33:16 <ykarel> any volunteer to propose this change? 14:33:27 <ralonsoh> I can 14:33:31 <ykarel> thx ralonsoh 14:33:35 <lajoskatona> I can check also, 14:33:44 <lajoskatona> thanks ralonsoh 14:34:06 <ykarel> #action ralonsoh to switch periodic jobs for unmaintained branches to periodic-weekly 14:34:13 <ykarel> thx lajoskatona for raising it 14:34:36 <ykarel> if nothing else, let's close early and have everyone 25 minutes back :) 14:35:09 <ykarel> thx everyone for joining 14:35:12 <ykarel> #endmeeting