14:01:31 <ykarel> #startmeeting neutron_ci
14:01:31 <opendevmeet> Meeting started Mon Sep 23 14:01:31 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:31 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:31 <opendevmeet> The meeting name has been set to 'neutron_ci'
14:01:36 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
14:01:46 <ralonsoh> hello
14:01:49 <ralonsoh> video?
14:02:33 <ralonsoh> no, not today, I think
14:02:36 <mlavalle> \o
14:02:41 <bcafarel> o/
14:02:46 <slaweq> o/
14:03:19 <ykarel> IRC today
14:04:32 <ykarel> hello everyone, let's start with topics
14:04:33 <ykarel> #topic Actions from previous meetings
14:04:41 <ykarel> ykarel to report bug for sdk failure and remove this job from periodic queue
14:05:06 <ykarel> reported https://bugs.launchpad.net/cinder/+bug/2081177
14:05:13 <ykarel> and proposed https://review.opendev.org/q/topic:%22bug/2081177%22
14:05:24 <ykarel> with this jobs green, so not removing these
14:05:50 <ykarel> #topic Stable branches
14:06:04 <ykarel> bcafarel, anything to add?
14:06:23 <ykarel> couple of patches merged in stable, no consistent failures seen in periodic stables
14:06:37 <bcafarel> yep, a bit behind on my backlog but branches look good (including new 2024.2)
14:06:43 <ykarel> apart from the wallaby one
14:06:50 <ykarel> k thx for the update
14:06:58 <ykarel> #topic Stadium projects
14:06:59 <bcafarel> thanks lajoskatona for filling the changes to switch to 2024.2 tempest jobs in stadium projects
14:07:20 <ykarel> vpnaas failure https://zuul.openstack.org/buildset/1810f2aa0ecb492db96fd6e3606c823a
14:07:28 <ykarel> Related to https://bugs.launchpad.net/neutron/+bug/2080072 ?
14:07:47 <ykarel> lajoskatona, anything to add? you checked the vpnaas failure?
14:08:38 <ykarel> seems lajoskatona is not around, let's move
14:08:47 <ykarel> #topic Rechecks
14:09:15 <ykarel> it was better this week, just 1/29 bare recheck, and rechecks were also less
14:09:28 <ykarel> #topic fullstack/functional
14:09:38 <ykarel> Too many open files https://launchpad.net/bugs/2080199
14:09:54 <ykarel> we still hitting this with the proposed ulimit bump patch
14:10:12 <ykarel> not merged, but in test patch we can see that
14:10:35 <ykarel> so need to narrow it down further
14:10:48 <lajoskatona> o/
14:11:16 <lajoskatona> Sorry I was late (lost my time sensing waiting for a doctors appointment with my sons....)
14:11:58 <ykarel> lajoskatona, is this reproducible locally? or not tried that? just to see what we can do to further narrow it down
14:12:12 <ykarel> currently this is the main issue we have
14:12:29 <ralonsoh> do you remember since when we have this issue? I don't
14:12:50 <ralonsoh> could be a FT patch introduced during the last weeks
14:12:58 <lajoskatona> ykarel,ralonsoh: I have to check it
14:13:16 <ykarel> i think we started seeing in first week of september
14:13:28 <ykarel> and it's seen in all the branches
14:14:26 <lajoskatona> some patches are green today also for vpnaas so, not sure if it is ageneral issue or just a glitch
14:14:46 <lajoskatona> hat was fwaas as I remember
14:15:28 <ykarel> lajoskatona, the vpnaas one related to https://bugs.launchpad.net/neutron/+bug/2080072 ?
14:16:47 <lajoskatona> possible, but it was merged more than 1 week ago, let me check
14:18:53 <lajoskatona> I check it offline and check if #2080072 and the patch for that can be related
14:19:12 <ykarel> thx lajoskatona
14:19:22 <ykarel> #action lajoskatona to check vpnaas failure
14:19:48 <ykarel> test_metadata_proxy_rate_limiting_ipv6
14:19:59 <ykarel> https://402540ecdd63d1ecb8cc-46682989e655431f9d8a4a56f6d0c4c2.ssl.cf1.rackcdn.com/929793/3/gate/neutron-functional-with-uwsgi/5a7d403/testr_results.html
14:20:09 <ykarel> happened again after new logs added by slaweq
14:20:22 <ykarel> slaweq can you check it further with new logs available now?
14:20:39 <slaweq> yes, I will try to find some time and look into that this week
14:20:46 <ykarel> thx slaweq
14:21:04 <ykarel> #action slaweq to check https://bugs.launchpad.net/neutron/+bug/2079048/comments/7
14:21:16 <ykarel> #topic Tempest/Scenario
14:21:23 <ykarel> designate scenario random failures https://bugs.launchpad.net/neutron/+bug/2081252
14:21:33 <ykarel> jobs green now since revert merged https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/930022
14:21:44 <ykarel> #topic Periodic
14:22:02 <ykarel> postgres job fails with  AttributeError: 'NoneType' object has no attribute 'id'
14:22:06 <ralonsoh> I'm checking this
14:22:08 <ykarel> https://e22ff80d6617e0aebdbb-fab3512704550cb902a3eea3d4491f2b.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-postgres-full/6b355b7/controller/logs/screen-q-svc.txt
14:22:11 <ralonsoh> I'll open a new bug
14:22:15 <ykarel> Another version of https://bugs.launchpad.net/neutron/+bug/2072567
14:22:21 <ykarel> thx ralonsoh
14:22:30 <ykarel> #action ralonsoh to check postgres issue
14:22:43 <ykarel> #topic Grafana
14:22:49 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
14:23:03 <ykarel> let's also have quick look to the dashboard
14:25:50 <ykarel> there are spikes in functional failure, and that's known issue
14:27:17 <ykarel> other jobs also have spikes across jobs in check queue, haven't checked the failures myself so not sure patch specific
14:27:24 <ykarel> but will watch it out this week
14:27:26 <ykarel> anything to add?
14:27:46 <ralonsoh> no thanks
14:28:27 <bcafarel> week was probably also a bit quiet wit the branching and close release
14:28:30 <bcafarel> and nothing to add either
14:28:37 <ykarel> k thx
14:28:39 <ykarel> #topic On Demand
14:28:47 <lajoskatona> I have one topic
14:28:49 <ykarel> anything else you would like to raise?
14:28:53 <ykarel> yes please
14:29:20 <lajoskatona> we have periodic jobs running for unmaintained branches, I think we can remove those
14:29:38 <lajoskatona> for stable we can start them but not for unmaintained
14:30:21 <ralonsoh> did other projects remove them?
14:30:32 <ykarel> but those are the only way to detect failures(when they actually start) when there are no activity on those branches? am i missing something?
14:30:34 <lajoskatona> good question
14:30:51 <lajoskatona> I don't know how other projects handle periodic jobs on older branches
14:31:18 <lajoskatona> ykarel: yes , that's true
14:31:30 <ralonsoh> we can have periodic-weekly for these branches
14:31:49 <lajoskatona> Another way is to keep them and I create zuul dashboard to follow them
14:31:53 <ykarel> yes periodic-weekly should be enough, but we need to have something
14:32:21 <lajoskatona> ok, let's keep then periodic-weekly
14:32:27 <ralonsoh> +1
14:32:35 <ykarel> +1
14:32:54 <bcafarel> +1 (at least unless we get general guidelines for all projects, good to keep an eye on these branches)
14:33:16 <ykarel> any volunteer to propose this change?
14:33:27 <ralonsoh> I can
14:33:31 <ykarel> thx ralonsoh
14:33:35 <lajoskatona> I can check also,
14:33:44 <lajoskatona> thanks ralonsoh
14:34:06 <ykarel> #action ralonsoh to switch periodic jobs for unmaintained branches to periodic-weekly
14:34:13 <ykarel> thx lajoskatona for raising it
14:34:36 <ykarel> if nothing else, let's close early and have everyone 25 minutes back :)
14:35:09 <ykarel> thx everyone for joining
14:35:12 <ykarel> #endmeeting