15:00:55 <ykarel> #startmeeting neutron_ci
15:00:55 <opendevmeet> Meeting started Mon Jan 27 15:00:55 2025 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:55 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:55 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:01:06 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:01:08 <mlavalle> \o
15:01:09 <slaweq> hi
15:01:14 <bcafarel> o/
15:01:34 <ralonsoh> hello
15:02:13 <lajoskatona> o/
15:02:48 <ykarel> Hi everyone let's start with the topics
15:02:49 <ykarel> #topic Actions from previous meetings
15:02:57 <ykarel> ralonsoh to check https://bugs.launchpad.net/neutron/+bug/2085421
15:03:21 <ralonsoh> this is related to the hash ring issue, one sec
15:03:34 <ralonsoh> https://bugs.launchpad.net/neutron/+bug/2095590
15:03:44 <ralonsoh> so I'm going to close it as duplicated
15:04:01 <ykarel> ok good, you confirmed if this also fixing dynamic routing jobs?
15:04:05 <mlavalle> one less bug, nice!
15:04:08 <ralonsoh> not yet
15:04:24 <ralonsoh> hmmm
15:04:33 <ralonsoh> maybe we should keep it until we confirm that
15:05:03 <ykarel> yes let's do that
15:05:09 <ykarel> thx ralonsoh for taking care of these
15:05:21 <ykarel> mlavalle to improve ipv6 test in https://bugs.launchpad.net/neutron/+bug/2095091
15:05:54 <mlavalle> I did look at it. Left a comment here: https://bugs.launchpad.net/neutron/+bug/2095091/comments/3
15:06:41 <mlavalle> since we have seen this only once, for now I propose doing nothing, unless the team feels we should address it sooner
15:08:01 <ykarel> hmm we can watch for this, not seen it again yet
15:08:17 <mlavalle> I can keep an eye on it
15:08:29 <ykarel> mlavalle, have you checked what slaweq suggested if we really need the string checks?
15:08:47 <mlavalle> I didn't look at that
15:09:14 <ykarel> ack
15:09:18 <mlavalle> looking at the strings is straightforward and if we need to improve the test it has an easy improvement
15:09:23 <ykarel> ykarel to report the issue with neutron-functional-fips job
15:09:35 <ykarel> reported https://bugs.launchpad.net/neutron/+bug/2095598
15:09:55 <ykarel> and was pointed out by ralonsoh that it might be fixed with a in review patch
15:10:10 <ykarel> https://review.opendev.org/c/openstack/neutron/+/939627
15:10:23 <ralonsoh> yes, we saw that reviewing a related patch
15:10:24 <ykarel> but test patch confirms it's not helping, so have to be relooked
15:10:42 <ykarel> asked sahid if he can check that
15:10:44 <ralonsoh> what is the testing aptch?
15:11:12 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/940128
15:11:15 <ykarel> just running that test multiple times in https://review.opendev.org/c/openstack/neutron/+/940128 with depends on the possible fix
15:11:28 <ykarel> yes that one
15:11:56 <ralonsoh> let me check that, last week I spent sometime on this method
15:12:01 <ralonsoh> is not very well implemented
15:12:11 <ralonsoh> I'll assign it to me
15:12:14 <ykarel> ralonsoh, ack thx
15:12:23 <ykarel> #action ralonsoh to check https://review.opendev.org/c/openstack/neutron/+/940128
15:12:38 <ykarel> ralonsoh, just that we hitting it in fips jobs which runs on centos 9-stream
15:12:51 <ykarel> i added what all changed since last success run there
15:13:16 <ykarel> so good to check that as well what triggered this issue
15:13:25 <ykarel> ok moving to next
15:13:27 <ykarel> ykarel to send fix for job names in grafana dashboard
15:13:34 <ykarel> pushed https://review.opendev.org/c/openstack/project-config/+/940065
15:13:37 <ykarel> already merged
15:13:47 <ykarel> #topic Stable branches
15:13:59 <ykarel> bcafarel, all yours ^
15:14:16 <bcafarel> quiet week, not too many backports, and CI looked stable there
15:14:22 <bcafarel> I did not have time to check periodic though
15:14:35 <ykarel> I checked periodic and it was good
15:14:39 <ykarel> thx for the updates
15:14:42 <bcafarel> nice :)
15:14:52 <ykarel> #topic Stadium projects
15:14:59 <ykarel> dynamic routing still broken https://zuul.openstack.org/build/cca145c899d84fd389aa911e3ddf60d1
15:15:02 <lajoskatona> vpnaas is failing
15:15:04 <ykarel> - https://bugs.launchpad.net/neutron/+bug/2085421
15:15:10 <ykarel> vpnaas functional https://zuul.openstack.org/build/aaac7e51e607437b9095a085ed2d2a7e
15:15:15 <ykarel> - broken with subprocess utility method removal https://review.opendev.org/c/openstack/neutron/+/939097
15:15:38 <lajoskatona> yes exactly
15:15:42 <lajoskatona> I proposed patch for it:
15:15:49 <lajoskatona> https://review.opendev.org/q/topic:%22bug/2087942%22+project:openstack/neutron-vpnaas
15:16:51 <ykarel> thx lajoskatona
15:16:58 <lajoskatona> one is to use eventlet.greendthread,subprocess the other to use native subprocess.popen, I suppose the last one is better in the long run, let's see if it runs into issue on CI first
15:17:16 <ykarel> dynamic routing ralonsoh will be confirming if that also fixed with patch, then we should be good for stadiums
15:17:28 <ralonsoh> ok
15:17:39 <ralonsoh> lajoskatona, yes, much better the second
15:17:39 <ykarel> #topic Rechecks
15:18:24 <ykarel> We still have couple of rechecks for merging patches due to known intermittent issue, hopefully will be better once all current fixes get's in
15:18:49 <ykarel> Bare rechecks wise quite good, no bare recheck out of total 38 rechecks
15:19:01 <ykarel> #topic fullstack/functional
15:19:30 <ykarel> - reproduced after logs added as part of https://bugs.launchpad.net/neutron/+bug/2079048
15:20:01 <ykarel> slaweq, if you could check if debug logs helped here ^?
15:20:17 <ykarel> since you were looking that already :)
15:22:08 <ykarel> can follow up on this later
15:22:51 <ykarel> #topic Tempest/Scenario
15:22:56 <slaweq> sure
15:23:10 <ykarel> #undo
15:23:10 <opendevmeet> Removing item from minutes: #topic Tempest/Scenario
15:23:38 <ykarel> #action slaweq to check latest failure with logs added for https://bugs.launchpad.net/neutron/+bug/2079048
15:23:43 <ykarel> thx slaweq
15:23:44 <ykarel> #topic Tempest/Scenario
15:23:52 <ykarel> - we still have wsgi switch related issues
15:24:04 <ykarel> hopefully will all be good once fixes from ralonsoh get's in
15:24:10 <ykarel> thx again ralonsoh for working on these
15:24:17 <ykarel> #topic grenade
15:24:32 <ykarel> seen similar failure twice neutron-ovs-grenade-dvr-multinode, ping failure for FIP
15:24:38 <ykarel> https://b3463c59c8f87072d3e9-87141d8cdbf10530595552debffaf82b.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovs-grenade-dvr-multinode/d964961/controller/logs/grenade.sh_log.txt
15:24:38 <ykarel> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f66/periodic/opendev.org/openstack/neutron/master/neutron-ovs-grenade-dvr-multinode/f66a79c/controller/logs/grenade.sh_log.txt
15:24:57 <ykarel> I recall seeing something similar in past too occassionally
15:25:27 <ykarel> Will try to find and link that else open new one
15:25:36 <ykarel> #action ykarel to check grenade failure
15:25:49 <ykarel> #topic Grafana
15:25:53 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:25:59 <ykarel> let's have a quick look here too
15:27:50 <ykarel> ok in gate we can see some failures for fullstack
15:28:37 <ykarel> those were related to l3 ha test, which i recall we have a known issue, will just link that
15:29:01 <ykarel> and one was related to rabbitmq setup which we seen very rarely in past
15:29:49 <ykarel> apart from that looks normal, can see spikes in check queues but those looks patches specific or known internmittent issues
15:29:53 <ykarel> anything else to add?
15:30:01 <mlavalle> not from me
15:30:08 <lajoskatona> nothing from me
15:30:21 <ralonsoh> no thanks
15:30:46 <slaweq> ykarel just one thing about this metadata failure
15:31:11 <slaweq> it failed on the patch for stable/2024.2 and my additional log was never backported there :/
15:31:27 <slaweq> I just checked that there is no info I wanted to check there
15:32:00 <opendevreview> Slawek Kaplonski proposed openstack/neutron stable/2024.2: [Functional tests] Add logging router interfaces in metadata IPv6 tests  https://review.opendev.org/c/openstack/neutron/+/940250
15:32:11 <ykarel> slaweq, ok thanks somehow i saw it was backported :)
15:32:17 <ykarel> thx for relooking
15:32:24 <opendevreview> Slawek Kaplonski proposed openstack/neutron stable/2024.1: [Functional tests] Add logging router interfaces in metadata IPv6 tests  https://review.opendev.org/c/openstack/neutron/+/940251
15:32:32 <ykarel> #topic On Demand
15:32:39 <ykarel> anything else you would like to add?
15:32:42 <mlavalle> nothing from me
15:32:48 <opendevreview> Slawek Kaplonski proposed openstack/neutron stable/2023.2: [Functional tests] Add logging router interfaces in metadata IPv6 tests  https://review.opendev.org/c/openstack/neutron/+/940252
15:32:52 <slaweq> now backports are proposed ^^
15:33:01 <ykarel> ++
15:33:02 <slaweq> sorry that I forgo about them earlier
15:33:23 <bcafarel> added to my pile
15:33:50 <mlavalle> \o
15:34:16 <ykarel> if nothing else, let's close early and have everyone 26 minutes back
15:34:26 <ykarel> thx everyone for joining
15:34:27 <ykarel> #endmeeting