15:03:19 <ykarel> #startmeeting neutron_ci
15:03:19 <opendevmeet> Meeting started Mon Nov 18 15:03:19 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:19 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:19 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:03:26 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:03:29 <mlavalle> ykarel: video or irc?
15:03:31 <slaweq> ykarel sure :)
15:03:35 <slaweq> o/
15:03:38 <mlavalle> \o
15:03:41 <ykarel> IRC
15:04:09 <haleyb> o/
15:04:33 <bcafarel> o/ but in another meeting
15:04:39 <ralonsoh> hi
15:05:17 <ykarel> hello everyone, let's start with the topics
15:05:23 <ykarel> #topic Actions from previous meetings
15:05:31 <ykarel> ralonsoh to check cleanup issue in test_dvr_ha_router_interface_mtu_update
15:05:55 <ralonsoh> I filled a LP bug and working on it
15:06:42 <ykarel> #link https://bugs.launchpad.net/neutron/+bug/2088154
15:06:46 <ykarel> ^ right?
15:06:49 <ralonsoh> yes, I was looking for that
15:06:52 <ralonsoh> exactly
15:06:55 <ykarel> thx ralonsoh
15:07:08 <ykarel> ralonsoh to open bug for sdk quota zone failure issue
15:07:22 <ralonsoh> working on it, I have a patch (that is not working(
15:07:33 <ralonsoh> https://review.opendev.org/c/openstack/openstacksdk/+/934762
15:07:41 <ralonsoh> I need to check how to retrieve the dns.quota
15:07:49 <ralonsoh> but low priority, is a random issue
15:07:54 <ykarel> thx ralonsoh
15:08:14 <ykarel> ykarel to check for failure in test_no_value_set_for_other_config_raises_exception
15:08:31 <ykarel> Based on the symptoms Looks similar to https://bugs.launchpad.net/neutron/+bug/1961740
15:08:46 <ykarel> as can see interface getting added/deleted/added quickly
15:08:54 <lajoskatona> o/
15:09:43 <ykarel> ykarel to log bug for ovn sync test failures in functional tests
15:09:50 <ykarel> Opened https://bugs.launchpad.net/neutron/+bug/2088423
15:09:59 <ykarel> slawek to check https://bugs.launchpad.net/neutron/+bug/2083609
15:13:11 <ykarel> for this we have some recent updates
15:13:12 <ykarel> https://review.opendev.org/c/openstack/neutron/+/934960
15:13:21 <slaweq> I was checking that recently
15:13:24 <ykarel> https://review.opendev.org/c/openstack/neutron/+/934677
15:14:45 <slaweq> I described in https://bugs.launchpad.net/neutron/+bug/2083609/comments/10 what I have found
15:15:13 <slaweq> I don't know anything else and I have no idea if we can somehow fix it really
15:16:19 <ykarel> me neither :(
15:16:46 <ykarel> thx slaweq for looking into it, if anyone has idea on this please have a look and update the bug
15:17:03 <ykarel> #topic Stable branches
15:17:22 <ykarel> stable branches are impacted by os-ken release https://bugs.launchpad.net/neutron/+bug/2088285
15:17:33 <ykarel> bcafarel, anything to add ?
15:18:04 <bcafarel> some back-and-forth in the reviews for 2023.1 unmaintained transition, but nothing bad
15:19:10 <lajoskatona> with os-ken what is the full solution? I saw patches from ralonsoh (thanks for it)
15:19:10 <bcafarel> Rodolfo backported FT fix https://review.opendev.org/q/I18cfb18abe481f47db870f210188e1a570844077 , should make stable CI even greener
15:19:18 <ykarel> yes 2023.1 CI also kind of broken with some projects yet to transition to unmaintained
15:19:39 <ralonsoh> we are revirting the patch in os-ken
15:19:55 <ralonsoh> if needed, we can propose a FRR container but not remoiving the quagga one
15:19:59 <lajoskatona> ahh, ok, but we dont have to backport the frr patches?
15:20:00 <ralonsoh> and make it configurable
15:20:04 <ralonsoh> no
15:20:20 <lajoskatona> ralonsoh: ack, let's have both than
15:20:46 <ykarel> lajoskatona, to unblock CI we going with pin https://review.opendev.org/c/openstack/requirements/+/935485
15:21:03 <ykarel> and then we can have os-ken revert + release + u-c update
15:21:32 <ykarel> or can also check the proposals from ralonsoh before the release
15:21:48 <ykarel> i.e to have something compatible with stable branches
15:21:53 <ralonsoh> and we have the broken releases blocked --> os-ken!=2.11.0,!=2.11.1
15:21:56 <ralonsoh> https://review.opendev.org/c/openstack/requirements/+/935485/2/global-requirements.txt#356
15:23:14 <opendevreview> Rodolfo Alonso proposed openstack/neutron master: Replace ``ReaderWriterLock`` with ``threading.RLock``  https://review.opendev.org/c/openstack/neutron/+/935524
15:23:34 <ykarel> k that's it for stable
15:23:36 <ykarel> #topic Stadium projects
15:23:52 <ykarel> weekly run red for all stadium projects
15:24:07 <ykarel> tempest jobs with os-ken release https://bugs.launchpad.net/neutron/+bug/2088285
15:24:11 <ralonsoh> pfffff
15:24:14 <ykarel> unit test jobs broken by https://review.opendev.org/c/openstack/neutron/+/934190
15:24:48 <ykarel> for tempest one we should have that soon unblocked
15:25:05 <ykarel> for unit test not checked if we have the patches open
15:25:14 <ykarel> lajoskatona, anything else to add for stadiums?
15:25:15 <lajoskatona> yeah that is due to the os-ken issue
15:25:30 <lajoskatona> nothing other than those ones
15:26:34 <ykarel> for unit test ones will be need another neutron release? or just update in stadium to point to new location will be enough?
15:27:46 <lajoskatona> good question
15:27:47 <ralonsoh> I think they have neutron as dependency and use master
15:27:47 <ykarel> from neutron.tests.unit.db import test_db_base_plugin_v2 -> from neutron.tests.common import test_db_base_plugin_v2
15:27:59 <ralonsoh> so it shouldn't be needed
15:28:13 <ykarel> ok then just the fixes on all stadium is needed
15:28:24 <ykarel> any volunteer to send those patches?
15:28:30 <ralonsoh> I can do it
15:28:33 <lajoskatona> I can
15:28:56 <ralonsoh> lajoskatona, ok, all yours, I'm still fixing other issues
15:28:58 <ralonsoh> thanks!!
15:29:04 <lajoskatona> :-)
15:29:17 <ykarel> #action lajoskatona to send stadium fixes for test_db_base_plugin_v2 renames
15:29:23 <ykarel> thx lajoskatona ralonsoh
15:29:40 <ykarel> #topic Rechecks
15:30:04 <ykarel> we still have some rechecks with known issues
15:30:33 <ykarel> good bare recheks down this week, 2/27. Let's keep avoiding bare rechecks
15:30:58 <ykarel> #topic fullstack/functional
15:31:10 <ykarel> test_update_minimum_bandwidth_queue
15:31:16 <ykarel> https://613acde820ad3e40c738-ebad28ee85ce63233d9ad88daf84e2e7.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/646ee8a/testr_results.html
15:31:27 <ykarel> found an old bug https://bugs.launchpad.net/neutron/+bug/2025740
15:31:48 <ykarel> where these tests were transitioned to serial but we still hitting it
15:31:53 <ykarel> not frequently though
15:32:47 <ykarel> test_non_existing
15:32:52 <ykarel> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_22b/934652/3/gate/neutron-functional-with-uwsgi/22bd251/testr_results.html
15:32:57 <ykarel> seen this also once
15:33:45 <ykarel> seems due to some slowness on pg to be created and ready when it was looked
15:33:48 <ralonsoh> test_non_existing --> I think we can add eventwaits
15:33:59 <ralonsoh> same as in other OVN tests
15:34:08 <ralonsoh> let me check this one
15:34:10 <ykarel> yes +1
15:34:17 <ralonsoh> I'll create a LP bug
15:34:24 <ykarel> ralonsoh, to open lp for test_non_existing
15:34:38 <ralonsoh> about the min-qos, because I wrote the code and the test, I'll check it too
15:34:38 <ykarel> #action ralonsoh to open lp for test_non_existing ft tests
15:35:17 <ykarel> ralonsoh, thx
15:35:30 <ykarel> #action ralonsoh to check for test_update_minimum_bandwidth_queue
15:35:42 <opendevreview> Merged openstack/os-ken master: Revert "Use FRR instead of Quagga when testing"  https://review.opendev.org/c/openstack/os-ken/+/935505
15:35:51 <ykarel> ralonsoh, both these are seen once recently, so can do check in lower priority
15:35:59 <ykarel> even in later weeks, so not to hurry
15:35:59 <ralonsoh> perfect
15:36:19 <ykarel> #topic Tempest/Scenario
15:36:42 <ykarel> #info os-ken issue https://bugs.launchpad.net/neutron/+bug/2088285 impacting most of the branches
15:36:49 <ykarel> neutron-ovn-tempest-full-multinode-ovs-master failed thrice during last week where many tests failed as neutron api fails to respond in 90 seconds, last three run passed though
15:36:56 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-full-multinode-ovs-master
15:37:13 <ykarel> just raising here to follow up next week, as latest runs are good
15:37:46 <ykarel> and don't know if there were anything that broke and fix it
15:37:59 <ykarel> so will watch for next runs
15:38:08 <ykarel> #topic Periodic
15:38:16 <ykarel> unmaintained/xena needs https://review.opendev.org/c/openstack/neutron/+/935497
15:38:45 <ykarel> please review ^
15:38:55 <ykarel> unmaintained/2021 needs https://review.opendev.org/c/openstack/ironic/+/934036
15:39:22 <ykarel> but now 2021 is blocked as not all projects transitioned to unmaintained, and default branch not switched to unmaintained
15:39:41 <ykarel> #topic Grafana
15:39:47 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:40:00 <ykarel> let's have a quick look here too
15:42:30 <ykarel> looks normal as we can see some spikes in gate for known issues
15:43:02 <slaweq> Neutron-tempest-plugin jobs in check queue looks like totally broken
15:43:14 <ykarel> yeap that's the os-ken issue
15:43:25 <slaweq> ahh, ok
15:43:47 <ykarel> #topic On Demand
15:43:55 <ykarel> anything else you would like to raise?
15:43:58 <ralonsoh> yes
15:44:07 <ralonsoh> https://review.opendev.org/c/openstack/releases/+/935527
15:44:13 <ralonsoh> that's all
15:44:46 <slaweq> +1 from me
15:44:50 <lajoskatona> +1 ,
15:45:03 <frickler> ralonsoh: is that fixing all the neutron issues? I can likely single-approve then
15:45:08 <ralonsoh> frickler, yes
15:45:42 <frickler> ok, will approve once CI is done, thx
15:45:51 <ykarel> thx everyone for joining
15:45:58 <ykarel> #endmeeting