15:02:37 <ykarel> #startmeeting neutron_ci
15:02:37 <opendevmeet> Meeting started Mon Jan 13 15:02:37 2025 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:37 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:37 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:02:47 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:02:49 <ralonsoh> hello
15:03:17 <lajoskatona> Hi
15:03:18 <slaweq> o/
15:04:31 <ykarel> ok let's start with the topics
15:04:42 <ykarel> #topic Actions from previous meetings
15:04:53 <ykarel> there were none for this week
15:04:58 <ykarel> #topic Stable branches
15:05:22 <ykarel> bernard not around today, but all good on stable
15:05:28 <ykarel> #topic Stadium projects
15:05:33 <ykarel> neutron-dynamic-routing still down with wsgi switch https://bugs.launchpad.net/neutron/+bug/2085421
15:05:47 <lajoskatona> neutron-dynamic-routng is red, yes with the wsgi
15:06:21 <lajoskatona> but am I right that  the neutron patch will solve it?
15:06:34 <ykarel> which neutron patch?
15:06:46 <ykarel> have we already tested that job with the patch you referring?
15:06:47 <opendevreview> Merged openstack/ovn-bgp-agent master: Fix text from CrLrpChassisChangeWithdrawEvent docstring  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936427
15:08:02 <lajoskatona> let me check
15:08:28 <lajoskatona> https://review.opendev.org/c/openstack/neutron/+/936829
15:08:53 <lajoskatona> no, a one for n-t-p
15:09:19 <lajoskatona> https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/938466
15:10:25 <lajoskatona> from last wee kmy memories tell me that we said that this one to limit the wsgi workers is a workaround but we can go with it temporarily, but after that other things just washed this topic from my mind
15:12:22 <ykarel> yeap that was a workaround if we want to consider
15:13:22 <ykarel> ralonsoh, do you already have some patch to fix ^ issue?
15:13:30 <ralonsoh> no, sorry
15:13:38 <ralonsoh> I didn't investigate the ndr issue
15:13:50 <ralonsoh> but I can start checking it tomorrow
15:13:56 <ralonsoh> you can assign it to me
15:14:54 <ykarel> ralonsoh, ack behavior wise we seeing those lock wait timeout errors here
15:14:57 <lajoskatona> yatin's patch with the wsgi worker number showed that the issue is related to the eventlet/wsgi topic
15:15:14 <lajoskatona> ykarel:exactly
15:15:36 <ykarel> #action ralonsoh to check https://bugs.launchpad.net/neutron/+bug/2085421
15:15:49 <lajoskatona> but I think it is fine to merge the workaround from ykarel, to not add to the piles of ralonsoh
15:16:14 <ralonsoh> hmmmm let's wait a bit
15:16:27 <ralonsoh> because reducing to 1 the concurrency will hide some problems
15:16:28 <ykarel> yeap can wait
15:16:39 <ralonsoh> so I would prefer not to merge this patch yet
15:17:18 <lajoskatona> ack
15:17:42 <ykarel> let's move
15:17:43 <ykarel> #topic Rechecks
15:18:04 <ykarel> so we still have good number of rechecks for these known issues with ovn
15:18:16 <ykarel> bare recheck wise we had 5/34
15:18:41 <ykarel> Let's check failures
15:18:41 <ykarel> #topic Unit tests
15:18:48 <ykarel> bandit issue https://bugs.launchpad.net/neutron/+bug/2093849 fixed with new 1.8.2 release
15:19:03 <ykarel> this already got fixed, thanks haleyb for checking
15:19:12 <ykarel> #topic fullstack/functional
15:19:13 <haleyb> :)
15:19:29 <ykarel> we still seeing some oom-kill on raxflex node
15:19:34 <ykarel> - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_740/periodic/opendev.org/openstack/neutron/master/neutron-functional/740c0fc/controller/logs/syslog.txt
15:19:34 <ykarel> - https://bcd85c4e99ffc7863f97-b20dd3fcccdb5478c1889ed1c3ad542c.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/294db5b/controller/logs/syslog.txt
15:19:34 <ykarel> - https://47b74d2027b9966dbc6f-6fe938074b11dd62ac4ccd867c30657d.ssl.cf2.rackcdn.com/938853/2/gate/neutron-functional/32345b2/controller/logs/syslog.txt
15:19:39 <ykarel> - https://bugs.launchpad.net/neutron/+bug/2091855 still happening less frequent though
15:19:56 <ykarel> i will check that again if we can further improve it
15:20:06 <ykarel> #action ykarel to check lp#2091855
15:20:12 <ykarel> #topic Tempest/Scenario
15:20:28 <ykarel> Here we mainly seeing ovn wsgi switch related issues
15:20:47 <ykarel> most of the failures that i saw were where lsp up event was missed
15:21:02 <ykarel> ralonsoh, do we have a bug for ^ specifically
15:21:03 <ralonsoh> yeah, this is killing me
15:21:14 <ralonsoh> one related
15:21:19 <ralonsoh> https://bugs.launchpad.net/neutron/+bug/2093327
15:21:34 <ralonsoh> but no, I need to open the bug for the missing LSP up event
15:21:39 <ykarel> hmm but that's different
15:21:43 <ralonsoh> yes
15:21:52 <ykarel> ok yes please let's have dedicated one for that issue
15:22:00 <ralonsoh> about the LSP up event, we see that in the OVN logs
15:22:05 <ralonsoh> but never received in the Neutron API
15:22:18 <opendevreview> Vasyl Saienko proposed openstack/neutron master: Drop dead code related to keystone-v3 in network segment ranges  https://review.opendev.org/c/openstack/neutron/+/939107
15:22:22 <ykarel> #action ralonsoh to open bug for missing lsp up event with wsgi switch
15:23:50 <ykarel> linking some of these failures here
15:23:53 <ykarel> https://a6fbb86efdf5860e1b02-ce677b1d4d068789ef14672b1b5d2453.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-with-neutron-lib-master/fc27ce5/testr_results.html
15:23:59 <ykarel> https://650bc3c55ab286ed9493-bc3c897401c8263f785d7df7482dd6e6.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-plugin-ovs-lts/cde340b/testr_results.html
15:24:06 <ykarel> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_18d/periodic/opendev.org/openstack/neutron/master/neutron-tempest-plugin-ovn-enforce-scope-old-defaults/18dbe6f/testr_results.html
15:24:13 <ykarel> https://2780d629a7a6d047ca0f-9244b66e59ac9fd5e3dbb52d3e6f0017.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-tempest-plugin-ovn-enforce-scope-old-defaults/091ece8/testr_results.html
15:24:20 <ykarel> https://e76b8a042504872b0b7d-f18cd0734c7002742f0baf633d0573f2.ssl.cf1.rackcdn.com/938800/3/gate/neutron-tempest-plugin-ovn/548dc8b/testr_results.html
15:24:55 <ralonsoh> yes, unfortunately we have several examples everyday
15:25:23 <lajoskatona> :-)
15:25:36 <ykarel> also nother tempest failure unrelated to wsgi switch
15:25:47 <ykarel> test_metadata_ipv6_only_network failed with malformed check
15:25:52 <ykarel> https://33d4ca1eb24f74c1535d-46fef9f34fd106c3774e2eeeca119539.ssl.cf5.rackcdn.com/938670/1/gate/neutron-tempest-plugin-openvswitch/878b8bb/testr_results.html
15:26:18 <ykarel> somehow the console output was malformed like cloud-init[515]: latestQueried th]: e metadata service over IPv6ci-info
15:26:24 <ykarel> so the expected string not found
15:26:30 <ykarel> i will report the issue for it
15:26:39 <ykarel> mtomaska1, fyi ^
15:26:51 <opendevreview> Vasyl Saienko proposed openstack/neutron-lib master: Remove tenant_id from segment rages responces  https://review.opendev.org/c/openstack/neutron-lib/+/939108
15:26:54 <ykarel> #action ykarel to report issue for ipv6 metadata test
15:27:15 <ykarel> #topic Periodic
15:27:19 <mtomaska1> ACK ykarel
15:27:30 <ykarel> here we mostly hit ovn wsgi related issues
15:27:47 <ykarel> pyroute2 functional issues are now resolved
15:27:56 <ralonsoh> ykarel, but is this one related to wsgi?
15:28:00 <ralonsoh> the metadata bug
15:28:08 <ykarel> ralonsoh, no
15:28:13 <ykarel> it's unrelated
15:28:26 <ralonsoh> ahhhh sorry (I need to read better...)
15:29:36 <ykarel> for pyroute2 Petr will likely be joining tomorrows neutron meeting
15:30:06 <ykarel> to tell more about upcoming pyroute2 changes and removing some legacy bits and adoption in neutron for it
15:30:35 <ykarel> https://github.com/svinota/pyroute2/issues/1225#issuecomment-2582071411 where he pointed it out
15:30:37 <lajoskatona> +1, good news, always happy to hear/read from him
15:30:57 <ralonsoh> and about using asyncio
15:31:03 <ykarel> just to update i will not be around tomorrow
15:31:59 <ykarel> ok that's it on failures
15:32:00 <ykarel> #topic Grafana
15:32:11 <ykarel> lets quicly look on https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:35:01 <ykarel> looks normal, check failures either known issues or related to patches
15:35:11 <ykarel> anything to add?
15:35:35 <ralonsoh> related to wsgi: https://review.opendev.org/c/openstack/neutron/+/938319
15:35:37 <ralonsoh> please review it
15:35:40 <ralonsoh> that's all
15:37:03 <ykarel> #topic On Demand
15:37:13 <ykarel> anything else you would like to raise?
15:38:25 <opendevreview> Slawek Kaplonski proposed openstack/neutron master: Make API policies for tags to be working with resource attributes  https://review.opendev.org/c/openstack/neutron/+/938135
15:39:59 <lajoskatona> Nothing from me
15:40:54 <opendevreview> Vasyl Saienko proposed openstack/neutron master: Install DVR src to MAC flow for AAPs  https://review.opendev.org/c/openstack/neutron/+/938657
15:41:15 <opendevreview> Vasyl Saienko proposed openstack/neutron master: Use is_cidr_host utils to detect if AAP ip is host in l3_dvr_db  https://review.opendev.org/c/openstack/neutron/+/939075
15:47:19 <haleyb> ralonsoh: i am looking at that now, lgtm
15:47:26 <ralonsoh> thanks
15:50:41 <ralonsoh> ykarel, hi, are you still there?
15:51:09 <ralonsoh> maybe he had a network disconnection
15:51:18 <ralonsoh> I think we can close the meeting for today, right?
15:51:23 <ykarel> #endmeeting