15:02:37 <ykarel> #startmeeting neutron_ci 15:02:37 <opendevmeet> Meeting started Mon Jan 13 15:02:37 2025 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:37 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:02:37 <opendevmeet> The meeting name has been set to 'neutron_ci' 15:02:47 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 15:02:49 <ralonsoh> hello 15:03:17 <lajoskatona> Hi 15:03:18 <slaweq> o/ 15:04:31 <ykarel> ok let's start with the topics 15:04:42 <ykarel> #topic Actions from previous meetings 15:04:53 <ykarel> there were none for this week 15:04:58 <ykarel> #topic Stable branches 15:05:22 <ykarel> bernard not around today, but all good on stable 15:05:28 <ykarel> #topic Stadium projects 15:05:33 <ykarel> neutron-dynamic-routing still down with wsgi switch https://bugs.launchpad.net/neutron/+bug/2085421 15:05:47 <lajoskatona> neutron-dynamic-routng is red, yes with the wsgi 15:06:21 <lajoskatona> but am I right that the neutron patch will solve it? 15:06:34 <ykarel> which neutron patch? 15:06:46 <ykarel> have we already tested that job with the patch you referring? 15:06:47 <opendevreview> Merged openstack/ovn-bgp-agent master: Fix text from CrLrpChassisChangeWithdrawEvent docstring https://review.opendev.org/c/openstack/ovn-bgp-agent/+/936427 15:08:02 <lajoskatona> let me check 15:08:28 <lajoskatona> https://review.opendev.org/c/openstack/neutron/+/936829 15:08:53 <lajoskatona> no, a one for n-t-p 15:09:19 <lajoskatona> https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/938466 15:10:25 <lajoskatona> from last wee kmy memories tell me that we said that this one to limit the wsgi workers is a workaround but we can go with it temporarily, but after that other things just washed this topic from my mind 15:12:22 <ykarel> yeap that was a workaround if we want to consider 15:13:22 <ykarel> ralonsoh, do you already have some patch to fix ^ issue? 15:13:30 <ralonsoh> no, sorry 15:13:38 <ralonsoh> I didn't investigate the ndr issue 15:13:50 <ralonsoh> but I can start checking it tomorrow 15:13:56 <ralonsoh> you can assign it to me 15:14:54 <ykarel> ralonsoh, ack behavior wise we seeing those lock wait timeout errors here 15:14:57 <lajoskatona> yatin's patch with the wsgi worker number showed that the issue is related to the eventlet/wsgi topic 15:15:14 <lajoskatona> ykarel:exactly 15:15:36 <ykarel> #action ralonsoh to check https://bugs.launchpad.net/neutron/+bug/2085421 15:15:49 <lajoskatona> but I think it is fine to merge the workaround from ykarel, to not add to the piles of ralonsoh 15:16:14 <ralonsoh> hmmmm let's wait a bit 15:16:27 <ralonsoh> because reducing to 1 the concurrency will hide some problems 15:16:28 <ykarel> yeap can wait 15:16:39 <ralonsoh> so I would prefer not to merge this patch yet 15:17:18 <lajoskatona> ack 15:17:42 <ykarel> let's move 15:17:43 <ykarel> #topic Rechecks 15:18:04 <ykarel> so we still have good number of rechecks for these known issues with ovn 15:18:16 <ykarel> bare recheck wise we had 5/34 15:18:41 <ykarel> Let's check failures 15:18:41 <ykarel> #topic Unit tests 15:18:48 <ykarel> bandit issue https://bugs.launchpad.net/neutron/+bug/2093849 fixed with new 1.8.2 release 15:19:03 <ykarel> this already got fixed, thanks haleyb for checking 15:19:12 <ykarel> #topic fullstack/functional 15:19:13 <haleyb> :) 15:19:29 <ykarel> we still seeing some oom-kill on raxflex node 15:19:34 <ykarel> - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_740/periodic/opendev.org/openstack/neutron/master/neutron-functional/740c0fc/controller/logs/syslog.txt 15:19:34 <ykarel> - https://bcd85c4e99ffc7863f97-b20dd3fcccdb5478c1889ed1c3ad542c.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/294db5b/controller/logs/syslog.txt 15:19:34 <ykarel> - https://47b74d2027b9966dbc6f-6fe938074b11dd62ac4ccd867c30657d.ssl.cf2.rackcdn.com/938853/2/gate/neutron-functional/32345b2/controller/logs/syslog.txt 15:19:39 <ykarel> - https://bugs.launchpad.net/neutron/+bug/2091855 still happening less frequent though 15:19:56 <ykarel> i will check that again if we can further improve it 15:20:06 <ykarel> #action ykarel to check lp#2091855 15:20:12 <ykarel> #topic Tempest/Scenario 15:20:28 <ykarel> Here we mainly seeing ovn wsgi switch related issues 15:20:47 <ykarel> most of the failures that i saw were where lsp up event was missed 15:21:02 <ykarel> ralonsoh, do we have a bug for ^ specifically 15:21:03 <ralonsoh> yeah, this is killing me 15:21:14 <ralonsoh> one related 15:21:19 <ralonsoh> https://bugs.launchpad.net/neutron/+bug/2093327 15:21:34 <ralonsoh> but no, I need to open the bug for the missing LSP up event 15:21:39 <ykarel> hmm but that's different 15:21:43 <ralonsoh> yes 15:21:52 <ykarel> ok yes please let's have dedicated one for that issue 15:22:00 <ralonsoh> about the LSP up event, we see that in the OVN logs 15:22:05 <ralonsoh> but never received in the Neutron API 15:22:18 <opendevreview> Vasyl Saienko proposed openstack/neutron master: Drop dead code related to keystone-v3 in network segment ranges https://review.opendev.org/c/openstack/neutron/+/939107 15:22:22 <ykarel> #action ralonsoh to open bug for missing lsp up event with wsgi switch 15:23:50 <ykarel> linking some of these failures here 15:23:53 <ykarel> https://a6fbb86efdf5860e1b02-ce677b1d4d068789ef14672b1b5d2453.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-with-neutron-lib-master/fc27ce5/testr_results.html 15:23:59 <ykarel> https://650bc3c55ab286ed9493-bc3c897401c8263f785d7df7482dd6e6.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-plugin-ovs-lts/cde340b/testr_results.html 15:24:06 <ykarel> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_18d/periodic/opendev.org/openstack/neutron/master/neutron-tempest-plugin-ovn-enforce-scope-old-defaults/18dbe6f/testr_results.html 15:24:13 <ykarel> https://2780d629a7a6d047ca0f-9244b66e59ac9fd5e3dbb52d3e6f0017.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-tempest-plugin-ovn-enforce-scope-old-defaults/091ece8/testr_results.html 15:24:20 <ykarel> https://e76b8a042504872b0b7d-f18cd0734c7002742f0baf633d0573f2.ssl.cf1.rackcdn.com/938800/3/gate/neutron-tempest-plugin-ovn/548dc8b/testr_results.html 15:24:55 <ralonsoh> yes, unfortunately we have several examples everyday 15:25:23 <lajoskatona> :-) 15:25:36 <ykarel> also nother tempest failure unrelated to wsgi switch 15:25:47 <ykarel> test_metadata_ipv6_only_network failed with malformed check 15:25:52 <ykarel> https://33d4ca1eb24f74c1535d-46fef9f34fd106c3774e2eeeca119539.ssl.cf5.rackcdn.com/938670/1/gate/neutron-tempest-plugin-openvswitch/878b8bb/testr_results.html 15:26:18 <ykarel> somehow the console output was malformed like cloud-init[515]: latestQueried th]: e metadata service over IPv6ci-info 15:26:24 <ykarel> so the expected string not found 15:26:30 <ykarel> i will report the issue for it 15:26:39 <ykarel> mtomaska1, fyi ^ 15:26:51 <opendevreview> Vasyl Saienko proposed openstack/neutron-lib master: Remove tenant_id from segment rages responces https://review.opendev.org/c/openstack/neutron-lib/+/939108 15:26:54 <ykarel> #action ykarel to report issue for ipv6 metadata test 15:27:15 <ykarel> #topic Periodic 15:27:19 <mtomaska1> ACK ykarel 15:27:30 <ykarel> here we mostly hit ovn wsgi related issues 15:27:47 <ykarel> pyroute2 functional issues are now resolved 15:27:56 <ralonsoh> ykarel, but is this one related to wsgi? 15:28:00 <ralonsoh> the metadata bug 15:28:08 <ykarel> ralonsoh, no 15:28:13 <ykarel> it's unrelated 15:28:26 <ralonsoh> ahhhh sorry (I need to read better...) 15:29:36 <ykarel> for pyroute2 Petr will likely be joining tomorrows neutron meeting 15:30:06 <ykarel> to tell more about upcoming pyroute2 changes and removing some legacy bits and adoption in neutron for it 15:30:35 <ykarel> https://github.com/svinota/pyroute2/issues/1225#issuecomment-2582071411 where he pointed it out 15:30:37 <lajoskatona> +1, good news, always happy to hear/read from him 15:30:57 <ralonsoh> and about using asyncio 15:31:03 <ykarel> just to update i will not be around tomorrow 15:31:59 <ykarel> ok that's it on failures 15:32:00 <ykarel> #topic Grafana 15:32:11 <ykarel> lets quicly look on https://grafana.opendev.org/d/f913631585/neutron-failure-rate 15:35:01 <ykarel> looks normal, check failures either known issues or related to patches 15:35:11 <ykarel> anything to add? 15:35:35 <ralonsoh> related to wsgi: https://review.opendev.org/c/openstack/neutron/+/938319 15:35:37 <ralonsoh> please review it 15:35:40 <ralonsoh> that's all 15:37:03 <ykarel> #topic On Demand 15:37:13 <ykarel> anything else you would like to raise? 15:38:25 <opendevreview> Slawek Kaplonski proposed openstack/neutron master: Make API policies for tags to be working with resource attributes https://review.opendev.org/c/openstack/neutron/+/938135 15:39:59 <lajoskatona> Nothing from me 15:40:54 <opendevreview> Vasyl Saienko proposed openstack/neutron master: Install DVR src to MAC flow for AAPs https://review.opendev.org/c/openstack/neutron/+/938657 15:41:15 <opendevreview> Vasyl Saienko proposed openstack/neutron master: Use is_cidr_host utils to detect if AAP ip is host in l3_dvr_db https://review.opendev.org/c/openstack/neutron/+/939075 15:47:19 <haleyb> ralonsoh: i am looking at that now, lgtm 15:47:26 <ralonsoh> thanks 15:50:41 <ralonsoh> ykarel, hi, are you still there? 15:51:09 <ralonsoh> maybe he had a network disconnection 15:51:18 <ralonsoh> I think we can close the meeting for today, right? 15:51:23 <ykarel> #endmeeting