15:00:40 <ykarel> #startmeeting neutron_ci 15:00:40 <opendevmeet> Meeting started Mon Mar 10 15:00:40 2025 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:40 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:40 <opendevmeet> The meeting name has been set to 'neutron_ci' 15:00:46 <ykarel> Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira 15:00:47 <mlavalle> \o 15:00:57 <ralonsoh> hello, video or IRC today? 15:01:05 <ralonsoh> I always forget 15:01:32 <ykarel> IRC 15:01:36 <ralonsoh> cool 15:02:03 <mlavalle> ++ 15:02:09 <slaweq> hi 15:02:29 <lajoskatona> Hi 15:02:38 <opendevreview> Merged openstack/neutron stable/2024.2: [ovn][trivial] Add 'empty_string_filtering' extension to OVN https://review.opendev.org/c/openstack/neutron/+/943417 15:02:42 <opendevreview> Merged openstack/neutron stable/2024.1: [ovn][trivial] Add 'empty_string_filtering' extension to OVN https://review.opendev.org/c/openstack/neutron/+/943418 15:02:46 <opendevreview> Merged openstack/neutron stable/2023.2: [ovn][trivial] Add 'empty_string_filtering' extension to OVN https://review.opendev.org/c/openstack/neutron/+/943419 15:03:26 <bcafarel> late o/ 15:03:42 <ykarel> k let's start with topics, others can join in meanwhile 15:03:47 <ykarel> #topic Actions from previous meetings 15:03:57 <ykarel> ralonsoh to check and open bug for privsep related failure in functional job 15:04:28 <ralonsoh> Sorry, I didn't open it yet 15:04:36 <ralonsoh> I was investigating it 15:05:05 <ykarel> ack thx for checking, will readd 15:05:10 <ykarel> #action ralonsoh to check and open bug for privsep related failure in functional job 15:05:21 <ykarel> ralonsoh to check and open bug for fullstack failures failing at wait_until_true from quite sometime, original suspect was https://review.opendev.org/c/openstack/neutron/+/937843/ 15:05:28 <ralonsoh> yes 15:05:30 <ralonsoh> #link https://bugs.launchpad.net/neutron/+bug/2101839 15:05:37 <ralonsoh> And I proposed https://review.opendev.org/c/openstack/neutron/+/943942 15:05:43 <ralonsoh> temporarily 15:06:55 <ykarel> thx ralonsoh 15:07:07 <ykarel> ohkk so eventlet reintroduced in the test 15:07:24 <ralonsoh> yes only for fullstack tempest 15:07:31 <ralonsoh> until we remove it from this FW 15:08:07 <ykarel> ok 15:08:16 <lajoskatona> I hope we will have time and compony support in the comming cycles for this eventlet work otherwise Neutron will be like the gordian knot 15:08:44 <ykarel> ++ 15:08:47 <ralonsoh> I finshed last week the metadata and I'm working now in the L3 agent 15:08:56 <ralonsoh> but yes, more hands are welcome 15:09:13 <ykarel> moving to next 15:09:16 <ykarel> ykarel to check and open bug for ovs tempest failures 15:09:33 <ykarel> opened https://bugs.launchpad.net/neutron/+bug/2101840 15:09:37 <ykarel> #link https://bugs.launchpad.net/neutron/+bug/2101840 15:10:00 <ykarel> it's kind of critical now as the ovs jobs failing quite frequently now 15:10:11 <ykarel> seems related to the eventlet cleanups we have ? 15:10:51 <ykarel> like https://review.opendev.org/c/openstack/neutron/+/942393 15:10:51 <lajoskatona> thanks fro checking, I just read your comment in lp, I check it 15:10:56 <ralonsoh> but the dhcp agent moved out of eventlet some months ago 15:11:28 <ykarel> from logs i see dhcp agents have high number of events to process like 200+ 15:11:28 <ralonsoh> right... it received some adaptations 15:11:40 <ralonsoh> but the complete removal was finished 2 weeks ago 15:13:07 <lajoskatona> I check if the eventlet removal is related or not, and check this bug 15:13:24 <ykarel> thx lajoskatona 15:13:39 <ykarel> #action lajoskatona to check https://bugs.launchpad.net/neutron/+bug/2101840 15:14:07 <ykarel> ykarel to open bug for ping failure post upgrade in grenade 15:14:17 <ykarel> #link https://bugs.launchpad.net/neutron/+bug/2101166 15:14:30 <ykarel> this also seen in ovs jobs, dvr multinode 15:14:54 <ykarel> not much frequent though 15:15:24 <ykarel> ykarel to open bug for tempest failure in test_established_tcp_session_after_re_attachinging_sg 15:15:31 <ykarel> #link https://bugs.launchpad.net/neutron/+bug/2101165 15:15:51 <ykarel> this also seen mostly in ovs jobs 15:16:02 <ykarel> and i recall seeing similar traces from long 15:16:10 <ykarel> didn't find an old bug so reported this to track 15:16:28 <ykarel> that's it for action items from previous week 15:16:29 <ykarel> #topic Stable branches 15:16:45 <ykarel> bcafarel, any update ^ 15:17:16 <bcafarel> a few backports last week (and merging), nothing looking bad in the CI runs there 15:17:18 <ykarel> couple of patches merged in stable 15:17:35 <ykarel> yeap merges in both stable and unmaintained went fine 15:17:50 <ykarel> in periodic i saw some failures but none of them consistent this week 15:18:13 <ykarel> last was openstacksdk but that got resolved already 15:18:23 <ykarel> thx bcafarel for the updates 15:18:31 <ykarel> #topic Stadium projects 15:18:35 <ykarel> all green in periodic 15:18:39 <ykarel> lajoskatona, anything to add 15:18:53 <lajoskatona> nothing from my side for stadiums 15:19:17 <ykarel> k thx 15:19:26 <ykarel> #topic Rechecks 15:19:52 <ykarel> we still have couple of rechecks to get patches merged due to known issues 15:20:04 <ykarel> bare recheck wise we had 4/39 15:20:15 <ykarel> out of those 3 in same patch, test patch for pyroute2 master 15:20:27 <ykarel> so good, let's keep avoiding bare rechecks 15:20:28 <ralonsoh> yes, sorry for that 15:20:39 <ralonsoh> Peter is testing pyroute2 with a branch created only for neutron 15:20:39 <ykarel> np those were from Petr 15:21:02 <ykarel> ++ thx for working on that 15:21:12 <ykarel> Now let's check failures 15:21:22 <ykarel> test_ovn_nb_sync_repair_delete_ovn_nb_db in stable 15:21:29 <ykarel> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_811/periodic/opendev.org/openstack/neutron/stable/2024.2/neutron-functional-with-uwsgi-fips/811c11f/testr_results.html 15:21:36 <ykarel> pushed backports https://review.opendev.org/q/Ibaeaa36dbbb9def10a163b4ee071cb432db5a383 15:21:52 <ykarel> all merged to stable 15:22:02 <ykarel> some randomly failures in functional 15:22:09 <ykarel> - https://27d33baddc3887bf0580-122b756d505eee79094990db0a134990.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-oslo-master/77dd0c1/testr_results.html 15:22:10 <ykarel> - https://6047f309b8432f0272cd-3f475577843c6519bda0c8c12c28da61.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/1d78049/testr_results.html 15:22:40 <ykarel> from failures could see traces related to sqlite but as seeing mainly in master and logger driver involved 15:22:40 <ralonsoh> ^^ I'll check them 15:22:51 <ykarel> proposed https://review.opendev.org/c/openstack/neutron/+/943937 15:23:11 <ykarel> we can check again once this merges 15:23:19 <ralonsoh> cool 15:23:35 <lajoskatona> +1 15:23:40 <ykarel> test_migration failed twice in last periodic runs in 2024.1 15:23:47 <ykarel> - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d55/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-fullstack-with-uwsgi-fips/d55b927/testr_results.html 15:23:48 <ykarel> - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b18/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-fullstack-with-uwsgi-fips/b18873b/testr_results.html 15:24:37 <ykarel> have to confirm if next run fails similarly or it was intermittent failure 15:25:25 <ykarel> i can check that tomorrow and report an issue for this in any case if consistent or not 15:25:38 <ralonsoh> thanks 15:25:51 <ykarel> #action ykarel to check next runs for fullstack fips and report issue with fullstack 15:26:19 <ykarel> test_configurations_are_synced_towards_placement(NIC Switch agent) 15:26:27 <ykarel> https://70ea00e7f434a8c178c6-dd4a29c0fe6beace2356eed9dd1f7a86.ssl.cf2.rackcdn.com/928586/18/gate/neutron-fullstack/cd00a4b/testr_results.html 15:26:41 <ykarel> lajoskatona, may be you recall ^ 15:26:55 <ykarel> i recall you looking into this in past 15:27:05 <ykarel> from perspective of timeout increase is needed or not 15:27:11 <lajoskatona> yes, occasionally this failed 15:27:24 <lajoskatona> I check this one also 15:28:00 <ykarel> thx lajoskatona 15:28:28 <ykarel> #action lajoskatona to check fullstack test failure test_configurations_are_synced_towards_placement(NIC Switch agent) 15:28:37 <ykarel> #topic Tempest/Scenario 15:28:50 <ykarel> ubuntu jammy job fails randomly https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-ovn-ubuntu-jammy&project=openstack/neutron 15:29:19 <ykarel> i noticed couple of failures where tests fail during ssh for missing keys 15:29:35 <ykarel> i recall some recent discussion but couldn't find it 15:29:40 <ralonsoh> I talked about this to haleyb|out last week 15:29:55 <ralonsoh> I just proposed to mark it as non-voting (the easy path...) 15:29:57 <ykarel> ohkk do we already have bug for this? 15:30:03 <ralonsoh> nope sorry 15:30:12 <ralonsoh> there was no pattern 15:30:22 <ykarel> ohkk let's start with bug and see if we need non voting or any fix for it 15:30:31 <ralonsoh> perfect 15:30:32 <ralonsoh> I'll open it 15:30:48 <ykarel> i recall one issue with ovn metadata in jammy, not sure if that's related but can check 15:31:56 <ykarel> https://bugzilla.redhat.com/show_bug.cgi?id=2172036 and https://bugs.launchpad.net/neutron/+bug/2007166 15:32:13 <ykarel> but let's open a new one and we can check if it's same ^ or different 15:32:31 <ykarel> #action ralonsoh to open an issue for random issues in ovn ubuntu job 15:32:36 <ykarel> thx ralonsoh 15:32:44 <ykarel> #topic Periodic 15:32:57 <ykarel> - pyroute2 functional, already being tracked in https://bugs.launchpad.net/neutron/+bug/2100261 15:33:08 <ykarel> #topic Grafana 15:33:16 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate 15:33:23 <ykarel> let's have a quick look here 15:35:06 <ykarel> can see some spike in ovs jobs, which we already discussed 15:35:19 <ykarel> and same in fullstack and functional 15:35:39 <ykarel> and other check pipeline failures likely patches specific 15:35:45 <ykarel> anything to add? 15:36:20 <ralonsoh> no thanks 15:37:06 <ykarel> k let's move 15:37:07 <ykarel> #topic On Demand 15:37:29 <ykarel> anything else you would like to raise? 15:37:42 <lajoskatona> nothing from me 15:41:18 <ykarel> k if nothing else, let's close early and have everyone 19 minutes back 15:41:25 <ykarel> #endmeeting