| *** ykarel__ is now known as ykarel | 03:59 | |
| ralonsoh | lajoskatona, hello! if you have a couple of mins, please check https://review.opendev.org/c/openstack/neutron/+/973366 | 08:57 |
|---|---|---|
| ralonsoh | thanks! | 08:57 |
| *** liuxie is now known as liushy | 09:12 | |
| lajoskatona | ralonsoh: looking | 09:34 |
| lajoskatona | ralonsoh: question for the tenant_id => project_id patches, should we wait with the merges till the dust settles around SDK (see https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5HHEYPZA6VIORX2XLBZGNMM2EVX2LR65/ ) | 09:35 |
| lajoskatona | ralonsoh: question amrk to end, sorry: ? | 09:35 |
| ralonsoh | lajoskatona, let me check | 09:35 |
| ralonsoh | the link is not opening... | 09:37 |
| ralonsoh | I don't know what is happening with the mailing list website, it is not loading | 09:39 |
| lajoskatona | yeah, now even the archives page is hanging, 1 hour back it was fine | 09:45 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron-lib master: [WIP] Add PVLAN api extension https://review.opendev.org/c/openstack/neutron-lib/+/975931 | 09:57 |
| ralonsoh | lajoskatona, ok, now I can read it. We have 2 pending patches | 09:58 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/974253 | 09:58 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/975711 | 09:58 |
| ralonsoh | The 1st one affects only testing | 09:59 |
| ralonsoh | the last one is quite big | 09:59 |
| ralonsoh | but it affects only a small portion of the l3 DB | 09:59 |
| ralonsoh | and the rest is testing only | 09:59 |
| ralonsoh | IMO, we should finish it and then fix what could be broken in other projects | 10:00 |
| lajoskatona | ack, thanks | 10:03 |
| opendevreview | Merged openstack/ovsdbapp master: Simplify venv with ProcessFixture https://review.opendev.org/c/openstack/ovsdbapp/+/975198 | 10:07 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Update FDB default configuration values https://review.opendev.org/c/openstack/neutron/+/975908 | 10:07 |
| opendevreview | Merged openstack/neutron master: Stop using tenant_* in context objects https://review.opendev.org/c/openstack/neutron/+/974253 | 11:05 |
| gibi | ralonsoh: lajoskatona: o/ does this failure rings a bell to you? https://paste.opendev.org/show/bxYcKroz4QKkDmVNGya9/ | 11:57 |
| gibi | I saw this in devstack during stack.sh in a clean env on master | 11:57 |
| gibi | Feb 09 10:42:09 aio neutron-periodic-workers[75394]: ERROR neutron.service [None req-69e77641-be9e-4a28-b683-7e91fce0642c None None] Unrecoverable error: please check log for details.: AttributeError: Can't pickle local object 'Mapper.__init__.<locals>.<lambda>' | 11:59 |
| ralonsoh | gibi, let me check | 12:01 |
| gibi | ralonsoh: thanks | 12:01 |
| ykarel | gibi, by chance in your local you using unreleased oslo libraries? | 12:10 |
| ykarel | as noticed it in our periodic jobs which runs with unreleased libraries and was about to report a bug | 12:10 |
| ykarel | i see couple of patches merged in oslo.service and those looks suspicious | 12:12 |
| ralonsoh | ykarel, thanks for checking! I don't see anything relevant in Neutron related to this | 12:15 |
| ralonsoh | gibi, is that a local install? Can you print, before the error, what class is each worker? | 12:16 |
| gibi | ykarel: I'm running that job LIBS_FROM_GIT="python-neutronclient,openstacksdk,python-novaclient,osc-placement,neutron-lib,os-vif,os-traits,oslo-messaging,oslo-service,oslo-config" | 12:17 |
| gibi | so it has oslo.service at | 12:17 |
| gibi | stack@aio:/opt/stack/oslo.service$ git log | 12:17 |
| gibi | commit 131c4624360eeff891650c64cb0b89aca2ab0140 (HEAD, origin/master, origin/HEAD, master) | 12:17 |
| gibi | Author: Ghanshyam Maan <gmaan@ghanshyammann.com> | 12:18 |
| gibi | Date: Thu Feb 5 21:12:56 2026 +0000 | 12:18 |
| gibi | Fix ServiceManager singleton race condition in tests | 12:18 |
| ykarel | thx gibi for confirming, reported https://bugs.launchpad.net/neutron/+bug/2141304 | 12:19 |
| gibi | ralonsoh: | 12:20 |
| gibi | ********************************** {{(pid=102460) log_opt_values /opt/stack/oslo.config/oslo_config/cfg.py:3013}} | 12:20 |
| gibi | Feb 09 12:20:09 aio neutron-periodic-workers[102460]: WARNING neutron.service [None req-29cbbab9-c2a5-4b78-b2b0-61aab880a8e4 None None] worker: <neutron.service.RpcWorker object at 0x70aa8c3ace90> <class 'neutron.service.RpcWorker'> | 12:20 |
| gibi | Feb 09 12:20:09 aio neutron-periodic-workers[102460]: WARNING neutron.service [None req-29cbbab9-c2a5-4b78-b2b0-61aab880a8e4 None None] worker: <neutron.worker.PeriodicWorker object at 0x70aa8d1bf470> <class 'neutron.worker.PeriodicWorker'> | 12:20 |
| gibi | Feb 09 12:20:09 aio neutron-periodic-workers[102460]: ERROR oslo_service.backend._threading.service [None req-29cbbab9-c2a5-4b78-b2b0-61aab880a8e4 None None] Service PeriodicWorker is not picklable with spawn. Make it spawn-safe before launching additional workers.: AttributeError: Can't pickle local object 'Mapper.__init__.<locals>.<lambda>' | 12:20 |
| ralonsoh | gibi, yeah, these are the expected workers | 12:20 |
| gibi | ykarel: thanks for reporting, I will restack with release oslo.service for now | 12:21 |
| ralonsoh | I think ykarel is right, this could be an issue in oslo.service | 12:21 |
| ralonsoh | ykarel, don't we have CI with oslo master?? | 12:21 |
| ykarel | ralonsoh, yes we catched it in periodic CI | 12:21 |
| ykarel | links are there in the bug | 12:22 |
| ykarel | it failed randomly | 12:22 |
| ralonsoh | ahhhh cool, perfect | 12:22 |
| ykarel | 3/6 failed since 4th feb | 12:22 |
| ralonsoh | ok so the error can be reduced to just a few days | 12:23 |
| ralonsoh | I'm talking to Daniel in the oslo channel | 12:24 |
| opendevreview | yatin proposed openstack/neutron master: [DNM] Check migration ovn with fixes https://review.opendev.org/c/openstack/neutron/+/972820 | 13:17 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron-lib master: [WIP] Add PVLAN api extension https://review.opendev.org/c/openstack/neutron-lib/+/975931 | 13:25 |
| opendevreview | Richard Cruise proposed openstack/ovn-octavia-provider master: Update flake8-import-order version to fix checks https://review.opendev.org/c/openstack/ovn-octavia-provider/+/976120 | 13:51 |
| opendevreview | Rodolfo Alonso proposed x/whitebox-neutron-tempest-plugin master: Change the order of the Neutron metadata service check https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/976123 | 14:23 |
| ralonsoh | ykarel, ^ | 14:23 |
| ykarel_ | thx ralonsoh | 14:24 |
| *** haleyb|out is now known as haleyb | 14:24 | |
| mblue_ | hi lajoskatona, are you fine with moving these tests to NTP? as discussed here: https://review.opendev.org/c/openstack/tempest/+/974752/comments/bac6c18d_549bc52e (cc ralonsoh) | 14:27 |
| lajoskatona | mblue_: Hi, If these are skipped in tempest currently, I think it is a good idea, You can perhaps add this topic to the team meeting on-demand agenda: https://wiki.openstack.org/wiki/Network/Meetings | 14:35 |
| ralonsoh | thanks lajoskatona! | 14:35 |
| mblue_ | thanks! | 14:37 |
| mblue_ | lajoskatona, ralonsoh: added as topic for on-demand tomorrow | 14:44 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Make the ``logging_rpc`` member a class property https://review.opendev.org/c/openstack/neutron/+/976130 | 14:55 |
| *** ykarel_ is now known as ykarel | 15:00 | |
| ykarel | #startmeeting neutron_ci | 15:01 |
| opendevmeet | Meeting started Mon Feb 9 15:01:52 2026 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:01 |
| ykarel | Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:01 |
| ykarel | It's on IRC today | 15:02 |
| haleyb | o/ | 15:02 |
| mtomaska | 0/ | 15:02 |
| ralonsoh | hello | 15:02 |
| bcafarel | o/ | 15:03 |
| lajoskatona | o/ | 15:03 |
| ykarel | Hi all | 15:04 |
| opendevreview | Maor Blaustein proposed x/whitebox-neutron-tempest-plugin master: [DNM] Fail api server test on purpose for debugging https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/942536 | 15:04 |
| ykarel | no action item from previous week | 15:04 |
| ykarel | #topic Stable branches | 15:04 |
| ykarel | setuptools 82.0.0 broke most jobs | 15:04 |
| ykarel | [ci][kolla] setuptools v82.0.0 breaks a bunch of things by removing pkg_resources - https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/374GJ6DM6DRPFF6QBWQ6UGMCDJVA4YLA/ | 15:04 |
| ykarel | being fixed in https://review.opendev.org/q/topic:%22pin-setuptools%22 | 15:05 |
| ykarel | no recent backports in neutron stable | 15:05 |
| ykarel | bcafarel, anything to add ? | 15:05 |
| bcafarel | no, it is quiet indeed (and stable from what I saw) | 15:07 |
| ykarel | ok thx | 15:08 |
| ykarel | #topic Stadium projects | 15:08 |
| ykarel | finally all green \o/ | 15:08 |
| ykarel | lajoskatona, anything else to add? | 15:08 |
| ralonsoh | cool! | 15:08 |
| lajoskatona | no, fine that it is green so great work, thanks everybody :-) | 15:09 |
| ykarel | ok thanks | 15:10 |
| ykarel | #topic Rechecks | 15:11 |
| ykarel | still have a few rechecks due to random failures in CI | 15:12 |
| ykarel | bare recheck wise not able to collect stats as getting some failures | 15:13 |
| ykarel | #topic fullstack/functional | 15:13 |
| ykarel | test_gateway_chassis_least_loaded_scheduler_anti_affinity FT failure seen twice | 15:13 |
| ykarel | test_gateway_chassis_least_loaded_scheduler_anti_affinity seen twice | 15:13 |
| ykarel | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dba/openstack/dba0dc581dea409ead6001e079a622ae/testr_results.html | 15:13 |
| ykarel | https://768a14d1f7e379a1573a-dd0601cbd7f0aa15c09a82437ec7e47b.ssl.cf1.rackcdn.com/openstack/c5ec4ca660ce400595bc2d78953b993f/testr_results.html | 15:13 |
| ykarel | Looks triggered by https://review.opendev.org/c/openstack/neutron/+/974997 - [OVN] Fix an issue in the L3 OVN scheduler | 15:13 |
| ralonsoh | i'll check it again | 15:14 |
| ykarel | thx ralonsoh | 15:14 |
| ykarel | #action ralonsoh to check test_gateway_chassis_least_loaded_scheduler_anti_affinity failures | 15:14 |
| ykarel | #topic grenade | 15:14 |
| ykarel | master grenade also impacted by setuptools 82.0.0, devstack fixes should help in clearing this too | 15:14 |
| ykarel | #topic Periodic | 15:15 |
| ykarel | - https://zuul.openstack.org/builds?job_name=openstacksdk-functional-devstack-networking&project=openstack%2Fneutron&branch=master&skip=0 broken | 15:15 |
| ykarel | [ci][nova][cinder][glance][devstack][swift] openstacksdk-functional-devstack job failing 100% https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/6Y34AM6TIZESGITCKJWSC6R7Y4SACPT3/ | 15:15 |
| ykarel | haleyb, any update you would like to share on top of ^ ? | 15:15 |
| ralonsoh | I think he pushed https://review.opendev.org/c/openstack/openstacksdk/+/976008 | 15:17 |
| haleyb | ykarel: i have a patch up | 15:17 |
| ralonsoh | but is still failing in manila | 15:17 |
| ykarel | #link https://review.opendev.org/c/openstack/openstacksdk/+/976008 | 15:17 |
| haleyb | yes, that one, and pep8 is broken as well but that's the pkg_resources issue i think | 15:17 |
| haleyb | and the -manila job was broken before the project_id change merged | 15:18 |
| haleyb | that was all i had | 15:19 |
| ykarel | for pep8 looks need to bump https://github.com/openstack/openstacksdk/blob/master/.pre-commit-config.yaml#L32 | 15:20 |
| ykarel | similar to https://review.opendev.org/c/openstack/python-octaviaclient/+/976112 | 15:20 |
| haleyb | ykarel: ah, my grep for flake8 was in * not .* :) | 15:21 |
| haleyb | will need similar change in other repos probably, think i had another patch failing pep8 job as well | 15:22 |
| ykarel | ok | 15:23 |
| ykarel | random failures https://zuul.openstack.org/builds?job_name=neutron-ovs-tempest-with-oslo-master&project=openstack%2Fneutron&branch=master&skip=0 | 15:23 |
| ykarel | - example https://zuul.openstack.org/build/d2e60278d3e84fdca35a137950e4324f | 15:23 |
| ykarel | failing as neutron-periodic-workers[73353]: ERROR neutron AttributeError: Can't pickle local object 'Mapper.__init__.<locals>.<lambda>' | 15:23 |
| ykarel | reported https://bugs.launchpad.net/neutron/+bug/2141304 | 15:24 |
| ykarel | ralonsoh was checking with oslo folks on this | 15:24 |
| ralonsoh | yes, herve is checking it now | 15:24 |
| ralonsoh | it seems to be an issue in oslo.service | 15:24 |
| ykarel | thx ralonsoh for the update | 15:25 |
| ykarel | #topic Grafana | 15:25 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate | 15:25 |
| ykarel | ok there was one failure in gate which i missed to report | 15:26 |
| ykarel | tempest-plugin-openvswitch job https://9860dbfd2852c0414b38-8af958d7dd3c07fe325f83683b2d5c0b.ssl.cf2.rackcdn.com/openstack/c1166a273e7c4fcfabf35ef84eb660fe/testr_results.html failed many tests | 15:27 |
| ykarel | couldn't spot the reason, can anyone else check this? i think i saw similar in past too | 15:28 |
| ralonsoh | I'll check it tomorrow morning | 15:28 |
| ykarel | thx ralonsoh | 15:29 |
| ykarel | #action ralonsoh to check failure in the openvswitch job | 15:29 |
| ykarel | #topic On Demand | 15:29 |
| ykarel | anything else you would like to raise? | 15:29 |
| lajoskatona | nothing from me, thanks | 15:30 |
| ralonsoh | nothing, thanks | 15:31 |
| ykarel | ok in that case let's close early and have everyone almost 27 minutes back | 15:33 |
| ykarel | thx everyone for joining | 15:33 |
| ykarel | #endmeeting | 15:33 |
| opendevmeet | Meeting ended Mon Feb 9 15:33:20 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:33 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2026/neutron_ci.2026-02-09-15.01.html | 15:33 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2026/neutron_ci.2026-02-09-15.01.txt | 15:33 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2026/neutron_ci.2026-02-09-15.01.log.html | 15:33 |
| ralonsoh | bye | 15:33 |
| lajoskatona | Bye! | 15:33 |
| mtomaska | o/ | 15:33 |
| bcafarel | o/ | 15:33 |
| opendevreview | yatin proposed openstack/neutron master: [DNM] Check migration ovn tests https://review.opendev.org/c/openstack/neutron/+/972820 | 15:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/neutron-vpnaas master: Fix ipsec config updates https://review.opendev.org/c/openstack/neutron-vpnaas/+/976135 | 16:02 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Remove configuration variable ``enable_signals``. https://review.opendev.org/c/openstack/neutron/+/976137 | 16:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/neutron-vpnaas master: Fix ipsec config updates https://review.opendev.org/c/openstack/neutron-vpnaas/+/976135 | 16:13 |
| opendevreview | Daniel Alvarez proposed openstack/ovn-bgp-agent master: [NB evpn] Fix race condition when adding routes https://review.opendev.org/c/openstack/ovn-bgp-agent/+/976146 | 16:54 |
| opendevreview | Merged openstack/neutron master: Enable ovs_create_tap in ovn multinode jobs https://review.opendev.org/c/openstack/neutron/+/973915 | 17:04 |
| opendevreview | Brian Haley proposed openstack/neutron-vpnaas master: Fix failing pep8 CI job https://review.opendev.org/c/openstack/neutron-vpnaas/+/976157 | 17:59 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: bgp: Use port mapping for learned routes https://review.opendev.org/c/openstack/neutron/+/975674 | 18:14 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: bgp: Support IPv6 coming from fabric https://review.opendev.org/c/openstack/neutron/+/976158 | 18:14 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: bgp: Fix configuring vrf id on BGP routers https://review.opendev.org/c/openstack/neutron/+/976159 | 18:14 |
| opendevreview | Brian Haley proposed openstack/neutron-vpnaas master: Fix failing pep8 CI job https://review.opendev.org/c/openstack/neutron-vpnaas/+/976157 | 18:21 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 18:39 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: bgp: Implement connecting BGP and Neutron world https://review.opendev.org/c/openstack/neutron/+/963223 | 20:58 |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: WIP: new job tempest-multinode-with-bgp https://review.opendev.org/c/openstack/neutron/+/962188 | 21:09 |
| opendevreview | Merged openstack/ovsdbapp master: Fix KEEP_VENV for recent OVN/OVS and single venvs https://review.opendev.org/c/openstack/ovsdbapp/+/973562 | 21:22 |
| opendevreview | Merged openstack/neutron-vpnaas master: Fix failing pep8 CI job https://review.opendev.org/c/openstack/neutron-vpnaas/+/976157 | 21:40 |
| opendevreview | Merged openstack/ovsdbapp master: Re-apply Improve Row stringification https://review.opendev.org/c/openstack/ovsdbapp/+/974956 | 22:11 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!