Monday, 2024-12-02

opendevreviewMerged openstack/neutron-tempest-plugin master: Update gate jobs as per the 2025.1 cycle testing runtime  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93327707:03
opendevreviewyatin proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 5 workers  https://review.opendev.org/c/openstack/neutron/+/93642907:07
opendevreviewMerged openstack/neutron master: Update tempest jobs for Ubuntu Noble (24.04)  https://review.opendev.org/c/openstack/neutron/+/93292007:23
ralonsohykarel, hey! https://review.opendev.org/c/openstack/neutron/+/936428/ is working fine but I found https://bugs.launchpad.net/neutron/+bug/208994007:39
ralonsohthis is also affecting the Neutron API start07:39
ralonsohI'm working on this now and I'll push a patch on top of ^07:39
ralonsoh(this is really taking more time than expected...)07:40
ralonsohthe code was not ready for WSGI07:40
ykarelralonsoh, thx yes can now see with the fixes "Hash Ring loaded. 4 active nodes. 0 offline nodes"07:44
ykarelralonsoh, also i changed your testproject to run with 5 workers and there it fails quite consistanly07:46
ykarelhttps://zuul.openstack.org/status?change=93642907:46
ykarelin cleanup_default_secgroup07:47
ykarelAll 5 nodes were found offline. This should never happen in a normal situation, please check the status of your cluster07:48
ykarelAll 0 nodes were found offline. This should never happen in a normal situation, please check the status of your cluster07:48
ralonsohyes, I think there is also an issue with the hash ring method that updates the registers07:48
ralonsohis always trying to update the registers but fails with a DB lock07:48
ralonsohykarel, https://zuul.openstack.org/status?change=936429 oh my...07:49
ralonsohwith 5 workers works horribly wrong07:49
ralonsohah aha ha07:49
ralonsohno no, there is afix for that07:49
ralonsohhttps://review.opendev.org/c/openstack/releases/+/93634507:50
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/93440907:50
ralonsohslaweq, ^ please check this releases patch07:50
ralonsohlets see if we can have it today07:51
ykarelralonsoh, that may help, i believed if root cause is fixed we shouldn't see those pg missings07:54
ralonsohykarel, this is because we execute initialization processes that in eventlet are executed in the initial process and then it forks the workers07:55
ralonsohthis is not happening in WSGI: all workers are created at the same time07:55
ralonsohwe have, for example, the problem of the tunnells initialization 07:55
ykarelyeap right, i meant if all such remaining cases are fixed we shouldn't hit issues like this(missing pgs). I still didn't get from logs what cleaned the pg08:06
zigoHello! About CVE-2024-53916, is Caracal affected?08:06
zigoralonsoh: Do you think I can backport https://review.opendev.org/c/openstack/neutron/+/935883 right away?08:07
slaweqralonsoh hi, done08:14
ralonsohzigo, I commented that in https://review.opendev.org/c/openstack/neutron/+/935883/comments/80105bcb_deee89cc08:17
ralonsohthis patch, once merged, needs to go up to 2023.208:17
ralonsohslaweq, thanks08:18
zigoralonsoh: I don't really need to wait for upstream for it to be merged, my question was regarding Debian packaging.08:20
ralonsohzigo, well, at least in the repository I would prefer to wait until we have enough reviews08:21
zigoOk.08:21
zigoralonsoh: When the patch is merged, just ping me, as I'm already preparing backport patches for Bobcat, Caracal and Dalmatian, I'll have the backport patches ready.08:30
ralonsohzigo, sure, I would like this patch to be merged today08:31
opendevreviewVasyl Saienko proposed openstack/neutron master: Setup DVR L2 arp for AAP with correct MAC  https://review.opendev.org/c/openstack/neutron/+/93680708:37
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN] Improve initial hash ring setup  https://review.opendev.org/c/openstack/neutron/+/93642808:37
zigogr808:57
opendevreviewVasyl Saienko proposed openstack/neutron master: Setup DVR L1 arp for AAP with correct MAC  https://review.opendev.org/c/openstack/neutron/+/93680708:57
zigoralonsoh: I haven't run tempest *yet*, though my backports (Bobcat, Caracal and Dalmatian) are all building fine (ie: all unit test passed). Will let you know when I have run tempest.09:27
ralonsohthanks09:28
opendevreviewRodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 4 workers  https://review.opendev.org/c/openstack/neutron/+/93642909:30
opendevreviewRodolfo Alonso proposed openstack/neutron master: Fast exit when initially creating tunnel allocations  https://review.opendev.org/c/openstack/neutron/+/93681309:30
opendevreviewRodolfo Alonso proposed openstack/neutron master: [OVN] Improve initial hash ring setup  https://review.opendev.org/c/openstack/neutron/+/93642810:30
opendevreviewRodolfo Alonso proposed openstack/neutron master: Fast exit when initially creating tunnel allocations  https://review.opendev.org/c/openstack/neutron/+/93681310:30
opendevreviewRodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 4 workers  https://review.opendev.org/c/openstack/neutron/+/93642910:30
opendevreviewyatin proposed openstack/neutron stable/2023.2: [FT] Check "Port_Binding" register exists before checking type  https://review.opendev.org/c/openstack/neutron/+/93681510:39
opendevreviewyatin proposed openstack/neutron stable/2024.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests  https://review.opendev.org/c/openstack/neutron/+/93681710:47
opendevreviewyatin proposed openstack/neutron stable/2024.1: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests  https://review.opendev.org/c/openstack/neutron/+/93681810:48
opendevreviewyatin proposed openstack/neutron stable/2023.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests  https://review.opendev.org/c/openstack/neutron/+/93681910:48
vsaienkoHello Neutron team, please review https://review.opendev.org/c/openstack/neutron/+/936807 fixes an issue when we configure mac address from port for allowed address pair address, while in aap mac address is different11:09
opendevreviewMerged openstack/neutron-lib master: Switch to neutron-functional job  https://review.opendev.org/c/openstack/neutron-lib/+/93626511:40
zigoralonsoh: Any obvious clue why I'm having this in tempest (with Caracal): https://paste.opendev.org/show/bwSmlgmuHU0SpOVMDsxy/11:43
zigo(I haven't started investigating, but maybe you know already?)11:43
zigoThe file neutron_tempest_plugin/api/admin/test_tag.py hasn't been updated between caracal and dalmatian, as much as I can tell.11:45
ralonsohzigo, because most probably you are missing both patches:11:46
ralonsoh* https://review.opendev.org/c/openstack/neutron/+/93601011:47
ralonsoh* https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93603611:47
zigoThanks, will try to add them too.11:47
opendevreviewyatin proposed openstack/neutron master: [CI] Temporary limit neutron uwsgi workers to 1  https://review.opendev.org/c/openstack/neutron/+/93682912:51
zigoralonsoh: All good for me in my Tempest CI under Caracal. :)12:53
zigo(Debian package + puppet-openstack based install, FYI)12:54
opendevreviewyatin proposed openstack/neutron-tempest-plugin master: Bump cirros version to 0.6.3  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93683012:59
opendevreviewyatin proposed openstack/neutron master: Bump cirros version to 0.6.3  https://review.opendev.org/c/openstack/neutron/+/93683113:03
opendevreviewJakub Libosvar proposed openstack/ovn-bgp-agent master: Split _ensure_ovn_network_link function  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/93640913:17
ralonsohzigo, good to know13:17
amiraslanaslaniHi everyone, I'm working on fixing an issue that spans multiple repositories: Octavia, Octavia Lib, and OVN Octavia Provider. Here’s the challenge I’m facing: The changes I've made in Octavia Lib depend on updates in Octavia and the updates in OVN Octavia Provider depend on changes in both Octavia and Octavia Lib. I want to submit these change on one Gerrit topic but this interdependency is causing test failures on Octavia Lib and O13:20
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Member batch actions to increase performance  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/93676513:25
ralonsohamiraslanaslani, do you know you can push patches with a link to a dependant patch13:29
ralonsohhttps://review.opendev.org/c/openstack/neutron/+/936428/6//COMMIT_MSG#1413:30
opendevreviewVasyl Saienko proposed openstack/neutron master: Setup DVR L1 arp for AAP with correct MAC  https://review.opendev.org/c/openstack/neutron/+/93680713:31
amiraslanaslaniralonsoh, thanks for your insight. no it's the first time i'm trying to contribute. then i will try this.13:36
frickleramiraslanaslani: for reference, see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#adding-a-dependency and https://docs.opendev.org/opendev/infra-manual/latest/developers.html#cross-repository-dependencies13:58
*** haleyb|out is now known as haleyb14:03
opendevreviewRodolfo Alonso proposed openstack/neutron master: WIP == [OVN] Create a OVN hash ring maintenance thread per worker  https://review.opendev.org/c/openstack/neutron/+/93683814:41
opendevreviewRodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 4 workers  https://review.opendev.org/c/openstack/neutron/+/93642914:42
opendevreviewRodolfo Alonso proposed openstack/neutron master: WIP == [OVN] Create a OVN hash ring maintenance thread per worker  https://review.opendev.org/c/openstack/neutron/+/93683814:45
opendevreviewRodolfo Alonso proposed openstack/neutron master: DNM - Test "neutron-ovn-tempest-ipv6-only-ovs*" with 4 workers  https://review.opendev.org/c/openstack/neutron/+/93642914:45
opendevreviewSlawek Kaplonski proposed openstack/neutron-lib master: Add definition of the 'qinq' api extension  https://review.opendev.org/c/openstack/neutron-lib/+/93684114:54
ykarel#startmeeting neutron_ci15:01
opendevmeetMeeting started Mon Dec  2 15:01:26 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
opendevmeetThe meeting name has been set to 'neutron_ci'15:01
ykarelPing list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira15:01
ralonsohhi15:01
slaweqo/15:01
ralonsohirc, right? no video15:01
ykarelyes irc15:02
opendevreviewBodo Petermann proposed openstack/neutron-lib master: VPNaaS-related device owner definitions  https://review.opendev.org/c/openstack/neutron-lib/+/93684215:02
haleybo/15:02
opendevreviewRodolfo Alonso proposed openstack/neutron stable/2024.2: Fix the tagging policy engine  https://review.opendev.org/c/openstack/neutron/+/93684315:02
mlavalle1\o15:03
ykarellet's start with topics15:04
ykarel#topic Actions from previous meetings15:04
opendevreviewRodolfo Alonso proposed openstack/neutron stable/2024.2: Fix the tagging policy engine  https://review.opendev.org/c/openstack/neutron/+/93684315:04
ykarelykarel to check/report issue for stadiums15:04
ykarel    https://bugs.launchpad.net/neutron/+bug/208967315:04
ykarel    https://review.opendev.org/c/openstack/releases/+/93623815:04
ykarel    https://review.opendev.org/q/topic:%22stadium-stable-jobs%2215:04
ykarel    https://review.opendev.org/c/openstack/neutron-fwaas/+/93673915:04
ykarelmost of the fixes already merged, 2 pending, please check those15:05
ykarel#topic Stable branches15:05
ykarelBernard not around today, from what i see all good in stable/unmaintained branches15:06
ykarel#topic Stadium projects15:06
ykarellajos seems not around today15:06
ykarelfrom weekly runs15:06
ykareldynamic-routing,fwaas,vpnaas,sfc,bgpvpn functional job failed with noble switch, https://review.opendev.org/c/openstack/neutron/+/932920 should clear it15:06
ykarelthe fix merged, so next run should be good15:07
ykarel#topic Rechecks15:07
ykarelwe still have random issues requiring rechecks15:07
ykareland had 4/56 bare rechecks15:07
ykarelnot bad15:07
ykarel#topic fullstack/functional15:08
ykareltest_virtual_port_host_update_upon_failover15:08
ykarel    - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_14e/935040/2/gate/neutron-functional-with-uwsgi/14e888f/testr_results.html15:08
ykarel    - proposed backport https://review.opendev.org/c/openstack/neutron/+/93681515:08
ykarelplease check the backport ^15:08
ykareltest_ovn_nb_sync_repair15:08
ykarel    - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9ef/936340/1/check/neutron-functional-with-uwsgi/9ef9cb1/testr_results.html15:08
ykarel    - proposed backports https://review.opendev.org/q/If1da6c5992aa4635da5a4b5c6eaa06db56d693b415:08
ykarelplease check ^ backports too15:08
ykareltest_ha_router_failover_host_failure15:08
ykarelFails like AssertionError: destination ip 42.0.0.248 is replying to ping from namespace test-28f80714-08a4-4d39-90f2-3ce92ea6217e, but it shouldn't15:09
ykarelseen couple of times15:09
ykarel    - https://5c003e9488bb945dd53b-63e63f4901d8c8037ad85f11ee8978c0.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-fullstack-with-pyroute2-master/06dd02e/testr_results.html15:09
ykarel    - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e5c/periodic/opendev.org/openstack/neutron/master/neutron-fullstack/e5cb4b0/testr_results.html15:09
ykarel    - https://92076981c91368a87195-3da5177dffc08bd93069f7a2b6d5d8cd.ssl.cf2.rackcdn.com/936428/2/check/neutron-fullstack/6f8e782/testr_results.html15:09
ykarelSeems same issue https://bugs.launchpad.net/neutron/+bug/2083609 just fails differently15:09
ykarelslaweq as you worked on that issue, if you can confirm ^15:10
slaweqI will take a look15:10
ykarelthx slaweq 15:11
ykarel#action slawek to check/confirm fullstack failures with lp #208360915:11
ykarel#topic Tempest/Scenario15:11
ykareltest_list_security_groups_list_all_tenants_filter15:11
ykarelseen once15:11
ykarelFails like Details: HTTP response body is invalid (None is not of type 'string'15:12
ykarelhttps://1a679a15ede886d68369-dfb11b8e45238c790fb951827c502144.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2024.1/neutron-ovn-tempest-postgres-full/c92c3d6/testr_results.html15:12
ykarelseems triggered by a concurrent test where description was None15:12
ykarelcan keep an eye for now, will report lp if see it again15:14
ralonsohlet me check that15:14
ralonsohthis could be as easy as always adding a description15:14
ralonsohwe do that in n-t-p15:14
ykarelthat test does does add the description, some other likely not15:15
ykarelbut yes adding explicit should clear it15:15
ralonsohI'll open a LP at least to document it15:15
ykarel#action ralonsoh to open bug for test test_list_security_groups_list_all_tenants_filter 15:16
ykarelthx ralonsoh 15:16
ykarelwe still have couple of random failures related to wsgi switch15:16
ykarelwe have couple of fixes and test patch in flight15:17
ykarelthx ralonsoh for working on those15:17
ralonsohworking on this now: https://review.opendev.org/c/openstack/neutron/+/93642915:17
ralonsoh(check the lower patches)15:17
ykarelmeanwhile we can also consider a CI workaround https://review.opendev.org/c/openstack/neutron/+/936829 to avoid such issues impacting gate15:18
ykarelbasically reducing api workers to 115:18
ralonsohI know this is hitting the CI frequently but not so much to be a problem right now15:18
ralonsohwith the current config, we find more issues every day15:19
ralonsohI would, for now, keep the current number of workers15:19
ykarelok if we can live with these failures, will keep it open if we want to go that way some time15:20
ykarel#topic Periodic15:20
opendevreviewBodo Petermann proposed openstack/neutron-vpnaas master: Switch to neutron-lib device owner constants  https://review.opendev.org/c/openstack/neutron-vpnaas/+/93684415:20
ykarelwith noble switch mariadb job is broken15:20
ykarelhttps://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-mariadb-full&project=openstack%2Fneutron&branch=master&skip=015:20
ykarelreported https://bugs.launchpad.net/devstack/+bug/209083515:21
ykarelalso pushed a fix for that, will do run test job for it15:21
ykarel#topic Grafana15:21
ykarelhttps://grafana.opendev.org/d/f913631585/neutron-failure-rate15:21
ykarellet's also have quick look here15:21
ykarelwe can see some spikes in ovn jobs, and those are mostly wsgi switch related issues15:23
ykarelapart from that it looks normal15:23
ykarelanything to add?15:23
ralonsohno15:23
slaweqnope15:24
mlavalle1no15:24
ykarel#topic On Demand15:24
ykareljust a small one 15:25
ykarelUpdating cirros to 0.6.3 https://review.opendev.org/q/topic:%22bug/2045549%2215:25
ykarelanything else you would like to raise?15:25
ralonsohno thanks15:26
mlavalle1no15:26
ykarelthx everyone for joining15:27
ykarel#endmeeting15:27
opendevmeetMeeting ended Mon Dec  2 15:27:16 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:27
opendevmeetMinutes:        https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-12-02-15.01.html15:27
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-12-02-15.01.txt15:27
opendevmeetLog:            https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-12-02-15.01.log.html15:27
ralonsohbye15:27
mlavalle1\o15:28
opendevreviewBrian Haley proposed openstack/neutron master: DNM: test neutron-lib Context class project_id change  https://review.opendev.org/c/openstack/neutron/+/93684515:28
opendevreviewRodolfo Alonso proposed openstack/neutron stable/2024.1: Fix the tagging policy engine  https://review.opendev.org/c/openstack/neutron/+/93684615:33
opendevreviewRodolfo Alonso proposed openstack/neutron stable/2023.2: Fix the tagging policy engine  https://review.opendev.org/c/openstack/neutron/+/93684915:34
ralonsohzigo, https://review.opendev.org/q/Id7d0c8e7ba37993b1084519d05e7e2eac095b81b15:34
opendevreviewBodo Petermann proposed openstack/neutron master: Set IP/MAC address on VPNaaS gateway port (OVN)  https://review.opendev.org/c/openstack/neutron/+/93685015:43
opendevreviewBodo Petermann proposed openstack/neutron master: Set IP/MAC address on VPNaaS gateway port (OVN)  https://review.opendev.org/c/openstack/neutron/+/93685015:44
opendevreviewyatin proposed openstack/neutron master: [DNM] check mariadb fix  https://review.opendev.org/c/openstack/neutron/+/93685316:04
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider master: Bump hacking  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/93685516:33
opendevreviewBodo Petermann proposed openstack/neutron-vpnaas master: Switch to neutron-lib device owner constants  https://review.opendev.org/c/openstack/neutron-vpnaas/+/93684416:33
opendevreviewMerged openstack/neutron stable/2024.2: Add policy enforcer for QoS policy "tags" service plugin  https://review.opendev.org/c/openstack/neutron/+/93634016:58
opendevreviewMerged openstack/neutron-tempest-plugin master: Bump cirros version to 0.6.3  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/93683016:58
ralonsohhaleyb, hey! If you have 1 minute: https://review.opendev.org/q/Ibc7d9b4dd196bed65cff73b79d78122f70aac1a717:06
ralonsohthanks!17:06
opendevreviewVasyl Saienko proposed openstack/neutron master: Setup DVR router ARP for AAP with correct MAC  https://review.opendev.org/c/openstack/neutron/+/93680717:07
haleybralonsoh: ack, done17:22
ralonsohthanks17:22
opendevreviewMerged openstack/neutron stable/2023.2: Add policy enforcer for QoS policy "tags" service plugin  https://review.opendev.org/c/openstack/neutron/+/93634417:38
opendevreviewMerged openstack/neutron master: Bump cirros version to 0.6.3  https://review.opendev.org/c/openstack/neutron/+/93683117:38
opendevreviewMerged openstack/neutron stable/2023.2: [FT] Check "Port_Binding" register exists before checking type  https://review.opendev.org/c/openstack/neutron/+/93681517:39
opendevreviewMerged openstack/neutron stable/2024.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests  https://review.opendev.org/c/openstack/neutron/+/93681717:39
opendevreviewMerged openstack/neutron stable/2024.1: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests  https://review.opendev.org/c/openstack/neutron/+/93681817:39
opendevreviewMerged openstack/neutron stable/2023.2: [OVN] Use the MySQL backend for the ``TestOvnNbSync`` tests  https://review.opendev.org/c/openstack/neutron/+/93681917:39
opendevreviewAslan proposed openstack/ovn-octavia-provider master: Use limited graph retrieval to improve performance  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/93686817:41
opendevreviewMerged openstack/neutron-fwaas stable/2024.2: Account for iptables-save output spacing differences  https://review.opendev.org/c/openstack/neutron-fwaas/+/93673917:42
opendevreviewJakub Libosvar proposed openstack/ovsdbapp master: Add bfd parameter for lr_route_add  https://review.opendev.org/c/openstack/ovsdbapp/+/93687719:40
opendevreviewJakub Libosvar proposed openstack/ovn-bgp-agent master: Use BFD sessions for local OVN  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/93688120:14
opendevreviewMerged openstack/neutron stable/2024.1: OVN metadata agent additional_chassis detection  https://review.opendev.org/c/openstack/neutron/+/93446120:33
opendevreviewMerged openstack/neutron master: Add a repr method to fixture ``Host``  https://review.opendev.org/c/openstack/neutron/+/93621920:33
-opendevstatus- NOTICE: Gerrit will have a short outage while we update to the latest 3.9 release in preparation for our 3.10 upgrade on Friday21:31

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!