Wednesday, 2024-09-11

opendevreviewDarin Chakalov proposed openstack/neutron-fwaas master: Fix router disable flow logging exception  https://review.opendev.org/c/openstack/neutron-fwaas/+/92790505:05
opendevreviewEduardo Olivares proposed openstack/ovn-bgp-agent master: WIP: add retries to get_device_port_at_ovs  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/92882207:14
slaweqralonsoh ykarel lajoskatona good morning, can one of you check https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/915369 when You will have few minutes? Thx in advance :)07:23
slaweqralonsoh ykarel lajoskatona and can you also check https://review.opendev.org/c/openstack/neutron-lib/+/923860? We already have stable/2024.2 in neutron-lib so master is open for next release already and would be nice to have api ref merged soon :)07:25
SvenKieskecan maybe someone with ovn knowledge look at this bug? stale since may and it even has a reproducer script attached with a graph and all, seems ovn is leaking FDs: https://bugs.launchpad.net/neutron/+bug/2063043 thanks!08:13
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/2024.1: Fix member subnet id on a fully populated LB  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/92888708:19
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/2023.2: Fix member subnet id on a fully populated LB  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/92888808:19
opendevreviewFernando Royo proposed openstack/ovn-octavia-provider stable/2023.1: Fix member subnet id on a fully populated LB  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/92889008:20
fricklerSvenKieske: seems froyo is the assignee08:35
froyoSvenkieske Hi!, I will take another look but some last patches should be fixed the issue08:36
SvenKieskefroyo: that would be very nice, thanks for any update! :)08:36
opendevreviewBodo Petermann proposed openstack/neutron-vpnaas master: Add reader context to get_ipsec_site_connection(s)  https://review.opendev.org/c/openstack/neutron-vpnaas/+/92889309:20
opendevreviewMerged openstack/neutron-lib master: Add port trusted vif extension  https://review.opendev.org/c/openstack/neutron-lib/+/92386010:01
dtantsurHi folks. In the ironic CI, we're seeing a high number of devstack failures like this:10:49
dtantsur2024-09-11 10:40:26.858426 | controller | + lib/neutron_plugins/services/l3:_neutron_configure_router_v6:416 :   sudo ip -6 addr replace 2001:db8::2/64 dev br-ex10:49
dtantsur2024-09-11 10:40:26.865813 | controller | RTNETLINK answers: Invalid argument10:49
dtantsurDoes it ring any bells?10:49
dtantsurexample: https://zuul.opendev.org/t/openstack/build/7d06160f4a0d4ea180378da764f9b66110:50
lajoskatonadtantsur: not as I can recall. Strange that Neutron gate is not affected (at least not for me, or it was not mentioned on the meetings) as it is coming from devstack11:37
dtantsurlajoskatona: we probably modify something around networking to enable a "baremetal" setup. But I don't even know what can cause this error in theory..12:09
lajoskatonadtantsur: I checked in opensearch and only ironic jobs affected with this issue12:11
lajoskatonadtantsur: opensearch links are still too long for IRC :-( but as I see first on 5 September12:12
dtantsurnarrows it down a bit12:12
dtantsurI'm still completely puzzled by why it happens and why it does not happen always12:12
dtantsurlajoskatona: do you see it on branches other than master?12:14
lajoskatonaonly master12:15
lajoskatonahttps://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover/?security_tenant=global#?_a=(discover:(columns:!(build_name,build_branch),interval:auto,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_q=(filters:!(),query:(language:kuery,query:'%20message:%22RTNETLINK%20answers:%20Invalid%20argument%22'))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,12:15
lajoskatonato:now))12:15
dtantsurThis URL redirects me to the login form12:26
lajoskatonaopenstack / openstack should work12:31
dtantsurthx12:39
opendevreviewMerged openstack/neutron-tempest-plugin master: Neutron&Designate DNS integration - some enhancements  https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/91536912:40
dtantsurNothing interesting seems to have merged around that time, weird12:43
haleybi wonder if the link up is tweaking something there? i can't reproduce it locally. That was added for Ironic. Maybe moving that after the address assignement?12:49
JayFI wonder if this is related to a neutron lib release. In the past we've had CI fail in the intervening time between library freeze and when we cut NGS13:35
JayFI thought we had fixed that systematically but IMBW13:36
haleybthis seems a little more basic, adding an IPv6 address to an interface isn't even touching neutron. almost like the kernel or package (openvswitch) changed in the image devstack is using13:51
*** whoami-rajat_ is now known as whoami-rajat14:04
haleybralonsoh, ihrachys: https://review.opendev.org/c/openstack/tempest/+/928471 seemed to have helped with the tempest failure, the "not ACTIVE" triggered and the wait succeeded14:07
JayFOne thing I'll look at when I dig into this in about an hour or so is providers. Because I think that time matches up when a CI cloud was at it14:10
JayFMakes me wonder if we're not properly respecting variables to disable IPv6 or similar14:10
ihrachyshaleyb: great to hear. I have one concern there - the loop may never end.14:16
ihrachysJayF: is ipv6 enabled in kernel? I think this may fail on stock ubuntus. (old versions?)14:17
ihrachyswhat's sysctl showing for net.ipv6.conf.all.disable_ipv614:17
* dtantsur is unable to find in the devstack logs14:23
dtantsurthese failures in the end of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7d0/928885/1/check/ironic-tempest-uefi-redfish-vmedia/7d06160/controller/logs/openvswitch/ovs-vswitchd_log.txt, are they expected?14:24
dtantsursame in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7d0/928885/1/check/ironic-tempest-uefi-redfish-vmedia/7d06160/controller/logs/openvswitch/ovsdb-server_log.txt14:24
haleybihrachys: the test will eventually get killed if that happens is what i figured. i mean we can just fail if we somehow loop 3 times or something14:36
opendevreviewMerged openstack/neutron stable/2024.1: Always get local vlan from port other_config  https://review.opendev.org/c/openstack/neutron/+/92695314:40
fricklerthe above failure is on raxflex. it may be related to the nodes not having a local IPv6 address there14:45
fricklerso invalid assumption in devstack, let's move over there14:46
ihrachyshaleyb: yeah but eventually is probably some very global timeout? is it killed with tempest?14:57
ihrachysdtantsur: I don't think it's "expected" but I can't say it's your issue. the message is about controller for openflow not being able to connect. the controller in this case is ovs-agent I think. the agent exits when these messages started to pop. so that's expected - if agent is down, then no controller connection possible,14:59
ihrachysthe agent is killed because devstack died - because of the ipv6 issue. so logs for ovs services or agent at this point are not relevant.15:00
haleybihrachys: i got rid of the while, it will work 99.999% of the time without it :)15:01
haleyb(not verified)15:01
ihrachysI don't think it's a good thing to do :)15:02
ihrachysI think we should spin; just do it controlled15:02
ihrachyshaleyb: re ironic issue above - maybe we should add sysctl dump to worlddump script? that would help to understand what their kernel state is better.15:03
haleybihrachys: sounds good re:ironic15:04
haleybihrachys: regarding tempest, i will agree to disagree on the while loop, as the failure we see is only about the port not being ACTIVE when we initially check. fixing this will allow us to merge the OVN wsgi change that is seeing the failure15:07
ihrachysposted https://review.opendev.org/c/openstack/devstack/+/928929 totally untested for sysctl for devstack15:11
ihrachyshaleyb: I am not against merging it. I'm saying we should not assume that a single repeat is enough15:11
opendevreviewLajos Katona proposed openstack/neutron master: DNM: test functional jobs  https://review.opendev.org/c/openstack/neutron/+/92895317:04
opendevreviewMerged openstack/neutron-vpnaas master: Add reader context to get_ipsec_site_connection(s)  https://review.opendev.org/c/openstack/neutron-vpnaas/+/92889317:15
opendevreviewMerged openstack/neutron stable/2023.1: Use oslo_service's SignalHandler for signals  https://review.opendev.org/c/openstack/neutron/+/92692617:30
opendevreviewMerged openstack/neutron stable/2023.2: Use oslo_service's SignalHandler for signals  https://review.opendev.org/c/openstack/neutron/+/92692421:09

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!