opendevreview | Darin Chakalov proposed openstack/neutron-fwaas master: Fix router disable flow logging exception https://review.opendev.org/c/openstack/neutron-fwaas/+/927905 | 05:05 |
---|---|---|
opendevreview | Eduardo Olivares proposed openstack/ovn-bgp-agent master: WIP: add retries to get_device_port_at_ovs https://review.opendev.org/c/openstack/ovn-bgp-agent/+/928822 | 07:14 |
slaweq | ralonsoh ykarel lajoskatona good morning, can one of you check https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/915369 when You will have few minutes? Thx in advance :) | 07:23 |
slaweq | ralonsoh ykarel lajoskatona and can you also check https://review.opendev.org/c/openstack/neutron-lib/+/923860? We already have stable/2024.2 in neutron-lib so master is open for next release already and would be nice to have api ref merged soon :) | 07:25 |
SvenKieske | can maybe someone with ovn knowledge look at this bug? stale since may and it even has a reproducer script attached with a graph and all, seems ovn is leaking FDs: https://bugs.launchpad.net/neutron/+bug/2063043 thanks! | 08:13 |
opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/2024.1: Fix member subnet id on a fully populated LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/928887 | 08:19 |
opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/2023.2: Fix member subnet id on a fully populated LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/928888 | 08:19 |
opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider stable/2023.1: Fix member subnet id on a fully populated LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/928890 | 08:20 |
frickler | SvenKieske: seems froyo is the assignee | 08:35 |
froyo | Svenkieske Hi!, I will take another look but some last patches should be fixed the issue | 08:36 |
SvenKieske | froyo: that would be very nice, thanks for any update! :) | 08:36 |
opendevreview | Bodo Petermann proposed openstack/neutron-vpnaas master: Add reader context to get_ipsec_site_connection(s) https://review.opendev.org/c/openstack/neutron-vpnaas/+/928893 | 09:20 |
opendevreview | Merged openstack/neutron-lib master: Add port trusted vif extension https://review.opendev.org/c/openstack/neutron-lib/+/923860 | 10:01 |
dtantsur | Hi folks. In the ironic CI, we're seeing a high number of devstack failures like this: | 10:49 |
dtantsur | 2024-09-11 10:40:26.858426 | controller | + lib/neutron_plugins/services/l3:_neutron_configure_router_v6:416 : sudo ip -6 addr replace 2001:db8::2/64 dev br-ex | 10:49 |
dtantsur | 2024-09-11 10:40:26.865813 | controller | RTNETLINK answers: Invalid argument | 10:49 |
dtantsur | Does it ring any bells? | 10:49 |
dtantsur | example: https://zuul.opendev.org/t/openstack/build/7d06160f4a0d4ea180378da764f9b661 | 10:50 |
lajoskatona | dtantsur: not as I can recall. Strange that Neutron gate is not affected (at least not for me, or it was not mentioned on the meetings) as it is coming from devstack | 11:37 |
dtantsur | lajoskatona: we probably modify something around networking to enable a "baremetal" setup. But I don't even know what can cause this error in theory.. | 12:09 |
lajoskatona | dtantsur: I checked in opensearch and only ironic jobs affected with this issue | 12:11 |
lajoskatona | dtantsur: opensearch links are still too long for IRC :-( but as I see first on 5 September | 12:12 |
dtantsur | narrows it down a bit | 12:12 |
dtantsur | I'm still completely puzzled by why it happens and why it does not happen always | 12:12 |
dtantsur | lajoskatona: do you see it on branches other than master? | 12:14 |
lajoskatona | only master | 12:15 |
lajoskatona | https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover/?security_tenant=global#?_a=(discover:(columns:!(build_name,build_branch),interval:auto,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_q=(filters:!(),query:(language:kuery,query:'%20message:%22RTNETLINK%20answers:%20Invalid%20argument%22'))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d, | 12:15 |
lajoskatona | to:now)) | 12:15 |
dtantsur | This URL redirects me to the login form | 12:26 |
lajoskatona | openstack / openstack should work | 12:31 |
dtantsur | thx | 12:39 |
opendevreview | Merged openstack/neutron-tempest-plugin master: Neutron&Designate DNS integration - some enhancements https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/915369 | 12:40 |
dtantsur | Nothing interesting seems to have merged around that time, weird | 12:43 |
haleyb | i wonder if the link up is tweaking something there? i can't reproduce it locally. That was added for Ironic. Maybe moving that after the address assignement? | 12:49 |
JayF | I wonder if this is related to a neutron lib release. In the past we've had CI fail in the intervening time between library freeze and when we cut NGS | 13:35 |
JayF | I thought we had fixed that systematically but IMBW | 13:36 |
haleyb | this seems a little more basic, adding an IPv6 address to an interface isn't even touching neutron. almost like the kernel or package (openvswitch) changed in the image devstack is using | 13:51 |
*** whoami-rajat_ is now known as whoami-rajat | 14:04 | |
haleyb | ralonsoh, ihrachys: https://review.opendev.org/c/openstack/tempest/+/928471 seemed to have helped with the tempest failure, the "not ACTIVE" triggered and the wait succeeded | 14:07 |
JayF | One thing I'll look at when I dig into this in about an hour or so is providers. Because I think that time matches up when a CI cloud was at it | 14:10 |
JayF | Makes me wonder if we're not properly respecting variables to disable IPv6 or similar | 14:10 |
ihrachys | haleyb: great to hear. I have one concern there - the loop may never end. | 14:16 |
ihrachys | JayF: is ipv6 enabled in kernel? I think this may fail on stock ubuntus. (old versions?) | 14:17 |
ihrachys | what's sysctl showing for net.ipv6.conf.all.disable_ipv6 | 14:17 |
* dtantsur is unable to find in the devstack logs | 14:23 | |
dtantsur | these failures in the end of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7d0/928885/1/check/ironic-tempest-uefi-redfish-vmedia/7d06160/controller/logs/openvswitch/ovs-vswitchd_log.txt, are they expected? | 14:24 |
dtantsur | same in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7d0/928885/1/check/ironic-tempest-uefi-redfish-vmedia/7d06160/controller/logs/openvswitch/ovsdb-server_log.txt | 14:24 |
haleyb | ihrachys: the test will eventually get killed if that happens is what i figured. i mean we can just fail if we somehow loop 3 times or something | 14:36 |
opendevreview | Merged openstack/neutron stable/2024.1: Always get local vlan from port other_config https://review.opendev.org/c/openstack/neutron/+/926953 | 14:40 |
frickler | the above failure is on raxflex. it may be related to the nodes not having a local IPv6 address there | 14:45 |
frickler | so invalid assumption in devstack, let's move over there | 14:46 |
ihrachys | haleyb: yeah but eventually is probably some very global timeout? is it killed with tempest? | 14:57 |
ihrachys | dtantsur: I don't think it's "expected" but I can't say it's your issue. the message is about controller for openflow not being able to connect. the controller in this case is ovs-agent I think. the agent exits when these messages started to pop. so that's expected - if agent is down, then no controller connection possible, | 14:59 |
ihrachys | the agent is killed because devstack died - because of the ipv6 issue. so logs for ovs services or agent at this point are not relevant. | 15:00 |
haleyb | ihrachys: i got rid of the while, it will work 99.999% of the time without it :) | 15:01 |
haleyb | (not verified) | 15:01 |
ihrachys | I don't think it's a good thing to do :) | 15:02 |
ihrachys | I think we should spin; just do it controlled | 15:02 |
ihrachys | haleyb: re ironic issue above - maybe we should add sysctl dump to worlddump script? that would help to understand what their kernel state is better. | 15:03 |
haleyb | ihrachys: sounds good re:ironic | 15:04 |
haleyb | ihrachys: regarding tempest, i will agree to disagree on the while loop, as the failure we see is only about the port not being ACTIVE when we initially check. fixing this will allow us to merge the OVN wsgi change that is seeing the failure | 15:07 |
ihrachys | posted https://review.opendev.org/c/openstack/devstack/+/928929 totally untested for sysctl for devstack | 15:11 |
ihrachys | haleyb: I am not against merging it. I'm saying we should not assume that a single repeat is enough | 15:11 |
opendevreview | Lajos Katona proposed openstack/neutron master: DNM: test functional jobs https://review.opendev.org/c/openstack/neutron/+/928953 | 17:04 |
opendevreview | Merged openstack/neutron-vpnaas master: Add reader context to get_ipsec_site_connection(s) https://review.opendev.org/c/openstack/neutron-vpnaas/+/928893 | 17:15 |
opendevreview | Merged openstack/neutron stable/2023.1: Use oslo_service's SignalHandler for signals https://review.opendev.org/c/openstack/neutron/+/926926 | 17:30 |
opendevreview | Merged openstack/neutron stable/2023.2: Use oslo_service's SignalHandler for signals https://review.opendev.org/c/openstack/neutron/+/926924 | 21:09 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!