| ralonsoh | Hi folks, some quick review patches from Takashi and Brian | 08:21 |
|---|---|---|
| ralonsoh | https://review.opendev.org/c/openstack/networking-sfc/+/967783 | 08:21 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron-lib/+/968732 | 08:21 |
| ralonsoh | and this one from me: | 08:21 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/968769 | 08:21 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: iPXE over IPv6 supported since OVN v23.06.0 https://review.opendev.org/c/openstack/neutron/+/968811 | 08:28 |
| ralonsoh | ah, another 2 easy patches: | 08:28 |
| ralonsoh | * https://review.opendev.org/c/openstack/neutron/+/968805 | 08:28 |
| ralonsoh | * https://review.opendev.org/c/openstack/neutron/+/968797 | 08:28 |
| blanson[m] | Hello, we're hitting a weird bug that I can't seem to find on launchpad using caracal neutron with ovs. Sometimes (sometimes every week, sometimes every two weeks), we running into a stack trace in the main thread, because the rpc_loop function of the ovs_agent (https://github.com/openstack/neutron/blob/unmaintained/2024.1/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L2746) ends up calling uninstall_flow | 08:45 |
| blanson[m] | from ofswitch, which in turn calls a parser in os_ken (https://github.com/openstack/os-ken/blob/unmaintained/2024.1/os_ken/ofproto/ofproto_v1_3_parser.py), but doesn't pass kwargs and so this (https://github.com/openstack/os-ken/blob/unmaintained/2024.1/os_ken/ofproto/ofproto_v1_3_parser.py#L891-L894) raises because of a ValueError ? (not enough values to unpack (expected 2, got 0)) | 08:45 |
| blanson[m] | Does that ring a bell to anyone ? | 08:45 |
| lajoskatona | blanson[m]: Hi, thanks, seems like we have it in zuul logs also: (https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_322/openstack/3223793519aa492583ed031e81fc0674/controller/logs/screen-q-agt.txt ) | 08:53 |
| ralonsoh | blanson[m], it would be better if you provide traces of the error, including logs from the OVS agent (better in debug mode) | 08:53 |
| ralonsoh | and, if possible, a reproducer or a root cause of the issue | 08:53 |
| blanson[m] | not sure I should paste these directly into irc, I shall make a proper bug report then | 08:54 |
| lajoskatona | blanson[m]: could you please open a bug with the details on launchpad? | 08:54 |
| ralonsoh | blanson[m], yes, much better in launchpad | 08:54 |
| lajoskatona | blanson[m]: that will be perfect, thanks | 08:54 |
| blanson[m] | https://bugs.launchpad.net/neutron/+bug/2133487 here you go | 09:07 |
| blanson[m] | we have plenty of logs so if you need more details just ask :) We'll debug it on our side aswell cause it's essentially crashing the ovs_agent everytime it happens and it needs a restart otherwise networking on the compute node is broken in so many ways | 09:08 |
| lajoskatona | blanson[m]: thanks | 09:10 |
| ralonsoh | blanson[m], hi, if you are going to debug this issue locally, please try to add some extra logs in the os-ken library | 09:19 |
| ralonsoh | in particular for kwargs in https://github.com/openstack/os-ken/blob/unmaintained/2024.1/os_ken/ofproto/ofproto_v1_3_parser.py#L891-L894 | 09:19 |
| ralonsoh | before and after the normalization method | 09:20 |
| blanson[m] | yh I was thinking of putting logs everywhere to get all the values passed around. I'll update the LP with the results aswell | 09:21 |
| ralonsoh | cool | 09:21 |
| skraynev | hello. could someone help to understand why policy was changed here: https://review.opendev.org/c/openstack/neutron/+/765847 I am wondering only about get_floatingip_port_forwarding . Previously it was allowed for member, but in this patch (and in current master branch) it requires reader role. and with member is not possible to get data | 11:15 |
| skraynev | @ralonsoh could you please help to solve this puzzle? ^^^ should it be only reader ? or member role also has to be added? | 11:34 |
| ralonsoh | let me check | 11:35 |
| ralonsoh | skraynev, that was changed in https://review.opendev.org/q/Ibff4c4f5b6d020fd598831a8a6e8ec0e2f559005 | 11:50 |
| skraynev | @ralonsoh yeah. but here is still ADMIN_OR_PARENT_OWNER_READER , without member on get_floatingip_port_forwarding. other endpoints require member, but here is reader | 11:51 |
| ralonsoh | let me check | 11:52 |
| ralonsoh | skraynev, can you describe the use case? So you have a member user and you try to read what FIPs? | 11:55 |
| skraynev | correct. I have user with role member, but without reader. and want to GET FIP port forwarding | 11:56 |
| skraynev | according policy it looks like I could not do it without role reader | 11:57 |
| ralonsoh | who created these port forwarding resources? | 11:58 |
| ralonsoh | skraynev, I can't reproduce the issue: I'm using admin/demo and demo/demo (a devstack deployment) | 12:02 |
| ralonsoh | even with a port, FIP and PF created by the admin user, I can list all with the demo user | 12:02 |
| ralonsoh | (in the same project) | 12:02 |
| ralonsoh | skraynev, can you open a LP bug with a reproducer? creating a new user, who creates the port, the FIP, etc | 12:03 |
| skraynev | @ralonsoh I forgot to mention, that I just generate polcies based on the fresh policy. Looks like on your installation bug is not reproduced, because deprecated rule is used by default: https://github.com/openstack/neutron/blob/7fa7737947092c77cb2c210fd18c5adbb81e5223/neutron/conf/policies/floatingip_port_forwarding.py#L63-L68 | 12:09 |
| skraynev | if to drop it from code - the new policy (with reader only) should be applied | 12:10 |
| ralonsoh | right, I have this | 12:13 |
| ralonsoh | "get_floatingip_port_forwarding": "(rule:admin_only) or (role:reader and rule:ext_parent_owner)" | 12:13 |
| ralonsoh | using oslopolicy-policy-generator | 12:13 |
| ralonsoh | skraynev, so I have role:reader but I can't reproduce your bug | 12:23 |
| ralonsoh | I'm still able to list the PF | 12:23 |
| skraynev | @ralonsoh does "demo" user have only "member" role? | 12:24 |
| ralonsoh | I have this: | 12:27 |
| ralonsoh | $ openstack role assignment list --user demo --names | 12:27 |
| ralonsoh | +-------------+--------------+-------+--------------+--------+--------+-----------+ | 12:27 |
| ralonsoh | | Role | User | Group | Project | Domain | System | Inherited | | 12:27 |
| ralonsoh | +-------------+--------------+-------+--------------+--------+--------+-----------+ | 12:27 |
| ralonsoh | | anotherrole | demo@Default | | demo@Default | | | False | | 12:27 |
| ralonsoh | | member | demo@Default | | demo@Default | | | False | | 12:27 |
| ralonsoh | +-------------+--------------+-------+--------------+--------+--------+-----------+ | 12:27 |
| ralonsoh | I can try deleting anotherrole | 12:27 |
| ralonsoh | The same: now "demo" has only "member" (one unique role) | 12:28 |
| skraynev | could you please. if it will not change behaviour, I will go to deeply check my installation and try to reproduce on devstack | 12:28 |
| ralonsoh | and the FIP was created by the admin user and the PF | 12:28 |
| skraynev | btw, do you have existing policy yaml file ? or use defaults? I think, maybe oslo-policy use deprecated version, but prints new version. | 12:31 |
| skraynev | @ralonsoh have you seen my last question about existing policy file? I have disconnect, so may be missed answer. | 13:08 |
| ralonsoh | and now you look disconnected again... | 13:31 |
| ykarel | #startmeeting neutron_ci | 15:01 |
| opendevmeet | Meeting started Mon Dec 1 15:01:14 2025 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:01 |
| ykarel | Ping list: bcafarel, lajoskatona, slawek, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:01 |
| ralonsoh | hello | 15:01 |
| bcafarel | o/ | 15:01 |
| ralonsoh | irc, right? | 15:01 |
| ykarel | Yeap it's IRC today | 15:01 |
| ykarel | #topic Actions from previous meetings | 15:04 |
| ykarel | lajoskatona to open bug for bagpipe failures and check | 15:04 |
| ykarel | #link https://bugs.launchpad.net/neutron/+bug/2132355 | 15:04 |
| ykarel | the fix was merged | 15:04 |
| ykarel | thx lajoskatona | 15:04 |
| ykarel | ralonsoh to check functional timeout failure logs and accordingly bump timeout or track as a bug | 15:05 |
| lajoskatona | o/ | 15:05 |
| ykarel | #link https://review.opendev.org/c/openstack/neutron/+/968210 | 15:05 |
| ykarel | thx ralonsoh | 15:05 |
| ralonsoh | yw | 15:05 |
| ykarel | #topic Stable branches | 15:05 |
| ykarel | all looks good in stable as per periodic | 15:05 |
| ykarel | also couple of patches merged in stable last week | 15:06 |
| bcafarel | yep | 15:06 |
| ykarel | #topic Stadium projects | 15:06 |
| ykarel | dynamic-routing still broken https://zuul.openstack.org/buildset/116018d1fc3248a1a10f86688cb34e94 | 15:06 |
| ykarel | Known one https://bugs.launchpad.net/neutron/+bug/2130631 | 15:07 |
| ykarel | lajoskatona, anything to add? | 15:07 |
| lajoskatona | nothing from me, only the bagpipe thing I had time last week | 15:07 |
| ykarel | ok thx | 15:08 |
| ykarel | #topic Rechecks | 15:08 |
| ykarel | we still have a few rechecks | 15:08 |
| ykarel | bare rechecks wise 1/7 bare recheck | 15:09 |
| ykarel | 1 was on some testing patch | 15:09 |
| ykarel | so let's keep avoiding bare rechecks | 15:09 |
| ykarel | now let's check the failures | 15:09 |
| ykarel | #topic fullstack/functional | 15:09 |
| ykarel | test_add_manager_overwrites_existing_manager | 15:10 |
| ykarel | https://b906e8bbacbb1d55ef2e-e8840ed1f498e2b83edb07bf05759e4c.ssl.cf5.rackcdn.com/openstack/e18b234e487d49e9853ec535778d8888/testr_results.html | 15:10 |
| ykarel | Looks need to reopen https://bugs.launchpad.net/neutron/+bug/2131024 | 15:10 |
| ralonsoh | I'll check it | 15:11 |
| ykarel | thx | 15:11 |
| ykarel | #action ralonsoh to check lp#2131024 | 15:11 |
| ykarel | test_create_bridges | 15:11 |
| ykarel | https://45b67cabc958c12316a8-de1ec222256e01db8c1f1f4d7a4b9170.ssl.cf2.rackcdn.com/openstack/a451afbef2b44cc8ae8251874b98ef4c/testr_results.html | 15:11 |
| ykarel | Looks need to reopen https://bugs.launchpad.net/neutron/+bug/2114909 | 15:12 |
| ykarel | as we still seeing it | 15:12 |
| ralonsoh | ok, I'll check it again too | 15:12 |
| ralonsoh | I think I pushed 2 patches recently | 15:12 |
| ykarel | #action ralonsoh to check lp#2114909 | 15:12 |
| ykarel | thx ralonsoh | 15:12 |
| ykarel | #topic Periodic | 15:12 |
| ykarel | https://zuul.openstack.org/builds?job_name=ironic-tempest-ovn-uefi-ipmi-pxe&project=openstack%2Fneutron&skip=0 | 15:13 |
| ykarel | new ovmf package fixed the issue, so all good now | 15:13 |
| ykarel | #topic Grafana | 15:13 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate | 15:13 |
| ykarel | let's have a quick check here too | 15:13 |
| ykarel | looks all good | 15:15 |
| lajoskatona | +1 | 15:16 |
| ralonsoh | I think so | 15:16 |
| ykarel | #topic On Demand | 15:16 |
| ykarel | anything else you want to discuss | 15:16 |
| ralonsoh | nothing from me | 15:16 |
| lajoskatona | nothing from my side | 15:16 |
| bcafarel | all good for me | 15:17 |
| ykarel | ok thx | 15:18 |
| ykarel | in that case let's close early and have everyone almost 40 minutes back | 15:18 |
| ykarel | #endmeeting | 15:18 |
| opendevmeet | Meeting ended Mon Dec 1 15:18:36 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:18 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2025/neutron_ci.2025-12-01-15.01.html | 15:18 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2025/neutron_ci.2025-12-01-15.01.txt | 15:18 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2025/neutron_ci.2025-12-01-15.01.log.html | 15:18 |
| ralonsoh | bye! | 15:18 |
| lajoskatona | o/ | 15:20 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [FT] Wait for the manager to be created (2) https://review.opendev.org/c/openstack/neutron/+/969107 | 15:53 |
| *** haleyb|out is now known as haleyb | 15:54 | |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/2025.2: Don't prevent setting CIDRs as allowed_address_pair for port https://review.opendev.org/c/openstack/neutron/+/969126 | 17:20 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/2025.1: Don't prevent setting CIDRs as allowed_address_pair for port https://review.opendev.org/c/openstack/neutron/+/969127 | 17:21 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron stable/2024.2: Don't prevent setting CIDRs as allowed_address_pair for port https://review.opendev.org/c/openstack/neutron/+/969128 | 17:21 |
| cardoe | So are changes like https://review.opendev.org/c/openstack/neutron/+/968508 okay to make? I'm trying to test my l2vni and my code needs real NetworkContext and PortContext... the port one is a bit bigger but I just want to make sure that's something you guys would want. | 17:35 |
| rm_work | We’re seeing a really strange issue with dnsmasq crashing (with a double-free error) when neutron replaces the additional hosts file (it is apparently designed to swap out the file inode to be threadsafe?) but I have a theory dnsmasq does not like this… just wondering if anyone else has seen similar issues. | 17:35 |
| rm_work | https://github.com/openstack/neutron/blob/stable/2024.2/neutron/agent/linux/dhcp.py#L1203 | 17:35 |
| cardoe | rm_work: I believe we've seen that on the Ironic side as well. There's a newer dnsmasq that might fix it. I know JayF | 18:08 |
| cardoe | was tracking the issue. | 18:08 |
| JayF | rm_work: yeah, make sure you're using the absolute latest dnsmasq release | 18:11 |
| opendevreview | Merged openstack/neutron master: [OVN] Add 'provider:physical_network' field if there is physnet https://review.opendev.org/c/openstack/neutron/+/968769 | 19:35 |
| opendevreview | Merged openstack/neutron master: Allow plugins to add periodics to maintenance worker https://review.opendev.org/c/openstack/neutron/+/939817 | 19:49 |
| opendevreview | Merged openstack/neutron master: [docs] Add OVS/OVN min requirement table https://review.opendev.org/c/openstack/neutron/+/967816 | 19:49 |
| opendevreview | Merged openstack/neutron-lib master: Replace deprecated warn_on_missing_entrypoint https://review.opendev.org/c/openstack/neutron-lib/+/968732 | 19:55 |
| opendevreview | Brian Haley proposed openstack/neutron unmaintained/2024.1: Don't prevent setting CIDRs as allowed_address_pair for port https://review.opendev.org/c/openstack/neutron/+/969155 | 21:03 |
| opendevreview | Merged openstack/neutron unmaintained/2024.1: Replace response body in after hook for 404 error https://review.opendev.org/c/openstack/neutron/+/968336 | 21:51 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change some common test code to use project_id https://review.opendev.org/c/openstack/neutron/+/968083 | 22:06 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change some unit test code to use project_id https://review.opendev.org/c/openstack/neutron/+/968084 | 22:07 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change some functional test code to use project_id https://review.opendev.org/c/openstack/neutron/+/968085 | 22:08 |
| rm_work | JayF: ah you’ve seen it? Is there an issue or something? | 22:08 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change test code to use project_id in dictionaries https://review.opendev.org/c/openstack/neutron/+/968414 | 22:08 |
| JayF | saying I've "seen" that bug doesn't do it justice. I've been tracking it for years. It's finally fixed and in a release though so if you're on latest dnsmasq, you found a new one that rhymes | 22:08 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change some unit test code to use project_id https://review.opendev.org/c/openstack/neutron/+/968084 | 22:09 |
| rm_work | We are literally .1 releases from newest on dnsmasq but can try it | 22:09 |
| JayF | https://bugs.launchpad.net/ironic/+bug/2026757 | 22:09 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change some functional test code to use project_id https://review.opendev.org/c/openstack/neutron/+/968085 | 22:09 |
| opendevreview | Brian Haley proposed openstack/neutron master: Change test code to use project_id in dictionaries https://review.opendev.org/c/openstack/neutron/+/968414 | 22:09 |
| JayF | rm_work: ^ that bug has all the information in it I have | 22:09 |
| JayF | rm_work: anything else has been forgotten in my grey matter :D | 22:09 |
| rm_work | Ok I was looking for the bug on the neutron board, so that’s why I didn’t find it I guess | 22:09 |
| JayF | it's still "new" in neutron, looks like they never did a bug intake on it | 22:10 |
| JayF | not that it was actionable by them, really | 22:10 |
| rm_work | Whelp. | 22:12 |
| rm_work | This has been messing with us for months I think | 22:13 |
| rm_work | Been trying to figure out our DHCP issues | 22:13 |
| JayF | well go upgrade your local friendly dnsmasq | 22:13 |
| JayF | and consider investing in implementing another dhcp implementation for neutron's agent, potentially | 22:13 |
| JayF | we (GR-OSS) looked at kea but abandoned the project when we realized all the good APIs were locked behind a paywall | 22:14 |
| rm_work | So… at Yahoo, we (specifically one guy not me) wrote a drop in replacement that worked amazingly | 22:15 |
| rm_work | But they didn’t open source it and I have been trying to get them to do it but don’t have good contacts there anymore | 22:15 |
| JayF | that was an isc-dhcp-server implementation which we thought about upstreaming but didn't because isc-dhcp-server is no longer supported | 22:15 |
| rm_work | The Python-trio one? | 22:15 |
| JayF | (ISC DHCP has so long unsupported that *I* was the one who was looking at upstreaming this during my small window that) | 22:16 |
| JayF | the one in place when I was at the big purp was a driver that talked to isc-dhcp-server aka original dhcpd | 22:16 |
| rm_work | I kinda thought you had left already heh | 22:16 |
| JayF | if it's python-trio, I know who wrote the "new" one you're talking about | 22:16 |
| JayF | and upstream likely doesn't want it | 22:16 |
| rm_work | I believe it was Zach lol | 22:16 |
| rm_work | It really did work fantastically | 22:17 |
| JayF | if it was written in the usual style of his stuff, adapting it to proper openstack style would be a heavy lift | 22:17 |
| rm_work | Likely true | 22:17 |
| rm_work | But damn it worked well :/ | 22:17 |
| JayF | sure, lots of things are unmaintainable and OK as long as they never need the maintenance lol | 22:17 |
| rm_work | Well, appreciate your response, we had been kinda tearing our hair out a little | 22:19 |
| rm_work | Granted our next step was to try an upgrade anyway but at least now we know why it was broken | 22:20 |
| JayF | there was a short window of dnsmasq with a lot of memory-crashy issues | 22:21 |
| JayF | right around a refactor to eliminate those kinda issues :D | 22:21 |
| JayF | as those things go, of course | 22:21 |
| rm_work | lol whelp | 22:23 |
| rm_work | So Ubuntu has no version above 2.9.0 apparently | 22:24 |
| JayF | that ticket has all the details about sru and how they fixed it, I'm not an ubuntu dude but I know the bug didn't express in our CI with the sru version they asked me to test | 22:24 |
| rm_work | Guess we’re just gonna build it | 22:29 |
| haleyb | there is version 2.91-1 in plucky and questing, 2.90-2 in noble | 22:54 |
| haleyb | $ rmadison dnsmasq | 22:58 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!