| opendevreview | Miguel Lavalle proposed openstack/neutron-tempest-plugin master: Test metadata query over IPv6 only network with OVS and LB https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926503 | 00:40 |
|---|---|---|
| opendevreview | Liushy proposed openstack/neutron master: [OVN] Support address group for ovn driver https://review.opendev.org/c/openstack/neutron/+/851509 | 03:19 |
| opendevreview | Liushy proposed openstack/neutron master: [OVN] Support address group for ovn driver https://review.opendev.org/c/openstack/neutron/+/851509 | 03:24 |
| ralonsoh | ykarel, hello! Are you around? I'm trying to debug https://review.opendev.org/c/openstack/neutron/+/924317, in particular one of the tests that is recurrently failing with these patches | 08:08 |
| ralonsoh | test_established_tcp_session_after_re_attachinging_sg | 08:08 |
| ralonsoh | checking the tempest logs, the issue seems to be in the first check, when the port is initially configured with the SSH rule and the test port (6666) | 08:08 |
| ralonsoh | but tempest cannot SSH into the VM | 08:08 |
| ralonsoh | well, no, that's not correct: it can SSH but cannot ping to port 6666 | 08:10 |
| ykarel | Hi ralonsoh | 08:10 |
| opendevreview | yatin proposed openstack/neutron master: Revert "Temporary mark ovs-rally job as non-voting" https://review.opendev.org/c/openstack/neutron/+/926613 | 08:10 |
| ralonsoh | btw, I can see the OVS agent applying the 6666 port rule | 08:12 |
| ralonsoh | ykarel, I'm checking https://6858d98d7a501e0358b0-54baced7b3de4d4a8f29448d2e5adbbf.ssl.cf2.rackcdn.com/925376/3/check/neutron-tempest-plugin-openvswitch-enforce-scope-old-defaults/9ba0ee0/controller/logs/index.html in particular, but I've seen the same problem in other cases | 08:15 |
| ralonsoh | the port related to these SG rules is created and deleted inmediatly | 08:15 |
| ykarel | ralonsoh, so that test failing consistantly in those patches? | 08:15 |
| ralonsoh | not constantly but very frequently | 08:15 |
| ykarel | as i recall seeing that fail in past but not that frequent | 08:16 |
| ralonsoh | not all ml2/ovs jobs are always failing but there is always one or two with this error | 08:16 |
| ralonsoh | yeah, me too, but with this patch this problem seems to be "improved" | 08:16 |
| ykarel | ohkk now i see it rebased on https://review.opendev.org/c/openstack/neutron/+/925376 | 08:19 |
| ykarel | which i had put -W couple of days back | 08:19 |
| ykarel | as the failure looked related as failures were quite frequent in that patch | 08:19 |
| ralonsoh | yes, right | 08:19 |
| ykarel | while the jobs were mostly stable | 08:19 |
| ykarel | https://review.opendev.org/c/openstack/neutron/+/925376/comments/dd6ad1ac_7aee4f6b | 08:20 |
| ralonsoh | yes, I saw this comment | 08:20 |
| ralonsoh | and I really don't know why this is happening with this patch | 08:20 |
| ralonsoh | and specifically with OVS (this is a change that affects the Neutron API wsgi module) | 08:21 |
| ralonsoh | but is not affecting ML2/OVN (with wsgi) | 08:21 |
| ralonsoh | as commented, the problem is always the same, checking the OVS agent logs: the VM port is created and right after deleted (I don't know why) | 08:23 |
| ykarel | @ralonsoh, me neither know how that patch is linked to the failures, i had just concluded based on pattern and didn't want to get it merged and have wider impact across patches unless and until we sure that's not related | 08:38 |
| ykarel | will check logs post lunch to see if can spot something | 08:38 |
| ralonsoh | for sure, we can't merge it right now | 08:38 |
| ralonsoh | I'm pinging a Nova folk to find out why n-compute is deleting this port (that is happening in other executions too) | 08:38 |
| ykarel | +1 | 08:38 |
| ralonsoh | ykarel, hey, only if you have time. I'm checking https://32db135de77b6f5b2e24-00cf68680e4576901ac5e284377f47f2.ssl.cf2.rackcdn.com/924317/7/experimental/neutron-tempest-plugin-openvswitch-distributed-dhcp/43f8b60/controller/logs/screen-q-agt.txt. The port "new" event is received at 12:41:15 | 09:12 |
| ralonsoh | Aug 19 12:41:15.653623 np0038215967 neutron-openvswitch-agent[61294]: DEBUG neutron.agent.common.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: {"data":[["18eba42b-2254-46a1-aeb8-2027e39235ed","old",null,257,null],["","new","tap9bc40eec-ae",-1,["map",[["attached-mac","fa:16:3e:ec:7b:05"],["iface-id","9bc40eec-ae67-4b56-9e75-551609520b | 09:12 |
| ralonsoh | 27"],["iface-status","active"],["vm-uuid","4623b509-a848-41a4-9fc2-1cf8227bd4ca"]]]]],"headings":["row","action","name","ofport","external_ids"]} {{(pid=61294) _read_stdout /opt/stack/neutron/neutron/agent/common/async_process.py:285}} | 09:12 |
| ralonsoh | but nova created the VM more than 1 minute before and decides to delete it at 12:41:14 | 09:13 |
| ralonsoh | Aug 19 12:41:14.998800 np0038215967 devstack@n-api.service[58447]: DEBUG nova.compute.api [None req-54123910-92d9-491b-b845-f8e7d37490dd tempest-StatefulNetworkSecGroupTest-1423213631 tempest-StatefulNetworkSecGroupTest-1423213631-project-member] [instance: 4623b509-a848-41a4-9fc2-1cf8227bd4ca] Going to try to terminate instance {{(pid=58447) delete /opt/stack/nova/nova/compute/api.py:2725}} | 09:13 |
| ralonsoh | Aug 19 12:41:15.024110 np0038215967 devstack@n-api.service[58446]: DEBUG nova.policy [None req-1e19f865-7f99-4da7-ae46-4965467a20d2 tempest-TrunkTest-1398874509 tempest-TrunkTest-1398874509-project-member] Policy check for os_compute_api:os-extended-server-attributes failed with credentials {'is_admin': False, 'user_id': 'e318cd2f500f41249a5a2f13772dc59b', 'user_domain_id': 'default', 'system_scope': None, | 09:13 |
| ralonsoh | 'domain_id': None, 'project_id': '7f724d856c9d4a2ab98d13c20f696d1e', 'project_domain_id': 'default', 'roles': ['reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=58446) authorize /opt/stack/nova/nova/policy.py:203}} | 09:13 |
| ralonsoh | so for some reason, we receive the port creation event just at the same time Nova decides to delete it because it never received the vif-plugged-event | 09:14 |
| ralonsoh | (next time I'll use https://paste.opendev.org/) | 09:14 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-tempest-plugin master: DNM == Wait for the ML2/OVS FW conntrack rules to be deleted https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926621 | 11:15 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Monkey patch the system libraries before calling them https://review.opendev.org/c/openstack/neutron/+/925376 | 11:15 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron-tempest-plugin master: DNM == Wait for the ML2/OVS FW conntrack rules to be deleted https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926621 | 11:23 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: DNM == Test neutron-tempest-plugin-openvswitch https://review.opendev.org/c/openstack/neutron/+/925205 | 11:23 |
| damiandabrowski | ralonsoh: I looked at https://review.opendev.org/c/openstack/neutron/+/907504 and I think it's unrelated to my issues :/ | 11:33 |
| damiandabrowski | I added OVN nat rules for 0.0.0.0/0 on my routers but it didn't help. | 11:33 |
| damiandabrowski | I also created two, small all-in-one environments(one with OVN and another one with OVS) with even simpler architecture: | 11:34 |
| damiandabrowski | https://i.ibb.co/gzjd604/Screenshot-from-2024-08-20-13-26-55.png | 11:34 |
| damiandabrowski | For OVN: vm-inner-* do not have internet connectivity, vm-outer-* have internet connectivity. It's also a bit weird that gateway for inner-router is down. | 11:34 |
| damiandabrowski | For OVS: all VMs have internet connectivity and all neutron ports are up | 11:34 |
| ralonsoh | damiandabrowski, 0.0.0.0/0 won't work, you need to add one by one the inner networks to the external OVN router | 11:38 |
| ralonsoh | as NAT entries | 11:38 |
| ralonsoh | did you use this patch in your OVN deployment? | 11:38 |
| damiandabrowski | ahh ok, i'll try to add NAT entries manually(for inner network, not 0.0.0.0/0) and if it won't work, I'll try to use above patch to make sure it's unrelated | 11:40 |
| ralonsoh | damiandabrowski, why don't you use this patch? | 11:40 |
| ralonsoh | that will be faster | 11:40 |
| damiandabrowski | yeah...you may be right, let me try to apply the patch right away | 11:41 |
| damiandabrowski | ralonsoh: I applied the patch, enabled ovn.ovn_router_indirect_snat, restarted all neutron and ovn services, executed OVN DB sync tool in repair mode and even recreated all routers/networks - no extra NAT rules were added | 12:31 |
| damiandabrowski | I'm currently trying to find out why | 12:31 |
| opendevreview | yatin proposed openstack/neutron stable/2024.1: Adopt to StandardAttribute load method change to "selectin" https://review.opendev.org/c/openstack/neutron/+/926629 | 12:33 |
| opendevreview | yatin proposed openstack/neutron stable/2023.2: Adopt to StandardAttribute load method change to "selectin" https://review.opendev.org/c/openstack/neutron/+/926631 | 12:36 |
| opendevreview | yatin proposed openstack/neutron stable/2023.1: Adopt to StandardAttribute load method change to "selectin" https://review.opendev.org/c/openstack/neutron/+/926633 | 12:37 |
| *** jamesdenton is now known as Guest972 | 13:25 | |
| *** jamesdenton_alt is now known as jamesdenton | 13:25 | |
| damiandabrowski | ralonsoh: okay, so I noticed that patch https://review.opendev.org/c/openstack/neutron/+/907504 works perfectly fine but for other scenarios | 13:41 |
| damiandabrowski | like the one described here: https://bugs.launchpad.net/neutron/+bug/1386041 | 13:41 |
| damiandabrowski | where "inner" router does not have default gateway and static routes are defined on both routers | 13:41 |
| damiandabrowski | in my case, both of my routers have default gateways, I don't define any static routes and technically I'm not doing nested NAT | 13:41 |
| damiandabrowski | so I think this patch should not be required in my case(and applying it does not help) | 13:41 |
| damiandabrowski | but there must be some other difference between OVS and OVN that breaks my architecture :/ | 13:42 |
| haleyb | damiandabrowski: that patch is being updated, it does not work exactly right | 13:43 |
| ralonsoh | damiandabrowski, then I don't know what is your particular case, please open a LP bug with a reproducer, including the networks, subnets, routers and extra routes added | 13:43 |
| damiandabrowski | haleyb: ahh ok, thanks for the info! For now I just want to make sure that I'm really affected by this issue or I need to dig somewhere else | 13:59 |
| damiandabrowski | ralonsoh: okok, done: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/2077430 | 13:59 |
| haleyb | #startmeeting networking | 14:00 |
| opendevmeet | Meeting started Tue Aug 20 14:00:33 2024 UTC and is due to finish in 60 minutes. The chair is haleyb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'networking' | 14:00 |
| haleyb | Ping list: bcafarel, elvira, frickler, mlavalle, mtomaska, obondarev, slaweq, tobias-urdin, ykarel, lajoskatona, jlibosva, averdagu, amotoki, haleyb, ralonsoh | 14:00 |
| mlavalle | \o | 14:00 |
| damiandabrowski | I also have all-in-one environment with described setup for both OVS and OVN, I can let you in if it can help. | 14:00 |
| ykarel | o/ | 14:00 |
| ihrachys | o/ | 14:00 |
| ralonsoh | hello | 14:00 |
| slaweq | o/ | 14:00 |
| bcafarel | o/ | 14:00 |
| obondarev | o/ | 14:00 |
| elvira | o/ | 14:01 |
| haleyb | #topic announcements | 14:01 |
| haleyb | We are now in Dalmatian release week (R - 6) | 14:01 |
| haleyb | Work on libraries should be wrapping up! | 14:02 |
| haleyb | Non-client library freeze: August 22nd, 2024 (R-6 week) | 14:02 |
| haleyb | Client library freeze: August 29th, 2024 (R-5 week) | 14:02 |
| mtomaska | o/ | 14:02 |
| haleyb | Dalmatian-3 milestone: August 29th, 2024 (R-5 week) | 14:02 |
| haleyb | With all that said, are there any neutron-lib changes that need attention and merging? | 14:02 |
| haleyb | https://review.opendev.org/c/openstack/neutron-lib/+/924700 perhaps? | 14:03 |
| ralonsoh | this is not a hard requirement | 14:03 |
| ralonsoh | but that will be perfect if merged | 14:03 |
| haleyb | ack | 14:04 |
| haleyb | ihrachys: there is this older one you have - https://review.opendev.org/c/openstack/neutron-lib/+/909044 | 14:04 |
| haleyb | it had some comments | 14:05 |
| ihrachys | forgot about it. I don't think it's important release wise anyway. | 14:05 |
| haleyb | if there are others please ping for reviews as the release patches will start coming | 14:06 |
| haleyb | ihrachys: ack, just trying to get things merged | 14:06 |
| slaweq | ralonsoh I just approved Your patch | 14:06 |
| ralonsoh | slaweq++ | 14:07 |
| haleyb | and i'll copy/paste my comment from last week | 14:07 |
| haleyb | Since we are at the end of the cycle, I would like to start using the priorities dashboard for patches in the "ready to merge" state. This could be older changes as well as new ones | 14:07 |
| mlavalle | +1 | 14:07 |
| slaweq | ++ | 14:07 |
| ykarel | +1 | 14:07 |
| haleyb | So if you think something is "ready to go" add an RP+1 so others can see it | 14:07 |
| haleyb | we have some old patches | 14:07 |
| haleyb | there is a link in the wiki, it's too long to paste | 14:08 |
| haleyb | and i think lajos added a comment about TaaS, which I don't think has RP settings | 14:08 |
| haleyb | but I'll add his work here if people have cycles | 14:09 |
| haleyb | #link https://review.opendev.org/q/topic:%22bug/2015471%22+status:open | 14:09 |
| haleyb | Tap Mirror for OVN | 14:09 |
| haleyb | and OVS | 14:09 |
| haleyb | and other comments or important things to share? | 14:10 |
| haleyb | I will mention again we are in the combined PTL/TC election cycle, there was an email to the -discuss list for those interested | 14:12 |
| slaweq | right, I was just going to say that | 14:12 |
| slaweq | thx haleyb :) | 14:12 |
| haleyb | As I mentioned, I will run again for PTL, I have the cycles :) | 14:14 |
| mlavalle | ++ | 14:14 |
| mlavalle | Thanks for your servicw | 14:14 |
| haleyb | #topic bugs | 14:15 |
| haleyb | I was the bug deputy last week, my report is at | 14:15 |
| haleyb | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/Y6B77PQKGGM2MZXBKCL73WQIIEEDBW2S/ | 14:15 |
| haleyb | and amazingly there was only one bug | 14:15 |
| slaweq | this time You were lucky with bugs :) | 14:15 |
| haleyb | very lucky | 14:15 |
| bcafarel | August 15 week, usually quiet time :) | 14:16 |
| haleyb | #link https://bugs.launchpad.net/neutron/+bug/2076916 | 14:16 |
| haleyb | IPv6 metadata issue with ml2/ovs and LB, mlavalle tracked down to gate setup | 14:16 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926503 | 14:17 |
| mlavalle | I have a comment regarding this bug | 14:17 |
| mlavalle | My first approach to fix it was to enable metadata in isolated networks (enable_isolated_metadata = true) | 14:18 |
| haleyb | that would fix the case where the dhcp-agent is providing the service | 14:19 |
| mlavalle | However, yesterday I discovered that while that fixed the IPv6 problem and allowed the new test case to pass, a couple of other unrelated test cases failed | 14:19 |
| mlavalle | so in the end I decided not to use an isolated network for the IPv6 test case and now everything seems to be good | 14:20 |
| mlavalle | however that let me to wonder whether we are testing isolated networks in neutron-tempest-plugin | 14:20 |
| mlavalle | and we are not | 14:21 |
| haleyb | ah, i just noticed you changed it to add a router interface, which is maybe better as it's like a "normal" network would be | 14:21 |
| mlavalle | yeah, now I use a router | 14:21 |
| mlavalle | but, isolated networks is something that we support, don't we | 14:21 |
| mlavalle | ? | 14:22 |
| ralonsoh | yes, using the DHCP namespace | 14:22 |
| haleyb | yes, fully supported, and a number of deployment tools set it | 14:22 |
| mlavalle | maybe we should be testing it in n-t-p | 14:22 |
| mlavalle | and yesterday I found that two test cases stareted to fail when I enebled isolated networks | 14:23 |
| haleyb | mlavalle: yes, more coverage would be good, probably would find some edge cases | 14:23 |
| mlavalle | in other words, maybe we are not paying attention enough and we regressing in this feature | 14:23 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron/+/926497 is one of those odd cases i've been working on | 14:23 |
| haleyb | mlavalle: we can either use the existing bug or open another to track the work | 14:24 |
| mlavalle | ok, I'm going to file a new LP with this discovery and I'm going to try to come up with a n-t-p test case | 14:24 |
| mlavalle | targeting specifically isolated networks | 14:25 |
| mlavalle | that's all | 14:25 |
| haleyb | mlavalle: thanks for fixing this and doing more testing | 14:25 |
| haleyb | any other bugs to discuss? | 14:26 |
| haleyb | #topic community-goals | 14:27 |
| haleyb | ralonsoh: do you want to talk about eventlet removal? i only quickly saw the related patches to track down the gate failure stopping your other patches from merging | 14:28 |
| ralonsoh | yes, I've spent more than one day debugging this | 14:28 |
| ralonsoh | and the only conclusion is that, in the 2 failing cases, we somehow are faster and don't allow the OVS agent to configure backend | 14:29 |
| ralonsoh | I'm testing with adding sleeps after the SG port updates | 14:29 |
| ralonsoh | I know is horrible but still testing | 14:29 |
| ralonsoh | that's all | 14:29 |
| haleyb | ralonsoh: thanks, and yes i understand just adding delays in tests is not great | 14:31 |
| haleyb | i will look at the reviews later | 14:31 |
| haleyb | the other community goal is neutronclient deprecation. i don't think lajos is here, but he has additional changes for horizon up | 14:32 |
| haleyb | #link https://review.opendev.org/q/topic:%22bug/1999774%22 | 14:32 |
| haleyb | i had no other community items | 14:33 |
| haleyb | #topic bugs | 14:33 |
| haleyb | i forgot in #bugs that this week rubasov is the deputy, next week is mlavalle | 14:34 |
| haleyb | is that good for both? | 14:34 |
| haleyb | although i don't thinkg rubasov is here today | 14:34 |
| ralonsoh | if he doesn't reply, I can do it this week | 14:34 |
| mlavalle | yes it is for me | 14:34 |
| mlavalle | rubasov said last week he is on PTO yesterday and today | 14:35 |
| mlavalle | but that he will catch up tomorrow | 14:35 |
| mlavalle | he was fine with this week | 14:35 |
| haleyb | great, thanks. oh right, he said he'll catch up in last weeks notes, and it's a slow week anyways | 14:36 |
| haleyb | ok, that's all the bug stuff i had | 14:36 |
| haleyb | #topic on-demand | 14:36 |
| haleyb | i had one topic, but will let any others go first | 14:36 |
| ralonsoh | nothing from me | 14:37 |
| haleyb | i just had a question on the update to the RBAC default override | 14:37 |
| mlavalle | nothing from me either | 14:37 |
| haleyb | #link https://review.opendev.org/c/openstack/neutron/+/926085 | 14:37 |
| haleyb | that patch seems fine, except for the designate scenario, but i didn't quite understand gmann's comment and if it means there was something broken | 14:38 |
| haleyb | slaweq: maybe if you can give your opinion in the review it would help move it along, i'm stumped | 14:40 |
| slaweq | sure, I will check it | 14:41 |
| haleyb | thanks, i'm sure he'd like to get that series all merged as it crosses many repos | 14:41 |
| haleyb | i had nothing else | 14:42 |
| haleyb | ykarel: CI meeting in person today? | 14:42 |
| ykarel | haleyb, yes video today | 14:42 |
| haleyb | great, that's in :17 for anyone attending | 14:43 |
| haleyb | thanks for coming, and don't forget to look at the priority dashboard :) | 14:43 |
| haleyb | and have a great week | 14:43 |
| haleyb | #endmeeting | 14:43 |
| opendevmeet | Meeting ended Tue Aug 20 14:43:37 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:43 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/networking/2024/networking.2024-08-20-14.00.html | 14:43 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/networking/2024/networking.2024-08-20-14.00.txt | 14:43 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/networking/2024/networking.2024-08-20-14.00.log.html | 14:43 |
| ralonsoh | bye | 14:43 |
| ykarel | o/ | 14:43 |
| ihrachys | haleyb: in nested snat experiments you did for 0.0.0.0/0 snat rule for ovn, when you had issues with fips; which ovn version did you run? | 14:44 |
| haleyb | ihrachys: i don't exactly know. I *thought* I tried to build from source at the time i sent the patch | 14:45 |
| ihrachys | haleyb: so like master? | 14:45 |
| haleyb | yes, it should have had that conntrack fix you mentioned for the issue | 14:46 |
| ihrachys | i'm being told something was broken circa 24.03 but not before. still need to check if earlier releases are good actually. | 14:46 |
| ihrachys | so I wanted to check if you tried with a pre 24.03. will do it myself of course. | 14:46 |
| haleyb | i just don't know what version, sorry | 14:47 |
| ihrachys | ack | 14:47 |
| ihrachys | haleyb: https://github.com/ovn-org/ovn/commit/40136a2f | 14:49 |
| mlavalle | haleyb: yeah, one of the failing test cases I saw yesteday was neutron_tempest_plugin.scenario.test_mtu.NetworkWritableMtuTest.test_connectivity_min_max_mtu. So it seems to be related to https://bugs.launchpad.net/neutron/+bug/2074207 | 14:52 |
| mlavalle | I'll investigate further before filing another LP. Good pointer, though. Thanks! | 14:53 |
| haleyb | ihrachys: interesting, i would have thought i had that based on the date but ??? | 14:55 |
| ihrachys | I'm told this commit actually breaks something in the scenario | 14:55 |
| ihrachys | so we're going to check if revert fixes it | 14:55 |
| ihrachys | in next day or two | 14:55 |
| haleyb | ah | 14:55 |
| haleyb | mlavalle: thanks for that, my next step of testing my fix for the dhcp-agent mtu issue was to look at the metadata agents and see if they need a similar fix | 14:56 |
| haleyb | basically, if just ipv4 subnets don't configure ipv6 metadata address, since the degenerate case of a small mtu could break it | 14:57 |
| ykarel | #startmeeting neutron_ci | 15:03 |
| opendevmeet | Meeting started Tue Aug 20 15:03:34 2024 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:03 |
| opendevmeet | The meeting name has been set to 'neutron_ci' | 15:03 |
| ykarel | Ping list: bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira | 15:03 |
| ykarel | This will be video meeting this time: https://meetpad.opendev.org/neutron-ci-meetings | 15:03 |
| ykarel | #topic Stable branches | 15:05 |
| ykarel | all good | 15:06 |
| ykarel | #topic Stadium projects | 15:06 |
| ykarel | rest all green apart from sfc https://bugs.launchpad.net/neutron/+bug/2068727 | 15:06 |
| ykarel | #action slawek to check for sfc failures in coming weeks and have some conclusion for dalmatian release | 15:11 |
| ykarel | #topic Rechecks | 15:12 |
| ykarel | #topic Tempest/Scenario | 15:12 |
| ykarel | pagination api test random failures | 15:12 |
| ykarel | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_34a/periodic/opendev.org/openstack/neutron/master/neutron-ovs-tempest-plugin-iptables_hybrid-nftables/34a2463/testr_results.html | 15:12 |
| ykarel | https://bugs.launchpad.net/neutron/+bug/2076328 | 15:12 |
| opendevreview | Terry Wilson proposed openstack/neutron unmaintained/yoga: ovn-metadata: Refactor events https://review.opendev.org/c/openstack/neutron/+/926655 | 15:14 |
| opendevreview | Terry Wilson proposed openstack/neutron unmaintained/yoga: Handle creation of Port_Binding with chassis set https://review.opendev.org/c/openstack/neutron/+/926656 | 15:14 |
| ykarel | https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926201 | 15:14 |
| ykarel | #topic Periodic | 15:16 |
| ykarel | post gres jobs with recent fixes now still randomly fails with | 15:16 |
| ykarel | AttributeError: 'NoneType' object has no attribute 'tags' | 15:16 |
| ykarel | - https://82f990e55ca70e41f871-36409e2f060733ae498eef8cd27f40f4.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-postgres-full/126119f/controller/logs/screen-q-svc.txt | 15:16 |
| ykarel | - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_360/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-postgres-full/3602d9e/controller/logs/screen-q-svc.txt | 15:16 |
| ykarel | #action ralonsoh to check failures in postgres job | 15:18 |
| ykarel | #topic Grafana | 15:18 |
| ykarel | https://grafana.opendev.org/d/f913631585/neutron-failure-rate | 15:18 |
| ykarel | #topic On Demand | 15:19 |
| opendevreview | Terry Wilson proposed openstack/neutron unmaintained/yoga: Handle creation of Port_Binding with chassis set https://review.opendev.org/c/openstack/neutron/+/926656 | 15:21 |
| ykarel | Neutron CI meeting time | 15:22 |
| ralonsoh | +1 thank you all | 15:22 |
| ykarel | #agreed to move CI meeting to Monday 2:00 UTC | 15:23 |
| ykarel | #endmeeting | 15:23 |
| opendevmeet | Meeting ended Tue Aug 20 15:23:24 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:23 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-08-20-15.03.html | 15:23 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-08-20-15.03.txt | 15:23 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/neutron_ci/2024/neutron_ci.2024-08-20-15.03.log.html | 15:23 |
| otherwiseguy | @haleyb: Just to prove that I promise I don't hate your customers. ;) https://review.opendev.org/c/openstack/neutron/+/926656 | 15:25 |
| otherwiseguy | I did a previous version that also backported kuba's refactor, but then remembered that originally when I proposed the dowstream patch, his refactor hadn't merged. So this was just the /1 revision--so it *should* work. | 15:27 |
| otherwiseguy | Also, we're making good progress on getting the 0.0.0.0/0 thing resolved with the OVN folks. It's looking like it'll be a backportable fix. | 15:27 |
| otherwiseguy | So just having code in neutron that uses your config option to toggle setting NAT.logical_ip=0.0.0.0/0 *hopefully* will work. | 15:28 |
| ralonsoh | ykarel, 1 min, if you have time | 15:34 |
| ralonsoh | https://95a57cfa3d60c01252ee-a75a90e4f62f1557c9429d321082a0da.ssl.cf5.rackcdn.com/925205/12/check/neutron-tempest-plugin-openvswitch-iptables_hybrid-4/346848e/testr_results.html | 15:34 |
| haleyb | otherwiseguy: hey, thanks! that was a much more compact patch than waht i had | 15:34 |
| ralonsoh | the command to remove the SG rule from the port is sent at 2024-08-20 13:00:59,045 | 15:34 |
| ralonsoh | the OVS agent starts processing it 20 secs later | 15:34 |
| otherwiseguy | haleyb: just took my brain a while to remember all of the details of putting the fix together. takes longer every day. ;) | 15:35 |
| ralonsoh | Aug 20 13:01:20.063875 np0038223743 neutron-openvswitch-agent[61344]: DEBUG neutron.agent.securitygroups_rpc [None req-65f7e1ba-e56f-4843-a499-232de9cbf47e tempest-StatefulNetworkSecGroupTest-48337440 tempest-StatefulNetworkSecGroupTest-48337440-project-member] Adding ['6589eb87-d607-41ec-9e54-8d8b906cfbc7'] devices to the list of devices for which firewall needs to be refreshed {{(pid=61344) _security_group_updated | 15:35 |
| ralonsoh | /opt/stack/neutron/neutron/agent/securitygroups_rpc.py:227}} | 15:35 |
| ralonsoh | that is an unacceptable delay, to be honest | 15:35 |
| haleyb | otherwiseguy: and regarding the snat thing, that's even better news, ihrachys did mention something about a possible revert, might mention to martin | 15:37 |
| ykarel | ralonsoh, yes that's bad :( | 15:45 |
| ralonsoh | ykarel, because we can increase the testing timeout, I' | 15:46 |
| ralonsoh | I'm going to document it and propose a patch | 15:46 |
| ralonsoh | (this american keyboard with the small enter is killing me) | 15:46 |
| ykarel | ralonsoh, but do you get why it started happening now? | 15:46 |
| ykarel | specific to this patch or something else? | 15:47 |
| ralonsoh | not really, but I'm not checking the Neutron API part, that is the only one affected by this patch | 15:47 |
| ralonsoh | I'm now* | 15:47 |
| ykarel | also there were couple of patches related to move to singlethread, those could be related? | 15:48 |
| ykarel | hmm but here in ovs agent log it doesn't look busy | 15:49 |
| ralonsoh | I found that there is a hiccup in the n-api os 20 seconds | 15:50 |
| ralonsoh | https://95a57cfa3d60c01252ee-a75a90e4f62f1557c9429d321082a0da.ssl.cf5.rackcdn.com/925205/12/check/neutron-tempest-plugin-openvswitch-iptables_hybrid-4/346848e/controller/logs/screen-neutron-api.txt | 15:50 |
| ralonsoh | from Aug 20 13:00:59.304551 | 15:50 |
| ralonsoh | to Aug 20 13:01:19.69590 there is nothing | 15:50 |
| ralonsoh | and this is just in the middle of the SG update | 15:50 |
| opendevreview | Merged openstack/neutron-lib master: Fix python modules docs build https://review.opendev.org/c/openstack/neutron-lib/+/925503 | 16:11 |
| opendevreview | Merged openstack/neutron-lib master: Add API extension ``tag-creation`` https://review.opendev.org/c/openstack/neutron-lib/+/924700 | 16:19 |
| haleyb | ralonsoh: regarding that 20 seconds, does that dbcounter entry tell us anything? SELECT=2155, etc? that should have been just after though, so i'm confused | 16:25 |
| opendevreview | Merged openstack/neutron master: Add tap_mirror to extension to OVN supported extensions https://review.opendev.org/c/openstack/neutron/+/905840 | 16:27 |
| haleyb | ralonsoh: there is also this entry in tempest_log.txt there around that time | 16:45 |
| haleyb | 2024-08-20 13:01:09.313 87890 INFO neutron_tempest_plugin.scenario.test_security_groups [-] Wait for conntrack invalid rules to be deleted | 16:45 |
| haleyb | but i can't find any code that prints that in codesearch :-/ | 16:45 |
| opendevreview | Brian Haley proposed openstack/neutron unmaintained/yoga: Handle creation of Port_Binding with chassis set https://review.opendev.org/c/openstack/neutron/+/926656 | 18:09 |
| opendevreview | Brian Haley proposed openstack/neutron unmaintained/zed: Handle creation of Port_Binding with chassis set https://review.opendev.org/c/openstack/neutron/+/926666 | 18:10 |
| opendevreview | Brian Haley proposed openstack/neutron unmaintained/zed: Handle creation of Port_Binding with chassis set https://review.opendev.org/c/openstack/neutron/+/926666 | 18:13 |
| otherwiseguy | @haleyb: it does look like reverting https://github.com/ovn-org/ovn/commit/40136a2f seems to fix FIP to FIP traffic using logical_ip=0.0.0.0/0. | 18:36 |
| otherwiseguy | Interestingly, the pings seem broken in 22.04, work in 23.09, and are broken again in main. :p | 18:39 |
| haleyb | otherwiseguy: thanks for the info, i'm going to ping martin now so he's aware, don't know if your plan is to revert or find a fix | 18:40 |
| otherwiseguy | haleyb: I'm not sure yet what their plan is either. But if your customers are on 22.04 by chance (like my devstack install on jammy) then it might work w/o any changes. | 18:41 |
| otherwiseguy | er, nm | 18:42 |
| otherwiseguy | if they were on 23.09. | 18:42 |
| otherwiseguy | ETOOMANYVERSIONS | 18:42 |
| otherwiseguy | getting ready to test to make sure that the nested pinging also works | 18:42 |
| haleyb | let me look, i know most are on 22.04.3 i believe | 18:43 |
| otherwiseguy | yeah, w/o the patch both fip2fip and nested outbound pinging (at least to the router ip 172.24.4.1 and fips) works. | 18:44 |
| haleyb | otherwiseguy: i pinged him he'll try and reproduce, thanks for looking into it | 21:03 |
| mlavalle | haleyb: would you nudge https://review.opendev.org/c/openstack/neutron/+/922264 over the edge, please? | 21:32 |
| haleyb | mlavalle: pushed | 21:34 |
| mlavalle | haleyb: :-) | 21:49 |
| opendevreview | Miguel Lavalle proposed openstack/neutron-tempest-plugin master: Test metadata query over IPv6 only network with OVS and LB https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926503 | 22:18 |
| opendevreview | Merged openstack/neutron master: Fix support of IPv6 only networks in OVN metadata agent https://review.opendev.org/c/openstack/neutron/+/922264 | 23:16 |
| opendevreview | Miguel Lavalle proposed openstack/neutron master: User defined router flavor driver with no LSP https://review.opendev.org/c/openstack/neutron/+/917800 | 23:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
