| opendevreview | liuxie proposed openstack/neutron master: Prevent internal IP change for floating IP https://review.opendev.org/c/openstack/neutron/+/889791 | 03:16 |
|---|---|---|
| opendevreview | liuxie proposed openstack/neutron master: Prevent internal IP change for floating IP https://review.opendev.org/c/openstack/neutron/+/889791 | 03:20 |
| opendevreview | morice proposed openstack/neutron master: [ovn] AZs distribution in L3 port scheduler https://review.opendev.org/c/openstack/neutron/+/892604 | 07:28 |
| opendevreview | Lajos Katona proposed openstack/neutron-dynamic-routing master: DNM: Test patch only https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/890806 | 07:36 |
| ralonsoh | haleyb, hey! Do you mind checking these two patches? | 08:14 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/888769/ | 08:15 |
| ralonsoh | https://review.opendev.org/c/openstack/neutron/+/888776 | 08:15 |
| ralonsoh | (in order). Both are fixing the issue we currentlty have with live migration in OVN with trunk ports | 08:15 |
| ralonsoh | testing patch: https://review.opendev.org/c/openstack/tempest/+/888770 | 08:15 |
| opendevreview | morice proposed openstack/neutron master: [ovn] AZs distribution in L3 port scheduler https://review.opendev.org/c/openstack/neutron/+/892604 | 08:18 |
| opendevreview | yatin proposed openstack/neutron-tempest-plugin master: [DNM] Test ovs latest https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/892766 | 08:42 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Use the new network HA parameter https://review.opendev.org/c/openstack/neutron/+/881742 | 08:43 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Fix race condition when creating two routers without HA network https://review.opendev.org/c/openstack/neutron/+/881826 | 08:43 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/885111 | 09:12 |
| fnordahl | lajoskatona: thanks alot for taking the time to test and review. Have not looked at the whole end-to-end thing in a while as it's been in review for so long, will check. It is my understanding that those extensions are supposed to be truly virtual and require no configuration but let's see whats up | 09:13 |
| lajoskatona | fnordahl: ok, please check it, if on top of the whole series there could be a doc or user config patch would be even better | 09:24 |
| fnordahl | lajoskatona: yes, we do owe docs for sure | 09:26 |
| opendevreview | Merged openstack/neutron-dynamic-routing master: Install os-ken from git repo https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/892450 | 09:43 |
| opendevreview | Merged openstack/neutron master: [OVN] Disable the mcast_flood_reports option for LSPs https://review.opendev.org/c/openstack/neutron/+/888127 | 10:15 |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron stable/2023.1: [OVN] Disable the mcast_flood_reports option for LSPs https://review.opendev.org/c/openstack/neutron/+/892772 | 10:41 |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron stable/zed: [OVN] Disable the mcast_flood_reports option for LSPs https://review.opendev.org/c/openstack/neutron/+/892773 | 10:41 |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron stable/yoga: [OVN] Disable the mcast_flood_reports option for LSPs https://review.opendev.org/c/openstack/neutron/+/892774 | 10:42 |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron stable/xena: [OVN] Disable the mcast_flood_reports option for LSPs https://review.opendev.org/c/openstack/neutron/+/892775 | 10:42 |
| opendevreview | Lucas Alvares Gomes proposed openstack/neutron stable/wallaby: [OVN] Disable the mcast_flood_reports option for LSPs https://review.opendev.org/c/openstack/neutron/+/892776 | 10:42 |
| opendevreview | Merged openstack/neutron master: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/891232 | 11:46 |
| opendevreview | Michal Arbet proposed openstack/neutron stable/yoga: Spread OVN metadata agent heartbeat response in time https://review.opendev.org/c/openstack/neutron/+/892745 | 12:07 |
| opendevreview | Michal Arbet proposed openstack/neutron stable/yoga: Send ovn heatbeat more often. https://review.opendev.org/c/openstack/neutron/+/892746 | 12:07 |
| opendevreview | Michal Arbet proposed openstack/neutron stable/yoga: Send ovn heatbeat more often. https://review.opendev.org/c/openstack/neutron/+/892746 | 12:13 |
| opendevreview | Merged openstack/os-ken master: [CI] Run a ml2/ovs job https://review.opendev.org/c/openstack/os-ken/+/892452 | 12:41 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == Add a "port" child table "porthardwareoffloadtype" https://review.opendev.org/c/openstack/neutron/+/882832 | 12:54 |
| *** TheJulia is now known as needs_brains_and_sleep | 13:04 | |
| *** needs_brains_and_sleep is now known as TheJulia | 13:23 | |
| haleyb | ralonsoh: looking... | 13:56 |
| ralonsoh | thanks | 13:56 |
| ralonsoh | (no rush) | 13:56 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: WIP == Add a "port" child table "porthardwareoffloadtype" https://review.opendev.org/c/openstack/neutron/+/882832 | 13:59 |
| TheJulia | Any neutron folks around who know ovn and who might be up for a mystery of why NAT suddenly gets used again in my devstack? | 14:32 |
| TheJulia | ... even though it is disabled. | 14:32 |
| ralonsoh | TheJulia, sorry, can you provide more context? | 14:33 |
| TheJulia | So, I have to turn off NAT for OVN because I can't pass TFTP packets | 14:34 |
| TheJulia | And I've done this on the test job I'm working on to verify ironic can use Neutron OVN | 14:34 |
| TheJulia | And, works wonderfully on two jobs, but then I have another job where we run instance rescue, which rebinds the network ports | 14:34 |
| TheJulia | and suddenly, we have MTU packet drops because NAT is suddenly back on | 14:35 |
| TheJulia | and OVN can't handle the inbound packet being too large | 14:35 |
| * TheJulia is also curious where 1430 bytes as a default MTU is coming from when the public bridge mtu in devstack is set to 1400 bytes | 14:35 | |
| TheJulia | https://f65b0483f01fbf97cb5c-1988f1bc3d637497f7692396b58d77ce.ssl.cf2.rackcdn.com/885087/49/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool/0990cd1/controller/logs/apache/ipxe_access_log.txt | 14:36 |
| TheJulia | you can see the IP changes, but it is the same vif on the private network | 14:36 |
| TheJulia | NAT is disabled by the devstack plugin in the change set | 14:36 |
| ralonsoh | TheJulia, what does it means turn off NAT? | 14:36 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/885087/49/devstack/lib/ironic#2241 | 14:38 |
| TheJulia | in that case, that is the router for the private network | 14:38 |
| TheJulia | in the linked apache logs, the 172 addresses are from the same instance with the vif detached/reattached from the first download of the same files | 14:40 |
| ralonsoh | TheJulia, do you happen to know at what point, in this script, the "enable_snat" flag is restored? | 14:40 |
| TheJulia | it is not restored in the script | 14:40 |
| TheJulia | and if we create a new vif on the network, later on you can see this in the apache log, there is no snat occuring | 14:41 |
| ralonsoh | TheJulia, let me find a way to reproduce it | 14:42 |
| ralonsoh | TheJulia, did you create a LP Neutron bug? | 14:42 |
| TheJulia | ralonsoh: not yet, I can't figure out how it has changed to begin with | 14:42 |
| TheJulia | it really makes no sense since I can confirm that the vif is always on the private network | 14:42 |
| TheJulia | it is frankly, the most bizar thing I've seen this week | 14:43 |
| TheJulia | I also need to look at the public bridge logic, I guess something has changed there since we're getting different defaults when we create networks | 14:45 |
| TheJulia | but that is aside from the NAT | 14:45 |
| ralonsoh | I'm trying to find something in the neutron logs right now | 14:46 |
| TheJulia | If i didn't have an explicit route to ensure my packets didn't get silently dropped due to the MTU, i would have never discovered the source natting reappearing | 14:46 |
| TheJulia | (that route just clamps the mtu/mss) | 14:46 |
| ralonsoh | TheJulia, sorry, I only see the command when the router GW is defined with 'enable_snat': False. And we execute the NAT rule deletion | 14:53 |
| ralonsoh | but then I see no other operation restoring it | 14:54 |
| ralonsoh | TheJulia, ups, hold on | 14:56 |
| ralonsoh | Aug 24 22:09:02.321100 np0035045764 neutron-server[112824]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-bd0cad9d-cfec-4fca-a502-cf9bf46eab43 None None] Running txn n=1 command(idx=0): AddNATRuleInLRouterCommand(_result=None, lrouter=neutron-cbfcbf42-6602-4e5d-95b4-60d5a50556f7, columns={'type': 'dnat_and_snat', 'logical_ip': '10.1.0.12', 'external_ip': '172.24.5.109', 'logical_port': '55e083e3-35a6-4a1f-a24b | 14:56 |
| ralonsoh | -d45308ae06c7', 'external_ids': {'neutron:fip_id': '982560e7-a6f3-4928-b090-6d72caa488e9', 'neutron:revision_number': '0', 'neutron:fip_port_id': '55e083e3-35a6-4a1f-a24b-d45308ae06c7', 'neutron:router_name': 'neutron-cbfcbf42-6602-4e5d-95b4-60d5a50556f7', 'neutron:fip_external_mac': 'fa:16:3e:4b:ad:6c', 'neutron:fip_network_id': '1def3279-3954-4727-9567-a141d2731047'}}) {{(pid=112824) do_commit /opt/stack/data/venv/ | 14:56 |
| ralonsoh | lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} | 14:56 |
| ralonsoh | does it match with the IP, private IP and router? | 14:56 |
| TheJulia | uhhhh hmm | 14:57 |
| TheJulia | I believe that ip/port matches | 14:57 |
| TheJulia | but we only detached and reattached the vif | 14:57 |
| ralonsoh | because this is actually restoring the previous nat rule deletion | 14:57 |
| TheJulia | and updated dhcp info | 14:57 |
| TheJulia | *how* ?! | 14:57 |
| TheJulia | that seems like a bug, tbh | 14:57 |
| ralonsoh | yeah, how? | 14:57 |
| ralonsoh | most probably | 14:57 |
| ralonsoh | I'll try to find out when/how are we "restoring" that value | 14:58 |
| TheJulia | okay | 14:58 |
| TheJulia | Thanks! I'm here basically all day but having to step away periodically since my wife is sick | 14:59 |
| ralonsoh | I've created an initial LP bug to track this problem | 15:02 |
| TheJulia | ralonsoh: oh, cool, that actually sort of unblocks me already! :) | 15:04 |
| racosta | ralonsoh, about this reported snat problem, from the log you posted the port seems to have a FIP configured, right? If so, with or without enable_snat on the router a dnat_and_snat type rule will be created and all traffic from/to this Private IP would be translated with 1:1 NAT mapping... The DNAT/SNAT rule for FIP is applied directly in the OVN controller on the chassis (openflow flows level). | 15:42 |
| ralonsoh | racosta, with enable_snat=False, Neutron removes the NAT rule for the private network to the GW IP | 15:49 |
| ralonsoh | logical_ip=10.1.0.0/26, external_ip=172.24.5.95 | 15:49 |
| ralonsoh | the FIP creates a 1:1 NAT rule | 15:49 |
| ralonsoh | 'logical_ip': '10.1.0.12', 'external_ip': '172.24.5.109' | 15:49 |
| ralonsoh | and yes, that is handled in the controller chassis or in the compute node, if DVR is enabled | 15:50 |
| frickler | I don't grasp the whole scenario yet, but maybe using address scope(s) to avoid nat+fips completely might be a better approach? | 15:52 |
| ralonsoh | that should work fine as is, the problem (maybe) is this new FIP added using the fixed IP of an ironic port | 15:55 |
| ralonsoh | I don't know why this FIP is created | 15:55 |
| TheJulia | so the fip originally gets created by the tempest job for an instance | 16:00 |
| ralonsoh | this FIP is matching the fixed IP previously used in the ipxe calls | 16:01 |
| TheJulia | it is a flat network | 16:01 |
| ralonsoh | it doesn't matter the network type | 16:02 |
| ralonsoh | the FIP is adding the NAT rule for 'logical_ip': '10.1.0.12', 'external_ip': '172.24.5.109' | 16:02 |
| TheJulia | the fip gets created by the tempest job to create the ability to ssh in | 16:02 |
| TheJulia | so fips *always* create nat rules now? | 16:02 |
| TheJulia | always create SNAT that is | 16:02 |
| ralonsoh | yes, 1:1 rules | 16:02 |
| TheJulia | was this the case with OVS in any configuration? | 16:03 |
| ralonsoh | the GW SNAT option is something like | 16:03 |
| ralonsoh | logical_ip=10.1.0.0/26, external_ip=172.24.5.95 | 16:03 |
| ralonsoh | network:GW IP | 16:03 |
| ralonsoh | TheJulia, what do you mean? | 16:03 |
| TheJulia | well, this worked just fine with OVS in configurations. Maybe it silently just sort of worked and was inherently broken, or it was not NAT'ed. I'll need to go dig through another jobs logs to be sure | 16:04 |
| TheJulia | but the bottom line ends up being that we can't pass TFTP traffic then | 16:05 |
| ralonsoh | no if the ftpf traffic cannot be SNATed | 16:05 |
| TheJulia | yeah, that shoots ironic in the foot hard | 16:05 |
| TheJulia | like... approaching unusable feature set territory | 16:05 |
| TheJulia | with OVN if that is the case | 16:05 |
| TheJulia | and we know operators use it today | 16:05 |
| ralonsoh | but I think the current behaviour is correct | 16:05 |
| ralonsoh | if you create a FIP, you are asking to NAT a fixed IP into a public IP | 16:06 |
| TheJulia | pulling up an ovs version of that job to look at the logs | 16:06 |
| TheJulia | okay, so we could nat that traffic with OVS and did in that case | 16:08 |
| TheJulia | the fact tftp just can't be forwarded back with OVN is... wow | 16:08 |
| TheJulia | That *really* hurts us | 16:08 |
| ralonsoh | so the issue is not the NAT but how OVN handles tftp traffic | 16:09 |
| TheJulia | well, how OVN handles tftp traffic ended up driving me to disable snat for the CI job to begin with | 16:09 |
| TheJulia | which should also *just work* | 16:09 |
| TheJulia | and it does, but I'm sort of back to square zero then | 16:10 |
| TheJulia | or I move forward, begin documenting, and we figure out a new plan | 16:10 |
| TheJulia | I'm guessing folks have been turning off OVN, this might actually explain some other things I've brushed up against in the past | 16:10 |
| ralonsoh | TheJulia, on Monday I'll ping OVN core folks about this issue | 16:11 |
| TheJulia | err, not turning off ovn, but turning off fip usage | 16:11 |
| TheJulia | ralonsoh: that is much appreciated, I guess ultimately no is an answer, just... let me know what they say :\ | 16:11 |
| ralonsoh | for sure | 16:11 |
| * TheJulia opens up her ovn-documentation.rst file and begins to add a new entry on fip usage :) | 16:12 | |
| TheJulia | ralonsoh: thanks for the assistance! | 16:12 |
| racosta | ralonsoh, regarding we talked about in Vancouver (OVN 22.03 compatibility with Neutron Ussuri - 16.x)... with recent OVN northd patch applied from the ovs-discuss that I sent the link yesterday (northd recompute), and a single backport in the Neutron bump revision number - to solve the some loop issue. We are running load tests with more than 5k router and 8k VMs in our OpenStack "Ussuri" Lab... maybe that changes a little bit the idea of | 17:29 |
| racosta | scaling number that we have of OVN with OpenStack/Neutron. | 17:29 |
| opendevreview | Merged openstack/neutron master: [OVN] Skip the port status UP update during a live migration https://review.opendev.org/c/openstack/neutron/+/888769 | 18:01 |
| opendevreview | Merged openstack/neutron master: [OVN][Trunk] Set the subports correct host during live migration https://review.opendev.org/c/openstack/neutron/+/888776 | 18:01 |
| ralonsoh | racosta, I'll check this tomorrow, I need to leave now, sorry | 18:18 |
| opendevreview | Miro Tomaska proposed openstack/neutron stable/2023.1: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/892748 | 18:24 |
| opendevreview | Miro Tomaska proposed openstack/neutron stable/zed: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/892749 | 18:25 |
| opendevreview | Miro Tomaska proposed openstack/neutron stable/yoga: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/892750 | 18:25 |
| opendevreview | Miro Tomaska proposed openstack/neutron stable/xena: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/892751 | 18:25 |
| opendevreview | Miro Tomaska proposed openstack/neutron stable/wallaby: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/892752 | 18:25 |
| *** tbachman_ is now known as tbachman | 18:31 | |
| opendevreview | Jakub Libosvar proposed openstack/neutron master: Forbid updating vnic type on a bound port https://review.opendev.org/c/openstack/neutron/+/892815 | 20:33 |
| opendevreview | Miro Tomaska proposed openstack/neutron master: [DHCP agent] Add route to OVN metadata port if exists https://review.opendev.org/c/openstack/neutron/+/886988 | 21:13 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!