| *** EugenMayer440 is now known as EugenMayer44 | 02:10 | |
| opendevreview | liuxie proposed openstack/neutron master: [OVN] Support address group for ovn driver https://review.opendev.org/c/openstack/neutron/+/851509 | 06:27 |
|---|---|---|
| tore | haleyb: don't know, what do you think? I was thinking to maybe re-file as an issue in the OVN issue tracker on GitHub (I don't think it's an OVS bug, so I'm not sure why ovs-discuss would be appropriate). That said, if the OVN people that did the bad commit are also using Launchpad (as ihrachys_ suggested in his comment) then maybe it can stay there? Being a bit of a | 07:22 |
| tore | newcomer it can be difficult sometimes to identify where Neutron ends and OVN begins… :) | 07:22 |
| lajoskatona | fnordahl: Hi, I started to play with external-gateway-multihoming feature (https://review.opendev.org/q/topic:2023-aa-l3-gw-multihoming ) | 07:30 |
| lajoskatona | fnordahl: and to tell the truth I lost on the API, how I can set enable_default_route_ecmp and enable_default_route_bfd? | 07:30 |
| opendevreview | Slawek Kaplonski proposed openstack/neutron master: [S-RBAC] Add service role in neutron policy https://review.opendev.org/c/openstack/neutron/+/886724 | 07:30 |
| lajoskatona | fnordahl: by the API def I should see these flags for external_gateway_info and I should be able to set it for the external gws, and for the router as a whole also | 07:33 |
| opendevreview | Merged openstack/ovsdbapp stable/train: Remove ovsdbapp-tempest-dsvm-networking-ovn-ovs-release job definition https://review.opendev.org/c/openstack/ovsdbapp/+/892459 | 07:33 |
| lajoskatona | fnordahl: I have enable_default_route_bfd & enable_default_route_ecmp True in my neutron.conf, do I need anything else? | 07:38 |
| frickler | ralonsoh: moving the MTU discussion over here, someone just showed me https://bugzilla.redhat.com/show_bug.cgi?id=1833813#c3 which seems related, too, but I'm not sure what it says about which scenario should be supported or not | 07:38 |
| ralonsoh | frickler, yes, this BZ is related. I'm finishing a couple of tasks now but I've raised this question internally | 07:39 |
| ralonsoh | but yes, this is related | 07:39 |
| ralonsoh | frickler, btw, the LP bug is https://bugs.launchpad.net/neutron/+bug/2028651 | 07:39 |
| frickler | ralonsoh: the latter is a different issue, not mtu related? | 07:42 |
| ralonsoh | frickler, it is, the problem is that you can't read private comments | 07:43 |
| ralonsoh | frickler, actually that was an issue in the kernel | 07:44 |
| ralonsoh | from the BZ | 07:44 |
| ralonsoh | Private | 07:44 |
| ralonsoh | Description Roman Safronov 2020-05-10 19:57:56 UTC | 07:44 |
| ralonsoh | Description of problem: | 07:44 |
| ralonsoh | On OSP16.1 installed with default MTU settings sending UDP/ICMP packets from internal network to external with sizes higher than MTU results with 100% packet loss regardless of setting ovn_emit_need_to_frag=True in ml2. | 07:44 |
| ralonsoh | [heat-admin@controller-0 ~]$ sudo podman exec -it neutron_api crudini --get /etc/neutron/plugins/ml2/ml2_conf.ini ovn ovn_emit_need_to_frag | 07:45 |
| ralonsoh | True | 07:45 |
| ralonsoh | [heat-admin@controller-0 ~]$ sudo ovs-appctl -t ovs-vswitchd dpif/show-dp-features br-int | grep "Check pkt length action" | 07:45 |
| ralonsoh | Check pkt length action: Yes | 07:45 |
| ralonsoh | External network- flat, MTU 1500 | 07:45 |
| ralonsoh | Internal network geneve, MTU 1442 | 07:45 |
| ralonsoh | Made the following attempts: | 07:45 |
| ralonsoh | NETCAT (from VM on internal network to external network) | 07:45 |
| ralonsoh | $ cat 1400_bytes_file | nc -u 10.0.0.178 65000 | 07:45 |
| ralonsoh | File reached the external network successfully - OK | 07:45 |
| ralonsoh | $ cat 1600_bytes_file | nc -u 10.0.0.178 65000 | 07:45 |
| ralonsoh | File did not reach the external network - NOT OK | 07:45 |
| ralonsoh | PING - Size smaller than MTU, works - OK | 07:45 |
| ralonsoh | $ ping 10.0.0.178 -s 1400 | 07:45 |
| ralonsoh | PING 10.0.0.178 (10.0.0.178) 1400(1428) bytes of data. | 07:45 |
| ralonsoh | 1408 bytes from 10.0.0.178: icmp_seq=1 ttl=63 time=4.60 ms | 07:45 |
| ralonsoh | ^C | 07:45 |
| ralonsoh | --- 10.0.0.178 ping statistics --- | 07:45 |
| ralonsoh | 1 packets transmitted, 1 received, 0% packet loss, time 0ms | 07:45 |
| ralonsoh | rtt min/avg/max/mdev = 4.602/4.602/4.602/0.000 ms | 07:45 |
| ralonsoh | PING - Size bigger than MTU, does not work, regardless of -M dont, -M want settings - NOT OK | 07:45 |
| ralonsoh | $ ping 10.0.0.178 -s 1600 -M dont | 07:45 |
| ralonsoh | PING 10.0.0.178 (10.0.0.178) 1600(1628) bytes of data. | 07:45 |
| ralonsoh | From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) | 07:45 |
| ralonsoh | From 192.168.1.1 icmp_seq=2 Frag needed and DF set (mtu = 1482) | 07:45 |
| ralonsoh | ^C | 07:45 |
| ralonsoh | --- 10.0.0.178 ping statistics --- | 07:45 |
| ralonsoh | 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms | 07:45 |
| ralonsoh | $ ping 10.0.0.178 -s 1600 -M want | 07:45 |
| ralonsoh | PING 10.0.0.178 (10.0.0.178) 1600(1628) bytes of data. | 07:46 |
| ralonsoh | From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) | 07:46 |
| ralonsoh | From 192.168.1.1 icmp_seq=2 Frag needed and DF set (mtu = 1482) | 07:46 |
| ralonsoh | ^C | 07:46 |
| ralonsoh | --- 10.0.0.178 ping statistics --- | 07:46 |
| ralonsoh | 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms | 07:46 |
| ralonsoh | maximal working size with ping 1414 | 07:46 |
| ralonsoh | The same scenarios were tested with OSP16.1 using ML2/OVS and same type of traffic was able to reach external network. | 07:46 |
| ralonsoh | Version-Release number of selected component (if applicable): | 07:46 |
| ralonsoh | 16.1-RHEL-8/RHOS-16.1-RHEL-8-20200505.n.0 | 07:46 |
| ralonsoh | How reproducible: | 07:46 |
| ralonsoh | 100% | 07:46 |
| ralonsoh | Steps to Reproduce: | 07:46 |
| ralonsoh | 1. Create external network, internal network, router, connect both networks to the router, keypair, security group with rules allowing icmp, ssh, udp. | 07:46 |
| ralonsoh | 2. Launch instance on the internal network using created keypair and security group | 07:46 |
| ralonsoh | 3. Using netcat try to send a UDP packet from the instance to the external network addreess which is smaller (e.g. 1400 bytes) and then bigger (e.g. 1600 bytes) than MTU size | 07:46 |
| ralonsoh | 4 Try to ping with same sizes | 07:46 |
| ralonsoh | Actual results: | 07:46 |
| ralonsoh | UDP/ICMP packets with size bigger than MTU do not reach external network | 07:46 |
| ralonsoh | Expected results: | 07:46 |
| ralonsoh | UDP/ICMP packets with size bigger than MTU reach external network successfully after fragmenting and reassembling | 07:46 |
| ralonsoh | Additional info: | 07:46 |
| ralonsoh | Roman Safronov 2020-05-10 19:59:47 UTC | 07:46 |
| ralonsoh | Blocks: 1547074 | 07:46 |
| ralonsoh | Private | 07:46 |
| ralonsoh | Comment 1 Roman Safronov 2020-05-10 20:08:34 UTC | 07:46 |
| ralonsoh | puppet-ovn-15.4.1-0.20200311045730.192ac4e.el8ost.noarch | 07:47 |
| ralonsoh | python3-networking-ovn-7.1.1-0.20200505113427.071bd83.el8ost.noarch | 07:47 |
| ralonsoh | ovn2.13-2.13.0-18.el8fdp.x86_64 | 07:47 |
| ralonsoh | openvswitch2.13-2.13.0-18.el8fdp.x86_64 | 07:47 |
| ralonsoh | Mike Burns 2020-05-10 20:20:21 UTC | 07:47 |
| ralonsoh | Internal Whiteboard: DFG:Networking Squad:OVN | 07:47 |
| ralonsoh | Private | 07:47 |
| ralonsoh | Comment 2 Roman Safronov 2020-05-10 20:31:04 UTC | 07:47 |
| ralonsoh | [heat-admin@controller-0 ~]$ ovn-nbctl show | 07:47 |
| ralonsoh | switch fef954a0-dfca-4a68-8b87-1c993a1ec2ce (neutron-3d88807d-1415-45e9-a9d5-9849c545d44b) (aka external) | 07:47 |
| ralonsoh | port fe4ea8f7-1163-44db-b1e4-60222155b430 | 07:47 |
| ralonsoh | addresses: ["fa:16:3e:f1:28:ed 10.0.0.178"] | 07:47 |
| ralonsoh | port provnet-3d88807d-1415-45e9-a9d5-9849c545d44b | 07:47 |
| ralonsoh | type: localnet | 07:47 |
| ralonsoh | addresses: ["unknown"] | 07:47 |
| ralonsoh | port 2e0ba96d-a448-41a9-892d-ae2d3a8d21e8 | 07:47 |
| ralonsoh | type: localport | 07:47 |
| ralonsoh | addresses: ["fa:16:3e:1d:44:e2 10.0.0.151"] | 07:47 |
| ralonsoh | port 05fbf01e-bb99-4129-8ddd-e2262fac3a7b | 07:47 |
| ralonsoh | type: router | 07:47 |
| ralonsoh | router-port: lrp-05fbf01e-bb99-4129-8ddd-e2262fac3a7b | 07:47 |
| ralonsoh | switch f366bc5f-166f-4e8c-a29e-ead50b5dfdc1 (neutron-040e120f-2743-4ab7-ae24-7864d21def9a) (aka internal_A) | 07:47 |
| ralonsoh | port 8537fd46-0eff-4d19-91b0-58031f18cfed | 07:47 |
| ralonsoh | type: router | 07:47 |
| ralonsoh | router-port: lrp-8537fd46-0eff-4d19-91b0-58031f18cfed | 07:47 |
| ralonsoh | port 3b68cc85-9fef-4485-8448-3fee5f4d83f4 | 07:47 |
| ralonsoh | addresses: ["fa:16:3e:f0:b7:a7 192.168.1.188"] | 07:47 |
| ralonsoh | port 877d4203-dff9-4968-bb58-990c3183a468 | 07:48 |
| ralonsoh | addresses: ["fa:16:3e:33:20:b9 192.168.1.84"] | 07:48 |
| ralonsoh | port 50086416-8719-4e7c-816a-d424b4f42d89 | 07:48 |
| ralonsoh | type: localport | 07:48 |
| ralonsoh | addresses: ["fa:16:3e:74:62:ec 192.168.1.2"] | 07:48 |
| ralonsoh | router 9586a768-8b0f-4c00-815e-ae05e94929f5 (neutron-ce6c4dbf-3698-4303-bd33-82f37f8975fd) (aka routerA) | 07:48 |
| ralonsoh | port lrp-05fbf01e-bb99-4129-8ddd-e2262fac3a7b | 07:48 |
| ralonsoh | mac: "fa:16:3e:92:da:29" | 07:48 |
| ralonsoh | networks: ["10.0.0.177/24"] | 07:48 |
| ralonsoh | gateway chassis: [9cb5b6d6-b468-47bc-a18c-0a9e4544ab72 0d2347bf-ae10-43c7-8a6c-7dad0e31205d 264ffaed-01d3-450b-8a7d-34b8211c5555] | 07:48 |
| ralonsoh | port lrp-8537fd46-0eff-4d19-91b0-58031f18cfed | 07:48 |
| ralonsoh | mac: "fa:16:3e:e2:8b:4c" | 07:48 |
| ralonsoh | networks: ["192.168.1.1/24"] | 07:48 |
| ralonsoh | nat 63687bcd-2304-4d13-9a07-ba9a41bf123f | 07:48 |
| ralonsoh | external ip: "10.0.0.177" | 07:48 |
| ralonsoh | logical ip: "192.168.1.0/24" | 07:48 |
| ralonsoh | type: "snat" | 07:48 |
| ralonsoh | nat efc57895-a5fa-4b2c-a110-9b706c10d918 | 07:48 |
| ralonsoh | external ip: "10.0.0.172" | 07:48 |
| ralonsoh | logical ip: "192.168.1.188" | 07:48 |
| ralonsoh | type: "dnat_and_snat" | 07:48 |
| ralonsoh | [heat-admin@controller-0 ~]$ ovn-nbctl list logical_router_port | 07:48 |
| ralonsoh | _uuid : 533d9460-88c5-494d-814a-67612688600e | 07:48 |
| ralonsoh | enabled : [] | 07:48 |
| ralonsoh | external_ids : {"neutron:network_name"=neutron-3d88807d-1415-45e9-a9d5-9849c545d44b, "neutron:revision_number"="8", "neutron:router_name"="ce6c4dbf-3698-4303-bd33-82f37f8975fd", "neutron:subnet_ids"="0b88748a-3c03-405d-a901-4e9ed3e4948d"} | 07:48 |
| ralonsoh | gateway_chassis : [0105d693-e120-41ee-adf0-1b2849736022, 080486c2-9e2b-45ee-959d-5c3bebe477b6, 6f6fcc99-258c-4d5d-921f-5400331d3ceb] | 07:48 |
| ralonsoh | ha_chassis_group : [] | 07:49 |
| ralonsoh | ipv6_prefix : [] | 07:49 |
| ralonsoh | ipv6_ra_configs : {} | 07:49 |
| ralonsoh | mac : "fa:16:3e:92:da:29" | 07:49 |
| ralonsoh | name : lrp-05fbf01e-bb99-4129-8ddd-e2262fac3a7b | 07:49 |
| ralonsoh | networks : ["10.0.0.177/24"] | 07:49 |
| ralonsoh | options : {gateway_mtu="1500"} | 07:49 |
| ralonsoh | peer : [] | 07:49 |
| ralonsoh | _uuid : a2cd2303-fa8a-405c-b3ce-70cd074f5595 | 07:49 |
| ralonsoh | enabled : [] | 07:49 |
| ralonsoh | external_ids : {"neutron:network_name"=neutron-040e120f-2743-4ab7-ae24-7864d21def9a, "neutron:revision_number"="3", "neutron:router_name"="ce6c4dbf-3698-4303-bd33-82f37f8975fd", "neutron:subnet_ids"="20ea6c57-d7d7-48bf-b373-6fef6180c8dc"} | 07:49 |
| ralonsoh | gateway_chassis : [] | 07:49 |
| ralonsoh | ha_chassis_group : [] | 07:49 |
| ralonsoh | ipv6_prefix : [] | 07:49 |
| ralonsoh | ipv6_ra_configs : {} | 07:49 |
| ralonsoh | mac : "fa:16:3e:e2:8b:4c" | 07:49 |
| ralonsoh | name : lrp-8537fd46-0eff-4d19-91b0-58031f18cfed | 07:49 |
| ralonsoh | networks : ["192.168.1.1/24"] | 07:49 |
| ralonsoh | options : {} | 07:49 |
| ralonsoh | peer : [] | 07:49 |
| ralonsoh | [heat-admin@controller-0 ~]$ ovn-nbctl list logical_switch | 07:49 |
| ralonsoh | _uuid : fef954a0-dfca-4a68-8b87-1c993a1ec2ce | 07:49 |
| ralonsoh | acls : [] | 07:49 |
| ralonsoh | dns_records : [57f7eda5-fc8c-47fb-921a-909b59b6e624] | 07:49 |
| ralonsoh | external_ids : {"neutron:mtu"="1500", "neutron:network_name"=external, "neutron:qos_policy_id"=null, "neutron:revision_number"="2"} | 07:49 |
| ralonsoh | forwarding_groups : [] | 07:49 |
| ralonsoh | load_balancer : [] | 07:50 |
| ralonsoh | name : neutron-3d88807d-1415-45e9-a9d5-9849c545d44b | 07:50 |
| ralonsoh | other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} | 07:50 |
| ralonsoh | ports : [15e9e7b4-b919-4fe8-a618-9211ea060f2e, 45f0234a-1319-4dad-bbd3-390f63a35e40, 8201a23b-ef36-4c67-927d-6dfafadd1fd7, cfb35313-f87c-4673-81aa-ff7ff31b7610] | 07:50 |
| ralonsoh | qos_rules : [] | 07:50 |
| ralonsoh | _uuid : f366bc5f-166f-4e8c-a29e-ead50b5dfdc1 | 07:50 |
| ralonsoh | acls : [] | 07:50 |
| ralonsoh | dns_records : [765572f9-eeb1-4b2d-91d6-c121bd78f696] | 07:50 |
| ralonsoh | external_ids : {"neutron:mtu"="1442", "neutron:network_name"=internal_A, "neutron:qos_policy_id"=null, "neutron:revision_number"="2"} | 07:50 |
| ralonsoh | forwarding_groups : [] | 07:50 |
| ralonsoh | load_balancer : [] | 07:50 |
| ralonsoh | name : neutron-040e120f-2743-4ab7-ae24-7864d21def9a | 07:50 |
| ralonsoh | other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} | 07:50 |
| ralonsoh | ports : [14f7769c-6bab-4a58-b0fc-c11537382aad, 5b57f30f-2822-492d-b178-2bc758311725, 86cbfff9-0c6b-4c15-a3cd-501ffbe5c683, d17bcb48-6db3-4ee6-9586-559b7c459cbf] | 07:50 |
| ralonsoh | qos_rules : [] | 07:50 |
| ralonsoh | CCIT 2020-05-11 04:31:15 UTC | 07:50 |
| ralonsoh | PM Score: 0 → 800 | 07:50 |
| ralonsoh | Private | 07:50 |
| ralonsoh | Comment 3 Numan Siddique 2020-05-11 16:39:30 UTC | 07:50 |
| ralonsoh | If the VMs are configured with 1442 MTU (geneve), then how can the packets greater than 1442 go out of the VM ? | 07:50 |
| ralonsoh | I don't know if it works with ML2OVS or not, but surely it will not work with OVN. | 07:50 |
| ralonsoh | We added the fragmentation support to address use case where the internal geneve networks are of jumbu mtus (i.e 8442), | 07:50 |
| ralonsoh | and the external network's MTU is lesser than that. | 07:50 |
| ralonsoh | @Daniel - When we worked on this in core OVN, this usecase is what we discussed right ? | 07:50 |
| ralonsoh | @can you please test out the scenario I mentioned above ? i.e internal networ MTU > external network MTU. | 07:50 |
| ralonsoh | If there is a requirement to support otherway round, I don't think it will work at the moment. | 07:51 |
| ralonsoh | Request to please raise RFE bugs for the OVN component. | 07:51 |
| ralonsoh | Thanks | 07:51 |
| ralonsoh | CC: nusiddiq@redhat.com | 07:51 |
| ralonsoh | Numan Siddique 2020-05-11 16:40:34 UTC | 07:51 |
| ralonsoh | CC: dalvarez@redhat.com | 07:51 |
| ralonsoh | Flags: needinfo?(dalvarez@redhat.com) | 07:51 |
| ralonsoh | Private | 07:51 |
| ralonsoh | Comment 4 Roman Safronov 2020-05-11 17:22:27 UTC | 07:51 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:51 |
| ralonsoh | I tested it also on an environment with internal network MTU 8442 and external 1500 and the result was like described in the bug, 100% packet loss. Both environments are still available for investigation. | 07:51 |
| ralonsoh | Packets greater than MTU should be fragmented and then reassembled on the other side. I tested this scenario with OSP16.1 ML2/OVS and some hardware home router. Both were able to transfer packets from internal network through nat to external network without packet loss and regardless of differences in MTU sizes. I can attach captures if needed, tested UDP and ICMP. | 07:51 |
| ralonsoh | So I expected the same behavior now with OVN after OVN UDP fragmentation support was added. I think customers would expect the same. | 07:51 |
| ralonsoh | Jakub Libosvar 2020-05-12 13:19:25 UTC | 07:51 |
| ralonsoh | Priority: unspecified → high | 07:51 |
| ralonsoh | CC: jlibosva@redhat.com | 07:51 |
| ralonsoh | Assignee: amuller@redhat.com → lmartins@redhat.com | 07:51 |
| ralonsoh | Private | 07:51 |
| ralonsoh | Comment 5 Roman Safronov 2020-05-12 13:39:26 UTC | 07:51 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:51 |
| ralonsoh | Not blocking BZ 1547074 anymore because as appeared it relates only to south(with bigger mtu) to north(with smaller mtu) scenario. | 07:51 |
| ralonsoh | No longer blocks: 1547074 | 07:51 |
| ralonsoh | CCIT 2020-05-13 04:32:10 UTC | 07:51 |
| ralonsoh | PM Score: 800 → 1400 | 07:51 |
| ralonsoh | James Smith 2020-06-01 16:49:17 UTC | 07:51 |
| ralonsoh | Blocks: 1842606 | 07:52 |
| ralonsoh | Private | 07:52 |
| ralonsoh | Comment 6 Lucas Alvares Gomes 2020-06-08 15:41:04 UTC | 07:52 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:52 |
| ralonsoh | (In reply to Roman Safronov from comment #4) | 07:52 |
| ralonsoh | > I tested it also on an environment with internal network MTU 8442 and | 07:52 |
| ralonsoh | > external 1500 and the result was like described in the bug, 100% packet | 07:52 |
| ralonsoh | > loss. Both environments are still available for investigation. | 07:52 |
| ralonsoh | > | 07:52 |
| ralonsoh | > Packets greater than MTU should be fragmented and then reassembled on the | 07:52 |
| ralonsoh | > other side. I tested this scenario with OSP16.1 ML2/OVS and some hardware | 07:52 |
| ralonsoh | > home router. Both were able to transfer packets from internal network | 07:52 |
| ralonsoh | > through nat to external network without packet loss and regardless of | 07:52 |
| ralonsoh | > differences in MTU sizes. I can attach captures if needed, tested UDP and | 07:52 |
| ralonsoh | > ICMP. | 07:52 |
| ralonsoh | > So I expected the same behavior now with OVN after OVN UDP fragmentation | 07:52 |
| ralonsoh | > support was added. I think customers would expect the same. | 07:52 |
| ralonsoh | Ok thanks for testing it out Roman. | 07:52 |
| ralonsoh | I am moving this bug to the OVN component because I don't think there is anything we can do from the OpenStack OVN driver perspective here. From the output/logs provided I can see that the OVN driver is setting the "gateway_mtu" option correctly in the Logical Router Port entry. | 07:52 |
| ralonsoh | @Numan or anyone else from core OVN, please feel free to move it back to us if you think that my take on this is incorrect and we need to do something else in the OpenStack OVN driver. | 07:52 |
| ralonsoh | Cheers, | 07:52 |
| ralonsoh | Lucas | 07:52 |
| ralonsoh | CC: ctrautma@redhat.com, jishi@redhat.com, ralongi@redhat.com | 07:52 |
| ralonsoh | Component: python-networking-ovn → ovn2.13 | 07:52 |
| ralonsoh | Version: 16.1 (Train) → RHEL 8.0 | 07:52 |
| ralonsoh | Assignee: lmartins@redhat.com → ovnteam@redhat.com | 07:52 |
| ralonsoh | Product: Red Hat OpenStack → Red Hat Enterprise Linux Fast Datapath | 07:53 |
| ralonsoh | QA Contact: ekuris@redhat.com → jishi@redhat.com | 07:53 |
| ralonsoh | engops-jira@redhat.com 2020-06-08 15:45:04 UTC | 07:53 |
| ralonsoh | Link ID: PnT-DevOps Jira FD-706 | 07:53 |
| ralonsoh | Private | 07:53 |
| ralonsoh | Comment 7 Numan Siddique 2020-06-08 16:01:11 UTC | 07:53 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:53 |
| ralonsoh | (In reply to Roman Safronov from comment #4) | 07:53 |
| ralonsoh | > I tested it also on an environment with internal network MTU 8442 and | 07:53 |
| ralonsoh | > external 1500 and the result was like described in the bug, 100% packet | 07:53 |
| ralonsoh | > loss. Both environments are still available for investigation. | 07:53 |
| ralonsoh | > | 07:53 |
| ralonsoh | > Packets greater than MTU should be fragmented and then reassembled on the | 07:53 |
| ralonsoh | > other side. I tested this scenario with OSP16.1 ML2/OVS and some hardware | 07:53 |
| ralonsoh | > home router. Both were able to transfer packets from internal network | 07:53 |
| ralonsoh | > through nat to external network without packet loss and regardless of | 07:53 |
| ralonsoh | > differences in MTU sizes. I can attach captures if needed, tested UDP and | 07:53 |
| ralonsoh | > ICMP. | 07:53 |
| ralonsoh | > So I expected the same behavior now with OVN after OVN UDP fragmentation | 07:53 |
| ralonsoh | > support was added. I think customers would expect the same. | 07:53 |
| ralonsoh | Hi Lucas, | 07:53 |
| ralonsoh | From OVN perspective, we send out the ICMP need frag to the sender of the packet. | 07:53 |
| ralonsoh | To debug further, I think the working pcap files for the case of ML2OVS would | 07:53 |
| ralonsoh | be helpful. | 07:53 |
| ralonsoh | Hi Roman - Is this possible? | 07:53 |
| ralonsoh | Thanks | 07:53 |
| ralonsoh | Numan | 07:53 |
| ralonsoh | Private | 07:53 |
| ralonsoh | Comment 8 Roman Safronov 2020-06-08 16:10:15 UTC | 07:54 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:54 |
| ralonsoh | Created attachment 1696154 [details] | 07:54 |
| ralonsoh | ML2/OVS ML2/OVN fragmentation parity testing log | 07:54 |
| ralonsoh | Roman Safronov 2020-06-08 16:10:47 UTC | 07:54 |
| ralonsoh | Attachment #1696154 - Attachment is private: 0 → 1 | 07:54 |
| ralonsoh | Roman Safronov 2020-06-08 16:10:48 UTC | 07:54 |
| ralonsoh | Comment 8 is private: 0 → 1 | 07:54 |
| ralonsoh | Private | 07:54 |
| ralonsoh | Comment 9 Roman Safronov 2020-06-08 16:24:51 UTC | 07:54 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:54 |
| ralonsoh | Numan, 'need to frag' ICMP is sent in case a IPv4 packet bigger than external network MTU reaches OVN router from the internal network. BTW, neither 'need to frag' nor IPv6 'packet too big' received in case of IPv6. Regarding IPv6, I think we need to open a new RFE, path MTU discovery does not work there by itself. You can see details in my bug https://bugzilla.redhat.com/show_bug.cgi?id=1839847 , that I opened on | 07:54 |
| ralonsoh | ignoring MTU by IPv6. | 07:54 |
| ralonsoh | But anyway, I think we can close this bug as NOTABUG, because, as was concluded in our discussion in the corresponding mail thread it is not expected from OVN router to perform any fragmentation. | 07:54 |
| ralonsoh | PS, I attached a log of my parity testing, with tcpdump output seen when sending tcp/udp packets using ML2/OVS and ML2/OVN, that I did couple of weeks ago. | 07:54 |
| ralonsoh | Private | 07:54 |
| ralonsoh | Comment 10 mcambria@redhat.com 2020-06-08 16:26:42 UTC | 07:54 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:54 |
| ralonsoh | (In reply to Roman Safronov from comment #0) | 07:54 |
| ralonsoh | [deleted] | 07:54 |
| ralonsoh | > Actual results: | 07:54 |
| ralonsoh | > UDP/ICMP packets with size bigger than MTU do not reach external network | 07:54 |
| ralonsoh | What is the next hop device for traffic leaving the VM? | 07:54 |
| ralonsoh | Are there tcpdump captures of the traffic leaving the VM and entering the this next hop? Capture on the next hop (hypervisor?) side, with GRO disabled will be needed. | 07:54 |
| ralonsoh | If not, can this be added to the "needednfo" | 07:54 |
| ralonsoh | CC: mcambria@redhat.com | 07:55 |
| ralonsoh | Private | 07:55 |
| ralonsoh | Comment 11 Roman Safronov 2020-06-08 17:00:41 UTC | 07:55 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:55 |
| ralonsoh | Michael, | 07:55 |
| ralonsoh | the next hop device for the traffic leaving the VM is OVN router. Capture on the next-hop network (external network) is empty. As far as I understood from our discussion via mail with Numan and Daniel it is not expected that OVN router will fragment traffic, packets will be dropped and it is OK. The RFE was actually to send a 'need to frag' informing the host that router is not able to fragment. And OVN router sends | 07:55 |
| ralonsoh | 'need to frag' as expected. | 07:55 |
| ralonsoh | Folks, | 07:55 |
| ralonsoh | in case somebody wants to investigate or get captures, I still have an environment with such setup, you are welcome to login to titan12.lab.eng.tlv2.redhat.com with root/1-8 to see the issue on a live system. | 07:55 |
| ralonsoh | Access to rhel-int (user/pass: cloud-user/1-8) is via vm1 that has a FIP (user/pass: cirros/gocubsgo) | 07:55 |
| ralonsoh | Private | 07:55 |
| ralonsoh | Comment 12 mcambria@redhat.com 2020-06-08 19:28:53 UTC | 07:55 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:55 |
| ralonsoh | Reproduced with Roman's help. | 07:55 |
| ralonsoh | cat 35k_bytes | nc -u 10.0.0.66 65000 | 07:55 |
| ralonsoh | [cloud-user@rhel-int ~]$ sudo tcpdump -n -i eth0 -n -vvvv 'udp and host 10.0.0.66' or icmp | 07:55 |
| ralonsoh | tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes | 07:55 |
| ralonsoh | 15:04:27.915021 IP (tos 0x0, ttl 64, id 531, offset 0, flags [DF], proto UDP (17), length 8220) | 07:55 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: [bad udp cksum 0xec68 -> 0xb036!] UDP, length 8192 | 07:55 |
| ralonsoh | 15:04:27.915323 IP (tos 0x0, ttl 64, id 532, offset 0, flags [DF], proto UDP (17), length 8220) | 07:55 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: [bad udp cksum 0xec68 -> 0xcdc4!] UDP, length 8192 | 07:55 |
| ralonsoh | 15:04:27.915417 IP (tos 0x0, ttl 64, id 533, offset 0, flags [DF], proto UDP (17), length 8220) | 07:55 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: [bad udp cksum 0xec68 -> 0xb05d!] UDP, length 8192 | 07:55 |
| ralonsoh | 15:04:27.915534 IP (tos 0x0, ttl 64, id 534, offset 0, flags [DF], proto UDP (17), length 8220) | 07:55 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: [bad udp cksum 0xec68 -> 0xcdc4!] UDP, length 8192 | 07:55 |
| ralonsoh | 15:04:27.915621 IP (tos 0x0, ttl 64, id 535, offset 0, flags [DF], proto UDP (17), length 3028) | 07:55 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: [bad udp cksum 0xd820 -> 0xb6f9!] UDP, length 3000 | 07:56 |
| ralonsoh | 15:04:27.918476 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 07:56 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 07:56 |
| ralonsoh | IP (tos 0x0, ttl 64, id 531, offset 0, flags [DF], proto UDP (17), length 8220) | 07:56 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: UDP, length 8192 | 07:56 |
| ralonsoh | 15:04:27.918643 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 07:56 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 07:56 |
| ralonsoh | IP (tos 0x0, ttl 64, id 532, offset 0, flags [DF], proto UDP (17), length 8220) | 07:56 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: UDP, length 8192 | 07:56 |
| ralonsoh | 15:04:27.918889 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 07:56 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 07:56 |
| ralonsoh | IP (tos 0x0, ttl 64, id 533, offset 0, flags [DF], proto UDP (17), length 8220) | 07:56 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: UDP, length 8192 | 07:56 |
| ralonsoh | 15:04:27.918921 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 07:56 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 07:56 |
| ralonsoh | IP (tos 0x0, ttl 64, id 534, offset 0, flags [DF], proto UDP (17), length 8220) | 07:56 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: UDP, length 8192 | 07:56 |
| ralonsoh | 15:04:27.919183 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 07:56 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 07:56 |
| ralonsoh | IP (tos 0x0, ttl 64, id 535, offset 0, flags [DF], proto UDP (17), length 3028) | 07:56 |
| ralonsoh | 192.168.1.101.40252 > 10.0.0.66.65000: UDP, length 3000 | 07:56 |
| ralonsoh | ^C | 07:56 |
| ralonsoh | 10 packets captured | 07:56 |
| ralonsoh | 10 packets received by filter | 07:56 |
| ralonsoh | 0 packets dropped by kernel | 07:56 |
| ralonsoh | [cloud-user@rhel-int ~]$ | 07:57 |
| ralonsoh | It looks like the stack isn't honoring the ICMP need to frag message, or expects something else do offload this function. | 07:57 |
| ralonsoh | I turned off tso (sorry, out of habit) and gso and it didn't help. | 07:57 |
| ralonsoh | ethtool -k eth0 output in next post. | 07:57 |
| ralonsoh | One thing I wanted to turn off was this, but I'm not allowed to. | 07:57 |
| ralonsoh | tx-gso-robust: on [fixed] | 07:57 |
| ralonsoh | Private | 07:57 |
| ralonsoh | Comment 13 mcambria@redhat.com 2020-06-08 19:29:05 UTC | 07:57 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:57 |
| ralonsoh | [cloud-user@rhel-int ~]$ sudo ethtool -k eth0 | 07:57 |
| ralonsoh | Features for eth0: | 07:57 |
| ralonsoh | rx-checksumming: on [fixed] | 07:57 |
| ralonsoh | tx-checksumming: on | 07:57 |
| ralonsoh | tx-checksum-ipv4: off [fixed] | 07:57 |
| ralonsoh | tx-checksum-ip-generic: on | 07:57 |
| ralonsoh | tx-checksum-ipv6: off [fixed] | 07:57 |
| ralonsoh | tx-checksum-fcoe-crc: off [fixed] | 07:57 |
| ralonsoh | tx-checksum-sctp: off [fixed] | 07:57 |
| ralonsoh | scatter-gather: on | 07:57 |
| ralonsoh | tx-scatter-gather: on | 07:57 |
| ralonsoh | tx-scatter-gather-fraglist: off [fixed] | 07:57 |
| ralonsoh | tcp-segmentation-offload: off | 07:57 |
| ralonsoh | tx-tcp-segmentation: off | 07:57 |
| ralonsoh | tx-tcp-ecn-segmentation: off | 07:57 |
| ralonsoh | tx-tcp-mangleid-segmentation: off | 07:57 |
| ralonsoh | tx-tcp6-segmentation: off | 07:57 |
| ralonsoh | generic-segmentation-offload: off | 07:57 |
| ralonsoh | generic-receive-offload: on | 07:57 |
| ralonsoh | large-receive-offload: off [fixed] | 07:58 |
| ralonsoh | rx-vlan-offload: off [fixed] | 07:58 |
| ralonsoh | tx-vlan-offload: off [fixed] | 07:58 |
| ralonsoh | ntuple-filters: off [fixed] | 07:58 |
| ralonsoh | receive-hashing: off [fixed] | 07:58 |
| ralonsoh | highdma: on [fixed] | 07:58 |
| ralonsoh | rx-vlan-filter: on [fixed] | 07:58 |
| ralonsoh | vlan-challenged: off [fixed] | 07:58 |
| ralonsoh | tx-lockless: off [fixed] | 07:58 |
| ralonsoh | netns-local: off [fixed] | 07:58 |
| ralonsoh | tx-gso-robust: on [fixed] | 07:58 |
| ralonsoh | tx-fcoe-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-gre-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-gre-csum-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-ipxip4-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-ipxip6-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-udp_tnl-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-udp_tnl-csum-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-gso-partial: off [fixed] | 07:58 |
| ralonsoh | tx-sctp-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-esp-segmentation: off [fixed] | 07:58 |
| ralonsoh | tx-udp-segmentation: off [fixed] | 07:58 |
| ralonsoh | tls-hw-rx-offload: off [fixed] | 07:58 |
| ralonsoh | fcoe-mtu: off [fixed] | 07:58 |
| ralonsoh | tx-nocache-copy: off | 07:58 |
| ralonsoh | loopback: off [fixed] | 07:58 |
| ralonsoh | rx-fcs: off [fixed] | 07:58 |
| ralonsoh | rx-all: off [fixed] | 07:58 |
| ralonsoh | tx-vlan-stag-hw-insert: off [fixed] | 07:59 |
| ralonsoh | rx-vlan-stag-hw-parse: off [fixed] | 07:59 |
| ralonsoh | rx-vlan-stag-filter: off [fixed] | 07:59 |
| ralonsoh | l2-fwd-offload: off [fixed] | 07:59 |
| ralonsoh | hw-tc-offload: off [fixed] | 07:59 |
| ralonsoh | esp-hw-offload: off [fixed] | 07:59 |
| ralonsoh | esp-tx-csum-hw-offload: off [fixed] | 07:59 |
| ralonsoh | rx-udp_tunnel-port-offload: off [fixed] | 07:59 |
| ralonsoh | tls-hw-tx-offload: off [fixed] | 07:59 |
| ralonsoh | rx-gro-hw: off [fixed] | 07:59 |
| ralonsoh | tls-hw-record: off [fixed] | 07:59 |
| ralonsoh | [cloud-user@rhel-int ~]$ | 07:59 |
| ralonsoh | Private | 07:59 |
| ralonsoh | Comment 14 mcambria@redhat.com 2020-06-08 20:02:08 UTC | 07:59 |
| ralonsoh | RED HAT CONFIDENTIAL | 07:59 |
| ralonsoh | I reset gso, tso etc. back to the way they were originally. | 07:59 |
| ralonsoh | Here is capture using TCP instead of UDP. | 07:59 |
| ralonsoh | cat 35k_bytes | nc -t 10.0.0.66 65000 | 07:59 |
| ralonsoh | [cloud-user@rhel-int ~]$ | 07:59 |
| ralonsoh | [cloud-user@rhel-int ~]$ sudo tcpdump -n -i eth0 -n -vvvv 'tcp and host 10.0.0.66 and port 65000' or icmp | 07:59 |
| ralonsoh | tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes | 07:59 |
| ralonsoh | 15:38:36.967607 IP (tos 0x0, ttl 64, id 56902, offset 0, flags [DF], proto TCP (6), length 60) | 07:59 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [S], cksum 0xcc7d (incorrect -> 0xc84f), seq 1811514524, win 28840, options [mss 1442,sackOK,TS val 221807159 ecr 0,nop,wscale 7], length 0 | 07:59 |
| ralonsoh | 15:38:36.969448 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) | 07:59 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [S.], cksum 0x6601 (correct), seq 2423028385, ack 1811514525, win 26844, options [mss 8960,sackOK,TS val 226898967 ecr 221807159,nop,wscale 7], length 0 | 07:59 |
| ralonsoh | 15:38:36.969464 IP (tos 0x0, ttl 64, id 56903, offset 0, flags [DF], proto TCP (6), length 52) | 07:59 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xcc75 (incorrect -> 0x1a12), seq 1, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 0 | 08:00 |
| ralonsoh | 15:38:36.969783 IP (tos 0x0, ttl 64, id 56904, offset 0, flags [DF], proto TCP (6), length 7202) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xe863 (incorrect -> 0x4d07), seq 1:7151, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 7150 | 08:00 |
| ralonsoh | 15:38:36.969846 IP (tos 0x0, ttl 64, id 56909, offset 0, flags [DF], proto TCP (6), length 1094) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [P.], cksum 0xd087 (incorrect -> 0x02f4), seq 7151:8193, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1042 | 08:00 |
| ralonsoh | 15:38:36.969878 IP (tos 0x0, ttl 64, id 56910, offset 0, flags [DF], proto TCP (6), length 5772) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xe2cd (incorrect -> 0xfb84), seq 8193:13913, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 5720 | 08:00 |
| ralonsoh | 15:38:36.970717 IP (tos 0x0, ttl 63, id 65194, offset 0, flags [DF], proto TCP (6), length 64) | 08:00 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xc6dc (correct), seq 1, ack 1, win 226, options [nop,nop,TS val 226898969 ecr 221807161,nop,nop,sack 1 {7151:8193}], length 0 | 08:00 |
| ralonsoh | 15:38:36.970730 IP (tos 0x0, ttl 64, id 56914, offset 0, flags [DF], proto TCP (6), length 1482) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x1587), seq 13913:15343, ack 1, win 226, options [nop,nop,TS val 221807162 ecr 226898969], length 1430 | 08:00 |
| ralonsoh | 15:38:36.971054 IP (tos 0x0, ttl 63, id 65195, offset 0, flags [DF], proto TCP (6), length 72) | 08:00 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0x52cc), seq 1, ack 1, win 366, options [nop,nop,TS val 226898969 ecr 221807161,nop,nop,sack 2 {13913:15343}{7151:8193}], length 0 | 08:00 |
| ralonsoh | 15:38:36.971146 IP (tos 0x0, ttl 64, id 56915, offset 0, flags [DF], proto TCP (6), length 1482) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x68f7), seq 1:1431, ack 1, win 226, options [nop,nop,TS val 221807163 ecr 226898969], length 1430 | 08:00 |
| ralonsoh | 15:38:36.971452 IP (tos 0x0, ttl 63, id 65196, offset 0, flags [DF], proto TCP (6), length 72) | 08:00 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0x4ca7), seq 1, ack 1431, win 506, options [nop,nop,TS val 226898970 ecr 221807163,nop,nop,sack 2 {13913:15343}{7151:8193}], length 0 | 08:00 |
| ralonsoh | 15:38:36.971465 IP (tos 0x0, ttl 64, id 56916, offset 0, flags [DF], proto TCP (6), length 2912) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd7a1 (incorrect -> 0x77c9), seq 1431:4291, ack 1, win 226, options [nop,nop,TS val 221807163 ecr 226898970], length 2860 | 08:00 |
| ralonsoh | 15:38:36.972598 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:00 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:00 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56904, offset 0, flags [DF], proto TCP (6), length 1482) | 08:00 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 1:1431, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:00 |
| ralonsoh | 15:38:36.972681 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:01 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:01 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56905, offset 0, flags [DF], proto TCP (6), length 1482) | 08:01 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 1431:2861, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:01 |
| ralonsoh | 15:38:36.973557 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:01 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:01 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56906, offset 0, flags [DF], proto TCP (6), length 1482) | 08:01 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 2861:4291, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:01 |
| ralonsoh | 15:38:36.973689 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:01 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:01 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56907, offset 0, flags [DF], proto TCP (6), length 1482) | 08:01 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 4291:5721, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:01 |
| ralonsoh | 15:38:36.973922 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:01 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:01 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56908, offset 0, flags [DF], proto TCP (6), length 1482) | 08:01 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 5721:7151, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:01 |
| ralonsoh | 15:38:36.974139 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:01 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:01 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56910, offset 0, flags [DF], proto TCP (6), length 1482) | 08:01 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 8193:9623, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:01 |
| ralonsoh | 15:38:36.974377 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:01 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:01 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56911, offset 0, flags [DF], proto TCP (6), length 1482) | 08:02 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 9623:11053, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:02 |
| ralonsoh | 15:38:36.974624 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:02 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:02 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56912, offset 0, flags [DF], proto TCP (6), length 1482) | 08:02 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 11053:12483, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:02 |
| ralonsoh | 15:38:36.974798 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:02 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:02 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56913, offset 0, flags [DF], proto TCP (6), length 1482) | 08:02 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 12483:13913, ack 1, win 226, options [nop,nop,TS val 221807161 ecr 226898967], length 1430 | 08:02 |
| ralonsoh | 15:38:36.975483 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:02 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:02 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56916, offset 0, flags [DF], proto TCP (6), length 1482) | 08:02 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 1431:2861, ack 1, win 226, options [nop,nop,TS val 221807163 ecr 226898970], length 1430 | 08:02 |
| ralonsoh | 15:38:36.975594 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:02 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:02 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56917, offset 0, flags [DF], proto TCP (6), length 1482) | 08:02 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 2861:4291, ack 1, win 226, options [nop,nop,TS val 221807163 ecr 226898970], length 1430 | 08:02 |
| ralonsoh | 15:38:37.177215 IP (tos 0x0, ttl 64, id 56918, offset 0, flags [DF], proto TCP (6), length 1482) | 08:02 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x4552), seq 1431:2861, ack 1, win 226, options [nop,nop,TS val 221807369 ecr 226898970], length 1430 | 08:02 |
| ralonsoh | 15:38:37.177802 IP (tos 0x0, ttl 63, id 65197, offset 0, flags [DF], proto TCP (6), length 72) | 08:02 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0x44e9), seq 1, ack 2861, win 646, options [nop,nop,TS val 226899176 ecr 221807369,nop,nop,sack 2 {13913:15343}{7151:8193}], length 0 | 08:02 |
| ralonsoh | 15:38:37.177821 IP (tos 0x0, ttl 64, id 56919, offset 0, flags [DF], proto TCP (6), length 1482) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x3eee), seq 2861:4291, ack 1, win 226, options [nop,nop,TS val 221807369 ecr 226899176], length 1430 | 08:03 |
| ralonsoh | 15:38:37.177883 IP (tos 0x0, ttl 64, id 56920, offset 0, flags [DF], proto TCP (6), length 1482) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x56bf), seq 4291:5721, ack 1, win 226, options [nop,nop,TS val 221807369 ecr 226899176], length 1430 | 08:03 |
| ralonsoh | 15:38:37.178160 IP (tos 0x0, ttl 63, id 65198, offset 0, flags [DF], proto TCP (6), length 72) | 08:03 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0x3ec6), seq 1, ack 4291, win 786, options [nop,nop,TS val 226899177 ecr 221807369,nop,nop,sack 2 {13913:15343}{7151:8193}], length 0 | 08:03 |
| ralonsoh | 15:38:37.178170 IP (tos 0x0, ttl 64, id 56921, offset 0, flags [DF], proto TCP (6), length 1482) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x33c0), seq 5721:7151, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:03 |
| ralonsoh | 15:38:37.178242 IP (tos 0x0, ttl 64, id 56922, offset 0, flags [DF], proto TCP (6), length 1482) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x477f), seq 8193:9623, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:03 |
| ralonsoh | 15:38:37.178302 IP (tos 0x0, ttl 63, id 65199, offset 0, flags [DF], proto TCP (6), length 72) | 08:03 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0x38f0), seq 1, ack 5721, win 850, options [nop,nop,TS val 226899177 ecr 221807369,nop,nop,sack 2 {13913:15343}{7151:8193}], length 0 | 08:03 |
| ralonsoh | 15:38:37.178312 IP (tos 0x0, ttl 64, id 56923, offset 0, flags [DF], proto TCP (6), length 2912) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd7a1 (incorrect -> 0x7392), seq 9623:12483, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 2860 | 08:03 |
| ralonsoh | 15:38:37.178476 IP (tos 0x0, ttl 63, id 65200, offset 0, flags [DF], proto TCP (6), length 64) | 08:03 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x6c73), seq 1, ack 8193, win 850, options [nop,nop,TS val 226899177 ecr 221807370,nop,nop,sack 1 {13913:15343}], length 0 | 08:03 |
| ralonsoh | 15:38:37.178485 IP (tos 0x0, ttl 64, id 56925, offset 0, flags [DF], proto TCP (6), length 1482) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x1956), seq 12483:13913, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:03 |
| ralonsoh | 15:38:37.178548 IP (tos 0x0, ttl 64, id 56926, offset 0, flags [DF], proto TCP (6), length 1482) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x2b6a), seq 15343:16773, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:03 |
| ralonsoh | 15:38:37.178606 IP (tos 0x0, ttl 63, id 65201, offset 0, flags [DF], proto TCP (6), length 64) | 08:03 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x66dd), seq 1, ack 9623, win 850, options [nop,nop,TS val 226899177 ecr 221807370,nop,nop,sack 1 {13913:15343}], length 0 | 08:03 |
| ralonsoh | 15:38:37.178617 IP (tos 0x0, ttl 64, id 56927, offset 0, flags [DF], proto TCP (6), length 2912) | 08:03 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd7a1 (incorrect -> 0x57a4), seq 16773:19633, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 2860 | 08:04 |
| ralonsoh | 15:38:37.178710 IP (tos 0x0, ttl 63, id 65202, offset 0, flags [DF], proto TCP (6), length 64) | 08:04 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x6c73), seq 1, ack 9623, win 850, options [nop,nop,TS val 226899177 ecr 221807370,nop,nop,sack 1 {12483:15343}], length 0 | 08:04 |
| ralonsoh | 15:38:37.178719 IP (tos 0x0, ttl 64, id 56929, offset 0, flags [DF], proto TCP (6), length 1482) | 08:04 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xfd67), seq 19633:21063, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:04 |
| ralonsoh | 15:38:37.178776 IP (tos 0x0, ttl 63, id 65203, offset 0, flags [DF], proto TCP (6), length 64) | 08:04 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x66dd), seq 1, ack 9623, win 850, options [nop,nop,TS val 226899177 ecr 221807370,nop,nop,sack 1 {12483:16773}], length 0 | 08:04 |
| ralonsoh | 15:38:37.178802 IP (tos 0x0, ttl 64, id 56930, offset 0, flags [DF], proto TCP (6), length 1482) | 08:04 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x6fe9), seq 9623:11053, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:04 |
| ralonsoh | 15:38:37.178858 IP (tos 0x0, ttl 63, id 65204, offset 0, flags [DF], proto TCP (6), length 72) | 08:04 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0xc6a8), seq 1, ack 9623, win 850, options [nop,nop,TS val 226899177 ecr 221807370,nop,nop,sack 2 {19633:21063}{12483:16773}], length 0 | 08:04 |
| ralonsoh | 15:38:37.178866 IP (tos 0x0, ttl 64, id 56931, offset 0, flags [DF], proto TCP (6), length 1482) | 08:04 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xf0eb), seq 11053:12483, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:04 |
| ralonsoh | 15:38:37.178928 IP (tos 0x0, ttl 63, id 65205, offset 0, flags [DF], proto TCP (6), length 72) | 08:04 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc89 (incorrect -> 0xc112), seq 1, ack 11053, win 850, options [nop,nop,TS val 226899177 ecr 221807370,nop,nop,sack 2 {19633:21063}{12483:16773}], length 0 | 08:04 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: Check the device ID to validate if a port is virtual https://review.opendev.org/c/openstack/neutron/+/892564 | 08:04 |
| ralonsoh | 15:38:37.178936 IP (tos 0x0, ttl 64, id 56932, offset 0, flags [DF], proto TCP (6), length 2912) | 08:04 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd7a1 (incorrect -> 0x57a4), seq 16773:19633, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 2860 | 08:04 |
| ralonsoh | 15:38:37.179159 IP (tos 0x0, ttl 63, id 65206, offset 0, flags [DF], proto TCP (6), length 64) | 08:04 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x1e3e), seq 1, ack 16773, win 850, options [nop,nop,TS val 226899178 ecr 221807370,nop,nop,sack 1 {19633:21063}], length 0 | 08:04 |
| ralonsoh | 15:38:37.179201 IP (tos 0x0, ttl 64, id 56934, offset 0, flags [DF], proto TCP (6), length 2912) | 08:04 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd7a1 (incorrect -> 0x46e0), seq 21063:23923, ack 1, win 226, options [nop,nop,TS val 221807371 ecr 226899178], length 2860 | 08:04 |
| ralonsoh | 15:38:37.181115 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:04 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:05 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56923, offset 0, flags [DF], proto TCP (6), length 1482) | 08:05 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 9623:11053, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:05 |
| ralonsoh | 15:38:37.181206 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:05 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:05 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56924, offset 0, flags [DF], proto TCP (6), length 1482) | 08:05 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 11053:12483, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:05 |
| ralonsoh | 15:38:37.181397 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:05 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:05 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56927, offset 0, flags [DF], proto TCP (6), length 1482) | 08:05 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 16773:18203, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:05 |
| ralonsoh | 15:38:37.181595 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:05 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:05 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56928, offset 0, flags [DF], proto TCP (6), length 1482) | 08:05 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 18203:19633, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:05 |
| ralonsoh | 15:38:37.182380 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:05 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:05 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56932, offset 0, flags [DF], proto TCP (6), length 1482) | 08:05 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 16773:18203, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:05 |
| ralonsoh | 15:38:37.182521 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:05 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:05 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56933, offset 0, flags [DF], proto TCP (6), length 1482) | 08:05 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 18203:19633, ack 1, win 226, options [nop,nop,TS val 221807370 ecr 226899177], length 1430 | 08:06 |
| ralonsoh | 15:38:37.182782 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:06 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:06 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56934, offset 0, flags [DF], proto TCP (6), length 1482) | 08:06 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 21063:22493, ack 1, win 226, options [nop,nop,TS val 221807371 ecr 226899178], length 1430 | 08:06 |
| ralonsoh | 15:38:37.182923 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:06 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:06 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56935, offset 0, flags [DF], proto TCP (6), length 1482) | 08:06 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 22493:23923, ack 1, win 226, options [nop,nop,TS val 221807371 ecr 226899178], length 1430 | 08:06 |
| ralonsoh | 15:38:37.183157 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:06 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:06 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63276, offset 0, flags [DF], proto TCP (6), length 1482) | 08:06 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 2867026017:2867027447, ack 2875809594, win 412, options [nop,nop,TS val 221807371 ecr 226899176], length 1430 | 08:06 |
| ralonsoh | 15:38:37.183174 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:06 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:06 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63277, offset 0, flags [DF], proto TCP (6), length 1482) | 08:06 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 1430:2860, ack 1, win 412, options [nop,nop,TS val 221807371 ecr 226899176], length 1430 | 08:06 |
| ralonsoh | 15:38:37.183317 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:06 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:06 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63278, offset 0, flags [DF], proto TCP (6), length 1324) | 08:06 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [P.], seq 2860:4132, ack 1, win 412, options [nop,nop,TS val 221807371 ecr 226899176], length 1272 | 08:06 |
| ralonsoh | 15:38:37.183561 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:06 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:06 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63279, offset 0, flags [DF], proto TCP (6), length 1482) | 08:07 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 4132:5562, ack 1, win 412, options [nop,nop,TS val 221807371 ecr 226899176], length 1430 | 08:07 |
| ralonsoh | 15:38:37.183650 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:07 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:07 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63280, offset 0, flags [DF], proto TCP (6), length 1314) | 08:07 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [P.], seq 5562:6824, ack 1, win 412, options [nop,nop,TS val 221807371 ecr 226899176], length 1262 | 08:07 |
| ralonsoh | 15:38:37.385267 IP (tos 0x0, ttl 64, id 56936, offset 0, flags [DF], proto TCP (6), length 1482) | 08:07 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x252b), seq 16773:18203, ack 1, win 226, options [nop,nop,TS val 221807577 ecr 226899178], length 1430 | 08:07 |
| ralonsoh | 15:38:37.385803 IP (tos 0x0, ttl 63, id 65207, offset 0, flags [DF], proto TCP (6), length 64) | 08:07 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x170b), seq 1, ack 18203, win 850, options [nop,nop,TS val 226899384 ecr 221807577,nop,nop,sack 1 {19633:21063}], length 0 | 08:07 |
| ralonsoh | 15:38:37.385822 IP (tos 0x0, ttl 64, id 56937, offset 0, flags [DF], proto TCP (6), length 1482) | 08:07 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x0160), seq 18203:19633, ack 1, win 226, options [nop,nop,TS val 221807577 ecr 226899384], length 1430 | 08:07 |
| ralonsoh | 15:38:37.385884 IP (tos 0x0, ttl 64, id 56938, offset 0, flags [DF], proto TCP (6), length 1482) | 08:07 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x75dd), seq 21063:22493, ack 1, win 226, options [nop,nop,TS val 221807577 ecr 226899384], length 1430 | 08:07 |
| ralonsoh | 15:38:37.385953 IP (tos 0x0, ttl 63, id 65208, offset 0, flags [DF], proto TCP (6), length 52) | 08:07 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xc21a), seq 1, ack 21063, win 850, options [nop,nop,TS val 226899384 ecr 221807577], length 0 | 08:07 |
| ralonsoh | 15:38:37.385960 IP (tos 0x0, ttl 64, id 56939, offset 0, flags [DF], proto TCP (6), length 1482) | 08:07 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x8e5b), seq 22493:23923, ack 1, win 226, options [nop,nop,TS val 221807577 ecr 226899384], length 1430 | 08:07 |
| ralonsoh | 15:38:37.386052 IP (tos 0x0, ttl 64, id 56940, offset 0, flags [DF], proto TCP (6), length 1482) | 08:07 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xeb07), seq 23923:25353, ack 1, win 226, options [nop,nop,TS val 221807577 ecr 226899384], length 1430 | 08:07 |
| ralonsoh | 15:38:37.386121 IP (tos 0x0, ttl 63, id 65209, offset 0, flags [DF], proto TCP (6), length 52) | 08:07 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xbc84), seq 1, ack 22493, win 850, options [nop,nop,TS val 226899384 ecr 221807577], length 0 | 08:07 |
| ralonsoh | 15:38:37.386130 IP (tos 0x0, ttl 64, id 56941, offset 0, flags [DF], proto TCP (6), length 1482) | 08:08 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xe570), seq 25353:26783, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899384], length 1430 | 08:08 |
| ralonsoh | 15:38:37.386201 IP (tos 0x0, ttl 64, id 56942, offset 0, flags [DF], proto TCP (6), length 1482) | 08:08 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xfd41), seq 26783:28213, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899384], length 1430 | 08:08 |
| ralonsoh | 15:38:37.386255 IP (tos 0x0, ttl 63, id 65210, offset 0, flags [DF], proto TCP (6), length 52) | 08:08 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xb6ee), seq 1, ack 23923, win 850, options [nop,nop,TS val 226899384 ecr 221807577], length 0 | 08:08 |
| ralonsoh | 15:38:37.386264 IP (tos 0x0, ttl 64, id 56943, offset 0, flags [DF], proto TCP (6), length 1482) | 08:08 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xf7ab), seq 28213:29643, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899384], length 1430 | 08:08 |
| ralonsoh | 15:38:37.386374 IP (tos 0x0, ttl 63, id 65211, offset 0, flags [DF], proto TCP (6), length 52) | 08:08 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xb158), seq 1, ack 25353, win 850, options [nop,nop,TS val 226899384 ecr 221807577], length 0 | 08:08 |
| ralonsoh | 15:38:37.386384 IP (tos 0x0, ttl 64, id 56944, offset 0, flags [DF], proto TCP (6), length 1482) | 08:08 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xd4ae), seq 29643:31073, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899384], length 1430 | 08:08 |
| ralonsoh | 15:38:37.386457 IP (tos 0x0, ttl 63, id 65212, offset 0, flags [DF], proto TCP (6), length 52) | 08:08 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xabc1), seq 1, ack 26783, win 850, options [nop,nop,TS val 226899384 ecr 221807578], length 0 | 08:08 |
| ralonsoh | 15:38:37.386463 IP (tos 0x0, ttl 63, id 65213, offset 0, flags [DF], proto TCP (6), length 52) | 08:08 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xa62a), seq 1, ack 28213, win 850, options [nop,nop,TS val 226899385 ecr 221807578], length 0 | 08:08 |
| ralonsoh | 15:38:37.386469 IP (tos 0x0, ttl 64, id 56945, offset 0, flags [DF], proto TCP (6), length 2912) | 08:08 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd7a1 (incorrect -> 0xc8ad), seq 31073:33933, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 2860 | 08:08 |
| ralonsoh | 15:38:37.386533 IP (tos 0x0, ttl 63, id 65214, offset 0, flags [DF], proto TCP (6), length 52) | 08:08 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0xa094), seq 1, ack 29643, win 850, options [nop,nop,TS val 226899385 ecr 221807578], length 0 | 08:08 |
| ralonsoh | 15:38:37.386542 IP (tos 0x0, ttl 64, id 56947, offset 0, flags [DF], proto TCP (6), length 1482) | 08:08 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x1966), seq 33933:35363, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 1430 | 08:08 |
| ralonsoh | 15:38:37.386600 IP (tos 0x0, ttl 64, id 56948, offset 0, flags [DF], proto TCP (6), length 458) | 08:09 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [FP.], cksum 0xce0b (incorrect -> 0xfece), seq 35363:35769, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 406 | 08:09 |
| ralonsoh | 15:38:37.386605 IP (tos 0x0, ttl 63, id 65215, offset 0, flags [DF], proto TCP (6), length 52) | 08:09 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0x9afe), seq 1, ack 31073, win 850, options [nop,nop,TS val 226899385 ecr 221807578], length 0 | 08:09 |
| ralonsoh | 15:38:37.386761 IP (tos 0x0, ttl 63, id 65216, offset 0, flags [DF], proto TCP (6), length 64) | 08:09 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x750a), seq 1, ack 31073, win 850, options [nop,nop,TS val 226899385 ecr 221807578,nop,nop,sack 1 {33933:35363}], length 0 | 08:09 |
| ralonsoh | 15:38:37.386772 IP (tos 0x0, ttl 64, id 56949, offset 0, flags [DF], proto TCP (6), length 1482) | 08:09 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0xcf17), seq 31073:32503, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 1430 | 08:09 |
| ralonsoh | 15:38:37.386930 IP (tos 0x0, ttl 63, id 65217, offset 0, flags [DF], proto TCP (6), length 64) | 08:09 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x6f74), seq 1, ack 32503, win 850, options [nop,nop,TS val 226899385 ecr 221807578,nop,nop,sack 1 {33933:35363}], length 0 | 08:09 |
| ralonsoh | 15:38:37.386938 IP (tos 0x0, ttl 64, id 56950, offset 0, flags [DF], proto TCP (6), length 1482) | 08:09 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xd20b (incorrect -> 0x916e), seq 32503:33933, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 1430 | 08:09 |
| ralonsoh | 15:38:37.387220 IP (tos 0x0, ttl 63, id 65218, offset 0, flags [DF], proto TCP (6), length 52) | 08:09 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc75 (incorrect -> 0x8a3c), seq 1, ack 35363, win 850, options [nop,nop,TS val 226899385 ecr 221807578], length 0 | 08:09 |
| ralonsoh | 15:38:37.387229 IP (tos 0x0, ttl 64, id 56951, offset 0, flags [DF], proto TCP (6), length 458) | 08:09 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [FP.], cksum 0xce0b (incorrect -> 0xfecd), seq 35363:35769, ack 1, win 226, options [nop,nop,TS val 221807579 ecr 226899385], length 406 | 08:09 |
| ralonsoh | 15:38:37.387418 IP (tos 0x0, ttl 63, id 65220, offset 0, flags [DF], proto TCP (6), length 64) | 08:09 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [.], cksum 0xcc81 (incorrect -> 0x5b81), seq 2, ack 35770, win 850, options [nop,nop,TS val 226899386 ecr 221807579,nop,nop,sack 1 {35363:35770}], length 0 | 08:09 |
| ralonsoh | 15:38:37.387645 IP (tos 0x0, ttl 63, id 65219, offset 0, flags [DF], proto TCP (6), length 52) | 08:09 |
| ralonsoh | 10.0.0.66.65000 > 192.168.1.101.50396: Flags [F.], cksum 0x88a3 (correct), seq 1, ack 35770, win 850, options [nop,nop,TS val 226899386 ecr 221807578], length 0 | 08:09 |
| ralonsoh | 15:38:37.387654 IP (tos 0x0, ttl 64, id 56952, offset 0, flags [DF], proto TCP (6), length 52) | 08:09 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], cksum 0xcc75 (incorrect -> 0x8b12), seq 35770, ack 2, win 226, options [nop,nop,TS val 221807579 ecr 226899386], length 0 | 08:09 |
| ralonsoh | 15:38:37.387748 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:09 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:10 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56945, offset 0, flags [DF], proto TCP (6), length 1482) | 08:10 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 31073:32503, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 1430 | 08:10 |
| ralonsoh | 15:38:37.387913 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:10 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:10 |
| ralonsoh | IP (tos 0x0, ttl 64, id 56946, offset 0, flags [DF], proto TCP (6), length 1482) | 08:10 |
| ralonsoh | 192.168.1.101.50396 > 10.0.0.66.65000: Flags [.], seq 32503:33933, ack 1, win 226, options [nop,nop,TS val 221807578 ecr 226899385], length 1430 | 08:10 |
| ralonsoh | 15:38:37.388866 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:10 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:10 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63298, offset 0, flags [DF], proto TCP (6), length 1482) | 08:10 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 13176:14606, ack 1, win 412, options [nop,nop,TS val 221807579 ecr 226899384], length 1430 | 08:10 |
| ralonsoh | 15:38:37.389016 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:10 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:10 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63299, offset 0, flags [DF], proto TCP (6), length 1482) | 08:10 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 14606:16036, ack 1, win 412, options [nop,nop,TS val 221807579 ecr 226899384], length 1430 | 08:10 |
| ralonsoh | 15:38:37.389169 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:10 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:10 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63301, offset 0, flags [DF], proto TCP (6), length 1482) | 08:10 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 17308:18738, ack 1, win 412, options [nop,nop,TS val 221807579 ecr 226899384], length 1430 | 08:10 |
| ralonsoh | 15:38:37.389311 IP (tos 0x48, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 562) | 08:10 |
| ralonsoh | 192.168.1.1 > 192.168.1.101: ICMP 10.0.0.66 unreachable - need to frag (mtu 1482), length 542 | 08:10 |
| ralonsoh | IP (tos 0x48, ttl 64, id 63302, offset 0, flags [DF], proto TCP (6), length 1482) | 08:10 |
| ralonsoh | 192.168.1.101.22 > 10.0.0.66.48360: Flags [.], seq 18738:20168, ack 1, win 412, options [nop,nop,TS val 221807579 ecr 226899384], length 1430 | 08:11 |
| ralonsoh | ^C | 08:11 |
| ralonsoh | 96 packets captured | 08:11 |
| ralonsoh | 96 packets received by filter | 08:11 |
| ralonsoh | 0 packets dropped by kernel | 08:11 |
| ralonsoh | [cloud-user@rhel-int ~]$ | 08:11 |
| ralonsoh | Private | 08:11 |
| ralonsoh | Comment 15 mcambria@redhat.com 2020-06-09 14:12:22 UTC | 08:11 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:11 |
| ralonsoh | It looks like OVN is sending the packets that trigger ICMP fragmentation needed. Is this intentional? | 08:11 |
| ralonsoh | According to https://tools.ietf.org/html/rfc1191 packets with DF=1 | 08:11 |
| ralonsoh | "If any of the datagrams | 08:11 |
| ralonsoh | are too large to be forwarded without fragmentation by some router | 08:11 |
| ralonsoh | along the path, that router will discard them and return ICMP | 08:11 |
| ralonsoh | Destination Unreachable messages with a code meaning "fragmentation | 08:11 |
| ralonsoh | needed and DF set" | 08:11 |
| ralonsoh | Shouldn't OVN be dropping these? | 08:11 |
| ralonsoh | In the tcp capture in #c14 all the data does eventually get to the destination [1]. But performance is not what it could be. Each too large packet generates an ICMP message. Some TCP segments are sent multiple times. Some are even re-sent with length > MSS long after the sender has seen the ICMP message with the new MSS. | 08:11 |
| ralonsoh | [1] I haven't diff'ed the sent file with the received file. I don't know if any bits were flipped. Right now I'm focusing on MTU/MSS. | 08:11 |
| ralonsoh | lorenzo bianconi 2020-06-10 08:58:30 UTC | 08:11 |
| ralonsoh | CC: lorenzo.bianconi@redhat.com | 08:11 |
| ralonsoh | Assignee: ovnteam@redhat.com → lorenzo.bianconi@redhat.com | 08:11 |
| ralonsoh | James Smith 2020-06-10 21:39:32 UTC | 08:11 |
| ralonsoh | No longer blocks: 1842606 | 08:11 |
| ralonsoh | Private | 08:11 |
| ralonsoh | Comment 16 lorenzo bianconi 2020-06-18 16:45:10 UTC | 08:11 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:12 |
| ralonsoh | Looking at the data attached to the bz and at the ovs/ovn/kernel code I guess we have 2 different scenario: | 08:12 |
| ralonsoh | DF (Don't Fragment) bit is set | 08:12 |
| ralonsoh | ------------------------------ | 08:12 |
| ralonsoh | OVN works as expected since it sends an ICMP error message (packet too big) for each incoming frame | 08:12 |
| ralonsoh | that exceed the gw MTU | 08:12 |
| ralonsoh | DF (Don't Fragment) bit is not set | 08:12 |
| ralonsoh | ---------------------------------- | 08:12 |
| ralonsoh | this scenario seems broken to me since OVN continues to send an ICMP error message (packet too big) | 08:12 |
| ralonsoh | even on fragmented traffic (fragment size is lower than gw MTU). I am to reproduce the issue locally | 08:12 |
| ralonsoh | and I spotted a possible root cause in the ovs kernel datapath. | 08:12 |
| ralonsoh | @Roman: is it ok to share a rhel kernel scratch build to test the fix in your environment? | 08:12 |
| ralonsoh | Regards, | 08:12 |
| ralonsoh | Lorenzo | 08:12 |
| ralonsoh | Flags: needinfo?(rsafrono@redhat.com) | 08:12 |
| ralonsoh | Private | 08:12 |
| ralonsoh | Comment 17 Roman Safronov 2020-06-22 08:55:50 UTC | 08:12 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:12 |
| ralonsoh | Lorenzo, | 08:12 |
| ralonsoh | I am afraid I have no capacity to verify with kernel scratch builds (I even have no idea how to do this correctly, just replace kernel on openstack nodes?). We are focusing on next OSP release issues, currently it is in code freeze stage. | 08:12 |
| ralonsoh | I would suggest to first verify the fix without openstack, may be someone from core ovn qa can help? | 08:12 |
| ralonsoh | Flags: needinfo?(rsafrono@redhat.com) | 08:12 |
| ralonsoh | Private | 08:12 |
| ralonsoh | Comment 18 lorenzo bianconi 2020-06-25 08:43:52 UTC | 08:12 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:13 |
| ralonsoh | (In reply to Roman Safronov from comment #17) | 08:13 |
| ralonsoh | > Lorenzo, | 08:13 |
| ralonsoh | > I am afraid I have no capacity to verify with kernel scratch builds (I even | 08:13 |
| ralonsoh | > have no idea how to do this correctly, just replace kernel on openstack | 08:13 |
| ralonsoh | > nodes?). We are focusing on next OSP release issues, currently it is in code | 08:13 |
| ralonsoh | > freeze stage. | 08:13 |
| ralonsoh | > I would suggest to first verify the fix without openstack, may be someone | 08:13 |
| ralonsoh | > from core ovn qa can help? | 08:13 |
| ralonsoh | Hi Roman, | 08:13 |
| ralonsoh | the fix I posted for the identified issue has been merged in net tree: | 08:13 |
| ralonsoh | https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=17843655708e1941c0653af3cd61be6948e36f43 | 08:13 |
| ralonsoh | I guess we should give it a whirl in your environment. Please let me know what is the best way to do. I guess I can backport it on rhel and provide you a build to test. | 08:13 |
| ralonsoh | Regards, | 08:13 |
| ralonsoh | Lorenzo | 08:13 |
| ralonsoh | Flags: needinfo?(rsafrono@redhat.com) | 08:13 |
| ralonsoh | Private | 08:13 |
| ralonsoh | Comment 19 Roman Safronov 2020-06-25 10:11:30 UTC | 08:13 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:13 |
| ralonsoh | > Hi Roman, | 08:13 |
| ralonsoh | > | 08:13 |
| ralonsoh | > the fix I posted for the identified issue has been merged in net tree: | 08:13 |
| ralonsoh | > | 08:13 |
| ralonsoh | > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/ | 08:13 |
| ralonsoh | > ?id=17843655708e1941c0653af3cd61be6948e36f43 | 08:13 |
| ralonsoh | > | 08:13 |
| ralonsoh | > I guess we should give it a whirl in your environment. Please let me know | 08:13 |
| ralonsoh | > what is the best way to do. I guess I can backport it on rhel and provide | 08:14 |
| ralonsoh | > you a build to test. | 08:14 |
| ralonsoh | > | 08:14 |
| ralonsoh | > Regards, | 08:14 |
| ralonsoh | > Lorenzo | 08:14 |
| ralonsoh | Hi Lorenzo, | 08:14 |
| ralonsoh | There is no better way to test it on my environment other than consume an existing FDP build. | 08:14 |
| ralonsoh | Is your fix already included in the latest FDP 20.E build? | 08:14 |
| ralonsoh | BTW, in case it does we are additionally blocked by the infra issue https://projects.engineering.redhat.com/browse/OSPNET-19 | 08:14 |
| ralonsoh | Flags: needinfo?(rsafrono@redhat.com) → needinfo?(lorenzo.bianconi@redhat.com) | 08:14 |
| ralonsoh | Private | 08:14 |
| ralonsoh | Comment 20 Daniel Alvarez Sanchez 2020-06-26 15:52:39 UTC | 08:14 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:14 |
| ralonsoh | Hey Roman, I think we need a new kernel. @Lorenzo can you patch Roman's setup with the new kernel (rhel 8.2) and re-run the tests? | 08:14 |
| ralonsoh | Flags: needinfo?(dalvarez@redhat.com) | 08:14 |
| ralonsoh | Private | 08:14 |
| ralonsoh | Comment 21 lorenzo bianconi 2020-06-26 16:23:46 UTC | 08:14 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:14 |
| ralonsoh | (In reply to Daniel Alvarez Sanchez from comment #20) | 08:14 |
| ralonsoh | > Hey Roman, I think we need a new kernel. @Lorenzo can you patch Roman's | 08:14 |
| ralonsoh | > setup with the new kernel (rhel 8.2) and re-run the tests? | 08:14 |
| ralonsoh | @Daniel: sure, I will provide a 8.2 scratch build (rpms) to install on the system | 08:14 |
| ralonsoh | Flags: needinfo?(lorenzo.bianconi@redhat.com) | 08:14 |
| ralonsoh | Brendan Conoboy 2020-06-29 03:34:45 UTC | 08:14 |
| ralonsoh | Pool ID: sst_networking_fdp | 08:14 |
| ralonsoh | Private | 08:14 |
| ralonsoh | Comment 22 lorenzo bianconi 2020-06-29 10:28:01 UTC | 08:14 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:15 |
| ralonsoh | related kernel bz: Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=1851888 | 08:15 |
| ralonsoh | Private | 08:15 |
| ralonsoh | Comment 23 Roman Safronov 2020-06-29 17:59:03 UTC | 08:15 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:15 |
| ralonsoh | As I don't have already my old environment where ovn router was sending ICMP 'need to frag' I installed the attached kernel on all nodes with 16.1 RC puddle (RHOS-16.1-RHEL-8-20200625.n.0). | 08:15 |
| ralonsoh | Nothing changed. The behavior is same as described in https://bugzilla.redhat.com/show_bug.cgi?id=1850962. Seems like we need to test this on a env where https://bugzilla.redhat.com/show_bug.cgi?id=1850962 is fixed. | 08:15 |
| ralonsoh | Private | 08:15 |
| ralonsoh | Comment 24 Roman Safronov 2020-06-29 18:11:22 UTC | 08:15 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:15 |
| ralonsoh | Addition to the comment, ovn version was ovn2.13-host-2.13.0-30.el8fdp.x86_64. | 08:15 |
| ralonsoh | Private | 08:15 |
| ralonsoh | Comment 25 Roman Safronov 2020-06-30 16:40:54 UTC | 08:15 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:15 |
| ralonsoh | Hi Lorenzo, | 08:15 |
| ralonsoh | I have an environment with ovn2.13-2.13.0-37.el8fdp.x86_64 and I would like to retest with the new kernel from https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=29756081 | 08:15 |
| ralonsoh | However I am not able to install kernel rpms: | 08:15 |
| ralonsoh | [heat-admin@controller-0 ~]$ sudo rpm --nodeps -ivh http://download.eng.bos.redhat.com/brewroot/work/tasks/6081/29756081/kernel-core-4.18.0-221.el8.bz1851888.test.cki.net.x86_64.rpm | 08:15 |
| ralonsoh | Retrieving http://download.eng.bos.redhat.com/brewroot/work/tasks/6081/29756081/kernel-core-4.18.0-221.el8.bz1851888.test.cki.net.x86_64.rpm | 08:15 |
| ralonsoh | curl: (22) The requested URL returned error: 403 Forbidden | 08:15 |
| ralonsoh | error: skipping http://download.eng.bos.redhat.com/brewroot/work/tasks/6081/29756081/kernel-core-4.18.0-221.el8.bz1851888.test.cki.net.x86_64.rpm - transfer failed | 08:15 |
| ralonsoh | Do you have an idea how to proceed? | 08:15 |
| ralonsoh | Flags: needinfo?(lorenzo.bianconi@redhat.com) | 08:15 |
| ralonsoh | Private | 08:15 |
| ralonsoh | Comment 26 Roman Safronov 2020-07-01 18:28:39 UTC | 08:15 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:16 |
| ralonsoh | Tested on RHOS-16.1-RHEL-8-20200625.n.0 with OVN from next FDP 20.E (not released yet) ovn2.13-2.13.0-37.el8fdp.x86_64 using a scratch kernel from https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=29756081 | 08:16 |
| ralonsoh | Tested with big ICMP and UDP packets (bigger than MTU of the external network) | 08:16 |
| ralonsoh | All works as expected according the following guidelines: | 08:16 |
| ralonsoh | DF=0 --> the device will learn the real MTU and after first pkt, the traffic flows | 08:16 |
| ralonsoh | DF=1 --> the router discard the traffic but the sender does not perform fragmentation so traffic does not flow | 08:16 |
| ralonsoh | Verified also that TCP traffic passes. | 08:16 |
| ralonsoh | Some details: | 08:16 |
| ralonsoh | [cloud-user@rhel-8-vm ~]$ ip route get 10.0.0.31 | 08:16 |
| ralonsoh | 10.0.0.31 via 192.168.1.1 dev eth0 src 192.168.1.34 uid 1000 | 08:16 |
| ralonsoh | cache | 08:16 |
| ralonsoh | [cloud-user@rhel-8-vm ~]$ ping -s 8900 10.0.0.31 -c 5 | 08:16 |
| ralonsoh | PING 10.0.0.31 (10.0.0.31) 8900(8928) bytes of data. | 08:16 |
| ralonsoh | From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) | 08:16 |
| ralonsoh | 8908 bytes from 10.0.0.31: icmp_seq=2 ttl=63 time=2.22 ms | 08:16 |
| ralonsoh | 8908 bytes from 10.0.0.31: icmp_seq=3 ttl=63 time=1.49 ms | 08:16 |
| ralonsoh | 8908 bytes from 10.0.0.31: icmp_seq=4 ttl=63 time=1.31 ms | 08:16 |
| ralonsoh | 8908 bytes from 10.0.0.31: icmp_seq=5 ttl=63 time=1.21 ms | 08:16 |
| ralonsoh | --- 10.0.0.31 ping statistics --- | 08:16 |
| ralonsoh | 5 packets transmitted, 4 received, +1 errors, 20% packet loss, time 12ms | 08:16 |
| ralonsoh | rtt min/avg/max/mdev = 1.208/1.557/2.222/0.398 ms | 08:16 |
| ralonsoh | [cloud-user@rhel-8-vm ~]$ | 08:16 |
| ralonsoh | [cloud-user@rhel-8-vm ~]$ ip route get 10.0.0.31 | 08:16 |
| ralonsoh | 10.0.0.31 via 192.168.1.1 dev eth0 src 192.168.1.34 uid 1000 | 08:16 |
| ralonsoh | cache expires 591sec mtu 1482 | 08:16 |
| ralonsoh | Flags: needinfo?(lorenzo.bianconi@redhat.com) | 08:16 |
| ralonsoh | Roman Safronov 2020-07-06 10:34:52 UTC | 08:16 |
| ralonsoh | Blocks: 1854084 | 08:17 |
| ralonsoh | RHEL Program Management Team 2020-07-07 14:23:56 UTC | 08:17 |
| ralonsoh | PM Score: 1400 → 600 | 08:17 |
| ralonsoh | lorenzo bianconi 2020-07-09 10:40:13 UTC | 08:17 |
| ralonsoh | Status: NEW → POST | 08:17 |
| ralonsoh | Roman Safronov 2020-07-22 09:45:21 UTC | 08:17 |
| ralonsoh | Depends On: Red Hat1854149 | 08:17 |
| ralonsoh | Roman Safronov 2020-07-23 17:09:09 UTC | 08:17 |
| ralonsoh | Depends On: Red Hat1851888 | 08:17 |
| ralonsoh | Private | 08:17 |
| ralonsoh | Comment 27 Roman Safronov 2020-09-30 13:44:31 UTC | 08:17 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:17 |
| ralonsoh | Tested on RHOS-16.1-RHEL-8-20200925.n.1 (ovn2.13-20.06.2-11.el8fdp.x86_64) with kernel updated on all overcloud nodes to kernel-4.18.0-193.20.1.el8_2 (AFAIU not released yet, according to Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=1860169 GA planned to 2020-10-20) | 08:17 |
| ralonsoh | With updated kernel (kernel-4.18.0-193.20.1.el8_2) udp fragmentation worked. With default kernel (4.18.0-193.19.1.el8_2.x86_64) fragmentation did not work. | 08:17 |
| ralonsoh | Federico Iezzi 2020-10-02 08:53:28 UTC | 08:17 |
| ralonsoh | CC: fiezzi@redhat.com | 08:17 |
| ralonsoh | Private | 08:17 |
| ralonsoh | Comment 28 Dan Williams 2020-11-10 14:50:58 UTC | 08:17 |
| ralonsoh | RED HAT CONFIDENTIAL | 08:17 |
| ralonsoh | kernel fixed at least as of RHSA-2020:4286-10 (kernel-4.18.0-193.28.1.el8_2) and FDP 20.H ovn release. | 08:17 |
| ralonsoh | Status: POST → CLOSED | 08:17 |
| ralonsoh | CC: dcbw@redhat.com | 08:17 |
| ralonsoh | Resolution: --- → ERRATA | 08:17 |
| ralonsoh | Last Closed: 2020-11-10 14:50:58 | 08:17 |
| ralonsoh | Jianlin Shi 2021-05-31 01:11:52 UTC | 08:17 |
| ralonsoh | QA Whiteboard: ovn/regression/bz1846300_ipv6_ignore_gateway_mtu | 08:17 |
| ralonsoh | Flags: qe_test_coverage+ | 08:18 |
| ralonsoh | sorry! | 08:18 |
| ralonsoh | what I've done... sorry | 08:18 |
| ralonsoh | frickler, I had an incorrect copy/paste buffer. Please check https://bugzilla.redhat.com/show_bug.cgi?id=1833813#c26 | 08:18 |
| frickler | oh, it has ended, wow | 08:25 |
| frickler | ralonsoh: well that does talk only about south->north traffik, right? I can try to test that, too, but that was the least probably scenario to me. All other directions were broken in my tests | 08:27 |
| ralonsoh | they also tested E/W traffic | 08:28 |
| frickler | which is also broken for me, in the scenario that I reduce the network MTU via neutron on one of the tenant networks | 08:29 |
| frickler | also the comment #26 that you link to doesn't mention the actual MTU on the instance that is being tested from | 08:30 |
| ralonsoh | the MTU instance interface must be the same of the network | 08:30 |
| ralonsoh | that value wasn't changed | 08:31 |
| frickler | yes, but which value does it have in that test? | 08:31 |
| ralonsoh | PING 10.0.0.31 (10.0.0.31) 8900(8928) bytes of data. | 08:31 |
| ralonsoh | so it was using jumbo frames in this network | 08:32 |
| frickler | that's the ping size, I want to see the mtu of eth0 or the network | 08:32 |
| ralonsoh | adn then | 08:32 |
| ralonsoh | From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) | 08:32 |
| ralonsoh | I don't have this value | 08:32 |
| frickler | yes, so the information in that test is incomplete, that's what I'm saying | 08:33 |
| frickler | also some earlier comment mentions ovn_emit_need_to_frag , is that something that neutron needs to set? or a deployer? | 08:34 |
| opendevreview | Elvira García Ruiz proposed openstack/neutron master: [OVN] Fix burst limit for stateless security groups https://review.opendev.org/c/openstack/neutron/+/892648 | 08:34 |
| ralonsoh | frickler, this is a ml2 config configuration parameter | 08:36 |
| ralonsoh | https://opendev.org/openstack/neutron/src/branch/master/neutron/conf/plugins/ml2/drivers/ovn/ovn_conf.py#L196 | 08:36 |
| ralonsoh | that is False by default | 08:36 |
| ralonsoh | so yes, this is something configurable | 08:36 |
| frickler | so that might resolve at least some of the issues. I'm going to test that in a bit. maybe it also should default to true by now, at least in devstack | 08:37 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN] Skip the port status UP update during a live migration https://review.opendev.org/c/openstack/neutron/+/888769 | 09:03 |
| opendevreview | Rodolfo Alonso proposed openstack/neutron master: [OVN][Trunk] Set the subports correct host during live migration https://review.opendev.org/c/openstack/neutron/+/888776 | 09:11 |
| racosta | Hi ralonsoh, If you know the answer... is there any reason for Neutron to create an entry in the HA_Chassis_Group table for each Logical Router (OVN case - 1:1 mapping)? shouldn't it just be for the external ports linked to the network nodes? | 11:25 |
| ralonsoh | racosta, I can't reply right now, give me 1 hour | 11:28 |
| racosta | Ok, no worries. It's just a technical question... I'm evaluating results on a large scale , and I've received feedback from the core folks at OVN about an inefficient recompute of the SB Port_Binding table entries (because of the HA_Chassis_Group table references). | 11:28 |
| ralonsoh | racosta, ah, that's easy: currently we have only two schedulers | 11:28 |
| ralonsoh | and both use HA_Chassis_Group to assign the LRP to this "HA_Chassis_Group" | 11:29 |
| ralonsoh | that's the reason | 11:29 |
| racosta | https://mail.openvswitch.org/pipermail/ovs-discuss/2023-August/052614.html | 11:29 |
| ralonsoh | racosta, I was in the middle of creating a OVN scheduler using the HA_Chassis, instead of the HA_Chassis_Group | 11:30 |
| ralonsoh | racosta, https://review.opendev.org/c/openstack/neutron/+/872033 | 11:31 |
| racosta | Hum, ok. I read your paper (https://rodolfo-alonso.com/ha_router_gateway_ports_in_ovn) and the openstack doc (https://docs.openstack.org/neutron/latest/admin/ovn/external_ports.html), but it wasn't clear to me. | 11:31 |
| ralonsoh | but I didn't have time to complete it | 11:31 |
| racosta | oh, nice! thanks for clarifying. | 11:32 |
| racosta | It's not a problem for now, the OVN core guys are working to improve it in future versions of OVN (considering the current use of HA_Chassis_Group table in the northd recompute process) | 11:34 |
| racosta | Hi iurygregory, nice to see another Brazilian guy here! congrats on your performance in the OpenInfra video/quiz in Vancouver ;) | 12:00 |
| iurygregory | racosta, hey o/ tks! :D | 12:02 |
| lucasagomes | ralonsoh, ihrachys_ hi there, if you have some time today or in the next few days mind taking a quick look at https://review.opendev.org/c/openstack/neutron/+/888127 ? Thanks | 12:43 |
| ralonsoh | sure | 12:45 |
| ihrachys_ | will do | 12:47 |
| lucasagomes | thanks! | 12:57 |
| opendevreview | Yann Morice proposed openstack/neutron master: Add two new l3 ovn schedulers (azchance, azleastloaded) that ensure LRP gateways are distributed over chassis in the different eligible AZs. https://review.opendev.org/c/openstack/neutron/+/892604 | 13:59 |
| opendevreview | Yann Morice proposed openstack/neutron master: [ovn] AZs distribution in L3 port scheduler https://review.opendev.org/c/openstack/neutron/+/892604 | 14:02 |
| frickler | ralonsoh: in my testing there was no change in any of the scenarios. looking at this code neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py it seems like it really only deals with the case that mtu(tenant) > mtu(external), no idea what would happen if we change that ">" to "!=". anyway I'm done with that issue for now | 14:09 |
| ralonsoh | frickler, sorry, I'm in a call now | 14:10 |
| mtomaska | o/ Looking for an example where unittest checks that something was logged using oslo_log. I am aware of python standard assertLogs but wondering if there is something special with oslo_log module. Thanks! | 14:19 |
| opendevreview | yatin proposed openstack/neutron stable/2023.1: Send ovn heatbeat more often. https://review.opendev.org/c/openstack/neutron/+/892592 | 14:57 |
| opendevreview | Brian Haley proposed openstack/neutron master: Catch non-existent entry failures better in ip_lib https://review.opendev.org/c/openstack/neutron/+/891236 | 15:22 |
| *** TheJulia is now known as confused | 15:34 | |
| *** confused is now known as TheJulia | 15:34 | |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/885111 | 15:37 |
| opendevreview | Fernando Royo proposed openstack/ovn-octavia-provider master: Allow multiple VIPs per LB https://review.opendev.org/c/openstack/ovn-octavia-provider/+/885111 | 16:07 |
| haleyb | mtomaska: something like test_periodic_resync_helper_with_event() ? I just searched for 'log.assert' and that was one line | 16:11 |
| mtomaska | thanks haleyb, Ill check it out | 16:26 |
| opendevreview | Miro Tomaska proposed openstack/neutron master: OVN Metadata handle process execeptions https://review.opendev.org/c/openstack/neutron/+/890986 | 17:47 |
| haleyb | tore: just noticed your response way back, but after reading the bug seems like ihrachys_ was going to ping Numan, so perhaps it's all set for now? | 17:55 |
| ihrachys_ | haleyb numan will be back in 2 weeks only. so it would help to have dump-flows | 17:56 |
| ihrachys_ | ah I see them attached, great | 17:56 |
| haleyb | haleyb: ack, and thanks for taking a look! | 17:56 |
| haleyb | doh, meant ihrachys_ :) | 17:57 |
| ihrachys_ | I will check these dumps next week if no one beats me to it. | 17:57 |
| tore | haleyb, ihrachys_: great! I'm off on a mini holiday, back Tuesday, so next week is perfect :) | 18:18 |
| opendevreview | Miro Tomaska proposed openstack/neutron master: Fix ovn-metadata agent sync of unused namespaces https://review.opendev.org/c/openstack/neutron/+/891232 | 19:19 |
| opendevreview | Brian Haley proposed openstack/neutron stable/2023.1: Remove duplicate rows in MySQL query output https://review.opendev.org/c/openstack/neutron/+/892598 | 21:33 |
| opendevreview | Brian Haley proposed openstack/neutron stable/2023.1: Add scope ID to the "GROUP BY" clause in ``get_scoped_floating_ips`` https://review.opendev.org/c/openstack/neutron/+/892599 | 21:33 |
| opendevreview | Brian Haley proposed openstack/neutron stable/2023.1: Add scope ID to the "GROUP BY" clause in ``get_scoped_floating_ips`` https://review.opendev.org/c/openstack/neutron/+/892599 | 21:47 |
| opendevreview | Brian Haley proposed openstack/neutron stable/2023.1: Add scope ID to the "GROUP BY" clause in ``get_scoped_floating_ips`` https://review.opendev.org/c/openstack/neutron/+/892599 | 21:55 |
| opendevreview | Merged openstack/neutron stable/2023.1: Send ovn heatbeat more often. https://review.opendev.org/c/openstack/neutron/+/892592 | 22:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!