| *** markvoelker has joined #openstack-neutron | 00:03 | |
| *** macz has quit IRC | 00:05 | |
| *** armax has quit IRC | 00:08 | |
| *** openstackgerrit has joined #openstack-neutron | 00:09 | |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack/neutron master: Add SR-IOV ML2 driver support for VLAN trunking. https://review.opendev.org/665467 | 00:09 |
|---|---|---|
| *** david-lyle is now known as dklyle | 00:09 | |
| *** frank_wang has joined #openstack-neutron | 00:12 | |
| *** markvoelker has quit IRC | 00:13 | |
| *** frankwang has quit IRC | 00:16 | |
| openstackgerrit | Manjeet Singh Bhatia proposed openstack/neutron master: Add SR-IOV ML2 driver support for VLAN trunking. https://review.opendev.org/665467 | 00:19 |
| openstackgerrit | Merged openstack/neutron stable/train: [nova][train] Docs: Remove deprecated RetryFilter https://review.opendev.org/689644 | 00:20 |
| *** weifan has quit IRC | 00:30 | |
| *** ociuhandu has joined #openstack-neutron | 00:31 | |
| *** weifan has joined #openstack-neutron | 00:32 | |
| *** weifan has quit IRC | 00:33 | |
| *** weifan has joined #openstack-neutron | 00:34 | |
| *** ociuhandu has quit IRC | 00:35 | |
| *** slaweq has joined #openstack-neutron | 00:46 | |
| *** frankwang has joined #openstack-neutron | 00:47 | |
| *** frank_wang has quit IRC | 00:49 | |
| *** ileixe has joined #openstack-neutron | 00:49 | |
| *** frank_wang has joined #openstack-neutron | 00:51 | |
| ileixe | amotoki: Hi there, I was lost my connection yesterday, and log tells you :) | 00:51 |
| *** slaweq has quit IRC | 00:51 | |
| *** frankwang has quit IRC | 00:54 | |
| *** spatel has joined #openstack-neutron | 00:59 | |
| *** weifan has quit IRC | 01:05 | |
| *** frankwang has joined #openstack-neutron | 01:09 | |
| *** frank_wang has quit IRC | 01:11 | |
| *** yamamoto has joined #openstack-neutron | 01:25 | |
| *** nanzha has joined #openstack-neutron | 01:33 | |
| *** frank_wang has joined #openstack-neutron | 01:40 | |
| *** frankwang has quit IRC | 01:44 | |
| *** mhen has quit IRC | 01:47 | |
| *** mhen has joined #openstack-neutron | 01:47 | |
| *** slaweq has joined #openstack-neutron | 01:47 | |
| *** nanzha has quit IRC | 01:48 | |
| *** nanzha has joined #openstack-neutron | 01:49 | |
| *** slaweq has quit IRC | 01:53 | |
| *** nanzha has quit IRC | 01:54 | |
| openstackgerrit | Deepak proposed openstack/neutron master: Add SR-IOV ML2 driver support for VLAN trunking. https://review.opendev.org/665467 | 01:54 |
| *** nanzha has joined #openstack-neutron | 01:55 | |
| *** weifan has joined #openstack-neutron | 02:05 | |
| *** markvoelker has joined #openstack-neutron | 02:14 | |
| *** nanzha has quit IRC | 02:18 | |
| *** markvoelker has quit IRC | 02:18 | |
| *** nanzha has joined #openstack-neutron | 02:19 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** brault has quit IRC | 02:24 | |
| *** brault has joined #openstack-neutron | 02:25 | |
| *** frankwang has joined #openstack-neutron | 02:25 | |
| *** frank_wang has quit IRC | 02:27 | |
| *** weifan has quit IRC | 02:31 | |
| *** frank_wang has joined #openstack-neutron | 02:42 | |
| *** frankwang has quit IRC | 02:45 | |
| *** slaweq has joined #openstack-neutron | 02:48 | |
| *** spsurya has joined #openstack-neutron | 02:50 | |
| *** slaweq has quit IRC | 02:53 | |
| *** weifan has joined #openstack-neutron | 03:05 | |
| *** dave-mccowan has quit IRC | 03:05 | |
| openstackgerrit | zhanghao proposed openstack/neutron-lib master: Move fwaas_v2_log constants to neutron-lib https://review.opendev.org/689623 | 03:12 |
| *** weifan has quit IRC | 03:20 | |
| *** rkukura has quit IRC | 03:34 | |
| *** slaweq has joined #openstack-neutron | 03:49 | |
| *** slaweq has quit IRC | 03:54 | |
| *** brault has quit IRC | 03:56 | |
| *** brault has joined #openstack-neutron | 03:56 | |
| *** spatel has quit IRC | 04:02 | |
| *** spatel has joined #openstack-neutron | 04:02 | |
| *** spatel has quit IRC | 04:03 | |
| *** frankwang has joined #openstack-neutron | 04:08 | |
| *** frank_wang has quit IRC | 04:10 | |
| *** frank_wang has joined #openstack-neutron | 04:37 | |
| *** frankwang has quit IRC | 04:41 | |
| *** nanzha has quit IRC | 04:43 | |
| *** nanzha has joined #openstack-neutron | 04:43 | |
| *** lajoskatona has joined #openstack-neutron | 04:45 | |
| *** slaweq has joined #openstack-neutron | 04:50 | |
| *** slaweq has quit IRC | 04:54 | |
| *** ileixe has quit IRC | 04:55 | |
| *** ileixe has joined #openstack-neutron | 04:57 | |
| *** do3meli has joined #openstack-neutron | 05:02 | |
| *** ratailor has joined #openstack-neutron | 05:09 | |
| *** frankwang has joined #openstack-neutron | 05:12 | |
| *** frank_wang has quit IRC | 05:14 | |
| *** ralonsoh has joined #openstack-neutron | 05:18 | |
| *** frank_wang has joined #openstack-neutron | 05:22 | |
| *** frankwang has quit IRC | 05:25 | |
| *** slaweq has joined #openstack-neutron | 05:28 | |
| *** gcheresh_ has joined #openstack-neutron | 05:28 | |
| *** janki has joined #openstack-neutron | 05:30 | |
| *** gcheresh_ has quit IRC | 05:39 | |
| *** weifan has joined #openstack-neutron | 05:39 | |
| *** weifan has quit IRC | 05:39 | |
| *** weifan has joined #openstack-neutron | 05:40 | |
| *** weifan has quit IRC | 05:44 | |
| openstackgerrit | Nate Johnston proposed openstack/neutron-lib master: Allow for both regular and bulk resource extenders https://review.opendev.org/689945 | 05:53 |
| *** slaweq has quit IRC | 05:59 | |
| *** slaweq has joined #openstack-neutron | 06:00 | |
| *** frankwang has joined #openstack-neutron | 06:01 | |
| *** frank_wang has quit IRC | 06:03 | |
| *** slaweq has quit IRC | 06:05 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Temporary disable CI job neutron-functional-python27 https://review.opendev.org/689957 | 06:05 |
| *** nanzha has quit IRC | 06:15 | |
| *** nanzha has joined #openstack-neutron | 06:15 | |
| *** rcernin has quit IRC | 06:16 | |
| *** markvoelker has joined #openstack-neutron | 06:17 | |
| *** sapd1 has joined #openstack-neutron | 06:21 | |
| *** markvoelker has quit IRC | 06:21 | |
| *** frank_wang has joined #openstack-neutron | 06:21 | |
| *** frankwang has quit IRC | 06:24 | |
| *** igordc has joined #openstack-neutron | 06:34 | |
| *** ircuser-1 has quit IRC | 06:35 | |
| *** aedc has quit IRC | 06:40 | |
| *** frankwang has joined #openstack-neutron | 06:43 | |
| *** frank_wang has quit IRC | 06:45 | |
| *** pcaruana has joined #openstack-neutron | 06:47 | |
| *** gcheresh_ has joined #openstack-neutron | 06:47 | |
| *** igordc has quit IRC | 06:48 | |
| *** bobmel has joined #openstack-neutron | 06:49 | |
| *** trident has quit IRC | 06:53 | |
| *** njohnston_ has joined #openstack-neutron | 06:53 | |
| *** njohnston has quit IRC | 06:54 | |
| *** frank_wang has joined #openstack-neutron | 06:54 | |
| *** papasuder has joined #openstack-neutron | 06:54 | |
| *** frankwang has quit IRC | 06:57 | |
| *** trident has joined #openstack-neutron | 06:58 | |
| papasuder | Hello all, Paweł here. I would I like to clarify one thing related to OVS agent and extension manager. During troubleshooting issue with port removal of skipped devices, I found that there is a line added by slaweq'u to remove the within ext_manager: https://review.opendev.org/#/q/I3cf5c57c7f232deaa190ab6b0129e398fdabe592 | 06:59 |
| papasuder | It made me a small hope that port will be removed, but.. there is a shadow of the whole story - if there is NO extension configured, no information about NOT removed skipped devices will be printed. | 07:00 |
| *** frank_wang has quit IRC | 07:00 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Reset timeout exception in DietTestCase when retrying https://review.opendev.org/689658 | 07:00 |
| *** frank_wang has joined #openstack-neutron | 07:00 | |
| papasuder | https://github.com/openstack/neutron/blob/bf98b2a12a352e5bc7bbee1994d44b82e7dfe928/neutron/agent/l2/l2_agent_extensions_manager.py#L52-L60 there is no ELSE for the FOR loop and due to that no information about NOT removed port is printed. | 07:01 |
| papasuder | Anyway, what I checked, I cannot find suitable L2 extenstion which can handle that port deletion: https://github.com/openstack/neutron/blob/bf98b2a12a352e5bc7bbee1994d44b82e7dfe928/setup.cfg#L113-L116 | 07:02 |
| papasuder | I am wondering if I understand it correctly. Case is to remove port from OVS integration bridge in case of "No such device" ofport = -1 | 07:02 |
| ralonsoh | papasuder, the goal of https://review.opendev.org/#/c/533318/1/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py if that: to remove the port if is not present in the integration brisge | 07:04 |
| ralonsoh | for https://github.com/openstack/neutron/blob/bf98b2a12a352e5bc7bbee1994d44b82e7dfe928/setup.cfg#L113-L116 | 07:04 |
| *** slaweq has joined #openstack-neutron | 07:04 | |
| ralonsoh | you can propose a patch to log that this object was not treated | 07:04 |
| ralonsoh | but is the same for handle_port | 07:04 |
| papasuder | All right, but is there any L2 extenstion which can be used for deleting port from OVS when ofport is -1? | 07:05 |
| ralonsoh | ofport -1 is an invalid port | 07:06 |
| ralonsoh | those ports are not considered | 07:07 |
| papasuder | But those port are treated as skipped devices, so on are in this part of code: https://github.com/openstack/neutron/blob/bf98b2a12a352e5bc7bbee1994d44b82e7dfe928/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1836-L1842 | 07:08 |
| papasuder | So those ports are being processed with line: self.ext_manager.delete_port(...) | 07:08 |
| *** frankwang has joined #openstack-neutron | 07:09 | |
| ralonsoh | if this happens, that means this port was added and in the same polling cycle, deleted | 07:10 |
| ralonsoh | and now it's not present in the int br | 07:10 |
| *** frank_wang has quit IRC | 07:11 | |
| *** cshen has joined #openstack-neutron | 07:11 | |
| *** frank_wang has joined #openstack-neutron | 07:12 | |
| *** sridharg has joined #openstack-neutron | 07:12 | |
| *** mjozefcz|lunch has joined #openstack-neutron | 07:12 | |
| *** ccamposr has joined #openstack-neutron | 07:14 | |
| *** frankwang has quit IRC | 07:15 | |
| papasuder | ralonsoh: yeap, that's what I have :) | 07:17 |
| *** frankwang has joined #openstack-neutron | 07:21 | |
| *** frank_wang has quit IRC | 07:23 | |
| *** bobmel has quit IRC | 07:24 | |
| *** bobmel has joined #openstack-neutron | 07:25 | |
| *** jawad_axd has joined #openstack-neutron | 07:26 | |
| *** njohnston_ is now known as njohnston | 07:28 | |
| *** bobmel has quit IRC | 07:29 | |
| *** aedc has joined #openstack-neutron | 07:32 | |
| *** frank_wang has joined #openstack-neutron | 07:32 | |
| mjozefcz|lunch | papasuder, \o | 07:34 |
| *** mjozefcz|lunch is now known as mjozefcz | 07:34 | |
| *** mjozefcz is now known as maciejjozefczyk | 07:34 | |
| *** jawad_axd is now known as gchristian | 07:35 | |
| *** frankwang has quit IRC | 07:36 | |
| *** jawad_axd has joined #openstack-neutron | 07:36 | |
| *** ajay33 has joined #openstack-neutron | 07:37 | |
| *** trident has quit IRC | 07:40 | |
| *** priteau has joined #openstack-neutron | 07:43 | |
| *** ratailor_ has joined #openstack-neutron | 07:43 | |
| *** trident has joined #openstack-neutron | 07:43 | |
| *** ivve has joined #openstack-neutron | 07:43 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/networking-ovn master: Add Design documentation for L3 HA Rescheduling https://review.opendev.org/653718 | 07:45 |
| *** janki has quit IRC | 07:46 | |
| *** ratailor has quit IRC | 07:46 | |
| *** jangutter has joined #openstack-neutron | 07:52 | |
| *** janki has joined #openstack-neutron | 07:52 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/networking-ovn master: Bump coverage job to 77% and don't skip Octavia driver https://review.opendev.org/690002 | 08:07 |
| *** jlibosva has joined #openstack-neutron | 08:18 | |
| *** tesseract has joined #openstack-neutron | 08:21 | |
| *** tesseract has quit IRC | 08:21 | |
| *** frankwang has joined #openstack-neutron | 08:24 | |
| *** frank_wang has quit IRC | 08:27 | |
| *** jawad_axd has quit IRC | 08:28 | |
| *** andyzon has joined #openstack-neutron | 08:29 | |
| ralonsoh | Hi folks. py27 FT tests is giving some problems since this weekend | 08:32 |
| ralonsoh | please check https://review.opendev.org/#/c/689658/ and https://review.opendev.org/#/c/689957/ | 08:32 |
| *** sapd1 has quit IRC | 08:33 | |
| *** lucasagomes has joined #openstack-neutron | 08:37 | |
| *** papasuder has quit IRC | 08:43 | |
| *** tssurya has joined #openstack-neutron | 08:53 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** papasuder has joined #openstack-neutron | 09:01 | |
| openstackgerrit | kangyufei proposed openstack/python-neutronclient master: Switch to Ussuri jobs https://review.opendev.org/690014 | 09:05 |
| *** davidsha has joined #openstack-neutron | 09:17 | |
| *** TomStappaerts has quit IRC | 09:17 | |
| *** weifan has joined #openstack-neutron | 09:24 | |
| *** frank_wang has joined #openstack-neutron | 09:33 | |
| *** weifan has quit IRC | 09:36 | |
| *** frankwang has quit IRC | 09:37 | |
| *** papasuder has quit IRC | 09:42 | |
| *** nanzha has quit IRC | 09:51 | |
| *** nanzha has joined #openstack-neutron | 09:53 | |
| *** frankwang has joined #openstack-neutron | 10:07 | |
| *** janki has quit IRC | 10:07 | |
| *** frank_wang has quit IRC | 10:09 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Add support for virtual port type https://review.opendev.org/676223 | 10:10 |
| *** ociuhandu has joined #openstack-neutron | 10:10 | |
| ivve | hello, i have a wierd problem with neutron 14.0.2 and wondering if anyone has any input on how i should troubleshoot this issue. whenever i migrate a machine from one host to another L2 between the instance and DHCP server is not established, once i restart the neutron openvswitch agent the vxlan mesh is bumped and it is reestablished.. the only neutron errors i keep getting from controller is "Exception during message handling: TooManyExternalNetworks: | 10:14 |
| ivve | More than one external network exists" | 10:14 |
| ivve | im running ovs, l3_ha and dvr | 10:16 |
| ivve | it is 100% consistent when moving instances between two hosts | 10:16 |
| *** pcaruana has quit IRC | 10:16 | |
| *** papasuder has joined #openstack-neutron | 10:16 | |
| *** papasuder has left #openstack-neutron | 10:18 | |
| *** markvoelker has joined #openstack-neutron | 10:19 | |
| *** pcaruana has joined #openstack-neutron | 10:21 | |
| *** markvoelker has quit IRC | 10:24 | |
| slaweq | ivve: can You check if You didn't hit https://bugs.launchpad.net/neutron/+bug/1824571 ? | 10:33 |
| openstack | Launchpad bug 1824571 in neutron "l3agent can't create router if there are multiple external networks" [Medium,Fix released] - Assigned to Miguel Lavalle (minsel) | 10:33 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Temporary disable CI job neutron-functional-python27 https://review.opendev.org/689957 | 10:36 |
| *** tbachman has quit IRC | 10:44 | |
| *** do3meli has quit IRC | 11:04 | |
| *** frank_wang has joined #openstack-neutron | 11:05 | |
| *** frankwang has quit IRC | 11:08 | |
| *** yamamoto has joined #openstack-neutron | 11:13 | |
| *** yamamoto has quit IRC | 11:19 | |
| f0o | Would this be the right place to talk about issues with os-vif ? | 11:19 |
| *** tonyb has joined #openstack-neutron | 11:31 | |
| ivve | slaweq: sorry was at lunch, im checking now | 11:36 |
| openstackgerrit | Gregoire Mahe proposed openstack/neutron master: Allow to parse keywords in dns labels https://review.opendev.org/686343 | 11:37 |
| *** cloudkitten has joined #openstack-neutron | 11:37 | |
| cloudkitten | is segments_db.py:get_networks_segments() the best way to query for a few segmentation_id ? | 11:38 |
| *** do3meli has joined #openstack-neutron | 11:39 | |
| *** do3meli has quit IRC | 11:39 | |
| *** do3meli has joined #openstack-neutron | 11:40 | |
| *** ratailor__ has joined #openstack-neutron | 11:43 | |
| *** ratailor_ has quit IRC | 11:46 | |
| ivve | slaweq: okay so im not sure how to answer you but here it goes: it do get the error in logs "Exception during message handling: TooManyExternalNetworks: More than one external network exists". HOWEVER. i have 14.0.2 installed on all controller and compute nodes...(!?) | 11:51 |
| ivve | so i do have multiple external networks in tenants | 11:51 |
| ivve | and i can reach the flips of each instance | 11:52 |
| ivve | dhcp works when the instance is deployed | 11:52 |
| ivve | this only occurs once the instance is migrated | 11:52 |
| ralonsoh | f0o, and #openstack-nova, because Nova is the project using this library | 11:53 |
| *** ratailor__ has quit IRC | 11:53 | |
| ivve | and if (after migration) i restart neutron-openvswitch-agent, the L2 mesh is updated and i can again communicate with dhcp | 11:53 |
| ivve | slaweq: so i guess my short answer is no | 11:54 |
| ivve | im on my way to submit a bug, just collecting information to paste | 11:54 |
| ivve | basically just wanted to know if there is something known before i create it or if im doing something wrong | 11:55 |
| *** beagles|afk is now known as beagles | 11:57 | |
| f0o | ralonsoh: ok good to know.. basically I'm running into an issue that only affects vlan-backed networks when using the patch introduced in https://bugs.launchpad.net/os-vif/+bug/1837252 (CVE-2019-15753 / OSSA-2019-004) Do you happen to know if this is neutron or nova's domain? | 11:58 |
| openstack | Launchpad bug 1837252 in os-vif stein "[OSSA-2019-004] Ageing time of 0 disables linuxbridge MAC learning (CVE-2019-15753)" [High,Fix committed] - Assigned to sean mooney (sean-k-mooney) | 11:58 |
| *** markvoelker has joined #openstack-neutron | 12:01 | |
| ralonsoh | f0o what's the problem you have? | 12:01 |
| sean-k-mooney | f0o: the changes in the patch that adressed that issue are not effectected by the segmentaion type of the nuetorn network | 12:03 |
| *** yamamoto has joined #openstack-neutron | 12:04 | |
| ralonsoh | that's true | 12:04 |
| sean-k-mooney | e.g. that woudl effect vxlan or vlan networks equally | 12:04 |
| f0o | ralonsoh: basically the ageing prevents instances from getting network connectivity when backed by vlan networks (vxlan works fine) | 12:04 |
| ivve | slaweq: also, that bug is fixed in the version i have deployed. 14.0.2 | 12:05 |
| f0o | fdb shows the instance MAC on the provider network instead of the tap interface, the bridge is thus not moving the packets into the tap interface (I got pcaps to show that) | 12:05 |
| ralonsoh | f0o ?? In what situation? because as sean commented the segmentation type does not affect | 12:05 |
| f0o | flipping the ageing parameter back to 0 (disable mac learning) fixes it | 12:05 |
| ralonsoh | the change is https://review.opendev.org/#/c/672834/3/vif_plug_ovs/linux_net.py@135 | 12:05 |
| sean-k-mooney | f0o: i have a hard time beliving that give that we had aging enable in all previous releases | 12:05 |
| f0o | Stein has it disabled but Train has it enabled | 12:05 |
| f0o | so Train is breaking for me | 12:06 |
| sean-k-mooney | yes and every release before stien had again enabled | 12:06 |
| sean-k-mooney | the change in stien was a copy paste error effectivly | 12:06 |
| *** kevinz has joined #openstack-neutron | 12:06 | |
| f0o | then I wonder why packets arent being forwarded to the tap interface when ageing is enabled | 12:06 |
| f0o | because the mac seems to be learned from the wrong interface | 12:06 |
| f0o | and I cant seem to figure out why | 12:06 |
| ralonsoh | f0o ok this is not a problem in the ageing config | 12:07 |
| ralonsoh | but a problem in the ARP table | 12:07 |
| ralonsoh | and how it was populated | 12:07 |
| *** maciejjozefczyk has quit IRC | 12:07 | |
| *** yamamoto has quit IRC | 12:07 | |
| sean-k-mooney | f0o: what interface was the mac learned on ? | 12:07 |
| *** yamamoto has joined #openstack-neutron | 12:07 | |
| f0o | seems reasonable - I only came across the ageing issue because I have a refence host in Stein that works fine and that stood out as the obvious one | 12:07 |
| sean-k-mooney | the vlan interface? | 12:07 |
| f0o | mac was learned on ens* instead of tap* | 12:08 |
| sean-k-mooney | f0o: was the vm migrated?? | 12:08 |
| f0o | nope | 12:08 |
| sean-k-mooney | that could happen if it was it should not happen if it a new vm | 12:08 |
| sean-k-mooney | e.g. it was spawned | 12:08 |
| f0o | I rebooted the host a few times even and the VMs are a few days old and rebooted fresh when the host comes up | 12:09 |
| sean-k-mooney | have you logged in to the vm via the console and tried to ping the dhcp server | 12:09 |
| f0o | yeah I cant acquire DHCP even. on tcpdump I see DHCP replies coming into the br* but not being forwarded to tap* | 12:09 |
| *** spatel has joined #openstack-neutron | 12:09 | |
| sean-k-mooney | the way i would debug this is login via consol | 12:10 |
| f0o | if I set a static IP on the VM it works but with packetloss | 12:10 |
| ralonsoh | are you using hybrid or direct ports? | 12:10 |
| sean-k-mooney | then add the ip manuall then ping the dhcp agent | 12:10 |
| sean-k-mooney | then start tracing the packet flow | 12:10 |
| sean-k-mooney | ralonsoh: its linux bridge | 12:10 |
| sean-k-mooney | so that is not a thing | 12:11 |
| f0o | I should've mentioned before, I'm using linuxbridge and not OVS | 12:11 |
| sean-k-mooney | in linux bridge world you have 1 linux bridge per neutron netwrok on the host and all taps are addded directly too it | 12:11 |
| ivve | i think i have the same issue you do. however it only occurs when i migrate a machine (im using OVS tho) | 12:11 |
| sean-k-mooney | and then a vlan or tunnel port is added to the linux bridge | 12:11 |
| ralonsoh | right | 12:11 |
| f0o | sean-k-mooney: exactly, for whatever reason the MAC is learned from the vlan ens256.3002 interface instead of tap* | 12:12 |
| sean-k-mooney | ivve: when you migrate a vm it can have learned the mac on the upstream interface form its previous location | 12:12 |
| ivve | even tho vxlan is up (vtep) mac isn't learned for the two dhcp servers and i can't reach them | 12:12 |
| f0o | so packets are not being forwarded correctly - However if I disable aging it just magically works. vxlan backed networks are not affected whatsoever | 12:12 |
| sean-k-mooney | but when the vm starts it will send a rarp to advitise its new location | 12:12 |
| f0o | so I image it being some weird arp_announce magic that makes a loop | 12:13 |
| sean-k-mooney | f0o: that should only happen if the vm was moved although | 12:13 |
| f0o | hrm | 12:13 |
| ivve | whenever i restart neutron-openvswitch-agent on the host it becomes populated and things start working again | 12:13 |
| ivve | (dhcp that is) | 12:13 |
| f0o | what would the solution be if it was migrated tho? | 12:13 |
| sean-k-mooney | f0o: it would age out | 12:14 |
| sean-k-mooney | and teh mac would be leand form the tap | 12:14 |
| f0o | so 5 minutes of network disruption? | 12:14 |
| sean-k-mooney | no | 12:14 |
| sean-k-mooney | it should see that the mac is not directly attached | 12:14 |
| f0o | so I guess my issue is that the MAC is not being learned from the tap then | 12:14 |
| sean-k-mooney | as the tap will have the mac adress | 12:14 |
| sean-k-mooney | and that should cause it to be removed form the ens... interface | 12:15 |
| *** spatel has quit IRC | 12:15 | |
| sean-k-mooney | f0o: did you confirm it has the mac you expect? | 12:15 |
| sean-k-mooney | ivve: i think you problem is similar but unrelated | 12:15 |
| sean-k-mooney | ivve: are you useing dvr? | 12:15 |
| *** yamamoto has quit IRC | 12:16 | |
| sean-k-mooney | ivve: and if so are you using vlan networks? | 12:16 |
| f0o | sean-k-mooney: the instance has fa:16:3e:0d:c0:42, the tap has fe:16:3e:0d:c0:42 (I did not override any base-mac settings) | 12:16 |
| f0o | would that be the issue? | 12:16 |
| sean-k-mooney | no they are the same which is what we expect | 12:16 |
| f0o | on the other hand, the vxlan tap and interface also differs in the fa/fe initial | 12:16 |
| sean-k-mooney | so the mac learning table shoudl know that the tap is the destiation for that mac | 12:17 |
| *** tbachman has joined #openstack-neutron | 12:17 | |
| sean-k-mooney | oh sorry missed that | 12:17 |
| ivve | sean-k-mooney: using l3_ha dvr and OVS | 12:17 |
| sean-k-mooney | ivve: do you have vlan networks? | 12:17 |
| ivve | yes, many | 12:17 |
| ivve | and yes many in the same tenant | 12:17 |
| sean-k-mooney | ok did you also enable the l2_population driver | 12:17 |
| ivve | (external) | 12:17 |
| slaweq | ivve: ok, please report new bug than | 12:17 |
| openstackgerrit | Gregoire Mahe proposed openstack/neutron master: Allow to parse keywords in dns labels https://review.opendev.org/686343 | 12:18 |
| sean-k-mooney | ivve: to user dvr with vlan on ovs you are requried to run the l2_population driver | 12:18 |
| slaweq | ivve: sorry, I'm in training this week so I don't have too much time to work on it but I will try to check it | 12:18 |
| sean-k-mooney | mechdriver that is | 12:18 |
| ivve | hmm just askin but, if l2_pop was disabled, would i ever get anything to work? | 12:19 |
| f0o | sean-k-mooney: when I grep `bridge fdb` for the MAC of a vlan-backed vm, I see: fa:16:3e:0d:c0:42 dev ens256.3002 master brqa50c5b7b-db | 12:19 |
| sean-k-mooney | it shoudl work i belive | 12:19 |
| sean-k-mooney | ivve: ^ | 12:19 |
| f0o | sean-k-mooney: however, when I do the same for a vxlan backed vm, I see: fa:16:3e:83:2f:b5 dev tapb798e733-ad master brq3e90bc8f-0f | 12:19 |
| sean-k-mooney | but there are some negitive sideffects | 12:19 |
| ivve | [ml2] \n type_drivers = flat,vlan,vxlan \n tenant_network_types = vxlan \n mechanism_drivers = openvswitch,l2population | 12:20 |
| sean-k-mooney | ivve: https://specs.openstack.org/openstack/neutron-specs/specs/kilo/neutron-ovs-dvr-vlan.html | 12:20 |
| sean-k-mooney | ivve: ok so its not that then | 12:20 |
| ivve | slaweq: no problem, as long as we don't migrate anything without forgetting to restart neutron-ovs-agent we should be fine :P | 12:21 |
| ivve | its just a hard bug to detect since you won't notice until lease expires | 12:22 |
| *** maciejjozefczyk has joined #openstack-neutron | 12:22 | |
| sean-k-mooney | ivve: hum the spec actuly does not mention l2 pop which is weired | 12:22 |
| sean-k-mooney | f0o: ok so can you check 2 things for me | 12:23 |
| sean-k-mooney | f0o: 1 what is the mac on the neutron port | 12:23 |
| sean-k-mooney | f0o: 2 what is the mack in the libvirt xml | 12:23 |
| ivve | im gonna check a few things in ovs | 12:23 |
| ivve | im quite sure thats where the problem lies | 12:23 |
| f0o | sean-k-mooney: Neutron: fa:16:3e:0d:c0:42 / libvirt: fa:16:3e:0d:c0:42 (libvirt interface output: http://paste.openstack.org/show/785468/) | 12:25 |
| sean-k-mooney | f0o: ok so neutron and libvirt agree and in the vm? you see the fa:16:3e:0d:c0:42 adress too? | 12:26 |
| f0o | yep inside the VM I have the same MAC | 12:26 |
| sean-k-mooney | so its only the mac that has teh fe address? | 12:26 |
| ralonsoh | sean-k-mooney, I think when seg type is VLAN, the code is https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L103-L106 | 12:27 |
| sean-k-mooney | is there another interface on the host that has the fa address | 12:27 |
| f0o | the tap has the fe: address but inside the VM I have fa: | 12:27 |
| ralonsoh | sean-k-mooney, but "should_provide_vlan" is something from NOva networks | 12:27 |
| ivve | oh and also, the config has: [agent]\n tunnel_types = vxlan \n l2_population = true \n arp_responder = true \n enable_distributed_routing = True | 12:27 |
| f0o | and bridge fdb maps the fa: to ens256.3002 instead of tapb2b8c5ff-8c | 12:27 |
| sean-k-mooney | ralonsoh right but we should not be modifing the mac in os-vif | 12:28 |
| f0o | I can see on the vxlan backed vm that the fa: is mapped to the tap* interface, so I assume this should be the same for the vlan backed vm | 12:28 |
| ralonsoh | sean-k-mooney, yes, this is another probleem | 12:28 |
| ralonsoh | I don't know where is this happening | 12:28 |
| sean-k-mooney | ralonsoh: by should not i ment i dont think we are | 12:28 |
| sean-k-mooney | e.g. i think something else modified it | 12:28 |
| ivve | hmm | 12:28 |
| ivve | is there a certain magic order neutron/ovs needs to be started? | 12:29 |
| ivve | i might have found it if it exists lol | 12:29 |
| *** yamamoto has joined #openstack-neutron | 12:29 | |
| ivve | like neutron components overall.. vs ovs components | 12:29 |
| sean-k-mooney | ovs should be started first | 12:29 |
| *** nanzha has quit IRC | 12:30 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Add support for virtual port type https://review.opendev.org/676223 | 12:30 |
| sean-k-mooney | although the neutorn agent will try to recoonect if it is started before ovs | 12:30 |
| f0o | sean-k-mooney: the moment I send traffic over from the vm (such as DHCP requests), bridge fdb relearns the MAC but still puts it on the ens* interface and not on the tap* interface | 12:30 |
| ivve | i just restarted ovs-agent on controller nodes and after a migration one dhcp server is now reachable while other is not | 12:30 |
| ivve | (dhcp agent was recently restarted) | 12:30 |
| ralonsoh | f0o who has the fe* address? | 12:31 |
| sean-k-mooney | f0o: have you run tshark or tcpdump on the tap to verify it has teh fa source mac | 12:31 |
| sean-k-mooney | ralonsoh: the tap device | 12:31 |
| f0o | ralonsoh: the tap device on the host side | 12:31 |
| f0o | sean-k-mooney: do you mean from inside an instance or from the hypervisor? | 12:31 |
| sean-k-mooney | specifcly tapb2b8c5ff-8c | 12:31 |
| sean-k-mooney | f0o: tshark/tcpdump on the hypervior with a packet sent form in the vm | 12:32 |
| f0o | 12:32:28.969742 fa:16:3e:0d:c0:42 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:0d:c0:42, length 300 | 12:32 |
| f0o | # tcpdump -neli tapb2b8c5ff-8c udp | 12:32 |
| sean-k-mooney | ok so its sent with the correct mac from insed the vm | 12:32 |
| f0o | if I tcpdump the br* I will get the reply | 12:33 |
| sean-k-mooney | ok so the anti spoof firewall rules are not droping it | 12:33 |
| sean-k-mooney | actully they might be | 12:33 |
| sean-k-mooney | do you see the reply on the tap | 12:33 |
| sean-k-mooney | or just the bridge | 12:33 |
| f0o | just the bridge | 12:33 |
| f0o | neutron-linuxbri-ib2b8c5ff-8 is the incomming chain that handles traffic for that VM afaik. I added a -j ACCEPT at the top of it to rule it out | 12:34 |
| *** nanzha has joined #openstack-neutron | 12:34 | |
| sean-k-mooney | f0o: can you check the mac address on the vlan interface for me | 12:36 |
| sean-k-mooney | is it the fa address by any chance? | 12:36 |
| f0o | ens256.3002: fa:16:3e:ba:fa:33 / brqa50c5b7b-db: fa:16:3e:ba:fa:33 / tapb2b8c5ff-8c: fe:16:3e:0d:c0:42 | 12:37 |
| sean-k-mooney | ok that is fine | 12:38 |
| sean-k-mooney | well the tap is wrong | 12:38 |
| sean-k-mooney | but the vlan and bridge have the correct mac | 12:38 |
| f0o | all tap* have fe instead of fa though | 12:38 |
| sean-k-mooney | i was just checking that https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L72-L85 worked corerctly | 12:38 |
| ralonsoh | os-vif is not defining the TAP mac | 12:39 |
| sean-k-mooney | ralonsoh: yep just finished going through the os-vif code to confirm that too | 12:40 |
| sean-k-mooney | ralonsoh: and libvirt has teh correct mac set and the vm sees the correct mac | 12:40 |
| ralonsoh | sean-k-mooney, the point is the vlan interface should route the traffic into the tap device | 12:40 |
| ralonsoh | the vlan interface has the correct mac | 12:41 |
| ralonsoh | and will filter the vlan traffic | 12:41 |
| sean-k-mooney | ralonsoh: yes if the mac was learned form the correct interface | 12:41 |
| sean-k-mooney | f0o: can you provide your arp table for that bridge? | 12:41 |
| f0o | bridge fdb? or ip neigh? | 12:41 |
| sean-k-mooney | no i mean the output of arp command | 12:42 |
| ralonsoh | brctl showmacs <bridge> | 12:42 |
| sean-k-mooney | i want to confirm that there is not a static arp entry | 12:42 |
| sean-k-mooney | "brctl showmacs <bridge>" would also be useful | 12:42 |
| f0o | http://paste.openstack.org/show/785470/ | 12:43 |
| f0o | both outputs | 12:43 |
| f0o | http://paste.openstack.org/show/785472/ brctl showstp | 12:43 |
| f0o | to be able to identify port 1/2 ;) | 12:43 |
| sean-k-mooney | f0o: is this just happening on this 1 vm by the way or is it happening on others too? | 12:45 |
| sean-k-mooney | you dont have any staic arp entries which is good. | 12:46 |
| sean-k-mooney | it looks like you have a mac adress conflict | 12:46 |
| f0o | it happens on any vm with vlan backed network on any hypervisor running Train (or well, the fix that re-enabled ageing) | 12:46 |
| sean-k-mooney | ok | 12:46 |
| sean-k-mooney | i think the root cause is that the tap has the wrong mac | 12:46 |
| *** dtantsur is now known as dtantsur|brb | 12:47 | |
| ralonsoh | agree | 12:47 |
| sean-k-mooney | if it had the correct mac the "is local" | 12:47 |
| ralonsoh | (but I don't know how this mac ended there) | 12:47 |
| f0o | sean-k-mooney: but how comes it works with vxlan backed networks? | 12:47 |
| sean-k-mooney | field would be ture and it woudl take prescendes | 12:47 |
| sean-k-mooney | im guessing there is a bug i the linux bridge agent | 12:47 |
| sean-k-mooney | libvirt has no awareness of the segmentaiton type | 12:48 |
| f0o | I just want to confirm that this line is correct: fa:16:3e:0d:c0:42 dev ens256.3002 master brqa50c5b7b-db | 12:48 |
| sean-k-mooney | nor does os-vif | 12:48 |
| sean-k-mooney | no it is not | 12:48 |
| f0o | because on all vms that arent vlan-backed but vxlan-backed it shows `dev tap*` | 12:48 |
| f0o | eventho the tap does not have fa: but also uses fe: | 12:48 |
| sean-k-mooney | it should be on the tap | 12:48 |
| sean-k-mooney | oh | 12:48 |
| ralonsoh | even in vxlan?? | 12:49 |
| f0o | yep | 12:49 |
| ralonsoh | ok | 12:49 |
| sean-k-mooney | the fe adress is on the tap for vxlan | 12:49 |
| f0o | I can paste an example sec | 12:49 |
| sean-k-mooney | ok i guess that might be a redherring then | 12:49 |
| ralonsoh | ?? | 12:49 |
| sean-k-mooney | ralonsoh: the fact that the tap has the fe address might not be the issue. | 12:50 |
| *** spatel has joined #openstack-neutron | 12:50 | |
| sean-k-mooney | i still think its strange and would liek to know why that is the case but if its the same for vxlan it implies that something esle is happening | 12:51 |
| f0o | sean-k-mooney: ralonsoh : http://paste.openstack.org/show/785473/ | 12:51 |
| ralonsoh | the weird thing is this transformation from fa:**** to fe:***** | 12:51 |
| f0o | the obvious difference is that vlan backed shows on the provider interface while vxlan shows on the tap interface | 12:52 |
| ralonsoh | nonono | 12:52 |
| f0o | maybe provider is the wrong word ehre | 12:52 |
| ralonsoh | this is ok | 12:52 |
| *** frankwang has joined #openstack-neutron | 12:52 | |
| ralonsoh | the vlan interfaces and the tap devices have the same mac | 12:52 |
| ralonsoh | fe:16:3e:83:2f:b5 dev tapb798e733-ad master brq3e90bc8f-0f permanent | 12:52 |
| ralonsoh | fe:16:3e:83:2f:b5 dev tapb798e733-ad vlan 1 master brq3e90bc8f-0f permanent | 12:52 |
| ralonsoh | this is correct | 12:52 |
| f0o | I was more referring to the first two lines | 12:53 |
| *** nanzha has quit IRC | 12:53 | |
| f0o | fa:16:3e:0d:c0:42 dev ens256.3002 master brqa50c5b7b-db - vs - fa:16:3e:83:2f:b5 dev tapb798e733-ad master brq3e90bc8f-0f | 12:53 |
| f0o | shouldnt the first also be pointing to a tap*? | 12:53 |
| f0o | specifiaclly tapb2b8c5ff-8c | 12:53 |
| *** frank_wang has quit IRC | 12:54 | |
| ralonsoh | it should, we don;t create named interfaces | 12:55 |
| ralonsoh | *create those names interfaces | 12:55 |
| *** spatel has quit IRC | 12:55 | |
| sean-k-mooney | os-vif does | 12:55 |
| f0o | if I run `bridge fdb replace fa:16:3e:0d:c0:42 dev tapb2b8c5ff-8c master` it doesnt change anything sadly, the bridge-fdb looks correct with having tap* as the target for the mac but the VM still wont get the traffic | 12:55 |
| sean-k-mooney | and neutron dose create the ens256.3002 interface | 12:55 |
| ralonsoh | sean-k-mooney, not with this naming convention | 12:56 |
| ralonsoh | sean-k-mooney, ok yes, we do L179 LB agent | 12:57 |
| sean-k-mooney | ralonsoh: os vif does https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L72-L76 | 12:58 |
| ralonsoh | and the LB agent with "name.vlan" | 12:58 |
| sean-k-mooney | ya | 12:58 |
| ralonsoh | sean-k-mooney, ok ok | 12:59 |
| sean-k-mooney | in both cases it will end up with ens256.3002 i think | 12:59 |
| ralonsoh | and LB is creating this by default | 12:59 |
| ralonsoh | but we are not providing the MAC address there | 12:59 |
| sean-k-mooney | we do it on both sides to ensure it is created | 12:59 |
| ralonsoh | when creating the vlan interface | 12:59 |
| sean-k-mooney | yes although we can set it on line 79 if we have one | 13:00 |
| ralonsoh | sean-k-mooney, the point is when we are creating a vlan interface | 13:00 |
| ralonsoh | if the MAC is not defined | 13:00 |
| f0o | neutron/tests/unit/objects/test_common_types.py and neutron_lib/tests/unit/objects/test_common_types.py seem to derive macs starting with fe:16:3e... | 13:01 |
| ralonsoh | the vlan interface will inherit the mac of the upper interface | 13:01 |
| ralonsoh | right now | 13:01 |
| ralonsoh | 6391: d2: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 | 13:01 |
| ralonsoh | link/ether fe:bd:48:40:96:26 brd ff:ff:ff:ff:ff:ff | 13:01 |
| ralonsoh | 6393: vlan_d2@d2: <BROADCAST,NOARP,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 | 13:01 |
| ralonsoh | link/ether fe:bd:48:40:96:26 brd ff:ff:ff:ff:ff:ff | 13:01 |
| sean-k-mooney | ralonsoh: yes and the only place we invoke ensuer_vlan_bridge https://github.com/openstack/os-vif/blob/734d3bfa5d1324537f50cc95221703d4c19a734f/vif_plug_linux_bridge/linux_bridge.py#L105 | 13:01 |
| sean-k-mooney | we dont pass a mac | 13:01 |
| *** nanzha has joined #openstack-neutron | 13:02 | |
| ralonsoh | and that's ok | 13:02 |
| sean-k-mooney | yep | 13:02 |
| ralonsoh | because it will create an interface inheriting the mac | 13:02 |
| sean-k-mooney | it means thats dead code we can safely delete | 13:02 |
| f0o | ralonsoh: the vlan interface has the underlying interface's mac | 13:02 |
| sean-k-mooney | e.g. its only there form nova networks | 13:02 |
| ralonsoh | what's the brqa50c5b7b-db bridge mac?? | 13:02 |
| *** yamamoto has quit IRC | 13:02 | |
| *** nweinber__ has joined #openstack-neutron | 13:02 | |
| sean-k-mooney | f0o: yep that is what we expect | 13:02 |
| sean-k-mooney | ralonsoh: the bridge mac shoudl be the same as teh vlan mac | 13:03 |
| ralonsoh | I know | 13:03 |
| f0o | brqa50c5b7b-db has the same mac as the vlan interface | 13:03 |
| *** spatel has joined #openstack-neutron | 13:03 | |
| ralonsoh | ok, those FDB rules are created wrongly by Neutron | 13:05 |
| *** yamamoto has joined #openstack-neutron | 13:05 | |
| sean-k-mooney | ralonsoh: did you fined the point in the code? | 13:06 |
| sean-k-mooney | i was looking at https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py#L776-L788 | 13:06 |
| ralonsoh | sean-k-mooney, just a guess | 13:06 |
| ralonsoh | but the LB agent should be my initial searching point | 13:06 |
| ralonsoh | yes, exactly | 13:06 |
| f0o | ralonsoh: how should the fdb rules look like? | 13:08 |
| ralonsoh | f0o, I'm not 100% sure | 13:08 |
| ralonsoh | just checking | 13:08 |
| f0o | because I just replaced the one pointing to ens256.3002 to point to tapb2b8c5ff-8c instead and it didnt change anything | 13:08 |
| f0o | is there any sysctl setting that might be in the way? I dont have so much trust in Ubuntu anymore to be honest | 13:09 |
| f0o | http://paste.openstack.org/show/785474/ sysctl of net.ipv4.conf.$interfaces | 13:11 |
| *** jlibosva has quit IRC | 13:12 | |
| openstackgerrit | Merged openstack/networking-ovn stable/rocky: Avoid port group creation race https://review.opendev.org/687568 | 13:13 |
| *** jlibosva has joined #openstack-neutron | 13:13 | |
| ralonsoh | sean-k-mooney, https://www.redhat.com/archives/libvir-list/2012-June/msg01330.html | 13:16 |
| ralonsoh | "vnet0 is the backend of the guest NIC, and its MAC addr | 13:16 |
| ralonsoh | is more or less irrelevant to functioning of the guest | 13:16 |
| ralonsoh | itself, since traffic does not originate on this NIC. | 13:16 |
| ralonsoh | The only important thing is that this TAP device must | 13:16 |
| ralonsoh | have a high value MAC address, to avoid the bridge | 13:16 |
| ralonsoh | device using the TAP device's MAC as its own. Hence | 13:16 |
| ralonsoh | when creating the TAP Device libvirt takes the guest | 13:16 |
| ralonsoh | MAC addr and simply sets the top byte to 0xFE" | 13:16 |
| ralonsoh | https://openstack.nimeyo.com/48447/openstack-dev-how-does-instances-tap-device-address-generate | 13:16 |
| sean-k-mooney | right i was reading https://www.iana.org/assignments/ethernet-numbers/ethernet-numbers.xhtml | 13:17 |
| sean-k-mooney | FE-00-00-00-00 to FE-FF-FF-FF-FF IPv4 Addr Holders | 13:17 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Add support for virtual port type https://review.opendev.org/676223 | 13:18 |
| sean-k-mooney | so the 0xFE prefix is reserved for macs with ipv4 connectivity | 13:18 |
| sean-k-mooney | 10-00-00-01-00 to FC-FF-FF-FF-FF is Unassigned | 13:18 |
| sean-k-mooney | and we default to values in teh FA range in neutron | 13:18 |
| *** dave-mccowan has joined #openstack-neutron | 13:25 | |
| *** sridharg has quit IRC | 13:29 | |
| *** dave-mccowan has quit IRC | 13:29 | |
| *** ajay33 has quit IRC | 13:38 | |
| *** jmlowe has quit IRC | 13:46 | |
| *** kevinz has quit IRC | 13:47 | |
| *** ociuhandu has quit IRC | 13:48 | |
| *** kevinz has joined #openstack-neutron | 13:48 | |
| sean-k-mooney | ralonsoh: are you happy that f0o issue is on the neutron side by the way if so ill leave it in neutrons capable hands | 13:49 |
| ralonsoh | sean-k-mooney, but this FE* mac is being set by libvirt | 13:49 |
| ralonsoh | we are not creating/modifying this mac | 13:49 |
| sean-k-mooney | yes | 13:49 |
| sean-k-mooney | although since that is done for all vms | 13:50 |
| *** trident has quit IRC | 13:50 | |
| sean-k-mooney | that is not the issue | 13:50 |
| *** ociuhandu has joined #openstack-neutron | 13:50 | |
| sean-k-mooney | the question is what is different in the l2 agent for vxlan and vlan | 13:50 |
| sean-k-mooney | that results in the different behavior | 13:50 |
| sean-k-mooney | given that libvirt, nova and os-vif do the same thing in both cases | 13:51 |
| *** mlavalle has joined #openstack-neutron | 13:51 | |
| f0o | this turned out to be quite a long thread all of a sudden :| | 13:52 |
| *** trident has joined #openstack-neutron | 13:54 | |
| *** do3meli has quit IRC | 13:54 | |
| sean-k-mooney | f0o: well it appears to be a ligitmate issue in your env and we are not seeing it in the gate so its non triveial to resolve | 13:54 |
| sean-k-mooney | the gate supostion is based on the fact the jobs dont explode | 13:55 |
| *** ociuhandu has quit IRC | 13:58 | |
| ivve | im getting nowhere with this issue. its so random and strange | 13:59 |
| ivve | however | 13:59 |
| *** jmlowe has joined #openstack-neutron | 13:59 | |
| ivve | something that bother me is that neutron 14.0.2 is still getting "too many external networks" error | 14:00 |
| f0o | sean-k-mooney: I understand and agree. I'm unsure if neutron/nova is even to blame or perhaps it's just an ubuntu kernel oddity as well... Because in the end it's the kernel who forwards the packets within the bridge and the packets do arrive and outbound packets are being forwarded. So something must be off on that regard... | 14:01 |
| ivve | could it be that l3_ha + dvr + ovs is still affected by https://bugs.launchpad.net/neutron/+bug/1824571 ? | 14:02 |
| openstack | Launchpad bug 1824571 in neutron "l3agent can't create router if there are multiple external networks" [Medium,Fix released] - Assigned to Miguel Lavalle (minsel) | 14:02 |
| ivve | and somehow causes my problem with dhcp and migrations | 14:02 |
| *** nweinber_ has joined #openstack-neutron | 14:04 | |
| *** cshen has quit IRC | 14:05 | |
| *** kevinz has quit IRC | 14:05 | |
| *** nweinber__ has quit IRC | 14:07 | |
| *** yamamoto has quit IRC | 14:07 | |
| *** do3meli has joined #openstack-neutron | 14:15 | |
| *** ociuhandu has joined #openstack-neutron | 14:17 | |
| *** andyzon has quit IRC | 14:19 | |
| *** andyzon has joined #openstack-neutron | 14:20 | |
| *** igordc has joined #openstack-neutron | 14:23 | |
| *** andyzon has quit IRC | 14:25 | |
| *** gibi is now known as gibi_off | 14:29 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Update bug contacts https://review.opendev.org/690094 | 14:32 |
| *** andyzon has joined #openstack-neutron | 14:33 | |
| *** slaweq has quit IRC | 14:33 | |
| *** maciejjozefczyk has quit IRC | 14:34 | |
| *** maciejjozefczyk has joined #openstack-neutron | 14:38 | |
| *** yamamoto has joined #openstack-neutron | 14:42 | |
| *** dtantsur|brb is now known as dtantsur | 14:42 | |
| *** dswebb has joined #openstack-neutron | 14:46 | |
| *** yamamoto has quit IRC | 14:46 | |
| ivve | sean-k-mooney: do you have experience with larger environments (3control and 50+ computes) and ovs? | 14:47 |
| ivve | i think the issue i have i related with many requests, even tho i have the error (multiple external networks) if i migrate 1 instance at a time it works fine. 10 at a time. not so much | 14:48 |
| *** gcheresh_ has quit IRC | 14:49 | |
| sean-k-mooney | the largest clust i have personally run is about 20 nodes but 50 shoudl not be hitting scaling issues | 14:49 |
| ivve | thats my thought as well | 14:49 |
| sean-k-mooney | i know people do hit issue when you get into the mid hundereds due to amqp bottelnecks | 14:50 |
| ivve | well restarting l3 agent on the controllernodes do generate errors and it can take a very long time to get them fully up and running | 14:50 |
| sean-k-mooney | although i also belive the neutron team was working on improvment there | 14:50 |
| ivve | but the error with multiple external networks is worrying because i can only see 1 fip namespace on the computes now | 14:50 |
| ivve | instead of 1 per flip | 14:51 |
| sean-k-mooney | im not actully familar with that error but its possibel it still exists. i mainly work on nova these days so i dont follow neutorn too closely | 14:51 |
| ivve | ah alright | 14:51 |
| ivve | that error was the reason i stayed in rocky for quite some time | 14:54 |
| ivve | but tested it in a smaller env and didn't have that issue | 14:54 |
| ivve | now i moved it all up and things start behaving this way | 14:54 |
| *** lajoskatona has quit IRC | 14:55 | |
| *** pcaruana has quit IRC | 14:58 | |
| *** rkukura has joined #openstack-neutron | 14:58 | |
| *** jlibosva has quit IRC | 14:58 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: [WIP] [OVS] Handle added/removed ports in the same polling iteration https://review.opendev.org/690098 | 14:59 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Reset timeout exception in DietTestCase when retrying https://review.opendev.org/689658 | 14:59 |
| dswebb | sean-k-mooney, this slide by the cern admins says much the same. https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/presentation-media/Evolution-of-OpenStack-Networking-at-CERN3.pdf | 15:00 |
| dswebb | though they had a pretty crazy neutron setup with lots of flat networks | 15:00 |
| sean-k-mooney | ya so ovns decaritvie distubted state model rathar then imperitive "hay this chagne go do somthing" apparch that we have in ml2/ovs sacalse much better | 15:02 |
| *** ratailor has joined #openstack-neutron | 15:03 | |
| *** ratailor has quit IRC | 15:03 | |
| *** hjensas has quit IRC | 15:04 | |
| sean-k-mooney | although some of there slides are questionable | 15:04 |
| sean-k-mooney | main line the refence to dpdk when comapreing ovn/odl and ml2 ovs. | 15:04 |
| sean-k-mooney | ml2/ovs has the best support for dpdk out of all 3 | 15:05 |
| sean-k-mooney | although odl and ovn have better routing support when usign dpdk | 15:05 |
| sean-k-mooney | they have worse vm support | 15:06 |
| *** slaweq has joined #openstack-neutron | 15:06 | |
| *** armax has joined #openstack-neutron | 15:06 | |
| *** mmethot has quit IRC | 15:09 | |
| *** mmethot has joined #openstack-neutron | 15:09 | |
| slaweq | haleyb: about neutron-lib, I wanted to raise this on PTG and ask if there is any new volunteer to take care of it | 15:10 |
| slaweq | haleyb: but if You would update this doc now, You can simply put my name there | 15:11 |
| slaweq | haleyb: actually I think I will need to do more updates to this doc now | 15:11 |
| slaweq | so I can take care of it if You want | 15:11 |
| *** ivve has quit IRC | 15:13 | |
| haleyb | slaweq: i sent out a review, and had put "Neutron PTL" for now, https://review.opendev.org/690094 | 15:13 |
| slaweq | haleyb: thx | 15:13 |
| *** otsukahy has joined #openstack-neutron | 15:13 | |
| *** otsukahy has quit IRC | 15:13 | |
| *** trident has quit IRC | 15:13 | |
| *** maciejjozefczyk has quit IRC | 15:13 | |
| *** otsukahy has joined #openstack-neutron | 15:14 | |
| ralonsoh | davidsha, hmmm next meeting is during the PTG | 15:21 |
| *** dswebb has quit IRC | 15:21 | |
| ralonsoh | we'll see if we need to cancel it or we can reschedule | 15:21 |
| davidsha | ralonsoh, ack, are you attending? | 15:21 |
| ralonsoh | nope | 15:21 |
| davidsha | same, it's up to you. | 15:22 |
| davidsha | Is there going to be some kind of stream for it actually? | 15:23 |
| *** trident has joined #openstack-neutron | 15:26 | |
| *** mattw4 has joined #openstack-neutron | 15:26 | |
| ralonsoh | sean-k-mooney, I'll do my best with the LB problem | 15:27 |
| *** do3meli has quit IRC | 15:27 | |
| *** slaweq has quit IRC | 15:28 | |
| *** rubasov has quit IRC | 15:32 | |
| *** hjensas has joined #openstack-neutron | 15:38 | |
| *** ksambor has quit IRC | 15:38 | |
| *** slaweq has joined #openstack-neutron | 15:40 | |
| *** cloudkitten has quit IRC | 15:41 | |
| *** otsukahy has quit IRC | 15:43 | |
| *** otsukahy has joined #openstack-neutron | 15:44 | |
| *** slaweq has quit IRC | 15:45 | |
| *** otsukahy has quit IRC | 15:48 | |
| *** andyzon has quit IRC | 15:50 | |
| *** ircuser-1 has joined #openstack-neutron | 15:54 | |
| *** rubasov has joined #openstack-neutron | 15:57 | |
| *** lucasagomes has quit IRC | 15:57 | |
| *** otsukahy has joined #openstack-neutron | 16:03 | |
| *** otsukahy has quit IRC | 16:03 | |
| *** jmlowe has quit IRC | 16:06 | |
| *** otsukahy has joined #openstack-neutron | 16:09 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Temporary disable CI job neutron-functional-python27 https://review.opendev.org/689957 | 16:11 |
| *** macz has joined #openstack-neutron | 16:12 | |
| *** rpittau is now known as rpittau|afk | 16:17 | |
| *** igordc has quit IRC | 16:20 | |
| f0o | sean-k-mooney ralonsoh - So about my issue, should I open an issue on launchpad or is there anything I should be doing? | 16:21 |
| ralonsoh | yes please, open a bug in launchpad | 16:21 |
| f0o | will do :) | 16:22 |
| *** ccamposr has quit IRC | 16:22 | |
| sean-k-mooney | f0o: yes we sould track it as a sperate bug rather then reuse the cve | 16:25 |
| *** hjensas has quit IRC | 16:31 | |
| *** nanzha has quit IRC | 16:38 | |
| *** pcaruana has joined #openstack-neutron | 16:41 | |
| *** davidsha has quit IRC | 16:43 | |
| *** pcaruana has quit IRC | 16:52 | |
| *** markvoelker has quit IRC | 16:54 | |
| *** bobmel has joined #openstack-neutron | 17:02 | |
| *** dtantsur is now known as dtantsur|afk | 17:02 | |
| *** armax has quit IRC | 17:05 | |
| *** bobmel has quit IRC | 17:05 | |
| *** markvoelker has joined #openstack-neutron | 17:12 | |
| *** spatel has quit IRC | 17:15 | |
| *** tbachman has quit IRC | 17:22 | |
| *** priteau has quit IRC | 17:22 | |
| *** bobmel has joined #openstack-neutron | 17:24 | |
| *** otsukahy has quit IRC | 17:24 | |
| *** weifan has joined #openstack-neutron | 17:27 | |
| *** tbachman has joined #openstack-neutron | 17:28 | |
| *** ozzzo has quit IRC | 17:32 | |
| *** tssurya has quit IRC | 17:32 | |
| *** ociuhandu_ has joined #openstack-neutron | 17:32 | |
| *** mmethot_ has joined #openstack-neutron | 17:35 | |
| *** mmethot has quit IRC | 17:35 | |
| *** ociuhandu has quit IRC | 17:36 | |
| *** bobmel has quit IRC | 17:38 | |
| *** ociuhandu_ has quit IRC | 17:39 | |
| openstackgerrit | Merged openstack/networking-bagpipe stable/train: assorted fixes in networking_bagpipe.bagpipe_bgp module https://review.opendev.org/688811 | 17:42 |
| *** ociuhandu has joined #openstack-neutron | 17:43 | |
| *** andyzon has joined #openstack-neutron | 17:46 | |
| *** ociuhandu has quit IRC | 17:47 | |
| *** armax has joined #openstack-neutron | 17:49 | |
| *** mhen has quit IRC | 17:51 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: [WIP] Use threads insted of greethreads in IP monitor https://review.opendev.org/690144 | 17:54 |
| ralonsoh | do we have another problem in the CI? | 17:55 |
| ralonsoh | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d78/689658/3/check/neutron-tempest-iptables_hybrid-fedora/d78280a/job-output.txt | 17:55 |
| ralonsoh | 2019-10-22 15:19:50.319322 | controller | Collecting tempest===22.1.0 (from -c /opt/stack/requirements/upper-constraints.txt (line 622)) | 17:55 |
| ralonsoh | 2019-10-22 15:19:50.340347 | controller | Could not find a version that satisfies the requirement tempest===22.1.0 (from -c /opt/stack/requirements/upper-constraints.txt (line 622)) (from versions: 10.0.0, 11.0.0, 12.0.0, 12.1.0, 12.2.0, 13.0.0, 14.0.0, 15.0.0, 16.0.0, 16.1.0, 17.0.0, 17.1.0, 17.2.0, 18.0.0, 19.0.0, 20.0.0, 21.0.0, 22.0.0) | 17:55 |
| ralonsoh | 2019-10-22 15:19:50.427785 | controller | No matching distribution found for tempest===22.1.0 (from -c /opt/stack/requirements/upper-constraints.txt (line 622)) | 17:55 |
| *** otsukahy has joined #openstack-neutron | 18:05 | |
| *** tbachman has quit IRC | 18:07 | |
| *** otsukahy has quit IRC | 18:09 | |
| *** njohnston has quit IRC | 18:10 | |
| *** njohnston has joined #openstack-neutron | 18:11 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Update bug contacts https://review.opendev.org/690094 | 18:13 |
| *** igordc has joined #openstack-neutron | 18:14 | |
| *** ivve has joined #openstack-neutron | 18:15 | |
| jrosser | ralonsoh: I asked about that tempest package issue in #infra earlier and it seems that the pypi cdn may be inconsistent | 18:16 |
| ralonsoh | ansssss | 18:16 |
| ralonsoh | jrosser, thanks!! | 18:16 |
| *** tbachman has joined #openstack-neutron | 18:18 | |
| *** weifan has quit IRC | 18:22 | |
| *** ralonsoh has quit IRC | 18:22 | |
| *** CeeMac has joined #openstack-neutron | 18:26 | |
| *** cshen has joined #openstack-neutron | 18:26 | |
| *** spsurya has quit IRC | 18:30 | |
| *** weifan has joined #openstack-neutron | 18:31 | |
| *** weifan has quit IRC | 18:39 | |
| *** cshen has quit IRC | 18:42 | |
| *** igordc has quit IRC | 18:47 | |
| *** igordc has joined #openstack-neutron | 18:56 | |
| *** weifan has joined #openstack-neutron | 18:59 | |
| *** andyzon has quit IRC | 19:00 | |
| *** weifan has quit IRC | 19:00 | |
| *** weifan has joined #openstack-neutron | 19:00 | |
| *** tbachman has quit IRC | 19:03 | |
| *** pcaruana has joined #openstack-neutron | 19:04 | |
| *** weifan has quit IRC | 19:06 | |
| *** jmlowe has joined #openstack-neutron | 19:09 | |
| *** pcaruana has quit IRC | 19:19 | |
| *** slaweq has joined #openstack-neutron | 19:20 | |
| *** yamamoto has joined #openstack-neutron | 19:26 | |
| *** henriqueof has joined #openstack-neutron | 19:27 | |
| henriqueof | I've been using UniFi Network Controller as an instance on OpenStack for a while, after updating from Rocky to Stein my users can't get redirected to captive portal anymore, is there any significant change that can cause it? | 19:27 |
| *** yamamoto has quit IRC | 19:31 | |
| *** weifan has joined #openstack-neutron | 19:31 | |
| *** otsukahy has joined #openstack-neutron | 19:32 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Clean-up ssl packages from bindep.txt https://review.opendev.org/690388 | 19:32 |
| *** hjensas has joined #openstack-neutron | 19:36 | |
| *** weifan has quit IRC | 19:38 | |
| *** maciejjozefczyk has joined #openstack-neutron | 19:40 | |
| *** weifan has joined #openstack-neutron | 19:45 | |
| *** slaweq has quit IRC | 19:46 | |
| *** slaweq has joined #openstack-neutron | 19:50 | |
| *** do3meli has joined #openstack-neutron | 19:54 | |
| *** do3meli has quit IRC | 19:56 | |
| *** nweinber_ has quit IRC | 20:00 | |
| openstackgerrit | Thomas Morin proposed openstack/networking-bagpipe stable/train: bagpipe-bgp: cleanly ignore RTC route of unsupported type https://review.opendev.org/690395 | 20:04 |
| openstackgerrit | Thomas Morin proposed openstack/networking-bagpipe stable/stein: bagpipe-bgp: cleanly ignore RTC route of unsupported type https://review.opendev.org/690396 | 20:04 |
| *** maciejjozefczyk has quit IRC | 20:04 | |
| *** otsukahy has quit IRC | 20:05 | |
| *** otsukahy has joined #openstack-neutron | 20:05 | |
| *** weifan has quit IRC | 20:07 | |
| *** weifan has joined #openstack-neutron | 20:09 | |
| *** weifan has quit IRC | 20:22 | |
| *** tbachman has joined #openstack-neutron | 20:24 | |
| *** weifan has joined #openstack-neutron | 20:24 | |
| *** weifan has quit IRC | 20:29 | |
| *** weifan has joined #openstack-neutron | 20:30 | |
| *** aedc has quit IRC | 20:32 | |
| *** do3meli has joined #openstack-neutron | 20:34 | |
| *** weifan has quit IRC | 20:34 | |
| *** weifan has joined #openstack-neutron | 20:41 | |
| *** otsukahy has quit IRC | 20:46 | |
| *** weifan has quit IRC | 20:46 | |
| *** weifan has joined #openstack-neutron | 20:49 | |
| *** cgoncalves has quit IRC | 20:49 | |
| *** colby_ has quit IRC | 20:50 | |
| *** weifan has quit IRC | 20:53 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Conclude adoption of new enginefacade in Neutron https://review.opendev.org/545501 | 20:55 |
| *** weifan has joined #openstack-neutron | 20:55 | |
| *** cgoncalves has joined #openstack-neutron | 20:55 | |
| *** aedc has joined #openstack-neutron | 20:56 | |
| *** aedc has quit IRC | 21:01 | |
| *** weifan has quit IRC | 21:01 | |
| *** aedc has joined #openstack-neutron | 21:03 | |
| *** weifan has joined #openstack-neutron | 21:03 | |
| *** weifan has quit IRC | 21:05 | |
| *** weifan has joined #openstack-neutron | 21:05 | |
| *** slaweq has quit IRC | 21:08 | |
| *** weifan has quit IRC | 21:10 | |
| *** weifan has joined #openstack-neutron | 21:13 | |
| *** weifan has quit IRC | 21:18 | |
| *** slaweq has joined #openstack-neutron | 21:19 | |
| *** weifan has joined #openstack-neutron | 21:23 | |
| *** slaweq has quit IRC | 21:23 | |
| NobodyCam | Good Afternoon Neutron folks, I had a off the wall question: is it possible to change the provider for a network with deployed resources. | 21:26 |
| *** mattw4 has quit IRC | 21:40 | |
| *** mattw4 has joined #openstack-neutron | 21:41 | |
| *** otsukahy has joined #openstack-neutron | 21:44 | |
| *** armax has quit IRC | 21:56 | |
| *** otsukahy has quit IRC | 22:01 | |
| *** otsukahy has joined #openstack-neutron | 22:04 | |
| *** otsukahy has quit IRC | 22:07 | |
| *** otsukahy has joined #openstack-neutron | 22:07 | |
| *** otsukahy has quit IRC | 22:14 | |
| *** otsukahy has joined #openstack-neutron | 22:15 | |
| *** weifan has quit IRC | 22:25 | |
| *** weifan has joined #openstack-neutron | 22:28 | |
| *** otsukahy has quit IRC | 22:29 | |
| *** weifan has quit IRC | 22:34 | |
| *** weifan has joined #openstack-neutron | 22:35 | |
| *** otsukahy has joined #openstack-neutron | 22:40 | |
| *** otsukahy has quit IRC | 22:44 | |
| *** ivve has quit IRC | 22:45 | |
| *** ociuhandu has joined #openstack-neutron | 22:46 | |
| *** ociuhandu has quit IRC | 22:51 | |
| *** weifan has quit IRC | 22:53 | |
| openstackgerrit | Merged openstack/neutron-specs master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/688233 | 22:53 |
| *** tkajinam has joined #openstack-neutron | 22:53 | |
| *** weifan has joined #openstack-neutron | 22:54 | |
| *** armax has joined #openstack-neutron | 22:55 | |
| *** weifan has quit IRC | 23:05 | |
| *** weifan has joined #openstack-neutron | 23:18 | |
| *** weifan has quit IRC | 23:25 | |
| *** yamamoto has joined #openstack-neutron | 23:28 | |
| *** weifan has joined #openstack-neutron | 23:30 | |
| *** weifan has quit IRC | 23:34 | |
| *** weifan has joined #openstack-neutron | 23:35 | |
| *** weifan has quit IRC | 23:42 | |
| *** macz has quit IRC | 23:43 | |
| *** weifan has joined #openstack-neutron | 23:44 | |
| *** weifan has quit IRC | 23:48 | |
| *** weifan has joined #openstack-neutron | 23:51 | |
| *** weifan has quit IRC | 23:54 | |
| *** igordc has quit IRC | 23:55 | |
| *** mattw4 has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!