| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: [WIP] Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815224 | 08:29 |
|---|---|---|
| opendevreview | Merged openstack/ansible-role-pki master: Add support for setting extended key usage https://review.opendev.org/c/openstack/ansible-role-pki/+/815007 | 12:27 |
| mgariepy | i'm around now. | 12:29 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova stable/wallaby: [WIP] Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815835 | 12:50 |
| spatel | jamesdenton morning | 13:03 |
| spatel | i have very strange stuff going on with OVS, its doing unicast flooding on all port even my mac age time is 1800 | 13:03 |
| mgariepy | hey | 13:08 |
| mgariepy | i'm around now :D i wasn't yesterday . | 13:08 |
| mgariepy | do you use ovs flows or iptables-hybride? | 13:08 |
| spatel | hmm let me check | 13:33 |
| spatel | i think yes @mgariepy | 13:33 |
| spatel | checking now | 13:34 |
| spatel | firewall_driver = openvswitch | 13:34 |
| spatel | is this what telling ovs to use as firewall? | 13:35 |
| jamesdenton | https://bugs.launchpad.net/neutron/+bug/1732067 - maybe? | 13:35 |
| spatel | my iptables is empty so i believe firewall running inside OVS | 13:35 |
| spatel | I already read that but didn't understand what to do @jamesdenton | 13:36 |
| spatel | its 3 year | 13:36 |
| spatel | where do i set explicitly_egress_direct=True ? | 13:37 |
| spatel | and how? | 13:37 |
| spatel | i don't have any DVR or tenant networking. I am using VLAN base provider | 13:38 |
| jamesdenton | TBH i have not looked at this in a while and am not sure it fixes the problem completely | 13:40 |
| jamesdenton | what version are you running here? | 13:41 |
| spatel | 2.13.3 | 13:45 |
| spatel | ubuntu 20.04 | 13:45 |
| spatel | wallaby | 13:45 |
| spatel | I am seeing flooding happening for outgoing traffic only.. not inbound | 13:45 |
| spatel | seems like bug in openflow which doesn't understand how to learn address | 13:46 |
| spatel | my fdb/show always showing age is 0 | 13:46 |
| spatel | that means its constantly refreshing age | 13:46 |
| spatel | I have post question in mailing list also to see if anyone aware otherwise i have to open bug for this. | 13:47 |
| jamesdenton | hrm, yeah not sure if related | 13:47 |
| jamesdenton | for grins, you might try adding explicitly_egress_direct=True to the agent config | 13:47 |
| mgariepy | is it configured as DVR ? | 13:48 |
| mgariepy | oops. rereading sorry missed the line. | 13:55 |
| mgariepy | my fdb are resetting at 300. | 13:55 |
| mgariepy | but i'm on V. | 13:55 |
| opendevreview | James Gibson proposed openstack/ansible-role-pki master: Slurp all server certs not just first one https://review.opendev.org/c/openstack/ansible-role-pki/+/815849 | 14:00 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova stable/wallaby: [WIP] Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815835 | 14:02 |
| spatel | I don't have DVR @mgariepy | 14:04 |
| spatel | do i need to rebuild VMs after setting explicitly_egress_direct=True in /etc/neutron/plugins/ml2/openvswitch_agent.ini | 14:09 |
| mgariepy | https://bugs.launchpad.net/neutron/+bug/1732067/comments/79 | 14:15 |
| mgariepy | O_o | 14:15 |
| spatel | hmm | 14:20 |
| spatel | lots of words here not able to understand some of them :( | 14:20 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Bump OpenStack-Ansible Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/815333 | 14:24 |
| mgariepy | spatel, what are your other config on your brides ? # ovs-vsctl list bridge |grep "name\|other_config" | 14:35 |
| spatel | https://paste.opendev.org/show/810255/ | 14:36 |
| spatel | if i want to add explicitly_egress_direct=True/False then in which section i should be adding it here /etc/neutron/plugins/ml2/openvswitch_agent.ini | 14:38 |
| mgariepy | in the agent section : [agent] | 14:41 |
| mgariepy | https://docs.openstack.org/neutron/latest/configuration/samples/openvswitch-agent.html | 14:41 |
| spatel | let me add and restart openvswitch to see if any difference.. not sure if i need to recreate all vm | 14:42 |
| mgariepy | you probably do not need. | 14:42 |
| mgariepy | neutron should refresh stuff on restart. | 14:42 |
| spatel | nice lets give it a try | 14:42 |
| mgariepy | maybe check / dumps the flows and ovs config to see what changes. | 14:43 |
| mgariepy | is it a deployment with ovn or simply ovs ? | 14:44 |
| mgariepy | if not dvr i guess it's only ovs. | 14:45 |
| spatel | without OVN | 14:46 |
| spatel | this is legacy neutron-openvswitch-agent based deployment but i am using ovs-dpdk | 14:46 |
| spatel | I don't have any DVR etc.. my compute nodes directly get public IP and using vlan based provider (no VXLAN no vRouter) | 14:47 |
| spatel | vm directly talk to physical infra | 14:47 |
| spatel | i have dump openflow and now changing option to see if this change anything | 14:49 |
| spatel | damn!!!! it fixed my issue | 14:51 |
| spatel | no flooding at all | 14:51 |
| mgariepy | i guess you did try to restart neutron before ? | 14:52 |
| spatel | https://paste.opendev.org/show/810259/ | 14:53 |
| spatel | it did changed flow | 14:54 |
| spatel | i am going to revert my change and see if my issue come back | 14:54 |
| spatel | i revert option and flooding not happening.. ( i should wait and see.. ) i think it will come back | 14:58 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: Do not install ceilometerclient https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/815470 | 14:59 |
| mgariepy | the flows looks quite similar to me. | 15:05 |
| mgariepy | they were re-created tho. | 15:05 |
| mgariepy | what sha of W are you using for neutron ? | 15:05 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova stable/wallaby: Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815835 | 15:06 |
| mgariepy | spatel, you saw this issue out of nowhere after an upgrade or it was always there and you didn't saw it ? | 15:12 |
| spatel | This is new deployment | 15:14 |
| mgariepy | ok. | 15:14 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815224 | 15:14 |
| mgariepy | was ovs restarted because of an upgrade or something ? | 15:14 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815224 | 15:15 |
| spatel | mgariepy i don't know i did upgrade neutron, this is brand new deployment and i didn't do any upgrade | 15:18 |
| spatel | mostly i do restart whole node after building and running all playbook to verify everything come back clean.. | 15:19 |
| spatel | i may need more experiment to prove this | 15:19 |
| mgariepy | if/when you dig into it deeper let me know | 15:24 |
| spatel | sure.. are you also dealing with this kind of issue? | 15:28 |
| mgariepy | not yet but i might at some point. i'm just really curious really haha | 15:29 |
| spatel | sure.. may be i have bad case and as you said my issue might be neutron restart but it doesn't make sense.. | 15:31 |
| mgariepy | yep indeed. | 15:32 |
| mgariepy | if there were no bugs it wouldn't be fun .. right ? | 15:40 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815224 | 15:41 |
| spatel | jamesdenton mgariepy i did some more documentation around dpdk performance - https://satishdotpatel.github.io/openstack-dpdk-with-intel-x550-nic-part2/ | 16:57 |
| mgariepy | nice | 17:04 |
| spatel | DPDK cpu isolation is very important, if you don't do that performance go down.. | 17:42 |
| spatel | i did loadtesting and i can see impact | 17:42 |
| mgariepy | yep if you need the network to behave dedicated cores would stabilize perfs instead of putting them in the same pool as the vms. | 17:48 |
| spatel | I am doing loadtesting with realtime audio traffic and i can see audio quality digression when cpu sharing happen | 17:50 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| spatel | trying to understand relation of n_rxq option | 17:51 |
| spatel | i meant this one - | 17:51 |
| spatel | ovs-vsctl set Interface phy0 options:n_rxq=2 | 17:51 |
| mgariepy | yep not sure what it does really. | 17:57 |
| spatel | also i have noticed big improvement when you use sibling CPU for PMD | 17:58 |
| -opendevstatus- NOTICE: mirror.bhs1.ovh.opendev.org filled its disk around 17:25 UTC. We have corrected this issue around 18:25 UTC and jobs that failed due to this mirror can be rechecked. | 18:43 | |
| mgariepy | ( ͡ಠ ʖ̯ ͡ಠ) https://paste.opendev.org/show/810263/ | 19:55 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!