| *** macz_ has quit IRC | 00:00 | |
| *** YoursTruly has quit IRC | 00:10 | |
| *** tosky has quit IRC | 00:23 | |
| *** YoursTruly has joined #openstack-ansible | 00:31 | |
| YoursTruly | @admin0 it took some time as I had to reinstall whole host (to clean up after last try) | 00:31 |
|---|---|---|
| YoursTruly | http://paste.openstack.org/show/800479/ | 00:31 |
| *** dpaclt has quit IRC | 01:53 | |
| *** YoursTruly has quit IRC | 02:00 | |
| *** rfolco has quit IRC | 02:20 | |
| *** gillesMo has quit IRC | 02:22 | |
| *** gillesMo has joined #openstack-ansible | 02:23 | |
| *** cshen has quit IRC | 02:24 | |
| *** raukadah is now known as chandankumar | 04:08 | |
| nsmeds | Hello folks. We've been troubleshooting an issue which only affects a few OpenStack subnets, where _some_ instances in those subnets fail to ping the gateway. Today we realized it's specifically subnets that conflict with IP ranges configured on the controller network servers. | 04:16 |
| nsmeds | e.g. 10.0.3.0/24 is the lxcbr0 network, and if a user creates that 10.0.3.0/24 subnet on OpenStack they will experience the problem. | 04:16 |
| nsmeds | Digging around but thought I'd check here: anyone run into something similar before? I'm hoping there is simply a setting we need to adjust. | 04:17 |
| nsmeds | I think it's specifically because the controller has `10.0.3.1` assigned to it, which conflicts with the subnets gateway. | 04:24 |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** sri_ has quit IRC | 06:33 | |
| *** shyamb has joined #openstack-ansible | 06:35 | |
| *** shyam89 has joined #openstack-ansible | 06:35 | |
| *** shyamb has quit IRC | 06:35 | |
| *** sri_ has joined #openstack-ansible | 06:36 | |
| *** shyamb has joined #openstack-ansible | 07:04 | |
| *** shyam89 has quit IRC | 07:07 | |
| *** shyamb has quit IRC | 07:09 | |
| *** shyamb has joined #openstack-ansible | 07:28 | |
| *** cshen has joined #openstack-ansible | 08:04 | |
| *** andrewbonney has joined #openstack-ansible | 08:08 | |
| *** ThiagoCMC has quit IRC | 08:12 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| *** jbadiapa has joined #openstack-ansible | 08:23 | |
| admin0 | nsmeds, are you on a flat network ? | 08:25 |
| admin0 | in all my cases, the controller is never a network node, its either network or compute, so i have not faced it before | 08:25 |
| *** shyamb has quit IRC | 08:55 | |
| *** macz_ has joined #openstack-ansible | 09:44 | |
| *** macz_ has quit IRC | 09:49 | |
| *** rfolco has joined #openstack-ansible | 10:13 | |
| openstackgerrit | Merged openstack/openstack-ansible master: [doc] Mention uwsgi overrides in doc https://review.opendev.org/c/openstack/openstack-ansible/+/761206 | 10:27 |
| *** shyamb has joined #openstack-ansible | 10:34 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Unify deployment of ceilometer files https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/762183 | 10:44 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Use mysql user instead of root https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/764449 | 11:37 |
| *** macz_ has joined #openstack-ansible | 11:38 | |
| *** macz_ has quit IRC | 11:42 | |
| *** pto_ has quit IRC | 11:51 | |
| *** pto has joined #openstack-ansible | 11:51 | |
| *** pto has joined #openstack-ansible | 11:52 | |
| *** shyamb has quit IRC | 11:55 | |
| *** tosky has joined #openstack-ansible | 11:57 | |
| *** rfolco is now known as rfolco|ruck | 13:00 | |
| *** macz_ has joined #openstack-ansible | 13:02 | |
| *** macz_ has quit IRC | 13:07 | |
| *** jbadiapa has quit IRC | 13:24 | |
| *** jbadiapa has joined #openstack-ansible | 13:35 | |
| *** tobberydberg_ has quit IRC | 13:53 | |
| *** tobberydberg has joined #openstack-ansible | 13:56 | |
| *** rfolco has joined #openstack-ansible | 14:00 | |
| *** rfolco|ruck has quit IRC | 14:01 | |
| *** pto has quit IRC | 14:02 | |
| *** cshen has quit IRC | 14:23 | |
| nsmeds | @admin0 ah interesting - your network hosts are dedicated network hosts and don't run other OpenStack services? Our "network" hosts also run the RabbitMQ, Galera, Neutron Server, Memcached, Repo and Utility containers. | 14:45 |
| nsmeds | The tenant network is using VXLANs. | 14:45 |
| nsmeds | For the instances which fail in those subnets, we've noticed the gateway MAC is incorrect. It has the MAC of the bridge/tap interface, not of an interface in the router netns. | 14:47 |
| *** rpittau is now known as rpittau|afk | 14:51 | |
| nsmeds | We're seeing the ARP reply that includes the wrong MAC address as well. Still digging. | 15:08 |
| *** chandankumar is now known as raukadah | 15:08 | |
| *** ThiagoCMC has joined #openstack-ansible | 15:17 | |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible master: Update variables for default Zun deployments and AIO https://review.opendev.org/c/openstack/openstack-ansible/+/763562 | 15:24 |
| openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: DNM: Update zun role to match current requirements https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/763141 | 15:26 |
| *** rfolco is now known as rfolco|ruck | 15:32 | |
| *** klamath_atx has joined #openstack-ansible | 16:02 | |
| ThiagoCMC | Hey guys, how to configure: `dhcp-option-force: "26,1500"` ? | 16:03 |
| ThiagoCMC | I'm trying: | 16:03 |
| ThiagoCMC | neutron_dnsmasq_neutron_conf_overrides: | 16:03 |
| ThiagoCMC | dhcp-option-force: "26,1500" | 16:03 |
| ThiagoCMC | No effect | 16:03 |
| ThiagoCMC | Any idea? | 16:03 |
| ThiagoCMC | I found this option here: https://bugs.launchpad.net/openstack-ansible/+bug/1697981 - but doesn't seems to be working | 16:05 |
| openstack | Launchpad bug 1697981 in openstack-ansible "Override neutron_dnsmasq_neutron_conf_overrides is broken" [Medium,Fix released] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard) | 16:05 |
| *** cshen has joined #openstack-ansible | 16:07 | |
| *** macz_ has joined #openstack-ansible | 16:09 | |
| *** macz_ has quit IRC | 16:10 | |
| *** macz_ has joined #openstack-ansible | 16:10 | |
| ThiagoCMC | I remember that Neutron was supposed to calculate the correct MTU for my instances but, my br-vxlan has MTU = 1550 and my instances are still with 1450. Why? | 16:16 |
| ThiagoCMC | Gonna try this: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/defaults/main.yml#L127 | 16:19 |
| ThiagoCMC | But also found this: https://review.opendev.org/c/openstack/openstack-manuals/+/302165/4/doc/networking-guide/source/adv-config-mtu.rst#5 - so, not sure about how to make it automatic... :-P | 16:22 |
| admin0 | nsmeds, when you plan for growth, then the controllers or 2-3 dedicated network nodes will be point of failures .. so in my case,i always make the compute servers also network nodes | 16:59 |
| admin0 | in that way, a failure of one will not hamper the whole network or others | 17:00 |
| admin0 | it will only hamper some networks | 17:00 |
| admin0 | but since dhcp and router are in HA | 17:00 |
| admin0 | its redundant | 17:00 |
| admin0 | ThiagoCMC, gave up on magnum already ? | 17:00 |
| *** tosky has quit IRC | 17:04 | |
| jrosser | ThiagoCMC: i left this as a note to myself in my user_variables.yml about MTU http://paste.openstack.org/show/800509/ | 17:26 |
| jrosser | dont forget to set MTU on your actual interfaces on the hosts too or it won't do what you expect | 17:26 |
| jrosser | nsmeds: the one time i saw traffic going between interfaces it shouldn't (a different context to yours), making sure reverse path filter was enabled stopped the wierd behaviour | 17:29 |
| jrosser | OSA disables RPF by default but imho that should not be necessary and we made this patch to allow it to be controlled https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/751265 | 17:30 |
| ThiagoCMC | admin0, no, I didn't! But I got extra NIC cards yesterday, for Ceph Public, and I had to reconfigure my deployment, now, I double checking all the MTUs. | 17:42 |
| ThiagoCMC | jrosser, awesome! Thank you! I'll give it a try now. | 17:43 |
| jrosser | you’ll need to adjust “provider” as that’s the network name | 17:44 |
| ThiagoCMC | The option: "dhcp-option-force=26,1500" inside of `/etc/neutron/dnsmasq-neutron.conf` had no effect. | 17:44 |
| jrosser | look at the interfaces that get created on the compute node | 17:45 |
| jrosser | ip a with whatever the flag is for extra detail | 17:45 |
| jrosser | until you get those != 1450 no amount of messing with dhcp will help | 17:45 |
| ThiagoCMC | Hmm... I see. The `br-vxlan` is 1550 but, the Linux `vxlan-XXX` are still being created with 1450. | 17:46 |
| ThiagoCMC | I thought that Neutron was inheriting the MTU from the br-vxlan automatically. | 17:47 |
| ThiagoCMC | Gonna check your notes now. | 17:47 |
| ThiagoCMC | jrosser, what's the "provider" here: `ml2_conf.ini: [DEFAULT] - physical_network_mtus = provider:1500`, the `br-vxlan`? | 17:52 |
| jrosser | thats the name of the network | 17:52 |
| jrosser | in OSA there is too much name=vxlan type=vxlan which ends up with those things being vxlan:vxlan | 17:53 |
| jrosser | and i just can't deal with the level of confusion that creates | 17:53 |
| ThiagoCMC | Like this: | 17:53 |
| ThiagoCMC | neutron_provider_networks: | 17:53 |
| ThiagoCMC | network_mappings: "vlan:br-vlan,vxlan:br-vxlan" | 17:53 |
| ThiagoCMC | ? | 17:53 |
| jrosser | yes so thats name:vlan | 17:54 |
| ThiagoCMC | So, the name would be "vxlan"... | 17:54 |
| ThiagoCMC | ok | 17:54 |
| ThiagoCMC | Thanks! | 17:54 |
| jrosser | i think so for a stock config | 17:54 |
| ThiagoCMC | it worth to mention that this must be the default somewhere, since I don't even have that configured! lol | 17:54 |
| ThiagoCMC | And from what I remember, the "vlan:XXX" doesn't have to be a bridge, like "br-vlan", it should work with the plain network device as well... | 17:55 |
| ThiagoCMC | I used to have just "vlan:eth4" | 17:56 |
| jrosser | sure, I guess it’s sort of historical and also to support the AIO config | 17:57 |
| ThiagoCMC | Cool | 17:57 |
| ThiagoCMC | Isn't it better to configure the br-vxlan MTU to 1554, instead of 1550? Isn't the VXLAN overhead +54 ? | 18:01 |
| ThiagoCMC | Maybe even to 1574 for IPv6? | 18:02 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Use mysql user instead of root https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/764449 | 18:16 |
| *** jbadiapa has quit IRC | 18:21 | |
| ThiagoCMC | jrosser, any idea why this error happened: http://paste.openstack.org/show/800511/ ? | 18:45 |
| ThiagoCMC | The "br-mgmt" have nothing to do with the VXLAN networks! O_O | 18:46 |
| *** CeeMac has quit IRC | 18:55 | |
| jrosser | ThiagoCMC: i would say you are inadvertently using br-mgmt for your vxlan traffic | 18:55 |
| jrosser | check the neturon plugin config file to see what got put in there | 18:56 |
| ThiagoCMC | You're right! I'm so glad that I came down to this to double check! The `plugins/ml2/linuxbridge_agent.ini` has its local_ip from the br-mgmt! How this happened? lol | 18:57 |
| ThiagoCMC | :-O | 18:57 |
| ThiagoCMC | My `openstack_user_config.yml` doesn't have that "unused" block under `provider_networks:`, related to the "br-vxlan". I'm guessing that's why! Since the linuxbridge-agent doesn't run in a container anymore, I removed that block! | 19:00 |
| jrosser | be-mgmt is the last ditch fallback for selecting the ip to use for the vtep | 19:00 |
| ThiagoCMC | O_O | 19:00 |
| jrosser | it’s automagic otherwise, or you can add local_ip (I’m guessing here...) to the neutron config to set it yourself | 19:01 |
| * jrosser not at computer to check | 19:01 | |
| ThiagoCMC | Ok, thanks! | 19:01 |
| *** tosky has joined #openstack-ansible | 19:03 | |
| *** cshen has quit IRC | 19:34 | |
| ThiagoCMC | jrosser, the linuxbridge agent local_ip is finally correct! I had to bring back this: http://paste.openstack.org/show/800512/ | 19:41 |
| ThiagoCMC | It feels weird to use this block of code for services that are "bare metal"... | 19:42 |
| ThiagoCMC | I mean, the provider_networks under global_overrides... | 19:42 |
| jrosser | well provider_networks rather confusingly does several things | 19:44 |
| jrosser | it wires containers to bridges | 19:44 |
| ThiagoCMC | Cool, learned the hardway! LOL | 19:44 |
| jrosser | but also it determines what the neutron physical network mappings are | 19:44 |
| jrosser | this is what tripped you up https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L47-L52 | 19:44 |
| ThiagoCMC | Hmmm | 19:45 |
| jrosser | and specifically this https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-tasks/dynamic-address-fact.yml#L20 | 19:48 |
| jrosser | which falls back to ansible_host | 19:48 |
| ThiagoCMC | Hmmm... Maybe it should not be called "container_networks.*"... :-P | 19:49 |
| *** cshen has joined #openstack-ansible | 19:49 | |
| ThiagoCMC | Not sure though... lol | 19:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Clean up barbican.conf https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/759084 | 19:59 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Allow multibackend support for Barbican https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/762691 | 19:59 |
| *** mugsie has quit IRC | 19:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Add deployment of the external libraries https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/762842 | 19:59 |
| *** mugsie has joined #openstack-ansible | 20:01 | |
| ThiagoCMC | My Neutron setup is still in bad shape... ;-( | 20:27 |
| ThiagoCMC | Gonna have fun this weekend! LOL | 20:27 |
| *** andrewbonney has quit IRC | 20:38 | |
| *** rfolco|ruck has quit IRC | 21:04 | |
| *** skelly has quit IRC | 21:23 | |
| *** raukadah has quit IRC | 21:24 | |
| *** NewJorg has quit IRC | 21:25 | |
| *** NewJorg has joined #openstack-ansible | 21:25 | |
| *** redrobot has quit IRC | 21:27 | |
| *** fanfi has joined #openstack-ansible | 21:43 | |
| *** macz_ has quit IRC | 22:03 | |
| *** yann-kaelig has joined #openstack-ansible | 23:05 | |
| *** cshen has quit IRC | 23:10 | |
| fanfi | hello, is there anyone who can help me please. I would like to install os-octavia but i am failed on this error: https://pastebin.com/rpRj8CWN | 23:39 |
| fanfi | only one node have the issue | 23:40 |
| *** tosky has quit IRC | 23:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!