| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/954681 | 03:06 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Generate ca_bundle during cert creation for standalone backend https://review.opendev.org/c/openstack/ansible-role-pki/+/954628 | 06:02 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Allow certificates to be installed by specifying them by name https://review.opendev.org/c/openstack/ansible-role-pki/+/954239 | 06:02 |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/954681 | 07:29 |
| f0o | Hi, I'm poking around stabilizing IPv6 for tenant networks and I'm struggling wrapping my head around how to distribute the tenant networks onto the Top-Of-Rack switches. I figured as much that I need to use OVN-BGP for this but I'm a bit confused what is going to run the BGPD. Is it going to be only the hypervisors? only the gateway-hosts? Both? Also what's the state of | 08:45 |
| f0o | `enable_distributed_ipv6` in [ovn] ? | 08:45 |
| f0o | Feels like IPv6 is still very much experimental in OpenStack/Neutron for anything that isnt a vlan/flat network | 08:45 |
| f0o | or at least the documentation is very obscure about it. Showing a million ways how things *could* work but no clear indication of what is the preferred/reference way | 08:46 |
| noonedeadpunk | f0o: so I think there are 2 options available (at least) | 08:49 |
| noonedeadpunk | if we are not going back to discussion of ovn-bgp-agent (which we had somer time ago), I'd guess it's worth to rely on old bgp dragent | 08:49 |
| noonedeadpunk | so it would be quite same setup as for ovs afaik | 08:49 |
| noonedeadpunk | I have not tried that personally with OVN, but I heard miultople times that it just works | 08:50 |
| noonedeadpunk | we're having IPv6, but with OVN driver we do have ovn-bgp-agent | 08:50 |
| f0o | my main fear is the placement of a FRR instance on our gateway hosts which would break those as they host FRR themselves for their own connectivity to the rest of the fabric | 08:51 |
| noonedeadpunk | so ipv6 and ipv4 work quite alike, except that for IPv6 we do have subnet pool and allocations | 08:51 |
| noonedeadpunk | so frr should be placed on each gateway node | 08:52 |
| f0o | I have IPv6 working entirely fine without BGP by just using a flat-network'ed ovn-router which then ties into every tenant's vxlan network. But this puts an imense stress on the gateway host becoming a singular bottleneck obviously | 08:52 |
| noonedeadpunk | if gateway != compute ofc | 08:52 |
| f0o | ok that's very good to know | 08:52 |
| f0o | then regardless what I do, I need to move the gateway hosts away from the Top-Of-Rack routers | 08:53 |
| noonedeadpunk | so in our scenario we set FRR on just cvouple of nodes which act as a gateway | 08:53 |
| noonedeadpunk | well | 08:53 |
| noonedeadpunk | maybe you can continue doing the same way with flat network... not 100% sure.... | 08:54 |
| noonedeadpunk | but regardless of bgp, what would you need to do, is create an address scope and allocation pool with IPv6 subnet in it | 08:55 |
| noonedeadpunk | then you create an external network or add a subnet from this subnet pool to existing one (it can be same as for ipv4) | 08:56 |
| noonedeadpunk | and client routers should allocate an own /64 or smth from this same subnet pool and add them to there internal geneve/vxlan network | 08:56 |
| noonedeadpunk | while connect their router to the public subnet as external one, so that router would get ipv6 as well | 08:57 |
| noonedeadpunk | traffic would still come through the gateway, but you kinda not limited to single router, and they are spread across multiple ones. | 08:57 |
| f0o | The way Out is not the problem here | 08:58 |
| f0o | but the way back in | 08:58 |
| noonedeadpunk | well, thus we announce the router IP and tenant subnet with ovn bgp agent | 08:59 |
| f0o | if I have /48 on vlan123; and set an allocation pool to distribute /64 to tenants. Then I create a Tenant Router which sits on that vlan123 network as well as on the tenant's vxlan432 which now is in charge of 2001:db8:1::/64, how would I know that it is charge of it and route the traffic to that specific tenant router? | 08:59 |
| f0o | so regardless how I spin this, it seems I will always need ovn-bgp-agent in some form or shape to make that revpath available to my network infrastructure | 09:00 |
| noonedeadpunk | but bgp dragent should be announcing that as well | 09:00 |
| noonedeadpunk | at least we had exact same setup in OVS with it | 09:00 |
| noonedeadpunk | (and I heard it does work in OVN) | 09:00 |
| f0o | and since the agent needs to sit on the gateway hosts, I will need to migrate those out of the current ToR infrastructure and place them on some new dedicated machines somewhere | 09:01 |
| f0o | or am I thinking this wrong? | 09:01 |
| noonedeadpunk | I think for IPv6 you need BGP in some way or form, yes | 09:02 |
| noonedeadpunk | though this could be second option if not ovn-bgp-agent: https://docs.openstack.org/neutron/latest/admin/config-bgp-dynamic-routing.html | 09:02 |
| noonedeadpunk | as you would need to re-do ipv4 as well for it | 09:02 |
| f0o | trying to find a place to upload a diagram to show where my brain-knot is :D | 09:04 |
| f0o | https://imgur.com/a/VWnJ0Cg | 09:04 |
| f0o | so if those two links with `?` become "Gateway Hosts" and "OVN-BGP-Agent" then I know a path forward | 09:05 |
| noonedeadpunk | so with ovn-bgp-agent, vlan ext net is really a virtual vlan, which is present only inside of OVN | 09:05 |
| noonedeadpunk | while you don't need to have it on switches | 09:06 |
| f0o | I do for distributed_fip tho | 09:06 |
| noonedeadpunk | instead, you need to figure you a peering vlan or jsut use default route for that | 09:06 |
| noonedeadpunk | then, ovn-bgp-agent create a vrf on ovn gateway node, where fakes the vlan as noop device. And then frr picks up addresses from VRF for announcement | 09:07 |
| noonedeadpunk | so if you do distributed fip, then I guess you'd need that on all computes actually | 09:07 |
| noonedeadpunk | but also you may apply that not for all vlan networks... | 09:08 |
| noonedeadpunk | you still can have just regular vlans, and announce on others... | 09:08 |
| noonedeadpunk | but yeah | 09:08 |
| noonedeadpunk | btw, fip is not really applicable for ipv6... not helpful if you wanna have dual stack network though | 09:08 |
| f0o | I know but the fip stuff already made me span the vlan to all computes | 09:09 |
| noonedeadpunk | right... | 09:09 |
| f0o | the dragent; that just needs to run on *some* host and sets foreign next-hop like a route-server right? | 09:09 |
| noonedeadpunk | well, I mean. if it's only about ipv6, and you don't wanna touch ipv4 - I'd suggest really give old dragent a try | 09:10 |
| noonedeadpunk | I think it runs on neutron-api, but not 100% sure tbh.... | 09:10 |
| noonedeadpunk | maybe indeed on gateways... | 09:10 |
| f0o | yeah I'm reading the docs right now and I'm intruiged if it doesnt become it's own nexthop | 09:10 |
| f0o | if it can just announce the prefix with the OVN router's interface as next-hop directly then that would be absolutely ideal | 09:10 |
| noonedeadpunk | well, ovn router does not have directly exposed interface, does it? | 09:11 |
| f0o | it does because the extnet is a vlan | 09:12 |
| noonedeadpunk | oh, ok, yes... | 09:12 |
| f0o | so I can place the extnet's v6 subnet as linknet | 09:12 |
| noonedeadpunk | I don't think it does that though tbh... | 09:12 |
| noonedeadpunk | but not sure | 09:12 |
| f0o | then it and the switches (or even ToR) has direct L2 access | 09:12 |
| f0o | so if I had some BGP speaker that just tells me "2001:db:1::/48 next-hop 2001:db:0::12ab | 09:12 |
| f0o | where 2001:db:0::12ab is the IP from the tenant's router external interface | 09:13 |
| f0o | although in reality the linknet might just be a ULA instead of a GUA | 09:14 |
| f0o | doesnt really matter wther ULA or GUA in the end. IPv6 is free anyway heh | 09:14 |
| f0o | gonna take a look at the dragent codebase and investigate what it annunces as next-hop and if I can just patch it to do what I have in mind | 09:15 |
| f0o | thanks for the inputs! | 09:15 |
| f0o | will make a PR if there's anything of value coming out (but first walking the dog) | 09:16 |
| f0o | noonedeadpunk: okay it seems that dragent is spot-on what I want/need to get this to work for me | 10:11 |
| noonedeadpunk | okay, awesome then :) | 10:11 |
| noonedeadpunk | there's a code in osa to make it work, at least for OVS | 10:11 |
| f0o | You dont' happen to have a link to the OSA docs for it, do you? :D | 10:11 |
| noonedeadpunk | no idea about ovn to have that said | 10:11 |
| f0o | I mean if I can just recycle some of it, that would be good enough. The dragent itself doesnt seem to care about OVN/OVS | 10:12 |
| noonedeadpunk | I think it was it. but not sure: https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#bgp-dynamic-routing-service-optional | 10:12 |
| noonedeadpunk | looking at local config overrides - that indeed seems it | 10:14 |
| noonedeadpunk | and yeah, just this line triggers a lot of logic under the curttain: https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/vars/main.yml#L300 | 10:15 |
| f0o | gonna have to do some grep'ing | 10:16 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Allow certificates to be installed by specifying them by name https://review.opendev.org/c/openstack/ansible-role-pki/+/954239 | 10:16 |
| noonedeadpunk | we just added `neutron_dynamic_routing.services.bgp.bgp_plugin.BgpPlugin` to `neutron_plugin_base` and that was it... | 10:16 |
| noonedeadpunk | and then adding speakers through API | 10:16 |
| f0o | yeah it doesnt seem to have any other relation to OVS/OVN in the role either | 10:17 |
| f0o | so that's cool | 10:17 |
| jrosser | i think we do the same, and took some care to ensure that the bgp session between the dragent and the upstream router happens over some interface/route that setup specifically for that | 10:19 |
| jrosser | usual business of not mixing up data plane and control planes | 10:19 |
| f0o | jrosser: yeah that's a given :) | 10:21 |
| f0o | need to grep for `network-agent_containers` to find what hosts map to it | 10:22 |
| f0o | but I seem to be too stupid today to find it. hound only shows it in inventory/env.d/neutron.yml where it is defined | 10:23 |
| noonedeadpunk | f0o: there should be no network-agent_containers for OVN by default.... | 10:35 |
| f0o | I'm struggling to connect the dots between neutron_bgp_dragent -> network-agent_containers -> network-agent_hosts; I see dragent belongs to agent_containers; but agent_containers and agent_hosts has no connection other than similar name. | 10:38 |
| noonedeadpunk | this indeed was group for ovs/lxb | 10:38 |
| noonedeadpunk | yeah, that's the name:) | 10:39 |
| noonedeadpunk | as it's split via [-|_]hosts$ | 10:39 |
| noonedeadpunk | same for containers | 10:39 |
| f0o | huh | 10:39 |
| f0o | that explains my brainknot | 10:40 |
| noonedeadpunk | dyumanic_inventory..... | 10:40 |
| noonedeadpunk | but it may add more things as well... | 10:40 |
| noonedeadpunk | (defining the group) | 10:40 |
| f0o | so safest is to create an own group that includes neutron_bgp_dragent and nothing else? | 10:41 |
| noonedeadpunk | yeah | 10:41 |
| f0o | or tbh I might just add neutron_bgp_dragent to the network-infra_containers group | 10:41 |
| f0o | because those would make the most sense to have them | 10:41 |
| noonedeadpunk | or that | 10:41 |
| noonedeadpunk | but it's lxc | 10:42 |
| noonedeadpunk | or well | 10:42 |
| noonedeadpunk | depending if you use lxc | 10:42 |
| f0o | current network-infra is also lxc | 10:42 |
| f0o | so should be fine | 10:42 |
| noonedeadpunk | I don't think it is? | 10:42 |
| noonedeadpunk | I mean 0- network-agent is not | 10:42 |
| f0o | network-infra_* is | 10:43 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/env.d/neutron.yml#L72-L73 | 10:43 |
| noonedeadpunk | yes, but neutron_bgp_dragent was on agents | 10:43 |
| f0o | yeah but I can change that :D | 10:43 |
| noonedeadpunk | and agents are kind of gateway nodes on ovn | 10:43 |
| f0o | to infra | 10:43 |
| noonedeadpunk | unless it announces only what is locally served | 10:44 |
| noonedeadpunk | but dunno | 10:44 |
| f0o | I just re-re-reading the ovn-bgp-agent docs; not that I missed some detail last time. Because dragent also took me 2-3x reading carefully to realize it has correct next-hop (in the way that I want/need them) | 10:44 |
| f0o | specifically https://docs.openstack.org/ovn-bgp-agent/latest/contributor/drivers/bgp_mode_stretched_l2_design.html | 10:45 |
| f0o | because this reads very similar to dragent | 10:45 |
| noonedeadpunk | it probably is very similar | 10:47 |
| noonedeadpunk | the problem with that specific one, is that I think it's SB DB oriented | 10:47 |
| noonedeadpunk | and that is not really reliable, as ovn does not intend to sb db being directly used, so they tend to change api/behavior it without much of grace period | 10:48 |
| f0o | so I should aim for NB DB? | 10:49 |
| noonedeadpunk | with ovn-bgp-agent - yes | 10:50 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!