| janno | Can someone explain to me what has happenend to this commit? https://opendev.org/openstack/openstack-ansible/commit/01765b335868db13656b7221915181126ce5a34a#diff-c2a42d2e96826f541495fed0648d2c0e0d97c3d6 | 09:44 |
|---|---|---|
| janno | You can find it within the history under https://opendev.org/openstack/openstack-ansible/commits/branch/master | 09:44 |
| janno | but it seems it has never beed merged to master? | 09:45 |
| janno | *been | 09:46 |
| noonedeadpunk | janno: you are right and it never was merged: https://review.opendev.org/c/openstack/openstack-ansible/+/804109 | 09:54 |
| janno | noonedeadpunk: Would there have been any chance of finding it myself? | 09:55 |
| noonedeadpunk | but lately we've also refactored how rabbitmq_upgrade flag behaves during an upgrade | 09:55 |
| noonedeadpunk | janno: in all commit messages there is a `Change-Id` - you can search by it in gerrit | 09:56 |
| janno | Ah, thank you! | 09:56 |
| f0o | noonedeadpunk: can I change network_mappings post-deployment? i.e. without having to recreate the neutron networks? I noticed that I cannot update network_interface_mappings because the interface is already a port on the other bridge and ovs-vsctl errors with that | 10:11 |
| noonedeadpunk | well, there's no logic of removing already defined interfaces from bridges, it can only add new ones... | 10:12 |
| f0o | yeah but here both br-ext and br-fip would have the same physical interface which is no possible with vsctl apparently | 10:13 |
| noonedeadpunk | so while you technically can change it - it will not always work if interface was already in use before | 10:13 |
| f0o | so now I'm thinking if I can just change network_mappings instead, would that cause neutron issues? I can do the port-changes manually | 10:14 |
| noonedeadpunk | you probably can create a veth pair?:) | 10:14 |
| noonedeadpunk | frankly - I never tried that... | 10:14 |
| f0o | re veth; you mean make a standard bridge, add the physical interface to it, create 2 veths add them to the bridge and also to the ovs-bridge? | 10:18 |
| noonedeadpunk | yeah, kinda | 10:19 |
| noonedeadpunk | or anotrher ovs brdige, just created manually | 10:20 |
| noonedeadpunk | But eventually, I think it should be possible to change m,ap[pigns during the runtime | 10:21 |
| f0o | arguably its cleaner if I'm allowed to change network_mappings, then I can set both to br-ext and it should just work right? | 10:21 |
| noonedeadpunk | But I'd guess that you'd need to trigger neutron-ovs-agent to re-wire things | 10:22 |
| noonedeadpunk | it's totally cleaner to change mappings | 10:22 |
| f0o | actually if I move it the other way around; there is no rewiring needed on the gateway nodes | 10:25 |
| f0o | because br-ext was only used for vlan typed networks whereas br-fip was the public shebang. So from the gateway node's view, br-ext is not used | 10:26 |
| f0o | so if I can change the network_mappings to use vlan:br-fip,fip:br-fip instead of the old vlan:br-ext,fip:br-fip and manually remove br-ext from gatewaynodes and do the port-swap, then it should work right? | 10:28 |
| f0o | all of this with the assumption that I can just change network_mappings :D | 10:28 |
| f0o | But wait what am I saying - since br-ext is not in use on gateway nodes, I can just del-br it and do the port swap | 10:30 |
| f0o | then I can just use network_interface_mappings as before which is a safe thing | 10:30 |
| f0o | I'm really doing a 180 over and over again here but I think I'm cooking something | 10:30 |
| f0o | sorry for being verbose, it did help tho! | 10:30 |
| noonedeadpunk | yeah, sure, it often helps when you start typing :) | 10:31 |
| noonedeadpunk | so feel free to | 10:31 |
| f0o | well if my shell drops connectivity we all know that I messed it up :D | 10:31 |
| noonedeadpunk | so if it was ovs, I'd say that you might need to run neutron-ovs-cleanup after changing mappings | 10:32 |
| noonedeadpunk | as iirc it triggers re-wiring for ports | 10:32 |
| f0o | well I think I got the briges sorted out | 11:24 |
| f0o | all of this is now banking on the hope that network_mappings can hold duplicates, namely "vlan:br-ext,fip:br-ext" | 11:26 |
| noonedeadpunk | oh | 11:26 |
| noonedeadpunk | I don't think you can do that | 11:26 |
| f0o | :| | 11:26 |
| noonedeadpunk | actually, I'm pretty sure y9ou can't | 11:26 |
| f0o | so then I'm doomed entirely because I cant merge vlan and fip | 11:27 |
| f0o | but I need them both to run on the same physical interface | 11:27 |
| noonedeadpunk | veth pairs ?:) | 11:28 |
| f0o | that sounds awful for performance | 11:28 |
| noonedeadpunk | (I know) | 11:28 |
| f0o | duplicating packets 3x at a minimum | 11:28 |
| f0o | can I merge network_vlan_ranges ? | 11:29 |
| noonedeadpunk | but I really don't think you can use same interface twice in there from neutron prespecitve | 11:29 |
| noonedeadpunk | Well, the problem is that you can't really swap physical interface for the vlan networks in neutron | 11:29 |
| noonedeadpunk | or well, you can't natively | 11:30 |
| noonedeadpunk | you probably can in db | 11:30 |
| noonedeadpunk | but then it's smth you totally need to spawn sandbox for and test | 11:30 |
| f0o | so what if I rewrite network_vlan_ranges from vlan:100:2000,fip:2000:3000 to vlan:100:3000, then network_mappings becomes as easy as "vlan:br-ext" ? | 11:30 |
| f0o | For the existing fip networks, I would edit the DB entry directly to make them "vlan" | 11:31 |
| f0o | would that work? | 11:31 |
| noonedeadpunk | it can work, yes | 11:31 |
| noonedeadpunk | but you totally would need to trigger re-wiring of interfaces mnually | 11:32 |
| noonedeadpunk | as neutron will not do that just on basis of db change | 11:32 |
| noonedeadpunk | and also I guess it depends if it's ovn ovs | 11:32 |
| f0o | no re-wiring needed | 11:33 |
| f0o | compute nodes only use br-ext, gateway nodes only use br-fip. merging them is just a matter of telling vlan:br-ext on compute and vlan:br-fip on gateway nodes | 11:33 |
| f0o | right? | 11:34 |
| noonedeadpunk | eh, dunno | 11:34 |
| noonedeadpunk | so you wnat to make a different mapping based o nthe group? | 11:34 |
| f0o | yeah | 11:34 |
| noonedeadpunk | you can do that | 11:34 |
| f0o | so that way both classes (compute,gateway) have their currently deployed ovs-bridge as the "vlan" provider. And as long as I make the networks that use `fip` (3 of em luckily) in the DB to use `vlan`, neutron should be happy (given that vlan_ranges are set to vlan:100:3000) | 11:35 |
| noonedeadpunk | but chaning db record for networks with swapping physicxal network name should require changes in ovs | 11:36 |
| noonedeadpunk | like to use another briddge in namespaces | 11:36 |
| f0o | I think you're generally right but when you have gatewaynodes the only wiring that exist is for your public/fip networks, not the vlan ones. And you arent changing that location | 11:37 |
| noonedeadpunk | so you still need to tell neutron to re-wire veth for namespaces from one bridge to another? | 11:37 |
| f0o | Thats what I'm avoiding by reusing the existing bridges on both classes, they just happen to be named differently between them | 11:37 |
| noonedeadpunk | or dunno, as you're using ovn it's probably different | 11:38 |
| noonedeadpunk | but then with ovn I guess you'd need to sync state from neutron db to ovn nb db | 11:38 |
| noonedeadpunk | (neutron-ovn-db-sync-util) | 11:38 |
| noonedeadpunk | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#ovn-database-population | 11:39 |
| f0o | what I did right now was to manually delete br-ext (vlan provider) from all gateway nodes and attached the physical interface of that bridge to the br-fip (fip provider where the public vlan is). I now moved all traffic from the gateway cross-connect to the switch ports, So this went pretty fine as the gateway nodes had no other ports in the br-ext bridge, it was really | 11:39 |
| f0o | unused with no traffic on it at all. | 11:39 |
| f0o | For the compute nodes the entire thing is reversed, they dont even have a br-fip set up to start with, they only have br-ext. | 11:40 |
| f0o | so to merge them, I claim vlan:br-ext on compute and vlan:br-fip on gateway nodes. This will not change any wiring. But I need to tell neutron that the 3 `fip` networks are now `vlan` | 11:40 |
| f0o | and that last part I can likely just do in the DB and restart the agents/servers | 11:41 |
| noonedeadpunk | yeah, so I'm not sure about that db part, as in ovn restarting agents won't give you anything pretty much | 11:41 |
| f0o | once all of that is done, I should be able to enable distributed fip making the compute node bind to the vlans of the old fip networks which are now correct vlan | 11:41 |
| noonedeadpunk | as for ovn iirc neutron is not caring much about some of these things being in config | 11:42 |
| f0o | I'm lowkey banking on that as well, and for each OVN chassis nothing has changed. The unused bridge in each respective class (comp,gate) is gone and the used one continues to get ports | 11:43 |
| f0o | that's my hope & dreams at least | 11:44 |
| f0o | I gut says this should work but I'm afraid of some hidden ovs attribute on the bridge or chassis or so that will wreck it :D | 11:45 |
| noonedeadpunk | I'm concerned about flows tbh, but dunno | 11:49 |
| f0o | ml2_vlan_allocations is the neutron table I should be concerned about right? | 11:53 |
| f0o | I will check the flows in a bit as well | 11:53 |
| noonedeadpunk | well, I mean. There's nothing in neutron which will understand that DB has changed and that it need to do anything with that. | 11:54 |
| noonedeadpunk | And neutron DB is kinda a reference of how to propogate NB DB | 11:54 |
| noonedeadpunk | so this data could be somewhere in nb db - then you'd need to do respective changes there as well | 11:55 |
| noonedeadpunk | but again - I feel not that confident when it comes to networking | 11:55 |
| f0o | as far as I understood it, ovn only cares about the physical mapping, in this case br-* - neutron only cares about the "other" mapping (vlan/fip in my case) where the network is assigned to. Since I'm not changing the physical mapping on any node, it should be fine no? | 11:56 |
| noonedeadpunk | yeah true | 11:57 |
| f0o | I do change that "other" mapping by merging fip+vlan physical_network | 11:57 |
| f0o | I checked that there are no networks with the same vlan IDs but different physical_network | 11:57 |
| f0o | like I said, my gutfeeling says this should be working fine but my brain tells me there's something I overlooked and I'm about to nuke everything | 11:58 |
| f0o | maybe I need a whisky first for... plausible deniability or something | 11:59 |
| noonedeadpunk | could be, hehe | 11:59 |
| f0o | ok so ml2_vlan_allocations needs to be cleaned up and networksegments needs the physical_network updated | 12:09 |
| f0o | there's no occurances of br-ext or br-fip in the neutron database | 12:09 |
| f0o | now to the ovs stuff | 12:09 |
| noonedeadpunk | no, it really care only about the mapped naming... | 12:10 |
| f0o | that was what I suspected, just happy to see it was correct :D | 12:11 |
| f0o | nb does not have br-* information (took a backup with ovsdb-client backup unix:///run/ovn/ovnnb_db.sock OVN_Northbound and grepped it) | 12:15 |
| noonedeadpunk | Yeah, I was just thinking that you wanted to rename "vlan"/"fip" in a way as well | 12:17 |
| f0o | ok I found something in the SB DB in the Chassis > Other_Config - there's an attribute ovn-bridge-mappings which reflects the original network_interface_mappings | 12:20 |
| f0o | this might need updating | 12:20 |
| f0o | SB DB also has the `network_name` option set on all Port_Bindings which are either fip or vlan | 12:22 |
| f0o | "all" was a bit wrong. On two. wonder what ports those are | 12:23 |
| spatel | noonedeadpunk I have upgraded zed to 2023.1 and encounter this error in keystone. I am not able to attach cinder volume to vm now. - https://paste.opendev.org/show/b637xEcAknJygEkAYAcM/ | 15:36 |
| spatel | Any clue or did you see this before? | 15:37 |
| f0o | Well this was a nightmare | 15:52 |
| f0o | it absolutely did not work whatsoever | 15:52 |
| noonedeadpunk | f0o: so it all broke? | 16:21 |
| f0o | yep | 19:08 |
| f0o | and it took a while to revert it all and recover | 19:08 |
| f0o | I think distributed fip is not something that works to switch in runtime | 19:09 |
| f0o | because I saw the packets arrive on the compute nodes, right vlan too - so L2 existed. However the MAC of the fip was the router's - and nothing replied to it | 19:10 |
| f0o | I tried detaching and reattaching the FIP, shutdown and start the VM, nothing changed | 19:11 |
| f0o | at first I thought I just need to retrigger port creation for the flows or something but nope... The compute nodes wouldnt reply to FIPs no matter what | 19:11 |
| noonedeadpunk | f0o_: and newly created fips were not working either? | 20:05 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!