| opendevreview | Verification of a change to openstack/ironic master failed: ci: focus ironic-tempest-bios-ipmi-direct-tinyipa https://review.opendev.org/c/openstack/ironic/+/942204 | 01:49 |
|---|---|---|
| TheJulia | cardoe: huh? not sure I understand | 02:18 |
| TheJulia | cardoe: might help to have a higher bandwidth discussion | 02:18 |
| cardoe | Sure thing. Tomorrow? The more I think about it our network automation must have done something wrong. skrobul and I were doing manual testing on that box and I bet it still had some override. But 100% Ironic and Anaconda are busted. | 02:20 |
| opendevreview | Vasyl Saienko proposed openstack/ironic master: Enable atop on jobs https://review.opendev.org/c/openstack/ironic/+/942340 | 06:45 |
| opendevreview | Vasyl Saienko proposed openstack/ironic-tempest-plugin master: Make sure fixed IPs are different for multitenancy tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/942414 | 07:20 |
| frickler | cardoe: sorry my day ended faster than expected yesterday. we still need to clean up the terms I think, like somehow I'm unsure what the difference between VXLAN and VNI is for you. also do you want to see encapsulated VXLAN packets on the baremetal server port? | 08:13 |
| opendevreview | cid proposed openstack/ironic-python-agent-builder master: More reliable TinyIPA builds with network retries https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/942369 | 08:21 |
| vsaienko | ironic community, please review simple patch that enables atop installation on devstack environments https://review.opendev.org/c/openstack/ironic/+/942340 should helm to troubleshoot performance issues and unstable CI | 09:08 |
| opendevreview | cid proposed openstack/ironic-python-agent-builder master: More reliable TinyIPA builds with network retries https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/942369 | 11:51 |
| cardoe | frickler: there is no difference between VXLAN and VNI for me. I’d like to be able to reuse the same VNI number in the same neutron. And yes I would like to see VXLAN encapsulated traffic to the bare metal port. Because I could have OVS VTEP running on that machine participating. | 12:28 |
| frickler | cardoe: so the vxlan port would be a trunk subinterface on that bare metal port? or how would the bare metal port get its l3 connectivity? so far in neutron, the L3 connectivity for vxlan is part of the globalserver setup and not port- or tenant-specific afaict | 12:58 |
| cardoe | So let's ignore that piece for now cause I feel like that's where my conversation gets hung up each time. | 13:08 |
| cardoe | I've got two separate VXLAN fabrics. Both can have VNI 2000 running on them. How can I allow for that allocation in Neutron today? | 13:09 |
| cardoe | With VLAN, you'd define separate physical_network values and then there's no conflict. | 13:09 |
| frickler | cardoe: in neutron, you cannot, because as I said, vnis are bound to the whole server and thus you cannot have multiple domains for them. you would first introduce a concept of VRFs in order to have different L3 domains and then you could maybe tie VNI ranges to those | 13:11 |
| cardoe | But that's still taking an OVS view and not a bare metal view. | 13:13 |
| cardoe | Because port binding happens according to the physical_network field. So if my server has value X for some ports and value Y for other ports. I've already got that separation. | 13:14 |
| cardoe | No VRF involved because its physically separated. | 13:15 |
| frickler | cardoe: yes, I'm talking only about the hypervisor/instance focused via that neutron currently has (or as I interpret it) | 13:15 |
| frickler | cardoe: but you still have a single routing domain on your baremetal node? how does OVS distinguish vxlan packets from those two ports? | 13:16 |
| cardoe | Absolutely. And I agree with you about the VRFs in that case. | 13:16 |
| frickler | s/via/view/ | 13:17 |
| cardoe | The VXLAN traffic isn't coming up into a single OVS. | 13:18 |
| cardoe | In one case the traffic is coming up into a device doing NVMe over TCP so that Cinder can plumb stuff over the top. | 13:18 |
| cardoe | Today Cinder really expects your networks to be setup already for all your storage stuff. | 13:19 |
| cardoe | So maybe it'll help to use some RedHat terms (well really Triple-O) but I'm interested in the "undercloud" case. The hypervisor/instance focus you speak of would be the "overcloud". | 13:20 |
| cardoe | So I'm having my cinder backend actually talk to neutron to setup that physical fabric. | 13:21 |
| cardoe | And then the regular traffic side is nova talking to neutron. | 13:21 |
| cardoe | I could run two different neutrons. One for nova and one for cinder. But then when it comes to Ironic, I cannot toggle back and forth. That's the crux of my problem. | 13:25 |
| frickler | cardoe: I see. maybe treating these sides as different availability_zones might work? not sure if that would solve the "vni domains" issue right away, but might be a concept where this could be added to | 13:25 |
| cardoe | Oh. Now we're gonna peel back some other gross hacks I did. :D | 13:26 |
| cardoe | So right now since I cannot get the neutron network node to join the VTEP proper. I'm creating VLAN segments for each leaf switch on that VXLAN network. It's an idea I copied from the Juniper and Cisco ML2 mechanism. | 13:28 |
| cardoe | And in that case since the regular traffic to a bare metal server is really exposed out as a VLAN, those segments store the actual VLAN that box is plugged into. We'll we're trunking ports so there many VLANs with the trunking plugin. But that's another aside. | 13:30 |
| cardoe | But the result is I've spread neutron network nodes running OVN into different leaf switches. Their AZ is named their leaf as well. The OVN bridge mapping to the trunk port is the name of the leaf switch. | 13:31 |
| cardoe | And OVN will happily use a VLAN and tag the traffic. | 13:32 |
| cardoe | I can't have HA cause <insert bug here that jamesdenton_ will probably remember> | 13:33 |
| frickler | cardoe: so "neutron network node" is the overcloud node provisioned by the undercloud ironic? or are you talking about neutron in the undercloud? because in the latter case I'd fail to see the ironic connection | 13:35 |
| cardoe | And a VXLAN network cannot ever have two segments that live in cabinets with two different OVN nodes. So for example if I have OVN node in F20-1 and F20-5. If the rest of those cabinets have usable servers. I cannot build with ironic/nova a server into F20-5 and have the network use OVN from F20-1. | 13:35 |
| cardoe | This is neutron in the undercloud exclusively. | 13:36 |
| cardoe | So think of any OpenStack public facing provider like Vexxhost or OVH or Rackspace. | 13:37 |
| cardoe | Their public facing OpenStack would be the overcloud. | 13:37 |
| cardoe | Let's pick Rackspace's OpenStack Flex for example which OpenDev is consuming now days. | 13:39 |
| cardoe | They need bare metal for their hypervisors and all those OpenStack services. | 13:39 |
| cardoe | They come my undercloud. They build machines with Nova using their hypervisor OS image. They define networks in my neutron that carry their VXLAN or Geneve or whatever traffic along with whatever management they need. | 13:41 |
| cardoe | They want their Cinder to be able to have their hypervisors hook up to storage and so they come to me and I configure those network paths as well. | 13:42 |
| cardoe | My neutron using exclusively doing those bare metal switch configurations. | 13:43 |
| cardoe | But I've started running OVN to provide some ease of use comforts. Like when the box is first coming up, it's default gateway might be an OVN router and there for it can communicate to their configuration management system and fetch whatever it needs. | 13:45 |
| cardoe | Bigger footprint though is that my neutron runs neutron-fwaas and actually drives bare metal devices that ultimately are what their overcloud uses as their infrastructure. | 13:46 |
| frickler | so you are running OVN on the baremetal node and integrating it into the neutron-controlled OVN? | 13:46 |
| cardoe | No. | 13:47 |
| cardoe | They are running OVN on the baremetal node and it's controlled by their neutron. | 13:47 |
| cardoe | They're oblivious to how I set it up. To them it's just some boxes that are plugged into the same switch. | 13:48 |
| cardoe | I'm running an OVN on my control plane machines. | 13:49 |
| frickler | so maybe let's move back one step, why do you need/want vxlan to the baremetal node, what's bad about terminating it on the leaf switch? | 13:50 |
| cardoe | Nothing bad about terminating it on the leaf switch. | 13:50 |
| cardoe | btw I'm happy to have the convo in a higher bandwidth medium as well. | 13:51 |
| cardoe | I'm terminating it on the leaf switch today. | 13:51 |
| cardoe | BUT there's bugs with neutron and OVN there. | 13:52 |
| cardoe | So like I said. I make VLAN segments on the VXLAN network with the physical_network field set to the leaf switch name for each VLAN segment. | 13:52 |
| cardoe | That limits my OVN node which is providing that router for example (or DHCP) to 4096 - $OVERHEAD networks it can service. | 13:53 |
| frickler | ah, I'd have expected the vlan ids to be specific only per switch port, but in this case things might become limited pretty soon indeed | 13:56 |
| cardoe | I've got well in excess of that many networks running on my fabric today. | 13:56 |
| cardoe | I had tried to poke ovn-bgp-agent without success. https://bugs.launchpad.net/ovn-bgp-agent/+bug/2017890 exists for really adding that support. A person on that created https://github.com/toreanderson/evpn_agent/ which I haven't tried yet. | 13:57 |
| frickler | that's tore in #openstack-neutron I assume | 13:58 |
| cardoe | I've brought it up during their calls. | 13:59 |
| frickler | one day I'll need to look into ovn-bgp-agent myself. so far I've been fine with just using n-d-r | 14:00 |
| cardoe | Anyway. I hope that kind of makes sense to the use case. I'm trying to see what the best way is to present it to the community at large so that when I make RFEs and such there's a way to tie back to the "why". | 14:06 |
| cardoe | The biggest rub right now is the "I'm wanting VNI 200,000 on the storage fabric AND on the network fabric." | 14:06 |
| frickler | I was just about to write that reasoning why you need to have overlap there might be a good start ;) | 14:07 |
| jamesdenton_ | cardoe seems like we are attempting to overload the existing vxlan type to describe physical vxlan fabrics when its original intention was to be used for the single (software) overlay fabric managed by neutron? Therefore, the concept of a 'physical_network' equivalent doesn't really apply/exist? (FWIW, i don't know if that was its original intention, or not). | 14:15 |
| cardoe | jamesdenton_: yep. I think that's the crux of the problem. | 14:17 |
| frickler | +1, well summarized | 14:18 |
| frickler | and indeed one possible solution path might involve creating a new provider-network-type | 14:20 |
| jamesdenton_ | having *tenant provisioned* baremetal nodes participate act as a VTEP isn't a requirement, either, AFAIK. But rather, the Neutron nodes handling L3, DHCP, etc would benefit as VTEPs against the *physical fabric* - since the VNI describes a network across the fabric while the VLAN is only locally significant to a particular leaf. I guess today we have to fumble with segments to make sure a network node gets plumbed | 14:22 |
| jamesdenton_ | correctly? But then i seem to recall your issue with the actual implementation (re: the first segment not actually having something to bind against) | 14:22 |
| cardoe | frickler: I thought about that... "vxlan_evpn" or something. I'm bad at naming stuff. | 14:23 |
| jamesdenton_ | obviously, rackvxlan™ | 14:23 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: nova driver - get_secure_boot volume boot https://review.opendev.org/c/openstack/sushy-tools/+/942456 | 14:26 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: nova driver - get_secure_boot volume boot https://review.opendev.org/c/openstack/sushy-tools/+/942456 | 14:29 |
| mnasiadka | cardoe: Have you seen that ovn-core is working on a BGP implementation? Maybe they are solving your issues? (And I assume once that BGP code lands - ovn-bgp-agent might be sort of redundant?) | 15:27 |
| cardoe | Maybe? We'll have to see cause ovn-bgp-agent doesn't do the needful today. That's why there's evpn_agent. | 15:29 |
| cardoe | There's no configuration option that I can somehow have the anaconda deploy interface have a "switch to provisioning network" step and then later "switch to tenant network"? | 15:38 |
| cardoe | As far as I can tell, the anaconda deploy interface never creates the provisioning network port and never puts the machine on the provisioning network. | 15:41 |
| cardoe | But it then grabs the glance data and puts it on the conductor HTTP server, which is accessible from the provisioning network. It also requires the box to poke the heartbeat_url. | 15:42 |
| cardoe | Both of which aren't accessible from the tenant network. But it's putting the box on the tenant network and asking it to PXE boot and load that stuff. | 15:43 |
| jamesdenton_ | cardoe if the expectation is to hit the conductor api then i would expect that node to have been placed on the provisioning network and not the tenant network. But if the original test case was a "flat" provisioning/tenant network, then i could see how this might've worked. Just speculating. | 16:25 |
| cardoe | Imma just wait until TheJulia tells me what to do. :D | 16:33 |
| jamesdenton_ | that is usually best | 16:34 |
| TheJulia | wait, what did I do?! | 17:13 |
| TheJulia | sorry, day off and checking on neighbors danes and other tasks have kept me busy | 17:14 |
| TheJulia | vsaienko: as an fyi, we have a hashtag you can apply to changes your desiring reviews of | 17:15 |
| TheJulia | ironic-week-prio | 17:15 |
| TheJulia | JayF: so, I think the ideal primitive of physical network should still apply because there are crazy folks who do things like A/B fabrics | 17:20 |
| TheJulia | cardoe: so.. to a provisioning network should be part of the higher level flow before the driver really gets into driving | 17:22 |
| cardoe | So the only place I see "add_provisioning_network" being called in a prepare() method on a DeployInterface is for AnsibleDeploy and CustomAgentDeploy. Neither of which AnacondaDeploy inherits from. | 17:26 |
| cardoe | So that's why I'm wondering if that's something I can configure? or do I literally need to make my own AnacondaDeploy that inherits from CustomAgentDeploy? | 17:26 |
| * TheJulia raises an eyebrow | 17:29 | |
| TheJulia | give me a couple minutes to take some notes down and I'll look at the code | 17:29 |
| cardoe | This is btw after I patched the glance_service code to stop doing what it's doing. Which I think you wanted to sync on TheJulia | 17:30 |
| TheJulia | distracted, a wild nobodycam is talking to me about switch configuration in person | 17:36 |
| TheJulia | okay, so its based upon the pxe interface | 17:41 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/pxe.py#L72-L74 | 17:42 |
| TheJulia | because of the modeling, there is not a separate provisioning network with that interface to deploy, it drops it on the tenant network and expects it to boot/deploy from that point | 17:43 |
| TheJulia | if you need to isolate and switch, I'd but a flag in.. I guess | 17:43 |
| cardoe | okay so custom class it is. :D | 17:46 |
| TheJulia | cardoe: oh, the permission check, yeah. I chatted with Jay about that last night | 17:53 |
| cardoe | Not the permissions check | 17:58 |
| TheJulia | the bug jayf created yesterday? | 18:05 |
| JayF | I think cardoe's bug was different | 18:05 |
| JayF | just in the same neighborhood | 18:05 |
| JayF | something about the structure of a dict changing | 18:06 |
| JayF | and our tests (which work) mocking the original glance structure, while the code in production gets a slightly modified structure | 18:06 |
| TheJulia | oh, that | 18:09 |
| TheJulia | yeah, that has me confused | 18:09 |
| TheJulia | but I don't have enough details to be actionable on it | 18:09 |
| TheJulia | at least, personally actionable | 18:09 |
| JayF | I think cardoe seemed to have his head wrapped around it well enough that I think (hope?) he might take a swing at it | 18:13 |
| TheJulia | ++ | 18:37 |
| cardoe | okay back at my desk | 18:41 |
| cardoe | Actually no. I need some advice. | 18:42 |
| TheJulia | welcome back! | 18:42 |
| cardoe | That's what I was hoping to bug ya for TheJulia | 18:42 |
| TheJulia | rutro! | 18:42 |
| cardoe | why is golangci-lint destroying my CPU... | 18:43 |
| cardoe | why am I touching golang code... these yaks need to go away. | 18:43 |
| cardoe | https://etherpad.opendev.org/p/cardoe-ironic-anaconda-img-props so that's the details I took about the issue. | 18:43 |
| cardoe | But I'm happy to jump on higher bandwidth convo to explain the issue and get advice on how to proceed. | 18:44 |
| TheJulia | hypothetical question: If I can't use libvirt to create a VM, and I need get it an IP to use on devstack, should I just bind it out to virbr0? | 18:45 |
| TheJulia | hmm, not sure I'm groking | 18:46 |
| TheJulia | let me go to my RV for a call | 18:46 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic-python-agent master: WIP Add ContainerHardwareManager https://review.opendev.org/c/openstack/ironic-python-agent/+/941714 | 18:50 |
| TheJulia | cardoe: o/ https://meet.google.com/dks-gcwx-bga | 18:55 |
| JayF | devstack test vms for ironic usage don't have internet access in provisioning network, seemingly, I assume this is expected? cc: satoshi | 19:12 |
| JayF | and assuming so, anyone have experience making it work? | 19:12 |
| TheJulia | yeah, that is expected | 19:30 |
| JayF | Yeah, I think I'm going to have to figure out how to change that, even if just as a oneoff | 19:52 |
| JayF | since literally the point of the incoming hardware manager is to run things from a container registry | 19:52 |
| JayF | I guess option b might be spinning up a local registry on devstack | 19:52 |
| JayF | satoshi: ^ that is actually the route I'd go for now; there should be some howtos on setting up a local container registry and you could access that by-ip | 19:54 |
| vsaienko | TheJulia thanks, I've applied this hastag for vlan aware VMs patches | 20:40 |
| JayF | if I haven't said so, it's nice to see you around contributing o/ thanks Vasyl | 20:52 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix glance metadata layout https://review.opendev.org/c/openstack/ironic/+/942496 | 21:19 |
| cardoe | You know what we need? Two copies of get_image_properties() with different call interfaces. | 21:20 |
| cardoe | And just for good luck, we'll inline another copy in another function. | 21:20 |
| shermanm | even something like this works (replace docker/quay/whatever) | 21:29 |
| shermanm | docker run -d -p 5000:5000 \ | 21:29 |
| shermanm | -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \ | 21:29 |
| shermanm | --restart always \ | 21:29 |
| shermanm | --name registry-docker.io registry:2 | 21:29 |
| shermanm | proxies all requests to e.g. 127.0.0.1/namespace/containername:tag to registry-1.docker.io/namespace/containername:tag | 21:30 |
| cardoe | TheJulia: so other random discoveries in the anaconda deploy... it already requires the heartbeat_url to be hit with the "end" message. When it gets that, it attempts to teardown the provisioning network and switch to the tenant network. | 22:03 |
| TheJulia | That is what I would have expected, fwiw | 22:06 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: OS vmedia: Update device on eject_image https://review.opendev.org/c/openstack/sushy-tools/+/942498 | 22:21 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: Openstack vmedia - refactor to pre-defined volumes https://review.opendev.org/c/openstack/sushy-tools/+/942499 | 23:00 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: Openstack vmedia - refactor to pre-defined volumes https://review.opendev.org/c/openstack/sushy-tools/+/942499 | 23:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!