| shermanm | JayF: done. 2169200, 2169779, and 2169776, and I tried to create reasonable regression tests to go with. | 00:49 |
|---|---|---|
| -opendevstatus- NOTICE: zuul processing is broken since about 03:00 UTC, investigation is in progress, please be patient | 08:02 | |
| opendevreview | Elod Illes proposed openstack/ironic-python-agent-builder stable/2025.2: DNM: CI health test https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/1009162 | 08:39 |
| opendevreview | Elod Illes proposed openstack/ironic-python-agent-builder stable/2026.1: DNM: CI health test https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/1009211 | 11:32 |
| opendevreview | OpenStack Proposal Bot proposed openstack/ironic-ui stable/2026.2: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic-ui/+/1009223 | 12:36 |
| mdfr | dtantsur : Hey I have a small question regarding filter_device: https://review.opendev.org/c/openstack/ironic-python-agent/+/954024 ; was it intentional that filter_device ships without a built-in USB NIC filter? I faced the same issue (PiKVM/BMC USB NICs with duplicate MACs breaking inspection...) | 13:05 |
| mdfr | and would an opt-in option (ipa-ignore-usb-network-interfaces) be welcome upstream? Or is it intended to let users write their own hardware manager to filter as they want? :) | 13:06 |
| opendevreview | Hollie Hutchinson proposed openstack/bifrost master: Bifrost NTP configuration https://review.opendev.org/c/openstack/bifrost/+/895691 | 13:15 |
| opendevreview | Elod Illes proposed openstack/ironic-python-agent-builder stable/2026.2: DNM: CI health test https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/1009236 | 13:17 |
| TheJulia | mdfr: I guess the thought which comes to mind is maybe go ahead and just propose code to filter it out by default, since any pattern which is becoming a common problem shouldn't require tuning by a human or AI to make the software work on the first attempt | 13:20 |
| mdfr | alright! thank you TheJulia I'll look into how to address this properly | 13:29 |
| TheJulia | for.. the duplicate mac stuffs coming from ?hpe? gear, I think we explicitly ignore the range because it's a wild west hardware range of self-addressing, I don't know what pikvm does, but it sounds like it is using something outside of that allocation | 13:44 |
| mdfr | The BMC I see enrolled as baremetal ports are mostly AMI: their virtual USB NIC gets enrolled, and we have to delete those ports by hand… I don't think it happens with our HPE nodes | 14:13 |
| mdfr | for the PiKVM, we enrolled a node with it plugged in, so its USB NIC's MAC became a port of that node. When we moved the PiKVM to another node, enrollment failed because that MAC was already registered to the first node's port (expected behavior) | 14:14 |
| sathieu | Hello, is there a way to have the Ironic <-> IPA communication only one way? (I would prefer Ironic -> IPA). i.e. can the lookup and hearthbeat be the other way? | 14:20 |
| * sathieu is needing this for security | 14:21 | |
| opendevreview | Merged openstack/ironic master: Fix file descriptor leak in Cinder client connections https://review.opendev.org/c/openstack/ironic/+/1005972 | 14:22 |
| JayF | sathieu: That's something that's been proposed several times but hits significant technical hurdles when we try. If you can articulate your specific security concern, I can help direct you toward best practices for mitigating it. | 14:29 |
| JayF | sathieu: For instance, in high-security environments, we typically suggest you deploy a separate ironic API cluster configured to only permit access from ramdisks. Then for the other API, configure it in the opposite manner with the ramdisk-accessible endpoints disabled. | 14:30 |
| JayF | We published an OSSN about this architecture earlier this year. It's going to be somewhere in the early 100s IIRC. | 14:30 |
| sathieu | OK. Here is our current setup : We are deploying Kubernetes with metal3 + baremetal-operator + ironic standalone. | 14:32 |
| sathieu | We use virtual-media (mostly Dell iDracs), no PXE, no DHCP no tftp. | 14:32 |
| TheJulia | mdfr: easiest path may also just be to detail a bug in launchpad against ironic or ironic-python-agent where you detail the exact Mac PiKVM is offering. | 14:33 |
| sathieu | Currently, cluster-api and bmo is in the administrative area (with specific network fabrics), | 14:34 |
| TheJulia | sathieu: can you detail the data flow which is concerning? Specifically, when you use virtual media, your setup in a far more secure case because what is called an "agent token" is pre-generated and embedded into each machine's ISO image. That token's existance locks out IPA from doing the traditional lookup action against the machine in terms of generating a fresh token, and that token is used for mutual authentication of the | 14:36 |
| TheJulia | ramdisk and ironic for commands/data to flow back and forth. | 14:36 |
| sathieu | The problem is that I want to move the ironic in the same cluster as BMO. And stream from IPA to this cluster is not allowed (we have rules). | 14:38 |
| JayF | I'll note that you might get some interesting suggestions if you go into the metal3 slack as well. They're going to have more specific experience on deploying using the operator on Kubernetes. | 14:40 |
| JayF | Not that we're going to stop helping you here, just that they might have a different perspective | 14:40 |
| sathieu | Problems with having Ironic in its current zone is that I have to create a k8s cluster only for this tool. And I need this 4 times because of our network topology | 14:40 |
| JayF | Yeah, I mean, you cannot get away from the inherent requirement that ironic, be able to talk to remote agents. | 14:41 |
| JayF | You can use HTTP proxies or other similar techniques to bridge networks together to make it work, but there's no way you can just turn it completely off | 14:41 |
| sathieu | It's ok for us if Ironic is initiating the communication. | 14:41 |
| JayF | I would be really, really surprised if someone in the Metal3 community hasn't got a Kubernetes-flavored answer to this problem | 14:41 |
| sathieu | OK. I'll ask in the #cluster-api-baremetal slack | 14:42 |
| JayF | We actually have a spec for that which was implemented at a previous job and ended up being an absolute mess. It's a pretty hard technical nut to crack. | 14:42 |
| JayF | I suspect that specification will end up deleted after the PTG. We have an item to clear out old specs. | 14:43 |
| JayF | I just wanted to point it out in case you found it that it didn't give you any false hope 😐 | 14:43 |
| sathieu | Is the problem ithat Ironic don't know how to reach the node before the lookup? | 14:45 |
| JayF | The [post authentication] data flow is basically "IPA calls /v1/heartbeat" -> "Ironic calls /v1/commands on IPA" (GET and later likely POST) -> "IPA calls /v1/heartbeat" | 14:48 |
| JayF | both the info IPA sends thru and the info that Ironic sends thru are needed | 14:48 |
| JayF | doing "IPA can't talk to Ironic, but Ironic can talk to IPA" is possible but not easy | 14:48 |
| JayF | basically things like, Ironic asks IPA what clean steps are available via a GET api call | 14:49 |
| sathieu | And having a permanent websocket is not possible? | 14:49 |
| JayF | I don't know if we've persued something like a persistent websocket, the model has been REST since the inception of the agent. It's one of those things where the comms model is more structural than the agent itself. | 14:49 |
| JayF | I'm sure there are ways around it -- some infrastructural, even (you can put *whatever* you want in the ramdisk itself before IPA starts to help with this) | 14:50 |
| JayF | that's part of why I pointed you at the larger k8s+ironic community; I'm not as up to date on the best practices for those environments (cut from a more old-school sysadmin cloth) | 14:50 |
| JayF | but anything is possible :) | 14:50 |
| sathieu | OK. Thanks | 14:51 |
| JayF | I'll make a note in our PTG notes that we've had someone asking for this :) | 14:52 |
| JayF | so your preference is "Ironic talks to IPA, IPA doesn't talk to Ironic" | 14:52 |
| JayF | we get this feature request in both forms :) | 14:53 |
| sathieu | +1 | 14:53 |
| JayF | sathieu: fwiw, kubernetes.slack.com #cluster-api-baremetal if you're not familiar | 14:53 |
| * JayF is in there and will be watching :) | 14:53 | |
| TheJulia | fwiw, using something like a websocket to tunnel from ironic's conductor to IPA hasn't been explored since ultimately the key challenge is IPA functionally operates as two threads which don't really directly talk today. It wouldn't be impossible, but then the question becomes how is it initiated because that only works with known static addresses up front and unfortnately many operate with DHCP or SLaaC so there are no real | 15:01 |
| TheJulia | consistent way to lock into that model for all comms because you still need the initial call from IPA to the ironic deployment, and ultimately IPA still needs to be able to access the requested artifact(s) which may be hosted by ironic or by 3rd party endpoints. | 15:01 |
| TheJulia | All sort of rests on how exactly you have your deployment configured as well. | 15:02 |
| JayF | did we make that better with eventlet migration? | 15:02 |
| JayF | or make it worse by splitting it further | 15:02 |
| TheJulia | ... I don't really remember | 15:02 |
| JayF | me either | 15:02 |
| TheJulia | I think it got a little better | 15:02 |
| JayF | I remember the pain of the migration | 15:02 |
| JayF | I don't remember how we ended up solving it lol | 15:02 |
| TheJulia | but the thread boundry is still there | 15:02 |
| JayF | yeah I think what we did was still spawn the wsgi server that | 15:02 |
| TheJulia | For the needful, I think multiprocessing | 15:02 |
| JayF | *late | 15:02 |
| TheJulia | yeah | 15:03 |
| JayF | sathieu: that's the real reason why: the lookup/heartbeat calls populate config, which we need to *start* the API server at all ^^ | 15:03 |
| * JayF needs to wait until post-caff to tech support in IRC, I knew ^ that the whole time | 15:03 | |
| TheJulia | I mean, its possible to develop a new access/protocol model, and then tunnel over that, but you still have artifact access in general to content with which is not something you really want to try and ship over a websocket. | 15:03 |
| JayF | oh yeah, the image itself | 15:04 |
| TheJulia | yup | 15:04 |
| JayF | I would assume in this kinda model, that's hosted in lower security zone | 15:04 |
| JayF | (which inherently makes the answer to this: put a limited ipa only ironic-api wherever you put the image host) | 15:04 |
| JayF | this is why we never implement this feature; the long tail of "reasons you need access" can't all be put aside | 15:04 |
| TheJulia | likely, but that is just a guess and who knows if anyone has tuned local download settings or not in the ironic deployment | 15:05 |
| JayF | TheJulia: we need a driver model that like, would just mount the disk on the conductor, then we wouldn't have to worry about all this agentic communuication (/s!) (/s /s /s /s) | 15:05 |
| TheJulia | I mean, there could be, but it is not an easy lift and is functional interaction refactoring at an extreme level, thus maybe not worth the squeeze | 15:05 |
| JayF | it's borderline a different deploy interface entirely | 15:06 |
| TheJulia | Hi, Cthulhu here, lets talk about iSCSI | 15:06 |
| JayF | the agent model is really built around the comms more than anything else tbh | 15:06 |
| * TheJulia hears everyone scurry away and sips coffee instead | 15:06 | |
| TheJulia | yeah, survivability, adaptability over singular paths | 15:06 |
| sathieu | The IPA image is downloaded from the BMC, in the same security area (almost) as where is want Ironic to be. | 15:07 |
| sathieu | The provisioned image is already is a lower security zone | 15:07 |
| sathieu | also, we are using static address. | 15:08 |
| JayF | yeah that's what I'm saying: can you put an ironic-api, in "ramdisk endpoints only" mode, in that same area as the image host | 15:08 |
| TheJulia | well, the BMC proxies the IO across the BMC network as it connects outbound to the http(s) endpoint where the custom ramdisk has been written. | 15:08 |
| JayF | as we lay out https://docs.openstack.org/security-notes/OSSN-0106.html | 15:08 |
| TheJulia | metal3 would have to support that, but even then, the BMC accessing the artifact is not even the same endpoint as ironic's API itself | 15:09 |
| sathieu | yes I can, but my goal is to get rid of anything in this zone (and avoid this silly 3-nodes cluster for only Ironic) | 15:09 |
| JayF | yeah O' | 15:09 |
| JayF | **I'm curious to see if there are any k8s-infra shaped answers to this | 15:09 |
| JayF | because this feels like it's screaming for some kinda network tunnelling to solve your problem | 15:09 |
| JayF | the question I'd lean on is this: why do other metal3 deployers, like Ericson Software Technologies, not need this security limitation when y'all do? Find that gap and crowbar on it until you find a secure architecture folks can all be happy with | 15:10 |
| sathieu | This is time for me to go. I'll ask on Slack later (probably tomorrow). Thanks for all! | 15:10 |
| JayF | Anyone who has deployed Ironic has had the pain you are in now, in some shape or another: bare metal provisioning needs a lot of network access, and network operators are loathe to give access :) | 15:11 |
| JayF | o/ | 15:11 |
| opendevreview | Merged openstack/ironic master: Fix OCI pre-shared basic auth credential handling https://review.opendev.org/c/openstack/ironic/+/1007765 | 16:47 |
| opendevreview | Merged openstack/ironic master: Ansible: allow disabling per-node python interpreter override https://review.opendev.org/c/openstack/ironic/+/1006616 | 17:00 |
| opendevreview | Verification of a change to openstack/ironic master failed: Validate ansible_playbooks_path against permitted directories https://review.opendev.org/c/openstack/ironic/+/1006848 | 17:00 |
| TheJulia | I likely need to rebase that, I'll take a look in a little while | 17:04 |
| opendevreview | Verification of a change to openstack/bifrost master failed: [CI] Upgrade master from 2026.2 and add slurp job https://review.opendev.org/c/openstack/bifrost/+/1007253 | 17:44 |
| TheJulia | Jay, fwiw, I elaborated a little bit more on that topic on the etherpad | 18:12 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Validate ansible_playbooks_path against permitted directories https://review.opendev.org/c/openstack/ironic/+/1006848 | 18:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Enforce ansible_playbooks_path by default https://review.opendev.org/c/openstack/ironic/+/1006849 | 18:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Write ansible configdrive to a private temporary directory https://review.opendev.org/c/openstack/ironic/+/1006850 | 18:22 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009336 | 18:47 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: doc: Consolidate Trait Based Networking documentation https://review.opendev.org/c/openstack/ironic/+/1009337 | 18:47 |
| cardoe | clif: ^ made some changes to examples and wording as I consolidated | 18:52 |
| cardoe | TheJulia: that first patch I think makes it possible to attach a port in a port group but also ensure that all tenant ports get attached as the use requests | 18:57 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent-builder master: Fix ipa-b ci: el10 polkit and epel invocations. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/1007981 | 19:04 |
| TheJulia | stevebaker[m]: I think ^^^ should unwedge ipa-b | 19:05 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent-builder master: Fix ipa-b ci: el10 polkit and epel invocations. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/1007981 | 19:20 |
| clif | cardoe: any chance we can separate the corrections from the consolidation? also do we really have to consolidate everything into one page essentially? | 19:21 |
| TheJulia | I think one page does sort of make sense | 19:31 |
| clif | if consensus is one page I'm fine, im just a separatist | 20:14 |
| cardoe | I can split the corrections. | 20:17 |
| cardoe | The pages had the same content. They just didn’t have the exact syntax. They talked about the syntax in a high level. | 20:23 |
| cardoe | The three pages ago disagreed with each other. e.g. one said there was an attach action which operated on ports or port groups and you needed to use is_port or is_portgroup to make it trigger correctly. Which isn’t true. | 20:25 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009336 | 20:41 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: doc: Consolidate Trait Based Networking documentation https://review.opendev.org/c/openstack/ironic/+/1009337 | 20:41 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: doc: Correct Trait Based Networking documentation https://review.opendev.org/c/openstack/ironic/+/1009360 | 20:41 |
| TheJulia | clif: I mean... they were not wrong really ;) | 20:47 |
| TheJulia | (meaning, separists and star wars) | 20:48 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/1003598 needs some pep8 love | 20:48 |
| JayF | this is related to the OSSA I am issuing today | 20:48 |
| JayF | please someone find a minute to rev this if you can, otherwise I'll swing back around to it when I'm done authoring the ossa | 20:48 |
| JayF | and I mean someone; literally anyone, it's a backport | 20:48 |
| TheJulia | ugh, that boundry down to 2025.1 | 20:49 |
| TheJulia | pulling it down to fix | 20:49 |
| JayF | I specifically didn't ping you in hopes someone less busy would get it first lol | 20:50 |
| TheJulia | I got blocked on a human driven process | 20:50 |
| TheJulia | so I had a few minutes, it was that or keep trying to get the firmware to flash on a router which is refusing to take firmware because it's firmware is broken | 20:50 |
| JayF | https://review.opendev.org/c/openstack/ossa/+/1009362 up for review | 20:52 |
| TheJulia | pep8 is ignoring my override, ugh | 20:56 |
| TheJulia | and locally it says its okay | 21:00 |
| TheJulia | wut | 21:00 |
| * TheJulia waits for ci | 21:00 | |
| *** priteau_ is now known as priteau | 21:38 | |
| TheJulia | It passed on recheck | 21:41 |
| JayF | TheJulia: dtantsur: rpittau: https://review.opendev.org/c/openstack/security-doc/+/1009383 please review when you get a chance. I will announce very early tomorrow. | 22:38 |
| opendevreview | Merged openstack/ironic stable/2026.1: Add image_server_auth_hosts to restrict credential scope https://review.opendev.org/c/openstack/ironic/+/1003587 | 22:57 |
| TheJulia | looks pretty good to me | 23:09 |
| opendevreview | Verification of a change to openstack/ironic stable/2025.2 failed: Add image_server_auth_hosts to restrict credential scope https://review.opendev.org/c/openstack/ironic/+/1003597 | 23:59 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!