Tuesday, 2026-07-28

iurygregoryAnyone have a Dell PowerEdge R450? Or any Server from 15G? seems like they use Numeric ID (1 vs System.Embedded.1 or 1 vs iDRAC.Embeedded.1)01:14
TheJuliaWhat’s going on?02:20
iurygregorybasically a machine where NumericDynamicSegmentsEnable is Disabled is freaking out...02:30
iurygregorythey have numeric ID for Manager, which I personally find really weird for a Dell machine..02:31
rpittau'morning! looking for final approval for https://review.opendev.org/c/openstack/bifrost/+/932175 thanks! :)06:26
opendevreviewMerged openstack/ironic-ui master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/ironic-ui/+/99868308:04
opendevreviewMerged openstack/ironic-prometheus-exporter master: Disable neutron-ovn-agent  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/99864508:43
opendevreviewRowan Johns proposed openstack/tenks master: Add support for persistent networking  https://review.opendev.org/c/openstack/tenks/+/99880608:45
opendevreviewMahnoor Asghar proposed openstack/ironic master: Power off before ejecting redfish virtual media on ramdisk cleanup  https://review.opendev.org/c/openstack/ironic/+/99864010:59
dtantsurTheJulia: a wall of meetings today, but I'll try, thanks for heads-up11:54
iurygregoryrpittau, +W12:03
opendevreviewcid proposed openstack/ironic master: devstack: Support the container console provider  https://review.opendev.org/c/openstack/ironic/+/99585412:54
opendevreviewcid proposed openstack/ironic master: Document the container console provider  https://review.opendev.org/c/openstack/ironic/+/99585512:54
opendevreviewcid proposed openstack/ironic master: A non-voting job for the container console provider  https://review.opendev.org/c/openstack/ironic/+/99893412:54
dtantsurTeam, could we get some eyes on https://review.opendev.org/c/openstack/ironic/+/998640 and https://review.opendev.org/c/openstack/ironic/+/998628 please? These affect Metal3 and cause regular failures in the BMO job.13:14
TheJuliadtantsur: yeah, I too have a wall of meetings today13:44
TheJulia:(13:44
rpittauMahnoor: in https://review.opendev.org/c/openstack/ironic/+/998640 is it intended that we're doing a hard power off and not a soft one ?13:52
TheJuliaSo going back to Iury's question, it appears dell has introduced a "feature" to allow devices to be named based upon ID order in a list as opposed to the static expected resource names based upon the internal names. The issue seems to be, and Iury is working to confirm this, that some resources are still on their old device names even though we construct requests and URLs based upon the values discovered in place. In other words, 13:52
TheJulialooking like a but in the BMC which needs to be fixed. 13:52
dtantsurrpittau: very intended. Soft power off may not even work.13:52
dtantsur(it does fail in the BMO CI quite regularly since it relies on the operating system being up and accepting ACPI events)13:52
rpittaudtantsur: ack, thanks14:00
opendevreviewMerged openstack/bifrost master: Use cirros 0.6.3 as default cirros version  https://review.opendev.org/c/openstack/bifrost/+/93217514:44
rpittau\o/15:22
JayFcardoe: TheJulia: after going to Vegas, it'16:01
JayFcardoe: TheJulia: it's the food mecca which will bring us all together. It just needs a WH.16:01
JayFcardoe: TheJulia: Bojangles, In-n-out, Whataburger all in walking distance16:02
TheJuliawhere is this mystical, magical place with all three in close proximity16:02
JayFthey have bojangles everywhere now. one in miracle mile shops, one being built in linq promenade16:03
TheJuliawait... wut16:18
TheJuliaI'm deeply disappointed there is not a biscuitville16:20
opendevreviewClif Houck proposed openstack/ironic master: Add kernel parameter parsing to _prepare_iso_image  https://review.opendev.org/c/openstack/ironic/+/99230716:21
opendevreviewClif Houck proposed openstack/ironic master: Add kernel parameter parsing to build_extra_pxe_options  https://review.opendev.org/c/openstack/ironic/+/99230816:21
opendevreviewClif Houck proposed openstack/ironic master: Utilize KernelCommmandLine.parse() instead of ironic.common.utils.parse_kernel_params  https://review.opendev.org/c/openstack/ironic/+/99230916:21
opendevreviewClif Houck proposed openstack/ironic master: Add a tool to parse kernel command lines  https://review.opendev.org/c/openstack/ironic/+/99264116:21
opendevreviewClif Houck proposed openstack/ironic master: Add a deprecation notice for disable_kernel_parameter_parsing  https://review.opendev.org/c/openstack/ironic/+/99575016:21
JayFTheJulia: you know if you and Summer ever want to go on a Vegas trip, just let me know when and where 😂16:32
dtantsurTheJulia: I'm not sure https://review.opendev.org/c/openstack/ironic-specs/+/998805 can even work, especially the power parts16:33
dtantsurunless we suddently become okay with embedding IPA into user's instances...16:33
TheJuliaMy reading is we pivot through the ramdisk again to restart the user instance16:37
TheJuliathe point on the user instance is they are inaccessible and unreachable until the machine soft reboots16:37
TheJuliaso its boot ramdisk, unlock devices, trigger soft reboot and pivot network to the instance network16:38
JayFtalk about features that might be hard to test in CI16:38
TheJuliayeah, super difficult16:38
dtantsurTheJulia: soft reboot? why wouldn't that lock the device again?16:38
TheJuliano16:38
TheJuliaso soft reboot is similar to the host just going "we're going back to the beginning" on the processor16:39
TheJuliawhere as hard is full on "RESET ALL THE THINGS"16:39
TheJuliaand it actually holds down the interrupt to trigger it on the bus16:39
TheJuliaso a hard reset, triggers all devices watching that interrupt as well which includes the disk controllers and ultimately disks because it permutates all the way across16:40
TheJuliaoften hard reset also includes cutting power16:40
dtantsuruh, interesting. except that.. to do network switch, we use power off, not reboot.16:40
dtantsurI think the spec overly focuses on API additions (redundant IMO) and less on semantics16:40
TheJuliathat is a great callout, for this we would need switch similar to what we do with rescue16:40
dtantsuryeah, we got something for the disable_power_off feature16:41
TheJuliayeah, that was sort of my take, and I mentally viewed it as rescue with the callout. I think it could be a little more verbose as well in the interface they propose, but they have a working thing downstream so... kind of giving them a little grace that they have a lot of those details basically figured out and didn't want to get too deep in the details on the spec16:41
JayFWhat's our take on "we don't want this complexity for a limited use case"? :| 16:42
dtantsurJayF: that's kind of where I'm getting with it. If it was just a bunch of service/deploy/clean steps, I'd be cool with it.16:44
dtantsurBut we're talking about entire new state transitions and a serious rethink of power API.16:44
TheJuliaWell, in the defense of the idea, we already do similar with rescue16:44
JayFand the more steps there are in a simple deployment, the more likely they are to break16:44
TheJuliabut yeah, we would need to wire in on the power management calls to unlock disks16:44
TheJuliaor just reset them16:44
JayFTheJulia: I think the fact rescue isn't frequently used (at least to my knowledge) is a nod to how that complexity drives folks way16:45
JayF**away16:45
dtantsurIf rescue was invented today, it would be a service step combined with "hold" most likely.16:45
TheJuliaWell, a lot of folks try to wedge that as a solution for things outside of our supported flows16:45
TheJuliadtantsur: yup16:45
TheJuliathe key difference under the hood here is hardware disk locking16:45
TheJuliaand the need to do the unlock which does, obviously have some impacts16:46
dtantsurI don't mind the feature but I do mind the scope of user-visible changes16:46
dtantsurPower actions becoming a state transition that uses IPA makes my head hurt.16:47
TheJuliayeah, that *feels* like it should be implied16:47
TheJuliabut I can see why they do that today16:47
TheJuliaits... weird and then how do you deal with that with the api, really, the api sort of needs to hide/mask that16:47
JayFI disagree a lot with that16:48
TheJuliawell, depending on version16:48
JayFit should be easy for an operator to know when an IPA ramdisk is gonna boot on their node16:48
JayFhiding it as an implementation detail to hide the immense complexity of something that should be a basic action doesn't really improve things16:48
TheJuliaI more so mean, if you send like api version 1.110, then you see the new states16:48
TheJuliabut the key at the end of the day is it troubleshootable with an up to date client and use context16:49
JayFI'm going to open this up for a full review, but I am skeptical I'll ever find a +1 for this spec unless there is significant complexity that can be removed16:50
dtantsurOr they do it as a service step, which has a nice property of not modifying fundamental assumptions in Ironic16:50
TheJuliaThe real question even if it sosmehow a service step, is then how do you enable power to be translated across unless we have some flag/mechanism to re-map the step invocation to a series of steps16:52
TheJulia(which was sort of what I was thinking when reading it, as an ideal place to be, just not sure there is a magical get there in advance)16:52
JayFMy answer to that would be similar to dtantsur's "this has no power control" flag16:53
dtantsurI don't quite know what you mean by "power to be translated across" and why it has to be several steps.16:54
dtantsurSo yeah, in this deployment, normal power actions don't really work. You need to use "power_on_and_unlock" or "reboot_and_unlock" service/clean steps instead.16:54
TheJuliasome way to almost replace task.driver.power.power_on to a list of steps16:54
TheJuliaMaybe the right thing is a power driver wrapper16:55
JayFthat feature comes in a bucket labelled "may also contain 100 security vulnerabilities"16:55
dtantsurYou cannot touch the power interface itself - there must still be a way to make the low-level call. Chicken-and-egg problem otherwise.16:55
JayFthis is a lot of complexity around one of the things we rely on the most to ensure security: powering off machines16:55
TheJuliawell, you make a superset power_interface which understands there is additional workflow, but yeah, that gets weird16:55
TheJuliaoh, we can always just power off the machine16:56
TheJuliathe disks are locked as a result upon restart16:56
dtantsurI'm not talking about "where to put code" problem, I'm talking about "power action is now a state transition" problem.16:56
TheJuliaIts the power_on which needs additional logic16:56
TheJuliafair16:56
dtantsurFor low-level logic, whatever they need can be implemented in a derivative of the agent deploy interface most likely. Make sure every time we reboot for any reason, we tell IPA to unlock on return.16:57
dtantsurThe user-facing power API will not work as expected, and I'd rather disable it for such nodes than rethink it entirely tbh16:58
JayF^^ that is gonna be why I'm going to almost certainly -1 it16:58
JayFIPA required for power actions, and IPA boots themselves requiring a power action16:58
JayFmeans special casing and complexity being added to one of our most common paths16:58
JayFwhen the real answer is: don't use this style of drive locking with cloud provisioning16:58
dtantsur(On this positive note, I'm going to get some dinner, see you :)16:59
TheJuliaThe risk they are trying to manage is also, to use such devices to explicitly address risk of theft of the drive16:59
TheJuliaor of the chassis16:59
TheJuliagoes back to one of the older problems, ensuring data encrypted at rest16:59
JayFI mean, I appreciate the problem. Doesn't mean that Ironic's design choices enable this solution for it.17:00
JayFWe have a very similar design in my downstream, but the vendor handles this part in the BMC, in hardware17:00
JayFbecause we understand that this model of disk locking doesn't fit with cloud server provisioning17:00
TheJulia... interesting17:00
TheJuliaso I was sort of thinking such a model where the bmc or even base firmware enables a case to some sort of control service would be ideal in our model, because we have to do it in advance and don't really have to think about all the workflow implications17:19
JayFyeah, I put a review on that with a strong -1 with a tl;dr: We gotta find a way to do this without booting IPA *to execute a power action*17:20
JayFand that I don't think that's possible with the hardware as laid out in the spec17:20
JayFyou need something BMC/firmware side to help plug things in17:20
TheJuliathe only real alternative I guess would be something like magical-sed-controller.efi which is signed and able to do it17:22
JayFyeah, most solutions to this problem in the wild are shaped like17:22
JayFremote server holds the key; BMC Queries remote server on boot to get the key and unlock the drive17:22
JayFremote server can say "no" if things are in wrong states, or whatnot17:22
TheJuliaThe other and likely challenge is an inbound connetion for the BMC to be driving that is likely a possible issue because the BMC is largely limited to the PLDM over the I2C bus17:32
TheJuliaso it implies the bmc has a worrisome level of access or engagement with controllers which is, questionable17:32
TheJuliaFWIW, I started a thread on the mailing list regarding threat modeling and perceptions. If anyone is interested, it would be good to chime in18:21
opendevreviewJulia Kreger proposed openstack/ironic bugfix/37.0: bugfix/stable: lock constraints for bugfix to 2026.1  https://review.opendev.org/c/openstack/ironic/+/99851119:30
TheJuliaJayF: I guess I'm curious why https://review.opendev.org/c/openstack/ironic-python-agent/+/997637/3/ironic_python_agent/inspector.py ?19:48
clifhas anyone run into the fact that mypy doesn't seem to understand Ironic's objects' `fields`? As in, annotate a variable or method args with something like `objects.Node`, then try to access one of it's fields directly19:53
clifmypy will complain about the type not having the attribute and the tox mypy target will fail19:53
clifI'm not sure if there's some way to have mypy take 'fields' into account19:54
JayFTheJulia: backwards compat without having to add `or CONF.inspection_callback_url==mdns` everywhere19:54
cardoeclif: yeah I have the same issue.19:54
JayFTheJulia: think about the unmanaged inspection + mdns use case :( 19:55
TheJuliayeah, but your overriding the value to it explicitly in that path by turning it on, but after the config loading19:55
JayFI'm turning it on /based on the inspection url/19:55
TheJuliayeah, but if your putting a knob in, that should be set upfront19:55
JayFwhich is an explicit opt in19:55
JayFthe goal of this PR: "only do mDNS if opted in" inspection_callback_url=mdns is a valid opt-in, just like [mdns]use_mdns=True is19:56
clifI haven't dug into it yet, I'm hoping there's something better than telling mypy not to type check the line cardoe 19:56
JayFTheJulia: see: ironic_python_agent/agent.py:215 for the equivalent, non-api behavior19:56
JayFs/api/inspection/19:56
TheJuliaokay, looking at it through the slightly different lens I guess makes sense19:57
TheJuliait just does feel weird19:57
JayFI'll note: ipa-use-mdns is hooked up 19:57
JayFto kernel command line19:57
JayFbecause if you can modify kernel cli, you can ALREADY set arbitrary commands including Ironic API19:58
JayFso my policy was "anything opted in explicitly, command line or in config file, wins"19:58
JayFthis specific use case edge is something I worked a nontrivial amount on :) 19:59
cardoeclif: yeah I hope there is as well.20:02
TheJuliaI guess makes sense, doesn't completely explicitly close it, but at least the two patches makes it way more opt in, the second one functionally is the real lockout  then :)20:02
JayFthe [not yet written] third patch, to set a default value to the conf the 2nd patch adds, is a "better fix" potentially20:04
JayFI also think there's a door clearly open to get rid of url=mdns overrides and require config file opt-in on master20:05
JayFthis was just written shaped in the ... least offensive, most backportable way possible20:05
TheJuliaYeah, I think it is then, the second patch really being the lockout on other values20:09
JayFTheJulia: to be explicit: 997637 resolves the issue where ipa-api-url being unset would automatically engage mDNS20:10
JayFTheJulia: now, someone must set api-url=mdns OR ipa-use-mdns=true / [mdns]/use_mdns=True for mDNS to engage20:11
JayF(ipa-api-url=null, ipa-use-mdns/use_mdns unset == doesn't automatically mDNS as it used to)20:12
JayFhttps://review.opendev.org/c/openstack/ironic-python-agent/+/997637/3#message-006d597301c69c8ed2c35b57b6b9d57c41f2a91e I put this in writing here20:12
TheJuliayeah, I guess in my brain I'm more coupling the two together20:12
TheJuliabut inspection only *IS* different20:13
JayFipa-api-url being unset can happen in *weird cases*20:13
JayFipa-api-url (or inspection-callback-url) == mdns is never an accident20:13
JayFand while I don't love there are separate entrypoints for unmanaged inspection vs regular ironic use case ... meh, it's not that bad :)20:13
TheJuliaI guess in the separation, we really don't need a doc update then20:13
TheJuliabut we should just deprecate the feature out20:13
JayFyeah, but to me, the worst security hit of this bug -- a configuration being MISSING meaning we just go willy-nilly into mDNS -- is resolved by 99763720:14
TheJuliayeah20:15
UnknownI've been playing with standalone ironic (metal3 container image to start with) and am amazed how well this has been put together. Thanks so much everyone!20:16
*** Unknown is now known as Mike--20:16
Mike--meh was not identified it seems20:16
JayFIt's only taken us [check notes] about 12 years ;)20:17
Mike--it's a very nice piece of software20:17
TheJuliaJayF: do I have anything else to review for you that I forgot while you were on PTO?20:18
JayFI don't know the answer to that question yet lol20:19
TheJulialol20:19
JayFI am still loading in my swapfile :)20:19
TheJuliaoh my20:19
TheJuliai told you not to put your swapfiles on floppy disks!20:19
TheJulia... I did that once20:19
clifcardoe: we may need to load oslo_versionedobjects.mypy plugin20:20
cliftrying out a patch to enable this20:23
cardoesweet20:23
opendevreviewJulia Kreger proposed openstack/ironic-specs master: Add spec for sonic and nvue driver replacements.  https://review.opendev.org/c/openstack/ironic-specs/+/99836320:31
TheJuliawould we have capacity... (as a group) to do some review jamming tomorrow ?20:36
TheJuliadtantsur: jfyi, I'm digging the diff on https://review.opendev.org/c/openstack/ironic-specs/+/995954/3..4/specs/approved/standalone-networking-v2.rst20:46
clifhuzzah progress20:47
opendevreviewClif Houck proposed openstack/ironic master: Add a custom configuration type KernelParameterString  https://review.opendev.org/c/openstack/ironic/+/99230620:48
opendevreviewClif Houck proposed openstack/ironic master: Add UnsafeKernelCommandLine class  https://review.opendev.org/c/openstack/ironic/+/99574920:48
opendevreviewClif Houck proposed openstack/ironic master: Add kernel parameter parsing to _prepare_iso_image  https://review.opendev.org/c/openstack/ironic/+/99230720:48
opendevreviewClif Houck proposed openstack/ironic master: Add kernel parameter parsing to build_extra_pxe_options  https://review.opendev.org/c/openstack/ironic/+/99230820:48
opendevreviewClif Houck proposed openstack/ironic master: Utilize KernelCommmandLine.parse() instead of ironic.common.utils.parse_kernel_params  https://review.opendev.org/c/openstack/ironic/+/99230920:48
opendevreviewClif Houck proposed openstack/ironic master: Add a tool to parse kernel command lines  https://review.opendev.org/c/openstack/ironic/+/99264120:48
opendevreviewClif Houck proposed openstack/ironic master: Add a deprecation notice for disable_kernel_parameter_parsing  https://review.opendev.org/c/openstack/ironic/+/99575020:48
opendevreviewClif Houck proposed openstack/ironic master: Enable `oslo_versionedobjects.mypy` plugin for mypy  https://review.opendev.org/c/openstack/ironic/+/99902020:48
JayFdtantsur: TheJulia: So fun story: I have a need on a near-immediate timescale for portgroups in ironic standalone networking.20:48
JayFcc clif ^20:48
TheJuliaclif: I'm now super happy I didn't start into clif's patches yet20:49
clifoh boy20:49
clifI mean, the mypy thing is minor20:49
clifI just put it up front because one of the patches in that chain benefits from it20:49
clifim happy about the versionedobjects mypy plugin because I was afraid it was going to be a pita to get mypy working for that20:52
clifstephenfin: kudos to you for creating that plugin, thank you20:53
TheJuliacid: hey, https://review.opendev.org/c/openstack/ironic/+/995853 has me a bit worried since tests are not included in the commit. Could we get https://review.opendev.org/c/openstack/ironic/+/995856 folded into 995853 ? 20:59
TheJuliaeasy backport: https://review.opendev.org/c/openstack/ironic/+/998018?usp=search21:28
cidTheJulia, alright. I will quickly do that now.21:28
TheJulialast call on https://review.opendev.org/c/openstack/sushy/+/99746321:31
opendevreviewMerged openstack/ironic-specs master: Firmware endpoint hardware identity  https://review.opendev.org/c/openstack/ironic-specs/+/98580021:32
opendevreviewVerification of a change to openstack/ironic master failed: Remove disabling the scope enforcement  https://review.opendev.org/c/openstack/ironic/+/99668721:33
opendevreviewMerged openstack/ironic-specs master: Asynchronous sensor data collection  https://review.opendev.org/c/openstack/ironic-specs/+/97275421:35
opendevreviewJulia Kreger proposed openstack/ironic-python-agent-builder stable/2025.2: Fixed rescue ConditionPathExists flag  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/99902521:38
opendevreviewcid proposed openstack/ironic master: Add a container provider for console containers  https://review.opendev.org/c/openstack/ironic/+/99585321:38
opendevreviewcid proposed openstack/ironic master: devstack: Support the container console provider  https://review.opendev.org/c/openstack/ironic/+/99585421:38
opendevreviewcid proposed openstack/ironic master: Document the container console provider  https://review.opendev.org/c/openstack/ironic/+/99585521:38
opendevreviewcid proposed openstack/ironic master: A non-voting job for the container console provider  https://review.opendev.org/c/openstack/ironic/+/99893421:38
TheJuliacardoe: I believe you've wanted this: https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/99765921:41
TheJuliaGeneral reminder: ironic-week-prio open items can use some reviews21:59
opendevreviewMerged openstack/networking-baremetal master: doc: include info about l2vni_subport_anchor_network and cross refs  https://review.opendev.org/c/openstack/networking-baremetal/+/99853022:04
TheJuliarpittau: when your up tomorrow, if you have any ideas on https://review.opendev.org/c/openstack/ironic/+/998511 lmk. Thanks in advance!22:12
opendevreviewMerged openstack/ironic master: Limit LLDP TLV value length and interfaces  https://review.opendev.org/c/openstack/ironic/+/99045222:23
opendevreviewMerged openstack/sushy master: Add openstack-python3-next-jobs  https://review.opendev.org/c/openstack/sushy/+/99776922:33
opendevreviewMerged openstack/sushy master: Remove Python 3.10 support  https://review.opendev.org/c/openstack/sushy/+/99777022:33
opendevreviewMerged openstack/ironic-python-agent-builder master: Add minimum pyproject.toml  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/99863922:40
opendevreviewMerged openstack/ironic master: Enable `oslo_versionedobjects.mypy` plugin for mypy  https://review.opendev.org/c/openstack/ironic/+/99902023:26
opendevreviewMerged openstack/networking-generic-switch master: Disable neutron-ovn-agent  https://review.opendev.org/c/openstack/networking-generic-switch/+/99864423:29

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!