| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/1000921 | 00:35 |
|---|---|---|
| opendevreview | Doug Goldstein proposed openstack/ironic bugfix/37.0: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/1000922 | 00:35 |
| opendevreview | Doug Goldstein proposed openstack/ironic bugfix/38.0: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/1000923 | 00:36 |
| opendevreview | Rico Lin proposed openstack/ironic master: Pass explicit virtual media protocol to Sushy https://review.opendev.org/c/openstack/ironic/+/997464 | 08:24 |
| opendevreview | Rico Lin proposed openstack/ironic master: Allow image publisher aliases for hardware types https://review.opendev.org/c/openstack/ironic/+/997806 | 08:26 |
| opendevreview | Rico Lin proposed openstack/ironic master: Pass explicit virtual media protocol to Sushy https://review.opendev.org/c/openstack/ironic/+/997464 | 08:32 |
| TheJulia | cardoe: regarding ricolin's change, I guess I've planted an idea on there. TBH, not a fan of it, but I can kind of see what/why and where he is trying to go. | 13:37 |
| TheJulia | That would be https://review.opendev.org/c/openstack/ironic/+/997806 | 13:37 |
| opendevreview | Merged openstack/ironic stable/2026.1: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/1000921 | 13:52 |
| opendevreview | Merged openstack/ironic bugfix/38.0: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/1000923 | 13:52 |
| opendevreview | Merged openstack/ironic bugfix/37.0: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/1000922 | 13:52 |
| *** hroy_ is now known as hroy | 14:20 | |
| opendevreview | Verification of a change to openstack/python-ironicclient master failed: Add --disable-ramdisk to runbook set/unset https://review.opendev.org/c/openstack/python-ironicclient/+/1000113 | 15:09 |
| cardoe | TheJulia: yeah I agree with you | 15:41 |
| TheJulia | I can see merging it with that in mind though, something always better than nothing, I just don't like explicit config mapping for the by name attribute. Honestly, the invocation could include a task as an option characteristic and there could just be a map matching option on the driver in the model. Just not a fan of human mapped config | 16:03 |
| *** hroy_ is now known as hroy | 16:16 | |
| opendevreview | Merged openstack/ironic-python-agent master: Enforce checksum algorithm validation for Nvidia NIC firmware https://review.opendev.org/c/openstack/ironic-python-agent/+/1000018 | 16:25 |
| JayF | TheJulia: I'm shocked your review of that didn't come with a -1 or -2, honestly. I'm not going to point my brain at it enough today to do so myself, but that config var SCREAMS "I'm a decoder ring" | 16:30 |
| opendevreview | Merged openstack/ironic-python-agent master: Change md5_enabled default to False https://review.opendev.org/c/openstack/ironic-python-agent/+/1000019 | 16:32 |
| opendevreview | Merged openstack/ironic master: Allow disable_ramdisk to be updated via runbook PATCH https://review.opendev.org/c/openstack/ironic/+/1000059 | 16:33 |
| TheJulia | I guess I'm mentally more lienent given its coming from a "I have a downstream hardware type" place, so its sort of their own made problem | 16:42 |
| TheJulia | but yeah, I'm really not a fan of it either | 16:42 |
| opendevreview | Merged openstack/ironic master: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/999701 | 16:46 |
| opendevreview | Merged openstack/ironic master: Add request context user tracking to node history records https://review.opendev.org/c/openstack/ironic/+/993901 | 16:46 |
| JayF | TheJulia: if they have a downstream hardware type, they can invest the time to help make it actually-pluggable or hold a downstream patch | 16:52 |
| JayF | I don't think the middle route of a mid-rare baked idea getting merged is the best one here | 16:52 |
| TheJulia | fair, I'm working on another item right now, I might propose a clean path forward if I am not fighting claude much longer | 16:52 |
| opendevreview | cid proposed openstack/ironic stable/2026.1: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/1000984 | 16:53 |
| opendevreview | cid proposed openstack/ironic stable/2025.2: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/1000985 | 16:54 |
| opendevreview | cid proposed openstack/ironic stable/2025.1: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/1000986 | 16:54 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: Add opt-in MTU management for switch ports https://review.opendev.org/c/openstack/networking-generic-switch/+/984832 | 17:08 |
| opendevreview | Merged openstack/python-ironicclient master: Add --disable-ramdisk to runbook set/unset https://review.opendev.org/c/openstack/python-ironicclient/+/1000113 | 17:20 |
| cardoe | JayF: TheJulia: you two know that I've got downstream hardware types... I'm working at sanding down the rough edges as well for those interfaces. | 17:21 |
| cardoe | But I'm doing what JayF said... until I can land said feature / functionality then I'm holding a downstream patch. | 17:21 |
| JayF | maybe you and Rico can find the intersection of how to make that more properly pluggable? | 17:21 |
| TheJulia | I think in this case, there are a few solutions as well, but someone just needs to do it | 17:22 |
| TheJulia | next patch will be that from me | 17:22 |
| cardoe | Yeah I've reached out to him. We're unfortunately in the literal worst time delta. He starts his day 9pm my time and ends his day 5am my time. | 17:22 |
| cardoe | It's not just this. He's using OpenStack Helm and working on supporting the new console stuff... which is something we're doing as well. | 17:23 |
| cardoe | He's also working on OVN backend for Octavia for baremetal. And we are as well. | 17:24 |
| JayF | we have all the tools to work async, it just stinks that makes it hard to transfer all the initial context | 17:24 |
| TheJulia | cardoe: do you know if he going to the summit? | 17:25 |
| cardoe | I don't | 17:25 |
| TheJulia | okay | 17:25 |
| opendevreview | cid proposed openstack/ironic unmaintained/2024.1: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/1000989 | 17:27 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add NVMe-oF volume connector and target type support https://review.opendev.org/c/openstack/ironic/+/991700 | 17:34 |
| opendevreview | cid proposed openstack/ironic unmaintained/2023.1: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/1000990 | 17:36 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Let boot interfaces declare image publisher settings https://review.opendev.org/c/openstack/ironic/+/1000996 | 18:16 |
| TheJulia | cardoe: JayF ^^^ | 18:16 |
| TheJulia | JayF: replied to your comment, technically we are moving the default there because the same interface is used | 18:25 |
| cardoe | TheJulia: so my review would be to not use a dict but instead define a dataclass with the fields. | 18:32 |
| cardoe | But I'm not sure how much that complicates the fallback case. | 18:33 |
| cardoe | Likely not that much actually. | 18:34 |
| cardoe | TheJulia: can I rudely push over the top? | 18:42 |
| TheJulia | go ahead, but the class being loaded defines it is my point, the original behavior to hw type strict mapping was always wrong | 18:46 |
| cardoe | agreed | 18:47 |
| TheJulia | so, the class being loaded in the driver, i.e. the boot interface, is the information point | 18:47 |
| TheJulia | makes it nice and direct too | 18:47 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: Let boot interfaces declare image publisher settings https://review.opendev.org/c/openstack/ironic/+/1000996 | 18:54 |
| cardoe | Hopefully you don't dislike that. | 18:56 |
| TheJulia | seems okay to me | 18:57 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: ROCE Enabled BMaaS https://review.opendev.org/c/openstack/ironic-specs/+/990413 | 19:58 |
| cardoe | Do we want a spec for me adding an API and access method for the LogService of Redfish? | 20:11 |
| TheJulia | what problem are you attempting to solve with that? | 20:12 |
| TheJulia | I guess I could see accessing logs handy if you've got a vendor on the phone on some level | 20:15 |
| cardoe | We don't un-enroll our nodes and all that jazz. We use Ironic power checker to ensure the boxes are powered on. | 20:15 |
| TheJulia | because the alternative is.. page level 3/4 to do that | 20:15 |
| cardoe | So I've got folks wanting iDRAC access still to these Dell boxes. | 20:15 |
| cardoe | So I'm being evil and saying "no" and making them express what they need want. | 20:16 |
| * TheJulia gets out the "evil" shirt to better understand the ask | 20:16 | |
| cardoe | One of the items they brought up are the hardware logs. It turns out that Redfish standardizes the hardware logs. IPMI did as well and they were called SEL but they're a subset of the actual log events. | 20:16 |
| TheJulia | yeah | 20:17 |
| TheJulia | true | 20:17 |
| cardoe | Yes. Today is "evil" shirt day for sure. | 20:17 |
| TheJulia | I could see "retreive hardware logs' being a reasonable thing and is functionally generic | 20:17 |
| TheJulia | because of LogService or ipmitool sel | 20:17 |
| cardoe | A little birdy in Austin tells me the Redfish LogService has got it all | 20:17 |
| TheJulia | oh, I'm sure it does | 20:18 |
| TheJulia | so question is, do we store the log or do we just go ask the bmc each time? Or do we cache it for a short time ? I guess the other question is "how much" | 20:19 |
| cardoe | https://github.com/dell/iDRAC-Redfish-Scripting/blob/master/Redfish%20Python/SupportAssistCollectionNetworkShareREDFISH.py | 20:19 |
| cardoe | That's a them specific feature that's required for certain RMA and such operations. | 20:19 |
| TheJulia | ugh | 20:19 |
| cardoe | So I'm gonna wire that up to a vendor passthru method for the idrac-redfish if that makes sense. | 20:20 |
| cardoe | So let's table that item. | 20:21 |
| cardoe | But the LogService interface isn't horrible. | 20:21 |
| TheJulia | I mean, it does sort of make sense, but at RMA I could see paging an "adminy" individual | 20:21 |
| TheJulia | but for general logservie, yeah | 20:21 |
| cardoe | But it's ambiguous because it exists on the Chassis and ComputerSystem object. We have a link in sushy but no implementation | 20:22 |
| cardoe | Each entry has an ID and you fetch that specific entry | 20:22 |
| cardoe | There's a fetch depth syntax that we've done in other places but alas it's "vendor defined" how to handle a large amount. | 20:23 |
| TheJulia | I haven't looked deeply at logservice before, but I know chassis also tends to be more bmc orientation, where computersystem is the instance which boots the host and they've moved some stuff around as time has moved on | 20:23 |
| TheJulia | yeah, afaik not universally supported though | 20:23 |
| cardoe | So I fear some vendors will paginate... some will truncate... some will truncate returning the first X items from the beginning of time and some will truncate returning the X most recent items. | 20:23 |
| cardoe | So I feel like the safest interface for us is to fetch a list of IDs on one endpoint and the other endpoint just look up individual IDs. | 20:24 |
| TheJulia | hmmm | 20:24 |
| TheJulia | quite possibly | 20:24 |
| TheJulia | seems... not great | 20:25 |
| TheJulia | but... its designed that way for some reason | 20:25 |
| cardoe | You know that we'll sit here and come up with all possible ways that a vendor will interpret "vendor defined" and then we'll go to actual hardware and discover yet another. | 20:25 |
| TheJulia | yeah | 20:25 |
| cardoe | We can certainly make our fetch accept a list of IDs if we want to optimize? | 20:26 |
| cardoe | Cause that data would have to traverse the RPC pathway from API node to conductor and back | 20:26 |
| TheJulia | so, looking at the model | 20:27 |
| TheJulia | I wonder if we can get to batch and retrieve | 20:27 |
| cardoe | So to give you an idea they'll see like a CPU fault or something like that and then know they need to swap the CPU out. | 20:29 |
| cardoe | or PSU fault and swap that out. | 20:29 |
| TheJulia | oh yeah | 20:29 |
| TheJulia | I grok that | 20:29 |
| TheJulia | I'm more trying to mentally model how we don't create a slog for 30 characters with like 4kb of network traffic to get that 30 characters | 20:29 |
| cardoe | Today our folks are slinging racadm commands against that gear. So I'm just thinking how I can make their interface be entirely the "baremetal" command. | 20:30 |
| TheJulia | ++ | 20:30 |
| cardoe | No you're absolutely right. I was looking at how to batch it as well. | 20:30 |
| cardoe | The vendor defined made me nervous. | 20:30 |
| TheJulia | there does seem to be some level of consistency | 20:31 |
| TheJulia | but.. yeah | 20:31 |
| TheJulia | I'd ask the conductor to get the "last 10" by default and return it over rpc | 20:31 |
| TheJulia | and make it user requestable just how much with an understanding or an upper bound limit in the config of 100-150 entries | 20:32 |
| cardoe | yeah we can do that. I'll whip up a spec. | 20:32 |
| TheJulia | most of the time I've had to look at hardware logs, it hasn't been *THAT* verbose | 20:32 |
| cardoe | Should we make it possible to request a specific one? | 20:32 |
| TheJulia | only when I have a bad memory stick | 20:32 |
| TheJulia | that feels more like modeling redfish into our api | 20:32 |
| cardoe | and then should we make it possible to look at Chassis and System or should we just merge the two together... The redfish example data actually has the endpoints returning different things but then for 1 item returning the same ID from both endpoints. And they do mention that it means its the same message logged to both. | 20:33 |
| TheJulia | ugh, yeah | 20:33 |
| TheJulia | so... maybe BMC log and system as separate logs | 20:34 |
| TheJulia | because, you can have one BMC with subsystems | 20:34 |
| cardoe | I've literally not even thrown a curl against the redfish endpoint on the gear I've got yet. | 20:34 |
| cardoe | But if its possible then someone will do it is how I feel. | 20:34 |
| cardoe | I'll take our convo and craft an initial spec and let folks provide some feedback. | 20:35 |
| cardoe | The other item that came up is that Ironic's node health is quite lacking. | 20:35 |
| cardoe | And I unfortunately had to agree. | 20:36 |
| cardoe | So we have get_node_health() on the management interface which in the redfish side calls the overall system rollup health status and returns that | 20:37 |
| cardoe | But there's technically subsystem / component level as well | 20:37 |
| cardoe | And once again there's chassis vs system | 20:37 |
| cardoe | You can have a bad memory card for the BMC for example and get_node_health() rolls up healthy | 20:39 |
| cardoe | You can have a component with a warning and get_node_health() rolls up healthy | 20:39 |
| cardoe | Claude's hot take of the Redfish spec is that the system health rollup is only triggered by errors on the child components not by warnings. | 20:40 |
| cardoe | It's a slippery slope of hardware monitoring though. Cause I can see that being expensive from a scraping of APIs if people are polling it. | 20:41 |
| cardoe | But as a one off tell me about this box... that's certainly got value | 20:42 |
| cardoe | the iDRAC has the overall and then component status on the front page when you login. | 20:42 |
| cardoe | https://github.com/dell/iDRAC-Redfish-Scripting/blob/master/Redfish%20Python/GetIdracLcLogsREDFISH.py that's Dell's script. 14G and newer hardware conforms to the Redfish spec. | 20:51 |
| cardoe | That's the LogService stuff. They call it the Lifecycle Controller Logs | 20:51 |
| TheJulia | EBRAINOVERFLOW :) | 21:03 |
| TheJulia | yeah, rollup is only errors which are faults | 21:04 |
| opendevreview | Merged openstack/ironic master: dev/test: Enable graphical console in localdev config https://review.opendev.org/c/openstack/ironic/+/1000566 | 21:06 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!