| *** Khaled is now known as KhaledAtOVH | 08:56 | |
| kblah | Hi everyone, I'm new here so I'm not sure if I've come to the right place. But I believe I did. I'm working on a task to enable the ironic introspection to be run in a more automated fashion, i.e. not only during enrollment or manually but (maybe) as a step in every cleaning of a node. The ultimate goal is to run introspection more often without having to use human | 10:02 |
|---|---|---|
| kblah | intervention. I've identified a few roadblocks in the ironic code base but I believe these can be overcome. But I'm not sure if my approach is architecturally sound and I'm curious what this particular group of experienced ironic users and devs thinks... | 10:02 |
| kblah | I'm thinking about introducing a new "clean step" which runs introspection as part of a regular (i.e. automated) cleaning of a node. | 10:03 |
| kblah | This approach has the benefit of not requiring many changes in the ironic code base an re-using the existing code for the introspection. However, I'm worried that this might affect security and open up a new path for an attacker. | 10:04 |
| kblah | To me, it also looks like this group has already thought about such a use case. See https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/inspect_utils.py#L371 which states: | 10:06 |
| kblah | # TODO(dtantsur): support active state inspection | 10:06 |
| kblah | I'm very much interested in your thoughts and ideas regarding such a feature for ironic. | 10:07 |
| *** mdfr2 is now known as mdfr | 10:25 | |
| cardoe | I actually wrote a spec for doing this that’s been approved. I just haven’t had anyone free to task with doing it on my team. https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/inspection-steps.html | 13:03 |
| cardoe | kblah: thanks for joining us and good to see someone else interested in this | 13:03 |
| TheJulia | Good morning! | 13:04 |
| TheJulia | (FWIW, today is a day off for all RH folk, myself included, but just felt like saying good morning!) | 13:04 |
| cardoe | I would start with the spec and feel free to provide any feedback to your thoughts. | 13:04 |
| cardoe | TheJulia: jealous. All our recharge days have landed on days I already had scheduled PTO | 13:05 |
| TheJulia | I don't even try to align stuff like that | 13:06 |
| TheJulia | And it doesn't help me that I'll be in the air on the way to the summit on the next holdiay as well | 13:06 |
| cardoe | I don’t either cause they announce them only like 2 weeks in advance and I’ve got my PTO set for like 6 months out. | 13:06 |
| TheJulia | wheeeeeee | 13:06 |
| TheJulia | kblah: Greetings! Sounds interesting. A step is the right model, really. I'm the person who created active state introspection in the past with the prior inspector, and active state is more "allowing ironic to have introspection data updates be received and processed". Its not the best solution because that was in-band, operator runs a command on the host and data gets posted to ironic. A very large partical accelerator basically | 13:14 |
| TheJulia | asked for it and I was happy to make it happen. I'm just not sure there is value to do it again or continue to do it when the machine is in an operating state. A few reasons why: 1) In-band could lead to security issues now that we're tightly coupled with ironic for inspection. So if supported, the defenses would need t be strong. 2) If only out of band inspection is done, it can also negatively impact performacne. Depending on the | 13:14 |
| TheJulia | vendor, your BMC queries turn into PLDM data points being walked over the i2c bus between the BMC and various controllers. That can do fun things like cause IO to disks to be paused when the BMC starts asking for details from the storage controllers. That *should* be getting better, but its still a risk. 3) Then there is a question of data validity. I guess this is the more subjective aspect but when the need was originally raised | 13:14 |
| TheJulia | to us, the idea was to collect updated disk serial numbers on long-lived hosts to keep data-center operations from having to do any sort of manual data entry if a drive failed in a raid set. Could still be reasonable I guess, but discussion and understanding of the case around active state data collection going to be key. | 13:14 |
| TheJulia | Sorry for the wall of text | 13:15 |
| JayF | Only writing allowed on day off, not editing lol | 13:23 |
| TheJulia | lol | 13:25 |
| TheJulia | What about "travel days?" | 13:25 |
| opendevreview | Ivan Anfimov proposed openstack/networking-generic-switch master: Replaced usage outdate egrep to grep https://review.opendev.org/c/openstack/networking-generic-switch/+/1002753 | 13:25 |
| TheJulia | ... I could load up scrivner on my tablet and and and... write the big next thing! | 13:27 |
| opendevreview | Ivan Anfimov proposed openstack/networking-generic-switch master: Replaced usage outdate egrep to grep https://review.opendev.org/c/openstack/networking-generic-switch/+/1002753 | 13:27 |
| TheJulia | </joking> | 13:27 |
| TheJulia | Ugh, I need to get down the desert floor today. WHEEEEEeeeeee | 13:28 |
| cardoe | TheJulia: you ever see the movie Old School with Will Ferrell? | 13:37 |
| cardoe | He's dumb as can be and there's a scene where he debates James Carville and it's about the only intelligent thing he says during the whole movie. But when he's done speaking it's like he blacked out. | 13:38 |
| cardoe | That wall of text made me think of that. | 13:38 |
| cardoe | We're all joking around and then suddenly you have a giant technically insightful response, apologize and then back to joking. | 13:39 |
| TheJulia | I have not, I'm not really a fan of Will Ferrell | 13:41 |
| cardoe | You and my wife. | 13:41 |
| kblah | cardoe: I've not seen this spec before and will read it asap! From a first glance, it looks like it might fit our requirements! | 13:42 |
| cardoe | When I was ~20 in college he was funny | 13:42 |
| TheJulia | Joking++ | 13:42 |
| TheJulia | (its how you build the bonds of trust and collaboration!) | 13:42 |
| kblah | TheJulia: the use case you described sounds rather complex, i.e. I'm not sure we're actually looking for introspection on nodes in state active but rather want to increase the cadence of introspections so that they're not only done once during enrollment. But what you describe regarding discs replaced in RAID sets is exactly what we thought of as well. Particularly if you | 13:45 |
| kblah | add SED to the mix, this might make things like replacing discs a lot easier (if not fully automated). If it could be done while the node is active, even better! | 13:45 |
| TheJulia | The challenge is more so how often you move the machine through the state machine. That work was before we had "servicing" as well, the current state model would likely be "servicing" + "go execute an inspect step", but milage may vary. The big thing with that particle accelerator is they have a huge data farm of ceph nodes and they wanted to track data about each disk as well, so those nodes and the "large data set nodes" were | 13:48 |
| TheJulia | known and managed by ironic, but had special rules around their engagement.. since they would be alive for a very long time compared to a machine which might be a hypervisor... maybe 9 months out of the year, or a machine for a specific 2 week workload | 13:48 |
| cardoe | So one fun one I'm noodling about here is nodes in "available" state. Maybe we get a new firmware that we want to baseline all the boxes to... say the iDRAC. So now I've got a runbook created which can do that. But I cannot use clean or service for nodes in available. I need to first move the node to manageable and then clean it with the runbook and then provide it back to available which re-runs cleaning. Not horrible but | 13:50 |
| cardoe | not time efficient. | 13:50 |
| cardoe | But maybe a BIOS update I would want to re-clean the box. | 13:51 |
| cardoe | I dunno. | 13:51 |
| JayF | Yeah, that's the reason we force that | 13:51 |
| JayF | we want to be 100% sure cleaning works on $curNodeState before putting data on it | 13:51 |
| JayF | and that also enforces a consistent platform you are deploying onto | 13:51 |
| cardoe | Well currently we update iDRAC versions out of band. I'm trying to bring this into Ironic. | 13:52 |
| JayF | cardoe: automated cleaning via runbook: managable -> configure per-node cleaning runbook as CLEAN_AFTER_BIOS_UPDATE_1_45 -> AVAILABLE with clean+update all in one | 13:52 |
| JayF | not what it was designed for, but it should work, you'll just have to opt in to the lesser-secure per node cleaning runbook override | 13:53 |
| cardoe | yeah I don't like the lesser-secure words. :-D | 13:53 |
| cardoe | Like I said... still noodling over the best way. | 13:53 |
| JayF | cardoe: or an IB HWM that checks for node['driver_info']['upgrade_firmware_bios_1_45'] before running the updates | 13:53 |
| JayF | there are lots of ways to sneaky-downstream modify cleaning :D | 13:54 |
| cardoe | Well I'd like to ultimately start writing some blog posts on the ironic site about usage patterns | 13:54 |
| cardoe | So just trying to come up with generic ways for folks. | 13:54 |
| JayF | I wonder if we shuld filter ironicbaremetal posts for ... good/supported ideas lol | 13:54 |
| JayF | there are lots of things possible in Ironic that I'd tell people not to do (but maybe elbow and wink afterwards, if you get the drift) | 13:55 |
| JayF | eh, if we have fake-nova-adoption documented, even vaguely, that's basically the worst of those cases lol | 13:55 |
| cardoe | So another thing I'm noodling... we've got the bios interface for setting bios settings... but there's also BMC settings via /Managers/<system-id>/Attributes which take the same shape as /Systems/<system-id>/BIOS/Attributes | 13:55 |
| cardoe | I'm wondering if we should expose setting the BMC settings. | 13:56 |
| cardoe | We've got some settings that we ensure via again out of Ironic process that I'd like to have a runbook for. | 13:56 |
| cardoe | Mentally I'm visualizing TheJulia's web UI as something a tech can run on a tablet for example and let's say they had to service that box, they can click on some runbooks which would just ensure that the box is good before walking away. | 13:58 |
| kblah | I'm new to Openstack and I hope this is not a stupid question: but what is the servicing state? Is it a state that was recently introduced because I've not yet seen it | 14:11 |
| JayF | kblah: https://docs.openstack.org/ironic/latest/admin/servicing.html | 14:11 |
| kblah | Thanks, JayF! | 14:14 |
| kblah | seems the first version that introduced servicing was 2023.2 so I just did not yet see it in action myself... | 14:17 |
| TheJulia | I believe that is correct version wise | 14:30 |
| TheJulia | cardoe: because we have to move out of scheduling availability, we need to change the state, but available -> service -> available seems... reasonable. | 14:31 |
| JayF | That's something that we should discuss some among cores. I think that leads to snowflakes in the AVAILABLE pool | 14:32 |
| JayF | I really like that you can't get to AVAILABLE without immediately passing a cleaning beforehand (if enabled). | 14:32 |
| cardoe | yeah like I said... I don't really like that escape hatch for a lot of applications. | 14:33 |
| TheJulia | cardoe: yeah, that was sort of what I was thinking and was the inspriation, because when I worked in a DC, one was just like 80 feet from my office door. The other was like 2800 feet. | 14:33 |
| TheJulia | and it was entirely hit or miss if the crash cart was even available in that DC, and the number of times I had to walk back to my office was a giant time sink | 14:34 |
| TheJulia | JayF: oh yeah, definitely | 14:34 |
| TheJulia | The alternative is to kick back to manageable, then clean, then re-provide. A little cumbersome to do on a fleet in an ad-hoc fashion. | 14:35 |
| JayF | we've got a handful of ways to shortcut that: disable auto cleanining on the provide, use auto-clean-via-runbook to batch your service/cleaning together (this especially makes sense for the fleet use case) | 14:37 |
| TheJulia | that disable auto-cleaning disables auto-cleaning on teardown | 14:38 |
| TheJulia | cardoe: anyway, add something to the ptg etherpad if you feel the case makes sense. | 14:38 |
| JayF | Call Ironic API: set driver_info[cleaning]=nuh-uh; call Ironic API: set provision_state provide; call Ironic API: unset driver_info[cleaning] | 14:38 |
| cardoe | yeah could do something like that. | 14:39 |
| TheJulia | oh, fair, yeah, you could set the node level automated_clean flag | 14:39 |
| cardoe | The other one is the BMC settings. | 14:39 |
| TheJulia | I guess there is a case to be made for enabling the operator to ensure there is eventual operational consistency across the nodes they are managing so a runbook is sort of ideal if you can match non-updated nodes | 14:40 |
| TheJulia | (we likely need a step which sets a flag to be a semaphore style indicator) | 14:40 |
| TheJulia | (that can then be viewed by the operator....) | 14:40 |
| TheJulia | ((even, if its in extra....)) | 14:40 |
| JayF | PTG session in 2029: "Oh no, we made steps turning complete" | 14:40 |
| JayF | s/turning/turing/ | 14:40 |
| TheJulia | JayF: your not wrong | 14:41 |
| TheJulia | Anyway, I need to get my day off going to retrieve the wife's car. | 14:41 |
| JayF | at some point just do the thing that software from wow to nginx have been doing for decades | 14:41 |
| JayF | "screw it, here's a lua interface, I give up" | 14:41 |
| * TheJulia twitches | 14:41 | |
| JayF | :) | 14:41 |
| * JayF -> AFK | 14:42 | |
| kblah | So I'm leaving for the week-end. Have a great week-end everyone and thanks for all of your hints and ideas so far! I look forward to contributing something useful here! | 15:29 |
| cardoe | Did I do something wrong with the api for the runbooks? https://docs.openstack.org/api-ref/baremetal/ I notice that the latest runbook API bits aren't there. | 18:19 |
| JayF | did you update api-ref/ in the Ironic repo? | 19:03 |
| JayF | we are supposed to enforce that for API changes, but it's often missed | 19:03 |
| JayF | https://docs.openstack.org/api-ref/baremetal/#runbooks-runbooks is saying "1.92 Runbook API was introduced" | 19:03 |
| JayF | so I think you found a gap | 19:03 |
| JayF | Please make sure to get those posted, or at an absolute minimum we should document the miss in lp so *someone* can | 19:03 |
| cardoe | well now I wanna see if Claude can right some check for me. | 20:56 |
| cardoe | I dunno why I didn't think to look in the api-ref directory. Thanks JayF. | 20:56 |
| opendevreview | Merged openstack/bifrost master: tox: Drop redundant injection of VIRTUAL_ENV variable https://review.opendev.org/c/openstack/bifrost/+/996872 | 21:05 |
| TheJulia | cardoe: w/r/t admin web ui/mobile app stuff. I’m hearing “wider view for a tablet”, and maybe “select and trigger run book”? Was there anything else standing out? | 21:41 |
| * TheJulia is stuck waiting for a guide car to come back to let us escape the flagged area in a long construction area | 21:42 | |
| cardoe | So I yolo'd something while having multiple convos with our folks. | 21:42 |
| cardoe | To be clear... yours works... this is just HTML and JavaScript and no real APIs. | 21:42 |
| cardoe | But I can show ya next week? | 21:43 |
| TheJulia | Sure! | 21:45 |
| opendevreview | Merged openstack/ironic master: Adopt port physical network for empty portgroups https://review.opendev.org/c/openstack/ironic/+/1002353 | 22:10 |
| opendevreview | Doug Goldstein proposed openstack/ironic bugfix/38.0: Adopt port physical network for empty portgroups https://review.opendev.org/c/openstack/ironic/+/1002835 | 22:14 |
| opendevreview | Doug Goldstein proposed openstack/ironic bugfix/37.0: Adopt port physical network for empty portgroups https://review.opendev.org/c/openstack/ironic/+/1002836 | 22:15 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: Adopt port physical network for empty portgroups https://review.opendev.org/c/openstack/ironic/+/1002837 | 22:16 |
| opendevreview | Merged openstack/ironic master: Make AutoDetectDeploy act more consistently https://review.opendev.org/c/openstack/ironic/+/1000074 | 22:31 |
| opendevreview | Doug Goldstein proposed openstack/ironic bugfix/38.0: Make AutoDetectDeploy act more consistently https://review.opendev.org/c/openstack/ironic/+/1002838 | 22:34 |
| opendevreview | Merged openstack/ironic master: Clean up deprecated send_sensor_data options https://review.opendev.org/c/openstack/ironic/+/1002012 | 22:40 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!