| stevebaker[m] | JayF: Would it help if we started publishing ironic-python-agent-base images[1] and signalling that this would be considered a default minimal image? | 00:04 |
|---|---|---|
| stevebaker[m] | [1] https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/997659 | 00:04 |
| TheJulia | The alternative then becomes... 3 images, which not much (if anything?) different between two? | 00:05 |
| stevebaker[m] | what would the 3rd image be? | 00:05 |
| TheJulia | apparently one with podman which can do bootc? but then I wonder what the first two's difference would be | 00:06 |
| TheJulia | I think this is largely perception overlapping with risk management | 00:06 |
| TheJulia | consensus being the goal | 00:06 |
| stevebaker[m] | I'm proposing we publish a ironic-python-agent-base image which doesn't have dhcp-all-interfaces or anything else we add to the ironic-python-agent image (including podman). Then we give some guidance in the docs for when to use the minimal ironic-python-agent-base or the full-fat ironic-python-agent | 00:10 |
| stevebaker[m] | Also is metal3 still working on publishing their own upstream ipa image? dtantsur | 00:15 |
| TheJulia | That is what I understand as well | 00:50 |
| opendevreview | Nuutti Hakala proposed openstack/ironic master: Add more JSON schema validation to API https://review.opendev.org/c/openstack/ironic/+/990440 | 06:12 |
| rpittau | TheJulia: the problem is that the mechanism to apply the upper constraints varies based on the job, changes to apply local or remote UCs are not in a single place | 07:16 |
| rpittau | TheJulia: I did some tests, I'm getting close to a solution | 07:46 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/37.0: Pin upper constraints locally for bugfix branch https://review.opendev.org/c/openstack/ironic/+/998511 | 07:57 |
| opendevreview | Jacob Anders proposed openstack/bifrost master: Add uninstall playbook to reverse bifrost install https://review.opendev.org/c/openstack/bifrost/+/999041 | 07:58 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add allow_grouping_reboots argument to firmware.update step https://review.opendev.org/c/openstack/ironic/+/998417 | 07:59 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Refactor firmware update SimpleUpdate submission and error dispatch https://review.opendev.org/c/openstack/ironic/+/999230 | 07:59 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Implement batched non-BMC firmware update orchestration https://review.opendev.org/c/openstack/ironic/+/999231 | 07:59 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Document batched firmware updates and add release note https://review.opendev.org/c/openstack/ironic/+/999232 | 07:59 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add allow_grouping_reboots argument to firmware.update step https://review.opendev.org/c/openstack/ironic/+/998417 | 08:16 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Implement batched non-BMC firmware update orchestration https://review.opendev.org/c/openstack/ironic/+/999231 | 08:16 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Document batched firmware updates and add release note https://review.opendev.org/c/openstack/ironic/+/999232 | 08:16 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/37.0: Pin upper constraints locally for bugfix branch https://review.opendev.org/c/openstack/ironic/+/998511 | 09:04 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic bugfix/38.0: Update .gitreview for bugfix/38.0 https://review.opendev.org/c/openstack/ironic/+/999246 | 09:18 |
| opendevreview | Merged openstack/ironic-python-agent master: Move container execution into a single method https://review.opendev.org/c/openstack/ironic-python-agent/+/998374 | 09:25 |
| dtantsur | stevebaker[m]: please check with Adam. We've wanted Metal3 IPA images for a long time, but it has slipped through the cracks for now. | 09:51 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic stable/2026.1: Another attempt to fix fast-track after inspection https://review.opendev.org/c/openstack/ironic/+/999255 | 10:17 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/37.0: Another attempt to fix fast-track after inspection https://review.opendev.org/c/openstack/ironic/+/999258 | 10:18 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/37.0: Power off before ejecting redfish virtual media on ramdisk cleanup https://review.opendev.org/c/openstack/ironic/+/999259 | 10:20 |
| dtantsur | Mahnoor: I assume we don't need to backport ^^ further? Or do we? Gerrit reports merge conflicts, and I haven't investigated further. | 10:22 |
| rpittau | TheJulia: this works https://review.opendev.org/c/openstack/ironic/+/998511 | 12:19 |
| dtantsur | le sigh | 13:14 |
| TheJulia | good morning | 13:24 |
| * TheJulia blinks | 13:26 | |
| TheJulia | I most definitely did not sleep well last night | 13:39 |
| iurygregory | sorry to hear that TheJulia =( | 13:44 |
| TheJulia | it is what it is | 13:51 |
| cardoe | JayF: I'm with ya on the podman in the image. | 15:12 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: Advance connectivity/driver modeling in NGS https://review.opendev.org/c/openstack/ironic-specs/+/997653 | 15:13 |
| cardoe | TheJulia: so is the ask for us to build images with that in it so that Metal3 can just consume upstream image? | 15:13 |
| TheJulia | Metal3 and other users fo the base image type, apparently | 15:13 |
| dtantsur | By the way, isn't it a bit odd that we wrote our own client for OCI images but then still rely on podman for bootc? | 15:16 |
| cardoe | also... | 15:16 |
| * cardoe table flips. | 15:16 | |
| TheJulia | no, because we have to download the artifact and we can stream that directly, podman is needed to execute the bootc command | 15:17 |
| TheJulia | so entirely disjointed cases | 15:17 |
| cardoe | Someone wants to know if the vnc console stuff can give read-only to the BMC... not just the console. | 15:17 |
| TheJulia | that we quickly conflate because podman command | 15:17 |
| dtantsur | ah, yeah, so not the format but rather bootc people don't know how to distribute software | 15:18 |
| TheJulia | oh, to them, everything is a container and why do anything different | 15:18 |
| dtantsur | TheJulia: silly question probably, but: bootc is just a Rust project, cannot we build it ourselves? | 15:19 |
| TheJulia | not really,because it requires the container itself | 15:19 |
| dtantsur | another aside, do we know if container inside container even works any well? | 15:19 |
| TheJulia | no idea | 15:20 |
| TheJulia | i think its supposed to, but that is outside of my realm/area | 15:20 |
| dtantsur | asking in case you ever need this to work in OpenShift | 15:20 |
| TheJulia | I feel like that is a cross the bridge when we get to it moment if it doesn't work. Our demand for it has died down in the short term, I think steve is just trying to get things he feel were unfinished/blockers to non-ocp use cases | 15:27 |
| dtantsur | Cool cool. We may even give it a try downstream as part of our general OCI image productization (which suddenly got more priority). | 15:27 |
| dtantsur | In fact, maybe bootc now is a bigger priority for us than for you :) | 15:28 |
| TheJulia | cardoe: .... Read only to the bmc?! like graphically? Do they not understand read-only ? | 15:28 |
| cardoe | yeah I think iDRAC has that mode. I dunno I gotta look. | 15:30 |
| TheJulia | it does, depending on version, but you have no mouse control either so.... | 15:43 |
| TheJulia | your just a viewer | 15:43 |
| cardoe | Well they wanted to see like the BIOS settings and the RAID settings... that stuff. | 15:58 |
| TheJulia | they can introspect bios setting data through the smbios data aiui, raid settings... that is oem tooling I guess | 16:08 |
| cardoe | They're mad I've taken away their ability to login to iDRAC | 16:18 |
| opendevreview | Esther Domfeh proposed openstack/ironic master: Add request context user tracking to node history records https://review.opendev.org/c/openstack/ironic/+/993901 | 16:35 |
| opendevreview | Esther Domfeh proposed openstack/ironic master: Add power tracking telemetry in node history https://review.opendev.org/c/openstack/ironic/+/996347 | 16:35 |
| JayF | cid: https://review.opendev.org/c/openstack/ironic-python-agent/+/998376 this is a really good catch. Probably advisory-worthy if it wasn't already inside a broken class. | 16:39 |
| TheJulia | cardoe: ironic also explicitly caches bios settings.... | 16:45 |
| TheJulia | granted, they will need their dell translation book for thenm | 16:45 |
| cid | JayF. Good find indeed. But I don't know about advisory. | 16:47 |
| cid | Maybe add it as a comment too. | 16:47 |
| JayF | I mean, we'll write up an OSSN about all this once they merge. | 16:47 |
| cid | Thanks for the reviews btw | 16:47 |
| TheJulia | "already broken" | 16:47 |
| JayF | I think there's value in noting about that brokenness, too | 16:47 |
| TheJulia | err, "wasn't already in a (already) broken class" | 16:47 |
| TheJulia | ++ | 16:47 |
| JayF | cid: a few -1 on that container hwm patch set, none of it major | 17:12 |
| JayF | cid: nice work \o/ | 17:12 |
| cid | oh thanks. I will loke at that in a few | 17:12 |
| TheJulia | super simple patch going back to november that I'm sure folks will appreciate: https://review.opendev.org/c/openstack/ironic/+/968361 | 18:57 |
| opendevreview | Merged openstack/ironic bugfix/37.0: Pin upper constraints locally for bugfix branch https://review.opendev.org/c/openstack/ironic/+/998511 | 18:57 |
| *** jroll04 is now known as jroll0 | 19:05 | |
| TheJulia | a review on https://review.opendev.org/c/openstack/ironic/+/992306 would be awesome | 19:11 |
| JayF | that's on my list, I'll bump it up | 19:14 |
| JayF | Just trying to get things reviewed or do you have a particular reason to want that in? | 19:14 |
| TheJulia | Question: do we want to get sushy released or some of the more outstanding items landed first | 19:19 |
| TheJulia | JayF: just trying to get the ironic-week-prio list down to sane | 19:19 |
| TheJulia | since we have so much just waiting | 19:19 |
| TheJulia | heh, https://review.opendev.org/c/openstack/sushy/+/989988 is big | 19:21 |
| JayF | oh no. seeing stacks of CI failures for centos checksum mismatches | 19:30 |
| JayF | they are from 7/28 so rechecking but :( | 19:30 |
| opendevreview | Merged openstack/sushy master: Allow explicit virtual media transfer protocols https://review.opendev.org/c/openstack/sushy/+/997463 | 19:33 |
| opendevreview | Merged openstack/ironic master: fix: TBN could not handle evaluations of items that were collections https://review.opendev.org/c/openstack/ironic/+/999138 | 19:37 |
| opendevreview | Merged openstack/ironic bugfix/38.0: Update .gitreview for bugfix/38.0 https://review.opendev.org/c/openstack/ironic/+/999246 | 19:37 |
| TheJulia | UGH | 19:41 |
| opendevreview | Merged openstack/ironic-python-agent master: Security: Make mDNS discovery explicitly opt-in https://review.opendev.org/c/openstack/ironic-python-agent/+/997637 | 19:49 |
| JayF | stevebaker[m]: Let's revisit the conversation about podman-in-image after the ContainerHardwareManager security fixes land. | 19:53 |
| opendevreview | Merged openstack/ironic-python-agent master: Allow limiting what config can be set in mDNS https://review.opendev.org/c/openstack/ironic-python-agent/+/997638 | 19:53 |
| JayF | TheJulia: dtantsur: ^ is someone going to write the third patch to populate that default config value? | 19:54 |
| TheJulia | you mean to disable it moving forward? I've been entirely overloaded, so it wasn't on my immediate radar | 20:07 |
| JayF | So I said this in the coresec sync: I wrote the framework to limit the things that we can set over mDNS | 20:07 |
| JayF | I did not populate the allowlist because I have zero context on that use case | 20:08 |
| TheJulia | OH | 20:08 |
| JayF | literally the last step to close the mDNS stuff is a two line patch to add a default value to the config I wrote | 20:08 |
| TheJulia | so, its literally None now, I was thinking of just moving to deprecation | 20:08 |
| TheJulia | honestly | 20:08 |
| TheJulia | since if someone wants to use it, go define it | 20:08 |
| JayF | None means "don't block anything" not "block everything" | 20:08 |
| TheJulia | oh, right, its continue in the loop | 20:09 |
| opendevreview | Merged openstack/ironic master: Add an option for OCI-specific TLS settings. https://review.opendev.org/c/openstack/ironic/+/968361 | 20:13 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: fix: TBN could not handle evaluations of items that were collections https://review.opendev.org/c/openstack/ironic/+/999333 | 20:18 |
| JayF | cid: is https://review.opendev.org/c/openstack/ironic/+/995856 still valid? | 20:23 |
| cid | Not at all. That's merged with another change now. | 20:23 |
| cid | Abandoning... | 20:24 |
| JayF | ack; ty | 20:24 |
| JayF | that's what I suspected | 20:24 |
| cardoe | thanks for the quick ack JayF | 20:29 |
| cardoe | I'm gonna try and put a for realz TBN workflow into my dev environment tomorrow with 2026.1 | 20:30 |
| JayF | I was talking to Clif about that today :) exciting to see it getting into the real world! | 20:30 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: Advance connectivity/driver modeling in NGS https://review.opendev.org/c/openstack/ironic-specs/+/997653 | 20:31 |
| TheJulia | cardoe: just don't forget to add z to every word :) | 20:32 |
| cardoe | Even more full circle.... the current OnMetal successor that's used internally would actually get replaced with our 2026.1 with TBN once we finish up the testing. | 20:32 |
| TheJulia | dtantsur: JayF: the spec w/r/t ngs should have everything addressed at this point, but if you don't feel like it has been please let me know | 20:33 |
| * TheJulia switches gears over to i-p-a | 20:33 | |
| JayF | cardoe: It's so generous of you to offer free bare metal servers to run our devstacks in as a thank you for me and clif still writing code for OnMetal | 20:35 |
| JayF | cardoe: I'll take one of those 512GB ram servers, please /s | 20:35 |
| cardoe | Just 1? Why not two? | 20:36 |
| JayF | one for me, one for clif | 20:36 |
| JayF | I'm not that greedy | 20:36 |
| JayF | unless you're physically giving me the server, in which case, I'll take as many chassis-full-of-silicon-gold as you wanna offer ;) | 20:36 |
| cardoe | I wonder what the price ratio of RAM vs gold in mm^2 or some way to compare is now days. | 20:36 |
| TheJulia | How much acid will it require to... umm... yeeeaaahhhhh. ;) | 20:38 |
| shermanm | so it seems like the socat OSSA fix might have broken the socat console entirely? https://bugs.launchpad.net/ironic/+bug/2148331 | 20:39 |
| shermanm | I haven't verified on master yet, but the 2024.1 backport was causing socat "command not found" errors for me, and reverting that commit fixed it | 20:39 |
| JayF | Please file a bug with the full output from running your console. | 20:40 |
| TheJulia | ++ | 20:40 |
| JayF | I am not exaggerating when I say you may be one of the only installations using this in the world. | 20:40 |
| JayF | So more specific = more terrific since none of us have test beds for this | 20:40 |
| TheJulia | I know someone who sort of does, but they have promised to propose some fixes and never did | 20:40 |
| shermanm | gotcha, just wanted to check first since it touched one of the ossa's | 20:40 |
| shermanm | what is everyone else doing to get a console on their baremetal instances if they break networking by accident? | 20:41 |
| JayF | shermanm: we've done approximately 9001 OSSAs in the last three months. I'd be more shocked if one of them /didn't/ break anything. Thanks for the report and it'll get looked at | 20:41 |
| JayF | Redfish graphical console support is extremely good, and getting better | 20:41 |
| JayF | but the "real" answer for most downstreams I've worked is "manual process outside of Ironic" | 20:42 |
| JayF | or rescue mode (Rackspace OnMetal used this)( | 20:42 |
| shermanm | @JayF gotcha! we've been on the socat console because for years it was the only thing that got you a console, in horizon, and was packaged via kolla+kolla-ansible | 20:43 |
| TheJulia | I <3 that claude just "knows" if it is an openstack change with me to whip me up a reno | 20:43 |
| TheJulia | oh, it did the wrong thing! | 20:43 |
| JayF | shermanm: yeah, IPMI usage in general is very much down | 20:43 |
| TheJulia | oh noes! | 20:43 |
| opendevreview | Merged openstack/ironic master: Add a custom configuration type KernelParameterString https://review.opendev.org/c/openstack/ironic/+/992306 | 20:43 |
| opendevreview | Merged openstack/ironic master: Add UnsafeKernelCommandLine class https://review.opendev.org/c/openstack/ironic/+/995749 | 20:43 |
| TheJulia | Supposidly there is a way of websocketing to some bmcs via redfish to do serial consoles | 20:43 |
| JayF | shermanm: not meaning that as a slight, just that literally we have nearly zero test beds for this stuff so good output is even more important | 20:43 |
| TheJulia | but I haven't spent any time on it | 20:43 |
| JayF | I don't wanna spend any time one it lol | 20:43 |
| JayF | we (well, Steve) built a whole damn onion router of VNC proxies for graphical console | 20:44 |
| shermanm | no worries at all, I prefer to report and propose fixes so it's not a patch I have to carry forever by myself... | 20:44 |
| TheJulia | ((the only thing it is missing is an actual onion router for it)) | 20:44 |
| shermanm | besides the whole "community good" aspect I mean | 20:44 |
| JayF | shermanm: in fact, Ironic gives advice now to folks specifically to disable the serial consoles. It can have perf implications on the server itself to redirect console to serial (real or fake) | 20:44 |
| JayF | if it's what you have, it's what you have, but there are reasons beyond "ipmi yuck" that we've driven away from it | 20:45 |
| JayF | but ipmi yuck is a *great* reason anyway lol | 20:45 |
| TheJulia | Yeah, BMCs + serial consoles creates the same basic interrupt overload you get with not using the framebuffer and logging to the console. | 20:45 |
| TheJulia | "oh, its a character, hold down interrupt 3!" | 20:45 |
| shermanm | I've been waiting for years for enough my my old hw to die and switch fully over to redfish, but... :shrug | 20:45 |
| TheJulia | "Oh, another character! hold it down!" | 20:46 |
| shermanm | we only just got everything onto uefi boot | 20:46 |
| JayF | shermanm: now *that* I can help with, and I don't even need a bug | 20:46 |
| shermanm | just a big hammer, right? | 20:46 |
| JayF | shermanm: where do I go with my collection of blunt instruments of IPMI destruction ;) | 20:46 |
| JayF | I mean, there are other ways | 20:46 |
| * TheJulia is giggling and Summer demands to know "do I need to be concerned?" | 20:46 | |
| JayF | cat /dev/random>/dev/dsp # I call this the "make someone else apply hammer to server" method | 20:46 |
| TheJulia | Oh the fun memories of making machine beep in a DC | 20:47 |
| JayF | TheJulia: shermanm: the real scary thing is that if there *is* some OSS community that would know how to brick hardware with just a root command line ... it'd be us | 20:47 |
| TheJulia | hush, we can't reveal our secret incantations. ;) | 20:48 |
| shermanm | I did put grub init-tunes onto our golden image before leaving a past job to troll my coworkers. | 20:48 |
| shermanm | 400 servers beeping chip-tunes slightly out of sync when you roll out a reboot | 20:50 |
| JayF | If someone had secure boot enabled, you could just nuke the keys from orbit now that they've rotated. | 20:50 |
| TheJulia | I had to write "what if an invading army goes for the data center" destruction a few years back. I may have also included piling road flares into the racks and actually burning the facility down.... at the end of the list. | 20:51 |
| JayF | the default keys won't boot anything modern anymore lol | 20:51 |
| shermanm | you can probably manage to corrupt the bios flash tbh | 20:51 |
| JayF | "In case of army attack, here are the actions I won't be taking as my cowardly-self will be fleeing the DC: 1) ...." | 20:51 |
| JayF | fun story, it's impossible to make a SATA disk that's both: 1) compliant with ATA standard and 2) requires firmwares to be signed/authenticated | 20:52 |
| JayF | you have to violate the standard by walling off the ATA-official firmware update mechanisms | 20:53 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Set a minimal default mdns config key list for IPA https://review.opendev.org/c/openstack/ironic-python-agent/+/999337 | 20:56 |
| TheJulia | JayF: there you go | 20:56 |
| JayF | TheJulia: given how it's implemented, does the actual api url need to be in that list, too? | 20:56 |
| TheJulia | claude doesn't *think* so | 20:57 |
| TheJulia | since they don't come from txt records | 20:57 |
| JayF | a very cursory look makes me agree | 20:58 |
| * TheJulia gets a pry bar and goes to the ironic code | 20:58 | |
| TheJulia | yeah, params are separate | 21:00 |
| TheJulia | Hi brain, what was next on today's lit | 21:02 |
| TheJulia | list | 21:02 |
| shermanm | I had actually been considering a "real" use for the serial console - log all of the output from node power-on until the ironic-python-agent checks in, and keep it if provisioning fails to debug pxe boot issues and related | 21:28 |
| clif | cardoe: please yell at me if something is broken in TBN | 21:28 |
| cardoe | will do | 21:29 |
| clif | it's also funny in a strange way to still be writing networking related code that will once again touch prod in rackspace | 21:29 |
| TheJulia | shermanm: Some folks I have known have wanted to use it to zmodem.... | 21:31 |
| TheJulia | JayF: so, your kind of right, it looks like one might need to furnish the inspector url in today's code path because only ironic runs one service now. | 21:34 |
| JayF | clif: just don't mention your hidden trap door bug that will trigger if cardoe doesn't hand over the ram peacefully 😂 | 21:50 |
| JayF | TheJulia: hjensas: I'm mashing buttons for Harold now | 21:53 |
| clif | "ram"somware | 21:53 |
| JayF | hiyoooooo | 21:53 |
| clif | :D | 21:53 |
| TheJulia | JayF: oh, thanks! he won't find them for a few weeks :) | 21:53 |
| JayF | he's on sabbatical? | 21:54 |
| TheJulia | PTO, he goes and gets lost for a few weeks once a year | 21:54 |
| JayF | sounds like fun | 21:54 |
| TheJulia | iknowrite! | 21:54 |
| opendevreview | Merged openstack/networking-generic-switch master: batching: Fix missed batch result watches https://review.opendev.org/c/openstack/networking-generic-switch/+/987867 | 22:00 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Set a minimal default mdns config key list for IPA https://review.opendev.org/c/openstack/ironic-python-agent/+/999337 | 22:02 |
| * TheJulia gives CI cookies | 22:05 | |
| opendevreview | Merged openstack/ironic master: Add [vnc]container_* console container options https://review.opendev.org/c/openstack/ironic/+/995852 | 22:29 |
| opendevreview | Merged openstack/ironic master: Add a container provider for console containers https://review.opendev.org/c/openstack/ironic/+/995853 | 22:32 |
| opendevreview | Merged openstack/ironic master: Add kernel parameter parsing to _prepare_iso_image https://review.opendev.org/c/openstack/ironic/+/992307 | 22:38 |
| opendevreview | Merged openstack/ironic master: Add kernel parameter parsing to build_extra_pxe_options https://review.opendev.org/c/openstack/ironic/+/992308 | 22:44 |
| opendevreview | Merged openstack/ironic stable/2026.1: fix: TBN could not handle evaluations of items that were collections https://review.opendev.org/c/openstack/ironic/+/999333 | 23:08 |
| opendevreview | Merged openstack/ironic master: devstack: Support the container console provider https://review.opendev.org/c/openstack/ironic/+/995854 | 23:15 |
| opendevreview | Merged openstack/ironic master: Document the container console provider https://review.opendev.org/c/openstack/ironic/+/995855 | 23:15 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!