| rpittau | good morning ironic! o/ | 07:42 |
|---|---|---|
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Build and publish updated CS10 and debian images https://review.opendev.org/c/openstack/ironic-python-agent/+/966513 | 10:22 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Build and publish updated CS10 and debian images https://review.opendev.org/c/openstack/ironic-python-agent/+/966513 | 10:38 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/31.0: Handle HTTP 400 and 409 race condition in Redfish power operations https://review.opendev.org/c/openstack/ironic/+/966516 | 10:54 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/30.0: Handle HTTP 400 and 409 race condition in Redfish power operations https://review.opendev.org/c/openstack/ironic/+/966517 | 10:57 |
| opendevreview | nidhi proposed openstack/ironic master: Add PCIe function fields to redfish inspection https://review.opendev.org/c/openstack/ironic/+/963179 | 11:25 |
| * cid Updates https://wiki.openstack.org/wiki/Meetings/Ironic with updates you might want to look at before the meeting :) | 13:12 | |
| *** dking is now known as Guest30846 | 13:12 | |
| *** Guest30846 is now known as dking | 13:14 | |
| rpittau | didn't we rmeove it ? https://bugs.launchpad.net/ironic/+bug/2129469 | 13:34 |
| dking | Good morning, folks. Are there Metal3 guys available? Particularly, could somebody confirm what the latest version tag should be for the metal3 ironic image? | 13:58 |
| rpittau | dking: latest version tag is... latest :) | 14:09 |
| rpittau | dking: https://quay.io/repository/metal3-io/ironic?tab=tags | 14:10 |
| dking | rpittau: Well, I do like to version tag, and somewhere back in the day, we had we had tags like "camp3-v1..." | 14:19 |
| cardoe | hello Ironic. As folks enjoy they're Monday before they get too swamped with the week's things... have a look over at https://review.opendev.org/q/hashtag:%22ironic-week-prio%22+status:open even if you're not a core member, you can help your own changes land by reviewing other changes. Your reviews are helpful for us all. | 14:19 |
| rpittau | dking: you're talking about 2 years ago :D | 14:20 |
| dking | rpittau: Yeah, that was a while ago. I'm just trying to sanity check. I'm asking because we're having some trouble with BMO re-populating Ironic. I'm pretty sure they're talking because Ironic logs have complaints about the UUIDs missing (because nothing is there), and I'm trying to work backwards. The last time this happened, it was a version mismatch. | 14:24 |
| rpittau | dking: I see, can this maybe help? https://book.metal3.io/version_support | 14:26 |
| dking | rpittau: That's very helpful. Thank you!. We're pretty close. I'll add that our local documentation. | 14:33 |
| opendevreview | cid proposed openstack/ironic master: Filter null NIC firmware versions from cache https://review.opendev.org/c/openstack/ironic/+/966567 | 14:33 |
| dking | rpittau: Would you have thoughts on where to start troubleshooting BMO not repopulating Ironic? | 14:33 |
| dtantsur | dking: I assume you've already checked BMO logs? | 14:36 |
| dtantsur | if you're using BMO from the main branch, we've recently dropped support for really old ironics and ironic-inspector | 14:37 |
| dking | dtantsur: Yeah, I'm looking there. It's complaining a lot about "could not get node for BIOS settings: host not registered" | 14:38 |
| dking | dtantsur: ...but that could perhaps just be a symptom? There's a few other errors, too, and it looks like it had been in crashloobackoff. | 14:40 |
| dking | "failed to wait for hostfirmwarecomponents caches to sync kind source: *v1alpha1.HostFirmwareComponents: timed out waiting for cache to be synced for Kind *v1alpha1.HostFirmwareComponents" | 14:42 |
| opendevreview | Merged openstack/ironic-specs master: Add 2026.1 workitems to index https://review.opendev.org/c/openstack/ironic-specs/+/966322 | 14:44 |
| cid | rpittau, re... PostgreSQL. Yup. It's been removed. I couldn't mark as "Won't fix" without a concrete reason. I remember the CI jobs for Postgres were removed back in 2024. | 14:54 |
| TheJulia | Good morning | 15:00 |
| janders | hey TheJulia o/ | 15:01 |
| dtantsur | dking: none of these immediately ring any bells.. | 15:01 |
| dtantsur | Who's our chair this time around? | 15:02 |
| dtantsur | #startmeeting ironic | 15:04 |
| opendevmeet | Meeting started Mon Nov 10 15:04:31 2025 UTC and is due to finish in 60 minutes. The chair is dtantsur. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:04 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:04 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:04 |
| janders | o/ | 15:04 |
| dtantsur | Well, hello everyone, I'm not even sure we have the meeting today, but I'm going to chair it! | 15:04 |
| TheJulia | o/ | 15:04 |
| dtantsur | Who is here for some ironic conversation? | 15:04 |
| TheJulia | \o | 15:05 |
| alegacy | o/ | 15:05 |
| * TheJulia is dancing in the corner | 15:05 | |
| dtantsur | Our agenda is where it usually is: | 15:05 |
| dtantsur | #link https://wiki.openstack.org/wiki/Meetings/Ironic Meeting agenda | 15:05 |
| rpittau | o/ | 15:05 |
| dtantsur | I'm giving the obviously undercaffeinated community members a couple more minutes to join | 15:06 |
| kubajj | o/ | 15:06 |
| mostepha[m] | \o | 15:06 |
| dtantsur | #topic Announcements / Reminders | 15:07 |
| cid | o/ | 15:07 |
| dtantsur | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/FQJSYWZIUX2LJKUGTVB274K4GB3T2QPS/ there is a proposal to expand the core team, current cores please take a look | 15:07 |
| TheJulia | Does the agenda need a coffee break? | 15:09 |
| dtantsur | #link https://specs.openstack.org/openstack/ironic-specs/priorities/2026-1-workitems.html here are the agreed workitems for 2026.1 | 15:09 |
| dtantsur | (please bear with me, I was not prepared to chair this meeting :) | 15:09 |
| dtantsur | Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash | 15:09 |
| dtantsur | Finally, 2026.1 Gazpacho Release Schedule https://releases.openstack.org/gazpacho/schedule.html | 15:10 |
| dtantsur | Anything else to announce or remind of? | 15:10 |
| * dtantsur sprinkles virtual coffee in the air | 15:10 | |
| dtantsur | I wonder how many people are still confused by the daylight changes :) | 15:11 |
| dtantsur | anyway | 15:11 |
| dtantsur | #topic Working Group Updates | 15:11 |
| dtantsur | TheJulia: should we close the eventlet WG or reuse it for further work like delayed tasks? | 15:12 |
| TheJulia | I just removed the line item | 15:12 |
| TheJulia | I'll try to rev the delayed task spec this week, fwiw | 15:12 |
| dtantsur | Good, please ping me if I miss the notification (I most certainly will) | 15:13 |
| dtantsur | alegacy: anything on standalone networking, except for the large patch chain to review? | 15:13 |
| dtantsur | #link https://review.opendev.org/q/topic:%22feature/standalone-networking%22 standalone networking patches | 15:13 |
| alegacy | No, no further updates. I managed to split the large patch into smaller chunks. I hope that's a better configuration. | 15:13 |
| dtantsur | Thanks, appreciated! | 15:14 |
| dtantsur | Meanwhile, I don't have anything new on the asyncio front (dragged into downstream priorities). cid, do you have anything? | 15:14 |
| cid | Not at all. | 15:14 |
| dtantsur | okie cool | 15:15 |
| dtantsur | #topic Bug Deputy Updates | 15:15 |
| dtantsur | cid: thats you again :) | 15:15 |
| cid | Yup :) | 15:15 |
| cid | There were a total of 12 bugs, 5 of which were RFEs | 15:15 |
| TheJulia | Clearly bug filing occurs with caffeine | 15:16 |
| TheJulia | :) | 15:16 |
| dtantsur | heh | 15:16 |
| dtantsur | Should we review the RFEs, do you think? | 15:16 |
| cid | :) | 15:16 |
| cid | I will just drop the full list here | 15:16 |
| cid | https://bugs.launchpad.net/networking-generic-switch/+bug/2130384: Treat a DPU like a switch | 15:16 |
| cid | https://bugs.launchpad.net/ironic/+bug/2130646: Enable the agent deploy interface to auto-switch to bootc | 15:16 |
| cid | https://bugs.launchpad.net/ironic/+bug/2130667: Enable Redfish interactions with network device functions | 15:16 |
| cid | https://bugs.launchpad.net/ironic/+bug/2129469: Add PostgreSQL database backend support | 15:16 |
| cid | https://bugs.launchpad.net/ironic/+bug/2129690: Make all nodes of a conductor-group write grub config | 15:16 |
| dtantsur | TheJulia: DPU-as-a-switch, is it spec worthy? | 15:16 |
| TheJulia | The first one, I filed, the broad idea is the ability for NGS to begin to treat a DPU like a switch | 15:16 |
| TheJulia | I don't think so, I think its more just a driver | 15:17 |
| rpittau | I think we can jsut close the postgresql one as cid suggested before | 15:17 |
| TheJulia | or drivers | 15:17 |
| dtantsur | (I know that we don't usually do specs for n-g-s) | 15:17 |
| TheJulia | Issue is, it can only be developed by someone with their hands on a card | 15:17 |
| dtantsur | Yeah, let's prepare a polite response for PostgreSQL. Something along the lines of "Alas, OpenStack moved on from it, and we cannot maintain another database on our own" | 15:17 |
| TheJulia | which is similar, really, to physical switcehs | 15:17 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Test advertised ip reachability before assigning it https://review.opendev.org/c/openstack/ironic-python-agent/+/963670 | 15:17 |
| dtantsur | TheJulia: I assume you don't have one handy? | 15:18 |
| TheJulia | dtantsur: w/r/t postgres, concur | 15:18 |
| TheJulia | dtantsur: I don't, I might be able to grab one out of a lab personally | 15:18 |
| TheJulia | its more a broad idea than anything else | 15:18 |
| TheJulia | since each card will be different in the end | 15:18 |
| * cid Goes to update the bug | 15:18 | |
| TheJulia | the second one, regarding auto-switching the device, I think stevebaker is taking a look at. | 15:19 |
| dtantsur | Yeah, I see. I don't mind rfe-approved on this, assuming we don't need to change any data model | 15:19 |
| TheJulia | s/device/driver/ | 15:19 |
| dtantsur | (this was about the DPU one) | 15:19 |
| TheJulia | I don't think so, I think the existing model works for that | 15:20 |
| dtantsur | I'm fine with the bootc one, but I know that JayF had reservations, so I'd not approve it until I hear from him | 15:20 |
| * TheJulia wonders if we should have sifted through these one at at ime | 15:20 | |
| cardoe | gah time change :/ | 15:20 |
| TheJulia | s/ime/time/ | 15:20 |
| TheJulia | Daylight savings time is indeed, evil. | 15:20 |
| TheJulia | Possibly the root of all evil. | 15:20 |
| rpittau | hopefully it will disappear from EU next year | 15:21 |
| TheJulia | (We voted it away here in california, but the US seems hellbent on ignoring our resolution) | 15:21 |
| dtantsur | heh | 15:22 |
| cardoe | We did as well in Alabama (to my shock) but we gotta wait just like CA. | 15:22 |
| TheJulia | Next up in the catecory of roots of all evil, cats seeking to cuddle with the cables behind the laptop | 15:22 |
| * TheJulia can't spell today | 15:22 | |
| TheJulia | So, back to RFEs | 15:22 |
| dtantsur | Cats tend to be.. energized ;) | 15:23 |
| cardoe | So I wanted to get to the changing of the deploy interface…. | 15:23 |
| cardoe | It feels like we need a way to change that dynamically-ish. From the nova-ironic side it could be an image property. | 15:23 |
| TheJulia | The third, network device functions, I see that as first step spend some time in sushy modeling the options and largely in sushy but it proposes a general idea of some clean steps to toggle the options around. | 15:23 |
| TheJulia | for going to something like anaconda, I think that makes a lot of sense | 15:24 |
| TheJulia | the same general idea is present though | 15:24 |
| dtantsur | Network device functions is cool, I wonder if you know how much of a mess NetworkInterfaces are... | 15:24 |
| * dtantsur summons janders | 15:24 | |
| TheJulia | aiui, fairly messy, unfortuantely. | 15:24 |
| cardoe | He’s in EU time zone no now? | 15:24 |
| janders | yes | 15:25 |
| janders | we had a chat with TheJulia in Paris about the NetworkAdapters bug I'm tackling | 15:25 |
| dtantsur | ah, NetworkAdapters. Are these different? | 15:25 |
| TheJulia | I was largely thinking we might be able to model it on a single platform | 15:25 |
| TheJulia | so.. | 15:25 |
| TheJulia | you basically need to walk from the system -> Network Adapters -> NetworkDeviceFunctions | 15:25 |
| dtantsur | aha, so you may be forced to ensure that something (IPA?) is running on the machine | 15:26 |
| * janders is looking at BMC mockups to see where things are tied together | 15:26 | |
| janders | ++ | 15:26 |
| janders | or at least have some of a fallback for pathological cases | 15:26 |
| TheJulia | dtantsur: based upon the model, not really. That is if vendors have tied things cleanly and/or we already have the MAC addresses | 15:26 |
| dtantsur | I thought the resource was not accessible when the OS is not running? | 15:27 |
| TheJulia | the host might need to be "on" | 15:27 |
| janders | dtantsur on some servers, yes | 15:27 |
| TheJulia | it really depends on the level of firmware and underlay architecture when it comes to what is done for the BMC to talk to other devices | 15:28 |
| janders | so we could 1)either have a requirement to have OS running when we do work on NetworkInterfaces or 2) try and see, if failure -> boot IPA | 15:28 |
| TheJulia | But the base idea is to take the devices, model them matching the network ports, and toggle network ports out of networking funciton | 15:28 |
| TheJulia | function | 15:28 |
| TheJulia | and towards a storage function | 15:28 |
| TheJulia | Ultimately, it will all depend on what BMCs return and the error handling which is needed there | 15:29 |
| dtantsur | Does it make sense to have something on the Port API to mark it? so that the steps themselves only enforce the value? | 15:29 |
| dtantsur | (similarly to target_raid_config?) | 15:29 |
| TheJulia | I did poke at an idrac10 which was off, and it seemed that the adapters were visible | 15:29 |
| janders | I concur | 15:29 |
| dtantsur | yeah, the problem is with HPE machines | 15:30 |
| janders | exactly | 15:30 |
| TheJulia | I was largely thinking we just use the category function and toggle it to "storage" or some other preconfigurable value | 15:30 |
| TheJulia | and then just dissuade from trying to use the interfaces for anything else. | 15:30 |
| TheJulia | which ties into TBN | 15:30 |
| dtantsur | ah, right, we have many more fields now that I'm not aware of :) | 15:30 |
| dtantsur | okay, I personally like the idea, but I'd prefer a bit more technical details (and maybe an RFE per step) | 15:31 |
| TheJulia | I'm not sure if I'll be able to start on it this year or not, still trying to determine the priorities until EoY | 15:31 |
| dtantsur | ack | 15:32 |
| TheJulia | It is seeming like VXLAN networking is more important on my front, realistically. | 15:32 |
| dtantsur | Then let's table it for now and get back once you have cycles? | 15:32 |
| TheJulia | I just wanted to get something recorded that is the seed of the idea first | 15:33 |
| TheJulia | onward! | 15:33 |
| dtantsur | So, the last is PXE HA | 15:34 |
| dtantsur | https://bugs.launchpad.net/ironic/+bug/2129690 | 15:34 |
| dtantsur | I'd *love* to have this for metal3 HA. I think you even had a draft spec some time ago TheJulia? | 15:34 |
| dtantsur | Like before, I doubt anyone has cycles for that | 15:35 |
| TheJulia | Yes, I know exactly how this would need to work | 15:35 |
| TheJulia | Realistically, it would take a Conductor RPC call to the remote conductors | 15:36 |
| dtantsur | I'm still worried about RPC multiplication, but I guess we can get back to it on the actual spec if anyone revives it.. | 15:36 |
| TheJulia | for them to do the needful in an abridged form with some additional details in terms of networking | 15:36 |
| TheJulia | Where it would need to be wired in, its actually not a big deal and if we have deferred tasks then its feasible to just use that as a mechanism as well | 15:37 |
| TheJulia | The key difference is somehow the other conductors need to so some amount of work and it will need to be a shared task under the hood. | 15:37 |
| dtantsur | yeah, there are interesting nuances | 15:38 |
| dtantsur | I'd also prefer the secondary conductors to refer to images from the primary one, not download them from the internet | 15:38 |
| dtantsur | (which is not fully HA though, so I dunno if it fixes the issue in question) | 15:38 |
| TheJulia | oh, yeah | 15:38 |
| TheJulia | that was something I had baked into the previous idea | 15:39 |
| TheJulia | just include a pointer to the other node | 15:39 |
| TheJulia | but the broad fear of conductor triggering 1 to N additional messages over RPC was sort of the hurdle the discussion died at | 15:39 |
| cardoe | So do we just need like "conductor can upload to an object storage that's not swift"? | 15:39 |
| TheJulia | for a super short lived "create a pointer", thats likely not a big deal | 15:39 |
| TheJulia | So | 15:40 |
| TheJulia | that goes into conductor failover theory | 15:40 |
| TheJulia | when a failure occurs, the conductor will automatically try to download artifacts | 15:40 |
| dtantsur | Another approach would be for secondary conductors to download images from the primary one | 15:41 |
| TheJulia | that is a greater multiplication risk, really | 15:41 |
| dtantsur | it is | 15:41 |
| TheJulia | the original idea was just generate a pointer to load config/artifacts from the first, and should failure occur and takeover triggers then the conductor and presumably new secondaries would get calls to create new records and new pointers | 15:42 |
| TheJulia | I guess | 15:42 |
| dtantsur | yeah, fair | 15:42 |
| TheJulia | the challenge is how much of a lag are we okay with... or not. | 15:42 |
| dtantsur | now someone needs to do the hardest parts: write/update the spec and the code :D | 15:42 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix storing inventory and plugin data in Swift https://review.opendev.org/c/openstack/ironic/+/966259 | 15:42 |
| TheJulia | yeah | 15:44 |
| dtantsur | Can we agree that the last one definitely needs-spec? | 15:44 |
| TheJulia | yeah, definitely | 15:44 |
| TheJulia | Its totally do-able, but requires agreement on which time/risk gap we're focused on as well | 15:45 |
| * dtantsur nods | 15:45 | |
| dtantsur | cid: anything else on bugs or rfes? | 15:45 |
| TheJulia | (this quickly turns into one of those "perfection is the enemy of good" things as well. | 15:45 |
| dtantsur | also true | 15:45 |
| * dtantsur suspects everyone else has falled asleep | 15:45 | |
| cid | That's all on bug deputy updates | 15:45 |
| dtantsur | Thanks! | 15:46 |
| dtantsur | #topic Open discussion | 15:46 |
| dtantsur | The floor is yours, crew | 15:46 |
| rpittau | I have one thing | 15:46 |
| dtantsur | you have our attention | 15:46 |
| TheJulia | dtantsur: mandatory coffee break at the beginning of meetings? | 15:46 |
| dtantsur | ++ | 15:47 |
| dtantsur | if it was not 4pm here.. | 15:47 |
| rpittau | when moving from tinyipa to DIB for integration tests in bifrost I just realized that the old GRUB can't really work well with big vmedia devices when in UEFI | 15:47 |
| TheJulia | dtantsur: agenda updated. | 15:47 |
| rpittau | meaning that the vmedia jobs will be broken for bookworm and jammy | 15:47 |
| rpittau | and not feeling very well for noble | 15:47 |
| TheJulia | Well, that is UEFI in general on some hardware as well. | 15:47 |
| dtantsur | will it work anywhere at all? | 15:48 |
| TheJulia | Is it the 512MB barrier? | 15:48 |
| rpittau | they run ok on centos10 as it has a more recent version of grub | 15:48 |
| rpittau | it';s smaller | 15:48 |
| TheJulia | SRSLY? | 15:48 |
| * TheJulia waits for a YA'RLY | 15:48 | |
| dtantsur | hmm, don't we have vmedia coverage in the devstack CI? the standalone job? | 15:48 |
| rpittau | anyway jsut wanted to say that I'm going to test with debian DIB | 15:48 |
| rpittau | but I would also like to propose alpine as an alternative | 15:48 |
| TheJulia | dtantsur: centos based afaik | 15:48 |
| dtantsur | TheJulia: then you had to ask O'RLY? | 15:48 |
| JayF | Alpine will introduce new testing service to us as it uses MUSL | 15:49 |
| dtantsur | TheJulia: ehhmmm? we have devstack jobs on centos?? | 15:49 |
| JayF | **surface | 15:49 |
| TheJulia | well, okay, ubuntu but they boot centos | 15:49 |
| JayF | I don't think it'll matter but it's worth noting | 15:49 |
| rpittau | true | 15:49 |
| rpittau | the alternative is to drop bookworm and ubuntu jammy entirely | 15:49 |
| rpittau | if debian based can't work | 15:50 |
| dtantsur | rpittau: hold on, does it mean that only DIB-building jobs are affected? | 15:50 |
| rpittau | yep | 15:50 |
| rpittau | I wrote that at the beginning :D | 15:50 |
| dtantsur | you.. did not? | 15:50 |
| dtantsur | DIB building jobs don't even use vmedia, so I'm still confused | 15:50 |
| rpittau | I wrote "when moving from tinyipa to DIB for integration tests in bifrost" | 15:51 |
| dtantsur | right, but not all integration tests are affected, only dibipa ones | 15:51 |
| dtantsur | which don't use vmedia, so... wut? | 15:51 |
| rpittau | I probably can't explain myself | 15:52 |
| rpittau | I'm converting ALL the jobs to DIBIPA | 15:52 |
| rpittau | https://review.opendev.org/c/openstack/bifrost/+/964404 | 15:52 |
| dtantsur | wait, why? | 15:52 |
| rpittau | because we don't support tinyipa anymore | 15:52 |
| dtantsur | we have IPA images published, we don't need to build them in literally every job | 15:52 |
| rpittau | we don't | 15:52 |
| TheJulia | why not? | 15:52 |
| dtantsur | (dibipa jobs build IPA using DIB, hence their name) | 15:52 |
| rpittau | in the integration jobs we don't build the images | 15:53 |
| rpittau | we used the published ones | 15:53 |
| TheJulia | So why don't published images work? | 15:53 |
| dtantsur | Ah, so you mean if we switch our official images to Debian, then.. oh | 15:53 |
| rpittau | because they're too big for uefi | 15:53 |
| rpittau | so | 15:53 |
| rpittau | if we switch to dib with cs9/10 they're super big >500 MB | 15:54 |
| rpittau | with debian we're around 300MB I think | 15:54 |
| dtantsur | CS9/10 is already know to work | 15:54 |
| rpittau | with recent version of grub, yes | 15:54 |
| rpittau | with grub on bookworm or jammy, nope | 15:54 |
| dtantsur | does it mean that the ironic's standalone jobs still use tinyipa? | 15:55 |
| opendevreview | Merged openstack/networking-generic-switch master: Drop remaining logic for linuxbridge-agent https://review.opendev.org/c/openstack/networking-generic-switch/+/965149 | 15:55 |
| TheJulia | Are the ipab results still 500+ MB? | 15:55 |
| rpittau | I'm ok making bookworm and jammy non-voting for the time being just to switch to DIB | 15:55 |
| rpittau | but I'm not sure debian based still make it work | 15:55 |
| TheJulia | I thought I posted patches to remove like 90+ MB | 15:56 |
| * TheJulia senses we're in the weeds and doing ourself a disservice from asynchronous discussion | 15:56 | |
| janders | I'd like to briefly touch on iRMC deprecation/removal before we wrap up | 15:56 |
| dtantsur | rpittau: could you check it? I think we might have pre-built images in the ipa-b location | 15:56 |
| TheJulia | rpittau: maybe some deep synchronous discussion after the meeting? | 15:56 |
| dtantsur | if debian IPA images is a workaround, we're fine | 15:56 |
| TheJulia | dtantsur: we do afaik | 15:57 |
| dtantsur | we don't need a call, we need more information | 15:57 |
| TheJulia | it might be the IPA location is not updated yet if we've not merged anything in that repo yet | 15:57 |
| rpittau | dtantsur, TheJulia, I checked the images, cs9 is 490MB, cds10 is 590MB | 15:57 |
| rpittau | https://tarballs.opendev.org/openstack/ironic-python-agent-builder/dib/files/ | 15:57 |
| dtantsur | rpittau: let's create a variation of your patch that uses https://tarballs.opendev.org/openstack/ironic-python-agent-builder/dib/files/ipa-debian-master.initramfs | 15:57 |
| dtantsur | and see if it passes | 15:57 |
| dtantsur | if yes, we're golden, just need to switch our official images | 15:58 |
| dtantsur | if not, we may need to actually seriously talk tomorrow | 15:58 |
| rpittau | yeah that's what I was going to do | 15:58 |
| rpittau | just wanted to check if we're open to the alpine alternative for CI | 15:58 |
| rpittau | alright | 15:58 |
| TheJulia | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/965984 will help, a lot. | 15:58 |
| dtantsur | we have some crazy options like using a different grub binary... but let's wait for the results | 15:58 |
| TheJulia | janders: might as well :) | 15:58 |
| janders | I made initial contact with Fujitsu. Most of the discussion is downstream related ( TheJulia I will loop you into this for OSP part ). The upstream-relevant part is FJ are open to deprecation/removal of iRMC from master if it remains in stable branches. | 15:59 |
| janders | question 1) is this doable | 15:59 |
| * dtantsur issues a 1 minute warning | 15:59 | |
| janders | question 2) how would they be proposing critical fixes if needed? straight to stable/whatever branch? | 15:59 |
| dtantsur | it's definitely doable, and we definitely won't remove code from stable branches | 15:59 |
| * dtantsur doubts they'll propose any fixes now | 16:00 | |
| TheJulia | 1) yes | 16:00 |
| dtantsur | I've already let janders know my opinion, so please others weigh in | 16:00 |
| TheJulia | 2) straight to the branch | 16:00 |
| janders | dtantsur I agree, unless there's some realisation about CVEs | 16:00 |
| dtantsur | like in ancient pysnmp? ;) | 16:00 |
| janders | haha they may need to get creative | 16:01 |
| janders | it's damage control at this stage anyway | 16:01 |
| dtantsur | anyway. does anyone object tho the approach that janders is proposing? | 16:01 |
| TheJulia | not at all, it is what it is | 16:01 |
| janders | sounds like we have way forward so I will keep pursuing that - there will be some downstream discussion followed by upstream comms once things are clear | 16:01 |
| janders | thank you for your insights | 16:02 |
| TheJulia | Thanks! | 16:02 |
| dtantsur | We need to wrap up, thank you folks! | 16:02 |
| janders | o/ | 16:02 |
| dtantsur | #endmeeting | 16:02 |
| opendevmeet | Meeting ended Mon Nov 10 16:02:20 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2025/ironic.2025-11-10-15.04.html | 16:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2025/ironic.2025-11-10-15.04.txt | 16:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2025/ironic.2025-11-10-15.04.log.html | 16:02 |
| cid | ++ | 16:02 |
| dtantsur | JayF: do we have any requests to support alpine? I thought they were more of base OS for containers... | 16:02 |
| JayF | dtantsur: It was mentioned as an option, I just wanted to point at musl v glibc. To be clear, any Ironic-impacting errors there wouldn't be our bugs, but I'd rather steer clear of other folks' bugs too :D | 16:03 |
| rpittau | btw I think we should grab the images from IPA not ipa-builder, hence we need https://review.opendev.org/c/openstack/ironic-python-agent/+/966513 | 16:04 |
| dtantsur | rpittau: sure, I meant to do it as an experiment, not final solution | 16:04 |
| dtantsur | rpittau: I dunno about publishing CS10 images without ever testing them | 16:05 |
| dtantsur | why? | 16:05 |
| rpittau | mmm that's true, and we're going to witch to debian anyway | 16:07 |
| rpittau | ok, I'll remove that | 16:07 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Build and publish updated debian images https://review.opendev.org/c/openstack/ironic-python-agent/+/966513 | 16:09 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [WIP] Remove tinyipa support and switch to debian IPA https://review.opendev.org/c/openstack/bifrost/+/964404 | 16:12 |
| rpittau | testing debian ipa on bookworm ^ | 16:13 |
| opendevreview | Merged openstack/bifrost master: [CI] upgrade from 2025.2 https://review.opendev.org/c/openstack/bifrost/+/966283 | 16:13 |
| JayF | I'm going to execute the approver/reviewer changes from the mailing list. This is your last few minutes of opportunity to object. | 16:15 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix storing inventory and plugin data in Swift https://review.opendev.org/c/openstack/ironic/+/966259 | 16:26 |
| opendevreview | Merged openstack/ironic-python-agent stable/2025.2: Fix for matching hints with lists of strings https://review.opendev.org/c/openstack/ironic-python-agent/+/966031 | 16:38 |
| opendevreview | nidhi proposed openstack/sushy master: Add complete LLDP fields to Port.Ethernet.LLDPReceive per DMTF Redfish v1.12.0 https://review.opendev.org/c/openstack/sushy/+/966616 | 16:44 |
| opendevreview | nidhi proposed openstack/sushy master: Add complete LLDP fields to Port.Ethernet.LLDPReceive per DMTF Redfish v1.12.0 https://review.opendev.org/c/openstack/sushy/+/966616 | 16:46 |
| opendevreview | nidhi proposed openstack/sushy master: Add complete LLDP fields to Port.Ethernet.LLDPReceive per DMTF Redfish v1.12.0 https://review.opendev.org/c/openstack/sushy/+/966616 | 16:47 |
| dking | dtantsur: When setting IRONIC_IP in BMO, does that default to an HTTP or HTTPS connection? | 16:55 |
| opendevreview | Verification of a change to openstack/ironic-python-agent-builder master failed: Wait up to 30 seconds for config drive https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/962673 | 16:56 |
| JayF | Gerrit auth changes have been finalized: cid cardoe should have approvers access now, kubajj has reviewer access now, mgoddard removed from approvers | 16:58 |
| JayF | Congratulations | 16:58 |
| kubajj | JayF: 🫡 thanks | 16:58 |
| cid | \o/. Thanks Ironic for the vote of confidence. | 16:59 |
| TheJulia | woot! | 17:00 |
| dtantsur | dking: there is no such option in BMO, did you mean ironic-image? | 17:00 |
| dtantsur | the BMO option is called IRONIC_ENDPOINT and includes the whole URI | 17:00 |
| cardoe | Thank you all :) | 17:01 |
| dking | dtantsur: I suppose that's why I couldn't find it. :) I think it might be sharing some tings in the configmap. I'm starting to grasp at straws. The Ironic endpoint is set correctly. I just found some line in the error logs that looked like it was trying to connect with the wrong handshake. | 17:03 |
| dtantsur | dking: the current deployment scripts in BMO are a mess. They do indeed mix options for BMO, option for Ironic and options for the CI. | 17:03 |
| dking | That probably explains that. I don't think that we're doing anything too custom here. | 17:05 |
| dtantsur | This all is one of the main reasons I started ironic-standalone-operator | 17:06 |
| dking | That might be worth looking at. At the moment, though, our setup has been pretty stable for a few years, and we're not quite ready to rock the boat. | 17:07 |
| dtantsur | I can imagine | 17:07 |
| dking | ... but after this latest update, something just isn't happy. Fortunately, it's not a production environment, but we'll be doing that soon. | 17:08 |
| dtantsur | dking: which versions do you use? | 17:09 |
| dking | dtantsur: So, currently, BMO has a pod continually going into CrashLoopBackoff, and also, our first indication of a problem was that Ironic was never repopulated. I see several issues in the logs... and I just noticed that it has errors that "spec.architecture" is an unknown field. | 17:12 |
| dking | dtantsur: BMO: v0.11.1, Ironic: v31.0.0 That's what we just updated to. | 17:13 |
| dtantsur | spec.architecture, huh? maybe you did not update CRDs properly? | 17:15 |
| dtantsur | spec.architecture was added in BMO 0.4, so something must be really off | 17:16 |
| dking | dtantsur: I'm ashamed to say that in this non-prod environment, we are using an older version than that. So, that might be why it's not there. | 17:17 |
| dking | We were around v0.3 | 17:18 |
| dtantsur | dking: did you upgrade from 0.3 to 0.11? Could it be that you did not upgrade CRDs or something went wrong there? | 17:20 |
| dking | dtantsur: I didn't do the upgrade myself, but yes, it was bumped directly from 0.3 to 0.11.1, and it doesn't look like anything further was done. | 17:24 |
| dking | These machines are currently running, so I'm hoping to not have to re-provision them if I can help it, but I do have at least one that can be completely removed as it was already ready to clean when we noticed the issue. | 17:24 |
| opendevreview | Clif Houck proposed openstack/ironic master: Trait Based Networking Simulator https://review.opendev.org/c/openstack/ironic/+/966202 | 17:25 |
| dtantsur | if only the code was bumped, it can explain the symptoms | 17:26 |
| dtantsur | (I'm not sure why you mention reprovisioning, BMHs themselves are fine probably) | 17:26 |
| dtantsur | I cannot provide specifics since I don't know how your installations and upgrades work | 17:27 |
| dking | dtantsur: Okay. So, forgive my ignorance, but what is the best way to update these BMH properly after the image update? | 17:27 |
| dtantsur | You don't need to update BMH | 17:28 |
| dtantsur | Make sure CRD (not CR!) are up-to-date. I.e. that you've applied the latest version of https://github.com/metal3-io/baremetal-operator/blob/main/config/base/crds/bases/metal3.io_baremetalhosts.yaml | 17:28 |
| dtantsur | (or, well, the 0.11 version) | 17:28 |
| dtantsur | This is how the fields are defined, they don't come from the BMO code implicitly | 17:29 |
| dking | Ah, thanks. I'll see if I can get that done. 8 versions is a big jump, so we should have probably taken more care, but that's why it's in dev. | 17:32 |
| dtantsur | True. I don't remember breaking changes but better safe than sorry. | 17:32 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/961498 (this is the tip of Trait Based Networking) could use a review | 17:41 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/965090 (r pittau's docs update for releasing) could use a review | 17:47 |
| TheJulia | I'm likely not going to be able to circle to reviews until much latter today, fwiw | 17:53 |
| JayF | Similarly I'll be mostly gone tomorrow so trying to get a review on all the things today :) | 18:07 |
| TheJulia | heh, I'm over here trying to get groundwork done so I can get reviews done later. joy! | 18:09 |
| TheJulia | rpittau: any thoughts on disabling or non-voting the arm64 job on ipa-b ? | 18:43 |
| JayF | Do we need some kind of Ironic-netowrking sync? I'm thinking about https://review.opendev.org/c/openstack/ironic-specs/+/940861/2#message-5800acbabaf63de96df0e7320f7d988d654cb67b and how with TBN, ironic-networking (standalone) and the VXLan work ongoing + maybe this tags thing, we should make sure we're not making a different decoder ring for each finger :D | 18:47 |
| JayF | cc clif | 18:48 |
| JayF | I'm all for figuring out things as we implement them but especially with so many moving parts it seems like an ounce of prevention could be worth a pound of cure here | 18:48 |
| TheJulia | we do likely need one, just hesitant to commit at the moment | 18:58 |
| JayF | same but I wouldn't hate us all making sure we're working together | 18:58 |
| JayF | for instance, in that spec, I could invision it turning into "syntax sugar" to activate TBN | 18:58 |
| JayF | even if we need a simpler door to scheduling, I'd rather not reimplment the world | 18:58 |
| JayF | especially since I may desire a phase 3 involving hooking this up to standalone ironic-networking :D | 18:59 |
| TheJulia | On my end it is more a issue of I have a ton of requirements and more so its indecision on which is the. most pressing | 19:22 |
| opendevreview | Merged openstack/ironic master: Fix storing inventory and plugin data in Swift https://review.opendev.org/c/openstack/ironic/+/966259 | 19:37 |
| opendevreview | Merged openstack/ironic stable/2025.2: Handle HTTP 400 and 409 race condition in Redfish power operations https://review.opendev.org/c/openstack/ironic/+/966290 | 20:14 |
| *** milan is now known as Guest30883 | 20:15 | |
| cardoe | JayF: I think it would be good to have an how will this work end to end. | 21:15 |
| JayF | My main thing is I just wanna make sure all the network folks are talking to each other | 21:15 |
| JayF | we have enough parallel work happening that in 6-12 months Ironic networking could be massively improved | 21:15 |
| JayF | but when that much work is going on at the same time the risk for making it worse (or more confusing... maybe more confusing that necessary) if we don't | 21:16 |
| opendevreview | Merged openstack/ironic master: fix: ensure that portgroup physical_network is updated for tests https://review.opendev.org/c/openstack/ironic/+/965985 | 22:02 |
| cardoe | JayF: agreed. happy to have a call at some point. | 22:27 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: pass along physical_network to neutron from the baremetal port https://review.opendev.org/c/openstack/ironic/+/964570 | 22:29 |
| cardoe | JayF: ^ like that I'd love to have a chat with how multiple VXLAN fabrics would work with scheduling. | 22:30 |
| cardoe | Cause we've said use category which could work but we've also said use category for bonding. | 22:30 |
| opendevreview | Merged openstack/ironic-python-agent master: Test advertised ip reachability before assigning it https://review.opendev.org/c/openstack/ironic-python-agent/+/963670 | 22:40 |
| JayF | you should be talking to clif about this more than me | 22:41 |
| JayF | and I think your concerns are sorta mixing TBN + non-TBN | 22:41 |
| cardoe | Happy to speak with clif as well. | 22:41 |
| JayF | "what bonds together" + "what VXLAN fabric to attach" from an Ironic standpoint would all be inside the TBN logic | 22:41 |
| JayF | I'm just saying I am speaking in general he has the specifics better nailed down | 22:41 |
| cardoe | Agreed | 22:41 |
| opendevreview | Merged openstack/ironic master: fix: local_link_connection inspection hook does not fail on missing port https://review.opendev.org/c/openstack/ironic/+/966373 | 22:47 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!