| opendevreview | Merged openstack/ironic master: Fix firmware update failure when task remains in RUNNING state https://review.opendev.org/c/openstack/ironic/+/970998 | 00:21 |
|---|---|---|
| opendevreview | Merged openstack/ironic master: Fix BIOS firmware not flashing when task completes before first poll https://review.opendev.org/c/openstack/ironic/+/970999 | 00:28 |
| opendevreview | Merged openstack/ironic master: Fix BMC resource validation to catch all sushy exceptions https://review.opendev.org/c/openstack/ironic/+/971000 | 00:36 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 00:38 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add retry logic for boot device changes during POST https://review.opendev.org/c/openstack/ironic/+/971150 | 00:38 |
| opendevreview | Merged openstack/ironic master: Fix introspection failure handling https://review.opendev.org/c/openstack/ironic/+/973560 | 01:07 |
| TheJulia | So, I'm starting to think that my headache is more a result of configuration, but haven't qtuie figured it out yet | 01:12 |
| TheJulia | Okay, I've got it binding | 01:59 |
| TheJulia | it was my own doing, but now need to get/ensure packets flow | 02:00 |
| TheJulia | That is for tomorrow | 02:00 |
| cardoe | :/ I was gonna push my update but it sounds like ya do it. | 02:34 |
| opendevreview | Julia Kreger proposed openstack/networking-baremetal master: WIP: l2vni baremetal mech driver https://review.opendev.org/c/openstack/networking-baremetal/+/973889 | 04:03 |
| TheJulia | Thats my current state, but if that does work entirely, then I suspect we sort of boil down to the far end port management. I need to take a look at your structure a little more closely | 04:04 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add retry logic for boot device changes during POST https://review.opendev.org/c/openstack/ironic/+/971150 | 07:42 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 07:42 |
| rpittau | good morning ironic! o/ | 07:43 |
| kubajj | good morning Ironic! o/ | 07:52 |
| kubajj | cardoe: dtantsur, Do I understand it correctly that the inspection rules were not 100% complete in Epoxy? (Want to retire inspector before upgrading further) | 07:53 |
| rpittau | janders cardoe what's the issue with the Metal3 integration job? Happy to help fix that | 07:53 |
| rpittau | kubajj: inspection rules were completed during epoxy (2025.1), at least for in-band inspection, but I would suggest using 2025.2 for out-of-band compatibility plus improvements and bug fixes | 08:50 |
| kubajj | rpittau: ok, we are trying to keep up at least with SLURPs, so we will start using them when we upgrade to Gazpacho | 09:01 |
| opendevreview | Merged openstack/python-ironicclient stable/2025.2: fix: report compatibility with 2025.2 release of Ironic https://review.opendev.org/c/openstack/python-ironicclient/+/973295 | 09:06 |
| janders | rpittau it looks like some artifact was missing from GH, seems like a lag between generation and publishing (subsequent runs passed) | 09:16 |
| janders | https://zuul.opendev.org/t/openstack/builds?job_name=metal3-integration | 09:23 |
| janders | it was repeatedly failing but recovered and has many good runs now | 09:23 |
| rpittau | janders: I see, CAPI release change cause some mess | 09:47 |
| rpittau | we could hardcode the version, dynamic resolution as we do now can fail on edge cases like this one | 09:47 |
| rpittau | somethin to discuss on metal3 meeting today maybe | 09:47 |
| janders | ++ | 09:47 |
| kubajj | cardoe: sorry, somehow I removed the "work in progress" from the change by commenting | 09:53 |
| dtantsur | kubajj: note that cardoe found a series of issues with the new implementation, so you may want to keep an eye on his patches | 09:54 |
| kubajj | dtantsur: yep, will do, thanks | 09:54 |
| rpittau | janders: without hardcoding, this should help preventing those issues https://github.com/metal3-io/metal3-dev-env/pull/1629 | 10:04 |
| opendevreview | Merged openstack/ironic master: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 11:02 |
| opendevreview | Merged openstack/ironic master: Add retry logic for boot device changes during POST https://review.opendev.org/c/openstack/ironic/+/971150 | 11:02 |
| janders | thanks rpittau | 11:34 |
| cardoe | kubajj: no worries. I'm aiming to have a local tool for testing inspection rules. | 14:00 |
| alegacy_ | dtantsur: thanks for your feedback on https://review.opendev.org/c/openstack/ironic/+/966470. I've addressed your comments. | 14:12 |
| * dtantsur will check after the meeting | 14:14 | |
| opendevreview | nidhi proposed openstack/ironic master: Add LLDP collect for DRAC Redfish inspection https://review.opendev.org/c/openstack/ironic/+/970630 | 14:15 |
| TheJulia | good morning | 14:22 |
| TheJulia | good morning again, because I'm a zombie | 14:25 |
| dtantsur | TheJulia: good morning, zombie Julia! | 14:26 |
| kubajj | eicieekddjnflkljehbtcnjgrftlubvrilkfnclinkcd | 14:32 |
| kubajj | oops, sorry 🥲 | 14:32 |
| dtantsur | Hello, kubajj's cat! | 14:32 |
| TheJulia | meow! pspsppspsppsmeow1 | 14:32 |
| cardoe | A couple of checks / backports if folks have time: https://review.opendev.org/c/openstack/python-ironicclient/+/973440 https://review.opendev.org/c/openstack/ironic/+/973532 https://review.opendev.org/c/openstack/ironic/+/973533 | 14:36 |
| cardoe | Honestly https://review.opendev.org/c/openstack/ironic/+/973557 is another big-ish one that I think improves the behavior and safety of VIF handling. | 14:37 |
| cardoe | rpittau: can I ask for a python-ironicclient stable/2025.2 release? Just so I've got a pypi installable version that reports compat with the 2025.2 API version of Ironic? | 14:38 |
| rpittau | cardoe: they're all up, the releases I mean, from 2024.2 to 2026.1 | 14:39 |
| cardoe | sweet thank you! | 14:39 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2025.2: Fix introspection failure handling https://review.opendev.org/c/openstack/ironic/+/974069 | 14:39 |
| TheJulia | has a release_mapping patch been posted for ironic? | 14:40 |
| cardoe | I don't see one | 14:42 |
| opendevreview | Doug Goldstein proposed openstack/ironic bugfix/33.0: Fix introspection failure handling https://review.opendev.org/c/openstack/ironic/+/974070 | 14:43 |
| cardoe | Do we have it documented how to make one? | 14:43 |
| TheJulia | You just map the current master versions to the released numbered version in the file | 14:43 |
| TheJulia | I think we link to an example in the docs, its a little different at cycle end because we update the release name to the version as well | 14:44 |
| rpittau | why the release_mapping? it | 14:44 |
| rpittau | it's ironicclient | 14:44 |
| rpittau | oh well, I guess we need one as next bugfix is coming out next week :D | 14:45 |
| TheJulia | yeah, for that | 14:46 |
| rpittau | cardoe: an example https://review.opendev.org/c/openstack/ironic/+/968306 | 14:48 |
| cardoe | So there's the node health patch which will bump the version... https://review.opendev.org/c/openstack/ironic/+/966946 do we want to let that land first? | 14:48 |
| TheJulia | If we can, it woudl be ideal | 14:49 |
| cardoe | I believe that's the last API bump we've got in the pipeline for 2026.1 | 14:49 |
| cardoe | I think that node health patch is pretty solid at this point. Assuming it passes tests I gave it my +2. I just re-reviewed it. | 14:49 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/973440 should land before the node health change to the client.... it fixes the test to actually do something instead of being papered over. The node health client patch continues the papering over the test so it conflicts. But that change shouldn't be in the node health client patch. | 14:53 |
| cardoe | if we're gonna land the node health API bump, rpittau do we want to hold off on 5.15.0 until we get the client patch in there to support that API? | 15:00 |
| cardoe | rpittau: actually... -1 on the 5.15.0.. cause we need to fix the portgroup API change | 15:00 |
| rpittau | cardoe: is the fix going to merge soon? | 15:01 |
| cardoe | I'll work on that this morning... then get back to networking with TheJulia | 15:01 |
| cardoe | rpittau: yep I'll link ya... https://review.opendev.org/c/openstack/python-ironicclient/+/973440 is needed first. | 15:01 |
| rpittau | ok, letm eblock that then | 15:01 |
| opendevreview | Merged openstack/python-ironicclient master: fix: test determining latest API version https://review.opendev.org/c/openstack/python-ironicclient/+/973440 | 15:21 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: feat: add 'physical_network' and 'category' to portgroup object https://review.opendev.org/c/openstack/python-ironicclient/+/974097 | 15:36 |
| cardoe | ^ adds API 1.102 and 1.103 support to the client | 15:36 |
| cardoe | We've already landed 1.104 | 15:36 |
| cardoe | 1.105 doesn't need a client change | 15:37 |
| cardoe | 1.106 is https://review.opendev.org/c/openstack/python-ironicclient/+/970562 | 15:37 |
| cardoe | 1.107 is already supported | 15:39 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: Filter portgroups by shard https://review.opendev.org/c/openstack/python-ironicclient/+/970562 | 15:41 |
| opendevreview | Merged openstack/ironic master: Add configuration options for Trait Based Networking https://review.opendev.org/c/openstack/ironic/+/968054 | 15:41 |
| cardoe | janders: please rebase https://review.opendev.org/c/openstack/ironic/+/966946 and we can land that | 15:43 |
| opendevreview | Clif Houck proposed openstack/ironic master: Integrate TBN with vif_attach https://review.opendev.org/c/openstack/ironic/+/973690 | 15:47 |
| opendevreview | Clif Houck proposed openstack/ironic master: WIP: Update TBN simulator for vif_attach planning https://review.opendev.org/c/openstack/ironic/+/973691 | 15:47 |
| clif | cardoe: ^^^ first patch that might possibly be functional wrt TBN + vif_attach? | 15:48 |
| clif | unit tests pass, I need to actually functionally test though | 15:48 |
| clif | there's some TODOs still that I should address before it merges but I feel like it's very close | 15:49 |
| clif | and I still need to update the sim to model vif_attach flow | 15:51 |
| cardoe | good deal. I'll give it a shot. | 15:51 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/974097 should be the API needful for your portgroup changes. | 15:51 |
| cardoe | We've got a ton of openstacksdk stuff in flight too. | 16:22 |
| cid | Erm, assuming I rechecked a change that I had open all along and forgot to reload it to know the latest state before doing, what happens if the PR is already merged :) | 16:24 |
| JayF | Nothing! | 16:25 |
| JayF | :D | 16:25 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: feat: add 'physical_network' and 'category' to portgroup object https://review.opendev.org/c/openstack/python-ironicclient/+/974097 | 16:26 |
| dtantsur | Yeah, nothing | 16:26 |
| dtantsur | So, folks. From the look of the code, on any "abort" action, we're doing a complete clean-up: disconnect virtual media, power off, etc. Was it intended? | 16:26 |
| dtantsur | I'm looking into a workflow which is similar to "abort the action, figure out the state of the node, maybe retry", but this behavior gets in the way | 16:27 |
| JayF | abort takes you to a fail state | 16:27 |
| dtantsur | (ref https://github.com/metal3-io/baremetal-operator/issues/2923) | 16:27 |
| JayF | fail state cleans up all the stuff | 16:27 |
| JayF | that's been the operator expectation long enough that I'd be very, very hesitant to change it | 16:27 |
| JayF | we've never approached the idea of e.g. resuming cleaning | 16:28 |
| JayF | with the assumption being a machine is only "cleaned" (or "verb'ed") until it did the process start to finish successfully | 16:28 |
| dtantsur | yeah, I'm not talking about resuming where we left | 16:28 |
| JayF | I'm not clear what you want then | 16:28 |
| dtantsur | more of: my personal assumption was that we clean up on the way *from* clean fail | 16:28 |
| dtantsur | I want the machine left intact for investigation without it going through any processes | 16:29 |
| dtantsur | (and staying there for any time if needed) | 16:29 |
| JayF | Hm. | 16:29 |
| JayF | I... don't really have good ideas for this. | 16:30 |
| JayF | maintenance would be how to do it if you were there pre-fail | 16:30 |
| dtantsur | Maintenance does not stop all actions in Ironic | 16:31 |
| JayF | It (at least last release I actively maintained a cloud) would effectively pause in-band cleaning | 16:31 |
| JayF | since hte heartbeat woudl be ignored | 16:31 |
| JayF | I'm sure it's not quite as clean for oob | 16:31 |
| dtantsur | Another complication: one of the possible actions is "just remove the node, it's too broken". It's possible in DEPLOYFAIL but not in DEPLOYWAIT IIRC. | 16:32 |
| dtantsur | Funnily, we have these options https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/conductor.py#L505-L518 and they default to *false* | 16:35 |
| dtantsur | But I don't think they fully work. And it looks like you assume they default to true :) | 16:36 |
| JayF | poweroff in cleanfail defaults to false | 16:43 |
| JayF | whcih means your desired behavior should already be there | 16:43 |
| JayF | and if it's not, it's a regression | 16:43 |
| dtantsur | I need to check it in reality (probably next week), but it looks like we also call tear_down_cleaning, which calls power off unconditionally.. | 16:43 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: feat: add 'physical_network' and 'category' to portgroup object https://review.opendev.org/c/openstack/python-ironicclient/+/974097 | 16:44 |
| cardoe | If someone else would +2 and +W, https://review.opendev.org/c/openstack/python-ironicclient/+/970562 that's another API tweak that's in gazpacho and makes the client compatible. Trying to push these through so that rpittau can make a gazpacho release of the client that's compatible with ironic gazpacho API. | 16:47 |
| JayF | I refuse to +2W that (I'm the author) :D | 16:48 |
| cardoe | Fair but then what about... https://review.opendev.org/c/openstack/python-ironicclient/+/974097 :-D | 16:49 |
| cardoe | JayF: I don't understand why your download performance patch breaks metal3 but it definitely does. | 16:49 |
| rpittau | we don't really need to add all the features now :) | 16:49 |
| JayF | cardoe: I couldn't even figure out where to get the damn logs to troubleshoot it, so I'm mildly screwed there unless I can get rpittau or dtantsur to help :) | 16:50 |
| rpittau | JayF: which patch ? | 16:50 |
| JayF | it certainly doesn't /seem/ like something that should only impact metal3 | 16:50 |
| cardoe | rpittau: https://review.opendev.org/c/openstack/ironic/+/964502 | 16:50 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/964502 | 16:51 |
| cardoe | rpittau: just trying to marshal in the last of the API tweaks to make everything compatible. | 16:51 |
| rpittau | waiting for the patch to spit new logs | 16:52 |
| rpittau | cardoe: ack | 16:52 |
| rpittau | cardoe: are the API related changes already released? | 16:53 |
| cardoe | Yes. | 16:53 |
| rpittau | ok | 16:53 |
| cardoe | This is just catching the client up. | 16:53 |
| rpittau | I blocked the release again | 16:54 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/970562 and https://review.opendev.org/c/openstack/python-ironicclient/+/974097 make the client compatible with what's currently in ironic master. | 16:54 |
| rpittau | ah | 16:54 |
| rpittau | then that's a nope | 16:54 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/974097 brings compatibly to what we released as bugfix/33.0 | 16:55 |
| rpittau | ok so anything in master can wait from client perspective | 16:55 |
| rpittau | otherwise we release a client that is not compatible with ironic latest release | 16:56 |
| opendevreview | Verification of a change to openstack/python-ironicclient master failed: Filter portgroups by shard https://review.opendev.org/c/openstack/python-ironicclient/+/970562 | 16:56 |
| rpittau | so https://review.opendev.org/c/openstack/python-ironicclient/+/970562 and https://review.opendev.org/c/openstack/python-ironicclient/+/974097 are nope | 16:56 |
| cardoe | ? | 16:57 |
| cardoe | gazpacho is master? | 16:57 |
| rpittau | yep | 16:57 |
| cardoe | So those are patches against master of the client? | 16:57 |
| rpittau | yeah but we don't need to wait for them for the release :) | 16:57 |
| cardoe | okay. I was just trying to get everything on equal level | 16:58 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/974097 is what's in bugfix/33.0 | 16:58 |
| rpittau | which is ok, but we can't release on client something that is not released on server | 16:58 |
| cardoe | Well there's already a few patches in client master for APIs that aren't released. | 16:59 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/955102 for example from December is in client master and that's only in ironic master | 17:00 |
| rpittau | then we wait for the client release | 17:00 |
| rpittau | I will do a bugfix anyway next week, or even end of this week | 17:00 |
| cardoe | The client code will omit things that are too new anyway. | 17:01 |
| rpittau | uhm the instance_name is in 33.0 | 17:01 |
| rpittau | so that client patch is legit | 17:01 |
| cardoe | oh you're right that was 1.04 | 17:02 |
| cardoe | er 1.104 | 17:02 |
| cardoe | so then yes https://review.opendev.org/c/openstack/python-ironicclient/+/974097 adds 1.102 and 1.103 support which is in 33.0 | 17:02 |
| rpittau | yeah | 17:02 |
| rpittau | I checked the list before submitting the release, I didn't spot anything weird | 17:02 |
| rpittau | ok | 17:02 |
| rpittau | so we can go ahead with the release after that | 17:03 |
| rpittau | I will block the others | 17:03 |
| cardoe | sounds good | 17:04 |
| cardoe | I'll work with janders to rebase his node health... I think we'll get him to renumber it to 1.109. And take cid's https://review.opendev.org/c/openstack/ironic/+/973016 as 1.108 since that's about to pass tests and not have any conflicts. | 17:05 |
| cardoe | I've gone through all the open Ironic changes and that's the extent of the API changes. | 17:09 |
| cardoe | At least those that seem ready for 2026.1 | 17:09 |
| cardoe | So rough line in the sand for 2026.1 will be 1.109 | 17:09 |
| rpittau | cardoe: sounds about right, I will review the release mapping before the bugfix release anyway, we can leave the final changes for the final G release | 17:12 |
| JayF | cardoe: I'm trying a new thing. Feeding claude-code the failure logs from that image test | 17:13 |
| rpittau | JayF: have you checked the ironic logs ? | 17:15 |
| rpittau | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e70/openstack/e704ed6f05c74fa1a620279f09023fe2/controller/before_pivoting/ironic.log | 17:15 |
| rpittau | HttpImageService.download() got an unexpected keyword argument 'checksum': TypeError: | 17:15 |
| JayF | oh, perfect | 17:16 |
| JayF | thank you | 17:16 |
| JayF | I can fix it! I have the technology! :D | 17:17 |
| JayF | claude came to the same conclusion too, fwiw | 17:17 |
| JayF | download logs -> point claude at them might not be the worst CI troubleshooting pattern | 17:17 |
| rpittau | I'm glad I'm as good as Claude :D | 17:17 |
| JayF | well, claude isn't busy doing 4000000 other things :) | 17:18 |
| rpittau | lol | 17:18 |
| JayF | if I had rpittau as a executable on my CLI to think for me, I'd be SUPER productive ;) | 17:18 |
| rpittau | thanks, too kind :) | 17:18 |
| cardoe | sudo rpittau is probably safer than sudo claude too | 17:51 |
| cardoe | Alright https://review.opendev.org/c/openstack/python-ironicclient/+/974097 passed tests and when that gets +W'd that'll unblock the client releases. | 17:53 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2025.2: fix redfish inspect system product name https://review.opendev.org/c/openstack/ironic/+/971487 | 18:08 |
| opendevreview | cid proposed openstack/ironic master: Type annotation guide https://review.opendev.org/c/openstack/ironic/+/974224 | 18:09 |
| * dtantsur welcomes rpittau-as-a-service | 20:33 | |
| dtantsur | Folks, I'll be out for this week, back on Monday (likely later than usual) | 20:33 |
| opendevreview | Julia Kreger proposed openstack/networking-baremetal master: WIP: l2vni baremetal mech driver https://review.opendev.org/c/openstack/networking-baremetal/+/973889 | 20:55 |
| opendevreview | Steve Baker proposed openstack/ironic master: WIP autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/973187 | 20:58 |
| opendevreview | Steve Baker proposed openstack/ironic master: Add an autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/973187 | 22:30 |
| opendevreview | Steve Baker proposed openstack/ironic master: Replace ironic-tempest-bios-ipmi-direct with -autodetect https://review.opendev.org/c/openstack/ironic/+/974260 | 22:30 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Improve image download performance https://review.opendev.org/c/openstack/ironic/+/964502 | 22:48 |
| opendevreview | Steve Baker proposed openstack/ironic master: Document autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/974262 | 23:01 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!