| stevebaker[m] | cool thanks. I'll dig into it | 00:00 |
|---|---|---|
| *** darmach4 is now known as darmach | 02:07 | |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 09:45 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 09:49 |
| dtantsur | I remember we discussed allowing "abort" on DEPLOYWAIT, have there been any objections? https://bugs.launchpad.net/ironic/+bug/2138310 | 13:50 |
| TheJulia | dtantsur: I think the consensus was "just do it, it makes sense the more we think about it", but a lack of time to cycle back | 14:28 |
| TheJulia | I remember we had a chorus teeth gnashing, but that quickly ceased when the whole idea sunk in | 14:29 |
| dtantsur | I can probably unleash claude on it. I'm quite annoyed by limited ways of getting out of states. | 14:42 |
| * dtantsur unleashes claude and gets back to other business | 14:44 | |
| TheJulia | so, I talked to nobodycam recently and he basically expressed furstration that we don't have a force state reset | 14:48 |
| dtantsur | force state reset is such a thing.. once you get into details, it all falls apart | 14:52 |
| dtantsur | like the resistence that https://bugs.launchpad.net/ironic/+bug/2133499 saw | 14:52 |
| * dtantsur looking for food, brb | 14:53 | |
| TheJulia | Eh, I think the pushback there was more because it seems logic/workflow with similar effects, more than anything else. Not... "My machine got bricked in this weird state, its fixed now" recovery | 15:00 |
| JayF | My biggest concern is if we get a reset state like Nova has is that it will be the hammer and operators will see everything as a nail. | 15:05 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Fix firmware update failure when task remains in RUNNING state https://review.opendev.org/c/openstack/ironic/+/970998 | 15:18 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Make post-firmware-update reboot conditional on component https://review.opendev.org/c/openstack/ironic/+/966344 | 15:20 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Fix firmware update failure when task remains in RUNNING state https://review.opendev.org/c/openstack/ironic/+/970998 | 15:21 |
| TheJulia | I wonder if we could "reset to prior or forward stable state" as options to force movement *instead* of just writing a blanket reset check | 15:28 |
| TheJulia | Makes it harder to use as a giant hammer | 15:29 |
| JayF | Or I was thinking for say reset to a state like manageable | 15:41 |
| JayF | Maybe a choice between that or active? | 15:41 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix firmware update failure when task remains in RUNNING state https://review.opendev.org/c/openstack/ironic/+/970998 | 15:53 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix BIOS firmware not flashing when task completes before first poll https://review.opendev.org/c/openstack/ironic/+/970999 | 15:53 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix BMC resource validation to catch all sushy exceptions https://review.opendev.org/c/openstack/ironic/+/971000 | 15:53 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 15:53 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add retry logic for boot device changes during POST https://review.opendev.org/c/openstack/ironic/+/971150 | 15:55 |
| dtantsur | TheJulia: I quite literally suggested "reset to enroll" in that RFE :) | 16:03 |
| dtantsur | from there, it's two small jumps to "active" | 16:03 |
| TheJulia | I'm not sure folks are even happy with two small jumps, tbh, but nobodycam is sort of in an edge case of trying to make the hardware better as well | 16:04 |
| dtantsur | I've wanted skip-step transitions for a long time, that would make it one :) | 16:05 |
| dtantsur | Without all this, I have to write instructions for production usage that includes wiping the Metal3 database to unwedge a node, sooo... it's hard to make it worse | 16:06 |
| TheJulia | yeah, that is not ideal. | 16:06 |
| dtantsur | if you want to get horrified, my current case is something like: deploywait -> (abort) -> deployfail -> (set maintenance) -> (delete) -> enroll -> (manage) -> manageable -> (adopt) -> active | 16:09 |
| dtantsur | but this is an odd case caused by OpenShift peculiarities. I don't necessarily want to implement $THAT in ironic. | 16:09 |
| TheJulia | Taking a step back, why are we getting wedged in deploywait? | 16:12 |
| dtantsur | OpenShift (or really rather ACM) peculiarities :) The actual deployment is handled by another component in this case. If that component reports success and Ironic does not, you get in the situation where a node is deployed and is a part of the cluster, while Ironic is waiting for IPA to come back. | 16:13 |
| dtantsur | What I do want to have is a way out for hopelessly broken nodes or the process itself is hopelessly wedged (image download taking ages or something) | 16:13 |
| JayF | Shouldn't the success signal to those other components be the success in ironic? | 16:18 |
| JayF | It sounds like it would be impossible for that problem to happen unless you're trying to play around with having multiple sources of Truth | 16:18 |
| dtantsur | JayF: oh, well. I'm not drunk enough to have these discussion, but tl;dr for a long time Ironic and that other thing have been developing entirely independently | 16:18 |
| TheJulia | some complicated history there | 16:18 |
| dtantsur | and for the past few years we're trying to find a way to integrate them | 16:18 |
| dtantsur | yeah | 16:18 |
| JayF | This honestly is a case of the exact thing that I was worried about this tool being used for. Instead of fixing a base architectural issue, we're going to add a workaround to make it so that you don't have to | 16:19 |
| TheJulia | I encourage the beverage and not discuss it at this point | 16:19 |
| dtantsur | heh | 16:19 |
| TheJulia | We can discuss another day when the brain is happier | 16:19 |
| dtantsur | JayF: I very explicitly said it's not the main use case for this tool | 16:19 |
| dtantsur | unless you have ideas how to make hardware never break.. | 16:19 |
| JayF | Trust me, I work with people who know exactly how to make that happen, leading to all failures being caused by ironic /s /s 😂 | 16:20 |
| dtantsur | heh | 16:20 |
| JayF | dtantsur: I have worked places where the presence of the Nova reset state tool was explicitly used to kick architectural issues with similar downsides down the road indefinitely | 16:20 |
| JayF | So while I know that's not why you're building it now, I can't help but wonder if it already existed. Would someone have just used it and not worried about the larger problem | 16:21 |
| dtantsur | I'm literally in a conversation with a person right now, trying to find a way to fix this split-brain problem. But it won't help with "hardware went kaboom" issue, nor will it help customers who are already in a weird state now. | 16:21 |
| dtantsur | Oh yeah. You know that Kubernetes has forced deletion of each and any resource? Imagine how much headache THAT causes for us? :) | 16:22 |
| dtantsur | "Cleaning is stuck for longer than I want to wait... force delete! Oh, why do things behave weirdly still? Why is something still happening to my machine??" | 16:22 |
| dtantsur | Which is to say, in my world the option already exists, it just breaks ironic/metal3 relationship each time and causes bugs and escalations for me | 16:23 |
| JayF | your story keeps parallelling mine so hard | 16:23 |
| JayF | except not k8s | 16:23 |
| dtantsur | heh, I'm not surprised | 16:23 |
| dtantsur | Side questions because I'm too lazy to look into the code: are deploy steps always abortable? or do they have an abortable flag like clean steps? Asking for claude | 16:24 |
| JayF | they need to respect abortable | 16:24 |
| JayF | if they don't, it's a bug | 16:24 |
| JayF | I could do a firmware update in deploy steps | 16:25 |
| dtantsur | Right, okay. My initial reaction was "wait, they're always abortable". And then "wait, maybe not". | 16:25 |
| TheJulia | If I'm recalling correctly, not always abortable in that we can't necessarilly stop mid-stream on an update | 16:26 |
| TheJulia | but we can abort additional I believe | 16:26 |
| * TheJulia starts building vxlan test lab | 16:26 | |
| dtantsur | I have doubts again, maybe I should just ask claude | 16:27 |
| TheJulia | claude knows our code surprisingly well | 16:27 |
| dtantsur | Oh noes, deploy steps don't have an abortable flag. | 16:27 |
| dtantsur | I will add it. But I guess I'll need to default it to True for compatibility. | 16:30 |
| dtantsur | "You're absolutely right!" (before even checking anything lol) | 16:31 |
| TheJulia | lol | 16:31 |
| JayF | dtantsur: this preference fixed the "you're absolutely right!" problem for me https://usercontent.irccloud-cdn.com/file/HP0WfLuo/image.png | 16:43 |
| JayF | I also have a preference somewhere else, for the claude.ai webapp, telling it to not personify itself | 16:43 |
| JayF | someone or something that never disagrees with you is maximally unreliable :) | 16:44 |
| dtantsur | I wonder where this configuration can be found in my case... | 16:56 |
| dtantsur | (this is a RH-managed account) | 16:56 |
| JayF | login to claude.ai, look under settings | 16:56 |
| JayF | it's an individual profile setting | 16:56 |
| dtantsur | I'm afraid our setup is much, MUCH more convoluted. I only have a google cloud login, where the thing is hosted. | 16:57 |
| JayF | oh, you're using like, API claude stuff then | 16:58 |
| JayF | not the sub | 16:58 |
| JayF | I have the $100/mo max sub | 16:58 |
| dtantsur | We have a massive corporate deployment here, I'm not sure how it even works | 16:59 |
| TheJulia | Yeah, it is... complicated | 16:59 |
| * TheJulia tries her first local multinode stack with OVN and ngs and it will surely explode | 17:01 | |
| TheJulia | oh, nice, already did | 17:01 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Allow aborting deployments in "wait call-back" state https://review.opendev.org/c/openstack/ironic/+/973279 | 17:19 |
| dtantsur | This ^^^ but I'll WIP it until I can review it with a fresh brain | 17:20 |
| dtantsur | comments welcoem though | 17:20 |
| JayF | dtantsur: there is a way to put a claude.md in your homedir with some instructions, putting a line (or similar) in that palce for you might have a similar result | 17:43 |
| dtantsur | true, yeah | 17:44 |
| JayF | I know you can tell claude.ai to remember something | 17:47 |
| JayF | and it says it does | 17:47 |
| * JayF going to check to see if it was cross-conversation | 17:47 | |
| * TheJulia wonders if I could get claude to sanity check my devstack configuration | 17:47 | |
| JayF | dtantsur: maybe just try telling claude to remember it? https://usercontent.irccloud-cdn.com/file/RyW26Lzt/image.png | 17:48 |
| dtantsur | Interesting! | 17:54 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: feat: skip invalid mac addr interfaces in redfish inspect add speed https://review.opendev.org/c/openstack/ironic/+/973283 | 18:03 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: feat: add 'vendor' and 'category' for port object https://review.opendev.org/c/openstack/python-ironicclient/+/973286 | 18:25 |
| cardoe | clif: ^ | 18:25 |
| cardoe | So how do I nudge my 2025.2 to use API 1.01? Cause it seems to want to negotiate 1.96 with OSC no matter what. Yet 1.101 is what it reports back as the maximum version | 18:35 |
| cardoe | using openstackclient 8.3.0 and I've pip installed python-ironicclient from master | 18:36 |
| JayF | python-ironicclient has a max microversion const iirc | 18:37 |
| JayF | if it's not had something pushed for 1.101, that's it | 18:37 |
| cardoe | LAST_KNOWN_API_VERSION = 104 from python-ironicclient | 18:40 |
| * TheJulia glares at neutron | 18:42 | |
| JayF | cardoe: can you curl the 1.101 endpoint you want | 18:45 |
| cardoe | yeah gonna do that. | 18:45 |
| * TheJulia senses a race condition in neutron's startup | 18:46 | |
| cardoe | Well looks like a pyc got me | 18:50 |
| cardoe | maybe | 18:51 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: feat: add 'vendor' and 'category' for port object https://review.opendev.org/c/openstack/python-ironicclient/+/973286 | 18:52 |
| cardoe | So baremetal port list --long --node <blah> shows me Category and Vendor but baremetal port show doesn't show me category and vendor | 18:53 |
| cardoe | baremetal port set --category blah <port> didn't spit an error but I don't see the value. | 18:53 |
| TheJulia | cardoe: quick question, on your setups, what is your default tenant network type? | 18:57 |
| cardoe | vxlan | 18:57 |
| TheJulia | Have you observed any reason why it refuses to fill in the available vni segments on first startup? | 18:58 |
| cardoe | Like the ranges? | 18:59 |
| TheJulia | yup | 19:00 |
| cardoe | We don't use the ranges in the config file. Purely API driven. | 19:00 |
| janders | folks, I would really appreciate reviews on https://review.opendev.org/c/openstack/ironic/+/966344 as well as the patches chained with it (5 in total). Probably 3 of these already had +2s but I had to rebase the lot. Thanks in advance! | 19:00 |
| TheJulia | hmm | 19:00 |
| TheJulia | What is super weird is if I reset my neutron-rpc-service It populates them with existing config | 19:01 |
| TheJulia | I'm guessing I need to see if i need to do deferred networking setup | 19:01 |
| * TheJulia has to find that flag again | 19:01 | |
| cardoe | yes if you reset neutron-rpc-service it should populate them | 19:04 |
| cardoe | because that's where the ml2 is running | 19:04 |
| TheJulia | Yeah, but super duper weird given its a clean/new stacking | 19:04 |
| cardoe | https://docs.openstack.org/api-ref/baremetal/#show-port-details doesn't include details about category and vendor... where's the source to that so I can update that to? | 19:06 |
| cardoe | the latest python-ironicclient advertises the maximum supported is 1.96 when 1.101 shipped for 2025.2 so it seems we're a bit out of date there. | 19:07 |
| cardoe | yeah stable/2025.2 branch has 1.96 | 19:08 |
| cardoe | So that was the issue. I pip installed from stable/2025.2 | 19:09 |
| cardoe | But clearly something I've got in https://review.opendev.org/c/openstack/python-ironicclient/+/973286 isn't right cause its not fully showing me the fields.. | 19:11 |
| TheJulia | cardoe: did you find the api-ref stuffs yet in the repo? | 19:11 |
| cardoe | yeah | 19:11 |
| TheJulia | k | 19:11 |
| cardoe | yeah something is screw-y here. | 19:16 |
| cardoe | curl-ing the port end point with API 1.101 doesn't give me category or vendor | 19:16 |
| cardoe | What's funny is the endpoint I linked above that's the List Ports by Node which doesn't document those fields actually returns them | 19:18 |
| cardoe | But the fields are documented on the Ports endpoints which don't return them | 19:18 |
| TheJulia | are you asking for details or the default view? sounds like the default view | 19:22 |
| cardoe | I'm using --debug on openstack baremetal port show <uuid> which gives me the curl command | 19:28 |
| TheJulia | what gets offered by the client version wise? | 19:29 |
| cardoe | Maxium is 1.101 and I edited my local stable/2025.2 to use 1.01 | 19:30 |
| cardoe | https://docs.openstack.org/api-ref/baremetal/#list-ports doesn't show the fields. | 19:30 |
| TheJulia | do they show when you ask for detail? | 19:30 |
| cardoe | Nope | 19:32 |
| cardoe | https://docs.openstack.org/api-ref/baremetal/#create-port I can create a port with vendor and category as documented | 19:32 |
| TheJulia | and if you show a port? | 19:32 |
| cardoe | Nope | 19:32 |
| * TheJulia blinks | 19:32 | |
| cardoe | They're invisible no matter the call | 19:32 |
| cardoe | I'm confirming that openstack baremetal port set and openstack baremetal port create work by MySQL commands | 19:33 |
| TheJulia | detail on a port show? | 19:33 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/port.py#L154-L170 | 19:35 |
| TheJulia | theres the problem | 19:35 |
| cardoe | That'll do it. | 19:37 |
| TheJulia | we need to create a bug, thats pretty annoying :( | 19:37 |
| JayF | I fed this chat to claude, I'll take a look at the bug | 19:39 |
| JayF | it's probably me or cid who wrote it | 19:39 |
| TheJulia | oh, I found out why my networks were not creating properly, looks like it was my brige mappings | 19:40 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix: port endpoints did not return vendor and category and fix docs https://review.opendev.org/c/openstack/ironic/+/973294 | 19:47 |
| cardoe | That makes sense | 19:47 |
| cardoe | JayF: https://review.opendev.org/c/openstack/ironic/+/973294 feel free to start with that | 19:48 |
| JayF | I'm already done | 19:48 |
| JayF | literally about to push | 19:48 |
| JayF | :( | 19:48 |
| cardoe | Human defeats AI! | 19:49 |
| JayF | more like we wasted half an hour | 19:49 |
| JayF | you finish yours | 19:49 |
| JayF | well not a half hour, just ten minutes | 19:49 |
| JayF | I just hate duplicate work | 19:49 |
| cardoe | I didn't realize you were working on it. I'd tabbed out when Julia linked the github link | 19:50 |
| cardoe | You can eyeball the client changes here.. https://review.opendev.org/c/openstack/python-ironicclient/+/973286 | 19:50 |
| cardoe | They should also be backported. | 19:51 |
| cardoe | And I'll make a patch to stable/2025.2 to fix the API range it claims it supports. | 19:51 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient stable/2025.2: fix: report compatibility with 2025.2 release of Ironic https://review.opendev.org/c/openstack/python-ironicclient/+/973295 | 19:54 |
| TheJulia | oooooooh ahhhh my stack is downloading ipa | 19:58 |
| opendevreview | Julia Kreger proposed openstack/ironic-prometheus-exporter master: Swap i-p-e over to use dib ipa builds. https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/973298 | 20:02 |
| cardoe | ^ retry limit? | 20:57 |
| -opendevstatus- NOTICE: An update to one of our base jobs roles broke another base job role. This update has been reverted and jobs should be working again. | 20:58 | |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: devstack: Ignore error with file existing on restack https://review.opendev.org/c/openstack/networking-generic-switch/+/973310 | 21:09 |
| cardoe | So with ironic-Prometheus-exporter can I still use rabbitmq notifications? I recall someone wanted to have multiple listeners in the past. | 21:12 |
| TheJulia | so, i-p-e is basically modeled on tapping the data stream to a prometheus format for each conductor, I just don't remember in regards to rabbitmq notifications | 21:30 |
| TheJulia | I think we discussed deprecating the sensor data stuffs out as well, but its all a blur | 21:30 |
| cardoe | Just sharing something stupid I had Claude do from my phone. We run Ironic in k8s so I said make me a CRD and a little side car to handle inspection rules and providing them to Ironic in a way it can consume them... https://github.com/openstack/ironic/compare/master...cardoe:ironic:claude/inspection-rules-crd-YQy1D | 21:44 |
| TheJulia | cardoe: I've got a partially stood up VM lab environment to do the needful with NGS, so I guess expect me to start focusing on those again soon, if we want to move the mech driver over to networking-baremetal soon, it might be good time wise. | 22:09 |
| cardoe | If you can give me some hints how to duplicate your setup i would be interested as well. I assume you are virtualizing a switch? | 22:14 |
| TheJulia | two VMs, second network interface just attached to a shared vm bridge | 22:17 |
| TheJulia | as an isolated network | 22:17 |
| TheJulia | My second VM is not the happiest, but NGS should still be able to give it orders for attachments | 22:18 |
| TheJulia | The base jobs will need to get swapped out to be more OVN-ey too, separate item ahead | 22:18 |
| TheJulia | There is some inner logic which is VM network based which... is overkill and doesn't make sense in the ngs devstack plugin too, but a few logical steps it should be better, overall. | 22:28 |
| cardoe | hmm okay. Cause I'd be more than happy to duplicate a test setup so we can work the end to end. | 22:30 |
| TheJulia | I might try to start stubbing out job configs tomorrow | 22:31 |
| TheJulia | The job expects a br-infra tunnel between the nodes as well, which we don't really have on local VMs | 22:32 |
| cardoe | What's emulating the switch? | 22:34 |
| TheJulia | in this case, I'm going to lean towards the testing idea I hinted at in the spec along with the last n-g-s patch I posted | 22:34 |
| TheJulia | https://review.opendev.org/c/openstack/networking-generic-switch/+/972765/1/networking_generic_switch/devices/netmiko_devices/ovs.py see line 32 :) | 22:35 |
| cardoe | ah that's not a bad idea | 22:37 |
| TheJulia | That way we can focs nah, that way for my test VMs, I can just treat is as vlan and still validate behavior/flow | 22:38 |
| TheJulia | and then, switch gears | 22:38 |
| TheJulia | I'd likely start with SONiC in software, but I don't remember if sonic or the cisco nexus 9k images support software vxlan stuffs either. My brain first wants to just get NGS started then mech driver started and then independenly we can hammer out the best end shape for mech driver and or reconcillation stuffs for ports per the spec | 22:39 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Simplify ovn vtep microversion logic https://review.opendev.org/c/openstack/ironic/+/971494 | 22:48 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: feat: add 'vendor' and 'category' for port object https://review.opendev.org/c/openstack/python-ironicclient/+/973286 | 23:16 |
| cardoe | https://review.opendev.org/c/openstack/openstacksdk/+/973321 as well | 23:22 |
| cardoe | well for some reason zuul is hung on a non-voting job with https://review.opendev.org/c/openstack/ironic/+/973294 but if someone can let me know if that's correct to be backportable I'd appreciate it. otherwise I'll fix it up. | 23:24 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
