| opendevreview | fivetime proposed openstack/networking-generic-switch master: Configure link aggregations declaratively, never their member ports https://review.opendev.org/c/openstack/networking-generic-switch/+/1005277 | 02:34 |
|---|---|---|
| opendevreview | fivetime proposed openstack/networking-generic-switch master: Route trunk subport changes to the aggregate as well https://review.opendev.org/c/openstack/networking-generic-switch/+/1005278 | 02:34 |
| opendevreview | fivetime proposed openstack/networking-generic-switch master: Honor VLAN management on default VLAN restore https://review.opendev.org/c/openstack/networking-generic-switch/+/1009666 | 02:56 |
| opendevreview | fivetime proposed openstack/networking-generic-switch master: arista: Configure 802.3ad aggregate interfaces https://review.opendev.org/c/openstack/networking-generic-switch/+/1005277 | 02:57 |
| opendevreview | fivetime proposed openstack/networking-generic-switch master: Configure repeated 802.3ad links only once https://review.opendev.org/c/openstack/networking-generic-switch/+/1005278 | 02:57 |
| opendevreview | Verification of a change to openstack/ironic master failed: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009336 | 03:40 |
| opendevreview | Nuutti Hakala proposed openstack/ironic-python-agent-builder master: Disable Glean DHCP fallback by default https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/1008917 | 05:08 |
| opendevreview | Nuutti Hakala proposed openstack/ironic master: Add more JSON schema validation to allocation API https://review.opendev.org/c/openstack/ironic/+/1009700 | 06:12 |
| opendevreview | Merged openstack/ironic master: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009336 | 06:35 |
| dtantsur | Team, I'm fighting with heavy cold, will probably stay afk for the most of the day. JFYI. | 06:55 |
| rpittau | dtantsur: take care! | 07:18 |
| opendevreview | Jed Preist proposed openstack/ironic master: Retry setting boot device when told to by BMC https://review.opendev.org/c/openstack/ironic/+/1009435 | 07:58 |
| opendevreview | Verification of a change to openstack/bifrost master failed: [CI] Upgrade master from 2026.2 and add slurp job https://review.opendev.org/c/openstack/bifrost/+/1007253 | 09:42 |
| rpittau | there's something really odd happening in bifrost CI, some kind of race on the debian disk image, I need to investigate deeper | 10:02 |
| rpittau | still the bifrost CI patches need to get in ASAP to fix the upgrade jobs | 10:02 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Use debian-minimal for DIB deployment images https://review.opendev.org/c/openstack/bifrost/+/1009744 | 10:09 |
| rpittau | found it ^ | 10:09 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Use debian-minimal for DIB deployment images https://review.opendev.org/c/openstack/bifrost/+/1009744 | 10:09 |
| opendevreview | Verification of a change to openstack/bifrost stable/2026.1 failed: [stable only] CI: Fall back to HTTPS when cirros 0.5.3 is not cached https://review.opendev.org/c/openstack/bifrost/+/1008072 | 11:54 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add Dell LC job status helper for multiple JIDs https://review.opendev.org/c/openstack/ironic/+/1005656 | 12:00 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add Redfish BootProgress gate helper https://review.opendev.org/c/openstack/ironic/+/1005657 | 12:00 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Verify firmware application after reboot before resuming https://review.opendev.org/c/openstack/ironic/+/1005658 | 12:00 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Validate BMC stability only after BMC firmware updates https://review.opendev.org/c/openstack/ironic/+/1005659 | 12:00 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Keep power on firmware update failures https://review.opendev.org/c/openstack/ironic/+/1009754 | 12:00 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Use debian-minimal for DIB deployment images https://review.opendev.org/c/openstack/bifrost/+/1009744 | 12:55 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Use debian-minimal for DIB deployment images https://review.opendev.org/c/openstack/bifrost/+/1009744 | 12:56 |
| opendevreview | Merged openstack/bifrost master: [CI] Upgrade master from 2026.2 and add slurp job https://review.opendev.org/c/openstack/bifrost/+/1007253 | 13:10 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.2: Make TBN comparisons against unset variables never match https://review.opendev.org/c/openstack/ironic/+/1009780 | 14:38 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: Make TBN comparisons against unset variables never match https://review.opendev.org/c/openstack/ironic/+/1009781 | 14:39 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.2: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009782 | 14:41 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009783 | 14:43 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: Only plug ports with a tenant VIF in configure_tenant_networks https://review.opendev.org/c/openstack/ironic/+/1009783 | 14:46 |
| opendevreview | Merged openstack/ironic bugfix/37.0: Add image_server_auth_hosts to restrict credential scope https://review.opendev.org/c/openstack/ironic/+/1003579 | 15:05 |
| opendevreview | Merged openstack/ironic-prometheus-exporter stable/2026.1: Ignore non-metrics notifications in PrometheusFileDriver https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/1004769 | 15:20 |
| * TheJulia hopes today is a nice and quiet day for folks | 15:21 | |
| rpittau | TheJulia: thanks, same to you :) | 15:50 |
| rpittau | have a great weekend folks! o/ | 15:50 |
| cardoe | rpittau: you too! | 15:57 |
| TheJulia | you too rpittau | 16:07 |
| TheJulia | Its a wild cardoe! | 16:08 |
| shermanm | bit of a debugging Q based on the bugs I'd reported previously. I've found 4 nodes that are in state "available", but have an instance_id set, and no matching nova instance. Because it's in state available, I can't use `undeploy` because that's not a valid transition. | 18:07 |
| shermanm | I can just unset the instance_id, but is there any other state I should track down and make sure is cleaned up? maybe VIFs? | 18:08 |
| JayF | that should only happen if you have folks calling ironic to tear down machines outside of nova | 18:11 |
| JayF | given the order of operations | 18:12 |
| JayF | unless you're running a patch or an extremely old version | 18:12 |
| shermanm | I'm still tracing how it got there, but right now i'm putting a runbook for how to fix it given that its what i see | 18:12 |
| JayF | still active with no instance_uuid is plausible; instance_uuid while in available is *weird*. Weird enough to make me wonder if they are failing on the front end (attempted provisioning) rather than being on the far end (attempted cleaning) | 18:12 |
| JayF | does it have anything interesting in instance_info? | 18:13 |
| shermanm | yep, instance_info still populated on all of them | 18:15 |
| shermanm | so, I think what happened is that it was the same "deploy failed" bugs that i reported on the nova driver | 18:16 |
| shermanm | but somehow it got out of deploy failed after that, maybe via admin commands, but is now in available? | 18:16 |
| TheJulia | there is a slim possibility that the start of a claim occured and a power sync lock took over | 18:26 |
| TheJulia | shermanm's shop being very IPMI heavy if I'm recalling past comments correctly, means that call is also a heavier call to complete | 18:26 |
| TheJulia | so the lock gets held longer, ec | 18:27 |
| shermanm | yeah, I'm pulling logs, I should have a full sequence of api requests and conductor logs | 18:27 |
| shermanm | but switching everything to redfish is finally on our roadmap for the next few months (yay!) | 18:27 |
| TheJulia | I'd expect the instance_uuid to end up in the nova logs, fwiw | 18:43 |
| shermanm | what initially triggered the issues was nova sending a delete while ironic has the node locked, failing to retry (#2169776), but still cleaning up the nova side and deleting the ports (#2169779). The logs for that part are here, but I understand it at least. https://gist.github.com/msherman64/aac8351dd1c78ffb0643c2777ef8ed3b | 18:45 |
| shermanm | but i'm still looking for the "deploy failed" -> "something" transition | 18:45 |
| shermanm | unfortunately the next logs I have are from 4 days later, with 'node moved to cleaning from manageable' and 'node moved to available from cleaning', but no smoking gun for "deploy failed -> manageable" | 18:50 |
| shermanm | but more concretely, I'm still not sure how to clean these nodes up without editing the DB | 18:52 |
| JayF | there are API commands to unset it | 18:54 |
| JayF | I am getting bad at building them from just memory, give me a sec | 18:55 |
| JayF | openstack baremetal node unset <IDENT> --instance-info --instance-uuid # should work | 18:56 |
| JayF | then you'll have a machine with 0 instance uuid or info, in available, ready to go | 18:56 |
| JayF | if you feel uncertain it's OK, you can also reclean it: | 18:56 |
| JayF | openstack baremetal node manage <IDENT> && openstack baremetal node provide <IDENT> | 18:56 |
| shermanm | blah, I wonder if this is another edge case for us because we have automated cleaning disabled. manage && provide doesn't fix it. | 18:58 |
| JayF | turning off automated cleaning is dangerous and scary | 18:59 |
| JayF | and will lead to all *kinds* of weird broken behavior -- including making race conditions racier because you can traverse the states so quickly | 18:59 |
| shermanm | it is, but it's been essentially unusable for us unfortunately. because our users are working with aggregates with usually only 1 node, the 10-30 minutes that cleaning takes gives them a no hosts available error for that entire duration. I'm fully aware of the risks w.r.t. nodes having old state, cleaning that up, etc, but I wasn't aware that it could affect how things show up in the ironic API like the instance info and | 19:01 |
| shermanm | so on | 19:01 |
| JayF | I'd say it's more, you're operating in an untested manner at scale | 19:02 |
| JayF | and so you're more likely to land in weird bugs :) | 19:02 |
| JayF | and yeah, targetting single nodes in an nova-fronted ironic leads to some ... unpleasant side effects like what you mention | 19:02 |
| JayF | I wonder if your case would be better served by use of Ironic directly | 19:02 |
| shermanm | it's a fair point, but I suspect it would be a big lift. our current workaround has been a small patch that wipes the disks at the start of deployment, rather than at the end of teardown, and a automation that runs cleaning when a user's lease on some resources ends (which is what manages the aggregates too) | 19:05 |
| JayF | you can do that without a patch, fwiw | 19:06 |
| JayF | deploy template that includes the erase step at an early priority | 19:06 |
| shermanm | I think we actually had a convo about this some time back, IIRC that requires setting some custom config/traits to apply that template on every node. At the time it wasn't possible to change "default" templates, has that changed? | 19:07 |
| cardoe | It's be probably good to write up a high level of how you use stuff and how many things you have | 19:07 |
| cardoe | So we can incorporate that into an operator use case | 19:08 |
| cardoe | There's some PTG topics around operator experience and asks on how operators use things | 19:08 |
| JayF | ++ what cardoe said, if it's not shaped in a way you can use it, come to PTG or file a bug asking for a shape that's more useful for you | 19:08 |
| JayF | it may be it can be done, and is badly documented, or we may need a new way to do something | 19:08 |
| shermanm | that's a good point. I am aware that "operator needs cleaning disabled" is a bit of a touchy topic ;) | 19:08 |
| JayF | he basically took that roundtrip with runbooks to remove some of the trait-y downsides | 19:09 |
| JayF | shermanm: I mainly bluster loudly about it when discussing in public b/c I don't want new opers to be like "that's convienient!" and follow in the footsteps | 19:09 |
| JayF | shermanm: the toggle exists because we know some people need it :) | 19:09 |
| shermanm | :) | 19:09 |
| shermanm | fwiw, manage -> adopt -> undeploy cleaned it up | 19:12 |
| shermanm | since I couldn't run undeploy from available or manageable | 19:12 |
| shermanm | I'd need to check it on a node when the issue reoccurs, but I suspect the cause here was the above "deploy failed" bug, then an allowed transition from deploy failed -> manageable, and then transition from manageable -> available which didn't clear the instanc einfo | 19:14 |
| JayF | FWIW; disabled cleaning or not, a node being in that weird of a state is 100% a bug and something if we can track down we will fix up | 19:19 |
| shermanm | +1, I'll file it with the above info at least. | 19:20 |
| JayF | One thing that'll be a key thing: if you can assure us this happened without any known network or rabbitmq blips | 19:25 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!