| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/997366 | 10:56 |
|---|---|---|
| TheJulia | good morning | 13:09 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/997366 | 13:49 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: ROCE Enabled BMaaS https://review.opendev.org/c/openstack/ironic-specs/+/990413 | 14:00 |
| dtantsur | You're really making me learn more scary sounding network stuff TheJulia :D | 14:05 |
| TheJulia | ... sorry? | 14:09 |
| dtantsur | ROCE | 14:09 |
| TheJulia | RoCE (I've heard pronounceded as RoCey or RoCe... regional dialects to add a "t" are also a thing | 14:10 |
| cardoe | It's relatively simple from an Ironic standpoint. We're just better capturing the state of the hardware and exposing the BIOS knobs in the right way rather than a random collection of flags. | 14:11 |
| dtantsur | nice! | 14:18 |
| JayF | Roak is more where I was going for pronunciation | 14:22 |
| JayF | Or rocky | 14:22 |
| TheJulia | yeah | 14:27 |
| TheJulia | I was talking to janders and it kept sounding like "rockt" | 14:27 |
| JayF | Were you listening to this while authoring 990413? https://www.youtube.com/watch?v=ob8TNqQw2hY | 14:30 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix BIOS firmware update failure detection on Dell iDRAC https://review.opendev.org/c/openstack/ironic/+/997366 | 14:31 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Fix IPA service not starting on CentOS 10 (systemd 257) https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/995589 | 15:12 |
| dtantsur | TheJulia: not sure if you missed the notifications: https://review.opendev.org/c/openstack/networking-generic-switch/+/981418 cannot be merged any more :( | 15:27 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997310 | 15:31 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Maintain driver dependencies as optional dependencies https://review.opendev.org/c/openstack/ironic/+/996882 | 15:37 |
| TheJulia | dtantsur: ugh, joy I knew that one was going to be awful | 15:39 |
| dtantsur | yeah, quite a bit of technical debt to pay there.. | 15:39 |
| TheJulia | it is what it is | 15:39 |
| * dtantsur nods | 15:39 | |
| cardoe | dtantsur: so don't take this the wrong way... but the terrifying network thing is your spec. | 15:54 |
| cardoe | I feel like we need a white board session. | 15:56 |
| dtantsur | LOL, I know, it terrifies myself | 15:56 |
| JayF | if we barely understand it, a deploying operator has no chance :) so we gotta find and resolve the decoder rings :D | 15:56 |
| dtantsur | I mean, yeah, this is why I'm trying to have a properly designed API now | 15:57 |
| dtantsur | guessing port.extra fields is much less fun | 15:57 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997310 | 15:57 |
| dtantsur | (in the ideal world, somebody much smarter and more knowledgeable would do this spec, but... reality) | 15:58 |
| JayF | I'll take driver_info{'what_the_hell_is': 'this'} for 200. | 15:58 |
| JayF | dtantsur: we are those people. we feel like we aren't because we understand enough to know how hard it is :D | 15:58 |
| JayF | and if I tell you that you are capable, you will continue to do it so I never will have to ;) | 15:58 |
| cardoe | Well it's the same thing I've been trying to discuss with Neutron folks. | 15:58 |
| cardoe | We adopt a quick spec for OVN only. | 15:59 |
| cardoe | But then there's a handful of bugs and asks about how to do it more extensively | 15:59 |
| cardoe | And whoopsie we didn't think of all that. | 15:59 |
| dtantsur | whoopsie! | 15:59 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997310 | 15:59 |
| JayF | we have more than our own share of whoopsies | 16:00 |
| * JayF looks at the security bug he's about to open | 16:00 | |
| cardoe | No doubt. hindsight is 20/20. | 16:00 |
| TheJulia | yup | 16:02 |
| JayF | https://bugs.launchpad.net/ironic-python-agent/+bug/2160143 is now open | 16:02 |
| dtantsur | \o/ | 16:05 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997310 | 16:19 |
| JayF | dtantsur: ^ sorry, had to nullify your +2, I used Gentoo syntax (Closes: not Closes-Bug:) on the bug :( | 16:20 |
| dtantsur | I'm bumping my review stats really well today :D | 16:23 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-specs master: WIP new API for standalone networking (+ EVPN L2VNI) https://review.opendev.org/c/openstack/ironic-specs/+/995954 | 16:26 |
| opendevreview | Merged openstack/ironic-python-agent-builder master: Uninstall unnecessary packages to reduce ramdisk size https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/993050 | 16:51 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: Backlog: NVMe-oF boot/data volumes https://review.opendev.org/c/openstack/ironic-specs/+/990532 | 17:26 |
| cardoe | Well if ya want you can read the completely rewritten spec... https://review.opendev.org/c/openstack/ironic-specs/+/984596 | 18:01 |
| opendevreview | Merged openstack/ironic-python-agent master: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997310 | 18:29 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent bugfix/11.6: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997446 | 19:32 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent stable/2025.2: Fix: Remove legacy agent_token fallback code https://review.opendev.org/c/openstack/ironic-python-agent/+/997449 | 19:50 |
| keekz | i'm hitting a weird issue with 1 of our environments when attempting to delete a baremetal node: https://gist.github.com/nicholaskuechler/6ce63fd5e773eb60212d85b9ff588075 - for the user and in the logs there are errors and a trace. i can re-add the node and delete again and get the same errors. has anyone else encountered this? | 19:58 |
| TheJulia | .... Uhh... hmmm | 20:49 |
| TheJulia | keekz: do you have notifications turned on? | 20:50 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/node.py#L3371-L3372 | 20:50 |
| keekz | yes but we don't have a good way to read notifications :( | 20:51 |
| keekz | why do you ask, should i dump all the messages and find the one for this delete? | 20:53 |
| keekz | actually you might have led me towards the answer. i think our rabbit is 100% full | 20:56 |
| TheJulia | so yeah heh | 20:57 |
| TheJulia | okay | 20:57 |
| TheJulia | its a straight forward bug fix... I think | 20:57 |
| TheJulia | I suspect | 20:57 |
| TheJulia | but yeah, your officially the only operator I know of really using notifications now | 20:57 |
| JayF | <-- | 20:57 |
| TheJulia | oh, okay | 20:57 |
| * TheJulia will go fix | 20:57 | |
| JayF | notifications are in heavy use | 20:57 |
| TheJulia | or at least try to | 20:57 |
| keekz | @jayf what's that look like for you, where do you send the notifications, what tools do you use to read them, etc? | 20:58 |
| JayF | keekz: I don't have that level of operational detail. It goes out over kafka into [black hole of operational magic] | 20:59 |
| keekz | i've hit weird issues before when the rabbit gets full, but we had previously set up ttls to purge messages older than like 3 days. something is funky in this environment though as it's using queues/exchanges we aren't ttl'ing. not sure how to share screenshots on old timey irc | 20:59 |
| keekz | i have around 10 or so queues named like 'ironic-neutron-agent-member-manager-pool-55c5b75a-60f4-4de5-89cf-1c045dca69bf' each with over 500k messages. i haven't seen these queues grow so large before | 21:01 |
| TheJulia | didn't you also have some issues with like retires being turned way up and timer counts also bein gturned way up? | 21:04 |
| TheJulia | Those should just be heartbeats afaik | 21:10 |
| keekz | i don't think we're doing anything special with retries / timer counts.. just the defaults afaik | 21:10 |
| TheJulia | keekz: what version of ironic is this? | 21:14 |
| keekz | 2026.1 with some recent cherrypicks | 21:15 |
| keekz | i call it the cardoe special | 21:15 |
| TheJulia | its a race condition it looks like | 21:15 |
| * cardoe looks up from neutron | 21:15 | |
| TheJulia | so, claude thinks something different a than me right now | 21:16 |
| cardoe | https://github.com/rackerlabs/ironic/tree/understack/2026.1 is the literal code | 21:16 |
| TheJulia | which is actually more in line with what your errors indicate as well | 21:16 |
| cardoe | It's stable/2026.1 with the runbooks v2 backported | 21:17 |
| TheJulia | the tl;dr is sort of along the lines the db's record deletion is immediate in your case where as its likely not in most other cases so things sort of go sideways later on | 21:18 |
| TheJulia | likely just super spiffy database | 21:19 |
| JayF | or rabbitmq slower than molasses? | 21:19 |
| TheJulia | eh, That likely doesn't play in here | 21:19 |
| TheJulia | the issue is the db has already committed the delete commit and then the task goes to unwind the task again which is wrong | 21:20 |
| JayF | oh that's a weird race | 21:20 |
| TheJulia | but it woudl work if that row is still lingering for a little bit | 21:20 |
| JayF | the DB holds the reservation | 21:20 |
| TheJulia | yeah, but destroy nukes the row | 21:20 |
| JayF | but deleting the row also deletes the reservation | 21:20 |
| TheJulia | but that is not always the fastest thing on DBs | 21:21 |
| TheJulia | yup | 21:21 |
| TheJulia | its literally a one-liner but I have claude making a unit test | 21:21 |
| TheJulia | oh, its worse of sorts | 21:24 |
| TheJulia | it the record still lingers for a little while and finds the lock record it seems | 21:24 |
| TheJulia | at least, thats its write-up | 21:24 |
| TheJulia | we've seen something like this before, I just don't remember how exaclty it presented | 21:26 |
| TheJulia | also, read replicas | 21:31 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix destroy_node race condition https://review.opendev.org/c/openstack/ironic/+/997458 | 21:32 |
| TheJulia | they will lag and still show a lock | 21:32 |
| TheJulia | so... yeah, this can happen | 21:32 |
| TheJulia | keekz: cardoe: There you go | 21:32 |
| cardoe | TheJulia: remember when you came to my Ted Talk about a TaskManager object having a None node and that being a really weird case and I'm trying to teach the type checking about it? | 21:33 |
| TheJulia | Aye, and I'm sure that fix is making you twitch | 21:34 |
| TheJulia | But it does short circuit the attempt to cleanup | 21:34 |
| keekz | thanks @thejulia! | 21:35 |
| cardoe | TheJulia: not really making me twitch but just highlighting the case where I think I'm right | 21:37 |
| TheJulia | We're going to need an asylumn for ironic contributors, aren't we?/ | 21:38 |
| cardoe | Let's just say Claude ran away and is sitting in the corner rocking back and forth crying. | 21:38 |
| TheJulia | Clearly claude forgot it's self hugging jacket! | 21:39 |
| cardoe | The reason I haven't make the patch yet is cause it's blown its 1M context window and threw itself into multiple compacting loops and forgot what it was doing. | 21:39 |
| cardoe | I think the right fix is that we don't actually grab the node in the __init__ of TaskManager | 21:39 |
| cardoe | We've provided a contextmanager __enter__ which just returns self. I think we move that logic into __enter__ | 21:40 |
| TheJulia | ++ | 21:40 |
| TheJulia | I think that makes a lot of sense actually | 21:40 |
| TheJulia | we might need to re-write some tests | 21:41 |
| TheJulia | but yeah | 21:41 |
| cardoe | Yeah. That's where it got bogged down and lost. | 21:41 |
| TheJulia | techincally if not already loaded acquire could populate it | 21:41 |
| TheJulia | that wouldn't really require test changes... | 21:41 |
| TheJulia | and sort of follows the same pattern | 21:41 |
| cardoe | acquire just returns self | 21:41 |
| cardoe | But yes | 21:42 |
| TheJulia | i thought it executes | 21:42 |
| cardoe | hrm maybe | 21:42 |
| cardoe | too much neutron in context right now | 21:42 |
| cardoe | I'm lost in oslo.policy hell | 21:43 |
| TheJulia | ugh | 21:43 |
| TheJulia | May I recommend a Tuacca Alexander as a soothing beverage to address all pains of RBAC | 21:43 |
| TheJulia | Sorry, Tuaca Alexander, only one C in Tuaca | 21:45 |
| JayF | it's like if jameison was vanilla | 21:45 |
| TheJulia | blending is also acceptable | 21:45 |
| TheJulia | +++ | 21:45 |
| JayF | (that was one of my best friends' drinks when I drank decades ago) | 21:45 |
| cardoe | Some objects don't have a project_id but users need to be able to read them. But the migration from tenant_id to project_id seems to have a band aid to always attempt to get project_id into the policy check. | 21:46 |
| JayF | From an Ironic POV, those objects need to be made project-aware | 21:46 |
| JayF | like runbooks are | 21:46 |
| JayF | to be seen by someone with project scope | 21:46 |
| JayF | unless you wanna start handing out system scoped reader roles like candy | 21:46 |
| TheJulia | I think he is talking of neutron, fwiw | 21:47 |
| TheJulia | We may want to see if claude can bring him a nice jacket | 21:47 |
| JayF | oooh | 21:47 |
| JayF | Neutron, same as the old tron | 21:47 |
| TheJulia | JayF wins | 21:47 |
| cardoe | Yeah the issue is that you cannot have both system scope and project scope | 21:49 |
| cardoe | The failure is "openstack router create --flavor my-flavor my-router" | 21:49 |
| cardoe | Cause a regular user can no longer lookup my-flavor | 21:50 |
| JayF | you should be making noise on the list about some of this | 21:50 |
| JayF | make sure to tag [rbac] so folks like g maan will see it can can help drive improvement | 21:50 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add two-step deferred actions cleanup DB methods https://review.opendev.org/c/openstack/ironic/+/994948 | 21:55 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add config options, reschedule exception, and action creation helper https://review.opendev.org/c/openstack/ironic/+/989225 | 21:55 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add deferred action launcher periodic and executor https://review.opendev.org/c/openstack/ironic/+/989226 | 21:55 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add deferred action cleanup periodic and stuck recovery https://review.opendev.org/c/openstack/ironic/+/989227 | 21:55 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add notifications for deferred action terminal states https://review.opendev.org/c/openstack/ironic/+/989228 | 21:55 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add metrics gauges and enhanced logging for deferred actions https://review.opendev.org/c/openstack/ironic/+/989229 | 21:55 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add upgrade check for deferred action backlog https://review.opendev.org/c/openstack/ironic/+/989230 | 21:56 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Decompose redfish firmware stability validation into deferred actions https://review.opendev.org/c/openstack/ironic/+/989231 | 21:56 |
| opendevreview | Merged openstack/ironic master: Drop unused bandit https://review.opendev.org/c/openstack/ironic/+/996883 | 22:21 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add TLS hardening for kickstart configdrive and inspection rule connections https://review.opendev.org/c/openstack/ironic/+/988804 | 23:32 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!