Monday, 2026-08-24

opendevreviewMerged openstack/ironic master: Start Xvfb and pass running display to x11vnc  https://review.opendev.org/c/openstack/ironic/+/100127404:28
opendevreviewMerged openstack/ironic master: Don't wrap a console container error twice  https://review.opendev.org/c/openstack/ironic/+/100170304:31
opendevreviewTakashi Kajinami proposed openstack/ironic master: Clean up deprecated send_sensor_data options  https://review.opendev.org/c/openstack/ironic/+/100201206:11
opendevreviewDmitry Tantsur proposed openstack/ironic bugfix/38.0: Redfish: retry transient 409 conflict on power-on  https://review.opendev.org/c/openstack/ironic/+/100204510:49
opendevreviewDmitry Tantsur proposed openstack/ironic bugfix/37.0: Redfish: retry transient 409 conflict on power-on  https://review.opendev.org/c/openstack/ironic/+/100204610:50
opendevreviewMerged openstack/ironic master: Add image_server_auth_hosts to restrict credential scope  https://review.opendev.org/c/openstack/ironic/+/99974410:51
opendevreviewMerged openstack/ironic bugfix/38.0: Fix periodic task processing a node via the wrong interface  https://review.opendev.org/c/openstack/ironic/+/100132810:51
opendevreviewMerged openstack/ironic bugfix/37.0: Fix periodic task processing a node via the wrong interface  https://review.opendev.org/c/openstack/ironic/+/100132910:51
opendevreviewMerged openstack/ironic stable/2026.1: Fix periodic task processing a node via the wrong interface  https://review.opendev.org/c/openstack/ironic/+/100132710:58
opendevreviewMerged openstack/ironic-specs master: Add spec for sonic and nvue driver replacements.  https://review.opendev.org/c/openstack/ironic-specs/+/99836311:07
iurygregorygood morning ironic, I'm back  o/11:08
opendevreviewMerged openstack/virtualbmc master: Drop unused test dependencies  https://review.opendev.org/c/openstack/virtualbmc/+/99785611:37
opendevreviewMerged openstack/ironic master: devstack: Fix multinode CI failures with unreachable API endpoint  https://review.opendev.org/c/openstack/ironic/+/100157511:51
opendevreviewMerged openstack/ironic stable/2026.1: Redfish: retry transient 409 conflict on power-on  https://review.opendev.org/c/openstack/ironic/+/100076711:51
opendevreviewMerged openstack/sushy-tools master: Remove note about old pip's behavior  https://review.opendev.org/c/openstack/sushy-tools/+/99818712:15
opendevreviewMerged openstack/sushy-tools master: Add openstack-python3-next-jobs  https://review.opendev.org/c/openstack/sushy-tools/+/99818612:16
stephenfino/ We've seen an uptick in failures in the ironic functional job for SDK recently. I don't believe anything has changed in SDK, so I think the issue may lie with Ironic12:17
stephenfinThe failures all look like this one https://zuul.opendev.org/t/openstack/build/22caafe9284545f4a34279d9a1c7c26412:18
dtantsurstephenfin: could put it on LP so that it's not lost in IRC?12:20
dtantsurlooks like something became racy..12:20
stephenfinWhile waiting for an allocation to be created, we get an error `no available nodes match the resource class baremetal-XXX`12:20
stephenfinsure thing12:20
dtantsurit looks like a race between two allocation tests, hmm12:23
dtantsurstephenfin: are SDK tests running sequentially or in parallel? has it changed?12:23
stephenfinNothing has changed in either the SDK implement or test nor in the test tooling (stestr, testools etc.) that I'm aware of12:24
stephenfiniirc, we execute test classes in parallel and the order of those is random, but individual test methods in a test class are sequential12:25
stephenfinhttps://bugs.launchpad.net/ironic/+bug/2164899 bug report here. I've included info as much as I could without digging into ironic itself12:35
stephenfin*as much info12:35
dtantsurThanks! I'm a bit worried about seeing two references to resource class baremetal-42612:35
dtantsurI wonder if we unironically (!) have a random number clash12:35
dtantsurstephenfin: dunno how reproducible the problem is, https://review.opendev.org/c/openstack/openstacksdk/+/1002079 may help highlighting it at least12:43
opendevreviewMerged openstack/ironic master: Switch ironic-standalone-redfish to autodetect deploy  https://review.opendev.org/c/openstack/ironic/+/99791012:46
stephenfinIt's no harm, though I will emphasise that none of this code has changed in SDK in some time and the CI was consistently green for months before this started last month'ish12:46
opendevreviewEsther Domfeh proposed openstack/bifrost master: Document enhanced node history CLI usage  https://review.opendev.org/c/openstack/bifrost/+/100208312:50
opendevreviewMerged openstack/networking-baremetal master: ruff: Fix outdated target-version  https://review.opendev.org/c/openstack/networking-baremetal/+/100146413:14
opendevreviewEsther Domfeh proposed openstack/bifrost master: Document enhanced node history CLI usage  https://review.opendev.org/c/openstack/bifrost/+/100208313:29
TheJuliao/ I'm going to be out again today. would be good to remind folks about the ptg etherpad.13:31
JayFdtantsur: I wonder if any of clif's nova-compute speedup patches landed14:18
JayFdtantsur: if it's about when resources become available .... timing of that may have recently changed (for the better)14:18
dtantsurJayF: I don't think these particular tests rely on nova14:24
JayFack, good to know14:24
clifJayF: not yet, got a -1 this morning I need to address14:55
Mahnoorno meeting today?15:04
* clif shrugs15:07
dtantsurJudging by the logs, JayF volunteered to run it15:07
clifyep15:07
dtantsurusing half of his brain, mind you15:08
JayFoh, hey15:08
JayF#startmeeting ironic15:08
opendevmeetMeeting started Mon Aug 24 15:08:11 2026 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.15:08
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:08
opendevmeetThe meeting name has been set to 'ironic'15:08
dtantsurhey, half of JayF!15:08
JayFI don't recall volunteering to do this15:08
JayFbut I am always happy to do it lol15:08
clifthe logs recall ;)15:08
clifo/15:08
JayFhow dare you bring my previous statements into this ;) 15:08
Mahnoorthe other half of the brain volunteered :) 15:08
JayFAs always, we're operating under the OpenInfra Code of Conduct, be nice ;) 15:09
JayF#topic Announcements/Reminders15:09
iurygregoryo/15:09
JayF#note Please review patches hashtagged ironic-week-prio at https://tinyurl.com/ironic-weekly-prio-dash15:09
JayFAlso tag any patches you need landed15:09
JayFIt's Aug 24 -- do you know where your release is? :D15:10
JayF#note It's R-5 (5 weeks until release); the hibiscus-3 milestone. Feature freeze, client library final freeze, requirement freeze, elections start15:11
cido/15:11
JayFGet those patches reviewed and if needed into the priority queue if you want it in 2026.215:11
JayF#topic Working Group Updates15:11
JayFAnything from the AsyncIO folks?15:11
iurygregoryI was out last two weeks, medical leave, no updates15:12
JayFmake sure to give your get_better coroutine the ability to preempt as needed :)15:12
JayFSecurity Coresec team update15:12
JayFOSSA-2026-008 was errata'd15:13
JayFWe also have some public security bugs which are open with patches for review15:14
JayFthey should be hashtagged as ironic-week-prio; please review them15:14
iurygregoryack15:14
JayF#note There are no Discussion Topics in the agenda; moving on15:14
JayFBug Deputy updates15:14
JayF#topic Bug Deputy updates15:14
JayFclif: or cid, depending on what "current" and "next" mean in the agenda15:15
JayFone of you should talk :)15:15
clifthat's me15:15
cid:D 15:16
clifThere's one new ironic bug filed this morning: https://bugs.launchpad.net/ironic/+bug/2164899 functional tests in openstack sdk sporadically fail possibly due to Ironic race condition15:16
clifhttp://bugs.launchpad.net/ironic-python-agent/+bug/2163748 there's also this IPA bug filed, which seems to need more discussion15:16
JayFthat race is likely a good one to target resolving before 2026.215:16
clifSomething about using verified discard to erase SATA SSDs?15:16
JayFYeah; that is an RFE that should be discussed during RFE section15:16
clifthose are the only new bugs I know of15:17
dtantsurI've just filed another one based on a downstream report: https://bugs.launchpad.net/ironic/+bug/216491615:17
clifno fair filing during my update ;)15:17
clifthat's all I've got for bug deputy updates15:17
JayF#topic RFE Review15:17
JayF#link http://bugs.launchpad.net/ironic-python-agent/+bug/216374815:17
JayFI have concerns about inlining this as a new, potential default implementation for erase_devices.15:18
JayFAs an optional method that has to be opted into by changing steps or setting config, +215:18
JayFWhich means I'm -1 to the RFE as proposed in the bug15:18
dtantsurI seriously think that we need something like allowed_erase_methods with an *ordered* list of methods that can be tried15:19
TheJuliaI think its a decent solution to the issue, and does have decent guarding and even verification, but I can see the need as dmitry has pointed out, to allow there can be an ordered list15:19
TheJuliaunfortunately that will require a bit of a rewrite of the existing logic15:20
TheJulia(of what is in IPA today)15:20
JayFI agree, I am not thrilled with gating support for a new hardware erase method behind a mechanism RFE. 15:20
JayFbut I think it's expectation-violating to wipe security locked devices without an opt-in15:20
TheJuliaI think the issue is more so, the hardware vendor is doing the locking and we can't unlock/override via the controller15:21
TheJuliaso its really a bug15:21
TheJuliabut, nuanced15:21
TheJuliawe should consider "shred" an absolute last resort action on modern storage devices15:21
JayFI would suggest finding a way to get this code in to "unbreak" people ASAP, without flipping the model upside down, and add a topic for PTG about making "how much erase to erase" a setting in IPA15:21
opendevreviewMerged openstack/ironic bugfix/38.0: Redfish: retry transient 409 conflict on power-on  https://review.opendev.org/c/openstack/ironic/+/100204515:21
JayFeven if it means in the meantime we're adding a clunky/temporary "use_discard_erase_method" or something15:22
TheJulia++ I think that makes sense15:22
JayFSo to confirm (please speak up if you disagree); consensus is: reject RFE in it's current form. Require it to be config-gated (default: off). Add an item to PTG (and maybe comment about this on the RFE) about making a global "how should we erase disks" option.15:24
TheJuliaSGTM, https://etherpad.opendev.org/p/ironic-ptg-2027.1 will have some text shortly15:24
opendevreviewTakashi Kajinami proposed openstack/ironic master: doc: Replace remaiming reference to legacy sensor_data options  https://review.opendev.org/c/openstack/ironic/+/100214415:25
TheJuliaokay, words added to the etherpad15:27
JayF#link https://bugs.launchpad.net/ironic-python-agent/+bug/2163748/comments/215:27
JayF#topic Open Discussion15:28
JayFAny items for general discussion among the team?15:28
dtantsurso https://bugs.launchpad.net/ironic/+bug/216491615:28
JayFOH, we also forgot to get a new bug deputy down. Someone volunteer please :)15:28
dtantsurI think it's caused by one of our security fixes15:28
TheJuliaAs a broad reminder, we need to begin surfacing the ptg discussion15:28
dtantsurwhatever introduced is_source_a_path, TheJulia, you may have context15:28
JayFisap came with anaconda driver iirc15:29
MahnoorI will be out fri and mon, dont want to volunteer this week for bug deputy15:29
TheJuliayeah, "a folder has all the contents"15:29
TheJuliasince there is no singular item, its the composite15:29
TheJuliashould be easy to put extra guards in, I thought we had all our request stuffs guarded, but I'm wondering if they are somehow hitting request pooling as well.15:30
JayFSo it seems like this is only a security issue if we don't have good connect timeouts on that HEAD, yeah?15:30
dtantsurHmm. It's not great that it allows DoS15:30
JayFThis isn't that different of a mechanism to setting deploy_ramdisk or image_source to a non-existent HTTP url and hammering it15:31
dtantsurWe do, but 60 seconds is quite a lot, and the reporter claims that eventlet doesn't really comply with them (on old versions)15:31
TheJuliaI've got a family medical issue going on, so I can't dig into it today, but yeah, we should likely tune/wind it to something tighter in some cases as well.15:31
opendevreviewMerged openstack/ironic bugfix/37.0: Redfish: retry transient 409 conflict on power-on  https://review.opendev.org/c/openstack/ironic/+/100204615:31
opendevreviewMerged openstack/ironic bugfix/37.0: Power off before ejecting redfish virtual media on ramdisk cleanup  https://review.opendev.org/c/openstack/ironic/+/99925915:31
opendevreviewMerged openstack/ironic bugfix/37.0: Another attempt to fix fast-track after inspection  https://review.opendev.org/c/openstack/ironic/+/99925815:31
JayFI don't disagree it can be improved, I just think it's a big, big stretch to say it's a security issue15:32
TheJuliaI'd say "mild security"15:32
TheJuliabut also, what rights are required to trigger verify?15:32
TheJuliaunder our normal rbac model15:33
JayFthe same rights one needs to trigger deploy/cleaning/servicing15:33
JayFwhich is why I'm kinda "eh" on if it's actually a security issue15:33
TheJuliayeah, in our normal model, its elevated15:33
TheJuliathe issue is noauth and basic is functionally rbacless and that is the base problem15:33
dtantsurI thought that deploy is pretty low privileges15:33
dtantsur(And it affects higher level orchestrators like metal3)15:34
TheJuliathat sort of opens that door, but we can't treat everything as "urgent security", because if everything is, nothing is.15:34
JayFbeing able to trigger a http call via an api call, is basically the crux of the security bit15:34
JayFand you can do "N" http calls with an actual-deploy, or a cleaning if configured in certain ways15:34
TheJuliaso, So that is where where I'm hitting at "mild"15:34
TheJuliayup15:34
JayFit's sorta the price you pay for being an API to deploy things, people can tell you to deploy BS and you listen :)15:35
JayFif we wanna harden, just gotta get a connect timeout that's wired thru so the operator can choose what fits them15:35
JayFgoing lower than 60s in the common case will break people I suspect15:35
TheJuliaor, we just have a tighter timeout for some operations15:35
TheJuliarealistically, "automatic" things, need a tighter timeout15:35
TheJulia60 seconds is "I'm deploying stuff on the moon"15:36
TheJuliaand space dust got in the way for a moment15:36
JayFdo not make that assumption15:36
JayF60 seconds might be15:36
JayFI am deploying this instance through a tiny VPN on a not-fully-provisioned new DC15:36
TheJuliaOkay, I'm deplying stuff over directway 2x dishes15:36
* TheJulia twitches15:36
TheJulia(128k down, ?8?k up15:36
TheJulia)15:37
JayFyeah, it's unfortunate that people try stuff so far outta whack, but we gotta make sure we're flexible enough for it15:37
JayFdefaults lower ++, but we gotta plumb thru a minimum setting15:37
TheJuliayup15:39
JayFAnything else for open discussion? I'll close the meeting at/around :45 if not15:39
TheJulianope15:41
cidSo, I had a question that came up during one of the reviews of the maintenance feature15:41
* cid tries to get a link15:41
cid#link https://review.opendev.org/c/openstack/ironic/+/1000752/1/ironic/common/exception.py#94215:42
JayFI would tl;dr the concern as the new maintenance API, as proposed, would not allow idempotent "re-setting" of maintenance on already-maintenanced machines15:44
cidThe change I have up changes what happens when a node is removed from maintenance state15:44
JayFIME with the kinda scripts that might set maintenance on somehting... I was wondering if we wanted to revisit the decision15:44
JayFexisting behavior is if you set maintenance when it's already set, it accepts the request and changes the reason to whatever you sent15:44
TheJuliare-setting on the old setting interface?15:44
JayF(this is also the behavior retained for old microversions)15:44
JayFre-setting via the new interface15:45
TheJuliaoh15:45
TheJuliahmm15:45
JayFhas this new limiting behavior15:45
TheJuliaI thought we were going to allow that15:45
* TheJulia is dazed and confused, and likely needs to get going anyway15:45
JayFyeah, this is behavior where I think either way is probably okay15:45
JayFand if nobody is attached to the "no duplicate maintenances", I'd be tempted to retain existing behavior15:45
JayFif the pair of (type,scope) match, don't conflict, just update reason15:46
JayFand teh node history SHOULD (as a requirement) retain the history of reasons15:46
JayFwhat we CANNOT do is add a row for each of those calls, or else something in a cron script like, once an hour or something, might insert 100+ node maintenance rows in a week if misconfigured15:46
JayFdtantsur: ^ do you have any thoughts on this?15:47
* dtantsur is trying to remember how metal3 uses maintenance..15:47
kubajjo/15:47
cid\o 15:48
dtantsurFrom API purist perspective, having a separate maintenance API does give us some wiggle room that a proper PUT/PATCH approach won't15:48
cidBasically, The code as-is, will not allow same type and scope by the same 15:48
cid*project15:48
dtantsurWe may seriously consider to microversion this15:48
JayFdtantsur: yeah, I think as written it's a better/cleaner API, but I also think that cleanliness will be annoying to operators15:48
JayFdtantsur: it 100000% will be 15:49
dtantsur(that being said, proper PATCH based API won't trigger the duplicate logic at all)15:49
JayFdtantsur: we're just talking about the behavior if someone does the similar behavior (sets an identical maintenance type twice in a row) on the new microversion15:49
dtantsurYeah, it's not terrible to reject that15:50
dtantsur(proper PATCH with E-Tags... okay, I'll stop dreaming)15:50
JayFyeah I think rejection is the nice pretty answer15:50
JayFbut I think it means a bash script that goes "should I maint this node?" needs 2x as much code aorund openstackclient15:50
JayFsince it'll have to check or handle the conflict15:51
dtantsurwhich may mean that our CLI needs --ignore-conflict / --retry-on-conflict=N15:51
dtantsur(although we use conflict for way too many things..)15:51
JayFI mean, the idea is sound15:52
dtantsurThen again, blindly overwriting existing maintenance is a sketchy thing to do, I agree with that15:52
TheJuliaAdd that to the ptg etherpad!15:52
TheJulia(meaning, the conflict stuffs)15:52
JayFyeah please don't punt the whole question to PTG15:52
JayFor esle I gotta find somethign else to keep cid busy for a while lol15:52
cid:D 15:53
TheJuliaoh, yeah, no15:53
Mahnoormy 2 cents: I would expect a failure only if it actually failed to set maintenance, if it was the same, should be okay imo15:53
JayFI like the idea about handling the UI shenanigans in the UI15:53
Mahnoorbut no strong feelings15:53
TheJuliaI'm meaning, the conflict stuff in general needs to be on the etherpad because that is "annoying"15:53
JayFMahnoor: ooh, that's an interesting third option15:53
JayFMahnoor: 1) type/scope but not reason match: fail. 2) type/scope/reason match: succeed as a noop 3) type/scope are unique: succeed as an op15:53
dtantsuryeah, we can just ignore completely identical requests15:53
JayFthat is the likely correct behavior15:54
JayFas it handles the script case15:54
JayFand keeps the API cleanly shaped15:54
JayFwith the only downside being that created_at won't get touched, but that means we'll still accurately report the first time a maint of (type, scope, reason) was created15:54
JayFso maybe not even a downside15:54
JayFhell yeah15:54
JayFand we can provide guidance to operators to make the reasons generic in documentation (e.g. don't add your own timestamp)15:55
JayFdtantsur: Mahnoor: cid: ^ agreed?15:56
dtantsur++15:56
Mahnoorsounds alright to but, can one not update the reason?  type/scope but not reason match: fail15:56
JayFthe existing behavior would clobber the old reason with the new one15:57
dtantsurI don't remember if we actually allow PATCH on reason15:57
Mahnoorsorry I meant the quote your point no. 215:57
JayFthe new API would not permit it, as you're only allowed one maintenance per type/scope combo15:57
Mahnoorah okay15:57
JayFeach maintenance in the new system is a DB row15:57
Mahnoorokay15:57
JayFcomment put into gerrit15:58
JayFwe are at time, basically, going to clsoe things out15:58
cidDoes that mean "reason" will have to be stricter or a string is still okay?15:58
cidAnyways, I think I got it.15:59
cidAny further updates can always be added to the change15:59
cid#link https://review.opendev.org/c/openstack/ironic/+/1000752/comment/80877bb0_0eafb097/15:59
JayFcid: lets work out the details in gerrit/code review or outside the meeting15:59
JayF#endmeeting15:59
opendevmeetMeeting ended Mon Aug 24 15:59:18 2026 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-08-24-15.08.html15:59
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-08-24-15.08.txt15:59
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-08-24-15.08.log.html15:59
JayFI think I've sussed out the functional test bug with claude cc: stephenfin 16:58
JayFlooks like a bug in the openstacksdk testing code directly: we set the node to power off without a wait=true, and the times the tests fail are when the sdk gets the next step run *before* the power off hits the db16:59
JayFstephenfin: I wish I knew you were working on it; I duped all your work. https://review.opendev.org/c/openstack/openstacksdk/+/1002174 is a more complete change though, and I think should win? plus you can +2 that one :D 17:17
stephenfinOh, apologies: I had assumed it would link against the bug. I associated openstacksdk with it and all 😕17:18
stephenfinbut yes, yours is definitely the more complete of the two17:19
JayFI tend to work over longer periods of time, I had a thread running (AI and mental) since the meeting lol17:19
JayFI am more of a complete 10 things in 10 time than 1 thing in 1 time :D 17:20
JayFlol17:20
*** Uggla is now known as Guest1624521:34
*** Uggla7 is now known as Uggla21:34
opendevreviewMerged openstack/ironic unmaintained/2024.1: Pin sushy-tools so we don't get into a python conflict  https://review.opendev.org/c/openstack/ironic/+/100138521:53
iurygregoryJayF, you mentioned the security patches are in the ironic-week-prio, I only see 15 patches most related to ngs and specs... I will check patches without the hashtag o/ 22:04
JayFiurygregory: look for the one adding the basic auth hostname list22:04
JayFiurygregory: and the autodetect cleaning fix22:04
JayFthose are the two at top of mind; if they are not ironic-week-prio please add it22:05
iurygregoryack!22:05
iurygregoryI've found one from cid https://review.opendev.org/c/openstack/ironic/+/1000989/ 22:05
JayFthose are just backports that need some love, the advisory was already errata'd22:06
iurygregoryseems like dtantsur has some thoughts on https://review.opendev.org/c/openstack/ironic/+/999907 , going to check the other22:07
JayFthat's a great review comment by dtantsur++22:08
JayFnice nice catch22:08
iurygregorywill look at the autodetect o/22:08
opendevreviewMerged openstack/ironic stable/2025.2: Fix socat console broken by shell-quoting  https://review.opendev.org/c/openstack/ironic/+/100098523:06

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!