Monday, 2026-05-11

opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: Switch batching from direct etcd3gw to tooz  https://review.opendev.org/c/openstack/networking-generic-switch/+/98805008:29
*** zigo_ is now known as zigo08:57
*** dmellado4717259 is now known as dmellado47172509:45
dtantsurAll jobs failing with "The specified regex doesn't match with anything"09:59
opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: Switch batching from direct etcd3gw to tooz  https://review.opendev.org/c/openstack/networking-generic-switch/+/98805011:27
opendevreviewcid proposed openstack/ironic master: Fix healthcheck intercepting all API routes  https://review.opendev.org/c/openstack/ironic/+/98810212:36
TheJuliaoh joy12:59
fricklerlook like that was only transient while tempest was preparing a new tag, should be green again with https://review.opendev.org/c/openstack/tempest/+/987912 merged13:04
opendevreviewMithun Krishnan Umesan proposed openstack/ironic master: Add TLS configuration to agent lookup response  https://review.opendev.org/c/openstack/ironic/+/98788713:37
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: create more specific update method for typing  https://review.opendev.org/c/openstack/python-ironicclient/+/98810613:49
cardoeJayF: is Karan done?13:50
dtantsurI can confirm that rechecking works13:55
JayFcardoe: he said to us in the MLH wrapup that he would finish it up in the next few days. That was weeks ago.14:00
JayFcardoe: sooooo... :| 14:01
TheJulia:(14:01
cardoeokay I'll rebase the stuff14:01
*** ykarel_ is now known as ykarel14:01
cardoeI'd like to cut a release with the typing stuff in it and runbook v2 for testing purposes14:01
cardoeand use that as feedback for me creating some work items for some folks to do follow ups for the 2026.2 if that's okay?14:02
JayFcardoe: we underutilize the lever of "file a good issue and smoeone will fix it" :) 14:06
cardoeYeah I would agree with that.14:20
* cardoe goes to google if there is a launchpad CLI.14:20
JayFThere is a pretty good LP API14:21
JayFI suggest vibe coding up a client14:21
JayFthat has worked for me in the past, atleast for pulling R/O statistics14:21
JayFdunno how much of a pain the auth dance is14:21
cardoeI know their official CLI is some Python-like REPL which I hated.14:22
cardoeBut vibe coding something is probably the best answer.14:22
cardoeThat'll make it use it more.14:22
cardoeCause I often write little markdown snippets of issues and what not and I can just pop those in.14:22
TheJuliaspeaking of launchpad, I randomly glanced at some of the really old items, and we're building up some "fixed committed" but not released items due to mismatch in repositories in the such. I've closed some out, but it would be good for us to overall give the older bugs a scrub. I also closed out some stuff that was just never going to move forward for drivers which have been removed and features since added14:22
JayFI suggest adding that as a task in the bug deputy doc14:23
TheJuliapossibly, although we may be at a point where its beyond the scope of a bug deputy doing triage14:24
opendevreviewJulia Kreger proposed openstack/ironic master: WIP: Default TLS minimum version to 1.3  https://review.opendev.org/c/openstack/ironic/+/98795314:47
* TheJulia awaits delicious CI results14:48
opendevreviewcid proposed openstack/ironic master: Fix anaconda deploy on multitenant networks  https://review.opendev.org/c/openstack/ironic/+/98812514:52
* cid Oh my god :D. did I just push a git log :man-facepalming:14:56
opendevreviewcid proposed openstack/ironic master: Fix anaconda deploy on multitenant networks  https://review.opendev.org/c/openstack/ironic/+/98812514:56
JayFcid: the best part is: I didn't even notice there was a bad patchset on that change. You outed yourself! :D 14:58
dtantsursame, I could not resist looking :-P14:58
cid:D. I have enough shame to last me the rest of the week :D. God!!!!!!14:58
JayFI kid but honestly it's like a major "green flag" in tech, especially OSS communities and infrastructure stuff, to point at your own screw ups14:59
JayFnothing worse than root causing an issue to someone having screwed up and trying to quickfix/cover it up14:59
TheJuliaheh15:00
mumesan[m]lol15:00
TheJuliaI believe it is time.15:00
JayFapparently I volunteered15:01
JayFit me15:01
JayF#startmeeting ironic15:01
opendevmeetMeeting started Mon May 11 15:01:08 2026 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
opendevmeetThe meeting name has been set to 'ironic'15:01
TheJuliao/15:01
rpittauo/15:01
JayFIt's a meeting! Hooray. As usual we operate under the OpenInfra Code of Conduct and the general attitude of "be kind, dudes" 15:01
TheJuliaJayF: I was about to go ahead and start the meeting :)15:01
JayF#topic Announcements/Reminders15:01
JayF#link https://tinyurl.com/ironic-weekly-prio-dash please review things hashtagged:ironic-week-prio -- to get your changes reviewed more quickly, hashtag them :) 15:02
dtantsuro/15:02
JayF#link https://releases.openstack.org/hibiscus/schedule.html It's R-20, do you know where your releases are? (It's OK, you don't need to know where they are, there are no deadlines)15:02
JayFI'm going to give until :05 for more folks to trickle in15:02
TheJuliaSurely the release is tucked in it's bed for a nap!15:03
clifo/15:03
mumesan[m]o/15:03
JayFOK, starting things15:04
JayF#topic Working Group Updates15:04
JayFI pruned this list down in the agenda, it seems like maybe teh Async group is the only one still relevant to the Hibiscus cycle, yeah?15:05
dtantsurI hope it is :)15:05
JayFAnything to say about it?15:06
dtantsurFor standalone networking, we're still working on the Metal3 side15:06
dtantsurNo updates for asyncio, and I'll probably pass the lead to one of my colleagues15:06
cido/15:06
JayFPlease feel free to edit the agenda if anyone wants to spin up any further working groups15:07
TheJuliaAside from async, I wonder if we'll want to treat deferred tasks similarly, but we actually need to begin to build momentum there.15:07
JayFwe probably should have the nova/ironic group there at some point15:07
TheJulia++15:07
JayFbut similarly to TheJulia we just need spare time to kickoff15:07
TheJuliayup15:07
JayF#topic Discussion Topics15:07
JayFWe have quite a few CI issues comments here.15:07
JayFAnd a lot of topics in general.15:07
TheJuliaSo, I think the glance image issue is resolved at this point15:07
JayFI think 99% of the are TheJulia-adjacent? /me hands over the podiuym15:07
TheJulialol15:07
JayFThe AKI/ARI stuff was weird, re: the glance image CI breakage15:08
JayFbut is a harsh reminder that it's a steep, steep price to embargo issues and work in quiet15:08
TheJuliaSo, there is an open question regarding the firefox PPA, I know steve has started work on trying to bring the graphical console testing job back into the running state. I know early last week, the launchpad PPA repo was unresponsive15:08
JayFwhich can lead to weird miscomms like what happened there15:08
TheJuliayup15:08
JayFTheJulia: I am generally concerned that such a job is, by it's nature, more likely to bitrot than most of our jobs... I wonder if it's reasonable to keep it nonvoting, experimental, periodic ... something to make it so it's not going to break the whole cloud if it breaks15:09
TheJuliaSkipping the glance stuff which we discussed/retrospected on last week15:09
TheJuliaSo, looking at our non-voting jobs, non-voting basically leads directly to bitrot15:10
TheJuliaover stupid stuff typically, but there is a point where we likely need to recognize we can't prefectly test everything15:10
TheJuliawhere that is, I don't know15:10
JayFI don't know how to solve this problem directly, but we have so many jobs, the matrix is so big, even a small % failure chance cascades15:10
TheJuliaWe likely need to have the human discussion of "what can be reasonably done" disjointed from the job voting level discussion15:11
JayFIt makes it feel more sisyphean than usual to work on CI15:11
TheJuliaYeah, agreed15:11
TheJuliaI haven't had a chance to look at what steve is doing exactly, cardoe did propose maybe doing a cached container image15:11
TheJuliabut we also saw quay.io break last week15:11
TheJuliaso...15:11
TheJuliadunno, as long as there is some aware and that we follow-up on the issues, we're likely in a better place15:12
TheJuliabut yeah, things like graphical are also a bit of a longer tail.15:12
TheJuliaAnyone have anything else to add before we move on ?15:12
cardoeit's just building a container each time and installing A LOT OF packages. Cause it's starting with a minimal ubuntu and installing firefox15:13
cardoeand X1115:13
TheJuliaeach download is an opportunity for job failure15:13
TheJuliaso, simplifying that would be for the best15:13
dtantsurCan we publish and reuse a base container?15:14
JayFyep, especially ones that don't use CI mirrors (I don't know if our container build does; but PPAs def. don't)15:14
TheJuliafwiw, it looks like the PPA mirror is back up, but it seems like we should just publish a container15:15
TheJuliaor a base container15:15
TheJuliastevebaker[m]: fyi the consensus seems to be ^15:15
TheJuliaonward?15:15
dtantsurI'm pondering the same thing for metal3 jobs btw15:15
dtantsurbuilding iPXE from source each time is annoying at best15:16
dtantsuryeah, onward, not going to diverge the conversation :)15:16
TheJuliait really wouldn't be a bad idea, although leveraging quay right now is why we had to take it out of voting at one point last week and then add it right back in15:16
* dtantsur nods15:16
dtantsurQuay is a pain in all parts of our team's bodies15:16
TheJuliaNext topic: During the PTG or right after it we discussed software raid and one item which was raised was software raid testing is limited, which has me thinking if we should put an explicit feature flag around sofware raid15:17
TheJuliaSpecifically: limited testing upstream, limited ability to test it, you need to accept it might not work or you might be trying to do something it can't do15:18
JayFI think any of our features which could: 1) expose additional security surface and 2) already require significant operator setup to work should be treated that way15:19
TheJuliaI mean, we could sink resources into testing it, but I think we're also heading into a state where distributions start backing off including modules because distros are more centered around end workstations with single devices and clouds, not servers with multiple devices and even then hardware raid is relatively inexpensive now15:19
JayFif they already have to do a bucketload of manual config, add "turn it on" to the list15:19
TheJuliaSo, anyone objecting to us adding a knob around software raid in general?15:19
* TheJulia waits the customary minute15:20
TheJuliaOkay, I think we've agreed then...15:20
TheJuliaOnward!15:20
TheJuliaNext topic: A quick one for sure: We're retiring virutalpdu right?15:21
* TheJulia hears crickets15:21
TheJuliacid: I think you have a patch up but the repo still has some testing someplace?15:21
cidI do !15:22
TheJuliaI think it needs to be marked in the releases stuff that its being retired, but I haven't done a retirement in a very long time15:22
cidI think it's currently blocked by reivews, if I remember correctly.15:22
TheJuliaoh, is there more than one ? because the one I found on the virtualpdu repo is blocked on tests15:22
JayFmaybe worth spinning up an etherpad with the steps and where we're at15:22
JayFso that folks can intervene to get reviews if needed?15:23
TheJulia++15:23
rpittaushould we announce that in the mailing list ?15:23
JayFjust links to the outstanding stuff pending review, and the next steps once they land15:23
TheJuliawe should!15:23
cidFollowing the project retirement guide, I think I am at a point where a liason will have to check off on the patch, then we get the retirement date to put on a follow on commit15:23
JayF!action cid to email list about virtualpdu retirement; create documentation around where things are and what is the next stsp15:23
opendevmeetJayF: Error: "action" is not a valid command.15:23
TheJuliaheh, agreed ?15:24
JayF#action cid to email list about virtualpdu retirement; create documentation around where things are and what is the next step15:24
TheJuliaheh15:24
TheJuliaokay15:24
TheJuliaAnything else on this topic?15:24
cidNot really15:24
TheJuliaFor our last topic on today's topic marathon, TLS versions! As some of you might be aware, I've been working on adding knobs for ciphers and tls versions, and I'm curious if folks have thoughts on just explicitly starting to default ironicy things to utilizing TLS 1.3 by default. Thoughts?15:25
JayFI don't mind that change; I do not want the PQC ciphers used by default.15:25
TheJuliaYeah, I'm not thinking of doing anything beyond asserting a default version at some point to 1.315:25
cardoeIs that for IPA or for everything like the BMCs?15:26
JayFThere are many folks outside of US jurisdictions worried about the introduction of net-new crypto algorithms as part of the PQC push.15:26
dtantsurWhat's the scope?15:26
cardoeCause afaik a lot of BMCs don't do 1.315:26
dtantsurIronic<->IPA traffic - sure. Ironic<->BMC rather no15:26
TheJuliaI presently have a change working its way thorugh CI which forces apache to 1.3, the downside in our model is we use apache and uwsgi (another item we should discuss at some point) front-ending it15:26
TheJuliaSo, I'm thinking for BMC access, we let it fall to system policy unless an operator wishes to configure it explicitly15:26
dtantsurapache - that's devstack-only, right? we cannot change how users configure the API?15:27
TheJuliabut frontend API wise, default wise, IPA wise, all 1.3 by default and make operators dial it back15:27
TheJuliawell, for those users, but we also have different launch models and controls all over the place15:27
TheJuliaAlternatively, my set of changes are starting to lock everything to 1.2, but since 1.3 actually seems to "just work" in devstack (like, jobs have passed when I edited apache config via sed, maybe 1.3 is okay15:28
TheJuliaagain, bmcs themselves are out of that scope, only getting the knob, not a default15:29
dtantsurbifrost may also be an okay target, although it might share its TLS settings with its vmedia server..15:29
TheJulia(in that that ecosystem moves way slower than OSes in general)15:29
JayFWe need to make sure this is about as loud of an upgrade note as possible, since this may need infra changes on the operator side as well. 15:29
JayFEven if we exclude BMCs15:29
TheJuliaYeah, there are already some loud notes in the release notes already in my patch series, I'd do a 1.3 change as a standalone change where I update defaults and unit tests15:30
TheJuliaSeems like we're arriving at "reasonable" ?15:30
TheJuliaAnyone seeking to object?15:30
JayFmaybe email the list if you want more input?15:31
JayFOperators aren't usually at this meeting.15:31
TheJuliadtantsur: the tls enpoint for getting vmedia default artifacts is a super good callout, but also that endpoint is externally managed15:31
TheJuliayeah, I'll take that as an action item15:31
TheJuliaCool, I guess we can proceed onward!15:32
* TheJulia returns the podium to the chair15:32
TheJuliaJayF: we can move on to bug deputy updates15:34
JayF#topic Bug Deputy Updates15:34
* JayF hands chair to cid :D 15:34
TheJulialol15:34
JayFThere are 7 new public bugs listed.15:34
cidlol15:34
cidYup15:34
JayFI'll note we had a security bug or two that went public as well that might not be in this list. 15:34
JayFAnything worth specific note? The full list is in the agenda.15:35
JayFLooks like the RFEs are things previously discussed at the PTG?15:36
cid7 new bugs and no new RFES... I went through the list this "morning" so my filter must have missed the security bugs15:36
cidI don't think there's anyting worthy of note.15:36
JayFwell it's more that they aren't "new"15:36
JayFthey come in private, ironic coresec triages them, and changes to public security if they are eligible15:37
JayFso basically coresec does bug triage part of this15:37
cidI understand there's a new action item to pay a little more attention to ancient bugs.15:37
cid++15:38
JayFYeah, I think we just have bugs that stick in Fix Committed instead of Fix Released15:38
JayFjust a task-thing to clean it up15:38
cidacknwoledge.15:39
cidhanding the chair over to the chair15:39
JayFwho wants to be the deputy next week?15:40
JayFSomeone should give CID a break from it lol15:40
cardoeI'd sign up if I knew I wasn't gonna drop the ball.15:41
JayFI mean, make an effort, if you miss you miss15:41
clifI can do it I suppose15:41
* JayF locks clif and cardoe in a cage match to see who ends up deputy15:42
cidlol :D15:42
* JayF puts down clif and moves on15:42
JayFThere is nothing for RFE Review.15:42
JayF#topic Open Discussion15:43
JayFAny items for discussion not previously added to the agenda?15:43
TheJulialol15:43
TheJuliaoh my15:43
JayFAlso call for volunteers to chair the next meeting.15:43
TheJuliathis meeting has brought a smile to my face15:43
TheJuliaI can chair next week.15:43
cardoeI'll be out at the end of the month and the beginning of next.15:44
TheJuliaglorius vacation?15:44
cardoejust as fyi15:44
cardoeyes15:44
TheJuliaNice!15:44
JayF#note cardoe to be unavailable end of May/early June15:45
JayFLast call for the meeting15:46
JayFHave a good week o/15:47
JayF#endmeeting15:47
opendevmeetMeeting ended Mon May 11 15:47:39 2026 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:47
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-05-11-15.01.html15:47
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-05-11-15.01.txt15:47
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-05-11-15.01.log.html15:47
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: Stop ignoring `assignment` in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98523815:49
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: Stop ignoring call-arg in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98524015:49
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: Stop ignoring index in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98524115:49
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: Stop ignoring dict-item in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98524215:49
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: Stop ignoring func-returns-value in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98524315:49
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: Stop ignoring list-item in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98524515:49
cardoe^ that rebases the parts of Karan's series that I think will pass and work15:49
JayFdtantsur: https://review.opendev.org/c/openstack/ossa/+/987764 please review, I'm going to announce this today15:50
dtantsurk15:56
dtantsurmeanwhile, if anyone wants one fewer red non-voting job: https://review.opendev.org/c/openstack/ironic/+/98767315:57
JayFthanks for that, wanted to make sure you were onboard before I landed it15:57
*** tkajinam_ is now known as tkajinam16:03
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: switch to new tox constraints syntax  https://review.opendev.org/c/openstack/python-ironicclient/+/98813916:08
cardoeOnce all this goes through, I'll dig into the merge conflicted parts.16:13
TheJuliapost sent to mailing list regarding tls 1.316:41
TheJuliaoverall, looks like the jobs mostly work, going to attempt the negative test now16:41
TheJuliaerr, glad I double checked, didn't actually do it16:43
opendevreviewJulia Kreger proposed openstack/ironic master: WIP: Default TLS minimum version to 1.3  https://review.opendev.org/c/openstack/ironic/+/98795316:45
opendevreviewMithun Krishnan Umesan proposed openstack/ironic-python-agent master: Apply TLS configuration from Ironic lookup response  https://review.opendev.org/c/openstack/ironic-python-agent/+/98814917:11
opendevreviewMithun Krishnan Umesan proposed openstack/ironic-python-agent master: Apply TLS configuration from Ironic lookup response  https://review.opendev.org/c/openstack/ironic-python-agent/+/98815017:14
opendevreviewMerged openstack/ironic master: Repair the ironic-standalone-operator job  https://review.opendev.org/c/openstack/ironic/+/98767317:16
opendevreviewJulia Kreger proposed openstack/ironic master: WIP: Default TLS minimum version to 1.3  https://review.opendev.org/c/openstack/ironic/+/98795318:09
TheJuliaokay, if my changes are at least "sort of working", then that should explode horribly18:10
TheJuliawell, it looks like tls version control knobs work18:49
opendevreviewMithun Krishnan Umesan proposed openstack/ironic-python-agent master: Apply TLS configuration from Ironic lookup response  https://review.opendev.org/c/openstack/ironic-python-agent/+/98815019:24
JayFI am reviewing a lot of patches now, and adding hashtags to a lot of them19:44
JayFmight be worth taking a look thru the tagged patches in an hour or two when I get to the bottom, I've already added 2 or 319:45
cardoeJayF: so I got the 100 nodes per conductor during the PTG.19:50
JayFcardoe: please pull your -1, it's fixed now https://review.opendev.org/c/openstack/ironic/+/940572 19:53
JayFcardoe: I don't remember that conversation, but I've never ever heard a number that low for conductor scaling in the ~13 years I've been working on Ironic19:53
cardoeI'm happy to use a better value.19:54
cardoeBut other than than I appreciate the feedback on the doc. I wanna make it better.19:54
JayFI'm thinking more 500:1 is a safe bet. I've personally seen places where that number was ... thousands:one19:54
JayFactually, a better concrete suggestion:19:54
JayF"Ironic conductor scaling needs can change drastically depending on the how you manage and provision nodes as well as how frequently you manage and provision nodes. Five hundred nodes per conductor, with extra capacity in case of failure, may be a good starting point if you don't have any data at all."19:55
cardoeI'm not trying to say these are the hard rules but just some examples of usage.19:55
cardoeThat's good. I like it.19:56
JayFI think we just need to hit the two Ironic-centric modes of op (standalone/bifrost-shaped + openstack-but-no-nova shaped) and the two nova-centric modes of op (Ironic is an admin only api -- Nova is the only thing users us + node.lessee is set on provisioning so users get some visibility and self-service19:56
JayF)19:56
JayFbasically: Ironic, Ironic+OpenStack, Nova+OpenStack (hidden Ironic), Nova+Ironic+OpenStack (automated lessee)19:57
TheJuliawow, https://docs.openstack.org/python-ironicclient/latest/doc-python-ironicclient.pdf is interesting to find in google search20:02
TheJulia"openstack llc"20:02
JayFour docs pages have been very, very badly indexed in google recently20:02
JayFI think the AI stuff hurting our infra + the infra attempts to block AI have hurt us in the search space20:03
JayFand/or just it's a more personal example of how search engines are rapidly becoming useless20:03
JayFcardoe: https://review.opendev.org/c/openstack/python-ironicclient/+/985240/2 that's a real CI failure, sadly :( 20:15
cardoeYeah readying a fix but got squirreled into neutron20:15
JayFI am trying to catch up on review20:16
JayFI've been heads down in other stuff but pushing it all aside to catch up on this today20:16
JayFdtantsur: (IDK who else would care?) bugfix/34.0 appears to have busted CI generally https://review.opendev.org/c/openstack/ironic/+/987920 -- I don't consume this so I'm not worrying about it, but I noticed because I was gonna try and land the security fix20:23
opendevreviewJay Faulkner proposed openstack/ironic-python-agent stable/2026.1: Preserve ESP filesystem label during software RAID relocation  https://review.opendev.org/c/openstack/ironic-python-agent/+/98817420:25
opendevreviewMerged openstack/python-ironicclient master: create more specific update method for typing  https://review.opendev.org/c/openstack/python-ironicclient/+/98810620:28
opendevreviewMerged openstack/python-ironicclient master: Stop ignoring `assignment` in tests  https://review.opendev.org/c/openstack/python-ironicclient/+/98523820:28
opendevreviewMerged openstack/networking-baremetal master: Clean up example local.conf  https://review.opendev.org/c/openstack/networking-baremetal/+/98800020:36
opendevreviewMerged openstack/sushy stable/2026.1: filter out drives reporting null size on inspect  https://review.opendev.org/c/openstack/sushy/+/98774520:47
opendevreviewMerged openstack/python-ironicclient master: switch to new tox constraints syntax  https://review.opendev.org/c/openstack/python-ironicclient/+/98813920:49
opendevreviewMerged openstack/sushy-tools master: Add --storage-pool CLI option  https://review.opendev.org/c/openstack/sushy-tools/+/98748621:07
opendevreviewMerged openstack/ironic-python-agent master: Preserve ESP filesystem label during software RAID relocation  https://review.opendev.org/c/openstack/ironic-python-agent/+/98670721:21
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: simplify is not None check  https://review.opendev.org/c/openstack/python-ironicclient/+/98817921:29
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: fix codespell config  https://review.opendev.org/c/openstack/python-ironicclient/+/98818021:29
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: remove unused script which does not pep8  https://review.opendev.org/c/openstack/python-ironicclient/+/98818121:29
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: switch to using pre-commit for lint checks  https://review.opendev.org/c/openstack/python-ironicclient/+/98818221:29
opendevreviewDoug Goldstein proposed openstack/python-ironicclient master: remove separate codespell job  https://review.opendev.org/c/openstack/python-ironicclient/+/98818321:29
opendevreviewMerged openstack/ironic master: Remove deprecated http_basic_* options  https://review.opendev.org/c/openstack/ironic/+/98642921:32
opendevreviewMerged openstack/ironic master: Document software RAID image compatibility caveats  https://review.opendev.org/c/openstack/ironic/+/98671821:32
opendevreviewMerged openstack/ironic master: Allow removing volume connectors on some states  https://review.opendev.org/c/openstack/ironic/+/94057221:32
opendevreviewMerged openstack/ironic-tempest-plugin master: Reuse base tempest code for common config sections  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/98749421:32
cardoeJayF: I switched us to pre-commit for python-ironicclient21:35
opendevreviewMerged openstack/networking-generic-switch stable/2026.1: Add Dell OS10 bond trunk support  https://review.opendev.org/c/openstack/networking-generic-switch/+/98606321:40
cardoeI wonder what else we've got left.21:42
opendevreviewMerged openstack/ironic master: fix: drop version from runbook traits v2  https://review.opendev.org/c/openstack/ironic/+/98680621:45
opendevreviewMerged openstack/ironic master: Clean up example local.conf  https://review.opendev.org/c/openstack/ironic/+/98800121:45
opendevreviewMerged openstack/ironic master: Trivial: debug_tracebacks_in_api=True for unit tests  https://review.opendev.org/c/openstack/ironic/+/98718421:45
opendevreviewMerged openstack/ironic master: Document provisioning a riscv machine  https://review.opendev.org/c/openstack/ironic/+/98746021:45
opendevreviewMerged openstack/ironic master: Fix healthcheck intercepting all API routes  https://review.opendev.org/c/openstack/ironic/+/98810221:45
opendevreviewDoug Goldstein proposed openstack/ironic stable/2026.1: Fix healthcheck intercepting all API routes  https://review.opendev.org/c/openstack/ironic/+/98818621:58
opendevreviewMerged openstack/networking-generic-switch stable/2025.2: Add Dell OS10 bond trunk support  https://review.opendev.org/c/openstack/networking-generic-switch/+/98606422:00
TheJuliaFor deferred actions, and ultimately what would surface as bulk actions for users to find out the "what" is pending, I feel the need for a quick poll.22:02
TheJuliaOption 1: /v1/jobs and /v1/nodes/<uuid>/jobs22:04
TheJuliaOption 2: /v1/actions and /v1/nodes/<uuid>/actions22:04
TheJuliaOption 3: /v1/todo and /v1/nodes/<uuid>/todo22:04
TheJuliaor, deferred tasks22:04
TheJulia(we need to solidify on a name, kthxbai)22:04
JayFDid you see my comment about how it might be two separate things?22:09
TheJuliaAlso, how long would we want to keep the details for each, specifically do keep in mind that we really need to keep the table pruned down for efficient index operations 22:09
TheJuliaSort of, and that was not how I read it on my first pass22:09
TheJuliaI just posted my comments, but first I want us to try and get to consistency22:09
JayFI'm saying what's been proposed is like, very tightly coupled to nodes22:09
JayFand that gives it a certain amount of power, but limits it's generic usefulness for e.g. bulk actions22:10
TheJuliayeah, and I get that, and ultimately I think we have to with our model22:10
TheJuliaso22:10
JayFso I'd say, just not make this generic. Just make it node actions, and have it be closely affiliated with a node22:10
TheJuliaI wrote many words about this, maybe too quickly, but it boils down tot he follwoing in my mind22:10
JayFeven if we need a /v1/whatever level endpoint, too, to roll them up for visibility22:10
JayF /v1/actions and "node actions" (/v1/nodes/{id}/actions) are the first implementation? something like that?22:10
TheJulia1) we really need to try and front load the identification of nodes, and the creation of the records to fully track the state. Otherwise we're then trying to account for state in follow-up and possibly missing/skipping nodes. Or we're building an eventual consistency engine, but then those are long running tasks on each conductor because immedaitely, we have to split requests by conductor.22:11
JayFTheJulia: ^ this is what I was trying to nod to. Keep the limited "one node action" scope, and have it be the first of many things in a category22:11
TheJulia2) If we look at where we're getting taxed, its single row inserts and and deletes in general22:12
TheJuliaThe delete problem, we know how to handle22:12
JayFand then [future milestone that doesn't have to be decided here] we maybe have a "Bulk Action" action type, which creates a bunch of node actions (or trickles them in "N" at a time)22:12
opendevreviewMerged openstack/ironic master: Update packaging configuration  https://review.opendev.org/c/openstack/ironic/+/98710522:12
TheJuliaThe insert, if we try to hydrate python objects and all that, then we're inefficient, but if we bulk insert, we're pretty much drawing a direct line for a conductor to pickup and track the work22:12
JayFYeah, I think we agree that the node-specific object, and thinking "one action at a time" and/or "linked list-style action chaining operating on one node"22:12
JayFis the path for now22:13
TheJuliaOh, okay, yeah22:13
JayFSo /v1/actions (proposed) would have a rollup of all actions the caller can see22:13
TheJuliaso, I was thinking that very first step could handle most of that22:13
JayFwhich today is Node Actions22:13
JayFand after Bulk Node Updates, might also be "Bulk Actions" or something else, that's similarly shaped but not exact22:13
JayFlike imagine /v1/actions has a type and an id with linking, in the response, to the fully hydrated /v1/nodes/id/actions/id22:14
TheJuliaI think the cleanest way to handle it is sort of keep it the exact same under the hood, but to model the breakup of the work to be hyper efficient on a bulk insert22:14
JayFSo I have a cluster with 100k nodes.22:14
* TheJulia spittakes22:14
TheJuliaokay22:14
JayFYou want me to call bulk actions and have us sync insert 100k lines into node actions22:14
* TheJulia goes for the booze22:14
JayFTheJulia: I don't have this today, but a former employer did, and had a use case for this API22:15
TheJuliaoh22:15
TheJuliaokay22:15
TheJuliayeah, so22:15
JayFthat's why I'm saying I am completely unwilling to say "do just in time node action submission one by one"22:15
JayFbecause it's adding a scaling cap to something that's inherently built for people at the edges of scaling22:15
TheJuliato permit for locking, it would likely need to be chunked under the hood, so the api has to "accept it" and send it off to a conductor to be a long lived item22:15
JayFa call to update all nodes' deploy_interface to autocorrect, for instance22:15
JayFyeah, which means we need a way to track that async task :) 22:15
TheJuliaits first thing is to "get that list enumerated 22:15
TheJuliaand from then, conductors could pick it up22:16
JayF"Hey, how many of those 100k nodes in my cluster have been updated?" can't be asked of the current API shape22:16
JayFor even "Hey, did you get any errors trying to submit 100k node actions?"22:16
TheJuliaideally, chunk into 250-500 so we don't entirely flood the cluster, but we keep conductors buys22:16
TheJuliabusy22:16
JayFexactly, but we'd need a meta-object to track that22:16
JayFone that I don't think we create today22:16
TheJuliaYeah, so I can only see that as viable as down to the individual node level22:17
JayFbut the need for which suggests not all actions will be "single node action" shaped22:17
TheJuliaso, okay22:17
TheJuliaso basically we've been talking the same thing, just using different words22:17
JayFthat's why I'm saying, punt anything/everything not aligned to 1 node/1 action, if we need a root-level rollup api, design it in a way that it can be used for other action types, too22:17
TheJuliaYeah, that is why I wanted the parent request to be identifable/trackable22:17
TheJuliaThe "done" is more than anything, getting the work enumerated"22:18
JayFyeah, so I'm thinking of this as like22:18
TheJulia"getting he work enumerated"22:18
JayF /v1/nodes/[id]/actions/[id] is primarily there to kill sleep()22:18
JayFhandle those cases today which are long running and ugly in threaded ironic22:18
TheJuliaso22:18
JayFideally we would JUST implement that much now, to limit scope22:18
JayFand not try to do a GET list(all pending node actions)22:19
TheJuliaI can't think of it as enitrely separate views of the same table structure22:19
JayFthere's a reason I keep saying the root level API would be a rollup: do we require a 1:1 mapping of /v1/endpoint to EndpointObject?22:19
TheJuliaGranted, if you ask for actions upon a node, you'll always get a very short list22:19
TheJuliaand that is different user oriented filtering22:19
JayFso like I'm saying if we can today: implement /v1/nodes[id]/actions (list of all actions FOR THAT NODE), and /v1/nodes/[id]/actions/[id] (detailed action) 22:19
JayFCOMPLETELY PUNT /v1/actions (or whatever we want to call it)22:20
TheJuliaso22:20
TheJuliano22:20
JayFand try to limit scope to sleep()ytime cases22:20
TheJuliaokay22:20
TheJuliaHow long do you expect entries to exist for a node?22:20
JayF$runtime_of_action, once it's done it'd be promoted to a node history item22:20
TheJuliaAnd operationally, if you need to know what is going on will you look at individual nodes and state, or the rollup endpoint?22:20
TheJuliabecause surely your not going to do 50-100k node gets to find out something today22:21
JayFoperationally, we're exposing data about operation we don't expose today22:21
JayFso I don't feel a strong responsibility to give rollup visibility22:21
JayFespecially if a failed node task leads to a last_error population and fail out as it does today22:21
TheJuliaI feel stuck between many strong asks and arguments22:21
JayFwe're taking data currently only in logs and putting it into an API endpoint22:21
TheJuliaand truthfully a clean view makes a lot of sense to me22:21
JayFeven if we don't have the cross-cutting view of it22:21
TheJuliaa distributed view shifts responsibility22:22
TheJuliaand burden22:22
JayFbe clear: what is a distributed view?22:22
JayFI could see that being used to describe both methods of operation22:22
TheJuliathe node level "get pending actions for this node"22:22
JayFI don't have a strong feel for needing that, fwiw22:22
JayFif we just wanted actions-by-id on the node I'd be OK22:22
TheJuliahonestly, I even question if that is usable22:22
TheJuliabecause your doing a point in time quick get, and that entry is likely to be gone by the time you look again22:23
TheJuliaand if your doing a bunch of stuff, you'll want to see the aggregate/clean view of everything22:23
JayFyeah, so just do /v1/nodes/[id]/actions/[id] 22:23
JayFthat is an antipattern IMO22:23
JayFand goes against the idea of scaling up22:23
TheJuliano, because you'll never have the id to even ask it for more info22:23
TheJuliaThe idea is get rid of sleeps, but also to be able to queue up the needful22:23
JayFbecause you should be able to trust Ironic to raise the errors if/when they happen22:24
JayFI see providing a dashboard-ready action GET endpoint as basically directing operators to do things that are bad ideas, generally22:24
TheJuliaso, if an action takes <30 seconds to trigger and its gotten to, it is on a quick path to disappearing in general22:24
TheJuliaso there is no time to go ask for more data22:24
JayFthe more I think about this the more I dislike the ability to get an aggregate list of pending actions, on or across nodes22:24
JayFsimply because it's information paralysis: it's not useful info to operators22:24
JayFThis is essentially what was an INFO log a year or two ago being promoted to /v1/stress-over-me22:25
TheJuliacan you elaborate on informational paralysis in this context22:25
JayFThis is essentially what was an INFO log a year or two ago being promoted to /v1/stress-over-me <--- this is EXACTLY what I mean22:25
TheJuliaspecifically, what you see as valuable for operators to know22:25
TheJuliaand in the time window they may care22:25
JayFI see it as valuable *to not tell operators what they don't need to konw* almost moreso than telling them what they need to know22:26
JayFI think operators need to know very little about node actions as proposed when they successfully complete22:26
JayFwhen they error, they need the info they get already today via node history/logs/last_error22:27
TheJuliaso, we're going to need to have a way to peer into this data set regardless to troubleshoot22:27
TheJuliabut maybe thats like a non-api endpoint thing22:27
TheJuliaand maybe its just all aggreated22:27
JayFif there's value in the info, it's only in an aggregate way, e.g. "how busy is conductor (group?) X" 22:27
JayFSo I flip the question: in the case of a non-failing node action, what info would *you* think an operator needs?22:27
TheJuliaSo, taking a step back to bulk node actions, how do you tell the user their node was covered, or do we just not provide that level of insight?22:27
JayFThis is what resolves to [null set] for me everytime22:28
JayFTheJulia: part of my proposal is saying "don't build the API shape for that yet"22:28
TheJuliaThats a super good point, "I have x queued items at this time, I'm launching y"22:28
JayFWhen we later do bulk actions, we'd have the freedom to implement some kinda meta-action22:29
JayFwhich would likely just have stuff like: action to perform, nodes to operate on [list of nodes], nodes completed [list of nodes] or similar, and we'd just batch them in groups of "N" into node actions22:29
JayF^^ + nodes_that_errored22:29
TheJuliawell, I have serious concerns on "meta actions" in general, unless the action itself is the enumeation of the work22:30
JayFthat would be the action :)22:30
TheJuliaokay22:30
JayFenumerating the work and batching it into the conductor22:30
TheJuliaokay, so, if folks agree we don't put an API in, again, since we've gone back and forth on this a few times22:30
TheJuliawe can deal with that later22:30
JayFTheJulia: I am too being careful to not accidentally reimplement celery-over-ironic22:30
TheJuliaor not at all22:30
TheJuliaand just try to keep the under the hood data to metrics/coutners22:30
TheJuliathat get logged/surfaced for ops metrics side folks22:31
JayFI like the idea of fetching action by ID for the future case, but I can't think of a use case for it now22:31
JayFassuming -- as I mentioned before -- any failures end up in node history22:31
TheJuliain the model I'm thinking, the historical id will rapidly become worhtless because the data will be gonezo.22:31
JayFand unless we soft deleted, like you said, it'd just be a "flappy" API in terms of things being visible/invisible quickly22:31
TheJuliaI'd prefer we don't try to soft delete unless we're actively purging the data22:32
JayFonly real time I'd GET /v1/nodes/blah/actions/lol would be to troubleshoot why lol is taking a long time that's not yet timeout-long22:32
TheJuliaI've been burned by "soft delete" modeling in the past and it took me 3 months to get out of that nightmare22:32
JayFwhich is ... not a pattern I want our operators to think about22:32
JayFTheJulia: yep, and our existing node history periodic is a piece of code that makes almost every operator I've seen WTF when they realize it22:32
JayFlol22:32
TheJuliasame, I really don't want node level listing of pending work which may be gone in moments22:32
TheJuliaJayF: because I got horribly burned once in the past22:33
JayFyeah, I think we agree on no API now, at least for single-node-action based stuff22:33
JayFthere's no case for it that's not better suited for node history or logs22:34
TheJuliaLike I called a friend of mine who works on some of the largest databases in existence, and discussed the nightmare I had22:34
JayFsince to have a node actions API that makes sense, we have to tackle soft deletes, and I don't want that and it hasn't even hurt me in the past lol22:34
JayFIt's weird; my operator instincts say I want the API. I want the visibility -- but my developer instincts know I'm just going to be enticing an operator to drill down into red herrings 22:35
TheJuliawell, we either do soft deletes and still purge it programatically, or we hard delete as we go, or we hard delete after some amount of time has passed22:35
TheJuliayeah, exactly22:35
TheJuliaunder no circumstances, do I think its right just to soft delete and punt the problem to operators to "clean up your tables"22:35
TheJuliaespescially since them doing that will take down ironic.22:36
TheJuliasince large delete locks block inserts.22:36
TheJuliawhich is why history is written the way it is22:36
JayFI was about to say "of course", but I assume if you're saying that it means...22:36
TheJulia?22:36
JayF> under no circumstances, do I think its right just to soft delete and punt the problem to operators to "clean up your tables"22:37
JayF^ of course22:37
JayFbut you wouldn't have said that if someone thought it was OK, I suspect lol22:37
JayF**unless someone thought it was OK22:37
JayFegad22:37
TheJuliaso plus side of this discussion: we're passionate and want it to move forward22:37
TheJuliaI'm okay if we soft delete it, but we're going to need a periodic to sit there just like node history and keep that thing pruned22:38
TheJuliaand we're likely going to have to keep the batch size down to a smallish limit22:38
TheJuliaWe'd have to do the math to figure out the right parameters22:38
JayFI'm OK with no API, I'm ok with soft deletes and rollup at the node level and get-by-id (TTL of what, an hour or three after it finishes with a max of "N" actions total?)22:39
JayFI'd strongly prefer we avoid a meta-rollup api (e.g. get all actions vs ^ get all actions for node X)22:39
TheJuliaSo I was thinkinging the done actions should stick around for no more than like minutes, not hours22:40
JayFjust because I'd prefer to figure out the shape of that as we're designing bulk node updates22:40
JayFI was thinking an hour and added the or three to be safe lol22:40
JayFso we're closer to aligned than not22:40
TheJuliaokay22:41
JayFI guess in busy clouds, 15 minutes vs 1 hour could be an appreciable amount of data22:41
TheJuliayeah22:41
TheJuliaand depends on how much churn as well22:41
TheJuliawe'd need to do some math22:41
JayFgoing back to my "an endpoint to help us be more efficient needs to be very scalable" 22:41
JayFI'd almost just say, phase 1: no API22:41
JayFif we wanna add an API later, great22:41
JayFor if people demand it22:41
TheJuliawell, the take I got from the PTG was that we absolutely needed a roll-up/aggregate view endpoint22:42
TheJuliabut I don't remember "why" exactly22:42
JayFfor the bulk node stuff, I think22:42
TheJuliadtantsur: if you happen to remember, ^22:42
JayFat least that's the use case I was pushing it for22:42
TheJuliaeh, if that was the only reason, we could punt on it22:42
JayFbut I realized we need more bite-sized chunks22:42
JayFPTG puts me in big-think mode22:42
TheJuliaof course22:43
TheJulia:)22:43
TheJuliacardoe: you too, if you have opinions, chime on in ^22:43
JayFI suggest putting a link to eavesdrop starting at the beginning of this chat in the gerrit lol22:43
TheJulialol22:43
JayFI don't think we started out agreeing but we got there, the whys were what drove us there22:43
JayFwe need to only do the stuff with a good, justifying why22:44
TheJuliaoh yeah, and truthfully talking through it, we were already pretty closely aligned22:44
TheJuliajust using different words/meanings22:44
cardoeoh dear... here I was making dinner22:44
TheJuliaDOOOOooooommmmm of IRC22:44
JayFthis is all logged, leave us alone and disable notifications when you're not working 22:44
TheJuliayes, go finish dinner22:44
JayFlife >>>>>>>>>>>>>>>>> ironic22:44
TheJuliaSpeaking of, I need to go begin a sauce in a little bit22:45
JayFfor instance, touch sand sometime :D https://usercontent.irccloud-cdn.com/file/lXEcAQ7V/life.png22:45
TheJuliaI have sand outside... except it is dry and hot22:47
TheJuliaI'll trade you sand!22:47
TheJuliawith the most dangerous substance known to mankind mixed in22:47
JayFants?22:47
TheJuliaWater!22:47
TheJuliaoh, wait, we have ants here22:47
TheJuliado you not have ants?22:47
JayFI didn't see em at Ocean Shores lol22:48
JayFday tripped to the beach yesterday morning, just spent half the day driving and being near the ocean22:48
JayFyou don't realize how much you need something like that until you do it22:48
TheJulia... many years ago I woke up around liek 2:30 AM22:48
TheJuliaAnd decided "I'm going to drive up the mountains and photograph the sunrise"22:49
TheJuliaand did it22:49
TheJuliaand it was such a refreshing experience22:49
JayFI wake up many mornings, weekend or weekday, like 5am +-30minutes, and I just decided that on Sunday I was gonna wake up and GTFO. Got to roadtrip my truck for the first time (and drive it on the sand! It's apparently allowed and is very normal in WA). 22:49
JayF"EV roadtrip" was not scary. The rivian charger worked, and the only hiccup was that mother's day was not the right day to eat at the Denny's across the street (it's okay, I was saved by Wendy and her baconator)22:50
TheJuliaheh22:50
TheJuliacool cool22:50
JayFyou are old hand at the EV stuff, it's still new to me22:51
TheJuliaOkay, I think my brain is reaching "GO MAKE PASTA SAUCE"22:51
JayFI mean, I'm reaching "just tell old man stories" phase, and it's like 9 minutes until my EOD22:51
JayFso same :D 22:51
cardoehttps://review.opendev.org/c/openstack/python-ironicclient/+/988183 that series switches the client to pre-commit and makes codespell enforcing22:51
TheJuliacardoe: dinner > ironic22:52
cardoeI’ll have to get caught up. You guys talked a lot.23:13
TheJuliapassion!23:34
opendevreviewMerged openstack/ironic stable/2026.1: Fix pickle error when oslo.service uses spawn mode  https://review.opendev.org/c/openstack/ironic/+/98755023:48

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!