Tuesday, 2026-01-27

cardoeDon't you know we evaluate everyone on LOC?00:01
JayFThat's how they did it 5 years ago. Now it's all based on how many plates^W AI agents you can keep spinning00:01
opendevreviewSteve Baker proposed openstack/ironic-tempest-plugin master: Test graphical console novnc connection  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/96315200:34
opendevreviewSteve Baker proposed openstack/ironic master: Add an autodetect deploy interface  https://review.opendev.org/c/openstack/ironic/+/97318701:35
opendevreviewSteve Baker proposed openstack/ironic master: Replace ironic-tempest-bios-ipmi-direct with -autodetect  https://review.opendev.org/c/openstack/ironic/+/97426001:35
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: Change default value of some redfish firmware update config  https://review.opendev.org/c/openstack/ironic/+/97480004:09
rpittaugood morning ironic! o/07:27
kubajjgood morning rpittau, and ironic! o/07:31
kubajjis anything wrong with the metal3-integration job in Ironic? I keep getting an Internal Server Error when getting collection version metadata for kubernetes.core:6.2.007:33
rpittaukubajj: it's a problem with the ansible galaxy site, it's been fixed yesterday though07:42
rpittaurecheck should work07:42
kubajjrpittau: will do, thanks (didn't want to waste recheck again)07:42
jandersgood morning Ironic o/08:26
abongalegood morning o/09:05
opendevreviewAbhishek Bongale proposed openstack/python-ironicclient master: Fix field names for selection and json  https://review.opendev.org/c/openstack/python-ironicclient/+/97394809:14
opendevreviewMerged openstack/python-ironicclient master: Filter portgroups by shard  https://review.opendev.org/c/openstack/python-ironicclient/+/97056210:40
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: Change default value of some redfish firmware update config  https://review.opendev.org/c/openstack/ironic/+/97480011:48
dtantsurcardoe: I don't understand why you say that the schema approach won't work..12:53
dtantsurwe literally do it for RAID12:53
dtantsurno objects to using something more sophisticated, but then we may need to fix other usages12:55
dtantsur* objections12:57
cardoeOpening a data file like that in the source tree won’t work correctly for many packaging13:12
cardoeMethods. That’s why pkg_resources was created in the first place.13:13
cardoeIt definitely won’t work when that path is not called out in the packaging metadata. Now pbr does some magical things but I wouldn’t count on that forever.13:14
dtantsurcardoe: then how does it work for RAID?13:14
dtantsurI think if your packaging arbitrary omits files, it may be a problem with packaging13:15
cardoeI’m speaking to the Python packaging spec.13:15
dtantsurWell, that part evidently works unless I'm missing a subtle difference to the existing cases13:16
* dtantsur still does not quite get why it should not work13:17
cardoehttps://importlib-resources.readthedocs.io/en/stable/using.html?utm_source=chatgpt.com13:18
dtantsurI've just built a wheel out of alegacy's patch, and it does contain the schemas13:18
cardoehttps://docs.python.org/3/library/pkgutil.html13:18
dtantsurI don't mind us switching to a better way, I just don't understand why block the patch13:18
cardoeYou’ve not -1’d patches cause there’s something technically incorrect that happens to pass tests?13:19
dtantsurdefine technically incorrect please13:20
cardoeYou cannot open a file along side and installed Python file the way it is being done.13:20
cardoeIt’s a violation of the Python packaging spec which is why there are a multitude of helpers over the years for this.13:20
dtantsurCitation needed13:21
cardoehttps://docs.python.org/3/library/importlib.resources.html13:22
cardoeI’ve literally just sent you now 3 places in the Python docs that say this.13:22
dtantsurNo, you did not13:23
dtantsur>>> from ironic.drivers.modules.network import ironic_networking13:23
dtantsur>>> len(ironic_networking._LAG_SCHEMA)13:23
dtantsur813:23
dtantsurIt literally works, installed in a venv from a wheel13:23
cardoeRead any of the 3 links.13:23
dtantsurStart with proving that RAID has never worked in Ironci pleas13:24
dtantsurMaybe, just maybe, nobody ever imports Ironic from a zip file13:25
dtantsurBut since they're already badly broken (even iPXE uses local files), I'm not sure why we need to block a patch o nit13:25
dtantsurI mean all this stuff that also does not work wip zips https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/pxe.py#L143-L14713:26
dtantsurcardoe: my problem with this request is that we're tasking alegacy with fixing something that we've done for ages, and where some pondering is required about the path forward (i.e. how to handle operator-provided paths that are not packaged)13:30
dtantsurJayF: since you also put -1 ^^^13:31
cardoeYou’ve always been the one advocating for the code base to13:32
cardoeImprove and be technically correct. I’m surprised to hear to push for what’s clearly wrong because it works now.13:33
cardoeOverride my vote and merge it anyway.13:33
dtantsurHeh, if I was pushing for everything to be perfect, you'd not be complaining about broken inspection rules :)13:33
dtantsurLook, whatever beef you have with me, could you please not apply it to alegacy? While we work in the same company, he's not even my teammate.13:34
dtantsurAnd he's doing a feature that nobody has dared taking for many years of taking about it.13:34
cardoeI have no beef or issue. I’m just saying I’m surprised. You’ve done a good job holding the project to a high standard.13:34
dtantsurAlso, my point is not to keep this behavior forever, my point is deciding on how to fix it within the core team and fixing it then. If you want me to do it - happy to.13:35
dtantsur(I also don't mind alegacy just moving the schemas to constants as JayF suggested)13:35
cardoeI was trying to ask yesterday if the schemas should be constants or files.13:35
dtantsurWe don't have a policy on that, to my best knowledge13:36
cardoeI even offered to convert them one way or the other if there was a preferred approach.13:36
dtantsurSome schemas can be overriden via configuration (network data, for instance), these make sense as files (and it's not trivial to use importlib for them because they can be operator-provided)13:37
cardoeWell when they are operator provided that path is different13:37
dtantsurI think RAID is the only one that we hardcode, maybe we could just move it to a constant13:38
dtantsurI think using constants for alegacy's work is a fair call (I'm personally happy with a follow-up).13:39
cardoeI did the -1 for awareness since it was gonna get merged in otherwise.13:40
cardoeiurygregory: just wondering if you're able to set Dell BIOS settings without booting into IPA?13:41
iurygregorycardoe, hey o/13:41
iurygregoryI haven't touched BIOS setting yet13:42
dtantsurIIRC we use IPA heartbeats as a sign that the machine is back13:42
iurygregorywe are focused on the firmware updates first, trying to address the issues we see in our testing..13:42
cardoedtantsur: yeah. And Dell has some weird thing that SOMETHING must call ExitBootServices() which we solve by booting IPA since the Linux kernel does that for us.13:43
dtantsuryep, correct13:44
cardoeI'm just looking at an email where apparently we got a bunch of stuff from Dell and the only enabled boot device is the hard disk. So I need to enable something first.13:44
* dtantsur is speechless13:45
cardoeAnd I was just thinking if I could have Ironic enroll the node and then ensure a boot device is enabled before moving to inspection or any other state.13:45
dtantsurSounds like a verify step to me13:46
cardoeYeah.13:47
dtantsur(in metal3, we wipe idrac jobs as a verify step to make sure virtual media works - a somewhat similar case)13:48
cardoeSo booting into IPA *AFTER* the BIOS change wouldn't be problematic. But right now it seems like Ironic wants it to boot into IPA prior to the BIOS change and then come back into IPA.13:48
dtantsurhmm, I think janders handled that for servicing, maybe we need to adapt his patch to cleaning13:49
dtantsurcardoe: I mean https://opendev.org/openstack/ironic/commit/1b2c01185c7907a1d17c4d90cf3bcd73fe34565813:50
cardoeYeah that makes sense.13:50
cardoeCurious to see how you add a verify step in Ironic13:50
dtantsurQuite similar to other steps https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/drac/management.py#L42113:51
dtantsurbut enabled in ironic.conf13:51
dtantsurhttps://opendev.org/openstack/ironic/src/branch/master/ironic/conf/conductor.py#L438-L44713:52
opendevreviewRiccardo Pittau proposed openstack/bifrost stable/2025.1: [stable only] Ensure firewalld is unmasked before enabling it  https://review.opendev.org/c/openstack/bifrost/+/97485513:53
dtantsurcardoe: but note that verify steps are designed to be rather fast, I'm not sure about 15 minutes of rebooting the machine.. I also don't think we have any framework for async steps (no VERIFYWAIT)13:54
cardoeI could at least check that the right boot device is enabled.14:03
JayFdtantsur: cardoe: I would prefer as constant but my -1 was based on the (unsure if flawed) understanding we needed to switch it or add something to the project toml. If that's not true feel free to land over the -1 14:09
JayFI won't be at a desk to update that vote for a bit14:09
cardoeThat's what I was saying that pbr might be overriding things for us.14:09
dtantsurit probably does14:10
dtantsurI've chatted with alegacy, he'll create a constant. No need to update the vote.14:10
JayFalegacy: My downstream got back to me about that log and said that not only was it okay from a security standpoint that it would be super duper useful14:39
alegacyJayF... ok, good enough for me, thanks.14:47
opendevreviewRiccardo Pittau proposed openstack/ironic master: Migrate database charset from UTF8MB3 to UTF8MB4  https://review.opendev.org/c/openstack/ironic/+/97443015:15
cardoerpittau: /home/zuul/src/opendev.org/openstack/bifrost/scripts/collect-test-info.sh: line 102: /home/zuul/openrc: No such file or directory15:23
opendevreviewDmitry Tantsur proposed openstack/ironic-specs master: Asynchronous sensor data collection  https://review.opendev.org/c/openstack/ironic-specs/+/97275415:34
dtantsurClaude things this specification ^^^ is "well-written with good technical detail, clear structure, and logical flow. The code examples are helpful and the reasoning is sound." You have no excuses not to review it :D15:36
cardoeI'll give it a look. Claude also says you can review some times in https://review.opendev.org/q/hashtag:ironic-week-prio+status:open ;)15:52
opendevreviewcid proposed openstack/ironic-specs master: A specification for a node power lock feature  https://review.opendev.org/c/openstack/ironic-specs/+/97489416:11
TheJuliadtantsur: if that was a bar for reviews, the VXLAN spec would have landed long ago ;)16:24
TheJuliaI’ll try to review it tomorrow, fwiw16:24
dtantsurwell, yeah, there you also have the "not being entirely clueless" aspect :D16:25
TheJuliaI have had several people remark a statement along the lines of: this vxlan stuff breaks my brain”16:26
TheJuliacardoe: oh, sounds like you might need a manageable -> cleaning -> manageable flow which then allows normal provide flow to proceed16:28
cardoeyep. but I've got the catch-22 of not being able to go into IPA16:29
cardoeTheJulia: did you see that person from ib systems asking about vxlan and bridging to physical hardware.16:29
cardoeI read their initial "this is what I was thinking" and I was like "oh my sweet summer child... I remember when I thought this too. then the hell of vxlan unleashed its face on me and I aged 20 years."16:30
TheJuliaHeh, where was this?!16:30
TheJuliacardoe:yeah, the catch-22 is sort of provide *expects* to be able to boot and yeah16:32
TheJuliaIt… may be as simple as some state machine changes to enable a manual cleaning on that level16:33
TheJulia*may*16:33
TheJuliaalthough, on aging 20 years, I don’t think anyone aged 20 years that I’ve talked to16:33
cardoehttps://review.opendev.org/c/openstack/neutron/+/96505316:34
cardoeTimeout reached while cleaning the node. Please check if the ramdisk responsible for the cleaning is running on the node. Failed on step {}. ORLY?16:35
TheJulialol16:35
TheJuliacardoe:yeah, that be automated cleaning at it’s finest!16:35
TheJuliaBoot the agent, find out what to do16:35
opendevreviewMerged openstack/ironic-python-agent-builder master: Drop redundant basepython  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/97443116:41
TheJuliaFWIW, I also commented on there.16:43
cardoeMuch better said than me.16:46
cardoeI also pinged ya on https://review.opendev.org/c/openstack/neutron/+/96541516:47
ContinuityHey ironic'er16:55
ContinuityIs anyone else being bitten by16:55
Continuityhttps://bugs.launchpad.net/neutron/+bug/199507816:55
TheJuliacardoe: yeah, I’d move it at this point. They are clearly disinterested.16:55
cardoeContinuity: yes16:56
TheJuliaCould this be an opportunity for more ironic-neutron-agent to be… evil ?!16:56
* TheJulia jumps on the next meeting16:56
Continuitycardoe: any Perls of wisdom on a work around, one of my engineers had a pretty good work around for 2022.2 but a load of code changed since then16:57
ContinuityShort of shouting at and manually changing it on the occasions when it goes wrong 16:57
ContinuityIt seems very... brittle. 16:57
opendevreviewVerification of a change to openstack/ironic master failed: Set columns in bios_settings table to bigint  https://review.opendev.org/c/openstack/ironic/+/96834816:58
cardoeWorking through this with the baremetal-l2vni and how to fix it fully.16:58
ContinuityJust to confirm we are talking about the same thing, as sometimes these are a little obtuse17:00
ContinuityCreating a new vlan backed provider network sometimes results in ha chassis and the ovn gateway chassis being out of sync17:00
ContinuityWhich then causes north south traffic form the baremetal node to be lost 17:01
cardoeI'm using vxlan -> vlan but yes.17:05
cardoeIt's in older OpenStacks. I've not run multiple in newer versions to confirm.17:06
cardoeBut that's cause I'm trying to get us updated to the baremetal-l2vni setup first and then work there.17:06
* TheJulia re-reads the bug17:24
ContinuityTheJulia: cadre: if we can work out a fix, or a work around I will personally buy you a case of your fav tipple17:24
opendevreviewMerged openstack/ironic master: Populate the conductor group field from config  https://review.opendev.org/c/openstack/ironic/+/97469417:30
opendevreviewMerged openstack/ironic-tempest-plugin master: Add project badge  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/97235717:30
TheJuliaHmmmmm17:36
TheJuliaSo, I’m not an OVN expert, but I feel like I’ve been on a crash course as of recent17:36
TheJuliaSeems like the “right” solution, is to evaluate the list of routers, and ensure the local ports are appropriately tagged out to the ha_chassis_group so they are appropriately scheduled out.  I wonder if we can update post deploy and just handle it in some reconciliation logic17:40
TheJuliaThat at-least begins to anchor the routers into a specific chassis group which the networks would as well17:41
TheJuliaContinuity: More of a pale ale person, but the brewery which makes my favorite is now like… a bazillion miles away and only makes a batch of their best every few years which makes me super sad. *cries*17:42
ContinuityYeah I was looking at the code for that whole area, its a little twisty and I keep getting lost :D17:42
ContinuityIt also seems to update EVERY time something changes 17:43
ContinuityWhich can lead to flip flopping 17:43
TheJuliaI think what Gabriel posited works for bare metal which is just vlans, but I think that will compound as we support connecting Geneve/VXLAN to VLAN switches17:45
TheJuliaYeah, because every time it recalculates, without the anchor, i could see it trying to land everything elsewhere and things just breaking17:46
TheJuliaIt’s like the agent needs to go and say “for every lrp attached external port, regardless of type, where a vlan or vlan segment is attached, ensure we have the chassis group anchoring it17:47
TheJuliaSince those are in groups of up to 5 hosts, it can continue to figure out where and let everything map17:47
JayF"I'm [snip] an OVN expert" - Julia ;) 17:52
TheJuliaRetro17:54
TheJuliaThere is likely also an opportunity to actually look and log “broken” cases17:55
TheJuliaLike, if your router exists, it has bound ports, but not everything is online because it’s own external port is not online because it can’t be brought online, ovn localnet ports as it relates to the router never engage17:56
TheJuliaAnd then never forward traffic. Your silently broken but don’t really know it because it is not blinking in your face that there is a misconfigured or broken state17:56
TheJuliaRutro, I might be becoming an OVN expert. Please send help.17:57
ContinuityYeah im in the same place.. its scary and I dont like it 17:57
TheJuliaI’d like my self hugging jacket in purple.17:58
TheJuliaContinuity: I’m fairly certain this is something we can address in networking-baremetal because this whole thing keeps going around and around, and we can at least take enough of the information to fix or highlight the issues17:59
TheJuliacardoe: Do you have any thoughts?18:00
cardoeJayF: "I'm [snip] OVN [snip]." - Julia18:01
JayFNow I know that's a lie, because I like Julia :D 18:01
JayFshe makes sense most of the time 18:02
cardoeTheJulia: uhh bagels are tasty?18:02
cardoeSo nothing that's necessarily backportable.18:02
TheJuliaJayF: Summer says “Can confirm… some of the time”18:02
TheJuliacardoe: moved on to the doctor’s office18:03
cardoeNeutron has recently acknowledged that the way the chassis IDs were being done by Neutron is all wrong.18:03
TheJulia The bagel was before this.18:03
cardoeAnd recently is like ~1 year.18:04
TheJuliacardoe: Is that the whole bug where they were like “the two chassis levels are bonkers”18:04
cardoeMaybe?18:04
cardoeSo my rough understanding is that they should allocate the port to the HA Chassis Group and then OVN will schedule it to the Chassis and then two fields are different and unique values.18:05
cardoeAnd Helen might know more here. But that's how I've understood the behavior.18:05
cardoeThe port can even get scheduled to multiple chassis (chassi? chassises?) but that's all recorded. 18:05
TheJuliaYeah, except it gets created on the front end without the ha chassis group (afaik), and then it just gets scheduled where it thinks it can18:06
cardoeRight. Which is wrong.18:06
TheJuliaPlural of chassis is chassis. I dislike the word so very much as a result.18:06
TheJuliaI mean, its not a bad model if the creation knows where it needs to land or something reconciles it back in18:06
TheJuliaIt is one of those, it doesn’t know what it can’t possibly entirely know upfront18:07
TheJuliaThings18:07
cardoeWell you can do that but then that's two different fields.18:07
cardoeSo there were some patches to neutron to fix this a while ago in gerrit. There wasn't a lot of tests to the patches.18:09
cardoeBut they got ignored and abandoned cause they auto-abandon old stuff.18:10
cardoeBut they fix the issue.18:10
cardoeIt's being addressed in neutron proper.18:11
cardoeFor the network node getting the proper VLAN plugged up, we need to get the right event off the right chassis18:14
cardoeAnd have the port with the right device ID18:14
TheJuliaI think the event matters a bit less as long as the ports get enough data for the mapping to occur18:20
TheJuliaBut, I guess sort of a different approach model, I always lean towards fix upfront if we can18:21
opendevreviewNahian Pathan proposed openstack/ironic master: Reduce API calls when collecting sensor data with redfish  https://review.opendev.org/c/openstack/ironic/+/95548418:40
ContinuityWhen I looked at the code, it feels like it Guesses18:43
ContinuityAs to how to sync them up18:44
ContinuityThere used to be a periodic that ran to try and sync stuff up, but that seems to ahve gone away18:44
ContinuityThe strange thing, is that when a new node is added to the network, it should try and reconcile then, and sometimes it does, 18:44
ContinuityIts just very flakey18:44
TheJuliaContinuity: do you need it to be very static, or would you be okay with any appropriate member of the matching physical networks being okay?19:09
JayFdtantsur: rpittau: cardoe: iurygregory: https://review.opendev.org/c/openstack/governance/+/974930 Please -1 this to indicate your willinness to continue as a DPL liason19:24
cardoeiurygregory: you still want https://review.opendev.org/c/openstack/ironic/+/972421 ?19:29
opendevreviewVerification of a change to openstack/networking-generic-switch master failed: Migrate setup configuration to pyproject.toml format  https://review.opendev.org/c/openstack/networking-generic-switch/+/97352619:54
cardoe^ I don't even see what failed?20:17
opendevreviewMerged openstack/ironic master: make inspection doc link better in the side menu  https://review.opendev.org/c/openstack/ironic/+/97457020:38
ContinuityTheJulia: static is fine as long as it is consistent and works first time20:53
opendevreviewMerged openstack/ironic bugfix/33.0: Fix introspection failure handling  https://review.opendev.org/c/openstack/ironic/+/97407021:01
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: Change default value of some redfish firmware update config  https://review.opendev.org/c/openstack/ironic/+/97480021:08
TheJuliaContinuity: so funny thing is that static is actually harder, but it would require almost managing placement of ha chassis groups21:11
TheJuliaBut I could almost see it desirable, as in if your underlying fabric prefers/needs MACs to stay pinned21:12
opendevreviewClif Houck proposed openstack/ironic master: WIP: Update TBN simulator for vif_attach planning  https://review.opendev.org/c/openstack/ironic/+/97369122:00
opendevreviewClif Houck proposed openstack/ironic master: Add default trait behavior to _vif_attach_tbn  https://review.opendev.org/c/openstack/ironic/+/97496022:00
opendevreviewSteve Baker proposed openstack/ironic master: Add an autodetect deploy interface  https://review.opendev.org/c/openstack/ironic/+/97318722:22
opendevreviewSteve Baker proposed openstack/ironic master: Replace ironic-tempest-bios-ipmi-direct with -autodetect  https://review.opendev.org/c/openstack/ironic/+/97426022:22
opendevreviewVerification of a change to openstack/ironic master failed: Remove old -ipmitool job definitions  https://review.opendev.org/c/openstack/ironic/+/97478923:08
opendevreviewNahian Pathan proposed openstack/ironic master: Reduce API calls when collecting sensor data with redfish  https://review.opendev.org/c/openstack/ironic/+/95548423:09
cardoeShould we https://opendev.org/openstack/ironic/src/commit/cd20dbc87e3b135954367820cb231e8bfdcd8a4c/ironic/drivers/modules/redfish/inspect.py#L184 create ports in redfish inspection or remove that and defer it to the hook?23:20
opendevreviewMerged openstack/ironic bugfix/33.0: fix: port endpoints did not return vendor and category and fix docs  https://review.opendev.org/c/openstack/ironic/+/97353323:39

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!