15:00:39 <JayF> #startmeeting Ironic
15:00:39 <opendevmeet> Meeting started Mon Dec  4 15:00:39 2023 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:39 <opendevmeet> The meeting name has been set to 'ironic'
15:00:42 <iurygregory> o/
15:00:51 <JayF> #topic Announcements/Reminder
15:00:56 <JayF> Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash
15:01:18 <dtantsur> o/
15:01:19 <JayF> No action items from previous meeting, skipping agenda item
15:01:28 <JayF> #topic Carcal release schedule
15:01:35 <JayF> #link https://releases.openstack.org/caracal/schedule.html
15:01:47 <JayF> Next milestone is R-17, Caracal-2 on Jan 11, take notice :)
15:01:54 <JayF> #topic Review Ironic CI Status
15:01:56 <JayF> How is CI?
15:02:26 <iurygregory> I think is in a good shape
15:02:41 <dtantsur> my changes have passed recently
15:02:45 <iurygregory> last week we were able to merge a few things without problems from what I remember
15:03:04 <JayF> That matches my experience, nice to hear, thanks for keeping the gate clean.
15:03:04 <iurygregory> we finally moved the snmp job from focal =) tks TheJulia
15:03:13 <JayF> #topic Bug deputy
15:03:31 <JayF> Julia was bug deputy last week, and we did some cleanup of bugtrackers
15:03:38 <JayF> TheJulia: anything to report from time as bug deputy?
15:03:51 <iurygregory> Julia is not available today
15:03:54 <dtantsur> I think she mentioned she wouldn't be around
15:03:57 <JayF> ack
15:04:02 <iurygregory> yeah
15:04:05 <JayF> Hopefully it went well, is someone signed up for enxt week?
15:04:21 <iurygregory> afaik nope
15:04:28 <JayF> Aight, I'll take a swing
15:04:31 <iurygregory> you mean this week right?
15:04:34 <JayF> #action JayF to bug deputy this week
15:04:36 <JayF> yep
15:04:40 <iurygregory> gotcha!
15:04:49 <JayF> Is this bugfix branch topic leftover from last week?
15:05:01 <JayF> #topic New bugfix branches proposed
15:05:22 <iurygregory> I think rpittau was doing something related to this, but he is out today
15:05:29 <JayF> Ack; we'll table the topic
15:05:34 <JayF> #topic RFE Review
15:05:37 <JayF> dtantsur has two
15:05:45 <iurygregory> :O
15:05:45 <JayF> #link https://bugs.launchpad.net/ironic/+bug/2045548
15:05:50 <dtantsur> I do indeed
15:06:00 <JayF> dtantsur: is the real only use case for this dual stack ip?
15:06:12 <dtantsur> JayF: for me - yes
15:06:23 <JayF> dtantsur: basically I'm a little worried this is going to end up as "Ironic API HA via 'yolo try all the api servers'" rather than properly load balanced
15:06:43 <dtantsur> Isn't properly the same?
15:06:58 <JayF> oh, I modeled this backwards
15:07:06 <JayF> I was mentally thinking of this as "N" callback urls *from* IPA
15:07:13 <JayF> but this is "N" callback urls *for* IPA to callback to
15:07:50 <dtantsur> Yeah, so IPA->Ironic. My personal pain is less about HA, more about a dual-stack Ironic with a V6-only node.
15:08:03 * iurygregory hides
15:08:15 <JayF> I don't have any objection to this directly; but I also think if rpittau and TheJulia is out that we don't really have strong quorum for RFE approval
15:08:20 <iurygregory> I know your pain dtantsur
15:08:27 <JayF> if you know one or both of them are +1 or at least +0 to it, I'm OK with it
15:08:49 <dtantsur> I can ask them later on. Putting it on the meeting means that they can at least see that in the scrollback.
15:08:53 <JayF> ++
15:09:05 <dtantsur> Let's treat this as a sanity check / announcement of intentions :)
15:09:32 <JayF> #info Bug 2045548 seems ripe for RFE approval, but no strong quorum of cores at meeting. Intention is to move forward if no objection once more cores look.
15:09:36 <dtantsur> The next one is more interesting, I believe
15:10:07 <dtantsur> #link https://bugs.launchpad.net/ironic/+bug/2045551 Auto-configuration of iPXE script for inspection
15:10:24 <JayF> So there's two things that confuse me about this:
15:10:41 <JayF> 1) is it still unmanaged inspection if we're generating PXE scripts around it in Ironic? hehe
15:11:00 <JayF> 2) when we talk about fallback scripts, is that like, generating an ipxe config which would boot anything if put in the correct spot? what is the correct spot?
15:11:21 <dtantsur> The current logic in boot.ipxe is roughly this:
15:11:54 <dtantsur> for mac in macs: try_chainload(http://server/$mac/boot.ipxe); chainload(http://server/fallback.ipxe)
15:12:09 <dtantsur> So, this last item is where inspection is normally hooked into, at least for standalone deployments
15:12:42 <JayF> basically "boot any specific configs we have, boot the generic config for anything else" (which in a default config is nothing/chainload to hdd I would assume?)
15:13:01 <dtantsur> Correct
15:13:21 <dtantsur> So far, this fallback script has to be hand-written (see examples from metal3; bifrost has the same)
15:13:44 <JayF> Is there a difference from what we document today as "unmanaged inspection" and "discovery" ?
15:14:24 <dtantsur> Technically the same thing, only the intention is slightly different
15:14:54 <JayF> ack
15:15:13 <JayF> my only real weirdness about this feature
15:15:24 <JayF> is we can't really enable it on *all* conductors on a given dhcp domain, right?
15:15:34 <JayF> hmm. I guess we could, because it's just an ipxe script
15:15:43 <JayF> it's just available should the dhcp server point at it
15:15:47 <dtantsur> Right. DHCP is a different story.
15:16:02 <JayF> dhcp is [magic word] UNMANAGED in that case, right? lol
15:16:22 <dtantsur> Right :D
15:16:31 <JayF> So the feature seems OK
15:16:40 <JayF> but I kinda hate the name?
15:16:48 <JayF> I know it fits in ironic contexts, but I don't think it signals to operators what the behavior change will be
15:16:59 <dtantsur> which exactly name?
15:17:16 <JayF> so my only thing would be, lets make 100% sure it's clear that if someone flips that switch what they're going tobe getting
15:17:30 <JayF> >  [pxe]generate_ipxe_inspection_script = True/False (default False)
15:18:01 <JayF> like, I feel like we could use a word like ... "universal"
15:18:01 <dtantsur> Yep. (Also, naming suggestions welcome)
15:18:01 <JayF> or "fallback"
15:18:01 <JayF> or "global"
15:18:01 <JayF> or "all of the things"
15:18:08 <JayF> I don't have an actual good suggestion but you get the idea :D
15:18:25 <dtantsur> This thing will be inspection-specific
15:18:36 <dtantsur> It's not like a fallback to something abstract
15:18:47 <JayF> that means ... less in a future world where everything is Ironic-based :D
15:18:55 <dtantsur> :D
15:20:00 <JayF> So I think similar adjudication for this, probably OK, lets get more feedback and maybe pick a stronger name?
15:20:06 <iurygregory> we probably need a shared doc with good naming ideas lol
15:20:14 <dtantsur> :D
15:20:17 <dtantsur> JayF++
15:20:46 <JayF> #info Bug 2045551 seems ripe for RFE approval, but no strong quorum of cores at meeting. Intention is to move forward once a better config var name is chosen and if no objection once more cores look.
15:21:09 <JayF> dtantsur: please update those RFE bugs with the comments from the meeting (link to the meeting logs once we get ti closed up would be great)
15:21:18 <JayF> #topic Open Discussion
15:21:26 <JayF> Nisha_Agarwal: mallik: You have a topic here
15:21:28 <Nisha_Agarwal> Hi Jayf
15:21:32 <Nisha_Agarwal> yup
15:21:52 <mallik> Hi JayF
15:22:01 <JayF> So first of all, the storyboard issues created; please create them in launchpad. We no longer use storyboard and I'm confused as to how new issues were created there.
15:22:18 <JayF> https://bugs.launchpad.net/ironic
15:22:20 <Nisha_Agarwal> We are testing the redfish driver with ilo6
15:22:26 <Nisha_Agarwal> JayF, ok
15:22:40 <Nisha_Agarwal> there are two features where we see the issue
15:22:49 <Nisha_Agarwal> one is Event Scusbscription
15:22:58 <Nisha_Agarwal> ilo6 is on higher redfish schema
15:23:16 <Nisha_Agarwal> while ironic is using the schema where attributes are deprecated
15:24:03 * iurygregory was the one to write the event subscription via vendor passthru
15:24:24 <Nisha_Agarwal> So sushy needs to enhance the redfish schema implementation and may be intelligently derive which schema version does the server has....
15:24:52 <Nisha_Agarwal> ilo5/SuperdomeFlex280 works with current implementation of event Subscription
15:25:06 <Nisha_Agarwal> while ilo6 has higher schema implemented
15:25:13 <dtantsur> So, ilo6 removes the deprecated fields completely?
15:25:19 <iurygregory> I think we would normally add the new attributes to sushy to match the new schema and keep backward compatibility I think
15:25:23 <Nisha_Agarwal> Looks like
15:25:49 <Nisha_Agarwal> iurygregory, yeah that should solve the issue....
15:25:50 <dtantsur> Sigh, okay. Patches are welcome if you have cycles (esp. since you can test on both hardware versions)
15:26:13 <Nisha_Agarwal> dtantsur, yup. We are working on the patches
15:26:27 <iurygregory> Nisha_Agarwal,  would also need to update the vendor passthru logic in ironic after sushy is ready
15:26:47 <Nisha_Agarwal> iurygregory, ok.
15:27:13 <Nisha_Agarwal> We will test end-to-end using ironic
15:27:26 <iurygregory> nice!
15:27:30 <Nisha_Agarwal> Secondly on out-Of-Band Raid
15:27:51 <Nisha_Agarwal> Ironic implements two parameters "VolumeType" and "Encrypted"
15:28:03 <Nisha_Agarwal> in addition to other parameters
15:28:31 <Nisha_Agarwal> VolumeType is deprecated and is not present in ilo6
15:29:07 <dtantsur> masghar: ^^^
15:29:30 <dtantsur> Nisha_Agarwal: what do you use instead for the RAID level?
15:30:25 <Nisha_Agarwal> "Encrypted" is not deprecated, but iLO6 has cautiouslu not impelmented it. The Storage Distinguished Technoligist has told that very few vendors have impelmented Encrypted
15:30:37 <Nisha_Agarwal> cautiously^^^
15:30:49 <Nisha_Agarwal> dtantsur, it is deprecated in favor of "RAIDType"
15:30:50 <dtantsur> It seems that we unconditionally send Encrypted:False. Not sure why, but I suspect we can just remove that.
15:31:23 <dtantsur> Ah. And we already pass RAIDType too. Again, I'm curious why we did that.
15:31:28 <Nisha_Agarwal> so i believe it should work with that payload. But i need to crosscheck with the engineer who is working on it
15:32:03 <masghar> Redfish driver, right? (just confirming)
15:32:18 <Nisha_Agarwal> Yes
15:32:19 <iurygregory> masghar, correct
15:32:31 <dtantsur> masghar: can you check if your Dell machine has RAIDType?
15:32:34 <Nisha_Agarwal> if we remove these two parameters the creation of RAID works
15:32:45 <masghar> Let me see
15:33:46 <Nisha_Agarwal> So we wante dto know if we could add a condition in redfish driver (sushy) specific to Proliants?
15:33:59 <Nisha_Agarwal> that would be allowed right?
15:34:33 <iurygregory> this wouldn't be a problem I would say
15:34:34 <dtantsur> Nisha_Agarwal: this is not immediately disallowed, but for what purpose?
15:34:50 <dtantsur> if VolumeType is deprecated and Encrypted is unused, we just drop them.
15:34:59 <JayF> Yeah, that is kinda a last resort if we can't use non-vendor-specific ways to figure out the right thing to do
15:35:11 <Nisha_Agarwal> dtantsur, ok.
15:35:36 <Nisha_Agarwal> there is one power bug also which was raised on ilo5. But it applies to ilo6 as well
15:36:04 <Nisha_Agarwal> that was raised by Gresearch i guess
15:36:13 <iurygregory> https://bugs.launchpad.net/sushy/+bug/2016307 ?
15:36:13 <JayF> Yeah, it's already been fixed, and the backport is back pretty far already
15:36:17 <Nisha_Agarwal> there it requires vendor specific check probably in sushy
15:36:45 <Nisha_Agarwal> iurygregory, not this
15:36:59 <iurygregory> oh ok, I've found this one when looking in launchpad
15:37:13 <JayF> https://opendev.org/openstack/sushy/commit/192d897ce97120db98aca6d6bb840c4243a4b5b3 is the fix I was thinking of
15:37:29 <Nisha_Agarwal> https://bugs.launchpad.net/ironic/+bug/2021995
15:37:40 <Nisha_Agarwal> yes the fix done above was partial
15:37:47 <Nisha_Agarwal> and doesnt solve the above issue
15:38:02 <Nisha_Agarwal> i had done a workaround fix in proliantutils for ilo5
15:38:33 <Nisha_Agarwal> but it still requires a small fix in sushy to have that workaround available for ilo6 as well
15:38:42 <Nisha_Agarwal> and probably for other vendors too
15:39:12 <Nisha_Agarwal> this may require vendor specific check is what i wante dto put forward
15:39:48 <JayF> Stuff like that is best evaluated in gerrit on the patch IMO
15:39:52 <dtantsur> I'd have to see the patch to judge
15:39:53 <dtantsur> yeah
15:39:58 <Nisha_Agarwal> dtantsur, ok
15:40:12 <JayF> Is there anythign else to discuss on this topic?
15:40:44 <Nisha_Agarwal> JayF, one more point about the mail thread on driver deprecation
15:40:50 <JayF> sure
15:41:48 <Nisha_Agarwal> the spec raised for this release item lists ilo driver under the heading "Drivers to be removed"
15:42:20 <Nisha_Agarwal> which looked incorrect and may discourage customers to use ilo/ilo5 driver for gen9/gen10
15:42:29 <JayF> so two things, firstly, it doesn't say that
15:42:31 <Nisha_Agarwal> This was a concern raised by my management
15:42:40 <JayF> it says: "Drivers to be marked for removal"
15:42:53 <Nisha_Agarwal> so just putting forward the concern
15:42:55 <JayF> also in the paragraph above, it says: "We do not intend to actively remove any of these drivers until it is clear any hardware they exclusively support has gone end of life. We are primarily taking this action to indicate to operators that they should be provisioning new hardware with Redfish-based drivers."
15:42:57 <Nisha_Agarwal> yup
15:43:10 <JayF> So, the context behind why we worded it this way, and put it so strongly
15:43:18 <JayF> and there's willingness to change it as long as it achieves this goal
15:43:31 <JayF> is we've seen the branded-drivers be very attractive to new operators of ironic
15:43:49 <JayF> so we'll have users choose to use an ilo driver on their ilo6, or a idrac{-wsman} driver on a newer dell
15:44:06 <JayF> even though for those devices, a redfish driver is a better choice, because they've heard of "idrac" or "ilo" but have never heard of "redfish"
15:44:19 <Nisha_Agarwal> hmm
15:44:33 <Nisha_Agarwal> ok
15:44:44 <JayF> I'll note that for many drivers, we already have it printing a deprecation message -- we didn't put that in for iLo
15:44:55 <Nisha_Agarwal> Thanks JayF that answers the query :) yes thanks for that
15:45:02 <JayF> and I thought it'd be worse for you all to leave you off that list entirely; I don't want HPE users to be completely lost on what driver to use
15:45:09 <Nisha_Agarwal> i saw the mail, but still i wanted to put forward the message
15:45:22 <JayF> so if there's suggestions on how to reword that, I'm happy to take edits to that spec, nothign is ever set in stone
15:45:31 <JayF> but the goal was to funnel new ironic operators to redfish driver by default
15:45:35 <Nisha_Agarwal> JayF, that should be great
15:45:58 <Nisha_Agarwal> I will check with management and reword if they are not ok with what you explained above
15:46:13 <Nisha_Agarwal> Thanks JayF
15:46:19 <JayF> no problem o/
15:46:23 <JayF> Anything else for open discsusion?
15:46:26 <Nisha_Agarwal> thats all
15:46:32 <iurygregory> I have one
15:46:43 <JayF> What's up iurygregory
15:47:02 <iurygregory> it's more a request than a discussion LOL
15:47:48 <iurygregory> related to how we could speed-up multipath checking in IPA I've created https://review.opendev.org/c/openstack/ironic-python-agent/+/902012
15:48:28 <JayF> Gotta be honest, I've not looked at that much because I don't have very much experience with multipath IO
15:48:40 <JayF> so I'm a little scared I'd approve something that'd break the underlying tech unknowingly
15:48:43 <iurygregory> is just my initial idea in the code, the commit explains what I'm trying to do to speed-up
15:49:07 <iurygregory> yeah, so if anyone is interested please look and provide your thoughts in the patch \o/
15:49:15 <JayF> #link https://review.opendev.org/c/openstack/ironic-python-agent/+/902012
15:49:27 <JayF> I'll look, but as I said, take any comments I put on it as advisory
15:49:37 <iurygregory> JayF, sure =)
15:49:39 <JayF> I'm not sure I've ever, once, been on a machine with multipath remote IO
15:50:12 <JayF> Anything else for Open Discussion?
15:50:12 <iurygregory> I don't have one, this is just by looking at ironic logs and looking at our logic for multipath
15:50:19 <JayF> makes sense
15:50:47 <iurygregory> tks!
15:51:46 <JayF> Last call?
15:53:04 <JayF> ty all o/
15:53:06 <JayF> #endmeeting