| *** dtantsur_ is now known as dtantsur | 02:24 | |
| jssfr | Huh. https://review.opendev.org/c/openstack/ironic-python-agent/+/925087 failed in Zuul, but running `tox` finds no fails on my local machine. Is there anything Zuul does differently so I can reproduce the issue? | 06:53 |
|---|---|---|
| rpittau | good morning ironic! o/ | 06:58 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/24.0: [bugfix only] Remove deleted lextudio packages https://review.opendev.org/c/openstack/ironic/+/926119 | 07:21 |
| iurygregory | good morning ironic | 10:45 |
| cid | I have updated ironic's wiki/meetings page with my bug deputy report, please review for correctness and contained report details ahead of the weekly meeting. | 12:14 |
| cid | https://wiki.openstack.org/wiki/Meetings/Ironic | 12:14 |
| iurygregory | https://bugs.launchpad.net/ironic/+bug/2076265 I do think it make sense to set importance high .. | 12:35 |
| iurygregory | https://bugs.launchpad.net/metalsmith/+bug/2076344 I would leave to metalsmith experts... but it makes sense to me, I would mark as low | 12:36 |
| iurygregory | yay for a new implementation on hardware vendor side https://bugs.launchpad.net/sushy/+bug/2075979 ... | 12:37 |
| iurygregory | or at least is seems like a new implementation on the first glance, I need more coffee before looking at bugs XD | 12:43 |
| rpittau | wasn't that already addressed ? | 12:44 |
| rpittau | oh no, nvm | 12:45 |
| rpittau | it could be an issue on the dell firmware | 12:46 |
| rpittau | cid: thanks for the report! | 12:47 |
| rpittau | cid: in https://bugs.launchpad.net/metalsmith/+bug/2076344 it mentions metalsmith 1.4.4 but the latest is 2.3.0 | 12:48 |
| rpittau | 1.4.4 is 2 years old | 12:48 |
| cid | rpittau, Hmm, so likely already fixed. How about people still using that version though | 12:51 |
| rpittau | that's a great question, 1.4.4 is from wallaby which is currently unmaintained, they should not use that | 12:52 |
| cid | ++, should I add a comment "to upgrade to a later version" in the bug? | 12:54 |
| cid | Or just leave it | 12:54 |
| rpittau | the minimum supported version is 1.10.1 which is from stable/2023.1 | 12:54 |
| rpittau | cid: yeah, suggest to use a more recent version to see if the bug is still present | 12:54 |
| rpittau | thanks! | 12:54 |
| cid | rpittau, tks | 12:55 |
| iurygregory | rpittau, yeah, I'm thinking about a firmware issue also =( | 13:00 |
| rpittau | cid: if you quickly update https://review.opendev.org/c/openstack/ironic/+/926082 I guess we can merge it as quickly :) | 13:07 |
| cid | rpittau: on it | 13:08 |
| opendevreview | cid proposed openstack/ironic master: Update error message https://review.opendev.org/c/openstack/ironic/+/926082 | 13:10 |
| TheJulia | good morning | 13:14 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-lib master: Fix invalid UTF-8 characters in execute output https://review.opendev.org/c/openstack/ironic-lib/+/926045 | 13:55 |
| rpittau | still unsure if we should ignore or log the invalid characters , reviews welcome | 13:56 |
| rpittau | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Aug 12 15:00:15 2024 UTC and is due to finish in 60 minutes. The chair is rpittau. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | o/ | 15:00 |
| dtantsur | o/ | 15:00 |
| rpittau | Hello everyone! | 15:00 |
| rpittau | Welcome to our weekly meeting! | 15:00 |
| rpittau | The meeting agenda can be found here: | 15:00 |
| rpittau | #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_August_12.2C_2024 | 15:00 |
| TheJulia | o/ | 15:00 |
| JayF | o/ | 15:00 |
| rpittau | #topic Announcements/Reminders | 15:01 |
| rpittau | #info Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio | 15:01 |
| rpittau | #link https://tinyurl.com/ironic-weekly-prio-dash | 15:01 |
| cardoe | o/ | 15:01 |
| rpittau | it's not looking bad, lots of new patches, some from last week | 15:01 |
| masghar | o/ | 15:02 |
| rpittau | #info 2024.2 Dalmatian Release Schedule | 15:02 |
| rpittau | #link https://releases.openstack.org/dalmatian/schedule.html | 15:02 |
| rpittau | we're at week -7 ! | 15:02 |
| cid | o/ | 15:02 |
| rpittau | we hsould start thinking about what's missing in the non-client and client libraries | 15:02 |
| rpittau | FF is in 2 weeks also | 15:02 |
| rpittau | #info PTL nominations period between August 14 and August 28 | 15:03 |
| rpittau | I have decided to run for a second mandate as PTL | 15:03 |
| iurygregory | nice =D | 15:03 |
| rpittau | :) | 15:03 |
| rpittau | sorry for the late notice, just life in the middle of everything, it's been a long summer and it hasn't ended yet! | 15:04 |
| JayF | Thanks Riccardo! | 15:04 |
| TheJulia | Awesome, thanks! | 15:04 |
| rpittau | my pleasure, really :) | 15:04 |
| rpittau | I'm glad I can do it | 15:05 |
| rpittau | #info the next OpenInfra PTG which will take place October 21-25, 2024 virtually! Registration is now open! | 15:05 |
| rpittau | #link https://ptg.openinfra.dev/ | 15:05 |
| rpittau | the ironic team has been registered | 15:05 |
| rpittau | please add your name and topics here | 15:05 |
| rpittau | https://etherpad.opendev.org/p/ironic-ptg-october-2024 | 15:05 |
| rpittau | we still have some time for topics | 15:06 |
| rpittau | anything else to announce/remind ? | 15:07 |
| rpittau | okey dokey, onward! | 15:07 |
| rpittau | #topic Review Ironic CI status | 15:08 |
| rpittau | anything worth mentioning from last week? | 15:08 |
| rpittau | I've only seen some instability on pkgs repos, but it recovered pretty quickly | 15:08 |
| rpittau | alright, great week for CI :D | 15:09 |
| rpittau | #topic Discussions | 15:09 |
| rpittau | nothing to discuss, unless anyone has anything to mention? | 15:09 |
| JayF | Is someone making sure old bugfix branches still get retired? | 15:10 |
| rpittau | JayF: I am :) | 15:10 |
| JayF | Awesome, thanks :) | 15:10 |
| rpittau | np! :) | 15:10 |
| rpittau | actually maybe I should write something down about the procedure, I'll take a note on that | 15:11 |
| TheJulia | ++ | 15:11 |
| rpittau | ok! | 15:11 |
| rpittau | anything else? | 15:11 |
| masghar | I would like to mention something | 15:12 |
| rpittau | masghar: please go ahead :) | 15:12 |
| masghar | Unfortunately I have not been able to complete the inspection rules work so far | 15:12 |
| masghar | (Have been a little overwhelmed with bugs and such downstream) | 15:12 |
| TheJulia | No worries, it happens :) | 15:13 |
| masghar | (Just a head's up) | 15:13 |
| rpittau | no worries masghar :) | 15:13 |
| masghar | Thanks, will try to carve out time for it | 15:13 |
| rpittau | thanks for the headsup | 15:13 |
| masghar | No problem, and thanks | 15:14 |
| masghar | (Thats it from me) | 15:14 |
| JayF | masghar: If we can document, in detail, what's done and what's left to be done, I think cid is willing to help pickup some of that work. I'm unsure if the knowledge transfer is worth it :) | 15:14 |
| JayF | masghar: so know that's an option at hand if you'd like to exercise it | 15:15 |
| masghar | I have a very tiny patch that I started a few months ago | 15:15 |
| masghar | I can explain my thought process to cid or whoever asks | 15:15 |
| masghar | I appreciate the offer of help :) | 15:16 |
| cid | masghar, JayF, In my todo for the week | 15:16 |
| rpittau | thanks cid :) | 15:17 |
| masghar | Thanks cid! | 15:17 |
| cid | nop! | 15:17 |
| * cid that should be no problems :) | 15:18 | |
| rpittau | anything else? otherwise I have one quick thing | 15:18 |
| rpittau | ok | 15:19 |
| rpittau | I forgot to mention that it's almost time for the highlights | 15:19 |
| rpittau | I will take care of them, at least start them, I'll be out at the beginning of Seprember when they're due, but I'll do my best to finish them before I leave for my PTO | 15:19 |
| rpittau | if you have anything you want to mention about this development cycle please let me know | 15:19 |
| iurygregory | ack | 15:20 |
| rpittau | alright, moving on | 15:20 |
| rpittau | #topic Bug Deputy Updates | 15:20 |
| rpittau | cid: thanks for taking car of that, anything to mention? | 15:21 |
| cid | So, I needed help triaging 4 bugs, I think two can be considered done, except these other two: | 15:21 |
| cid | https://bugs.launchpad.net/sushy/+bug/2075979 | 15:21 |
| cid | https://bugs.launchpad.net/sushy/+bug/2075980 | 15:21 |
| cardoe | So the first one is mine. | 15:21 |
| TheJulia | I suspect yours is a valid but, but I'm having trouble understanding exactly what, the partition, in the api is | 15:22 |
| cardoe | Essentially sushy does not see those NICs. Dell says they conform to Redfish, etc. | 15:22 |
| rpittau | cardoe: which version of iDRAC9 firmware are you using ? cause that could be a firmware issue | 15:22 |
| cardoe | It doesn't matter. It's the same for at least half a dozen versions. Both 6.x and 7.x | 15:23 |
| rpittau | ok, that's what I wanted to exclude :) | 15:23 |
| TheJulia | its sort of a bit of a framing issue, and my comment sort of reflect this, we need to understand how to frame it and I think we're missing context. | 15:23 |
| cardoe | Yeah I'm sure I'm not providing the right details. | 15:24 |
| TheJulia | Well, your providing what you have :) | 15:24 |
| cardoe | So basically when ironic calls `ethernet_interfaces.summary` on Sushy. Those NICs are excluded. | 15:24 |
| TheJulia | I guess sushy needs to understand "which one is actually important" | 15:25 |
| TheJulia | and identify when to check/reconcile them together | 15:25 |
| cardoe | So the ramdisk inspector sees NIC.Slot.1-1 as ens2f0np0 for example. Which would imply NIC.Slot.1-1-1 | 15:27 |
| dtantsur | I see a big problem in that there is no link between the two EthernetInterface objects | 15:27 |
| cardoe | But then the other port on that card is NIC.Slot.1-2 for example... it's called ens2f1np1 which implies NIC.Slot.1-2-2 | 15:28 |
| TheJulia | dtantsur: ... that is indeed a huge issue | 15:29 |
| dtantsur | A workaround would be to ignore the Health record if its values are null (as opposed to unhealthy) | 15:29 |
| TheJulia | but, 1-1-1 is a noted partition, but still sort of goes back to what is the partition in the context | 15:29 |
| cardoe | I'm gonna get someone from Dell's firmware team on a call. | 15:29 |
| TheJulia | +++++++ | 15:29 |
| dtantsur | ++ | 15:30 |
| dtantsur | "InterfaceEnabled": false is very concerning (but we don't look at it.. yet?) | 15:30 |
| cardoe | But I did want to at least bubble this up to you guys and see if I could come up with an acceptable way to map them and put that in sushy-oem-idrac or something. | 15:30 |
| TheJulia | I guess that explains why it fails | 15:30 |
| TheJulia | the partition has no mac | 15:30 |
| cardoe | Right | 15:30 |
| TheJulia | so more info needed, I guess next item? | 15:32 |
| cardoe | So another weird wrinkle | 15:32 |
| cardoe | When I set the devices as PXE or HTTPS bootable from Dell's HTTP UI | 15:32 |
| cid | Next item is: https://bugs.launchpad.net/sushy/+bug/2075980 | 15:32 |
| cid | Then, one last one from TheJulia: https://bugs.launchpad.net/ironic-python-agent/+bug/2076367 (sounded like something that needs to be discussed, so...)https://bugs.launchpad.net/sushy/+bug/2075980https://bugs.launchpad.net/sushy/+bug/2075980 | 15:32 |
| cardoe | The values that Ironic pulls out from the BIOS are "NIC.Slot.1-1" and "NIC.Embedded.1-1-1" | 15:33 |
| dtantsur | Ahh ohh. 2075980 is a long standing pain point | 15:33 |
| cardoe | Yeah 2075980 is mine as well. Happy to write patches. | 15:33 |
| dtantsur | cardoe: what's your timezone? I'm happy to chat about the potential fix when I'm not boiling alive in this bloody weather. | 15:34 |
| JayF | Is there something sneakily not straightforward about 2075980? | 15:34 |
| cardoe | I'm CST (or is it CDT right now?) | 15:34 |
| dtantsur | JayF: yes, figuring out what do use instead of IPA to detect the finish of the operation and the subsequent reboot. | 15:34 |
| dtantsur | it could be simple, but we need to take a careful look (and ideally involve janders and iurygregory) | 15:35 |
| JayF | why is "flip the power on and wait" just like we do for going ACTIVE on deployment not sufficient? | 15:36 |
| dtantsur | JayF: wait for what? | 15:36 |
| cardoe | So when I locally patched it, Ironic got mad that the node went away for a while. | 15:36 |
| TheJulia | power doesn't even necessarily need to be on. | 15:36 |
| JayF | cardoe just pointed at the thing I was trying to see | 15:36 |
| dtantsur | Yeah, during BIOS settings or firmware updates, the machine is doing $weird_things for several minutes | 15:36 |
| iurygregory | yeah | 15:36 |
| iurygregory | in firmware updates the bmc will be unresponsive for some time also | 15:37 |
| cardoe | Ironic was happy knowing that it would come back to the IPA after a while. | 15:37 |
| iurygregory | at least with iDRAC the UI goes down XD | 15:37 |
| dtantsur | yeah, we cannot even be sure the BMC behaves during the process | 15:37 |
| dtantsur | so it's "wait for $something and retry if the BMC is not reachable or returns HTTP 5xx) | 15:38 |
| TheJulia | I think we're going down a rabbit hole | 15:38 |
| dtantsur | PTG topic? :) | 15:38 |
| TheJulia | but maybe disjoint it from the overall issue | 15:38 |
| rpittau | sounds like it :) | 15:38 |
| TheJulia | Yes, there are issues, but that is not blocking to trying to fix cardoe | 15:38 |
| TheJulia | or at least, cardoe and his efforts are input into that discussion | 15:39 |
| TheJulia | so we shouldn't inadvertently block | 15:39 |
| cardoe | So if I just updated 1 setting and let it expect to come back after a while that was fine. | 15:39 |
| JayF | my question of "where is the hard here" has been more than sufficiently answered :D | 15:39 |
| cardoe | But you can't chain 2 clean/service steps | 15:40 |
| TheJulia | cardoe: quite possibly, but you'll need to provide input to the discussions :) | 15:40 |
| cardoe | Kick the bug back to me to write up more details when running with a patched Ironic allowing that behavior | 15:40 |
| cardoe | Fair? | 15:40 |
| iurygregory | i'm a bit confused by "you can't chain 2 clean/service steps" .-. | 15:41 |
| dtantsur | yeah, let's start with that, and maybe let's have a high-bandwidth discussion afterwards | 15:41 |
| TheJulia | iurygregory: i think it means, the second step might fail | 15:41 |
| iurygregory | TheJulia, oh ok! | 15:41 |
| TheJulia | iurygregory: because we don't have a complete understanding | 15:41 |
| JayF | Yeah it sounds like the workaround is setting cleaning timeout so high it just works | 15:41 |
| JayF | I've seen similar workaround for in-band steps that rebooted outta band of ironic | 15:42 |
| rpittau | or it doesn't :) | 15:42 |
| rpittau | reboot time is so unpredictable with bios/bmc updates | 15:42 |
| iurygregory | well, I had a funny iLO bug when doing firmware update (but only happened when doing between two specific versions) | 15:42 |
| dtantsur | I think TheJulia has rightfully hinted that we should collect more information and thoughts before coming up with a solution :) | 15:43 |
| iurygregory | even increasing the timeout the node went to clean failed because it failed to power on after reboot | 15:43 |
| rpittau | yes, I agree | 15:43 |
| TheJulia | dtantsur: bingo | 15:43 |
| iurygregory | ++ | 15:43 |
| TheJulia | onward? | 15:45 |
| rpittau | yeah | 15:45 |
| rpittau | any other bug to check ? | 15:45 |
| cid | ++ | 15:45 |
| rpittau | otherwise I think we're good for today :) | 15:45 |
| rpittau | oh wait | 15:45 |
| rpittau | any volunteer for the bug deputy this week ? | 15:46 |
| cid | Happy to do it again | 15:46 |
| cid | This could be a topic for another day: | 15:46 |
| cid | https://bugs.launchpad.net/ironic-python-agent/+bug/2076367 | 15:46 |
| cid | And my observation is worthy of note too: A non-core bug deputy might need to be able to revert the status of a bug that shows as 'In Progress' when the assignee has abandoned it. | 15:46 |
| rpittau | right | 15:47 |
| JayF | I am surprised the ironic-drivers group we added you to doesn't have that ability | 15:47 |
| JayF | cid: I'd say 2076367 is Low | 15:47 |
| dtantsur | JayF: me too | 15:47 |
| TheJulia | Yeah, there is an opportunity to make IPA a little smarter there, but definitely low priority | 15:47 |
| JayF | It's behavior that's existed for years, that's mildly annoying but the real price paid is minimal (5 seconds?) | 15:47 |
| TheJulia | eh... 20+ locally | 15:48 |
| TheJulia | at least, it feels like 20+ | 15:48 |
| TheJulia | in a VM | 15:48 |
| JayF | it's a trivial enough fix in any event ... if os.path.exists() | 15:48 |
| JayF | (on the various ipmi device locations) | 15:48 |
| rpittau | looks like a low priority indeed, and a quick fix | 15:48 |
| * cid Updated 2076367 bug's important. | 15:49 | |
| rpittau | cid: thanks for volunteering, again! :D | 15:49 |
| rpittau | I forgot one more thing! | 15:50 |
| rpittau | I will be out next monday, so someone will have to take care of the meeting and meeting notes, please :) | 15:50 |
| cid | no p | 15:50 |
| JayF | I can run it if you want | 15:51 |
| rpittau | thanks JayF, much appreciated | 15:51 |
| rpittau | alright, I think that's it for today | 15:52 |
| rpittau | thanks everyone! | 15:52 |
| rpittau | #endmeeting | 15:52 |
| opendevmeet | Meeting ended Mon Aug 12 15:52:23 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:52 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2024/ironic.2024-08-12-15.00.html | 15:52 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2024/ironic.2024-08-12-15.00.txt | 15:52 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2024/ironic.2024-08-12-15.00.log.html | 15:52 |
| opendevreview | cid proposed openstack/ironic master: Follow up to the runbooks change (#922142) https://review.opendev.org/c/openstack/ironic/+/925917 | 15:53 |
| opendevreview | cid proposed openstack/ironic master: Follow up to the runbooks change (#922142) https://review.opendev.org/c/openstack/ironic/+/925917 | 16:02 |
| cardoe | Anything we can do to nudge some of the PRs along? A number from the weekly tag are just waiting on workflows. | 16:04 |
| TheJulia | I'm heads down the next couple days, I can devote time to reviewing later this week | 16:07 |
| JayF | I'm in a similar boat, but given I have a meeting at EOD I'll dedicate everything after that to reviewing, might get some of them landed if it's not already reviewed by me. | 16:10 |
| rpittau | good night! o/ | 16:26 |
| opendevreview | Merged openstack/ironic master: DevStack: enable the new in-band inspection by default https://review.opendev.org/c/openstack/ironic/+/925688 | 17:01 |
| cardoe | There's no "baremetal node eject" like command to let me just nuke a node from the DB even if its in "active" state? | 18:12 |
| cardoe | Or like delete this node but don't actually tear it down? | 18:12 |
| cardoe | Also if I told you Dell just told me... "try this rando firmware blob as an iDRAC update" | 18:13 |
| JayF | "delete" removes a node from Ironic management entirely | 18:16 |
| JayF | "undeploy" deprovisions a node and initiates automated cleaning. There are tricks for disabling automated cleaning per node, but it's one of those situations where -- usually -- if you think you need it, there's *probably* a more appropriate path to get where you're going. | 18:16 |
| cardoe | Definitely not a feature I'm looking for. More along the lines of testing. | 18:59 |
| TheJulia | testing wise, just to rip a node out people just set maintenance and then delete a node | 19:03 |
| cid | o/ | 19:05 |
| cardoe | Thank you TheJulia | 20:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!