15:00:13 <JayF> #startmeeting Ironic
15:00:13 <opendevmeet> Meeting started Mon May 19 15:00:13 2025 UTC and is due to finish in 60 minutes.  The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:13 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:13 <opendevmeet> The meeting name has been set to 'ironic'
15:00:19 <dtantsur> o/
15:00:20 <iurygregory> o/
15:00:22 <JayF> Welcome to the weekly Ironic meeting, I'm Jay, I'll be your host today :D
15:00:23 <kubajj> o/
15:00:24 <rpittau> o/
15:00:30 <frickler> \o
15:00:34 <cid> o/
15:00:34 <JayF> As always our meetings are operated under the OpenInfra Code of conduct
15:00:41 <JayF> #note  Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash
15:01:00 <JayF> #note It's R-19 week! 2025.2 Flamingo Release Schedule https://releases.openstack.org/flamingo/schedule.html
15:01:14 <JayF> Honestly this is usually the point where I'd say we'll wait for quorum
15:01:19 <JayF> but I think we're already loaded in :D
15:01:29 <JayF> #topic Working Group Updates: Standalone Networking
15:01:31 <masghar> (o/)
15:01:36 <JayF> Any update on Standalone Networking work?
15:01:52 <dtantsur> not from me. alegacy?
15:03:09 <JayF> I'm going to assume there's nothing to update; we can revisit if folks come around later.
15:03:19 <JayF> #topic Discussion Topics: Phasing off Python 3.9 in Tests
15:03:26 <JayF> There are several related PRs linked on this topic
15:03:38 <JayF> #link https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/950152 use Python 3.12 in DIB CS9 image
15:03:51 <JayF> #link https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/950235 remove tinycore tests (does not support Python > 3.9)
15:04:03 <JayF> #link https://review.opendev.org/c/openstack/bifrost/+/949861 pinning UCs in bifrost CS9 jobs
15:04:17 <rpittau> yeah they just represent what's currently happening
15:04:17 <rpittau> python 3.9 support in UCs is not going to last for long
15:04:29 <JayF> It seems like, at least to me, there's general consensus we're likely better off removing our need for a radically tiny ramdisk than trying to chase tinyipa or a replacement forever
15:04:43 <JayF> Now would be a great time to make noise if you don't agree :)
15:04:53 <rpittau> it is really a huge effort to use anything higher than pythgon 3. 9 in tinycore at the moment
15:05:19 <JayF> My research into using gentoo could get us a smaller one, but not smaller enough to prevent our need to rework the tests
15:05:20 <dtantsur> Is there any official position by the TCL team?
15:05:41 <rpittau> meaning that you need to compile it yourself or use a tool like pyenv, which is still a big effort due to the chroot spaghetti stuff
15:06:02 <JayF> https://forum.tinycorelinux.net/index.php/board,31.0.html they appear to release approximately once a year
15:06:12 <JayF> and the most recent update was 4/26/2025 with no reference to python
15:06:16 <rpittau> yep
15:06:25 <JayF> I'll note I also don't know tinycore well enough to know if that is separate
15:06:33 <rpittau> fun fact: python 3.12 for tinycore exists only for ARM
15:06:43 <JayF> we can run tests under arm :P
15:06:58 <rpittau> me shrugs
15:06:59 <JayF> I prefer the approach of getting multinode working and trying to rework our tests to use realistic ramdisks
15:07:04 <JayF> it moves us closer to what our customers use
15:07:08 <rpittau> sounds great to me
15:07:19 <JayF> gets us into the business of making our software/CI better rather than chasing a distro which I think none of us love :)
15:07:40 <rpittau> I used to love it, now it's yet another Stockholm syndrome I have :D
15:08:00 <JayF> It's (seemingly) one  of those things that's really cool as a 1 time project; really painful to maintain for years
15:08:14 <rpittau> I agree
15:08:44 <JayF> So is there anything else to talk about around the discussion topic of python 3.9? I'll give a couple minutes
15:11:01 <JayF> Aight, next topic
15:11:05 <JayF> #topic Bug Deputy Updates
15:11:12 <JayF> 1 new bug: https://bugs.launchpad.net/ironic/+bug/2110916
15:11:15 <iurygregory> o/
15:11:20 <JayF> 1 new RFE: https://bugs.launchpad.net/ironic/+bug/2110694
15:11:29 <JayF> 3 bugs closed, 3 triaged
15:11:39 <JayF> Whoever was the bug deputy feel free to speak up with more detail :)
15:11:45 <iurygregory> it was me
15:11:46 <JayF> Also we need a bug deputy for next week
15:11:52 <JayF> \p
15:12:13 <iurygregory> so last wed Michael Sherman (not sure about his irc nick) created this rfe
15:12:44 <JayF> The RFE seems sensible, I just wanna make sure it's unique :D
15:13:10 <JayF> Yeah the real hard part of the RFE is around handing credentials and uploading the image to glance
15:13:15 <iurygregory> based on the details, seems like we retired, so maybe he wants to work on it?
15:13:31 <iurygregory> wondering how we should reach out to him about it
15:13:34 <JayF> I am +1 to the feature existing, a little nervous about someone implementing it specless
15:13:47 <iurygregory> agree
15:14:21 <iurygregory> I can add the needs-spec to it
15:14:45 <iurygregory> and mention we briefly talked at the weekly meeting
15:14:47 <dtantsur> Yeah, especially since there is a prior art already
15:15:15 <TheJulia> o/
15:15:19 <TheJulia> sorry, got super distracted this morning
15:15:26 <iurygregory> np TheJulia o/
15:15:35 <JayF> Sounds like needs-spec is the answer, and I think he was already down that path.
15:15:40 <JayF> \o
15:15:50 <frickler> nick should be shermanm[m]
15:16:01 <JayF> Do we have someone who wants to bug deputy this week?
15:16:15 <TheJulia> I can watch the bugs this week
15:16:17 <JayF> I'll note for US-ians: many of us will have Monday off next week (memorial day). So take this into consideration when volunteering
15:16:21 <iurygregory> frickler, ack
15:16:31 <JayF> Skipping RFE review, we just kinda did that in the bug deputy section
15:16:42 <JayF> #note Next bug deputy: Julia
15:16:45 <JayF> #topic Open Discussion
15:16:59 <JayF> re: next week's Monday being Memorial Day in the US, would we like to keep the meeting?
15:17:02 <JayF> I will not be here.
15:17:35 <TheJulia> I should not be here next monday :)
15:17:44 <TheJulia> I may take Friday off as well, who knows!
15:17:59 <JayF> I'd be +1 to just cancelilng next week
15:18:33 <rpittau> fine to me
15:18:33 <dtantsur> I'll be here but I'm fine with cancelling
15:18:42 <iurygregory> ++
15:19:02 <opendevreview> Julia Kreger proposed openstack/ironic master: DNM: CI Science - Expand the multinode job  https://review.opendev.org/c/openstack/ironic/+/950206
15:19:07 <iurygregory> JayF, you want to send an email or should I?
15:19:08 <JayF> #note Meeting next week cancelled due to US Memorial Day holiday. Next meeting June 2 (if my math is right :D)
15:19:08 <cid> cancel, ++
15:19:12 <JayF> iurygregory: I'll do it
15:19:17 <iurygregory> ack
15:19:18 <JayF> Anything else for Open Discussion?
15:20:20 <rpittau> CI related
15:20:20 <rpittau> something odd with metal3 jobs
15:20:42 <rpittau> not sure if anyeone else noticed but there are auth issues with sushy
15:21:04 <rpittau> like 2025-05-19 13:20:52.997 1 WARNING sushy.connector [None req-f7d87eb6-999d-4248-a780-1a044a68911a - - - - - -] Session authentication appears to have been lost at some point in time. Connectivity may have been lost during a prior session refresh. Attempting to re-authenticate.: sushy.exceptions.AccessError: HTTP POST https://192.168.111.1:8000/redfish/v1/SessionService/Sessions returned code 401. Base.1.0.GeneralError:
15:21:04 <rpittau> Authorization required Extended information: [{'@odata.type': '/redfish/v1/$metadata#Message.1.0.0.Message', 'MessageId': 'Base.1.0.GeneralError'}]
15:21:39 <rpittau> please check the other jobs as well
15:21:39 <rpittau> just want to exclude it's Python 3.12 only
15:22:13 <rpittau> if anyone wants to have a look see the ironic logs here https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ed/openstack/3ed3085139af4072b86cd6c990706577/controller/before_pivoting/ironic.log
15:23:00 <TheJulia> That is a super weird edge to fall into to get that error
15:23:16 <TheJulia> :(
15:23:34 <rpittau> yeah
15:23:34 <rpittau> I don't get it
15:23:34 <rpittau> it was working until yesterday?
15:23:51 <JayF> so we should check for requirements/constraints changes?
15:24:26 <JayF> https://opendev.org/openstack/requirements/commits/branch/master looks like a lot of bumps for internal libraries two days ago
15:24:36 <JayF> inc keystoneauth1/keystonemiddleware
15:24:54 <JayF> might be a place to dig from, if you could lock to the u-c from three days ago and see if it repros
15:25:51 <rpittau> latest uc updated sushy to 5.6.0
15:25:52 <TheJulia> well, if memory serves the way to get into that case is you think you've authenticated, but in the process you lost connectivity *or* the remote side never confirmed your session
15:26:17 <TheJulia> so so when you try to use what you believe is valid, then you error and you need to reauthenticate, the code *should* be re-authenticating
15:27:27 <rpittau> sushytools logs look very weird
15:27:35 <rpittau> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ed/openstack/3ed3085139af4072b86cd6c990706577/controller/before_pivoting/sushy-tools.log
15:28:12 <TheJulia> maybe time to add more debug logging to sushy-tools ? I'm guessing that is all re-auth requests but why are we getting 401s being sent back
15:28:16 <TheJulia> that seems like a bug
15:28:45 <rpittau> yeah
15:29:34 <rpittau> I have to drop in like 10 minutes, I can get another look tomorrow if no one gets to it before
15:29:46 <JayF> May I suggest we continue this troubleshooting async/outside of meeting thne?
15:30:34 <iurygregory> ++
15:30:50 <rpittau> yep
15:30:57 <TheJulia> yeah, wrap the the meeting :)
15:30:58 <JayF> #endmeeting