| iurygregory | JayF, regarding bugfix branches, we don't need them downstream anymore from what I've checked, I will let rpittau confirm this =) | 00:45 |
|---|---|---|
| opendevreview | Verification of a change to openstack/ironic stable/xena failed: Fix Cinder Integration fallout from CVE-2023-2088 https://review.opendev.org/c/openstack/ironic/+/883580 | 05:44 |
| adam-metal3 | Hello Ironic! So I am bit confused I thought the Ironic community has migrated back to launchpad for planning, but I only see bugreports there, is it so, that bugs should be reported on launchpad and new feature proposals on StoryBoard? | 06:41 |
| dtantsur | adam-metal3: everything goes to launchpad | 07:58 |
| dtantsur | you can tag feature requests as [RFE] | 07:58 |
| adam-metal3 | dtantsur: Thanks! | 07:59 |
| rpittau | good morning ironic! o/ | 07:59 |
| adam-metal3 | rpittau; hi ! | 08:00 |
| rpittau | JayF, iurygregory, we don't use bugfix branches anymore downstream BUT we're going to use them for metal3, specifically for the ironic container, and BTW I'm going to request a new bugfix branch this week :) | 08:00 |
| rpittau | hey adam-metal3 :) | 08:00 |
| *** Continuity_ is now known as Continuity | 08:28 | |
| opendevreview | Verification of a change to openstack/bifrost master failed: Reduce multiple tasks to task with loop https://review.opendev.org/c/openstack/bifrost/+/877186 | 10:57 |
| iurygregory | good morning Ironic | 11:21 |
| iurygregory | rpittau, "bugfix/18.1 (xena) (should be supported until June 2023)" JayF wants to remove the branches that are reaching the eol time, there is an email on openstack-discuss "[ironic] Retirement of ironic-prometheus-exporter bugfix/2.1 branch" but involves the other projects we have =) | 11:23 |
| rpittau | iurygregory: bugfix branches have a life expectancy of 6 months, after that time they're not supported and they should probably be retired/removed, so I'm fine with that | 11:27 |
| rpittau | if 18.1 is to be supported until June 2023, then after June will be EoL :) | 11:29 |
| opendevreview | Merged openstack/bifrost master: Set [inspector]require_managed_boot explicitly https://review.opendev.org/c/openstack/bifrost/+/877469 | 12:01 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 12:47 |
| JayF | rpittau: the reason that one bug fixed branch is labeled June 2023, is last time someone told me they are only 6 months old and I wanted to retire them I was told not to do that one until June 2023. | 13:32 |
| JayF | So it's sort of hard to keep track of what being used and what's not being used when all I have is random comments in IRC every now and then 😄 | 13:33 |
| rpittau | JayF: I thought we clarified the lifecycle of the bugfix branches, but apparently we haven't written anything down about that :/ | 13:39 |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Replace Cirros references with generic descriptors https://review.opendev.org/c/openstack/bifrost/+/883943 | 13:41 |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Improve downloaded deployment image support https://review.opendev.org/c/openstack/bifrost/+/884888 | 13:41 |
| rpittau | JayF: I'm going to review the bugfix branches we still have and update the etherpad according to the 6 months support time, and since I'm there I will propose a patch to make that official since I can't find any mention of that anywhere in the docs | 13:47 |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Improve downloaded deployment image support https://review.opendev.org/c/openstack/bifrost/+/884888 | 13:52 |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Improve downloaded deployment image support https://review.opendev.org/c/openstack/bifrost/+/884888 | 13:53 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Be explicit about bugfix branches support lifetime https://review.opendev.org/c/openstack/ironic/+/884889 | 13:55 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Be explicit about bugfix branches support lifetime https://review.opendev.org/c/openstack/ironic/+/884889 | 13:57 |
| rpittau | JayF: I've updated the etherpad and cleaned up the supported bugfix branches | 14:26 |
| JayF | rpittau: thank you so much, I'll look at it when I can point my brain upstream | 15:00 |
| rpittau | no problem | 15:01 |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Improve downloaded deployment image support https://review.opendev.org/c/openstack/bifrost/+/884888 | 15:03 |
| *** Continuity__ is now known as Continuity | 15:15 | |
| JayF | I'm really confused. ilo-redfish is mentioned as a driver in the ilo docs, but I can't find reference to it in the ironic repo (git grep ilo-redfish) | 15:18 |
| JayF | anyone know what's up? | 15:18 |
| JayF | rpittau: ack. If you have time, can you email list about the removed bugfix/{} branches and banner their retirement? I can push the buttons in a couple weeks | 15:25 |
| JayF | if not, let me know and I'll lookup which ones you pulled and do it myself | 15:25 |
| TheJulia | uhh... ilo internally switches out to redfish | 15:27 |
| TheJulia | ilo5 I think is what they mean, maybe? | 15:27 |
| rpittau | JayF: will try to do it before EOD | 15:28 |
| JayF | yeah it's very badly written | 15:31 |
| JayF | rpittau: thank you, no real rush. I just want to billboard at all before we retire them :D | 15:31 |
| TheJulia | JayF: the other conundrum is sone "HPE" gear, is not "Proliant" and doesn't just work with the ilo driver | 15:32 |
| TheJulia | in those cases, stock "redfish" is the way | 15:32 |
| JayF | so context: I have a bug coming in soon from my downstream | 15:33 |
| JayF | node is reserved, in AVAILABLE (no tgt prov state), no maintenance | 15:33 |
| TheJulia | reserved as in lock is held ? | 15:33 |
| JayF | last thing Ironic conductor did was power status loop, found the node was wrong power status, tried to tell the node BMC to power on to make it match | 15:33 |
| JayF | yes | 15:33 |
| JayF | and the power change timed out | 15:34 |
| JayF | so I'm going to dig a bit to see if we're missing err handling somewhere, because that's what it smells like | 15:34 |
| TheJulia | ... and the lock didn't get released? | 15:34 |
| JayF | yep | 15:34 |
| JayF | it was locked for like a week until the operator restarted the service today | 15:34 |
| TheJulia | did the conductor crash in that process? | 15:34 |
| TheJulia | weird | 15:34 |
| JayF | there was no reported crash, I asked, but I don't have direct access to the logs | 15:35 |
| TheJulia | so, it *should* have released the lock regardless on task exit, unless the task never exited... | 15:35 |
| TheJulia | was there any stalled sockets to the remote bmc? | 15:35 |
| JayF | there is no direct access to the conductor nodes, for security reasons | 15:35 |
| JayF | we only have access to logs/db | 15:35 |
| JayF | but the error *was in the log* that it failed to update the power status | 15:35 |
| JayF | so it at least knows it failed, even if it hung at that point | 15:36 |
| TheJulia | yeah | 15:36 |
| TheJulia | weird | 15:36 |
| JayF | I feel like it's either going to be something horrible around locking or someone missed an error catch/throwing wrong exception/etc | 15:36 |
| JayF | I'm going to dig it, but Scott is supposed to be filing an issue with redacted logs/output | 15:36 |
| JayF | (two folks from my downstream, Scott and Ross, will be in Vancouver too \o/) | 15:37 |
| TheJulia | this may also we need to find a reproducer or something to get a little more info | 15:37 |
| TheJulia | it is *super* weird | 15:37 |
| JayF | yeah audible WTF was garnered from me | 15:38 |
| JayF | especially since it seems all conducotr-internal | 15:38 |
| JayF | so my "rabbitmq hopped your message away" default answer failed :P | 15:38 |
| TheJulia | yeah | 15:40 |
| TheJulia | the rabbit monster went "om nom nom" | 15:40 |
| JayF | oooh, I wonder if I should have Scott check | 15:41 |
| JayF | to see if the power status loop on that conductor died after that | 15:41 |
| TheJulia | that would be even more bizar | 15:41 |
| JayF | it would at least be a failure mode I've seen before | 15:41 |
| JayF | power status loop crash or hangs on IO, leaves one node in a bad state | 15:41 |
| JayF | either way, I'll dig it and see if I get anywhere | 15:42 |
| rpittau | JayF: sent :) | 15:54 |
| rpittau | see ya all tomorrow! o/ | 15:54 |
| JayF | thank you | 15:58 |
| JayF | TheJulia: heh. The plot thickens but makes sense | 16:10 |
| JayF | node was flapping power state due to a reboot loop (failing to pxe) | 16:11 |
| JayF | when the power status loop comes around, it sees node as "power off" | 16:11 |
| JayF | by the time it tells the bmc to power on, it's already on and the iLo gets angry | 16:11 |
| JayF | the nodes end state? target_power_state: power on; power_state: power on; reservation [hostname] | 16:11 |
| JayF | which makes very much sense as an edge that could happen | 16:11 |
| TheJulia | interesting | 17:42 |
| JayF | it's written up here: https://bugs.launchpad.net/bugs/2021995 | 17:42 |
| JayF | I'll keep it updated with any interesting information I ask for and get to try and troubleshoot it | 17:42 |
| JayF | sadly, we can't tell if the power status loop was still running; the only node it was performing actinos on was the broken one | 17:43 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB API for Firmware and Object https://review.opendev.org/c/openstack/ironic/+/883062 | 17:51 |
| iurygregory | funny thing I noticed, the class name in models.py should match the the object class .-. (I wasn't expecting this...) | 17:53 |
| iurygregory | I've decided to squash the object patch with the db api layer since they have a lot in common | 17:54 |
| TheJulia | Makes sense, a common quash | 18:01 |
| TheJulia | squash | 18:01 |
| JayF | https://platform9.com/docs/bare-metal/overview this is interesting | 20:16 |
| JayF | hosted Ironic, apparently | 20:16 |
| JayF | Is anyone here, or a part of the community in general, affiliated? | 20:16 |
| JayF | https://superuser.openinfra.dev/articles/ironic-bare-metal-case-study-platform9/ | 20:23 |
| clarkb | John Studarous was involved in doing openstacky things at platform 9 a while back | 20:24 |
| JayF | just happy to hear of another company using Ironic to do good stuff | 20:27 |
| JayF | I'll have to browbea^W ask them nicely to contribute back ;) | 20:27 |
| NobodyCam | Good afternoon Openstack folks | 21:40 |
| NobodyCam | Happy hump day | 21:40 |
| TheJulia | good afternoon! | 22:39 |
| NobodyCam | o/ howdy TheJulia | 22:45 |
| NobodyCam | it shouldn't be so difficult to get Devstack running :head desk: | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!