| stevebaker[m] | cardoe: I have proposed this https://review.opendev.org/c/openstack/kolla/+/987109 | 04:58 |
|---|---|---|
| stevebaker[m] | cardoe: also is this the error you see when a previous session was just stopped? I have an understanding of the problem now, and maybe the start of a solution https://kommodo.ai/i/wIz7Twyw3BRbDTohCtep | 05:17 |
| blanson[m] | Hello ! I am having a bit of an issue with the redfish inspection, and more specifically sushy from what I can gather ? It seems that at least on HPE servers, the redfish api returns a null size for all drive bays that aren't populated, but the drives_sizes_bytes function in sushy doesn't discriminate and tries to get the size for null drives, resulting in type errors ? I'm on 2025.1 but this does not look patched on master ? is | 08:30 |
| blanson[m] | this something you guys are aware of or should I just raide a bug and patch for it ? | 08:30 |
| blanson[m] | {... (full message at <https://matrix.org/oftc/media/v1/media/download/AY4rTlXZNG9HjOo5Yr8qIjIGtD1NSrfDuliwg22CQNxWtcctiHyZ3avSpS93lSMWE6LiWsWNCqgjR41zahxABhRCeeOk0JKgAG1hdHJpeC5vcmcvdXBIUmdBQkdIZFZwU2ZGeWpDeElod2ZO>) | 08:56 |
| opendevreview | Bertrand Lanson proposed openstack/sushy master: filter out drives reporting null size on inspect https://review.opendev.org/c/openstack/sushy/+/987133 | 09:32 |
| dtantsur | blanson[m]: I support the just do it approach | 11:42 |
| blanson[m] | haha yeah I figured as much tho zuul isn't too happy about my 5 word change and idk why | 11:50 |
| dtantsur | "HttpException: 415: Client Error for url: https://10.209.128.148/image/v2/images/67339791-b6f2-4d9b-a1be-a9d146b40640/file, Unsupported Media Type" | 11:57 |
| dtantsur | broken glance? Oo | 11:57 |
| opendevreview | Bertrand Lanson proposed openstack/sushy master: filter out drives reporting null size on inspect https://review.opendev.org/c/openstack/sushy/+/987133 | 12:00 |
| dtantsur | Safety check mbr on gpt failed because MBR PTE 1 has invalid boot flag: oslo_utils.imageutils.format_inspector.SafetyViolation: MBR PTE 1 has invalid boot flag | 12:01 |
| dtantsur | https://opendev.org/openstack/glance/commit/e2725ffa19c4ffc3d30c637df1a9c74484429518 | 12:02 |
| dtantsur | If anyone has a minute, could you return the value ^^^ in our devstack plugin to at least unblock the CI? | 12:04 |
| cardoe | stevebaker[m]: yes that’s the issue I see. | 12:08 |
| blanson[m] | also another question, inspection rules don't seem to run on redfish inspect ? | 12:11 |
| blanson[m] | is that intentional ? | 12:11 |
| dtantsur | cardoe: ^^^ (I don't remember the current status) | 12:21 |
| cardoe | He’s running 2025.1. They don’t run in that version. That was a recent release feature. | 12:22 |
| dtantsur | makes sense | 12:22 |
| ykarel | Hi team is the issue with ironic jobs where glance image create fails with , Unsupported Media Type known one? | 12:47 |
| ykarel | Example https://zuul.openstack.org/builds?job_name=ironic-tempest-ovn-uefi-ipmi-pxe&branch=master&skip=0 | 12:47 |
| ykarel | looks triggered by https://review.opendev.org/c/openstack/glance/+/964569 | 12:48 |
| dtantsur | ykarel: I've raised it on #openstack-glance already | 12:49 |
| ykarel | dtantsur, thanks dtantsur | 12:50 |
| opendevreview | mabulgu proposed openstack/ironic master: Improve vmedia insertion error messages https://review.opendev.org/c/openstack/ironic/+/982626 | 13:09 |
| blanson[m] | cardoe: I don't think the code is on master either ? should I just make a patch for it or is it morte involved than that ? | 13:11 |
| blanson[m] | more involved* | 13:11 |
| cardoe | To run inspection rules on redfish? It absolutely is. | 13:11 |
| blanson[m] | I only see the apply_rules call for in-band inspection ? https://github.com/openstack/ironic/blob/master/ironic/conductor/inspection.py#L148 inspect_hardware does not call it or am I missing something ? | 13:16 |
| TheJulia | good morning folks | 13:16 |
| blanson[m] | or shouldn't it call it and I'm trying to do something stupid here ? | 13:17 |
| TheJulia | https://bugs.launchpad.net/ironic/+bug/2150794 is just wonderful... :( | 13:19 |
| dtantsur | sigh | 13:30 |
| guilhermesp___ | i think that was my case of turning into ipmi these days and ricolin helped filling this out | 13:38 |
| iurygregory | good morning ironic | 13:40 |
| iurygregory | guilhermesp___, ipmi is dead, move on! :D | 13:40 |
| TheJulia | iurygregory: brutal | 13:43 |
| TheJulia | guilhermesp___: I guess the frustrating thing, at a single cup of coffee and my plate-o-work overflowing standpoint at the moment, is I think the reason we explicitly stopped sending that extra field was because supermicro was doing the exact opposite and we had to do similar with ipmi at a point as well. | 13:44 |
| iurygregory | TheJulia, yeah :D (I can only do that because I know Gui from a long time hehe) | 13:44 |
| TheJulia | or, maybe it was cisco, I don't remember and clearly need more coffee | 13:45 |
| TheJulia | iurygregory: still ;) | 13:45 |
| guilhermesp___ | iurygregory: bruuuuuuuuuuutaaaaaaaaallllllll :P | 13:48 |
| iurygregory | :D | 13:48 |
| guilhermesp___ | yeah i mean i lost so much time last week becasue i refused to upgrade bmc firmware and ended up just switching to ipmi to deliver to user, but i guess i will have more supermicros to play with redfish soon | 13:49 |
| iurygregory | <3 hopefully | 13:49 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: pecan: Drop redundant force_canonical https://review.opendev.org/c/openstack/ironic/+/987096 | 13:51 |
| cardoe | blanson[m]: https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/redfish/inspect.py#L193 | 14:09 |
| cardoe | dtantsur, TheJulia: right now I think getting CI working again needs to be our priority. | 14:10 |
| dtantsur | see #openstack-glance | 14:10 |
| TheJulia | yeah, editing the agenda now | 14:11 |
| TheJulia | dtantsur: I didn't see a bug for this issue yet, is that right? | 14:11 |
| cardoe | If I grok the scroll back we've got some issue with glance but the ironic-vnc-container fails to build because the PPA it relies on changed dependencies so the packages conflict. | 14:11 |
| cardoe | I know stevebaker[m] is working to get kolla to build a copy of the ironic-vnc-container so we can just use that but afaik the container doesn't build yet? | 14:11 |
| blanson[m] | @cardoe thank you, I'm dumb. | 14:12 |
| TheJulia | Likely chicken/egg issues where the glance one has us dead in the water unless we want to retool the plugin at the moment | 14:12 |
| dtantsur | TheJulia: not really - busy morning | 14:13 |
| cardoe | There's also a bunch of patches that I saw over the weekend where someone is changing out devstack stuff to focus more on OVS vs OVN in Neutron. | 14:13 |
| TheJulia | every monday is busy ;) | 14:13 |
| dtantsur | indeed | 14:13 |
| TheJulia | I'm putting an item into the agenda now with some detail | 14:13 |
| cardoe | I feel like we can land those just to fix the devstack issues. | 14:13 |
| TheJulia | Just to get the discussion started | 14:13 |
| cardoe | But we should have an honest convo about calling OVS "deprecated" and making OVN the target support. | 14:14 |
| cardoe | Since Neutron is deprecating their OVS support for OVN. | 14:14 |
| cardoe | This is the Ironic + Neutron case. | 14:14 |
| cardoe | Not any other case with OVS that I'm talking about. | 14:14 |
| cardoe | I've been asked to prep some issues and items for the Nova meeting wrt to Ironic so working that right now. | 14:15 |
| TheJulia | okay | 14:24 |
| TheJulia | so, uhhhh brains | 14:24 |
| TheJulia | cardoe: I tried to raise this during the ptg and felt like I was getting pushback aside from just slowly pushing the changes under the general radar to align our CI jobs | 14:25 |
| cardoe | Who was concerned? | 14:26 |
| TheJulia | my perception was that everyone wanted to "keep" it even though neutron slowly removing is going to be a thing | 14:26 |
| JayF | My perception is the OVN-on-baremetal case is still noticibly worse than OVS-on-baremetal case, is that not an accurate perception anymore? | 14:27 |
| JayF | I also know OVS->OVN migrations are painful and want to save our downstreams from forced migrations | 14:27 |
| TheJulia | .... I feel like we're rapidly approaching the point where maybe we need to have a high bandwidth meeting | 14:28 |
| * TheJulia first needs a second cup of coffee | 14:28 | |
| JayF | I am just tossing in my main motivation for OVS+Ironic continuing to exist: everytime we hand down a significant upgrade migration, it dregs up all the "OpenStack is a PITA to upgrade" stuff :| | 14:29 |
| TheJulia | while feeding the coffee maker, a likely starting point is also to highlight current state in our docs with updated context/perception | 14:30 |
| TheJulia | JayF: and a partial side effect of trying not to actually makes it worse at times :( | 14:31 |
| JayF | Maybe so, but it is frustrating to be the place where dependencies making decisions to swap backends exposes to users | 14:32 |
| TheJulia | yup | 14:32 |
| TheJulia | so, step 0, is bring more clarity, let me quickly revise the OVN docs | 14:33 |
| cardoe | Well our docs say that OVS is the recommended path. And so we're giving folks a migration in the future by recommending that. | 14:43 |
| JayF | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon May 4 15:00:15 2026 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| JayF | Good morning Ironic'ers, time for our Monday meeting. Who's here? | 15:00 |
| JayF | As usual we operate the meeting under the OpenInfra CoC | 15:00 |
| TheJulia | o/ | 15:01 |
| dtantsur | o/ | 15:02 |
| JayF | Tw^W Three does not a quorum make. Is May 4 a holiday somehwere? | 15:02 |
| TheJulia | yes, Much of the EU AFAIK | 15:03 |
| dtantsur | May 1 was | 15:03 |
| TheJulia | or.. was that Ireland only that is May 4th? | 15:03 |
| dtantsur | likely | 15:03 |
| JayF | We have a beefy enough meeting agenda that I'm skeptical of the value of running through it if we don't have quorum. | 15:04 |
| JayF | #link https://wiki.openstack.org/wiki/Meetings/Ironic | 15:04 |
| TheJulia | I know cardoe has interest in the first discussion topic and we should attempt to unwedge our state of being | 15:04 |
| JayF | My suggestion is to cancel the meeting if it's just us three. | 15:05 |
| TheJulia | cardoe: likely time to eradicate that "ovs is the path" text, fwiw | 15:05 |
| JayF | Does that count as an official vote with one "yes" an "I need coffee", and an abstention? lol | 15:07 |
| TheJulia | maybe? | 15:07 |
| * TheJulia dreads taht | 15:07 | |
| TheJulia | that | 15:07 |
| clif | o/ sorry im late | 15:08 |
| TheJulia | Am I the one who always needs coffee? | 15:08 |
| clif | I do | 15:08 |
| JayF | TheJulia: I just didn't charge dtantsur with needing coffee since he's not in a TZ currently experiencing AM | 15:08 |
| dtantsur | heh | 15:08 |
| dtantsur | yeah, 5pm is not great for more coffee | 15:09 |
| JayF | #note There are 4 contributors at the meeting, this does not meet quorum and so the primary agenda items will not be discussed. | 15:09 |
| JayF | #topic Announcements/Reminders | 15:09 |
| JayF | Please review code hashtagged with "ironic-week-prio" and hashtag your changes that are ready for review. | 15:09 |
| JayF | Also, it's R-21 in the release cycle. Do you know where your Hibiscus tea is? | 15:10 |
| JayF | Skipping Working Group Updates, Discussion Topics, and new bugs as we do not have quorum. | 15:10 |
| JayF | There are 14 new bugs on the agenda (!) thanks cid for deputying. | 15:11 |
| JayF | I assume he will continue as deputy unless we have another volunteer. | 15:11 |
| JayF | #topic Open Discussion | 15:11 |
| JayF | Is there anything we need to discuss in this quorum-less meeting that cannot wait until next week? I see several things on the agenda about broken CI, but I think that's being worked separately | 15:12 |
| TheJulia | its dead in the water until we reach some level of consensus | 15:12 |
| * cardoe waves. | 15:12 | |
| TheJulia | hi cardoe! | 15:12 |
| JayF | TheJulia: can we be more specific? | 15:12 |
| * TheJulia gives cardoe coffee | 15:12 | |
| TheJulia | sure | 15:12 |
| cardoe | coffee good | 15:13 |
| TheJulia | so, our CI is dead in the water right now because x86_64 kernels when uploaded as raw artifacts, now get blocked form glance due to a change landing | 15:13 |
| cardoe | UK has "Early May Bank Holiday" aka we wanted May the Forth Be With You off of work. | 15:13 |
| TheJulia | dan smith believe we can just use the aki/ari path and they should still upload | 15:13 |
| TheJulia | cardoe: how very starwars of them ;() | 15:13 |
| TheJulia | err ;) | 15:13 |
| * TheJulia can't type today | 15:13 | |
| JayF | Wasn't aki/ari specifically what we were told to *not* use? | 15:13 |
| TheJulia | anyway, back to glance | 15:13 |
| TheJulia | JayF: yup, yeah | 15:14 |
| JayF | is there a reason the answer isn't "revert the change in Glance until a conversation happens"? | 15:14 |
| cardoe | So this is just a band aid? | 15:14 |
| TheJulia | JayF: it seems like it was miscommunication, we can unwedge by taking one of the following three paths | 15:14 |
| JayF | Because it's 100% unacceptable for us to document to users to lie about the type of image they are using | 15:14 |
| TheJulia | 1) Just set the config to permit it and document it and move on. Not great for operators. 2) Try and revert it out, but it sounds like they are intending to make raw unusable for us. 3) Try ari/aki again which previously broke I believe. :\ 4) Question why use glance for kernel/ramdisk artifacts to boot hosts. | 15:15 |
| TheJulia | so, 3^H4 possible paths | 15:15 |
| JayF | 2 is the short term path | 15:16 |
| JayF | and I will lean on gouthamr and the TC to have that service land such a revert given they blatantly broke us | 15:16 |
| TheJulia | I think they will refuse the revert since they intend on adding more blocks on use of "raw" | 15:16 |
| JayF | We are a single community at the base of it, and landing software that completely bricks another project is unacceptable | 15:16 |
| TheJulia | To be clear, it doesn't completely brick is. It just bricks our CI, and sets us up for a world of hurt perception wise regardless | 15:17 |
| JayF | it *does* brick us, from an operator perpspective | 15:18 |
| TheJulia | fair | 15:18 |
| JayF | if they used master glance, and pulled this change in | 15:18 |
| JayF | they just got f*in bricked | 15:18 |
| TheJulia | dead in the water unless they go change the glance config | 15:18 |
| JayF | sadl; | 15:19 |
| JayF | I'm so annoyed by gerrit revert UI right now | 15:19 |
| JayF | going to do it manually | 15:19 |
| JayF | AFAICT it's impossible to do it in such a way that gerrit is OK with it, even a manual DCO got kicked out for no DCO | 15:20 |
| TheJulia | eww :( | 15:20 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Trivial: debug_tracebacks_in_api=True for unit tests https://review.opendev.org/c/openstack/ironic/+/987184 | 15:21 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Update admin doc on ovn networking https://review.opendev.org/c/openstack/ironic/+/987185 | 15:21 |
| TheJulia | JayF: regarding the ovn discussion, hopefully ^ makes things a bit more clear | 15:21 |
| JayF | #link https://review.opendev.org/c/openstack/glance/+/987186 Revert "Remove default to relaxed MBR safety checks" [NEW] | 15:22 |
| JayF | Aight, we still are technically in the meeting. | 15:24 |
| JayF | Anything else beofre I close it? | 15:24 |
| cardoe | Not from me. I'm gonna try and highlight some ironic/nova rubs | 15:24 |
| cardoe | So if you have any specific ones call them out | 15:24 |
| JayF | Please do the legwork for the Nova/Ironic working group | 15:25 |
| TheJulia | heh | 15:25 |
| JayF | and use that as the mechanism for this grievancing lol | 15:25 |
| JayF | Thanks for the 4 people who showed up, extra thanks to the 3 on time. We'll hopefully have a real meeting next week o/ | 15:26 |
| JayF | #endmeeting | 15:26 |
| opendevmeet | Meeting ended Mon May 4 15:26:14 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:26 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-05-04-15.00.html | 15:26 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-05-04-15.00.txt | 15:26 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2026/ironic.2026-05-04-15.00.log.html | 15:26 |
| cardoe | Sorry. I have a 1:1 at the start of the ironic meeting that I cannot move since the time zone change. | 15:26 |
| TheJulia | cardoe: maybe one century, daylight savings time will be no more. | 15:27 |
| TheJulia | JayF: please let me know if the ovn doc update makes sense and re-contextualizes concerns, perhaps. | 15:30 |
| JayF | I don't have bandwidth for that today or anytime soon, I can take a look real quick and give you a high level look, but I am literally buried under a pile of urgent stuff | 15:31 |
| JayF | not least of which is publishing the damn work items doc which I think all of 3 people reviewed | 15:31 |
| TheJulia | welcome to the island of overloaded | 15:33 |
| JayF | TheJulia: that doc update doesn't help with the real problem: how do we sell to operators to do a migration without it being for "bs openstack reasons" .... we are asking for a big undertaking with minimal benefit. | 15:35 |
| JayF | I know some large % of that is not Ironic's doing | 15:35 |
| TheJulia | I mean, I tried to plant some of that context upfront | 15:35 |
| TheJulia | but fair enough | 15:35 |
| JayF | this is all downside from the operator standpoint | 15:35 |
| JayF | unless we can change that, I will be unlikely to want to push the buttons for this migration even if OVS is rotting | 15:36 |
| TheJulia | btw, the priorities doc, now has a -1, it can be ignored or done in follow-up. I have zero capacity for attestaiton this cycle | 15:36 |
| TheJulia | So, to be clear, that comes off as pure resistance to change when OVN is actually far superior performance wise, but yes, without being an expert in the lower levels, its hard to really build that justificaiton when you just want something to minimally work | 15:37 |
| TheJulia | so, I guess what I need is a whole section "why you want to do this" | 15:38 |
| TheJulia | coupled with "why it shouldn't be awful for your baremetal" | 15:38 |
| JayF | "Pure resistance to change *that doens't have upside from a working operators standpoint*" | 15:38 |
| JayF | that's the entire thing | 15:38 |
| TheJulia | yeah, you don't know what you don't knwo. | 15:38 |
| JayF | we're giving a project to downstreams with zero ammo for justifying it other than "you know openstack, upgrades take forever" | 15:38 |
| cardoe | So I think the reason is simply upstream decided they only had the bandwidth for one approach. | 15:44 |
| cardoe | Thank you operator for continuing to use the software. If features and functionality are important to you get involved past saying I want X. | 15:45 |
| cardoe | There’s one case to be had when there are interested contributors and upstream is turning folks away. | 15:45 |
| cardoe | There’s another case when the folks supporting a feature all move to another feature and say if nobody steps up to maintain their previous feature it will go away. | 15:46 |
| JayF | cardoe: I don't have a constructive response to that. I don't think our operators give a damn and if we provide enough friction they'll leave us alone, no matter if we write a damn disseratation about how right it is to migrate | 15:49 |
| opendevreview | Harald Jensås proposed openstack/networking-generic-switch master: GenericSwitchNetmikoConfigError error details https://review.opendev.org/c/openstack/networking-generic-switch/+/985029 | 15:50 |
| cardoe | JayF: I was just trying to point out that our docs still say OVN bad... OVS good... use OVS... as do our testing defaults... when the Neutron folks have said OVS will go away unless someone steps up to maintain it. We're gonna be in the same broken boat in a couple of cycles. | 15:51 |
| JayF | Yep. I just think we can only press the "painful upgrade + change the mental model of your cluster" so many times | 15:52 |
| JayF | and the last time I looked at the button it was raggedy and about to break | 15:52 |
| opendevreview | Verification of a change to openstack/ironic stable/2026.1 failed: ci: stable: disable metal3 CI jobs https://review.opendev.org/c/openstack/ironic/+/986812 | 15:53 |
| JayF | and an "open source harder" argument really runs the wrong way with me, given the amount of contribution my employer gives vs company size (we don't deserve to get a "oh well, should've maintained harder" even if others do) | 15:53 |
| cardoe | Well then it's gonna fall on Ironic to maintain OVS | 15:54 |
| JayF | oh no! We'll lose the zero help we get maintaining Ironic networking /s | 15:55 |
| JayF | I know it's not that simple but it does feel like we're our own vertical | 15:55 |
| JayF | I shouldn't have this conversation today. I'm not in a good place and it's extremely unlikely to go anywhere constructive. | 15:56 |
| dtantsur | Poor little launchpad, what happened | 16:02 |
| TheJulia | huh? | 16:03 |
| dtantsur | it's down | 16:03 |
| TheJulia | did it go kaboom while I was in the shower | 16:03 |
| TheJulia | uuuugh | 16:03 |
| dtantsur | back apparently | 16:04 |
| TheJulia | \o/ | 16:04 |
| dtantsur | well, not back enough to leave comments.. I guess I'll just wait a bit | 16:07 |
| TheJulia | :( | 16:08 |
| TheJulia | le-sigh | 16:08 |
| cid | ironic o/. | 16:39 |
| cid | feedback on this patch is that it has to be a major release, is that common practice early in the cycle? https://review.opendev.org/c/openstack/releases/+/986434 | 16:39 |
| JayF | Major releases at the beginning of the cycle are cheap | 16:39 |
| JayF | meaning like, it's going to be nondisruptive to people | 16:40 |
| dtantsur | oh, we're at a release point already? OO | 16:40 |
| JayF | It's H-1 next week iirc | 16:40 |
| cid | So it seems. JayF, tks. It is! | 16:41 |
| cid | I will go ahead and update the patch then. | 16:41 |
| dtantsur | We're one month since G, no? | 16:41 |
| dtantsur | but yeah, it should be next week indeed (not this week though?) | 16:41 |
| cid | Yeah, 1 month | 16:42 |
| dtantsur | We're supposed to release every two months, hence my surprise | 16:42 |
| JayF | The -1 timing is off the RC deadline IIRC | 16:43 |
| dtantsur | we literally finalized corresponding Metal3 releases last week (okay, we had an unplanned delay there) | 16:43 |
| JayF | it's 2 months after cycle-with-rc projects push their RC | 16:43 |
| TheJulia | if we remove molds, then its 36.0, fwiw. | 16:43 |
| TheJulia | but I'm not sure that has landed yet | 16:43 |
| dtantsur | hmm, it would be good to get the molds removal in | 16:43 |
| TheJulia | yeah | 16:44 |
| cid | TheJulia, not landed all places yet. I think it's in on the master branch | 16:44 |
| cid | Let me confirm | 16:44 |
| dtantsur | 35.0.0 was on Mar 19, I see. I'd rather wait until next week, but yeah. Time flies. | 16:45 |
| cid | Apparently not. blocked by CI | 16:45 |
| JayF | If it's not in master, we shouldn't release yet | 16:48 |
| JayF | and it can't go into master until either glance lands https://review.opendev.org/c/openstack/glance/+/987186 or we change our devstack config to something that I don't think is good for operators | 16:50 |
| cid | Yeah, I contemplated the retry I just issued | 16:52 |
| cid | I guess it will not change anything. | 16:52 |
| cardoe | IMHO, we should get the sushy release and bump the dependency on sushy as well. | 16:52 |
| cardoe | master won't work with the currently released sushy | 16:53 |
| cardoe | https://review.opendev.org/c/openstack/releases/+/986893 | 16:53 |
| cardoe | Specifically you need a18eaf3 | 16:54 |
| cid | I saw your comment on the Ironic release mapping change and was about to propose one then saw the patch is up already. | 16:55 |
| cid | Just waiting for votes from release liason. | 16:55 |
| cardoe | Well sushy needs our folks to vote. | 16:57 |
| cid | yup | 17:02 |
| opendevreview | Verification of a change to openstack/ironic master failed: Remove deprecated configuration molds feature https://review.opendev.org/c/openstack/ironic/+/986703 | 17:08 |
| TheJulia | going back to ovn, I've got claude trying to do a docs cleanup pass with providing additional context to address JayF and cardoe's comments | 17:40 |
| JayF | I don't think my concern is completely undoable. It's just an unfortunate reality. I've been very grumpy today due to that damned reality. | 17:41 |
| TheJulia | likewise :( | 17:43 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Update docs and guides regarding OVN networking https://review.opendev.org/c/openstack/ironic/+/987185 | 18:02 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM: Revert "Replace reference to ari/aki" https://review.opendev.org/c/openstack/ironic/+/987239 | 18:12 |
| TheJulia | Question regarding sushy, shouldn't we also release master and update the dependency accordingly. I'd like to make progress on tls controls but sushy is soft of the break point at present. | 18:41 |
| noonedeadpunk | hey folks! I am having a weird issue now with Ironic while I am trying to spawn an osa sandbox (pretty much attempting to fix CI and drop out old inspector code). | 18:50 |
| noonedeadpunk | So wsgi app loads, throws no errors, but it doesn't do anything either, and replies 200 with any random bs request | 18:51 |
| noonedeadpunk | so neither versioning, nothing really is routable somehow. | 18:51 |
| noonedeadpunk | it feels that an api-paste.ini is required or smth, but I don't see it neither in setup guide, nor in devstack setup | 18:52 |
| noonedeadpunk | so I wonder if smth has changed between 2025.2 -> 2026.1 in terms of running API which could lead to smth like that? | 18:53 |
| noonedeadpunk | As comparing of devstack config vs what I have does not really highlights issue... | 18:53 |
| noonedeadpunk | the configs look like that: https://paste.openstack.org/show/bNhkvxmnWla1UlDyhfs9/ | 18:57 |
| noonedeadpunk | but this is a noop sandbox where I expect just API to work or at least to load the app... | 18:57 |
| TheJulia | So, regarding the firefox PPA. Could we just disable that portion of the jobs for now so we're un-wedged? | 18:57 |
| TheJulia | stevebaker[m]: ^ | 18:58 |
| stevebaker[m] | TheJulia: The jobs choose Containerfile.ubuntu or Containerfile.centos to match the host distro. Why not switch to Containerfile.centos regardless? That might be less disruptive at this point | 18:59 |
| TheJulia | noonedeadpunk: there was some stuff to address pickable changes related to service launches towards the end of cycle, but I don't think so. | 19:00 |
| TheJulia | stevebaker[m]: OS of the container, or OS of the host? | 19:00 |
| stevebaker[m] | The distro of the container matches the distro of the host. Arbitrarily just for test coverage | 19:00 |
| TheJulia | so, your advocating we kick ubutnu entirelly to centos which is something the rest of the devsack stuff is not even tested on today | 19:01 |
| TheJulia | ... | 19:01 |
| * TheJulia needs a vacation | 19:02 | |
| stevebaker[m] | Just until we have base images in place which have the packages installed | 19:03 |
| stevebaker[m] | centos 9 specifically | 19:03 |
| TheJulia | that seems like a lot with no gain when the cleanest path is to isolate the logic to a variable and -NV that job for now while we work on long term "ubuntu ppa is never coming back or we need to switch to $something_else" paths | 19:04 |
| noonedeadpunk | but just to double-check - there's indeed no api-paste was introduced and routing/loaded middlewares are not depending on it? | 19:06 |
| TheJulia | AIUI, added as a generic injection skaffold to enable operators to do additional logic in isolated ways | 19:08 |
| stevebaker[m] | How unavailable is PPA currently? I have had builds succeed 11 hours ago | 19:09 |
| TheJulia | CI and I are both unable to reach the server | 19:09 |
| clif | is launchpad still broken | 19:09 |
| TheJulia | it was working for a time this morning and is down again | 19:09 |
| clif | :( | 19:09 |
| clif | oh I got a page load after about 10 seconds, exciting | 19:10 |
| stevebaker[m] | OK. So in theory the toggle to enable and test graphical consoles in CI is whether ir-novnc service is enabled. Disabling that service on all jobs would be enough | 19:11 |
| TheJulia | yeah, I get connection refused trying to get to the PPA endpoint | 19:11 |
| TheJulia | by chance do you know the ones where it should be expected to work and pass today? | 19:12 |
| noonedeadpunk | TheJulia: So... I think I have pin-pointed the behavior to [healthcheck]/enabled = True | 19:19 |
| noonedeadpunk | once I comment it out, things are working again... | 19:20 |
| noonedeadpunk | So... according to this logic, it's always will respond with healthcheck only? https://opendev.org/openstack/ironic/src/branch/master/ironic/api/app.py#L148-L149 | 19:21 |
| TheJulia | apps afaik are nested | 19:23 |
| TheJulia | normally I would say file a bug in launchpad with enough context to reproduce without being an OSA expert | 19:23 |
| noonedeadpunk | I would expect them to be... | 19:23 |
| noonedeadpunk | :D | 19:23 |
| noonedeadpunk | well, I can make that quite reproducible, as eventually I could reproduce even without uwsgi and just with defualt wsgi server | 19:25 |
| stevebaker[m] | ir-novnc is enabled by default and only disabled on ironic-tempest-ipa-wholedisk-direct-multinode. Disabling it | 19:25 |
| stevebaker[m] | ..in devstack/settings would be easiest. I'm putting a change together now | 19:25 |
| noonedeadpunk | I wonder if I just add this property to devstack - if that would fail CI right away | 19:25 |
| noonedeadpunk | I'll do some digging/reproduce guide and report a bug I guess | 19:26 |
| TheJulia | stevebaker[m]: go ahead, but expect a compound change to fix it since we have more than one break right now | 19:26 |
| noonedeadpunk | as it feels that healthcheck just intercepts all requests today | 19:27 |
| opendevreview | Steve Baker proposed openstack/ironic master: CI: disable ir-novnc to avoid PPA package installs https://review.opendev.org/c/openstack/ironic/+/987258 | 19:29 |
| stevebaker[m] | TheJulia: OK ^ can be rolled into a compound change | 19:29 |
| TheJulia | cool cool, we can also make multinode non-voting too for now, I'll cry about that later | 19:30 |
| noonedeadpunk | to be fair, for non-OSA experts, it was just `python3 -m venv /openstack/venvs/ironic-32.1.0.dev109/; /openstack/venvs/ironic-32.1.0.dev109/bin/pip install ironic systemd-python PyMySQL pymemcache python-memcached` | 19:34 |
| noonedeadpunk | (u-c up to the taste) | 19:34 |
| opendevreview | Brian Haley proposed openstack/ironic master: Make ironic-tempest-bios-ipmi-autodetect non-voting https://review.opendev.org/c/openstack/ironic/+/987260 | 19:40 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix CI (aki/ari && novncproxy containers) https://review.opendev.org/c/openstack/ironic/+/987239 | 19:48 |
| cardoe | TheJulia: yes we can release sushy master as well. | 20:10 |
| cardoe | noonedeadpunk: I've seen the same problem... are you using uWSGI or the ironic shim binary to start? | 20:10 |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: Drop Python 3.10 https://review.opendev.org/c/openstack/ironic-ui/+/987280 | 20:37 |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: Drop Python 3.10 https://review.opendev.org/c/openstack/ironic-ui/+/987280 | 20:38 |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: Update packaging configuration https://review.opendev.org/c/openstack/ironic-ui/+/987281 | 20:39 |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: Update packaging configuration https://review.opendev.org/c/openstack/ironic-ui/+/987281 | 20:41 |
| opendevreview | Ivan Anfimov proposed openstack/ironic-ui master: Update packaging configuration https://review.opendev.org/c/openstack/ironic-ui/+/987281 | 20:41 |
| TheJulia | stevebaker[m]: nope, sadness, jobs fail with it being disabled :\ | 20:51 |
| TheJulia | hmm, likely an issue in the plugin, quick break and back to it | 20:52 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix CI (aki/ari && novncproxy containers) https://review.opendev.org/c/openstack/ironic/+/987239 | 21:06 |
| TheJulia | stevebaker[m]: my own mistake, hopefully its better with this. | 21:11 |
| TheJulia | ricolin: o/, regarding https://bugs.launchpad.net/ironic/+bug/2150794 wasthe vendor being recorded by ironic? | 21:32 |
| TheJulia | so, first pass wise on change 987239, looks like CI is passing, but only a couple smaller jobs so far, the bulk are still running | 21:41 |
| guilhermesp___ | TheJulia: im about to logout here now but as far as i remember, i did see vendor="super micro" or something along those lines, is that it? | 22:01 |
| TheJulia | yeah, just get us the exact string which ironic detecs | 22:02 |
| TheJulia | detects | 22:02 |
| guilhermesp___ | 'vendor': 'super micro computer inc.' | 22:02 |
| TheJulia | ugh | 22:02 |
| guilhermesp___ | on my live node here | 22:02 |
| TheJulia | okay | 22:02 |
| TheJulia | so, https://review.opendev.org/c/openstack/ironic/+/987239 is looking pretty good, I get JayF has concerns and the stance change ain't great because its confusing, but we an clarify it in post if at all possible to unbrick CI. | 22:23 |
| JayF | I'm not going to -1 that to unblock us, I just don't accept that it's a good final state (yet) | 22:24 |
| cardoe | I workflowed it just to get us unbusted. | 23:33 |
| cardoe | But I don't think this is the best form. | 23:33 |
| TheJulia | Yeah, to be clear, I'm not super thrilled by it all | 23:43 |
| *** hroy_ is now known as hroy | 23:47 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!