| opendevreview | Bartosz Bezak proposed openstack/tenks master: CI: move do debian trixie https://review.opendev.org/c/openstack/tenks/+/988970 | 07:16 |
|---|---|---|
| Sandzwerg[m] | Ohai Ironic, I have a question regarding the "max_concurrent_clean" What are the implications of increasing this? The comment in the sample config https://docs.openstack.org/ironic/2025.2/configuration/sample-config.html mentions it's a security mechanism (the cleaning itself) but what is the bottle neck for this? Conductors? file transfer of the ramdisk? | 10:10 |
| priteau | Hello Ironic team. I am curious if you are planning to publish CentOS Stream 10 IPA images for stable branches? I can only see master ones. | 10:16 |
| kubajj | Sandzwerg[m]: it is a security mechanism in a sense - my data centre has 10,000 nodes - if somebody accidentally executed cleaning of the whole DC and I realise within an hour, since the max_concurrent_clean is set to 50, I might have only lost ~100 nodes and not all 10k | 12:17 |
| Sandzwerg[m] | kubajj: Hmkay. But that means increasing it in principle should not cause issues in other places of the process if I have no issue with someone deleting a hundreds of servers? We have a customer who rebuilds his complete landscape of a ~300-400 nodes every other week. Cleaning only takes ~10 minutes or so but they still delete faster than that, servers end up in error etc. I already increased the limit but I'm unsure if I want to | 12:28 |
| Sandzwerg[m] | increase it further. I've seen that in latest there is already "max_concurrent_per_conductor_group" but we're a bit behind that. | 12:28 |
| kubajj | Sandzwerg[m]: we have it set to 250 per region and never had issues - how many conductor groups do you have? | 12:30 |
| Sandzwerg[m] | oh sooo many. Because we currently have one group per block of HW we buy/install etc. But every group is only a single conductor | 12:31 |
| kubajj | (I would imagine that a reasonable amount should be ok, as long as your conductors are reasonably specced) | 12:31 |
| kubajj | Sandzwerg[m]: then it should be ok ™ | 12:32 |
| Sandzwerg[m] | Alright, thanks a lot :) | 12:32 |
| TheJulia | good morning | 12:51 |
| TheJulia | So yeah, the max_concurrent stuffs are all just count gates on the maximum number of already existing nodes in that state to guard the exact case kubajj indicated. Upstream ships a relatively reasonable default although folks with thousands of servers do likely need to tune it just based upon their workloads/patterns. We've only had a handful of people bring it up as an issue/challenge/observation outside of "thank you | 12:55 |
| TheJulia | for having that knob" because occassionally runaway scripts can do some damage. | 12:55 |
| Sandzwerg[m] | Sounds good, thanks for explaining | 13:36 |
| TheJulia | fwiw, the only times people really notice it is when they are doing massive sweeps and most often are folks going "thank you this saved my cloud" which is a super weird feeling but we're glad we built a useful guardrail | 13:41 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Add a Metal3 baremetal-operator functional job https://review.opendev.org/c/openstack/ironic/+/988766 | 13:55 |
| Sandzwerg[m] | TheJulia: I offer a customer who rebuilds like 400(?) nodes in batches of 50 but now they delete and & deploy instead of rebuild, thus the delete is mostly instant but the cleaning is not and then b00m, even with a limit of 100. But that is probably also affected by https://review.opendev.org/c/openstack/ironic/+/989431 | 14:22 |
| TheJulia | yeouch | 14:22 |
| TheJulia | yeah, that is not fun | 14:22 |
| TheJulia | one of the operators I know IRL I think forces rebuilds but they tightly control *most* of the baremetal management interaction and don't let end users individually make decisions on what to precisely do with each machine. | 14:24 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-prometheus-exporter master: Fix issues with DevStack data collection https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/988442 | 14:40 |
| JayF | Sandzwerg[m]: there are also known latency issues between cleaning completing on ironic end and machines becoming available in nova-computer | 14:50 |
| JayF | Sandzwerg[m]: that's a major reason why the rebuild workflow is nice, especially if your users have already done the work of scheduling to nodes that are spread across the dc | 14:51 |
| TheJulia | Yeah, most high capacity/high load users actually prefer rebuild on some level because they are staying on the same gear | 14:51 |
| Sandzwerg[m] | They did rebuild before, not sure why they switched. I doubt I would be able to convince them to return to it :/ | 15:01 |
| Sandzwerg[m] | I'm also checking hardware from a new vendor with https://github.com/openshift-metal3/bmctest so far I don't get over enroll because ironic/sushy seems to try to use a wrong redfish path. It tries to use /redfish/v1/Systems/systems but the actual path, that I also see as member in /redfish/v1/Systems is /redfish/v1/Systems/system (singular vs plural). I doubt the path is hardcoded. Any idea how to debug this further? Is there a | 15:07 |
| Sandzwerg[m] | -vvv or something? Because if it's a bug I'd like to raise it with the vendor as soon as possible | 15:07 |
| TheJulia | There is this odd slice of folks that sometimes view "fresh" machines as better. I've heard of someone doing something similar user wise when they discovered they had worn out the flash on the storage system | 15:07 |
| TheJulia | Sandzwerg[m]: standard wise, there is a lower level link your supposed to base your next query on in the standard | 15:08 |
| TheJulia | Sandzwerg[m]: so, you sort of need to look at GET /v1/redfish/ as a starting point just to be on the safe side | 15:08 |
| Sandzwerg[m] | yeah from there the members all look fine, to my human eye and just going link by link in the browser works fine | 15:10 |
| Sandzwerg[m] | Or could it be that the rendered page in a browser has differnt information than the json? | 15:11 |
| Sandzwerg[m] | nope, curl shows me the same | 15:12 |
| TheJulia | the curl should be right, I'm wondering if it is a bug, but I'm strugglign to wrap my brain around it since I'm also in discussion a DMTF Redfish call at the moment :) | 15:15 |
| Sandzwerg[m] | How fitting :) | 15:36 |
| TheJulia | Yeah.... I guess ;) | 15:37 |
| TheJulia | Sandzwerg[m]: It might be good to open a bug somewhere about that, but if you can try to reproduce with redfish and a manual connection request, I'm wondering if the hint for the systems url is off, but just seems funky overall | 15:42 |
| Sandzwerg[m] | yeah question would be where. ironic? sushy? I see that if I use the wrong URL it doesn't work but I don't get why it's using the wrong path. But I can provide jsons and the log from the bmctest | 15:44 |
| TheJulia | well, give it a spin with sushy, trying to manually connect *without* giving a system hint so it does autodiscovery attempts | 15:45 |
| Sandzwerg[m] | Alright | 15:56 |
| Sandzwerg[m] | hmm tried https://docs.openstack.org/sushy/latest/reference/usage.html#creating-and-using-a-sushy-client-with-sessions as that works without me stating the system ID/path and yeah that works all fine so far. | 16:32 |
| Sandzwerg[m] | I'll drop for now but I'll continue tomorrow. If someone can tell me what ironic is doing with sushy at this stage I'll try to reproduce it | 16:34 |
| TheJulia | I think we just ask sushy to figure it out, which makes it weird | 16:36 |
| cardoe | TheJulia, Sandzwerg[m]: I'm gonna disagree. Sushy absolutely does it correct. | 17:42 |
| cardoe | I looked at this code not that long ago to validate and tell someone they are wrong. | 17:42 |
| TheJulia | so that doesn't explain the issue Sandzwerg[m] is encountering unless its data driven from the bmc | 17:43 |
| cardoe | Maybe the same vender | 17:43 |
| cardoe | vendor | 17:43 |
| cardoe | So sushy reads the @odata.id object for Systems. Which has been required since version 1.0 of the redfish spec | 17:44 |
| cardoe | Which is going to be a collection of systems | 17:45 |
| TheJulia | yeah, absolutely | 17:45 |
| cardoe | And if you didn't specify the endpoint, it takes the first returned value and uses that. | 17:45 |
| JayF | I mean, I've spoken here about multi-headed servers I've encountered | 17:46 |
| cardoe | oh no doubt but in that case you need to specify for sure | 17:46 |
| JayF | those blew up with weird errors because the first system in the list didn't contain much of anything we expected | 17:46 |
| JayF | and in that case, we had to specify redfish_system_id | 17:46 |
| JayF | so I wonder if Sandzwerg[m] has hardware that's pathological in a slightly different way. | 17:47 |
| cardoe | My box has "contoso" as it's first entry | 17:48 |
| cardoe | Which if I told you returns back byte for byte the same data as the sample JSON files from DMTF's site probably wouldn't shock you. | 17:50 |
| JayF | DMTF Rule #15287: "[X] You must ship a system object that fully complies with the API spec" | 17:51 |
| JayF | Now they can make the other systems whatever crazy crap they want | 17:51 |
| JayF | (^ is a joke) | 17:51 |
| cardoe | I want to laugh and cry at the same time. | 17:52 |
| cardoe | Sandzwerg[m]: your box should return something like https://opendev.org/openstack/sushy/src/branch/master/sushy/tests/unit/json_samples/system_collection.json | 17:53 |
| cardoe | If you wanna be lazy like me curl -sku <user>:<pass> https://<my.ip>/redfish/v1/Systems/ | 17:54 |
| cardoe | Assuming from your comments above your @odata.id Systems Collection does point to /Systems/ | 17:54 |
| cardoe | My point here is that Sushy IMHO is correct. If the user specifies it, use what is specified. If the user does not specify, use the first system returned by the BMC. | 17:55 |
| cardoe | Even when you have accelerator hardware augmenting the chassis or blade computer chassis, you're suppose to return the most significant system or parent system first. | 17:57 |
| cardoe | So rule 48920 trumps run 15287! | 17:58 |
| * TheJulia twitches at the use of a particular word | 18:07 | |
| TheJulia | JayF: go check 2026.1's release to see if that language works w/r/t multi-headed servers of doom | 18:08 |
| TheJulia | cardoe: yeah, I was thinking it was correct, albeit i was also reminded earlier some folks want to model everything by walking an entire bmc too, so our model might be a little different than what an OEM or bmc designer thinks too | 18:10 |
| JayF | TheJulia: ? | 18:10 |
| JayF | I am confused | 18:10 |
| cardoe | TheJulia: The language I'm quoting is from the latest spec release. | 18:10 |
| cardoe | Don't try to nerd snipe me. | 18:11 |
| cardoe | The vendor already tried. | 18:11 |
| TheJulia | https://www.dmtf.org/sites/default/files/standards/documents/DSP0268_2026.1.html#computersystem-1280 see the bullet under: 6.28.1 Description | 18:20 |
| TheJulia | cardoe: you quite literally have contoso gear!? lucky you! | 18:25 |
| JayF | awesome \o/ | 18:37 |
| JayF | I've passed that on to my downstream | 18:38 |
| JayF | I think it would've covered the last case | 18:38 |
| Sandzwerg[m] | cardoe: Here is the full chain from redfish root to Systems and member https://paste.opendev.org/show/brC2VaGoBE4hJpKGwZpF/ | 18:41 |
| Sandzwerg[m] | This is a freak system where two smaller boxes are combined to a single one. This is against the first BMC who, AFAIK also has the controll over the whole "partition" | 18:41 |
| Sandzwerg[m] | And apart from "I´d like a newline after the final closing }" the responses look fine to me? | 18:42 |
| Sandzwerg[m] | But I won't rule out that I'm holding it wrong or this system behaves differently. That's why we test it now | 18:43 |
| TheJulia | oh, wow, it really is at /redfish/v1/Systems/system | 18:45 |
| Sandzwerg[m] | yes, that naming is quiet odd. But most others I've seen are either /1/ or Systems.embedded.1 (or something like that) in the dell case | 18:49 |
| Sandzwerg[m] | It's running a modified openBMC AFAIK and if there is a bug we can report it. We already found two values who had the wrong type but I'm not sure whats wrong here. | 18:50 |
| TheJulia | I'd have to wrap my head around the code when I'm not deeply focused on other items | 19:29 |
| Sandzwerg[m] | It's all fine. Heads are important | 19:30 |
| Sandzwerg[m] | Just for the record the last part ofironic log from the bmctest including the error: https://paste.opendev.org/show/bwLfpf46EOJcv95z7t1y/ | 19:31 |
| TheJulia | okay, that is... bizzar | 19:57 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!