| janders | TheJulia w/r/t 979828: I left Konstantinos a comment (in response to yours) - will try to get him to test it. There's some downstream context here which I would like him to check. | 00:24 |
|---|---|---|
| janders | In principle, I agree with you, if unsure let's err on the side of an extra reboot - it's not like we're provisioning machines every day. They get provisioned once and run for months/years and they better run well (not so easy if config is slightly off). However provisioning performace is important for the folks, they don't like unnecessary | 00:25 |
| janders | reboots. So trying to help them make things faster where possible. | 00:25 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: [WIP] Add settable health status to ComputerSystem resource https://review.opendev.org/c/openstack/sushy-tools/+/980381 | 00:35 |
| cardoe | janders: I’m gonna try to get some time to reload the BIOS settings series into real hardware area tomorrow and test it some more. | 00:44 |
| janders | cardoe thank you, that would be great! | 00:49 |
| cardoe | JayF: I would like the glance image to be obvious as to what type it is rather than a basic Boolean. That’s what I’m saying. | 00:54 |
| rpittau | good morning ironic! happy friday ! o/ | 08:00 |
| kubajj | good morning rpittau, and Ironic! o/ | 09:02 |
| rpittau | hey kubajj :) | 09:06 |
| opendevreview | Jakub Jelinek proposed openstack/ironic master: Make local_gb an int https://review.opendev.org/c/openstack/ironic/+/980408 | 11:00 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: novadriver: Add rescue-based virtual media support https://review.opendev.org/c/openstack/sushy-tools/+/980255 | 12:15 |
| opendevreview | Harald Jensås proposed openstack/sushy-tools master: Verify power state after rescue operations https://review.opendev.org/c/openstack/sushy-tools/+/980426 | 12:15 |
| TheJulia | janders: I guess my only real worry is use of the pending/future value field is not the current value unless there *is* a reboot. Otherwise fine in principal, the issue is what was done on the idrac10 and maybe that is worthy of a request for DMTF to clarify the field's usage | 13:36 |
| nathharp | hey - is it appropriate to ask usage questions here, or should that go to the mailing list? | 13:39 |
| TheJulia | hey nathharp, go right ahead, here is fine! :) | 13:39 |
| kubajj | regarding https://review.opendev.org/c/openstack/ironic/+/980408 does anybody knows where the default value of local_gb property comes from? | 13:40 |
| nathharp | Thank you. I'm having a headache with IPA + Redfish virtual media boot + DHCPless. I've had it working on a system a while ago, but now I'm failing to get any network connectivity. Unhelpfully I also don't appear to be able to get dynamic-login using rootpwd= - I suspect due to $ and or " vs '. Is | 13:42 |
| nathharp | https://docs.openstack.org/ironic-python-agent/latest/admin/troubleshooting.html#ironic-python-agent-builder-dynamic-login still expected to work? | 13:42 |
| TheJulia | Hmm, IPA I think, looking | 13:42 |
| TheJulia | kubajj: oh, self determining although should be int actually.... | 13:43 |
| TheJulia | well, cast to string | 13:43 |
| TheJulia | its the size of the disk chosen as the root device -1 | 13:43 |
| nathharp | argh - forget my question, this time it works! | 13:43 |
| TheJulia | which is chosen by the agent based upon config or by the selection crtieria to find the default device | 13:44 |
| TheJulia | nathharp: using dynamic-login or devuser elements from dib? | 13:45 |
| TheJulia | I don't know if anyone has used dynamic-login with IPA, I would expect it to work though. Devuser is a good alternative for a one-off host troublehsooting albeit in both cases you have to build an IPA ramdisk with an element | 13:46 |
| kubajj | TheJulia: what I meant is for the test cases where something goes wrong, then it says: | 13:47 |
| kubajj | # The default value of the `local_gb` property is left | 13:47 |
| kubajj | # unchanged | 13:47 |
| kubajj | self.assertEqual(task.node.properties.get('local_gb'), '10') | 13:47 |
| kubajj | But where does the '10' come from? | 13:47 |
| TheJulia | oh, likely the test class itself | 13:48 |
| TheJulia | oh, looking at it, the test class defines four options | 13:49 |
| TheJulia | and test_root_Device_skip_list explciitly excludes the first | 13:50 |
| kubajj | TheJulia: my bad, I missed the get_test_node setting it in ironic/tests/unit/db/utils.py | 13:52 |
| TheJulia | no worries | 13:52 |
| TheJulia | nathharp: fwiw, there is a length limit to the kernel command line and what gets parsed, so always possible you might be super close to it | 13:55 |
| TheJulia | IPA I *believe* | 13:59 |
| nathharp | it ended up taking both escaping $ with a second $, but also using single rather than double quotes. With double quotes $foo was being interpreted, so only half of the string was actually being passed | 13:59 |
| TheJulia | err | 13:59 |
| TheJulia | wrong window! | 13:59 |
| TheJulia | oh joy, yeah, that is why I sort of avoid dynamic-login and lean towards devuser | 14:00 |
| TheJulia | Lets just be glad we still have a kernel command line | 14:00 |
| nathharp | indeed. Thanks anyway for your help. Now I'm in I can at least see that IPA definitely isn't applying an IP address...... | 14:01 |
| TheJulia | So, IPA itself doesn't do it, glean does | 14:02 |
| TheJulia | so, that is sort of the first place I'd look along with the syntax of the network configuration data | 14:02 |
| nathharp | sorry, yeah that's what I meant - "my IPA instance isn't getting an IP address". I'm testing a backport of https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/962673 but it looks like I'm missing glean entirely | 14:05 |
| TheJulia | oh, yeah simple-init as an image element to your image builder and that will pull in glean | 14:06 |
| TheJulia | Since your using an older branch, you may need to opt for it explicitly | 14:06 |
| opendevreview | Julia Kreger proposed openstack/networking-baremetal master: fix neutron port reconcilator https://review.opendev.org/c/openstack/networking-baremetal/+/980437 | 14:08 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2025.2: fix: MAC address handling was not always normalized https://review.opendev.org/c/openstack/ironic/+/980371 | 14:11 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: Move multicast helper to device_utils. https://review.opendev.org/c/openstack/networking-generic-switch/+/980158 | 14:13 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: Arista eos multicast vxlan https://review.opendev.org/c/openstack/networking-generic-switch/+/980159 | 14:14 |
| opendevreview | Jakub Jelinek proposed openstack/ironic master: Make local_gb an int https://review.opendev.org/c/openstack/ironic/+/980408 | 14:17 |
| kubajj | TheJulia: I did not update the other test cases, because there is so many that have either `local_gb` or `memory_mb` as strings. 🥲 | 14:18 |
| TheJulia | I guess the big thing will be to see if anything explodes, I highly doubt it will | 14:19 |
| shermanm | have a a bit of a success to share, that ssh connection reuse patch to NGS dropped our "best case" time to configure ports on os10 from 20 seconds to ~5 seconds, and we're seeing similar improvements to the median times | 14:46 |
| TheJulia | shermanm: that is awesome news | 14:49 |
| shermanm | I was just pulling the times out of the neutron wsgi logs , e.g. `"DELETE /v2.0/ports/8c729094-4b2f-4259-8590-c9126db096df HTTP/1.1\" status: 204 len: 149 time: 4.8874485"`, but it's gotten fast enough that I now need to distinguish from the "no-op" api calls to ports missing local_link_info. I had just been doing an "exclude < 5s" | 14:53 |
| shermanm | if it's of interest, I may have a PR that adds NGS timing info to a debug log or similar, I wanted to get statistics on how long each ml2 request spends waiting for a lock, waiting for switch connect, and waiting for the change to "apply", broken out by which switch | 14:55 |
| opendevreview | Merged openstack/ironic master: Support ramdisk deploy with Glance image_source https://review.opendev.org/c/openstack/ironic/+/978828 | 15:01 |
| opendevreview | Merged openstack/ironic master: Add ramdisk to autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/978829 | 15:01 |
| nathharp | TheJulia: yep, our IPA image builds were missing glean. Added that + now I'm working! | 15:03 |
| nathharp | and by missing glean, I mean missing simple-init | 15:05 |
| TheJulia | yeah, horribly named, sorry about that | 15:06 |
| cardoe | kubajj: rg -t python local_gb | grep -v tests | 15:43 |
| kubajj | cardoe: will do, thanks | 15:46 |
| cardoe | nathharp: you need to escape the $ with two | 15:47 |
| cardoe | shermanm: might be worthwhile to expose it as a prometheus /metrics endpoint | 15:48 |
| shermanm | I was thinking that, but it seemed like a larger change to try and land | 15:49 |
| nathharp | cardoe: yeah, I had done that too (without it conductor won't start), but even after that it would come out wrong: eg rootpwd=/VjVAFYGvEINAZKuIjHVMq2nNtMa1vWU0J0o1gH0/W1 instead of rootpwd=$6$gZmMmrjzEeAmDRLk$dOfw/VjVAFYGvEINAZKuIjHVMq2nNtMa1vWU0J0o1gH0/W1 | 15:49 |
| opendevreview | Clif Houck proposed openstack/ironic master: Fix bug in TBN NoMatch.__str__ https://review.opendev.org/c/openstack/ironic/+/980477 | 15:54 |
| opendevreview | Clif Houck proposed openstack/ironic master: WIP: Update TBN simulator for milestone 2 https://review.opendev.org/c/openstack/ironic/+/980478 | 15:54 |
| TheJulia | Any last reviews for end of cycle? | 16:21 |
| opendevreview | Jakub Jelinek proposed openstack/ironic master: Make local_gb an int https://review.opendev.org/c/openstack/ironic/+/980408 | 16:21 |
| TheJulia | shermanm: so, timing overall seems like a good idea to log, maybe behind a boolean option though | 16:26 |
| JayF | if my ramdisk stuff hasn't landed yet, please that TheJulia | 16:29 |
| * JayF checks | 16:29 | |
| TheJulia | not yet, next | 16:29 |
| TheJulia | -1from ci | 16:30 |
| JayF | read my comment :) | 16:30 |
| JayF | it's already fixed, I just tossed a recheck on it | 16:30 |
| TheJulia | refresh your gerrit ;) | 16:30 |
| JayF | I had to unbreak the job | 16:30 |
| JayF | we jinx'd the recheck :P | 16:31 |
| JayF | the failure was just because the autodetect job got ramdisk added to autodetect but not enabled with the default change | 16:31 |
| JayF | and the guardrail I asked steve to put it is the one I crashed into lol | 16:31 |
| opendevreview | Harald Jensås proposed openstack/networking-baremetal master: Use other_config for bridge-mapping lookup (OVN) https://review.opendev.org/c/openstack/networking-baremetal/+/980374 | 16:37 |
| opendevreview | Harald Jensås proposed openstack/networking-baremetal master: Allow hostname in l2vni_network_nodes.yaml config https://review.opendev.org/c/openstack/networking-baremetal/+/980375 | 16:37 |
| JayF | fwiw the parts I care about already landed, it'd be nice to have default-on this release but not critical for my use case | 16:47 |
| TheJulia | fair fair | 16:47 |
| JayF | we have two really good use case stories for our videos though: more control for integrated nova users (autodetect ramdisk + TBN + vxlan), networking launch for standalone | 16:48 |
| JayF | autodetect bootc for standalone, too | 16:48 |
| TheJulia | ++ | 16:49 |
| TheJulia | Reminds me, should go look at that empty deck | 16:49 |
| TheJulia | Any thoughts/feels on https://review.opendev.org/c/openstack/ironic/+/977569 and the patches attached? | 16:55 |
| JayF | I've read all the code | 16:55 |
| JayF | The lack of review is a decision moreso than a task prioritization | 16:56 |
| JayF | I can't put my finger on it, but something about those changes make me nervous | 16:56 |
| JayF | not something I can quantify so I'd never put it on the change, but enough I've looked at it twice and didn't feel like a +2 | 16:56 |
| JayF | it's a pretty risky change if you don't trust hardware may just be the heart of my uncertainty | 16:57 |
| JayF | </topic> <topic> I was telling my downstream today about a downstream thing we had a Rackspace OnMEtal | 16:57 |
| JayF | we had some devices I didn't trust to ATA erase properly, so we added "QA" to our cleaning steps | 16:57 |
| JayF | dd some random string to some random location on disk, secure erase, dd back out that random location and ensure it's not what we expect | 16:58 |
| JayF | he suggested it might be something we could do upstream. In terms of timing, it probably wouldn't add too much to cleaning, and we wouldn't enable it by default | 16:58 |
| dtantsur | Sounds cool, I'd even enable it by default if we can somehow roll it into the configdrive support | 16:59 |
| JayF | what do you mean by roll it into configdrive support? | 17:00 |
| dtantsur | Ah, wait, I'm too tired to think properly | 17:01 |
| dtantsur | You write the string right before the erasing, not in deploy time.. | 17:01 |
| dtantsur | Yep, works for me | 17:01 |
| JayF | the context of the conversation was w/r/t out-of-band disk erasing being scary so sadly won't fix that issue | 17:03 |
| JayF | but if folks are onboard I'm going to implement it | 17:03 |
| JayF | only question I have: should it be a separate clean step or just baked into erase_block_devices? | 17:03 |
| JayF | baking it in == you save 2-4 heartbeat round trips, but have no knob to turn it on/off and the error may be more confusing | 17:03 |
| dtantsur | it's odd to have a step that has a hard dependency on another step.. | 17:03 |
| JayF | separate steps == longer but more clear | 17:03 |
| JayF | dtantsur: not really, we do it implicitly with frequency: think about interactions between cleaning and in-band raid, for instance | 17:05 |
| JayF | dtantsur: the entire priority system exists because you may need ordering. We used that a *lot* at Rackspace when we had intensive in-band cleaning | 17:05 |
| dtantsur | okay, but this step would not even work with an out-of-tree cleaning step because the other step won't write the magic string, right? | 17:06 |
| JayF | so if I integrated it, it'd likely be shaped like: https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/hardware.py#L1097 line above this: dd string to disk; line below this, check for dd'd string | 17:08 |
| TheJulia | my whole concern with out of band disk cleaning is largely that for the bmc to list a device, it needs to be recognized. The bmc might not show *all* of the devices which need to be cleaned | 17:32 |
| TheJulia | example: evil data center attackers mysterious weird thumbdrive in the back of the host | 17:32 |
| TheJulia | I replied on doug's change, fwiw. Being pragmatic I +2'ed it | 17:39 |
| JayF | Yeah, our threat model is a little different as a nonpublic cloud, but I'd have a lot of trouble *ever* being comfy with OOB disk erasure in a public cloud | 18:28 |
| TheJulia | Yeah, its one of those things if you have appropriate physical controls or just... trust your DC staff its a whole different ballgame | 18:39 |
| opendevreview | Julia Kreger proposed openstack/networking-baremetal master: fix neutron port reconcilator https://review.opendev.org/c/openstack/networking-baremetal/+/980437 | 18:44 |
| cardoe | So TheJulia/JayF wrt https://review.opendev.org/c/openstack/ironic/+/977569 the original variant I had worked fine on Dell and HPE boxes. | 21:59 |
| cardoe | Tested with Dell R7615 / R740 and then HPE DL380 Gen10? or Gen11? | 21:59 |
| cardoe | The current version I don't know. I had planned on testing it today but -ETIME due to some other fires. | 21:59 |
| cardoe | If the final state doesn't make it into 2026.1 then I'll be absolutely patching it into our build though. | 22:00 |
| cardoe | Cause without that change its not possible to reset a box to a default state via clean. | 22:00 |
| cardoe | As far as disk cleaning goes, I understand the unease with ATA erase. Erasing disks is a complex problem in the modern NAND / NOR days and then rotating media with CMR. | 22:04 |
| cardoe | Fundamentally ATA erase works by implementing raw encryption and the controller has a key stored internally. ATA erase works by changing that key. | 22:05 |
| JayF | so you'd think that, cardoe, but the very bad no good DOM (direct-on-motherboard) disks they got for onmetal v1 actually, when you called for an ATA erase, it blocked for 30 minutes. We later figured out, by inspecting smart stats, they implemented a DBAN in the firmwrae. | 22:09 |
| cardoe | Yeah so that's the issue. | 22:10 |
| cardoe | I worked with NIST at the time on it. And the folks working on that were good and boxed the spec in really well. | 22:11 |
| cardoe | It was the "virtual disk atop physical disk" area that those vendors got some wiggle room on. | 22:12 |
| cardoe | And that's where I think the problem is today. | 22:12 |
| cardoe | Cause every flash drive is a virtual disk to a physical disk for example. | 22:12 |
| cardoe | So I agree with you on the write a little canary out there and see if my canary is there. It's a good way to do it. | 22:13 |
| JayF | re: your patch, knowing it was tested on dell and hpe makes me feel a bit better and I will circle back this weekend | 22:14 |
| cardoe | so patchset 3 is what ran against all the hardware | 22:22 |
| cardoe | It's up to 8 right now. I'm not certain what's changed. | 22:23 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!