| hjensas | cardoe: Regarding the dynamic segments inclusion in NetworkContext, I'm ok with either: a) your reverse order fixed backport + tbachman's revert or b) the other fix I proposed. - I have a feeling tbachman is more happy with a) so I proposed that. | 09:00 |
|---|---|---|
| *** darmach3 is now known as darmach | 09:58 | |
| *** darmach0 is now known as darmach | 10:13 | |
| tbachman_ | Hi TheJulia ! I had a question about a patch. | 11:20 |
| tbachman_ | For https://review.opendev.org/c/openstack/networking-baremetal/+/977329, the release of the dynamic segment is done in a postcommit. Was there a reason you used the postcommit instead of the precomit? | 11:20 |
| opendevreview | cid proposed openstack/networking-baremetal master: Actually ignore devices with no driver configured https://review.opendev.org/c/openstack/networking-baremetal/+/1002480 | 12:32 |
| cardoe | tbachman_: other mechanisms can block on a precommit so now you’ve deleted but then we should be rolling back. I think we would just need to test that much better. | 12:42 |
| tbachman_ | k. I was just thinking that the advantage to keeping it in the precommit is that all DB state (including the segment) gets committed or rolled back atomically. | 12:43 |
| tbachman_ | I think the intent for MD's for the precommit was to keep those methods down to a minimum, in order to not block that processing path (not saying that all MDs honor that intent - just that I think that was the goal) | 12:45 |
| TheJulia | Good morning | 12:52 |
| tbachman_ | TheJulia: good morning! | 13:09 |
| TheJulia | I'm on a slow start this morning, caffine hasn't quite yet hit me :) | 13:10 |
| * tbachman_ is fully caffeinated | 13:19 | |
| TheJulia | It is still dark outside.... | 13:20 |
| TheJulia | Also, precommit is just *supposed* to be a preflight validation as I understand it. | 13:44 |
| TheJulia | FWIW, Reverse order unwinding was the agreed upon path forward to properly do this stuff during the last PTG | 13:47 |
| opendevreview | cid proposed openstack/networking-baremetal master: Actually ignore devices with no driver configured https://review.opendev.org/c/openstack/networking-baremetal/+/1002480 | 14:19 |
| opendevreview | cid proposed openstack/networking-baremetal master: Actually ignore devices with no driver configured https://review.opendev.org/c/openstack/networking-baremetal/+/1002480 | 14:24 |
| blanson[m] | Hello ironic ! | 14:28 |
| blanson[m] | quick question, are you guys aware this behavior with redfish on HPE systems, where the machine will stop seeing the storage controllers when powered down because ???? which causes the raid config to fail if the machine is powered-off ? or is it the first time you hear about it ? I've discovered it while trying to provision ProLiant DL365 Gen11, made a quick workaround with a runbook task that reboots the server so it's always on | 14:30 |
| blanson[m] | but I wonder if it's a storage controller problem, an HPE problem, or bit of both ? | 14:30 |
| cardoe | Likely ambiguity in the spec allows the vendor to do whatever they want. Or maybe no ambiguity and the vendor does what they want. | 14:33 |
| blanson[m] | maybe even a me problem and I missed something stupid ? | 14:34 |
| cardoe | nah. | 14:34 |
| cardoe | It's philosophy difference between HPE and Dell for example. | 14:35 |
| cardoe | Dell's BMC speaks to the components and caches data which allows redfish queries to be more stable and provide data back a bit quicker. | 14:35 |
| cardoe | While HPE gets all the data in real time | 14:36 |
| cardoe | The design problem when driving HPE hardware via API is that you need to power it on but you don't want it booting. You want the box to power on and initialize all the subsystems and hold there. | 14:37 |
| cardoe | It's actually some of the same problems around Dell's and their firmware updates. | 14:37 |
| cardoe | The firmware update job for some components isn't "completed" until something calls UEFI ExitBootServices. | 14:38 |
| cardoe | Certain boot loader configurations don't even call ExitBootServices and let the Linux kernel call it. | 14:39 |
| cardoe | But back to your example of configuring the RAID... there might not even be something on the host to boot. | 14:40 |
| cardoe | Historically Ironic used IPA and booted into that always. But as these operations support more of these Out-of-Band changes, we've been changing the APIs to allow for that. But we're caught by the issue above. | 14:40 |
| cardoe | I think it's a good call out. It's worth filing a bug. | 14:41 |
| cardoe | Just mentally I've thought about writing a little UEFI module in the past which boots up and just sits there. And maybe we can use that with Ironic to give these boxes a state of boot up and hold. | 14:42 |
| TheJulia | blanson[m]: so, yes we've seen it. To cardoe's point HPE wants to represent current state over PLMD which is the i2c bus level signaling protocol, so if there is no power to the device, the device doesn't respond to pldm commands much less can't even be reached via PLDM. | 14:45 |
| TheJulia | (also, the pldm accessible devices basically drops to the NIC interfaces on the motherboard which have standby power.) | 14:45 |
| TheJulia | If you read into pldm signaling as well, that device initalization only happens once power to the main board is fully turned on and the entire substrate system begins to initialize. | 14:46 |
| TheJulia | Hopefully that paints the entire picture! | 14:47 |
| blanson[m] | hum, yh got u, I didn't know about this philosophy diff between the 2. TheJulia and even then, the storage controller takes its sweet time to be available, it's a mess. well, probbly my solution of forcing the node up is the least awful one then ? this + having a polling step that wiats for the storage route not to return 400 anymore :( | 14:49 |
| TheJulia | So, we've mentally had this concept of hold/wait, it wouldn't be awful to extend the wait to be a "wait for some subsystem to be online in the bmc", but would of course require some code | 14:51 |
| cardoe | It's not terrible but it's less than ideal for you as the end user. | 14:51 |
| TheJulia | sort of yeah, but its the right knob if your trying to deploy or do service steps, because you can saddle the step right after power_on | 14:51 |
| JayF | that kinda logic might need deferred tasks, yeah? | 14:52 |
| TheJulia | and just shit and hold. Its not great obviously, but it does sort of address the challenge if your needing to apply storage/raid config or do firmware upgrades facilitated over pldm from the bmc to the storage controller | 14:53 |
| TheJulia | JayF: basically, yes | 14:53 |
| TheJulia | JayF: this is where I remind the world that a first pass is up for review ;) | 14:53 |
| TheJulia | It would be relatively trivial for a step to reschedule itself of sorts in a "wait for storage subsystem to be online" which could be represented as a step and become a deferred task which holds the node provision state as-is until it *can* progress | 14:54 |
| cardoe | JayF: I planned on building on the deferred tasks for this to work | 14:54 |
| TheJulia | my dream of solving neutron callback issues solved by john the tuba guy's patches. | 14:54 |
| TheJulia | in a poll model, unfortuantely :( | 14:54 |
| cardoe | what patches? | 14:55 |
| cardoe | blanson[m]: in fact since you've got some HPE gear that I don't have... wanna confirm something for me.. | 14:55 |
| cardoe | curl https://yourbox/redfish/v1/Systems/System.Embedded.1/ | jq -r .BootProgress | 14:56 |
| cardoe | You might have to change that system identifier for HPE cause that's the Dell one. | 14:56 |
| TheJulia | cardoe: https://review.opendev.org/q/topic:%22deferred_tasks%22 | 14:56 |
| cardoe | If you throw that curl in a loop against your box while you're working on it. | 14:56 |
| TheJulia | fwiw, it won't be System.Embedded.1, but the serial number of the host | 14:57 |
| cardoe | Yeah that's it. | 14:57 |
| cardoe | So SystemHardwareInitializationComplete is the state that I've found PLDM to be fully alive. | 14:57 |
| TheJulia | That *should* be complete | 14:58 |
| TheJulia | err | 14:58 |
| TheJulia | correct | 14:58 |
| TheJulia | *however*, if you read the pldm spec | 14:58 |
| TheJulia | initailziation and alive is disjointed from "ready" | 14:58 |
| cardoe | A lot of hardware likes to transition through OEM for a bit. | 14:58 |
| cardoe | So I yoloed a UEFI module ages ago and it holds the box at OSBootStarted | 14:59 |
| TheJulia | because for pldm to say its initialized, its only able to switch through the device, i.e. the i2c bus switch *and* the endpoint is responsive, but hasn't reached the running state where the device is ready for work | 14:59 |
| cardoe | and everything is actually alive and working at that point cause it's gotta be | 14:59 |
| TheJulia | yup | 15:00 |
| cardoe | There's some OS state field as well which it poked | 15:00 |
| TheJulia | less supported, afaik | 15:00 |
| cardoe | I think I reported something like DougHold as the value | 15:01 |
| TheJulia | but, if memory recalls the UEFI pre-boot state requires pldm and hardware to be fully online to reach as well. | 15:01 |
| cardoe | But I just poked someone internally and they confirmed they were able to see that on Dell hardware. | 15:01 |
| cardoe | The idea at least (once I clear off some of the other bits on my plate) was to clear that up a little bit and work on top of the deferred tasks to get Ironic to boot it up and hold the box at that point and mark steps as needing it held and checking for the box to be there as a requirement step. | 15:03 |
| TheJulia | Yeah, OSBootStarted is actually a flag which the OS toggles by exiting UEFI runtime by a hardware driver getting loaded. | 15:03 |
| cardoe | Yep | 15:03 |
| cardoe | Er.. | 15:03 |
| cardoe | No. That's UEFI ExitBootServices which gives you UEFI runtime services | 15:03 |
| cardoe | OSRunning can even give up UEFI runtime as well. | 15:04 |
| TheJulia | Well, you still have UEFI runtime then | 15:04 |
| cardoe | Yep. | 15:04 |
| TheJulia | its once you load a driver, UEFI is *supposed* to begin self-unwinding | 15:04 |
| TheJulia | anyway, I don't have the state tables and their mappings handy in front o fme | 15:05 |
| cardoe | I used to have a crazy whiteboard with this. | 15:05 |
| cardoe | That was so dried up that it'd never erase. | 15:05 |
| cardoe | But that was $JOB-1 | 15:05 |
| cardoe | TL;DR we'll get there some day | 15:06 |
| TheJulia | Yeah, the only way it all ever "clicked" for me was reading a pile of PLDM documentation | 15:09 |
| blanson[m] | cardoe: I get { "LastBootTimeSeconds": 142.0, "LastState": "OSRunning" } | 15:17 |
| TheJulia | I'm drawing a blank, but I remember at least vendor doesn't properly update LastState | 15:27 |
| TheJulia | FWIW, my reasoning for wiring in sushy support for BootProgress was cases exactly like this | 15:27 |
| opendevreview | cid proposed openstack/networking-baremetal master: Validate device options on startup https://review.opendev.org/c/openstack/networking-baremetal/+/1002522 | 15:29 |
| clif | cardoe: are you still messing with TBN? run into any problems there? | 17:45 |
| cardoe | ish... haven't touched it in a few days / week | 17:48 |
| cardoe | yes there was an issue and I posted a patch | 17:48 |
| cardoe | https://review.opendev.org/c/openstack/ironic/+/1002353 | 17:48 |
| cardoe | But I haven't actually tested it through the flow. | 17:48 |
| cardoe | basically the portgroup was created without the physical_network which is supposed to remain in lock step with the ports involved. | 17:49 |
| JayF | I coulda swore I put a vote on that | 17:49 |
| TheJulia | huh, a vote on what? | 17:59 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/1002353 (and I did; after making that comment) | 19:23 |
| cardoe | I still need to try it out. | 19:53 |
| JayF | mark it wip or W-1 if you don't think it's covered in tests and don't feel confident without manual testing | 19:54 |
| JayF | please :) | 19:54 |
| cardoe | Well I know it’s correct from an API usage standpoint. | 20:15 |
| cardoe | What I don’t know is if the dynamic port group creation by TBN comes through those code paths. | 20:15 |
| cardoe | I was able to reproduce the issue via API calls and this fixed that case. | 20:16 |
| JayF | OH | 20:17 |
| JayF | so this was happening *at runtime* | 20:17 |
| JayF | port gets added to dynamic portgroup | 20:17 |
| JayF | no physnet | 20:17 |
| JayF | kaboom | 20:17 |
| JayF | that's not only a fix then, that should be backported | 20:17 |
| cardoe | That's why I flagged it as should backport | 20:21 |
| cardoe | I even recall we went back and forth a little bit on the reviews of this because we had to get the behavior right with different API versions and upgrade. | 20:21 |
| cardoe | The first patch I did was the easiest cause it mutated the object in the validate_portgroup() method but that felt wrong. So I created a helper function from where we set it in 1 place. Called the helper there. And then in the path that was missed called the function there. | 20:23 |
| cardoe | Claude wrote me the tests and I asked it to ensure the TBN path was covered which it said it was. | 20:23 |
| cardoe | But I feel like this past week it's been a little overly confident with a blank stare in its eyes. | 20:24 |
| TheJulia | Was there anyhthing for us to update in relation to user survey? | 21:20 |
| JayF | I don't think so | 21:25 |
| * TheJulia wonders if it is nap time | 21:27 | |
| TheJulia | ... or at a minimum, migraine meds | 21:30 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!