| opendevreview | Adam McArthur proposed openstack/ironic-specs master: Add emergency bulk operations https://review.opendev.org/c/openstack/ironic-specs/+/952533 | 01:15 |
|---|---|---|
| adamcarthur5 | Hi TheJulia, the spec is in a place where I would be happy to receive another review + suggested next steps. cc cid | 01:17 |
| adamcarthur5 | I would love advise on https://review.opendev.org/c/openstack/ironic/+/953352 if possible folks | 03:01 |
| adamcarthur5 | cc stephenfin | 03:01 |
| TheJulia | adamcarthur5: sure, I'll try to take a look tomorrow | 03:13 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix Sensor Data values https://review.opendev.org/c/openstack/ironic/+/953395 | 03:36 |
| opendevreview | Merged openstack/sushy master: Add retry logic to connection failures https://review.opendev.org/c/openstack/sushy/+/953248 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Create service plugin to configure security groups https://review.opendev.org/c/openstack/networking-generic-switch/+/949834 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Support security groups in netmiko base class https://review.opendev.org/c/openstack/networking-generic-switch/+/949835 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Add security group support to netmiko_cisco_nxos https://review.opendev.org/c/openstack/networking-generic-switch/+/952314 | 03:59 |
| opendevreview | Merged openstack/ironic master: Fix Sensor Data values https://review.opendev.org/c/openstack/ironic/+/953395 | 06:04 |
| rpittau | good morning ironic! o/ | 07:11 |
| queensly[m] | Good morning | 07:31 |
| abongale | Good morning ironic! | 08:29 |
| mumesan[m] | Good morning! | 08:31 |
| Continuity | Morning | 08:38 |
| dtantsur | TheJulia: "eventlet hiding a lot of stuff" is what I suspected too | 09:35 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Graceful way for hardware managers to ignore certain devices https://review.opendev.org/c/openstack/ironic-python-agent/+/954024 | 10:23 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: Graceful way for hardware managers to ignore certain devices https://review.opendev.org/c/openstack/ironic-python-agent/+/954024 | 10:29 |
| cid | Ironic, o/ | 11:11 |
| cid | I see this new possible "quick fix" bug: https://bugs.launchpad.net/ironic-python-agent/+bug/2115721. Just wondering if this is something we think is worth patching upstream? It seems small enough to be handled downstream if needed, especially given the temporary nature of the --experimental_verify flag in FIO. | 11:11 |
| cid | adamcarthur5, I will ensure to review your updates ++. | 11:12 |
| cid | TheJulia: I indeed will love to listen in :) | 11:13 |
| stephenfin | adamcarthur5: replied | 12:15 |
| * TheJulia drinks coffee while listening to folks have a managerial sort of meeting | 12:21 | |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add Tempest tests for inspection rules in Ironic https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/951761 | 12:42 |
| TheJulia | cid: dtantsur: do y'all have time to chat? | 13:17 |
| dtantsur | Let me check the state of cooking.. | 13:18 |
| TheJulia | y'all being "southern US" for "you all" | 13:18 |
| dtantsur | yes, I am | 13:21 |
| cid | If you were to start the meeting now, I will be able to join in around ~20mins. | 13:21 |
| TheJulia | I have another meeting in ~40 | 13:21 |
| TheJulia | I can always sync with dmitry and then sync with you cid, if that works | 13:21 |
| cid | So, short meeting! | 13:21 |
| cid | TheJulia. Works! | 13:21 |
| TheJulia | hopefully | 13:21 |
| TheJulia | ok | 13:21 |
| TheJulia | dtantsur: https://meet.google.com/ofx-avoq-eni | 13:21 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Add schema for inspection rules API (versioning) https://review.opendev.org/c/openstack/ironic/+/954038 | 13:45 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Add schema for inspection rules API (requests) https://review.opendev.org/c/openstack/ironic/+/954039 | 13:45 |
| dtantsur | TheJulia: okay, yeah, we have stuff: https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues | 13:51 |
| dtantsur | I'm still thinking how I could wire something compatible *before* we pull the trigger on migration | 13:52 |
| dtantsur | I guess it would be a (green) thread in the conductor that is reading from the queue and invoking actual methods | 13:52 |
| dtantsur | well, that won't allow us to read the response in a sane manner, will it | 13:53 |
| TheJulia | I'd suspect multiprocessing might still hold the answer in the multiprocessing style, in other words maybe have two parts using the same object initialized by multiprocessing | 13:54 |
| dtantsur | yeah, it seems to have some primitive wrappers around unix sockets or pipes https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients | 13:55 |
| TheJulia | for example, on load of conductor/rpc_service.py, create the queue and have each side work it via the object | 13:55 |
| TheJulia | that way its initailized upfront and each side can interact with it indepedently | 13:55 |
| dtantsur | mmm, we'll have fun time matching requests to responses | 13:57 |
| dtantsur | it could be easier to actual load the whole JSON RPC mechanism, just through a unix socket | 13:57 |
| TheJulia | yeah | 13:57 |
| * dtantsur will think more, walkie time now | 13:57 | |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/ironic master: Add manual clean and automated verify steps to set BMC clock via Redfish Manager https://review.opendev.org/c/openstack/ironic/+/953477 | 14:11 |
| TheJulia | rofl, in doing what I discussed, I can see why some of the other weirdness occured | 14:17 |
| TheJulia | which means eventlet was *never* completely executing service startup | 14:17 |
| Sandzwerg[m] | Hello wonderful people of ironic, as usual I have a question. I'm still trying to get secure boot working. I have issues with the centos images, with the pre-build images by you as well as when I build it myself with the ironic-agent-builder. But centos 9 seems to use a real old shim version so that might be related. When I try it with ubuntu noble the bootloader/esp seems to work but then it doesn't find kernel & ramdisk, if I | 14:37 |
| Sandzwerg[m] | try to build it with the ubuntu-signed element that fails because in noble the package name changed. (From what I found it seems to be the normal kernel images by know). Is there a known working esp & kernel & ramdisk for secure boot? If not what would be the easiest way to change/adapt the ubunut or ubuntu-signed element? I briefly looked at the docs but I don't think I want to build something from the ground up I just to make | 14:37 |
| Sandzwerg[m] | sure some packages like shim are included. But maybe missunderstood the documentation and it's easier then I thought? I'm grateful for any pointers. | 14:37 |
| TheJulia | Sandzwerg[m]: I'll try to respond, but it may be ~45 minutes, need to walk dog and have a meeting right after getting the dog back in side | 14:40 |
| Sandzwerg[m] | Thanks, have a good walk :) | 14:42 |
| TheJulia | okay, I think the issue your hitting signing issues. So the tl;dr is each vendor's shim only recognizes their own signed artifacts because it only knows their signing key. The other aspect it sounds like you might be hitting, and percieving it as "old shim" is the signing keys doing the verification on the firmware side are supposed to recognize each signed msft key, there are two of them, and shim may ship from vendors | 14:59 |
| TheJulia | with different versions. | 14:59 |
| TheJulia | That different version is supposed to be recognized by the firmware, but might not be, so it might result in some weirdness | 15:00 |
| TheJulia | There is also an aspect with VMs and what ships, so I guess the question is what are you trying to boot with secure boot. Knowing that will help ground the conversation as well :) | 15:00 |
| TheJulia | dtantsur: approx 298 tests use _start_service ;) | 15:01 |
| Sandzwerg[m] | In this case I have a dell R760 node. If I take the centos9 iso and boot that I think I was able to boot it with secure boot. At least I'm in general able to boot secure boot enabled stuff if I deploy it before. But now I need to get IPA to support secure boot. To my understand that means the ESP that I build must fit to the kernel & ramdisk I use for IPA and both need to include shim & must be signed. | 15:11 |
| Sandzwerg[m] | My first issue was the ESP. For Centos I still have no esp that "works" even when I build it with the (IMHO) correct shim in a secure booted centos9 VM | 15:12 |
| Sandzwerg[m] | For ubuntu I now have a esp that doesn't give me a "there was an error with secure boot" (or something along these lines) right at the start, also build in a secure boot enabled & booted ubuntu VM | 15:14 |
| Sandzwerg[m] | But then the ramdisk & kernel are not found. My current working theory: because it's not build with shim as I didn't saw shim during the install log | 15:16 |
| opendevreview | Merged openstack/ironic-specs master: Add support for standalone network configuration https://review.opendev.org/c/openstack/ironic-specs/+/952126 | 15:26 |
| Sandzwerg[m] | Like, if I sent the output of the ironic-python-agent-builder and grep for packages I find stuff related to efi, grub, kernel but no shim, so my current theory is that shim is missing | 15:51 |
| Sandzwerg[m] | I need to drop now but I'll be back tomorrow. Have a nice timezone o/ | 16:01 |
| dtantsur | TheJulia: uuuuugh | 16:09 |
| * TheJulia appears like an emergency medical hologram | 16:10 | |
| TheJulia | "Please state the nature of the technical emergency" | 16:10 |
| dtantsur | lol | 16:10 |
| cid | lol | 16:11 |
| TheJulia | Sandzwerg[m]: reading | 16:11 |
| TheJulia | Sandzwerg[m]: so, ubuntu failing that way seems super weird, how are you booting? virtual media? | 16:12 |
| TheJulia | Sandzwerg[m]: what might help is a file list of the the media and the content layout in the esp image | 16:12 |
| Sandzwerg[m] | Yes this is all virtual media, sorry I forgot to mention it | 16:13 |
| TheJulia | The caveat which can also exist I guess is preferred file paths can be burned into shim as well | 16:14 |
| TheJulia | Each vendor's shim is vendor customized | 16:14 |
| dtantsur | TheJulia: forgot to say: wait_for_start existed because otherwise we're signaling systemd that the start-up was successful before the actual start-up | 16:17 |
| dtantsur | I guess we're regressing in this area now, this is worth adding to known issues or something | 16:18 |
| TheJulia | yeah, absolutely | 16:18 |
| dtantsur | (assuming we even have the systemd notification still) | 16:18 |
| TheJulia | since we're forking off a thread, yeah. I think the idea moving forward is to try and enhance our pre-flight checking as much as possible for known issues and fail before the new process is launched | 16:19 |
| TheJulia | Anyhow, one step at a time | 16:19 |
| dtantsur | yeah, I guess stuff like config or driver initialization can be done much earlier than init_host | 16:21 |
| TheJulia | Config is, driver init init is a bit harder and in reality it might end up being done twice, one just as a "sanity check for purposes of failing start" | 16:28 |
| dtantsur | yeah, make sure all entrypoints are there. Maybe a new method to validate configuration only. | 16:29 |
| TheJulia | ++ | 16:30 |
| dtantsur | btw if oslo.service doesn't have systemd support at all any more (and it looks so), we'll break bifrost and any similar systemd unit configurations | 16:33 |
| dtantsur | they're set up to wait for sd_notify | 16:33 |
| TheJulia | Yeah, I've got an item on my team's backlog to start trying to figure out what/how we can move that forward | 16:34 |
| dtantsur | I think it's a matter of copying the relevant code from the eventlet backend, which got lost for some reason | 16:35 |
| dtantsur | unless it's in cotyledon already | 16:35 |
| TheJulia | That i didn't check | 16:35 |
| TheJulia | I just created a story for someone to dig into it | 16:35 |
| dtantsur | ++ | 16:35 |
| TheJulia | I don't know if it will be on our plans in the next coupld of sprints though, I think undrestanding wise, we will want to be able to take our stack of patches to move to threading and have an a/b comparison to corretly inform tuning/docs | 16:36 |
| TheJulia | ooooohhh ahhhhhh now only 113 tests fail! | 16:43 |
| dtantsur | progress! | 16:43 |
| TheJulia | should be better on this next run. The furstraitng thing is some of the tests break/expect the state resolution logic to have also executed (ugh!) | 16:46 |
| TheJulia | 110.... | 17:15 |
| TheJulia | dtantsur: back to no steps returned :\ At least getting closer https://www.irccloud.com/pastebin/8mhw1E3g/ | 17:29 |
| Sandzwerg[m] | <TheJulia> "The caveat which can also..." <- So far to me it seems that shim is missing in the kernel & ramdisk image but is present in the esp. | 18:00 |
| TheJulia | 24 tests... | 18:03 |
| TheJulia | yeah, it doesn't need to be in the image itself | 18:03 |
| TheJulia | but... if its not there then weirdness can occur | 18:03 |
| TheJulia | uhhh in any code expecting it to be a filesystem | 18:03 |
| Sandzwerg[m] | I'm trying to figure out how I can edit a dib element to include it, or that's the plan for tomorrow | 18:06 |
| TheJulia | its a package... shim-signed I believe | 18:07 |
| TheJulia | so, easy short term test, add -p shim-signed | 18:08 |
| TheJulia | to the dib build command line | 18:08 |
| TheJulia | 21 tests | 18:09 |
| TheJulia | muahahaha, 11 | 18:46 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Replace GreenThreadPoolExecutor in conductor https://review.opendev.org/c/openstack/ironic/+/952939 | 22:58 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Set the backend to threading. https://review.opendev.org/c/openstack/ironic/+/953683 | 22:58 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Cleanup tests for eventlet migration https://review.opendev.org/c/openstack/ironic/+/954073 | 22:58 |
| TheJulia | cid: I touched up the green thread pool executor patch and put it in line with cleaning up the tests | 22:58 |
| TheJulia | my backend swap is going to need the A/B performance comparision testing (and more unit tests fixed/retooled since we have to go from booleans for some things to flags for concurrency/interaction across threads. | 22:59 |
| * TheJulia calls it a day | 23:00 | |
| TheJulia | err, mabe not | 23:11 |
| TheJulia | maybe not | 23:11 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Set the backend to threading. https://review.opendev.org/c/openstack/ironic/+/953683 | 23:15 |
| TheJulia | cid: dtantsur: unit tests pass on ^, so we'll just need to figure out single process service stuff for inter-process comms, and then benchmark a before and after if I understand where w'ere at correctly. | 23:16 |
| * TheJulia really goes away now | 23:16 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!