| rpittau | good morning ironic! o/ | 07:08 |
|---|---|---|
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [PoC] Render the redfish interop profile in the docs https://review.opendev.org/c/openstack/ironic/+/920574 | 07:35 |
| dtantsur | On PTO until next week, see you on Tuesday! | 07:35 |
| dtantsur | The IRC bouncer will be off, reach out using email if needed. | 07:35 |
| rpittau | dansmith: enjoy! :) | 07:43 |
| rpittau | lol | 07:43 |
| rpittau | disappeared already :D | 07:43 |
| opendevreview | Merged openstack/ironic-inspector master: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic-inspector/+/920710 | 08:17 |
| sylvr | Hello ! I'm missing redfish_address in my idrac-redfish driver, but when I tried to add one (drac.host) instead of getting the IP address of the BMC it just replace it with "drac.host" which cause the verification state to fail... I'm using kayobe and I don't know what config to change | 08:36 |
| sylvr | I've been told that my BMC (idrac) are too old to support redfish too, but couldn't manage to use idrac-wsman ^^ | 08:36 |
| sylvr | here are some "logs" https://pastebin.com/R6rTJGfN | 08:41 |
| sylvr | I also have some mismatch on my driver, I'm no sure why/how I did that | 08:42 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Enable voting for codespell CI https://review.opendev.org/c/openstack/ironic-python-agent/+/920192 | 09:35 |
| cid | Hey sylvr , I will look at the log, but I won't be of much help. | 09:43 |
| cid | The TZs will align soon | 09:45 |
| cid | In the mean time, looking at more documation will likely be a good way to go | 09:46 |
| opendevreview | Merged openstack/ironic-python-agent master: Fix issues caused/found by new codespell https://review.opendev.org/c/openstack/ironic-python-agent/+/920366 | 10:03 |
| sylvr | cid: I'm reading continuously, but between kayobe and its configuration and ironic/bifrost it's hard to keep track of everything... I tried configuring bifrost/ironix to use idrac-wsman in the kayobe-config, but it won't and instead mention idrac-redfish everywhere... | 10:16 |
| cid | sylvr: I know that feeling :( | 10:33 |
| cid | sylvr: Well, I think you might not be configuring everything explicitly, hence, it's using some defaults. | 10:54 |
| cid | I might be wrong, but the problem here seems to be a mismatch, like the hardware interface for instance. | 11:00 |
| cid | I will defer to people that know more than me, I just as much confused :D | 11:00 |
| iurygregory | good morning ironic | 11:13 |
| iurygregory | sylvr, what is the model of your dell machine? | 11:14 |
| iurygregory | is iDRAC 8 or 9? | 11:15 |
| iurygregory | tonyb, I think we are good to go with https://review.opendev.org/c/openstack/project-config/+/904012 seems like openstackci is on PyPi for the ironic-prometheus-exporter | 11:57 |
| iurygregory | I didn't recover the access to pypi, so we can keep the branch I've created since I don't know if they checked already... | 11:59 |
| iurygregory | that moment you noticed you should take the day off because is public holiday and you totally forgot | 12:29 |
| rpittau | iurygregory: if you don't want it I'll gladly take it :D | 12:29 |
| iurygregory | rpittau, maybe I can switch to next week lol :D | 12:31 |
| sylvr | cid: yeah I think so too, I'm looking into getting more info on my hardware! | 12:34 |
| sylvr | iurygregory: I'm gonna check iDRAC version ! | 12:35 |
| sylvr | iurygregory: I have iDRAC 8 | 12:41 |
| sylvr | and IPMI over lan is disabled in my iDRAC config | 12:45 |
| iurygregory | I never used iDRAC with ipmi.., but based on what you mentioned the interfaces configured for your ironic node are using redfish right? | 12:46 |
| iurygregory | depending on the firmware version of iDRAC it may or may not work... | 12:46 |
| iurygregory | and since it is mapping to redfish, you would probably need to set the necessary params in the driver_info before moving forward | 12:47 |
| sylvr | well redfish seems to be the default now yes | 12:47 |
| sylvr | I tried to set the missing params (redfish_address) but I don't know why, I can't specify the redfish driver to use the iDRAC's IP per host | 12:48 |
| iurygregory | https://paste.opendev.org/show/bRUg1Er9vLdc5E7dViIp/ | 12:50 |
| iurygregory | normally this would be the configs you need to set | 12:50 |
| sylvr | the IPMI driver gets the correct IP, so if I can't manage to make redfish or idrac-wsman work, I may update the config of all my idrac to accept IPMI over lan... | 12:50 |
| iurygregory | we deprecated idrac-wsman in Caracal | 12:51 |
| iurygregory | there is a typo in my paste, /edfish/v1/Systems/System.Embedded.1 -> /redfish/v1/Systems/System.Embedded.1 | 12:52 |
| sylvr | ironic version : 23.0.2.dev6 | 12:53 |
| sylvr | okay, I got the driver info from the doc there https://docs.openstack.org/ironic/latest/admin/drivers/idrac.html , but how do you get redfish to dynamically get the IP of the iDRAC the same way IPMI does ? is it related to kayobe conf ? | 12:54 |
| iurygregory | dynamically get the ip... you mean the bmc address change from time to time? | 12:55 |
| sylvr | no, but I have like 18 node ^^ | 12:56 |
| iurygregory | ipmi gets the ip from the config you provided in driver_info | 12:56 |
| iurygregory | driver_info | {'ipmi_address': '172.16.10.20 from the paste | 12:56 |
| sylvr | yup, that was automatic, I think its from kayobe | 12:57 |
| iurygregory | oh, so you probably have a place where you define the info for your nodes right? | 12:57 |
| iurygregory | some yaml etc | 12:57 |
| iurygregory | I never used kayobe so I'm not sure | 12:57 |
| sylvr | well, there should be some sort of automatic discovery process, as I didn't specified the idrac's IP anywhere (they get their IP from a DHCP server) | 12:59 |
| sylvr | I'm going to check on the kayobe IRC | 12:59 |
| iurygregory | yeah, not sure how kayobe is doing things, maybe worth asking in the kolla channel | 12:59 |
| sylvr | yup, thanks for your insight ! | 13:00 |
| JayF | iurygregory: I feel dumb for not suggesting that to sylvr earlier /o\ | 13:00 |
| JayF | often those of us who work on the service projects don't always know the deployment projects well (or only are familiar with a single flavor of them) | 13:00 |
| iurygregory | JayF, it happens =) | 13:01 |
| sylvr | I was told to ask over here too, but issues changes, and I'm getting closer to figure out why it's not working but I think I need to change scope several times (sometimes I mess up kayobe's config, sometimes it's ironic's config...) | 13:05 |
| JayF | Yeah, and it's tough for us to know what level it's busted on too | 13:05 |
| TheJulia | GOOD MORNING | 13:05 |
| iurygregory | kayobe should probably have a way to set "redfish" details in driver info | 13:06 |
| TheJulia | err | 13:06 |
| JayF | but most openstack things, even though it can be a pain to initially configure, once it's working you're in good shape | 13:06 |
| JayF | GOOD MORNING (ugt) | 13:06 |
| TheJulia | sorry about the all caps, I guess a cat walked on to the keyboard last night | 13:06 |
| iurygregory | but if didn't, you can probably override the config after kayobe created the node | 13:06 |
| JayF | Here I thought you were just a bit cheery ;) | 13:06 |
| sylvr | iurygregory: I'd like to try to fix the driver manually, but how can I supply the IP's address of all my iDRAC | 13:10 |
| iurygregory | sylvr, well, use the same ip you have in the ipmi_address =) | 13:12 |
| iurygregory | set redfish_address to https://172.16.10.20 for the node mentioned in the logs | 13:12 |
| sylvr | that changes for every node, does that means I have to update my driver before trying to manage any single node ? | 13:12 |
| iurygregory | yup... | 13:13 |
| sylvr | hum... | 13:13 |
| iurygregory | I would change in one | 13:13 |
| iurygregory | have all the configs redfish_address redfish_username redfish_verify_ca redfish_system_id | 13:13 |
| iurygregory | and see how it goes, before changing 18 nodes... | 13:14 |
| sylvr | yup, I'm going to try that first to be sure it can be managed using redfish! | 13:15 |
| iurygregory | ++ | 13:15 |
| iurygregory | fingers crossed | 13:15 |
| JayF | We try to avoid as many "you need to update this for each node" scenarios as possible, but the BMC address is one that's very difficult to get away from | 13:22 |
| JayF | (and creds in general) | 13:22 |
| TheJulia | Also, one of those things you sort of have to manage since use of dhcp with BMCs is not really advisable. | 13:23 |
| TheJulia | and you should have varying credentials for each BMC | 13:23 |
| sylvr | TheJulia: that makes sense for security | 13:26 |
| * JayF is still adding more ironic features to his cern slides | 13:27 | |
| JayF | node history and conductor graceful shutdown came up in a chat yesterday and I realized I missed them | 13:28 |
| JayF | I don't think I fully understood how awesome conductor graceful shutdown was | 13:28 |
| JayF | (until now) | 13:28 |
| sylvr | deleting a node then cold boot the node should make it appear again (discovered and inspected) right? | 13:29 |
| TheJulia | In theory if so configured to do so, but your likely the first person who has talked about using discovery of hardware in... quite soem time | 13:31 |
| TheJulia | *some | 13:32 |
| sylvr | okaaaay it took me some time to figure out that I have to update the driver_info, in the node ! | 13:36 |
| sylvr | baremetal node set UUID --driver etc etc | 13:36 |
| sylvr | and node baremetal node create --driver | 13:36 |
| sylvr | is not what I need | 13:37 |
| TheJulia | ahh, that would quickly create confusion | 13:38 |
| sylvr | yuup, and now I understand why redfish_verify_ca=false is needed! | 13:39 |
| sylvr | and now I finally have a different error : Failed to get power state for node 8869a17e-1755-413c-9f7c-5e7acf714819. Error: Redfish connection failed for node 8869a17e-1755-413c-9f7c-5e7acf714819: Unable to connect to https://172.16.10.19/r edfish/v1/. Error: HTTPSConnectionPool(host='172.16.10.19', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c: | 13:40 |
| TheJulia | ... ouch | 13:40 |
| sylvr | do you know what this means ? x) | 13:42 |
| TheJulia | Kind of, yeah | 13:42 |
| TheJulia | so, you mentioned idrac8 gear | 13:43 |
| JayF | Everything but discovery is in-tree for Inspector as of 2024.1, right? | 13:43 |
| TheJulia | I'd *first*check the settings and certificate on the BMC. Maybe also try updating the firmware | 13:44 |
| jrosser | ^ i had a bunch of old dell r730 which needed a firmware update before i could use them with ironic | 13:44 |
| jrosser | for same SSL error reasons | 13:45 |
| TheJulia | Issue boils down to the diffie-hellman key being offered (dh) by the BMC is just insufficient | 13:45 |
| sylvr | okay, that make sense too, I've been given "old server" | 13:45 |
| sylvr | okay ! | 13:45 |
| JayF | Honestly I wonder if you could just flip ssl off | 13:46 |
| JayF | update your redfish address to remove the "s", then have ironic update it for you | 13:46 |
| JayF | I have no idea if dell hardware is capable of this | 13:46 |
| JayF | (and certainly switch it back after you update if you try this) | 13:46 |
| TheJulia | ... I know some of the hardware some vendors ship explicitly rejects attempts to let you use redfish over http | 13:47 |
| TheJulia | Turning redfish on my old supermicro was not a "quick" operation | 13:47 |
| sylvr | JayF: I tried removing the 's' already ;) and yeah, it fail to connect | 13:48 |
| JayF | Worth a shot :( | 13:48 |
| jrosser | i had a whole ton of trouble making r730 era dells work nicely with ironic | 13:49 |
| jrosser | tbh i would not do that again, ever | 13:49 |
| TheJulia | :( | 13:52 |
| sylvr | okay, so introspection rules for redfish where incomplete (on my version) and it's patched for future kayobe release, but I still have to update my iDRAC in the mean time... I hope this won't be too much trouble ^^ thanks a lot ! | 13:57 |
| rpittau | I was lurking in the background and following the discussion, seen a lot of issues with idrac8 downstream | 13:57 |
| rpittau | sylvr: idrac8 is indeed old, it's out of support since February and unfortunately the latest released firmware (2.85.85.85) is not super reliable | 13:57 |
| rpittau | I would still give it a shot to an upgrade as TheJulia suggested | 13:57 |
| sylvr | yup! | 13:58 |
| TheJulia | I seem to remember 2.76.76.76 being semi-reliable, but that is literally the version which added "basic redfish" support | 13:58 |
| TheJulia | or am I confusing that with basic virtual media | 13:59 |
| TheJulia | I might be mixing that up | 13:59 |
| jonmills_nasa | Hello Ironic team, I have a question about using ironic.hardware.interfaces.bios = redfish with Supermicro servers (X11 or X12 generation). Is this known to work? | 14:04 |
| jonmills_nasa | I am able to curl the redfish URI for the bios and see the settings there.... | 14:05 |
| jonmills_nasa | However, running 'openstack baremetal node bios setting list <node_name>' results in zero output (not an error...just nothing at all) | 14:06 |
| TheJulia | jonmills_nasa: what state are the nodes in? | 14:08 |
| jonmills_nasa | TheJulia: manageable | 14:10 |
| TheJulia | jonmills_nasa: so we don't directly at the time of request proxy through to the BMC, because that would be bad and create a vulnerability. We cache when we move nodes through cleaning if memory serves | 14:10 |
| TheJulia | so if you trigger cleaning, ironic should cache the settings | 14:10 |
| jonmills_nasa | TheJulia: interesting. Is it possible to update the settings on SMC hardware using this Redfish interface, and might that trigger the caching of the settings? | 14:11 |
| TheJulia | The only place where ironic will break that pattern is setting boot devices on demand | 14:12 |
| TheJulia | You'd have to get the nodes into cleaning if memory serves, I don't know if anyone has used or tried that on supermicro gear | 14:12 |
| TheJulia | iurygregory: you might know ^ | 14:12 |
| * iurygregory reads | 14:13 | |
| TheJulia | jonmills_nasa: I *do* believe setting does result in caching first in the use model, but it still involves that clean step exec if memory serves | 14:13 |
| iurygregory | ok, using redfish with Supermicro X11 and X12 right? | 14:15 |
| jonmills_nasa | Yessir | 14:15 |
| iurygregory | we tested in X11 (but virtualmedia) | 14:15 |
| iurygregory | so I would assume it would just work (vmedia requires another license fwiw) | 14:15 |
| jonmills_nasa | All ours are licensed. I'll have to test out a few things | 14:16 |
| jonmills_nasa | Thank you for your help | 14:17 |
| iurygregory | I don't remember problems with retrieving bios settings from it | 14:17 |
| TheJulia | two quick and easy tempest test fixes: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/918462 and https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/919762 | 14:36 |
| opendevreview | cid proposed openstack/ironic master: GRUB conf template compatibility with arm server https://review.opendev.org/c/openstack/ironic/+/920836 | 14:46 |
| opendevreview | cid proposed openstack/ironic master: Flexible IPMI credential persistence method configuration https://review.opendev.org/c/openstack/ironic/+/917229 | 14:57 |
| opendevreview | cid proposed openstack/ironic master: Flexible IPMI credential persistence method configuration https://review.opendev.org/c/openstack/ironic/+/917229 | 14:59 |
| opendevreview | cid proposed openstack/ironic master: Flexible IPMI credential persistence method configuration https://review.opendev.org/c/openstack/ironic/+/917229 | 15:00 |
| JayF | So, question: I've been poking at notifications today | 15:25 |
| JayF | and it looks like in some cases, we might be putting a lot of extranneous information into a notification | 15:25 |
| JayF | e.g. step description | 15:25 |
| JayF | Is that intentional? Specifically asking re: description | 15:25 |
| TheJulia | Yeah, the couple folks I know that use notifications tend to actually be making decisions and triggering processes based on what we might consider extraneous. Then again, the description might be one which truly is | 15:26 |
| JayF | description absolutely made me WTF | 15:32 |
| JayF | but it's also kinda an API? | 15:32 |
| JayF | so I'm not sure we could change it anyway | 15:32 |
| JayF | if it ends up being a pain at scale for any deployers, we can consider our options | 15:33 |
| TheJulia | kind of, yeah | 15:36 |
| JayF | Refreshing a question from earlier today: I'm writing a slide up on inspector deprecation; AFAICT the only real piece missing from in-tree inspection at this point is discovery (which we know is mildly problematic) | 15:47 |
| JayF | just looking for someone to ack that, as the docs label in-band in-tree inspection as experimental | 15:47 |
| JayF | but I can't see anything that would suggest it's not feature complete | 15:47 |
| TheJulia | inspection rules? | 16:05 |
| rpittau | good night! o/ | 16:06 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: reboot the node in basic ops tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/918462 | 16:27 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Exclude ramdisk tests with tinycore in uefi mode https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/919762 | 16:27 |
| opendevreview | cid proposed openstack/ironic master: Provision ARM (aarch64) fake-bare-metal-vms https://review.opendev.org/c/openstack/ironic/+/915441 | 17:30 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Handle invalid configuration for vif test execution https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/919544 | 17:34 |
| TheJulia | Curious, does anyone know of anyone using vendor passthrough stuffs actively | 18:24 |
| iurygregory | TheJulia, downstream we provide support for the EventSubscription via vendor passthrough | 18:27 |
| iurygregory | this is the only case I remember | 18:28 |
| TheJulia | off of the node right? | 18:28 |
| iurygregory | yeah | 18:28 |
| *** clarkb is now known as Guest8089 | 19:49 | |
| *** Guest8089 is now known as clarkb | 20:04 | |
| cid | o/ | 20:29 |
| *** clarkb is now known as Guest8094 | 21:15 | |
| *** Guest8094 is now known as clarkb | 21:19 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!