| TheJulia | Anyway, either some help or future hardware type would be helpful. Downside of snmp is even though it is old and everything else, it is still quite popular. | 00:04 |
|---|---|---|
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: [WIP] Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 01:38 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 04:23 |
| opendevreview | OpenStack Proposal Bot proposed openstack/ironic master: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic/+/923809 | 04:43 |
| rpittau | good morning ironic! o/ | 07:23 |
| rpittau | TheJulia: thanks for the email on snmp | 07:24 |
| rpittau | JayF: I believe I shared it all here https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/921998, plus some patches that are merged on DIB, I see you added the one for the profile change which I didn't have the time to do (downstream priorities overrule) | 07:26 |
| iurygregory | good morning ironic | 09:20 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 10:49 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 11:02 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 11:20 |
| opendevreview | Jacob Anders proposed openstack/sushy-tools master: Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 11:24 |
| sylvr | Hello ! it seems I have an issue with my ironic service being unable to store/process the introspection data from my IPAramdisk | 12:22 |
| TheJulia | sylvr: what are the symptoms your encountering? | 12:44 |
| sylvr | all my nodes are stuck in enroll state and the driver_info field are empty | 12:46 |
| sylvr | when checking the logs on the IPA, everything looks fine (to me at least) | 12:46 |
| sylvr | and the logs on the seed/bifrost don't report error | 12:47 |
| TheJulia | So your expecting inspection to populate all of your driver_info fields? | 12:50 |
| sylvr | no, I configured them with kayobe (username/password) but the ipmi_address should be populated by node with the result of the introspection right ? | 12:52 |
| TheJulia | No guarentee it will and far less likely on newer hardware because that requires the inband ipmi bridge to be enabled *and* the host to be configured with a static (non-dhcp/dynamic) address which can be determiend over the IPMI bridge | 12:52 |
| sylvr | I'm using old hardware | 12:53 |
| TheJulia | Furthermore you cannot inspect a node we have zero information about | 12:53 |
| sylvr | the node are registered in bifrost, they're booting through PXE the IPAramdisk and the heartbeat is OK | 12:54 |
| TheJulia | so a network port is known | 12:54 |
| TheJulia | is the ipmi inbahd interface enabled on the hardware? | 12:55 |
| TheJulia | inband | 12:55 |
| sylvr | I'm guessing yes since I can run "ipmitool lan print" locally on the node (after ssh on the IPA) and get the BMC addresses | 12:56 |
| TheJulia | okay | 12:56 |
| TheJulia | so did you create an introspection rule to publish the ipmi_address? or do you have hardware discovery enabled? | 12:56 |
| TheJulia | if you have nodes pre-created, discovery won't work | 12:56 |
| TheJulia | since you already have nodes | 12:57 |
| TheJulia | so the only thing I can think is an introspection rule | 12:57 |
| sylvr | (auto)discovery, the rules are set uisng Kayobe | 12:57 |
| TheJulia | so yeah, you've short circuited the process then by having them enrolled in advance | 12:57 |
| sylvr | what do you mean ? | 12:57 |
| TheJulia | So if a node is already known by ironic, then discovery path where ipmi_address is set, won't be executed nor set | 12:58 |
| TheJulia | since it is no longer discovery | 12:58 |
| sylvr | the node are known by ironic when they're booting the IPAramdisk, not before in my case | 12:59 |
| TheJulia | discovery only works when the node is unknown to ironic | 13:00 |
| TheJulia | i.e. the ramdisk boots and Ironic knows nothing about the node | 13:00 |
| sylvr | yes | 13:00 |
| TheJulia | what do you get with ipmitool lan print? | 13:00 |
| iurygregory | Cisco why you do this to me... | 13:01 |
| TheJulia | iurygregory: "Because we can!" | 13:01 |
| iurygregory | TheJulia, https://paste.opendev.org/show/be1nxWaZ5Hhn3BZYYTx9/ | 13:01 |
| sylvr | TheJulia : https://pastebin.com/br9AVjm7 | 13:02 |
| iurygregory | this is just interesting... | 13:02 |
| TheJulia | how so? | 13:02 |
| TheJulia | sylvr: okay then, what about inspector's logs? What do you see in them? | 13:03 |
| sylvr | grep -i error /var/log/kolla/ironic/ironic.log : https://pastebin.com/2U3H4irD | 13:07 |
| sylvr | error in /var/log/kolla/ironic-inspector/inspector.log : https://pastebin.com/6mD7NrPi | 13:07 |
| sylvr | info&debug in /var/log/kolla/ironic-inspector/inspector.log : https://pastebin.com/0ekcTuK3 | 13:07 |
| iurygregory | TheJulia, just the whole description of the params XD is weird since I never saw in any hardware before | 13:09 |
| rpittau | TheJulia, JayF, iurygregory, can you please have a look at cid's patch https://review.opendev.org/c/openstack/ironic/+/923570? kind of a sensitive change, thanks! | 13:19 |
| iurygregory | rpittau, will add to my list | 13:20 |
| rpittau | thanks! | 13:21 |
| iurygregory | rpittau, TheJulia when someone asks "is it imperative to have the ejectmedia available? can we do it manually if needed?" | 13:22 |
| iurygregory | how should I react to it? | 13:22 |
| iurygregory | "M4 are EoL for Cisco, and due to their policies i can not open a case for EoL hw" | 13:23 |
| rpittau | facepalm intensifies | 13:23 |
| iurygregory | can I cry? | 13:23 |
| opendevreview | Merged openstack/ironic-inspector master: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic-inspector/+/923811 | 13:23 |
| opendevreview | cid proposed openstack/ironic master: Follow-up to change; #922951 breaks RAID https://review.opendev.org/c/openstack/ironic/+/923570 | 13:25 |
| sylvr | TheJulia : I need to disconnect from IRC, joinable via mail | 13:36 |
| opendevreview | Merged openstack/sushy-tools master: Add support for BIOS update emulation https://review.opendev.org/c/openstack/sushy-tools/+/909500 | 13:37 |
| opendevreview | Verification of a change to openstack/ironic master failed: Localdev: disable caching and related periodic https://review.opendev.org/c/openstack/ironic/+/923453 | 13:42 |
| TheJulia | Have a wonderful weekend folks, I'm taking the day off | 13:53 |
| rpittau | TheJulia: enjoy! | 13:55 |
| iurygregory | TheJulia, enjoy | 14:01 |
| iurygregory | rpittau, the EjectMedia works, but they give a strange response =X {"error":{ "code":"InternalError", "message":"/usr/local/share/lua/5.1/dkjson.lua:397: bad argument #1 to 'strfind' (string expected, got nil)" }, can we handle this in sushy? =( I know is not the best scenario but seems like Cisco won't provide firmware updates for the hardware they are using... | 14:03 |
| TheJulia | ... that is pretty bad | 14:09 |
| rpittau | iurygregory: that really look like a bug in the firmware, and I don't think should be on us to do anything honestly | 14:09 |
| TheJulia | I wonder what the string is and what evil can be performed | 14:10 |
| * TheJulia wonders if that is a CVE... | 14:10 | |
| TheJulia | it does leak context into the internals of th eBMC | 14:10 |
| TheJulia | Powered by Lua | 14:10 |
| TheJulia | s/th\ eBMC/the\ BMC/ | 14:10 |
| rpittau | the problem is that all the is gone out of support | 14:10 |
| iurygregory | yeah | 14:10 |
| TheJulia | still a potential issue | 14:10 |
| rpittau | yep, that's why it should not be used in production :) | 14:11 |
| iurygregory | and we have customer case.. I'm talking with the TSE =X | 14:11 |
| TheJulia | yeah, best to have customer apply vendor pressure | 14:11 |
| TheJulia | Although, some countryies/governmants might find that to be cve-ey in nature and expect that issue to be fixed by hardware vendor if the hardware was shipped at all in the last 5 years | 14:12 |
| iurygregory | sadly M4 are EoL for Cisco, and due to their policies i can not open a case for EoL hw, | 14:12 |
| iurygregory | is it imperative to have the ejectmedia available? | 14:12 |
| iurygregory | can we do it manually if needed? | 14:12 |
| iurygregory | this was the answer from the customer when I told it was a bad firmware.. | 14:12 |
| TheJulia | i think it is imperitive, we shouldn't leave a bootable thing which has injected it's own bootloader record | 14:12 |
| TheJulia | the value is automation and avoiding the sharp edges to the technology | 14:13 |
| TheJulia | this is a nearly unavoidable sharp edge to that server line | 14:13 |
| rpittau | if they want to press the eject media button on hundreds of server who are we to stop them? | 14:13 |
| iurygregory | rpittau, will ironic like they doing this? XD | 14:14 |
| iurygregory | them* | 14:14 |
| TheJulia | not responsible if they bootup tot he wrong workload before they hit eject | 14:14 |
| rpittau | this ^ | 14:14 |
| iurygregory | the install will probably go crazy if they eject at the wrong time | 14:15 |
| TheJulia | https://www.youtube.com/watch?v=1umEXpGHc0E | 14:15 |
| iurygregory | and if ironic ejects and this message causes error in the process the install will also fail | 14:15 |
| iurygregory | so I'm like "I have no idea on what to do now" | 14:16 |
| TheJulia | Nothing you can do | 14:19 |
| TheJulia | I forgot how sad that song was | 14:20 |
| TheJulia | I should have remembered given what happened to a friend around when it was released | 14:20 |
| rpittau | :/ | 14:21 |
| TheJulia | On a plus side, YouTube has started 90’s alt rock hits | 14:26 |
| iurygregory | rpittau, do you want me to close the bug mentioning is the hardware with this problem is eol? or we should consider a fix for the insert at least? | 14:33 |
| rpittau | the insert part may be useful and that's actually something that we can handle, so I guess we can leave it open | 14:34 |
| iurygregory | rpittau, awesome. I will keep that open and improve some tests when I'm back from PTO | 14:35 |
| iurygregory | now I'm not in a hurry to wrap up the patch | 14:35 |
| rpittau | yep, thanks :) | 14:35 |
| nick_ | hi all. i'm troubleshooting a problem running ironic in kubernetes (with openstack-helm). in kubernetes, a volume cannot contain a hard link to anything in a different volume. so i'm trying to set the config to use a single volume. i came across this tmpdir setting: https://opendev.org/openstack/ironic/src/branch/master/ironic/common/utils.py#L254-L256 - is the only way to set tmpdir via the `TMPDIR` environment variable? | 14:41 |
| TheJulia | nick_: As a service project, we're unaware of what openstack-helm has put in place configuration wise. We generally take such configuration via a configuration file, but it looks like there is some fallback logic logic which calls this helper https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir | 14:48 |
| TheJulia | figures.... | 14:49 |
| * TheJulia sighs | 14:49 | |
| JayF | rpittau: cid had a minor typo fix, your +2 was lost here (mine was added) https://review.opendev.org/c/openstack/ironic/+/923570 | 14:53 |
| JayF | rpittau: so you can land it if ya want | 14:53 |
| rpittau | oh! didn't see the sneaky print there :) | 14:53 |
| rpittau | approved! | 14:54 |
| nick_ | so i can set tmpdir in ironic.conf? i tried putting it in the default section but ironic didn't honor it. it only worked when i used a TMPDIR environment variable i set. but i'd much prefer to put this in ironic.conf if possible | 14:54 |
| JayF | print('cool') | 14:54 |
| JayF | nick_: let me grep the same config file for "temp" and "tmp", I have an environment to generate one | 14:55 |
| JayF | but even if you have to set an ENV, you should be able to do that e.g. in a systemd service file | 14:55 |
| JayF | nick_: [DEFAULT]/tmpdir -- it's documented in the sample config | 14:56 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/conf/default.py#L417 | 14:58 |
| TheJulia | tempdir, not tmpdir | 14:58 |
| nick_ | i see, it's `tempdir`, not `tmpdir` | 14:58 |
| rpittau | bye everyone have a great weekend! o/ | 15:03 |
| nick_ | thanks all! | 15:05 |
| opendevreview | Merged openstack/ironic master: Localdev: disable caching and related periodic https://review.opendev.org/c/openstack/ironic/+/923453 | 15:45 |
| opendevreview | Verification of a change to openstack/ironic master failed: Follow-up to change; #922951 breaks RAID https://review.opendev.org/c/openstack/ironic/+/923570 | 16:05 |
| Matthew | Hello all, excited to joining this chat room as a OpenStack lover! | 16:15 |
| *** Matthew is now known as Guest1289 | 16:15 | |
| JayF | o/ | 16:50 |
| cid | Hello, you're welcome Matthew | 17:16 |
| opendevreview | Verification of a change to openstack/ironic master failed: Follow-up to change; #922951 breaks RAID https://review.opendev.org/c/openstack/ironic/+/923570 | 17:58 |
| opendevreview | cid proposed openstack/python-ironicclient master: Add CLI support for runbooks https://review.opendev.org/c/openstack/python-ironicclient/+/924101 | 20:10 |
| opendevreview | cid proposed openstack/python-ironicclient master: Add CLI support for runbooks https://review.opendev.org/c/openstack/python-ironicclient/+/924101 | 20:24 |
| iurygregory | bye everyone, see you on July 22 o/ | 20:32 |
| opendevreview | cid proposed openstack/ironic master: Self-Service via Runbooks https://review.opendev.org/c/openstack/ironic/+/922142 | 20:55 |
| cid | Happy weekend o// | 21:02 |
| JayF | o/ | 21:03 |
| opendevreview | Merged openstack/ironic master: Follow-up to change; #922951 breaks RAID https://review.opendev.org/c/openstack/ironic/+/923570 | 21:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!