| *** zbitter is now known as zaneb | 03:33 | |
| shajizad | Good morning ironic | 06:28 |
|---|---|---|
| shajizad | For my pull requests, Zuul succeeded in the VIF patch while openstacksdk-functional-devstack-networking-extfailed for the virtual media one. Could somebody help me find what the issue is? The patch is here: https://review.opendev.org/c/openstack/openstacksdk/+/921957 | 06:28 |
| rpittau | good morning ironic! o/ | 07:10 |
| rpittau | dtantsur: re redfish interop profile: IMHO the thing to do is just keep the docs updated every time we touch anything related to redfish | 07:11 |
| rpittau | shajizad: let me finish wake up and I'll get to it :) | 07:12 |
| cid | Good morning o/ | 07:41 |
| rpittau | shajizad: I don't think the function job is failing because of your patch, the openstacksdk-functional-devstack-networking-ext job does not run baremetal tests | 07:57 |
| shajizad | I see | 08:13 |
| shajizad | Should I just do recheck then? | 08:13 |
| shajizad | Oh you already did t | 08:14 |
| shajizad | s/t/it/ | 08:14 |
| rpittau | shajizad: yup, rechecked, if it fails again maybe it's time to ask in openstacksdk channel about it | 08:17 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Remove CentOS Stream 8 leftovers https://review.opendev.org/c/openstack/bifrost/+/922789 | 08:31 |
| *** mgoddard- is now known as mgoddard | 10:12 | |
| dtantsur | a new low hanging fruit found while researching our redfish code https://bugs.launchpad.net/ironic/+bug/2070485 | 10:44 |
| iurygregory | good morning Ironic | 10:47 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Update the redfish interoperability profile https://review.opendev.org/c/openstack/ironic/+/920574 | 10:54 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [PoC] Tox environment to verify the redfish profile https://review.opendev.org/c/openstack/ironic/+/921302 | 10:55 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Update the redfish interoperability profile https://review.opendev.org/c/openstack/ironic/+/920574 | 11:19 |
| dtantsur | I declare this finished ^^. Enjoy reviewing! | 11:19 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [PoC] Tox environment to verify the redfish profile https://review.opendev.org/c/openstack/ironic/+/921302 | 11:20 |
| cid | Just out of curiosity... I've noticed that the Ironic codebase has quite a number of TODOs. | 11:47 |
| cid | Is there a dedicated process or tool that manages these TODOs? | 11:47 |
| cid | More like, how does the community ensure they are addressed in a timely manner? | 11:47 |
| dtantsur | We do not | 11:48 |
| dtantsur | If you know how to fix any of these, feel free to just do it. | 11:48 |
| cid | Got it! That's what I was driving at. | 11:48 |
| shajizad | rpittau: I changed the commit message for both patches. Now the vmedia one succeeded and VIF failed. I don't think my patch was the problem because it passed earlier | 12:15 |
| shajizad | openstacksdk-functional-devstack | 12:15 |
| rpittau | shajizad: yeah, that job is nor runnning bare metal tests AFAICS, please report the issue in the openstacksdk channel | 12:16 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add RPC calls and handlers needed to support async service steps https://review.opendev.org/c/openstack/ironic/+/922811 | 12:26 |
| iurygregory | great finding janders =) | 12:28 |
| *** tosky_ is now known as tosky | 12:37 | |
| opendevreview | Jacob Anders proposed openstack/ironic master: Make redfish firmware update a service step https://review.opendev.org/c/openstack/ironic/+/922815 | 12:41 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add RPC calls and handlers needed to support async service steps https://review.opendev.org/c/openstack/ironic/+/922811 | 12:48 |
| janders | ty iurygregory o/ | 12:48 |
| janders | missed some version bumps, sorted now | 12:48 |
| dtantsur | ah damn, was about to write a review :D | 12:49 |
| janders | still needs some work (tests, reno) but I wanted to get initial reviews to reflect any changes in these | 12:49 |
| dtantsur | janders: tests yes, not sure what would a release note contain - the calls are not wired in yet, right? | 12:50 |
| janders | dtantsur yeah you are right | 12:50 |
| janders | it's kinda bad these are missing but so what if nothing is trying to use the missing calls yet | 12:51 |
| janders | OK two done, I will now redo the Targets patch removing the decorator from it | 12:54 |
| janders | sushy-tools stuff I will leave for tomo | 12:54 |
| janders | (needs a more substantial respin also) | 12:54 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] add Targets to firmware.update on multi system BMCs https://review.opendev.org/c/openstack/ironic/+/922438 | 13:00 |
| janders | OK so I think these three patches is a reasonably logical way to slice it up, when you have time have a look and leave me feedback, I will work towards getting these closer to merge-ready tomorrow (and will also get onto sushy-tools) | 13:02 |
| janders | when combined together these patches work pretty well but I expect they will need some work and tidying up | 13:03 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Remove CentOS Stream 8 leftovers https://review.opendev.org/c/openstack/bifrost/+/922789 | 13:56 |
| opendevreview | Verification of a change to openstack/ironic master failed: Remove deprecated idrac wsman driver interfaces https://review.opendev.org/c/openstack/ironic/+/922340 | 15:00 |
| opendevreview | Verification of a change to openstack/ironic master failed: Remove unused parameter in ansible driver https://review.opendev.org/c/openstack/ironic/+/921540 | 15:31 |
| opendevreview | Merged openstack/networking-generic-switch stable/2023.2: [CI] Remove not defined job from queue https://review.opendev.org/c/openstack/networking-generic-switch/+/920659 | 15:32 |
| opendevreview | Merged openstack/networking-generic-switch stable/2023.1: [CI] Remove not defined job from queue https://review.opendev.org/c/openstack/networking-generic-switch/+/920660 | 15:34 |
| opendevreview | Verification of a change to openstack/ironic master failed: Fix execution of node servicing steps exposed by IPA's HardwareManager https://review.opendev.org/c/openstack/ironic/+/922024 | 15:48 |
| dtantsur | [ERROR] /opt/stack/devstack/inc/meta-config:191 bogus config file specification $NOVA_CPU_CONF ($NOVA_CPU_CONF=/etc/nova/nova-cpu.conf, /etc/nova is not a directory) | 15:51 |
| dtantsur | I'm sorry, WHAT | 15:51 |
| * dtantsur asking on #openstack-qa | 15:55 | |
| rpittau | good night! o/ | 16:03 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [DNM] Test the proposed CI fix https://review.opendev.org/c/openstack/ironic/+/922847 | 16:13 |
| JayF | dtantsur: fwiw with that local API doc patch, I'm going to finish the testing and do a video on the results too | 16:19 |
| JayF | dtantsur: so your questions are heard, are good, and I'll dig in when I go down that path | 16:19 |
| JayF | I'm 99% sure with network_interface=noop it reduces the amount of stuff you need in order to get a node to validate and a deployment to work (you don't have to create ports) | 16:20 |
| dtantsur | yeah, but I think you don't really need the rest | 16:20 |
| dtantsur | (I personally did some changes a while ago to make sure Ironic can deduce missing required interfaces) | 16:21 |
| JayF | I 100% know we need more than in the existing config | 16:22 |
| JayF | because we had an ipmi-something interface still enabled | 16:22 |
| JayF | which blew up b/c I didn't have ipmitool on my local machine | 16:22 |
| JayF | and I validated that with my changes, it works without ipmitool installed | 16:22 |
| JayF | so there is *something* needed, I 100% believe it could be /less something/ than what I've done | 16:23 |
| dtantsur | sure, you disable IPMI. what I'm speculating about is that you might be okay with only overriding enabled_hardware_interfaces and enabled_network_interfaces. | 16:23 |
| dtantsur | I expect the rest to be deduced | 16:23 |
| JayF | I think the one that blew me up was ipmi console interfaces | 16:25 |
| JayF | but I'd have to repro to be 100% sure | 16:25 |
| dtantsur | the default is only no-console: https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/default.py#L115 | 16:26 |
| dtantsur | you may be forced to set boot/deploy interfaces if you want to avoid having real implementations (the ipxe one will probably fail on creating boot.ipxe) | 16:26 |
| dtantsur | power/management, while being required, should default to fake in the end | 16:27 |
| JayF | I'm 99.999% sure still it was ipmi console | 16:27 |
| dtantsur | everything else already defaults to their no-* implementations | 16:27 |
| JayF | and it wasn't looking at enabled_ it was looking at something else | 16:27 |
| JayF | let me see if I can re-repro quickly locally | 16:27 |
| dtantsur | This is all I have locally: | 16:28 |
| dtantsur | enabled_hardware_types = fake-hardware | 16:28 |
| dtantsur | enabled_boot_interfaces = fake | 16:28 |
| dtantsur | enabled_deploy_interfaces = fake | 16:28 |
| JayF | nothing in vendor_interfaces, either? | 16:28 |
| dtantsur | it's also optional, so no-vendor by default | 16:29 |
| dtantsur | (I seriously wonder if we should have something like ./tools/start-fake-ironic.sh that just does the necessary) | 16:29 |
| JayF | yes | 16:29 |
| JayF | dtantsur: reproduced | 16:30 |
| JayF | dtantsur: can you move ipmitool out of the way on your box and see if you reproduce? | 16:30 |
| dtantsur | gimme a minute | 16:31 |
| JayF | https://gist.github.com/jayofdoom/e2b508c4eda133e31acad841f6eaf1b5 | 16:32 |
| dtantsur | hmm, interesting, it does try to use ipmitool | 16:33 |
| dtantsur | which is something we should really fix rather than documenting workarounds | 16:33 |
| dtantsur | daaaamn | 16:34 |
| dtantsur | JayF: https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/default.py#L165-L166 | 16:34 |
| JayF | dtantsur: yep, I knew it was somewhere | 16:34 |
| dtantsur | I kinda think it's a bug.. but also fixing it might break people | 16:34 |
| JayF | which is likely why it is how it is | 16:34 |
| JayF | I'd suggest instead, we should not make the absense of ipmitool blow up ironic | 16:34 |
| JayF | but I'm not sure how to structure that without setting up a footgun | 16:34 |
| dtantsur | Right. We ask Ironic to enable an interface that cannot possibly work. | 16:35 |
| dtantsur | maybe we still change it with an "upgrade" release note | 16:35 |
| JayF | but that interface is /vendor passthru/ | 16:35 |
| JayF | if there was ever a case to just fail and be okay with it | 16:35 |
| JayF | vendor passthru is it | 16:35 |
| dtantsur | But was there? So far, Ironic verifies preconditions on load. | 16:36 |
| JayF | honestly leaving the "explode if no ipmitool" in the actual-driver but making the vendorpassthru one just throw errors when that happens | 16:36 |
| JayF | but was there /what/? | 16:36 |
| dtantsur | An even more interesting solution would be to raise a warning that the hardware interface is not used by any hardware type. And not load it. | 16:36 |
| JayF | Do VendorPassthru methods even care about hardware type? | 16:37 |
| JayF | becaeuse we have both node vendor passthru and top level, right? | 16:37 |
| dtantsur | It's a normal hardware interface, it can only be attached to hardware types that enable it | 16:37 |
| JayF | the top level driver passthru is supposed to work even without a node | 16:37 |
| JayF | or am I remembering an old deprecated part of our api | 16:37 |
| dtantsur | ... and now I remember that fake-hardware can use any interface :( | 16:37 |
| JayF | heartbeat/lookup, for example, were not node-specific | 16:38 |
| JayF | when they were passthrus | 16:38 |
| dtantsur | Driver passthru still require a driver, which is baked by a hardware type | 16:38 |
| JayF | (as opposed to the send_raw passthru for ipmi driver, which was on a node path) | 16:38 |
| dtantsur | anyway, my suggestion cannot work in the presence of fake-hardware | 16:38 |
| JayF | yeah | 16:38 |
| JayF | what /is/ in ipmi vendor passthru | 16:38 |
| * JayF wonders if we can just deprecate it lol | 16:38 | |
| dtantsur | the raw command? can be useful, dunno | 16:39 |
| * dtantsur rarely uses IPMI nowadays | 16:39 | |
| JayF | I've not seen an ipmi install of ironic in production in a while | 16:39 |
| dtantsur | JayF: could you at least document the issue you've discovered in a bug? even if we cannot immediately come up with a solution | 16:39 |
| JayF | but I also don't work for the sorta places that have that slow of a hardware lifecycle | 16:39 |
| JayF | yep, I wasn't sure it was a bug when I found it | 16:39 |
| JayF | whch is why I documented around it instead of calling it a bug | 16:39 |
| JayF | (in fact, I said it was a bug in IRC when I found it then walked it back later lol) | 16:40 |
| dtantsur | okay, we need to document changing enabled_vendor_interfaces too | 16:40 |
| JayF | yeah it was one of those things where I shotgunned it to make it more futureproof | 16:40 |
| JayF | but I think building *up* from an empty config will be better | 16:40 |
| JayF | and easier to maintain | 16:40 |
| JayF | I'm going to call this a "low" bug | 16:45 |
| dtantsur | yep | 16:46 |
| JayF | https://bugs.launchpad.net/ironic/+bug/2070678 and self-triaged | 16:46 |
| JayF | bug deputy triaged that one with 0ms of latency! lol | 16:46 |
| dtantsur | good job! but you'll struggle to show further improvement on your next performance review! | 16:48 |
| dtantsur | :D | 16:48 |
| JayF | my boss just adds a gold star sticker to the side of the 5 stars in the HR app | 16:52 |
| JayF | turning it up to 11, so to speak | 16:53 |
| dtantsur | Nice :) | 16:53 |
| JayF | it's kinda funny taking your statement in context of Ironic | 16:53 |
| JayF | b/c it totally has happened here | 16:53 |
| JayF | "Wow! A bare metal server via an API in like, an hour?! wow!!!!" | 16:53 |
| JayF | (fast forward a decade) | 16:53 |
| JayF | every third conversation is me telling someone complaining about deployments taking 20 minutes that the reboot takes longer than everything else combined :P | 16:54 |
| dtantsur | lol, so true | 16:54 |
| JayF | ironic: the only software that works faster when a server has less ram | 16:54 |
| dtantsur | haha, also very true | 16:54 |
| JayF | there's maybe a short half-comedy talk in that | 16:55 |
| JayF | ironic ironies | 16:55 |
| dtantsur | janders and I will need to look into avoiding some reviews into IPA during servicing at some point | 16:55 |
| JayF | I would be a fan of that work | 16:55 |
| JayF | if servicing with disable_ramdisk still boots IPA with only an OOB step | 16:55 |
| JayF | I will be a sad panda | 16:55 |
| JayF | but ... not surprised knowing how it all works | 16:55 |
| JayF | my "IPA must be able to detect the hardware to decide on available steps" chickens are coming home to roost | 16:56 |
| dtantsur | The problem with disable_ramdisk is even simpler: we use IPA to detect when hardware is actually ready for the next step | 16:57 |
| JayF | because heartbeat drives the whole bus | 16:57 |
| JayF | it really is all the IPA design decisions coming home to get me, eh | 16:58 |
| dtantsur | yeah, and because Redfish does not always tell us when the hardware is truly ready | 16:58 |
| dtantsur | oh, it's quite late here apparently, talk to you tomorrow! | 16:59 |
| JayF | o/ have a good one | 16:59 |
| JayF | krozcheck (sp?) just sat down beside me! | 18:00 |
| JayF | formerly of the community here | 18:01 |
| clarkb | ya he lives in seattle | 18:10 |
| clarkb | or lived in seattle its been a while | 18:10 |
| JayF | What are the side effects of setting use_storyboard:false in projects.yaml? I'm looking, trying to address https://bugs.launchpad.net/ironic-python-agent-builder/+bug/2056146 and want to ensure the obvious path (updating projects.yaml) is the right one ... I suspect there's something hiding here if I didn't already flip that switch during the migration | 18:28 |
| JayF | (that bug is just that the issues tab https://opendev.org/openstack/ironic-python-agent-builder goes to storyboard) | 18:29 |
| clarkb | fungi: ^ is probably the best one toe ask. I think it mostly has to do with initial setup and after the fact the impact either way is minimal | 18:29 |
| clarkb | oh ya we use it to set the bug links in gitea | 18:29 |
| clarkb | if you toggle it off of storyboard it should switch the link in gitea to lp. | 18:29 |
| JayF | I'm mainly worried that it's going to change something on storyboard side locking out old Ironic bugs or something | 18:29 |
| fungi | correct, and no need to explicitly set it false, it defaults to false so just remove the line | 18:29 |
| JayF | that's the reason I'm asking about side effects :D | 18:29 |
| fungi | it won't lock out old anything | 18:30 |
| JayF | beautiful, patch incoming | 18:30 |
| fungi | we don't delete bugs from sb regardless, i follow a manual process to adjust the project description and hide it from the new bugtask autocomplete by setting its inactive flag when a project moves off sb | 18:31 |
| fungi | mainly to dissuade anyone from creating new stories/tasks for a project there | 18:32 |
| JayF | and of course I also now discover https://opendev.org/x/ipa-example-hardware-managers needs retirement | 18:34 |
| JayF | (they now live in ipa repo, examples/, and have for *years*) | 18:34 |
| JayF | fyi fix is in the gate now for the devstack ci failures https://review.opendev.org/c/openstack/devstack/+/881580 when that lands I think Ironic should be good | 19:05 |
| JayF | cid: dtantsur: Did we ever email the list about ironic-bug-dashboard taking config files now? Might be worth broadcasting it's useful for other projects. | 19:11 |
| *** dking is now known as Guest10879 | 19:12 | |
| *** Guest10879 is now known as dking | 19:13 | |
| dking | If anybody happens to still be around, does anybody know of a way to abort a stuck "cleaning" state? Or better, to troubleshoot what it is stuck doing? | 19:14 |
| cid | o/ | 19:14 |
| dtantsur | dking: cleaning means it's doing something actively, like getting stuck in the middle of some network request. Since it involves locking, unfortunately, your only bet is to restart the responsible conductor. | 19:56 |
| *** dmellado0755 is now known as dmellado075 | 21:06 | |
| JayF | yeah restart the conductor that has it locked is the answer to that, and probably will be ~forever. It indicates something failed/hung and without knowing what, we can't keep things in a sane state :( often I've seen this in cases where you have some backing store for ironic on a network share | 22:44 |
| JayF | e.g. if you were sharing image cache directories on conductors using nfs (don't do this) | 22:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!