| *** JayF_ is now known as JayF | 03:48 | |
| janders | good morning Ironic o/ Happy New Year 2026 | 08:27 |
|---|---|---|
| abongale | good morning ironic o/ | 09:18 |
| Continuity | Happy new year o/ | 09:58 |
| Continuity | Afternoon all, does anyone have any ideas on a dirty fix for https://bugs.launchpad.net/ironic/+bug/2129260 | 13:39 |
| Continuity | Absolutely killing me | 13:39 |
| Continuity | :D | 13:39 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add hardware health monitoring via management interface https://review.opendev.org/c/openstack/ironic/+/966946 | 13:46 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 14:06 |
| iurygregory | janders, fyi we will need to put on the chain again, after I push the new updates | 14:08 |
| * dtantsur has provided some feedback on 971152 | 14:22 | |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Make post-firmware-update reboot conditional on component https://review.opendev.org/c/openstack/ironic/+/966344 | 14:27 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add hardware health monitoring via management interface https://review.opendev.org/c/openstack/ironic/+/966946 | 14:57 |
| janders | Meeting time? | 15:01 |
| iurygregory | I'm wondering if we have quorum | 15:02 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Fix firmware update failure when task remains in RUNNING state https://review.opendev.org/c/openstack/ironic/+/970998 | 15:03 |
| dtantsur | We don't seem to have quorum... | 15:08 |
| JayF | what year is it | 15:09 |
| JayF | :) | 15:09 |
| iurygregory | what are we doing here? | 15:11 |
| cardoe | 3025 | 15:11 |
| iurygregory | cardoe, lol | 15:11 |
| cardoe | Space travel is possible but Ironic still needs to kick and provision the servers before we can do that. | 15:12 |
| iurygregory | 100% | 15:12 |
| TheJulia | happy new year folks | 15:17 |
| JayF | cardoe: OpenStack in Spaaaccccceeeeeeee | 15:17 |
| JayF | cardoe: (I no-lie think this may actually already be happening but I'm not 100%) | 15:17 |
| TheJulia | Wait, are we no longer using the deloraen and laughablly getting it up to 88 mph?!? | 15:18 |
| cardoe | iurygregory: does https://review.opendev.org/c/openstack/ironic/+/970998 need a backport? | 15:21 |
| iurygregory | cardoe, from the tests we have been doing (yes) | 15:23 |
| iurygregory | mostly it affects HPE, so yeah =) | 15:23 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Add hardware health monitoring via management interface https://review.opendev.org/c/openstack/ironic/+/966946 | 15:26 |
| TheJulia | rpittau: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bc1/openstack/bc1ee99cd2e64a4cae3782d564ef68b9/logs/testvm1_console.log | 15:34 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: Soft power off before cleaning up inspection ramdisk https://review.opendev.org/c/openstack/ironic/+/970388 | 15:34 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 15:46 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 15:47 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 15:47 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 15:53 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 15:55 |
| opendevreview | Jacob Anders proposed openstack/python-ironicclient master: Add support for node health status field https://review.opendev.org/c/openstack/python-ironicclient/+/967055 | 15:57 |
| Continuity | JayF: https://bugs.launchpad.net/ironic/+bug/2129260 any ideas on how to work around this at the moment. We tried the patch that was in the Gerrit, but no dice. | 16:00 |
| JayF | Can you put detailed feedback on the gerrit patch? | 16:01 |
| JayF | The other thing you could do to help is if you could help us get a good reproduction case that works in a devstack | 16:01 |
| JayF | I've heard vague reports of the issue but have not personally reproduced it and have not had a user willing to work with me to troubleshoot | 16:01 |
| Continuity | I can work with you... We have an environment we are standing up at the moment. | 16:02 |
| JayF | ack; would be very interested to get a report on the patch I pushed | 16:02 |
| Continuity | Im not sure its related to checksumming, as with disable_file_checksum = True in [conductor] which should (if im reading the code correctly) should stop all checksumming we see the same issue during the download | 16:03 |
| Continuity | Ill get the patch re-applied and do some more testing | 16:03 |
| JayF | Hm. Interesting. | 16:03 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Fix BIOS firmware not flashing when task completes before first poll https://review.opendev.org/c/openstack/ironic/+/970999 | 16:06 |
| Continuity | Im rebuilding the images now, anything specifically you need me to check | 16:08 |
| Continuity | Or want me to try | 16:08 |
| JayF | I mean, that patch is my best effort to fix it given the details I got from John Garbutt | 16:08 |
| JayF | just more information in general | 16:08 |
| JayF | if you have an idea where it's freezing, put some log lines around it | 16:08 |
| Continuity | Wilco | 16:09 |
| Continuity | I am right that having disable_file_checksum = True should stop any checksumming | 16:09 |
| * TheJulia makes more coffee and thinks it is time to revisit vxlan | 16:15 | |
| dtantsur | Crashing sushy-tools by causing `libvirt: XML-RPC error : Failed to create socket: Too many open files` was not on my plans for today :D | 16:32 |
| * TheJulia raises an eyebrow | 16:33 | |
| TheJulia | interesting! | 16:33 |
| dtantsur | I'm testing my asyncio metrics collection against 1000 copies of just 2 actual VMs. With 500 async workers. It's not very surprising, I would say. | 16:34 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [PoC] Unedited AI dump: demonstrate collecting sensors asynchronously https://review.opendev.org/c/openstack/ironic/+/970842 | 16:38 |
| TheJulia | yeah, not very surprising then :) | 16:39 |
| dtantsur | I'll try the fake backend. We need to get to 3500 nodes in the end :) | 16:40 |
| dtantsur | TheJulia: btw, if you have any free brainpower in the near future, I'd like your thoughts on the approach in https://review.opendev.org/c/openstack/ironic/+/970450 (don't pay attention to tests and details of implementation - can be hallucinated a bit) | 16:41 |
| TheJulia | okay, I'll try to sit down and do that later this week | 16:51 |
| JayF | the default chunk_size for iterating over a requests response object? 128 bytes | 16:53 |
| JayF | that's the chunk size we're using in our code, which means we're doing several hundred million iterations through that loop for a 40GB image | 16:54 |
| JayF | I'm pushing ann updated version of my patch that might help Continuity ^ | 16:54 |
| TheJulia | yeowie | 16:55 |
| Continuity | Thanks JayF | 16:55 |
| Continuity | Yeah I was getting to the point where I thought maybe it was teh download not the checksum | 16:56 |
| JayF | I pointed claude at it, my original patch, our chat, the bug | 16:56 |
| JayF | and just paired through it and it identified it | 16:56 |
| JayF | I would've never even thought about the chunk size on the requests iteration | 16:56 |
| Continuity | chatGPT just came to the same conclusion :D | 16:57 |
| JayF | (FWIW I validated the assumptions and have the change working through right now) | 16:58 |
| Continuity | Thanks JayF | 17:00 |
| Continuity | Ill get it onto our cluster tonight/tomorrow and report back | 17:00 |
| JayF | let me get something up and passing CI | 17:00 |
| JayF | if you are happy with it we can land+backport it | 17:00 |
| Continuity | Wilco | 17:00 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Fix BMC resource validation to catch all sushy exceptions https://review.opendev.org/c/openstack/ironic/+/971000 | 17:00 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix async periodic on redfish for servicewait https://review.opendev.org/c/openstack/ironic/+/971581 | 17:08 |
| opendevreview | Dmitry Tantsur proposed openstack/sushy-tools master: memoize: do not leak open connections https://review.opendev.org/c/openstack/sushy-tools/+/972222 | 17:11 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Improve image download performance https://review.opendev.org/c/openstack/ironic/+/964502 | 17:11 |
| dtantsur | Another fun one ^^ | 17:11 |
| JayF | Continuity: ^ obviously need to see if our CI likes it :) | 17:12 |
| JayF | dtantsur: geeze that's a footgun | 17:12 |
| dtantsur | yep | 17:12 |
| dtantsur | But nobody noticed until I tried listing 3500 fake systems :D | 17:13 |
| JayF | lol | 17:13 |
| TheJulia | I do love seeing when things fall together nicely. | 17:18 |
| JayF | Continuity: so for the checksum-it case, the orignal patch was fixing chunk size too -- because TransferHelper was using a 1MB chunk size | 17:26 |
| Continuity | So your old code should have worked as well then? | 17:28 |
| Continuity | Hmm, let me deploy it again and see what I can see | 17:28 |
| JayF | IF you are on the code path doing the checksum inline | 17:29 |
| JayF | that calls out to checksum_utils.TransferHelper | 17:29 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Improve image download performance https://review.opendev.org/c/openstack/ironic/+/964502 | 17:32 |
| opendevreview | Dmitry Tantsur proposed openstack/sushy-tools master: Fake backend: allow disabling persistent memoization https://review.opendev.org/c/openstack/sushy-tools/+/972228 | 17:41 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [PoC] Unedited AI dump: demonstrate collecting sensors asynchronously https://review.opendev.org/c/openstack/ironic/+/970842 | 17:41 |
| dtantsur | I'm now at the point where I reliably get timeouts from sushy-tools on collecting so many sensors :) | 17:42 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 17:55 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Fix heartbeat for steps with requires_ramdisk=False https://review.opendev.org/c/openstack/ironic/+/971152 | 17:56 |
| TheJulia | cardoe: when you have some time, we should chat about vxlan :) | 17:59 |
| cardoe | okay. I'm just making myself some lunch so in a bit? | 18:00 |
| TheJulia | I have a call starting in 1.5hr | 18:00 |
| TheJulia | That will be 1hr most likely | 18:00 |
| TheJulia | cardoe: a starting thought, I'm thinking of dropping "Bridging Fabrics" from the spec I have and letting the neutron type2 work roll forward and then also trying to roll forward your approach as well I think I've got a few possible ideas which are better, but its just a much heavier lift | 18:46 |
| cardoe | Sure we can do that. | 18:46 |
| cardoe | I'm free for a bit | 18:46 |
| TheJulia | free now? | 18:46 |
| TheJulia | https://meet.google.com/epr-vztw-oev | 18:48 |
| TheJulia | cardoe: ^ | 18:51 |
| cardoe | https://review.opendev.org/c/openstack/neutron/+/965053 | 18:58 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: VXLAN networking https://review.opendev.org/c/openstack/ironic-specs/+/959401 | 22:27 |
| TheJulia | cardoe: ^^^ please review/follow-up. Specifically the same unknown is the whole interaction as it relates to trunk subports, but the broad idea is starting to be painted | 22:31 |
| iurygregory | seems like metal3-integration is broken .-. | 23:42 |
| iurygregory | unknown: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://quay.io/v2/metal3-io/ironic-standalone-operator/blobs/sha256:765b0ba032b579668e68acbc99b5f949cee9f868dc0c0326472377b996baa10c: 502 Bad Gateway | 23:42 |
| iurygregory | 502 yay | 23:43 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!