| hroy_ | estherd[m], good morning! :) | 04:30 |
|---|---|---|
| hroy_ | I see that in https://review.opendev.org/c/openstack/ironic/+/996347 the NodeHistory version set includes 1.4, shouldn't it be only till 1.3, or am I missing something? | 04:31 |
| hroy_ | also, did you get a chance to send out an email to openstack mailing list requesting folks to share some use cases they have | 04:32 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix fast-track deployment failure with autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/999699 | 09:31 |
| rpittau | dtantsur: I think this ^ will fix the issue we're seeing in BMO e2e tests in metal3 from time to time, looks like a race condition | 09:35 |
| opendevreview | cid proposed openstack/ironic master: Fix socat console broken by shell-quoting https://review.opendev.org/c/openstack/ironic/+/999701 | 09:48 |
| dtantsur | rpittau: nice catch! | 10:06 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Refresh the scaling guide in the reference architecture https://review.opendev.org/c/openstack/ironic/+/997998 | 10:38 |
| dtantsur | TheJulia: updated ^^^ | 10:38 |
| opendevreview | Julia Kreger proposed openstack/ironic bugfix/37.0: security: Fix poisoning of the allowed origin list https://review.opendev.org/c/openstack/ironic/+/998017 | 10:43 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/37.0: Power off before ejecting redfish virtual media on ramdisk cleanup https://review.opendev.org/c/openstack/ironic/+/999259 | 10:43 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/37.0: Another attempt to fix fast-track after inspection https://review.opendev.org/c/openstack/ironic/+/999258 | 10:43 |
| stephenfin | clif: RE: o.vo mypy plugin: thank gibi. I just took what he had and got it mergeable https://github.com/gibizer/ovo-mypy-plugin | 11:03 |
| stephenfin | (I'm only back to work today) | 11:03 |
| stephenfin | Could someone sanity check https://review.opendev.org/c/openstack/openstacksdk/+/999686 for us, please? | 11:24 |
| dtantsur | Oooops! | 11:55 |
| TheJulia | good morning | 13:10 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix Redfish inspection without Storage API https://review.opendev.org/c/openstack/ironic/+/999718 | 13:26 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Unify non-BMC firmware updates under batched state machine https://review.opendev.org/c/openstack/ironic/+/999720 | 13:33 |
| opendevreview | Jacob Anders proposed openstack/ironic master: Document batched firmware updates and add release note https://review.opendev.org/c/openstack/ironic/+/999721 | 13:33 |
| TheJulia | folks, a relatively quick review on https://review.opendev.org/c/openstack/ironic/+/999656 would be appreciated | 13:36 |
| dtantsur | FYI: the BMO job got broken, and it may be on the Ironic side | 13:40 |
| dtantsur | ah, it may be the thing that rpittau posted this morning | 13:44 |
| opendevreview | cid proposed openstack/ironic master: Drop the unused container_conf_file option https://review.opendev.org/c/openstack/ironic/+/998386 | 13:46 |
| opendevreview | cid proposed openstack/ironic master: doc: Fix the container-based steps guide https://review.opendev.org/c/openstack/ironic/+/998387 | 13:46 |
| opendevreview | cid proposed openstack/ironic master: Verify registry TLS by default for agent containers https://review.opendev.org/c/openstack/ironic/+/999724 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: security: Fix the container allowlist not being enforced https://review.opendev.org/c/openstack/ironic-python-agent/+/998375 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Prevent container steps from shadowing existing steps https://review.opendev.org/c/openstack/ironic-python-agent/+/998376 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Only honor container step priorities while cleaning https://review.opendev.org/c/openstack/ironic-python-agent/+/998377 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Accept whitespace separated container options https://review.opendev.org/c/openstack/ironic-python-agent/+/998378 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Check that the configured container runtime is present https://review.opendev.org/c/openstack/ironic-python-agent/+/998379 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Report errors reading the container steps file https://review.opendev.org/c/openstack/ironic-python-agent/+/998380 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Ignore the transport prefix when matching allowed containers https://review.opendev.org/c/openstack/ironic-python-agent/+/998381 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Drop the unused container_conf_file option https://review.opendev.org/c/openstack/ironic-python-agent/+/998382 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Rename container_clean_step to generic_container_step https://review.opendev.org/c/openstack/ironic-python-agent/+/998383 | 13:46 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Verify registry TLS by default https://review.opendev.org/c/openstack/ironic-python-agent/+/999725 | 13:46 |
| dtantsur | rpittau: either https://review.opendev.org/c/openstack/ironic/+/999699 does not work, or we're not actually testing with new ironic | 13:47 |
| dtantsur | the fact that we missed the breakage initially makes me suspect the latter... | 13:47 |
| rpittau | mmmm let me triple check then | 13:57 |
| dtantsur | Testing a revert in https://github.com/metal3-io/ironic-image/pull/1150 (while also checking whether the job picks up ironic changes at all) | 14:11 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Check that the configured container runtime is present https://review.opendev.org/c/openstack/ironic-python-agent/+/998379 | 14:23 |
| iurygregory | good morning ironic | 14:26 |
| rpittau | dtantsur: so it looks like we're testing autodetect in ironic CI, but not with fast_track | 14:29 |
| rpittau | so the "broken" part is skipped | 14:30 |
| dtantsur | rpittau: I mean the ironic-image CI when we enabled autodetect | 14:30 |
| rpittau | oh! I think BMO e2e is the only place then | 14:31 |
| dtantsur | Exactly. But why did it succeed if now it fails more or less always? | 14:31 |
| dtantsur | rpittau: there is a suspicious gap though. stevebaker[m]'s patch merged on Jul 24th, BMO failures started around Jul 30th per https://zuul.opendev.org/t/openstack/builds?job_name=metal3-baremetal-operator-functional&project=openstack%2Fironic&branch=master&skip=0 | 14:33 |
| dtantsur | I wonder if there is an interplay with https://review.opendev.org/c/openstack/ironic/+/998628 too | 14:34 |
| rpittau | I considered this ^ but I excluded it for some reason that I don't recall now | 14:36 |
| dtantsur | rpittau: I'm like 90% sure that these patches together break BMO | 14:37 |
| rpittau | ok, let' | 14:38 |
| rpittau | s give it another look | 14:38 |
| dtantsur | rpittau: what makes me confident is the fact that your 38.0 PR fails with this error, while master BMO does not. Master BMO is pinned to an old version because https://github.com/metal3-io/ironic-image/pull/1127 got misteriously closed | 14:38 |
| dtantsur | Not BMO, sorry, ironic-image | 14:38 |
| rpittau | right, that I noticed too | 14:38 |
| rpittau | well, I can test that actually | 14:39 |
| dtantsur | rpittau: in your patch, it does not look like we receive any heartbeats. The only one I see was rejected because the node was locked. | 14:42 |
| rpittau | mmm ok let me check the logs, we're talkgin about the BMO job, right ? | 14:45 |
| dtantsur | rpittau: yep. Ironic https://c16ba2cc0921cdf6518d-f5e06fd9232d0500a75b50f2bf1ad7e1.ssl.cf2.rackcdn.com/openstack/d2916c46bc864f899ab24da3e8bb0cb2/ubuntu-noble/logs/baremetal-operator-system/ironic-service/ironic-service-7bfd6498bb-qr89f/ironic.log IPA https://c16ba2cc0921cdf6518d-f5e06fd9232d0500a75b50f2bf1ad7e1.ssl.cf2.rackcdn.com/openstack/d2916c46bc864f899ab24da3e8bb0cb2/ubuntu-noble/network-data/bmo-e2e-1-serial0.log | 14:46 |
| rpittau | ok, let's see | 14:46 |
| dtantsur | Here is how I see it: before my fix, fast-track was just broken, and we used to do a normal deployment. | 14:46 |
| dtantsur | Now fast-track works during inspection, but if the heartbeat never arrives at the right time, deployment is broken instead. | 14:46 |
| rpittau | soemthing on the conductor side ? | 14:47 |
| dtantsur | The right answer seems to be "we need to send agent_url during inspection", but I don't know how to produce a backportable fix | 14:47 |
| rpittau | anyway it looks like just adding HeartbeatMixin does not fix the issue | 14:47 |
| dtantsur | I'm also not sure why autodetect seems to play a role | 14:47 |
| dtantsur | well, it's an improvement. but the node is locked, so we still cannot record agent URL | 14:48 |
| dtantsur | (another "thank you" to our love for free-form JSON fields...) | 14:48 |
| dtantsur | rpittau: where does the agent_url validation happen? can we park the node until the next heartbeat? | 14:50 |
| opendevreview | Merged openstack/ironic master: Drop nonexistent irrelevant-files; update the list in metal3-jobs https://review.opendev.org/c/openstack/ironic/+/998301 | 14:50 |
| dtantsur | (this is going to make the whole thing ~ 30 seconds longer, which is also terrible) | 14:50 |
| opendevreview | Merged openstack/ironic master: Portgroup shard filter bypasses scope to project https://review.opendev.org/c/openstack/ironic/+/999656 | 14:52 |
| rpittau | the agent_url sohuld be stored after HeartbeatMixin state check | 14:54 |
| rpittau | but it never happen | 14:54 |
| rpittau | the upgrade_lock fails because inspector is holding the lock, so yeah we need to send that during the inspection | 14:54 |
| rpittau | we can park the node in DEPLOYWAIT ? | 14:55 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Check that the configured container runtime is present https://review.opendev.org/c/openstack/ironic-python-agent/+/998379 | 14:57 |
| dtantsur | Technically, we can pause a step until the next heartbeat | 14:57 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Report errors reading the container steps file https://review.opendev.org/c/openstack/ironic-python-agent/+/998380 | 14:57 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Ignore the transport prefix when matching allowed containers https://review.opendev.org/c/openstack/ironic-python-agent/+/998381 | 14:57 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Drop the unused container_conf_file option https://review.opendev.org/c/openstack/ironic-python-agent/+/998382 | 14:57 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Rename container_clean_step to generic_container_step https://review.opendev.org/c/openstack/ironic-python-agent/+/998383 | 14:57 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Verify registry TLS by default https://review.opendev.org/c/openstack/ironic-python-agent/+/999725 | 14:57 |
| rpittau | ok, makes sense,I can proabbly add that to the fix patch | 14:59 |
| dtantsur | rpittau: dunno if a 30 sec delay can be considered acceptable.. but let's see if it even helps | 15:01 |
| dtantsur | I'd rather revert the autodetect switch tbh. Then update Ironic in ironic-image. Then see how we re-introduce autodetect. | 15:02 |
| dtantsur | IF autodetect has a part in it at all | 15:02 |
| rpittau | I'll write something quickly | 15:02 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix fast-track deployment failure with autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/999699 | 15:50 |
| rpittau | dtantsur: new code makes bmo e2e test fails -> https://github.com/metal3-io/ironic-image/pull/1151 | 15:54 |
| rpittau | let's see how the fix behaves now | 15:54 |
| dtantsur | rpittau: if we go down ^^^ path, maybe we need to add the 3rd part to heartbeat manager in the conductor: if agent_url is not recorded and node.reservation is set, fail immediately with CONFLICT | 15:55 |
| rpittau | oh ok | 15:55 |
| dtantsur | rpittau: why: current upgrade_lock happens in a thread and never reaches IPA, so it cannot retry quickly | 15:55 |
| dtantsur | there is conflict logic in IPA which cannot be reached this way | 15:55 |
| rpittau | mmm I was taking that into consideration, ok | 15:56 |
| rpittau | I was NOT! | 15:56 |
| dtantsur | NOOOOO!! | 15:56 |
| dtantsur | :D | 15:56 |
| rpittau | :D | 15:56 |
| dtantsur | rpittau: meanwhile, https://github.com/metal3-io/ironic-image/pull/1150 should tell us if autodetect plays a role | 15:56 |
| rpittau | ack | 15:56 |
| dtantsur | (I've changed it to also update the Ironic pin) | 15:56 |
| dtantsur | Unrelated, but I'd appreciate reviews on https://review.opendev.org/c/openstack/ironic/+/999718. It's blocking testing redfish inspection with sushy-tools. | 15:58 |
| dtantsur | on this note, I'll go try to cool down | 15:59 |
| opendevreview | Mohammed Naser proposed openstack/sushy master: Handle incomplete TaskMonitor responses https://review.opendev.org/c/openstack/sushy/+/999741 | 16:06 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix fast-track deployment failure with autodetect deploy interface https://review.opendev.org/c/openstack/ironic/+/999699 | 16:13 |
| opendevreview | Yorick proposed openstack/sushy master: system: guard HttpBootUri writes against iDRAC 10 removal https://review.opendev.org/c/openstack/sushy/+/999743 | 16:20 |
| cardoe | Boy... mnaser opening that change against sushy TaskMonitor makes me look at the API and say... wat? | 16:46 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add image_server_auth_hosts to restrict credential scope https://review.opendev.org/c/openstack/ironic/+/999744 | 16:57 |
| TheJulia | JayF: replied on https://review.opendev.org/c/openstack/ironic/+/999744 | 17:02 |
| TheJulia | other ironic-cores, might be good to take a look | 17:03 |
| TheJulia | cardoe: the redfish API or what vendors do with it? | 17:03 |
| JayF | Yeah I mean, you're not wrong per se, but I also suspect we'll have operators who will not be able to use the feature as written | 17:03 |
| TheJulia | so then do we globify everything? | 17:05 |
| TheJulia | and even then, the other stuff is all CDN'ed anyhow | 17:05 |
| TheJulia | do we open that door, or not I guess is the bigger question | 17:06 |
| JayF | I was thinking something simpler than a glob might be easier to implement | 17:06 |
| JayF | "example.com" matches example.com | 17:06 |
| JayF | ".example.com" matches *.example.com | 17:06 |
| TheJulia | so, allow pure domain name matching in addition to FQDN matching | 17:06 |
| JayF | this would also allow `.internal` as a config | 17:06 |
| TheJulia | true | 17:06 |
| JayF | this is basically the case I'm thinking, re: internal domains | 17:06 |
| TheJulia | xupdating | 17:08 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add image_server_auth_hosts to restrict credential scope https://review.opendev.org/c/openstack/ironic/+/999744 | 17:26 |
| adam-metal3 | hello | 18:12 |
| TheJulia | greetings | 18:17 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/2026.1: Portgroup shard filter bypasses scope to project https://review.opendev.org/c/openstack/ironic/+/999762 | 20:28 |
| opendevreview | Jay Faulkner proposed openstack/ironic bugfix/38.0: Portgroup shard filter bypasses scope to project https://review.opendev.org/c/openstack/ironic/+/999763 | 20:33 |
| opendevreview | Jay Faulkner proposed openstack/ironic bugfix/37.0: Portgroup shard filter bypasses scope to project https://review.opendev.org/c/openstack/ironic/+/999764 | 20:33 |
| opendevreview | Jay Faulkner proposed openstack/ironic bugfix/34.0: Portgroup shard filter bypasses scope to project https://review.opendev.org/c/openstack/ironic/+/999765 | 20:34 |
| JayF | RFR https://review.opendev.org/c/openstack/ossa/+/999767 [OSSA-2026-033] Ironic portgroup scoping | 20:43 |
| cardoe | Reviewed and merged. | 23:17 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!