| TheJulia | I tagged https://review.opendev.org/c/openstack/ironic/+/941136 as ironic-week-prio | 00:14 |
|---|---|---|
| opendevreview | Verification of a change to openstack/ironic master failed: bootc deploy interface - for bootable containers https://review.opendev.org/c/openstack/ironic/+/937897 | 00:26 |
| opendevreview | Verification of a change to openstack/ironic master failed: bootc deploy interface - for bootable containers https://review.opendev.org/c/openstack/ironic/+/937897 | 02:38 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: move hooks execution into shareable module https://review.opendev.org/c/openstack/ironic/+/940332 | 03:48 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow multiple inspection interfaces to load hooks https://review.opendev.org/c/openstack/ironic/+/940333 | 03:48 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface https://review.opendev.org/c/openstack/ironic/+/933066 | 03:48 |
| opendevreview | Will Szumski proposed openstack/networking-generic-switch master: Adds vlan aware VMs support for Cumulus NVUE and DellOS10 https://review.opendev.org/c/openstack/networking-generic-switch/+/940966 | 07:42 |
| rpittau | good morning ironic! o/ | 08:06 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [docs] Fix link to metal3-integration page https://review.opendev.org/c/openstack/ironic/+/941214 | 08:34 |
| *** jroll09 is now known as jroll0 | 08:45 | |
| rpittau | JayF: I'm going through the ironic-lib deprecation patches left, I see only inspector left before the actual removal, let me know if you need help anywhere | 09:49 |
| rpittau | any core please review https://review.opendev.org/c/openstack/ironic-python-agent/+/941089 this time is in the right branch! :) | 09:53 |
| opendevreview | Merged openstack/ironic-prometheus-exporter master: Trivial: Fix TypeError from None value being returned in sensor data https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/940878 | 11:10 |
| opendevreview | Merged openstack/ironic master: CI: Only artificially pin down the mtu in multinode https://review.opendev.org/c/openstack/ironic/+/941136 | 11:48 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: move hooks execution into shareable module https://review.opendev.org/c/openstack/ironic/+/940332 | 13:33 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow multiple inspection interfaces to load hooks https://review.opendev.org/c/openstack/ironic/+/940333 | 13:33 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface https://review.opendev.org/c/openstack/ironic/+/933066 | 13:33 |
| TheJulia | Good morning | 14:09 |
| JayF | rpittau: I haven't been able to get that inspector patch to pass grenade. It looks like grenade is generally broken on that Branch. | 14:16 |
| opendevreview | Merged openstack/ironic master: [docs] Fix link to metal3-integration page https://review.opendev.org/c/openstack/ironic/+/941214 | 14:36 |
| opendevreview | Verification of a change to openstack/ironic master failed: bootc deploy interface - for bootable containers https://review.opendev.org/c/openstack/ironic/+/937897 | 14:54 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: OCI: Send the auth header to IPA https://review.opendev.org/c/openstack/ironic/+/941252 | 15:06 |
| TheJulia | o/ shermanm | 15:14 |
| shermanm | o/ | 15:49 |
| vsaienko | hello ironic community, please add to your review list https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/941054 - should improve CI stability with slow nodes, when cloudinit does not setup ssh keys in time | 16:15 |
| TheJulia | vsaienko: by chance, were you thinking this is because ovn jobs were sometimes failing? | 16:47 |
| keekz | od | 16:48 |
| vsaienko | yes, it might be easily. I've faced with this issue on one of my patches | 16:50 |
| JayF | is there already a retry or something? that patch seems to be a straightforward improvement | 16:55 |
| TheJulia | vsaienko: yeah, turns out the ovn job has been breaking because of mtu limitations. The existing ssh code tries for a long time fwiw | 17:00 |
| TheJulia | vsaienko: I put up a patch which has been merged which should fix mtu handling. I'm keeping an eye on that | 17:01 |
| TheJulia | "should fix" | 17:01 |
| rpittau | good night! | 17:03 |
| TheJulia | Goodnight! | 17:04 |
| vsaienko | I've seen it here https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/940678, to be more precise https://zuul.opendev.org/t/openstack/build/c675c25988b941b09f18620ddfd9bf2c/logs its not ovn job | 17:06 |
| opendevreview | Merged openstack/ironic master: bootc deploy interface - for bootable containers https://review.opendev.org/c/openstack/ironic/+/937897 | 17:15 |
| vsaienko | please add to review queue https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/941013 - adds scenario tests for vlan aware VMs passed here https://review.opendev.org/c/openstack/networking-generic-switch/+/928490 | 17:22 |
| vsaienko | and 1 line patch that enables trunk extension for tinyipa-multinode job https://review.opendev.org/c/openstack/ironic/+/941023 | 17:23 |
| vsaienko | thanks in advance | 17:23 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic master: Remove Ipmi from the default enabled vendor interfaces https://review.opendev.org/c/openstack/ironic/+/941153 | 17:29 |
| TheJulia | I'm becoming less convinced some of our tempest jobs need to continue to live | 17:38 |
| TheJulia | Looks like the job issues vsaienko is looking at are also mtu related | 17:39 |
| TheJulia | https://8c4653dff114d5897250-284808df0480792a2bc66ed6147fab6e.ssl.cf2.rackcdn.com/940678/8/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/c675c25/compute1/logs/syslog.txt | 17:39 |
| JayF | TheJulia: it does feel like (and the bug I found yesterday is informing this tbh) we pretend our matrix is larger than it is by testing a bunch of different drivers, when we don't have a big enough matrix for features | 17:46 |
| JayF | we focus on the things unlikely to break (the ipmi <> vbmc interactions that haven't changed in years) and not on things more likely to break (edge case features) | 17:46 |
| JayF | I would be easily sold on lowering the jobs we run in total, or moving some to experimental and asking people to run them if it's meaningful | 17:47 |
| TheJulia | We do that, then have overly complex scenario testing which also is fragile | 17:47 |
| TheJulia | We've done a lot of that, we just need to keep doing that | 17:47 |
| TheJulia | also, the trend in job naming is largely about drivers, not about what occurs under the hood | 17:47 |
| TheJulia | that is... a miscommunication on our part but also a tendancy to lean towards more the status quo | 17:47 |
| cardoe | unrelated JayF, https://review.opendev.org/c/openstack/nova/+/941227 | 18:04 |
| cardoe | That's how I intended on it going. | 18:04 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic master: Create Support Resources Page https://review.opendev.org/c/openstack/ironic/+/941293 | 19:19 |
| opendevreview | Steve Baker proposed openstack/ironic master: Utility functions for graphical console drivers https://review.opendev.org/c/openstack/ironic/+/939505 | 19:33 |
| opendevreview | Steve Baker proposed openstack/ironic master: Add ironic-novncproxy service https://review.opendev.org/c/openstack/ironic/+/939191 | 19:33 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Migrate from ironic-lib https://review.opendev.org/c/openstack/ironic-inspector/+/939407 | 21:14 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Fix missing ironic-inspector-status when GLOBAL_VENV is True https://review.opendev.org/c/openstack/ironic-inspector/+/935363 | 21:15 |
| JayF | I seriously need some assistance getting CI to pass ^^ on inspector grenade; even just a rubber duck session or something would be helpful | 21:21 |
| JayF | rpittau: you mentioned a willingness to help at some point, can I cash in on that? :D | 21:21 |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Migrate from ironic-lib https://review.opendev.org/c/openstack/ironic-inspector/+/939407 | 21:26 |
| * JayF asking infra to hold the node | 21:26 | |
| opendevreview | Jay Faulkner proposed openstack/ironic-inspector master: Migrate from ironic-lib https://review.opendev.org/c/openstack/ironic-inspector/+/939407 | 21:30 |
| cardoe | How many yaks can I shave on the way... https://github.com/canonical/cloud-init/issues/6018 | 21:38 |
| JayF | ifname0.100 # for vlan 100 | 21:40 |
| JayF | is very standard naming | 21:40 |
| JayF | the fact that the actual assigned names from the OS are so long that adding a vlan id makes it too long is infuriating :| | 21:40 |
| cardoe | Well OpenStack doesn't officially have "name" as a field. Though it's not prohibited. | 21:41 |
| JayF | for ironic configdrives, that's in the virt driver | 21:42 |
| JayF | which at least makes it seem more doable | 21:42 |
| cardoe | Yep. I can specify my own network_data.json and be good. | 21:42 |
| cardoe | But this is the case of letting nova generate it. | 21:43 |
| cardoe | Which leads me back to my ask of giving baremetal ports a "name" that doesn't have to be unique. | 21:45 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Enable embedded authentication passing https://review.opendev.org/c/openstack/ironic-python-agent/+/941300 | 21:47 |
| TheJulia | Given the character length limit, its not just a baremetal issue | 21:48 |
| TheJulia | its network interfaces in general in linux focusing on desktop use cases | 21:48 |
| JayF | fair, but only baremetal in nova context provides the larger configdrive where it's impacting | 21:48 |
| JayF | so even though it's not bare metal, it's only ironic-facing from an openstack perspective | 21:48 |
| TheJulia | because we have more than virtual interfaces | 21:48 |
| JayF | welcome to the real world, neo | 21:48 |
| TheJulia | yup | 21:48 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: OCI: Send the auth header to IPA https://review.opendev.org/c/openstack/ironic/+/941252 | 21:49 |
| opendevreview | Merged openstack/sushy-tools master: Openstack vmedia - add glance file upload support https://review.opendev.org/c/openstack/sushy-tools/+/939636 | 22:33 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: move hooks execution into shareable module https://review.opendev.org/c/openstack/ironic/+/940332 | 23:07 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow multiple inspection interfaces to load hooks https://review.opendev.org/c/openstack/ironic/+/940333 | 23:07 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface https://review.opendev.org/c/openstack/ironic/+/933066 | 23:07 |
| cardoe | well good news... nobody's inspection hooks throw an exception in the preprocess() method... total aside... it's annoying that the "post" hook is __call__() | 23:08 |
| cardoe | Man... OVN really wants to throw down with me. | 23:15 |
| cardoe | I blame jamesdenton ultimately because he's using his own ironic and not my ironic right now. | 23:16 |
| cardoe | So this is a little bit in that bare metal operator mindset TheJulia and maybe my approach is stupid. Cause I try lots of stupid stuff until I find something that makes sense. | 23:16 |
| cardoe | So previously I was using VLAN networks with OVN in my experimental setup. | 23:20 |
| cardoe | We're ultimately using VXLAN in a fabric and my internal customers can make their own networks which will be VNIs in the fabric. They can make trunk ports and all that fun stuff. | 23:21 |
| cardoe | So I've switched back to the VXLAN network type. Which NGS doesn't support yet but we're using Nautobot for that aspect. | 23:21 |
| cardoe | So just kicking around the idea of letting these tenant networks use DHCP and ovn-metadata-agent so I'd need a neutron network node on there. | 23:25 |
| cardoe | Not sure if that makes sense as a broader feature set. But I do see some rough edges and I know the docs really punt on some of those OVN pieces. What I'm not sure is if that's a use case that should be supported or not. | 23:26 |
| TheJulia | I have feelings about OVN, very mixed feelings | 23:31 |
| JayF | For https://review.opendev.org/c/openstack/ironic-inspector/+/939407, it's actually successfully inspecting the node; it just also boots one at the end and that bit is failing. | 23:32 |
| TheJulia | I think it has been pondered and discussed, but someone just needs to... for lack of a better way to put it, just post some code | 23:32 |
| TheJulia | JayF: failing to boot how? | 23:32 |
| JayF | timeout | 23:33 |
| JayF | https://opendev.org/openstack/ironic-tempest-plugin/src/branch/master/ironic_tempest_plugin/tests/scenario/test_introspection_basic.py#L108 | 23:33 |
| JayF | I'm just trying to understand why we need this check | 23:33 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Improve check_vm_connectivity https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/941054 | 23:33 |
| TheJulia | JayF: got an example job? | 23:33 |
| JayF | https://zuul.opendev.org/t/openstack/build/2cbcdd0b28e842209587b61857c352b9/logs | 23:33 |
| TheJulia | thanks | 23:33 |
| JayF | it's very possible the node for this failure has been held | 23:34 |
| JayF | I asked infra to | 23:34 |
| JayF | looks like it failed to pxe boot, perhaps | 23:34 |
| TheJulia | for a deploy yeah | 23:38 |
| JayF | if my reading of the logs is right | 23:38 |
| JayF | I think dhcp updated about 10 seconds after that boot | 23:38 |
| TheJulia | I don't see why we need to then deploy the node at this point | 23:38 |
| JayF | the dhcp agent logs were littered with complaints about haproxy | 23:38 |
| JayF | yeah, you're on the same line of thinking as m=e | 23:38 |
| JayF | this ccheck is overzealous and asking us to do a lot of things in a row | 23:38 |
| TheJulia | I *could* sort of see it when critical records were required to be populated for nova-secheduler | 23:38 |
| TheJulia | but... its placement time | 23:38 |
| TheJulia | and has been for many moons | 23:39 |
| JayF | I'm pushing a patch to remove this step | 23:39 |
| JayF | if someone wants to -1 it they can fix the grenade job 😠| 23:39 |
| JayF | (I'm not 100% serious about that, but I'm close to my end on this one) | 23:40 |
| opendevreview | Jay Faulkner proposed openstack/ironic-tempest-plugin master: Inspector tests need not provision afterwards https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/941302 | 23:41 |
| JayF | cardoe: can you please volunteer yourself on the ironic dpl thread | 23:43 |
| JayF | cardoe: you half-did here, but I am trying to get it all on mailing list for documentation purposes | 23:44 |
| TheJulia | it also looks like the network is ipv6 enabled as well | 23:44 |
| TheJulia | so it gets v6 when the job likely needs to be locked | 23:44 |
| JayF | I don't know what you mean by that | 23:45 |
| TheJulia | checkout https://dbcb5b6346554cd3ec60-39089c3b9838d3ab017ce571be5f6059.ssl.cf2.rackcdn.com/939407/10/check/ironic-inspector-grenade/2cbcdd0/controller/logs/ironic-bm-logs/node-1_no_ansi_2025-02-11-22%3A54%3A26_log.txt | 23:45 |
| TheJulia | compare to https://dbcb5b6346554cd3ec60-39089c3b9838d3ab017ce571be5f6059.ssl.cf2.rackcdn.com/939407/10/check/ironic-inspector-grenade/2cbcdd0/controller/logs/ironic-bm-logs/node-1_no_ansi_2025-02-11-22%3A40%3A13_log.txt | 23:45 |
| JayF | oh, are you saying the v6 made it fail-faster | 23:45 |
| JayF | as opposed to retrying? | 23:46 |
| JayF | that second link, it still has v6 ll addresses? | 23:46 |
| TheJulia | no IP_VERSION in https://dbcb5b6346554cd3ec60-39089c3b9838d3ab017ce571be5f6059.ssl.cf2.rackcdn.com/939407/10/check/ironic-inspector-grenade/2cbcdd0/controller/logs/old/.localrc_auto.txt | 23:47 |
| TheJulia | no, v6 is slower | 23:47 |
| TheJulia | and ipxe tries both | 23:47 |
| TheJulia | okay | 23:48 |
| JayF | so we should set IP_VERSION=4 in that grenade? | 23:48 |
| TheJulia | checkout https://dbcb5b6346554cd3ec60-39089c3b9838d3ab017ce571be5f6059.ssl.cf2.rackcdn.com/939407/10/check/ironic-inspector-grenade/2cbcdd0/controller/logs/grenade.sh_log.txt and look for IP_VERSION | 23:48 |
| JayF | 2025-02-11 22:13:32.914 | ++ /opt/stack/new/ironic/devstack/lib/ironic:source:514 : IRONIC_IP_VERSION=4 | 23:49 |
| TheJulia | eventually it goes from IP_VERSION=4 to IP_VERSION=4+6 | 23:49 |
| JayF | OH | 23:49 |
| JayF | is neutron's devstack plugin overriding? | 23:49 |
| TheJulia | devstack itself, when it upgrades | 23:49 |
| TheJulia | I guess that is okay, but I bet we're updating a network to be v6 enabled | 23:50 |
| JayF | yes, almost certainly | 23:50 |
| JayF | why isn't this breaky in ironic? | 23:50 |
| TheJulia | Feb 11 22:42:15.020195 np0039827749 neutron-dhcp-agent[210863]: DEBUG neutron.agent.dhcp.agent [-] neutron.agent.dhcp.agent.DhcpAgentWithStateReport method _port_delete called with arguments ({'port_id': '7ca34c82-38f6-4aec-afd2-21a0902b4306', 'network_id': '6f3b3eb9-d707-416a-af16-b4b18403cdd1', 'fixed_ips': [{'subnet_id': '2865a371-0133-43a1-b243-e45b84ffc6e6', 'ip_address': '10.1.0.38'}, {'subnet_id': | 23:52 |
| TheJulia | '1fc6b1e0-60c6-4ed1-a9ac-933f9fc80724', 'ip_address': 'fdc8:c1b:8b6e:0:5054:ff:fe6b:e283'}] | 23:52 |
| TheJulia | its likely intermittent | 23:52 |
| TheJulia | and timing related | 23:52 |
| JayF | and inspection does like, bonus things | 23:52 |
| JayF | so we lose the race vs winning it in the "usual" case | 23:52 |
| TheJulia | yeah, cleanest thing, honestly, just delete the excess boot and move on | 23:52 |
| JayF | I thought I was on a scavenger hunt for post-upgrade- | 23:53 |
| JayF | IP_VERSION-value-changing | 23:53 |
| TheJulia | so, in the prior vresion, it should be 4+6 actually, just... weird | 23:53 |
| JayF | IP_VERSION doesn't even show up in ironic-inspector job defs | 23:55 |
| TheJulia | so it gets default from devstack | 23:55 |
| TheJulia | which is 4+6 | 23:55 |
| JayF | which changed from 4->4+6 | 23:55 |
| JayF | if our before is 4, yeah? | 23:55 |
| TheJulia | well, seems like it started 4 | 23:55 |
| TheJulia | but yeah, shouldbe 4+6 and ironic is defaulting out to 4 | 23:55 |
| JayF | afaict, looking at stable/2024.2, we don't set IP_VERSION wither | 23:58 |
| TheJulia | Yeah, we've long inhereted the base default and lean towards v4 | 23:58 |
| TheJulia | but seeing the job getting a v6 announcement enough to cause it to to break since we don't have v6 records for network booting in place then | 23:58 |
| JayF | i'll mark the itp change as ironic-week-prio | 23:58 |
| JayF | and hopefully we can land it | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!