| TheJulia | Oh, that explains it the ! | 02:48 |
|---|---|---|
| TheJulia | Then | 02:48 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Store driver objects in a global dict https://review.opendev.org/c/openstack/networking-generic-switch/+/949830 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Move _is_port_supported, _is_port_bound to utils https://review.opendev.org/c/openstack/networking-generic-switch/+/949831 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Improve test coverage of NetmikoSwitch commands https://review.opendev.org/c/openstack/networking-generic-switch/+/949832 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Clean up add_subports_on_trunk, del_subports_on_trunk https://review.opendev.org/c/openstack/networking-generic-switch/+/949833 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Create service plugin to configure security groups https://review.opendev.org/c/openstack/networking-generic-switch/+/949834 | 03:59 |
| opendevreview | Steve Baker proposed openstack/networking-generic-switch master: Support security groups in netmiko base class https://review.opendev.org/c/openstack/networking-generic-switch/+/949835 | 03:59 |
| rpittau | good morning ironic! o/ | 07:16 |
| dtantsur | rpittau: if you have a second: https://review.opendev.org/c/openstack/ironic/+/945482 (requires for future IrSO work) | 07:51 |
| rpittau | ack, approved | 07:52 |
| dtantsur | thank you (and good morning) | 07:55 |
| rpittau | :) | 07:55 |
| frickler | ironic-grenade failure in gate, it isn't obvious to me if this is just a race condition in devstack or some deeper issue, maybe someone wants to take a look? https://zuul.opendev.org/t/openstack/build/04584e9ae5ca4b1c978b3b6f941c15a2 | 08:20 |
| frickler | what is the status of https://specs.openstack.org/openstack/ironic-specs/specs/16.0/in-band-deploy-steps.html, completed? the raid docs still only mention creating config via clean step, not during deployment? | 08:35 |
| dtantsur | frickler: the general work is completed, yes. The docs might be outdated, I'm sure you can use the apply_configuration step at least with software RAID. | 08:36 |
| frickler | dtantsur: ok, I'll look into that, thx. you don't happen to have an example command handy, do you? ;) | 08:38 |
| dtantsur | I'm afraid I do not. It should not be much different from the normal flow: set target RAID config, add the raid.apply_configuration step to your deploy request or deploy template. | 08:41 |
| dtantsur | We should have generic example for the latter. | 08:41 |
| dtantsur | Btw, Redfish RAID is also supported as a deploy step: https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/redfish/raid.py#L739-L743 | 08:42 |
| dtantsur | And it seems that I was wrong: the RAID config must be passed explicitly, not via target_raid_config. | 08:43 |
| dtantsur | frickler: ^^ | 08:43 |
| frickler | dtantsur: thx for the pointer | 08:46 |
| opendevreview | Verification of a change to openstack/ironic master failed: Allow configuring RPC client TLS separately from server https://review.opendev.org/c/openstack/ironic/+/945482 | 08:53 |
| rpittau | dtantsur: after (a lot of) tests I believe running bifrost with python 3.12 on CS9 requires quite a lot, maybe too much, effort | 09:11 |
| rpittau | the problem is that we can run ironic on python3.12 in a separate venv, but bifrost itself requires ansible, with some system integration modules (e.g. selinux, policycoreutils) which do not exist precompiled for CS9 | 09:11 |
| rpittau | compiling them is possible but it will modify some system python modules, besides being not a small task, and we'll have to maintain them | 09:11 |
| rpittau | I guess pinning upper-constraints is the best alternative, until we're ready to switch to CS10 | 09:11 |
| rpittau | TL;DR yeah, I'm giving up :D | 09:11 |
| rpittau | another alternative would be compiling policycoreutils for the venv, but again need to maintain that working | 09:13 |
| jovial | Do you still use the python-ironic-inspector-client to create introspection rules for the built-in inspector (as opposed to the legacy standalone service)? | 09:16 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Fix python interpreter when installing in venv https://review.opendev.org/c/openstack/bifrost/+/949518 | 09:19 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Default ansible to version 10.x https://review.opendev.org/c/openstack/bifrost/+/948245 | 09:21 |
| jovial | ahh, I've found: https://review.opendev.org/c/openstack/python-ironicclient/+/937628 to add support for inspection rules | 09:30 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Pin upper-constraints in CS9 jobs for Py3.9 compatibility https://review.opendev.org/c/openstack/bifrost/+/949861 | 09:35 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Pin upper-constraints in CS9 jobs for Py3.9 compatibility https://review.opendev.org/c/openstack/bifrost/+/949861 | 09:46 |
| iurygregory | good morning Ironic | 11:13 |
| dtantsur | rpittau: have you considered a separate venv for ansible? | 12:17 |
| dtantsur | it's ugly, but well... there is no good solution | 12:17 |
| dtantsur | jovial: thank you for a reminder :) | 12:17 |
| dtantsur | rpittau: hmm, why does CI pass on https://review.opendev.org/c/openstack/bifrost/+/949518 if CS 9 is not working? | 12:21 |
| rpittau | dtantsur: CS9 is working at the moment, UCs have python 3.9 again | 12:29 |
| dtantsur | ahhh, good | 12:29 |
| rpittau | dtantsur: I've considered multiple venvs, I actually started in that direction, and I got lost at some point! :D | 12:37 |
| rpittau | it's a big effort, I mean I can keep working on that maybe for the long term | 12:37 |
| dtantsur | Hmm, probably not worth it. We can wait and see how the situation develops. And ask the requirements team to give us heads-up when they're forced to remove the 3.9 constraints: at that moment we can pin the u-c | 12:37 |
| rpittau | yeah, I'm trying to give the requirements team a hand to maintain UC, so I can keep an eye there too | 12:39 |
| dtantsur | ++ | 12:39 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Add schema for node firmware API https://review.opendev.org/c/openstack/ironic/+/945943 | 12:39 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Ensure parameter transform happens early https://review.opendev.org/c/openstack/ironic/+/948795 | 12:42 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Add schema for node firmware API https://review.opendev.org/c/openstack/ironic/+/945943 | 12:43 |
| TheJulia | good morning | 13:17 |
| dtantsur | morning TheJulia | 13:21 |
| TheJulia | rpittau: grenade is toast right now, rechecking won't fix it | 13:31 |
| rpittau | TheJulia: yep, I realized after the second recheck :) | 13:32 |
| * TheJulia suspects her devstack machine is also toast | 13:32 | |
| okamitok[m] | Good morning, thanks for the guidance the other day. The issue was infact with my network config the auto assigned neutron dhcp was sharing an IP with one of the interfaces on the controller node. | 13:34 |
| okamitok[m] | Working through the last details now dhcp is working but the host didn't pxe boot so going through the configs to see where I may have another misconfig. | 13:35 |
| TheJulia | yeah, you can't co-mingle those network ranges between static configuration and the network used by neutron | 13:40 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Pin upper-constraints in CS9 jobs for Py3.9 compatibility https://review.opendev.org/c/openstack/bifrost/+/949861 | 13:59 |
| TheJulia | well, bare devstack on master correctly populates the url | 14:32 |
| TheJulia | hmmmm | 14:43 |
| * TheJulia wonders if grenade has finally bitrot too far | 14:58 | |
| TheJulia | okay, I think I have a solution | 15:05 |
| TheJulia | walking dog first | 15:05 |
| dtantsur | Walking dog instead of debugging grenade is a solution of its own :) | 15:11 |
| JayF | has grenade ever actually caught an upgrade issue in ironic? | 15:16 |
| * dtantsur cannot remember | 15:18 | |
| TheJulia | dtantsur: ++ | 15:26 |
| opendevreview | Kaifeng Wang proposed openstack/ironic master: Fix unhandled exception during node tear down https://review.opendev.org/c/openstack/ironic/+/949909 | 15:29 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Fix anaconda job post OSSA-2025-001 https://review.opendev.org/c/openstack/ironic/+/949529 | 15:29 |
| dtantsur | Folks, I'm collecting work items at the top of https://etherpad.opendev.org/p/ironic-eventlet-removal | 15:30 |
| dtantsur | feel free to assign any to yourself (cc JayF cid) | 15:30 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Set an explicit neutron endpoint for grenade https://review.opendev.org/c/openstack/ironic/+/949910 | 15:31 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: cleanup outdated rbac scope enforce jobs https://review.opendev.org/c/openstack/ironic/+/949912 | 15:38 |
| opendevreview | Kaifeng Wang proposed openstack/ironic master: Fix unhandled exception during node tear down https://review.opendev.org/c/openstack/ironic/+/949909 | 15:48 |
| TheJulia | okay, looks like my change has grenade past where it was failing | 15:50 |
| JayF | thank you, good work \o/ | 15:53 |
| cid | dtantsur, ++ | 15:56 |
| TheJulia | JayF: dtantsur: I replied on https://review.opendev.org/c/openstack/ironic/+/946677, I guess the big question we need to ask ourselves, do we want to detect and log in possible cases where we might not be able to fix the metadata due to users not using a network driver which can populate it. I guess the reality is much more, the standalone users are more likely to be willing and able to fix their own metadata issues, | 16:06 |
| TheJulia | where as in the integrated context... we're sort of stuck fixing up stuff like this. Truthfully, I do want to add a "generate and inject metadata" knob in general, but just because I feel like it might help reduce the integrated decoder ring required for users who use neutron or any other future network_interface which has logic behind it | 16:06 |
| JayF | I also wonder if we have standalone users | 16:06 |
| JayF | with flat/noop | 16:06 |
| JayF | just with a dhcp server there and creds baked into image | 16:07 |
| JayF | not even caring a drop about network metadata | 16:07 |
| JayF | that's mainly what I'm curious about; if people are sending metadata and it's invalid, it's not unreasonable to ask them to fix it | 16:07 |
| dtantsur | This is a significant proportion of my customers | 16:07 |
| dtantsur | Missing mtu or services is **not** an invalid metadata | 16:07 |
| JayF | I think it's wise to hide this feature behind a flag, and maybe even hiding the "validate metadata" behind a flag | 16:08 |
| JayF | so ironic will still try to generate it as best as it can, but won't go unhappy if it doesn't have e.g. mtu | 16:08 |
| dtantsur | I don't want noise in the logs when mtu is not present | 16:08 |
| dtantsur | I have enough problems with people picking up random sentences from our logs and filing them as bugs :-/ | 16:09 |
| TheJulia | dtantsur: so, missing mtu on a neutron integrated case is atrocious | 16:10 |
| TheJulia | and problematic | 16:10 |
| dtantsur | Right, I'm fine if you blow up in the Neutron's case :) | 16:10 |
| TheJulia | standalone, don't really care, hopefully they are not re-encapping things like crazy | 16:10 |
| dtantsur | For reference, this is the template which Metal3 uses when network data is generated by CAPM3: https://github.com/metal3-io/metal3-dev-env/blob/599def1e551bb5bc9304823a31a4222110aca319/tests/roles/run_tests/templates/metal3datatemplate-template.yaml#L22 | 16:10 |
| dtantsur | it only has MTU's for VLAN's (and I wonder if hardcoding 1500 is even correct) | 16:10 |
| TheJulia | dtantsur: re logs, fair enough | 16:11 |
| TheJulia | JayF: I put mtu behind a flag in the next patch | 16:11 |
| TheJulia | you guys are really hyper focusing on mtus | 16:12 |
| JayF | I only mentioned mtu because dmitry did :) | 16:12 |
| TheJulia | dtantsur: I'll move up the interface check which should streamline the flow, and keep it out the check/correction logic for loging and all that | 16:12 |
| TheJulia | likely friday | 16:12 |
| dtantsur | I'm using it as an example since it will clearly fail for many cases | 16:12 |
| dtantsur | thanks! | 16:12 |
| TheJulia | well, fail to be able to but the way it is written currently it should just log and move on | 16:13 |
| dtantsur | another real world example (sorry, slack link): https://kubernetes.slack.com/archives/CHD49TLE7/p1734445107520569?thread_ts=1734344907.625989&cid=CHD49TLE7 | 16:14 |
| * TheJulia feels completely out of energy | 16:17 | |
| TheJulia | Looks like https://review.opendev.org/c/openstack/ironic/+/949910 is good, grenade job is running tempest tests now | 16:20 |
| TheJulia | so *way* past where it was failing | 16:20 |
| Sandzwerg[m] | Has opendev a possibility to upload images to share them here? I assume it doesn't work if I paste something in matrix but I'm not sure if the bridge will handle this. | 16:22 |
| JayF | I don't believe so, no | 16:23 |
| JayF | I'd use an external image host | 16:23 |
| Sandzwerg[m] | yeah, let's see whats hip and hopefully not sketchy | 16:24 |
| TheJulia | heh | 16:26 |
| Sandzwerg[m] | https://postimg.cc/mPsLG5qN it seems we run into https://opendev.org/openstack/ironic-python-agent/src/branch/unmaintained/zed/ironic_python_agent/errors.py#L321 I hoped to get more logs today but had no luck so far and now the deployment is broken for different reasons. | 16:26 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Pin upper-constraints in CS9 jobs for Py3.9 compatibility https://review.opendev.org/c/openstack/bifrost/+/949861 | 16:27 |
| TheJulia | ugh, different reasons == not fun | 16:28 |
| TheJulia | Sandzwerg[m]: anyhow, yeah, that sure is looking like it is thinking it should have an EFI partition to put artifacts into | 16:28 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Pin upper-constraints in CS9 jobs for Py3.9 compatibility https://review.opendev.org/c/openstack/bifrost/+/949861 | 16:30 |
| Sandzwerg[m] | That's the thing it does, as far as I can tell. I threw qemu-img info and virt-filesystems on all the images I see, the working and the non working and they look pretty similar | 16:31 |
| TheJulia | I'm super curious about openstack image show output for images | 16:32 |
| Sandzwerg[m] | https://paste.opendev.org/show/b37O8ib2og5YZgk1OoHP/ | 16:33 |
| Sandzwerg[m] | I can also provide these | 16:33 |
| TheJulia | Hmm, all have an efi partition, so thats a good sign | 16:35 |
| TheJulia | .... its really thinking its a partition image then | 16:35 |
| Sandzwerg[m] | https://paste.opendev.org/show/bG6s46qvvhq63mRd6xJU/ | 16:36 |
| Sandzwerg[m] | The test sles is the SLES that is failing, the ubuntu is the noble-something from the output before, just the most recent ubuntu cloud image in qcow upload without setting anything else apart from container and disk format. The last ist a ubuntu from us that get's deployed but then doesn't find it's root disk (I suspect because ironic installs an boot loader) | 16:38 |
| Sandzwerg[m] | But for the issue I described the SLES is the main issue for now | 16:38 |
| TheJulia | wow, okay | 16:39 |
| TheJulia | so yeah, an agent log would be awesome | 16:39 |
| TheJulia | because something is going on and I don't udnerstand it | 16:39 |
| TheJulia | if nobody screams in short order, I'm going to workflow https://review.opendev.org/c/openstack/ironic/+/949910 with a single core to unbrick CI | 16:41 |
| Sandzwerg[m] | That is a customer image, also based on SLES, that works. The virt-filesystem output should inclue it https://paste.opendev.org/show/by4yb0RNQgkXl6h8pmpH/ | 16:41 |
| Sandzwerg[m] | TheJulia: Still working on that. Something else is blocking more deployments currently unless a colleague can fix it for me. And the automatc log upload after deployments fails for this node. Probably some network foo in the test region | 16:42 |
| TheJulia | yeah, that image then I'd also expect to be fine and work | 16:42 |
| JayF | TheJulia: I did the approval as a single core on that ci fix | 16:43 |
| TheJulia | ugh, fun | 16:43 |
| TheJulia | JayF: cool cool, thanks | 16:43 |
| Sandzwerg[m] | I also tried an upstream opensuse-leaf-something that gets proberly deploed as well | 16:43 |
| Sandzwerg[m] | Maybe I have a weg to get agent logs.. (deploys somewhere else again) | 16:49 |
| opendevreview | Merged openstack/ironic unmaintained/yoga: OSSA-2025-001: Disallow unsafe image file:// paths https://review.opendev.org/c/openstack/ironic/+/949177 | 17:02 |
| Sandzwerg[m] | logs! https://limewire.com/d/sTurV?lmwr-ftv=1#MI217oScOM (file.io apparently is now limewire :<) | 17:06 |
| TheJulia | lol, they have a stock price on their webpage | 17:07 |
| TheJulia | wow | 17:07 |
| * JayF checks a calendar | 17:08 | |
| JayF | What's next? A food delivery service called kazaa? | 17:09 |
| Sandzwerg[m] | TIL paste.opendev has a size limit. | 17:09 |
| TheJulia | okay, wow | 17:16 |
| * TheJulia cracks open the agent code | 17:16 | |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: trivial: Fix missing extra space in logging https://review.opendev.org/c/openstack/ironic-python-agent/+/949920 | 17:17 |
| TheJulia | so it writes as a whole disk imae | 17:33 |
| TheJulia | image | 17:33 |
| Sandzwerg[m] | But from understanding it then tries to add a bootloader, which shouldn't be necessary as a whole-disk-image should include everything? | 17:35 |
| cardoe | TheJulia: btw... https://review.opendev.org/c/openstack/nova/+/941227 | 17:46 |
| cardoe | Your stuff is a bit related to https://bugs.launchpad.net/neutron/+bug/1705084 | 17:47 |
| TheJulia | yeah, I'd just do it in ironic at this point | 17:47 |
| cardoe | You know what I'm gonna say right? | 17:48 |
| cardoe | I'm not against what ya did. Just gonna say big happy family. | 17:49 |
| JayF | I mean, big happy family is one thing. Having enough people and time to wire something through forty-seven layers is another :) | 17:50 |
| JayF | I can greatly appreciate a two-pronged approach. Not the first time we've done it (e.g. automated_lessee) | 17:50 |
| cardoe | I'll be your champion. | 17:51 |
| Sandzwerg[m] | So wonderful people of ironic, I wish you a good rest of the day. I'm off for today and tomorrow. But I'll read everything on Monday, have a good weekend o/ | 17:52 |
| JayF | \o | 17:52 |
| cardoe | Today is the day of the shoe on the other foot from yesterday's skyline. | 17:57 |
| cardoe | Cause now I'm looking at the patch queue that my team is carrying, the Flex team is carrying, and mnaser has mentioned before. And they're all submitted against respective upstream projects but just crickets. | 17:58 |
| TheJulia | Sandzwerg[m]'s issue has me semi-stumped, but it looks like something is going sideways in manage_uefi and cascades from there | 18:01 |
| JayF | One of the things I sorta do is ensure any GR-OSS openstack patches get mentioned in channels | 18:02 |
| JayF | like if they sit for a week, ping the channel again | 18:02 |
| Sandzwerg[m] | TheJulia: hmkay. If there is anything I should do or check ping me. As I have on-call during the weekend I have my work laptop with me, and might be able to send something to the colleague trying to debug it from image side. | 18:03 |
| TheJulia | Sandzwerg[m]: have a good weekend, I'll try to dig more and see if I can figure out what is going on | 18:04 |
| Sandzwerg[m] | Thanks ❤️ | 18:10 |
| TheJulia | Sandzwerg[m]: my theory is that it is dos MBR partition table, and strictly speaking that is not supposed to be used with EUFI booting. What happens from there is partx failds to add the partitions laid down, why I have no idea. | 18:22 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: WIP: Remove eventlet https://review.opendev.org/c/openstack/networking-baremetal/+/947985 | 18:37 |
| opendevreview | Merged openstack/ironic master: Set an explicit neutron endpoint for grenade https://review.opendev.org/c/openstack/ironic/+/949910 | 18:49 |
| opendevreview | Jay Faulkner proposed openstack/ironic-specs master: Trait based port selection and dynamic portgroups https://review.opendev.org/c/openstack/ironic-specs/+/945642 | 19:52 |
| JayF | > neutron-server[63436]: CRITICAL neutron.plugins.ml2.managers [-] The 'EntryPoint(name='baremetal', [snip] 'Backend already set to 'eventlet', cannot reinitialize with 'threading''.: oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' | 20:45 |
| JayF | I think networking-baremetal will be unable to be non-eventleted until after neutron is | 20:46 |
| JayF | likely same for ngs | 20:49 |
| Sandzwerg[m] | <TheJulia> "Sandzwerg: my theory is that..." <- Hmmm could be I'll ask the coworker to look into that. | 22:08 |
| TheJulia | JayF: maybe we need to make it smart enough to handle that transition? | 22:19 |
| TheJulia | Surely, there is a way | 22:20 |
| JayF | I suspect if I remove the init_backend stuff | 22:20 |
| JayF | and remove the backend being specified from the oslo messaging init | 22:20 |
| JayF | that we could /maybe/ make it eventlet agnostic | 22:20 |
| JayF | let me try that | 22:20 |
| JayF | hold please :D | 22:20 |
| TheJulia | I'm going to recheck som estuff and lay down, got a migraine from having to fasting labs today | 22:21 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: WIP Remove eventlet, maybe? https://review.opendev.org/c/openstack/networking-baremetal/+/947985 | 22:23 |
| JayF | feel better o/ | 22:24 |
| opendevreview | Julia Kreger proposed openstack/ironic master: trivial: add missing exception to agent code path docstrings https://review.opendev.org/c/openstack/ironic/+/949776 | 22:25 |
| TheJulia | I rechecked a number of things, the world should be much happier | 22:28 |
| JayF | awesome, thank you | 22:30 |
| opendevreview | Verification of a change to openstack/ironic master failed: Allow configuring RPC client TLS separately from server https://review.opendev.org/c/openstack/ironic/+/945482 | 23:03 |
| opendevreview | Verification of a change to openstack/ironic master failed: Allow to unprovision instance from service wait states https://review.opendev.org/c/openstack/ironic/+/948301 | 23:05 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!