| opendevreview | Merged openstack/sushy master: Update hacking to 7.0.0 https://review.opendev.org/c/openstack/sushy/+/984093 | 00:57 |
|---|---|---|
| opendevreview | Merged openstack/ironic master: Fix IPA boot during out-of-band service steps https://review.opendev.org/c/openstack/ironic/+/983010 | 02:42 |
| opendevreview | Merged openstack/sushy master: tls: Short circuit retries on hard tls errors https://review.opendev.org/c/openstack/sushy/+/984071 | 02:43 |
| opendevreview | Merged openstack/ironic-python-agent master: Enforce TLS 1.2 minimum across all connections https://review.opendev.org/c/openstack/ironic-python-agent/+/982174 | 02:59 |
| opendevreview | Merged openstack/ironic-python-agent master: ci: Fix the functional test job https://review.opendev.org/c/openstack/ironic-python-agent/+/984054 | 02:59 |
| opendevreview | Doug Goldstein proposed openstack/ironic stable/2026.1: Fix IPA boot during out-of-band service steps https://review.opendev.org/c/openstack/ironic/+/984894 | 03:12 |
| *** ykarel_ is now known as ykarel | 04:11 | |
| Uggla | Hi all, FYI Nova ptg agenda almost completed --> https://etherpad.opendev.org/p/nova-2026.2-ptg | 09:52 |
| cardoe | Thanks Uggla | 12:18 |
| TheJulia | good morning | 13:15 |
| TheJulia | cardoe: you around/ | 13:48 |
| cardoe | TheJulia: yep | 13:57 |
| cardoe | I didn't do it. Unless I was suppose to and then I did. | 13:58 |
| TheJulia | I think I'm getting there while also waking up | 13:58 |
| TheJulia | trying to figure out why some of our multinode jobs are busted hard | 13:58 |
| kubajj | TheJulia: Is attestation a common thing? I just learned that it exists | 14:01 |
| TheJulia | \o/ | 14:02 |
| TheJulia | First round of success! | 14:02 |
| cardoe | ah I was wondering if you figured out why it failed. | 14:05 |
| TheJulia | compute nodes are failing specifically it looks like | 14:05 |
| TheJulia | https://zuul.opendev.org/t/openstack/build/cf164e7ba8c345f88b62a4540e9d88b7 | 14:05 |
| cardoe | cool.. I had a box stuck in bios.apply_configuration cause it wouldn't get to booting and now have restarted conductor and so even though we fixed the booting on it since its not abortable I cannot take it out of that state without a DB edit I don't think | 14:09 |
| cardoe | TheJulia: is multinode using Q_AGENT=openvswitch or something else? like ovn? | 14:09 |
| TheJulia | so, they are set with openvswitch it looks like | 14:10 |
| TheJulia | err, maybe not actually | 14:11 |
| TheJulia | "openvswitch" | 14:11 |
| TheJulia | okay, its rooted in the change you made | 14:18 |
| TheJulia | and moves the overall pattern from "does the router exist", if so do the thing, to "does the network exist and if so unconditionally run these commands | 14:19 |
| TheJulia | so basically, the issue is the network has been defined, but the namespace may still not exist, so we traded one race condition for another | 14:19 |
| cardoe | Yep. I know it's related to my change... :/ | 14:21 |
| cardoe | Cause previously it was a grep for any netns that matched and ran that command against anyone that existed | 14:22 |
| TheJulia | yeah, we just need to also ensure a namespace is present before we try to run the commands | 14:22 |
| TheJulia | We don't care what, just that it is | 14:22 |
| cardoe | And I hit the issue that I had 2 with 1 being cleaned up and disappearing while it ran so I tried to instead make it target the specific one that should exist. | 14:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix guarding logic ovs qrouter logic https://review.opendev.org/c/openstack/ironic/+/984950 | 14:27 |
| TheJulia | That should do it | 14:27 |
| TheJulia | JayF: Any interest in growing the review jams? I've got some folks interested. | 14:29 |
| JayF | I'm happy to add people to my existing invite for the downstream team. | 14:30 |
| JayF | I'm happy to participate in some that other folks organize as well. I don't think I have the juice to organize a fresh one | 14:30 |
| TheJulia | I think that would be pretty awesome | 14:30 |
| JasonParoly[m] | please add me to the invite if possible, and thank you | 14:32 |
| TheJulia | mumesan[m] abongale you both expressed interest to me as well. I would also be happy to have a calendar item to join in | 14:33 |
| TheJulia | Calendars, the most evil items on the planet ;) | 14:33 |
| JayF | Just DM me email addresses and I'll get em added | 14:37 |
| JayF | or put them in here and brave the spam lol | 14:37 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix guarding logic ovs qrouter logic https://review.opendev.org/c/openstack/ironic/+/984950 | 14:39 |
| TheJulia | abongale, mumesan[m], JasonParoly[m] ^^^ | 14:42 |
| TheJulia | All the spam! | 14:43 |
| abongale | :JayF :TheJulia done | 14:43 |
| TheJulia | FYI, a couple quick reviews on https://review.opendev.org/c/openstack/ironic/+/984950 would be appreicated so I can un-brick n-g-s's gate | 14:44 |
| cardoe | TheJulia: can we not change to check for the specific qrouter instead of ANY qrouter again? | 14:52 |
| cardoe | oh you did the update | 14:52 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: fix guarding logic ovs qrouter logic https://review.opendev.org/c/openstack/ironic/+/984950 | 14:52 |
| TheJulia | yeah, and hopefully fixed the bash being grumpy syntax error now too! :) | 14:53 |
| cardoe | TheJulia: how's that bash gonna work with sudo ip netns list | grep "" when qrouter_id is empty? | 14:53 |
| TheJulia | ugh | 14:53 |
| TheJulia | well | 14:54 |
| TheJulia | empty, it shouldn't matter | 14:54 |
| TheJulia | present, something will be there | 14:54 |
| TheJulia | should be okay, I would think | 14:54 |
| TheJulia | fails open, but is guarded by awk | 14:55 |
| TheJulia | because when ip netns list returns nothing, it literally returns nothing | 14:56 |
| cardoe | So is https://review.opendev.org/c/openstack/ironic-specs/+/984596 too vague or good? | 16:10 |
| * dtantsur bookmarks | 16:13 | |
| dtantsur | immediate comment: s/is_oob/requires_ramdisk/ (in my experience, oob is not clear for people outside of our bubble) | 16:14 |
| TheJulia | Just broadly speaking, I'm also worried by a focus on "let me try to do everything OOB" and there are just some aspects which don't really just work out of band or even if they did, is it really a good idea is a question folks may need to ponder. I am hesitant to plant a seed that can grow to a more OOB oriented model just because of the because we can should we questions start to be big items to consider. | 16:21 |
| * TheJulia hopes that makes sense | 16:21 | |
| JayF | +++++ | 16:22 |
| JayF | I haven't read the spec but ++++++ | 16:22 |
| JayF | the more redfish API interface Ironic needs to work, the less likely things are to work | 16:22 |
| dtantsur | There is no hard dependency though | 16:22 |
| TheJulia | well, and that doesn't prohibit great ideas from surfacing, just... everyone's context will be different | 16:22 |
| dtantsur | But I can imagine a valid argument for having a separate list of hooks (and possible a separate scope for inspection rules) for this new action | 16:23 |
| dtantsur | (I'm asking in the spec about what happens with ports, for example) | 16:23 |
| TheJulia | "Just because risc-v bmc thingie has a bit bang firmware interface, doesn't necessarily mean we should try to ask the bmc to re-flash the OS in it's entirety" but then also go to a multi block device server and UEFI. | 16:23 |
| dtantsur | I have a permanently lingering customer request to update inspection data for provisioned instances, so I can see the motivation though | 16:25 |
| TheJulia | part of me *really* wonders "service" with a step is the right thing for an active node | 16:25 |
| TheJulia | dunno, I need to ponder that one overall | 16:26 |
| TheJulia | and I too, totally get the motivation | 16:26 |
| dtantsur | good point about servicing. cardoe, have you considered that? this would remove a few concerns around the state machine. | 16:26 |
| TheJulia | okay, my devstack change at least gets the mulitnode job back to failing the way we expect it to fail sometimes | 16:29 |
| dtantsur | we all need a bit stability in our job. for example, the multinode job failing. | 16:32 |
| dtantsur | :) | 16:32 |
| TheJulia | i know right! | 16:32 |
| TheJulia | I'm tempted to feed the failed job logs to claude just to see what it might think the issue is and see if it comes up with an answer | 16:33 |
| TheJulia | I'm sure its likely a cross-node comms issue, but it could also be MTU and getting that failure rate to be less would enable us to not hit the same sort of issue through comfort with the job... like... actually passing. | 16:33 |
| dtantsur | claude tends to give food for thought, if nothing else | 16:34 |
| TheJulia | Yeah | 16:34 |
| TheJulia | I think that job network boots as well, I should likely change that (if I haven't already done so) to be vmedia based | 16:35 |
| TheJulia | at could also be that job just takes a long time and is variable based upon the overall resulting layout | 16:46 |
| TheJulia | and under the hood retries | 16:46 |
| cardoe | I’m done with a servicing step for inspection | 16:48 |
| TheJulia | done or down | 16:50 |
| TheJulia | ? | 16:50 |
| TheJulia | Is this a throw down?!? | 16:50 |
| TheJulia | or a hoedown?!? | 16:50 |
| * dtantsur misses throwing wet cats | 16:50 | |
| * TheJulia smells burnout smoke | 16:50 | |
| TheJulia | Its too early for a cat meeting | 16:51 |
| cardoe | Yes it’s a hoedown. | 16:51 |
| cardoe | I would be okay with a servicing / cleaning step. | 16:51 |
| dtantsur | It's funny how because inspection and rescue more or less predate manual cleaning and servicing, they're independent operations | 16:52 |
| TheJulia | There is nothing saying we also couldn't directly map certian verbs to explicit steps | 16:53 |
| TheJulia | (of course, that could also be bonkers confusing) | 16:53 |
| * dtantsur nods | 16:53 | |
| cardoe | What interface ya want it in? Or leave it in inspect? | 16:53 |
| dtantsur | I'd allow steps on the inspect interface, yes | 16:54 |
| dtantsur | Then you don't need is_oob because the "agent" inspect simply won't have the necessary step | 16:54 |
| dtantsur | And no concerns around the new state transitions | 16:54 |
| cardoe | I wanna stick a date/time in the inspect data too so that I know how stale it is. | 16:54 |
| TheJulia | There could be verb transitions which require explicit steps as well | 16:54 |
| TheJulia | service is basically one of those | 16:55 |
| dtantsur | cardoe: big +1 to updated_at on inspection data | 16:55 |
| TheJulia | You need to give us steps or we're going to do the wrong thing potentially becuase your in advanced workflows | 16:55 |
| dtantsur | (checking out now, good evening folks) | 16:56 |
| TheJulia | fyi folks, I'm planning on taking tomorrow off for my mental health and to be charged up sufficient to survive ptg week | 16:57 |
| TheJulia | sufficiently | 16:57 |
| cardoe | Good idea | 17:01 |
| TheJulia | I'm burning hard right now, and largely its just because I'm exhausting context switches quickly | 17:03 |
| TheJulia | a lot of stuff going on | 17:03 |
| TheJulia | I just need one more review on https://review.opendev.org/c/openstack/ironic/+/984950 so I can recheck a pile of NGS stuff | 17:04 |
| cardoe | I'm just wondering if I'm doing https://review.opendev.org/c/openstack/ironic-specs/+/984598 correct | 17:13 |
| cardoe | Cause I was gonna go update all the specs to when they were implemented. | 17:13 |
| JayF | ah didn't realize 984950 has a revision, +2a | 17:14 |
| TheJulia | cardoe: that looks right I think | 17:21 |
| TheJulia | you moved the link from not-implemented to 36.0 right? | 17:21 |
| TheJulia | (this is the sort of thing gerrit doesn't provide great visibility for, unfortunately | 17:21 |
| TheJulia | fyi, it looks like we may ahve some additional ci failures, albeit transient ones due to package updates | 17:38 |
| TheJulia | just check the job los | 17:38 |
| TheJulia | logs | 17:41 |
| cardoe | TheJulia: yeah first link from not-implemented to 36.0 | 17:41 |
| TheJulia | ++ | 17:41 |
| cardoe | TheJulia: yeah it looks like the Ubuntu package mirrors went sideways a little bit ago | 17:42 |
| cardoe | dtantsur: https://review.opendev.org/c/openstack/ironic/+/984663 that was to address your feedback btw. | 17:47 |
| TheJulia | Gah, we're going to have to recheck that job since we got more post_failures | 18:24 |
| TheJulia | (really, pre, but I thank the job for at least giving us a little bit of an idea | 18:24 |
| opendevreview | Verification of a change to openstack/ironic master failed: ci: fix guarding logic ovs qrouter logic https://review.opendev.org/c/openstack/ironic/+/984950 | 18:36 |
| opendevreview | Merged openstack/ironic-python-agent master: trivial: fix imports and other test race changes https://review.opendev.org/c/openstack/ironic-python-agent/+/983917 | 18:40 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: Tune multinode CI job VM resources for stability https://review.opendev.org/c/openstack/ironic/+/985024 | 19:04 |
| TheJulia | so claude's analysis is in huge part the variable ci node performance can really hit that job heavily, which also can result in increased memory pressure as well, so some slight tuning plus timeout extension should yield us to a happier place. It's not so quick anaylsis is basically that IPA took 13+ minutes | 19:06 |
| TheJulia | so, as we know, second cpu can have a huge impact there on test VMs | 19:06 |
| opendevreview | Verification of a change to openstack/ironic master failed: ci: fix guarding logic ovs qrouter logic https://review.opendev.org/c/openstack/ironic/+/984950 | 19:16 |
| opendevreview | Harald Jensås proposed openstack/networking-generic-switch master: GenericSwitchNetmikoConfigError error details https://review.opendev.org/c/openstack/networking-generic-switch/+/985029 | 19:23 |
| cardoe | dtantsur: TheJulia: I think for the out of band inspect I want to nuke https://opendev.org/openstack/ironic/src/commit/192e2e8e7055b5e1d48114d9b4d8459f91939a55/ironic/drivers/modules/redfish/inspect.py#L169 as a precursor | 19:56 |
| cardoe | I'm gonna toss this stuff into a PTG session and maybe we can hash out the design there. | 19:57 |
| JayF | yeah, I'm going to chew on that spec a while for sure | 19:58 |
| JayF | it feels like we're setting up parallel generic pipelines | 19:58 |
| JayF | one for step-based flows, one for inspect-flows | 19:58 |
| JayF | I don't know how to better arrange it, so I'm not saying it's a -1 or anything, just trying to think about where the lines can be drawn and if there's any common thread we can pull thru both of em | 19:58 |
| TheJulia | cardoe: I think for at least if the state is active then yeah, likely not the best idea to add new ports | 19:59 |
| TheJulia | but then I wonder, what if someone adds a network card | 19:59 |
| cardoe | Well I personally use inspection hooks to add the ports. | 19:59 |
| cardoe | The comments on the add port stuff are that it's needed for IPA to have a port. But we're out of band so I don't know why we'd need a port. | 20:00 |
| TheJulia | true | 20:01 |
| TheJulia | and really then, if your out of bound only, your there more for "i replaced a drive", unless you rebooted to get a card | 20:01 |
| cardoe | That's the case I'm thinking of. The box was active. It had a fault in some piece of hardware. It was powered off, repaired and now powered on. | 20:02 |
| cardoe | I'm happy if the workflow is maintenance=true -> run service steps -> maintenance=false like you and Dmitri suggested. | 20:03 |
| TheJulia | okay | 20:04 |
| TheJulia | you've got me agreeing then | 20:04 |
| cardoe | Right now a quick summary of what we've got is a custom iPXE that POSTs the PCI IDs and hardware data about the box back to an API. That API returns a 200 if the hardware is the same that was in there at deployment time. Otherwise it returns a 4xx message and the box doesn't continue to boot. | 20:06 |
| cardoe | Looking to Ironic-ify this operator workflow. | 20:07 |
| * TheJulia twitches slightly | 20:07 | |
| TheJulia | I mean, that is a thing one can do! | 20:08 |
| cardoe | It's a sanity check. | 20:08 |
| cardoe | There's other things as well. | 20:09 |
| TheJulia | of course | 20:09 |
| cardoe | But this isn't the only place that's had some process like this. | 20:10 |
| cardoe | Attestation checks is part of that. | 20:10 |
| opendevreview | Merged openstack/ironic-specs master: record the runbook traits API as implemented https://review.opendev.org/c/openstack/ironic-specs/+/984598 | 20:11 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: fix: make it possible to unset local-link-connection on baremetal port https://review.opendev.org/c/openstack/python-ironicclient/+/985047 | 21:28 |
| cardoe | ohhhhh how many yaks deep can we get todayyyyyyyyyyyy | 21:28 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient stable/2025.1: [stable only] pin setuptools to fix pkg_resources errors https://review.opendev.org/c/openstack/python-ironicclient/+/985049 | 21:31 |
| JayF | cardoe: https://review.opendev.org/c/openstack/ironic-specs/+/984596/1#message-3e128202fa61b831e26cce8253fe6290b63853d8 got a review on your spec | 21:35 |
| JayF | cardoe: likely gonna add a yak to yer stack | 21:35 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix case where neutron ports are not tracked with the baremetal port https://review.opendev.org/c/openstack/ironic/+/985050 | 21:35 |
| cardoe | JayF: that's fine I think I now want it as a service step already | 21:36 |
| JayF | yeah mainly just thinking about the case | 21:36 |
| JayF | what if one of your level 1 openstack admins saw a node in this state | 21:36 |
| JayF | what would they assume? In many places, it might look like a failed delete or bad state (instance_uuid set in INSPECT* states) or operators might be used to seeing INSPECT state and thinking "X" is the next step | 21:37 |
| cardoe | Maybe a better start would be to write down some "possible operator operations" | 21:40 |
| cardoe | e.g. "As Ollie the Operator, I want to X" | 21:40 |
| * cardoe chokes on yak hair. | 21:40 | |
| TheJulia | cardoe: not opposed | 21:51 |
| cardoe | To letting me choke on all this shaved yak hair? Well that's mean.... | 21:52 |
| TheJulia | choking, no. I was quietly ordering folgers decaf in a "yak shaver" box | 21:58 |
| cardoe | https://review.opendev.org/c/openstack/python-ironicclient/+/985049 can we do that? And then backport to 2024.2? Then I can clean up the queue of stuck patches too. | 22:02 |
| TheJulia | sure, done | 22:03 |
| TheJulia | err | 22:03 |
| TheJulia | thats not a backport | 22:03 |
| opendevreview | Verification of a change to openstack/python-ironicclient stable/2025.1 failed: [stable only] pin setuptools to fix pkg_resources errors https://review.opendev.org/c/openstack/python-ironicclient/+/985049 | 22:03 |
| TheJulia | okay, better now | 22:03 |
| cardoe | Right because we've changed stuff... we cannot go to new enough depends in 2025.1 and 2024.2 cause of the upper-constraints.txt and we've fixed it by going to newer deps in 2025.2 | 22:04 |
| cardoe | another fun fact... virtual-media ignores pxe_enabled=True and binds all ports so if you use neutron.fail_on_port_binding_failure=True it doesn't work | 22:07 |
| cardoe | Definitely looking forward to us iterating TBN to like v6 where the magical options can be explicit. | 22:08 |
| JayF | To be explicit: GR's current use cases for TBN are settled, I don't anticipate us being the driver for additional features at the moment. | 22:19 |
| JayF | and we should fix that vmedia shenanigans :( it explains something I had to walk my downstream through that confused me | 22:19 |
| opendevreview | Merged openstack/python-ironicclient stable/2025.1: [stable only] pin setuptools to fix pkg_resources errors https://review.opendev.org/c/openstack/python-ironicclient/+/985049 | 22:23 |
| cardoe | JayF: I mean what's the right fix? You probably want neutron.fail_on_port_binding_failure cause cid wrote it and defaulted it to true. | 22:34 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient stable/2024.2: [stable only] pin setuptools to fix pkg_resources errors https://review.opendev.org/c/openstack/python-ironicclient/+/985055 | 22:36 |
| JayF | cardoe: s/pxe_enabled/ramdisk_enabled/ and then respect it | 22:44 |
| JayF | I think my downstream hacked in something to respect pxe_enabled | 22:45 |
| JayF | but that was a while ago and I was mainly just on the outsides of that | 22:45 |
| JayF | from a workflow standpoint, we shouldn't care if the port is pxe enabled or not, we just care about "What of these ports need to succeed in binding in order to ramdisk" | 22:45 |
| cardoe | JayF: well we're talking about different things. Cause this takes all the ports that are on the node and if the boot driver has "pxe_boot" in it then we just require the pxe_enabled=True ports to bind... if the boot driver doesn't have that then all the ports... I tried to use virtual-media. | 22:52 |
| cardoe | virtual-media doesn't have pxe_boot | 22:52 |
| cardoe | I've got ports that got created by inspection with pxe_enabled=false | 22:53 |
| JayF | so what I'm saying is | 22:54 |
| JayF | we need to genericize the concept | 22:54 |
| JayF | it shouldn't just be "these ports can pxe", it should be "these ports can be used by IPA" | 22:54 |
| cardoe | Oh yes agreed. | 22:54 |
| JayF | which in that case, means we can make pxe_enabled instead mean, e.g. "ramdisk_enabled" and respect that setting for vmedia, too | 22:55 |
| JayF | whether we rename the field in name or only in spirit is up to you ;) | 22:55 |
| cardoe | There's other quirky nuances in there that I really hate. | 22:55 |
| JayF | maybe we can factor out a decoder ring or two :) or at least you can learn to hate the *reason* the code is hateable instead lol | 22:56 |
| cardoe | Technically UEFI can have some ports that run PXEClient and some that run HTTPClient for booting | 22:56 |
| cardoe | I've got a patch somewhere to add to the docs that tell people you must keep both values in sync manually. | 22:57 |
| JayF | port.capabilities [uefi_http, uefi_net, bios_pxe, ipa_enabled] | 22:57 |
| JayF | or something like that | 22:57 |
| JayF | IDK | 22:57 |
| cardoe | yeah that's probably the right answer. | 22:57 |
| cardoe | I think I threw something on the PTG for this topic | 22:57 |
| JayF | I suspect neutron has some prior art in this too | 22:57 |
| cardoe | But that's a good suggestion. | 22:57 |
| JayF | so maybe we can use some bits of their model with modes/capabilities extended for BM cases | 22:57 |
| * JayF hand waves away his lack of neutron understanding :D | 22:57 | |
| JayF | similar to how we (ab?)use traits | 22:58 |
| cardoe | First go get a tri-state boolean and integrate it into the code base. | 22:58 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient stable/2024.2: [stable only] pin setuptools to fix pkg_resources errors https://review.opendev.org/c/openstack/python-ironicclient/+/985055 | 23:04 |
| opendevreview | Doug Goldstein proposed openstack/python-ironicclient master: fix: make it possible to unset local-link-connection on baremetal port https://review.opendev.org/c/openstack/python-ironicclient/+/985047 | 23:06 |
| TheJulia | Hey now, I might want to add some more traits for attestation ;) | 23:20 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!