| opendevreview | Jad Haj Yahya proposed openstack/ironic master: Fix Lenovo SR645 V3 boot loop after RHCOS install https://review.opendev.org/c/openstack/ironic/+/986597 | 05:27 |
|---|---|---|
| opendevreview | Jad Haj Yahya proposed openstack/ironic master: Fix Lenovo SR645 V3 boot loop after RHCOS install https://review.opendev.org/c/openstack/ironic/+/986597 | 11:11 |
| opendevreview | Jad Haj Yahya proposed openstack/ironic master: Fix Lenovo SR645 V3 boot loop after RHCOS install https://review.opendev.org/c/openstack/ironic/+/986597 | 11:33 |
| cardoe | janders: you hear from iury? He was going to send an invite to me but I haven’t gotten anything | 11:44 |
| janders | cardoe yes, let me resend | 11:44 |
| iurygregory | good morning ironic | 12:07 |
| cardoe | Morning. | 12:22 |
| cardoe | iurygregory: I’ve got a naughty box with a BIOS update… https://etherpad.opendev.org/p/cardoe-bad-bootprogress | 12:23 |
| iurygregory | cardoe, OMG | 12:24 |
| TheJulia | ewwwwww | 12:56 |
| TheJulia | cardoe: did it stay that way? | 13:06 |
| TheJulia | was it having an OEM party in setup? | 13:06 |
| cardoe | Yeah and that's normal. | 13:06 |
| TheJulia | Inquiring minds must know on the bare metal sitcom. | 13:06 |
| TheJulia | wut | 13:06 |
| cardoe | So when its displaying their graphical progress bar of stuff (like doing the BIOS settings change) it reports OEM | 13:06 |
| TheJulia | oh, okay | 13:07 |
| TheJulia | That... is sort of expected | 13:07 |
| TheJulia | translation: they have sub-states that they couldn't build consensus on | 13:07 |
| cardoe | curl 'https://10.97.16.13/redfish/v1/Systems/System.Embedded.1' | jq '.BootProgress.LastState' reported 'null' when I expected to see Ironic print a log message. | 13:07 |
| TheJulia | and they are hiding in that, so the oem data might say something like "memory checking" | 13:07 |
| cardoe | conductor needs to be restarted to even attempt to poll the state of the box | 13:07 |
| TheJulia | oh, we break down there? :( | 13:08 |
| cardoe | yeah the box is stuck in "clean wait" | 13:09 |
| cardoe | and has booted into the previous OS | 13:09 |
| cardoe | We tried to use custom runbooks for cleaning to add a step to ensure that the right BIOS settings were always on the box to ensure we could get into IPA | 13:10 |
| cardoe | What I'm wondering is if BootProgress was null instead of a dict and an exception was thrown inside of the function and our @node_periodic wrapper doesn't catch any exception and then stops checking? | 13:20 |
| cardoe | or if our code to check if any pending changes are there is bad and throwing an exception | 13:20 |
| cardoe | I just cannot believe an exception is quietly swallowed. | 13:20 |
| cardoe | But I just don't know where it's going | 13:21 |
| TheJulia | That is possible | 13:21 |
| TheJulia | We HAVE seen some weird edge cases like that in the past, where in state change we catch it literally in that transition so what we get it sort of out of spec or doesn't make sense | 13:21 |
| TheJulia | This would be a more egregious example, but yeah | 13:22 |
| iurygregory | wow, that is crazy .-. | 13:22 |
| cardoe | oh wtf | 13:23 |
| TheJulia | cardoe: big words please ;) | 13:23 |
| cardoe | It's not swallowed. It's just not passing the check and we're coming out of an unlogged case. | 13:23 |
| TheJulia | oh, that would do it as well | 13:24 |
| cardoe | So I'm visually comparing the step data which are the settings we expected to be changed. | 13:24 |
| cardoe | To what's actually on the box. | 13:24 |
| cardoe | And HttpDev1Interface and HttpDev2Interface are swapped. | 13:24 |
| cardoe | From what the step has in it. | 13:24 |
| cardoe | bah and it just logged that | 13:25 |
| cardoe | oh but (not on IRC) restarted the conductor. Lemme got smack them with a noodle. Cause they ruined my test environment. | 13:26 |
| cardoe | But yeah if a setting doesn't change (or cannot change) there's no way to abort out now. | 13:27 |
| cardoe | I feel like we should have a timeout value in that to just fail the step after a while. | 13:27 |
| TheJulia | cardoe: that seems like why wet cats exist, although instead of tossing them, I suggest you dry them and let them cuddle with you on your lap/sholders/head while you are on the needful "why did you do that!?!" call ;) | 13:27 |
| cardoe | ohhh I see why | 13:28 |
| * cardoe shakes fist at Dell. | 13:28 | |
| * TheJulia notes step 2 is ????? and step 3 is feline world domination | 13:28 | |
| TheJulia | do tell! | 13:28 |
| cardoe | TheJulia: doggos are nicer. cats come at me with hate lasers | 13:28 |
| cardoe | So we set the HttpDev1Interface and enable it. We're "clearing" the HttpDev2Interface to the default but also disabling it. When the field is disabled you cannot change the interface value. | 13:29 |
| TheJulia | cardoe: you have to let them sit on your head so they may capture your brain's thermal output. | 13:30 |
| iurygregory | cat with lasers do exist | 13:30 |
| iurygregory | :D | 13:30 |
| TheJulia | so do sharks! | 13:30 |
| iurygregory | yup! | 13:31 |
| cardoe | I have a little welsh terrier that is the more cat than some cats. | 13:33 |
| cardoe | yeah so the API accepts that change for pending. And then the job that's created drops the interface name change cause of the disabled. | 13:33 |
| TheJulia | My corgi was basically raised by cats... | 13:34 |
| cardoe | Still don't know why the main interface name didn't change either... I can understand the disabled one. | 13:34 |
| cardoe | But yeah... we need an escape hatch when the BIOS settings haven't applied for a while. Cause right now the only thing I can do is edit the DB or manually change the box until ironic is happy. | 13:34 |
| cardoe | After restarting conductor to make it restart the BIOS settings checks. | 13:35 |
| cardoe | So some sharp edges be here. | 13:35 |
| cardoe | Anyone have any further on https://review.opendev.org/c/openstack/ironic/+/984663 or can I +W it? | 13:36 |
| cardoe | I wanna write some tempest tests and the client side today. | 13:36 |
| TheJulia | There are always sharp edges around firmware/bios settings, unfortunately we have to find them the painful way :( | 13:38 |
| iurygregory | I would say is ok to +W | 13:38 |
| cardoe | TheJulia: yep that's my goal this cycle | 13:39 |
| iurygregory | was about to mention releasenote, but up to you cardoe o/ | 13:40 |
| TheJulia | cardoe: one comment left on that, please take a look, but otherwise I workflowed it | 13:40 |
| TheJulia | iurygregory: I think he needs to revisit it anyway but it can be in follow-up | 13:40 |
| iurygregory | TheJulia was faster hehe | 13:40 |
| iurygregory | yeah, I also didn't want to re-run full CI because of a releasenote, unless something was really necessary | 13:41 |
| cardoe | TheJulia: so can you point me to an example of what to do with the version? | 13:41 |
| TheJulia | I'm seriously wondering if you even need to specify the version | 13:42 |
| TheJulia | your just doing object hydration right? | 13:43 |
| TheJulia | truthfully, objects should be hydrated at the object layer, not the db model | 13:43 |
| TheJulia | and interaction layer | 13:43 |
| frickler | there seems to be a regression in ipa for raid setups, see https://bugs.launchpad.net/ironic/+bug/2150502 | 13:51 |
| TheJulia | so, distros are now shipping singed artifacts which don't grok software raid | 13:53 |
| TheJulia | gee, sounds like software raid is on it's way out then. | 13:53 |
| TheJulia | (kind of a hard take, but it means caring has dropped to zero as computers are disposable then) | 13:53 |
| TheJulia | problem 3 on that seems like it needs to be addressed in the dib build process. problem 2 seems like something Ironic can fix | 13:54 |
| frickler | yes, problem 2 specifically is the regression I was referring to, should've been more explicit | 13:55 |
| TheJulia | frickler: looks like that is non-secure-bootable as a result of problem 1 in general | 13:55 |
| TheJulia | to be fair, and focus discussions, it might be worthwhile to futher delineate that item since we can't really fix #1's base issue, #2 doable, #3 much more a starting state issue which is distro specific i.e. likely a dib element fix | 13:56 |
| frickler | well not sure how much influence the larger ironic community could have on grub packagers regarding #1, but I agree only #2 is directly actionable. maybe some better documentation for #1 could be helpful and could see some future where I do that myself | 13:59 |
| TheJulia | Ironic basically has no pull. Some minor respect at RH, but they are focused on minimizing the cases they must support and trying to direct to systemd-boot | 14:01 |
| TheJulia | better docs on #1 would be totally fair | 14:02 |
| frickler | so maybe switching to gummiboot would be the path forward. tbh I never looked at that before | 14:05 |
| cardoe | iurygregory: I still think we have a bug because the settings change that ironic via sushy would have done return back a 400 error... so before we can even get into that deadlock case of checking... we should have seen the 400 and not proceeded. | 14:06 |
| TheJulia | I'm not sure we can really force/assert that forward at the ironic level. The issue seems more rooted at the cases folks contributing on that level care about are reducing in focus to the most common use cases and software raid I guess is becoming a fringe approach. That itself likely needs to be a doc callout though. | 14:10 |
| JayF | I was hoping to land https://review.opendev.org/c/openstack/ironic-specs/+/986018 today but there are still members of the community who haven't reviewed it | 14:27 |
| JayF | I'm going to land it sometime after 9 today my time (~90-120 minutes) ; if you haven't provided feedback please do so | 14:28 |
| JayF | as I'll be recording my PTG summary video based on the contents of this very soon | 14:28 |
| TheJulia | ack ack | 14:39 |
| opendevreview | Armin proposed openstack/ironic master: fix: oci image service handling webserver_verify_ca when it's string https://review.opendev.org/c/openstack/ironic/+/986668 | 14:53 |
| opendevreview | Merged openstack/ironic master: build runbook traits via ORM relationship in create_runbook https://review.opendev.org/c/openstack/ironic/+/984663 | 15:00 |
| TheJulia | https://bugs.launchpad.net/ironic/+bug/2150628 sounds like a fun one, I bet the vendor is not being captured/identified as we expect for lenovo gear :( | 15:27 |
| TheJulia | I requested baremetal node show output, I guess this is coming out of Red Hat's own work, but we're going to need that output to understand | 15:28 |
| opendevreview | Armin proposed openstack/ironic master: fix: oci image service handling webserver_verify_ca when it's string https://review.opendev.org/c/openstack/ironic/+/986668 | 16:09 |
| iurygregory | cardoe, yeah if we get 400 we should not proceed, but I seem to remember places where with 400 we would retry some operations, not sure if its our case here | 16:49 |
| iurygregory | TheJulia, Jad is from my team | 16:50 |
| TheJulia | cool, but we can't reproduce in upstream context | 17:03 |
| iurygregory | yeah ... | 17:03 |
| JayF | TheJulia: https://bugs.launchpad.net/ironic/+bug/2148317 is now public if/when you wanna push to gerrit | 17:46 |
| opendevreview | Julia Kreger proposed openstack/ironic master: security: validate molds url against swift in keystone catalog https://review.opendev.org/c/openstack/ironic/+/986698 | 17:48 |
| TheJulia | ^^^ | 17:49 |
| JayF | cid: can you please work with TheJulia to draft an OSSA for this issue and get the patches backported. This is another public security bug, like the console one we did the other day. | 17:50 |
| JayF | Since it's not embargoed, you can propose the patch to add the OSSA directly to the ossa/ repo for code review by Ironic+VMT | 17:50 |
| cardoe | I commented on the release note about wording. | 17:54 |
| cardoe | I don't even know what molds is. | 17:54 |
| TheJulia | fair, I'm going to start working on a follow-up patch to rip molds entirely | 17:54 |
| TheJulia | I do want to verify CI passes, but really its way outside the tested path as well | 17:55 |
| cid | JayF, Alright, let me see to that. | 17:56 |
| cid | iurygregory, when you get some review time, I had proposed release cuts for a security fix the other day. I believe we need a vote from the release liaison, and I also need the review since it's my first release cut PR. | 17:56 |
| cid | master, and the 4 maintained stable branches | 17:57 |
| TheJulia | JayF: I'll revise that patch as soon as I have initial feedback from CI, doug's feedback is solid but I need to wordsmith it a little | 18:03 |
| JayF | I will be heading to the airport in mere hours, so you may have to get +2s from others, but there are others now :D | 18:03 |
| TheJulia | yeah, I don't think that will be much of an issue in like an hour | 18:04 |
| JayF | I said hours but I haven't packed so from upstream POV I'll be gone sooner lol | 18:04 |
| iurygregory | cid, sure, on it | 18:05 |
| TheJulia | dude, go pack. | 18:05 |
| * TheJulia goes and retreives frozen pizzas for the wifey | 18:05 | |
| iurygregory | cid, Dalmatian went EOL no? or it was other branch /me checks | 18:05 |
| cid | Dalmation is 2024.2 which should be maintained. | 18:06 |
| iurygregory | https://review.opendev.org/c/openstack/releases/+/984974 | 18:07 |
| iurygregory | not really... | 18:08 |
| iurygregory | in theory we probably won't need https://review.opendev.org/c/openstack/releases/+/986449/ since we can update https://review.opendev.org/c/openstack/releases/+/984974 | 18:08 |
| iurygregory | and include in the eol tag.. | 18:08 |
| iurygregory | at least I think it would make sense, not sure what others think | 18:09 |
| iurygregory | would you be ok in updating https://review.opendev.org/c/openstack/releases/+/984974 to include the right commit ? | 18:09 |
| cid | I am a little confused about what you mean | 18:10 |
| TheJulia | so, until it has merged and the tag is set, we still need to | 18:11 |
| TheJulia | Truthfully, odds are we'll backport on unmaintaed as well | 18:11 |
| TheJulia | but only once we land it | 18:11 |
| iurygregory | https://review.opendev.org/c/openstack/releases/+/984974 is not yet merged so we don't have the tag in place | 18:11 |
| iurygregory | so we can just point the tag to the latest commit (which is the one with your security fix) | 18:12 |
| iurygregory | and we won't need "26.1.6" tag | 18:12 |
| iurygregory | we would have "2024.2-eol" | 18:13 |
| cid | Oh, so, instead of the 2024.2-eol tag. | 18:13 |
| cid | I will push a patch now that you're still around and you see if that's what you mean :D | 18:14 |
| cid | Give me a moment | 18:14 |
| iurygregory | I'm still around all day, just my working hours aren't normal :D | 18:14 |
| iurygregory | like, I'm going to the gym in 15min and I will be back working at night :D | 18:15 |
| iurygregory | so I will check things again later | 18:15 |
| cid | ++, tks. | 18:15 |
| cardoe | So question on policy and service users.... working on some of my mapping stuff. | 18:24 |
| TheJulia | will I cry? | 18:26 |
| cardoe | If I have my service users "nova" and "ironic" for example in service:service (domain:project) but then I have the owner of my nodes as infra:baremetal (domain:project form again). Should the "nova" user have the "service" role on infra:baremetal and authenticate in nova.conf with domain:project of infra:baremetal because I've changed rbac_service_project_name to "baremetal". | 18:29 |
| cardoe | Which initially felt correct. | 18:30 |
| cardoe | But now I'm looking at adding some other Ironic nodes with the noop driver for example which I don't want nova to interact with.... again... seems to work fine... | 18:30 |
| cardoe | But then I theoretically want a "nova like scheduler" for those other devices.... | 18:31 |
| cardoe | So those other devices maybe live in infra:gavinbelsonsignaturebox So I don't know how my other thing would get the service role permission to those devices. Am I abusing Ironic too much? Or am I off the rails here? | 18:32 |
| cardoe | Do I need to go sit in the time out corner? | 18:34 |
| TheJulia | lol | 18:36 |
| TheJulia | No, no timeout corner for you. | 18:36 |
| cid | iurygregory, I have updated the commit | 18:37 |
| TheJulia | So, the issue is merging a "service project" view of access, which is a weird sudo elevation of sorts which is configurable in ironic and the policy (yay, projects not wanting to do system scope) I think the "ironic" way is to scope limit both and leverage owner concepts | 18:38 |
| TheJulia | so bifrucate on owner and have the rest of the fleet in the other "owner project" | 18:38 |
| TheJulia | does that make sense? | 18:38 |
| TheJulia | or do I need to go to the timeout corner? | 18:38 |
| cardoe | That's what I'm trying to do is bifurcate on owner and not make nova or anything else mad | 18:40 |
| cardoe | it looks like the endpoints nova hits might be "(role:reader and system_scope:all) or (role:service and system_scope:all) or rule:service_role" with the rule expanding to "(role:service and project_name:%(config.service_project_name)s)" | 18:40 |
| TheJulia | so, config.service_project_name is not int the filter | 18:41 |
| TheJulia | service_role rule is booleanified if memory serves on startup | 18:41 |
| TheJulia | so, not do that, and not really rely upon the service modeling at all but have the service project it uses be the owner | 18:41 |
| cardoe | It's copied from [DEFAULT]rbac_service_project_name to config.service_project_name on startup. | 18:42 |
| TheJulia | If I'm properly groking the question | 18:42 |
| TheJulia | oh, cool | 18:42 |
| cardoe | Well if I let nova see the gavin belson boxes, it throws a fit that they don't have resource_classes and properties and other stuff. | 18:42 |
| TheJulia | .... huh | 18:43 |
| TheJulia | that seems... odd | 18:43 |
| cardoe | They're essentially stand ins for networking automation. | 18:43 |
| cardoe | So I have a project "julia" and you decide that you want a Gavin Belson Signature box so I assign it to you as a lessee just like Nova/Ironic interaction with a real device. But as part of that I can create Neutron ports on your network and "vif attach" through TBN can do the magic. | 18:44 |
| cardoe | So I guess in a round about way we can blame Clif and JayF for this idea cause I was like hrm... I wonder if TBN could do the heavy lifting here. | 18:45 |
| cardoe | I might also need to step away to have some lunch and be randomly pinging people on IRC cause the welshie believes she's entitled to sit in my lap right now and moving would ruin her day. | 18:46 |
| TheJulia | sounds like the welshie is a cat ;) | 18:48 |
| cardoe | sphinx-build: error: unrecognized arguments: -W -j auto -E -b html doc/source doc/build/html | 19:03 |
| cardoe | That's weird when I run "tox -e docs" | 19:03 |
| cardoe | Well I let Claude rewrite my personas and deployments docs. I don't love it yet. But is there some way to see the rendered version when built with Zuul? | 19:07 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: docs: add deployment scenarios and user personas guide https://review.opendev.org/c/openstack/ironic/+/986702 | 19:11 |
| cardoe | I'll push it anyway since I've gotta take the kids for braces. | 19:11 |
| opendevreview | Julia Kreger proposed openstack/ironic master: security: validate molds url against swift in keystone catalog https://review.opendev.org/c/openstack/ironic/+/986698 | 19:13 |
| clif | cardoe: huh, docs is failing for me locally now, there's a bunch of warnings now about some api docs not being included in any toctree | 19:25 |
| clif | is that what you're running into? my main question is how did that sneak in without the build failing in gerrit | 19:26 |
| cardoe | I dunno. I didn’t have api errors. | 19:33 |
| clif | also I have seen this error before: `sphinx-build: error: unrecognized arguments: -W -j auto -E -b html doc/source doc/build/html` but it shouldn't fail the build | 19:33 |
| clif | it's like a spurious invocation but I haven't been able to track down where it happens | 19:34 |
| clif | hmmm maybe my tree is unclean somehow | 19:34 |
| TheJulia | cardoe, iurygregory, clif: https://review.opendev.org/c/openstack/ironic/+/986698 could use quick reviews | 19:35 |
| TheJulia | stevebaker[m]: thanks! | 19:35 |
| cardoe | Reviewed. | 19:36 |
| clif | +2 | 19:39 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove deprecated configuration molds feature https://review.opendev.org/c/openstack/ironic/+/986703 | 20:03 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Preserve ESP filesystem label during software RAID relocation https://review.opendev.org/c/openstack/ironic-python-agent/+/986707 | 20:39 |
| guilhermesp___ | hi ppl! aside from my saga with software raid ( i made it to work with ubuntu at least ) -- im still facing a blocker with supermicro deprovisioning process.... tried to put together what i observed so far and things ive tried https://paste.openstack.org/raw/bfcnkrwJYFWrmvHBDeA5/ | 20:54 |
| guilhermesp___ | thanks for 986707 btw TheJulia . I might try it as soon as i uncover this issue ( which happens with raid or non-raid ) | 20:55 |
| TheJulia | I didn't try it, but its what claude is thinking and seems straight forward | 20:55 |
| TheJulia | guilhermesp___: are you using ipmi or redfish? | 20:56 |
| guilhermesp___ | i tried both | 20:56 |
| guilhermesp___ | so sticking with ipmi right now | 20:56 |
| TheJulia | so, basically with ipmi, supermicro doesn't really convey reality | 20:57 |
| TheJulia | and they are super opinionated yet also in the club of "not investing in ipmi" | 20:57 |
| guilhermesp___ | i was thinking that there was no investing on redfish actually lol | 20:58 |
| TheJulia | so, if you could collect data with redfish, likely better | 20:58 |
| guilhermesp___ | ive even read some release notes from latest bios version | 20:58 |
| TheJulia | with some supermicro gear, I've literally had to query raw data out | 20:58 |
| TheJulia | which is a pain | 20:58 |
| TheJulia | eh, I doub't it given DMTF is active and IPMI is a dead standard. | 20:59 |
| TheJulia | There is totally this pool of folks who want to just send raw data and not document it | 21:00 |
| guilhermesp___ | yesterday i read this https://www.supermicro.com/Bios/softfiles/28082/H11DSU-iN_BIOS_v3_5_Release_Notes.pdf , specifically BIOS R3.0 (07/12/2024) changelog, item #25 and i had hope. but hope was destroyed on the subsequent test | 21:00 |
| TheJulia | but... redfish is more about surfacing allt he details | 21:00 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Document software RAID image compatibility caveats https://review.opendev.org/c/openstack/ironic/+/986718 | 21:09 |
| opendevreview | Stephen Finucane proposed openstack/python-ironicclient master: Add missing py.typed file, typing classifier https://review.opendev.org/c/openstack/python-ironicclient/+/986719 | 21:10 |
| iurygregory | poor guilhermesp___ =( good luck bro | 21:21 |
| TheJulia | cardoe: that audit discussion might actually bring about system scope ;) | 21:21 |
| TheJulia | at least, more adoption of it | 21:21 |
| guilhermesp___ | thx iurygregory lol im persistent, i might try more tomorrow. Enough for today | 21:26 |
| iurygregory | ++ | 21:40 |
| opendevreview | Merged openstack/ironic master: security: validate molds url against swift in keystone catalog https://review.opendev.org/c/openstack/ironic/+/986698 | 21:50 |
| iurygregory | cardoe, we lost you | 22:10 |
| TheJulia | ?!?!?!? | 22:11 |
| iurygregory | we are having a meeting | 22:11 |
| TheJulia | ahh, didn't realize | 22:11 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix RBAC field redaction when owner not in requested fields https://review.opendev.org/c/openstack/ironic/+/986723 | 22:55 |
| TheJulia | dtantsur: that is exactly what I thought was going on when we were talking | 22:56 |
| opendevreview | Jay Faulkner proposed openstack/ironic stable/2026.1: security: validate molds url against swift in keystone catalog https://review.opendev.org/c/openstack/ironic/+/986725 | 23:05 |
| iurygregory | dinner time now :D | 23:12 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!