| opendevreview | Merged openstack/python-ironicclient stable/2024.2: Fix parallel initial version negotiation https://review.opendev.org/c/openstack/python-ironicclient/+/977408 | 02:08 |
|---|---|---|
| opendevreview | Merged openstack/python-ironicclient stable/2025.1: Fix parallel initial version negotiation https://review.opendev.org/c/openstack/python-ironicclient/+/977407 | 02:09 |
| *** hroy_ is now known as hroy | 03:06 | |
| clif | gm ironic o/ | 12:25 |
| TheJulia | good morning folks | 12:51 |
| Sandzwerg[m] | Morning ironic | 13:14 |
| Sandzwerg[m] | What would be the easiest way for me to get a "standalone" ironic to add a node to test if redfish etc works? Devstack? Something else? We got access to new hardware but only via jumphost so adding it to our current cluster is not possible. | 13:16 |
| dtantsur | Sandzwerg[m]: standalone - bifrost? | 13:16 |
| dtantsur | Even as a Metal3 person, I often use bifrost as a very easy way to get Ironic without literally anything else. | 13:17 |
| Sandzwerg[m] | Never used that but I'll take a look and ask dumb questions if I fail, thanks :D | 13:18 |
| dtantsur | let's put it this way: it's as easy as openstack software even gets :D (which is not what normal humans generally call easy, but you're in this business, you know) | 13:19 |
| dtantsur | Sandzwerg[m]: we also have this thing here for the specific case of testing nodes, but somewhat opinionated: https://github.com/openshift-metal3/bmctest | 13:20 |
| dtantsur | maybe it's useful for you | 13:20 |
| Sandzwerg[m] | Sounds both useful. I'll dig some documentation | 13:25 |
| * TheJulia crosses her fingers hoping her long stopped devstack VM stack a new stack without any issues | 15:19 | |
| opendevreview | Jay Faulkner proposed openstack/ironic-specs master: Add 2026.2 Work Items document https://review.opendev.org/c/openstack/ironic-specs/+/986018 | 16:10 |
| dtantsur | TheJulia: hey, rbac question. Is it intended that even project admins cannot read last_error by default? | 16:37 |
| dtantsur | This looks like a usability nightmare | 16:37 |
| JayF | dtantsur: they are similarly blocked from node history then, I presume? | 16:38 |
| TheJulia | depends on the modeling, because if it is an owner project admin, yes, if a lessee project admin, maybe not?! | 16:38 |
| JayF | last_error can contain infrastructural things -- like hostnames -- that you wouldn't want exposed to external users | 16:38 |
| TheJulia | That ^^ | 16:38 |
| dtantsur | Yeah, but how do you use Ironic without it? | 16:39 |
| JayF | In a standalone Ironic + multitenant model, you don't trust project owner w/infra information | 16:39 |
| JayF | dtantsur: support ticket -> system admin "plz2fix" | 16:39 |
| dtantsur | Ugh. I'd never use a system like that. | 16:39 |
| TheJulia | if your using ironic as a lessee, your node is deployed by that point | 16:39 |
| JayF | dtantsur: I don't disagree we should make it perhaps easy/obvious/well documented how to loosen that control | 16:39 |
| dtantsur | TheJulia: no lessees involved | 16:39 |
| dtantsur | Just a normal project admin, the way bifrost+keystone is installed | 16:39 |
| TheJulia | I don't have the rbac context loaded from backup tapes right now, but I think we may have put a separate policy knob around that which could be tuned | 16:40 |
| dtantsur | Yeah, but who would *not* want it tuned? "As an operator, I want to make sure failures are impossible to debug"? :) | 16:40 |
| JayF | dtantsur: node.owner is not, in all use cases, the infra operator | 16:41 |
| JayF | that's the reason the knob has to exist | 16:41 |
| TheJulia | I mean, it all depends on the security posture and honestly the ironic user was always modeled as more the system scoped users with graduated access down to the project scoped admin | 16:41 |
| TheJulia | this goes to the root and response of the original openstack rbac bug | 16:41 |
| TheJulia | and truthfully, all we can do make a reasonable attempt and pivot with feedback when we get it | 16:42 |
| JayF | I'll note I am not a user of this use case currently; but if my downstream ever stopped use of Nova we'd almost certainly move to this model with these expectations :/ | 16:42 |
| TheJulia | then we adjust, its not a big deal | 16:42 |
| dtantsur | I understand the reasoning, but ironic is not usable without access to last_error or service logs | 16:42 |
| TheJulia | dmitry, I hear you but your maybe not understanding my response | 16:43 |
| TheJulia | We can change, that is okay | 16:43 |
| TheJulia | beating up the issue over a shift in use expectations doesn't help | 16:43 |
| JayF | Having an easy knob to turn doesn't even mean it has to default in the way I want it to | 16:43 |
| TheJulia | that is also another possibility, based on the discussion and the embracing of project scoped admins, I'd likely revisit that and make it more towards what dmitry is saying | 16:44 |
| JayF | but I also think it's sad to have the expectation that without the ability to troubleshoot directly, Ironic node ownership is not a useful item. I think that reliability expectation is different per environment | 16:44 |
| * TheJulia entirely shifts gears now | 16:44 | |
| dtantsur | JayF: well, it's useful until something goes wrong, even if something is as trivial as a bad checksum | 16:45 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/common/policy.py#L538-L545 | 16:45 |
| TheJulia | so, either there is a bug someplace, or there is a larger issue | 16:45 |
| TheJulia | what roles does the user have ? | 16:45 |
| JayF | dtantsur: this honestly is kinda interesting in context of the maintenance discussion in the PTG: we *should* have some way to define what errors are infra (5xx style errors) vs usage/image/permissions (4xx/3xx) | 16:46 |
| dtantsur | true | 16:46 |
| JayF | in the same way we have ideas of maintenance which might operate at different levels | 16:46 |
| * dtantsur figures out the roles | 16:46 | |
| TheJulia | hold on | 16:46 |
| TheJulia | Please go back to my question, what roles does the user have | 16:46 |
| dtantsur | Trying, trying | 16:46 |
| TheJulia | because it should be visible to an owner | 16:46 |
| dtantsur | Bloody openstack CLI, I cannot remember so many UUIDs..... | 16:47 |
| JayF | owner/lessee being where that split is by default does make sense | 16:47 |
| JayF | for the higher security case bifurcating into user v system errors is almost required to have sensible error reporting in bad image cases as pointed out by dmitry | 16:47 |
| TheJulia | I'm sort of wondering if dmitry is using an admin roled user which is somehow missing reader | 16:47 |
| TheJulia | but... things would look super weird there because its expected in the model | 16:47 |
| * TheJulia has no idea how that would *look* API surface wise | 16:48 | |
| dtantsur | TheJulia: bingo. Bifrost only creates "admin" roles. | 16:48 |
| dtantsur | I guess the roles are not hierarchical, are they? | 16:48 |
| TheJulia | AFAIK they are supposed to be | 16:48 |
| TheJulia | that sounds like a keystone bug?! | 16:48 |
| TheJulia | But maybe... devstack masks over it | 16:49 |
| TheJulia | or... maybe... uhhhhh | 16:49 |
| * dtantsur dies inside | 16:49 | |
| * TheJulia joins dtantsur | 16:49 | |
| TheJulia | Is it beer'o'clock yet? | 16:49 |
| TheJulia | Anyhow, seriously speaking, in any proper environment, admin is expected to have "member", and "reader" as well | 16:50 |
| TheJulia | it should be heiarchical, but if you have custom handling like maybe what we ended up with bifrost's "keystone" logic, maybe that gets bypassed. Honestly, I don't know, I know in devstack the last time I looked it was doing a pattern from the pre-bake heiarchical changes so maybe that got kept which is horrific | 16:51 |
| dtantsur | So, a bifrost bug? But when can I list my nodes then? :) | 16:51 |
| dtantsur | s/when/why/ | 16:51 |
| dtantsur | (I'm also surprised I can create/delete users without having the system scope.. I don't understand keystone) | 16:52 |
| TheJulia | so, can you verify your roles from keystone itself? | 16:52 |
| dtantsur | If you tell me how.. | 16:52 |
| TheJulia | yeah, its all in that project's scope | 16:52 |
| TheJulia | great question | 16:53 |
| TheJulia | uhhhhhh openstack role list? maybe | 16:53 |
| opendevreview | Victor proposed openstack/ironic master: Truncate node history event to avoid DB column overflow https://review.opendev.org/c/openstack/ironic/+/981399 | 16:53 |
| dtantsur | TheJulia: it's all roles, not just mine | 16:53 |
| TheJulia | openstack role assignment list --user <user-name-or-id> --effective --names | 16:54 |
| TheJulia | that is what google is suggesting and I do believe that is heading in the right direction because you have to ask keystone for the role assignment data | 16:54 |
| dtantsur | Well, I have all roles on my project: admin, manager, member, reader | 16:54 |
| TheJulia | you mean to your user in the project? | 16:55 |
| dtantsur | yep | 16:55 |
| TheJulia | Okay, so your a system scoped admin, manager, member, reader | 16:56 |
| TheJulia | that explains why list gets nodes | 16:56 |
| TheJulia | If you look at the node itself, is the owner set to the project's uuid ? | 16:56 |
| dtantsur | Hmmhmmm, I have a bit too many projects here | 16:57 |
| dtantsur | why do I have both "baremetal" and "admin"? | 16:57 |
| dtantsur | It may be a bifrost stuff-up in the end | 16:58 |
| TheJulia | yeah | 16:58 |
| TheJulia | and pre-baked defaults | 16:58 |
| dtantsur | It still confusing why I, being a super admin that can read anything, cannot read last_error :) | 17:00 |
| dtantsur | TheJulia: wasn't https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/tasks/keystone_setup.yml#L159-L167 supposed to create me a role on the baremetal project? | 17:01 |
| TheJulia | super admin is system scope | 17:01 |
| dtantsur | ah, wait, it's a normal user | 17:01 |
| TheJulia | oh yeah, so, thats why | 17:01 |
| TheJulia | its a mismatched state | 17:01 |
| TheJulia | at least, that is my gut feeling | 17:02 |
| TheJulia | so, bifrost bug | 17:02 |
| dtantsur | Mmm, wait. No. | 17:02 |
| TheJulia | I bet if we change that to grant member, reader, it will behave exactly as desired | 17:02 |
| dtantsur | Do you know if I can understand which user and project I'm using now? Through CLI? | 17:02 |
| TheJulia | I just don't have that context on my mind at the moment | 17:04 |
| TheJulia | there is a way | 17:04 |
| TheJulia | just I don't know it. If you look at the response headers, I think X-Project-ID gets included in the response | 17:04 |
| TheJulia | I.. Think | 17:04 |
| dtantsur | I think we're back to square one, unfortunately. I was looking at a wrong user. I must be bifrost_user, and I have all roles on project baremetal. | 17:04 |
| TheJulia | this is all so blurry to me at this point that I just don't remember how the middleware is doing that | 17:04 |
| dtantsur | My nodes are owned by project baremetal. But I cannot read their last_error. | 17:05 |
| TheJulia | debug log and spit out the oslo_policy logging by editing it and comparing on the backend because its about what the server gets as effective on the backend | 17:06 |
| dtantsur | I seriously wonder how some of our policy.check invokations even work. We don't pass a node there, how is node.owner read? | 17:19 |
| dtantsur | I have a gut feeling that we need to replace many cases of direct policy.check accesses with our check_owner_policy helper | 17:20 |
| dtantsur | I'll file a bug, I cannot spend more time on this, unfortunately | 17:20 |
| JayF | cardoe: https://review.opendev.org/c/openstack/ironic/+/984663/2#message-dd27186859190048de5d6e7b6c76090889384339 this is a trivial thing to fix if you get a sec | 17:23 |
| dtantsur | Ohhhhhhhh, uhhhhh, lol | 17:24 |
| dtantsur | The policy works when "owner" is included in --fields but not without it | 17:24 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: [WIP] Experimental support for ARM64 hosts on Debian https://review.opendev.org/c/openstack/bifrost/+/920057 | 17:24 |
| cardoe | JayF: well we didn't backport the original addition of the API so I don't think we're gonna backport that. | 17:24 |
| dtantsur | https://bugs.launchpad.net/ironic/+bug/2150573 | 17:26 |
| JayF | cardoe: ...huh? Oooooh. this bug only impacts runbooks "v2" style runbooks? | 17:48 |
| JayF | dtantsur: lolsob | 17:49 |
| JayF | that's a hilariously sad bug | 17:49 |
| cardoe | JayF: yes. This was to address your or dtantsur's feedback on the "v2" runbooks but it had already gotten a couple of +2 and we wanted to land another DB change on top of it so I said I'd make changes in a follow on. | 17:50 |
| JayF | cid: https://review.opendev.org/c/openstack/ironic/+/984663/2#message-dd27186859190048de5d6e7b6c76090889384339 you might wanna re-evaluate your vote based on what cardoe says | 17:50 |
| JayF | cardoe: we might not approve it r/n just so it doesn't go from 0 review -> landed in like 5 minutes, but if nobody else does, self-approve tomorrow or poke me about it? | 17:51 |
| cid | :D | 17:51 |
| cid | So true and we missed missed it. | 17:51 |
| JayF | me and you are anchored to when runbooks v1 was implemented | 17:52 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Allow removing volume connectors on some states https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/981859 | 18:29 |
| cardoe | JayF: you got access to openstack-exporter right? | 18:47 |
| cardoe | What about the helm-charts? | 18:47 |
| JayF | openstack-exporter's only active maintainer is Adam MacArthur for my team. For some definition of "Active" (Adam is a PhD student with a plate that is almost always full -- but if you get someone in the community reviewing patches they have a path to maintainership) | 18:58 |
| cardoe | Well I'm interested in the helm-charts which are an adjacent repo. | 20:40 |
| cardoe | keekz from my team is gonna fork it to start fixing issues. | 20:41 |
| JayF | we don't use that and have no work in it | 20:46 |
| JayF | I'm not sure if Adam got repo or org keys but I'd generally feel uncomfy asking him to use them on a repo we don't operate on anyway | 20:46 |
| cardoe | Well I'd be happy to propose merging the helm-chart into the main repo and publishing it with code releases. | 20:51 |
| cardoe | Or we can just fork it and go from there. | 20:51 |
| cardoe | The separate repo used to be a thing when you couldn't use OCI for charts since they used a repo for the chart metadata. | 20:51 |
| *** mdfr8 is now known as mdfr | 21:10 | |
| JayF | yeah again the ideal path for that to take is to get active in that repo so we can get >1 maintainer | 21:18 |
| JayF | I doubt Adam has helm experience, but imbq | 21:18 |
| JayF | **imbw | 21:18 |
| keekz | both repos are in the same org (openstack-exporter) | 21:20 |
| keekz | is this an 'openstack' project? if so, i feel like it should be in the openstack org, so we have clear ownership etc :) | 21:21 |
| JayF | It's not an openstack project. It's a github org (openstack-exporter) with one repo that Adam on my team (openstack-exporter/openstack-exporter) has been trying to improve. | 21:21 |
| JayF | I wouldn't be -1 to a move into openstack; but there is a community there that we are trying to reignite through actually reviewing and landing the code | 21:22 |
| JayF | and from my PoV that's just moving deckchairs around until we actually get the project in a decent state | 21:22 |
| keekz | yeah i'd make prs to the helm charts if they'd actually get reviewed. i was under the impression that entire org/project was dead since other folks have already forked it and improved it | 21:23 |
| JayF | We advocated and got Adam admin access at some level (unsure if repo or org) | 21:23 |
| JayF | but as I indicated earlier; I will not be one to push Adam to use that access in a repo he's not even touching | 21:23 |
| JayF | the best route is to get reviewing in that repo, talk to Adam (sharpz7) there, and get yourself as a maintainer too | 21:24 |
| JayF | we are desparate for more hands right now, Adam is highly part time and I am not gonna smear his work there any thinner | 21:24 |
| JayF | keekz: cardoe: noting this conversation is likely better held in that github org -- please post something there and ping me+adam if needed (I'm jayofdoom over there) | 21:29 |
| cardoe | I was tabbed out. | 21:30 |
| cardoe | keekz: throw a PR out there | 21:30 |
| keekz | i will have kiro raise a pr and we'll what happens :) my change is going to be fairly small, the helm chart assumes it creates/owns the secret, but we need to supply our own secret and have it use that instead, so it should be optional. | 21:33 |
| JayF | To be clear: I do not think Adam is reviewing helm charts nor has the knowledge base to do so. | 21:34 |
| keekz | yeah that's fine, i understand. | 21:34 |
| JayF | we are ... walking a line here | 21:34 |
| JayF | trying not to just trample the project, build some momentum with multiple contributors | 21:34 |
| JayF | without pissing off the old maintainer who retains admin | 21:35 |
| keekz | it's fine, i think we mostly thought it was abandoned | 21:35 |
| JayF | it was, honestly | 21:36 |
| keekz | at least we know at least one repo in that org has activity 👍 | 21:36 |
| JayF | it took us 6 months and a threatened fork to get keys to the repo | 21:36 |
| TheJulia | brraaaains? | 21:38 |
| cardoe | none here | 21:39 |
| cardoe | fix node_periodic() to not completely die on exception | 21:39 |
| cardoe | TheJulia: how's the wife? | 21:39 |
| TheJulia | Doing pretty good, although some swelling and hating the world is naturally occuring | 21:39 |
| TheJulia | I need to go get her more ice cream at some point before dinner time | 21:40 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!