Tuesday, 2024-09-17

opendevreviewcid proposed openstack/ironic master: [WIP] Add inspection rules  https://review.opendev.org/c/openstack/ironic/+/91830301:21
opendevreviewOpenStack Proposal Bot proposed openstack/ironic-inspector master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/ironic-inspector/+/92770102:18
opendevreviewcid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92895809:29
opendevreviewcid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92895812:34
TheJuliagood morning12:59
guilhermesphello ppl! i was wondering if you can help me with a operational issue with ironic caracal down here.. 13:21
guilhermespso i have a cloud which is a mix of regular qemu kvms and 2 baremetals. The baremetals are all good in available state. We should expect them, once in available state, to tell nova-compute-ironic and then the resource tracker tells placement the node is a resource provider, right? 13:22
TheJuliaGood morning!13:25
TheJuliaSo slightly different: nova-compute services (ideally, only *one* since hash ring use is deprecated and I think gets removed ?next? cycle in favor of shard key usage) with the ironic driver should poll ironic and then update the resource tracker *and* about the node's resource_class so it can be scheduled upon.13:27
TheJuliaIf you don't see the the resource_class populated in placement, I'd start looking at the nova-compute service logs to see what they are seeing, but then again, you might just be missing something silly like the resource_class value on the baremetal13:28
guilhermespmornings Julia! Yeah... ive been trying to get something meaningful in nova logs ( enabled debug as well ) but nothing relevant. I have set up the resource class ( called baremetal and flavor uses CUSTOM_BAREMETAL='1')  and all but.. even me creating the resource class manually in placement still returns no valid hosts, and scheduler logs saying that the resource class doesnt exist. Pretty weird13:31
guilhermespthats the overcloud, undercloud runs ironic too, and it works flawless13:31
TheJuliaguilhermesp: so crazy question, can the account nova runs with *see* the nodes?13:33
TheJuliaundercloud/overcloud? Are you running some sort of caracal based tripleo?13:33
guilhermespnope, its a vexxhost/atmosphere deployment. We have undercloud there which doesnt setup nova cells since we just want to provision baremetals there, and the overcloud we want both kvms and baremetals... 13:35
guilhermespyeah let me check if we have implied roles for reader role, im pretty sure we had13:35
TheJuliaguilhermesp: you have to keep in mind, a project scoped admin in ironic can *only* see nodes it is permitted to see or that it's project created13:36
TheJuliawe have that whole extra owner/lessee model on top13:36
TheJulia(which yes, means one could actually use the same ironic for both realistically if they could back the auth)13:37
TheJuliaHey! PTG idea: Diverse auth backends?!?13:37
TheJuliaThe one ironic to rule them all plan13:37
guilhermespyeah i do see errors in ironic-api of issues for permissions to list nodes. But i was talking to rico lin and he said it was something related to openstack-exporter since the requests were returning 200 and right after that, a permission issue. Something like "rejecting authoriaztion: baremetal:node_list:all is disallowed by policy. It tries with both admin and ironic users 13:40
TheJuliaguilhermesp: the permission model is nested, so permissions to just get a blanket list of everything is reserved for system scoped accounts13:41
TheJuliaIt then falls to a project scoped filter which applies the user's project to the query to match the owner and lessee fields in the db13:42
TheJuliaso your basically always going to see an error on project scoped requests, but they are non-fatal, it is just oslo.policy being loud logging wise in ways which help for other requests which fail13:42
TheJuliabut, that is a substantial hint to what is going on13:42
guilhermespgotcha TheJulia . k let me try digging more on nova-compute logs, i will let you all know the outcome if find the issue here.. it could be something really silly yeah, but i mean... using ironic is pretty simple, just make the node available, set a resource class, set a flavor and thats it :P  13:46
TheJuliaguilhermesp: please share that irc log with rico :)13:46
guilhermespyup i did that 5 minutes ago13:47
TheJuliacool cool13:47
guilhermespbtw, i was talking to iury yesterday13:47
TheJuliaso one thing with nova-compute is it should independently log the number of nodes found13:47
guilhermesphe asked for some stickers in indianapolis :P 13:47
guilhermespi will be there13:47
TheJuliaOkay, I'll actually try to drive down to my home this weekend and see if I can find the stickers13:47
* TheJulia keeps forgetting13:47
TheJuliaThe last few months hav ebeen crazy for me13:48
TheJuliaI'm trying to come up with talk titles for OpenInfra Days NA 2025, and I'm thinking "CVEs, why your upstream team needs to plan for them."13:49
TheJuliaAnd talk about how one ripples into your capacity if you have downstream processes13:49
TheJuliaOpenInfra Days NA 2025 will be colocated with SCALE13:52
opendevreviewDoug Goldstein proposed openstack/networking-baremetal master: avoid attribute error on bad password or config  https://review.opendev.org/c/openstack/networking-baremetal/+/92965014:05
cardoeAs I missed the container convo from yesterday. :/14:07
cardoeI would definitely be interested / involved / invested in container building.14:07
TheJuliacardoe: I missed it to, but I *do* wonder, what level of container building? I know some folks are highly preferring to just use the kolla built containers, or at least its base model and tooling. I know others (who are still asleep and will be for a few hours) might prefer that to change to something $else, but there is that whole ship momentum behind it already14:08
cardoeSo comically OpenStack has 2 official projects to build containers for OpenStack components.14:09
cardoekolla and loci14:09
cardoeI use both... :(14:10
TheJuliayeah, that is slightly problematic14:10
cardoemetal3 uses neither14:11
cardoevexxhost has a fork of loci14:11
TheJuliametal3 is not an openstack project14:11
cardoeSure. I'm just speaking from what I know of operators.14:11
TheJuliaahh, yeah14:11
cardoekolla is fairly opinionated. It's got its own entry point and specific paths that things need to be at. They aren't necessarily Kubernetes friendly cause a lot of assumptions are made about --privileged being passed.14:13
TheJuliaYeah, and I think as folks want to make tighter containers, they might be looking to re-evaluate that as well14:14
cardoeloci on the other hand is used by OSH and is pretty minimalist. But building them involves runes and magical IPs and I think you need to catch a leprechaun every few weeks otherwise it stops building.  14:14
TheJuliahmm, that seems... concerning at the same time14:15
TheJuliafrom the pipeline aspect *and* magical IPs14:15
TheJuliasince IPs are always the most infrastucture operator opinionated piece which has burned my team time after time after time14:16
cardoeSo loci is a tighter container for sure. It's just building a container of the openstack/requirements repo and stuffing all the wheels there. Then runs an nginx at a specific IP on a docker network that's expected to serve up those wheels to the other builds.14:16
TheJuliaI literally had a customer case where they are demanding the ability to just have a duplicate running ipv6 conflict and we're like "nope, no, not going to happen"14:16
TheJuliaso very close to upstream then14:17
TheJuliawhich is a nightmare for packagers14:17
TheJuliasince you have to work that requirements tree heavily if you just use it14:17
cardoeI love OSH and I hate OSH. Because it ignores every projects best practices and tries to learn them on its own.14:17
cardoeSo dealing with packagers demands around changes has it's own overlay system.14:18
cardoeI do think there's an answer for having a minimalist build that matches what packagers would want and could be extended and could be consumed by kolla.14:19
cardoeI think loci is closer to that minimal environment. But would require folks to come together.14:20
cardoeFor us, my rough sketched out plan (except I lost the person I was gonna task with this) was to start with loci. And then layer what kolla wants on top.14:20
opendevreviewDmitry Tantsur proposed openstack/ironic-specs master: Support for hardware that cannot be powered off  https://review.opendev.org/c/openstack/ironic-specs/+/92665414:40
dtantsurJayF: quite a rework ^^14:40
JayFheh you must have raced my review14:48
JayFyeah I found that literally a minute after you updated it14:48
JayFwas afraid I re-reviewed an old version but timing was just perfect14:49
cardoejust to confirm if someone else knows, the dnsmasq backend doesn't send signals correct?14:49
JayFwhat does that question mean?14:52
JayFwe HUP dnsmasq on config update14:52
JayFthat's where the bug lies in it where it sometimes goes kaput14:52
cardoehttps://opendev.org/openstack/ironic/src/commit/f082c24d2a672836dead33eae86e610a7a2afecf/ironic/dhcp/dnsmasq.py#L135 that seems to imply no more SIGHUP?15:14
JayFhmm, I'm surprised by that but I know that's been used+tested so it's just something I didn't know15:16
*** dking is now known as Guest373815:19
*** Guest3738 is now known as dking15:21
dkingI have a cleaning fail because I had a node reboot, and since it had been in power on for a long time, it got a new DHCP lease, and now it fails because agent_url and callback_url do not match. Are there some good solutions to force the cleaning to fail or proceed?15:23
dkingI can't change the DHCP lease at this time without introducing some undesirable risk in production. I cannot simply abort the cleaning because the API refuses to do so in the cleaning state. 15:24
JayFit sounds like you didn't restart cleaning properly then?15:24
JayFwhat's the state it was in, what'd you run?15:24
JayFironic should regenerate those bits at restart of cleaning, so it's strange to hear old values causing a headache15:25
dkingJayF: Correct. The cleaning wasn't able to proceed. The cleaning was hung due to an error (which should be fixed now), but since the server was rebooted in the middle of cleaning, it won't proceed because of the new IP.15:26
JayFso what state is/was the node in?15:26
dkingMy main goal right now is to abort the cleaning process and start over, but the timeout seems to be very long.15:26
JayFbtw at :30 I turn into a pumpkin15:26
JayFwhat state is it in right now?15:27
JayFprovision_state to be explicit15:27
dkingBefore cleaning? I'm not sure. Either manageable or available, probably the later. It is now in "cleaning".15:27
dkingNot clean wait, but "cleaning".15:28
JayFokay, if/when it gets to cleanwait you should be able to call set-provision-state abort15:28
JayFmaybe even try it now but it's unlikely15:28
JayFonly other way to knock it loose outside of the DB would be to restart the conductor handling it15:28
JayF(or, as you observed, wait for timeout)15:28
dkingYeah, abort won't work now. I can't even set it to maintenance. And if it made it any further, I'd probably be happy.15:28
JayFmaintenance would do the opposite you want to 15:29
JayFit'd keep it in cleaning indef, because we'd ignore heartbeats iirc15:29
JayFI gotta go record a podcast, sorry, won't be back for like 90 o/15:29
dkingWell, sometimes, I've seen it fail after coming out of mainteance. 15:29
dking...but fortunately, I think that it just cycled. Maybe I scared it by coming into chat. :)15:30
* TheJulia suspects "work from outside" is up next for today15:50
* TheJulia is only semi paying attention to IRC at the moment, too many asks/pings15:53
opendevreviewDmitry Tantsur proposed openstack/bifrost master: WIP: start working on Ubuntu 24.04 support  https://review.opendev.org/c/openstack/bifrost/+/92889515:55
JayFthose are always fun :) I got to learn F# is apparently a whitespace based language too16:25
* TheJulia twitches16:29
TheJuliaGIIITTTT16:50
* JayF playing a game of "SCM" or "country rage" 16:52
* JayF 's dog understands "git" as a command now lol16:53
JayF(for those unaware, it's like "get out" except you just say "GIT"; and no, you don't add an "er done" at the end)16:53
TheJuliaSCM16:55
TheJuliatoats scm16:55
TheJuliait ate a commit somehow16:55
TheJuliaand I found it in the prior commit16:55
jssfrgit reflog to the rescue?16:55
TheJuliawhich I found squahsed in on a prior commit16:55
TheJuliaI figured it out and am fixing it, but... blaaarg16:56
TheJuliaJoin us tonight for our news at 11 story: git rebase gone wrong16:56
TheJulia"When rebases go sideways"16:58
JayFgit add -p can be helpful for that situation16:59
JayFI assume you know but if not :)17:00
JayFaccidentally git commit --amend when I don't mean to amend is a common failure for me lol17:00
jssfroh yeah, especially during rebases17:00
jssfrwhen I marked a commit for an edit and I don't notice I get thrown into conflict resolution before that, bonus if rerere resolves the conflict17:01
TheJuliawhat was weird was I was in a rebase conflict resolution before it happened and suddenly git was like "no, your not" and I was just flabbergasted17:02
TheJuliaI must have though I was on another folder and amended a commit by accident17:02
TheJuliaand yeah17:02
cardoehttps://docs.openstack.org/releasenotes/ironic/unreleased.html looking at the release notes... should we be consistent in [section]/field as the syntax?17:33
JayFcardoe: I would +1 a patch that did that; although I'll note outside of release notes we can reference directly to the config file17:42
TheJuliaconsistency is key, at the same time historical style has varried a lot to [section]key18:13
TheJuliasince / is not part of the key name it is just a human formed delimiter anyhow18:13
TheJulianot to be confused with human form replicators18:14
opendevreviewDoug Goldstein proposed openstack/sushy master: add pyproject.toml to support pip 23.1  https://review.opendev.org/c/openstack/sushy/+/92969318:14
cardoeTheJulia: yeah I'm more just curious what the syntax should be and maybe making a check for that.18:50
cardoeTheJulia: should we also rebase https://review.opendev.org/c/openstack/ironic/+/927518 ?18:50
cardoeI didn't know if you were happy with it.18:50
TheJuliarebase at will!18:53
TheJuliaRebase parties encouraged18:55
TheJuliaThey are like Lightswitch raves18:55
* TheJulia might have dated herself with that reference19:12
JayFhttps://www.youtube.com/watch?v=GtQpThwWQtQ19:18
JayFwait, that cuts off right before the line :( 19:19
JayFoh well, the idea is still there, and the song is the best part anyway19:19
JayFTheJulia: you wanna feel dated? The first time I watched this was on the strongbad email video podcast on my **ZUNE 30G**19:19
TheJuliaYour zune ran flash?19:24
TheJuliaI'm impressed19:25
cardoeTheJulia: I just don't want to be grounded...19:25
TheJuliait suddenly occurred to me that strongbad was clearly a software engineer19:25
TheJuliaEarth Grounds are the best kind of grounds. Warm and Fuzzy without the electrons flowing.19:26
TheJulia#electricaljoke19:26
cardoeI will tell that one to my electrician next time I see him.19:27
JayFTheJulia: they did video versions of them forthe video cast19:27
cardoeJayF: https://review.opendev.org/c/openstack/sushy/+/929693 I'm cheating and doing the easier one.19:27
JayFTheJulia: they'd even put a little clip at the end of the stuff you'd get in the flash from clicking the random stuff19:27
JayFnot sure if that was first run or some republish anyway, I was sorta obsessed at that point in time with finding video podcasts because of how cool it was to just have a bunch of funny videos in my pocket (who would've thought how far we'd come)19:28
TheJuliaJayF: nice19:28
cardoeSo now stupid gerrit question... how do I make a branch that has another commit that depends on that change and the two commits are local without gerrit complaining at me to only submit one commit and squash them?19:28
cardoeYou know... like a real branch with multiple commits.19:28
TheJuliayou do it locally and stack them19:29
TheJuliajust like you realistically would with gerrit19:29
clarkbfwiw gerrit shouldn't complain to only submit one and squash anything19:29
TheJuliawhen you then next run "git review", it will reconcile the difference and restack upstream patches19:29
TheJuliaor you can do things like squash/fixup where appropriate19:30
cardoeI swear last time I tried it complained that I can only submit 1 commit in a review.19:30
clarkbgit review warns you when you are pushing more than one commit because people have historically accidentally pushed up hundreds of changes. So git review is saying "heres a list of commits you are about to push make sure you intend to do so"19:30
cardoeI'm LKML koolaid wrt to commits. Each one should stand on its own and the series tells a story.19:30
clarkbbut it just wants you to confirm that you intend on pushing more than one to avoid accidents.19:31
clarkbya thats exactly how gerrit operates19:31
JayFthat's mostly fair, but when doing a large feature it can be nice to stack 'em19:31
clarkbyou're just using git push to an ssh/http server instead of git am email stuff19:31
JayFcardoe: we kinda are similar, fwiw, because we have every commit has to pass CI19:32
JayFcardoe: so while you might merge an interface that's unused, we should *never* have the HEAD of master nonworking19:32
* JayF points at the word should for emphasis19:33
cardoeI agree with that.19:34
JayFof course, master was broken w/r/t qemu security for 10 years, so ymmv :D 19:38
JayFwe do the best we can tho19:38
TheJuliacardoe: There just needs to be change-id values and then it can submit them all19:59
TheJuliaand then magic happens19:59
TheJuliamagic()19:59
opendevreviewDoug Goldstein proposed openstack/sushy master: switch to pre-commit for style checks  https://review.opendev.org/c/openstack/sushy/+/92970420:07
cardoeWell I'll be.20:08
cardoeAnyway, hopefully okay changes. Those errors that I had to clean up in sushy happen for me locally always so not really sure how that passes in Zuul today.20:09
cardoeThe big changes are after that. Like just bringing in ironic's pre-commit results in 110 files needing a change.20:12
JayF👀20:13
TheJuliabrraaains20:57
* TheJulia doubles JayF's surprise20:58
cardoeJayF: I'll sling it but you might cringe. I'll fix get these 2 landed.21:11
JayFcontext?21:11
opendevreviewcid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92895821:12
cardoeJayF: the sushy pre-commit / flake8 / etc.21:27
JayFah, yeah, it seems fine to me21:27
JayFwell, it's failing CI for what appears to be config reasons21:38
JayFbut the general shape is okay21:38
opendevreviewDoug Goldstein proposed openstack/sushy master: switch to pre-commit for style checks  https://review.opendev.org/c/openstack/sushy/+/92970422:08
opendevreviewcid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92895823:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!