Friday, 2024-08-30

ContinuityMorning nova has a max build setting, does ironic have a similar setting to set how many machines ironic can attempt to build at the same time07:59
dtantsurContinuity: I believe so, let me check11:28
ContinuityThanks, having done some reading it looks like it relies on the max_concurrent_build of nova, but per conductor, so a max_concurrent_build in nova of 2 would be 6 if you have three conductors11:29
ContinuityUnless im misreading11:29
dtantsurhttps://docs.openstack.org/ironic/latest/configuration/config.html#conductor.max_concurrent_deploy11:29
TheJulia... Funny thing, we've actually had someone hit that ;)13:48
cardoemorning ironic14:36
TheJuliagood morning14:46
* TheJulia cries due to 4k14:46
cardoeI missed your patch on that so I'm out of the loop.14:48
TheJuliatl;dr we're starting to see hardware which has raid controllers which apparently only support 4k virtual sector size14:49
TheJuliawith LBA, and that renders GPT and all partitioning incompatible with 512 byte based layouts14:49
cardoeLovely14:53
opendevreviewDoug Goldstein proposed openstack/sushy master: add a link to the release notes in the README  https://review.opendev.org/c/openstack/sushy/+/92759814:55
cardoeCould we make a release of sushy-oem-idrac?14:57
TheJuliaHmmm... I wonder if anyone is available to cut that release right now15:01
cardoeLike no maintainer or you mean today? Cause there's no rush from me. There's just not been a release in 2 years and there's 2 fixes in there that I use and install from git rather than a release.15:04
TheJuliacardoe: today, specifically15:24
TheJuliaI think Riccardo or Iury would be the ones who need to do so15:25
JayFI think I have access to do so as well, but I haven't worked a lot on that project so I would definitely prefer someone more familiar15:29
JayFOr idk, I might have lost that access when I stopped being ptl? Because I don't think I'm technically a release liaison...15:29
TheJuliayeah, you have to be a release liaison15:29
dtantsurJayF, TheJulia, I think Riccardo gave ~ everyone access to releasing sushy-oem-idrac16:18
dtantsurit's pity that gerrit UI is no longer useful to check groups and access rights16:19
dtantsurah, wait, it's just hidden in another tab16:19
dtantsurhttps://review.opendev.org/admin/groups/51dc1df6f2ef25fe66385fca6f72c8feac3559dd,members16:19
dtantsurincludes ironic-core16:20
* dtantsur sighs and deletes Ilya16:20
dtantsurI can do the release but please remind me on Monday: I don't want to do releasing after 6pm on Friday16:21
dtantsurcardoe: ^^16:23
cardoeWill do. Thank you dtantsur 16:23
cardoeSo for an addition of the Ports interface to sushy, should I make a blueprint to discuss the addition? I’m just not sure if it needs the interop profile to change to say it’s an endpoint that is hit.17:49
JayFIronic doesn't use blueprints; our typical process is file a bug in launchpad, tag it `rfe` and put a description in there17:50
JayFif the description is detailed enough, you can often get it approved without a spec by asking in the next ironic meeting; if it gets tagged `rfe-approved` it's good to go17:51
JayFalternatively, if it's not detailed enough or the feature is complex enough to want more detail, it'll be tagged `needs-spec` and you'll have to put a spec in ironic-specs17:51
JayFgenerally speaking we are OK with detailed RFE bugs for things that aren't like, revolutionary new features or things that need lots of planning/coordination17:51
opendevreviewJay Faulkner proposed openstack/ironic master: add pyproject.toml to support pip 23.1  https://review.opendev.org/c/openstack/ironic/+/92754417:59
opendevreviewJay Faulkner proposed openstack/ironic master: Enable WSGI module entrypoint for Ironic  https://review.opendev.org/c/openstack/ironic/+/92763517:59
cardoeJayF: not sure if it requires even a spec or rfe... basically I wanna bring in https://www.dmtf.org/sites/default/files/standards/documents/DSP0268_2024.2.html#port-113018:19
JayFI'd 18:20
JayF**I'd just file the bug, it's trivial to file an rfe bug, and if you think it doesn't need much detail that's your call :D 18:20
cardoeSure. My questions are really what's the version of Redfish we're tracking so I use the correct version. Does it require a change to the interop profile?18:21
cardoeBut I'll make the bug.18:21
JayF Those are *exactly* the kind of questions that generally come up when discussing an RFE18:31
JayFit's less for us about wanting the paperwork, and more about wanting the conversation and to get that conversation documented18:31
JayFI have taken the ironic-specs template and used it downstream because the headers worked really well as a "checklist" to ensure you've thought about all he high level impacts of a change.18:32
cardoeYeah like I've wanted the sushy library to say which version of the redfish spec it tracks.18:33
JayFI think the reality of it is that it supports as many as we see? lol18:33
cardoeWell no. Each source file maps to a specific version of each model.18:34
* JayF is showing his hand-waving level knowledge of sushy18:34
cardoeSo like above my link was to Redfish Model Spec 2024.2 which includes Port v1.13.0 with backwards compat to Port v1.3.018:34
cardoeFor example sushy uses EthernetInterface v1.4.0 per the comment in that source file... # https://redfish.dmtf.org/schemas/EthernetInterface.v1_4_0.json18:35
cardoeBut Redfish 2024.2 has EthernetInterface v1.12.2 https://www.dmtf.org/sites/default/files/standards/documents/DSP0268_2024.2.html#ethernetinterface-112218:37
cardoeEthernetInterface v1.4.0 is from Redfish 2017.218:37
cardoeEthernetInterface v1.7.0+ uses the Port interface. And https://bugs.launchpad.net/sushy/+bug/2075979 is a non-bug by using EthernetInterface v1.10.0 and newer.18:39
cardoeYou see how you just proved that I need to make a RFE bug....18:41
JayFyeah, like I said, I think it's as much about the process of thinking it through moreso than creating paperwork18:43
cardoeLike I mentioned yesterday, we have our own Redfish library for dealing with our gear. It's got a bunch of tests and it hits hardware.18:44
cardoeI'm refactoring it to be a shim over sushy (and sushy-oem-idrac)18:45
cardoeAnd looking to add all the support into sushy proper and upstream it.18:46
opendevreviewJay Faulkner proposed openstack/ironic master: Support Automatic Lessee from instance metadata  https://review.opendev.org/c/openstack/ironic/+/92488718:46
cardoeI commented on your other two ironic's JayF 18:51
* cid nips out for the week. o/19:03
JayFalready replied to one of the comments19:04
JayFreally until I get an "ironic-api can run in wsgi" flag in devstack, I'm nervous about merging that19:04
TheJulia.... I thought it was by default19:15
JayFI know in devstack, restart ir-api does a thing19:18
JayFso I assumed we ran the ironic-api process 19:18
JayFope, you're 100% right19:18
JayFtyvm for that pointer19:19
JayFExecStart = /usr/bin/uwsgi --procname-prefix ironic-api --ini /etc/ironic/ironic-uwsgi.ini --venv /opt/stack/data/venv19:19
JayFaha, makes sense19:19
JayFthat means I should be able to completely close the loop on testing in those patches; score19:19
opendevreviewJay Faulkner proposed openstack/ironic master: Enable WSGI module entrypoint for Ironic  https://review.opendev.org/c/openstack/ironic/+/92763519:28
opendevreviewJay Faulkner proposed openstack/ironic master: add pyproject.toml to support pip 23.1  https://review.opendev.org/c/openstack/ironic/+/92754419:28
JayFthat should change our configuration to use the new module I added19:28
JayFI'm going to see if it passes CI, but the more I think about it, the more I think the path should be ironic.api.wsgi:application, since we're launching an ironic api19:28
JayFbut I'm mainly curious if it works, moving it around if it does is trivial19:28
opendevreviewJay Faulkner proposed openstack/ironic master: [doc] Add instructions on making big fake-BM nodes  https://review.opendev.org/c/openstack/ironic/+/92764519:39
opendevreviewJay Faulkner proposed openstack/ironic master: [doc] Add instructions on making big fake-BM nodes  https://review.opendev.org/c/openstack/ironic/+/92764519:49
JayFadamcarthur5: ^ would that have been useful for the issue we troubleshooted today19:49
adamcarthur5Hmm... I think so? But it would be something that I would expect as a new-user I wouldn't need to mess with. 19:50
adamcarthur5So I doubt I would look at that documentation directly? 19:50
JayFYou should fix that habit at the end :D at least look for the doc, we have a lot of troubleshooting stuff in our docs20:00
JayFand the protip is to use your preferred git search (I use ripgrep) to find stuff in the docs20:01
JayFespecially the contributor docs though, we've done a pretty good job of keeping those updates20:01
JayF*updated20:01
TheJuliaSo, regarding my recent 4k block device size pain, I've posted a change to at least begin to open a path. https://review.opendev.org/c/openstack/diskimage-builder/+/927550  I'm really wondering if we should add smarts/capabilities into ironic to enable forcing/asserting20:44
TheJuliafor devices, and maybe massaging out, dunno20:44
TheJuliamassaging out the stream of data/disk image on disk20:44
TheJuliaThat may not be realistic20:44
TheJuliaI *guess* the other thing is how in the world would an operator discover their hardware *requires* 4k blocks ?!?20:44
JayFI know this isn't a good answer for users, but like, geeze we bend over backwards to keep backwards compat20:47
JayFit's really annoying when we get burned by some other technology not being :(20:48
TheJuliaTruthfully, it boils down to "GPT starts at the second logical block"20:51
TheJuliaand the byte size just being different20:51
JayFI'm just surprised that the tooling doesn't handle that20:54
JayFhmmm20:54
TheJuliahow can it?20:54
JayFI guess whole disk images make that hilariously hard20:54
JayFTheJulia: I was in "just run parted [blah]" mindset, lol20:54
TheJuliaheh, yeah20:54
TheJuliathose tools get it based upon the underlying block device20:54
JayFis there a way to make a one-size-fits-all whole disk image these days?20:55
TheJulianope, there is not20:55
JayFeven if it means giving up a few k at the front of the disk20:55
JayFeven UEFI only? even if we toss BIOS out?20:55
TheJuliaI talked to one of the large operators and their answer is they just force everything to 512 bytes in their envioronment20:55
JayFthis is absolutely miserable20:55
TheJulianope, even with uefi20:55
TheJuliaif you have a 4k block disk, firmware doesn't recognize the the partition table because it is looking at the wrong bytes on disk20:56
TheJuliaWe think of it as a software problem, why can't it just look at the 512 byte area, but that is not how the firmware is designed20:57
TheJulianor the standard20:57
TheJuliaTruthfully, bios is actually kind of better for this20:57
JayFwell that's where my head was20:57
TheJuliabut maybe not20:57
JayFhybrid-MBR style uefi20:57
TheJuliaI don't remember the pointer location formatting style for bios booting20:57
JayFI think that was a microsoftism though? was that ever standardized?20:57
TheJuliaif it is a block or bytes into disk20:58
TheJuliaI think it has to be aligned, so it is likely block address which means it wouldn't work either20:58
JayFah, that makes sense20:58
TheJuliawe have the ability to build those sorts of hybrid uefi/bios images with dib, fwiw20:59
JayFso if you translated it (not saying this exists in the real world anywherre)20:59
JayFit implies we'd end up with a 4k MBR instead of a 512byte mbr20:59
TheJuliayup20:59
TheJuliano idea if 4k disks are even supportable with MBR style booting either, it might just work if the target is properly aligned21:00
TheJuliawhich it *should* be since I beleive O_DIRECT is used for that write21:00
JayFyeah that's what I'm saying, just mentally extending the model :)21:00
TheJuliaso, fun thought, mathmatically, GPT is too big to have a 512 byte and a 4096 byte record structure on disk21:01
TheJuliaor maybe antifun "oh noes!"21:01
TheJuliahttps://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_entries_(LBA_2%E2%80%9333)21:02
TheJulia2-33 *on* 512 bytes, its like 2-5 with 4k blocks21:03
JayFso basically some hardware vendor invented their own crap that's not even standardized21:03
JayFand we've been handed the pieces21:03
TheJuliaeh, not really, just nobody cared about cross-compatability21:04
* JayF cares :)21:04
JayFnobody at $vendor cares :D 21:04
TheJuliawell, nobody cared in whatever working group cooked up 4k disks21:05
JayFif we find out whatever group it is, we should leave a bag of flaming hdd on their porch /s21:05
TheJuliamagnesium case HDDs right?21:05
TheJuliaThere would likely need to also *not* be much alunimun if there is magnesium21:08
JayFI think the devil there is in the idiom or something like that :D 21:10
TheJuliaheh21:11
TheJuliaoh, covering HDDs in rainbows?!21:11
TheJulialots of rainbows!21:13
JayFYeah, but you can't find the partition table at the end of the rainbow21:16
TheJuliaTrue, sadly.21:17
* TheJulia wonders about the hdparm --set-sector-size option....21:20
JayFblock size only really impacts spinny disks, yah?21:23
TheJuliaoutstanding question though, how does one discover they need to use a different disk image...21:23
* TheJulia suspects maybe we need to evaluate root_device and record it as a trait if it is 4k21:23
JayFThis is a place where my knowledge of old school hardware sometimes pollutes my knowledge about current one :)21:23
JayFwell my thought was more, if we can find a universal way to flip the setting21:24
JayFdefault to flipping it21:24
TheJulianah, flash/nvme also impacted21:24
JayFand if someone disables it, they can have the pieces21:24
JayFack21:24
TheJuliabecause it is fundimentanlly the smallest chunk which can be read or written21:24
JayFso would be interesting to see if hdparm/nvme/etc gives us a route to disable (and if in the real world it works)21:24
TheJuliaNobodyCam mentioned he has something in his downstream hardware manager, I wonder if it is hdparm or not21:25
TheJuliaI'll try to ask him later, I think I'm going to call it a day soon and go to the post office21:27
JayFsame, I want to leave some hours so I can see if my nova patch gets a fresh review from non-USians on Monday21:27
JayFin fact... o/21:27
TheJuliahave a great weekend!21:27
JayFhave a good long weekend!21:27
TheJulia:)21:28
cardoeSo does that mean it can be bad idea o’clock? OCI format for images. With ability to package one up for multiple platforms and multiple block sizes. You download only the correct one.23:53
* TheJulia glances at whiskey23:54
TheJuliaThat actually might not be a bad idea, tbh23:54

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!