Thursday, 2025-02-20

opendevreviewMerged openstack/sushy-tools master: Openstack - get/set_boot_image support volume boot  https://review.opendev.org/c/openstack/sushy-tools/+/94221200:03
opendevreviewSatoshi Shirosaka proposed openstack/ironic master: Remove multinancy page  https://review.opendev.org/c/openstack/ironic/+/94230600:03
opendevreviewVerification of a change to openstack/ironic-tempest-plugin master failed: Add retries while waiting for SSH on server  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94200900:09
opendevreviewcid proposed openstack/ironic master: Follow-up: Apply Inspection Rules  https://review.opendev.org/c/openstack/ironic/+/94211200:18
opendevreviewcid proposed openstack/ironic master: API/Testing: Inspection rules migration  https://review.opendev.org/c/openstack/ironic/+/93921700:18
fricklerI've noticed some unstable looking failures in the devstack ironic job lately, maybe someone wants to take a look? https://zuul.opendev.org/t/openstack/builds?job_name=ironic-tempest-bios-ipmi-direct-tinyipa&project=openstack/devstack06:12
opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: doc: Rework support matrix for trunk driver  https://review.opendev.org/c/openstack/networking-generic-switch/+/94233807:35
opendevreviewVasyl Saienko proposed openstack/ironic-tempest-plugin master: Validate network data for portgroups  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94067807:36
opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: doc: Rework support matrix for trunk driver  https://review.opendev.org/c/openstack/networking-generic-switch/+/94233807:42
opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: doc: Rework support matrix for trunk driver  https://review.opendev.org/c/openstack/networking-generic-switch/+/94233807:48
vsaienkoTheJulia: thanks for review, I've pushed a new revision https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/940678, I've added latest microversion in requests with portgroups, as if not specify it client uses smallest version available.07:48
opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: doc: Rework support matrix for trunk driver  https://review.opendev.org/c/openstack/networking-generic-switch/+/94233807:51
opendevreviewMichal Nasiadka proposed openstack/networking-generic-switch master: doc: Rework support matrix for trunk driver  https://review.opendev.org/c/openstack/networking-generic-switch/+/94233807:53
opendevreviewVasyl Saienko proposed openstack/ironic-tempest-plugin master: Validate network data for portgroups  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94067807:53
rpittaugood morning ironic! o/08:07
opendevreviewVasyl Saienko proposed openstack/ironic master: Enable atop on jobs  https://review.opendev.org/c/openstack/ironic/+/94234008:14
opendevreviewVasyl Saienko proposed openstack/ironic-tempest-plugin master: Validate network data for portgroups  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94067808:34
rpittaubefore I forget again, don't you forget to register for the next OpenInfra vPTG! :)10:49
rpittau#link http://ptg.openinfra.dev/10:49
rpittauthe topics etherpad is ready to be populated!10:50
rpittau#link https://etherpad.opendev.org/p/ironic-ptg-april-202510:50
opendevreviewVasyl Saienko proposed openstack/ironic-tempest-plugin master: Validate network data for portgroups  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94067811:53
opendevreviewVasyl Saienko proposed openstack/networking-generic-switch master: Adds vlan aware VMs support for Cumulus NVUE and DellOS10  https://review.opendev.org/c/openstack/networking-generic-switch/+/94096612:00
opendevreviewJulia Kreger proposed openstack/ironic master: ci: set a longer nova tempest build interval  https://review.opendev.org/c/openstack/ironic/+/94235513:24
opendevreviewJulia Kreger proposed openstack/ironic master: trivial: follow-up on ftp enable/disable flag  https://review.opendev.org/c/openstack/ironic/+/94235713:37
opendevreviewVerification of a change to openstack/ironic master failed: Fix devstack plugin for centos  https://review.opendev.org/c/openstack/ironic/+/94172013:58
TheJuliacardoe: Could you take a glance at https://review.opendev.org/c/openstack/ironic/+/941343 please13:59
TheJuliaThanks in advance!13:59
cardoeI +2’d it. I went through that flow yesterday when I started writing up some of my flow and how it interacts.14:03
cardoeHis code is correct for L3 routed networks for sure.14:05
cardoeI just can’t phrase it correctly to neutron folks that L2 networks can’t do the right thing.14:06
cardoeI tried to explain this week that I have two VXLAN based networks that are physically separate and I need physical_network. And I keep getting told that it’s an overlay network not a physical one.14:06
TheJuliajust in irc ?14:06
cardoeYes at the weekly meeting.14:07
TheJuliaSo I guess they are unable to wrap their head around the physical modeling14:07
TheJulia... hmmmmm14:07
TheJuliacould it be time to request a higher bandwidth call?14:08
cardoeMaybe. I’m going to finish my write up first and share it with them.14:12
opendevreviewVerification of a change to openstack/ironic master failed: ci: focus ironic-tempest-bios-ipmi-direct-tinyipa  https://review.opendev.org/c/openstack/ironic/+/94220414:33
fricklercardoe: so you want someone (neutron?) to set up bridging between those two vxlans? or between a neutron vxlan and a physical port (aka flat network)?14:33
* TheJulia wonders if there is a slight difference in terminology14:39
opendevreviewcid proposed openstack/ironic master: More reliable TinyIPA build with network retries  https://review.opendev.org/c/openstack/ironic/+/94236814:51
opendevreviewcid proposed openstack/ironic-python-agent-builder master: More reliable TinyIPA build with network retries  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/94236914:51
fricklerpretty sure there is, what would the world be like if terminology was unique and well defined like in maths? ;-)15:02
fricklerwho is the resident expert for redfish virtual-media things? is there a list of vendors or similar that support the USB-for-configdrive option? seems supermicro doesn't, not that that would surprise me15:04
dtantsurfrickler: it's usually someone from our team :) But we don't have such a list. In fact, we have serious doubts that it works on real hardware.15:05
dtantsurI think we found Dell disliking more than one virtual media device at the same time.15:05
fricklerdtantsur: I'm wondering whether one could merge the config-drive data into the boot iso. teaching cloud-init about that might be another step, but if I could access them via some custom tooling that would already be very helpful15:07
dtantsurfrickler: I think we partly do that when Ironic is building the ISO. It's going to be a problem when the configdrive is provided as a blob.15:11
fricklerah, hmm, but that would still be an iso blob, which could be mounted and copied from?15:12
dtantsurfrickler: it will be in realistic cases, but strictly speaking we put no restrictions on what a configdrive is.15:12
dtantsur(it's not an objections, just something we need to consider)15:13
dtantsurBy the way, are we talking about the "ramdisk deploy" / live ISO use case?15:14
dtantsurOr are you looking for having a USB device connected to a normally provisioned instance?15:14
dtantsur(should have asked that as the first thing)15:14
fricklerok, I was under the assumption that it should always mimic what one would see from the metadata service15:14
dtantsurWe don't prevent people from doing crazy things in this area :)15:14
dtantsurAgain, not a blocking concern, just maybe we'll need a way to opt out15:15
fricklerI'm actually talking about ramdisk deploy, for the live iso the image isn't (re)built by ironic afaict?15:15
dtantsurRiiiiight15:15
dtantsurYeah, it sounds like an entirely new feature15:16
dtantsurNow, my concern would be: can we actually reliably rebuild an arbitrary CD image?15:17
fricklerdoes encoding metadata in a specifically named ipmi/redfish user account count as crazy thing? ;-D that's my current workaround15:17
dtantsurIt's not just files, I believe, it also has some sort of a bootloader-ish section?15:17
dtantsurwut?15:17
dtantsurfrickler: sounds scary to me, unless I'm missing something15:18
fricklerwell I essentially just need three bytes or digits. so I create an extra user META123 that isn't actually used and can check for that on the server with ipmitool => deployment proceeds happily15:19
dtantsurOh, that's just as cursed as I thought :D15:20
dtantsurYou can probably (ab)use UEFI variables for that too15:20
dtantsurCoreOS images have a cool feature: they leave a designated place on the image where you can put your first-boot stuff15:21
dtantsurI wish something like this was more common15:21
fricklerfor the bootloader stuff, there's the esp_image thing already, and that mostly works for me. just chasing some bug that seems related to https://bugs.launchpad.net/ironic/+bug/208171915:21
frickleroh, UEFI variables is also a good idea, that might even be more generic15:22
fricklerdtantsur: like just "echo my-metadata | dd skip=well-known-offset of=image-file"? that sounds nice, too15:23
dtantsuryeah, along these lines15:23
dtantsurThis allows us (in OpenShift) to have a service that serves node-specific ISO images on fly15:23
fricklermight be possible to have some dib element that does something similar15:24
* dtantsur nods15:25
fricklerdtantsur: that was pretty helpful, thx. /me goes back to isolating the esp_image/bootloader bug now15:28
dtantsurOn the topic of bugs: https://bugs.launchpad.net/ironic/+bug/2098977 is very annoying, and I'll be grateful if someone with spare cycles could do the code reshuffling15:34
cardoefrickler: no. neutron controls bare metal networks when integrated with Ironic and ML2 mechanisms like networking-generic-switch. I'm trying to work with physical VXLANs.15:51
cardoedtantsur: 100% agree with that bug. I believe what we need to do is have a "sushy.quirks.<vendor>" or some such.15:52
dtantsurI'm going to file another bug like that (HPE not supporting changing boot mode)16:02
dtantsur... by "like that" I mean: could benefit from quirks16:02
opendevreviewMerged openstack/sushy master: explicit oem extn names and fix parallel tests  https://review.opendev.org/c/openstack/sushy/+/94216616:02
cardoedtantsur: so rather than us trying 20 different methods we have a quirks and either when we connect to the device we apply the quirks to the connection or maybe we magic load the quirks16:04
dtantsurIt's a good idea (probably tie it together with the oem rework)16:05
dtantsurTheJulia: the HPE bug I promised you https://bugs.launchpad.net/ironic/+bug/209898316:07
fricklercardoe: in my world there is no such thing as "physical vxlan". from what you've said in the neutron channel last week, for you this means a vxlan with the source address for the encapsulation tied to a specific physical network? else you need to share your definition for it16:18
cardoeYou're right. VXLAN isn't really physical.16:20
cardoeThere's a few nuances here that all tie together.16:23
cardoeSo I have a box. It's physically plugged into 4 different switches.16:23
cardoeTwo switches make up a leaf pair on fabric X. Two switches make up a leaf pair on fabric Y.16:24
cardoeToday Ironic marries up neutron ports to bare metal ports using the physical_network field and then pxe_enabled field.16:26
cardoeSo what I've started doing is creating VLAN segments for my VXLAN network16:30
cardoeWith a physical_network field.16:30
cardoeThat at least let's me land on one of two correct ports. But still doesn't let me fully select (there's a different bug for that)16:31
cardoeBut now I've got a network segment range for VXLAN for my "primary" VXLAN network.16:32
cardoetenants go and create a network and they get a VNI assigned from that pool.16:33
cardoeBut for my OTHER network, I want to create a VNI as a provider. Since neutron doesn't let me have a physical_network for VXLAN, I cannot have two pools.16:33
cardoeWhich the answer might be oh just run two neutrons. But then how do I have Ironic know which Neutron to talk to for port binding?16:36
shermanmtbh "physical network" sounds like the right abstraction to me, all it really defines is: "this set of segmentation ids is logically independent from that set"16:43
shermanmI don't see it as different from the vlan case? You have layer 2 traffic, with protocol specific l2 segment IDs, and you want to specify where those are valid16:43
vsaienkoironic community, I'm checking this failed deployment https://zuul.opendev.org/t/openstack/build/5f838eed9dec42ba83f55713d376c234/logs and it looks like request didn't reach to nova-compute ironic. It stuck somewhere between conductors and superconductor. Are you aware of any opeend bugs? and do we really need to use this deployment topology with cells? When its not a feature of ironic itself 17:00
rpittaubye everyone see you on Monday o/17:01
opendevreviewMerged openstack/ironic master: Fix devstack plugin for centos  https://review.opendev.org/c/openstack/ironic/+/94172017:02
TheJulia... I think it is required17:22
masgharHello ironic o/17:42
TheJuliahello!17:42
masgharWondering if this is still a valid bug: https://bugs.launchpad.net/ironic/+bug/163031917:42
TheJuliaSo, it is valid, but odds are we'll never "fix" it17:43
masgharBecause no matter how much we try API errors can not be perfect? 17:45
masgharOr for lack of time?17:45
JayFI imagine opportunity cost is a big reason17:46
TheJulialack of time + desire to avoid breaking changes + it really is trying to drive us towards perfect when we've been around so long that making substantial changes like that also is not ideal for users17:46
JayFeven if we had the time to do it; we could use that time to do much more valuable stuff instead17:47
TheJuliayeah17:47
masgharMakes sense! Shall I close it, then?17:47
TheJuliaI'd mark it wontfix at this point17:47
masgharAlright, thanks!17:47
TheJuliaThank you!17:48
masghar:)17:48
masgharI think this one, too is not too important to do? https://bugs.launchpad.net/ironic/+bug/159672618:23
opendevreviewVerification of a change to openstack/ironic master failed: Trivial: Enable disabling tftp setup  https://review.opendev.org/c/openstack/ironic/+/94174218:28
JayFlooking19:13
JayFmasghar: that's one of those things that likely would be JFDI if it needed to be done; no need for a bug ticket calling for it19:14
JayF(JFDI = just ... do it)19:14
JayFI think we might need to update https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html#running-unit-tests-locally now that more stuff is integrated into the pep8 job (cc: satoshi) -- I don't think codespell is a target in most places anymore19:18
cardoeSo some stupid questions wrt to me fighting ESXi with skrobul and jamesdenton_ 19:25
cardoeVMware has their own boot loader which I need to run instead of IPA.19:26
cardoeJust trying to look for the best way to do this in a way that could be upstreamable.19:26
JayFanaconda driver is what you wanna use as the example19:26
JayFit boots an enterprise linux installer19:26
JayFonly real anaconda-level integration is a callback in our default ks template19:27
cardoeokay. do we run any tempest tests or anything with it today?19:27
cardoeThere's quite a bit of if boot_mode != "kickstart" (do this stuff) else special case.19:27
cardoeI cannot get the anaconda one to work. It always throws an error that stage2_id is missing.19:28
JayFyep there is :( 19:28
JayFI knoow the anaconda one worked at some point, I thought we had CI for this but I think it bitrotted19:28
cardoeI've added debug logging and the image properties come back and it sees stage2_id19:29
JayFcern uses it for sure cc: kubajj 19:29
JayFmaybe kuba has some insight?19:29
JayF> ironic-standalone-anaconda https://zuul.opendev.org/t/openstack/build/27bf264b40764bfabc2e8968f08369a6 : SUCCESS in 52m 19s19:30
JayFruns in ironic-tempest-plugin19:30
JayFbut note it's standalone, so if the error is in the nova/ironic handoff, that could 100% be a bug19:31
JayFthat could sneak past ci19:31
opendevreviewVerification of a change to openstack/ironic-tempest-plugin master failed: Add retries while waiting for SSH on server  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/94200919:39
cardoehttps://opendev.org/openstack/ironic/src/commit/d3c74c6a4bc6917cc97e834d4d8db26ad9d6e905/ironic/drivers/modules/deploy_utils.py#L623 basically that line barfs that stage2_id is missing19:44
cardoeBut https://opendev.org/openstack/ironic/src/commit/d3c74c6a4bc6917cc97e834d4d8db26ad9d6e905/ironic/drivers/modules/deploy_utils.py#L543 has it19:45
kubajjJayF: sorry, we just managed to deploy it and first users are using it in production, no insight in particular, just that it works (when I tested it)19:46
cardoeJayF: the other funky bit is that VMware needs a lot more files than just stage2, kernel and ramdisk19:51
cardoeWe've got a handful of files with hardcoded things on a glance image like "ks_cfg": <glance uuid> which will write out "ks.cfg" for the node. But then we've got "stage2_id": <glance uuid> which will write out a stage2.$disk_file_extension (where disk_file_extension is another image property)19:52
cardoeI'm wondering if we should have a more generic mechanism? something configured in ironic.conf as to what image properties can be used?19:55
cardoeI was also thinking about virtual media boot. It wouldn't be a hard lift to genisoimage this for Anaconda and VMware.19:56
cardoeIn both of those cases, you'd really want a "base ISO" and then you're dropping some customization in.19:57
cardoeI'll annoyingly ping TheJulia as well to weigh in.19:57
cardoeThe real answer is probably toss this on the PTG agenda.19:57
JayF100% a PTG topic20:02
JayFand when I was porting in some of the anaconda driver20:02
JayFit was very clear that there may be space for a generic "installer" driver20:02
JayFbut we had zero call for it, and other automatic distro install processes (e.g. ubuntu/debian + presseed) are not as mature20:03
JayFso it wasn't worth it at the time20:03
JayFbut I am onboard in theory if we can find a good shape for it20:03
TheJuliawait, what am I chiming in on?20:04
TheJuliaBanana bread is tasty, does that work?!20:04
cardoeOne day some of these gremlins will be solved and they won't be throwing yaks in front of me to shave.20:04
* TheJulia reads20:05
cardoeTheJulia: TLDR maybe virtual media boot will be better if you bring us all banana bread20:05
TheJuliacardoe: so, regarding more generic mechanisms, I'm all for it. I think i wired it so stage2 could be omitted when I revamped the docs, I think I put both examples in20:05
JayFit's cardoe's giant plan to cause Ironic to implode with irony by adding an esxi driver /s 20:06
TheJuliaI feel bad, but at the same time anaconda is a hyper-super-hyper specific opinionated path and I'm the wrong person to be driving that ship because  I'm just a servant leader20:06
TheJuliaso #PatchesWelcome20:06
cardoeI wish I could poke a group at large to drive it as well.20:06
TheJuliaeh, I'm more a fan of small groups of strategic things 20:07
TheJuliapersonally :)20:07
TheJuliabut seriously, that driver (and also the way RH internally uses anaconda is much more of a multi-stage setup with some lead in to do a whole install. That is obviously different from vmware so things do get a bit more weird20:08
TheJuliabut! virtualmedia->banana_bread == happiness()20:08
TheJuliaoh, yeah, the skipping existing stage2 is currently contingent on the image_source ending with a path20:09
TheJuliasounds like in your case it might not, or might be entirely omit-able?20:09
TheJulia(which, I guess in theory is toats valid with linux, but pointless maybe with vmware anaconda?!)20:10
TheJuliacardoe: officialy, I wouldn't really have any objection to a "modify an iso" interface to do injection20:51
opendevreviewSatoshi Shirosaka proposed openstack/ironic-python-agent master: WIP Add ContainerHardwareManager  https://review.opendev.org/c/openstack/ironic-python-agent/+/94171420:51
TheJuliacardoe: the whole base requirement would really just be how to make it repeatable and abstractable too20:51
JayFSo satoshi has been running into an intermittant bug around (seemingly) reboot_requested with his example IPA stuff21:09
JayFImage not found [uuid of correct ipa image] errors on the reboot21:09
JayF*however* I've not been able to reproduce cleanly21:09
cardoeokay I think I feel like I'm taking crazy pills... https://etherpad.opendev.org/p/cardoe-ironic-anaconda-img-props the image properties are nested in another sub key called properties?21:12
cardoeYeah so that's the question... if you set image properties should be in nested key called "properties" with the parent key of "properties" being a bunch of other stuff.21:17
cardoevsaienko: since I know you're here... https://etherpad.opendev.org/p/nova-trunk-details-94122721:23
* JayF wonders if it's possible we're accepting aki/ari and then bail on it during a reboot_requested21:25
JayFseems impossible though21:25
cardoeWhat I don't know is if Ironic is wrong or if Glance is wrong.21:28
JayFI have a hypothesis, perhaps?21:29
JayFmore of either a shot in the dark, or a tendency to wanna blame myself for breakages21:29
JayFhttps://github.com/openstack/nova/commit/93b90d2b6a58df26f6fbf0ff0fe20cf37fcbf8af21:30
JayFI wouldn't expect that to be breaky, but it absolutely moved some things around21:30
cardoeWell no this is on the "get_image_properties()" call inside of Ironic21:31
JayFokay good21:31
cardoeWhich is talking to Glance.21:31
JayFcardoe: oh we are chasing cousin bugs21:31
* JayF is trying to find out why satoshi's devstack's glance occassionally goes "WHAT IMAGE? I DON'T KNOW THAT UUID"21:32
JayFthen literally the next second it works21:32
JayFat least the failure is reproducable now21:32
cardoeIt's an Ironic thing.21:34
cardoehttps://github.com/openstack/ironic/blob/d3c74c6a4bc6917cc97e834d4d8db26ad9d6e905/ironic/common/glance_service/service_utils.py#L2621:35
cardoeIf the "properties" aren't one of those things. It gets moved to a nested "properties" key.21:35
cardoeAll of the tests use a flattened properties key like you get from glance.21:35
cardoeThe "translate_from_glance()" is skipped in a bunch of tests because they mock the caller of that with the raw data that glance returns21:38
JayFhttps://etherpad.opendev.org/p/wtf-ipa-reboot-requested is the gist of where I'm at21:40
JayFwe're poking in the same area but unrelated breakage21:40
cardoeYep. It's the same issue.21:41
cardoeYou wanna join me and jamesdenton_ on Zoom? we've got like 15 before we have to run.21:41
JayFI can but tbh will be digging my own issue21:42
JayFmaybe we luck out and collide21:42
JayFif I accidentally do work for Rackspace past-me is going to whack me in the head though /s :P  21:42
jamesdenton_:D21:43
cardoenah your issue is different.21:43
cardoeconductor cannot see your image21:43
JayFincoming test with this code change https://usercontent.irccloud-cdn.com/file/g8gQVgGL/image.png21:49
* TheJulia returns from lunch22:02
opendevreviewcid proposed openstack/ironic-python-agent-builder master: More reliable TinyIPA builds with network retries  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/94236922:05
opendevreviewJulia Kreger proposed openstack/ironic master: ci: focus ironic-tempest-bios-ipmi-direct-tinyipa  https://review.opendev.org/c/openstack/ironic/+/94220422:16
cardoeTheJulia: you don't wanna do that.23:08
cardoeJayF: managed to confirm that flattening the dual properties into one made the anaconda deploy interface work23:08
TheJuliacardoe: do which item?!?23:08
TheJuliaoh... wow23:09
JayFcardoe: https://bugs.launchpad.net/ironic/+bug/209927623:12
TheJuliagrrrr... now my corgi wants to go for a walk23:15
TheJuliaThanks Jay! ;)23:15
JayFwoof :D 23:19
cardoeWell maybe. It seems like it doesn’t go onto the deployment network? I dunno getting info relayed.23:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!