Thursday, 2026-03-26

priteauHello Nova team. I have just hit the `mkfs.vfat: Label can be no longer than 11 characters` issue while live migrating an instance. There is a fix in Nova but backports are waiting for review. Could you please merge? https://review.opendev.org/c/openstack/nova/+/95200308:47
priteauUggla: Would it be possible to merge these backports?10:04
Ugglapriteau, I think so. I'll try to have them merged.10:09
priteauThank you!10:09
gibipriteau: approved the backort now10:16
opendevreviewDoug Szumski proposed openstack/nova master: Stop corrupting ephemeral volumes during cold migration  https://review.opendev.org/c/openstack/nova/+/94090010:36
priteauThanks gibi. I might ping you for the next ones!10:46
gibipriteau: sure10:47
nathharphey knowledgable nova people.   I'm having problems with driving Ironic via nova-compute: when building multiple instances, i see errors like: "status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n resource provider generation conflict  ", "code": "placement.concurrent_update".   Is this likely to be an issue with nova, or placement?   This is a 2025.1 cloud10:58
nicolairuckelhttps://specs.openstack.org/openstack/nova-specs/specs/2025.1/approved/vtpm-live-migration.html Is it correct, that there is nothing to be implemented for the user mode since in that case live migration is not possible?11:23
opendevreviewMerged openstack/nova stable/2025.1: libvirt: Use common naming convention for ephemeral disk labels  https://review.opendev.org/c/openstack/nova/+/95200312:50
priteaugibi: 2025.1 backport has merged, thanks! 2024.2 backport is ready too: https://review.opendev.org/c/openstack/nova/+/95200412:56
*** bauzas3 is now known as bauzas13:07
opendevreviewBalazs Gibizer proposed openstack/nova master: Default native threading for conductor and compute  https://review.opendev.org/c/openstack/nova/+/98223713:29
opendevreviewKamil Sambor proposed openstack/nova master: Replace eventlet with standard threading in novncproxy  https://review.opendev.org/c/openstack/nova/+/97608914:32
opendevreviewMerged openstack/nova stable/2024.2: libvirt: Use common naming convention for ephemeral disk labels  https://review.opendev.org/c/openstack/nova/+/95200415:09
opendevreviewKamil Sambor proposed openstack/nova master: Replace eventlet with standard threading in novncproxy  https://review.opendev.org/c/openstack/nova/+/97608916:06
clifsean-k-mooney: could you give me some context on ProviderTree? does the tree represent a heirarchical relationship between nodes? I also see ProviderTree calls out the ironic driver specifically for having 'large number of roots' 17:11
sean-k-mooneyclif: placement is a forset data stucre where we have a forest of provider that are them selve aranged in a tree stucture with root RPs representign a comptue node and potically nested resouce provider as childer repcesting gpus or pci device on that compute host17:15
sean-k-mooneyclif: in the context of ironic each ironic node is create in placement as a seperate resouce provider with invetory of 1 resouce class fo custom_baremtal_something17:15
sean-k-mooneyso a single nova compute agent with the ironic driver manage a set of trees of depht 117:16
clifah, ok17:16
sean-k-mooneythe uuid of the placment resouce provider is the ironic node uuid which is alsoce the nova compute node uuid17:16
sean-k-mooneyso it a 1:1:1 mapping between compute_node in the nova db, resouce provider in the placment db and node in the ironic db17:17
clifso each tree is a single ironic node and each tree's children is information about the ironic node?17:17
sean-k-mooneyclif: for almost every other dirver each nova comptue agent manages exactly 1 compute node and 1 root resouce provider17:18
sean-k-mooneyclif: for hte ironic diver we dont create childer sinc you cant subdevide ironic hosts17:18
JayF(this is, I presume, the root cause for how so much perf was left on the table in the Ironic case?)17:18
sean-k-mooneywe could in the future but we didnt have a suecase today17:18
sean-k-mooneyJayF: yes more or less17:19
sean-k-mooneyJayF: the first botlenec (the deep copy of the providers manged by this compute agent) 17:20
sean-k-mooneyis cheap if the number of provider is 1-8 max in 99% of cases17:20
sean-k-mooneyits really expensive if you have 100s or 100s espcially if your  doing it in a loop 17:20
clifcould we create a seperate provider implementation for ironic driver use?17:21
sean-k-mooneyJayF: so i think that is 1 of the hopefully simple things to fix17:21
sean-k-mooneyclif: we try not to have any virt driver awareness in teh compute manager or resouce tracker17:21
sean-k-mooneyand that is where the provider stuff lives17:21
sean-k-mooneybut short answer is yes i think we can optimize it 17:22
clifalright17:22
sean-k-mooneyjsut without the need for an if ironic check17:22
sean-k-mooneyclif: that is what https://review.opendev.org/c/openstack/nova/+/980676 is ment to do17:22
sean-k-mooneyspecifically https://review.opendev.org/c/openstack/nova/+/980676/1/nova/compute/resource_tracker.py#131417:23
clifright, I'm trying to load enough nova context into my noggin to get to the point where I can understand what's going on with that patch, the perf sim, etc17:23
sean-k-mooneywhen we get the provider tree for a spcific node that shoudl copy only that node RP insated fo every provider in our provider cash https://review.opendev.org/c/openstack/nova/+/980676/1/nova/scheduler/client/report.py#93517:24
clifit seems reasonable on the surface17:25
sean-k-mooneyclif: ya that a non zeor amount of context to load17:25
sean-k-mooneythe other optimisation that i think has prommis is this patch https://review.opendev.org/c/openstack/nova/+/980679/117:26
sean-k-mooneyso that that one does is very simple17:26
sean-k-mooneytoday we have a for loop over every ironic node managed by this agent (limited via shard key or peer_list)17:26
sean-k-mooneythe patch just kick the processing of each node out into a thread pool17:27
sean-k-mooneyso you can actully do them in parallel an tune it via a config option17:27
sean-k-mooneydefaulting to 1 for backward compatiablity17:27
JayFAnd I hope this is all building up, once the identified low hanging fruit are picked, to a deeper look at if there's things we can do to improve it further. I am skeptical that this will be enough to bring us within an acceptable startup time :)17:29
clifI will continue to delve17:29
clifthanks for the context sean17:29
sean-k-mooneyJayF: these were the hotspots in the current profiling but if we can get a beter repoducer with the ironic fake driver or soemthign elese we can see what the next bottel neck is17:30
JayFironic devstack plugin support for fake driver nodes is now #2 on my todo list (I got the PRs up removing 15k worth of retired driver code lines from Ironic)17:30
sean-k-mooneyJayF: we could as you suggested skip the placement management entirly in nova and delegate that to ironic but i dont know if that will really help17:31
sean-k-mooneywe may need to consider other change liek batch updating placement which may  need an api change17:31
JayFsean-k-mooney: basically in the places we poll Ironic currently, any way to switch that to something more evented17:31
JayFsean-k-mooney: or batching into placement, yep17:31
sean-k-mooneywell yes so we coudl extend the nova external event api for ironic17:31
JayFsean-k-mooney: either way once we pluck the low hanging fruit I want us to take a look at doing the bigger stuff to help; myself and clif can be the hands making the action work17:32
sean-k-mooneythat woudl allow use to get ride of poolign for power state for example17:32
JayFI just don't always know which way is up in Nova :)17:32
sean-k-mooneyJayF: if you are not familary with https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events17:32
sean-k-mooneythat is how cidner and nova tell nova that the sate of soemthign has changed or a workflow has completed in an event based manner17:33
JayFInteresting17:33
sean-k-mooneypower-update might actuly be used by ironic already17:33
sean-k-mooneybut there is no reason that we coudl not add other event if we had a reason too17:34
JayFI am 99.999999% it's not17:34
* JayF is checking17:34
sean-k-mooneyim just tyring to think what ealse is power related that it could be17:34
JayFyep we 100% do17:34
JayFTIL17:35
sean-k-mooneyhttps://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id6917:35
JayFhttps://opendev.org/openstack/ironic/src/branch/master/ironic/common/nova.py#L9117:35
sean-k-mooneyyep so i think this was added so we can turn off the power monitring perodic in ironic deployments17:35
JayFif we could just trigger update of placement relevant info from Ironic e.g. on certain state changes17:35
JayFI don't care if any of that comm goes Ironic<>Nova<>Placement or Ironic<>Placement17:36
sean-k-mooneywell ironic can go do placmeent adn do some limited things today17:36
JayFespecially if we were able to somehow get those calls to impact the node cache inside the virt driver itself17:36
sean-k-mooneyJayF: so if you tell me waht you want to update there i can say if it doable today or not without nova changes17:37
JayFbecause then we can do the "reconciliation job on a timer" change like you did that I already -1'd17:37
JayFsean-k-mooney: I don't know that answer (yet); but whatever happens when a node goes unavailable (e.g. cleaning or move out of AVAILABLE state) and whatever happens when a node goes available (anything->AVAILABLE state)17:37
JayFPlus maybe something that would trigger on a node trait add to expose that trait to placement17:38
sean-k-mooneyso the way to model that in placement today is to set reserved=total on the rp17:38
sean-k-mooneythat is something that ironic can do directly17:38
JayFRight now my goal is to get all these written down so we can evaluate them after the lower hanging fruit have been plucked17:38
JayFbecause hopefully by then we'll have devstack w/fake driver nodes17:39
sean-k-mooneyack sound good17:39
JayFand can have real data for what is helpful vs what just seems helpful at a surface read17:39
* JayF going to go toss it in an etherpad for the shared ptg sesh17:39
sean-k-mooneyJayF: you know if we do deploy with the fake driver with a 1000s inically devstack is definlly going to cail because it will time out waiting for nova-comptue to start :P17:43
sean-k-mooneyso maybe ill try with 10 and then add them once it up17:44
JayFsean-k-mooney: Ironic has a long, long, long history of knowing where all devstack timeouts are so we can set them high enough to make an infra team member weep17:44
JayFlol17:44
JayFsean-k-mooney: maybe add 1000 fake nodes, only put 3 of them in a shard managed by nova-compute by default or something :D either way I'll figure it out17:45
JayFsean-k-mooney: clif: I put some notes in here https://etherpad.opendev.org/p/ironic-nova-compute-startup-2026.217:47
sean-k-mooneyclif: if you want to rebase of rearrage those wip patches feel free17:47
sean-k-mooneyi wont get back to them for a while but if they are helpful to you go for it17:48
clifthank you, will do17:51
melwittgmaan: hey, I have a hopefully easy patch to fix openstacksdk usage for using endpoint discovery for unified limits if you might be able to look at your convenience https://review.opendev.org/c/openstack/nova/+/97510618:51
erlonsean-k-mooney++ melwitt++ thanks for the througly review on the RBD XML update functionality for migration process patch19:04
sean-k-mooneyerlon: im not sur eif keeping the sepreate small funciton for nova provisioned rbd storage is bettere or not for what its worth19:05
sean-k-mooneyi could be preswaded either way19:05
sean-k-mooneybut i woudl really prefer to see a functional repoducer for that19:05
erlonmelwitt: did you tried the XML "<ip family="ipv6" address="[2001:db8:ca2:2::1]" prefix="64"/>" with a ipv6 addr without the []? did it pass the checks?19:07
erlonsean-k-mooney: you mean to have that code inside other function like _update_volume_xml?19:10
erlonI believe that having the code in a separate function is better for readability and testing19:11
erlonAnother thing I was considering, is that when I brought it up in the meeting, we discussed wanting another managed command to perform the bulk update. I didn't start to think about how I'll do that implementation, but having things modularized usually help to reuse code.19:14
sean-k-mooneyerlon: my conder is driffet between the nova and cidner voulem management19:15
sean-k-mooneyerlon: like currently you are just updating the xml but not th bdms right19:16
melwitterlon: yes I did I pasted the output in the code block of my comment, I ran the virt-xml-validate CLI without brackets (passed) and with brackets (failed)19:16
sean-k-mooneyi dont recall if we store rbd info in teh bdms for nova provisioned stroage19:16
sean-k-mooneymelwitt: its possible that libvirt version depenent19:17
melwittno we don't, BDMs with rbd info is for cinder volume only19:17
sean-k-mooneyok then that not an issue with the current patch so19:17
melwittsean-k-mooney: yes true. it would be ideal if we could test it for real somehow but regardless I think it's safest to do it without brackets bc that's how XML is currently generated for rbd19:18
sean-k-mooneywe have a tempest-ipv6 job in our gate 19:18
sean-k-mooneyi just doen not ahve ceph in it19:18
sean-k-mooneybut i think we can look at that in a followup19:18
melwittI posted a comment on the review but I am wondering if the patch is actually needed right? bc AFAICT all cases should be covered already for nova local/ephemeral rbd19:18
melwittI wonder if people just want the patch to force an update to XML right away instead of waiting for a lifecycle event. just not sure I get the use case19:19
sean-k-mooneyso its not if you reboot the vm19:20
sean-k-mooneybut if we try to live migrate with the old mons i think the dest vm will fail to connect to the storage19:20
melwittceph handles all of mon IP update transparently while the guest is running and it's not clear from the bug report if the lack of XML update there results in a failure or error of some kind19:20
sean-k-mooneyand the migration will fail as a result if those mons are note aviabel19:20
melwittright and if you reboot the VM the existing code already grabs fresh mon IPs from ceph19:20
melwittok, that is what I wanted to know. I wish they had pasted the error in the bug19:21
sean-k-mooneyya so unless libvirt is proxiyng that info without telling us i suspect that is the failure mode19:21
melwittok, if it's actually a failure and not just a "I will feel better if the XML is updated now vs later" then I think the patch is useful19:21
erlonmelwitt, what would be the lifecicle events that would update that?19:21
melwitterlon: basically anything that calls _hard_reboot() internally so a stop/start, a hard reboot, an unshelve, I'm not sure what else off the top of my head. the main point is as long as the guest is running, the xml does not need an update, that's for later when it's starting up from being shutdown19:23
erlonafic, ephemeral rbds are not updated unless the instance is sheved/unshelved19:23
sean-k-mooneyit should be updated any tiem teh domain is redefiend19:23
melwitterlon: however, if live migration would in fact _fail_ without the xml updated, then that is the problem that makes the patch needed. it was just not clear to me from the bug report whether this is a real failure or if someone did dumpxml and saw old mon IPs and that made them believe there is a problem19:24
erlonyes, but the use case is to keep the instance running. 19:24
sean-k-mooneyso cold migrate, shleve hard,reboot,evacuate19:24
melwittyup every time it's redefined. for local/ephemeral RBD to be clear. Cinder volume is NOT that way19:24
melwittfor Cinder volume, currently, you have to live migrate to make it consume new mon IPs for its next xml redefinition. and I wonder if that might be the confusion19:25
erlonmelwitt: it does fail since, when RBD is trying to connect to one of the monitors, if the monitor times out, rbd will not retry19:25
melwittack ok19:26
erlonmelwitt: no, the patch clearly handles that difference between Cinder back the volumes and RBD ephemerals and config drivers 19:27
erlon*cinder backed19:27
sean-k-mooneyerlon: sure but its addign supprot for somethign we do not supprot19:28
sean-k-mooneychangign the mon ips for ceph cluster attached to a nova instnace19:28
sean-k-mooneywe do not actully supprot that even if it works19:28
sean-k-mooneywe can make it better but it still operaotr error to ever do that19:28
melwittto be clear, if live migration is failing with local RBD disk when new mon IP happens, then I am personally ok with the patch. and I have a draft comment written up about that if that is the case19:29
sean-k-mooneyim ok with the patch to make thing bettter as long as we are clear this is not a supproted operation that operaors shoudl do freely19:29
melwittI had thought that as long as the guest remains running, that ceph handles the change transparently and we should not need to update any XML but I was not 100% and realized I could be wrong19:29
melwitt*not 100% sure19:30
sean-k-mooneyin the cidner case it causes db cupprtion on our side at least as the connection info will be out of sync 19:30
melwittwell it's weird actually bc for live migration specifically, we _are_ refreshing connection_info. so ironically cinder volume works fine in this case already19:30
sean-k-mooneymelwitt: i bielive that is only true if 1 of the osd remains active at least that was a limitat of krbd in the past19:30
erlonmelwitt, yes, it does fail in that case19:30
melwittsean-k-mooney: I see19:31
sean-k-mooneymelwitt: actully ya your right we get an updated atachment19:31
sean-k-mooneybut only if we live migrate19:31
melwittok, since failure is confirmed, I will post my draft comments that explain why I think the patch is ok19:31
sean-k-mooneyif we jsut hard reboto the vm wont come back up if none of the old mon ips are reachable19:32
erlonIf you hard reboot, it will get the correct addresses from the monitor19:33
melwittsean-k-mooney: right. what we are missing for cinder volume is the domain redefine case. bc we didn't want to be pulling connection_info needlessly for 99.999% cases. but I actually had an idea about that, how we could do a cheaper check by calling ceph and comparing to xml, and only if it's different call cinder for fresh connection_info19:33
sean-k-mooneywwe coudl i jsut dont know if we ever want to really suprot this19:33
melwittsorry not compare to xml but compare to bdm19:34
sean-k-mooneywell either woruld work i guess btu i dont like dependignon the xml19:34
sean-k-mooneyif we can avoid it19:34
melwittyeah, I am not looking at it as much as supporting a thing we said it unsupported but rather making everything consistent19:34
sean-k-mooneymy conder with this in genrel is it opens us up to a buch of bugs relate dto snapshot ro other operatoin with stale info19:34
erlonsean-k-mooney: this is something that is floating around for a long time, and many people has showed interest in having this fixed. The bug is the 2nd in heat/affteced: https://bugs.launchpad.net/nova/+bug/145264119:35
sean-k-mooneywe coudl totally burn all of those down but im not sure what we are opening our selve up to if we try19:35
melwittbecause today, we have a mon IP update for ephemeral RBD hard reboot, we have a mon update for cinder volume RBD live migration, but the vice versa for each are missing. and that to me seems ... not great19:35
gmaanmelwitt: checking19:35
sean-k-mooneyerlon: yep its not a new compleaint19:36
sean-k-mooneyi would question the validiy of the bug since we intentally iddint suprpot this in the past19:36
sean-k-mooneyerlon: what i really wish we coudl do is jsut use fqdns for the mons instead of ips19:37
sean-k-mooneyor multicast/vip ips or simialr 19:37
erlonThat would not come without problems as well, since you would have to rely on DNS 19:38
erlonWe have discussed in this possibility19:38
sean-k-mooneyanyways erlon  if we build out the funcitonal tests to demostrate and validate the fix im ok with improving it as long as we dont docuemnt that htis is somethign operators shoudl expect to work in general19:38
melwittyeah to be clear, for me it is the inconsistent behavior. but, I do also wonder if it could open up any problems we don't expect. I can't think of anything but of course I guess anything is possible. I am not that much of a RBD expert19:38
erlonsean-k-mooney, so with that I would assume that a nova managed command would be out of the table19:39
sean-k-mooneywe do not suprpot krbd but when we had it at a workaroudh chanig the mon ip coudl cuase the host kernel to soft lock19:39
sean-k-mooneysimilar to how iowrites  to iscsi block device that are disconnected would hang for ever19:39
sean-k-mooneyalthoguht that was a long time ago so may that less terribel now19:39
sean-k-mooneyerlon: nova mange wont help here19:40
melwittgmaan: thanks :) also your ack would be appreciated on this backport proposed that fixes a http 500 => http 400 and if you think that is ok for stable https://review.opendev.org/c/openstack/nova/+/98054019:40
sean-k-mooneyerlon: it woudl for cinder19:40
sean-k-mooneybut not for nova provisioned volumes19:40
sean-k-mooneywe dont use nova-mange to update xmls19:40
erlonDid you would be that once a ceph mon list had changed, it will be possible to fix all vms without having to migrate or shelve them19:41
melwittwe don't but we have for cinder volume nova-manage for refreshing bdm connection_info which would in turn update XML19:41
sean-k-mooneyright but we do it via the contoelr nodes by updating the db19:41
erlonbut I believe that would be something we could live without19:41
melwittok I think you just said that19:41
melwittyeah19:41
sean-k-mooneyi dont think we should ever add a nova mange command that talks to libvirt and modifes the runing domain19:42
melwittyeah so for rbd local/ephemeral if you hard rebooted all the VMs then you would get all the new mon IPs today. but I get that a lot of operators don't want to be rebooting all the guests19:42
sean-k-mooneyya so "healing it" on live migratoin is ok19:42
sean-k-mooneyhealign it in place to me voids yoru warrenty even with somthgn liek nova-mnage19:43
sean-k-mooneywe have alwasy said out of band updates to the domain (any write to libvirt not form nova-compute) voids the warrenty on the domain19:43
sean-k-mooneyit would be very easy to create security issues by fat fingering the cli19:44
sean-k-mooneyand it woudl be inharently racy with api actoins too19:44
opendevreviewsean mooney proposed openstack/nova master: [WIP] how borken is ipv6 only ceph.  https://review.opendev.org/c/openstack/nova/+/98230219:53
opendevreviewsean mooney proposed openstack/nova master: [WIP] how borken is ipv6 only ceph.  https://review.opendev.org/c/openstack/nova/+/98230219:54
erlonsean-k-mooney: regarding the direct change/replace all monitors in the function, I understand that comparing that with the old values give us the opportunity to issue a warning, but it should not cause any impact on running instances, since we are just manipulating an in memory XML object, that will be sent to the destination host19:55
sean-k-mooneyright but if that warning ever fires it means that an operator did somethign wrong19:56
sean-k-mooneyso it warning them that thre may be problems that need to be fixed 19:56
gmaanmelwitt: done, backport of 500->400 lgtm too20:04
melwittthank you gmaan 20:04
sean-k-mooneylooking at zuul there is currently no job wiht ipv6 and (ceph or cinder) in the name so im not going to hold my breath on ^ working20:06
sean-k-mooneyit shoudl be doable we woudl jsut need to tweak the devstack plugin to resovle what ever it finds20:07
melwittdunno if this is a known issue but I noticed grenade-skip-level-always (non-voting) is failing on stable/2025.1 with "AttributeError: module 'setuptools.build_meta' has no attribute 'prepare_metadata_for_build_editable'. Did you mean: 'prepare_metadata_for_build_wheel'?"21:32
melwitthttps://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack%2Fnova&branch=stable%2F2025.1&skip=021:32
sean-k-mooneymelwitt: that might be related to jammy and setuptools/pbr versions when upgrading form 2024.121:45
sean-k-mooney2024.2 has not got the pkg_resouces fixes21:45
sean-k-mooneyat least not in the requriemetn repo and has not got the newerever pbr release as a result21:45
sean-k-mooneyhttps://review.opendev.org/c/openstack/requirements/+/976602 needs https://review.opendev.org/c/openstack/requirements/+/976903 to be cherry picked first21:47
sean-k-mooneywell it might not be that expiclty but its related to setup_tools and pbr and i think those mibg help21:49
sean-k-mooney melwitt  https://github.com/openstack/nova/commit/42de440471abf5d1953eacd21f191c1f0ff28ef721:51
sean-k-mooneyso ya its to do with the 2024.1 install21:51
sean-k-mooney2024.1 is not actully supproted for upgrades anymore21:51
sean-k-mooneyso we proably shoudl just turn that job off permently21:52
opendevreviewMerged openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery  https://review.opendev.org/c/openstack/nova/+/97510622:15
opendevreviewMerged openstack/nova stable/2025.2: api: Handle empty imageRef alongside null for local BDM check  https://review.opendev.org/c/openstack/nova/+/98054022:27
opendevreviewMerged openstack/nova stable/2025.1: Add functional reproducer for bug 2125030  https://review.opendev.org/c/openstack/nova/+/97986422:27
opendevreviewMerged openstack/nova stable/2025.1: Move cleanup of vTPM secret from driver to compute  https://review.opendev.org/c/openstack/nova/+/97986522:35
opendevreviewMerged openstack/nova stable/2025.1: Preserve vTPM state between power off and power on  https://review.opendev.org/c/openstack/nova/+/97986622:36
opendevreviewMerged openstack/nova stable/2025.2: Preserve UEFI NVRAM variable store  https://review.opendev.org/c/openstack/nova/+/97876622:55
opendevreviewMerged openstack/nova stable/2025.1: Fix string format specifier  https://review.opendev.org/c/openstack/nova/+/96133722:56

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!