Monday, 2026-01-12

*** mhen_ is now known as mhen02:57
opendevreviewTakashi Kajinami proposed openstack/nova master: Use ServiceCatalog class for identity v3 API  https://review.opendev.org/c/openstack/nova/+/97303805:43
opendevreviewTakashi Kajinami proposed openstack/nova master: Use ServiceCatalog class for identity v3 API  https://review.opendev.org/c/openstack/nova/+/97303805:45
opendevreviewTakashi Kajinami proposed openstack/nova master: Use ServiceCatalog class for identity v3 API  https://review.opendev.org/c/openstack/nova/+/97303805:57
opendevreviewFabian Wiesel proposed openstack/nova master: WIP: Re-create volume backed root before destroy  https://review.opendev.org/c/openstack/nova/+/94109509:23
opendevreviewFabian Wiesel proposed openstack/nova master: WIP: Re-create volume backed root before destroy  https://review.opendev.org/c/openstack/nova/+/94109510:46
opendevreviewFabian Wiesel proposed openstack/nova master: WIP: Re-create volume backed root before destroy  https://review.opendev.org/c/openstack/nova/+/94109510:51
opendevreviewFabian Wiesel proposed openstack/nova master: WIP: Re-create volume backed root before destroy  https://review.opendev.org/c/openstack/nova/+/94109510:52
opendevreviewTakashi Kajinami proposed openstack/nova master: Use ServiceCatalog class for identity v3 API  https://review.opendev.org/c/openstack/nova/+/97303812:30
opendevreviewMasanori Kuroha proposed openstack/nova master: Copy applied provider config  https://review.opendev.org/c/openstack/nova/+/94830414:03
opendevreviewMax Harmathy proposed openstack/nova master: Fix crash due to access to uninitialized variable  https://review.opendev.org/c/openstack/nova/+/97310314:19
gibiUggla: I will be pretty absent during our today's meeting. Nothing to report from eventlet as I was on PTO and haven't loaded context yet.15:44
gibiI will read back later. Sorry.15:44
Ugglagibi no worries thanks for letting me know.15:53
dansmithso... meeting or no?16:03
Ugglayep16:04
Uggla#startmeeting nova16:05
opendevmeetMeeting started Mon Jan 12 16:05:31 2026 UTC and is due to finish in 60 minutes.  The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot.16:05
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:05
opendevmeetThe meeting name has been set to 'nova'16:05
UgglaHello everyone16:05
dansmitho/16:05
fwieselo/16:05
UgglaSorry I'm a little late today.16:05
sean-k-mooneyo/16:05
gmaano/16:05
tkajinamo/16:06
nicolairuckelo/16:06
elodilleso/16:06
bauzaso/ (late due to other priorities)16:07
UgglaLet's start16:08
Uggla#topic Bugs (stuck/critical) 16:08
Uggla#info No Critical bug16:08
Uggla#topic Gate status16:08
Uggla#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:09
Uggla#link https://etherpad.opendev.org/p/nova-ci-failures-minimal16:09
Uggla#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status16:09
Uggla#info Please look at the gate failures and file a bug report with the gate-failure tag.16:09
Uggla         #info Please try to provide a meaningful comment when you recheck16:09
UgglaI have no seen something special about the gates.16:10
UgglaSomething you'd like to mention ?16:10
sean-k-mooneyi saw sopme tox timesout last week16:11
sean-k-mooneynot sure if they are still happeing 16:11
sean-k-mooneyso maybe just keep an eye out. i think in some case the 3.12 jobs were incorrectly compiling python with pyvenv16:12
sean-k-mooneyalthoguh i have not really looked into that much since it did not seam to be a blanket failure16:12
gmaanI think I saw in tox cover job last week but rest other were ok16:13
sean-k-mooneyya so lets just keep an eye on it16:13
tkajinamhttps://review.opendev.org/c/openstack/openstack-zuul-jobs/+/97097316:13
sean-k-mooneyoh it got fixed16:14
tkajinamI think that py3.12 compiled during jobs is resolved by ^^^16:14
tkajinamyeah16:14
tkajinamthere is also another which migrates py313 to Debian Trixie to avoid pyenv. just fyi. https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/97003616:14
tkajinam(this is still pending on reviews)16:14
sean-k-mooneyah cool16:15
sean-k-mooneyit normally doesnt take that long but it was adding 8 mins to the job that timed out16:15
sean-k-mooneyso on a slow node it was wastign time 16:16
sean-k-mooneywe can probaly move on16:16
Ugglayep thx for the info16:17
jayaanandhi16:17
Uggla#topic Release Planning 16:17
Uggla #link https://releases.openstack.org/gazpacho/schedule.html16:17
Uggla#info Nova deadlines are set in the above schedule16:17
Uggla#info 7 weeks before feature freeze16:17
Uggla#info PTG etherpad for 2026.1 is available: https://etherpad.opendev.org/p/nova-2026.1-ptg16:17
Uggla#topic Review priorities 16:19
Uggla#link https://etherpad.opendev.org/p/nova-2026.1-status16:19
Uggla^ still not updated sorry for that.16:19
jayaanandI am from NetApp. We are planning to implement FlexGroup Backend for Ceph-like Ephemeral Storage Performance on NFS  similar to Ceph by overriding RAW format16:19
Ugglajayaanand, please wait the open discussion point16:20
jayaanandSent email to Openstack discussion group 16:20
jayaanandOk16:20
Uggla#topic OpenAPI 16:20
Uggla#link: https://review.opendev.org/q/topic:%22openapi%22+(project:openstack/nova+OR+project:openstack/placement)+-status:merged+-status:abandoned16:20
Uggla#info still 20 remaining atm.16:20
Uggla#topic Stable Branches16:21
sean-k-mooneyjayaanand: we shoudl dicsuss that at the end but that will need a spec and it may be contoverial16:21
* Uggla giving the mic to elodilles16:21
opendevreviewTakashi Kajinami proposed openstack/nova master: libvirt: Extend functional test coverage of UEFI boot guests  https://review.opendev.org/c/openstack/nova/+/96926316:21
sean-k-mooneyUggla: for openapi16:21
opendevreviewTakashi Kajinami proposed openstack/nova master: libvirt: Add basic xml generation for firmware auto selection  https://review.opendev.org/c/openstack/nova/+/96908516:21
opendevreviewTakashi Kajinami proposed openstack/nova master: libvirt: Add capability to load loader and nvram from xml  https://review.opendev.org/c/openstack/nova/+/96908616:21
opendevreviewTakashi Kajinami proposed openstack/nova master: libvirt: Add capability to load ssm feature from existing xml  https://review.opendev.org/c/openstack/nova/+/96913116:21
opendevreviewTakashi Kajinami proposed openstack/nova master: libvirt: Use firmware auto-selection by libvirt  https://review.opendev.org/c/openstack/nova/+/96913216:21
elodilles#info stable gates seem to be in good state16:21
sean-k-mooneyI got through all the serise before going on pto16:21
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:21
elodillesthat's all from me about stable16:21
gmaanstephenfin: yeah, its on me. I will try to review as much and soon i can16:21
gmaansean-k-mooney: ^^16:21
* elodilles passing back the mic to Uggla 16:21
tkajinamelodilles, I wonder if we merged all of the backports of the os-vif job update ?16:21
elodillestkajinam: not yet16:22
sean-k-mooneygmaan: no worries ill loop back as needed16:22
gmaan++16:22
elodillestkajinam: https://review.opendev.org/q/topic:fix-zuul-config-error-nova16:22
tkajinamelodilles, ah, ok16:22
elodilles^^^ the patches need stable core reviews o:)16:23
tkajinam^^^ it would be better to merge these before stable branches are still stable :-)16:23
tkajinams/before/while/16:23
Ugglasorry I was bit fast on the openapi topic. glad if gmaan, you can review these patches.16:23
Ugglasean-k-mooney, thanks for the review, hopefully we could finish this cycle.16:24
tkajinamUggla, no problem16:24
Ugglasomething more you'd like to add on either openapi or stable branches ?16:25
sean-k-mooneywe can likely move on. ill take a look at the zuul erros16:26
Uggla#topic vmwareapi 3rd-party CI efforts Highlights16:26
* Uggla passing the mic to fwiesel, who seems to have provided patches.16:26
fwieselHi, so I have finally hooked up the oslo.vmware repo to the CI... sorry for the oversight.16:26
fwieselStill working on a patch to fix the final problems (https://review.opendev.org/c/openstack/nova/+/941095), but I did run into another bug in Cinder vmdk driver there.16:27
sean-k-mooneyi know gibi was intersted in feedback on the eventlet related patches16:27
sean-k-mooneyfor oslo.vmware16:28
fwieselYes, I retriggered the job on his patch and provided a comment that it didn't cause any regressions.16:28
sean-k-mooneyawsome16:29
fwieselSo, hopefully I get another cinder patch in and then we should see tempest succeed.16:29
fwieselThat's from my side.16:29
Ugglathx fwiesel.16:30
Ugglaskipping next point about eventlet removal because gibi was on pto. And as he mentioned earlier, nothing new yet on that side.16:32
Uggla#topic Open discussion 16:32
Ugglajayaanand, please go ahead.16:32
nicolairuckelI'd like to talk about my NVRAM patch later16:33
Ugglanicolairuckel please go ahead then jayaanand.16:33
nicolairuckelhttps://review.opendev.org/c/openstack/nova/+/95968216:34
nicolairuckelIt's about that patch.16:34
nicolairuckeltkajinam had some comments and I wasn't sure if my answers really addressed all of his concerns.16:35
nicolairuckelAnd I was wondering if there's anything still needed for a (hopefully) final review.16:35
jayaanandSure 16:36
jayaanandWe are planning to implement NetApp FlexGroup Backend for Ceph-like Ephemeral Storage Performance on NFS  similar to Ceph by overriding RAW format16:36
jayaanandCan you please let us know if we are in right direction 16:37
nicolairuckelAfter that I'd like to work on fixing the behaviour for cold migration. I think tkajinam is implementing at least some parts of that already so it would be nice if we'd be able to coordinate that a bit.16:37
sean-k-mooneyjayaanand: let finish the nvram topic first16:37
tkajinamnicolairuckel, not really, but if we want to address that (detect change in nvram template and clear var) we have to import the machanism to detect existing nvram template from my change, and we likely have to clear it during cold migration16:37
sean-k-mooneytkajinam: so we woudl liek to backprot the nvram fixes if we can16:38
tkajinamnicolairuckel, the core problem is that cold migration provides no way to tell which nvram template is used before migration, because xml is generated from scratch at the destination and we don't pass the previous xml16:38
sean-k-mooneyso ideally we want to fix that first with as littel depency on the firmeware detection serise16:38
nicolairuckelAnd then after tkajinam's patch, it should be possible to fix the cold migration, correct? So it makes sense to wait for that.16:39
sean-k-mooneytkajinam: for cold migration we will need to handel that yes. that may not be backprotable16:39
sean-k-mooneybut at lest the hard reboot case shoudl be simpler.16:39
tkajinamnicolairuckel, no. it provides only base code to detect required reset for hard-reboot (and start) only16:39
sean-k-mooneyso for hard reboot we should not be reseting the nvram file ever16:40
nicolairuckeltkajinam had an example with secure_boot where it needs to be reset in the comments.16:40
sean-k-mooneyi guess we coudl create it if and onloy if it is not present16:41
nicolairuckelhttps://review.opendev.org/c/openstack/nova/+/959682/comments/ae045d85_9e58a5a516:41
sean-k-mooneyin case it got deleted out of band but we should expect it to exit in general when hard rebooting16:41
sean-k-mooneywhy woudl secure boot requrie it16:42
tkajinamsean-k-mooney, yeah but the core problem is that nvram code may be changed, because nova always re-select it when an instance is started ...16:42
sean-k-mooneyso wheter a vms uses securie boot is ment to be fixed at the tiem teh vm is first created16:43
sean-k-mooneyeven with os_secure_boot=optional16:43
dansmithisn't this stuff that can be discussed on the review?16:44
tkajinamyes16:44
dansmithboth because we have a big audience here, but also these sorts of reasoning things seem best to capture in the gerrit history16:44
sean-k-mooneywell the didcion is ther form last year16:45
sean-k-mooneyi disagree with tkajinam usecase however but we can contiue to dicuss it ther16:45
dansmithI've reviewed the patch a little in the past, but I think that we probably need sean-k-mooney to help ack the workflow and lifecycle constraints (and maybe tkajinam) but.. that is best preserved in the gerrit comments for other reviewers I think16:46
nicolairuckelsorry, I just brought it up here because I didn't get any replies on the review anymore and wanted to know if there's anything I'd need to do before my patch can get merged.16:46
nicolairuckelBut I'd be happy to discuss it there. :)16:46
dansmithnicolairuckel: yeah it's fine to raise the awareness here, I'm just saying I don't think we need to have this full discussion in the meeting16:46
dansmithjust MHO16:46
nicolairuckelI'll mark the comment as unresolved now and I guess then we can move on for now.16:47
dansmithand also,16:47
dansmithif it's this complicated, perhaps we should have a spec explaining the theory of when we want to nuke NVRAM, not, and the constraints around firmware that come with it16:47
dansmithbecause that's helpful for the review, otherwise I'm left kinda helpless, sure that I don't know all these intricacies and thus unable to approve16:48
sean-k-mooneyfor me that is simply its baisclly never ok to od it unless perhaps rebuild16:48
dansmiththat would be my assumption as well16:48
sean-k-mooneyeven then its qutionable as ti will brake bitlocker16:49
dansmithbasically, like a real computer with actual NVRAM16:49
sean-k-mooneyyep16:49
sean-k-mooneywe dont currently snapshto the nvram so i dont think rebuild shoudl recreated it. however it might be reasonable fi rebuilding to a diffent image entirly.16:49
sean-k-mooneymy general hope was we woudl fix preseving it for hard reboot in the current patch16:50
sean-k-mooneyand then adress everything else sperately16:50
dansmithagain, this is sounding more like a spec than a bug fix, given it's going to introduce new behavior and affect many operations16:50
sean-k-mooneymaybe but to me this was just restorign the behvior we orginally intended16:51
sean-k-mooneybut i agree its non trivial and we need to docuemnt the lifecylce16:51
dansmithI didn't ever have intent for this, so don't lump me into that "we".. perhaps you mean what _you_ intended, and that's fine, but regardless it's going to be more of a change from where we are16:52
dansmithespecially if we're doing things like only recreating if rebuilding to a different image, etc16:52
sean-k-mooneywell that actully more of a a comment on the related vtpm feature rather then nvram16:53
sean-k-mooneywe said we wanted to adress both the same way16:53
sean-k-mooneyand for vtpm we very explcitly wanted to preserve the data across lifecycle operations16:53
nicolairuckelI mean, there were a few bug reports were people expected the behaviour from my patch.16:54
sean-k-mooneylets give the last 5 mins to jayaanand 16:54
Ugglaguys, can we stop and let some time for jayaanand16:54
jayaanandthank you 16:54
jayaanandWe are planning to implement NetApp FlexGroup Backend for Ceph-like Ephemeral Storage Performance on NFS  similar to Ceph by overriding RAW format16:55
sean-k-mooneyjayaanand: if i understand your propsoal form the list you want to intoduce nfs awarenes to the image_types backend by adding a new backend16:55
dansmithjayaanand: I dunno what "override RAW" means, but that's almost certainly not going to fly.. either way, we need a detailed spec proposed to review to explain what you're planning and then we can review16:55
jayaanandThis cow operations will be taken care at storage 16:56
dansmithwe're not going to get into the details of that in a meeting, and certainly not with five minutes left, until we have some context to build16:56
jayaanandOk, Has anyone explored or implemented storage-side cloning for other NFS vendors?16:57
sean-k-mooneyno not in nova16:57
sean-k-mooneywe do not creat vendor specific backend in nova in general16:57
sean-k-mooneyso for us to supprt that in nova we woudll ideally want ti wot work for any generic nfs server16:58
jayaanandOk, concerns is any architectural red flags with this approach in the Nova driver? NFS Ephemeral volumes 16:58
sean-k-mooneythe main concern with adding a nfs images_type backend is there is a lot fo existing technial debt in that part of the code base16:59
dansmithyes, I have red flag concerns because of what sean-k-mooney said and also because server-side snapshotting of files nova is accessing is already complicated and different when we do it with cinder16:59
dansmithyes, I also don't really want to be adding new image backends to the current implementation16:59
bauzasagreed17:00
* dansmith has to run17:00
sean-k-mooneyits something we can try and dicsus about more but if it would only work for netapp that proably not someting we would pursue. we would also need to dicuss how this will be tested in ci. all of witch would be captures in a spec17:00
jayaanandOk, we have multiple customers looking for this feature... What is way out?any suggestions 17:01
sean-k-mooneyi think we need to know what the problem statement is first17:01
tkajinamjayaanand, I think the general preference is to use cinder volume in case you want better features. I still don't understand why you really need it within ephemeral volumes17:01
sean-k-mooneyperhaps there si a way to adress that without requriing a new backend17:01
tkajinamthat's something I expect to be covered by that statement17:01
sean-k-mooneytkajinam: i think the main issue is the copy tiem if your using a raw image. obvioulsy there is no copy invoveld for qcow since we just share the backing file form the cache outise of the initial spawn17:02
jayaanandWe want to leverage copy on write technology from ONTAP 17:03
tkajinamok but I think that can be also eased by image cache volume in cinder/glance I guess17:03
jayaanandInstead of qcow2 format 17:03
sean-k-mooneyi think we will need more context on hwo that work and if tis a generic feature in nfs itslef or some propeirty extnsion17:04
jayaanandWe explored cinder volumes. Direct ONTAP volumes give better performance 17:04
sean-k-mooneybut you said using nfs right so taht woudl still just be a file of some form on disk17:05
sean-k-mooneywell a file on a file system17:05
jayaanandIt's propeirty on top of NFS17:05
sean-k-mooneyright os that also problematic17:06
jayaanandYes NFS on compute node17:06
jayaanandFile on ONTAP with better copy on write 17:06
sean-k-mooneyack can you create a draft spec, it does not have to be compelte with your problemstatemen use cases and any relvente context/info on the feature17:07
tkajinamhttps://specs.openstack.org/openstack/nova-specs/17:07
tkajinamhttps://specs.openstack.org/openstack/nova-specs/readme.html17:08
jayaanandSure, i will create, thank you for the advice 17:08
UgglaCool, we are already overtime, last minute thing to add ? If not then we will close.17:08
jayaanandNeed to understand how right OS that is problematic?17:09
womaxI had something I wanted to talk about but it can wait for next week17:09
womaxand i think it can be discussed outside of meetings17:09
Ugglawomax thanks, if you can update the agenda, so it will be identified.17:10
jayaanandSure 17:10
womaxUggla: ok i'll do that17:10
UgglaThanks for joining this meeting. Closing now.17:11
Uggla#endmeeting17:11
opendevmeetMeeting ended Mon Jan 12 17:11:24 2026 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:11
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2026/nova.2026-01-12-16.05.html17:11
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-01-12-16.05.txt17:11
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2026/nova.2026-01-12-16.05.log.html17:11
jayaanandThank you 17:12
sean-k-mooneyjayaanand: googleign thie a liitle ONTAP wafl filestiem when exported vis nfs 4.2 can transparently supprot servier side cloning when you use qemu-im or cp with the --reflink link flag17:17
sean-k-mooneyprovided you have ONTAP 9.817:18
jayaanandtrue, ONTAP handles RAW with native compression and dedupe. this with avoid double virtualization17:19
sean-k-mooneyso im wonderign if the flat backend woudl be enough for your usecase17:20
sean-k-mooneyor raw17:20
sean-k-mooneyif we ensure nova propelry uses that feature when nfs 4.2 is used17:20
jayaanandraw is better handled at ONTAP 17:20
sean-k-mooneysure but we already supprot raw format17:21
jayaanandfile is facade hear, internally it is RAW at WAFL17:21
sean-k-mooneywhat im saying is it may be possible to not add a new backend at all17:21
sean-k-mooneyand simple use the more efficnet copy method when tis supproted by the file systems17:22
jayaanandwe won't introduce new format. we will user RAW with ONTAP copy on write capabilities 17:22
jayaanandwe will delegate any clone operation to ONTAP for quick VM creation17:22
jayaanandon top of RAW17:23
jayaanandthese are NFS files at ONTAP on top of FlexGroup17:23
sean-k-mooneyright the feedback im providieng is im not sure that we will accpate a vendor speciics storage backend in nova17:24
sean-k-mooneyand im tryign to understand if we can achive the same thing by jsut doing "cp --reflink=always source.raw destination.raw"17:24
MaxLamprecht[m]Interestingly I heard yesterday also about the nfs4.2 native server-side-copy/file-cloning feature. Sadly ONTAP 9.16 currently returns NFS4 error not supported. I guess it‘s not implemented by netapp or hidden by some config flag or so.17:24
sean-k-mooneywhen doign the copy fo the image17:24
sean-k-mooneywell nfs v3 supprot is deprecated in general17:25
sean-k-mooneywe ahve recommend 4.2 or a bove for many years now17:25
sean-k-mooneythat recomendation orgianlly came form the qemu storage maintaienrs17:25
jayaananddocs.netapp.com/us-en/ontap/nfs-admin/ontap-support-nfsv42-concept.html17:25
sean-k-mooneyi.e. to move to v4+ due to lockign issues17:26
jayaanandMax, almost all of our customers on NFS 4+17:26
jayaanandyou can look into your specific issue17:26
MaxLamprecht[m]Yes we also use nfs4.2 :)17:26
jayaanandwe tried "if we can achive the same thing by jsut doing "cp --reflink=always source.raw destination.raw""... this is introducing conversion overhead at ONTAP17:27
sean-k-mooneyhttp://nfsv4bat.org/Documents/ConnectAThon/2010/ss-copy-spec.pdf that netaps presetion on the feature form 201017:27
sean-k-mooneybut maybe it never got enabeld or is behind a config flag17:28
jayaanandFor direct storage on the ONTAP backend, QCOW2 images are automatically converted to the raw format17:30
jayaanandhttps://www.netapp.com/media/10720-tr-4067.pdf17:31
sean-k-mooneyok but i am not suggestign you woudl use qcow im suggestign you configre nova to use raw format but when we copy the raw file from the casche we woudl do that with --reflink so that there would nto be a data copy17:31
sean-k-mooneyso you woudl set https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.force_raw_images to true17:32
jayaanandhttps://www.irccloud.com/pastebin/OCDzzgTM/17:32
sean-k-mooneyand images-type=raw or images_type=flat17:32
jayaanandok, let me check if reflink will trigger clone at ONTAP17:36
jayaanandi will explore and share my findings17:36
sean-k-mooneyack17:36
sean-k-mooneysimply supproting a generic nfs feature like server side copies would be a much smaller chnage if that is suffeient to fullfile the usecase17:37
jayaanandsure, we want to use RAW format to take advantage of propitiatory de-dupe, compression and COW 17:38
jayaanandand avoid double virtualization at storage layer and save VM cycles for QCOW2 maintenance 17:39
opendevreviewsean mooney proposed openstack/nova master: Support os-vif TAP pre-creation for OVS/OVN ports  https://review.opendev.org/c/openstack/nova/+/97314918:05
opendevreviewsean mooney proposed openstack/os-vif master: Add TAP device pre-creation support for OVS/OVN  https://review.opendev.org/c/openstack/os-vif/+/97123118:06
opendevreviewsean mooney proposed openstack/nova master: Support os-vif TAP pre-creation for OVS/OVN ports  https://review.opendev.org/c/openstack/nova/+/97314918:25

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!