16:01:39 <Uggla> #startmeeting nova
16:01:39 <opendevmeet> Meeting started Tue Aug  5 16:01:39 2025 UTC and is due to finish in 60 minutes.  The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:39 <opendevmeet> The meeting name has been set to 'nova'
16:01:51 <Uggla> Hello everyone
16:02:02 <jssfr> o/
16:02:04 <jssfr> NicolaiRuckel and me would like to bring up a topic in the "Open Discussion" slot
16:02:08 <jssfr> (jfyi)
16:02:29 <fwiesel> o/
16:02:44 <Uggla> jssfr, no pb but this week few people are available.
16:03:01 <jssfr> thanks for the heads up :)
16:03:08 <gmaan> o/
16:04:04 <Uggla> awaiting people to join, but I'm pretty sure we will not have quorum.
16:04:36 <Uggla> Please raise your hand for  showing you are part of the meeting.
16:05:07 <NicolaiRuckel> o/
16:05:40 <dansmith> o/
16:06:03 <Uggla> Let's start
16:06:09 <Uggla> #topic Bugs (stuck/critical)
16:06:16 <Uggla> #info No Critical bug
16:06:58 <melwitt> o/
16:07:04 <Uggla> #topic Gate status
16:07:14 <Uggla> hey melwitt, good to see you
16:07:23 <Uggla> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:07:29 <Uggla> #link https://etherpad.opendev.org/p/nova-ci-failures-minimal
16:07:36 <Uggla> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status
16:07:48 <Uggla> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:07:58 <Uggla> #info Please try to provide a meaningful comment when you recheck
16:08:06 <Uggla> #info Gate was blocked last week with "openstack.exceptions.SDKException: Image creation failed: 'latin-1' codec can't encode character '\u2019' in position 662: ordinal not in range(256) in openstack.tests.functional.image.v2.test_image.TestImage.test_tags".
16:08:13 <Uggla> #info https://review.opendev.org/c/openstack/openstacksdk/+/956369 fixed it
16:08:49 <Uggla> #topic tempest-with-latest-microversion job status
16:08:58 <Uggla> #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-with-latest-microversion&skip=0
16:09:10 <Uggla> gmaan, do you have something to share ?
16:10:28 <gmaan> ah, no update on this.
16:10:47 <Uggla> thanks, so moving on
16:10:55 <Uggla> #topic Release Planning
16:11:02 <Uggla> #link https://releases.openstack.org/flamingo/schedule.html
16:11:07 <Uggla> #info Nova deadlines are set in the above schedule
16:12:15 <Uggla> Time is flying, a bit less than 4 weeks before FF.
16:12:26 <Uggla> #topic Review priorities
16:12:36 <Uggla> #link https://etherpad.opendev.org/p/nova-2025.2-status
16:13:12 <Uggla> I guess the file is updated, please let me know if something is wrong in it.
16:13:29 <Uggla> #topic OpenAPI
16:13:37 <Uggla> #link: https://review.opendev.org/q/topic:%22openapi%22+(project:openstack/nova+OR+project:openstack/placement)+-status:merged+-status:abandoned
16:13:59 <Uggla> #info up from 13 to 32 remaining. (I guess Stephen created new ones and follow up)
16:14:24 <Uggla> I hope we can see an end about this.
16:15:01 <Uggla> I'm going to skip stable branch topic because Elod is on pto.
16:15:28 <Uggla> #topic vmwareapi 3rd-party CI efforts Highlights
16:15:36 <Uggla> fwiesel, something on our side ?
16:15:44 <fwiesel> #info fwiesel will be on PTO until the 9/9
16:15:54 <fwiesel> Otherwise, no news from me.
16:16:07 <fwiesel> Uggla: back to you
16:16:23 <Uggla> fwiesel, I wish you'll enjoy your PTO.
16:16:33 <fwiesel> Thanks!
16:17:25 <Uggla> Also skiping Gibi's point about eventlet removal due to Gibi's well deserved pto.
16:17:59 <Uggla> #topic Open discussion
16:18:17 <melwitt> I have a small topic
16:18:18 <Uggla> So jssfr, giving the mic to you.
16:18:26 <jssfr> thanks!
16:18:38 <jssfr> so we're currently looking into two topics which are somewhat related:
16:18:41 <jssfr> first we're rebasing https://review.opendev.org/c/openstack/nova/+/621646
16:18:53 <jssfr> second I proposed a draft patch for a vTPM data preservation issue a couple weeks ago: https://review.opendev.org/c/openstack/nova/+/955657
16:19:22 <sean-k-mooney> o/ sorry was doign something else
16:19:26 <jssfr> o/
16:19:38 <jssfr> I also think that sean-k-mooney proposed something similar to the vTPM draft for UEFI NVRAM preservation
16:20:00 <jssfr> we'd like to hear whether we're heading the right direction by rebasing #621646 regarding UEFI, or whether there might be a better way.
16:20:17 <sean-k-mooney> i was suggeing bring it up in this meetign yes
16:20:20 <melwitt> oh, good. this was the topic I wanted heh (vTPM data)
16:20:24 <sean-k-mooney> or rather that melwitt  bring it up
16:20:35 <jssfr> I also saw only now that you replied to #955657, because I had my inbox sorting messed up, reading that now.
16:20:59 <sean-k-mooney> melwitt: do you want to summerize
16:21:08 <jssfr> read it
16:21:12 <melwitt> sure
16:21:33 <melwitt> we are looking for input and consensus about handling the recently raised issue in the community regarding vTPM data that is lost across guest reboots https://bugs.launchpad.net/nova/+bug/2118888 and https://review.opendev.org/c/openstack/nova/+/955657 the tl;dr is that this is a latent problem that originates in libvirt
16:22:08 <melwitt> libvirt added a flag for this VIR_DOMAIN_UNDEFINE_KEEP_TPM a couple of years after vTPM was added in Nova and jssfr has proposed a patch for using the flag ^
16:22:20 <jssfr> (which will need a bit of work, such as adding tests)
16:22:30 <jssfr> (but I can do that)
16:22:53 <sean-k-mooney> there is a related but seperate issue that related to the nvram data
16:23:08 <sean-k-mooney> again there are now flag to supprot keeping or removing the nvram file
16:23:32 <jssfr> (UEFI NVRAM bug is https://bugs.launchpad.net/nova/+bug/1785123, patch-with-merge-conflicts is https://review.opendev.org/c/openstack/nova/+/621646 )
16:23:37 <dansmith> so if we do that, we need to do our own deleting of the TPM file when we undefine to actually delete the instance?
16:23:51 <jssfr> dansmith, no, because we can just not pass that flag when we truly want to delete
16:23:57 <sean-k-mooney> dansmith: no. we can pass the keep or delte flag as approtiate
16:23:58 <jssfr> in my draft patch, I tied it to the delete_storage bool IIRC
16:24:01 <dansmith> oh it's a flag _to_ undefine?
16:24:04 <jssfr> yes :)
16:24:06 <melwitt> yes
16:24:07 <dansmith> okay
16:24:26 <dansmith> ah I see yes, okay
16:24:57 <sean-k-mooney> so we have a number of placess we undefine the vm today wher the data shoudl not be deleted, hard-reboot is one but also some volume operations require use to temporlly undefien the domain
16:25:47 <melwitt> we are thinking about, how to fix this for older versions. the flag addition could be technically seen as a "feature" but IMHO the impact is quite significant and personally I would think about backporting the flag fix to older versions that could be reasonably expected to use libvirt >= 8.9.0 where the flag was introduced
16:26:13 <melwitt> wondering what others think of this
16:26:37 <dansmith> shifting behaviors in a library that we have to account for is not a feature, IMHO
16:26:37 <jssfr> I think a backport would be quite appropriate
16:26:44 <jssfr> and we are going to backport to at least 2024.1 anyway
16:27:15 <jssfr> so we could contribute those backported patches at least.
16:27:27 <sean-k-mooney> dansmith: the wrinkel is libvirt uncondtionally delete the data when we added vtpm supprot and the option to preseve it was added 2 years later
16:27:44 <dansmith> yeah understand
16:28:04 <sean-k-mooney> with that said the didnt really document that which is  partly why we missed it but i agre we shoudl not require a feature to "do the write thign"
16:28:05 <jssfr> sean-k-mooney, so you're saying it's not shifting behaviour in libvirt, but simply a bug which existed since forever but which nobody noticed?
16:28:30 <sean-k-mooney> jssfr: yes, however that does not mean i belive we shoudl consider it a feature
16:28:51 <sean-k-mooney> i supprot the idea of fixing this condtional on the libvirt verison
16:28:54 <dansmith> it's shifting behavior and our failure to account for it is a bug not a feature
16:28:55 <jssfr> (I get the impression that "feature" versus "bug" is an important distinction here, but I'm not sure why. backporting/support policies?)
16:29:05 <sean-k-mooney> and possibel backporting that based on dicusssion with teh stable team
16:29:20 <melwitt> I agree with dansmith
16:29:41 <sean-k-mooney> dansmith: well the default never changed in libvirt. it still deleted on undefine by defualt. they just added the ablity to keep it in later release
16:29:56 <sean-k-mooney> so do we agree to treat this a logic bug in nova
16:30:03 <sean-k-mooney> i.e. that we didnt account for how this actully works
16:30:24 <sean-k-mooney> and if so are we ok with usign the flags to adress that
16:30:58 <sean-k-mooney> jssfr: yes the stable plociy does not allow feature backports upstream
16:31:05 <jssfr> aha, thanks!
16:31:11 <sean-k-mooney> jssfr: there are very limited excptions to that
16:31:52 <jssfr> noted, thanks.
16:32:25 <melwitt> I support backporting use of the flags (with libvirt version conditional)
16:32:40 <sean-k-mooney> im +1 on treating it as a but and fixing it by approatbly passing the flags to undefine based on if we are deleting the instance or just undefining for some other reason
16:33:30 <NicolaiRuckel> this applies to both vTPM and UEFI, right?
16:33:33 <sean-k-mooney> s/as a but/as a bug/
16:33:49 <sean-k-mooney> so its the same patther and the same logic bug
16:34:07 <sean-k-mooney> so i think we could apply the same direction to both
16:34:34 <melwitt> +1
16:34:39 <jssfr> sean-k-mooney, does this only apply for the accidental deletion on undefine for UEFI, or also for the loss of NVRAM data during migrations?
16:34:39 <sean-k-mooney> the specific libvirt verions may differ for the relevent flags
16:34:46 <sean-k-mooney> so we shoudl fix both as seperate patches
16:35:17 <sean-k-mooney> jssfr: nvram data is actully copied during live migration i belive
16:35:38 <sean-k-mooney> cold migration is sperate. we shoudl focus on the non move operations cases first
16:36:00 <jssfr> NicolaiRuckel, any evidence of nvram preservation during live migrations when initially rebased https://review.opendev.org/c/openstack/nova/+/621646 ?
16:36:15 <sean-k-mooney> im not sure which subset of opertions loose nvram data but fixing all fo them is not going to be a singel bug
16:36:27 <jssfr> understood, thanks.
16:36:31 <sean-k-mooney> jssfr: i belive its copied by qemu
16:36:51 <jssfr> aha
16:37:13 <jssfr> NicolaiRuckel, we'll have to double-check that in the devstack, without the rebased patches.
16:37:23 <NicolaiRuckel> yeah, I didn't try it yet
16:37:41 <NicolaiRuckel> There were so many failing test cases that I wanted to clean that up first.
16:38:02 <sean-k-mooney> i would sugget fixign one bug first then the other
16:38:18 <sean-k-mooney> i dont really have a prefence in order but proably vtpm might be less work
16:38:49 <sean-k-mooney> for nvram there is existing tech debt that need to be unwornd related to rebuild
16:38:53 <jssfr> sean-k-mooney, okay, I'll get to work on making the vTPM patch ready for merge this week then.
16:39:22 <jssfr> probably not going to *finish* it this week though. What is the time we need to submit the polished patch (incl. tests) for it to be realistically merged for Flamingo?
16:39:48 <sean-k-mooney> you have about 3 weeks
16:39:54 <jssfr> okay, so 1w before FF
16:39:55 <jssfr> noted
16:40:03 <sean-k-mooney> well you have until FF
16:40:14 <jssfr> oh I thought that was in 4 weeks.
16:40:26 <Uggla> A bit less than 4 weeks
16:40:37 <jssfr> right
16:40:38 <sean-k-mooney> after which it woudl need a backport but its not a regressin intoduced in this cycle so between FF and RC1 we dont fix a lot of latent bugs
16:40:54 <sean-k-mooney> after rc 1 is cut the branch woudl be reopen for bugfixes again
16:41:00 <Uggla> FF will be August 28th
16:41:15 <jssfr> it should be doable to finish it in two weeks, so I'll aim for a submission before or on 2025-08-19
16:41:18 <sean-k-mooney> yep that why i said about 3 weeks
16:41:38 <NicolaiRuckel> I'll continue with rebasing/fixing the UEFI patch in the meantime.
16:44:45 <sean-k-mooney> i think we can move on if there are other topic unless there are any final objections?
16:45:24 <jssfr> I'd probably jump out of the meeting then. I'll stay reachable here as usual though if there's any later remarks. Thanks for all the input!
16:46:15 <Uggla> ok I guess melwitt point was covered ^
16:46:33 <Uggla> anything else to discuss ?
16:47:40 <jssfr> okay, text you all later, thanks!
16:52:59 <Uggla> As we are almost on the top of the hour, I'll skip bug scrubbing.
16:53:44 <Uggla> Thanks all, see you next week.
16:53:50 <Uggla> #endmeeting