Tuesday, 2024-06-04

*** elodilles_ooo is now known as elodilles06:32
zigoI'm still getting the issue that Caracal doesn't like cpu_models = EPYC-Milan (nova-compute exists with "Unacceptable CPU info: CPU doesn't have compatibility"), even tough "virsh capabilities" shows that my host's CPU really is EPYC-Milan. Is the _check_cpu_compatibility somehow broken?!?12:42
zigoHow can I check what it does?12:43
dansmithbauzas: do you want to +W this or should I? https://review.opendev.org/c/openstack/nova-specs/+/90765414:12
bauzasdansmith: do this14:13
dansmithbauzas: ?14:13
sean-k-mooneybauzas: does that mean you will or that dan should14:16
sean-k-mooneycause i was going to ask dansmith why he did not +w but then i saw this converstaion14:17
dansmithI thought bauzas said several times he wanted to review that, so I held off the +W but I also didn't realize you had +2d yet14:17
dansmithI was actually going to it to see the status and saw you had +2d14:18
sean-k-mooneyack14:18
dansmithbut if bauzas isn't going to in short order, we should just send it I think14:18
dansmithwe're running out of time14:18
sean-k-mooneyi just the email update for your +2 and was wondering but was waiting for our downstream converation to come to an end before changing topic14:19
sean-k-mooneyanyway im cool leaving it to either of you to +w i was wondering if my +2 had been lost or not14:20
bauzasok, ok, I can try to give it a shot14:23
dansmithbauzas: can you clarify what you're saying? _are_ you going to review that?14:28
bauzasdansmith: sorry I had to take my daughter from school14:52
bauzasif you want, I can review it indeed14:52
dansmithbauzas: we're just trying to figure out if I should approve it, or of you *want* to review it first14:52
bauzasdansmith: okay, so my point is that if you think all is good, you can approve it14:55
dansmithdone14:56
opendevreviewMerged openstack/nova-specs master: Re-propose specs for ephemeral encryption  https://review.opendev.org/c/openstack/nova-specs/+/90765415:04
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Jun  4 16:00:41 2024 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzashey folks16:00
elodilleso/16:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:00
bauzasshould be a quick one hopefully16:01
erlon\o16:01
marlinc\o16:01
kgube\o16:01
auniyalo/16:01
Luzio/16:02
marlincI forgot to put my topic in the agenda, is that still oke?16:02
bauzasshoot it in the agenda, sure16:02
bauzasokay, let's start16:02
bauzas#topic Bugs (stuck/critical) 16:02
bauzas#info No Critical bug16:03
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:03
bauzasanything about buuugs ?16:03
fwiesel\o16:03
gibio/16:03
bauzaslooks not16:03
bauzaslet's move on then16:03
bauzas#topic Gate status 16:04
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:04
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures-minimal16:04
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:04
bauzas(all greens, huzzah)16:04
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:04
bauzas#info Please try to provide meaningful comment when you recheck16:04
bauzasI haven't seen any important gate failure16:04
bauzasand you ?16:05
bauzasok, np16:06
bauzas#topic Release Planning 16:06
bauzas#link https://releases.openstack.org/dalmatian/schedule.html16:07
bauzasas a reminder, nova deadlines are ^16:07
bauzas#info Dalmatian-2 in 4 weeks16:07
bauzastick-tock16:07
bauzasnothing planned as a review day until milestone-2 where we will have a spec review day16:07
bauzasanything else about our schedule ?16:07
erlonIs there a freeze day for bug fixes?16:07
erlonI only see a freeze for specs16:08
erlonah, I see now, seems common to all projects16:09
bauzaswe can merge bugfixes until RC116:09
bauzasafter RC1, we then branch master for Dalmatian16:09
erlonok, so, what kind of changes are accepted after RC1?16:10
bauzasthen, in between RC1 and GA, we can't merge any bugfix, ony regression fixes16:10
erlonhmm, ok, makes sense16:10
bauzasbut we can still merge bugfixes in master16:10
erlonRight, that will be available for the next release16:10
bauzassince master will then be E (and no longer D) after RC116:10
bauzasso basically, we can merge any bugfixes anytime16:11
bauzasfor master16:11
sean-k-mooneythe only soft freeze is really betwee Feature freeze and RC116:11
bauzasbut if you want your fix to be on a specific release, then either before RC1 or after GA by backporting it 16:12
sean-k-mooneywe can merge a bugfix during that period but prefer to limit them ot regressions16:12
erlonright16:12
bauzassean-k-mooney: not for bugfixes16:12
bauzasanyway, I think you have your answer16:12
bauzascan we move ?16:12
bauzas#topic Review priorities 16:13
sean-k-mooneybauzas: we have prevsiously ased reviews not to merge bugfixes for bugs not intoduced in the current release in that period. but yes we can move on16:13
bauzas#link https://etherpad.opendev.org/p/nova-dalmatian-status16:13
bauzasnothing to say but please look at it ^16:13
bauzas#topic Stable Branches 16:13
erlonI have a request here16:13
erlonfor Review priorities 16:13
erlonI would like to get some attention on the Shared security groups patches?16:14
bauzas#undo16:14
opendevmeetRemoving item from minutes: #topic Stable Branches 16:14
erlons/?/\./g16:14
marlincI might have put my blueprint in the wrong meeting section, I'm not sure if it is this or open discussion16:14
sean-k-mooneyno that at the end16:15
bauzasmarlinc: we'll discuss this on the open discussion16:15
sean-k-mooneywhich is where we should discuss the shared security group patches :)16:15
marlincThank you :)16:15
bauzaserlon: for your series, this will be reviewed like any other 16:15
erlonow, came on, thas a review priority topic :)16:15
bauzashttps://etherpad.opendev.org/p/nova-dalmatian-status?#L3716:16
bauzasyeah16:16
erlonRight, I just want to make sure that I don't miss any deadlines on that one16:16
sean-k-mooneyyes and no its not identifed as a review priortiy form my perspective16:16
bauzasgiven your blueprint was accepted16:16
sean-k-mooneybut ill try and review it after the meeting16:16
sean-k-mooneylooks like its passing ci now16:16
erlonAnd I also have a special request on that16:16
bauzasditto here, I already have a lot of other series to look at16:16
bauzasmoving on then16:16
bauzas#topic Stable Branches 16:17
bauzaselodilles: heya16:17
elodilleso/16:17
elodilles#info stable gates should be mostly OK16:17
elodilles(i've seen some intermittent failures, otherwise, nothing special)16:17
elodilles#info stable release proposed for 2024.1 Caracal: https://review.opendev.org/c/openstack/releases/+/92128716:17
elodilleswe had Bobcat and Antelope stable releases some weeks ago,16:18
elodillesmaybe we can release now Caracal as well ^^^16:18
elodillesfeel free to comment on the release patch16:18
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:18
elodillesadd any issue if you see one ^^^16:18
elodillesand that's all from me16:19
bauzasthanks16:20
bauzasI'll look at the caracal patch16:20
elodillesbauzas: thx in advance16:21
bauzas+1d fwiw16:21
bauzas#topic vmwareapi 3rd-party CI efforts Highlights 16:21
bauzasfwiesel: are you here ?16:21
fwieselHi, yes... 16:21
fwiesel#info No updates.16:21
fwieselStill little progress from my side currently. Sorry about that.16:22
fwieselAny questions or comments?16:22
bauzasnot from me16:22
bauzasmoving on then16:22
bauzas#topic Open discussion 16:23
bauzaserlon: nova handling of virDomainGetJobStats() errors16:23
erlonso, I posted a link on the wiki16:23
erlon https://gist.githubusercontent.com/sombrafam/8f177cbc4e153c328a242811bc24650e/raw/67190ffce7f036c7c3d3628fda15327cef7a41da/nova-compute.log16:23
bauzas(woah, we have 4 topics to discuss today so please discuss everyone by only 5-10 mins maxc)16:23
erlonThis bug, is happening every time that we try to do a lot of migrations16:23
bauzasjust file a blueprint bug report16:24
erlonFor some reason the source host stops responding, and it gets to the time out. I want to know if would be okay to add a new Handler for this kind of exception there:16:24
erlonhttps://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L66916:24
bauzasand ping us anytime on the chat16:24
sean-k-mooneyerlon: does that imply you you have set the max conncurrent live migration over 116:24
erlonah ok, sounds good16:24
erlonI don't know the exact configuration but very likely16:25
sean-k-mooneyhttps://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_concurrent_live_migrations 16:25
sean-k-mooneythat should bassicly never be set to anything excpet 116:25
erlonSo why does that exist? Which case should it apply?16:26
sean-k-mooneythere are usecases for it to be != 1 but unless you using 100G networking its not recommended16:26
bauzasit just says it can't connect to the libvirt API16:26
sean-k-mooneyit exists for usecases wehere a single core is not enough to saturate the network link16:26
bauzasthere could be different reasons why libvirt is resetting the connection16:26
erlonIt says that because it time it out due to a short keepalive16:27
sean-k-mooneybauzas:right but since they said this was load related16:27
sean-k-mooneynot doning concurrent live migrations may help16:27
bauzasoh, missed that sentence, indeed good point16:27
bauzasdon't try to max out the number of calls you make to libvirt16:27
bauzaslike any other API, it has limits16:28
erlonThis comes from the libvirt logs: "At the same time in libvirtd logs: `May 28 13:00:39 ps5-ra1-n6 libvirtd[612268]: internal error: connection closed due to keepalive timeout`"16:28
erlonSo, that's why we know thats a timeout error16:28
bauzaserlon: to answer your original question, no, I wouldn't recommend to modify Nova to handle that libvirt exception16:28
bauzasparticularly when the bug comes from libvirt, not nova itself16:29
erlonBut see, that function does exactly that. it handles only libvirt exceptions16:30
sean-k-mooneythere are some cases where retrying at the nova level is correct16:30
sean-k-mooneythis may or may not be one of them 16:30
bauzasspecific and meaningful libvirt exceptions, that's the point :)16:30
sean-k-mooneycan you right this up as a bug report if you have not already16:30
sean-k-mooneyand we can see see if this is such a case16:31
bauzasretrying on a generic connection failed is never a good idea16:31
erlonok, I will as soon as I get a reproducer16:31
sean-k-mooneybauzas: we do that all the time for rest calls16:31
bauzashonestly, you have my opinion16:31
bauzassean-k-mooney: to placement16:31
sean-k-mooneythis is us doing a read call to check the status fo the job16:31
sean-k-mooneybauzas: and neutorn and cinder16:31
erlonI was actually thinking and just returning, an empty info and leave nova call it again in the next periodic task, since this is just a info report16:31
sean-k-mooneyperhaps or catching it an logging a warning16:32
sean-k-mooneyrather then a trace back16:32
bauzasI wouldn't hide 16:33
sean-k-mooneyanywya if you have a repoducer we can discuss the solution in gerrit16:33
bauzasif I were catching that exception16:33
erlonThe issue with the current process is that, although it triggers the section again, the16:34
erlonperiodic tasks do not complete the migration. Consequently, while the VM is migrated16:34
erlonto the destination host, it is not properly cleaned up on the source host.16:34
bauzashonestly, exception giving a better log to the operator saying 'libvirt is getting weirdo, see", I don't see the benefit of adding another handler to a periodic16:34
opendevreviewMerged openstack/nova stable/2023.2: Improve logging at '_numa_cells_support_network_metadata'  https://review.opendev.org/c/openstack/nova/+/90084016:34
bauzasanyway, we have three other topics to discuss16:34
bauzaserlon: please file a bug report and come to us once's done16:34
erlonOkay, let's move, thanks16:34
bauzasand yeah, a reproducer may help16:34
bauzasparticularly with some mitigation actions16:35
bauzaslike, I know libvirt gives me this but we can workaround by that16:35
bauzasthat way, we could log a better warning than just "heh, look, that's what I got from libvirt"16:35
bauzasmoving on anyway16:36
bauzasnext item (2 of 4)16:36
bauzas marlinc: rotation_rate blueprint: https://blueprints.launchpad.net/nova/+spec/rotation-rate16:36
bauzasI guess you're asking for a specless approval ?16:36
marlincHonestly I am not entirely sure what is best, I thought specless might work for this but I'm not sure16:37
bauzasthat's the point16:37
bauzasyou're saying you would get the detail from the cinder connection info16:37
bauzasdo we already have that ?16:37
sean-k-mooneyif we add a new compute service version for this and a min compute service version check i think that woudl resolve most if not all the upgrade concerns16:37
marlincYes so we have an internal Cinder driver for our own storage platform which does give that property in the cinder connection info, we didn't have change anything in Cinder itself16:38
bauzasI would honestly then recommend to write a spec 16:38
sean-k-mooneymarlinc: is there an upstream driver that support this16:38
bauzasthere could be some upgrade and compatibility concerns I may see16:38
sean-k-mooneymarlinc: without that we can proceed with this in nova16:38
sean-k-mooneywe do not add supprot for out of tree drivers in other project16:38
marlincNo there is currently no upstream driver that has this16:38
bauzasthen we can't just modify nova to blindly check anything that's not upstream16:39
sean-k-mooneyso implementing this in the lvm or a diffent intree cinder driver would likely be a requirement if we did this for cinder only16:39
bauzasspec it is16:39
bauzasand I guess you probably need to discuss that with the cinder folks16:39
marlincOke I will look into creating a spec and also see if I can get this implemented in a in-tree driver16:40
bauzasthanks16:41
bauzasping me anytime if you require assistance on the paperwork process16:41
marlincI'm going to have to see how to implement the min compute service version check and bump16:41
sean-k-mooneymarlinc: i would proably add it to the lvm driver personally, i think it woudl be relvitly simple to do16:41
bauzasbefore doing anything with version checks, please check with the cinder folks about populating the rotation rate into the connection info details16:41
bauzasyeah, lvm seems the easiast approach16:42
marlinc(Also actually the functional test for live migration, honestly I have never implemented something so complex as I have no experience at all with the testing framework and how it works in Nova)16:42
bauzasbut that's not my own garden :)16:42
marlincEspecially since there is no existing volume based live migration functional I could see16:42
bauzasmarlinc: once you get a go from the cinder folks about exposing the rotation rate, come to me16:42
sean-k-mooneymarlinc: if cinder accpet the enhancement we can revisit that and or help you with that16:43
marlincAlright16:43
bauzascool16:43
bauzasmoving on then16:43
marlincThank you, I'll update our internal ticket16:43
bauzasmarlinc: I assume you know how to reach the cinder folks?16:43
marlincWell, I'm going to assume #openstack-cinder and I'll check their contribution documentation16:44
bauzasI have no potential design disapproval on using a specific connection info detail for setting the right value in the xml16:44
sean-k-mooneymarlinc: yep thats the correct channel16:45
bauzasbut you'll need to write a nova spec for explaining that it'll require a recent enough cinder and new computes16:45
sean-k-mooneyby the way there may be a usecase here for nova local storage too.16:45
marlincI was maybe a bit afraid though about the special rotation_rate 1 that is libvirt specific16:45
bauzasand testing will be interesting to be discussed in the spec proocess16:45
marlincBut that is probably something for the spec16:45
bauzasthe rotation rate is storage specific16:45
bauzashow you tune it for the guest is libvirt specific indeed16:46
marlincThat is why, right now we return rotation_rate 1 for SSDs, however that might not be smart from a cinder -> nova integration perspective16:46
marlincIt is a magic value16:46
sean-k-mooneywell really your tryign to say "this is an ssd" or not16:46
bauzaswe're overtime and we still have two topics to discuss16:46
sean-k-mooneyafnd there may be better way to express that16:47
bauzasbut I think we may need to discuss that from a cross-project perspective16:47
sean-k-mooneyya lets continue this converatoin after the meeting16:47
marlincYes that is right now the primary use case though we also use it to set it to 7200 for HDDs16:47
marlincAlright thank you16:47
bauzasnext one then (3 of 4)16:47
bauzasLuzi: floating ip behavior (assign fip: https://bugs.launchpad.net/neutron/+bug/2060808  ,   remove fip: https://bugs.launchpad.net/nova/+bug/2060812)16:47
Luzithese two bugs describe changes with the neutron handling of floating ips (compared to the deprecated nova code). 1. one allows allocating a floating ip to a vm even if it is attached to another vm ("stealing" it). The 2. one does not check the VM when removing a floating ip, resulting in always removing the ip (even if the vm-name was correct, and the ip a mistake)16:47
bauzasare you asking for reviews on the nova patch ?16:48
LuziI raised that at the PTG with Neutron, but they wanted Nova poeple to look over it, and tell, whether you want to change these bahaviors or not16:48
bauzasoh my bad16:48
Luzi(i normally don't have time to attend the Nova meetings)16:49
LuziI would be glad if you could check out these bugs again and give your input on it16:49
bauzasLuzi: well, it's hard to comment those bugs on a limited timeframe16:49
Luziyeah, we can discuss this on launchpad also16:50
bauzascould you please go back to the nova channel on another time ?16:50
Luzisure16:50
bauzasLuzi: well, what's your timezone, please remind me ?16:50
LuziUTC-116:50
bauzasokay, would that work if you would ping us again on tomorrow UTC afternoon ?16:51
Luziyeah i can do taht16:51
bauzasthanks16:51
bauzaslast one16:52
bauzaskgube: re-proposed extend volume completion spec: https://review.opendev.org/c/openstack/nova-specs/+/91713316:52
kgubeHi, this is just a review request16:52
bauzascool, we'll review all the proposed specs indeed16:52
kgubeThe spec has been reproposed from last cycle and not much has changed16:52
bauzasI think I need to remember the exact situation we had last cycle16:52
bauzasand why this didn't get enough traction16:53
bauzasI think we were basically awaiting cinder's feedback, right?16:53
kgubeThe cinder dependencies had to get merged16:53
bauzasyah that16:53
bauzasand the cinder spec was accepted ?16:53
kgubebut cinderclient now supports the feature16:53
bauzasokay, so where are we exactly on the cinder side ?16:54
bauzaseverything eventually got merged ?16:54
kgubewell, there are some cinder changes left, but they depend on the nova change again16:54
bauzasyeah16:54
bauzasI remember that16:54
bauzasbut okay, that's on our plate now16:55
bauzasI'll then review the spec reproposal16:55
sean-k-mooneythey depend on https://review.opendev.org/c/openstack/nova/+/873560 right16:55
kgubethanks!16:55
sean-k-mooneywhich was waiting for the cinder client release16:55
bauzasyeah16:55
sean-k-mooneywhich has now happened16:55
bauzasexactly16:55
sean-k-mooneyok then if that can be rebased16:56
bauzasI don't see anything controversial but I'll just doublecheck16:56
kgubeyeah, i still need to fix the build for this16:56
sean-k-mooneywe can review and then cidner can complete the rest16:56
bauzasbefore blindly reapproving16:56
bauzascool16:56
bauzasI think the dust has settled then16:56
bauzasare we good with wrapping up the meeting then ?16:57
bauzaslooks so16:58
bauzasthanks all16:58
bauzas#endmeeting16:58
opendevmeetMeeting ended Tue Jun  4 16:58:20 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:58
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2024/nova.2024-06-04-16.00.html16:58
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-06-04-16.00.txt16:58
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2024/nova.2024-06-04-16.00.log.html16:58
opendevreviewmelanie witt proposed openstack/nova master: Consolidate vTPM and ephemeral encryption secret creation  https://review.opendev.org/c/openstack/nova/+/91209423:17
opendevreviewmelanie witt proposed openstack/nova master: libvirt: Add optional 'description' kwarg to create_secret()  https://review.opendev.org/c/openstack/nova/+/91998623:17
opendevreviewmelanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/87093223:17
opendevreviewmelanie witt proposed openstack/nova master: Validate key manager create access in API  https://review.opendev.org/c/openstack/nova/+/91998923:17
opendevreviewmelanie witt proposed openstack/nova master: Clean up unused ephemeral encryption libvirt secrets  https://review.opendev.org/c/openstack/nova/+/91999023:17
opendevreviewmelanie witt proposed openstack/nova master: libvirt: "Auto heal" ephemeral encryption secrets on guest launch  https://review.opendev.org/c/openstack/nova/+/91999123:17
opendevreviewmelanie witt proposed openstack/nova master: Support (resize|cold migration) with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/87093323:17
opendevreviewmelanie witt proposed openstack/nova master: Support live migration with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/90551223:18
opendevreviewmelanie witt proposed openstack/nova master: Support rebuild with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/87093923:18
opendevreviewmelanie witt proposed openstack/nova master: Support encrypted source images for qcow2  https://review.opendev.org/c/openstack/nova/+/91999223:18
opendevreviewmelanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase  https://review.opendev.org/c/openstack/nova/+/87093623:18
opendevreviewmelanie witt proposed openstack/nova master: Add backing_encryption_secret_uuid to BlockDeviceMapping  https://review.opendev.org/c/openstack/nova/+/90796023:18
opendevreviewmelanie witt proposed openstack/nova master: Support encrypted backing files for qcow2  https://review.opendev.org/c/openstack/nova/+/90796123:18
opendevreviewmelanie witt proposed openstack/nova master: Support rescue with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/87367523:18
opendevreviewmelanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/87093723:18
opendevreviewmelanie witt proposed openstack/nova master: Support cross cell resize with ephemeral encryption for qcow2  https://review.opendev.org/c/openstack/nova/+/90959523:18
opendevreviewmelanie witt proposed openstack/nova master: libvirt: Introduce support for raw with LUKS  https://review.opendev.org/c/openstack/nova/+/88431323:18
opendevreviewmelanie witt proposed openstack/nova master: Remove use of (hw:|hw_)ephemeral_encryption_format  https://review.opendev.org/c/openstack/nova/+/92134323:18
opendevreviewmelanie witt proposed openstack/nova master: Create EncryptOptions object for ephemeral encryption  https://review.opendev.org/c/openstack/nova/+/92134423:18
opendevreviewmelanie witt proposed openstack/nova master: Add Glance standardized encryption image properties  https://review.opendev.org/c/openstack/nova/+/92134523:18

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!