| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 02:50 |
|---|---|---|
| opendevreview | Bhavana proposed openstack/nova-specs master: Add spec for MIG-aware PCI Host Weigher https://review.opendev.org/c/openstack/nova-specs/+/998753 | 05:18 |
| opendevreview | chandan kumar proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | 05:54 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | 06:06 |
| opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Add test for MTU overwrite during live migration https://review.opendev.org/c/openstack/nova/+/995803 | 08:07 |
| opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Preserve source MTU during live migration https://review.opendev.org/c/openstack/nova/+/995809 | 08:36 |
| opendevreview | Lajos Katona proposed openstack/nova master: Add regression test to repoduce bug 1854844 https://review.opendev.org/c/openstack/nova/+/991772 | 08:36 |
| gibi | sean-k-mooney: thanks for the TDX review, and good catch on the cgroups problem. I agree with having a config option until libvirts add capacity reporting | 08:44 |
| gibi | I will test out scsi support on my machine | 08:44 |
| sean-k-mooney | ack | 08:45 |
| gibi | and I confirmed that serial access works | 08:45 |
| sean-k-mooney | that good | 08:45 |
| sean-k-mooney | at least one of the concoles will | 08:45 |
| gibi | yeah :) | 08:45 |
| gibi | super secure VM, you cannot even access it :) | 08:45 |
| sean-k-mooney | :) | 08:45 |
| gibi | nobody can. done. | 08:45 |
| sean-k-mooney | most secure | 08:45 |
| sean-k-mooney | so i was debating soemting as well | 08:46 |
| sean-k-mooney | if you have enabel tdx | 08:46 |
| sean-k-mooney | shoudl we automaticlly select hw_video_model="none" for you | 08:46 |
| sean-k-mooney | if its not set in the image | 08:46 |
| sean-k-mooney | technially when its not set in the image the libvirt driver choose for you | 08:47 |
| sean-k-mooney | it used to chosoe cirros and we updated it to virtio-gpu | 08:47 |
| sean-k-mooney | but it could take into account tdx | 08:47 |
| gibi | hm that is not a bad idea. I would say we can add that as a follow up | 08:48 |
| sean-k-mooney | yep can be a followup | 08:48 |
| gibi | currently we document that you need to set none explicitly, and we can relax that later | 08:48 |
| gibi | that is a nice UX improvement | 08:48 |
| sean-k-mooney | the real reason i said this feature feel like it shoudl be experimatal is the amount of specifcal casing you need to do to make it work at all combiend with the fact it only works with host-passthough | 08:48 |
| sean-k-mooney | that last part makes me quite uncofrrotable | 08:49 |
| sean-k-mooney | i like host-passthough for performace | 08:49 |
| sean-k-mooney | but we have never had a feature that only works when that is used before | 08:49 |
| sean-k-mooney | so i think if you ahve not selected host-passtouhg we shoudl likely avoid reporting tdx supprot as a trait | 08:50 |
| sean-k-mooney | even when the host supprot it | 08:50 |
| sean-k-mooney | with that sait | 08:50 |
| sean-k-mooney | *said | 08:50 |
| sean-k-mooney | im hoping the host-passthough requirement is tempory | 08:50 |
| sean-k-mooney | so if future qemu cpu models supprot tdx | 08:51 |
| sean-k-mooney | then we dont want the extra code in nova to filter the triats | 08:51 |
| opendevreview | Lajos Katona proposed openstack/nova master: libvirt: Preserve queue_size during live migration https://review.opendev.org/c/openstack/nova/+/995841 | 08:51 |
| sean-k-mooney | taht why i didnt suggest that in the review but ya it makes me a bit uneasy | 08:51 |
| gibi | I'm not sure that host-passthrough requirement is valid on cpu level or just a consequence of this bug https://bugs.launchpad.net/nova/+bug/2158927/comments/1 | 08:53 |
| gibi | antia: ^^ could you confirm which is the case? | 08:54 |
| gibi | sean-k-mooney: antia: fyi I'm on PTO Thru/Fri this week but I'm back on Monday | 08:55 |
| sean-k-mooney | gibi: ack | 08:55 |
| gibi | we have FF next Thursdays I assuem | 08:55 |
| sean-k-mooney | so cpu_models = Nehalem is an intel model | 08:55 |
| gibi | assume | 08:55 |
| sean-k-mooney | and it normlaly works on amd | 08:56 |
| sean-k-mooney | but i have seen it not work in some cases | 08:56 |
| sean-k-mooney | have you tested host-model | 08:56 |
| antia | amd has a cpu model with AMD SEV-SNP support (EPYC-v4), but there is none with Intel TDX support | 08:56 |
| gibi | in this case it triggers the feature precondition check without proper image meta passed causing the issue in that bug | 08:56 |
| antia | TDX requires qemu to have -cpu host which is where the requirement for host-passthrough comes from | 08:58 |
| sean-k-mooney | antia: but is that being worked on to fix in qemu | 08:58 |
| sean-k-mooney | antia: its totally fine if that is a limaition in the current qemu | 08:59 |
| sean-k-mooney | but long term that is not somethign we shoudl continue to require | 08:59 |
| sean-k-mooney | we generaly try not ot add feature that do not supprot live migration | 08:59 |
| sean-k-mooney | the excption to that is if its a hardare limiation | 09:00 |
| sean-k-mooney | which this mostly is today | 09:00 |
| sean-k-mooney | (mix of hardare and qemu limiation) | 09:00 |
| sean-k-mooney | as long as there is a path to eventuly supproting live migration with TDX or SEV instance that fine but i woudl hate to force operator to choose between secure vms and 5 9s of uptime for workloads | 09:01 |
| gibi | yeah regardless of the model support live migration is not supported for SEV or TDX | 09:01 |
| sean-k-mooney | gibi: by the way, our current cirros image dont actully supprot uefi boot | 09:03 |
| sean-k-mooney | you can enabel it but the guest wont actully boot | 09:03 |
| sean-k-mooney | qemu/libirt will boot the vm but i think the image just dont have the efi partion requried and the relevent grub config | 09:04 |
| gibi | I use an ubuntu image locally to be able to try attestation as that anyhow needs an extra kernel module in each case | 09:04 |
| sean-k-mooney | ya so on that | 09:04 |
| sean-k-mooney | im kind wish we didnt enable that as part of this feature | 09:04 |
| sean-k-mooney | that really fells like it shoudl have been out of scope | 09:04 |
| antia | removing the need for -cpu host was marked as future work, but not sure where it is today | 09:05 |
| gibi | you mean attestation? | 09:05 |
| antia | removing the need for -cpu host was marked as future work | 09:05 |
| antia | ops | 09:05 |
| antia | https://lists.nongnu.org/archive/html/qemu-devel/2025-04/msg00097.html | 09:05 |
| sean-k-mooney | yes enabel attestation in a portable way is a seprate feature in my view | 09:05 |
| gibi | sean-k-mooney: sure nova does nothing really with attestation | 09:05 |
| sean-k-mooney | so i woudl have wanted that to be its own spec adn have its own extra spec | 09:05 |
| sean-k-mooney | gibi: but we used too | 09:05 |
| sean-k-mooney | we had supprot for opens attstation and trusted boot before | 09:06 |
| gibi | the sev one works out of the box via the cpu itself. the tdx works via a local socket added automatically by libvirt to the guest | 09:06 |
| sean-k-mooney | and we riped it out | 09:06 |
| gibi | so we don't carry any code | 09:06 |
| sean-k-mooney | and we have been asked looks at supprot keylime in the future | 09:06 |
| gibi | and documented nothing | 09:06 |
| gibi | regarding attestation | 09:06 |
| sean-k-mooney | gibi: rgith btu we dotn supprot atested boot today | 09:06 |
| sean-k-mooney | meaning you cant file a bug agasint nova if it does not work | 09:07 |
| antia | this is not attested boot though | 09:07 |
| gibi | sure, that is fine by me :) | 09:07 |
| sean-k-mooney | as long as we dont have to fix bugs related to attestation for tdx im ok with it | 09:07 |
| antia | yeah that is more or less why it was included in the documentation to remove any expectation of Nova doing attestations | 09:08 |
| sean-k-mooney | antia: if the tdx attestation feature is not related to messered secure boot what is it | 09:08 |
| sean-k-mooney | antia: well the docs dont say that its unsuprpoted right. can we just not allcoate teh socket? | 09:08 |
| sean-k-mooney | so that thing we are not testing or cliaming to supprot is not enabled | 09:09 |
| gibi | sean-k-mooney: I would let libvirt do the default that allows anybody who want to try a chance to make it work | 09:10 |
| sean-k-mooney | so i dont htink these type of side channle shoud lbe aviabel to folks to hook up like that | 09:11 |
| sean-k-mooney | like if live migration for exmaple get implemtne only when the socket is not used | 09:11 |
| sean-k-mooney | we woudl not want ot have to chosoe betwen possiebl breakign folks that set it up and making live migration work | 09:11 |
| gibi | we expect that if we use the libvirt default then that will work with future features | 09:11 |
| gibi | what if not having the socket will break the live migration/ | 09:12 |
| gibi | ? | 09:12 |
| gibi | we should not do hypoteticals. Use the underlying default is the safest in my eyes | 09:12 |
| sean-k-mooney | its not that the socket will break it its we are enabling a feature that we dont know the side effect of | 09:12 |
| gibi | we are not enabling it, libvirt does. | 09:13 |
| sean-k-mooney | so we are not addign it to the domain | 09:13 |
| sean-k-mooney | ok | 09:13 |
| gibi | and the whole reason to have these features is to be able to prove that your VM is secure | 09:13 |
| sean-k-mooney | no | 09:14 |
| gibi | so blocking the possibility of that actively is against the goal | 09:14 |
| sean-k-mooney | its to encypt the guest memory | 09:14 |
| sean-k-mooney | proving its secure is its own feature | 09:14 |
| gibi | but how would you know that it is encrypted? | 09:14 |
| antia | encrypting the guest memory is somewhat useless unless it can be proven | 09:14 |
| sean-k-mooney | that is my point attestation of any kind shoudl ahve been its own dedicated feature | 09:14 |
| sean-k-mooney | gibi: im not sure you shoudl knwo that as a normal user or how | 09:15 |
| sean-k-mooney | beyond the flavor extra spec if that is expsoed to you in the releven tcloud | 09:15 |
| gibi | but the premise is that you don't trust the provider so you need a way to be safe | 09:15 |
| gibi | so if the provider tells you it is encrypted you wont trust that either | 09:16 |
| sean-k-mooney | this socket conenct to a deamon run on the host by the provdier | 09:16 |
| sean-k-mooney | you cant trust that eitehr | 09:16 |
| gibi | I guess there are mechanism to make that work. I guess it is just the channel not the source of truth | 09:17 |
| antia | this goes beyond the scope I think. Intel has documentation on this process | 09:17 |
| sean-k-mooney | antia im trying not to second guess the spec | 09:18 |
| gibi | sean-k-mooney: if this helps I'm OK to say in the doc that nova does not actively support attestation. So we can close bugs. But actively preventing attestation I'm not OK with | 09:18 |
| sean-k-mooney | im just highlihgt that i woudl have ask for this to be remvoed if i had reviewed the spec | 09:18 |
| sean-k-mooney | so im highlighting the maintaince concern here not saying we ahve to remove it | 09:18 |
| sean-k-mooney | gibi: yes that woudl help | 09:18 |
| gibi | then lets do that | 09:18 |
| antia | that I can add | 09:18 |
| gibi | thanks both of you | 09:19 |
| sean-k-mooney | gibi: im just sentitve to the fact that folks have come to depend on the internals of some feature that we didnt intnet to be public in the past | 09:19 |
| sean-k-mooney | i.e. cloud-init incorrectly usign our dmi/smbios code to detech that its oepnstack | 09:19 |
| gibi | I think in case of AMD you cannot even prevent attestation from libvirt if encryption is on | 09:20 |
| sean-k-mooney | even though that only ever worked for libvirt | 09:20 |
| sean-k-mooney | ack | 09:20 |
| sean-k-mooney | antia: for what its woth the feature more or less looks ok | 09:21 |
| sean-k-mooney | the main blocker for me is you missing the libvirt version check and the cgroups issue i mentioned in teh review | 09:21 |
| antia | thanks for reviewing! | 09:21 |
| sean-k-mooney | antia: once those 2 issues are adressed i think we can more or less proceed with the series | 09:21 |
| antia | the cgroup issue I am resolving right now | 09:22 |
| sean-k-mooney | cool | 09:22 |
| gibi | sean-k-mooney: about the version check | 09:23 |
| antia | I am not strictly against a version check for TDX, but I find it unnecessary. The same isn't done for SNP and it is not needed since Libvirt capabilities can handle it | 09:23 |
| gibi | yeah ^^ | 09:23 |
| gibi | so libvirt only reports the domcap if libvirt + qemu is supporting the feature | 09:24 |
| sean-k-mooney | oh we already have that supproted via domcaps | 09:24 |
| gibi | yeah | 09:24 |
| sean-k-mooney | and we are usign that to fileter the traits reporitng | 09:24 |
| gibi | the flag yes, the capacity does not | 09:24 |
| sean-k-mooney | ok then jsut the cgroups issue | 09:24 |
| gibi | yepp | 09:24 |
| antia | okay good | 09:24 |
| sean-k-mooney | sorry i missed that | 09:24 |
| gibi | no worries | 09:25 |
| gibi | so about scsi. On my machine the VM is created successfully but cannot boot. It stops at the EFI setup not seeing a boot device | 09:40 |
| antia | that is consistent with my previous experience | 09:56 |
| sean-k-mooney | that is becaus of the qemu firmware image yes? | 10:27 |
| sean-k-mooney | so it may or may not work dependin on that | 10:27 |
| sean-k-mooney | im not sure if we shoudl relaly block it in nova but we shoudl call it out in the docs | 10:27 |
| gibi | yepp call it out in the doc | 10:39 |
| gibi | (and yes I tested without TDX that the same image boots from scsi) | 10:40 |
| sean-k-mooney | ack, i partly wonder if we are jsut mising setting somethign liek the iommu options we recnelty stop setting for sev | 10:40 |
| sean-k-mooney | but i guess we can follow up with improvment next cycle if we discover a way to supprot it | 10:41 |
| sean-k-mooney | and jsut treat it as a minor bugfix | 10:41 |
| gibi | OK | 10:41 |
| opendevreview | Bhavana proposed openstack/nova-specs master: Add spec for MIG-aware PCI Host Weigher https://review.opendev.org/c/openstack/nova-specs/+/998753 | 11:16 |
| sean-k-mooney | im generally not sure ^ is something that is really in nova scope now | 11:52 |
| sean-k-mooney | cybrog will need a way to provide its own way to experss schdulign prioties in the future | 11:53 |
| sean-k-mooney | be it a weigher or somethign more generic in nova | 11:53 |
| sean-k-mooney | but if we are actully intending this fucntionatly to go out of nova then addign mig aware ness to nova is questionable | 11:54 |
| sean-k-mooney | nova can mange vgpus on PGUS configure with mig | 11:54 |
| sean-k-mooney | but its is not aware mig is in use | 11:54 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [DNM]: Troubleshoot race condition in unshelve vif plug https://review.opendev.org/c/openstack/nova/+/1001294 | 12:35 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [WIP]Add task_state = None as pre-condition of unshelve https://review.opendev.org/c/openstack/nova/+/1001485 | 12:35 |
| gibi | gmaan: ^^ I have hunch based on your last comment | 12:35 |
| sean-k-mooney | unshivlign while still shevle offloading? | 12:36 |
| gibi | yepp | 12:36 |
| gibi | it seems like we have a race | 12:36 |
| gibi | the offloading removes the instance.host while unshelve is already claimed | 12:36 |
| sean-k-mooney | ya i can see that being a problem | 12:36 |
| sean-k-mooney | also ya addign task state to the decoreato rmakes seesne | 12:36 |
| gibi | and it might happen more often in the grenade job as it uses a single conductor setup instead of a super and a cell conductor | 12:36 |
| gibi | so the two instance.save happens in the same conductor | 12:37 |
| sean-k-mooney | maybe although i dont think that would make it that much slower with onloy 1-2 vms | 12:37 |
| sean-k-mooney | *1-2 computes | 12:37 |
| gibi | yeah, I'm just guessing here | 12:37 |
| sean-k-mooney | i think the race hunch is a good one | 12:37 |
| sean-k-mooney | jus tnot entirly sure why its more common in grenade | 12:38 |
| gibi | we will see. if the precondition makes the tempest test fail early that will prove it | 12:38 |
| gibi | it can be something with threading + single conductor as well. I think only grenade is doing that now | 12:38 |
| sean-k-mooney | is the tempest tst properly waiting for it to be shleve offloaded or just unshilving straight away | 12:39 |
| gibi | threading allows more random task switching | 12:39 |
| gibi | sean-k-mooney: even if the tempest waits for the shelve_offload state we might set that early with a task state | 12:39 |
| gibi | let me check | 12:39 |
| sean-k-mooney | gibi: well im thinking that tempest shoudl retry the unshleve | 12:39 |
| sean-k-mooney | at least on a 409 | 12:40 |
| sean-k-mooney | which is what that decoreator shoudl be returning | 12:40 |
| sean-k-mooney | since the transtion form shelve to shelve offloaded is configurabel and depend on if its BFV or BFI | 12:40 |
| sean-k-mooney | or using ceph | 12:41 |
| gibi | https://github.com/openstack/tempest/blob/fa340b1f5fff8f57a018be206644c81b6d36b3d4/tempest/common/compute.py#L388-L400 | 12:41 |
| sean-k-mooney | so i dont think we can write the test to know which to expect | 12:41 |
| gibi | so it depends on the tempest config what we are waiting for | 12:41 |
| sean-k-mooney | but it can retyr the unshelve on 409 conflict and still be correct | 12:41 |
| gibi | I can imagine we end up only waiting for SHELVED state and issue an unshelve | 12:42 |
| sean-k-mooney | yep | 12:42 |
| sean-k-mooney | but as i said for images_type=rbd and bfv we go drictly to shelve_offloaded i think | 12:42 |
| gibi | OK we will see and if it is the case then we need to fix nova and tempest as well | 12:42 |
| sean-k-mooney | regradless fo the offload time | 12:42 |
| sean-k-mooney | ya we should have ci reulst in a while and we can see | 12:43 |
| gibi | yepp | 12:45 |
| dansmith | gibi: I just +2'd this if you want to re-add yours after the test coverage fix: https://review.opendev.org/c/openstack/nova/+/998571 | 14:15 |
| gibi | dansmith: ohh sure. Thanks for the reviews there | 14:23 |
| gibi | Uggla: I left some feedback in the repro and fix in https://review.opendev.org/c/openstack/nova/+/999978 | 14:30 |
| gibi | I agree this is a valid repro but for the fix we might need to make a step back and see if we can make a wider fix | 14:30 |
| gibi | having an instance without task state and without instance.host during a period of evac sounds a source of different issuees | 14:30 |
| gibi | Uggla: on the same note I will need to drop at 17:45 latest from the upstream triage call today | 14:45 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [DNM]: Troubleshoot race condition in unshelve vif plug https://review.opendev.org/c/openstack/nova/+/1001294 | 14:47 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [WIP]Add task_state = None as pre-condition of unshelve https://review.opendev.org/c/openstack/nova/+/1001485 | 14:47 |
| gibi | grr ^^ that was in merge conflict while I thought I just waiting for the CI result | 14:47 |
| Uggla | Upstream triage meeting | 15:29 |
| Uggla | meet.google.com/zjr-rxus-hzj | 15:29 |
| gmaan | gibi: thanks, I was thinking to do some check on conductor side but this is better | 15:48 |
| gmaan | gibi: sambork sorry i missed eventlet meeting today. if anything for me to review, please let me know. | 15:49 |
| sambork | np it was mostly about perf test | 15:52 |
| gmaan | ack | 15:55 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 16:50 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 17:11 |
| melwitt | gmaan: can you remind me if there is a fix idea for https://bugs.launchpad.net/nova/+bug/2160254 ? anecdotally I'm seeing a lot of failures on patches I'm following | 18:01 |
| gmaan | melwitt: not yet, I did not debug it yet. most probably I will be able to get on this after FF | 18:02 |
| gmaan | is it happening too frequently ? | 18:03 |
| melwitt | seems so but that is just my personal observation on patches I'm following. I guess I should check opensearch | 18:03 |
| gmaan | k, let me see if I can get some time today. it may get worst during FF week :) | 18:05 |
| melwitt | gmaan: looks like it's just my bad luck, I see only one hit per day at most | 18:06 |
| gmaan | ok, I used to see it mulitple times in the week when i reported bug but agfter that not much occurance, but thanks for reminder, i will see if there is any obious mistake in test | 18:07 |
| gmaan | gibi: i saw notification failure in functional threading job, it seems 'start' migration event happened instead of 'end' https://zuul.opendev.org/t/openstack/build/07c0f0aecebb42fc99ec209ad6a9df3e/log/job-output.txt#13762 | 19:17 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Create services required thread pool executors during service initialization https://review.opendev.org/c/openstack/nova/+/1001572 | 20:56 |
| gmaan | dansmith: gibi: ^^ this is the further refactoring of thread pool executors as discussed in original change | 21:01 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211 | 21:20 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!