| *** mhen_ is now known as mhen | 02:57 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | 09:13 |
|---|---|---|
| opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | 09:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466 | 10:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:39 |
| gibi | sean-k-mooney: dansmith: sambork: ^^ I had first light with nova-compute in threading mode. I was able to boot and delete VMs in a local devstack. Let see how this fails on the gate with more complex operations | 10:40 |
| sean-k-mooney | gibi: cool | 10:42 |
| sambork | gibi, nice! change looks less impressive than I think it would :) | 10:42 |
| sean-k-mooney | gibi: i have not looked yet but did you just make spawn a noop | 10:43 |
| gibi | I made the Tpools running the calls on the caller thread, and basically swiched all our executors to ThreadPools (already in tree) with the env variable. I needed to convince oslo.service not to fork a worker process for compute. And that is it. Compute just started working :) | 10:45 |
| gibi | probably fails in edge cases | 10:45 |
| gibi | but it boots and deletes VMs for me locally | 10:46 |
| sean-k-mooney | yep im reviweing th echange now i pushed qution to https://review.opendev.org/c/openstack/nova/+/965466 | 10:46 |
| gibi | I will dig a bit into the libvirt direct greenthread usage as that might only work just by luck | 10:47 |
| gibi | but the tpools are neutered | 10:47 |
| gibi | when running with native threading | 10:47 |
| sean-k-mooney | i dont see wehre you did that by the way | 10:47 |
| sean-k-mooney | oh | 10:47 |
| sean-k-mooney | its in earlier pathces | 10:48 |
| gibi | yepp earlier patches | 10:48 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/956089/12 and https://review.opendev.org/c/openstack/nova/+/956090/12 | 10:48 |
| sean-k-mooney | ya ok | 10:48 |
| gibi | one for the vfs and one for the libvirt tpool | 10:48 |
| sean-k-mooney | VFs? | 10:48 |
| gibi | hm | 10:49 |
| sean-k-mooney | do you mean rbd? | 10:49 |
| gibi | then I missed one | 10:49 |
| gibi | I did the rbd | 10:49 |
| gibi | but there is on in nova/virt/disk/vfs/guestfs.py:from eventlet import tpool | 10:49 |
| sean-k-mooney | oh virtual file systems | 10:49 |
| sean-k-mooney | not Virutal fucntions | 10:49 |
| sean-k-mooney | i was like how does sriov come into this | 10:49 |
| gibi | :) | 10:50 |
| gibi | anyhow there are more work to do | 10:50 |
| gibi | but I'm happy with this for now :) | 10:50 |
| sean-k-mooney | have you considerd putting a dnm with an eventlet poison on calls to monkey patch | 10:51 |
| sean-k-mooney | basiclly to prove that no one calls it in the job via a lib ectra | 10:51 |
| sean-k-mooney | importing eventlet is kind of ok provide its not monkey patch but tha twoudl be a nice addtion even if we dont watn to ship that | 10:52 |
| gibi | yeah I want to do that. I cannot do it right now as we still import it. But we are close to get rid of those imports. | 10:52 |
| sean-k-mooney | well im not saying poison the import | 10:52 |
| sean-k-mooney | im saying poison the eventlet.monkey_patch function | 10:53 |
| gibi | ahh OK | 10:53 |
| gibi | didn't we do that already? | 10:53 |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Update the api-ref for unshelve https://review.opendev.org/c/openstack/nova/+/938054 | 10:53 |
| sean-k-mooney | we coudl also https://opendev.org/openstack/nova/src/commit/7712e9193a2fc02454d73a43949a056b977076dd/nova/monkey_patch.py | 10:53 |
| gibi | https://github.com/openstack/nova/blob/30bf8c1025ea24823ebde3eacede4ea34ec6be8c/nova/monkey_patch.py#L115-L131 | 10:54 |
| sean-k-mooney | so that you first patch | 10:54 |
| sean-k-mooney | ya so your right | 10:54 |
| sean-k-mooney | we did | 10:54 |
| sean-k-mooney | so ok that good | 10:54 |
| sean-k-mooney | so it possibe a lib is doing eventlet.spawn | 10:54 |
| sean-k-mooney | we know that if you dont monkey patch and you call the restul object that cause the eventlet to run | 10:55 |
| sean-k-mooney | blockign the caller until complete | 10:55 |
| gibi | yeah | 10:55 |
| sean-k-mooney | i wonder if we could also poison eventlet.spawn? | 10:55 |
| gibi | so if the libvirt event handling works it works bec;se of that | 10:56 |
| sean-k-mooney | honestly this is probaly overkill | 10:56 |
| sean-k-mooney | lets see how ci fairs but this is great progress | 10:56 |
| sean-k-mooney | if we do what dan suggested for spliting the venvs into one with eventlet and one without | 10:57 |
| sean-k-mooney | we could slithgly update our code to do condtional imports | 10:57 |
| sean-k-mooney | and when eventlet is not there use the threaded mode even if no env var is set | 10:58 |
| nicolairuckel | Can anyone help me explaining the issue here? https://gitlab.com/libvirt/libvirt/-/issues/828 i was under the assumption that libvirt should take care of any permissions but either that's not the case or I'm not able to explain the problem well enough. | 10:58 |
| nicolairuckel | maybe jkulik? | 10:59 |
| sean-k-mooney | nicolairuckel: so honestly i need to trace the code for your orgianl patch and make sure we are not changing anything along the way | 10:59 |
| sean-k-mooney | i parlly wonder if this is an edge case with libvirt dynmaic ownership feature | 11:00 |
| sean-k-mooney | when you start a domain that is manage by libvirt libvier changes the ownwership of the root disk for example to be owned by qemu | 11:00 |
| sean-k-mooney | and then reverts it back when the domain is stopped | 11:00 |
| sean-k-mooney | i dont knwo if tis is doing something similar fo ht nvram file | 11:01 |
| sean-k-mooney | the nvram file as far as i am aware shoudl nto be owned by nova | 11:01 |
| sean-k-mooney | i shoudl be owned by either swtpm user or qemu | 11:01 |
| nicolairuckel | You mean those could still be issues caused by the first version of my patch? | 11:01 |
| sean-k-mooney | im unsure but that or soemthign else nova is doing that predates your patch | 11:02 |
| nicolairuckel | I understand. | 11:02 |
| nicolairuckel | The owner and group of the file are the same as for the disk (libvirt-qemu and kvm respectively) if that helps. | 11:03 |
| nicolairuckel | So at least that looks correct to me. | 11:04 |
| sean-k-mooney | if you dont change this how does it break | 11:13 |
| sean-k-mooney | i.e. if you dont update form 600 to 644 | 11:13 |
| sean-k-mooney | i assume the swtpm cant acess it | 11:13 |
| sean-k-mooney | what i think we need to do is test this on a clean env | 11:14 |
| sean-k-mooney | ill see if i can set up a devstack today on a clean vm and look at the owner ship after the inial spawn in some detail | 11:14 |
| sean-k-mooney | i think 640 woudl make sense assuming the swtpm user and qemu user and in the same group | 11:14 |
| jkulik | but we got some information that we were looking for, right? | 11:17 |
| jkulik | > When shutting down the VM libvirt also ought to return the permissions the file had before the VM was started. | 11:17 |
| jkulik | so trying to figure out if we change the permissions somewhere, would be a good next step | 11:17 |
| sean-k-mooney | jkulik: nicolairuckel i have been talkign to the libvirt maintainer interlly and they dont see a clear way that this could be a libvirt bug but also 600 should be ok | 11:20 |
| sean-k-mooney | provide the libvirt dynmaic ownership feature is enabeld | 11:20 |
| sean-k-mooney | nicolairuckel: do you have that disabeld in your env | 11:20 |
| sean-k-mooney | jkulik: 600 shoudl be valid provided the file si chown'd to be owned by qemu before qemu starts | 11:21 |
| jkulik | sean-k-mooney: I get that. I thought the point was, that it doesn't get re-chowned after the VM is shut down. | 11:26 |
| sean-k-mooney | jkulik: we need to confirm the behvior and who is creatign the nvram file | 11:26 |
| sean-k-mooney | and with what permisson | 11:27 |
| jkulik | yes. I agree with that. | 11:27 |
| jkulik | > If the NVRAM image is created by libvirt on first start of the VM it will have 600 permissions and will (originally) be owned by root; which will be restored after the VM is turned off. | 11:29 |
| jkulik | ^ from the ticket. sounds plausible to me. when are we creating the NVRAM - do we rely on libvirt? | 11:29 |
| nicolairuckel | I have a stacktrace of the error message in the libvirt issue, sean-k-mooney | 11:30 |
| nicolairuckel | I need to look into the dynamic ownership since my colleague set up the Dev Stack. | 11:31 |
| sean-k-mooney | nicolairuckel: lookign at the env its not supproted anymore | 11:33 |
| sean-k-mooney | python 3.10 is | 11:33 |
| sean-k-mooney | but your using ubuntu 22.04 | 11:33 |
| sean-k-mooney | which is not part of the testign runtimes | 11:33 |
| sean-k-mooney | you are using libvirt 8.0.0 which is our current min version https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L220 | 11:34 |
| sean-k-mooney | but its possible that there is some odd interaction here happeing with the version | 11:35 |
| sean-k-mooney | im goign to try and replciate this on ubuntu 24.04 and see what happens | 11:35 |
| sean-k-mooney | we will or likely shoudl move to libvirt 10.0.0 as our min version this cycle | 11:36 |
| sean-k-mooney | taht or current NEXT_MIN_LIBVIRT_VERSION | 11:36 |
| opendevreview | OpenStack Release Bot proposed openstack/nova master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/nova/+/965491 | 11:38 |
| opendevreview | OpenStack Release Bot proposed openstack/os-vif master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/os-vif/+/965493 | 11:38 |
| opendevreview | OpenStack Release Bot proposed openstack/osc-placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/osc-placement/+/965495 | 11:38 |
| opendevreview | OpenStack Release Bot proposed openstack/placement master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/placement/+/965497 | 11:39 |
| nicolairuckel | sean-k-mooney, thank you! | 11:39 |
| sean-k-mooney | that raises the minum to the minium we test with i.e. what ship in the disto with the oldest livbirt cover by the testing runtimes which siubuntu 24.04 | 11:39 |
| opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: reno: Update master for unmaintained/2024.1 https://review.opendev.org/c/openstack/python-novaclient/+/965500 | 11:39 |
| opendevreview | Masanori Kuroha proposed openstack/nova master: Copy applied provider config https://review.opendev.org/c/openstack/nova/+/948304 | 11:45 |
| gibi | bauzas: dansmith: can I get a second set of eyes on this to readd +2 https://review.opendev.org/c/openstack/nova/+/937275 The devstack dependency is merged | 12:15 |
| gibi | this is also a pretty simple test only change around the same topic https://review.opendev.org/c/openstack/nova/+/963036 | 12:16 |
| bauzas | Uggla: fwiw the slot is not booked for nova | 14:01 |
| bauzas | grizzly-FriB1 is empty | 14:01 |
| dansmith | looks like it is to me | 14:01 |
| Uggla | bauzas sounds good to me | 14:02 |
| bauzas | oh doh, my maths | 14:02 |
| bauzas | this is 2pm UTC now | 14:03 |
| Uggla | I have not booked grizzly-FriB5 because I have the secret expectation we will manage to finish early. | 14:03 |
| * bauzas ragesquits about daylight savings | 14:03 | |
| bauzas | Uggla: I'll need to skip the last hour then for us due to the tc meeting | 14:03 |
| Uggla | ok | 14:04 |
| gibi | sean-k-mooney: we have some tempest results with threaded compute. It is not good but not terrible https://review.opendev.org/c/openstack/nova/+/965467/1#message-518b76afc6b252d2f42c48d65f69f6570d9ecb8e | 14:19 |
| gibi | e.g. we have passing complex tests | 14:20 |
| gibi | 2025-10-31 13:07:09.576696 | controller | {2} tempest.scenario.test_instances_with_cinder_volumes.TestInstancesWithCinderVolumes.test_instances_with_cinder_volumes_on_all_compute_nodes [348.575935s] ... ok | 14:20 |
| nicolairuckel | sean-k-mooney, does https://gitlab.com/libvirt/libvirt/-/issues/828#note_2858916854 mean it works with a more recent Ubuntu version? | 14:30 |
| sean-k-mooney | nicolairuckel: i have not tried your patch yet | 14:33 |
| sean-k-mooney | but that confirm it shoudl not need to be 644 | 14:33 |
| sean-k-mooney | 600 shoudl be enough | 14:33 |
| sean-k-mooney | nicolairuckel: ill pull in your patch a bit later today when i get time | 14:33 |
| nicolairuckel | I see | 14:34 |
| sean-k-mooney | what this is telling me is your patch should not need to chown or chmod the file | 14:34 |
| nicolairuckel | That's something. | 14:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Allow to perform parallel live migrations https://review.opendev.org/c/openstack/nova/+/955784 | 15:27 |
| bauzas | oh doh, the TC session is also very interesting | 15:55 |
| * bauzas feels stuck in between walls | 15:55 | |
| sean-k-mooney | bauzas:oh the crypto stuff | 16:01 |
| sean-k-mooney | bauzas: not very impornat | 16:01 |
| sean-k-mooney | we will need to adress post quantum crypto eventually | 16:01 |
| sean-k-mooney | but not in 2026.1 | 16:01 |
| sean-k-mooney | my take is we finish the eventlet removal work first then we can think about how to adapt the minimal part of nova that will be impacted | 16:02 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make libvirt Tpool proxying conditional https://review.opendev.org/c/openstack/nova/+/956090 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [vncproxy]Handle ssl.wrap_socket removal in py312 https://review.opendev.org/c/openstack/nova/+/955915 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Do not fork compute workers in native threading mode https://review.opendev.org/c/openstack/nova/+/965466 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make guestfs' Tpool usage optional https://review.opendev.org/c/openstack/nova/+/965821 | 16:03 |
| opendevreview | Merged openstack/nova master: [CI][nova-next]test with placement ac optimizations https://review.opendev.org/c/openstack/nova/+/937275 | 16:29 |
| opendevreview | Merged openstack/nova master: [func]Test with optimize_for_wide_provider_trees https://review.opendev.org/c/openstack/nova/+/963036 | 16:30 |
| dansmith | Uggla: is the kid noise coming from you? | 17:29 |
| Uggla | dansmith, I apologize, forget to mute... | 17:30 |
| Uggla | dansmith thanks for letting me know | 17:30 |
| bauzas | Uggla: still in the TC meeting, but I see nova done for the PTG, I guess I won't need to join once the session ends at 5:40pm UTC ? | 17:33 |
| Uggla | I next still cyborg. | 17:34 |
| Uggla | s/I/in/ | 17:34 |
| Uggla | I mean, I have set next to end. But we are still discussing cyborg | 17:34 |
| Uggla | I guess we are not far from the end. | 17:35 |
| bauzas | ack, I'll try to join as soon as the TC session stops | 17:38 |
| bauzas | Uggla: well, false promise I made :) Happy Halloween everyone :) | 18:03 |
| Uggla | oh yes, Happy Halloween ! | 18:13 |
| cardoe | Uggla: I hear you're trying to get bauzas to drink the Rust Kool-aid. :-D | 18:59 |
| Uggla | cardoe, yes I'm a rust big fan, so trying to communicate the love. | 19:04 |
| opendevreview | melanie witt proposed openstack/nova master: Add hw:tpm_secret_security extra spec validation https://review.opendev.org/c/openstack/nova/+/940197 | 21:13 |
| opendevreview | melanie witt proposed openstack/nova master: Add handling for vTPM secret permission error https://review.opendev.org/c/openstack/nova/+/963648 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `user` secret security https://review.opendev.org/c/openstack/nova/+/942502 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `host` secret security https://review.opendev.org/c/openstack/nova/+/941795 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: Add vtpm_secret_(uuid|value) to LibvirtLiveMigrateData https://review.opendev.org/c/openstack/nova/+/952628 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `host` secret security https://review.opendev.org/c/openstack/nova/+/941483 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: bump service version and require it for live migration https://review.opendev.org/c/openstack/nova/+/962051 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support instances with `deployment` secret security https://review.opendev.org/c/openstack/nova/+/942021 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: support live migration of `deployment` secret security https://review.opendev.org/c/openstack/nova/+/925771 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: test live migration between hosts with different security https://review.opendev.org/c/openstack/nova/+/952629 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: add late check for supported TPM secret security https://review.opendev.org/c/openstack/nova/+/956975 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: opt-in to new TPM secret security via resize https://review.opendev.org/c/openstack/nova/+/962052 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: TPM: add documentation and reno for live migration https://review.opendev.org/c/openstack/nova/+/962889 | 21:14 |
| opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | 21:14 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!