| *** ricolin_ is now known as ricolin | 00:16 | |
| opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | 00:44 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | 00:48 |
| opendevreview | melanie witt proposed openstack/nova master: unified limits: Fix openstacksdk usage for endpoint discovery https://review.opendev.org/c/openstack/nova/+/975106 | 01:27 |
| *** mhen_ is now known as mhen | 02:39 | |
| opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | 07:23 |
| opendevreview | huanhongda proposed openstack/nova master: Live migration with iothreads https://review.opendev.org/c/openstack/nova/+/975000 | 07:27 |
| opendevreview | huanhongda proposed openstack/nova master: numa: Respect allocation ratios for non-pinned NUMA instances https://review.opendev.org/c/openstack/nova/+/971409 | 08:07 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | 10:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | 10:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | 10:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | 10:35 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | 10:35 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:35 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | 10:35 |
| gibi | bauzas: now we have the executor shutdown at compute service stop ^^ | 10:36 |
| bauzas | all cool, doing some backport work atm but I'll try to look at it today | 10:37 |
| gibi | OK thanks | 10:40 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: SubclassSignatureTestCase to use NoDBTestCase as base https://review.opendev.org/c/openstack/nova/+/974861 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | 10:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Enable mypy on nova/utils.py https://review.opendev.org/c/openstack/nova/+/969936 | 10:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | 10:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | 10:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 10:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | 10:57 |
| gibi | fwiesel, sean-k-mooney : I see the issue behind the stack trace in the oslo.vmware native threaded run. This is probably a bug with the libvirt driver too. I will do the needful | 13:43 |
| sean-k-mooney | gibi: ack | 14:07 |
| sean-k-mooney | what was it | 14:07 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975162 | 14:27 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975163 | 14:27 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975164 | 14:29 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2025.1: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975165 | 14:29 |
| gibi | sean-k-mooney: https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/utils.py#L605 this should be conditional as not all executors has a name | 14:32 |
| gibi | yet another refactoring (for the future) to ensure all our executors are named and created in the same way | 14:33 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Reproduce bug/2130881 https://review.opendev.org/c/openstack/nova/+/975166 | 14:33 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2024.2: Init virt driver before use https://review.opendev.org/c/openstack/nova/+/975167 | 14:33 |
| sean-k-mooney | yep i was just going to say | 14:33 |
| sean-k-mooney | that or well we shoudl jsut not depend on it for the warning | 14:33 |
| opendevreview | sean mooney proposed openstack/nova master: Add regression test to repoduce bug 2139351 https://review.opendev.org/c/openstack/nova/+/975171 | 15:24 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 15:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | 15:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | 15:39 |
| gibi | sean-k-mooney: btw there are bigger issues in the vmware logs than that stack trace. | 15:44 |
| gibi | [10197.783268] env[89459]: WARNING nova.utils [None req-08c5913a-0cad-45f4-991c-76f091773eb3 tempest-ServersTestJSON-388458959 tempest-ServersTestJSON-388458959-project-member] The MainProcess.default pool does not have free threads so the task <function ComputeManager.build_and_run_instance.<locals>._locked_do_build_and_run_instance at 0x7647d040cee0> will be queued. If this happens repeatedly then | 15:44 |
| gibi | the size of the pool is too small for the load or there are stuck threads filling the pool. | 15:44 |
| gibi | this is the reason why VM builds are failing. Something is using up our thread pool | 15:45 |
| sean-k-mooney | gibi: i notice a few other tracebaces ya but i was nto sure wht are "expected error" vs related to event let liek the "expected" os-brick errors | 15:45 |
| sean-k-mooney | so to get the message that means somting is callign spawn right? | 15:46 |
| sean-k-mooney | as in it can only get to that executor by going thorugh the nova.utils metods | 15:46 |
| sean-k-mooney | which means it has to be in hte nova side fo the vmware driver rather then oslo.vmware | 15:47 |
| gibi | yepp _locked_do_build_and_run_instance calling the spawn. That is normal, but somewho we run out of workers in the pool so somethign is hanging | 15:47 |
| gibi | https://github.com/openstack/nova/blob/59a7093915298973c72b6d1749a6acd27e0045a9/nova/compute/manager.py#L2455 | 15:48 |
| gibi | we need a run with thread_pool_statistic_period=0 to get all the places that puts work into that pool to see when we start running out of workers | 15:49 |
| gibi | I can look at differently to see if all VM boots are failing or not | 15:50 |
| gibi | to try to get what was the first failed boot on the compute and see previous successful boot (if any) doing something that does not return a worker | 15:51 |
| sean-k-mooney | sounds like a resonable plan of attack | 15:53 |
| sean-k-mooney | you could jsut add a dnm to make 0 the default for testing | 15:54 |
| opendevreview | Merged openstack/os-vif master: Add TAP device pre-creation support for OVS/OVN https://review.opendev.org/c/openstack/os-vif/+/971231 | 16:04 |
| opendevreview | Merged openstack/os-vif master: Fixed bridge name when per_port_bridge is used https://review.opendev.org/c/openstack/os-vif/+/966410 | 18:33 |
| gmaan | sean-k-mooney: thanks for reply on Fair lock comments, I am good on that except one new comment https://review.opendev.org/c/openstack/nova/+/973438/comment/ff4bb8ea_53533e58/ | 22:36 |
| gmaan | sean-k-mooney: if that comment make sense then i am ok to fix that in follow up because current change is in mergable state. let me know. | 22:37 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!