| opendevreview | Ghanshyam proposed openstack/nova master: Add manager graceful shutdown timeout and wait https://review.opendev.org/c/openstack/nova/+/975586 | 02:00 |
|---|---|---|
| gmaan | bauzas: this is ready, gate is failed due to known broken nova-next job otherwise it pass the gate https://review.opendev.org/c/openstack/nova/+/975242/4 | 02:01 |
| opendevreview | Ghanshyam proposed openstack/nova master: Add manager graceful shutdown timeout and wait https://review.opendev.org/c/openstack/nova/+/975586 | 02:02 |
| melwitt | sean-k-mooney: thanks, +2 | 02:18 |
| *** mhen_ is now known as mhen | 02:30 | |
| opendevreview | Ghanshyam proposed openstack/nova master: WIP: Use 2nd RPC server in compute operations https://review.opendev.org/c/openstack/nova/+/975588 | 04:16 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: stop using _get_mem_encryption_config() for SEV checks https://review.opendev.org/c/openstack/nova/+/967970 | 08:06 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: refactor memory encryption trait logic for extensiblity https://review.opendev.org/c/openstack/nova/+/967971 | 08:06 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: make RP creation independent of specific encryption models https://review.opendev.org/c/openstack/nova/+/967972 | 08:06 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: refactor _guest_configure_mem_encryption() for extensibility https://review.opendev.org/c/openstack/nova/+/967973 | 08:06 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: adjust requirement checks for mem_encryption guests https://review.opendev.org/c/openstack/nova/+/967974 | 08:06 |
| opendevreview | Taketani Ryo proposed openstack/nova master: mem-enc: introduce a check between mem_encryption and locked_memory https://review.opendev.org/c/openstack/nova/+/971300 | 08:06 |
| sean-k-mooney | gmaan: gibi speakign of the nova-next jobs if i can get a second core to review https://review.opendev.org/c/openstack/nova/+/975542 that will fix it | 10:10 |
| sean-k-mooney | actully bauzas ^ if your around you might have time to take a look also | 10:19 |
| bauzas | sure, I saw the gate failing | 10:20 |
| sean-k-mooney | its basically the same issue we had with the disk serial for device_type=lun in lun mode the phsical aspect fo the backing lun like the serial and in this case sector size cannot be overriden by qemu/libvirt the are presented as is form the storage array to the vm so the fix is to just not set those parmaters in the xml for lun type volumes | 10:25 |
| bauzas | yup, I reviewed it | 10:26 |
| *** ralonsoh_ is now known as ralonsoh | 10:26 | |
| bauzas | thanks for amending the fixture, as I said | 10:26 |
| sean-k-mooney | i spoke to the virt folks about it yesterday and they have created a rhel/libvirt bug to actully have libvirt validate this in the future instead of having ti fail at the qemu level | 10:26 |
| sean-k-mooney | bauzas ++ thanks once thats merged ill reply to mel's email to let folk know its safe to recheck | 10:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | 11:15 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove spawn_after https://review.opendev.org/c/openstack/nova/+/975396 | 11:15 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128 | 11:15 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172 | 11:15 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | 11:15 |
| gibi | bauzas: ^^ removed the print() call | 11:15 |
| *** debian is now known as Guest1463 | 12:21 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Troubleshoot deadlocks in threading mode https://review.opendev.org/c/openstack/nova/+/975515 | 12:32 |
| opendevreview | sean mooney proposed openstack/nova-specs master: add pci passthrough groups spec for 2026.2 https://review.opendev.org/c/openstack/nova-specs/+/975635 | 13:27 |
| opendevreview | sean mooney proposed openstack/nova-specs master: add pci passthrough groups spec for 2026.2 https://review.opendev.org/c/openstack/nova-specs/+/975635 | 13:36 |
| *** ralonsoh_ is now known as ralonsoh | 14:19 | |
| opendevreview | Max proposed openstack/nova master: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/975651 | 16:04 |
| sean-k-mooney | gibi: ya so that was the orginal comemnt on the delay behavior. which is being adressed by https://review.opendev.org/c/openstack/nova/+/974445/14 so ill review it properly after i grab somethign to drink but we can likely proceed. | 16:14 |
| sean-k-mooney | note i need to recheck the gate fix again because it had a network timeout downloaing pip packages form teh mirror | 16:14 |
| sean-k-mooney | gibi: i was thinking of https://review.opendev.org/c/openstack/nova/+/922497 and https://review.opendev.org/c/openstack/nova/+/905287/9 which on master is now https://github.com/openstack/nova/blob/master/nova/image/glance.py#L580-L586 so it looks like in threaded mode we jsut execute it inline and block on the upload | 16:39 |
| sean-k-mooney | so unless we move snapshots to the long runting thread pool that will tie up an rpc thread for the entirty fo the galnce snapshot upload | 16:40 |
| sean-k-mooney | gibi: when you took over the eventlet work you didnt add a dedicated io thread pool but rather a defult thread pool but we didnt update that code to dispatch the upload to the thread pool. | 16:42 |
| gibi | sean-k-mooney: thanks for the pointers. I don't see how that io executor helps, the usage in RBD is spawning a task to the executor and immediately blocking on waiting for the result. Logically that is the same as executing the function on the caller thread in my eyes (in native threading) | 17:18 |
| gibi | https://review.opendev.org/c/openstack/nova/+/917962/8/nova/storage/rbd_utils.py#56 | 17:19 |
| sean-k-mooney | i wrote it before dan told me that the rbd client was doing that internally | 17:19 |
| sean-k-mooney | so its not useful for that | 17:20 |
| gibi | ack | 17:20 |
| sean-k-mooney | so the up shot is we have 1 less executor then i tought we had (i.e. we dont have a dedicated io executor for long running file and api calls | 17:20 |
| sean-k-mooney | ) | 17:21 |
| sean-k-mooney | it was orgilly suggestign puting the neutron netrokign call on that but i guess the only thing that impoant to consider is the _upload_data function | 17:22 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/image/glance.py#L580-L586 | 17:22 |
| gibi | OK, I will check the _upload_data codepath | 17:22 |
| sean-k-mooney | im not sure we want to block the RPC thread with doing the file upload for snapshots | 17:23 |
| sean-k-mooney | but as dan also pointed out unless we have a futre we can pass in or simialr i dont know hwe we are going to deal with returning the image uuid that is created form this call | 17:24 |
| sean-k-mooney | when you do openstack server image create we need to return teh glance image id for the snapshot form the api if i recall correctly | 17:24 |
| gibi | yepp, I need to think about all these. | 17:24 |
| sean-k-mooney | doign it inline is fine if we move thie operation entrily to teh long runing thread pool but th ereply path is still an unknow i guess | 17:26 |
| gibi | sean-k-mooney: food for thought. How do you feel about https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L232 as a way out of deadlocks due to limited executor sizes? | 17:26 |
| gibi | DynamicThreadPoolExecutor | 17:26 |
| sean-k-mooney | i have not looked at that before but i was debeating about creatin a new executor propsosla that handeled prities canclation and delayed exection | 17:27 |
| gibi | for the delayed execution I have this WIP in futurist https://review.opendev.org/c/openstack/futurist/+/975508 | 17:27 |
| sean-k-mooney | gibi: but i dont feel terible about one that cna grow as a way out | 17:27 |
| gibi | yeah we still need to limit our parallel work due to IO and overall memory usage but at least the deadlock becomes something else :) | 17:28 |
| sean-k-mooney | so the lib that watcher and zuul uses has lots of interest feature but the reason im not really suggesting that at the morment is that lib is 4? yes int a v4 rewirte which break backward compatiblity | 17:28 |
| sean-k-mooney | *4 years? | 17:29 |
| sean-k-mooney | gibi: ill try and rominate on it. im not a huge fan of async io. i dont hate it but i was also wonderign if we could/should consider limited use fo an asynio executor | 17:30 |
| sean-k-mooney | if we were to use the DynamicThreadPoolExecutor | 17:32 |
| sean-k-mooney | are you suggesting we woudl not set a max workers or set it relitivly high with the expcation taht in reality it will stay small? | 17:32 |
| gibi | I have to read on what the Dynamic supports regarding sizing but I see that it can grow and shrink | 17:33 |
| sean-k-mooney | """ If threads are not created often in your application, the number of idle | 17:33 |
| sean-k-mooney | threads may stay high for a long time. To avoid it, you can call | 17:33 |
| sean-k-mooney | :py:meth:`.maintain` periodically to keep the number of threads within | 17:33 |
| sean-k-mooney | the thresholds.""" | 17:33 |
| sean-k-mooney | we might need to do that ^ | 17:34 |
| sean-k-mooney | gibi: i dont hate that idea if im honest. but its proably the rpc pool in oslo service that woudl be the most benifical to use that with right? | 17:35 |
| sean-k-mooney | sorry in oslo.messaging | 17:36 |
| gibi | yeah that too :) | 17:36 |
| sean-k-mooney | i know we can set an executor type when we create the rpc servier im wondering if we can just provide an rpc instance | 17:37 |
| sean-k-mooney | *executor instance | 17:37 |
| sean-k-mooney | ya its just the name https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/rpc/server.py#L226-L240 | 17:38 |
| sean-k-mooney | so we woudl have ot update https://github.com/openstack/oslo.messaging/blob/master/pyproject.toml#L53 | 17:41 |
| sean-k-mooney | or add a new value | 17:41 |
| sean-k-mooney | might be worth suggeting that to the oslo folks and see what they think | 17:41 |
| dansmith | I'm not sure how the dynamic one really helps | 18:03 |
| dansmith | you still have to set your max to the upper limit of what you're willing tolerate.. it uses less memory in slow times, which is a benefit, but it's still the same problem when you hit the limit | 18:03 |
| gmaan | yeah, it does not solve the limit issue. I think we should be gradually learning about those limit issue (after solving the known one) and keep improving the defaults limit or better separate executor if any specific operations causing problem | 18:06 |
| * gibi need more brains | 18:19 | |
| gibi | https://etherpad.opendev.org/p/eventlet-executor-semaphore-limit this is my brain dump looking at the long running task executor possibilities | 18:36 |
| gibi | we have a list of pros and cons to choose from | 18:36 |
| gmaan | thanks for catching those in etherpad | 18:38 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Move the concurrent builds to its own Executor https://review.opendev.org/c/openstack/nova/+/975694 | 19:01 |
| gibi | this is the very raw idea of optionA) from the etherpad ^^ not tested at all :) | 19:01 |
| gibi | and that marks my end of day | 19:01 |
| gibi | o/ | 19:01 |
| opendevreview | Marius Leustean proposed openstack/nova-specs master: Add spec for handling orphan BDMs from late RPC processing https://review.opendev.org/c/openstack/nova-specs/+/975698 | 19:43 |
| opendevreview | Merged openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542 | 22:20 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!