| opendevreview | Merged openstack/openstacksdk master: heat: Fix stack query tag parameters https://review.opendev.org/c/openstack/openstacksdk/+/1005901 | 04:55 |
|---|---|---|
| opendevreview | Merged openstack/openstacksdk master: placement: Add support for bulk update, delete resource provider inventories https://review.opendev.org/c/openstack/openstacksdk/+/1006242 | 04:57 |
| opendevreview | Merged openstack/openstacksdk master: placement: Add support for usages https://review.opendev.org/c/openstack/openstacksdk/+/1006243 | 04:57 |
| opendevreview | Pranali Deore proposed openstack/openstacksdk master: image: Parse hw_vif_multiqueue_enabled as BoolStr https://review.opendev.org/c/openstack/openstacksdk/+/1006895 | 06:29 |
| opendevreview | Moritz Tanner proposed openstack/ansible-collections-openstack master: image: Perform a web-download import when uri is given https://review.opendev.org/c/openstack/ansible-collections-openstack/+/1006899 | 07:16 |
| opendevreview | Moritz Tanner proposed openstack/ansible-collections-openstack master: image: do a web-download import when uri is given https://review.opendev.org/c/openstack/ansible-collections-openstack/+/1006899 | 07:19 |
| opendevreview | Matt Crees proposed openstack/ansible-collections-openstack master: Fix: Include target_project_id in RBAC creation https://review.opendev.org/c/openstack/ansible-collections-openstack/+/964684 | 08:33 |
| opendevreview | Matt Crees proposed openstack/ansible-collections-openstack master: Fix: Include target_project_id in RBAC creation https://review.opendev.org/c/openstack/ansible-collections-openstack/+/964684 | 08:35 |
| opendevreview | Vladislav Terekhov proposed openstack/openstacksdk master: resource: Fix CSV query parameter formatting for strings https://review.opendev.org/c/openstack/openstacksdk/+/1005733 | 11:47 |
| andreykurilin | Hi folks! Is anyone available to talk about `atexit.register(self.close)` in the connection init method? It seems like a problem for horizon + urllib3 2+ . I can raise a bug report, but want to discuss it first | 13:23 |
| andreykurilin | The problem - Horizon of 2026.1 started to leak memory. Quite a lot in comparison to 2025.1 . After digging into an affected process, I found ~3,5k of openstacksdk objects in the memory. Each one stores a connection pool with a complete chain of other objects. Horizon never calls close for most of openstack connection it creates, which means that all of them (as far as I understand) keeps to be attached to uwsgi process due to | 13:53 |
| andreykurilin | atexit.register | 13:53 |
| andreykurilin | https://github.com/psf/requests/pull/6767 changed the behavior of default ssl context. Every pooled connection now owns a CA-loaded SSLContext which makes the difference in memory consumption by leaked objects | 13:56 |
| stephenfin | andreykurilin: What do you think is the issue from the SDK side? | 14:38 |
| andreykurilin | `atexit.register` call which makes openstack conn “immortal” :) I’m doing some experiments right now, it seems like switching to weakref.finalize should solve the problem - it would fire close “as soon” as openstacksdk object is no longer used | 14:41 |
| andreykurilin | @stephenfin https://github.com/openstack/openstacksdk/blob/stable/2026.1/openstack/cloud/openstackcloud.py#L279-L280 this part I’m talking about | 14:43 |
| stephenfin | as you likely know, the whole point of that is to ensure things are correctly closed before the Python process exits. gtema is out this week (I think) but he added that some time back https://github.com/openstack/openstacksdk/commit/2b4aeff6d3972761da9c4a6a8b696103664a758b | 14:45 |
| andreykurilin | Yes, it has been there since 2023. I believe it has not caused much problems as leaked requests connections were cheap | 14:46 |
| andreykurilin | Having thousands of connections in memory was taking like couple megabytes as they were sharing the same ssl context | 14:46 |
| andreykurilin | Horizon (uwsgi) is long running process which means that atexit may never fire until redeploy or OOM | 14:49 |
| stephenfin | Right. My reason for pointing that out wasn't to say that this isn't an issue because it's old code. Rather, it was merely to point out that the code is there for a reason | 14:51 |
| stephenfin | I wonder how practical it would be to allow Connection to be used as a context manager and to use that in-tree in Horizon. Or just have horizon ensure it calls `close()` when it's done? | 14:52 |
| andreykurilin | Considering the review speed of horizon changes - near impossible | 14:53 |
| andreykurilin | + it is hard to ensure that all horizon dashboards follow this. For example, octavia-dashboard may create dozens openstacksdk objs for a single user request, no memorize usage | 14:54 |
| stephenfin | fair points | 14:54 |
| stephenfin | weakref.finalize sounds like a good shout. I recall us using that in one of the oslo projects at some point for a similar issue. Neutron uses it too, iirc (?) | 14:56 |
| stephenfin | codesearch.o.o tells we oslo.rootwrap is the lib I was thinking of | 14:56 |
| stephenfin | *me | 14:56 |
| andreykurilin | Ok, I can submit a change for that. Probably, I can try to run it for sometime in our prod to ensure that it fixes horizon memory leakage (5 uwsgi processes ate 30Gb of ram, ridiculous) | 15:05 |
| opendevreview | Andriy Kurilin proposed openstack/openstacksdk master: Replace atexit with weakref.finalize https://review.opendev.org/c/openstack/openstacksdk/+/1006972 | 15:26 |
| stephenfin | andreykurilin: Change looks fine. Couple of small requests though assuming we want to backport that | 16:03 |
| rm_work | stephenfin: do you know if danchild is here? or somewhere to chat with? | 16:10 |
| stephenfin | I don't think so. He should be on k8s Slack though | 16:11 |
| stephenfin | (btw, I have seen your comments on #openstack-oslo. I just haven't gotten back to that series yet) | 16:11 |
| rm_work | yeah no worries | 16:24 |
| rm_work | I'm back to working on the oslo.messaging side | 16:25 |
| rm_work | has been a lot of good collab with Furukawa-san on the LINE side there (and they are the ones that actually use Consul) so I didn't want to change TOO much from what he had last tested and confirmed, but all of those changes were small enough (and the split was identical at the end of the chain to the original) so that was fine | 16:26 |
| rm_work | we've both done pretty extensive testing on the functionality, the code itself though is ... mostly copied from the original driver and modified/extended by AI, so ... yeah, trying not to give you giant slop bomb to review, but it definitely falls into the category of "I have personally reviewed the shape and the behavior but not every line" T_T | 16:28 |
| rm_work | same with gophercloud | 16:28 |
| rm_work | my testing has been pretty exhaustive, and the code is live and in use actively (for both gopher and rpc actually) in my production cloud deployments, so that is the confidence level I have at least 😄 | 16:29 |
| opendevreview | Andriy Kurilin proposed openstack/openstacksdk master: Replace atexit with weakref.finalize https://review.opendev.org/c/openstack/openstacksdk/+/1006972 | 16:58 |
| opendevreview | Brian Haley proposed openstack/openstacksdk master: Make openstacksdk-functional-devstack-networking-ext OVN-based https://review.opendev.org/c/openstack/openstacksdk/+/1006592 | 17:34 |
| opendevreview | Merged openstack/python-openstackclient master: Migrate accelerator commands from python-cyborgclient https://review.opendev.org/c/openstack/python-openstackclient/+/1004707 | 19:24 |
| opendevreview | Merged openstack/python-openstackclient master: Add functional tests and Zuul job for accelerator commands https://review.opendev.org/c/openstack/python-openstackclient/+/996698 | 19:24 |
| opendevreview | melanie witt proposed openstack/openstacksdk master: accelerator: Add device status field and enable/disable support https://review.opendev.org/c/openstack/openstacksdk/+/1004720 | 21:09 |
| opendevreview | melanie witt proposed openstack/openstacksdk master: accelerator: Add device status field and enable/disable support https://review.opendev.org/c/openstack/openstacksdk/+/1004720 | 22:00 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!