| opendevreview | Ivan Anfimov proposed openstack/openstack-ansible-plugins master: Use kernel-modules-extra on RHEL for kernel headers task https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/996460 | 02:31 |
|---|---|---|
| opendevreview | OpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/996587 | 04:17 |
| opendevreview | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.opendev.org/c/openstack/openstack-ansible/+/996587 | 08:39 |
| kleini | I am still on 31.1.3, sorry for that. I encounter now that the periodic task in nova-compute, which should run every minute, did not run like for 20 minutes on some computes. Is there something known in those regards on Epoxy, what the possible cause is? | 10:05 |
| kleini | I found nova-compute burning one full CPU core - 100% in top. After restart, that is gone and nova-compute down to 0%. | 10:18 |
| noonedeadpunk | I wasn't spotting anything like that tbh | 10:19 |
| kleini | So urgent time to upgrade to 31.3.0 or even to 33.0.0 | 10:25 |
| noonedeadpunk | I think we were mostly running 31.0.1 though | 10:42 |
| kleini | https://paste.opendev.org/show/bB3TjOURD65gRucyLvMS/ <- found something interesting | 11:41 |
| mgariepy | hmm interesting. is one of the placement api containers/server acting weird? | 11:54 |
| kleini | I think, this is not the cause. That is some side effect, when getting the nova-compute process out of burning one CPU core. nova-compute opened the HTTP connection to haproxy but never sent the request for the periodic update of the compute node state. | 12:03 |
| kleini | see in error message: "your browser didn't sent a request." that is haproxy response to nova-compute client not sending a request. | 12:03 |
| mgariepy | is nova-compute out of file descriptor? | 12:04 |
| mgariepy | does it happens to all compute or only a handfull of them ? | 12:05 |
| kleini | all nova-compute on all compute hosts burn one full CPU core. so, they're all stuck at the same point. will investigate, what the cause is. | 12:07 |
| mgariepy | i am very curious on what can causes that. | 12:08 |
| mgariepy | did they start all at the same time ? | 12:08 |
| kleini | This already runs now for months and mostly newly spawning VMs spread across 18 compute nodes. In this case more than 10 VMs should be spawned through server group affinity on the same compute node, revealing this issue. | 12:11 |
| kleini | It looks like it is somewhere towards connection to libvirtd going stale or something. | 12:14 |
| kleini | https://bugs.launchpad.net/nova/+bug/2147424 <- and here we are! | 12:14 |
| kleini | libvirt-daemon 10.0.0-2ubuntu8.11 gotcha! | 12:16 |
| mgariepy | ho nice. | 12:19 |
| mgariepy | only 2 years. | 12:21 |
| mgariepy | it was a quick fix.. patch waiting in CI since june 2024 | 12:21 |
| kleini | The issue is, looking at it with "py-spy record --native" immediately resolves the issue as the zombie TCP connection is dropped. | 12:22 |
| noonedeadpunk | hm.... I can recall patching livbirt for 10 | 12:26 |
| noonedeadpunk | I think the fix I have in mind was published in 10.0.0-2ubuntu8.12 or smth like that | 12:26 |
| noonedeadpunk | but it was about https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2133183 | 12:27 |
| kleini | I see 10.0.0-2ubuntu8.14 available. | 12:27 |
| kleini | hmm, no quick fix in sight | 12:44 |
| mgariepy | cronjob running py-spy every minutes! | 13:34 |
| kleini | systemd_exporter and alert in Grafana load > 0.8 over 30m -> fire | 13:38 |
| mgariepy | i guess the nova patches linked should reduce the frequency tho. since it will make less calls. | 13:39 |
| opendevreview | Dmitriy Chubinidze proposed openstack/openstack-ansible-os_magnum master: Remove Heat-related variables and entities https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/996411 | 14:20 |
| -Guest12850- NOTICE: The Gerrit service on review.opendev.org will be offline briefly one hour from now, at 21:00 UTC, while we rename a project. | 20:00 | |
| -Guest12850- NOTICE: The Gerrit service on review.opendev.org is going offline momentarily at 21:00 UTC while we rename a project, but will return within a few minutes. | 20:58 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!