| opendevreview | Merged openstack/nova master: db: add indexes for source/dest migrations https://review.opendev.org/c/openstack/nova/+/958556 | 00:04 |
|---|---|---|
| *** erlon6 is now known as erlon | 01:25 | |
| opendevreview | ChungWon Lee proposed openstack/nova master: Fix port remain while error on update port with multi network https://review.opendev.org/c/openstack/nova/+/983672 | 07:20 |
| opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | 07:57 |
| opendevreview | Fabian Wiesel proposed openstack/nova master: conductor: reset placement and auth clients after fork https://review.opendev.org/c/openstack/nova/+/983676 | 08:34 |
| gibi | melwitt: dansmith: sean-k-mooney: here is the result for a 13 measurement on the PCI device tpool proxy situation https://paste.openstack.org/show/bKHhndSgMIVJtRn1HQLe/ | 09:18 |
| gibi | I see some runtime increase but the timing is pretty noisy | 09:19 |
| gibi | https://www.pasteboard.co/w-LCl8LzsgIk.png | 09:19 |
| gibi | the minimum run times are increasing I would say | 09:23 |
| MaxLamprecht[m] | gibi: can you share the libvirt version you have running? | 09:36 |
| opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | 09:57 |
| opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | 09:59 |
| opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | 10:01 |
| gibi | MaxLamprecht[m]: 10.0.0-2ubuntu8.12 | 10:30 |
| MaxLamprecht[m] | The increase of the minimum run time sounds similar to my reported finding. I assume that if you are updating to libvirt > 10.5 this constant increase is gone. | 10:48 |
| gibi | hm, interesting. That would suggest that the actual increase is due to libvirt and not due to nova itself | 11:01 |
| gibi | so if that is a theory then that suggests something accumlates on the libvirt side and a libvirt restart without a compute restart could clear it | 11:08 |
| gibi | MaxLamprecht[m]: do you have a link to the bug / bugfix in libvirt 10.5 that supposed to help? | 11:09 |
| gibi | sorry I saw it in the bug | 11:10 |
| gibi | thanks for adding it there | 11:10 |
| gibi | it seems we both wrote a comment to that bug around the same time | 11:10 |
| gibi | hence I missed your comment | 11:10 |
| gibi | until now | 11:10 |
| gibi | meanwhile I tried what happens when libvirt is restarted in my env while keep the nova-compute running | 11:11 |
| gibi | indeed after the libvirt restart the timing of the _get_pci_passthrough_devices call is back to the value it was at the start of my measurement run | 11:12 |
| MaxLamprecht[m] | gibi: thx for testing :) I guess that confirms it | 11:13 |
| gibi | I will try to pull in libvirt > 10.5 to see if that also proves it | 11:20 |
| sean-k-mooney | gibi: well libvir tdoes have a nodedevd which si listening to udev events and has a chache compoent | 11:20 |
| MaxLamprecht[m] | We observed this half a year ago and had massive freezes of nova-compute after having libvirt running for multiple days/months. I thought this is also related to this bug https://bugs.launchpad.net/nova/+bug/2091033. Thats also why I referenced it in my caching MR und mentioned the libvirt issue in the releasenote of it. | 11:20 |
| sean-k-mooney | so maybe that was buggy in older version and degraged over tiem with repeated calls | 11:21 |
| sean-k-mooney | gibi: the system i have had up for almost 4 months is runing 11.3.0 for what it worth | 11:22 |
| sean-k-mooney | and the perodic tiems were ~5seconds total | 11:22 |
| sean-k-mooney | sorry nova-compute as up 3 weeks 6 days ago and libvirt 2 months 13 days ago | 11:24 |
| sean-k-mooney | so with newer libvirt maybe it does not happen | 11:25 |
| sean-k-mooney | witht that said Memory: 14.1G (peak: 28.6G) for libvirtd seams... high | 11:25 |
| sean-k-mooney | PID: 729853 Shared: 22496K Name: libvirtd | 11:37 |
| sean-k-mooney | PID VSZ RSS PSS COMMAND | 11:37 |
| sean-k-mooney | 729853 1752908 48344 0 libvirtd | 11:37 |
| sean-k-mooney | i guess that actully ok | 11:38 |
| sean-k-mooney | that only 22MB of shared memory not 22G | 11:38 |
| sean-k-mooney | gibi: can you check https://paste.opendev.org/show/bnD6HOACPqVczxGCqYBi/ with your libvirts pid if you have not already replaced it | 11:39 |
| sean-k-mooney | i wonder if your seeing high memory usage form it or not | 11:39 |
| gibi | Apr 08 11:06:56 aio systemd[1]: libvirtd.service: Consumed 12min 42.721s CPU time, 1.7G memory peak, 24.0K memory swap peak. | 11:40 |
| gibi | this was the libvirt restart after more than 13 hours of usage | 11:41 |
| gibi | after 40 minutes passed since and the now it is on | 11:42 |
| gibi | Memory: 205.9M (peak: 206.7M) | 11:42 |
| gibi | same libvirt version still | 11:42 |
| sean-k-mooney | ack, i was conderd because this is what min shows https://paste.opendev.org/show/bxt9j6Mtzs4Is99u9nbv/ | 11:44 |
| sean-k-mooney | i guess it might be countign all the shile thread/processes | 11:46 |
| sean-k-mooney | https://paste.opendev.org/show/bEi3uoHoWJ7JNjcYXSA7/ | 11:46 |
| sean-k-mooney | bug given free tell me im using 16G total i dont think 14 of that is used by libvirt so im unsure how much we can really trust that systemd is reprotign in this case | 11:48 |
| sean-k-mooney | that proably not relvent to this issue however | 11:48 |
| sean-k-mooney | Uggla: would tuesday 13:00 utc work for you for the nova/cyborg cross project time? | 12:52 |
| sean-k-mooney | its the hour before the nova sessiosn start and we coudl have it in the cyborg room | 12:52 |
| sean-k-mooney | that means we can wrap the cyborg session in the following hour when nova is startign up and i can then proably join the nova room for the rest of the day | 12:53 |
| sean-k-mooney | since watcher is not starting its ptg until wednesday | 12:53 |
| sean-k-mooney | for wednesday and thrusday i do not have a conflict with wathcer/nova on the last hour of the windows as wather is tartign and fisniing 1 hour early then nova each day | 12:55 |
| sean-k-mooney | so that would be the other option unless you prefered monday? | 12:55 |
| opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | 16:00 |
| * gibi running the same measurement than last night with libvirt 11.3 (on debian 13) | 17:30 | |
| opendevreview | Victor proposed openstack/nova master: ironic: Abort build on fatal image validation errors https://review.opendev.org/c/openstack/nova/+/983753 | 17:48 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add spec for virtiofs live and cold migration https://review.opendev.org/c/openstack/nova-specs/+/983801 | 21:17 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
