| kata-irc-bot | <jic5760> It is not working.. qemu says "Property 'cache-size' not found". | 09:20 |
|---|---|---|
| kata-irc-bot | <jic5760> https://github.com/kata-containers/kata-containers/issues/4524 | 09:45 |
| kata-irc-bot | <cameron4mcdermott> Hey folks, I have been doing some investigation into memory usage for Kata guest workloads and how that corresponds to memory usage in the host. It looks to be that freeing memory in the guest does not cause memory to be free on the host, but free page reporting could be a potential solution to this; more info in the thread. | 11:20 |
| kata-irc-bot | <cameron4mcdermott> I have mainly been using cloud-hypervisor to figure this one out, but it looks like allocating memory inside the guest causes the RSS of the allocated memory which acts as the backing for guest's RAM to grow in accordance to the allocated memory, this is of course expected. When later freeing this memory, the memory consuption reported inside the guest drops to what it was prior to the allocation as expected, but the | 11:21 |
| kata-irc-bot | reported RSS of the memory on the host never drops back to the amount it was prior to the allocation. This can be a bit of an issue for pods that run for a long time, there may be a period in time when the guest requires more memory, thus causing page allocation to the memory segement on the host, but this never gets freed back to the host after usage, thus not allowing other pods to make use of this now unused memory. Over time this can lead | 11:21 |
| kata-irc-bot | to memory exhaustion on the host. I have been looking into the virtio ballooning feature available to clh, and in release v22.0, the cloud-hypervisor team added support for free page reporting, allowing the VMM to report unused pages back to the host so they may be allocated elsewhere. There was a discussion a while ago, prior to the v22 release that spoke about making use of ballooning driver to achieve something similar to this | 11:21 |
| kata-irc-bot | (https://katacontainers.slack.com/archives/C879ACQ00/p1618345774082700). With the new addition of free page reporting, it seems to be a more transparent solution than the full ballooning option originally discussed, enabling this also seems to be quite straightforward. I setup cloud hypervisor with 2GB of initial memory and set the balloon option to be size 0 with freePageReporting=on, I also had to enable CONFIG_VIRTION_BALLOON in the guest | 11:21 |
| kata-irc-bot | kernel but my tests show: before enabling free page reporing: initial clh rss: 103MB initial ch_ram rss: 81MB after allocating 1024MB: clh rss: 1131MB ch_ram rss: 1109MB after freeing: clh rss: 1131MB ch_ram rss: 1110MB after enabling: initial clh rss: 101MB initial ch_ram rss: 77MB after allocating 1024MB: clh rss: 1132MB ch_ram rss: 1108MB after freeing: clh rss: 122MB ch_ram rss: 88MB Before enabling free page | 11:21 |
| kata-irc-bot | reporting the RSS never drops below the 1024GB allocation, after enabling, the RSS drops to slightly above its initial value (before the 1GB allocation). Does anyone have any context, thoughts or are there any potential drawbacks of enabling this? | 11:21 |
| kata-irc-bot | <cameron4mcdermott> @eric.ernst @jose.carlos.venegas.m @samuel.ortiz - from discussion previously | 11:23 |
| kata-irc-bot | <fidencio> cc @sebastien.boeuf here | 11:25 |
| kata-irc-bot | <eric.ernst> Do you actually see memory exhaustion? It should be limited by pod cgroup, at the very least? | 14:18 |
| kata-irc-bot | <cameron4mcdermott> yes I agree its still limited by the pod cgroup, its probably not a direct cause of memory exhaustion, but over time we see our nodes slowly consume more memory and when looking at the VM process RSS vs the actual resource usage of the workloads inside the VMs, these start to drift more over time. So the main thing here is being able to reclaim some of that unused memory so we can ensure we are using the node efficiently as | 14:24 |
| kata-irc-bot | possible. | 14:24 |
| kata-irc-bot | <eric.ernst> When you free are you clearing caches as well in the guest? I'd expect consumption to be able to go up to container limits still in caching. | 14:33 |
| kata-irc-bot | <cameron4mcdermott> https://gist.github.com/Champ-Goblem/ac76fe582e1cbe1b4ecb8b6c66f5f814 I have created a GIST with hopefully all the relevant information. • `allocate-memory.cpp` is the simple program I wrote to test this out • `mem-alloc.yaml` is the deployment config I use to test with Kata • `virtcontainers-change.diff` is the temporary patch I made to enable the freePageReporting in cloud-hypervisor • `debug-process.md` illustr | 15:12 |
| kata-irc-bot | steps I use to measure this and briefly explains the results Let me know if the GIST helps and happy to provide more info if needed | 15:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!