I’d like to discuss the potential implementation of KSM (Kernel Same-page Merging) in Incus to save on RAM usage.
I currently run 50 identical containers, each consuming 4GB of RAM. Implementing KSM could save up to 196GB of RAM.
How can KSM be utilized with Incus? Is it possible to implement this technology? Are there any plans to use this technology?
This has been discussed before but not fully resolved. This feature could be extremely beneficial in data centers as well as under standard conditions.
I tried using KSM with Incus VMs and CTs, but unfortunately, it doesn’t work as expected
In theory, KSM should work well with QEMU instances, but it seems not to be the case with Incus.
Would it still be feasible for Incus to leverage the new kernel API to enable KSM for containers or VMs? Has anyone tried this approach or have insights on the potential challenges involved?
Looking forward to the maintainers’ input and any experiences from the community.
I didn’t use ksmtuned (used in Proxmox) to configure KSM.
It’s encouraging to see that you got some positive results with Incus VM after launching some VMs. I’ll follow your steps and check the /sys/kernel/mm/ksm/pages_shared and /sys/kernel/mm/ksm/run values.
I am using ksmtuned from the Ubuntu repositories.
I checked the RAM usage before and after enabling KSM by launching 7 VMs. Here are the commands I used to disable KSM and the results:
$ cat /sys/kernel/mm/ksm/pages_shared
835
$ echo 2 | sudo tee /sys/kernel/mm/ksm/run
$ cat /sys/kernel/mm/ksm/pages_shared
0
RAM usage:
Before KSM: 9024MiB
After KSM: 8885MiB
RAM saved: 139MiB
RAM usage of one VM: 127MiB
In theory, the RAM saved should be equal to the amount used by each VM. However, when all virtual machines are turned off except for one, it is clear that the RAM saved is not as much as expected.
One VM Memory: 5441MiB
Seven VMs Memory: 8966MiB
Has anyone else tried this approach or have additional tips on configuring KSM for optimal performance with Incus VMs & CTs?
I also tested KSM (ksm-wrapper) and UKSM technologies in different programs, and they turned out to be very inefficient.
Are there any plans to add a native Memory Deduplication feature in Incus? Implementing this technology would be highly beneficial for advancing Incus.
We’re not very likely to be doing that. For VMs, you can turn on the kernel ksm and let it do its thing as you’ve noticed. For containers, KSM is a bit of a PITA, we tried it before but it’s a mess to make it work consistently.
Then spectre/meltdown happened and KSM became a big no-no for a lot of people as you actively don’t want your VM or container memory to be merged with another tenant on the same system as now being able to read memory pages near what you think is yours may actually have you read another guest’s memory.
And lastly, these days the craze is around confidential computing which goes the complete other direction by instead pushing for VMs (and in some cases containers) to have their own page tables and have their entire memory encrypted with a per-instance key, making it completely impossible to do this kind of merging.
I understand the challenges and security concerns associated with KSM, especially in the context of Spectre/Meltdown and the current trend towards confidential computing.
While a native Memory Deduplication feature would have been beneficial, I see how the risks and industry trends make it less viable.
Thanks again for your insights and the work you do with Incus.
I appreciate the guidance on using the new API to enable KSM per process. I’ll try creating a .so file and using LD_PRELOAD to enable KSM for dynamic linked programs.
Thanks for providing the updated version on GitHub!
Could you please guide me on how to use this code with Incus? I’m interested in understanding how to apply it to enable KSM for VMs or containers managed by Incus.
I don’t think you can apply it globally actually.
If you have a systemd service that you specially want to do ksm,you can add the LD_PRELOAD to its environment definition.
I also tested with Incus by adding the value to the service file and launching 10 Ubuntu 24.04 containers. Everything seemed to work, but as I mentioned earlier, the savings were not as significant as I had hoped:
it will only enable ksm for incus daemon itself, not the containers.
containers are running inside new “sections”,which means they do not inherit the ksm settings.
You need to hack systemd config in every containers to load the library.
As I said, the way you use will only enable ksm for incus itself.
There should be /etc/systemd/system.conf in your container. You can add LD_PRELOAD to DefaultEnvironment so that the services will load it automatically.>systemd-system.conf
Note that shell session does not inherit the env so you need to add it to profiles if want it for shells.
You can check whether a process is KSMed by “cat /proc/process_pid/ksm_*”
Currently I have only saved 0.21GiB memory for host_and_2_container.
Also this ld_preload way is rather hacky and does not work for static binaries. If someone knows systemd people,then maybe it’s better to implement this directly in systemd since it’s the system manager and can do anythings(like passing ksm settings to process started).