[Memory Deduplication] Implement New Technology to Incus

Incus
1

Hi everyone,

I’d like to discuss the potential implementation of KSM (Kernel Same-page Merging) in Incus to save on RAM usage.

I currently run 50 identical containers, each consuming 4GB of RAM. Implementing KSM could save up to 196GB of RAM.

How can KSM be utilized with Incus? Is it possible to implement this technology? Are there any plans to use this technology?

This has been discussed before but not fully resolved. This feature could be extremely beneficial in data centers as well as under standard conditions.

Looking forward to your thoughts and suggestions.

Thank you!

2

IIRC for virtual machines, QEMU flags the virtual machine memory pages to allow KSM to work properly, so that should work with Incus VM I guess (?).

For containers, I think this will require some work, but since few kernel versions it seems that is possible to set it at process and/or cgroup level: mm: add new api to enable ksm per process · torvalds/linux@d7597f5 · GitHub

I don’t have any experience in this domain so I will let the maintainers answers on this one.

3

Thank you for your response!

I tried using KSM with Incus VMs and CTs, but unfortunately, it doesn’t work as expected

In theory, KSM should work well with QEMU instances, but it seems not to be the case with Incus.

Would it still be feasible for Incus to leverage the new kernel API to enable KSM for containers or VMs? Has anyone tried this approach or have insights on the potential challenges involved?

Looking forward to the maintainers’ input and any experiences from the community.

Thanks again!

1 Like
4

I will try with Incus VM on my side, did you use ksmtuned (used in Proxmox) to configure KSM ?

5

I made a small test on my side with Incus VM, seems working a bit after launching some VM:

$ cat /sys/kernel/mm/ksm/pages_shared
0
$ cat /sys/kernel/mm/ksm/run
0
$ echo 1 | sudo tee /sys/kernel/mm/ksm/run
$ cat /sys/kernel/mm/ksm/run
1
# Launch 10 VMs
$ incus launch images:ubuntu/24.04 v1 --vm -p vm -c limits.memory=4GiB
...
$ cat /sys/kernel/mm/ksm/pages_shared
832
2 Likes
6

Thank you for testing it on your side!

I didn’t use ksmtuned (used in Proxmox) to configure KSM.

It’s encouraging to see that you got some positive results with Incus VM after launching some VMs. I’ll follow your steps and check the /sys/kernel/mm/ksm/pages_shared and /sys/kernel/mm/ksm/run values.

I am using ksmtuned from the Ubuntu repositories.

I checked the RAM usage before and after enabling KSM by launching 7 VMs. Here are the commands I used to disable KSM and the results:

$ cat /sys/kernel/mm/ksm/pages_shared
835

$ echo 2 | sudo tee /sys/kernel/mm/ksm/run

$ cat /sys/kernel/mm/ksm/pages_shared
0

RAM usage:

  • Before KSM: 9024MiB
  • After KSM: 8885MiB

RAM saved: 139MiB

RAM usage of one VM: 127MiB

In theory, the RAM saved should be equal to the amount used by each VM. However, when all virtual machines are turned off except for one, it is clear that the RAM saved is not as much as expected.

  • One VM Memory: 5441MiB
  • Seven VMs Memory: 8966MiB

Has anyone else tried this approach or have additional tips on configuring KSM for optimal performance with Incus VMs & CTs?

Thanks again for your help and insights!

7

I also tested KSM (ksm-wrapper) and UKSM technologies in different programs, and they turned out to be very inefficient.

Are there any plans to add a native Memory Deduplication feature in Incus? Implementing this technology would be highly beneficial for advancing Incus.

How feasible would this idea be, @stgraber?

Regards.

8

We’re not very likely to be doing that. For VMs, you can turn on the kernel ksm and let it do its thing as you’ve noticed. For containers, KSM is a bit of a PITA, we tried it before but it’s a mess to make it work consistently.

Then spectre/meltdown happened and KSM became a big no-no for a lot of people as you actively don’t want your VM or container memory to be merged with another tenant on the same system as now being able to read memory pages near what you think is yours may actually have you read another guest’s memory.

And lastly, these days the craze is around confidential computing which goes the complete other direction by instead pushing for VMs (and in some cases containers) to have their own page tables and have their entire memory encrypted with a per-instance key, making it completely impossible to do this kind of merging.

2 Likes
9

Thank you for the detailed explanation.

I understand the challenges and security concerns associated with KSM, especially in the context of Spectre/Meltdown and the current trend towards confidential computing.

While a native Memory Deduplication feature would have been beneficial, I see how the risks and industry trends make it less viable.

Thanks again for your insights and the work you do with Incus.

Regards.

10

If your kernel version is recent that includes mm: add new api to enable ksm per process · torvalds/linux@d7597f5 · GitHub
then
You can use this code:

#include <stdio.h>
#include <sys/prctl.h>
void __attribute__((constructor)) load() {
int ret = -255;
ret = prctl(67,1,0,0,0);
if(ret >=0) {
fprintf(stderr,"KSM ON !\n");
}else{
fprintf(stderr,"KSM error %d\n",ret);
}
}

to create a .so that can be LD_PRELOAD to any (dynamic linked) programs that enables KSM for them.

2 Likes
11

Thank you for sharing the code snippet!

I appreciate the guidance on using the new API to enable KSM per process. I’ll try creating a .so file and using LD_PRELOAD to enable KSM for dynamic linked programs.

I’ll let you know how it goes.

Thanks again for your help!

Regards.

12

You can access a updated one,which ignores shell by default.

2 Likes
13

Thanks for providing the updated version on GitHub!

Could you please guide me on how to use this code with Incus? I’m interested in understanding how to apply it to enable KSM for VMs or containers managed by Incus.

I appreciate your help!

Regards.

14

I don’t think you can apply it globally actually.
If you have a systemd service that you specially want to do ksm,you can add the LD_PRELOAD to its environment definition.

You can learn more here c - What is the LD_PRELOAD trick? - Stack Overflow
You’ll need a development environment to compile&test.

You can also patch a binary to add the library to its dependency so that ld_preload no longer needed.

1 Like
15

I tried using your code, but unfortunately, it didn’t work for me:

ubuntu@ksm:~$ LD_PRELOAD=/home/ubuntu/libfksm.so librewolf
ERROR: ld.so: object '/home/ubuntu/libfksm.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

root@ksm:/home/ubuntu# cat /sys/kernel/mm/ksm/pages_sharing
0

When I used my code, it worked successfully:

ubuntu@ksm:~$ LD_PRELOAD=/home/ubuntu/enable_ksm.so librewolf

root@ksm:/home/ubuntu# cat /sys/kernel/mm/ksm/pages_sharing
3276

I also tested with Incus by adding the value to the service file and launching 10 Ubuntu 24.04 containers. Everything seemed to work, but as I mentioned earlier, the savings were not as significant as I had hoped:

nano /usr/lib/systemd/system/incus.service

[Service]
Environment=LD_PRELOAD=/home/ubuntu/enable_ksm.so

sudo systemctl daemon-reload
sudo systemctl restart incus

root@ksm:/home/ubuntu# cat /sys/kernel/mm/ksm/pages_sharing
947

Unfortunately, I couldn’t get your code to work, and I’m not sure what the issue might be:

nano /usr/lib/systemd/system/incus.service

[Service]
Environment=LD_PRELOAD=/home/ubuntu/libfksm.so

sudo systemctl daemon-reload
sudo systemctl restart incus

root@ksm:/home/ubuntu# cat /sys/kernel/mm/ksm/pages_sharing
0

PS: Here’s the code I used:

echo '
#include <sys/mman.h>
#include <sys/prctl.h>
#include <stdio.h>

__attribute__((constructor)) static void on_load(void) {
    if (prctl(PR_SET_MEMORY_MERGE, 1, 0, 0, 0) != 0) {
        perror("prctl");
    }
}' > enable_ksm.c

gcc -shared -fPIC -o enable_ksm.so enable_ksm.c

Thank you for your help!

16

it will only enable ksm for incus daemon itself, not the containers.
containers are running inside new “sections”,which means they do not inherit the ksm settings.

You need to hack systemd config in every containers to load the library.

1 Like
17

Unfortunately, I’m not sure how to modify the systemd configuration in each container. However, I got the idea from this post and tried the following:

root@ksm:/home/ubuntu# LD_PRELOAD=/home/ubuntu/enable_ksm.so /usr/libexec/incus/incusd --group incus-admin

After launching 10 Ubuntu 24.04 containers, I saved only 4 megabytes of RAM:

root@ksm:/home/ubuntu# echo $[$(cat /sys/kernel/mm/ksm/pages_sharing)*$(getconf PAGE_SIZE)/1024 ]KB
4744KB

Each container consumes about 22 megabytes of RAM:

root@ksm:/home/ubuntu# incus info u10 | grep Memory
Memory usage:
Memory (current): 22.95MiB

I appreciate your help!

Regards.

18

As I said, the way you use will only enable ksm for incus itself.

There should be /etc/systemd/system.conf in your container. You can add LD_PRELOAD to DefaultEnvironment so that the services will load it automatically.>systemd-system.conf
Note that shell session does not inherit the env so you need to add it to profiles if want it for shells.

1 Like
19

Thank you for the guidance.

I recreated the containers and edited the file in each one as follows:

sudo sed -i 's|^#DefaultEnvironment=.*|DefaultEnvironment=LD_PRELOAD=/usr/local/lib/enable_ksm.so|' /etc/systemd/system.conf

After restarting all the containers, I observed the following result:

root@ksm:~# echo $[$(cat /sys/kernel/mm/ksm/pages_sharing)*$(getconf PAGE_SIZE)/1024 ]KB
3916KB

Container memory usage:

root@ksm:~# incus info u1 | grep Memory
Memory usage:
Memory (current): 22.26MiB

As you can see, the expected RAM savings of 22 megabytes were not achieved, even though each container is identical.

Could there be something I did incorrectly? Or is KSM inherently inefficient in containers?

Thanks again for your help and insights!

20

You can check whether a process is KSMed by “cat /proc/process_pid/ksm_*”
Currently I have only saved 0.21GiB memory for host_and_2_container.

Also this ld_preload way is rather hacky and does not work for static binaries. If someone knows systemd people,then maybe it’s better to implement this directly in systemd since it’s the system manager and can do anythings(like passing ksm settings to process started).

1 Like