How to limit an LXD container for maximum swap space

LXD
1

Hi,
Is it possible to define a maximum swap space limit for an LXD container?
Is there a quick way to check the currently used swap space by a container processes? Or, it is necessary just to summarize the VmSwap values in /proc/nnn/status for all container processes?

Thank you,
Leonid

2

You can’t limit the swap space of a given container but you can tweak how likely a container is to swap.

lxc info should show you the total memory and swap usage for the container, for individual processes, VmSwap is likely the easiest unless you have systemd or something in the container grouping processes into memory cgroups which you could then query.

3

Unfortunately I don’t see swap usage information neither in the /sys/fs/cgroup/memory/lxc/container nor in the /sys/fs/cgroup/systemd/lxc/container folder 


4

Did you boot your kernel with the option enabling swap tracking?

5

swapaccount=1

6

To be honest, I didn’t know about this option. Thank you very much!
Leonid Fainshtein

7

I’ve added swapaccount to the kernel command line. Now I see a few memory.memsw.* files in the cgroups folders. I paid attention that the values in the memory.limit_in_bytes and in the memory.memsw.limit_in_bytes files are the same. In other words, if a container is limited for 512MiB:

lxc config set <container> limits.memory 512MiB
lxc config set <container> limits.memory.enforce hard

then the 512MiB limit is actually the RSS + SWAP limit.
Unfortunately I didn’t find in LXD an option to limit RSS and SWAP separately as it is possible in Docker. According to the LXD documentation (https://lxd.readthedocs.io/en/stable-3.0/containers/#container-configuration) we have limits.memory.swap that enables/disables the swapping for a container and there is no a parameter that defines the maximum swap size like Docker --memory-swap parameter (https://docs.docker.com/config/containers/resource_constraints/)
I think that the swap usage control is critical for system containers where we have many processes with sometimes very hardly predictable memory requirements. Therefore, one container can easily kill the whole server by allocating the all available swap space.
Are there any plans to add the swap allocation control to LXD?
Thank you,
Leonid

1 Like
8

We may eventually add a separate option for swap limits but that may have to wait until we’ve switched over to CGroupV2 as there is little point in adding new CGroupV1 features at this point.

One issue with adding swap control is that reporting of swap limits is pretty crappy, so software that’s actively looking at /proc/meminfo may end up doing the wrong thing, effectively assuming they can swap more than they actually can, leading to triggering of the OOM killer which is never a particularly pleasant experience.

So adding swap control would likely come together with improvements to swap handling in lxcfs to make sure we don’t make things worse in the process.

9

Is it not enough just to define

memory.limit_in_bytes=<max RAM>
memory.memsw.limit_in_bytes=<max RAM + max SWAP>

10

For the limit to be effective, yes, but userspace will only realize there’s a limit when it tries to reach past it, which isn’t a very good experience usually.

11

I’m trying to understand what is the ‘user space’ you have mentioned.
I don’t think that, let’s say, mysql checks whether there is a free memory before it calls malloc().
So, if there is a RAM+SWAP limit set then the service that is running inside of container fails when it tries to get some memory beyond the limit. I think that it is good because of the following reasons:

  • only specific container is affected. The server continues to work normally
  • the memory allocation problem is clearly exposed. The monitoring tools hopefully will alarm either about process failure inside of container or about growing value in the container related memory.memsw.failcnt file
  • in many cases once the problem is detected, it can be quickly fixed: increase limits for particular container, fix possible bugs in the software etc
  • no need to deal with almost not responsive server when it is not possible to investigate the current status and understand the problem reasons. Usually the only way to get the system working again is just to reboot the server.
12

Hello Stephan,
Could you give a brief explanation about the “user space” you mentioned in your post? To be honest, I didn’t understand the problem

Thank you in advance,
Leonid

13

mysql doesn’t look at /proc/meminfo but some others definitely do, the Java Virtual Machine is one of those which actively look at /proc/meminfo to size its memory allocation.

14

I got it. Thank you.
I still think that not controlling the swap may cause much more serious problems like the whole server stuck.
Is there any ‘hook’ where it is possible to define a custom script that will be executed automatically when a container has started? If so it is possible to change the memory.memsw.limit_in_bytes in this script.
Also, I think that it is not a good idea to automatically populate the memory.memsw.limit_in_bytes file with RAM size configured for container. I suggest to put a big number there instead.
Leonid

15

You may be able to manually set those values by using raw.lxc for them.

16

Ok, thank you very much! I’ll try to do it.