Some questions on limits.memory.enforce = soft

LXD
1

As I understand it, with the “soft” setting, a container can exceed its allocated memory, as long as the host memory is not full. If so:

  1. Is the amount of memory a container can “over-allocate” always 100% of host memory, or can it be limited somehow (aside from hard limits)?
  2. I assume it would be possible for a “soft” container to use up all available (free) memory on the host. What happens when the host is out of memory, is the “soft” container (and only that container) blocked from allocating new memory until back down at its original soft limit? And when it is, presumably it can then start all over again?
  3. Is host swap space included in the total “available memory”, or kept separate? I’d like to experiment with swap for long running containers with little activity, but wouldn’t want it to affect what is considered total memory available.

Regarding #2, I am a bit wary of trying out soft limits based on what I think I’ve understood so far since it seems like hypothetically, other containers could end up with large parts of their memory in swap or get nothing at all just because one container ran away and used up all the host memory, refusing to de-allocate it.

I guess another way to phrase it would be: what is the point of soft limits? Because if you have no limit, that already means all containers share the host memory equally. With hard limits, everything is sectioned up. If you give all containers soft limits, they can again grab all the host memory they want – so is the point of “soft” simply to have a mechanism to force the usage down when 100% usage is reached, which wouldn’t happen without limits?

Thank you in advance.

2 
7. Soft limits

Soft limits allow for greater sharing of memory. The idea behind soft limits
is to allow control groups to use as much of the memory as needed, provided

a. There is no memory contention
b. They do not exceed their hard limit

When the system detects memory contention or low memory, control groups
are pushed back to their soft limits. If the soft limit of each control
group is very high, they are pushed back as much as possible to make
sure that one control group does not starve the others of memory.

Please note that soft limits is a best-effort feature; it comes with
no guarantees, but it does its best to make sure that when memory is
heavily contended for, memory is allocated based on the soft limit
hints/setup. Currently soft limit based reclaim is set up such that
it gets invoked from balance_pgdat (kswapd).

That’s from the cgroup doc at https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

We’ve not seen wide adoption of this and I’m not sure how it all works with cgroup2 these days. In theory the way you could use it is by having some containers that are allowed to burst their memory usage through a soft limit while the rest have hard limits in place. Should the system experience memory pressure, those with hard limits won’t really be impacted whereas those with soft limits will start encountering issues until they’re back below their limit.

1 Like
3

I’m a little bit confused, but maybe I’m just not understanding it correctly. For example:

The idea behind soft limits is to allow control groups to use as much of the memory as needed, provided (…) they do not exceed their hard limit

This would imply that you can set both a soft and a hard limit, since “exceeding [the] hard limit” would be a criteria for not being allowed to use more memory, but from what I understand limits.memory.enforce can only be set to either hard or soft?

Should the system experience memory pressure, those with hard limits won’t really be impacted

Does this mean that hard limits somehow “reserve” memory capacity on the host, even if it’s not used? My impression was that given a total host memory of 16G, if you have:

  • foo (6G hard limit, utilizing 1G)
  • bar (10G soft limit, utilizing 15G)

Wouldn’t this mean that even if bar is prohibited from allocating new memory due to pressure, foo is still incapable of using all of its hard-limited memory allowance until it’s been freed from bar?

I’m also curious if the presence of swap would make any difference in a scenario like this, with or without settings like limits.memory.swap.priority – would it make a soft-limited container have its memory end up in swap to facilitate meeting the “hard” requirements for a container if it had higher priority?

4

Correct, the documentation above comes from the kernel documentation where it’s possible to have both soft and hard limits concurrently. LXD does not support that.

No, it doesn’t, but when under memory pressure, the system will prioritize memory evacuation from those using soft limits over those using hard limits. It may also impact who gets ultimately targeted by the OOM killer.

I would expect so. Basically if running out of space, I would expect the kernel to prevent new allocations from a container that’s got a soft limit which will typically cause memory to be swapped out to make up space for the requested allocation.

1 Like