Strange RAM usage on host

LXD
1

Hello,

On many hosts I’ve a strange problem: RAM usage increase suddenly (for example +20GB), and then stay this high, until I drop cache (echo 3 > /proc/sys/vm/drop_caches , need to do it several times).

I can’t figure out what is using this memory, as it is NOT reported to by used as “cache” in ‘top’.
The top (ordered by ram usage) before drop_cache: https://prnt.sc/lxd45p (29GB used)
The top after dropping cache: https://prnt.sc/lxd64e (dropped to 5GB)

Munin RAM graph: http://prntscr.com/lxdbj3
Could this be related to LXD ?

2

And that memory is reported as actual use by say free -m rather than as bufferred/cache (which would be normal)?

It could be ZFS’s cache as that may get weirdly reported, are you using zfs and if so, how big of a zpool do you have?

3

Hello Stéphane,

yes for free-m:

free -m

total used free shared buff/cache available
Mem: 64048 35412 25593 28 3043 27911
Swap: 1021 0 1021
35GB used here instead of < 5GB (3GB buuf/cache only).

zpool list

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
default 796G 53.6G 742G - 5% 6% 2.12x ONLINE -

On this host there is even no VM yet.

lxc list

±-----±------±-----±-----±-----±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-----±------±-----±-----±-----±----------+

Only one storage volume:

lxc storage volume list default

±-------±-----------±------------±--------+
| TYPE | NAME | DESCRIPTION | USED BY |
±-------±-----------±------------±--------+
| custom | references | | 0 |
±-------±-----------±------------±--------+

And FYI the ‘top’ of this machine sorted by ram usage desc: http://prntscr.com/lxl4md
And munin ram usage: http://prntscr.com/lxl5sz

Using lxd 3.8

4

I found that in most cases it’s related to file copy, so probably ZFS cache.
Strange it’s not reported as cache.

grep ^c /proc/spl/kstat/zfs/arcstats

c 4 33579814912
c_min 4 33554432
c_max 4 33579814912

seems the 33GB are here…
looking deeper this way, thank you Stéphane

Trying this: https://www.solaris-cookbook.eu/linux/linux-ubuntu/debian-ubuntu-centos-zfs-on-linux-zfs-limit-arc-cache/

1 Like
5

Thanks for the info.

Default config with 32GB ram. 50% is recommended.
c_min 4 1047002752 (1gbits)
c_max 4 16752044032 (max 16gbits)

6

hi,
I am having the same problem, that the container is displaying the cache from the host as used memory:

Host:
free -mh
total used free shared buff/cache available
Mem: 125G 32G 1.3G 1.3G 91G 89G
Swap: 8.0G 80M 7.9G

Container:
free -mh
total used free shared buffers cached
Mem: 96G 95G 741M 0B 0B 119M
-/+ buffers/cache: 95G 860M
Swap: 8.0G 80M 7.9G

I’ts the only container on this system.
In my case its mainly not the zfs cache, its the dentry
that is using 91 GB of the cache.

The main question is, how can I get LXD not to assume the cache from the host as used memory?

I am using lxd 3.0.1

7

That’d likely be a LXCFS issue, make sure you’re running current LXCFS first, current version is 3.0.3, it may be something that’s been fixed already.

9

hi,
I updated it to 3.0.3 shorty after your post, but the ram usage increased again.

In the meantime I found out, that if I use slabtop on the hardware it showed 60 GB used, what is shown on the hardware as cache, but inside the container the dentry usage was tagged as used memory, not as cache.

After I did a echo 2 > /proc/sys/vm/drop_caches and waited 15 min to finish, the cache was emptied and the memory inside the container was free too.

Is there a possibility to limit the dentry space, or get lxd to display the dentry space as cache as in the physical host?