Strange RAM usage on host


(Johann) #1

Hello,

On many hosts I’ve a strange problem: RAM usage increase suddenly (for example +20GB), and then stay this high, until I drop cache (echo 3 > /proc/sys/vm/drop_caches , need to do it several times).

I can’t figure out what is using this memory, as it is NOT reported to by used as “cache” in ‘top’.
The top (ordered by ram usage) before drop_cache: https://prnt.sc/lxd45p (29GB used)
The top after dropping cache: https://prnt.sc/lxd64e (dropped to 5GB)

Munin RAM graph: http://prntscr.com/lxdbj3
Could this be related to LXD ?


(Stéphane Graber) #2

And that memory is reported as actual use by say free -m rather than as bufferred/cache (which would be normal)?

It could be ZFS’s cache as that may get weirdly reported, are you using zfs and if so, how big of a zpool do you have?


(Johann) #3

Hello Stéphane,

yes for free-m:

free -m

total used free shared buff/cache available
Mem: 64048 35412 25593 28 3043 27911
Swap: 1021 0 1021
35GB used here instead of < 5GB (3GB buuf/cache only).

zpool list

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
default 796G 53.6G 742G - 5% 6% 2.12x ONLINE -

On this host there is even no VM yet.

lxc list

±-----±------±-----±-----±-----±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-----±------±-----±-----±-----±----------+

Only one storage volume:

lxc storage volume list default

±-------±-----------±------------±--------+
| TYPE | NAME | DESCRIPTION | USED BY |
±-------±-----------±------------±--------+
| custom | references | | 0 |
±-------±-----------±------------±--------+

And FYI the ‘top’ of this machine sorted by ram usage desc: http://prntscr.com/lxl4md
And munin ram usage: http://prntscr.com/lxl5sz

Using lxd 3.8


(Johann) #4

I found that in most cases it’s related to file copy, so probably ZFS cache.
Strange it’s not reported as cache.

grep ^c /proc/spl/kstat/zfs/arcstats

c 4 33579814912
c_min 4 33554432
c_max 4 33579814912

seems the 33GB are here…
looking deeper this way, thank you Stéphane

Trying this: https://www.solaris-cookbook.eu/linux/linux-ubuntu/debian-ubuntu-centos-zfs-on-linux-zfs-limit-arc-cache/


(Tom) #5

Thanks for the info.

Default config with 32GB ram. 50% is recommended.
c_min 4 1047002752 (1gbits)
c_max 4 16752044032 (max 16gbits)


(Dev Zero) #6

hi,
I am having the same problem, that the container is displaying the cache from the host as used memory:

Host:
free -mh
total used free shared buff/cache available
Mem: 125G 32G 1.3G 1.3G 91G 89G
Swap: 8.0G 80M 7.9G

Container:
free -mh
total used free shared buffers cached
Mem: 96G 95G 741M 0B 0B 119M
-/+ buffers/cache: 95G 860M
Swap: 8.0G 80M 7.9G

I’ts the only container on this system.
In my case its mainly not the zfs cache, its the dentry
that is using 91 GB of the cache.

The main question is, how can I get LXD not to assume the cache from the host as used memory?

I am using lxd 3.0.1


(Stéphane Graber) #7

That’d likely be a LXCFS issue, make sure you’re running current LXCFS first, current version is 3.0.3, it may be something that’s been fixed already.