Hi,
I’m having some issues with containers seeing their buffered/cached memory as used. I thought it had something to do with memory limits, but it still comes to a point where services get OOM’ed killed, after I’ve disabled them.
I run docker inside the container and might be something related to that. Atleast it’s easy to reproduce by spinning up a bunch of docker containers inside the LXC.
lxc config show compute-xlarge-1
architecture: x86_64
config:
image.build_ver: 10f16957b38b3df1cb525889296835c4868c3d4661a7fcd1040d78da1c364379
image.name: base_image
image.os: ubuntu
image.version: "20.04"
limits.cpu: 26-33
limits.memory: 16GB
security.nesting: "true"
user.access_interface: eth1
volatile.base_image: 42aa515d369bf585f35c774fa10f8dae74087ac952aa7cce4c59c19e93c5a4ae
volatile.eth0.host_name: veth65d4e87d
volatile.eth0.hwaddr: 00:16:3e:b4:95:cd
volatile.eth1.host_name: veth897966a1
volatile.eth1.hwaddr: 00:16:3e:50:ab:87
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.uuid: b5626105-3bc9-4b57-9262-e5dd4a89bea8
ephemeral: false
profiles:
- default
- vlan
stateful: false
description: ""
Fresh start of compute-xlarge-1
free -m
total used free shared buff/cache available
Mem: 15258 115 14842 0 301 15143
Swap: 0 0 0
After I’ve created 100 nginx docker containers, with docker run -ti -d nginx
957 MB used, this is still fair, as we have the 100 containers running.
free -m
total used free shared buff/cache available
Mem: 15258 957 12457 5 1843 14300
Swap: 0 0 0
It more than doubled the amount of memory used, after I’ve stopped and deleted all the 100 containers. This seems odd.
free -m
total used free shared buff/cache available
Mem: 15258 2414 11815 0 1028 12843
Swap: 0 0 0
Thought it might be the docker service, well stopping that lowered usage a little bit, but still quite high for what’s running
free -m
total used free shared buff/cache available
Mem: 15258 1513 12824 0 920 13744
Swap: 0 0 0
ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 169640 8300 ? Ss 17:25 0:01 /sbin/init
root 82 0.0 0.4 133580 64536 ? Ss 17:25 0:01 /lib/systemd/systemd-journald
root 117 0.0 0.0 21608 3036 ? Ss 17:25 0:00 /lib/systemd/systemd-udevd
systemd+ 172 0.0 0.0 27056 5276 ? Ss 17:25 0:00 /lib/systemd/systemd-networkd
root 200 0.0 0.0 237312 4164 ? Ssl 17:25 0:00 /usr/lib/accountsservice/accounts-daemon
root 209 0.0 0.0 8536 1432 ? Ss 17:25 0:00 /usr/sbin/cron -f
message+ 211 0.0 0.0 7384 3040 ? Ss 17:25 0:00 /usr/bin/dbus-daemon --system --address=systemd: --n
root 214 0.0 0.0 29528 14592 ? Ss 17:25 0:01 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-
syslog 215 0.0 0.0 154864 2976 ? Ssl 17:25 0:00 /usr/sbin/rsyslogd -n -iNONE
root 218 0.0 0.0 16804 3088 ? Ss 17:25 0:00 /lib/systemd/systemd-logind
daemon 221 0.0 0.0 3792 1032 ? Ss 17:25 0:00 /usr/sbin/atd -f
root 235 0.0 0.0 7352 1232 console Ss+ 17:25 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud conso
root 236 0.0 0.0 108084 14552 ? Ssl 17:25 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unat
root 237 0.0 0.0 12176 4680 ? Ss 17:25 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 start
root 258 0.0 0.0 232716 3612 ? Ssl 17:25 0:00 /usr/lib/policykit-1/polkitd --no-debug
root 509 0.0 0.0 8960 3256 ? Ss 17:25 0:00 bash
root 2809 0.0 0.0 8960 3936 ? Ss 17:31 0:00 bash
root 3045 0.0 0.0 9140 4888 ? S+ 17:31 0:04 htop
root 43901 0.0 0.0 10616 3304 ? R+ 17:37 0:00 ps aux
Memory consumption comes to a reasonable level after clearing cache with the following command on the host
sync && echo 3 > /proc/sys/vm/drop_caches
free -m
total used free shared buff/cache available
Mem: 15258 767 14360 0 130 14491
Swap: 0 0 0
Any help would be much appreciated!