Containers use swap from host. Normally the SwapTotal in /proc/meminfo inside the container is the same as that on the host. However, since the latest snap lxd upgrade (version 4.2, 15564) all containers on some hosts show a SwapTotal of 0.
host ~ # grep SwapTotal /proc/meminfo
SwapTotal: 33554428 kB
container ~ # grep SwapTotal /proc/meminfo
SwapTotal: 0 kB
A reboot of the host does not solve this problem.
Is your kernel booted with swapaccount=1? If not, this is unlikely to work.
Hmm. On none of the boxes swapaccount=1 has been set. Note that I’m not trying to limit swap for containers.
This box (A) the containers have the same swap report as the host:
uname -r
4.15.0-50-generic
grep CONFIG_MEMCG_SWAP /boot/config-*
/boot/config-4.15.0-101-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-101-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-50-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-50-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-99-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-99-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
grep -r GRUB_CMDLINE_LINUX /etc/default/grub*
/etc/default/grub:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub:GRUB_CMDLINE_LINUX=""
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 modprobe.blacklist=btrfs"
On a similar host (B), the containers report:
grep ^Swap /proc/meminfo
SwapCached: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
uname -r
4.15.0-99-generic
grep CONFIG_MEMCG_SWAP /boot/config-*
/boot/config-4.15.0-101-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-101-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-66-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-66-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-99-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-99-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
grep -r GRUB_CMDLINE_LINUX /etc/default/grub*
/etc/default/grub:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub:GRUB_CMDLINE_LINUX=""
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 modprobe.blacklist=btrfs"
Ok and what’s the host value for those?
It’s most likely related to recent changes in swap reporting in lxcfs but we’ll need to know more to see what’s going on.
A:~# grep ^Swap /proc/meminfo
SwapCached: 42572 kB
SwapTotal: 33554428 kB
SwapFree: 33208572 kB
A-cntr:~# grep ^Swap /proc/meminfo
SwapCached: 0 kB
SwapTotal: 33554428 kB
SwapFree: 33208572 kB
B:~# grep ^Swap /proc/meminfo
SwapCached: 0 kB
SwapTotal: 33554428 kB
SwapFree: 33554428 kB
B-cntr:~# grep ^Swap /proc/meminfo
SwapCached: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
@kbos Can you show:
On both systems?
A:~# ps aux | grep lxcfs
root 4231 0.0 0.0 234944 1632 ? Ssl 2019 1:30 /usr/bin/lxcfs /var/lib/lxcfs/
root 5483 0.0 0.0 163344 1512 ? Sl Apr15 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 5625 0.0 0.0 160976 152 ? Sl Feb18 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 7897 0.0 0.0 163344 1532 ? Sl May06 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 11630 0.0 0.0 384760 1896 ? Sl Feb19 0:08 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 11679 0.2 0.0 752600 3124 ? Sl 2019 867:10 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 12808 0.0 0.0 382580 1024 ? Sl Jan23 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 21432 0.0 0.0 160964 200 ? Sl 2019 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 21865 0.0 0.0 604128 568 ? Sl 2019 103:11 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 24113 0.0 0.0 163168 1272 ? Sl Apr02 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 27468 0.0 0.0 13136 1064 pts/0 S+ 14:43 0:00 grep --color=auto lxcfs
root 28360 0.0 0.0 95440 136 ? Sl 2019 0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 32010 0.0 0.0 678052 1688 ? Sl 2019 15:58 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root@lxh-sysa-015:~# readlink /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs
A:~# ls -l /usr/bin/lxcfs
-rwxr-xr-x 1 root root 18504 Mar 31 18:20 /usr/bin/lxcfs
B:~# ps aux | grep lxcfs
root 3655 0.0 0.0 13136 1120 pts/0 S+ 14:43 0:00 grep --color=auto lxcfs
root 14295 0.0 0.0 234944 1680 ? Ssl Jun11 0:02 /usr/bin/lxcfs /var/lib/lxcfs/
root 25206 0.1 0.0 770676 12556 ? Sl Jun11 32:45 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
B:~# readlink /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs
Oops, can you try readlink -f /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe instead?
A:~# readlink -f /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs
B:~# readlink -f /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs
@brauner more SWAP fun. I also noticed here that systems with swapaccount disabled do not see any swap rather than see the host amount, that feels like a bug.
@brauner more SWAP fun. I also noticed here that systems with swapaccount disabled do not see any swap rather than see the host amount, that feels like a bug.
I’m this close to rm mm/swap* in the kernel sources.
Christian
Do you have any memory limits set on your containers? I had the same problem after the snap updated and found the swap was visible in containers with memory limits as below
limits.memory must be set to some valuelimits.memory.enforce must be set to hard
limits.memory.swap must be set to false
I had only limits.memory. Tried this (with container restart):
config:
limits.memory: 1GB
limits.memory.enforce: hard
limits.memory.swap: "false"
Didn’t help though.
You said earlier you that you didn’t have swap accounting enabled, is that still the case? It’s enabled on my server.
Indeed. I’ve still not enabled swap accounting on the system (I don’t really need swap accounting). For testing I’ve enabled swap accounting on one box and these limits you suggested do help to show SwapTotal and SwapFree (no container restart needed).
Can we get this fix in 4.0/stable? The workaround suggested by @kbos works. But I have containers, that should use all available memory. So setting limits.memory is not an option for me.
Can we expect a fix in the 4.0 LTS branch? The workaround does not seem to actually work.
(Everything below is copied from: No swap inside containers when using snap LXD )
Does this workaround actually work?
After enabling “swapaccount=1” via GRUB file and setting the above mentioned parameters I can now see 10GB swap inside my container, but it is never used.
I tried filling a tmpfs inside the container as well as letting processes run out of memory.
In both cases the RAM fills up until it is completely full and processes are being killed by the kernels OOM killer, all while swap stays at 0 bytes used.
Does anybody else experience this issue or has found a solution?
Thank you