Invalid SwapTotal in /proc/meminfo (SwapTotal 0)

Containers use swap from host. Normally the SwapTotal in /proc/meminfo inside the container is the same as that on the host. However, since the latest snap lxd upgrade (version 4.2, 15564) all containers on some hosts show a SwapTotal of 0.

host ~ # grep SwapTotal /proc/meminfo
SwapTotal: 33554428 kB

container ~ # grep SwapTotal /proc/meminfo
SwapTotal: 0 kB

A reboot of the host does not solve this problem.

Is your kernel booted with swapaccount=1? If not, this is unlikely to work.

Hmm. On none of the boxes swapaccount=1 has been set. Note that I’m not trying to limit swap for containers.

This box (A) the containers have the same swap report as the host:

uname -r
4.15.0-50-generic

grep CONFIG_MEMCG_SWAP /boot/config-*
/boot/config-4.15.0-101-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-101-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-50-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-50-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-99-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-99-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set

grep -r GRUB_CMDLINE_LINUX /etc/default/grub*
/etc/default/grub:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub:GRUB_CMDLINE_LINUX=""
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 modprobe.blacklist=btrfs"

On a similar host (B), the containers report:

grep ^Swap /proc/meminfo 
SwapCached:            0 kB
SwapTotal:             0 kB
SwapFree:              0 kB

uname -r
4.15.0-99-generic

grep CONFIG_MEMCG_SWAP /boot/config-*
/boot/config-4.15.0-101-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-101-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-66-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-66-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set
/boot/config-4.15.0-99-generic:CONFIG_MEMCG_SWAP=y
/boot/config-4.15.0-99-generic:# CONFIG_MEMCG_SWAP_ENABLED is not set

grep -r GRUB_CMDLINE_LINUX /etc/default/grub*
/etc/default/grub:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub:GRUB_CMDLINE_LINUX=""
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
/etc/default/grub.d/50-cloudimg-settings.cfg:GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 modprobe.blacklist=btrfs"

Ok and what’s the host value for those?

It’s most likely related to recent changes in swap reporting in lxcfs but we’ll need to know more to see what’s going on.

A:~# grep ^Swap /proc/meminfo
SwapCached:        42572 kB
SwapTotal:      33554428 kB
SwapFree:       33208572 kB
A-cntr:~# grep ^Swap /proc/meminfo
SwapCached:            0 kB
SwapTotal:      33554428 kB
SwapFree:       33208572 kB

B:~# grep ^Swap /proc/meminfo
SwapCached:            0 kB
SwapTotal:      33554428 kB
SwapFree:       33554428 kB
B-cntr:~# grep ^Swap /proc/meminfo
SwapCached:            0 kB
SwapTotal:             0 kB
SwapFree:              0 kB

@kbos Can you show:

  • ps aux | grep lxcfs
  • readlink /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe

On both systems?

A:~# ps aux | grep lxcfs
root      4231  0.0  0.0 234944  1632 ?        Ssl   2019   1:30 /usr/bin/lxcfs /var/lib/lxcfs/
root      5483  0.0  0.0 163344  1512 ?        Sl   Apr15   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root      5625  0.0  0.0 160976   152 ?        Sl   Feb18   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root      7897  0.0  0.0 163344  1532 ?        Sl   May06   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     11630  0.0  0.0 384760  1896 ?        Sl   Feb19   0:08 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     11679  0.2  0.0 752600  3124 ?        Sl    2019 867:10 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     12808  0.0  0.0 382580  1024 ?        Sl   Jan23   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     21432  0.0  0.0 160964   200 ?        Sl    2019   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     21865  0.0  0.0 604128   568 ?        Sl    2019 103:11 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     24113  0.0  0.0 163168  1272 ?        Sl   Apr02   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     27468  0.0  0.0  13136  1064 pts/0    S+   14:43   0:00 grep --color=auto lxcfs
root     28360  0.0  0.0  95440   136 ?        Sl    2019   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     32010  0.0  0.0 678052  1688 ?        Sl    2019  15:58 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root@lxh-sysa-015:~# readlink /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs

A:~# ls -l /usr/bin/lxcfs
-rwxr-xr-x 1 root root 18504 Mar 31 18:20 /usr/bin/lxcfs


B:~# ps aux | grep lxcfs
root      3655  0.0  0.0  13136  1120 pts/0    S+   14:43   0:00 grep --color=auto lxcfs
root     14295  0.0  0.0 234944  1680 ?        Ssl  Jun11   0:02 /usr/bin/lxcfs /var/lib/lxcfs/
root     25206  0.1  0.0 770676 12556 ?        Sl   Jun11  32:45 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid

B:~# readlink /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs

Oops, can you try readlink -f /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe instead?

A:~# readlink -f /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs

B:~# readlink -f /proc/$(cat /var/snap/lxd/common/lxcfs.pid)/exe
/bin/lxcfs

@brauner more SWAP fun. I also noticed here that systems with swapaccount disabled do not see any swap rather than see the host amount, that feels like a bug.

@brauner more SWAP fun. I also noticed here that systems with swapaccount disabled do not see any swap rather than see the host amount, that feels like a bug.

I’m this close to rm mm/swap* in the kernel sources.

Christian

Do you have any memory limits set on your containers? I had the same problem after the snap updated and found the swap was visible in containers with memory limits as below

  • limits.memory must be set to some value
  • limits.memory.enforce must be set to hard
  • limits.memory.swap must be set to false

I had only limits.memory. Tried this (with container restart):

config:
  limits.memory: 1GB
  limits.memory.enforce: hard
  limits.memory.swap: "false"

Didn’t help though.

You said earlier you that you didn’t have swap accounting enabled, is that still the case? It’s enabled on my server.

Indeed. I’ve still not enabled swap accounting on the system (I don’t really need swap accounting). For testing I’ve enabled swap accounting on one box and these limits you suggested do help to show SwapTotal and SwapFree (no container restart needed).

Can we get this fix in 4.0/stable? The workaround suggested by @kbos works. But I have containers, that should use all available memory. So setting limits.memory is not an option for me.

Can we expect a fix in the 4.0 LTS branch? The workaround does not seem to actually work.


(Everything below is copied from: No swap inside containers when using snap LXD )

Does this workaround actually work?

After enabling “swapaccount=1” via GRUB file and setting the above mentioned parameters I can now see 10GB swap inside my container, but it is never used.

I tried filling a tmpfs inside the container as well as letting processes run out of memory.
In both cases the RAM fills up until it is completely full and processes are being killed by the kernels OOM killer, all while swap stays at 0 bytes used.

Does anybody else experience this issue or has found a solution?
Thank you