Hi,
I’m having an unusual problem that is cause only by certain software (firefox, chromium and a few other which I haven’t identified just yet but I suspect mysql).
Host information:
lxd-host # cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
lxd-host # uname -a
Linux 5.0.0-1026-gcp #27~18.04.1-Ubuntu SMP Fri Nov 15 07:40:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
lxd-host # cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.0.0-1026-gcp root=UUID=xxx ro scsi_mod.use_blk_mq=Y swapaccount=1 console=ttyS0
lxd-host #
lxd-host # apt-cache policy lxd
lxd:
Installed: 3.0.3-0ubuntu1~18.04.1
Candidate: 3.0.3-0ubuntu1~18.04.1
Version table:
[..]
lxd-host #
lxd-host # free -h
total used free shared buff/cache available
Mem: 3.6G 1.7G 136M 37M 1.8G 1.6G
Swap: 0B 0B 0B
lxd-host #
lxd-host # lxc config show xfce
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20190813.1)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: "20190813.1"
image.version: "18.04"
limits.memory: 256MB
limits.memory.enforce: hard
limits.memory.swap: "false"
volatile.base_image: 2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
volatile.eth0.hwaddr: 00:16:3e:11:c7:0f
volatile.idmap.base: "0"
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.power: RUNNING
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
lxd-host #
The memory limits are applied correctly for most software, eg: python.
root@xfce:~# free -h
total used free shared buff/cache available
Mem: 244M 146M 61M 7.3M 36M 97M
Swap: 0B 0B 0B
root@xfce:~# python
Python 2.7.15+ (default, Oct 7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = " " * 33554432
>>> b = " " * 33554432
>>> c = " " * 33554432
Killed
root@xfce:~#
However, if I start firefox/chromium, the container will start swapping heavily (although the host doesn’t have swap and container has hard limits, swap false) bringing down the host completely (load average goes over 100 and runs out of i/o completely).
Here’s how it looks:
root@xfce:~# sleep 30; killall -9 chromium-browser
root@xfce:~#
(this is to prevent the host from failing as my ssh connection will hang)
<start chromium-browser / firefox under x2go/rdesktop/etc>
root@xfce:~# while true; do free -h; sleep 3; done [5/33]
total used free shared buff/cache available
Mem: 244M 154M 39M 7.2M 49M 89M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 147M 46M 7.2M 49M 96M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 167M 160K 11M 76M 76M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 208M 36K 12M 35M 35M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 222M 92K 12M 21M 22M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 223M 100K 12M 21M 21M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 146M 56M 7.3M 40M 97M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 146M 56M 7.3M 40M 97M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 147M 56M 7.3M 40M 97M
Swap: 0B 0B 0B
total used free shared buff/cache available
Mem: 244M 147M 56M 7.3M 41M 97M
Swap: 0B 0B 0B
lxd-host # while true; do cat /proc/loadavg;lxc info xfce|grep -B 3 -i swap; sleep 3; done
1.60 2.03 1.97 3/1052 8583
Memory usage:
Memory (current): 227.24MB
Memory (peak): 310.05MB
Swap (current): 131.07kB
1.60 2.03 1.97 4/1051 8606
1.47 1.99 1.96 1/1052 8632
1.43 1.98 1.95 5/1079 8700
Memory usage:
Memory (current): 255.96MB
Memory (peak): 310.05MB
Swap (current): 28.67kB
1.43 1.98 1.95 4/1091 8757
1.40 1.96 1.95 8/1097 8784
Memory usage:
Memory (current): 255.96MB
Memory (peak): 310.05MB
Swap (current): -57344B
1.40 1.96 1.95 1/1095 8813
Memory usage:
Memory (current): 255.98MB
Memory (peak): 310.05MB
Swap (current): -94208B
1.93 2.06 1.98 3/1095 8836
Memory usage:
Memory (current): 255.96MB
Memory (peak): 310.05MB
Swap (current): 20.48kB
2.41 2.16 2.01 2/1096 8846
2.41 2.16 2.01 2/1096 8856
3.10 2.31 2.06 3/1097 8878
3.57 2.42 2.10 4/1097 8886
Memory usage:
Memory (current): 255.93MB
Memory (peak): 310.05MB
Swap (current): -208896B
3.57 2.42 2.10 2/1096 8898
4.33 2.59 2.16 3/1096 8923
4.33 2.59 2.16 2/1112 9063
5.50 2.87 2.25 2/1110 9077
5.14 2.83 2.24 3/1052 9110
5.14 2.83 2.24 1/1052 9142
By now the killall -9 took place, otherwise the loadavg would keep increasing until the host would crash as it wouldn’t have any i/o remaining.
Any idea what would cause this? Please let me know if I can provide any other information.
“For science” I tried docker and set the same limits. Firefox would be killed by the OOM killer within a few seconds, so it behaves as expected.
Thank you.