Strange problem with lxc limits.memory

I have a strange situation with the limits.memory:

When the memory size is ≥ 64GiB, it takes no effect, but once I had set another memory size, and set it 64GiB again, it will work:

The launch command:

lxc launch -p default -p k8s -s ssd dev-physical-0-19:centos-8-base dev-host-0-76-k8s-data-node -c limits.cpu=32 -c limits.memory=64GiB

Then the problem:

root@dev-physical-0-19:~# lxc exec  dev-host-0-76-k8s-data-node  -- sh -c "free -h "
              total        used        free      shared  buff/cache   available
Mem:          376Gi        77Mi       376Gi        42Mi        53Mi       376Gi
Swap:            0B          0B          0B

root@dev-physical-0-19:~# lxc config set dev-host-0-76-k8s-data-node limits.memory=64GiB
root@dev-physical-0-19:~# lxc exec dev-host-0-76-k8s-data-node -- sh -c "free -h "
              total        used        free      shared  buff/cache   available
Mem:          376Gi        77Mi       376Gi        42Mi        53Mi       376Gi
Swap:            0B          0B          0B

root@dev-physical-0-19:~# lxc config set  dev-host-0-76-k8s-data-node   limits.memory=48GiB
root@dev-physical-0-19:~# lxc exec  dev-host-0-76-k8s-data-node   -- sh -c "free -h "
              total        used        free      shared  buff/cache   available
Mem:           48Gi        77Mi        47Gi        42Mi        53Mi        47Gi
Swap:            0B          0B          0B

root@dev-physical-0-19:~# lxc config set  dev-host-0-76-k8s-data-node   limits.memory=64GiB
root@dev-physical-0-19:~# lxc exec  dev-host-0-76-k8s-data-node   -- sh -c "free -h "
              total        used        free      shared  buff/cache   available
Mem:           64Gi        77Mi        63Gi        42Mi        53Mi        63Gi
Swap:            0B          0B          0B

All of my lxd servers have the same problem.

ENV Info:

root@dev-physical-0-19:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 21.10
Release:        21.10
Codename:       impish

root@dev-physical-0-19:~# lxd version
4.24

root@dev-physical-0-19:~# lxc profile show default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    limits.read: 10000MB
    limits.write: 10000MB
    path: /
    pool: lxc-storage
    type: disk
name: default
used_by:
...

root@dev-physical-0-19:~# lxc profile show k8s
config:
  boot.autostart: "true"
  limits.memory.swap: "false"
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,br_netfilter
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw cgroup:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /sys/module/nf_conntrack/parameters/hashsize
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk
  aadisable2:
    path: /dev/kmsg
    source: /dev/kmsg
    type: unix-char
  aadisable3:
    path: /sys/fs/bpf
    source: /sys/fs/bpf
    type: disk
  aadisable4:
    path: /proc/sys/net/netfilter/nf_conntrack_max
    source: /proc/sys/net/netfilter/nf_conntrack_max
    type: disk
name: k8s
used_by:
...

The logs of lxd service has nothing about this.

I have make some progress:
With the k8s profile, the launch command can not set it correctly. I have to launch it and then config it:

root@dev-physical-0-17:~# lxc launch -p default dev-physical-0-19:centos-8-base c1 -c limits.cpu=1 -c limits.memory=256MiB
Creating c1
Starting c1                                
root@dev-physical-0-17:~# lxc exec c1 -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:          256Mi        44Mi       203Mi       8.0Mi       8.0Mi       211Mi
Swap:         256Mi          0B       256Mi
root@dev-physical-0-17:~# lxc stop c1
root@dev-physical-0-17:~# lxc delete c1
root@dev-physical-0-17:~# lxc launch -p k8s -p default dev-physical-0-19:centos-8-base c1 -c limits.cpu=1 -c limits.memory=256MiB
Creating c1
Starting c1                                
root@dev-physical-0-17:~# lxc exec c1 -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:          376Gi        50Mi       376Gi        17Mi        19Mi       376Gi
Swap:            0B          0B          0B
root@dev-physical-0-17:~# lxc stop c1 
root@dev-physical-0-17:~# lxc delete c1
root@dev-physical-0-17:~# lxc launch -p k8s -p default dev-physical-0-19:centos-8-base c1 
Creating c1
Starting c1                                
root@dev-physical-0-17:~# lxc config set c1 limits.cpu=1 limits.memory=256MiB
root@dev-physical-0-17:~# lxc exec c1 -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:          256Mi        46Mi       190Mi        17Mi        19Mi       209Mi
Swap:            0B          0B          0B

But when I restart the container, and the limits.memory is not working again.

root@dev-physical-0-17:~# lxc config set c1 limits.memory 1GiB
root@dev-physical-0-17:~# lxc config set c1 limits.memory 256MiB
root@dev-physical-0-17:~# lxc exec c1 -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:          256Mi        46Mi       190Mi        17Mi        19Mi       209Mi
Swap:            0B          0B          0B
root@dev-physical-0-17:~# lxc restart c1
root@dev-physical-0-17:~# lxc exec c1 -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:          376Gi        45Mi       376Gi        17Mi        17Mi       376Gi
Swap:         0.0Ki       0.0Ki          0B

The cgroup file also can not be set correctly:

root@dev-physical-0-17:~# cat /sys/fs/cgroup/memory/lxc.payload.c1/memory.limit_in_bytes 
9223372036854771712

Any ideas?

What distribution, kernel and LXD package is this?

root@dev-physical-0-17:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 21.10
Release:        21.10
Codename:       impish
root@dev-physical-0-17:~# uname -a
Linux dev-physical-0-17 5.13.0-30-generic #33-Ubuntu SMP Fri Feb 4 17:03:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@dev-physical-0-17:~# lxd version
4.24
root@dev-physical-0-17:~# snap list 
Name           Version       Rev    Tracking         Publisher   Notes
core           16-2.54.4     12821  latest/stable    canonical✓  core
core18         20220309      2344   latest/stable    canonical✓  base
core20         20220318      1405   latest/stable    canonical✓  base
distrobuilder  2.0           1125   latest/stable    stgraber    classic
lxd            4.24-c92c0b2  22754  latest/stable/…  canonical✓  -
snapd          2.54.4        15177  latest/stable    canonical✓  snapd

I think may be the 「k8s」profile allow the 「cgroup fs:rw」 that leads to this problem…?

:innocent: :innocent: :innocent:

root@dev-physical-0-19:~# lxc config set dev-host-0-53-qa-enterprise-standalone limits.memory 16GiB
root@dev-physical-0-19:~# lxc exec  dev-host-0-53-qa-enterprise-standalone  bash 
[root@dev-host-0-53-qa-enterprise-standalone ~]# free -h 
              total        used        free      shared  buff/cache   available
Mem:          376Gi       3.3Gi       366Gi       741Mi       6.4Gi       373Gi
Swap:            0B          0B          0B
[root@dev-host-0-53-qa-enterprise-standalone ~]# exit 
exit
root@dev-physical-0-19:~# lxc config set dev-host-0-53-qa-enterprise-standalone limits.memory 24GiB
root@dev-physical-0-19:~# lxc exec  dev-host-0-53-qa-enterprise-standalone  bash 
[root@dev-host-0-53-qa-enterprise-standalone ~]# free -h 
              total        used        free      shared  buff/cache   available
Mem:           24Gi       3.3Gi        14Gi       741Mi       6.4Gi        20Gi
Swap:            0B          0B          0B
[root@dev-host-0-53-qa-enterprise-standalone ~]# exit 
exit
root@dev-physical-0-19:~# cat /sys/fs/cgroup/
blkio/            cpuacct/          cpuset/           freezer/          memory/           net_cls/          net_prio/         pids/             systemd/          
cpu/              cpu,cpuacct/      devices/          hugetlb/          misc/             net_cls,net_prio/ perf_event/       rdma/             unified/          
root@dev-physical-0-19:~# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
9223372036854771712
root@dev-physical-0-19:~# lxc config set dev-host-0-53-qa-enterprise-standalone limits.memory 16GiB
root@dev-physical-0-19:~# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
9223372036854771712
root@dev-physical-0-19:~# lxc exec dev-host-0-53-qa-enterprise-standalone -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:           16Gi       3.4Gi       6.2Gi       741Mi       6.4Gi        12Gi
Swap:            0B          0B          0B
root@dev-physical-0-19:~# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
9223372036854771712
root@dev-physical-0-19:~# 

I have already solved the problem. The problem it that when the profile has the above 2 seetings at the same time, the 「limits.memory」 will not work.

security.privileged: “true”
limits.memory.swap: “false”

It must be a bug, please fix it on a new version of lxd.

root@dev-physical-0-17:~# lxc profile show test
config:
  limits.memory.swap: "false"
  security.privileged: "true"
description: ""
devices: {}
name: test
used_by:
- /1.0/instances/c1
root@dev-physical-0-17:~# lxc -p default -p test launch dev-physical-0-19:centos-8-base c1 -c limits.cpu=1 -c limits.memory=256MiB
Creating c1
Starting c1                                
root@dev-physical-0-17:~# lxc exec c1 -- sh -c "free -h"
              total        used        free      shared  buff/cache   available
Mem:          376Gi        34Mi       376Gi       8.0Mi       8.0Mi       376Gi
Swap:            0B          0B          0B

Please can you log the issue over at Issues · lxc/lxd · GitHub

Thanks

Done. The issue is log at the lxd 「limits.momory」is not working when turn on the setting: 「security.privileged: “true” 」and 「limits.memory.swap: “false”」 · Issue #10154 · lxc/lxd · GitHub

1 Like