depam
October 18, 2022, 2:22pm
1
Hi,
I just find out recently that the limits.memory is not respected when I added it in profiles.
lxd 5.6-794016a 23680 latest/stable canonical✓ -
snapd 2.57.2 17029 latest/stable canonical✓ snapd
lxc config show rhel-worker-node01 --expanded | less
architecture: x86_64
config:
image.architecture: amd64
image.description: Rockylinux 8 amd64 (20221017_02:07)
image.os: Rockylinux
image.release: "8"
image.serial: "20221017_02:07"
image.type: squashfs
image.variant: cloud
limits.cpu: "4"
limits.memory: 8GB
linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,br_netfilter
raw.lxc: "lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw
sys:rw"
security.nesting: "true"
security.privileged: "true"
and on container
lxc exec rhel-worker-node01 -- free -h
total used free shared buff/cache available
Mem: 62Gi 2.0Gi 59Gi 16Mi 1.1Gi 60Gi
Swap: 0B 0B 0B
It looks like it is still using the host memory. I tried shutdown, restart and recreate but it seems to be the same. Anyone knows why is it so? Thanks in advance.
depam
October 18, 2022, 2:25pm
2
Also i noticed that one weird that hosts sees the memory limits but going to docker container it runs, it seems it can see actual memory. But i guess its how cgroup works for docker on lxd?
root@POLOVM152160:~# lxc exec RHEL-cassandra01 -- free -h
total used free shared buff/cache available
Mem: 3.7Gi 2.5Gi 0.0Ki 0.0Ki 1.2Gi 1.2Gi
Swap: 0B 0B 0B
root@POLOVM152160:~# lxc exec RHEL-cassandra01 -- docker exec -ti db_cassandra_01 free -h
total used free shared buff/cache available
Mem: 62G 27G 21G 509M 14G 33G
Swap: 0B 0B 0B
root@POLOVM152160:~#
sdeziel
October 18, 2022, 5:19pm
3
I suspect the lxc.mount.auto=proc:rw you have in raw.lxc is preventing the lxcfs mount of /proc/meminfo which would explain why you are seeing the full host RAM.
1 Like
I suppose, something is wrong with the limits.memory. Here are some actions, Regards.
---------------------------------------Host system-----------------------------------
indiana@mars:~$ free -h
total used free shared buff/cache available
Mem: 15Gi 7.1Gi 2.4Gi 231Mi 6.1Gi 7.9Gi
Swap: 0B 0B 0B
---------------------------------------Host system-----------------------------------
indiana@mars:~$ lxc launch images:alpine/edge/cloud test
Creating test
Starting test
indiana@mars:~$ lxc config set test limits.memory=2GiB
indiana@mars:~$ lxc exec test -- free -h
total used free shared buff/cache available
Mem: 15.6G 13.1G 2.4G 196.2M 71.9M 2.0G
Swap: 0 0 0
indiana@mars:~$ lxc config show test
architecture: x86_64
config:
image.architecture: amd64
image.description: Alpine edge amd64 (20221013_13:01)
image.os: Alpine
image.release: edge
image.requirements.secureboot: "false"
image.serial: "20221013_13:01"
image.type: squashfs
image.variant: cloud
limits.memory: 2GiB
volatile.base_image: a0ef203cabd21ba6390eef390228e6fd143331ae4273af47640843604c5e1bf9
volatile.cloud-init.instance-id: a86386ce-17db-4850-a905-978b2196d1bc
volatile.eth0.host_name: veth6613c407
volatile.eth0.hwaddr: 00:16:3e:75:91:3b
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
volatile.uuid: 4d982e7e-8341-4a40-b267-998634cdb427
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
indiana@mars:~$ lxc shell test
test:~# free -h
total used free shared buff/cache available
Mem: 15.6G 13.1G 2.4G 196.2M 71.9M 2.0G
Swap: 0 0 0
sdeziel
October 18, 2022, 7:04pm
5
Alpine’s free uses sysinfo rather than consulting /proc/meminfo, see LXD 5.1 has been released for how to make it “see” the right memory limit.
1 Like
Thanks @sdeziel for valuable information.
1 Like
depam
October 19, 2022, 11:16am
7
Hi @sdeziel @cemzafer , thanks for info. It looks like this affects all and not distribution specific? I can see different behaviour for e.g. ubuntu:20.04 and rockylinux/8/cloud. I will try with that parameters recently introduced. Thanks
depam
October 20, 2022, 2:08pm
8
Hi on version 5.0.1-9dcf35b, it seems like that parameter is not allowed?
Config parsing error: Unknown configuration key: security.syscalls.intercept.sysinfo
Press enter to open the editor again or ctrl+c to abort change
sdeziel
October 20, 2022, 2:22pm
9
@depam that is correct, that was added in version 5.1.
depam
October 20, 2022, 2:43pm
10
I just did snap refresh to latest/stable but still doesn’t seem to work
~$ lxc config show RHEL-cassandra01 --expanded | head -n 20
architecture: x86_64
config:
image.architecture: amd64
image.description: Rockylinux 8 amd64 (20221020_02:07)
image.os: Rockylinux
image.release: "8"
image.serial: "20221020_02:07"
image.type: squashfs
image.variant: cloud
limits.cpu: "2"
limits.memory: 4GB
linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
raw.lxc: "lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw
sys:rw"
security.nesting: "true"
security.privileged: "true"
security.syscalls.intercept.sysinfo: "true"
$ lxc exec RHEL-cassandra01 -- free -h
total used free shared buff/cache available
Mem: 62Gi 95Mi 62Gi 8.0Mi 18Mi 62Gi
Swap: 0B 0B 0B
sdeziel
October 20, 2022, 3:55pm
11
@depam , I don’t think that RHEL’s free uses the sysinfo call to retrieve the memory info. If that’s indeed right, I still think the lxc.mount.auto=proc:rw setting you are using is getting in the way.
depam
October 20, 2022, 5:47pm
12
Thanks @sdeziel . I just now saw the video and looks it working with just these parameters:
security.nesting: "true"
security.syscalls.intercept.mknod: "true"
security.syscalls.intercept.setxattr: "true"
security.syscalls.intercept.sysinfo: "true"
1 Like
