Container stopped working

LXD
10

Hey,

I got the same issue on CentOS 7.

[root@centos7-test ~]# lxc --version
3.21
[root@centos7-test ~]# uname -a
Linux centos7-test.novalocal 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@centos7-test ~]# cat /etc/*release*
CentOS Linux release 7.7.1908 (Core)

Here’s how the container was launched:

lxc launch images:ubuntu/18.04/cloud NAME

Issue is fixed after:

lxc config set NAME security.privileged true

If it’s any help I can PM you access to this VM, it’s just a for testing.

11

Hi,

That would be quite helpful, yes.

12

I am also running into the same issue after upgrading to Centos 7.7 my current os and LXD versions are as follows:

CentOS Linux release 7.7.1908 (Core)
Linux 3.10.0-1062.12.1.el7.x86_64

lxc --version
3.21

I can confirm that setting the container to privileged does work, however I would prefer not to do so.

13

3.22 should behave, refreshing should get you that build now.

14

Thank you very much, however I can no longer start any containers after updating to 3.22. Even privileged containers now fail with the following error:

Console log:

systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA >+APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT >+GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN >-PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.4 LTS!

Set hostname to .
Failed to install release agent, ignoring: No such file or directory
Failed to create /lxc.payload/mycontainer/init.scope control group: Invalid argument
Failed to allocate manager object: Invalid argument
[!!!] Failed to allocate manager object, freezing.
Freezing execution.

15

Can you show cat /proc/self/mountinfo from within that container as well as cat /var/snap/lxd/common/lxd/logs/NAME/lxc.conf from the host?

16

The container is an Ubuntu 18.04 container from the ubunut:18.04 image. The contents of /proc/self/mountinfo are:

root@mycontainer:~# cat /proc/self/mountinfo
1213 453 0:185 /rootfs / rw,relatime master:342 - zfs Fast_Storage/system/lxd/containers/mycontainer rw,seclabel,xattr,posixacl
1214 1213 0:189 / /dev rw,relatime - tmpfs none rw,seclabel,size=492k,mode=755,uid=1000000,gid=1000000
1215 1213 0:188 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
1216 1213 0:190 / /sys rw,relatime - sysfs sysfs rw,seclabel
1217 1214 0:5 /fuse /dev/fuse rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1218 1214 0:5 /net/tun /dev/net/tun rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1219 1215 0:101 / /proc/sys/fs/binfmt_misc rw,relatime master:434 - binfmt_misc binfmt_misc rw
1220 1216 0:92 / /sys/fs/fuse/connections rw,relatime master:421 - fusectl fusectl rw
1221 1216 0:23 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:20 - pstore pstore rw
1222 1216 0:6 / /sys/kernel/debug rw,relatime master:25 - debugfs debugfs rw
1223 1216 0:17 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:7 - securityfs securityfs rw
1224 1214 0:15 / /dev/mqueue rw,relatime master:26 - mqueue mqueue rw,seclabel
1225 1214 0:91 / /dev/lxd rw,relatime - tmpfs tmpfs rw,seclabel,size=100k,mode=755
1226 1214 0:90 /mycontainer /dev/.lxd-mounts rw,relatime master:332 - tmpfs tmpfs rw,seclabel,size=100k,mode=711
1227 1216 0:191 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs none rw,seclabel,size=10240k,mode=755,uid=1000000,gid=100000
0
1250 1215 0:89 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1251 1215 0:89 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1252 1215 0:89 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1253 1215 0:89 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1254 1215 0:89 /proc/stat /proc/stat rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1255 1215 0:89 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1256 1215 0:89 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1257 1216 0:89 /sys/devices/system/cpu/online /sys/devices/system/cpu/online rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1258 1213 0:89 / /var/lib/lxcfs rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1228 1227 0:89 /cgroup/blkio /sys/fs/cgroup/blkio rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1229 1227 0:89 /cgroup/cpu /sys/fs/cgroup/cpu rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1230 1227 0:89 /cgroup/cpuset /sys/fs/cgroup/cpuset rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1231 1227 0:89 /cgroup/devices /sys/fs/cgroup/devices rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1232 1227 0:89 /cgroup/freezer /sys/fs/cgroup/freezer rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1233 1227 0:89 /cgroup/hugetlb /sys/fs/cgroup/hugetlb rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1234 1227 0:89 /cgroup/memory /sys/fs/cgroup/memory rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1235 1227 0:89 /cgroup/net_cls /sys/fs/cgroup/net_cls rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1236 1227 0:89 /cgroup/perf_event /sys/fs/cgroup/perf_event rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1237 1227 0:89 /cgroup/pids /sys/fs/cgroup/pids rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1238 1227 0:89 /cgroup/systemd /sys/fs/cgroup/systemd rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1239 1214 0:5 /full /dev/full rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1240 1214 0:5 /null /dev/null rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1241 1214 0:5 /random /dev/random rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1242 1214 0:5 /tty /dev/tty rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1243 1214 0:5 /urandom /dev/urandom rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1244 1214 0:5 /zero /dev/zero rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1245 1214 0:84 /9 /dev/console rw,relatime - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=666
454 1215 0:189 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime - tmpfs none rw,seclabel,size=492k,mode=755,uid=1000000,gid=1000000
455 1214 0:192 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,seclabel,gid=1000005,mode=620,ptmxmode=666,max=1024
456 1214 0:192 /ptmx /dev/ptmx rw,nosuid,noexec,relatime - devpts devpts rw,seclabel,gid=1000005,mode=620,ptmxmode=666,max=1024
457 1214 0:193 / /dev/shm rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,uid=1000000,gid=1000000
458 1213 0:194 / /run rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,mode=755,uid=1000000,gid=1000000
459 458 0:195 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,seclabel,size=5120k,uid=1000000,gid=1000000

The contents of /var/snap/lxd/common/lxd/logs/mycontainer/lxc.conf are:

lxc.log.file = /var/snap/lxd/common/lxd/logs/mycontainer/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/mycontainer/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/30499/exe callhook /var/snap/lxd/common/lxd 51 start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 51 stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 51 stop
lxc.tty.max = 0
lxc.uts.name = mycontainer
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/mycontainer
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/mycontainer:/dev/.lxd-mounts
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/mycontainer/rootfs
lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth11e74629

Thank you again for your help, please let me know what else I can provide.

17

@brauner anything we messed up in lxcfs with the file re-org which would explain why it’s not mounting a fake cgroupfs in this case?

The cgroup:mixed is the one we need for liblxc to mount something on /sys/fs/cgroup right?

18

Just to do a sanity check I spun up a Centos 7.7 vagrant image and installed the LXD 3.22 snap there and got the same results:

[root@localhost ~]# lxc console mycontainer --show-log

Console log:

systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.4 LTS!

Set hostname to .
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to install release agent, ignoring: No such file or directory
Failed to create /lxc.payload/mycontainer/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!] Failed to allocate manager object, freezing.
Freezing execution.

19

No, we also need lxc.hook.mount which isn’t present in this config, afaict.

20

Ah, I see it’s in the include.

21

But from the output it shows that is mounting a fake cgroupfs, no?

22

Hello,

Have exactly the same issue after upgrade from 3.21 to 3.22 (on arm64). Tried both versions from snap (rev 13718 3.22/stable and rev 13764 3.22/candidate).

Inside containers I see only 1 process init:

root@ns01:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  12116  3068 ?        Ss   14:13   0:00 /sbin/init
root       205  0.0  0.0   7300  3364 ?        Ss   14:13   0:00 bash
root       215  0.0  0.0   8680  2764 ?        R+   14:13   0:00 ps aux

In logs:

systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture arm64.

Welcome to Ubuntu 18.04.1 LTS!

Set hostname to <ns01>.
Failed to create compat systemd cgroup /lxc.payload/ns01/init.scope: Invalid argument
Failed to attach 1 to compat systemd cgroup /lxc.payload/ns01/init.scope: No such file or directory
Failed to open pin file: No such file or directory
Failed to allocate manager object: No such file or directory
[ESC[0;1;31m!!!!!!ESC[0m] Failed to allocate manager object, freezing.
Freezing execution.

Set to “security.privileged” and server reboot didn’t help.

23

We’ve reproduced the issue on a local test system.
@brauner will investigate shortly.

24

Tentative fix merged in lxcfs and being included in the candidate snap now.
We’ll validate on CentOS 7 before pushing to stable later today.

25

Seems to work fine here. Just need to wait for arm64 to be done building before releasing to stable.

26

Rolled out to stable.

27

Confirmed, arm64 version

  stable:         3.22        2020-03-16 (13771) 62MB -

works now without issues.

Thanks for fix that.

28

Good to hear, sorry about that.

29

Thanks for the fix.