Container stopped working

LXD
1

Hello,

I have a container that was working fine until this past weekend. Over the weekend, the host was powered down and brought back up. After the restart, my container stopped working in the following ways:

symptom/failure #1: systemctl doesn’t work
When I attempt to run systemctl (in order to start tomcat and apache), I get the following error message.

System has not been booted with systemd as init system (PID 1). Can't operate.

symptom/failure #2:
LXC container no longer has network connectivity. Unable to even ping the host. (Also tried a DNS lookup with no success.) The IPv4 address the container gets is 127.0.0.1. When it was working, its address was 10.129.221.80.

symptom/failure #3:
Cannot shutdown the container normally. Running ‘shutdown’ gives the similar error as when I try to run systemctl.

System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to talk to init daemon.

Even trying to shut it down from the host using “lxc stop” doesn’t work. Although “lxc stop” with force does halt the container.

Other information that may be relevant
The container is running Ubuntu 18.04.3. The host is Centos 7. (Yes, weird I know. But it is what it is for now.)

The LXC software could have been updated as part of the power cycle. The LXC version is currently 3.21, which I believe is relatively recent. I had not run a software update on the host in months. So that tells me that this power cycle caused the LXC version upgrade. So perhaps an LXC upgrade is behind these failures?

Can anyone think of a reason why all these failures would pop up all at once. I’m thinking there must be a single cause to this, as everything was working before the power cycle.

Thanks for any assistance anyone can provide.

2

When I run

lxc console <container> --show-log

I get

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted.
[!!!!!!] Failed to mount API filesystem, freezing. Freezing execution.

This is likely related?

3

Right, that usually indicates that your host system doesn’t have the name=systemd cgroup properly mounted, containers running systemd are then prevented by the kernel from setting up that mount, causing this failure to boot.

4

The ‘solution’ I found to my problems is to make the container privileged. Once I made the container privileged, it worked like it did before. Is this the right approach or am I leaving open a significant security vulnerability?

I’m sure that I didn’t have the container as privileged before. So not sure why things stopped working due to the power cycle.

Thanks for the info.

5

Privileged containers on a CentOS kernel can rather trivially obtain root access on the host, so not good for security, no.

On a broken system, can you show grep cgroup /proc/self/mountinfo on the host?

6

Sure, here it is:

25 18 0:21 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:8 - tmpfs tmpfs ro,mode=755
26 25 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
28 25 0:24 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,blkio
29 25 0:25 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,cpuacct,cpu
30 25 0:26 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,hugetlb
31 25 0:27 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,perf_event
32 25 0:28 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,net_prio,net_cls
33 25 0:29 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,pids
34 25 0:30 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,cpuset,clone_children
35 25 0:31 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,devices
36 25 0:32 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,freezer
37 25 0:33 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,memory
7

Ok, so name=systemd is there… that’s odd.

@brauner any ideas? It’s the second report we get of some CentOS 7 systems not being able to mount a new copy of name=systemd inside a container…

Our own test system hasn’t been hit though.

8

name=systemd must be writable I think, especially when using fully unprivileged containers. If that’s not the problem then I’m not sure. There are quite a bunch of fixes sitting in the LXC master branch that all rework cgroup handling though and we’re about to release 4.0.

9

Note that liblxc itself is quite happy, things only blow up when systemd tries to mount it.

10

Hey,

I got the same issue on CentOS 7.

[root@centos7-test ~]# lxc --version
3.21
[root@centos7-test ~]# uname -a
Linux centos7-test.novalocal 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@centos7-test ~]# cat /etc/*release*
CentOS Linux release 7.7.1908 (Core)

Here’s how the container was launched:

lxc launch images:ubuntu/18.04/cloud NAME

Issue is fixed after:

lxc config set NAME security.privileged true

If it’s any help I can PM you access to this VM, it’s just a for testing.

11

Hi,

That would be quite helpful, yes.

12

I am also running into the same issue after upgrading to Centos 7.7 my current os and LXD versions are as follows:

CentOS Linux release 7.7.1908 (Core)
Linux 3.10.0-1062.12.1.el7.x86_64

lxc --version
3.21

I can confirm that setting the container to privileged does work, however I would prefer not to do so.

13

3.22 should behave, refreshing should get you that build now.

14

Thank you very much, however I can no longer start any containers after updating to 3.22. Even privileged containers now fail with the following error:

Console log:

systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA >+APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT >+GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN >-PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.4 LTS!

Set hostname to .
Failed to install release agent, ignoring: No such file or directory
Failed to create /lxc.payload/mycontainer/init.scope control group: Invalid argument
Failed to allocate manager object: Invalid argument
[!!!] Failed to allocate manager object, freezing.
Freezing execution.

15

Can you show cat /proc/self/mountinfo from within that container as well as cat /var/snap/lxd/common/lxd/logs/NAME/lxc.conf from the host?

16

The container is an Ubuntu 18.04 container from the ubunut:18.04 image. The contents of /proc/self/mountinfo are:

root@mycontainer:~# cat /proc/self/mountinfo
1213 453 0:185 /rootfs / rw,relatime master:342 - zfs Fast_Storage/system/lxd/containers/mycontainer rw,seclabel,xattr,posixacl
1214 1213 0:189 / /dev rw,relatime - tmpfs none rw,seclabel,size=492k,mode=755,uid=1000000,gid=1000000
1215 1213 0:188 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
1216 1213 0:190 / /sys rw,relatime - sysfs sysfs rw,seclabel
1217 1214 0:5 /fuse /dev/fuse rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1218 1214 0:5 /net/tun /dev/net/tun rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1219 1215 0:101 / /proc/sys/fs/binfmt_misc rw,relatime master:434 - binfmt_misc binfmt_misc rw
1220 1216 0:92 / /sys/fs/fuse/connections rw,relatime master:421 - fusectl fusectl rw
1221 1216 0:23 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:20 - pstore pstore rw
1222 1216 0:6 / /sys/kernel/debug rw,relatime master:25 - debugfs debugfs rw
1223 1216 0:17 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:7 - securityfs securityfs rw
1224 1214 0:15 / /dev/mqueue rw,relatime master:26 - mqueue mqueue rw,seclabel
1225 1214 0:91 / /dev/lxd rw,relatime - tmpfs tmpfs rw,seclabel,size=100k,mode=755
1226 1214 0:90 /mycontainer /dev/.lxd-mounts rw,relatime master:332 - tmpfs tmpfs rw,seclabel,size=100k,mode=711
1227 1216 0:191 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs none rw,seclabel,size=10240k,mode=755,uid=1000000,gid=100000
0
1250 1215 0:89 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1251 1215 0:89 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1252 1215 0:89 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1253 1215 0:89 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1254 1215 0:89 /proc/stat /proc/stat rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1255 1215 0:89 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1256 1215 0:89 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1257 1216 0:89 /sys/devices/system/cpu/online /sys/devices/system/cpu/online rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1258 1213 0:89 / /var/lib/lxcfs rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1228 1227 0:89 /cgroup/blkio /sys/fs/cgroup/blkio rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1229 1227 0:89 /cgroup/cpu /sys/fs/cgroup/cpu rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1230 1227 0:89 /cgroup/cpuset /sys/fs/cgroup/cpuset rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1231 1227 0:89 /cgroup/devices /sys/fs/cgroup/devices rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1232 1227 0:89 /cgroup/freezer /sys/fs/cgroup/freezer rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1233 1227 0:89 /cgroup/hugetlb /sys/fs/cgroup/hugetlb rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1234 1227 0:89 /cgroup/memory /sys/fs/cgroup/memory rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1235 1227 0:89 /cgroup/net_cls /sys/fs/cgroup/net_cls rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1236 1227 0:89 /cgroup/perf_event /sys/fs/cgroup/perf_event rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1237 1227 0:89 /cgroup/pids /sys/fs/cgroup/pids rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1238 1227 0:89 /cgroup/systemd /sys/fs/cgroup/systemd rw,nosuid,nodev,relatime master:326 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,default_permissions,allow_other
1239 1214 0:5 /full /dev/full rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1240 1214 0:5 /null /dev/null rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1241 1214 0:5 /random /dev/random rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1242 1214 0:5 /tty /dev/tty rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1243 1214 0:5 /urandom /dev/urandom rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1244 1214 0:5 /zero /dev/zero rw,nosuid master:2 - devtmpfs devtmpfs rw,seclabel,size=65899660k,nr_inodes=16474915,mode=755
1245 1214 0:84 /9 /dev/console rw,relatime - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=666
454 1215 0:189 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime - tmpfs none rw,seclabel,size=492k,mode=755,uid=1000000,gid=1000000
455 1214 0:192 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,seclabel,gid=1000005,mode=620,ptmxmode=666,max=1024
456 1214 0:192 /ptmx /dev/ptmx rw,nosuid,noexec,relatime - devpts devpts rw,seclabel,gid=1000005,mode=620,ptmxmode=666,max=1024
457 1214 0:193 / /dev/shm rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,uid=1000000,gid=1000000
458 1213 0:194 / /run rw,nosuid,nodev - tmpfs tmpfs rw,seclabel,mode=755,uid=1000000,gid=1000000
459 458 0:195 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,seclabel,size=5120k,uid=1000000,gid=1000000

The contents of /var/snap/lxd/common/lxd/logs/mycontainer/lxc.conf are:

lxc.log.file = /var/snap/lxd/common/lxd/logs/mycontainer/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/mycontainer/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/30499/exe callhook /var/snap/lxd/common/lxd 51 start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 51 stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 51 stop
lxc.tty.max = 0
lxc.uts.name = mycontainer
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/mycontainer
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/mycontainer:/dev/.lxd-mounts
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/mycontainer/rootfs
lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth11e74629

Thank you again for your help, please let me know what else I can provide.

17

@brauner anything we messed up in lxcfs with the file re-org which would explain why it’s not mounting a fake cgroupfs in this case?

The cgroup:mixed is the one we need for liblxc to mount something on /sys/fs/cgroup right?

18

Just to do a sanity check I spun up a Centos 7.7 vagrant image and installed the LXD 3.22 snap there and got the same results:

[root@localhost ~]# lxc console mycontainer --show-log

Console log:

systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 18.04.4 LTS!

Set hostname to .
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to install release agent, ignoring: No such file or directory
Failed to create /lxc.payload/mycontainer/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!] Failed to allocate manager object, freezing.
Freezing execution.

19

No, we also need lxc.hook.mount which isn’t present in this config, afaict.

20

Ah, I see it’s in the include.