[Solved: Not a bug] Strange cgroups bug(?) on Ubuntu 21.10

LXD
#1

Summary:
On an Ubuntu 21.10 host, run lxc launch images:78ad7419038a centos7. The container starts, but hangs in /sbin/init. On the console is cgroup2: Unknown parameter 'mode'

This happened when I did my first dist upgrade of a LXD host. All the containers froze on startup, and wouldn’t get past that.

To get some more debugging, I then set the init to be /bin/bash:

lxc stop --force centos7
lxc config set centos7 raw.lxc 'lxc.init.cmd = /bin/bash'
lxc start centos7 --console
To detach from the console, press: <ctrl>+a q

[root@centos7 /]# exec /sbin/init
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

I also found this ticket - https://github.com/lxc/lxc/issues/4072 - which references a similar problem.

Works fine on 21.04, doesn’t work on 21.10.

#2

And as an almost instant update, for those that are coming along later, this is NOT an issue with LXD/LXC.

The problem is that the systemd that is being launched in the container does not understand cgroups2, and the host does not support cgroups1.

You can fix this by (on the host) adding systemd.unified_cgroup_hierarchy=0 to the kernel command line. For example, the relevant parts of /etc/default/grub are now:

GRUB_CMDLINE_LINUX_DEFAULT="mitigatons=off"
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 systemd.unified_cgroup_hierarchy=0"

Adding that makes everything work again. This will be something you will need to keep there forever, basically, as the C7 systemd will never support cgroups2.

#3

We have a new feature in LXD 4.23 (to be released later this week) which combined with some tweaks on our image server should lead to clear errors when attempting to launch CentOS 7, Ubuntu 16.04 or other distros that will fail on pure cgroup2.

2 Likes
#4

We just encountered this error with centos7 containers that failed to boot after upgrading from ubuntu 20.04 to 22.04.

Upgrading to snap lxd 4.23/candidate did not fix this issue as the fix requires install from new C7 images I assume?

So for existing upgrade path, the only fix for us was

systemd.unified_cgroup_hierarchy=0

grub/kernel config on host.

#5

All the new image requirements do is alert you to that same requirements as you’ve used, rather than the unclear error message you get from the container.

#6

And yes they will need to use updated images.

1 Like