Fedora 33: CentOS7 guest not working (cannot determine cgroup)

LXD
1

I have recently changed OS to Fedora 33 and I am currently having some issues with CentOS 7 guests. Others guests, including CentOS8 seems to work ok.

I can start the container, and exec into it fine, but systemd is not running, i.e:

# systemctl status
Failed to get D-Bus connection: No such file or directory

If I get the log I see the following with an unpriviledged container:

lxc console fleet-werewolf --show-log
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement

Console log:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

    Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
    Cannot determine cgroup we are running in: No such file or directory
    Failed to allocate manager object: No such file or directory
    [!!!!!!] Failed to allocate manager object, freezing.

If I try making the container priviledged I get:
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement

Console log:

systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

I have attempted to disable cgroupv2 by calling:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierachy=0"

And rebooted but without help.

On my other distro the same image seems to works fine.

2

The symptoms seems to be exactly the same as described here: Container stopped working but only for CentOS7 guest images.

Some more info based on the above:

On host/CentOS8 guest/CentOS7 guest:

cat /proc/self/mountinfo | grep "cgroup"
1522 1469 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,seclabel

CentOS 8 (working):

sudo cat /var/snap/lxd/common/lxd/logs/finer-hen/lxc.conf
lxc.log.file = /var/snap/lxd/common/lxd/logs/finer-hen/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/finer-hen/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:rw:force
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/1313/exe callhook /var/snap/lxd/common/lxd "default" "finer-hen" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "finer-hen" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "finer-hen" stop
lxc.tty.max = 0
lxc.uts.name = finer-hen
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/finer-hen
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/finer-hen:/dev/.lxd-mounts
lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth5a180d5f
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/finer-hen/rootfs

CentOS7 (seems very much the same):

sudo cat /var/snap/lxd/common/lxd/logs/centos7/lxc.conf
lxc.log.file = /var/snap/lxd/common/lxd/logs/centos7/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/centos7/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:rw:force
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/1313/exe callhook /var/snap/lxd/common/lxd "default" "centos7" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "centos7" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "centos7" stop
lxc.tty.max = 0
lxc.uts.name = centos7
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/centos7
lxc.idmap = u 0 1000000 1000000000
lxc.idmap = g 0 1000000 1000000000
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/centos7:/dev/.lxd-mounts
lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth0a3d7cbb
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/centos7/rootfs
3

Upgrading to Fedora 34 and set cgroups to verison 1 with command:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

solve the issue for me.