Containers Fail To Start Ubuntu 21.10

LXD
1

Hello all,

I’m running into a persistent issue on multiple machines. Two machines I upgraded from Ubuntu 21.04 to 21.10, LXC containers no longer start.

On a system with a clean install of Ubuntu 21.10, lxc containers were able to start after installing LXD. Then on reboot, the containers no longer start.

I’m able to launch containers, in that they download the requested image from the images repo, and the container appears in lxc list. It just refuses to start.

I’ve tried removing the snap with --purge so that it keeps nothing (none of my containers are production, they’re all to test and experiment so I treat them as ephemeral) and then installing the snap again, and the issue persists. On the clean install of 21.10, I switched the snap to latest/candidate so it’s now running 4.21. Still no joy.

Here’s the output of journalctl -u snap.lxd.daemon -n 300 which includes creating a container and then migrating the snap to 4.21 and trying to start the container again

Dec 11 08:45:57 TheRossFrame systemd[1]: Started Service for snap application lxd.daemon.
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: => Preparing the system (21902)
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Loading snap configuration
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up mntns symlink (mnt:[4026532684])
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/storage-pools
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up mount propagation on /var/snap/lxd/common/lxd/devices
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up persistent shmounts path
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ====> Making LXD shmounts use the persistent path
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ====> Making LXCFS use the persistent path
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up kmod wrapper
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Preparing /boot
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Preparing a clean copy of /run
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Preparing /run/bin
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Preparing a clean copy of /etc
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Preparing a clean copy of /usr/share/misc
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up ceph configuration
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up LVM configuration
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up OVN configuration
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Rotating logs
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Setting up ZFS (2.0)
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Escaping the systemd cgroups
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ====> Detected cgroup V2
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Escaping the systemd process resource limits
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Increasing the number of inotify user instances
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Increasing the number of keys for a nonroot user
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: ==> Disabling shiftfs on this kernel (auto)
Dec 11 08:45:57 TheRossFrame lxd.daemon[56761]: => Starting LXCFS
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: Running constructor lxcfs_init to reload liblxcfs
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: mount namespace: 5
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: hierarchies:
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: 0: fd: 6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: Kernel supports pidfds
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: Kernel does not support swap accounting
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: api_extensions:
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - cgroups
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - sys_cpu_online
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_cpuinfo
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_diskstats
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_loadavg
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_meminfo
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_stat
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_swaps
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - proc_uptime
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - shared_pidns
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - cpuview_daemon
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - loadavg_daemon
Dec 11 08:45:57 TheRossFrame lxd.daemon[56906]: - pidfds
Dec 11 08:45:58 TheRossFrame lxd.daemon[56761]: => Starting LXD
Dec 11 08:45:59 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:45:59-0600 lvl=warn msg=" - Couldn’t find the CGroup devices controller, device access control won’t work"
Dec 11 08:45:59 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:45:59-0600 lvl=warn msg=" - Couldn’t find the CGroup freezer controller, pausing/resuming containers won’t work"
Dec 11 08:45:59 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:45:59-0600 lvl=warn msg=" - Couldn’t find the CGroup network priority controller, network priority will be ignored"
Dec 11 08:46:01 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:46:01-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:46:01 TheRossFrame lxd.daemon[56761]: => LXD is ready
Dec 11 08:46:29 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:46:29-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:46:30 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:46:30-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:46:33 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:46:33-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:46:34 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:46:34-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:52:17 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:52:17-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:52:18 TheRossFrame lxd.daemon[56917]: t=2021-12-11T08:52:18-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:56:41 TheRossFrame systemd[1]: Stopping Service for snap application lxd.daemon…
Dec 11 08:56:41 TheRossFrame lxd.daemon[66136]: => Stop reason is: snap refresh
Dec 11 08:56:41 TheRossFrame lxd.daemon[66136]: => Stopping LXD
Dec 11 08:56:43 TheRossFrame lxd.daemon[56761]: => LXD exited cleanly
Dec 11 08:56:43 TheRossFrame lxd.daemon[66136]: ==> Stopped LXD
Dec 11 08:56:43 TheRossFrame systemd[1]: snap.lxd.daemon.service: Deactivated successfully.
Dec 11 08:56:43 TheRossFrame systemd[1]: Stopped Service for snap application lxd.daemon.
Dec 11 08:56:52 TheRossFrame systemd[1]: Started Service for snap application lxd.daemon.
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: => Preparing the system (22046)
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Loading snap configuration
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Setting up mntns symlink (mnt:[4026532684])
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Setting up kmod wrapper
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Preparing /boot
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Preparing a clean copy of /run
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Preparing /run/bin
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Preparing a clean copy of /etc
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Preparing a clean copy of /usr/share/misc
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Setting up ceph configuration
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Setting up LVM configuration
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Setting up OVN configuration
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Rotating logs
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Setting up ZFS (2.0)
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Escaping the systemd cgroups
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ====> Detected cgroup V2
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Escaping the systemd process resource limits
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: ==> Disabling shiftfs on this kernel (auto)
Dec 11 08:56:52 TheRossFrame lxd.daemon[56906]: Running destructor lxcfs_exit
Dec 11 08:56:52 TheRossFrame lxd.daemon[66629]: => Starting LXCFS
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: Running constructor lxcfs_init to reload liblxcfs
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: mount namespace: 5
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: hierarchies:
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: 0: fd: 6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: Kernel supports pidfds
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: Kernel does not support swap accounting
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: api_extensions:
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - cgroups
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - sys_cpu_online
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_cpuinfo
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_diskstats
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_loadavg
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_meminfo
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_stat
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_swaps
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - proc_uptime
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - shared_pidns
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - cpuview_daemon
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - loadavg_daemon
Dec 11 08:56:53 TheRossFrame lxd.daemon[66749]: - pidfds
Dec 11 08:56:54 TheRossFrame lxd.daemon[66629]: => Starting LXD
Dec 11 08:56:54 TheRossFrame lxd.daemon[66762]: t=2021-12-11T08:56:54-0600 lvl=warn msg=" - Couldn’t find the CGroup devices controller, device access control won’t work"
Dec 11 08:56:54 TheRossFrame lxd.daemon[66762]: t=2021-12-11T08:56:54-0600 lvl=warn msg=" - Couldn’t find the CGroup freezer controller, pausing/resuming containers won’t work"
Dec 11 08:56:54 TheRossFrame lxd.daemon[66762]: t=2021-12-11T08:56:54-0600 lvl=warn msg=" - Couldn’t find the CGroup network priority controller, network priority will be ignored"
Dec 11 08:56:55 TheRossFrame lxd.daemon[66762]: t=2021-12-11T08:56:55-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:56:55 TheRossFrame lxd.daemon[66629]: => LXD is ready
Dec 11 08:56:57 TheRossFrame lxd.daemon[66762]: t=2021-12-11T08:56:57-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”
Dec 11 08:56:58 TheRossFrame lxd.daemon[66762]: t=2021-12-11T08:56:58-0600 lvl=eror msg=“Error reading host’s cpuset.cpus”

I see errors for reading host cpuset, and I’m not sure how else to proceed. Any advice is much appreciated.

2

What’s the error you’re getting when the container fails to start?

3

No error is sent to the output. After sending the command “lxc start container” it accepts the command, but then running. Lxc list shows the container is still in a stopped state.

4

Okay, so that normally implies that the init process in the container immediately crashed.

lxc console --show-log NAME may be useful in such cases.

5

Thanks for pointing out the console log. I will have to write that down for later. Here is the output:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

So there it is. Mounting cgroup is not permitted. I’m still not sure where to go with this, since I would assume all of the permissions are set when the snap uncompresses itself. Although I will admit that I don’t understand how that whole side of things works.

6

What container images are those?

It sounds like you’re running older distros which don’t support cgroup2 on a system that only has cgroup2 enabled (Ubuntu 21.10).

You can force Ubuntu back into cgroup1 mode by booting your system with systemd.unified_cgroup_hierarchy=false set on the kernel command line.

7

These were all launched using the images repository. I have tried the following:

images:ubuntu/20.04
images:ubuntu/21.10
images:kali

All behave identically with the same console output from the previous post.

8

What does lxc config show --expanded look like for one of those?

9

Here is the expanded results for the 20.04 container.

architecture: x86_64
config:
image.architecture: amd64
image.description: Ubuntu focal amd64 (20211210_08:39)
image.os: Ubuntu
image.release: focal
image.serial: “20211210_08:39”
image.type: squashfs
image.variant: default
volatile.base_image: f6734866c479569f4c99bda7564096903372b395c94afd585933a5145a53a870
volatile.eth0.hwaddr: 00:16:3e:bb:26:65
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘’
volatile.last_state.power: STOPPED
volatile.uuid: 543f7967-3c23-4a4d-8335-60267de44a3f
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”
10

Not too sure what’s going on in your case, we usually don’t see quite that much breakage because of cgroup2.

In any case, booting your host system with systemd.unified_cgroup_hierarchy=false passed to the kernel should take care of this.