Container won't start - lxc_rootfs_init - Bad file descriptor

Hello - following a system update recently my LXC container will not start.
This container runs Zoneminder.
I’ve attempted to start the container with …
lxc-start zoneminder -o /zfsdata/scratch/lxc-start-zoneminder.log -l DEBUG
and here is the tail end of that log file where things seems to go wrong …

lxc-start zoneminder 20210509204155.138 INFO     start - start.c:lxc_init:855 - Container "zoneminder" is initialized
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-1) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-1)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-1)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-2) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-2)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-2)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-3) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-3)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-3)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-4) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-4)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-4)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-5) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-5)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-5)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-6) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-6)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-6)
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:__cgroup_tree_create:771 - File exists - Creating the final cgroup 12(lxc.monitor.zoneminder-7) failed
lxc-start zoneminder 20210509204155.138 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_tree_create:831 - File exists - Failed to create monitor cgroup 12(lxc.monitor.zoneminder-7)
lxc-start zoneminder 20210509204155.138 DEBUG    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1056 - Failed to create cgroup lxc.monitor.zoneminder-7)
lxc-start zoneminder 20210509204155.188 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1070 - The monitor process uses "lxc.monitor.zoneminder-8" as cgroup
lxc-start zoneminder 20210509204155.188 DEBUG    storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "zfs"
lxc-start zoneminder 20210509204155.188 ERROR    conf - conf.c:lxc_rootfs_init:557 - Bad file descriptor - Failed to open "zfsdata/lxcstore/zoneminder"
lxc-start zoneminder 20210509204155.189 ERROR    start - start.c:__lxc_start:2045 - Failed to handle rootfs pinning for container "zoneminder"
lxc-start zoneminder 20210509204155.189 WARN     cgfsng - cgroups/cgfsng.c:cgfsng_payload_destroy:538 - Uninitialized limit cgroup
lxc-start zoneminder 20210509204155.189 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:868 - Received container state "ABORTING" instead of "RUNNING"
lxc-start zoneminder 20210509204155.189 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start zoneminder 20210509204155.189 ERROR    lxc_start - tools/lxc_start.c:main:311 - To get more details, run the container in foreground mode
lxc-start zoneminder 20210509204155.189 ERROR    lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start zoneminder 20210509204155.189 WARN     cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:955 - Device or resource busy - Failed to destroy 12(lxc.monitor.zoneminder-8)

And here’s the containers config file …

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist archlinux --release current --arch amd64
# Template script checksum (SHA-1): 1ba3a6d6544626d6e64c7b8f1a51f6022c5e5f8f
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = x86_64

# Container specific configuration
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = zfs:zfsdata/lxcstore/zoneminder
lxc.uts.name = zoneminder
lxc.start.auto = 1
lxc.start.order = 0
lxc.start.delay = 0
lxc.group = onboot

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.mtu = 1500

lxc-checkconfig shows …

LXC version 4.0.9
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:


Cgroup v2 mount points:
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup namespace: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

The system is running Arch Linux with the following relevant packages installed:
lxc 1:4.0.9-1
systemd 248-5
linux 5.10.35-1-lts
zfs-dkms 2.0.4-2

Would appreciate some help troubleshooting this!

I’ll check right in the morning.

Ok, this is already fixed in master and I’ve backported it to our stable-4.0 branch. We’ll likely release 4.0.10 very soon which will have this fix. So this is:


and