Dear all!
We’re evaluating the full LXD/LXC/LXCFS 5.0.0 LTS chain and it seems to work fine in most cases. But when creating a nested container, we’re running into said error. I’m out of ideas and knowledge on what I am missing.
~# lxc info --show-log local:c2
[...]
lxc c2 20220503094300.251 ERROR cgfsng - cgroups/cgfsng.c:__initialize_cgroups:3274 - Not a directory - Failed to open 6/net_prio
lxc c2 20220503094300.251 ERROR cgfsng - cgroups/cgfsng.c:initialize_cgroups:3434 - Not a directory - Failed to initialize cgroups
lxc c2 20220503094300.252 ERROR cgroup - cgroups/cgroup.c:cgroup_init:33 - Bad file descriptor - Failed to initialize cgroup driver
lxc c2 20220503094300.252 ERROR start - start.c:lxc_init:865 - Failed to initialize cgroup driver
lxc c2 20220503094300.252 ERROR start - start.c:__lxc_start:2008 - Failed to initialize container "c2"
lxc c2 20220503094327.883 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:869 - No such file or directory - Failed to receive the container state
lxc 20220503094327.884 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220503094327.884 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"
lxc 20220503094327.884 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220503094327.884 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"
For reporting purposes I mostly followed the instructions on Nested containers in LXD | Ubuntu, but I had to switch to using a privileged first container or I run into another issue (which we can look at after separately or so - unprivileged nested container would be nice).
I highly believe it worked fine with 4.24 in this exact configuration, but I can’t prove that right now. If needed, I could make an effort to test.
So, the host:
~$ lsb_release -a
No LSB modules are available.
Distributor ID: Devuan
Description: Devuan GNU/Linux 4 (chimaera)
Release: 4
Codename: chimaera
~$ uname -a
Linux manderinli 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
~$ cat /etc/subuid /etc/subgid
root:500000:196608
root:500000:196608
The full chain is built using ubuntu deb source packages as reference, where applicable and maintained (hello lxd) but mostly patched sysvinit in. See bottom of the post for more details if needed.
So we end up with those packages installed on the host:
~$ dpkg -l | grep lx
ii liblxc-common 1:5.0.0-2xxx1~4.1 amd64 Linux Containers userspace tools (common tools)
ii liblxc1 1:5.0.0-2xxx1~4.1 amd64 Linux Containers userspace tools (library)
ii lxcfs 5.0.0-1xxx1~4.1 amd64 FUSE based filesystem for LXC
ii lxd 5.0.0-1xxx1~4.1 amd64 Container hypervisor based on LXC - daemon
ii lxd-client 5.0.0-1xxx1~4.1 amd64 Container hypervisor based on LXC - client
~$ dpkg -l | grep cgroup
ii cgroupfs-mount 1.4+devuan1 all Light-weight package to set up cgroupfs mounts
~$ lxc launch ubuntu:jammy c1privubuntu -c security.nesting=true -c security.privileged=true
Creating c1privubuntu
Starting c1privubuntu
~$ lxc exec c1privubuntu bash
# using snaps lxd and i did not modify any /etc/subuid /etc/subgid, I just use what is shipped in ubuntu:jammy
root@c1privubuntu:~# lxd init --auto
root@c1privubuntu:~# lxc launch ubuntu:jammy c2
Creating c2
Starting c2
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart c2 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/c2/lxc.conf:
Try `lxc info --show-log local:c2` for more info
Error see above at the beginning of the post. I’m out of ideas what the culprit here is.
Details to the packages if needed:
LXD is built based on GitHub - lxc/lxd-pkg-ubuntu at dpm-bionic but with a lot of updates to account for all the changes in the (inline) dependencies, compilation and new binaries since then. I was updating this through several 4.x releases and I’m fairly familiar with it - but there could be mistakes! (Re-)Added sysvinit stuff to make it work with devuan.
LXC is usually built based on GitHub - lxc/lxc-pkg-ubuntu at dpm-jammy, but for the jammy release you did a sneaky 5.0.0 prerelease build 
, so that source is based on the real source package of Ubuntu – Details of source package lxc in jammy. Patched to ship with sysvinit and dependency to cgroupfs-mount, since we don’t have systemd here to do cgroup stuff.
LXCFS is built based on lxc / lxcfs · GitLab as referenced by Ubuntu – Details of source package lxcfs in jammy. Also shipping the old sysvinit stuff.
