Error mounting rootfs on Debian

Compiled v4.22 from source on Ubuntu, then moved the binaries and dependencies to a fresh Debian 11 machine, upon launching a container it can’t mount the rootfs. It’s trying in /usr/lib/x86_64-linux-gnu/lxc, which doesn’t make much sense, any advice appreciated.

Kernel:

Linux debian 5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) x86_64 GNU/Linu

cat /proc/self/cgroup:

12:devices:/user.slice
11:freezer:/
10:memory:/user.slice/user-0.slice/session-1.scope
9:cpu,cpuacct:/user.slice
8:rdma:/
7:blkio:/user.slice
6:net_cls,net_prio:/
5:hugetlb:/
4:perf_event:/
3:cpuset:/
2:pids:/user.slice/user-0.slice/session-1.scope
1:name=systemd:/user.slice/user-0.slice/session-1.scope
0::/user.slice/user-0.slice/session-1.scope

cat /proc/1/mounts:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=483896k,nr_inodes=120974,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=99984k,mode=755 0 0
/dev/sda / ext4 rw,relatime,errors=remount-ro 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10610 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=99980k,nr_inodes=24995,mode=700 0 0
tmpfs /var/lib/lxd/shmounts tmpfs rw,relatime,size=100k,mode=711 0 0
tmpfs /var/lib/lxd/devlxd tmpfs rw,relatime,size=100k,mode=755 0 0

lxc info --show-log test:

lxc test 20220126044307.872 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1129 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.test"
lxc test 20220126044307.875 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1129 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.test"
lxc test 20220126044307.896 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1547 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc test 20220126044307.907 ERROR    conf - conf.c:lxc_mount_rootfs:1258 - No such file or directory - Failed to access to "/usr/lib/x86_64-linux-gnu/lxc". Check it is present
lxc test 20220126044307.907 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3155 - Failed to setup rootfs for
lxc test 20220126044307.907 ERROR    conf - conf.c:lxc_setup:3291 - Failed to setup rootfs
lxc test 20220126044307.907 ERROR    start - start.c:do_start:1218 - Failed to setup container "test"
lxc test 20220126044307.908 ERROR    sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc test 20220126044307.908 WARN     network - network.c:lxc_delete_network_priv:3183 - Failed to rename interface with index 2 from "eth0" to its initial name "vethcf18372a"
lxc test 20220126044307.908 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:859 - Received container state "ABORTING" instead of "RUNNING"
lxc test 20220126044307.912 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "test"
lxc test 20220126044307.913 WARN     start - start.c:lxc_abort:1012 - No such process - Failed to send SIGKILL via pidfd 30 for process 1547
lxc 20220126044312.982 WARN     commands - commands.c:lxc_cmd_rsp_recv:124 - Connection reset by peer - Failed to receive response for command "get_state"

With the build of LXC you’re using, /usr/lib/x86_64-linux-gnu/lxc will need to exist as an empty directory on your system. This is used to stage the container’s mounts (inside a mount namespace).

Thanks Stéphane, also needed to create the rootfs directory, was a bit thrown off by the location.

A note on the install docs, it says minimum Go 1.16 but had to upgrade to 1.17, also the raft and dqlite dependencies are now in lxd-4.22/vendor not go/deps

Cheers mate.