Incus fails to start any containers post migration from LXD

I recently migrated a LXD host to Incus and immediately had issues post-migration. I cannot start any containers and seem to be getting a ton of permission issues.

root@ip-10-0-49-70:/opt/incus/lib/lxc/rootfs# incus start ilm-backend
Error: Failed to run: /opt/incus/bin/incusd forkstart ilm-backend /var/lib/incus/containers /run/incus/ilm-backend/lxc.conf: exit status 1
Try `incus info --show-log ilm-backend` for more info
root@ip-10-0-49-70:/opt/incus/lib/lxc/rootfs# incus info --show-log ilm-backend
Name: ilm-backend
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2023/08/24 17:55 UTC
Last Used: 2024/02/22 21:03 UTC

Backups:
+---------+----------------------+----------------------+---------------+-------------------+
|  NAME   |       TAKEN AT       |      EXPIRES AT      | INSTANCE ONLY | OPTIMIZED STORAGE |
+---------+----------------------+----------------------+---------------+-------------------+
| backup0 | 2024/02/22 20:00 UTC | 2024/02/23 20:00 UTC | NO            | NO                |
+---------+----------------------+----------------------+---------------+-------------------+

Log:

lxc ilm-backend 20240222210340.733 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:fchowmodat:1619 - No such file or directory - Failed to fchownat(42, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc ilm-backend 20240222210340.819 ERROR    conf - ../src/lxc/conf.c:lxc_mount_rootfs:1428 - Permission denied - Failed to access to "/opt/incus/lib/lxc/rootfs". Check it is present
lxc ilm-backend 20240222210340.819 ERROR    conf - ../src/lxc/conf.c:lxc_setup_rootfs_prepare_root:3993 - Failed to setup rootfs for
lxc ilm-backend 20240222210340.819 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4368 - Failed to setup rootfs
lxc ilm-backend 20240222210340.819 ERROR    start - ../src/lxc/start.c:do_start:1272 - Failed to setup container "ilm-backend"
lxc ilm-backend 20240222210340.819 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc ilm-backend 20240222210340.829 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "vethf7b03bad"
lxc ilm-backend 20240222210340.829 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:878 - Received container state "ABORTING" instead of "RUNNING"
lxc ilm-backend 20240222210340.829 ERROR    start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "ilm-backend"
lxc ilm-backend 20240222210340.829 WARN     start - ../src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 43 for process 5133
lxc 20240222210341.478 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240222210341.484 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

We tracked it down on IRC to a bad permission on /opt.

When something like this happens, it’s useful to re-check directory permissions from / all the way to /opt/incus/lib/lxc/rootfs and make sure that an unprivileged user has at least traversal permission (1/+x bit set).

Cannot thank you enough man.

1 Like