None of the containers can be started!

itviewer@JackMa:~$ incus start ubuntu22
Error: Failed to run: /opt/incus/bin/incusd forkstart ubuntu22 /var/lib/incus/containers /run/incus/ubuntu22/lxc.conf: exit status 1
Try `incus info --show-log ubuntu22` for more info
itviewer@JackMa:~$ incus info --show-log ubuntu22
Name: ubuntu22
Description: dev
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/11/19 21:28 CST
Last Used: 2025/11/28 20:17 CST

Log:

lxc ubuntu22 20251128121708.289 ERROR    utils - ../src/lxc/utils.c:safe_mount:1334 - Permission denied - Failed to mount "/var/lib/incus/containers/ubuntu22/credentials" onto "/opt/incus/lib/lxc/rootfs/dev/.incus-systemd-credentials"
lxc ubuntu22 20251128121708.289 ERROR    conf - ../src/lxc/conf.c:mount_entry:2202 - Permission denied - Failed to mount "/var/lib/incus/containers/ubuntu22/credentials" on "/opt/incus/lib/lxc/rootfs/dev/.incus-systemd-credentials"
lxc ubuntu22 20251128121708.289 ERROR    conf - ../src/lxc/conf.c:lxc_setup:3908 - Failed to setup mount entries
lxc ubuntu22 20251128121708.289 ERROR    start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "ubuntu22"
lxc ubuntu22 20251128121708.289 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc ubuntu22 20251128121708.293 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "eth0" to its initial name "veth9d88c9aa"
lxc ubuntu22 20251128121708.293 ERROR    start - ../src/lxc/start.c:__lxc_start:2119 - Failed to spawn container "ubuntu22"
lxc ubuntu22 20251128121708.293 WARN     start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 6692
lxc ubuntu22 20251128121708.293 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:832 - Received container state "ABORTING" instead of "RUNNING"
lxc 20251128121708.344 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - 连接被对方重置 - Failed to receive response
lxc 20251128121708.344 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

I haven’t used incus for a while, and when I tried to start a container today, I encountered an error. I couldn’t start any container, even though I could create a new container. After successfully creating the container, starting it immediately failed.

  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.14.0-29-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "24.04"
  project: default
  server: incus
  server_clustered: false
  server_event_mode: full-mesh
  server_name: JackMa
  server_pid: 7236
  server_version: "6.18"
  storage: btrfs
  storage_version: 6.6.3
  storage_supported_drivers:
  - name: btrfs
    version: 6.6.3
    remote: false
  - name: dir
    version: "1"
    remote: false
  - name: truenas
    version: 0.7.3
    remote: true

Can you show stat -c"%U:%G %a" / /var /var/lib /var/lib/incus /var/lib/incus/containers /var/lib/incus/containers/ubuntu22

@stgraber Could you help me confirm if this is a problem of the btrfs storage driver in the new version incus? My current testing is:

  • Incus 6.18、6.19.1 with the dir storage driver works
  • Incus 6.18、6.19.1 with the btrfs storage driver cannot start any containers.
  • Incus 6.16、6.17 with the btrfs storage driver works.

My OS environment is Ubuntu 24.04.3, with kernel version 6.14.0-29-generic.

stat -c"%U:%G %a" / /var /var/lib /var/lib/incus /var/lib/incus/containers /var/lib/incus/containers/test
root:root 755
root:root 755
root:root 755
root:root 711
root:root 711
root:root 777

On a clean Ubuntu 24.04 system:

root@v1:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.3 LTS
Release:	24.04
Codename:	noble
root@v1:~# uname -a
Linux v1 6.8.0-88-generic #89-Ubuntu SMP PREEMPT_DYNAMIC Sat Oct 11 01:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@v1:~# incus admin init
Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, dir, truenas) [default=btrfs]:    
Create a new BTRFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=5GiB]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=incusbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: 
root@v1:~# incus launch images:debian/13 d13
Launching d13
root@v1:~# incus list                            
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4          |                      IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| d13  | RUNNING | 10.247.145.176 (eth0) | fd42:509d:44b7:50a5:1266:6aff:fe53:b136 (eth0) | CONTAINER | 0         |
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
root@v1:~# 

Same system after installing the HWE kernel:

root@v1:~# uname -a
Linux v1 6.14.0-36-generic #36~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 15 15:45:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
root@v1:~# incus list
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4          |                      IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| d13  | RUNNING | 10.247.145.176 (eth0) | fd42:509d:44b7:50a5:1266:6aff:fe53:b136 (eth0) | CONTAINER | 0         |
+------+---------+-----------------------+------------------------------------------------+-----------+-----------+
root@v1:~# 

Can you show incus storage list -fyaml?

itviewer@JackMa:~$ empty-incus.sh 
==> Deleting all instances for project: default
==> Deleting all images for project: default
==> Deleting all profiles for project: default
==> Deleting all networks
Network incusbr0 deleted
==> Deleting all storage pools
Storage pool default deleted
itviewer@JackMa:~$ cat <<EOF | incus admin init --preseed
config:
  core.https_address: :8443
  images.auto_update_interval: 168
networks:
- config:
    ipv4.address: auto
    ipv4.firewall: false
    ipv6.address: none
    ipv6.firewall: false
  description: ""
  name: incusbr0
  type: bridge
  project: default
storage_pools:
- config:
    source: /mnt/test
    volatile.initial_source: /mnt/test
  description: ""
  name: default
  driver: btrfs
profiles:
- config:
    limits.cpu: '16'
    limits.memory: 50%
  description: ""
  devices:
    eth0:
      name: eth0
      network: incusbr0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null
EOF
itviewer@JackMa:~$ incus launch images:alpine/edge alpine
Launching alpine
Error: Failed instance creation: Failed to run: /opt/incus/bin/incusd forkstart alpine /var/lib/incus/containers /run/incus/alpine/lxc.conf: exit status 1
itviewer@JackMa:~$ incus storage list -fyaml
- config:
    source: /mnt/test
    volatile.initial_source: /mnt/test
  description: ""
  name: default
  driver: btrfs
  used_by:
  - /1.0/images/915d1a6323bd396bad3e10621a591c420f52e9ce5134d9c0325d4b1d106c9586
  - /1.0/instances/alpine
  - /1.0/profiles/default
  status: Created
  locations:
  - none

I repeated your command above using the loop device without any problems. I’m using a separate btrfs partition on a physical disk. Is there any difference?

itviewer@JackMa:~$ incus admin init
Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (dir, truenas, btrfs) [default=btrfs]: 
Create a new BTRFS pool? (yes/no) [default=yes]: 
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: 
Size in GiB of the new loop device (1GiB minimum) [default=30GiB]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=incusbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: 
itviewer@JackMa:~$ incus launch images:alpine/edge alpine
Launching alpine
itviewer@JackMa:~$ incus list
+--------+---------+-----------------------+------------------------------------------------+-----------+-----------+
|  NAME  |  STATE  |         IPV4          |                      IPV6                      |   TYPE    | SNAPSHOTS |
+--------+---------+-----------------------+------------------------------------------------+-----------+-----------+
| alpine | RUNNING | 10.101.101.159 (eth0) | fd42:66c7:77d0:477d:1266:6aff:fe1f:a3f5 (eth0) | CONTAINER | 0         |
+--------+---------+-----------------------+------------------------------------------------+-----------+-----------+

Can you show:

  • stat /mnt
  • stat /mnt/test
root@v1:~# incus storage create test btrfs source=/mnt/test
Storage pool test created
root@v1:~# incus launch images:debian/13 d14 --storage test
Launching d14
root@v1:~# incus list d14                        
+------+---------+---------------------+------------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4         |                      IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+---------------------+------------------------------------------------+-----------+-----------+
| d14  | RUNNING | 10.247.145.3 (eth0) | fd42:509d:44b7:50a5:1266:6aff:fe28:3143 (eth0) | CONTAINER | 0         |
+------+---------+---------------------+------------------------------------------------+-----------+-----------+

root@v1:~# stat /mnt
  File: /mnt
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 8,2	Inode: 63938       Links: 3
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-12-01 04:24:23.676978228 +0000
Modify: 2025-12-01 04:24:17.690011727 +0000
Change: 2025-12-01 04:24:17.690011727 +0000
 Birth: 2025-11-30 08:04:45.127977829 +0000

root@v1:~# stat /mnt/test
  File: /mnt/test
  Size: 214       	Blocks: 32         IO Block: 4096   directory
Device: 0,48	Inode: 256         Links: 1
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-12-01 04:24:14.000000000 +0000
Modify: 2025-12-01 04:24:39.661145972 +0000
Change: 2025-12-01 04:24:39.661145972 +0000
 Birth: 2025-12-01 04:24:14.000000000 +0000
root@v1:~# 

with incus 6.17

itviewer@JackMa:~$ incus launch images:alpine/edge alpine
Launching alpine
itviewer@JackMa:~$ incus list                      
+--------+---------+-----------------------+------+-----------+-----------+
|  NAME  |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+--------+---------+-----------------------+------+-----------+-----------+
| alpine | RUNNING | 10.207.205.118 (eth0) |      | CONTAINER | 0         |
+--------+---------+-----------------------+------+-----------+-----------+
itviewer@JackMa:~$ stat /mnt
  File: /mnt
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 259,13  Inode: 9324289     Links: 5
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-12-01 10:10:53.657814293 +0800
Modify: 2025-12-01 09:16:45.821902843 +0800
Change: 2025-12-01 09:16:45.821902843 +0800
 Birth: 2022-10-05 20:13:04.647020531 +0800
itviewer@JackMa:~$ stat /mnt/test
  File: /mnt/test
  Size: 214             Blocks: 32         IO Block: 4096   directory
Device: 0,44    Inode: 256         Links: 1
Access: (0700/drwx------)  Uid: ( 1000/itviewer)   Gid: ( 1000/itviewer)
Access: 2025-12-01 12:31:23.578495595 +0800
Modify: 2025-12-01 12:31:33.492248791 +0800
Change: 2025-12-01 12:31:33.492248791 +0800
 Birth: 2025-12-01 09:14:47.000000000 +0800

with incus 6.19.1

itviewer@JackMa:~$ incus launch images:alpine/edge alpine
Launching alpine
Error: Failed instance creation: Failed to run: /opt/incus/bin/incusd forkstart alpine /var/lib/incus/containers /run/incus/alpine/lxc.conf: exit status 1
itviewer@JackMa:~$ stat /mnt
  File: /mnt
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 259,13  Inode: 9324289     Links: 5
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-12-01 10:10:53.657814293 +0800
Modify: 2025-12-01 09:16:45.821902843 +0800
Change: 2025-12-01 09:16:45.821902843 +0800
 Birth: 2022-10-05 20:13:04.647020531 +0800
itviewer@JackMa:~$ stat /mnt/test
  File: /mnt/test
  Size: 214             Blocks: 32         IO Block: 4096   directory
Device: 0,44    Inode: 256         Links: 1
Access: (0700/drwx------)  Uid: ( 1000/itviewer)   Gid: ( 1000/itviewer)
Access: 2025-12-01 12:33:48.708463835 +0800
Modify: 2025-12-01 12:35:34.611377552 +0800
Change: 2025-12-01 12:35:34.611377552 +0800
 Birth: 2025-12-01 09:14:47.000000000 +0800

UUID=2676e0da-50ec-4d28-b59d-a75ba685a806 /mnt/test btrfs defaults 0 0

The btrfs partition is mounted using the above configuration in /etc/fstab. And the btrfs partition has been deleted and reformatted multiple times to ensure that no previous content has affected it.

Yeah, those permissions on the root of that partition are wrong.

chown 0:0 /mnt/test
chmod 0755 /mnt/test

Great! The containers can now be started with incus 6.19.1.

But how did this happen? Why doesn’t incus 6.17 require the same permission changes above?

itviewer@JackMa:~$ sudo chown 0:0 /mnt/incus/
itviewer@JackMa:~$ sudo chmod 0755 /mnt/incus/
itviewer@JackMa:~$ incus start debian13 
itviewer@JackMa:~$ incus start ubuntu22
itviewer@JackMa:~$ incus list
+----------+---------+--------------------+------+-----------+-----------+
|   NAME   |  STATE  |        IPV4        | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+--------------------+------+-----------+-----------+
| debian13 | RUNNING | 10.89.3.221 (eth0) |      | CONTAINER | 0         |
+----------+---------+--------------------+------+-----------+-----------+
| ubuntu22 | RUNNING | 10.89.3.132 (eth0) |      | CONTAINER | 0         |
+----------+---------+--------------------+------+-----------+-----------+

The error message your first posted answers that. The systemd credentials feature requires the container to be able to bind-mount a directory that’s stored within its volume. The container runs unprivileged, so it can’t cross through a 0700 folder which is what your btrfs filesystem’s root directory is set to.

The problem seems to have been found: I’ve been formatting the partition as btrfs using the system’s built-in graphical disk utility (gnome-disks). The problem disappears when I switch to mkfs.btrfs or have incus create a clean storage pool and format the partition.

If you’re interested or just for fun, you can try formatting the partition using the gnome-disks, to see if the problem can be reproduced.

Is it possible that gnome-disks uses the older btrfs tool, and while incus 6.17 is compatible with this format, but versions after 6.18 only support the latest btrfs format?

My guess is that gnome-disks sees that you’re running this from a user session and tries to be helpful in setting the owner and permissions at the root of the new filesystem so that only your user can access it.

That makes sense if you’re formatting a USB stick, doesn’t make as much sense if you’re formatting system storage.