How to troubleshoot and resolve "Error status" after incus migrate failed?

I tried to run incus-migrate downloaded from the releases page. It seemed to work, but then starting the VM fails:

$ sudo ./bin.linux.incus-migrate.x86_64 
[sudo] password for xrd: 
The local Incus server is the target [default=yes]: yes
Would you like to create a container (1) or virtual-machine (2)?: 2
Name of the new instance: windows-dev
Please provide the path to a disk, partition, or qcow2/raw/vmdk image file: windows-dev.qcow2
Does the VM support UEFI booting? [default=yes]: 
Does the VM support UEFI Secure Boot? [default=yes]: 

Instance to be created:
  Name: windows-dev
  Project: default
  Type: virtual-machine
  Source: windows-dev.qcow2
  Source format: qcow2

Additional overrides can be applied at this stage:
1) Begin the migration with the above configuration
2) Override profile list
3) Set additional configuration options
4) Change instance storage pool or volume size
5) Change instance network

Please pick one of the options above [default=1]: 
Converting image "windows-dev.qcow2" to raw format before importing


Instance windows-dev successfully created 

$ incus start windows-dev
Error: mkdir /var/lib/incus/virtual-machines/windows-dev/config: no space left on device
Try `incus info --show-log windows-dev` for more info

$ incus info --show-log windows-dev
Name: windows-dev
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2025/02/05 14:17 EST
Last Used: 1969/12/31 19:00 EST
Error: open /var/log/incus/windows-dev/qemu.log: no such file or directory

Then, all my running VMs are down and now in an error stated.

For example, now when I run this command to start my server I see this:

$ incus start community-public-do-vm-v2
Error: The instance cannot be started as in Error status
$ incus start windows-dev
Error: mkdir /var/lib/incus/virtual-machines/windows-dev/config: no space left on device

If I use info --show-log I see this:

$ incus info --show-log community-public-do-vm-v2
Name: community-public-do-vm-v2
Status: ERROR
Type: virtual-machine
Architecture: x86_64
PID: 5572
Created: 2024/12/01 16:26 EST
Last Used: 2025/01/01 20:46 EST

Resources:
  Processes: 0
  Disk usage:
    root: 4.69GiB
Error: open /var/log/incus/community-public-do-vm-v2/qemu.log: no such file or directory

Indeed, there is no file in that directory.

incus storage list shows a degraded pool but my machine is not on that one:

$ incus storage list
+---------------+--------+--------------------------------------------+-------------+---------+-------------+
|     NAME      | DRIVER |                   SOURCE                   | DESCRIPTION | USED BY |    STATE    |
+---------------+--------+--------------------------------------------+-------------+---------+-------------+
| default2      | zfs    | /var/snap/lxd/common/lxd/disks/default.img |             | 9       | UNAVAILABLE |
+---------------+--------+--------------------------------------------+-------------+---------+-------------+
| incus-default | zfs    | /var/lib/incus/disks/incus-default.img     |             | 7       | CREATED     |
+---------------+--------+--------------------------------------------+-------------+---------+-------------+

zpool list shows this:

$ zpool list
NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
incus-default  29.5G  28.6G   937M        -         -    91%    96%  1.00x    ONLINE  -

I definitely have more disk space, but unsure how to read that error.

$ df -h 
Filesystem                                                Size  Used Avail Use% Mounted on
udev                                                       63G     0   63G   0% /dev
tmpfs                                                      13G  8.7M   13G   1% /run
efivarfs                                                  128K   13K  111K  10% /sys/firmware/efi/efivars
/dev/nvme0n1p2                                            1.8T  1.1T  632G  64% /
tmpfs                                                      63G  168K   63G   1% /dev/shm
tmpfs                                                     5.0M   12K  5.0M   1% /run/lock
tmpfs                                                      63G     0   63G   0% /run/qemu
/dev/nvme0n1p1                                            1.1G  6.2M  1.1G   1% /boot/efi
tmpfs                                                     100K     0  100K   0% /var/lib/incus/shmounts
tmpfs                                                     100K     0  100K   0% /var/lib/incus/guestapi
incus-default/virtual-machines/community-public-do-vm-v2  7.3M  7.3M     0 100% /var/lib/incus/storage-pools/incus-default/virtual-machines/community-public-do-vm-v2
incus-default/virtual-machines/discourse-plugin-dev       7.3M  7.3M     0 100% /var/lib/incus/storage-pools/incus-default/virtual-machines/discourse-plugin-dev
incus-default/virtual-machines/extrastatic-build-01       7.3M  7.3M     0 100% /var/lib/incus/storage-pools/incus-default/virtual-machines/extrastatic-build-01
incus-default/virtual-machines/katarismo-community        7.3M  7.3M     0 100% /var/lib/incus/storage-pools/incus-default/virtual-machines/katarismo-community

I am unsure how to resolve the storage issue.

$ incus stop --force community-public-do-vm-v2
$ incus start community-public-do-vm-v2
Error: symlink incus-agent /var/lib/incus/virtual-machines/community-public-do-vm-v2/config/lxd-agent: no space left on device
Try `incus info --show-log community-public-do-vm-v2` for more info

The filesystem definitely has more space:

$ cd /var/lib/incus/virtual-machines/windows-dev
$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  1.8T  1.1T  632G  64% /

Not sure how to read zfs list:

$ zfs  list
NAME                                                                                                  USED  AVAIL  REFER  MOUNTPOINT
incus-default                                                                                        28.6G     0B    24K  legacy
incus-default/buckets                                                                                  24K     0B    24K  legacy
incus-default/community-public-do-vm                                                                 7.18M     0B  7.18M  legacy
incus-default/containers                                                                               24K     0B    24K  legacy
incus-default/custom                                                                                   48K     0B    24K  legacy
incus-default/custom/default_community-public-do-vm                                                    24K     0B    24K  legacy
incus-default/deleted                                                                                1.16G     0B    24K  legacy
incus-default/deleted/buckets                                                                          24K     0B    24K  legacy
incus-default/deleted/containers                                                                       24K     0B    24K  legacy
incus-default/deleted/custom                                                                           24K     0B    24K  legacy
incus-default/deleted/images                                                                         1.16G     0B    24K  legacy
incus-default/deleted/images/3a48ee3f93ba8f240763b9d2dbfe9703a69455c4aa8dd30df2e66bb84a2bed59          27K     0B    26K  legacy
incus-default/deleted/images/3a48ee3f93ba8f240763b9d2dbfe9703a69455c4aa8dd30df2e66bb84a2bed59.block   398M     0B   398M  -
incus-default/deleted/images/6478a871004d0abdfc4a20f6485f6752725dc7a25c8f771928674502df338dee          27K     0B    26K  legacy
incus-default/deleted/images/6478a871004d0abdfc4a20f6485f6752725dc7a25c8f771928674502df338dee.block   397M     0B   397M  -
incus-default/deleted/images/bed54994642571edc2eecca20db23a61e1fc66d5ce520bb76dd6712239916061          27K     0B    26K  legacy
incus-default/deleted/images/bed54994642571edc2eecca20db23a61e1fc66d5ce520bb76dd6712239916061.block   397M     0B   397M  -
incus-default/deleted/virtual-machines                                                                 24K     0B    24K  legacy
incus-default/images                                                                                   24K     0B    24K  legacy
incus-default/virtual-machines                                                                       27.4G     0B    24K  legacy
incus-default/virtual-machines/community-public-do-vm-v2                                             7.17M     0B  7.18M  legacy
incus-default/virtual-machines/community-public-do-vm-v2.block                                       4.69G     0B  5.03G  -
incus-default/virtual-machines/discourse-plugin-dev                                                  7.17M     0B  7.18M  legacy
incus-default/virtual-machines/discourse-plugin-dev.block                                            3.04G     0B  3.39G  -
incus-default/virtual-machines/extrastatic-build-01                                                  7.17M     0B  7.18M  legacy
incus-default/virtual-machines/extrastatic-build-01.block                                            3.26G     0B  3.63G  -
incus-default/virtual-machines/katarismo-community                                                   7.17M     0B  7.18M  legacy
incus-default/virtual-machines/katarismo-community.block                                             3.61G     0B  3.98G  -
incus-default/virtual-machines/move-of-17335539195442412177                                          7.20M     0B  7.19M  legacy
incus-default/virtual-machines/windows-dev                                                             24K     0B    24K  legacy
incus-default/virtual-machines/windows-dev.block                                                     12.7G     0B  12.7G  -

Can you show:

  • incus config show --expanded windows-dev
  • zfs list -t all
$ incus config show --expanded windows-dev
architecture: x86_64
config:
  volatile.apply_template: copy
  volatile.cloud-init.instance-id: 20efc33f-c566-4cfe-ac62-5c4fc866d678
  volatile.eth0.hwaddr: 00:16:3e:af:80:ce
  volatile.uuid: 46587b3b-7510-4fb3-a3f7-ee870d23d5d2
  volatile.uuid.generation: 46587b3b-7510-4fb3-a3f7-ee870d23d5d2
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: incus-default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
$ zfs list -t all
NAME                                                                                                           USED  AVAIL  REFER  MOUNTPOINT
incus-default                                                                                                 28.6G     0B    24K  legacy
incus-default/buckets                                                                                           24K     0B    24K  legacy
incus-default/community-public-do-vm                                                                          7.18M     0B  7.18M  legacy
incus-default/community-public-do-vm@migrate                                                                     0B      -  7.18M  -
incus-default/containers                                                                                        24K     0B    24K  legacy
incus-default/custom                                                                                            48K     0B    24K  legacy
incus-default/custom/default_community-public-do-vm                                                             24K     0B    24K  legacy
incus-default/deleted                                                                                         1.16G     0B    24K  legacy
incus-default/deleted/buckets                                                                                   24K     0B    24K  legacy
incus-default/deleted/containers                                                                                24K     0B    24K  legacy
incus-default/deleted/custom                                                                                    24K     0B    24K  legacy
incus-default/deleted/images                                                                                  1.16G     0B    24K  legacy
incus-default/deleted/images/3a48ee3f93ba8f240763b9d2dbfe9703a69455c4aa8dd30df2e66bb84a2bed59                   27K     0B    26K  legacy
incus-default/deleted/images/3a48ee3f93ba8f240763b9d2dbfe9703a69455c4aa8dd30df2e66bb84a2bed59@readonly           1K      -    26K  -
incus-default/deleted/images/3a48ee3f93ba8f240763b9d2dbfe9703a69455c4aa8dd30df2e66bb84a2bed59.block            398M     0B   398M  -
incus-default/deleted/images/3a48ee3f93ba8f240763b9d2dbfe9703a69455c4aa8dd30df2e66bb84a2bed59.block@readonly     0B      -   398M  -
incus-default/deleted/images/6478a871004d0abdfc4a20f6485f6752725dc7a25c8f771928674502df338dee                   27K     0B    26K  legacy
incus-default/deleted/images/6478a871004d0abdfc4a20f6485f6752725dc7a25c8f771928674502df338dee@readonly           1K      -    26K  -
incus-default/deleted/images/6478a871004d0abdfc4a20f6485f6752725dc7a25c8f771928674502df338dee.block            397M     0B   397M  -
incus-default/deleted/images/6478a871004d0abdfc4a20f6485f6752725dc7a25c8f771928674502df338dee.block@readonly     0B      -   397M  -
incus-default/deleted/images/bed54994642571edc2eecca20db23a61e1fc66d5ce520bb76dd6712239916061                   27K     0B    26K  legacy
incus-default/deleted/images/bed54994642571edc2eecca20db23a61e1fc66d5ce520bb76dd6712239916061@readonly           1K      -    26K  -
incus-default/deleted/images/bed54994642571edc2eecca20db23a61e1fc66d5ce520bb76dd6712239916061.block            397M     0B   397M  -
incus-default/deleted/images/bed54994642571edc2eecca20db23a61e1fc66d5ce520bb76dd6712239916061.block@readonly     0B      -   397M  -
incus-default/deleted/virtual-machines                                                                          24K     0B    24K  legacy
incus-default/images                                                                                            24K     0B    24K  legacy
incus-default/virtual-machines                                                                                27.4G     0B    24K  legacy
incus-default/virtual-machines/community-public-do-vm-v2                                                      7.17M     0B  7.18M  legacy
incus-default/virtual-machines/community-public-do-vm-v2.block                                                4.69G     0B  5.03G  -
incus-default/virtual-machines/discourse-plugin-dev                                                           7.17M     0B  7.18M  legacy
incus-default/virtual-machines/discourse-plugin-dev.block                                                     3.04G     0B  3.39G  -
incus-default/virtual-machines/extrastatic-build-01                                                           7.17M     0B  7.18M  legacy
incus-default/virtual-machines/extrastatic-build-01.block                                                     3.26G     0B  3.63G  -
incus-default/virtual-machines/katarismo-community                                                            7.17M     0B  7.18M  legacy
incus-default/virtual-machines/katarismo-community.block                                                      3.61G     0B  3.98G  -
incus-default/virtual-machines/move-of-17335539195442412177                                                   7.20M     0B  7.19M  legacy
incus-default/virtual-machines/move-of-17335539195442412177@snapshot-12_4_2024                                16.5K      -  7.19M  -
incus-default/virtual-machines/windows-dev                                                                      24K     0B    24K  legacy
incus-default/virtual-machines/windows-dev.block                                                              12.7G     0B  12.7G  -
incus-large-zfs                                                                                               50.8G   129G    24K  legacy
incus-large-zfs/buckets                                                                                         24K   129G    24K  legacy
incus-large-zfs/containers                                                                                      24K   129G    24K  legacy
incus-large-zfs/custom                                                                                          24K   129G    24K  legacy
incus-large-zfs/deleted                                                                                        144K   129G    24K  legacy
incus-large-zfs/deleted/buckets                                                                                 24K   129G    24K  legacy
incus-large-zfs/deleted/containers                                                                              24K   129G    24K  legacy
incus-large-zfs/deleted/custom                                                                                  24K   129G    24K  legacy
incus-large-zfs/deleted/images                                                                                  24K   129G    24K  legacy
incus-large-zfs/deleted/virtual-machines                                                                        24K   129G    24K  legacy
incus-large-zfs/images                                                                                         384M   129G    24K  legacy
incus-large-zfs/images/225c9c6091d8aa5bf93d7ebd02f2e81cfa99f5ff780f9fe8775f655a2f44a3e1                         26K   500M    26K  legacy
incus-large-zfs/images/225c9c6091d8aa5bf93d7ebd02f2e81cfa99f5ff780f9fe8775f655a2f44a3e1@readonly                 0B      -    26K  -
incus-large-zfs/images/225c9c6091d8aa5bf93d7ebd02f2e81cfa99f5ff780f9fe8775f655a2f44a3e1.block                  384M   129G   384M  -
incus-large-zfs/images/225c9c6091d8aa5bf93d7ebd02f2e81cfa99f5ff780f9fe8775f655a2f44a3e1.block@readonly           0B      -   384M  -
incus-large-zfs/virtual-machines                                                                              50.5G   129G    24K  legacy
incus-large-zfs/virtual-machines/vivoh-runner-02                                                              7.18M   493M  7.19M  legacy
incus-large-zfs/virtual-machines/vivoh-runner-02.block                                                        4.37G   129G  4.73G  -
incus-large-zfs/virtual-machines/windows-install                                                              7.17M   493M  7.17M  legacy
incus-large-zfs/virtual-machines/windows-install.block                                                        46.1G   129G  46.1G  -

Your incus-default pool shows 0B of available space, so that’d explain the failures.
It’s a loop backed pool, so you should be able to grow it with incus storage set incus-default size=50GiB

1 Like

That’s the solution. Thank you.

It feels a bit counterintuitive to see that the set command grows the storage pool. Is my feeling off?

It does for loop backed storage pools like this one.
This obviously doesn’t work for pools that aren’t backed by an Incus-controlled file.