Incus 6.5 -- Incus VM not starting in Cluster

Incus
1

@stgraber After upgrading to Incus 6.5, my incus VM’s (Only in an incus Cluster, incus standalone server VM’s are fine) I am seeing the following:

scott@vmscloud-incus:~$ incus list
+--------------+---------+------+------+-----------------+-----------+----------------+
|     NAME     |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |    LOCATION    |
+--------------+---------+------+------+-----------------+-----------+----------------+
| Desktop-2404 | STOPPED |      |      | VIRTUAL-MACHINE | 0         | vmscloud-incus |
+--------------+---------+------+------+-----------------+-----------+----------------+
scott@vmscloud-incus:~$ incus start Desktop-2404
Error: Failed setting up device via monitor: Failed adding block device for disk device "root": Failed adding block device: Failed to find file descriptor with matching flags=0x4002

Incus VM appears fine:

scott@vmscloud-incus:~$ incus info --show-log Desktop-2404
Name: Desktop-2404
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Location: vmscloud-incus
Created: 2024/05/06 20:51 UTC
Last Used: 2024/08/23 13:39 UTC

Backups:
+------------------------------------+----------------------+------------+---------------+-------------------+
|                NAME                |       TAKEN AT       | EXPIRES AT | INSTANCE ONLY | OPTIMIZED STORAGE |
+------------------------------------+----------------------+------------+---------------+-------------------+
| Desktop-2404-20240710044158.tar.gz | 2024/07/10 04:41 UTC |            | NO            | NO                |
+------------------------------------+----------------------+------------+---------------+-------------------+

Log:

I also get the error trying to create a new Incus VM in the cluster:

scott@vmscloud-incus:~$ incus launch images:ubuntu/24.04 Test --vm
Launching Test
Error: Failed instance creation: Failed setting up device via monitor: Failed adding block device for disk device "root": Failed adding block device: Failed to find file descriptor with matching flags=0x4002

All my incus 6.5 standalone nodes are not having the error. Only the cluster members are seeing the issue.

image

scott@vmscloud-incus:~$ incus storage volume list default
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+----------------+
|      TYPE       |                               NAME                               | DESCRIPTION | CONTENT-TYPE | USED BY |    LOCATION    |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+----------------+
| image           | 8e9500e6b23d3c9b9dbae32429f37317c73dbeae247aee53c6a2e55413260461 |             | filesystem   | 1       | vmsfog-incus   |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+----------------+
| image           | a3c24e392238c81c942194a92f515edfb99ebd166dfac10d3166d9f0084bc63b |             | filesystem   | 1       | incus-member   |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+----------------+
| image           | a249ceb0ce903d6e1dcb6cf109288b1066c48c36b79f0263b14f60279edde2b6 |             | block        | 1       | vmsfog-incus   |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+----------------+
| virtual-machine | Desktop-2404                                                     |             | block        | 1       | vmscloud-incus |
+-----------------+------------------------------------------------------------------+-------------+--------------+---------+----------------+

scott@vmscloud-incus:~$ incus cluster list
+----------------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|      NAME      |            URL            |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS |      MESSAGE      |
+----------------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| incus-member   | https://incus-member:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+----------------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| vmscloud-incus | https://172.16.1.219:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|                |                           | database        |              |                |             |        |                   |
+----------------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| vmsfog-incus   | https://vmsfog-incus:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+----------------+---------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
2

Hmm, that error comes from QEMU. The Incus 6.5 packages come with QEMU 9.1 so something may have happened there.

Are all your machines running the same distribution, version of distribution and kernel or is there a difference between the working and broken ones?

3

I ran into the same issue after upgrading.

My host is running Debian 12.7, Linux 6.10.7-zabbly+ and zfs-2.2.5-1.

My VM is also Debian 12.

Error: Failed setting up device via monitor: Failed adding block device for disk device "root": Failed adding block device: Failed to find file descriptor with matching flags=0x4002
4

My correction. This is not limited to cluster nodes. It appears that all my incus servers are unable to start a VM. If the VM was started before the upgrade, it is still running. If the VM is restarted, then we see the error.

Error: Failed setting up device via monitor: Failed adding block device for disk device "root": Failed adding block device: Failed to find file descriptor with matching flags=0x4002

Yes, all incus servers are running QEMU 9.1.

1 Like
5

Different storage config on the cluster nodes?

6

Please note above. It appears this is on all nodes. The storage pool is zfs.

7

Got it reproduced on standalone here.

1 Like
8

The really interesting part is that you do not see the problem unless your either create a new VM or restart an existing VM. I have several incus VM’s on systems upgraded to 6.5 that are still running only because they have not been restarted.

Gee sorry about giving you extra work. I think for about the last 40x years that I have always been the first to find the smallest problems in a piece of system software almost immediately. Hopefully it won’t be too major.

9
10

Is there a workaround for now?

11

New build is in progress, should be available within 2-3 hours.

1 Like
12

The fix should be in the stable repository now.

13

I can report that it is working here.

14

i got the same error on all of my servers now! i am using archlinux,
can you please help me how can i fix it at least shortly until main solution comes out?

i get this error on all VMS :
Error: Unable to locate matching firmware: [{Code:/usr/share/ovmf/x64/OVMF_CODE.4m.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.4m.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.4m.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.4m.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.4m.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.4m.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.fd}]

i have to mention it happened right after incus update to version 6.5

15

You have to downgrade qemu. I think you need 9.0 and not 9.1.

16

I didn’t know how to link to the Incus chat. This topic came up recently.

1000005124
10000051241152×2496 469 KB

17
k4my4b
2024-09-14 18:29:32 UTC
Incus

I've temporarily reverted back to qemu 9.0 and that's fixed it. for anyone in the same position ether:

pacman -U /var/cache/pacman/pkg/qemu-*9.0*.pkg.tar.zst

OR
Download the packages from the archives Index of /packages/q/

18

didnt help! i downgrade all of my qemu packages to version 9.0.2-1
still incus fail to start my vms with same error

19

CC: @stgraber

I am not sure what the next step is. I am sure a solution is close, plus or minus timezone differences and conference obligations.

20

sudo rm /var/lib/incus/storage-pools/<storagepool-name>virtual-machines/<VM>/qemu.nvram fixed the issue

1 Like