Cannot create containers in a pool with the linstor storage backend

I have a storage pool with linstor as the backend,


and the backend of linstor is zfs

If I try to create a container using the command like below,

incus init images:alpine/3.21 test-images-alpine --no-profiles --ephemeral -s default

I get the following error at first.

Creating test-images-alpine
Error: Failed instance creation: Failed creating instance from image: Clone operation failed

When I try again, the error changes to this.

Error: Failed instance creation: Failed creating instance from image: Multiple resource definitions found for volume b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e

If I use linstor resource-definition list --show-props Aux/Incus/name Aux/Incus/type Aux/Incus/content-type to list the resources on linstor, it seems that there are two b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e, with one marked as delete and another is marked as ok.

โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š ResourceName                                  โ”Š Port โ”Š ResourceGroup โ”Š Layers       โ”Š State  โ”Š Aux/Incus/name                                                                โ”Š Aux/Incus/type   โ”Š Aux/Incus/content-type โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š incus-volume-1dee77099f3c4f98820a9c486e75237b โ”Š 7014 โ”Š default       โ”Š DRBD,STORAGE โ”Š DELETE โ”Š incus-volume-b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e โ”Š images           โ”Š filesystem             โ”Š
โ”Š incus-volume-1fcf22e2018647e5aa29d6bf38baf7c6 โ”Š 7007 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-hxhome-dev                                                       โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-5b677bbe593b45058596a16d779b7229 โ”Š 7010 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-minecraft                                                        โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-5d7457b8b02a4b859fe0d1b3c0f7cb6c โ”Š 7004 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-app-mysql                                                        โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-6cd8d18d6a1a4d98b5baa1c839ba6801 โ”Š 7006 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-app-registry                                                     โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-58c36de3bdcb49d6a3d65b6f0fed09a6 โ”Š 7012 โ”Š default       โ”Š DRBD,STORAGE โ”Š DELETE โ”Š incus-volume-018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac โ”Š images           โ”Š filesystem             โ”Š
โ”Š incus-volume-86dc0b858f014949bd3b28fed0a8d70f โ”Š 7008 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-compile-wsl                                                      โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-487cfa9cf19846f8bf2ae801317bf4ed โ”Š 7003 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-app-manager                                                      โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-2937114d649f4f3da50cc6f05970c6f8 โ”Š 7000 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c โ”Š images           โ”Š filesystem             โ”Š
โ”Š incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0 โ”Š 7013 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e โ”Š images           โ”Š filesystem             โ”Š
โ”Š incus-volume-db7bdd249ed4447b98eb9e7c2d01d0dc โ”Š 7009 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-compile-openwrt                                                  โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-e788aa8b851640548f33f5ddcfb9d000 โ”Š 7001 โ”Š default       โ”Š DRBD,STORAGE โ”Š DELETE โ”Š incus-volume-a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c โ”Š images           โ”Š filesystem             โ”Š
โ”Š incus-volume-eedb60a2f5c0498cb46adbd99954a9cf โ”Š 7005 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-app-alist                                                        โ”Š containers       โ”Š filesystem             โ”Š
โ”Š incus-volume-f5de31d7f8934c749bd69d8b1c87967b โ”Š 7011 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac โ”Š images           โ”Š filesystem             โ”Š
โ”Š incus-volume-fe587ef9113c4912afa81f6f9ab663b4 โ”Š 7002 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok     โ”Š incus-volume-hxhome-windows                                                   โ”Š virtual-machines โ”Š block                  โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

I can see the image using incus image list

+-------+--------------+--------+----------------------------------------+--------------+-----------+---------+----------------------+
| ALIAS | FINGERPRINT  | PUBLIC |              DESCRIPTION               | ARCHITECTURE |   TYPE    |  SIZE   |     UPLOAD DATE      |
+-------+--------------+--------+----------------------------------------+--------------+-----------+---------+----------------------+
|       | a8560b36e8b8 | no     | docker.io/library/alpine (OCI)         | x86_64       | CONTAINER | 3.47MiB | 2025/05/04 01:41 CST |
+-------+--------------+--------+----------------------------------------+--------------+-----------+---------+----------------------+
|       | b670fe21654f | no     | Alpine 3.21 amd64 (20250503_13:00)     | x86_64       | CONTAINER | 3.09MiB | 2025/05/04 01:45 CST |
+-------+--------------+--------+----------------------------------------+--------------+-----------+---------+----------------------+

And if I try to delete the image usingincus image delete b670fe21654f, I can get similar error again.

Error: Error deleting image "b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e" from storage pool "default": Multiple resource definitions found for volume b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e

Besides, I have several containers and vms which are migrated from other backends to linstor and they are working fine now.

The version of incus is 6.12, for zfs it is zfs-2.3.1-1,zfs-kmod-2.3.1-1 and for linstor it is 1.31.0 (a187af5c85a96bb27df87a5eab0bcf9dd6de6a34).

So, whatโ€™s going on and what should I do next in order to create the container?

Could you run linstor rd d incus-volume-1dee77099f3c4f98820a9c486e75237b and linstor rd d incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0 and try again if there is no error? Or report the error that LINSTOR gives you.

linstor rd d incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0 runs successfully with the output:

SUCCESS:
    Resource definition 'incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0' marked for deletion.
SUCCESS:
    (nas.hxhome.local) Resource 'incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0' [DRBD] deleted.
SUCCESS:
    (nas.hxhome.local) Volume number 0 of resource 'incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0' [ZFS-Thin] deleted
SUCCESS:
    Resource 'incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0' on 'nas.hxhome.local' marked for deletion
SUCCESS:
Description:
    Resource definition 'incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0' deleted.
Details:
    Resource definition 'incus-volume-cbaef3aef11b44f5bcbc22a4b486a5f0' UUID was: d56f4352-c622-4437-9ead-6bd070bb888e

But linstor rd d incus-volume-1dee77099f3c4f98820a9c486e75237b fails

SUCCESS:
    Resource definition 'incus-volume-1dee77099f3c4f98820a9c486e75237b' marked for deletion.
ERROR:
    (nas.hxhome.local) ZfsThinProvider did not delete the volume ZfsData [Node: 'nas.hxhome.local', Rsc: 'incus-volume-1dee77099f3c4f98820a9c486e75237b', VlmNr: '0', RscNameSuffix: ]
Show reports:
    linstor error-reports show 681646F6-F3666-000011

and linstor error-reports show 681646F6-F3666-000011 shows:

ERROR REPORT 681646F6-F3666-000014

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 10:21:45
Node:                               nas.hxhome.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'processResource', Source file 'StorageLayer.java', Line #296

Error message:                      ZfsThinProvider did not delete the volume ZfsData [Node: 'nas.hxhome.local', Rsc: 'incus-volume-1dee77099f3c4f98820a9c486e75237b', VlmNr: '0', RscNameSuffix: ]

Error context:
        An error occurred while processing resource 'Node: 'nas.hxhome.local', Rsc: 'incus-volume-1dee77099f3c4f98820a9c486e75237b''
Call backtrace:

    Method                                   Native Class:Line number
    processResource                          N      com.linbit.linstor.layer.storage.StorageLayer:296
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1369
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1412
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1442
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1365
    processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:386
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:228
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

I deleted one image successfully and run incus init again, still got an error Error: Failed instance creation: Failed creating instance from image: Clone operation failed

Hmmm something wrong happened thereโ€ฆ did you by any chance try to mount any resource without using LINSTOR or Incus commands?
Iโ€™d try deleting the volume directly on ZFS side, then trying to delete it from Incus, and if deletion from Incus doesnโ€™t work, deleting it on LINSTOR, then on Incus.
I have no idea why that happens, weโ€™ve not encountered the problem even once when developing the LINSTOR driverโ€ฆ

yeah, I manualy deleted images on zfs, incus and linstor, after that I still have Clone operation failed when creating containers. I guess there is something wrong in the creation process, the volume is created but not used by the incus or sth like that.

Ok, Iโ€™m out of ideasโ€ฆ could you give Incus logs with debug verbosity of the failing operation? There may be some useful LINSTOR debug info there.

I tried several times, and I found that sometime the incus init would stuck after unpacking image.

DESCRIPTION
Creating app-pxehelper
Retrieving image: Unpacking image: 100% (4.41GB/s)

And the linstor will show a cloning state

linstor resource-definition list --show-props Aux/Incus/name Aux/Incus/type Aux/Incus/content-type
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š ResourceName                                  โ”Š Port โ”Š ResourceGroup โ”Š Layers       โ”Š State   โ”Š Aux/Incus/name                                                                โ”Š Aux/Incus/type โ”Š Aux/Incus/content-type โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š incus-volume-0ac4a3c3ee644b63908b1e46c8702451 โ”Š 7001 โ”Š default       โ”Š DRBD,STORAGE โ”Š CLONING โ”Š incus-volume-018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac โ”Š images         โ”Š filesystem             โ”Š
โ”Š incus-volume-eeb73fd5e1484723b45ec471e7f1d524 โ”Š 7000 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok      โ”Š incus-volume-018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac โ”Š images         โ”Š filesystem             โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

and the zfs will shows both the volumes exists but they have different size

hx-pool/incus                                                                                       22.3M  1.83T   105K  legacy
hx-pool/incus/_deleted_incus-volume-ecc6597dad2940cba70e9d6cb27fcaa5_00000_2025-05-04T15-41-53-535  11.1M  1.83T  11.1M  -
hx-pool/incus/incus-volume-0ac4a3c3ee644b63908b1e46c8702451_00000                                   54.6K  1.83T  54.6K  -
hx-pool/incus/incus-volume-eeb73fd5e1484723b45ec471e7f1d524_00000                                   11.1M  1.83T  11.1M  -

Could it be something to do with place_count? I have set incus storage set default linstor.resource_group.place_count=1 as I only have one device currently.

I found something seems useful in the linstor err show. It says drbdmeta check-resize failed after a processAfterClone, seems to be related to the Clone operation failed.

ERROR REPORT 68170F0E-6584A-000001

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:40:07
Node:                               nas.hxdw.local
Thread:                             clone_ZFS_COPY(incus-volume-ecc6597dad2940cba70e9d6cb27fcaa5/0->incus-volume-bab1791fa687479ab1c882b06b76e67c/0)

============================================================

Reported error:
===============

Category:                           LinStorException
Class name:                         StorageException
Class canonical name:               com.linbit.linstor.storage.StorageException
Generated at:                       Method 'processAfterClone', Source file 'DrbdLayer.java', Line #2041

Error message:                      drbdmeta check-resize failed

ErrorContext:


Call backtrace:

    Method                                   Native Class:Line number
    processAfterClone                        N      com.linbit.linstor.layer.drbd.DrbdLayer:2041
    processAfterClone                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1770
    cleanupDevices                           N      com.linbit.linstor.clone.CloneService:204
    postClone                                N      com.linbit.linstor.clone.CloneService:544
    lambda$startClone$2                      N      com.linbit.linstor.clone.CloneService:395
    run                                      N      com.linbit.linstor.clone.CloneDaemon:171
    run                                      N      java.lang.Thread:1583

Caused by:
==========

Category:                           LinStorException
Class name:                         ExtCmdFailedException
Class canonical name:               com.linbit.extproc.ExtCmdFailedException
Generated at:                       Method 'drbdMetaCheckResize', Source file 'DrbdAdm.java', Line #410

Error message:                      The external command 'drbdmeta' exited with error code 1


ErrorContext:
  Description: Execution of the external command 'drbdmeta' failed.
  Cause:       The external command exited with error code 1.
  Correction:  - Check whether the external program is operating properly.
- Check whether the command line is correct.
  Contact a system administrator or a developer if the command line is no longer valid
  for the installed version of the external program.
  Details:     The full command line executed was:
drbdmeta 1001 v09 /dev/zvol/hx-pool/incus/incus-volume-bab1791fa687479ab1c882b06b76e67c_00000 internal check-resize

The external command sent the following output data:


The external command sent the following error information:
no suitable meta data found :(




Call backtrace:

    Method                                   Native Class:Line number
    drbdMetaCheckResize                      N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:410
    processAfterClone                        N      com.linbit.linstor.layer.drbd.DrbdLayer:2026
    processAfterClone                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1770
    cleanupDevices                           N      com.linbit.linstor.clone.CloneService:204
    postClone                                N      com.linbit.linstor.clone.CloneService:544
    lambda$startClone$2                      N      com.linbit.linstor.clone.CloneService:395
    run                                      N      com.linbit.linstor.clone.CloneDaemon:171
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

I found someone using linbit says his volume can not resize on a zfs backend.

Getting zfs exception when resizing due to incompatible size to to volblocksize

Yeah thatโ€™s not enough for me, Iโ€™d need full debug logsโ€ฆ

If you only have one node, then a place count of 1 is perfectly fine.

Yeah thatโ€™s bad.

Ok, probably related, although Iโ€™m running LINSTOR with zfsthin volumes day to day without any problemโ€ฆ

Iโ€™ll open an issue on the Incus side so that we try to cleanup the bad resource if it exists.

The full debug log is quite longโ€ฆ

incus launch images:alpine/3.21 c2 --storage default --debug
DEBUG  [2025-05-04T19:22:08+08:00] Connecting to a local Incus over a Unix socket
DEBUG  [2025-05-04T19:22:08+08:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2025-05-04T19:22:08+08:00] Got response struct from Incus
DEBUG  [2025-05-04T19:22:08+08:00]
        {
                "config": {
                        "core.https_address": "[::]:8443",
                        "storage.linstor.controller_connection": "http://172.17.1.1:3370"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "network_sriov",
                        "console",
                        "restrict_dev_incus",
                        "migration_pre_copy",
                        "infiniband",
                        "dev_incus_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "dev_incus_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "backup_compression",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "images_all_projects",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry",
                        "init_preseed",
                        "storage_volumes_created_at",
                        "cpu_hotplug",
                        "projects_networks_zones",
                        "network_txqueuelen",
                        "cluster_member_state",
                        "instances_placement_scriptlet",
                        "storage_pool_source_wipe",
                        "zfs_block_mode",
                        "instance_generation_id",
                        "disk_io_cache",
                        "amd_sev",
                        "storage_pool_loop_resize",
                        "migration_vm_live",
                        "ovn_nic_nesting",
                        "oidc",
                        "network_ovn_l3only",
                        "ovn_nic_acceleration_vdpa",
                        "cluster_healing",
                        "instances_state_total",
                        "auth_user",
                        "security_csm",
                        "instances_rebuild",
                        "numa_cpu_placement",
                        "custom_volume_iso",
                        "network_allocations",
                        "zfs_delegate",
                        "storage_api_remote_volume_snapshot_copy",
                        "operations_get_query_all_projects",
                        "metadata_configuration",
                        "syslog_socket",
                        "event_lifecycle_name_and_project",
                        "instances_nic_limits_priority",
                        "disk_initial_volume_configuration",
                        "operation_wait",
                        "image_restriction_privileged",
                        "cluster_internal_custom_volume_copy",
                        "disk_io_bus",
                        "storage_cephfs_create_missing",
                        "instance_move_config",
                        "ovn_ssl_config",
                        "certificate_description",
                        "disk_io_bus_virtio_blk",
                        "loki_config_instance",
                        "instance_create_start",
                        "clustering_evacuation_stop_options",
                        "boot_host_shutdown_action",
                        "agent_config_drive",
                        "network_state_ovn_lr",
                        "image_template_permissions",
                        "storage_bucket_backup",
                        "storage_lvm_cluster",
                        "shared_custom_block_volumes",
                        "auth_tls_jwt",
                        "oidc_claim",
                        "device_usb_serial",
                        "numa_cpu_balanced",
                        "image_restriction_nesting",
                        "network_integrations",
                        "instance_memory_swap_bytes",
                        "network_bridge_external_create",
                        "network_zones_all_projects",
                        "storage_zfs_vdev",
                        "container_migration_stateful",
                        "profiles_all_projects",
                        "instances_scriptlet_get_instances",
                        "instances_scriptlet_get_cluster_members",
                        "instances_scriptlet_get_project",
                        "network_acl_stateless",
                        "instance_state_started_at",
                        "networks_all_projects",
                        "network_acls_all_projects",
                        "storage_buckets_all_projects",
                        "resources_load",
                        "instance_access",
                        "project_access",
                        "projects_force_delete",
                        "resources_cpu_flags",
                        "disk_io_bus_cache_filesystem",
                        "instance_oci",
                        "clustering_groups_config",
                        "instances_lxcfs_per_instance",
                        "clustering_groups_vm_cpu_definition",
                        "disk_volume_subpath",
                        "projects_limits_disk_pool",
                        "network_ovn_isolated",
                        "qemu_raw_qmp",
                        "network_load_balancer_health_check",
                        "oidc_scopes",
                        "network_integrations_peer_name",
                        "qemu_scriptlet",
                        "instance_auto_restart",
                        "storage_lvm_metadatasize",
                        "ovn_nic_promiscuous",
                        "ovn_nic_ip_address_none",
                        "instances_state_os_info",
                        "network_load_balancer_state",
                        "instance_nic_macvlan_mode",
                        "storage_lvm_cluster_create",
                        "network_ovn_external_interfaces",
                        "instances_scriptlet_get_instances_count",
                        "cluster_rebalance",
                        "custom_volume_refresh_exclude_older_snapshots",
                        "storage_initial_owner",
                        "storage_live_migration",
                        "instance_console_screenshot",
                        "image_import_alias",
                        "authorization_scriptlet",
                        "console_force",
                        "network_ovn_state_addresses",
                        "network_bridge_acl_devices",
                        "instance_debug_memory",
                        "init_preseed_storage_volumes",
                        "init_preseed_profile_project",
                        "instance_nic_routed_host_address",
                        "instance_smbios11",
                        "api_filtering_extended",
                        "acme_dns01",
                        "security_iommu",
                        "network_ipv4_dhcp_routes",
                        "network_state_ovn_ls",
                        "network_dns_nameservers",
                        "acme_http01_port",
                        "network_ovn_ipv4_dhcp_expiry",
                        "instance_state_cpu_time",
                        "network_io_bus",
                        "disk_io_bus_usb",
                        "storage_driver_linstor",
                        "instance_oci_entrypoint",
                        "network_address_set",
                        "server_logging",
                        "network_forward_snat",
                        "memory_hotplug"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "auth_user_name": "root",
                "auth_user_method": "unix",
                "environment": {
                        "addresses": [
                                "172.17.1.1:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIICFzCCAZygAwIBAgIRAIRlXkOc1nSst1x+nTSIytcwCgYIKoZIzj0EAwMwOTEZ\nMBcGA1UEChMQTGludXggQ29udGFpbmVyczEcMBoGA1UEAwwTcm9vdEBuYXMuaHhk\ndy5sb2NhbDAeFw0yNTA1MDMxMjAwMDJaFw0zNTA1MDExMjAwMDJaMDkxGTAXBgNV\nBAoTEExpbnV4IENvbnRhaW5lcnMxHDAaBgNVBAMME3Jvb3RAbmFzLmh4ZHcubG9j\nYWwwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATCpNPMZOyBhaXPk88YG7xVyrDYRq5h\n7Rm8bm/p0p0x1j5bNE4XAXsRdrS3tQ3zUc3CT5iP5o8JvHV5By9fgAwBNJOtsOkK\nPxNDi9MAQ+2eCJzCj1kx+DnWghZ6iDAIz3OjaDBmMA4GA1UdDwEB/wQEAwIFoDAT\nBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMDEGA1UdEQQqMCiCDm5h\ncy5oeGR3LmxvY2FshwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMD\nA2kAMGYCMQCt21plG2lWucpciJwLmeC9YA0VT1SvxB9ijG9doiTWyaWT6UDdZYWZ\nxZRU6m4lpGMCMQC/K63N6oVa5ANjiwfCoMjHW9CIcshrBBvy/IiTUnyw77e1QIf1\nCqOCPSxt+kHqK+k=\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "65e4b3e0acef697b795efc94816b8a4773e6d63d02adbac2d3ed788f3b3ef21f",
                        "driver": "lxc | qemu",
                        "driver_version": "6.0.4 | 9.0.4",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "uevent_injection": "true",
                                "unpriv_binfmt": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "6.8.0-58-generic",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Ubuntu",
                        "os_version": "24.04",
                        "project": "default",
                        "server": "incus",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "nas.hxdw.local",
                        "server_pid": 43425,
                        "server_version": "6.12",
                        "storage": "linstor",
                        "storage_version": "1.31.0 / 9.2.13",
                        "storage_supported_drivers": [
                                {
                                        "Name": "btrfs",
                                        "Version": "6.6.3",
                                        "Remote": false
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "linstor",
                                        "Version": "1.31.0 / 9.2.13",
                                        "Remote": true
                                },
                                {
                                        "Name": "lvmcluster",
                                        "Version": "2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0",
                                        "Remote": true
                                },
                                {
                                        "Name": "zfs",
                                        "Version": "2.3.1-1",
                                        "Remote": false
                                }
                        ]
                }
        }
Launching c2
DEBUG  [2025-05-04T19:22:08+08:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0/storage-pools/default"
DEBUG  [2025-05-04T19:22:08+08:00] Got response struct from Incus
DEBUG  [2025-05-04T19:22:08+08:00]
        {
                "config": {
                        "drbd.auto_add_quorum_tiebreaker": "true",
                        "drbd.on_no_quorum": "suspend-io",
                        "linstor.resource_group.name": "default",
                        "linstor.resource_group.place_count": "1",
                        "linstor.resource_group.storage_pool": "incus",
                        "linstor.volume.prefix": "incus-volume-",
                        "source": "default",
                        "volatile.pool.pristine": "true"
                },
                "description": "",
                "name": "default",
                "driver": "linstor",
                "used_by": [
                        "/1.0/images/018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac",
                        "/1.0/images/8178b6bfe5767a363cb2a27f0befcd72b766f09eae7f7fe0884919686261cb63",
                        "/1.0/instances/app-pxehelper",
                        "/1.0/instances/c1"
                ],
                "status": "Created",
                "locations": [
                        "none"
                ]
        }
DEBUG  [2025-05-04T19:22:08+08:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2025-05-04T19:22:08+08:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG  [2025-05-04T19:22:08+08:00] Sending request to Incus                      etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2025-05-04T19:22:08+08:00]
        {
                "architecture": "",
                "config": {},
                "devices": {
                        "root": {
                                "path": "/",
                                "pool": "default",
                                "type": "disk"
                        }
                },
                "ephemeral": false,
                "profiles": null,
                "stateful": false,
                "description": "",
                "name": "c2",
                "source": {
                        "type": "image",
                        "certificate": "",
                        "alias": "alpine/3.21",
                        "server": "https://images.linuxcontainers.org",
                        "protocol": "simplestreams",
                        "mode": "pull",
                        "allow_inconsistent": false
                },
                "instance_type": "",
                "type": "container",
                "start": true
        }
DEBUG  [2025-05-04T19:22:08+08:00] Got operation from Incus
DEBUG  [2025-05-04T19:22:08+08:00]
        {
                "id": "39b83465-f301-4d43-9324-b8c06ccadebd",
                "class": "task",
                "description": "Creating instance",
                "created_at": "2025-05-04T19:22:08.37760526+08:00",
                "updated_at": "2025-05-04T19:22:08.37760526+08:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "instances": [
                                "/1.0/instances/c2"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DEBUG  [2025-05-04T19:22:08+08:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0/operations/39b83465-f301-4d43-9324-b8c06ccadebd"
DEBUG  [2025-05-04T19:22:08+08:00] Got response struct from Incus
DEBUG  [2025-05-04T19:22:08+08:00]
        {
                "id": "39b83465-f301-4d43-9324-b8c06ccadebd",
                "class": "task",
                "description": "Creating instance",
                "created_at": "2025-05-04T19:22:08.37760526+08:00",
                "updated_at": "2025-05-04T19:22:08.37760526+08:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "instances": [
                                "/1.0/instances/c2"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
Retrieving image: Unpacking image: 100% (4.04GB/s)

And nothing after Retrieving image: Unpacking image: 100% (4.04GB/s), with linstor shows the corresponding volume is cloning.

From afternoon to now, the linstor shows the following errors:
(I pasted 8 linstor err show together.)

root@nas:~# linstor err list
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š Id                    โ”Š Datetime            โ”Š Node             โ”Š Exception                                                                         โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š 68170F0E-6584A-000000 โ”Š 2025-05-04 15:40:07 โ”Š S|nas.hxdw.local โ”Š LinStorException: None 0 exit from: [timeout, 0, bash, -c, set -o pipefail;...    โ”Š
โ”Š 68170F0E-6584A-000001 โ”Š 2025-05-04 15:40:07 โ”Š S|nas.hxdw.local โ”Š StorageException: drbdmeta check-resize failed                                    โ”Š
โ”Š 68170F0E-6584A-000002 โ”Š 2025-05-04 15:41:36 โ”Š S|nas.hxdw.local โ”Š ImplementationError: ZfsThinProvider did not delete the volume ZfsData [Node: ... โ”Š
โ”Š 68170F0E-6584A-000003 โ”Š 2025-05-04 15:41:54 โ”Š S|nas.hxdw.local โ”Š ImplementationError: ZfsThinProvider did not delete the volume ZfsData [Node: ... โ”Š
โ”Š 68170F0E-6584A-000004 โ”Š 2025-05-04 15:41:57 โ”Š S|nas.hxdw.local โ”Š ImplementationError: ZfsThinProvider did not delete the volume ZfsData [Node: ... โ”Š
โ”Š 68170F0E-6584A-000005 โ”Š 2025-05-04 15:45:47 โ”Š S|nas.hxdw.local โ”Š ImplementationError: The DeviceManager service caught an unhandled exception o... โ”Š
โ”Š 68170F0E-6584A-000006 โ”Š 2025-05-04 19:20:41 โ”Š S|nas.hxdw.local โ”Š ImplementationError: The DeviceManager service caught an unhandled exception o... โ”Š
โ”Š 68170F0E-6584A-000007 โ”Š 2025-05-04 19:22:13 โ”Š S|nas.hxdw.local โ”Š ImplementationError: The DeviceManager service caught an unhandled exception o... โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
root@nas:~# linstor err show 68170F0E-6584A-000000
ERROR REPORT 68170F0E-6584A-000000

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:40:07
Node:                               nas.hxdw.local
Thread:                             clone_ZFS_COPY(incus-volume-ecc6597dad2940cba70e9d6cb27fcaa5/0->incus-volume-bab1791fa687479ab1c882b06b76e67c/0)

============================================================

Reported error:
===============

Category:                           LinStorException
Class name:                         LinStorException
Class canonical name:               com.linbit.linstor.LinStorException
Generated at:                       Method 'run', Source file 'CloneDaemon.java', Line #137

Error message:                      None 0 exit from: [timeout, 0, bash, -c, set -o pipefail; zfs send --embed --large-block hx-pool/incus/incus-volume-ecc6597dad2940cba70e9d6cb27fcaa5_00000@CF_c96f254b | zfs receive -F hx-pool/incus/incus-volume-bab1791fa687479ab1c882b06b76e67c_00000 && zfs destroy -r hx-pool/incus/incus-volume-bab1791fa687479ab1c882b06b76e67c_00000@%]

ErrorContext:
  Description: Clone command failed
  Cause:       cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one


Call backtrace:

    Method                                   Native Class:Line number
    run                                      N      com.linbit.linstor.clone.CloneDaemon:137
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000001
ERROR REPORT 68170F0E-6584A-000001

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:40:07
Node:                               nas.hxdw.local
Thread:                             clone_ZFS_COPY(incus-volume-ecc6597dad2940cba70e9d6cb27fcaa5/0->incus-volume-bab1791fa687479ab1c882b06b76e67c/0)

============================================================

Reported error:
===============

Category:                           LinStorException
Class name:                         StorageException
Class canonical name:               com.linbit.linstor.storage.StorageException
Generated at:                       Method 'processAfterClone', Source file 'DrbdLayer.java', Line #2041

Error message:                      drbdmeta check-resize failed

ErrorContext:


Call backtrace:

    Method                                   Native Class:Line number
    processAfterClone                        N      com.linbit.linstor.layer.drbd.DrbdLayer:2041
    processAfterClone                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1770
    cleanupDevices                           N      com.linbit.linstor.clone.CloneService:204
    postClone                                N      com.linbit.linstor.clone.CloneService:544
    lambda$startClone$2                      N      com.linbit.linstor.clone.CloneService:395
    run                                      N      com.linbit.linstor.clone.CloneDaemon:171
    run                                      N      java.lang.Thread:1583

Caused by:
==========

Category:                           LinStorException
Class name:                         ExtCmdFailedException
Class canonical name:               com.linbit.extproc.ExtCmdFailedException
Generated at:                       Method 'drbdMetaCheckResize', Source file 'DrbdAdm.java', Line #410

Error message:                      The external command 'drbdmeta' exited with error code 1


ErrorContext:
  Description: Execution of the external command 'drbdmeta' failed.
  Cause:       The external command exited with error code 1.
  Correction:  - Check whether the external program is operating properly.
- Check whether the command line is correct.
  Contact a system administrator or a developer if the command line is no longer valid
  for the installed version of the external program.
  Details:     The full command line executed was:
drbdmeta 1001 v09 /dev/zvol/hx-pool/incus/incus-volume-bab1791fa687479ab1c882b06b76e67c_00000 internal check-resize

The external command sent the following output data:


The external command sent the following error information:
no suitable meta data found :(




Call backtrace:

    Method                                   Native Class:Line number
    drbdMetaCheckResize                      N      com.linbit.linstor.layer.drbd.utils.DrbdAdm:410
    processAfterClone                        N      com.linbit.linstor.layer.drbd.DrbdLayer:2026
    processAfterClone                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1770
    cleanupDevices                           N      com.linbit.linstor.clone.CloneService:204
    postClone                                N      com.linbit.linstor.clone.CloneService:544
    lambda$startClone$2                      N      com.linbit.linstor.clone.CloneService:395
    run                                      N      com.linbit.linstor.clone.CloneDaemon:171
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000002
ERROR REPORT 68170F0E-6584A-000002

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:41:36
Node:                               nas.hxdw.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'processResource', Source file 'StorageLayer.java', Line #296

Error message:                      ZfsThinProvider did not delete the volume ZfsData [Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c', VlmNr: '0', RscNameSuffix: ]

Error context:
        An error occurred while processing resource 'Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c''
Call backtrace:

    Method                                   Native Class:Line number
    processResource                          N      com.linbit.linstor.layer.storage.StorageLayer:296
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1369
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1412
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1442
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1365
    processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:386
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:228
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000003
ERROR REPORT 68170F0E-6584A-000003

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:41:54
Node:                               nas.hxdw.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'processResource', Source file 'StorageLayer.java', Line #296

Error message:                      ZfsThinProvider did not delete the volume ZfsData [Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c', VlmNr: '0', RscNameSuffix: ]

Error context:
        An error occurred while processing resource 'Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c''
Call backtrace:

    Method                                   Native Class:Line number
    processResource                          N      com.linbit.linstor.layer.storage.StorageLayer:296
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1369
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1412
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1442
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1365
    processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:386
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:228
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000004
ERROR REPORT 68170F0E-6584A-000004

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:41:57
Node:                               nas.hxdw.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'processResource', Source file 'StorageLayer.java', Line #296

Error message:                      ZfsThinProvider did not delete the volume ZfsData [Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c', VlmNr: '0', RscNameSuffix: ]

Error context:
        An error occurred while processing resource 'Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c''
Call backtrace:

    Method                                   Native Class:Line number
    processResource                          N      com.linbit.linstor.layer.storage.StorageLayer:296
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1369
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1412
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1442
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1365
    processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:386
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:228
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000005
ERROR REPORT 68170F0E-6584A-000005

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 15:45:47
Node:                               nas.hxdw.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'devMgrLoop', Source file 'DeviceManagerImpl.java', Line #842

Error message:                      The DeviceManager service caught an unhandled exception of type AccessToDeletedDataException.


Call backtrace:

    Method                                   Native Class:Line number
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:842
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583

Caused by:
==========

Category:                           RuntimeException
Class name:                         AccessToDeletedDataException
Class canonical name:               com.linbit.linstor.AccessToDeletedDataException
Generated at:                       Method 'checkDeleted', Source file 'AbsCoreObj.java', Line #67

Error message:                      Access to deleted Volume

ErrorContext:   Details:     Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c', VlmNr: '0'


Call backtrace:

    Method                                   Native Class:Line number
    checkDeleted                             N      com.linbit.linstor.core.objects.AbsCoreObj:67
    getVolumeNumber                          N      com.linbit.linstor.core.objects.Volume:403
    getVlmNr                                 N      com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject:67
    getVlmNr                                 N      com.linbit.linstor.clone.CloneService$CloneInfo:623
    compareTo                                N      com.linbit.linstor.clone.CloneService$CloneInfo:677
    compareTo                                N      com.linbit.linstor.clone.CloneService$CloneInfo:564
    cpr                                      N      java.util.concurrent.ConcurrentSkipListMap:393
    doGet                                    N      java.util.concurrent.ConcurrentSkipListMap:551
    containsKey                              N      java.util.concurrent.ConcurrentSkipListMap:1294
    contains                                 N      java.util.concurrent.ConcurrentSkipListSet:225
    startClone                               N      com.linbit.linstor.clone.CloneService:320
    startClone                               N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1121
    processClone                             N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:861
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:242
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000006
ERROR REPORT 68170F0E-6584A-000006

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 19:20:41
Node:                               nas.hxdw.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'devMgrLoop', Source file 'DeviceManagerImpl.java', Line #842

Error message:                      The DeviceManager service caught an unhandled exception of type AccessToDeletedDataException.


Call backtrace:

    Method                                   Native Class:Line number
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:842
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583

Caused by:
==========

Category:                           RuntimeException
Class name:                         AccessToDeletedDataException
Class canonical name:               com.linbit.linstor.AccessToDeletedDataException
Generated at:                       Method 'checkDeleted', Source file 'AbsCoreObj.java', Line #67

Error message:                      Access to deleted Volume

ErrorContext:   Details:     Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c', VlmNr: '0'


Call backtrace:

    Method                                   Native Class:Line number
    checkDeleted                             N      com.linbit.linstor.core.objects.AbsCoreObj:67
    getVolumeNumber                          N      com.linbit.linstor.core.objects.Volume:403
    getVlmNr                                 N      com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject:67
    getVlmNr                                 N      com.linbit.linstor.clone.CloneService$CloneInfo:623
    compareTo                                N      com.linbit.linstor.clone.CloneService$CloneInfo:677
    compareTo                                N      com.linbit.linstor.clone.CloneService$CloneInfo:564
    cpr                                      N      java.util.concurrent.ConcurrentSkipListMap:393
    doGet                                    N      java.util.concurrent.ConcurrentSkipListMap:551
    containsKey                              N      java.util.concurrent.ConcurrentSkipListMap:1294
    contains                                 N      java.util.concurrent.ConcurrentSkipListSet:225
    startClone                               N      com.linbit.linstor.clone.CloneService:320
    startClone                               N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1121
    processClone                             N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:861
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:242
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~# linstor err show 68170F0E-6584A-000007
ERROR REPORT 68170F0E-6584A-000007

============================================================

Application:                        LINBITยฎ LINSTOR
Module:                             Satellite
Version:                            1.31.0
Build ID:                           a187af5c85a96bb27df87a5eab0bcf9dd6de6a34
Build time:                         2025-04-08T09:36:27+00:00
Error time:                         2025-05-04 19:22:13
Node:                               nas.hxdw.local
Thread:                             DeviceManager

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'devMgrLoop', Source file 'DeviceManagerImpl.java', Line #842

Error message:                      The DeviceManager service caught an unhandled exception of type AccessToDeletedDataException.


Call backtrace:

    Method                                   Native Class:Line number
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:842
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583

Caused by:
==========

Category:                           RuntimeException
Class name:                         AccessToDeletedDataException
Class canonical name:               com.linbit.linstor.AccessToDeletedDataException
Generated at:                       Method 'checkDeleted', Source file 'AbsCoreObj.java', Line #67

Error message:                      Access to deleted Volume

ErrorContext:   Details:     Node: 'nas.hxdw.local', Rsc: 'incus-volume-bab1791fa687479ab1c882b06b76e67c', VlmNr: '0'


Call backtrace:

    Method                                   Native Class:Line number
    checkDeleted                             N      com.linbit.linstor.core.objects.AbsCoreObj:67
    getVolumeNumber                          N      com.linbit.linstor.core.objects.Volume:403
    getVlmNr                                 N      com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject:67
    getVlmNr                                 N      com.linbit.linstor.clone.CloneService$CloneInfo:623
    compareTo                                N      com.linbit.linstor.clone.CloneService$CloneInfo:677
    compareTo                                N      com.linbit.linstor.clone.CloneService$CloneInfo:564
    cpr                                      N      java.util.concurrent.ConcurrentSkipListMap:393
    doGet                                    N      java.util.concurrent.ConcurrentSkipListMap:551
    containsKey                              N      java.util.concurrent.ConcurrentSkipListMap:1294
    contains                                 N      java.util.concurrent.ConcurrentSkipListSet:225
    startClone                               N      com.linbit.linstor.clone.CloneService:320
    startClone                               N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1121
    processClone                             N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:861
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:242
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:335
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1150
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:780
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:676
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

root@nas:~#

And here is the output of linstor resource-definition, the volume that started cloning in the afternoon is still cloning now.

 linstor resource-definition list --show-props Aux/Incus/name Aux/Incus/type Aux/Incus/content-type
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š ResourceName                                  โ”Š Port โ”Š ResourceGroup โ”Š Layers       โ”Š State   โ”Š Aux/Incus/name                                                                โ”Š Aux/Incus/type โ”Š Aux/Incus/content-type โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š incus-volume-0ac4a3c3ee644b63908b1e46c8702451 โ”Š 7001 โ”Š default       โ”Š DRBD,STORAGE โ”Š CLONING โ”Š incus-volume-018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac โ”Š images         โ”Š filesystem             โ”Š
โ”Š incus-volume-4d77d48a03b04333aa9b1b0864eddc21 โ”Š 7004 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok      โ”Š incus-volume-b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e โ”Š images         โ”Š filesystem             โ”Š
โ”Š incus-volume-a120c7a2c06649c984b6456ed74aa1e6 โ”Š 7005 โ”Š default       โ”Š DRBD,STORAGE โ”Š CLONING โ”Š incus-volume-b670fe21654f3c57b803f978ba6592a4d3acc39acf59cb0af31f80b48020d03e โ”Š images         โ”Š filesystem             โ”Š
โ”Š incus-volume-ad0b40c332b04606b1af82955143f352 โ”Š 7003 โ”Š default       โ”Š DRBD,STORAGE โ”Š CLONING โ”Š incus-volume-8178b6bfe5767a363cb2a27f0befcd72b766f09eae7f7fe0884919686261cb63 โ”Š images         โ”Š filesystem             โ”Š
โ”Š incus-volume-dc2dd21987374bccb9c7a8dd09af81ab โ”Š 7002 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok      โ”Š incus-volume-8178b6bfe5767a363cb2a27f0befcd72b766f09eae7f7fe0884919686261cb63 โ”Š images         โ”Š filesystem             โ”Š
โ”Š incus-volume-eeb73fd5e1484723b45ec471e7f1d524 โ”Š 7000 โ”Š default       โ”Š DRBD,STORAGE โ”Š ok      โ”Š incus-volume-018f2ac7fbc42f7ec515ba4f1907f07e7e527c8af95b5e94cf27596872d034ac โ”Š images         โ”Š filesystem             โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

I found the first error log is different, and it says cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one.
I actually tested the zfs, linstor and incus following How to set up LINSTOR with Incus in a VM with an unencrypted pool, everything was working fine. Then I tried to repeat these operations on the physical machine where I had the encrypted pool, I configured everything successfully, migrated the existing containers and VMs from the previous storage, everything worked perfectly until I tried to create a new container, then the problem occurred.

We have not checked LINSTOR with encrypted pools unfortunately.

1 Like

I confirmed that the problem Iโ€™m experiencing is related to the zfs encrypted pool on linstor, and I filed an issue with detailed steps to reproduce it.

In the case of using a zfs encrypted pool as the linstor storage backend and linstor as the incus storage backend, incus launch and incus init do not work properly.

Hope someone can fix it.
Although I feel like the problem seems to be more related to linstor rather than incus.

Thanks for the detailed report. Iโ€™ll try to have a look next week, but I canโ€™t guarantee anything :slight_smile:

1 Like