Linstor: "Unable to start cloning" error

webbird83 · December 30, 2025, 11:45am

Hi everybody!

Kind of new in the Incus-world, and to be honest I have kind of an impression that my approach is over-complicated, although I could not yet find a better option. Please be calm with me if the question is too weird

What i try to do:

Have a IncusOS in my home lab
Have a Linstor Controller on a VPS
Have a S3 compatible storage on very cheap device (Raspberry PI at a friend’s house) as backup of one of the storage pools of the IncusOS

The idea: Use the “Snapshot Shipping” feature of Linstor to backup the IncusOS data to the S3 compatible storage (see Controlling Data Replication with Snapshot Shipping - LINBIT). I am aware that this is kind of a corner-case usage of Linstor, but I could not come up with a better backup/restore solution (scheduled, incremental, deduplicated, encrypted - my limiting resource is primarily the bandwidth of the Raspberry PI).

# Setting configure the Linstor backend in Incus
incus config set incusos: storage.linstor.controller_connection=http://192.168.122.116:3370
incus config set incusos: storage.linstor.satellite.name=18e79163-b792-4521-a7cf-d2956550223d

# Create "MyLinstorPool" as zfs volume - it is supposed to be the actual incus storage pool that takes part in the Linstor storage pool
incus storage create MyLinstorPool zfs size=20GB
linstor storage-pool create zfs 18e79163-b792-4521-a7cf-d2956550223d MyLinstorPool MyLinstorPool

# Create a "remote" Incus storage pool - a pool that uses the Linstor driver now
incus storage create remote linstor linstor.resource_group.storage_pool=MyLinstorPool linstor.resource_group.place_count=1

Now my problem: If I create an instance in the “remote” pool, I get a ApiCallError:

incus launch images:debian/12 c1 --storage remote
Launching c1
Error: Failed instance creation: Failed creating instance from image: Unable to start cloning resource definition: json: cannot unmarshal object into Go value of type client.ApiCallError

The Linstor error log shows this:

user@vps:~$ sudo journalctl -u linstor-controller -n 100 | grep -i "clone\|error\|remote"
Dec 30 11:35:58 vps Controller[699]: 2025-12-30 11:35:58.191 [grizzly-http-server-0] INFO  LINSTOR/Controller/717627 SYSTEM - REST/API RestClient(192.168.122.236; 'Go-http-client/1.1')/CloneRscDfn
Dec 30 11:35:58 vps Controller[699]: 2025-12-30 11:35:58.226 [grizzly-http-server-0] ERROR LINSTOR/Controller/ SYSTEM - No suitable storage pools found for cloning. [Report number 69537AF1-00000-000013]
user@vps:~$ linstor err show 69537AF1-00000-000013
ERROR REPORT 69537AF1-00000-000013

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Controller
Version:                            1.33.1
Build ID:                           95da7940d6efb6a39ea303c5f37b03478a6fab0b
Build time:                         2025-12-22T16:04:57+00:00
Error time:                         2025-12-30 11:35:58
Node:                               vps
Thread:                             grizzly-http-server-0
Access context information

Identity:                           PUBLIC
Role:                               PUBLIC
Domain:                             PUBLIC

Peer:                               RestClient(192.168.122.236; 'Go-http-client/1.1')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'findCloneStoragePools', Source file 'CtrlRscDfnApiCallHandler.java', Line #1231

Error message:                      No suitable storage pools found for cloning.

Error context:
        No suitable storage pools found for cloning.
ApiRcException entries: 
  Message:     No suitable storage pools found for cloning.
  NumericCode: -4611686018427386908


Asynchronous stage backtrace:
    
    Error has been observed at the following site(s):
        *__checkpoint ⇢ Clone resource-definition
    Original Stack Trace:

Call backtrace:

    Method                                   Native Class:Line number
    findCloneStoragePools                    N      com.linbit.linstor.core.apicallhandler.controller.CtrlRscDfnApiCallHandler:1231

Suppressed exception 1 of 1:
===============
Category:                           RuntimeException
Class name:                         OnAssemblyException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at:                       Method 'findCloneStoragePools', Source file 'CtrlRscDfnApiCallHandler.java', Line #1231

Error message:                      
Error has been observed at the following site(s):
        *__checkpoint ⇢ Clone resource-definition
Original Stack Trace:

Error context:
        No suitable storage pools found for cloning.
Call backtrace:

    Method                                   Native Class:Line number
    findCloneStoragePools                    N      com.linbit.linstor.core.apicallhandler.controller.CtrlRscDfnApiCallHandler:1231
    cloneRscDfnInTransaction                 N      com.linbit.linstor.core.apicallhandler.controller.CtrlRscDfnApiCallHandler:1411
    lambda$cloneRscDfn$7                     N      com.linbit.linstor.core.apicallhandler.controller.CtrlRscDfnApiCallHandler:851
    doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:178
    lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:101
    call                                     N      reactor.core.publisher.MonoCallable:72
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:128
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8833
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2570
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Flux:8848
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2570
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Flux:8848
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
    onNext                                   N      reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:129
    completePossiblyEmpty                    N      reactor.core.publisher.Operators$BaseFluxToMonoOperator:2096
    onComplete                               N      reactor.core.publisher.MonoCollect$CollectSubscriber:145
    onComplete                               N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner:261
    checkTerminated                          N      reactor.core.publisher.FluxFlatMap$FlatMapMain:850
    drainLoop                                N      reactor.core.publisher.FluxFlatMap$FlatMapMain:612
    drain                                    N      reactor.core.publisher.FluxFlatMap$FlatMapMain:592
    onComplete                               N      reactor.core.publisher.FluxFlatMap$FlatMapMain:469
    checkTerminated                          N      reactor.core.publisher.FluxFlatMap$FlatMapMain:850
    drainLoop                                N      reactor.core.publisher.FluxFlatMap$FlatMapMain:612
    drain                                    N      reactor.core.publisher.FluxFlatMap$FlatMapMain:592
    onComplete                               N      reactor.core.publisher.FluxFlatMap$FlatMapMain:469
    complete                                 N      reactor.core.publisher.Operators:137
    subscribe                                N      reactor.core.publisher.FluxIterable:144
    subscribe                                N      reactor.core.publisher.FluxIterable:83
    subscribe                                N      reactor.core.publisher.Flux:8848
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:202
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8833
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2570
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Flux:8848
    onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:430
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner:251
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2570
    onSubscribeInner                         N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:150
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner:246
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:193
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8833
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2570
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.InternalMonoOperator:76
    subscribe                                N      reactor.core.publisher.MonoUsing:102
    subscribe                                N      reactor.core.publisher.Mono:4576
    subscribeWith                            N      reactor.core.publisher.Mono:4642
    subscribe                                N      reactor.core.publisher.Mono:4542
    subscribe                                N      reactor.core.publisher.Mono:4478
    subscribe                                N      reactor.core.publisher.Mono:4450
    doFlux                                   N      com.linbit.linstor.api.rest.v1.RequestHelper:345
    clone                                    N      com.linbit.linstor.api.rest.v1.ResourceDefinitions:394
    invoke                                   N      jdk.internal.reflect.DirectMethodHandleAccessor:103
    invoke                                   N      java.lang.reflect.Method:580
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker:159
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.

Unclear: I found several notes that enrypted ZFS is unsuable in combination with Linstor, and a bug that is very similar (In the case of using a zfs encrypted pool as the linstor storage backend and linstor as the incus storage backend, incus launch and incus init do not work properly. · Issue #2075 · lxc/incus · GitHub) - although both the incus error and the Linstor error are a little bit different.

Could you please help me?

bensmrs · December 30, 2025, 1:08pm

I wouldn’t say it’s a corner-case usage, it’s a pretty sane backup mechanism.

webbird83:

Now my problem: If I create an instance in the “remote” pool, I get a ApiCallError:
incus launch images:debian/12 c1 --storage remote
Launching c1
Error: Failed instance creation: Failed creating instance from image: Unable to sta

Weird… Maybe we’re not catching an error on Incus side.

Could you show the usual linstor node list, linstor storage-pool list and linstor resource-group list, please?

That’s correct. Encrypted ZFS doesn’t work with LINSTOR. If you want encrypted volumes, you need to use LUKS. If you tried using encrypted ZFS volumes, we can’t help you any further, I’m afraid.

webbird83 · December 30, 2025, 1:43pm

Thanks Benjamin, for this super quick reply!

Here the infos you have been asking for:

user@vps:~$ linstor node list
╭──────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                                 ┊ NodeType  ┊ Addresses                    ┊ State  ┊
╞══════════════════════════════════════════════════════════════════════════════════════════╡
┊ 18e79163-b792-4521-a7cf-d2956550223d ┊ SATELLITE ┊ 192.168.122.236:3366 (PLAIN) ┊ Online ┊
╰──────────────────────────────────────────────────────────────────────────────────────────╯

user@vps:~$ linstor storage-pool list
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node                                 ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                                                ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ 18e79163-b792-4521-a7cf-d2956550223d ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ 18e79163-b792-4521-a7cf-d2956550223d;DfltDisklessStorPool ┊
┊ MyLinstorPool        ┊ 18e79163-b792-4521-a7cf-d2956550223d ┊ ZFS      ┊ MyLinstorPool ┊     7.76 GiB ┊     18.50 GiB ┊ True         ┊ Ok    ┊ 18e79163-b792-4521-a7cf-d2956550223d;MyLinstorPool        ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
user@vps:~$ linstor resource-group list
╭──────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter                  ┊ VlmNrs ┊ Description                     ┊
╞══════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2                 ┊        ┊                                 ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ remote        ┊ PlaceCount: 1                 ┊        ┊ Resource group managed by Incus ┊
┊               ┊ StoragePool(s): MyLinstorPool ┊        ┊                                 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────╯

Regarding the encryption: As far as I understood, IncusOS ALWAYS uses encrypted ZFS (in contrast to “pure” Incus). If this is a misunderstanding, I would appreciate if you could correct me. The only other “match” between Linstor and IncusOS that I could find is of type file-system (which on the other hand does not provide all the efficient snapshot features).

Again, thank you so much!

stgraber · December 30, 2025, 2:06pm

IncusOS has a create-volume command to create a volume (dataset in practice) for use with either Incus or Linstor.

When a volume is created for use by Linstor, that dataset is marked as unencrypted.

bensmrs · December 30, 2025, 2:06pm

Ok, I just wanted to be extra sure that you set up the proper place count.
I need to clarify something: is 192.168.122.116 your local IncusOS instance or your remote VPS that’s somehow visible from your local network?

If it’s your local instance, I’d highly suggest setting the IP to 127.0.0.1
If it’s your remote VPS, then here’s your mistake: you need a local LINSTOR satellite on your Incus server (this is what allows you to create DRBD disks that get mounted by Incus).

EDIT: to continue debugging, I’d need to know what 192.168.122.116 and 192.168.122.236 are, as that could lead to very different diagnoses

I’m by no means an IncusOS expert, but I’d say that only applies to local volumes, as ZFS encryption is not very well supported by replicated storage drivers (this is the unfortunate case of LINSTOR).

stgraber · December 30, 2025, 2:08pm

In your case, it looks like you’ve created a file backed ZFS pool to use with Linstor, so that won’t be encrypted.

stgraber · December 30, 2025, 2:28pm

See here for how to create a dataset for use by Linstor directly on IncusOS managed storage:

Not that I would expect this to be the source of the issue here, just a better way to do what you’re currently doing by avoiding a layer of indirection and by getting to use what’s likely a bigger chunk of your storage.

webbird83 · December 30, 2025, 4:19pm

Wow guys, are you responsive Thanks so much!

Ok, at the moment I am still experimenting in a virtualized environment. Ultimately I want to go on a hardware setup.

Role	Usage	Experimental Address	Final Address
IncusOS	VMs/Containers/Data on a Mini PC	192.168.122.236	10.11.12.15
VPS	Linstore-Controller, … on a publicly accessible VPS	192.168.122.116	10.11.12.13
Backup	minio (S3) on Raspberry PI	192.168.122.8	10.11.12.14

Answering your questions:

192.168.122.116 is the VPS (in my experimental setup)
- It is NOT the Linstor satellite, just the controller
The IncusOS (192.168.122.236) is supposed to be the one and only satellite (at the same time providing storage and using it itself for VMs/Containers/Data)
- My idea was here: Linstor would be clever enough that I can somehow make a pool or volume part of the distributed storage space (even though there is just one participant) and directly use it in Incus, and I could use Linstor-Controller just make the “scheduler” for the backups to the remote S3 storage.

Thanks Stéphane! Thanks for the link. I tried to use this (in various ways, since, to be honest, I did not really understand the concept of pools/volumes/resources), but I failed

incus config set incusos: storage.linstor.controller_connection=http://192.168.122.116:3370
incus config set incusos: storage.linstor.satellite.name=18e79163-b792-4521-a7cf-d2956550223d

# Creating a new volume in the "local" ZFS pool
incus admin os system storage create-volume -d '{"pool":"local","name":"mylinstorvolume5","use":"linstor"}'
linstor storage-pool create zfs 18e79163-b792-4521-a7cf-d2956550223d mylinstorvolume5 local/mylinstorvolume5
incus storage create mylinstorvolume5 linstor source=mylinstorvolume5 linstor.resource_group.storage_pool=mylinstorvolume5 linstor.resource_group.place_count=1

Unfortunately the error remains somehow:

Instance creation failed
Failed creating instance from image: Unable to start cloning resource definition: json: cannot unmarshal object into Go value of type client.ApiCallError

ser@vps:~$ sudo journalctl -u linstor-controller -n 100 | grep -i "clone\|error\|remote"
Dec 30 17:05:23 vps Controller[699]: 2025-12-30 17:05:23.343 [grizzly-http-server-1] INFO  LINSTOR/Controller/4cf9c4 SYSTEM - REST/API RestClient(192.168.122.236; 'Go-http-client/1.1')/CloneRscDfn
Dec 30 17:05:23 vps Controller[699]: 2025-12-30 17:05:23.352 [grizzly-http-server-1] ERROR LINSTOR/Controller/ SYSTEM - No suitable storage pools found for cloning. [Report number 69537AF1-00000-000022]

I am almost sure that I use the functions to create volume/pool/resource wrong or in the wrong order Can you see the problem?

bensmrs · December 30, 2025, 4:43pm

That feels pretty unsafe, as if you lose the connection with your VPS, you’ll be unable to operate the cluster.

For the actual problem, I’m out of ideas. No suitable storage pools found for cloning suggests that there are constraints that cannot be satisfied by the currently defined storage pools, but I don’t know which (and I’m not really focused right now).

webbird83 · December 30, 2025, 5:34pm

You are right - maybe the overall solution is somehow too complex.

Actually what I want to do is pretty simple: Have a full backup on a different machine:

Incremental (so that I can select from various earlier times when rolling back)
Encrypted (so that the backup machine can be set up at friend’s home without giving them full access to all data)
Compressed/Deduplicated (so that the transfer is possible even with very low bandwidth)
Scheduled (so that I can run the backup only during the maintenance window at night time)
Partial recovery (possibility to roll back only a part of the VMs/Containers/Volumes)

Do you guys know anything in IncusOS that suits these needs? (I researched a lot of options already, but there is always one little thing that does not work)

Backup tools like kopia would fulfill all these needs, if they just had access to the /var/lib/incus filesystem. I am wondering, if there would be interest in someone implementing a kopia backup service (or application) into IncusOS.

Would there be interest? Or am I on a wrong track?
(I could maybe try to start a proof-of-concept, if you guys say this is sensible and you could give me some architectural directions)

rkunze · January 2, 2026, 10:34am

I am using BorgBackup (https://www.borgbackup.org/) for that. BorgBackup is extremely efficient at deduplicating and has fully encrypted backups.

What I do is:

export individual containers/VMs or storage volume to a tar file with incus export/ incus storage volume export
feed the resulting tar file to borg with borg import-tar ...

Restore is basically the same in reverse, borg export-tar ... and then incus import .... And using import-tar instead of just backing up the tar files lets me access the backed up individual files in containers or storage volumes as well (for VMs it does not really matter because the root fs for a VM is exported to tar as a blob instead of indicidual files in a directory tree).

And what I’d really like to see in incus is a way to export to / import from stdin directly, without the need to create potentially huge intermediate tar archives on disk…

bensmrs · January 2, 2026, 12:41pm

Incus is admittedly a bit lacking backup-wise. I’ll personally be working on a connector for Bareos in January, but it may not suit everybody (especially homelabbers), as it’s a fairly hard to configure piece of software.

webbird83 · January 2, 2026, 4:57pm

Thanks for sharing your experience!

I thought about a solution like this as well, but having these big intermediate files was worrying me. First of all they eat up a lot of resources (both disk space and computating time), which of course is somehow a waste. And secondly it causes lots of write cycles to my SSDs, which will make them probably fail sooner than later.

Thanks @bensmrs for sharing this. Haven’t ever heard of it - but as you say, this might be more targetted to other people than me

At the moment I am really a little lost on the backup topic. I have to admit that (although I really love the IncusOS approach!!!) I have started now experiments with Proxmox (and their great Proxmox Backup Server). I would prefer IncusOS in several ways - but without having the feeling that I can recover from a disaster, I would not dare to use it

It’s hard to imagine that I am the only one needing this - I assume, I just don’t see the most obvious approach Does anybody have a geo-redundant backup solution set up (via low bandwidth connection)?

rkunze · January 2, 2026, 5:41pm

Depends on how you define as “low bandwith”, and on how much your data changes between increments. Actually, it’s a bit of the wrong question - with a good incremental solution (like BorgBackup, or even simply using incus copy --refreshprovided that both ends run the same storage system and that system supports incremental transfers), the bandwidth you need for the daily increments pretty much depends only on how much your data changes between increments.

That said, I’ve been running my BorgBackup solution for a system that runs the office IT of a (very) small software development company (Mailserver, GIT repositories, test systems, …) with daily incremental backups over a 10 Mbit connection without any problems.

ykazakov · January 26, 2026, 10:28am

I just got exactly the same error when trying to setup Linstor in a fresh IncusOS cluster:

The reasone turned out to be the size of the ZFS pool. Apperently, 20 GB is not enough!

If using 20GB volume, the empty Incus volume created using the command incus storage volume create remote fsvol would occupy over 10 GB:

# linstor storage-pool list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ remote               ┊ incus-os1 ┊ ZFS      ┊ linstore ┊    18.89 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;remote               ┊
┊ remote               ┊ incus-os2 ┊ ZFS      ┊ linstore ┊    18.89 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;remote               ┊
┊ remote               ┊ incus-os3 ┊ ZFS      ┊ linstore ┊     8.73 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;remote               ┊
┊ remote               ┊ incus-os4 ┊ ZFS      ┊ linstore ┊     8.73 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;remote               ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Consequently, cloning/copying of this volume fails:

% incus storage volume copy remote/fsvol remote/fsvol1
Error: Unable to start cloning resource definition: json: cannot unmarshal object into Go value of type client.ApiCallError

It looks like Linstor has a very conservative estimation of the free space required for the cloning operation and probably checks if the free space on the same storage pool (since ZFS can only create snapshots on the same pool) is at least as large as the volume size. Consequently the clone operation fails with the error:

Using a volume on the local pool probably failed for the same reason:

When I used a 50 GB disk for IncusOS (the minimal space requirement), only about 16 GB was available in the local storage pool:

root@linstor-controller1:~# linstor storage-pool create zfs incus-os1 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool create zfs incus-os2 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool create zfs incus-os3 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool create zfs incus-os4 zfs_pool local/linstor
root@linstor-controller1:~# linstor sp l
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    15.16 GiB ┊     16.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    14.87 GiB ┊     16.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    14.98 GiB ┊     16.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    15.80 GiB ┊     16.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I changed the disk size to 100 GB (had to reinstall IncusOS as there seem to be no easy way to expand the existing ZFS file system on the last disk partition), and that now was sufficiently large:

root@linstor-controller1:~# linstor storage-pool list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    63.81 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    64.34 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    64.34 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    64.43 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
% incus storage volume create remote fsvol
Storage volume fsvol created
% incus storage volume copy remote/fsvol remote/fsvol1
Storage volume copied successfully!
% incus launch images:debian/12 c1 --storage remote
Launching c1
% incus storage volume attach remote fsvol c1 /mnt
 % incus launch images:debian/13 v1 --storage remote --vm -c migration.stateful=true
Launching v1
root@linstor-controller1:~# linstor resource-definition list --show-props Aux/Incus/name Aux/Incus/type Aux/Incus/content-type
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                                  ┊ ResourceGroup ┊ Layers       ┊ State ┊ Aux/Incus/name                                                                ┊ Aux/Incus/type   ┊ Aux/Incus/content-type ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-777ec6eb092d4384b92b522d54cb664c ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-internal_fsvol1                                                  ┊ custom           ┊ filesystem             ┊
┊ incus-volume-ae49130a7dab40539d5dd578a23600b8 ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-internal_fsvol                                                   ┊ custom           ┊ filesystem             ┊
┊ incus-volume-bf67ea1f5e93447ea12505c4528282c4 ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-20afac0c21620ccf1950a160c750acd4e8b692207a4a1130a810b82e515f5fdc ┊ images           ┊ block                  ┊
┊ incus-volume-c4bc004dbf514f1688fc0c5fa5a93867 ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-internal_v1                                                      ┊ virtual-machines ┊ block                  ┊
┊ incus-volume-d44304e38e0d4913ba856324299fb618 ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-internal_c1                                                      ┊ containers       ┊ filesystem             ┊
┊ incus-volume-e5405ac549f8476a900532125247ec5b ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-bb5d397a2780afaa037416b3469e6bc72f00e369011dfb5e6540faeeed60dcd7 ┊ images           ┊ filesystem             ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Still, a bit puzzled by the space consumption. Over 40 GB seems a bit large for just one container, one VM and one empty storage pool with a copy:

root@linstor-controller1:~# linstor storage-pool list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    42.96 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    43.40 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    21.39 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    21.82 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Just for the record (mainly to myself), here is the complete setup of Linstor (without SSL) from a freshly created IncusOS cluster:

% incus cluster ls
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|   NAME    |                          URL                          |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS |      MESSAGE      |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os1 | https://[fd42:470f:55fe:4d36:216:3eff:fe03:6f67]:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|           |                                                       | database         |              |                |             |        |                   |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os2 | https://[fd42:470f:55fe:4d36:216:3eff:fe0e:fea4]:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os3 | https://[fd42:470f:55fe:4d36:216:3eff:fe89:7f20]:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os4 | https://[fd42:470f:55fe:4d36:216:3eff:feda:d82f]:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+

Launch a container for the Linstor controller in the internal project. I used Debian 13 with the instructions from here (just to match the version in IncusOS), but using Ubuntu 24.04 as in the tutorial also worked.

% incus project switch internal
% incus launch images:debian/13 linstor-controller1
% incus exec linstor-controller1 -- bash
root@linstor-controller1:~# apt install ca-certificates
# create:
root@linstor-controller1:~# cat /etc/apt/sources.list.d/linbit.sources 
Types: deb
URIs: https://packages.linbit.com/public
Suites: proxmox-9
Components: drbd-9
Architectures: amd64
Signed-By: /etc/apt/keyrings/linbit.asc
root@linstor-controller1:~# cat /etc/apt/keyrings/linbit.asc 
-----BEGIN PGP PUBLIC KEY BLOCK-----

xsFNBGZsIboBEADLvvRG52QXsGL7d33xOYQzxjriCdLsXTnBhsC/bQc4dpQBfAp7
FEYhp93NcrwwPXLp5thYHfg3RF/O5pbyvpAnKZJt0rXMEvwy2K3JVJL7ddYs4cU6
I4m1n6W+YYQydEeMmZUFc77DNwQ74Dw3Xx/natz1dK2Y9S0pyBPb2ZMtWwDMuDpV
P/Y/W/wuDMWusbtSjVb+4TXJy046vPBobzi2jtQyaPJFfqwRiYqJYrQmsnns7L3F
ksd63Y4veqK9rvNiYDVXZNsXja0An59rUdwhMlHj/wJWhr23SnSbcQv3hkvY/I19
YT5FD9Y9srmRG2tIiQbK2rAjLqjKNGvYINRBlwnLDCCnoiCLJTahi2CT90aOr0Ae
7QKITi9cpDZJO+/vaLw7JnHEmhpt5W1WySFtSgSQWIYdkhF2Rx1aOkQwT/Ig6rDV
tkA8CjJ4ACMYXAx+zh+pxIRg2JNkLoGdl7pc/+5f4VV9iup3VW5U9MoC3KzZBDGJ
/DNF9x/1Rkf/H7tP3pYU3UmaPOCMvOx3zuwo1qqEUT9pKxybZo+lOPdNVgZbAuKY
C2jJa3Mv64IqyBds9tfZY8Sj+pbhfSvgTSmXR0WARXPS3po1PjUicnrStc+cWfEY
g7vh4Wn2EsExlgmdARvDl1TgENQCLshTI2jTIPu9XXUZlDVltGJZuGem4wARAQAB
zSlMSU5CSVQgUGFja2FnZSBhbmQgUmVwb3NpdG9yeSBTaWduaW5nIEtlecLBjgQT
AQoAOBYhBE5ThVRnJtE8tkmHLPwFox24Jv5IBQJmbCG6AhsDBQsJCAcCBhUKCQgL
AgQWAgMBAh4BAheAAAoJEPwFox24Jv5IaTIP/0qHRPuK0cKPsMEw7Bx+y9liST9y
XC6xEbcWEPm9qkjW73/ZdTLRKI4Ty9UGH7Q0zQ1otUdGYOolidtDAYKS2V3++PwQ
mbESJJiKOeKs1rjXeEBUr0usyhsL3P4/hFLCMf1ctgZNWklanu/N/aMQVjTQoYJu
rDCxq1y58/3hAyrWQ2kITHjVvpP5cXqXpDlwiLKy+oYNxvUdE6FUzNbuQ6htzWyQ
ugmMyjjqjlfD6gC7OqDcXk9eUf7AHbgf5+UQ+RbjMZ+YBoH9gihL/1TI1Ith50NX
5cVRMyscZ9inWFK8Fw2ubD8ZifmIQfrMZJMW0iajgJhX0GAqczl54Ihb08EODeRO
82oOgoeIJ7H14y2WfYA7Pb2Zap6WLrvA+k3arsr94aSDGutpThTBABR9F7F3fOLo
8dlv+jGXnsHtTeU9qh2+ZifyPZ3BVukyizoQk+TDxOavqNryQc7zLpwKsGymRJjP
M3ODV1fUVm6hLgPSMJP0tLakvK76+o/RCr5z5gCUyxjnnV0pbkYLhhE7FtLeSoPb
rxmNRzSFfdXk2uD1idlh54559JdSSh/HzqdZ1biizLs+xJK8JZPZI+3F/whVylwU
iwOhLRojCowfN2DhiiAencH0fsjgec3PT51bdCU6agvH8f6l1ivAuRb1RGT9WGpo
Xe+WxMjgfi4lbTsF
=m29F
-----END PGP PUBLIC KEY BLOCK-----
root@linstor-controller1:~# apt install linstor-controller linstor-client
root@linstor-controller1:~# systemctl enable --now linstor-controller

Enable the Linstor (sattilite?) service on every IncusOS node:

% incus admin os service edit linstor --target incus-os1
% incus admin os service edit linstor --target incus-os2
% incus admin os service edit linstor --target incus-os3
% incus admin os service edit linstor --target incus-os4
% incus admin os service edit linstor --target incus-os2
% incus admin os service show linstor --target incus-os1                       
WARNING: The IncusOS API and configuration is subject to change

config:
  enabled: true
  listen_address: ""
state: {}

Add the cluster nodes to as satilites inside the Linstor controller container:

% incus cluster ls
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|   NAME    |                          URL                          |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS |      MESSAGE      |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os1 | https://[fd42:470f:55fe:4d36:216:3eff:fe03:6f67]:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|           |                                                       | database         |              |                |             |        |                   |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os2 | https://[fd42:470f:55fe:4d36:216:3eff:fe0e:fea4]:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os3 | https://[fd42:470f:55fe:4d36:216:3eff:fe89:7f20]:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| incus-os4 | https://[fd42:470f:55fe:4d36:216:3eff:feda:d82f]:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+-----------+-------------------------------------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
root@linstor-controller1:~# linstor node create incus-os1 fd42:470f:55fe:4d36:216:3eff:fe03:6f67 --node-type satellite
root@linstor-controller1:~# linstor node create incus-os2 fd42:470f:55fe:4d36:216:3eff:fe0e:fea4 --node-type satellite
root@linstor-controller1:~# linstor node create incus-os3 fd42:470f:55fe:4d36:216:3eff:fe89:7f20 --node-type satellite
root@linstor-controller1:~# linstor node create incus-os4 fd42:470f:55fe:4d36:216:3eff:feda:d82f --node-type satellite
root@linstor-controller1:~# linstor node list
╭──────────────────────────────────────────────────────────────────────────────────────╮
┊ Node      ┊ NodeType  ┊ Addresses                                           ┊ State  ┊
╞══════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-os1 ┊ SATELLITE ┊ FD42:470F:55FE:4D36:216:3EFF:FE03:6F67:3366 (PLAIN) ┊ Online ┊
┊ incus-os2 ┊ SATELLITE ┊ FD42:470F:55FE:4D36:216:3EFF:FE0E:FEA4:3366 (PLAIN) ┊ Online ┊
┊ incus-os3 ┊ SATELLITE ┊ FD42:470F:55FE:4D36:216:3EFF:FE89:7F20:3366 (PLAIN) ┊ Online ┊
┊ incus-os4 ┊ SATELLITE ┊ FD42:470F:55FE:4D36:216:3EFF:FEDA:D82F:3366 (PLAIN) ┊ Online ┊
╰──────────────────────────────────────────────────────────────────────────────────────╯

Add a ZFS volume to the local Incus storage pool on each node and use it to create a Linstor storage pool:

% incus admin os system storage create-volume -d '{"pool":"local","name":"linstor","use":"linstor"}' --target incus-os1
% incus admin os system storage create-volume -d '{"pool":"local","name":"linstor","use":"linstor"}' --target incus-os2
% incus admin os system storage create-volume -d '{"pool":"local","name":"linstor","use":"linstor"}' --target incus-os3
% incus admin os system storage create-volume -d '{"pool":"local","name":"linstor","use":"linstor"}' --target incus-os4
root@linstor-controller1:~# linstor storage-pool create zfs incus-os1 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool create zfs incus-os2 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool create zfs incus-os3 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool create zfs incus-os4 zfs_pool local/linstor
root@linstor-controller1:~# linstor storage-pool list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    63.81 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    64.34 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    64.34 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    64.43 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Create the remote storage pool in Incus:

% incus ls
+---------------------+---------+------+--------------------------------------------------+-----------------+-----------+-----------+
|        NAME         |  STATE  | IPV4 |                       IPV6                       |      TYPE       | SNAPSHOTS | LOCATION  |
+---------------------+---------+------+--------------------------------------------------+-----------------+-----------+-----------+
| linstor-controller1 | RUNNING |      | fd42:cc7e:3e0d:c769:1266:6aff:fe2e:502c (eth0)   | CONTAINER       | 0         | incus-os1 |
+---------------------+---------+------+--------------------------------------------------+-----------------+-----------+-----------+
% incus config set storage.linstor.controller_connection="http://[fd42:cc7e:3e0d:c769:1266:6aff:fe2e:502c]:3370"  
% incus storage create remote linstor --target incus-os1  
% incus storage create remote linstor --target incus-os2
% incus storage create remote linstor --target incus-os3
% incus storage create remote linstor --target incus-os4  
% incus storage create remote linstor linstor.resource_group.storage_pool=zfs_pool
root@linstor-controller1:~# linstor resource-group list
╭─────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter             ┊ VlmNrs ┊ Description                     ┊
╞═════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2            ┊        ┊                                 ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ remote        ┊ PlaceCount: 2            ┊        ┊ Resource group managed by Incus ┊
┊               ┊ StoragePool(s): zfs_pool ┊        ┊                                 ┊
╰─────────────────────────────────────────────────────────────────────────────────────╯

Do I understand it correctly, that with this setup the volume local/linstor is unencrypted, even though the local storage pool is encrypeted?

stgraber · January 26, 2026, 5:32pm

Yes, that’s correct. The "use":"linstor" instructs IncusOS to make that dataset unencrypted so Linstor can work properly.

ykazakov · January 28, 2026, 7:22am

I just tried launching a few further containers on the remote Linstor storage pool, and quickly ran out of space:

% incus launch images:debian/12 c2 --storage remote --target incus-os2
Launching c2
% incus launch images:debian/12 c3 --storage remote --target incus-os2
Launching c3
Error: Failed instance creation: Failed creating instance from image: Unable to start cloning resource definition: json: cannot unmarshal object into Go value of type client.ApiCallError

It looks like each container takes 10 GB of memory, regardless which image it is based on (I used the same image).

% incus storage volume list remote -c cnU
+--------------+------------------------------------------------------------------+----------+
| CONTENT-TYPE |                               NAME                               |  USAGE   |
+--------------+------------------------------------------------------------------+----------+
| block        | 819f725c9c226a74ad38507878ae8840155cedd999961fb19e4649d5df187b41 |          |
+--------------+------------------------------------------------------------------+----------+
| block        | v1                                                               | 10.00GiB |
+--------------+------------------------------------------------------------------+----------+
| filesystem   | ada8dade02c14346ff5576d9b662676744328d0ad7fb903e2b9a28ada7a310d7 |          |
+--------------+------------------------------------------------------------------+----------+
| filesystem   | c1                                                               | 10.00GiB |
+--------------+------------------------------------------------------------------+----------+
| filesystem   | c2                                                               | 10.00GiB |
+--------------+------------------------------------------------------------------+----------+
| filesystem   | fsvol                                                            | 10.00GiB |
+--------------+------------------------------------------------------------------+----------+
| filesystem   | fsvol1                                                           | 10.00GiB |
+--------------+------------------------------------------------------------------+----------+

Is that so by design? Seems like a waste of memory to me.

stgraber · January 28, 2026, 7:27am

Yes, default volume size for block volumes is 10GiB.
You can change volume.size in the pool config to change the default.

ykazakov · January 28, 2026, 3:23pm

Thanks! I did not realize that linstor driver can be only used with (block) volumes of a fixed size. Reducing the value of volume.size did the job!

% incus storage create remote linstor linstor.resource_group.storage_pool=zfs_pool volume.size=1GiB
root@linstor-controller1:~# linstor storage-pool list  
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    61.36 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    62.90 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    63.92 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    63.19 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
% incus launch images:debian/13 c1 --storage remote --target incus-os1                       
Launching c1
% incus launch images:debian/13 c2 --storage remote --target incus-os1
Launching c2
% incus exec c1 -- df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1001   974M  410M  497M  46% /
root@linstor-controller1:~# linstor storage-pool list  
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    61.36 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    62.90 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    60.14 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    59.41 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

It is strange, however, that the volumes are placed not on the nodes on which the containers are running (the free capacity of zfs_pool on incus-os1 did not change).

I also noticed that deleting containers and even the whole storage pool does not seem to reclaim the free space:

% incus rm --force c1 c2
% incus storage show remote
...
used_by:
- /1.0/images/64920d55360957aa79cf7ba6e556807682e1b2c79f68c7b7c6de22c0fa520eee?project=internal
% incus image rm 64920d553609
% incus storage rm remote
Storage pool remote deleted
root@linstor-controller1:~# linstor storage-pool list  
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS      ┊ local/linstor ┊    61.36 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS      ┊ local/linstor ┊    62.90 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS      ┊ local/linstor ┊    60.14 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS      ┊ local/linstor ┊    59.41 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@linstor-controller1:~# linstor storage-pool delete incus-os4 zfs_pool
SUCCESS:
Description:
    Node: incus-os4, Storage pool name: zfs_pool deleted.
Details:
    Node: incus-os4, Storage pool name: zfs_pool UUID was: a79c62e8-211d-476a-8952-50065f32d311
% incus admin os system storage delete-volume -d '{"pool":"local","name":"linstor"}' --target incus-os4                
WARNING: The IncusOS API and configuration is subject to change

Are you sure you want to delete the storage volume? (yes/no) [default=no]: yes
Error: Failed to run: zfs destroy local/linstor: exit status 1 (cannot destroy 'local/linstor': filesystem has children
use '-r' to destroy the following datasets:
local/linstor/_deleted_incus-volume-e40ab3bdf3bb4cde8568746981df3b47_00000_2026-01-28T15-01-49-069@CF_b613b48f
local/linstor/_deleted_incus-volume-e40ab3bdf3bb4cde8568746981df3b47_00000_2026-01-28T15-01-49-069
local/linstor/_deleted_incus-volume-7ae22c24243946bd8533db80a8fbc644_00000_2026-01-28T15-01-51-071@CF_8cea9842
local/linstor/_deleted_incus-volume-7ae22c24243946bd8533db80a8fbc644_00000_2026-01-28T15-01-51-071
local/linstor/_deleted_incus-volume-26f71efb494244d699e102a633eddb47_00000_2026-01-28T15-02-35-647@CF_b613b48f
local/linstor/_deleted_incus-volume-26f71efb494244d699e102a633eddb47_00000_2026-01-28T15-02-35-647@CF_8cea9842
local/linstor/_deleted_incus-volume-26f71efb494244d699e102a633eddb47_00000_2026-01-28T15-02-35-647)

Does Incus cleanup these volumes local/linstor/_deleted_incus-volume-***?

I tried to force the scrub, but these did not remove the deleted volumes.

% incus admin os system storage scrub-pool -d '{"name":"local"}' --target incus-os4

I could only remove them by adding the setting "force": true

% incus admin os system storage delete-volume -d '{"pool":"local","name":"linstor","force":true}' --target incus-os4

ykazakov · January 29, 2026, 3:56pm

I realized that in order to use storage more economically, i.e., allocate only as much as is actually used, I should have used the zfsthin driver:

linstor storage-pool create zfsthin incus-os1 zfs_pool local/linstor
...
% incus launch images:debian/13 c1 --storage remote --target incus-os1           
Launching c1
~ % incus exec c1 -- df -h /                                             
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1001   9.8G  412M  8.9G   5% /
root@linstor-controller1:~# linstor storage-pool list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node      ┊ Driver   ┊ PoolName      ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                     ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ incus-os1 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os2 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os2;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os3 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os3;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ incus-os4 ┊ DISKLESS ┊               ┊              ┊               ┊ False        ┊ Ok    ┊ incus-os4;DfltDisklessStorPool ┊
┊ zfs_pool             ┊ incus-os1 ┊ ZFS_THIN ┊ local/linstor ┊    60.96 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os1;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os2 ┊ ZFS_THIN ┊ local/linstor ┊    62.38 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os2;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os3 ┊ ZFS_THIN ┊ local/linstor ┊    62.65 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os3;zfs_pool             ┊
┊ zfs_pool             ┊ incus-os4 ┊ ZFS_THIN ┊ local/linstor ┊    62.70 GiB ┊     66.50 GiB ┊ True         ┊ Ok    ┊ incus-os4;zfs_pool             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Still, after deleting the pool, there appear to be some leftover volumes hanging around:

root@linstor-controller1:~# linstor resource-definition list --show-props Aux/Incus/name Aux/Incus/type Aux/Incus/content-type
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                                  ┊ ResourceGroup ┊ Layers       ┊ State ┊ Aux/Incus/name                                                                ┊ Aux/Incus/type ┊ Aux/Incus/content-type ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-17a5da32a5e44e629fd6e9df7a23089c ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-1d6d44ec70dae04eaf674bdc5fd9011cd43004d5834b629669e01ce321842619 ┊ images         ┊ filesystem             ┊
┊ incus-volume-c16e321b862049329de25c2ba6b01df7 ┊ remote        ┊ DRBD,STORAGE ┊ ok    ┊ incus-volume-internal_c1                                                      ┊ containers     ┊ filesystem             ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
~ % incus rm c1 --force                                                             
~ % incus image rm 1d6d44ec70dae04eaf674bdc5fd9011cd43004d5834b629669e01ce321842619
~ % incus storage remove remote
Storage pool remote deleted

root@linstor-controller1:~# linstor storage-pool delete incus-os1 zfs_pool
root@linstor-controller1:~# linstor storage-pool delete incus-os2 zfs_pool
root@linstor-controller1:~# linstor storage-pool delete incus-os3 zfs_pool
root@linstor-controller1:~# linstor storage-pool delete incus-os4 zfs_pool

~ % incus admin os system storage delete-volume -d '{"pool":"local","name":"linstor"}' --target incus-os1 
WARNING: The IncusOS API and configuration is subject to change

Are you sure you want to delete the storage volume? (yes/no) [default=no]: yes
~ % incus admin os system storage delete-volume -d '{"pool":"local","name":"linstor"}' --target incus-os2
WARNING: The IncusOS API and configuration is subject to change

Are you sure you want to delete the storage volume? (yes/no) [default=no]: yes
Error: Failed to run: zfs destroy local/linstor: exit status 1 (cannot destroy 'local/linstor': filesystem has children
use '-r' to destroy the following datasets:
local/linstor/_deleted_incus-volume-17a5da32a5e44e629fd6e9df7a23089c_00000_2026-01-29T15-46-31-324@CF_391051bd
local/linstor/_deleted_incus-volume-17a5da32a5e44e629fd6e9df7a23089c_00000_2026-01-29T15-46-31-324
local/linstor/_deleted_incus-volume-c16e321b862049329de25c2ba6b01df7_00000_2026-01-29T15-46-17-760@CF_391051bd
local/linstor/_deleted_incus-volume-c16e321b862049329de25c2ba6b01df7_00000_2026-01-29T15-46-17-760)
~ % incus admin os system storage delete-volume -d '{"pool":"local","name":"linstor"}' --target incus-os3
WARNING: The IncusOS API and configuration is subject to change

Are you sure you want to delete the storage volume? (yes/no) [default=no]: yes
~ % incus admin os system storage delete-volume -d '{"pool":"local","name":"linstor"}' --target incus-os4
WARNING: The IncusOS API and configuration is subject to change

Are you sure you want to delete the storage volume? (yes/no) [default=no]: yes
Error: Failed to run: zfs destroy local/linstor: exit status 1 (cannot destroy 'local/linstor': filesystem has children
use '-r' to destroy the following datasets:
local/linstor/_deleted_incus-volume-c16e321b862049329de25c2ba6b01df7_00000_2026-01-29T15-46-17-761@CF_391051bd
local/linstor/_deleted_incus-volume-c16e321b862049329de25c2ba6b01df7_00000_2026-01-29T15-46-17-761
local/linstor/_deleted_incus-volume-17a5da32a5e44e629fd6e9df7a23089c_00000_2026-01-29T15-46-31-325@CF_391051bd
local/linstor/_deleted_incus-volume-17a5da32a5e44e629fd6e9df7a23089c_00000_2026-01-29T15-46-31-325)