Could not make resource available on node: 404 Not Found

Incus
1

Hi,
I have recently configure a linstor cluster and test with my incus server, I’m not using incus cluster just one host.
Here are my settings:

indiana@incusrv01:~$ incus storage ls
+---------+---------+-------------+---------+---------+
|  NAME   | DRIVER  | DESCRIPTION | USED BY |  STATE  |
+---------+---------+-------------+---------+---------+
| default | zfs     |             | 10      | CREATED |
+---------+---------+-------------+---------+---------+
| remote  | linstor |             | 0       | CREATED |
+---------+---------+-------------+---------+---------+
indiana@incusrv01:~$
indiana@incusrv01:~$
indiana@incusrv01:~$ incus launch images:ubuntu/24.04 c1 --storage remote
Launching c1
Error: Failed instance creation: Failed creating instance from image: Could not get device path for filesystem creation: Could not make resource available on node: 404 Not Found

indiana@incusrv01:~$ incus storage show remote
config:
  drbd.auto_add_quorum_tiebreaker: "true"
  drbd.on_no_quorum: suspend-io
  linstor.resource_group.name: remote
  linstor.resource_group.place_count: "2"
  linstor.resource_group.storage_pool: nvmepool
  linstor.volume.prefix: incus-volume-
  source: remote
  volatile.pool.pristine: "true"
description: ""
name: remote
driver: linstor
used_by: []
status: Created
locations:
- none

Any suggestion?
Regards.

2

How did you install LINSTOR? Can I see some info about the nodes, resource groups and storage pools? (linstor -m n l, linstor -m rg l and linstor -m sp l)

3

Sure, My system is Ubuntu24.04 and use linbit-ubuntu-linbit-drbd9-stack-noble.sources.
Here are my settings:

indiana@linstor01:~$ linstor -m n l
[
  [
    {
      "name": "linstor01",
      "type": "COMBINED",
      "props": {
        "CurStltConnName": "default",
        "NodeUname": "linstor01"
      },
      "net_interfaces": [
        {
          "name": "default",
          "address": "192.168.1.101",
          "satellite_port": 3366,
          "satellite_encryption_type": "PLAIN",
          "is_active": true,
          "uuid": "c5511036-949b-4d94-b344-16e372909024"
        }
      ],
      "connection_status": "ONLINE",
      "uuid": "41038b55-e626-4e7e-9978-8c050d963a21",
      "storage_providers": [
        "DISKLESS",
        "LVM",
        "LVM_THIN",
        "ZFS",
        "ZFS_THIN",
        "FILE",
        "FILE_THIN",
        "REMOTE_SPDK",
        "EBS_INIT",
        "EBS_TARGET"
      ],
      "resource_layers": [
        "DRBD",
        "LUKS",
        "NVME",
        "WRITECACHE",
        "CACHE",
        "BCACHE",
        "STORAGE"
      ],
      "unsupported_providers": {
        "SPDK": [
          "IO exception occured when running 'rpc.py spdk_get_version': Cannot run program \"rpc.py\": error=2, No such file or directory"
        ],
        "STORAGE_SPACES": [
          "This tool does not exist on the Linux platform."
        ],
        "EXOS": [
          "'/bin/bash -c 'cat /sys/class/sas_phy/*/sas_address'' returned with exit code 1",
          "'/bin/bash -c 'cat /sys/class/sas_device/end_device-*/sas_address'' returned with exit code 1"
        ],
        "STORAGE_SPACES_THIN": [
          "This tool does not exist on the Linux platform."
        ]
      }
    },
    {
      "name": "linstor02",
      "type": "SATELLITE",
      "props": {
        "CurStltConnName": "default",
        "NodeUname": "linstor02"
      },
      "net_interfaces": [
        {
          "name": "default",
          "address": "192.168.1.102",
          "satellite_port": 3366,
          "satellite_encryption_type": "PLAIN",
          "is_active": true,
          "uuid": "0806a70b-55ee-4c5e-b4d2-8891b1826910"
        }
      ],
      "connection_status": "ONLINE",
      "uuid": "de8377f6-315b-4e42-989f-48a5ae30b85f",
      "storage_providers": [
        "DISKLESS",
        "LVM",
        "LVM_THIN",
        "ZFS",
        "ZFS_THIN",
        "FILE",
        "FILE_THIN",
        "REMOTE_SPDK",
        "EBS_INIT",
        "EBS_TARGET"
      ],
      "resource_layers": [
        "DRBD",
        "LUKS",
        "NVME",
        "WRITECACHE",
        "CACHE",
        "BCACHE",
        "STORAGE"
      ],
      "unsupported_providers": {
        "SPDK": [
          "IO exception occured when running 'rpc.py spdk_get_version': Cannot run program \"rpc.py\": error=2, No such file or directory"
        ],
        "STORAGE_SPACES": [
          "This tool does not exist on the Linux platform."
        ],
        "EXOS": [
          "'/bin/bash -c 'cat /sys/class/sas_phy/*/sas_address'' returned with exit code 1",
          "'/bin/bash -c 'cat /sys/class/sas_device/end_device-*/sas_address'' returned with exit code 1"
        ],
        "STORAGE_SPACES_THIN": [
          "This tool does not exist on the Linux platform."
        ]
      }
    },
    {
      "name": "linstor03",
      "type": "SATELLITE",
      "props": {
        "CurStltConnName": "default",
        "NodeUname": "linstor03"
      },
      "net_interfaces": [
        {
          "name": "default",
          "address": "192.168.1.103",
          "satellite_port": 3366,
          "satellite_encryption_type": "PLAIN",
          "is_active": true,
          "uuid": "910564cb-b04e-46a4-96c3-5d48c809b632"
        }
      ],
      "connection_status": "ONLINE",
      "uuid": "ee61ad36-172e-4517-9d40-35e94403281c",
      "storage_providers": [
        "DISKLESS",
        "LVM",
        "LVM_THIN",
        "ZFS",
        "ZFS_THIN",
        "FILE",
        "FILE_THIN",
        "REMOTE_SPDK",
        "EBS_INIT",
        "EBS_TARGET"
      ],
      "resource_layers": [
        "DRBD",
        "LUKS",
        "NVME",
        "WRITECACHE",
        "CACHE",
        "BCACHE",
        "STORAGE"
      ],
      "unsupported_providers": {
        "SPDK": [
          "IO exception occured when running 'rpc.py spdk_get_version': Cannot run program \"rpc.py\": error=2, No such file or directory"
        ],
        "STORAGE_SPACES": [
          "This tool does not exist on the Linux platform."
        ],
        "EXOS": [
          "'/bin/bash -c 'cat /sys/class/sas_phy/*/sas_address'' returned with exit code 1",
          "'/bin/bash -c 'cat /sys/class/sas_device/end_device-*/sas_address'' returned with exit code 1"
        ],
        "STORAGE_SPACES_THIN": [
          "This tool does not exist on the Linux platform."
        ]
      }
    }
  ]
]

indiana@linstor01:~$ linstor -m rg l
[
  [
    {
      "name": "DfltRscGrp",
      "select_filter": {
        "place_count": 2
      },
      "uuid": "a52e934a-9fd9-44cb-9db1-716dcd13aae3"
    },
    {
      "name": "remote",
      "description": "Resource group managed by Incus",
      "props": {
        "DrbdOptions/auto-add-quorum-tiebreaker": "true",
        "DrbdOptions/Resource/on-no-quorum": "suspend-io"
      },
      "select_filter": {
        "place_count": 2,
        "storage_pool": "nvmepool",
        "storage_pool_list": [
          "nvmepool"
        ]
      },
      "uuid": "935568b9-34bf-4529-8023-74e2dba32739"
    }
  ]
]

indiana@linstor01:~$ linstor -m sp l
[
  [
    {
      "storage_pool_name": "DfltDisklessStorPool",
      "node_name": "linstor01",
      "provider_kind": "DISKLESS",
      "static_traits": {
        "SupportsSnapshots": "false"
      },
      "free_capacity": 9223372036854775807,
      "total_capacity": 9223372036854775807,
      "free_space_mgr_name": "linstor01;DfltDisklessStorPool",
      "uuid": "4de7cfd9-0900-488a-9c5c-cbbe6d9166e4",
      "supports_snapshots": false,
      "external_locking": false
    },
    {
      "storage_pool_name": "DfltDisklessStorPool",
      "node_name": "linstor02",
      "provider_kind": "DISKLESS",
      "static_traits": {
        "SupportsSnapshots": "false"
      },
      "free_capacity": 9223372036854775807,
      "total_capacity": 9223372036854775807,
      "free_space_mgr_name": "linstor02;DfltDisklessStorPool",
      "uuid": "3fb056b8-53a0-48c7-816b-ce04ee481b2e",
      "supports_snapshots": false,
      "external_locking": false
    },
    {
      "storage_pool_name": "DfltDisklessStorPool",
      "node_name": "linstor03",
      "provider_kind": "DISKLESS",
      "static_traits": {
        "SupportsSnapshots": "false"
      },
      "free_capacity": 9223372036854775807,
      "total_capacity": 9223372036854775807,
      "free_space_mgr_name": "linstor03;DfltDisklessStorPool",
      "uuid": "b22d40a4-540b-45de-b2f7-3d3c4b6ec74c",
      "supports_snapshots": false,
      "external_locking": false
    },
    {
      "storage_pool_name": "nvmepool",
      "node_name": "linstor01",
      "provider_kind": "LVM_THIN",
      "props": {
        "StorDriver/internal/lvmthin/thinPoolMetadataPercent": "10.42",
        "StorDriver/StorPoolName": "linstor_nvmepool/nvmepool",
        "StorDriver/internal/AllocationGranularity": "4096",
        "StorDriver/internal/lvmthin/thinPoolGranularity": "128"
      },
      "static_traits": {
        "Provisioning": "Thin",
        "SupportsSnapshots": "true"
      },
      "free_capacity": 249810944,
      "total_capacity": 249810944,
      "free_space_mgr_name": "linstor01;nvmepool",
      "uuid": "799257f0-1f45-425a-b0cb-7f55e39d6c41",
      "supports_snapshots": true,
      "external_locking": false
    },
    {
      "storage_pool_name": "nvmepool",
      "node_name": "linstor02",
      "provider_kind": "LVM_THIN",
      "props": {
        "StorDriver/internal/lvmthin/thinPoolMetadataPercent": "10.42",
        "StorDriver/StorPoolName": "linstor_nvmepool/nvmepool",
        "StorDriver/internal/AllocationGranularity": "4096",
        "StorDriver/internal/lvmthin/thinPoolGranularity": "128"
      },
      "static_traits": {
        "Provisioning": "Thin",
        "SupportsSnapshots": "true"
      },
      "free_capacity": 249810944,
      "total_capacity": 249810944,
      "free_space_mgr_name": "linstor02;nvmepool",
      "uuid": "abcc8b2a-d3fa-45bf-bf80-29bbf4afa223",
      "supports_snapshots": true,
      "external_locking": false
    },
    {
      "storage_pool_name": "nvmepool",
      "node_name": "linstor03",
      "provider_kind": "LVM_THIN",
      "props": {
        "StorDriver/internal/lvmthin/thinPoolMetadataPercent": "10.42",
        "StorDriver/StorPoolName": "linstor_nvmepool/nvmepool",
        "StorDriver/internal/AllocationGranularity": "4096",
        "StorDriver/internal/lvmthin/thinPoolGranularity": "128"
      },
      "static_traits": {
        "Provisioning": "Thin",
        "SupportsSnapshots": "true"
      },
      "free_capacity": 249810944,
      "total_capacity": 249810944,
      "free_space_mgr_name": "linstor03;nvmepool",
      "uuid": "1e75bede-07c7-4383-bf5a-cc4858c8f63a",
      "supports_snapshots": true,
      "external_locking": false
    }
  ]
]

Regards.

4

Nothing comes to my mind… Can I see the result of incus config show?

5

Sure, there it is,

indiana@incusrv01:~$ incus config show
config:
  core.https_address: :8443
  storage.linstor.controller_connection: http://192.168.1.101:3370
6

Great, now the problem is obvious to me :slight_smile:
Would you mind running incus config set storage.linstor.satellite.name=linstor01 and try again?
If it works, we’ll discuss a bit about your design. If it doesn’t, I’ll have to plug my brain :slight_smile:

7

Nah I guess it won’t work. To be able to use LINSTOR on an Incus node, you need a LINSTOR satellite installed on that node (even if this satellite has no data).

8

Something is happening right here strangely when I execute the following command,

indiana@incusrv01:~$ incus config unset storage.linstor.controller_connection
Error: Failed to connect to Linstor: Failed to create Linstor client: Get "http://localhost:3370/v1/controller/version": dial tcp 127.0.0.1:3370: connect: connection refused

May be it is related with it.

9

Humm, I have changed the config, now another error comes in.

indiana@incusrv01:~$ incus config show
config:
  core.https_address: :8443
  storage.linstor.controller_connection: http://192.168.1.101:3370
  storage.linstor.satellite.name: linstor01

indiana@incusrv01:~$ incus launch images:ubuntu/24.04/cloud c1 --storage remote
Launching c1
Error: Failed instance creation: Failed creating instance from image: Failed to run: mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0 /dev/drbd1000: exit status 1 (mke2fs 1.47.0 (5-Feb-2023)
The file /dev/drbd1000 does not exist and no size was specified.)
10

You are not supposed to unset the storage.linstor.controller_connection key if you don’t have a good reason to. Please read very carefully the installation guide that we wrote.

Regardless of the underlying storage setup, all Incus nodes should run the LINSTOR satellite service to be able to mount volumes on the node.

Basically, you tell Incus how to connect to a LINSTOR controller with storage.linstor.controller_connection (which is a string directly interpreted by LINSTOR), but you still need to mount your data locally. The way it’s done is through a local satellite, which needs to be installed on your Incus node. Then, if your local LINSTOR satellite has a different name from your Incus server (shouldn’t happen except with very specific use cases), you can override it with storage.linstor.satellite.name.

Soooo, please install a diskless satellite on your Incus server :slight_smile:

11

Thanks for the clarification but If I understand correctly, the configuration right here is enough for the linstor configuration right?

indiana@incusrv01:~$ incus config show
config:
  core.https_address: :8443
  storage.linstor.controller_connection: http://192.168.1.101:3370
  storage.linstor.satellite.name: linstor01
12

Actually you should drop the storage.linstor.satellite.name, sorry for that; I proposed it while still trying to understand your environment. Your initial setup is ok, you’re only missing a local satellite.

13

If this is the case, the service is running without any problem.

indiana@incusrv01:~$ sudo systemctl status linstor-satellite
● linstor-satellite.service - LINSTOR Satellite Service
     Loaded: loaded (/usr/lib/systemd/system/linstor-satellite.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-04-13 13:26:50 UTC; 5s ago
   Main PID: 3881 (java)
      Tasks: 37 (limit: 38378)
     Memory: 178.4M (peak: 188.4M)
        CPU: 5.430s
     CGroup: /system.slice/linstor-satellite.service
             ├─3881 java -Xms32M -Xmx2G -XX:+CrashOnOutOfMemoryError -classpath "/usr/share/linstor-server/lib/conf:/usr/share/linstor-server/lib/*" com.linbit.linstor.core.Satellite --logs=/var/log/linstor-satellite --config-directory=/e>
             └─3934 drbdsetup events2 all

Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.685 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'FileEventService' of type FileEventService
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.685 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'DrbdEventService-1' of type DrbdEventService
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.685 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'DrbdEventPublisher-1' of type DrbdEventPublisher
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.685 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'SnapshotShippingService' of type SnapshotShippingService
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.686 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'BackupShippingS3Service' of type BackupShippingS3Service
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.686 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'BackupShippingL2LService' of type BackupShippingL2LService
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.686 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'DeviceManager' of type DeviceManager
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.686 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - Starting service instance 'CloneService' of type CloneService
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.687 [DeviceManager] INFO  LINSTOR/Satellite/040715 SYSTEM - DeviceManager waiting for service-condition
Apr 13 13:26:54 incusrv01 Satellite[3881]: 2025-04-13 13:26:54.694 [Main] INFO  LINSTOR/Satellite/ff8be3 SYSTEM - NetComService started on port /0:0:0:0:0:0:0:0:3366
14

Your satellite must be added to your LINSTOR cluster. From your previous LINSTOR command output, it looks like it’s not the case.