Cephfs storage pool: "Storage pool does not support instance type" when trying to create new instance

Incus
,
1

Abstract: I’m trying to run a proof of concept ceph cluster (I’m not a ceph expert) and I’ve added a cephfs storage pool to my local incus, but when I try to start a instance using that storage, I get the error Error: Failed creating instance record: Failed initializing instance: Storage pool does not support instance type

Context:

  • We have currently use incus and are looking to start making use of ceph for distributed storage. Before we commit, we’re trying to run a proof-of-concept to better understand key things about ceph and its integration with incus
  • We have taken some old machine and deployed a 3 machine ceph cluster (only ceph running on it)
    • cephadm was used for the deployment and configuration
    • a cephfs pool has been created in our cluster, called incus-test
  • As a future ceph cluster should not be under the full control of incus (there will be other users), we do not wish to expose the incus cluster to the “admin” keyring
  • To test incus use with the ceph cluster, I am attempting to configure my local workstation’s incus to make use of the remote cephfs.
    • local incus version is 6.5

With that, I setup a client in ceph:

# (from a ceph shell in the cluster)
ceph fs authorize incus-test client.incus-local / rw

I put the resulting configuration onto my local workstation under /etc/ceph/ceph.client.incus-local.keyring, along with a minimal ceph.conf generated by ceph config generate-minimal-conf.

I can verify this filesystem and access exists, as I’m able to mount it on my workstation.

I then added the filesystem as a storage pool in incus:

incus storage create ceph-test cephfs source=incus-test cephfs.user.name=incus-local ceph

which responds with Storage pool ceph-test created. (I assume that) I can confirm that the storage pool is connected, because:

  • there’s a client mentioned in the ceph dashboard
  • incus storage info ceph-test correctly lists the total space of the filesystem
  • I can see a mount entry for the ceph cluster ips onto /var/lib/incus/storage-pools/ceph-test with type ceph

However, when I then try to launch an instance, using that storage pool, I get an error:

$ incus launch jammy fully-ceph-test --storage ceph-test
Error: Failed creating instance record: Failed initializing instance: Storage pool does not support instance type

Note: jammy is an alias of a “Ubuntu jammy amd64 (cloud)” container image

I’m finding it difficult to figure out what knowledge I’m lacking here:

  • I cannot find any references to that error message, except for the code
  • I haven’t seen anything to suggest there’s any storage driver that does not support a given instance type.
  • I cannot determine if this is an error related to permissions on the filesystem, as the only mention I’ve seen of not using the admin.keyring for your incus ↔ ceph configuration is this issue related to 6.10

I’m fairly certain this is some user error, but I’ll need some guidance on where I’ve gone wrong or what I’m missing.

Thank in advance!

2

I’ve become aware of the additional p and s capability flags that can be added to a cephfs client definition:

Thinking that incus might need these additional permissions to make proper use of cephfs, I recreated a new client with that permission, deleted the existing incus storage pool (incus storage delete ceph-test) and recreated it with the new client permissions, but got the same result.

3

See CephFS - cephfs - Incus documentation

It clearly states The cephfs driver can only be used for custom storage volumes with content type filesystem

Block custom storage volumes, images or instances cannot be stored on cephfs, you need to use rbd for that (ceph driver).

4

Ah, I was unaware of the distinction between a “container/virtual machine” storage type and a “custom” storage type (defined here: About storage pools, volumes and buckets - Incus documentation)

I thought the only distinction was between content types (block vs filesystem vs iso).

Thanks for your help!