Storage in LVM raw vs incus managed

1

Hello,

I was just trying to use LVM and thin-provising to increase the performance of my VMs and add extra flexibility, however I’ve a question about what incus storage volume create really does.

Considering I’ve this setup:

pvcreate /dev/nvme0n1p4
vgcreate vg-incus /dev/nvme0n1p4
lvcreate --thinpool incus-thinpool -L 190G vg-incus

incus storage create incus-lvm lvm \
  source=vg-incus \
  lvm.vg_name=vg-incus \
  lvm.thinpool_name=incus-thinpool \
  lvm.vg.force_reuse=true \
  lvm.use_thinpool=true

incus profile device set default-lvm root pool incus-lvm

incus launch images:debian/13 vm-lvm --profile default-lvm --vm -d root,size=10GB

If I want to add an extra data drive to my VM I can simply:

incus storage volume create incus-lvm vm-lvm-data size=20GB
incus config device add vm-lvm data disk source=vm-lvm-data pool=incus-lvm path=/mnt/data

However, this will show up inside the vm as filesystem incus_data.

Alternatively I might create a block device instead like this and format it inside the VM as ext4:

incus storage volume create incus-lvm vm-lvm-raw --type=block size=20GB
incus config device add vm-lvm fast disk source=vm-lvm-raw pool=incus-lvm

What is really happening in the first case? It is a virtiofs-backed disk? What about the second case?

My top priority is performance and I also need to be able run sqlite DBs properly from the data disk. Should I just opt for the second option? What are the catches?

I suppose the second option will always outperform whatever Incus is doing with virtiofs. My dd tests showed about the same performance.

Much appreciated.

2

First case is a LV that’s formatted and mounted by Incus on the host and then made available to your instances either through 9p/virtiofs (VMs) or bind-mount (containers). That’s going to be the best option if you want something you can share between multiple instances.

The second case is a unformatted LV created by Incus but then passed to the VM directly as an additional raw disk. The VM can format it and do whatever it wants with it, but this cannot safely be shared with other VMs (unless you’re using a clustered filesystem) and it cannot be exposed to containers at all.

1 Like
3

Thanks for the confirmation. So, in the second case I can expect native performance on the second case without potential virtiofs/fuse/9p shenanigans.

The second case is a unformatted LV created by Incus but then passed to the VM directly as an additional raw disk.

The way I did it, is it okay? Won’t it create issues with the Incus management of the LVM or the raw disk in some way?

Thanks.