I"m using incus on Truenas 25.04. I set my storage pool for incus to /mnt/cache which is a zfs mirror. I was able to create an run several lxc containers and am attempting to run a vm.
I just discovered incus-migrate and I’m attempting to use it to migrate an unraid vm. I was scared of what incus-migrate may do so I created a dataset on a new zfs pool. The pool is temp and the dataset was called BlueIris. So I use incus storage create to create a new pool at temp/BlueIris. That seemed to work.
Now I see a bunch of datasets inside temp/BlueIris. I see ‘buckets’, ‘containers’, ‘custom’, ‘deleted’, ‘images’, and ‘virtual-machines’.
My question is this: why can’t I see those datasets from the terminal? Also, since is set cache to be the default pool for incus, why do I not see a similar structure there? I don’t see anything at all.
Could you do a incus storage list? It’s possible that you created a storage pool using the dir driver instead of the zfs one.
root@vault[/mnt/cache]# incus storage list
±-------------±-------±------------±--------±--------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
±-------------±-------±------------±--------±--------+
| BlueIrisPool | zfs | | 1 | CREATED |
±-------------±-------±------------±--------±--------+
| cache | zfs | | 6 | CREATED |
±-------------±-------±------------±--------±--------+
Here is that same storage pool as seen from TN dataset view.
What is the output of incus storage show cache and incus storage show BlueIrisPool?
And could you paste the output of zfs list?
root@vault[/data]# incus storage show cache
config:
source: cache/.ix-virt
zfs.pool_name: cache/.ix-virt
description: “”
name: cache
driver: zfs
used_by:
- /1.0/instances/BlueIris2
- /1.0/instances/ark
- /1.0/storage-pools/cache/volumes/custom/BlueIris
- /1.0/storage-pools/cache/volumes/custom/Test
- /1.0/storage-pools/cache/volumes/custom/Win11_24H2_English_x64.iso
- /1.0/storage-pools/cache/volumes/custom/virtio-win-0.1.266.iso
status: Created
locations:
- none
root@vault[/data]# incus storage show BlueIrisPool
config:
source: temp/BlueIris
volatile.initial_source: temp/BlueIris
zfs.pool_name: temp/BlueIris
description: “”
name: BlueIrisPool
driver: zfs
used_by:
- /1.0/instances/BlueIris
status: Created
locations:
- none
root@vault[/data]#
root@vault[/data]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
Backup 1.82T 1.69T 96K /mnt/Backup
Backup/Duplicacy 1.82T 1.69T 1.82T /mnt/Backup/Duplicacy
boot-pool 5.68G 219G 96K none
boot-pool/ROOT 5.66G 219G 96K none
boot-pool/ROOT/25.04.0 2.91G 219G 174M legacy
boot-pool/ROOT/25.04.0/audit 5.96M 219G 5.96M /audit
boot-pool/ROOT/25.04.0/conf 6.97M 219G 6.97M /conf
boot-pool/ROOT/25.04.0/data 952K 219G 952K /data
boot-pool/ROOT/25.04.0/etc 7.21M 219G 6.18M /etc
boot-pool/ROOT/25.04.0/home 116K 219G 116K /home
boot-pool/ROOT/25.04.0/mnt 17.0M 219G 17.0M /mnt
boot-pool/ROOT/25.04.0/opt 96K 219G 96K /opt
boot-pool/ROOT/25.04.0/root 164K 219G 164K /root
boot-pool/ROOT/25.04.0/usr 2.51G 219G 2.51G /usr
boot-pool/ROOT/25.04.0/var 193M 219G 5.12M /var
boot-pool/ROOT/25.04.0/var/ca-certificates 96K 219G 96K /var/local/ca-certificates
boot-pool/ROOT/25.04.0/var/lib 148M 219G 27.4M /var/lib
boot-pool/ROOT/25.04.0/var/lib/incus 120M 219G 120M /var/lib/incus
boot-pool/ROOT/25.04.0/var/log 39.2M 219G 10.2M /var/log
boot-pool/ROOT/25.04.0/var/log/journal 29.0M 219G 29.0M /var/log/journal
boot-pool/ROOT/25.04.1 2.75G 219G 174M legacy
boot-pool/ROOT/25.04.1/audit 584K 219G 584K /audit
boot-pool/ROOT/25.04.1/conf 7.06M 219G 7.06M /conf
boot-pool/ROOT/25.04.1/data 928K 219G 928K /data
boot-pool/ROOT/25.04.1/etc 7.11M 219G 6.18M /etc
boot-pool/ROOT/25.04.1/home 116K 219G 116K /home
boot-pool/ROOT/25.04.1/mnt 128K 219G 128K /mnt
boot-pool/ROOT/25.04.1/opt 96K 219G 96K /opt
boot-pool/ROOT/25.04.1/root 156K 219G 156K /root
boot-pool/ROOT/25.04.1/usr 2.53G 219G 2.53G /usr
boot-pool/ROOT/25.04.1/var 34.7M 219G 4.31M /var
boot-pool/ROOT/25.04.1/var/ca-certificates 96K 219G 96K /var/local/ca-certificates
boot-pool/ROOT/25.04.1/var/lib 27.8M 219G 26.9M /var/lib
boot-pool/ROOT/25.04.1/var/lib/incus 692K 219G 692K /var/lib/incus
boot-pool/ROOT/25.04.1/var/log 2.39M 219G 1.02M /var/log
boot-pool/ROOT/25.04.1/var/log/journal 1.37M 219G 1.37M /var/log/journal
boot-pool/grub 8.41M 219G 8.41M legacy
cache 684G 1.09T 182G /mnt/cache
cache/.ix-virt 129G 1.09T 96K legacy
cache/.ix-virt/buckets 96K 1.09T 96K legacy
cache/.ix-virt/containers 20.3G 1.09T 96K legacy
cache/.ix-virt/containers/ark 20.3G 1.09T 20.9G legacy
cache/.ix-virt/custom 108G 1.09T 96K legacy
cache/.ix-virt/custom/default_BlueIris 102G 1.19T 56K -
cache/.ix-virt/custom/default_Test 56K 1.09T 56K -
cache/.ix-virt/custom/default_Win11_24H2_English_x64.iso.iso 5.45G 1.09T 5.45G -
cache/.ix-virt/custom/default_virtio-win-0.1.266.iso.iso 695M 1.09T 695M -
cache/.ix-virt/deleted 541M 1.09T 96K legacy
cache/.ix-virt/deleted/buckets 96K 1.09T 96K legacy
cache/.ix-virt/deleted/containers 96K 1.09T 96K legacy
cache/.ix-virt/deleted/custom 96K 1.09T 96K legacy
cache/.ix-virt/deleted/images 540M 1.09T 96K legacy
cache/.ix-virt/deleted/images/c6294bd147a98c6a988964c34b62543951c4d6b685590afae117bff46c627345 540M 1.09T 540M legacy
cache/.ix-virt/deleted/virtual-machines 96K 1.09T 96K legacy
cache/.ix-virt/images 540M 1.09T 96K legacy
cache/.ix-virt/images/2bcb42037af28fe673937917c40d121d45c507dbc3bfdc171113de599c4f0755 540M 1.09T 540M legacy
cache/.ix-virt/virtual-machines 13.4M 1.09T 96K legacy
cache/.ix-virt/virtual-machines/BlueIris2 13.2M 487M 13.2M legacy
cache/.ix-virt/virtual-machines/BlueIris2.block 56K 1.09T 56K -
cache/.system 1.86G 1.09T 1.55G legacy
cache/.system/configs-ae32c386e13840b2bf9c0083275e7941 6.30M 1.09T 6.30M legacy
cache/.system/cores 120K 1024M 120K legacy
cache/.system/netdata-ae32c386e13840b2bf9c0083275e7941 311M 1.09T 311M legacy
cache/.system/nfs 112K 1.09T 112K legacy
cache/.system/samba4 3.20M 1.09T 3.20M legacy
cache/appdata 278G 1.09T 278G /mnt/cache/appdata
cache/apps 81.1G 1.09T 81.1G /mnt/cache/apps
cache/apps/portainer 404K 1.09T 404K /mnt/cache/apps/portainer
cache/ark 907M 1.09T 907M /mnt/cache/ark
cache/ix-apps 11.1G 1.09T 180K /mnt/.ix-apps
cache/ix-apps/app_configs 2.79M 1.09T 2.78M /mnt/.ix-apps/app_configs
cache/ix-apps/app_mounts 96K 1.09T 96K /mnt/.ix-apps/app_mounts
cache/ix-apps/docker 10.9G 1.09T 10.9G /mnt/.ix-apps/docker
cache/ix-apps/truenas_catalog 256M 1.09T 256M /mnt/.ix-apps/truenas_catalog
cache/users 280K 1.09T 280K /mnt/cache/users
tank 488G 13.5T 488G /mnt/tank
tank/immich 1.10M 13.5T 151K /mnt/tank/immich
tank/immich/backups 140K 13.5T 140K /mnt/tank/immich/backups
tank/immich/library 140K 13.5T 140K /mnt/tank/immich/library
tank/immich/pgData 140K 13.5T 140K /mnt/tank/immich/pgData
tank/immich/profile 140K 13.5T 140K /mnt/tank/immich/profile
tank/immich/thumbs 140K 13.5T 140K /mnt/tank/immich/thumbs
tank/immich/upload 140K 13.5T 140K /mnt/tank/immich/upload
tank/immich/video 140K 13.5T 140K /mnt/tank/immich/video
temp 42.6G 7.10T 96K /mnt/temp
temp/BlueIris 42.6G 7.10T 96K legacy
temp/BlueIris/buckets 96K 7.10T 96K legacy
temp/BlueIris/containers 96K 7.10T 96K legacy
temp/BlueIris/custom 96K 7.10T 96K legacy
temp/BlueIris/deleted 576K 7.10T 96K legacy
temp/BlueIris/deleted/buckets 96K 7.10T 96K legacy
temp/BlueIris/deleted/containers 96K 7.10T 96K legacy
temp/BlueIris/deleted/custom 96K 7.10T 96K legacy
temp/BlueIris/deleted/images 96K 7.10T 96K legacy
temp/BlueIris/deleted/virtual-machines 96K 7.10T 96K legacy
temp/BlueIris/images 96K 7.10T 96K legacy
temp/BlueIris/virtual-machines 42.6G 7.10T 96K legacy
temp/BlueIris/virtual-machines/BlueIris 7.73M 492M 7.73M legacy
temp/BlueIris/virtual-machines/BlueIris.block 42.6G 7.10T 42.6G -
vault 18.9T 5.67T 174K /mnt/vault
vault/media 18.4T 5.67T 18.4T /mnt/vault/media
vault/misc 496G 5.67T 496G /mnt/vault/misc
root@vault[/data]#
Was cache created from TrueNAS’s interface? If so, then I’m not familiar with what iX Systems does to initialize the pool. But at the very least, we can see that it does exist in a hidden directory under /mnt/cache/.ix-virt
EDIT: scratch that third sentence, had a brain fart.
the cache storage pool was created with TN ui, yes.
Yeah, I think I understand why I’m not seeing cache now. Thanks for replying. If you see anything “weird” with what I pasted please let me know.
Sorry about that second part in my earlier comment.
So when Incus creates a storage pool with the ZFS driver, it doesn’t actually mount the datasets as noted by the legacy mounpoint. This means that Incus is leveraging the mount syscalls manually to mount the directory.
As an example, here is my zfs list:
NAME USED AVAIL REFER MOUNTPOINT
shuttle 2.49G 424G 320K /shuttle
shuttle/incus 1.94G 424G 320K legacy
shuttle/incus/buckets 320K 424G 320K legacy
shuttle/incus/containers 544K 424G 320K legacy
shuttle/incus/containers/workstations_debian13 224K 424G 415M legacy
shuttle/incus/custom 320K 424G 320K legacy
shuttle/incus/deleted 1.88M 424G 320K legacy
shuttle/incus/deleted/buckets 320K 424G 320K legacy
shuttle/incus/deleted/containers 320K 424G 320K legacy
shuttle/incus/deleted/custom 320K 424G 320K legacy
shuttle/incus/deleted/images 320K 424G 320K legacy
shuttle/incus/deleted/virtual-machines 320K 424G 320K legacy
shuttle/incus/images 1.94G 424G 320K legacy
shuttle/incus/images/3481efd7cb3c85fd2215557165093d43f31cceca0cf497b630ba7b24af44b522 302M 424G 302M legacy
shuttle/incus/images/a3c2133137f0b6321c4d52acd3c2cfc2b69d8fcc1d92eac48d27a524276006d8 415M 424G 415M legacy
shuttle/incus/images/d78cf18c8976d3348a7953698182e3b31f31d5c6c93e61a6e6e6af66b568b734 374M 424G 374M legacy
shuttle/incus/images/eb7ec38152a187619b1004f15f922a44cbbc11e46c6443bd119a6bd74854877f 268M 424G 268M legacy
shuttle/incus/images/fb0d71e5d06d57fa6962629023d9b285a42e2a7e9aecab918030939dd8645957 374M 424G 374M legacy
shuttle/incus/images/fd74e4363232c197b36eba71adb9260bda10c88eec3793e1a809b7682b6c06bd 252M 424G 252M legacy
shuttle/incus/virtual-machines 320K 424G 320K legacy
shuttle/var 556M 424G 320K none
shuttle/var/lib 556M 424G 320K none
shuttle/var/lib/incus 556M 424G 556M /var/lib/incus
yeah, looks basically the same. I created the BI one just as a test since I didn’t know what incus-migrate would do.
If you’re interested you can also take a look here for more documentation as to what the legacy mountpoint does: zfsprops.7 — OpenZFS documentation
You can also see where Incus is mounting the datasets using mount | grep, in my case, mount | grep shuttle which lists (nothing of note is actually listed because no instances are running):
shuttle/var/lib/incus on /var/lib/incus type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
shuttle on /shuttle type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
EDIT: with an instance running it shows:
shuttle/var/lib/incus on /var/lib/incus type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
shuttle on /shuttle type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
shuttle/incus/containers/workstations_moving-bison on /var/lib/incus/storage-pools/shuttle/containers/workstations_moving-bison type zfs (rw,relatime,seclabel,xattr,posixacl,casesensitive)
