Incus ignoring storage.backups_volume after migration from LXD

Incus
1

I migrated from LXD to Incus 6.0.0 on Ubuntu (specifically 6.0.0-1ubuntu0.2). I run occasional storage volume exports from cron like so:

/usr/bin/incus storage volume export pool1 foo_data_1 /data/backup/incus/20241122/foo_data_1-backup-11-22-2024.tar.zst --compression=zstd --optimized-storage --volume-only

Under LXD I set storage.backups_volume to avoid the export filling up the relatively small /var/lib partition during the backup. I confirmed this config migrated over to Incus:

root@fusion:~# incus config show
config:
  backups.compression_algorithm: none
  core.metrics_address: :8443
  storage.backups_volume: pool1/backupscratch

This worked fine under LXD. Since the migration, my backups have been failing for the larger volumes. The error makes it quite clear that it’s using /var/lib/incus and not the backups_volume:

Error: Create volume backup: Backup create: Failed to copy file content "/var/lib/incus/backups/incus_backup_zfs2760447516": io: read/write on closed pipe

To verify this, I ran an export in real time and watched the incus directory. Sure enough, a file is generated there each time. Nothing about the pool has changed, so I can’t explain why it’s choosing to use /var/lib.

root@fusion:~# incus storage list
+-------+--------+----------------+-------------+---------+---------+
| NAME  | DRIVER |     SOURCE     | DESCRIPTION | USED BY |  STATE  |
+-------+--------+----------------+-------------+---------+---------+
| pool1 | zfs    | rpool/DATA/lxd |             | 57      | CREATED |

Any help would be appreciated.

2

Can you restart Incus and then show cat /proc/self/mountinfo?
Showing zfs list -t all may also be helpful.

3

Oh and ls -lh /var/lib/incus as well.

4

Well, you set me on the right path. /var/lib/incus/backups was not symlinked to the intended volume. It was a standard directory with “custom” and “instances” subfolders.

I was able to resolve things by creating a new volume from scratch and changing the config variable, which then resulted in the backups directory becoming a symlink to the correct (new) location.

I did attempt to re-set things to the original volume after that. It complained that the volume was not empty. I’m not sure if this factored into the migration failing or not? I also don’t know why it wasn’t empty, but I would guess some backup files had gotten orphaned there over the life of the server and just never cleaned up.