Problems Creating after Full Backup Restore

Incus
,
1

Hi. My server had a semi-catastrophic failure yesterday (power outage). But I was able to salvage the data from the hard drive using the full backup instructions on this page incus/doc/backup.md at main · lxc/incus · GitHub

The basic steps were:

  • made a tarball of /var/lib/incus
  • backed up /etc/subuid and /etc/subgid
  • rebuilt the incus server on the same hardware
  • installed debian 13
  • installed incus from apt
  • stopped the incus services and deleted /var/lib/incus
  • restored /var/lib/incus from the tarball
  • restored /etc/subuid and /etc/subgid
  • restarted incus services

To my amazement, all my instances came back up! Yay!

But not all is well. I figured it was time for a backup strategy, so I tried incus export and I get this error:

depingus@server:~$ sudo incus export subversion subversion.tar.gz

Error: Create backup: Backup create: Failed to run: btrfs subvolume snapshot /var/lib/incus/storage-pools/default/containers/subversion /var/lib/incus/storage-pools/default/backup.2046340658/subversion: exit status 1 (ERROR: Not a Btrfs subvolume: Invalid argument)

After some useless flopping about, I tried to create a new instance with incus launch and got basically the same error:

depingus@server:~$ sudo incus launch images:debian/13 test --profile default

Launching test
Error: Failed instance creation: Failed creating instance from image: Failed to run: btrfs subvolume snapshot /var/lib/incus/storage-pools/default/images/5d2252f068069ee81cc0ece671f3a2cac09d9ba04d09b6b393c6641a3005f2e8 /var/lib/incus/storage-pools/default/containers/test: exit status 1 (ERROR: Not a Btrfs subvolume: Invalid argument)

I think the key error is ERROR: Not a Btrfs subvolume: Invalid argument. I feel like there’s a permissions issue with the storage pool on the new server, but I don’t know what to change.

I’ve got a very default install of Incus 6.0.4 running on Debian 13.2. My / directory is a btrfs filesystem. If anyone has any ideas, I would much appreciate the help.

2

Yeah, it sounds like you have a btrfs storage pool directly under /var/lib/incus/storage-pools/default, when you made a tarball of that, you lost all of the btrfs copy-on-write stuff, all the submodules, all the snapshots, …

To be able to create new instances again, it should be as simple as going through incus image list and deleting them all with incus image delete, then the images will be re-downloaded and turned back into btrfs subvolumes.

For the existing instances, that’s a bit trickier as their btrfs subvolume is gone.
One trick that may work could be to move them to a dir storage pool and then back to the btrfs storage pool.

Basically something like:

incus storage create tmp-dir dir
incus move subversion --storage tmp-dir
incus move subversion --storage default
incus storage delete tmp-dir

That may still cause some issues because of the lack of subvolume though depending on how the migration logic integrates with it.

3

Thanks for getting back to me. You are correct, about the btrfs storage pool. During the debian install, I changed / to btrfs. This is what my disks look like now after installing incus.

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 447.1G  0 disk 
├─sda1   8:1    0   976M  0 part /boot/efi
├─sda2   8:2    0 438.3G  0 part /var/lib/incus/storage-pools/default
│                                /
└─sda3   8:3    0   7.9G  0 part [SWAP]

Unfortunately, neither suggestions worked. Basically the same error.

I can’t delete existing images:

depingus@server:~$ sudo incus image delete 0612eb1528e6
Error: Error deleting image "0612eb1528e63ae5afbcc030da85262cf72e0ac4ecf73ffc26a27f8c8997ab54" from storage pool "default": Failed deleting subvolume "/var/lib/incus/storage-pools/default/images/0612eb1528e63ae5afbcc030da85262cf72e0ac4ecf73ffc26a27f8c8997ab54": Failed to run: btrfs subvolume delete /var/lib/incus/storage-pools/default/images/0612eb1528e63ae5afbcc030da85262cf72e0ac4ecf73ffc26a27f8c8997ab54: exit status 1 (ERROR: Not a Btrfs subvolume: Invalid argument)

And I can’t move the existing instances to a new storage:

depingus@server:~$ sudo incus storage create tmp-dir dir
Storage pool tmp-dir created
depingus@server:~$ sudo incus move subversion --storage tmp-dir
Error: Migration operation failure: Instance move to destination failed: Create instance from copy: Create instance volume from copy failed: Failed to run: btrfs subvolume snapshot /var/lib/incus/storage-pools/default/containers/subversion /var/lib/incus/storage-pools/default/backup.2691048933/subversion: exit status 1 (ERROR: Not a Btrfs subvolume: Invalid argument)

This gave me some ideas to test. Turns out I’m able to create new instances in new storage. And even do incus export on those new isntances. That gave me the idea to create a storage pool just for backups, and did incus config set storage.backups_volume to the new storage. But export just threw the same error as before. It can’t create the snapshot subvolume.

I still have the tarball I made of /var/lib/incus on the original server, but I wiped the drive when I rebuilt the server. So the btrfs copy-on-write stuff is gone.

4

Can you show:

incus admin sql global "SELECT * FROM storage_pools"
incus admin sql global "SELECT * FROM storage_pools_config"

We may be able to trick Incus into thinking the storage pool is a dir backend, so you can get the data out at least.

5

Just to confirm, I can access the data inside the recovered instances. The instances are actually up and running on the rebuilt server.

Here is the storage info:

depingus@server:~$ sudo incus admin sql global "SELECT * FROM storage_pools"
+----+---------+--------+-------------+-------+
| ID |  NAME   | DRIVER | DESCRIPTION | STATE |
+----+---------+--------+-------------+-------+
| 1  | default | btrfs  |             | 1     |
+----+---------+--------+-------------+-------+

depingus@server:~$ sudo incus admin sql global "SELECT * FROM storage_pools_config"
+----+-----------------+---------+-------------------------+--------------------------------------+
| ID | STORAGE POOL ID | NODE ID |           KEY           |                VALUE                 |
+----+-----------------+---------+-------------------------+--------------------------------------+
| 2  | 1               | 1       | source                  | /var/lib/incus/storage-pools/default |
+----+-----------------+---------+-------------------------+--------------------------------------+
| 3  | 1               | 1       | volatile.initial_source | /var/lib/incus/storage-pools/default |
+----+-----------------+---------+-------------------------+--------------------------------------+
6

Okay, so you could probably do incus admin sql global "UPDATE storage_pools SET driver='dir' WHERE id=1"to temporarily make Incus think the storage driver is the simple dir storage.

That should let you incus export everything as well as use incus delete on the instances that need to be re-imported.

I’d probably recommend getting rid of the storage pool entirely and then re-create it cleanly and re-import things from the exports, that way all the btrfs subvolumes will be right again.

1 Like
7

Thank you so much! This worked! I was able to delete the images and export the instances out of the broken default storage.

Then I created a new storage with sudo incus storage create newstore btrfs, and imported the instances. They started right up. Lastly, I edited the default profile to point to the new storage so that any new instances go to the right location.

I wasn’t able to delete the default storage. I don’t know enough about btrfs to understand whats happening here. It looks like the incus default storage took over the entire /dev/sda2 partition where the whole OS lives, and then newstore is just looped in on top or inside of it?

I don’t know. But everything is working for now! Maybe over the holidays I can rebuild the server again, this time…with proper exported instances to import cleanly.

Thanks again! I really appreciate your time, help, and the work you do on incus!

Storage info for the curious:

depingus@server:~$ sudo incus storage delete default
Error: Failed to remove directory "/var/lib/incus/storage-pools/default": remove /var/lib/incus/storage-pools/default: device or resource busy

depingus@server:~$ sudo incus storage show default
config:
  source: /var/lib/incus/storage-pools/default
  volatile.initial_source: /var/lib/incus/storage-pools/default
description: ""
name: default
driver: dir
used_by: []
status: Created
locations:
- none

depingus@server:~$ sudo incus storage info default
info:
  description: ""
  driver: dir
  name: default
  space used: 45.25GiB
  total space: 438.28GiB
used by: {}

depingus@server:~$ sudo ls -lah /var/lib/incus/storage-pools/default
total 0
drwxr-xr-x 1 root root  0 Dec 19 13:53 .
drwx--x--x 1 root root 30 Dec 19 13:20 ..

depingus@server:~$ sudo lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0    30G  0 loop /var/lib/incus/storage-pools/newstore
sda      8:0    0 447.1G  0 disk 
├─sda1   8:1    0   976M  0 part /boot/efi
├─sda2   8:2    0 438.3G  0 part /var/lib/incus/storage-pools/default
│                                /
└─sda3   8:3    0   7.9G  0 part [SWAP]

depingus@server:~$ sudo btrfs filesystem show
Label: none  uuid: 368a42d5-35da-4770-a341-d23edace3755
	Total devices 1 FS bytes used 45.09GiB
	devid    1 size 438.28GiB used 53.02GiB path /dev/sda2

Label: 'newstore'  uuid: 829e49d3-1bf8-4459-91d5-6a07123ce85b
	Total devices 1 FS bytes used 16.32GiB
	devid    1 size 30.00GiB used 18.52GiB path /dev/loop0