Corrupted lvm volumes - can't delete default storage

Incus
1

Hi,

I’m non english native speaker, and I think I made several mistakes that leaded to this : all my containers (3) et vms (2) are down since I thought I made the right configuration with space quotas on my lvm thinpool provisionning storage configuration, but it seems I might have not understand a lot of stuff. My lvs & default.img are corrupted & I tried so many different things (e2fsck was kind of the last resort) thanks to all the super work of everyone here & thanks to documentation, but it didn’t do & I write here to you, anyway thanks lot to anyone involved in Incus <3.

I made backups of so many stuff (storage, network, profile, etc) in yml format.

Now I just want to incus import backup.tar.gz of my safe backups (before everything “exploded”) into a new ZFS or DIR storage pool. But I can’t manage to delete my default corrupted lvm storage :

incus storage delete default
Error: The storage pool is currently in use

I tried this “solution” but I end up with these messages saying my containers & vms are down, which I know :

printf 'config: {}\ndevices: {}' | sudo incus profile edit default
Error: The following instances failed to update (profile change still saved):
 - Project: default, Instance: xxx: Failed to write backup file: Failed to mount LVM logical volume: Failed to mount "/dev/default/virtual-machines_xxx" on "/var/lib/incus/storage-pools/default/virtual-machines/xxx" using "ext4": invalid argument
 - Project: default, Instance: xxx: Failed to write backup file: Failed to mount LVM logical volume: Failed to mount "/dev/default/containers_xxx" on "/var/lib/incus/storage-pools/default/containers/xxx" using "ext4": structure needs cleaning
 - Project: default, Instance: xxx: Failed to write backup file: Failed to mount LVM logical volume: Failed to mount "/dev/default/containers_xxx" on "/var/lib/incus/storage-pools/default/containers/xxx" using "ext4": structure needs cleaning
 - Project: default, Instance: xxx: Failed to write backup file: Failed to mount LVM logical volume: Failed to mount "/dev/default/virtual-machines_xxx" on "/var/lib/incus/storage-pools/default/virtual-machines/xxx" using "ext4": invalid argument
 - Project: default, Instance: xxx: Failed to write backup file: Failed to mount LVM logical volume: Failed to mount "/dev/default/containers_xxx" on "/var/lib/incus/storage-pools/default/containers/containers_xxx" using "ext4": structure needs cleaning

And then end up with Error: The storage pool is currently in use whatever I do.

Now I want to just delete this storage that incus created in the beggining when I created lvm storage, but I can’t & it take too much “ghost space” (I don’t understand how incus & lvm communicate but I guess I should let incus do the work here ?) in my partition I want to recreate new storage.

I want to avoid erasing all incus stuff I have (profiles, network, macvlan, etc) like this post may suggest, only lvm storage-pool.

Thanks for the help here,

2

So what’s the state of LVM? Did you manually delete the VG/PV already?

Sounds like those instances need to go away. If you can’t delete them the normal way, you may need to do some database surgery to get rid of them.

Though if you’re looking at deleting everything on the machine, wouldn’t you be better off just wiping Incus entirely (Basically delete everything in /var/lib/incus/ and reboot the system).

1 Like
3

Thanks a lot @stgraber for your answer, I am very lucky.

I also thought it would be easier to wipe out & redo it all but I don’t know how to easily “rebuild” everything without re-doing every configuration stuff, is there a way to export configuration files (network, macvlan, share disks, size of containers/virtual machines, etc) then import it ?

EDIT : a bit more precisions

4

Sorry, states are « STOPPED », « LV Status » are « NOT available », nothing appear anymore inside df -h, vgchange --activate y won’t help too.

5

I wiped out it all because everything was super buggy at this point, thanks a lot

6

I have no clue why, but my problem came back with Trixie upgrade, lxd seemed to have come back but I don’t think it’s related. I still have this problem with my backup-volume that I can’t delete.

7

I did manage to restart incus following what was in this post.

/etc/lvm/lvm.conf changing thin_check_options = [ “-q”, “--clear-needs-check-flag” ] modify the flag to --auto-repair so it becomes thin_check_options = [ “-q”, “–auto-repair” ]

EDIT : sorry for multiple posts, I’ll try inform here how I got everything up, but I guess I’m gonna have to redo everything, I really never managed to understand how incus & lvm works as I always have problems with it and with space left in my volumes/thinpool etc :

pvs
  PV                          VG               Fmt  Attr PSize   PFree
  /dev/mapper/nvme0n1p3_crypt tatati-vg lvm2 a--  406.17g    0
Can't resize lvm backup volume storage
8

A thin pool is itself a logical volume. If you create a thin pool and allocate all remaining space in your volume group to it, then your volume group will be 100% full. And this is what incus does by default [^1].

But you can then create child logical volumes inside the thin pool. The space remaining in the thin pool is visible separately. (Look at the output of “lvs” to see)

[^1] You can override this with options to incus storage create

  • lvm.use_thinpool=false if you want to use regular LVM logical volumes, with no thin pool (snapshots will not work, or be very large and slow - I’ve not tried doing this)
  • lvm.thinpool_name=MyThinPool size=100GiB to set the thinpool name and/or size, so it doesn’t use 100% of the space in the volume group
  • lvm.vg.force_reuse=true if you want incus to use an existing LVM volume group and share it with non-incus logical volumes.

Thin pools allow efficient snapshots of logical volumes, which is why incus uses them by default.
Here’s a presentation and lab which might clarify what thin pools are and how they work.

I can’t help you with your stuck volumes though, as I’ve no idea how you got the system into that state, nor what version of incus you’re using, nor why you have lxd installed at the same time. Another wipe and clean install, with no lxd, might be the simplest answer. Use incus 7.0 LTS from the zabbly repos - not anything ancient that comes packaged with Debian.