I have an Incus 6.12 instance on Debian Trixie with a loopback LVM pool (I would have set it up to use block device, but it refused to integrate into the existing LVM, so I went for the loopback option for testing).
Today, one of the containers decided to write so much data to the pool that the underlying disk got completely filled up. So I resized the host system root partition to make enough room, everything is fine again on that front. However, Incus is not able to bring up the containers anymore:
Failed to activate LVM logical volume "/dev/incus-pool/containers_Pi--hole": Failed to run: lvchange --activate y --ignoreactivationskip /dev/incus-pool/containers_Pi--hole: exit status 5 (Check of pool incus-pool/IncusThinPool failed (status:64). Manual repair required!)
And this is where it gets strange: the path /dev/incus-pool does not even exist! The pool itself is present in /var/lib/incus/disk/incus-pool.img, and also the loopback seems to be set:
losetup -a
/dev/loop0: [65024]:525125 (/var/lib/incus/disks/incus-pool.img)
Also, LVM seems to see everything correctly:
root@zim:/home/locu# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
IncusThinPool incus-pool twi---tz-- 49.89g
containers_ArchiSteamFarm incus-pool Vwi---tz-k 10.00g IncusThinPool
containers_Jellyfin incus-pool Vwi---tz-k 10.00g IncusThinPool
containers_Pi--hole incus-pool Vwi---tz-k 10.00g IncusThinPool
containers_Pi--hole-current_working incus-pool Vri---tz-k 10.00g IncusThinPool containers_Pi--hole
containers_tvstuff incus-pool Vwi---tz-k 10.00g IncusThinPool
containers_tvstuff-current_working incus-pool Vri---tz-k 10.00g IncusThinPool containers_tvstuff
images_c4f17b293ea6413a120b169de518c2a75c72c311281cc45ddabbcc8500be4c2e incus-pool Vwi---tz-k 10.00g IncusThinPool
home zim-vg -wi-ao---- <66.30g
root zim-vg -wi-ao---- <72.94g
swap_1 zim-vg -wi-ao---- 976.00m
root@zim:/home/locu# pvs
PV VG Fmt Attr PSize PFree
/dev/loop0 incus-pool lvm2 a-- <50.00g 0
/dev/nvme1n1p3 zim-vg lvm2 a-- 475.96g 235.77g
root@zim:/home/locu# vgdisplay incus-pool
--- Volume group ---
VG Name incus-pool
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 188
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 8
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <50.00 GiB
PE Size 4.00 MiB
Total PE 12799
Alloc PE / Size 12799 / <50.00 GiB
Free PE / Size 0 / 0
VG UUID d9kUDu-oYaI-qZ2M-LPFG-tQSn-yF9l-bKZAK1
root@zim:/home/locu# lvdisplay incus-pool
--- Logical volume ---
LV Name IncusThinPool
VG Name incus-pool
LV UUID A8ycwe-K2EN-3RN9-FtJs-XM0n-r9UZ-bLoND0
LV Write Access read/write
LV Creation host, time zim, 2025-04-07 13:12:43 +0200
LV Pool metadata IncusThinPool_tmeta
LV Pool data IncusThinPool_tdata
LV Status NOT available
LV Size 49.89 GiB
Current LE 12773
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/containers_Pi--hole
LV Name containers_Pi--hole
VG Name incus-pool
LV UUID pufF3d-ewG4-nVbM-SwsV-3U0G-lWc0-XYxH3i
LV Write Access read/write
LV Creation host, time zim, 2025-04-07 13:42:33 +0200
LV Pool name IncusThinPool
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/containers_tvstuff
LV Name containers_tvstuff
VG Name incus-pool
LV UUID X72SyS-RHpi-6K7m-X6kd-d9HA-MlGw-1QUKSd
LV Write Access read/write
LV Creation host, time zim, 2025-04-07 19:54:37 +0200
LV Pool name IncusThinPool
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/containers_tvstuff-current_working
LV Name containers_tvstuff-current_working
VG Name incus-pool
LV UUID kWkfJp-Gxgu-iGxB-ugsE-Gu3M-mhg2-QoB8qR
LV Write Access read only
LV Creation host, time zim, 2025-04-15 12:58:44 +0200
LV Pool name IncusThinPool
LV Thin origin name containers_tvstuff
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/containers_Pi--hole-current_working
LV Name containers_Pi--hole-current_working
VG Name incus-pool
LV UUID 30mOOT-jXW5-TPUH-6DsR-yPAZ-gcXZ-dd198R
LV Write Access read only
LV Creation host, time zim, 2025-04-15 12:58:56 +0200
LV Pool name IncusThinPool
LV Thin origin name containers_Pi--hole
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/containers_Jellyfin
LV Name containers_Jellyfin
VG Name incus-pool
LV UUID eYrXcG-Nmfl-UQWl-TVaA-AATA-Q6jn-v23Lzk
LV Write Access read/write
LV Creation host, time zim, 2025-04-15 13:00:55 +0200
LV Pool name IncusThinPool
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/containers_ArchiSteamFarm
LV Name containers_ArchiSteamFarm
VG Name incus-pool
LV UUID zLNFGL-VhRm-JC5Q-KnzD-J1WX-UnJX-cOI6Zs
LV Write Access read/write
LV Creation host, time zim, 2025-04-27 21:08:45 +0200
LV Pool name IncusThinPool
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
--- Logical volume ---
LV Path /dev/incus-pool/images_c4f17b293ea6413a120b169de518c2a75c72c311281cc45ddabbcc8500be4c2e
LV Name images_c4f17b293ea6413a120b169de518c2a75c72c311281cc45ddabbcc8500be4c2e
VG Name incus-pool
LV UUID McXd6w-aKd1-qIv9-ulgk-bI2x-TcLD-utzBKw
LV Write Access read/write
LV Creation host, time zim, 2025-04-28 15:38:44 +0200
LV Pool name IncusThinPool
LV Status NOT available
LV Size 10.00 GiB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
I am not certain if this is related to the .img file being corrupted in some way, the LVM in the img file being broken, some updates that came in since the last reboot, or some other completely unrelated error, but I know that everything was working fine this morning with multiple containers running.
My best guess currently is that the problem lies somewhere between the loopback and LVM since the LVM data also points to /dev/incus-pool.
So: How do I do the manual repair that is requested and get everything running again?