LVM stoarge container doesn't start

LXD
42

Its a big log!!

56660.332552] EXT4-fs error (device dm-6): ext4_validate_block_bitmap:395: comm lxd: bg 4: bad block bitmap checksum
[56660.335812] EXT4-fs error (device dm-6): ext4_validate_block_bitmap:395: comm lxd: bg 5: bad block bitmap checksum
[56660.339008] EXT4-fs error (device dm-6): ext4_validate_block_bitmap:395: comm lxd: bg 6: bad block bitmap checksum
[56660.342264] EXT4-fs warning (device dm-6): ext4_end_bio:323: I/O error 10 writing to inode 8178 (offset 0 size 8192 starting block 230912)
[56660.342270] Buffer I/O error on device dm-6, logical block 230912
[56660.344412] Buffer I/O error on device dm-6, logical block 230913
[56660.402689] JBD2: Detected IO errors while flushing file data on dm-6-8
[56660.404711] buffer_io_error: 24 callbacks suppressed
[56660.404715] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56660.418905] Aborting journal on device dm-6-8.
[56660.425953] EXT4-fs error (device dm-6): ext4_put_super:935: Couldn’t clean up the journal
[56660.428231] EXT4-fs (dm-6): Remounting filesystem read-only
[56660.455279] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56660.458312] JBD2: recovery failed
[56660.458316] EXT4-fs (dm-6): error loading journal
[56660.966089] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56661.014366] JBD2: recovery failed
[56661.014371] EXT4-fs (dm-6): error loading journal
[56661.522489] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56661.549972] JBD2: recovery failed
[56661.549977] EXT4-fs (dm-6): error loading journal
[56662.058095] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56662.111365] JBD2: recovery failed
[56662.111370] EXT4-fs (dm-6): error loading journal
[56662.617096] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56662.639887] JBD2: recovery failed
[56662.639891] EXT4-fs (dm-6): error loading journal
[56663.147681] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56663.164301] JBD2: recovery failed
[56663.164306] EXT4-fs (dm-6): error loading journal
[56663.672437] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56663.697270] JBD2: recovery failed
[56663.697275] EXT4-fs (dm-6): error loading journal
[56664.205546] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56664.231275] JBD2: recovery failed
[56664.231280] EXT4-fs (dm-6): error loading journal
[56664.739626] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56664.749338] JBD2: recovery failed
[56664.749342] EXT4-fs (dm-6): error loading journal
[56665.272564] JBD2: recovery failed
[56665.272567] EXT4-fs (dm-6): error loading journal
[56665.781094] buffer_io_error: 1 callbacks suppressed
[56665.781098] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56665.806243] JBD2: recovery failed
[56665.806247] EXT4-fs (dm-6): error loading journal
[56666.314546] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56666.342176] JBD2: recovery failed
[56666.342181] EXT4-fs (dm-6): error loading journal
[56666.850481] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56666.876932] JBD2: recovery failed
[56666.876937] EXT4-fs (dm-6): error loading journal
[56667.385245] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56667.404933] JBD2: recovery failed
[56667.404938] EXT4-fs (dm-6): error loading journal
[56667.913581] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56667.936744] JBD2: recovery failed
[56667.936749] EXT4-fs (dm-6): error loading journal
[56668.445003] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56668.468530] JBD2: recovery failed
[56668.468534] EXT4-fs (dm-6): error loading journal
[56668.976796] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56669.000504] JBD2: recovery failed
[56669.000509] EXT4-fs (dm-6): error loading journal
[56669.509071] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56669.512722] JBD2: recovery failed
[56669.512728] EXT4-fs (dm-6): error loading journal
[56670.021228] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56670.025030] JBD2: recovery failed
[56670.025035] EXT4-fs (dm-6): error loading journal
[56670.533426] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[56670.543151] JBD2: recovery failed
[56670.543156] EXT4-fs (dm-6): error loading journal
[66161.634623] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66161.662399] JBD2: recovery failed
[66161.662404] EXT4-fs (dm-6): error loading journal
[66162.171296] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66162.205892] JBD2: recovery failed
[66162.205896] EXT4-fs (dm-6): error loading journal
[66162.714509] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66162.725775] JBD2: recovery failed
[66162.725781] EXT4-fs (dm-6): error loading journal
[66163.234550] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66163.262596] JBD2: recovery failed
[66163.262601] EXT4-fs (dm-6): error loading journal
[66163.771489] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66163.793069] JBD2: recovery failed
[66163.793073] EXT4-fs (dm-6): error loading journal
[66164.301925] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66164.327120] JBD2: recovery failed
[66164.327125] EXT4-fs (dm-6): error loading journal
[66164.836307] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66164.862142] JBD2: recovery failed
[66164.862146] EXT4-fs (dm-6): error loading journal
[66165.370863] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66165.397319] JBD2: recovery failed
[66165.397324] EXT4-fs (dm-6): error loading journal
[66165.906134] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66165.929821] JBD2: recovery failed
[66165.929826] EXT4-fs (dm-6): error loading journal
[66166.439093] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66166.460234] JBD2: recovery failed
[66166.460239] EXT4-fs (dm-6): error loading journal
[66166.969477] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66166.991798] JBD2: recovery failed
[66166.991803] EXT4-fs (dm-6): error loading journal
[66167.500970] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66167.505493] JBD2: recovery failed
[66167.505498] EXT4-fs (dm-6): error loading journal
[66168.014722] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66168.019072] JBD2: recovery failed
[66168.019076] EXT4-fs (dm-6): error loading journal
[66168.528047] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66168.532281] JBD2: recovery failed
[66168.532286] EXT4-fs (dm-6): error loading journal
[66169.041365] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66169.061051] JBD2: recovery failed
[66169.061055] EXT4-fs (dm-6): error loading journal
[66169.570227] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66169.595179] JBD2: recovery failed
[66169.595184] EXT4-fs (dm-6): error loading journal
[66170.104486] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66170.122630] JBD2: recovery failed
[66170.122635] EXT4-fs (dm-6): error loading journal
[66170.631623] Buffer I/O error on dev dm-6, logical block 1034, lost async page write
[66170.647596] JBD2: recovery failed
[66170.647600] EXT4-fs (dm-6): error loading journal
[66171.157050] Buffer I/O error on dev dm-6, logical block 1034, lost async page write

43

Here is some command output . have deleted another lvm for clean space but contanier deleted but lvm disk doesn’t delete.

root@lxd1:~# e2fsck -yf /dev/my-lvm/containers_Smart-DB
e2fsck 1.44.1 (24-Mar-2018)
e2fsck: No such file or directory while trying to open /dev/my-lvm/containers_Smart-DB
Possibly non-existent device?

root@lxd1:~# vgscan --cache
WARNING: /dev/my-lvm/containers_Smart–DB: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/custom_vdisk01: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_Smart–DB-nSmart–DB–backup: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_EID–DB-14122020: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_Smart–DB-Smart–DB: Thin’s thin-pool needs inspection.
Reading volume groups from cache.
Found volume group “vg1” using metadata type lvm2
Found volume group “my-lvm” using metadata type lvm2
Found volume group “ubuntu-vg” using metadata type lvm2

root@lxd1:~# lvchange -ay /dev/vg1/vol1

root@lxd1:~# e2fsck -yf /dev/my-lvm/containers_Smart-DB
e2fsck 1.44.1 (24-Mar-2018)
e2fsck: No such file or directory while trying to open /dev/my-lvm/containers_Smart-DB
Possibly non-existent device?

root@lxd1:~# lvmdiskscan
WARNING: /dev/my-lvm/containers_Smart–DB: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/custom_vdisk01: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_Smart–DB-Smart–DB–backup: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_ID–DB-14122020: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_Smart–DB-Smart–DB: Thin’s thin-pool needs inspection.
/dev/loop0 [ 99.21 MiB]
/dev/zd0 [ 2.00 TiB] LVM physical volume
/dev/ubuntu-vg/ubuntu-lv [ <248.50 GiB]
/dev/loop1 [ <99.15 MiB]
/dev/sda1 [ 1.09 TiB]
/dev/vg1/vol1 [ <2.00 TiB] LVM physical volume
WARNING: /dev/my-lvm/containers_Smart–DB: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/custom_vdisk01: Thin’s thin-pool needs inspection.
/dev/sda9 [ 8.00 MiB]
WARNING: /dev/my-lvm/containers_Smart–DB-nSmart–DB–backup: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_EID–DB-14122020: Thin’s thin-pool needs inspection.
WARNING: /dev/my-lvm/containers_Smart–DB-Smart–DB: Thin’s thin-pool needs inspection.
/dev/sdb1 [ 1.09 TiB]
/dev/sdb9 [ 8.00 MiB]
/dev/sdc1 [ 1.09 TiB]
/dev/sdc9 [ 8.00 MiB]
/dev/sdd1 [ 1.09 TiB]
/dev/sdd9 [ 8.00 MiB]
/dev/sde1 [ 512.00 MiB]
/dev/sde2 [ 1.00 GiB]
/dev/sde3 [ <248.50 GiB] LVM physical volume
1 disk
12 partitions
0 LVM physical volume whole disks
3 LVM physical volumes

44

@tomp Plz check it :pensive:

45

Hi,

I’m not sure how to proceed, the LVM thinpool looks to be quite corrupted, and potentially the filesystems in the thin volumes too.

If it were me, I’d be thinking about deleting the volume group entirely and starting again afresh, and checking that the hard disk itself is still functioning correctly after the power cut.

Perhaps a final step before that is to try this article here (I found it by searching, no warranty etc), as it guides you through the repair process.