Error: Failed to mount LVM logical volume: Failed to mount using "ext4": input/output error

LXD
,
1

Hello.
I have a problem with the different machines when I try to start containers:

DBUG[12-17|01:05:31] Sending request to LXD                   method=GET url=http://unix.socket/1.0/operations/9840d5ec-d18a-43a8-b9f1-d44cfebe356d etag=
DBUG[12-17|01:05:31] Got response struct from LXD 
DBUG[12-17|01:05:31] 
	{
		"id": "9840d5ec-d18a-43a8-b9f1-d44cfebe356d",
		"class": "task",
		"description": "Starting instance",
		"created_at": "2021-12-17T01:05:31.449567441+03:00",
		"updated_at": "2021-12-17T01:05:31.449567441+03:00",
		"status": "Running",
		"status_code": 103,
		"resources": {
			"instances": [
				"/1.0/instances/gluster-0"
			]
		},
		"metadata": null,
		"may_cancel": false,
		"err": "",
		"location": "none"
	} 
Error: Failed to mount LVM logical volume: Failed to mount "/dev/default/containers_gluster--0" on "/var/snap/lxd/common/lxd/storage-pools/default/containers/gluster-0" using "ext4": input/output error

I can remove the default pool but I have to save my data from containers storage.
Logs:

[Fri Dec 17 01:34:42 2021] XFS (dm-16): Mounting V5 Filesystem
[Fri Dec 17 01:34:42 2021] XFS (dm-16): Ending clean mount
[Fri Dec 17 01:34:42 2021] xfs filesystem being mounted at /var/lib/docker/devicemapper/mnt/b2c40441045b7604b293f3d6bdd362186e4fa460a0c84e7e4393e5c50e33fd53-init supports timestamps until 2038 (0x7fffffff)
[Fri Dec 17 01:34:43 2021] XFS (dm-16): Unmounting Filesystem
[Fri Dec 17 01:34:43 2021] XFS (dm-16): Mounting V5 Filesystem
[Fri Dec 17 01:34:43 2021] XFS (dm-16): Ending clean mount
[Fri Dec 17 01:34:43 2021] xfs filesystem being mounted at /var/lib/docker/devicemapper/mnt/79f78fd0dceb3d39cafc4a7a0fec6241bc4d44479bbbb8035b9229464ea9ab3c supports timestamps until 2038 (0x7fffffff)
[Fri Dec 17 01:34:44 2021] XFS (dm-17): Mounting V5 Filesystem
[Fri Dec 17 01:34:44 2021] JBD2: Detected IO errors while flushing file data on dm-14-8
[Fri Dec 17 01:34:45 2021] EXT4-fs warning (device dm-4): ext4_end_bio:315: I/O error 3 writing to inode 137143 (offset 0 size 0 starting block 371200)
[Fri Dec 17 01:34:45 2021] XFS (dm-17): Ending clean mount
[Fri Dec 17 01:34:45 2021] containerd[205161]: segfault at 55f2f6938000 ip 000055f2f6938000 sp 00007ffdc5326bc8 error 14 in containerd[55f2f6b38000+d0a000]
[Fri Dec 17 01:34:45 2021] Code: Bad RIP value.
[Fri Dec 17 01:34:45 2021] xfs filesystem being mounted at /var/lib/docker/devicemapper/mnt/79f78fd0dceb3d39cafc4a7a0fec6241bc4d44479bbbb8035b9229464ea9ab3c-init supports timestamps until 2038 (0x7fffffff)
[Fri Dec 17 01:34:45 2021] EXT4-fs warning (device dm-4): ext4_end_bio:315: I/O error 3 writing to inode 131413 (offset 0 size 4096 starting block 622318)
[Fri Dec 17 01:34:45 2021] EXT4-fs warning (device dm-4): ext4_end_bio:315: I/O error 3 writing to inode 131357 (offset 0 size 4096 starting block 622319)
[Fri Dec 17 01:34:47 2021] XFS (dm-17): Unmounting Filesystem
[Fri Dec 17 01:34:47 2021] XFS (dm-16): Unmounting Filesystem
[Fri Dec 17 01:34:47 2021] EXT4-fs warning: 3 callbacks suppressed
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 714084)
[Fri Dec 17 01:34:47 2021] buffer_io_error: 11 callbacks suppressed
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714084
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714085)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714085
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 714086)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714086
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714087)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714087
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 714088)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714088
[Fri Dec 17 01:34:47 2021] JBD2: Detected IO errors while flushing file data on dm-14-8
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714089)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714089
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 714090)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714090
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714091)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714091
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 714092)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714092
[Fri Dec 17 01:34:47 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714093)
[Fri Dec 17 01:34:47 2021] Buffer I/O error on device dm-14, logical block 714093
[Fri Dec 17 01:34:47 2021] XFS (dm-16): Mounting V5 Filesystem
[Fri Dec 17 01:34:48 2021] XFS (dm-16): Ending clean mount
[Fri Dec 17 01:34:48 2021] xfs filesystem being mounted at /var/lib/docker/devicemapper/mnt/79f78fd0dceb3d39cafc4a7a0fec6241bc4d44479bbbb8035b9229464ea9ab3c supports timestamps until 2038 (0x7fffffff)
[Fri Dec 17 01:34:48 2021] XFS (dm-16): Unmounting Filesystem
[Fri Dec 17 01:34:48 2021] XFS (dm-16): Mounting V5 Filesystem
[Fri Dec 17 01:34:48 2021] XFS (dm-16): Ending clean mount
[Fri Dec 17 01:34:49 2021] xfs filesystem being mounted at /var/lib/docker/devicemapper/mnt/f7a6583e808a6f56f15a0318d20f7d4759d13e135b82daadaa7a1cb0edc06bb2-init supports timestamps until 2038 (0x7fffffff)
[Fri Dec 17 01:34:50 2021] containerd[205282]: segfault at 55c8c5c7c000 ip 000055c8c5c7c000 sp 00007ffe3e3a3278 error 14 in containerd[55c8c5e7c000+d0a000]
[Fri Dec 17 01:34:50 2021] Code: Bad RIP value.
[Fri Dec 17 01:34:51 2021] XFS (dm-16): Unmounting Filesystem
[Fri Dec 17 01:34:52 2021] EXT4-fs warning: 17 callbacks suppressed
[Fri Dec 17 01:34:52 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 271260 (offset 0 size 8192 starting block 652042)
[Fri Dec 17 01:34:52 2021] buffer_io_error: 19 callbacks suppressed
[Fri Dec 17 01:34:52 2021] Buffer I/O error on device dm-14, logical block 652042
[Fri Dec 17 01:34:52 2021] Buffer I/O error on device dm-14, logical block 652043
[Fri Dec 17 01:34:52 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 271260 (offset 0 size 8192 starting block 652044)
[Fri Dec 17 01:34:52 2021] Buffer I/O error on device dm-14, logical block 652044
[Fri Dec 17 01:34:52 2021] Buffer I/O error on device dm-14, logical block 652045
[Fri Dec 17 01:34:54 2021] EXT4-fs warning (device dm-4): ext4_end_bio:315: I/O error 3 writing to inode 137143 (offset 0 size 0 starting block 371200)
[Fri Dec 17 01:34:54 2021] Buffer I/O error on device dm-4, logical block 371197
[Fri Dec 17 01:34:54 2021] Buffer I/O error on device dm-4, logical block 371198
[Fri Dec 17 01:34:54 2021] Buffer I/O error on device dm-4, logical block 371199
[Fri Dec 17 01:34:54 2021] Buffer I/O error on device dm-4, logical block 371200
[Fri Dec 17 01:34:54 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714105)
[Fri Dec 17 01:34:54 2021] Buffer I/O error on device dm-14, logical block 714105
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 271260 (offset 0 size 8192 starting block 714106)
[Fri Dec 17 01:34:55 2021] Buffer I/O error on device dm-14, logical block 714106
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 271260 (offset 0 size 8192 starting block 652046)
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 652048)
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 652049)
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 652050)
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 652051)
[Fri Dec 17 01:34:55 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 652052)
[Fri Dec 17 01:34:56 2021] containerd[205334]: segfault at 5628d898d000 ip 00005628d898d000 sp 00007ffd550a0308 error 14 in containerd[5628d8b8d000+d0a000]
[Fri Dec 17 01:34:56 2021] Code: Bad RIP value.
[Fri Dec 17 01:34:57 2021] JBD2: Detected IO errors while flushing file data on dm-14-8
[Fri Dec 17 01:34:57 2021] EXT4-fs warning: 6 callbacks suppressed
[Fri Dec 17 01:34:57 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 271260 (offset 0 size 8192 starting block 652057)
[Fri Dec 17 01:34:57 2021] buffer_io_error: 14 callbacks suppressed
[Fri Dec 17 01:34:57 2021] Buffer I/O error on device dm-14, logical block 652057
[Fri Dec 17 01:34:57 2021] Buffer I/O error on device dm-14, logical block 652058
[Fri Dec 17 01:34:57 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 271260 (offset 0 size 8192 starting block 714108)
[Fri Dec 17 01:34:57 2021] Buffer I/O error on device dm-14, logical block 714108
[Fri Dec 17 01:34:57 2021] Buffer I/O error on device dm-14, logical block 714109
[Fri Dec 17 01:34:57 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 131835 (offset 0 size 4096 starting block 714110)
[Fri Dec 17 01:34:57 2021] Buffer I/O error on device dm-14, logical block 714110
[Fri Dec 17 01:34:57 2021] EXT4-fs warning (device dm-14): ext4_end_bio:315: I/O error 3 writing to inode 132091 (offset 0 size 4096 starting block 714111)
[Fri Dec 17 01:34:57 2021] Buffer I/O error on device dm-14, logical block 714111

How to solve the problem?

2

That suggests physical disk corruption mixed in with CPU errors.

I’d start by rebooting the system and see how it behaves then.

3

I have tried to reboot all hosts. Cpu utilisation is only 7%. Physical disk corruption? In two different machines?

4

That or you have two versions of LVM fighting for metadata.

Is that using the LXD snap and if it is, are you using LVM separately from LXD on those systems?

If so, you should try snap set lxd lvm.external=true and then reboot the system.
This will cause the LXD snap to rely on the system’s LVM version rather than the built-in one.

5

I don’t know exactly. I installed LXD from snap and then run the init script and this one created a default pool. I ran the command in your message and then rebooted hosts. Also after a long timeout, I get the same error.