I can’t install docker in lxc container:
dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
Config error: [Errno 30] Read-only file system: '/var/log/dnf.log': '/var/log/dnf.log'
Configuration:
architecture: x86_64
config:
image.architecture: amd64
image.description: Almalinux 8 amd64 (20211028_23:08)
image.os: Almalinux
image.release: "8"
image.serial: "20211028_23:08"
image.type: squashfs
image.variant: default
volatile.base_image: de28512a689d129b4b205e140df9cdc2284766ee667d704af45ce0b9f63c29d4
volatile.eth0.host_name: veth4f48ffba
volatile.eth0.hwaddr: 00:16:3e:f4:f8:4b
volatile.eth0.name: eth0
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
volatile.uuid: 3061e636-6395-4aa6-b633-adf911d7c9b6
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
Profile configuration:
config: {}
description: Default LXD profile
devices:
eth0:
nictype: bridged
parent: virbr10
type: nic
root:
path: /
pool: default
type: disk
name: default
used_by:
- /1.0/instances/gluster-0
- /1.0/instances/test-master
How to fix it?
stgraber
November 13, 2021, 8:35pm
2
I’d suggest running dmesg, this often is the sign of some kind of storage corruption.
I can’t understand what do error lines in dmesg mean. Can you explain?
[Sat Nov 13 23:53:22 2021] xfs filesystem being mounted at /var/lib/docker/devicemapper/mnt/f7a6583e808a6f56f15a0318d20f7d4759d13e135b82daadaa7a1cb0edc06bb2-init supports timestamps until 2038 (0x7fffffff)
[Sat Nov 13 23:53:22 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117847)
[Sat Nov 13 23:53:22 2021] Buffer I/O error on device dm-7, logical block 117847
[Sat Nov 13 23:53:22 2021] EXT4-fs warning (device dm-10): ext4_end_bio:315: I/O error 3 writing to inode 135025 (offset 0 size 0 starting block 410093)
[Sat Nov 13 23:53:22 2021] Buffer I/O error on device dm-10, logical block 410093
[Sat Nov 13 23:53:22 2021] EXT4-fs warning (device dm-10): ext4_end_bio:315: I/O error 3 writing to inode 135025 (offset 102686720 size 8192 starting block 410094)
[Sat Nov 13 23:53:22 2021] Buffer I/O error on device dm-10, logical block 410094
[Sat Nov 13 23:53:22 2021] Buffer I/O error on device dm-10, logical block 410095
[Sat Nov 13 23:53:23 2021] XFS (dm-12): Unmounting Filesystem
[Sat Nov 13 23:53:25 2021] JBD2: Detected IO errors while flushing file data on dm-10-8
[Sat Nov 13 23:53:25 2021] EXT4-fs warning (device dm-10): ext4_end_bio:315: I/O error 3 writing to inode 131501 (offset 0 size 4096 starting block 601849)
[Sat Nov 13 23:53:25 2021] Buffer I/O error on device dm-10, logical block 601849
[Sat Nov 13 23:53:25 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117848)
[Sat Nov 13 23:53:25 2021] Buffer I/O error on device dm-7, logical block 117848
[Sat Nov 13 23:53:28 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117849)
[Sat Nov 13 23:53:28 2021] Buffer I/O error on device dm-7, logical block 117849
[Sat Nov 13 23:53:30 2021] JBD2: Detected IO errors while flushing file data on dm-10-8
[Sat Nov 13 23:53:31 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117850)
[Sat Nov 13 23:53:31 2021] Buffer I/O error on device dm-7, logical block 117850
[Sat Nov 13 23:53:31 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 270065 (offset 0 size 4096 starting block 117851)
[Sat Nov 13 23:53:31 2021] Buffer I/O error on device dm-7, logical block 117851
[Sat Nov 13 23:53:33 2021] JBD2: Detected IO errors while flushing file data on dm-7-8
[Sat Nov 13 23:53:34 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117852)
[Sat Nov 13 23:53:34 2021] Buffer I/O error on device dm-7, logical block 117852
[Sat Nov 13 23:53:37 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117853)
[Sat Nov 13 23:53:37 2021] Buffer I/O error on device dm-7, logical block 117853
[Sat Nov 13 23:53:40 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 117854)
[Sat Nov 13 23:53:40 2021] Buffer I/O error on device dm-7, logical block 117854
[Sat Nov 13 23:53:43 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 2154074)
[Sat Nov 13 23:53:43 2021] Buffer I/O error on device dm-7, logical block 2154074
[Sat Nov 13 23:53:46 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 2154075)
[Sat Nov 13 23:53:46 2021] Buffer I/O error on device dm-7, logical block 2154075
[Sat Nov 13 23:53:49 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 2154076)
[Sat Nov 13 23:53:49 2021] Buffer I/O error on device dm-7, logical block 2154076
[Sat Nov 13 23:53:50 2021] EXT4-fs warning (device dm-10): ext4_end_bio:315: I/O error 3 writing to inode 131565 (offset 0 size 4096 starting block 601850)
[Sat Nov 13 23:53:50 2021] Buffer I/O error on device dm-10, logical block 601850
[Sat Nov 13 23:53:52 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 2154077)
[Sat Nov 13 23:53:52 2021] Buffer I/O error on device dm-7, logical block 2154077
[Sat Nov 13 23:53:53 2021] EXT4-fs warning (device dm-10): ext4_end_bio:315: I/O error 3 writing to inode 135025 (offset 0 size 0 starting block 410095)
[Sat Nov 13 23:53:53 2021] Buffer I/O error on device dm-10, logical block 410095
[Sat Nov 13 23:53:53 2021] EXT4-fs warning (device dm-10): ext4_end_bio:315: I/O error 3 writing to inode 135025 (offset 102694912 size 8192 starting block 410096)
[Sat Nov 13 23:53:53 2021] Buffer I/O error on device dm-10, logical block 410096
[Sat Nov 13 23:53:53 2021] Buffer I/O error on device dm-10, logical block 410097
[Sat Nov 13 23:53:54 2021] JBD2: Detected IO errors while flushing file data on dm-10-8
[Sat Nov 13 23:53:55 2021] EXT4-fs warning (device dm-7): ext4_end_bio:315: I/O error 3 writing to inode 521223 (offset 0 size 4096 starting block 2154078)
[Sat Nov 13 23:53:55 2021] Buffer I/O error on device dm-7, logical block 2154078
But, okay, let’s imagine it is a problem with FS. Need I to remove all lxc volumes and recreate 'em again?
stgraber:
df -h
[root@srv1-prod ~]# lxc storage list
+---------+--------+--------------------------------------------+-------------+---------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY |
+---------+--------+--------------------------------------------+-------------+---------+
| default | lvm | /var/snap/lxd/common/lxd/disks/default.img | | 7 |
+---------+--------+--------------------------------------------+-------------+---------+
[root@srv1-prod ~]# cat /proc/partitions
major minor #blocks name
8 0 976762584 sda
8 1 1048576 sda1
8 2 975712256 sda2
11 0 1048575 sr0
252 0 73400320 dm-0
252 1 7471104 dm-1
252 2 894836736 dm-2
7 0 101824 loop0
7 1 43196 loop1
7 2 74328 loop2
7 3 63316 loop3
7 4 33220 loop4
7 5 43148 loop5
7 6 74844 loop6
7 7 104857600 loop7
7 8 2097152 loop8
252 3 104857600 dm-3
252 4 10485760 dm-4
7 9 12695312 loop9
252 5 1048576 dm-5
252 6 10596352 dm-6
252 8 10596352 dm-8
252 9 10596352 dm-9
252 7 9768960 dm-7
252 11 9768960 dm-11
252 10 9768960 dm-10
252 13 9768960 dm-13
[root@srv1-prod ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.4G 0 3.4G 0% /dev
tmpfs 3.4G 0 3.4G 0% /dev/shm
tmpfs 3.4G 9.3M 3.4G 1% /run
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
/dev/mapper/ol_srv1--prod-root 70G 18G 53G 25% /
/dev/loop1 43M 43M 0 100% /var/lib/snapd/snap/snapd/13831
/dev/loop4 33M 33M 0 100% /var/lib/snapd/snap/snapd/13640
/dev/loop0 100M 100M 0 100% /var/lib/snapd/snap/core/11993
/dev/loop3 62M 62M 0 100% /var/lib/snapd/snap/core20/1169
/dev/loop2 73M 73M 0 100% /var/lib/snapd/snap/lxd/21780
/dev/loop6 74M 74M 0 100% /var/lib/snapd/snap/lxd/21858
/dev/loop5 43M 43M 0 100% /var/lib/snapd/snap/certbot/1582
/dev/sda1 1014M 363M 652M 36% /boot
/dev/mapper/ol_srv1--prod-home 853G 6.0G 847G 1% /home
/dev/dm-4 10G 476M 9.6G 5% /var/lib/docker/devicemapper/mnt/f7a6583e808a6f56f15a0318d20f7d4759d13e135b82daadaa7a1cb0edc06bb2
tmpfs 1.0M 0 1.0M 0% /var/snap/lxd/common/ns
gluster-0.prod:/main 9.2G 2.2G 6.6G 25% /shared
tmpfs 688M 0 688M 0% /run/user/0
[root@srv1-prod ~]#
On the host docker works fine.
stgraber
November 13, 2021, 9:02pm
6
Can you show snap get -d lxd?
stgraber:
snap get -d lxd
[root@srv1-prod ~]# snap get -d lxc
{}
[root@srv1-prod ~]# snap get -d lxd
{}
stgraber
November 13, 2021, 9:10pm
8
Okay, given you’re using LVM on your host, it’d be good to do snap set lxd lvm.external=true so that LXD and your host use the both LVM version and configuration. I don’t know that it’s necessary the source of your problem, but it’s certainly possible that host LVM tools got confused with LXD’s own setup and caused some trouble.
Once that’s done, I’d recommend rebooting the host system, then see if the issues go away on their own. If not, you’d either want to recreate the instanc
Thanks. After snap set lxd lvm.external=true & reboot, it worked.