LXD container filesystem read only

LXD
1

I am facing difficulty in writing any files inside the lxd container because the filesystem has become read only for some reason.

The host filesystem is working fine and I don’t see any difficulty in writing/reading any files on it. It’s only happening particularly on the LXD container.

It’s a loop disk BTRFS filesystem for the container. I have tried all the possible options like rebooting server, ensuring their enough disk space on the host and inside container and also tried setting property to ro false using btrfs tool but nothing worked so far.

I tried this btrfs tool command:

sudo btrfs prop set -ts /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/containers/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/ ro false

Got this as response:
ERROR: failed to set flags for /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/containers/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/: Read-only file system

Would really appreciate a quick help on this. Thanks!

2

Hello!

Please show cat /proc/1/mountinfo from the problematic container.

3

Please can you show lxc storage show <pool> and are there any problems reported in sudo dmesg?

4

I typed in this command but I don’t see the loop device for the container (/dev/loop27):

cat /proc/1/mountinfo | grep loop

Response:

146 30 7:0 / /snap/gnome-calculator/884 ro,nodev,relatime shared:31 - squashfs /dev/loop0 ro
91 30 7:1 / /snap/snapd/17576 ro,nodev,relatime shared:35 - squashfs /dev/loop1 ro
274 30 7:2 / /snap/core22/275 ro,nodev,relatime shared:36 - squashfs /dev/loop2 ro
280 30 7:3 / /snap/bare/5 ro,nodev,relatime shared:144 - squashfs /dev/loop3 ro
286 30 7:4 / /snap/gnome-42-2204/29 ro,nodev,relatime shared:147 - squashfs /dev/loop4 ro
292 30 7:5 / /snap/gnome-characters/741 ro,nodev,relatime shared:150 - squashfs /dev/loop5 ro
298 30 7:7 / /snap/gnome-characters/781 ro,nodev,relatime shared:153 - squashfs /dev/loop7 ro
304 30 7:6 / /snap/core18/2620 ro,nodev,relatime shared:156 - squashfs /dev/loop6 ro
310 30 7:8 / /snap/gnome-logs/115 ro,nodev,relatime shared:159 - squashfs /dev/loop8 ro
316 30 7:9 / /snap/core22/310 ro,nodev,relatime shared:162 - squashfs /dev/loop9 ro
322 30 7:11 / /snap/gnome-system-monitor/178 ro,nodev,relatime shared:165 - squashfs /dev/loop11 ro
328 30 7:10 / /snap/gnome-3-38-2004/119 ro,nodev,relatime shared:168 - squashfs /dev/loop10 ro
334 30 7:13 / /snap/gtk-common-themes/1534 ro,nodev,relatime shared:171 - squashfs /dev/loop13 ro
340 30 7:12 / /snap/gnome-3-34-1804/77 ro,nodev,relatime shared:174 - squashfs /dev/loop12 ro
346 30 7:14 / /snap/core18/2632 ro,nodev,relatime shared:177 - squashfs /dev/loop14 ro
352 30 7:15 / /snap/gnome-42-2204/44 ro,nodev,relatime shared:180 - squashfs /dev/loop15 ro
358 30 7:17 / /snap/gnome-3-34-1804/72 ro,nodev,relatime shared:183 - squashfs /dev/loop17 ro
364 30 7:18 / /snap/core20/1695 ro,nodev,relatime shared:186 - squashfs /dev/loop18 ro
370 30 7:16 / /snap/core20/1634 ro,nodev,relatime shared:189 - squashfs /dev/loop16 ro
376 30 7:19 / /snap/gnome-calculator/920 ro,nodev,relatime shared:192 - squashfs /dev/loop19 ro
382 30 7:20 / /snap/snapd/17336 ro,nodev,relatime shared:195 - squashfs /dev/loop20 ro
388 30 7:21 / /snap/lxd/23983 ro,nodev,relatime shared:198 - squashfs /dev/loop21 ro
462 30 7:22 / /snap/gnome-3-38-2004/115 ro,nodev,relatime shared:268 - squashfs /dev/loop22 ro
469 30 7:23 / /snap/gnome-system-monitor/181 ro,nodev,relatime shared:272 - squashfs /dev/loop23 ro
618 30 7:24 / /snap/lxd/23972 ro,nodev,relatime shared:416 - squashfs /dev/loop24 ro
627 30 7:25 / /snap/gtk-common-themes/1535 ro,nodev,relatime shared:422 - squashfs /dev/loop25 ro
636 30 7:26 / /snap/gnome-logs/112 ro,nodev,relatime shared:428 - squashfs /dev/loop26 ro

I also ran lsblk to confirm that I have loop27 device available. lsblk response:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0   2,5M  1 loop /snap/gnome-calculator/884
loop1         7:1    0  49,7M  1 loop /snap/snapd/17576
loop2         7:2    0  70,4M  1 loop /snap/core22/275
loop3         7:3    0     4K  1 loop /snap/bare/5
loop4         7:4    0 414,4M  1 loop /snap/gnome-42-2204/29
loop5         7:5    0   704K  1 loop /snap/gnome-characters/741
loop6         7:6    0  55,6M  1 loop /snap/core18/2620
loop7         7:7    0   476K  1 loop /snap/gnome-characters/781
loop8         7:8    0   696K  1 loop /snap/gnome-logs/115
loop9         7:9    0  72,8M  1 loop /snap/core22/310
loop10        7:10   0 346,3M  1 loop /snap/gnome-3-38-2004/119
loop11        7:11   0   2,6M  1 loop /snap/gnome-system-monitor/178
loop12        7:12   0   219M  1 loop /snap/gnome-3-34-1804/77
loop13        7:13   0  81,3M  1 loop /snap/gtk-common-themes/1534
loop14        7:14   0  55,6M  1 loop /snap/core18/2632
loop15        7:15   0 446,3M  1 loop /snap/gnome-42-2204/44
loop16        7:16   0  63,2M  1 loop /snap/core20/1634
loop17        7:17   0   219M  1 loop /snap/gnome-3-34-1804/72
loop18        7:18   0  63,2M  1 loop /snap/core20/1695
loop19        7:19   0   2,6M  1 loop /snap/gnome-calculator/920
loop20        7:20   0    48M  1 loop /snap/snapd/17336
loop21        7:21   0 136,7M  1 loop /snap/lxd/23983
loop22        7:22   0 346,3M  1 loop /snap/gnome-3-38-2004/115
loop23        7:23   0   1,5M  1 loop /snap/gnome-system-monitor/181
loop24        7:24   0 136,5M  1 loop /snap/lxd/23972
loop25        7:25   0  91,7M  1 loop /snap/gtk-common-themes/1535
loop26        7:26   0   556K  1 loop /snap/gnome-logs/112
loop27        7:27   0   1,1T  0 loop 
sda           8:0    0 931,5G  0 disk 
├─sda1        8:1    0   512M  0 part 
└─sda2        8:2    0   931G  0 part /new
nvme0n1     259:0    0   1,9T  0 disk 
├─nvme0n1p1 259:1    0   512M  0 part /boot/efi
└─nvme0n1p2 259:2    0   1,9T  0 part /
5

Please can you also show lxc storage info <pool> too.

You may have filled your storage pool up.

6

Let’s look on the full cat /proc/1/mountinfo output.

7

Hi @tomp ,

Result for lxc storage show vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7:

config:
  size: 250GB
  source: /var/snap/lxd/common/lxd/disks/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7.img
description: ""
name: vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
driver: btrfs
used_by:
- /1.0/instances/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
- /1.0/profiles/pro-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
status: Created
locations:
- none

And sudo dmesg does show some mount issues in between but they don’t seem to be related to local host disk.

8

Can we see them please

9

Full response for:

cat /proc/1/mountinfo

25 30 0:23 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
26 30 0:5 / /proc rw,nosuid,nodev,noexec,relatime shared:14 - proc proc rw
27 30 0:6 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=65773320k,nr_inodes=16443330,mode=755
28 27 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
29 30 0:25 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=13159688k,mode=755
30 1 259:2 / / rw,relatime shared:1 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
31 25 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
32 27 0:26 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw
33 29 0:27 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
34 25 0:28 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
35 34 0:29 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:10 - cgroup2 cgroup rw
36 34 0:30 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,xattr,name=systemd
37 25 0:31 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:12 - pstore pstore rw
38 25 0:32 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:13 - efivarfs efivarfs rw
39 34 0:33 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,hugetlb
40 34 0:34 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,cpu,cpuacct
41 34 0:35 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,freezer
42 34 0:36 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,cpuset,clone_children
43 34 0:37 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,net_cls,net_prio
44 34 0:38 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,memory
45 34 0:39 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,devices
46 34 0:40 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,perf_event
47 34 0:41 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,blkio
48 34 0:42 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,pids
49 34 0:43 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:25 - cgroup cgroup rw,rdma
50 26 0:44 / /proc/sys/fs/binfmt_misc rw,relatime shared:26 - autofs systemd-1 rw,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=46850
51 27 0:45 / /dev/hugepages rw,relatime shared:27 - hugetlbfs hugetlbfs rw,pagesize=2M
52 25 0:8 / /sys/kernel/debug rw,relatime shared:28 - debugfs debugfs rw
53 27 0:21 / /dev/mqueue rw,relatime shared:29 - mqueue mqueue rw
144 30 259:1 / /boot/efi rw,relatime shared:30 - vfat /dev/nvme0n1p1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
146 30 7:0 / /snap/gnome-calculator/884 ro,nodev,relatime shared:31 - squashfs /dev/loop0 ro
85 30 8:2 / /new rw,relatime shared:32 - ext4 /dev/sda2 rw
87 25 0:22 / /sys/kernel/config rw,relatime shared:33 - configfs configfs rw
89 25 0:47 / /sys/fs/fuse/connections rw,relatime shared:34 - fusectl fusectl rw
91 30 7:1 / /snap/snapd/17576 ro,nodev,relatime shared:35 - squashfs /dev/loop1 ro
274 30 7:2 / /snap/core22/275 ro,nodev,relatime shared:36 - squashfs /dev/loop2 ro
280 30 7:3 / /snap/bare/5 ro,nodev,relatime shared:144 - squashfs /dev/loop3 ro
286 30 7:4 / /snap/gnome-42-2204/29 ro,nodev,relatime shared:147 - squashfs /dev/loop4 ro
292 30 7:5 / /snap/gnome-characters/741 ro,nodev,relatime shared:150 - squashfs /dev/loop5 ro
298 30 7:7 / /snap/gnome-characters/781 ro,nodev,relatime shared:153 - squashfs /dev/loop7 ro
304 30 7:6 / /snap/core18/2620 ro,nodev,relatime shared:156 - squashfs /dev/loop6 ro
310 30 7:8 / /snap/gnome-logs/115 ro,nodev,relatime shared:159 - squashfs /dev/loop8 ro
316 30 7:9 / /snap/core22/310 ro,nodev,relatime shared:162 - squashfs /dev/loop9 ro
322 30 7:11 / /snap/gnome-system-monitor/178 ro,nodev,relatime shared:165 - squashfs /dev/loop11 ro
328 30 7:10 / /snap/gnome-3-38-2004/119 ro,nodev,relatime shared:168 - squashfs /dev/loop10 ro
334 30 7:13 / /snap/gtk-common-themes/1534 ro,nodev,relatime shared:171 - squashfs /dev/loop13 ro
340 30 7:12 / /snap/gnome-3-34-1804/77 ro,nodev,relatime shared:174 - squashfs /dev/loop12 ro
346 30 7:14 / /snap/core18/2632 ro,nodev,relatime shared:177 - squashfs /dev/loop14 ro
352 30 7:15 / /snap/gnome-42-2204/44 ro,nodev,relatime shared:180 - squashfs /dev/loop15 ro
358 30 7:17 / /snap/gnome-3-34-1804/72 ro,nodev,relatime shared:183 - squashfs /dev/loop17 ro
364 30 7:18 / /snap/core20/1695 ro,nodev,relatime shared:186 - squashfs /dev/loop18 ro
370 30 7:16 / /snap/core20/1634 ro,nodev,relatime shared:189 - squashfs /dev/loop16 ro
376 30 7:19 / /snap/gnome-calculator/920 ro,nodev,relatime shared:192 - squashfs /dev/loop19 ro
382 30 7:20 / /snap/snapd/17336 ro,nodev,relatime shared:195 - squashfs /dev/loop20 ro
388 30 7:21 / /snap/lxd/23983 ro,nodev,relatime shared:198 - squashfs /dev/loop21 ro
462 30 7:22 / /snap/gnome-3-38-2004/115 ro,nodev,relatime shared:268 - squashfs /dev/loop22 ro
469 30 7:23 / /snap/gnome-system-monitor/181 ro,nodev,relatime shared:272 - squashfs /dev/loop23 ro
618 30 7:24 / /snap/lxd/23972 ro,nodev,relatime shared:416 - squashfs /dev/loop24 ro
627 30 7:25 / /snap/gtk-common-themes/1535 ro,nodev,relatime shared:422 - squashfs /dev/loop25 ro
636 30 7:26 / /snap/gnome-logs/112 ro,nodev,relatime shared:428 - squashfs /dev/loop26 ro
645 50 0:52 / /proc/sys/fs/binfmt_misc rw,relatime shared:434 - binfmt_misc binfmt_misc rw
657 29 0:25 /snapd/ns /run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=13159688k,mode=755
672 657 0:4 mnt:[4026533388] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
947 30 0:57 / /var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700
960 947 0:4 mnt:[4026533390] /var/snap/lxd/common/ns/shmounts rw - nsfs nsfs rw
933 947 0:4 mnt:[4026533388] /var/snap/lxd/common/ns/mntns rw - nsfs nsfs rw
977 30 0:67 / /var/lib/docker/overlay2/5644b157b5119e5cab55379aa6dfbe868c0631dcb6ebc3352fec971edb05a4b9/merged rw,relatime shared:531 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/3FLRAO43BCDGX2Y4MB6GZCWME6:/var/lib/docker/overlay2/l/4J5TXY3JIVL7JYRT56PHLR5NQN:/var/lib/docker/overlay2/l/ZWPGEQK5I6GX2UZXX7LOIE6CBN:/var/lib/docker/overlay2/l/I57EHMCHDTUAUPPCYD6FJPFKNB:/var/lib/docker/overlay2/l/7ZJ7K5WH4H2CXLYJVE2Q57RL7E:/var/lib/docker/overlay2/l/XUZ3QRSAENLNHXDRWGTZXA2L46:/var/lib/docker/overlay2/l/FDCRN7NEIID7AWOYCQDVOFR6OE:/var/lib/docker/overlay2/l/2EV7HEAEMOCLZCS32C5LPKXKV5:/var/lib/docker/overlay2/l/VEDFNVFTRKG4YGH6LMQ5LLQ6DK:/var/lib/docker/overlay2/l/GEFNHMZFYSDOF4ECJTVJUS4QGT:/var/lib/docker/overlay2/l/EO6P3GO4ONWXBPPBLDRHMD3RSJ:/var/lib/docker/overlay2/l/VQZWDTI7WMSBHBJ5HQUFTVHY67,upperdir=/var/lib/docker/overlay2/5644b157b5119e5cab55379aa6dfbe868c0631dcb6ebc3352fec971edb05a4b9/diff,workdir=/var/lib/docker/overlay2/5644b157b5119e5cab55379aa6dfbe868c0631dcb6ebc3352fec971edb05a4b9/work,xino=off
1082 29 0:4 net:[4026533428] /run/docker/netns/297dbc5b5a2b rw shared:546 - nsfs nsfs rw
1269 29 0:95 / /run/user/1001 rw,nosuid,nodev,relatime shared:630 - tmpfs tmpfs rw,size=13159684k,mode=700,uid=1001,gid=1001
560 30 0:51 / /var/lib/docker/overlay2/68676e3c10e622c5e3b8c3bf224cc46768cee132205d2cb0c750948b5baff604/merged rw,relatime shared:343 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/K3RR6QGQONR4NLE77HISGGQ5MV:/var/lib/docker/overlay2/l/MP2ZR2XM4QUCI4PAX7PRJXWLYQ:/var/lib/docker/overlay2/l/JQL24LBNIDLXYQICIDLCW2RO7U:/var/lib/docker/overlay2/l/S5QHT2JCJLENVHLRZEDJ7WMTGS:/var/lib/docker/overlay2/l/OUOARJTLE7KBT5XIAWW7HBDEBU:/var/lib/docker/overlay2/l/GZJBYXV54XMLBY2TSN5XXS4FSK:/var/lib/docker/overlay2/l/LTHAJ6GZSBC5YHOJMYI4A4B5RA:/var/lib/docker/overlay2/l/FBXQ52JWZRRAOOXEGPXK6DTNBH:/var/lib/docker/overlay2/l/RFBVXMJY4ORSOOSOEJCI5H3Y2T:/var/lib/docker/overlay2/l/4QDKBFOWVTGOXRKSALOYW4XVL3:/var/lib/docker/overlay2/l/JHWRZA7MX4Y5JXSIFGUN5BDQZM:/var/lib/docker/overlay2/l/CNOM3XWWIJD6JTPLFP7DQLTD7P:/var/lib/docker/overlay2/l/FHV2WZZ25EIGRHOPQHSHQPTJF5:/var/lib/docker/overlay2/l/LHH3JEDEOGWLTHANMLDS3TE5BC:/var/lib/docker/overlay2/l/FRTFDDFDO72PFSS4MCSQQVYL5J,upperdir=/var/lib/docker/overlay2/68676e3c10e622c5e3b8c3bf224cc46768cee132205d2cb0c750948b5baff604/diff,workdir=/var/lib/docker/overlay2/68676e3c10e622c5e3b8c3bf224cc46768cee132205d2cb0c750948b5baff604/work,xino=off
828 29 0:4 net:[4026533247] /run/docker/netns/07914f722b32 rw shared:348 - nsfs nsfs rw
10

image001 (3)
image001 (3)736×272 25.2 KB

Observed this mount issue in dmesg.

11

There’s the problem then. Looks like corruption perhaps.

12

this looks like cat /proc/1/mountinfo issued from the host, but not from container :wink:

13

What does lxc storage info <pool> show?

14

yep, that’s a clean sign of a corruption of your filesystem.

15

Oh my bad, I thought you asked for host info. Below is for the container:

1204 989 0:68 /containers/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/rootfs / rw,relatime master:553 - btrfs /dev/loop27 ro,ssd,space_cache,user_subvol_rm_allowed,subvolid=278,subvol=/containers/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
1205 1204 0:84 / /dev rw,relatime - tmpfs none rw,size=492k,mode=755
1206 1204 0:83 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
1207 1208 0:83 /sys/net /proc/sys/net rw,nosuid,nodev,noexec,relatime - proc proc rw
1208 1206 0:83 /sys /proc/sys ro,relatime - proc proc rw
1209 1206 0:83 /sysrq-trigger /proc/sysrq-trigger ro,relatime - proc proc rw
1211 1204 0:85 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs rw
1212 1211 0:85 /devices/virtual/net /sys/devices/virtual/net rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
1210 1205 0:82 / /dev/mqueue rw,relatime - mqueue mqueue rw
1213 1205 0:6 /fuse /dev/fuse rw,nosuid,relatime master:2 - devtmpfs udev rw,size=65773320k,nr_inodes=16443330,mode=755
1214 1205 0:6 /net/tun /dev/net/tun rw,nosuid,relatime master:2 - devtmpfs udev rw,size=65773320k,nr_inodes=16443330,mode=755
1215 1208 0:52 / /proc/sys/fs/binfmt_misc rw,relatime master:434 - binfmt_misc binfmt_misc rw
1216 1211 0:32 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime master:13 - efivarfs efivarfs rw
1217 1211 0:47 / /sys/fs/fuse/connections rw,relatime master:34 - fusectl fusectl rw
1218 1211 0:31 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
1219 1211 0:22 / /sys/kernel/config rw,relatime master:33 - configfs configfs rw
1220 1211 0:8 / /sys/kernel/debug rw,relatime master:28 - debugfs debugfs rw
1221 1211 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
1222 1211 0:23 /kernel/tracing /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
1223 1205 0:83 / /dev/.lxc/proc rw,relatime - proc proc rw
1224 1205 0:85 / /dev/.lxc/sys rw,relatime - sysfs sys rw
1225 1205 0:66 / /dev/lxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
1226 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu2.dev-dri-card2 /dev/dri/card2 rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1227 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu2.dev-dri-renderD130 /dev/dri/renderD130 rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1228 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu2.dev-nvidia2 /dev/nvidia2 rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1229 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu2.dev-nvidiactl /dev/nvidiactl rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1230 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu3.dev-dri-card3 /dev/dri/card3 rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1231 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu3.dev-dri-renderD131 /dev/dri/renderD131 rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1232 1205 259:2 /var/snap/lxd/common/lxd/devices/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/unix.gpu3.dev-nvidia3 /dev/nvidia3 rw,relatime master:528 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
1233 1205 0:65 /lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7 /dev/.lxd-mounts rw,relatime master:530 - tmpfs tmpfs rw,size=100k,mode=711
1234 1206 0:64 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1235 1206 0:64 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1236 1206 0:64 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1237 1206 0:64 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1238 1206 0:64 /proc/slabinfo /proc/slabinfo rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1239 1206 0:64 /proc/stat /proc/stat rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1240 1206 0:64 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1241 1206 0:64 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1242 1211 0:64 /sys/devices/system/cpu/online /sys/devices/system/cpu/online rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1243 1204 0:64 / /var/snap/lxd/common/var/lib/lxcfs rw,nosuid,nodev,relatime master:529 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1244 1205 0:86 / /dev/pts rw,nosuid,noexec,relatime - devpts /proc/self/fd/44 rw,gid=5,mode=620,ptmxmode=666,max=1024
1245 1205 0:86 /ptmx /dev/ptmx rw,nosuid,noexec,relatime - devpts /proc/self/fd/44 rw,gid=5,mode=620,ptmxmode=666,max=1024
1246 1205 0:86 /0 /dev/console rw,nosuid,noexec,relatime - devpts /proc/self/fd/44 rw,gid=5,mode=620,ptmxmode=666,max=1024
1005 1208 0:84 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime - tmpfs none rw,size=492k,mode=755
1004 1205 0:87 / /dev/shm rw,nosuid,nodev - tmpfs tmpfs rw
1006 1204 0:88 / /run rw,nosuid,nodev - tmpfs tmpfs rw,size=13159688k,mode=755
1007 1006 0:89 / /run/lock rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=5120k
1008 1211 0:90 / /sys/fs/cgroup ro,nosuid,nodev,noexec - tmpfs tmpfs ro,mode=755
1009 1008 0:29 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup2 rw
1010 1008 0:30 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,xattr,name=systemd
1011 1008 0:33 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,hugetlb
1012 1008 0:36 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpuset,clone_children
1013 1008 0:35 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,freezer
1014 1008 0:38 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory
1015 1008 0:40 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,perf_event
1016 1008 0:39 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices
1017 1008 0:34 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpu,cpuacct
1018 1008 0:42 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids
1019 1008 0:43 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,rdma
1020 1008 0:41 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio
1021 1008 0:37 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,net_cls,net_prio
1022 1205 0:91 / /dev/hugepages rw,relatime - hugetlbfs hugetlbfs rw,pagesize=2M
1023 1204 0:68 /containers/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7/rootfs/snap /snap rw,relatime shared:554 master:553 - btrfs /dev/loop27 ro,ssd,space_cache,user_subvol_rm_allowed,subvolid=278,subvol=/containers/lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
1024 1023 0:92 / /snap/core18/2074 ro,nodev,relatime shared:555 - fuse.snapfuse snapfuse ro,user_id=0,group_id=0,allow_other
1025 1023 0:93 / /snap/lxd/21029 ro,nodev,relatime shared:556 - fuse.snapfuse snapfuse ro,user_id=0,group_id=0,allow_other
1026 1023 0:94 / /snap/snapd/12398 ro,nodev,relatime shared:557 - fuse.snapfuse snapfuse ro,user_id=0,group_id=0,allow_other
1283 1006 0:88 /snapd/ns /run/snapd/ns rw,nosuid,nodev - tmpfs tmpfs rw,size=13159688k,mode=755```
16 
lxc storage info vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7

info:
  description: ""
  driver: btrfs
  name: vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
  space used: 1.00TiB
  total space: 1.05TiB
used by:
  instances:
  - lxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
  profiles:
  - pro-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
17

Looks like you’ve probably filled the pool too.

18

Also looks like there has been manual resizing of the loop file and BTRFS filesystem as LXD originally recorded a size of 250GB, but clearly that isn’t the case anymore.

Possibly the filesystem has been manually resized incorrectly.

19

Yes, we manually increased the filesystem size from initial 250GB to 1TB. But inside the container it still shows that 45GB of disk space is left.

Used this approach to increase the filesystem size:

sudo truncate -s +100GB /var/snap/lxd/common/lxd/disks/$volume.img
loopDevice=$(echo $(sudo nsenter -t $(pgrep daemon.start) -m df | grep $volume) | cut -d' ' -f1)
sudo losetup -c $loopDevice
sudo nsenter --mount=/run/snapd/ns/lxd.mnt /snap/lxd/current/bin/btrfs filesystem resize max /var/snap/lxd/common/lxd/storage-pools/$volume
20

Ohk. Is it corruption of host SSD? Or just the filesystem for this container?