(Snap) LXD cannot Resize default BTRFS storage pool

cesar-ayuuk · November 30, 2020, 4:32pm

Hello,

I was trying to resize my storage pool (from 100G to 900G) following this topic (Snap) LXD Resize default BTRFS storage pool . However, i neither see my pool is resizing nor cannot start my containers anymore.

lxd version is 4.8

lxc storage list:

default | | btrfs | /var/snap/lxd/common/lxd/disks/default.img

lxc storage show default

config:
size: 100GB
source: /var/snap/lxd/common/lxd/disks/default.img
description: “”
name: default
driver: btrfs
used_by:

/1.0/images/some-images
/1.0/instances/container-name
/1.0/instances/container-name
/1.0/instances/nextcloud
/1.0/profiles/default
status: Created
locations:
none

lxc start container

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart nextcloud /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/nextcloud/lxc.conf:
Try lxc info --show-log nextcloud for more

lxc info --show-log nextcloud

Log:

lxc nextcloud 20201130162534.547 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.monitor.nextcloud”
lxc nextcloud 20201130162534.549 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.payload.nextcloud”
lxc nextcloud 20201130162534.556 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1573 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc nextcloud 20201130162534.667 ERROR start - start.c:start:2089 - Input/output error - Failed to exec “/sbin/init”
lxc nextcloud 20201130162534.668 ERROR sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 7)
lxc nextcloud 20201130162534.672 WARN network - network.c:lxc_delete_network_priv:3185 - Failed to rename interface with index 0 from “eth0” to its initial name “vethc19948a5”
lxc nextcloud 20201130162534.672 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:860 - Received container state “ABORTING” instead of “RUNNING”
lxc nextcloud 20201130162534.672 ERROR start - start.c:__lxc_start:1999 - Failed to spawn container “nextcloud”
lxc nextcloud 20201130162534.672 WARN start - start.c:lxc_abort:1013 - No such process - Failed to send SIGKILL via pidfd 30 for process 10025
lxc 20201130162534.841 WARN commands - commands.c:lxc_cmd_rsp_recv:126 - Connection reset by peer - Failed to receive response for command “get_state”

Thanks in advance!

stgraber · November 30, 2020, 5:12pm

Look at dmesg, it sounds like your kernel isn’t happy.

cesar-ayuuk · November 30, 2020, 5:43pm

Thanks for the quick reply @stgraber.

This is the log from dmesg:

[ 136.972745] loop15: rw=0, want=128459968, limit=128456328
[ 136.972755] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 88, rd 17, flush 0, corrupt 0, gen 0
[ 136.972824] BTRFS error (device loop15): error reading free space cache
[ 136.972833] BTRFS warning (device loop15): failed to load free space cache for block group 55864983552, rebuilding it now
[ 136.974353] BTRFS error (device loop15): csum mismatch on free space cache
[ 136.974361] BTRFS warning (device loop15): failed to load free space cache for block group 59086209024, rebuilding it now
[ 136.975910] BTRFS info (device loop15): the free space cache file (64454918144) is invalid, skip it
[ 136.979341] attempt to access beyond end of device
[ 136.979343] loop15: rw=2049, want=130379152, limit=128456328
[ 136.979345] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 89, rd 17, flush 0, corrupt 0, gen 0
[ 136.979543] attempt to access beyond end of device
[ 136.979544] loop15: rw=2049, want=130379664, limit=128456328
[ 136.979546] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 90, rd 17, flush 0, corrupt 0, gen 0
[ 136.979942] attempt to access beyond end of device
[ 136.979944] loop15: rw=2049, want=130379152, limit=128456328
[ 136.979945] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 91, rd 17, flush 0, corrupt 0, gen 0
[ 136.980078] attempt to access beyond end of device
[ 136.980078] loop15: rw=2049, want=130379664, limit=128456328
[ 136.980079] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 92, rd 17, flush 0, corrupt 0, gen 0
[ 136.980249] attempt to access beyond end of device
[ 136.980250] loop15: rw=2049, want=130380176, limit=128456328
[ 136.980251] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 93, rd 17, flush 0, corrupt 0, gen 0
[ 136.981836] attempt to access beyond end of device
[ 136.981837] loop15: rw=2049, want=130379152, limit=128456328
[ 136.981839] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 94, rd 17, flush 0, corrupt 0, gen 0
[ 136.981990] attempt to access beyond end of device
[ 136.981991] loop15: rw=2049, want=130380176, limit=128456328
[ 136.981992] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 95, rd 17, flush 0, corrupt 0, gen 0
[ 136.982121] attempt to access beyond end of device
[ 136.982122] loop15: rw=2049, want=130379152, limit=128456328
[ 136.982123] BTRFS error (device loop15): bdev /dev/loop15 errs: wr 96, rd 17, flush 0, corrupt 0, gen 0

I’m not sure how to proceed with this?

stgraber · November 30, 2020, 6:24pm

What does df -h look like on your system?

cesar-ayuuk · November 30, 2020, 6:39pm

df -h:
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 2.0M 1.6G 1% /run
/dev/mapper/vgubuntu-root 1.2T 27G 1.1T 3% /
tmpfs 7.8G 183M 7.6G 3% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/loop2 243M 243M 0 100% /snap/chromium/1411
/dev/loop3 65M 65M 0 100% /snap/gtk-common-themes/1514
/dev/loop1 56M 56M 0 100% /snap/core18/1885
/dev/loop0 241M 241M 0 100% /snap/chromium/1399
/dev/loop5 31M 31M 0 100% /snap/snapd/9721
/dev/loop6 52M 52M 0 100% /snap/snap-store/498
/dev/loop4 70M 70M 0 100% /snap/lxd/18402
/dev/loop7 56M 56M 0 100% /snap/core18/1932
/dev/loop8 256M 256M 0 100% /snap/gnome-3-34-1804/36
/dev/loop9 50M 50M 0 100% /snap/snap-store/467
/dev/loop10 65M 65M 0 100% /snap/gtk-common-themes/1513
/dev/loop11 218M 218M 0 100% /snap/gnome-3-34-1804/60
/dev/loop12 70M 70M 0 100% /snap/lxd/18324
/dev/loop13 31M 31M 0 100% /snap/snapd/9607
/dev/sda1 511M 7.8M 504M 2% /boot/efi
/dev/loop14 163M 163M 0 100% /snap/gnome-3-28-1804/145
tmpfs 1.0M 0 1.0M 0% /var/snap/lxd/common/ns
tmpfs 1.6G 32K 1.6G 1% /run/user/1000

and lsblk if needed:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 240.8M 1 loop /snap/chromium/1399
loop1 7:1 0 55.3M 1 loop /snap/core18/1885
loop2 7:2 0 243M 1 loop /snap/chromium/1411
loop3 7:3 0 64.8M 1 loop /snap/gtk-common-themes/1514
loop4 7:4 0 69.8M 1 loop /snap/lxd/18402
loop5 7:5 0 31M 1 loop /snap/snapd/9721
loop6 7:6 0 51M 1 loop /snap/snap-store/498
loop7 7:7 0 55.4M 1 loop /snap/core18/1932
loop8 7:8 0 255.6M 1 loop /snap/gnome-3-34-1804/36
loop9 7:9 0 49.8M 1 loop /snap/snap-store/467
loop10 7:10 0 64.4M 1 loop /snap/gtk-common-themes/1513
loop11 7:11 0 217.9M 1 loop /snap/gnome-3-34-1804/60
loop12 7:12 0 69.4M 1 loop /snap/lxd/18324
loop13 7:13 0 31M 1 loop /snap/snapd/9607
loop14 7:14 0 162.9M 1 loop /snap/gnome-3-28-1804/145
loop15 7:15 0 348.3G 0 loop
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
└─sda2 8:2 0 238G 0 part
├─vgubuntu-root 253:0 0 1.1T 0 lvm /
└─vgubuntu-swap_1 253:1 0 976M 0 lvm [SWAP]
sdb 8:16 0 931.5G 0 disk
└─sdb1 8:17 0 931.5G 0 part
└─vgubuntu-root 253:0 0 1.1T 0 lvm /

stgraber · November 30, 2020, 6:46pm

Ok, what does ls -lh /var/snap/lxd/common/lxd/disks/default.img show at this stage?

If the file is properly 900GB large already, then your best bet may be to reboot the system, that should clear any kernel confusion about the size of the loop device.

The lsblk output doesn’t show a 900GB large loop device so it’s making me think that something didn’t behave during that part of the resize.

cesar-ayuuk · November 30, 2020, 7:04pm

ls -lh /var/snap/lxd/common/lxd/disks/default.img:

-rw------- 1 root root 349G Nov 30 13:34 /var/snap/lxd/common/lxd/disks/default.img

I rebooted the machine like 5 times and the issue remains.

stgraber · November 30, 2020, 7:07pm

Okay, so you said you were going for 900GB but the backing file is just 349GB right now.
So let’s try fixing that first with:

truncate -s 900G /var/snap/lxd/common/lxd/disk/default.img
ls -lh /var/snap/lxd/common/lxd/disk/default.img
losetup -a

stgraber · November 30, 2020, 7:08pm

When that looks good, we can force a loop refresh and then look at btrfs.

cesar-ayuuk · November 30, 2020, 7:58pm

~~Now~~ it is as ~~expected~~.
-rw------- 1 ~~root~~ ~~root~~ ~~900G~~ ~~Nov~~ 30 14:48 /var/snap/~~lxd~~/common/~~lxd~~/~~disks~~/~~default~~.~~img~~

stgraber · November 30, 2020, 8:03pm

Ok, what’s the losetup -a output?

cesar-ayuuk · November 30, 2020, 8:05pm

losetup -a
/dev/loop1: [64768]:2884001 (/var/lib/snapd/snaps/chromium_1411.snap)
/dev/loop8: [64768]:2883620 (/var/lib/snapd/snaps/lxd_18324.snap)
/dev/loop15: [64768]:4587611 (/var/snap/lxd/common/lxd/disks/default.img)
/dev/loop6: [64768]:2884086 (/var/lib/snapd/snaps/gnome-3-34-1804_60.snap)
/dev/loop13: [64768]:2883696 (/var/lib/snapd/snaps/snap-store_467.snap)
/dev/loop4: [64768]:2891770 (/var/lib/snapd/snaps/gnome-3-34-1804_36.snap)
/dev/loop11: [64768]:2893003 (/var/lib/snapd/snaps/snap-store_498.snap)
/dev/loop2: [64768]:2885959 (/var/lib/snapd/snaps/core18_1885.snap)
/dev/loop0: [64768]:2883629 (/var/lib/snapd/snaps/chromium_1399.snap)
/dev/loop9: [64768]:2884123 (/var/lib/snapd/snaps/gtk-common-themes_1514.snap)
/dev/loop7: [64768]:2884280 (/var/lib/snapd/snaps/snapd_9607.snap)
/dev/loop14: [64768]:2885032 (/var/lib/snapd/snaps/snapd_9721.snap)
/dev/loop5: [64768]:2883799 (/var/lib/snapd/snaps/gtk-common-themes_1513.snap)
/dev/loop12: [64768]:2883687 (/var/lib/snapd/snaps/lxd_18402.snap)
/dev/loop3: [64768]:2884426 (/var/lib/snapd/snaps/gnome-3-28-1804_145.snap)
/dev/loop10: [64768]:2884439 (/var/lib/snapd/snaps/core18_1932.snap)

stgraber · November 30, 2020, 8:06pm

Run losetup -c /dev/loop15

cesar-ayuuk · November 30, 2020, 8:14pm

@stgraber thanks a lot for your help. Now to resize my btrfs file system i should run:

btrfs filesystem resize max /var/snap/lxd/common/lxd/storage-pools/default

however, when i do ls -lh /var/snap/lxd/common/lxd/storage-pools/default
total0

Or should i run instead btrfs filesystem resize max /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default

where ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default
rwxr-xr-x 1 root root 200 Nov 27 20:01 ./
drwx–x--x 3 root root 4096 Nov 27 20:01 …/
drwx–x--x 1 root root 46 Nov 27 22:30 containers/
drwx–x--x 1 root root 18 Nov 27 22:56 containers-snapshots/
drwx–x--x 1 root root 0 Nov 27 20:01 custom/
drwx–x--x 1 root root 0 Nov 27 20:01 custom-snapshots/
drwx–x--x 1 root root 128 Nov 27 20:01 images/
drwx–x--x 1 root root 0 Nov 27 20:01 virtual-machines/
drwx–x--x 1 root root 0 Nov 27 20:01 virtual-machines-snapshots/

stgraber · November 30, 2020, 8:16pm

Try:

nsenter --mount=/run/snapd/ns/lxd.mnt /snap/current/lxd/bin/btrfs filesystem resize max /var/snap/lxd/common/lxd/storage-pools/default

cesar-ayuuk · November 30, 2020, 8:24pm

It looks like tehre no such directory.
nsenter --mount=/run/snapd/ns/lxd.mnt /snap/current/lxd/bin/btrfs filesystem
nsenter: failed to execute /snap/current/lxd/bin/btrfs: No such file or directory

stgraber · November 30, 2020, 8:24pm

/snap/lxd/current/bin/btrfs, sorry

cesar-ayuuk · November 30, 2020, 8:36pm

That worked! Thanks a lot @stgraber. I was a bit confused when running lxc storage show default and it still shows the 100G but lxc storage info default gives the 900GB.

I really appreciate your help!

I thought i’d be able to start my containers now but it seems i cannot.
lxc start nextcloud
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart nextcloud /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/nextcloud/lxc.conf:
Try lxc info --show-log nextcloud for more info

cesar-ayuuk · November 30, 2020, 9:22pm

It looks like btrfs is still complainig.
[ 1185.125901] btrfs_print_data_csum_error: 131294 callbacks suppressed
[ 1185.125904] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.125966] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126024] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126081] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126138] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126175] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126232] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126269] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126308] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1185.126346] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.129895] btrfs_print_data_csum_error: 137283 callbacks suppressed
[ 1190.129898] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.129932] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.129967] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130001] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130044] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130077] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130110] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130139] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130169] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1190.130202] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.133928] btrfs_print_data_csum_error: 131082 callbacks suppressed
[ 1195.133932] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.133983] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134040] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134094] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134148] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134202] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134257] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134303] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134360] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1195.134415] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.137933] btrfs_print_data_csum_error: 142839 callbacks suppressed
[ 1200.137935] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.137971] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138006] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138040] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138073] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138108] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138141] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138175] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138209] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1200.138244] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.141937] btrfs_print_data_csum_error: 144889 callbacks suppressed
[ 1205.141940] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142000] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142049] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142098] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142146] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142194] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142242] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142287] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142335] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1205.142383] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.145976] btrfs_print_data_csum_error: 141641 callbacks suppressed
[ 1210.145978] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146011] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146053] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146092] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146132] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146171] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146210] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146240] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146271] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1210.146313] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.149998] btrfs_print_data_csum_error: 141085 callbacks suppressed
[ 1215.150001] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150036] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150079] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150118] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150159] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150199] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150237] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150268] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150299] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1215.150341] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154038] btrfs_print_data_csum_error: 141575 callbacks suppressed
[ 1220.154041] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154080] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154118] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154155] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154191] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154227] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154264] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154300] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154337] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1220.154373] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158026] btrfs_print_data_csum_error: 142226 callbacks suppressed
[ 1225.158028] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158129] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158166] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158205] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158244] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158281] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158323] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158353] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158383] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1225.158423] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162053] btrfs_print_data_csum_error: 141275 callbacks suppressed
[ 1230.162056] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162097] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162150] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162199] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162248] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162295] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162345] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162382] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162434] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1230.162483] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166034] btrfs_print_data_csum_error: 139347 callbacks suppressed
[ 1235.166037] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166074] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166109] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166147] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166186] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166224] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166261] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166300] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166338] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1235.166372] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170117] btrfs_print_data_csum_error: 140721 callbacks suppressed
[ 1240.170120] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170169] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170216] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170258] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170300] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170342] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170383] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170424] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170466] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1240.170508] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174088] btrfs_print_data_csum_error: 139637 callbacks suppressed
[ 1245.174091] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174137] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174178] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174217] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174257] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174296] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174335] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174374] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174414] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1245.174453] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178186] btrfs_print_data_csum_error: 141667 callbacks suppressed
[ 1250.178189] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178228] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178266] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178303] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178339] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178376] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178414] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178450] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178487] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1
[ 1250.178524] BTRFS warning (device loop15): csum failed root 259 ino 37148 off 4096 csum 0x8941f998 expected csum 0xe5e907a7 mirror 1

stgraber · November 30, 2020, 9:43pm

lxc storage show will always show the initial requested size as the size property, that’s fine and can be safely ignored. lxc storage info showing the 900GB indicates that btrfs seems correctly configured now.

That being said you clearly have some corruption caused by the original partially broken state which is still causing issue.

Run nsenter --mount=/run/snapd/ns/lxd.mnt /snap/lxc/current/bin/btrfs scrub start /var/snap/lxd/common/lxd/storage-pools/default