Rebooted a server and since then the lxd container is not starting. It is giving an error saying:
Error: Storage pool "vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7" unavailable on this server
Need help on this as I am not able to access this container and the data inside. I don’t want to loose this data. Could you please help on this?
Response of command:
sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
Response:
time="2023-04-24T13:08:58+02:00" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
time="2023-04-24T13:08:58+02:00" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
time="2023-04-24T13:09:12+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:09:23+02:00" level=warning msg="Failed to initialize fanotify, falling back on inotify" err="Failed to initialize fanotify: invalid argument"
time="2023-04-24T13:09:23+02:00" level=error msg="Error getting disk usage" err="Storage pool is unavailable on this server" instance=qblxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7 instanceType=container project=default
time="2023-04-24T13:10:25+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:10:46+02:00" level=error msg="Error getting disk usage" err="Storage pool is unavailable on this server" instance=qblxc-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7 instanceType=container project=default
time="2023-04-24T13:11:39+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:12:52+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:14:06+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:15:19+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:16:33+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:17:47+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:19:00+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:20:14+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:21:27+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:22:41+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
time="2023-04-24T13:23:54+02:00" level=error msg="Failed mounting storage pool" err="Failed to mount \"/dev/loop27\" on \"/var/snap/lxd/common/lxd/storage-pools/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7\" using \"btrfs\": invalid argument" pool=vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7
LXD version: 5.13
Any solutions or suggestions on this team?
Hi, sorry I can’t really help out but perhaps could get the boulder moving…
Could that loop device:
/dev/loop27
be taken over by something else so what does this command output say about /dev/loop27 ?:
losetup -a
However I have a feeling that the losetup output above on the host system is not relevant to LXD.
Is the image still there under this path?:
/var/snap/lxd/common/lxd/disks/
https://linuxcontainers.org/lxd/docs/latest/explanation/storage/#loop-disk
Previously losetup -a was not showing this loop device. So I ran this command:
losetup -L -f --show /var/snap/lxd/common/lxd/disks/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7.img
Then only it showed this:
losetup -a:
/dev/loop1: [66306]:12593121 (/var/lib/snapd/snaps/core22_583.snap)
/dev/loop19: [66306]:12593251 (/var/lib/snapd/snaps/gnome-3-34-1804_90.snap)
/dev/loop27: [66306]:12981309 (/var/snap/lxd/common/lxd/disks/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7.img)
/dev/loop17: [66306]:12593449 (/var/lib/snapd/snaps/gnome-logs_115.snap)
/dev/loop8: [66306]:12593411 (/var/lib/snapd/snaps/gnome-system-monitor_181.snap)
/dev/loop25: [66306]:12593292 (/var/lib/snapd/snaps/gnome-3-38-2004_119.snap)
/dev/loop15: [66306]:12593352 (/var/lib/snapd/snaps/core20_1828.snap)
/dev/loop6: [66306]:12593377 (/var/lib/snapd/snaps/snapd_18933.snap)
/dev/loop23: [66306]:12583841 (/var/lib/snapd/snaps/gnome-characters_741.snap)
/dev/loop13: [66306]:12592968 (/var/lib/snapd/snaps/gtk-common-themes_1535.snap)
/dev/loop4: [66306]:12593200 (/var/lib/snapd/snaps/core18_2714.snap)
/dev/loop21: [66306]:12592934 (/var/lib/snapd/snaps/gnome-system-monitor_178.snap)
/dev/loop11: [66306]:12583051 (/var/lib/snapd/snaps/bare_5.snap)
/dev/loop2: [66306]:12593256 (/var/lib/snapd/snaps/gnome-42-2204_87.snap)
/dev/loop0: [66306]:12592952 (/var/lib/snapd/snaps/gnome-3-38-2004_137.snap)
/dev/loop18: [66306]:12593444 (/var/lib/snapd/snaps/gnome-characters_781.snap)
/dev/loop9: [66306]:12593306 (/var/lib/snapd/snaps/core20_1852.snap)
/dev/loop26: [66306]:12583827 (/var/lib/snapd/snaps/gtk-common-themes_1534.snap)
/dev/loop16: [66306]:12593287 (/var/lib/snapd/snaps/core22_607.snap)
/dev/loop7: [66306]:12592954 (/var/lib/snapd/snaps/core18_2721.snap)
/dev/loop24: [66306]:12583025 (/var/lib/snapd/snaps/gnome-3-34-1804_77.snap)
/dev/loop14: [66306]:12593307 (/var/lib/snapd/snaps/snapd_18596.snap)
/dev/loop5: [66306]:12593333 (/var/lib/snapd/snaps/gnome-calculator_926.snap)
/dev/loop22: [66306]:12593085 (/var/lib/snapd/snaps/gnome-logs_112.snap)
/dev/loop12: [66306]:12593147 (/var/lib/snapd/snaps/lxd_24758.snap)
/dev/loop3: [66306]:12593218 (/var/lib/snapd/snaps/lxd_24643.snap)
/dev/loop20: [66306]:12593090 (/var/lib/snapd/snaps/gnome-calculator_920.snap)
/dev/loop10: [66306]:12593019 (/var/lib/snapd/snaps/gnome-42-2204_68.snap)
Yes, disk is still there on this path.
OK, so with a (new) ZFS loop device I don’t see mounts via losetup, but I do with BTRFS, so it depends on the driver.
I see that loop27 is now mounted as such but is the pool still not available in LXD, and if so were there other logs since this losetup command?
It will be a good idea to get a backup of this IMG,
Yes, I took a backup of this IMG. But the pool is still not available in LXD. I don’t have other logs for losetup.
I also tried recovering btrfs using restore and related commands but nothing resolved it so far.
Is there any way to get back the container data?
I’m not experienced enough to say whether getting the entire container out of the storage pool and into another container is workable, but providing the file system in very messed up, we should be able to get specific data, e.g. MySQL data and logs, or a user’s home directory.
I created a BTRFS loopback storage pool device and a container on it, and was able to mount the IMG and drill down into the file system mount point to get the test file created in the container.
mount -vt btrfs -o loop /var/snap/lxd/common/lxd/disks/{yourdisk}.img /{mount-point}
e.g.
mount -vt btrfs -o loop /var/snap/lxd/common/lxd/disks/btrfsloop1.img /mnt
cp /mnt/containers/btrfstest1/rootfs/root/btrfstest1.txt /home/someuser/restore/
It would be interesting to see whether the above mount will work in your scenario.
Hi! I tried but it didn’t work.
Got this error:
mount: /mnt/backup/restore: wrong fs type, bad option, bad superblock on /dev/loop27, missing codepage or helper program, or other error.
Wonderful, so it sounds like your IMG is damaged or the file system within it is, which is why LXD cannot open it.
What does this command output show?:
file /var/snap/lxd/common/lxd/disks/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7.img
Sadly I don’t have experience with BTRFS so I’m at the end of my usefulness here. Hopefully someone else with BTRFS experience can jump in here.
Perhaps these are useful:
https://markandruth.co.uk/2019/03/16/recovering-from-unmountable-btrfs-filesystem-issues
With this one I’d do this on a different and spare system with a copy of the IMG:
This is the response of the command you provided:
/var/snap/lxd/common/lxd/disks/vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7.img: BTRFS Filesystem label "vol-ce9e2a76-8b0c-437b-81d4-4c4f47f147e7", sectorsize 4096, nodesize 16384, leafsize 16384, UUID=bc75b9c2-25be-4180-a00a-1a4afec0715b, 406211244032/1201999998976 bytes used, 1 devices
Hi. If you haven’t already, I recommend that you find other forums relating to BTRFS which increase your chances of someone knowing more about the file system and help you recover its’ contents.