Containers on storage-pool won't start

LXD
1

I forgot to clear out some btrfs snapshots yesterday and the root volume filled up (~400MB left), this has happened many times before on 2.x. I cleaned up snapshots and rebooted the server, but now any containers with a disk in a storage-pool won’t start saying that it can’t find them, but they are there. I’m running LXD in a snap.

t=2019-10-12T09:05:09-0800 lvl=info msg="LXD 3.18 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-10-12T09:05:10-0800 lvl=eror msg="Failed to start container 'scrypt': Common start logic: Failed to start device 'scrypt_blockchains': Source path /var/snap/lxd/common/lxd/storage-pools/hdd/custom/scrypt_blockchains doesn't exist for device scrypt_blockchains"   
t=2019-10-12T09:05:20-0800 lvl=eror msg="Failed to start container 'x11': Common start logic: Failed to start device 'x11_blockchains': Source path /var/snap/lxd/common/lxd/storage-pools/hdd/custom/x11_blockchains doesn't exist for device x11_blockchains"               
t=2019-10-12T09:05:25-0800 lvl=eror msg="Failed to start container 'neoscrypt': Common start logic: Failed to start device 'neoscrypt_blockchains': Source path /var/snap/lxd/common/lxd/storage-pools/hdd/custom/neoscrypt_blockchains doesn't exist for device neoscrypt_blockchains"

Here is the storage-pool:

# ls -lh /var/snap/lxd/common/lxd/storage-pools/hdd/custom/
total 0
drwxr-xr-x 1  1001001  1001001  90 Nov 24  2018 betelgeuse_data
drwxrwxrwx 1  1000000  1000000 218 Nov 24  2018 mythtv_data
drwxr-xr-x 1  1001000  1001000  24 Nov 23  2018 neoscrypt_blockchains
drwxrws--- 1 www-data www-data 202 Nov 24  2018 owncloud_data
drwxr-xr-x 1  1001000  1001000 114 Nov 23  2018 scrypt_blockchains
drwxr-xr-x 1  1001000  1001000 174 Jun 22 17:11 sha256_blockchains
drwxr-xr-x 1  1000000  1000000   0 Nov 23  2018 vpntest_storage
drwxr-xr-x 1  1001000  1001000  48 Nov 23  2018 x11_blockchains

I not sure if I have db corruption or what and can’t find the lxd.db file as mentioned it other threads.

Any containers not using this storage pool start up just fine.

Please advise.

2

Maybe this isn’t related to filling up. I took another snapshot, then removed everything in /var/snap/lxd/common/lxd/ and copied (reflink) it over from my snapshot earlier this morning before the file system filled up and I was able to start all the containers. I then rebooted to make sure everything would be okay and I’m back in the same situation.

3

Okay, so I put in /etc/fstab to mount /mnt/storage/lxd to /var/snap/lxd/common/lxd/storage-pools/hdd. When I umount /var/snap/lxd/common/lxd/storage-pools/hdd it was able to start all the containers. I ran:

+------+-------------+--------+---------------------------------------------+---------+
| NAME | DESCRIPTION | DRIVER |                   SOURCE                    | USED BY |
+------+-------------+--------+---------------------------------------------+---------+
| hdd  |             | btrfs  | /mnt/storage/lxd                            | 703     |
+------+-------------+--------+---------------------------------------------+---------+
| nvme |             | btrfs  | /var/snap/lxd/common/lxd/storage-pools/nvme | 128     |
+------+-------------+--------+---------------------------------------------+---------+

So it looks like it’s picking up the right place now (I think I did the mount because previous versions didn’t like storage pools outside of the namespace(?)).

I commented out the mount and rebooted and it came up this time.

Now I need to check that may backup scripts don’t break with this change.