Snap lxd not seeing containers in storage

I’ve tried to migrate from the apt lxd to the new Snap installed lxd, but my containers aren’t being seen by the new lxd. I did:
snap install lxd
lxd migrate # seemed to be successful, removed old apt packages
systemctl stop snap.lxd.daemon
cd /var/snap/lxd/common/
mv lxd lxd.orig
ln -s /home/srv/lxd/ lxd # all my containers are in /home/srv/lxd/, which is on a BTRFS partition
rebooted
systemctl stop snap.lxd.daemon

The problem is now that there’s nothing in my storage pool:
lxc storage show local:default
config:
source: /var/snap/lxd/common/lxd/storage-pools/default
volatile.initial_source: /var/snap/lxd/common/lxd/storage-pools/default
description: “”
name: default
driver: btrfs
used_by:
- /1.0/profiles/default
status: Created
locations:
- none

lxc list local:
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

So for some reason it’s not picking up my containers

Symlinks in general are a very bad idea for a number of reasons when used with LXD.
You should be using a bind-mount instead which tends to be a lot more reliable in such cases.

Anyway, what do you see in /var/snap/lxd/common/lxd/containers?

1 Like

I also just ran into this migration issue and placed the following line in /etc/fstab in order to bind-mount the old LXD storage into the new snap location.

/var/lib/lxd /var/snap/lxd/common/lxd bind defaults,bind

Before mounting it in the new place (/var/snap/lxd/common/lxd) I renamed the folder that came with the snap and created a fresh lxd subfolder. Then used mount /var/snap/lxd/common/lxd.

Thanks both, in the end I decided to just re-create the containers since I wanted to upgrade them to Ubuntu 18.04 anyway.

After running for a day there’s still a bit of a problem. The first time that the snap.lxd.daemon service starts after a reboot it crashes out due to /var/snap/lxd/common/lxd being a symlink. The second time it starts (i.e. systemctl start snap.lxd.daemon) it starts without issue. The logs from the first time after a reboot:

Aug 16 20:10:58 timserver lxd.daemon[6711]: mkdir: cannot create directory '/var/snap/lxd/common/lxd': File exists
Aug 16 20:10:58 timserver systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE

The symlink:

$ ls -l '/var/snap/lxd/common/lxd'
lrwxrwxrwx 1 root root 14 Aug  3 15:30 /var/snap/lxd/common/lxd -> /home/srv/lxd/

Doing bind mounts adds a lot of complexity to a setup that I’d rather avoid. Is there a way instead to edit the storage pool so that everything except storage pools lives under /var/snap/lxd/common/lxd/ (not a symlink) and just the storage-pool (which I assume takes up most of the space) lives on /home/srv/lxd/?

(/home/srv/lxd/ is on a BTRFS partition, whereas /var/snap/lxd/common/lxd/ is on a standard Ext4)

Yes, you could just have configured LXD to use an existing btrfs pool (/home/srv/lxd) during init.

If you’re re-doing your setup from scratch, during init you want to select:

  • Create new storage pool
  • Choose name (doesn’t matter)
  • Choose btrfs
  • Say no to creating a new btrfs pool
  • Enter /home/srv/lxd as the location of the existing pool/dataset (that path shouldn’t exist, LXD will create the subvolume for you)

You can probably re-shuffle things without starting from scratch, but for that you’d need to delete the storage pool with lxc storage delete after removing any container or image on it and removing it from the default profile. Then move /home/srv/lxd to /var/snap/lxd/common/lxd. Then restart LXD and re-add the storage pool with lxc storage create default btrfs source=/home/srv/lxd before adding it back to the default profile.

In the end I went with the bind mount option. I did a lxd init on both and started from scratch. On my Ubuntu 18.04 host machine that all worked fine, but on my 16.04 host the same setup seems to have hit LXD dqlite DB problems. I’ve put all the details in https://github.com/lxc/lxd/issues/4935