Mount location for additional storage pools

LXD
1

I’m trying to use an encrypted filesystem for the storage pool but after a reboot, it defaults back to the snap directory /var/snap/lxd/common/lxd/storage-pools/cryptpool/containers/. After some trial and error, it works this way with a regular directory. This is similar to the Storage pool not mounted? question last week.

First my environment since it may be relevant.
openSUSE Leap 42.3
Kernel 4.4.87-25-default
snapd 2.27.6-1.1
lxd 2.18 (installed via snap)

I’m using the “dir” device because this will eventually reside on an encrypted device. ZFS doesn’t work on openSUSE (module not found?), I’m not going to deal with out of disk space issues from btrfs metadata and I don’t know CEPH. So dir it is.

Let’s create a storage pool in /srv. This discussion indicates snap’s sandboxing of lxc is at fault.

# lxc storage create cryptpool dir source=/srv/lxc
error: mkdir /srv/lxc: read-only file system

Ok, so let’s try a directory in /var/snap.

# lxc storage create cryptpool dir source=/var/snap/cryptdata/storage/
# lxc storage list
+-----------+-------------+--------+------------------------------------------------+---------+
|   NAME    | DESCRIPTION | DRIVER |                     SOURCE                     | USED BY |
+-----------+-------------+--------+------------------------------------------------+---------+
| cryptpool |             | dir    | /var/snap/cryptdata/storage/                   | 1       |
+-----------+-------------+--------+------------------------------------------------+---------+
| default   |             | dir    | /var/snap/lxd/common/lxd/storage-pools/default | 0       |
+-----------+-------------+--------+------------------------------------------------+---------+

Looks good so far. Let’s launch an instance.

# lxc launch images:opensuse/42.3 test-cryptpool
Creating test-cryptpool
Starting test-cryptpool
# lxc stop test-cryptpool
# lxc list
+-----------------+---------+------+------+------------+-----------+
|      NAME       |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------+---------+------+------+------------+-----------+
| test-cryptpool  | STOPPED |      |      | PERSISTENT | 0         |
+-----------------+---------+------+------+------------+-----------+

Reboot and launch another instance.

# lxc launch images:opensuse/42.3 test-cryptpool2
Creating test-cryptpool2
Starting test-cryptpool2
# lxc stop test-cryptpool2
# lxc list
+-----------------+---------+------+------+------------+-----------+
|      NAME       |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+-----------------+---------+------+------+------------+-----------+
| test-cryptpool  | STOPPED |      |      | PERSISTENT | 0         |
+-----------------+---------+------+------+------------+-----------+
| test-cryptpool2 | STOPPED |      |      | PERSISTENT | 0         |
+-----------------+---------+------+------+------------+-----------+

Looks like the second container was stored in /var/snap/cryptdata/storage.

# lxc storage list
+-----------+-------------+--------+------------------------------------------------+---------+
|   NAME    | DESCRIPTION | DRIVER |                     SOURCE                     | USED BY |
+-----------+-------------+--------+------------------------------------------------+---------+
| cryptpool |             | dir    | /var/snap/cryptdata/storage/                   | 3       |
+-----------+-------------+--------+------------------------------------------------+---------+
| default   |             | dir    | /var/snap/lxd/common/lxd/storage-pools/default | 0       |
+-----------+-------------+--------+------------------------------------------------+---------+

Everything looks good. Let’s see where the containers are actually stored. Whoops. The container after the reboot resides in a different location.

# ls -ld /var/snap/cryptdata/storage/containers/* /var/snap/lxd/common/lxd/storage-pools/*/containers/*
drwxr-xr-x+ 4 1000000 1000000 77 Oct 12 15:59 /var/snap/cryptdata/storage/containers/test-cryptpool
drwxr-xr-x+ 4 1000000 1000000 77 Oct 12 16:02 /var/snap/lxd/common/lxd/storage-pools/cryptpool/containers/test-cryptpool2

I turned on debug logging to see what’s happening. Here’s the lxd.log.

ephemeral=false lvl=info msg="Creating container" name=test-cryptpool2 t=2017-10-12T20:02:15+0000
lvl=dbug msg="Initializing a DIR driver." t=2017-10-12T20:02:15+0000
ephemeral=false lvl=info msg="Created container" name=test-cryptpool2 t=2017-10-12T20:02:15+0000
lvl=dbug msg="Creating DIR storage volume for container \"test-cryptpool2\" on storage pool \"cryptpool\"." t=2017-10-12T20:02:15+0000
lvl=dbug msg="Shifting root filesystem \"/var/snap/lxd/common/lxd/containers/test-cryptpool2/rootfs\" for \"test-cryptpool2\"." t=2017-10-12T20:02:16+0000
lvl=dbug msg="Created DIR storage volume for container \"test-cryptpool2\" on storage pool \"cryptpool\"." t=2017-10-12T20:02:17+0000
lvl=dbug msg="Success for task operation: 29f1b72a-f9c0-463b-bd28-2c3ec27486c7" t=2017-10-12T20:02:17+0000
ip=@ lvl=dbug method=GET msg=handling t=2017-10-12T20:02:17+0000 url=/1.0/containers/test-cryptpool2

Why is it “Shifting root filesystem”? Why doesn’t lxc/lxd create the container in the specified location? Do I need to specify /var/snap/lxd/common/lxd/storage-pools/cryptpool as the “dir” when creating the storage?

Any insight is greatly appreciated.

2

“Shifiting root filesystem” is about uid/gid changes on initial container startup, that part is fine.

The rest I’m pretty sure can all be explained by a bug we’re currently working on:

3

Yup, that’s it. Will LXD bind-mount all storage pools if needed when it’s restarted? Since I plan to have the underlying storage encrypted, it will not be available during boot. The storage will be unlocked sometime after boot, either manually or automatically.

4

Yeah, LXD will do the mount whenever it’s started if it’s not already in place, so your script can do the crypsetup dance, then unmounts any existing bind-mount and then restart lxd. That should work fine once the bug above is fixed.