Is it safe to create an LVM-backed storage pool that can be shared with other logical volumes?

At https://lxd.readthedocs.io/en/latest/storage/

Note that LXD will assume it has full control over the ZFS pool or dataset. It is recommended to not maintain any non-LXD owned filesystem entities in a LXD zfs pool or dataset since LXD might delete them.

I don’t see anything that says that other backends like LVM or BTRFS aren’t subject to this limitation. I assume that’s not the case, but given how bad it would be if lxd started deleting other logical volumes I thought I’d check.

If it is safe to use lxd in existing lvm or btrfs volumes, I’ll take a stab at filing a docs clarification.

Andrew,

From my experience, you can easily re-use an existing BTRFS partition for LXD. Simply spin up a new server, format the target disk (ie: /dev/sdb1) with BTRFS, and then run the lxd init command. During the init command, specify “dir” backend and then mount /dev/sdb1 to /var/snap/lxd/storage-pools/default (or, whatever you called the storage pool name).

Note: After the lxd init command is run, you will need to rename /var/snap/lxd/storage-pools/default to /var/snap/lxd/storage-pools/default.old, make a new directory called /var/snap/lxd/storage-pools/default, manually mount /dev/sdb1 to /var/snap/lxd/storage-pools/default, and then copy the contents of the “old” directory to the new one. Add an entry to /etc/fstab to ensure the volume gets mounted on (re)boot. Probably best to reboot the server once all this is done to make sure everything is working 100%.

Hope this makes sense.

Yes, this does! Have you done the same on LVM by chance?

Not sure about LVM - I don’t use it. But, it should work.

I discovered that you can specify a ZFS pool or a ZFS dataset. If you specify a dataset, it won’t touch the rest of the pool, such as if you destroy the storage pool.

To me, it seems like it’d be better in docs and UI text to suggest specifying a dataset, with whole-pools being the edge case. I’ll see about filing some PRs.

I would recommend against using LVM if shared with non-LXD users, at least if using the snap.

The reason is that the snap includes its own set of LVM management utilities. If management of the LVM volumes is done via LXD and also via the system LVM commands, you can run into inconsistent views of the LVM metadata. (See e.g. https://github.com/lxc/lxd/issues/4445) At least when using lvmetad, which is on by default. I always assumed this was just a cosmetic issue, and also usually did lvscan --cache before using the system LVM commands, but one day my LVM thinpool (which was keeping both LXD volumes as well as other volumes) got corrupted, on hardware that is otherwise completely fine. I’m not sure the interference with LXD’s LVM management was the reason for this, so please take this as purely anecdotal, but it was enough for me to stay away from attempting this again.

(For the record, I managed to recover the data from the thinpool. If anyone finds themselves in the same situation, I’ve recorded my process at https://github.com/traktofon/thinpool-unscramble .)

2 Likes