Seeking for advices to build a serious LXD host

1

Hello everyone,

I’ve been experimenting with LXD in my home server for about two months and I’m happy with it. Currently, my LXD host is Ubuntu 18.04 with everything installed in a single HDD (with LUKS encrypted). Now I’m thinking about serious hardware upgrade and probably gonna replace HDD by SSDs. Here is what i want to achieve:

1) Native ZFS format on a separate drive, just for LXD storage pool
I’m aware of benefits ZFS brings. Having a separate, small SSD just for OS install would grant me flexibility for future upgrade of OS/LXD. However I don’t know what is the best and safest way to do it. Would you use that separate drive just for the block device (when running lxd init)? Or would you mount this drive as entire /var/lib/lxd?
Here is what @brauner suggested :

mount --bind /dev/sdb /var/lib/lxd

But on the other hand, LXC 2.0 documentation says we shouldn’t do that:

ZFS doesn’t have to (and shouldn’t be) mounted on /var/lib/lxd

So at this point i’m pretty clueless about what to do. In the scenario that I re-install OS or change the OS drive, I want to be able to easily get everything up and running again.

2) Redundancy with security
With 2 drives just for ZFS pool, I want to have mirrored vdevs in 1x2 configuration which is similar to RAID1 as lots of suggestion I found on internet. And it should be encrypted.

There are two methods I could think of: ZFS on LUKS and native encryption on ZFS. Native encryption sound great but from information I’ve red, it’s very new and unproven. Also, the lengthy and hacky process for that doesn’t look user-friendly at all. That guide also requires user to compile from source code because current Ubuntu zfsutils-linux package won’t support that feature until version 0.8.0.

So my question is what would you do for a production system?

On a side note, would self-encrypting drive be overall better than software encryption? In the OS drive, /boot partition is still unencrypted and vulnerable to evil maid attack. But I can move LUKS drives to another machine and recover my data, meanwhile SED would require support from motherboard (for ATA password) and proprietary software to achieve full disk encryption (in case of Samsung drives).

In conclusion, I’m looking for strategies/practices to build an LXD system which is secure, redundant and flexible in scenarios like moving ZFS pool to other host, reinstalling OS, expanding ZFS pool…

2

For storage, I’d recommend full-disk LUKS with the resulting volumes in a ZPOOL. If those drivers aren’t SSDs, then you could also carve out some space on your SSD for another LUKS partition to use for ZFS CACHE

As for how to use it with LXD. Just mounting a full zpool on /var/lib/lxd (or /var/snap/lxd/common/lxd if using the snap) is a bad idea and something we have a hard time handling during upgrades.

Instead, you may want to consider doing something like:

  • zpool create external … -m none
  • zfs create external/lxd-data -o mountpoint=/var/lib/lxd
  • zfs create external/lxd-storage -o mountpoint=none

Restart LXD so that it’s loaded using the new storage for its data, and then when running lxd init, you can tell LXD to use ZFS for the storage pool with external/lxd-storage as the dataset it should be using.

That way LXD will be happy with the way it handles storage pools (and you can easily create more of them, set quotas, …) and the LXD data itself is also stored on ZFS but in a separate dataset that you can separately snapshot/backup.

1 Like
3

Hey thanks for your reply. As I understand, you advised against native encryption feature of ZFS and I should proceed with following diagram (which I took from this blog post)?

However with this approach, I would have to unlock each of encrypted drives (either manually or by automated script) every time I restart the server. Would this trigger any problem in some scenarios; e.g. I have an automated script to start my containers at system startup and ZFS/LXD can’t find my zpool because drives isn’t unlocked yet?

If ZFS-on-LUKS is way to go, I would try to build an automated script for unlocking and mounting the drives. This script would reside on OS drive which is also LUKS encrypted, so I wouldn’t have to worry about my encryption passphrase being exposed in plain text.

4

Yeah, cryptsetup can take a file as the decryption key if I remember correctly, so you just need to make that this runs from within the OS LUKS partition before LXD runs, that should be reasonably straightforward to do with the right systemd jobs conditions.