"lxd init" fails to find zfs tool

I decided to try to get LXD going on my main home server rather than just on my laptop, mainly because said server has over a terabyte of unallocated disk space. It’s running Debian 9 (stretch), fully updated, and I just installed snapd and zfsutils-linux from stretch last night, then installed LXD using snap install lxd. But lxd init fails to find zfs:

$ sudo /snap/bin/lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]: lxd
Name of the storage backend to use (btrfs, ceph, dir, lvm, zfs) [default=zfs]:
Create a new ZFS pool? (yes/no) [default=yes]:
Would you like to use an existing block device? (yes/no) [default=no]: yes
Path to the existing block device: /dev/system/lxd-zfs
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML “lxd init” preseed to be printed? (yes/no) [default=no]:
Error: Failed to create storage pool ‘lxd’: the “zfs” tool is not enabled
$ which zfs
/sbin/zfs
$ sudo env | grep ^PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Kernel 4.9.0-3-amd64, amd64 userspace, LXD 3.0.0. Is this due to snap confinement forbidding lxd from being aware that zfs exists on $PATH, or something like that?

The lxd executable sees a different $PATH.
Try

snap run --shell lxd

and you get into the shell of the lxd snap.
The zfs tool is in fact at /snap/lxd/current//zfs-0.7/bin/zfs.

The function that does the check is at https://github.com/lxc/lxd/blob/master/lxd/storage_zfs_utils.go#L16 and https://github.com/lxc/lxd/blob/master/lxd/storage_zfs.go#L49

It is more likely that your Debian kernel does not have ZFS kernel support (does it have the FUSE version of ZFS?). It is not clear why you get that exact error message.

It seems that that exact error message can only happen if exec.LookPath("zfs") fails, so I’m not sure I see how it could possibly be a failure in kernel support. And no, I’ve installed zfs-dkms, and my kernel has native ZFS support as a result:

$ lsmod | grep zfs
zfs 2707456 0
zunicode 331776 1 zfs
zavl 16384 1 zfs
zcommon 53248 1 zfs
znvpair 90112 2 zcommon,zfs
spl 98304 3 znvpair,zcommon,zfs
$ grep zfs /proc/filesystems
nodev zfs

Indeed, there are no indications that’s a kernel issue.

The zfs executables in the snap are

$ tree /snap/lxd/current/zfs-0.*
zfs-0.6
├── bin
│   ├── zfs
│   └── zpool
└── lib
    ├── libnvpair.so -> libnvpair.so.1.0.1
    ├── libnvpair.so.1 -> libnvpair.so.1.0.1
    ├── libnvpair.so.1.0.1
    ├── libuutil.so -> libuutil.so.1.0.1
    ├── libuutil.so.1 -> libuutil.so.1.0.1
    ├── libuutil.so.1.0.1
    ├── libzfs_core.so -> libzfs_core.so.1.0.0
    ├── libzfs_core.so.1 -> libzfs_core.so.1.0.0
    ├── libzfs_core.so.1.0.0
    ├── libzfs.so -> libzfs.so.2.0.0
    ├── libzfs.so.2 -> libzfs.so.2.0.0
    ├── libzfs.so.2.0.0
    ├── libzpool.so -> libzpool.so.2.0.0
    ├── libzpool.so.2 -> libzpool.so.2.0.0
    └── libzpool.so.2.0.0
zfs-0.7
├── bin
│   ├── zfs
│   └── zpool
└── lib
    ├── libnvpair.so -> libnvpair.so.1.0.1
    ├── libnvpair.so.1 -> libnvpair.so.1.0.1
    ├── libnvpair.so.1.0.1
    ├── libuutil.so -> libuutil.so.1.0.1
    ├── libuutil.so.1 -> libuutil.so.1.0.1
    ├── libuutil.so.1.0.1
    ├── libzfs_core.so -> libzfs_core.so.1.0.0
    ├── libzfs_core.so.1 -> libzfs_core.so.1.0.0
    ├── libzfs_core.so.1.0.0
    ├── libzfs.so -> libzfs.so.2.0.0
    ├── libzfs.so.2 -> libzfs.so.2.0.0
    ├── libzfs.so.2.0.0
    ├── libzpool.so -> libzpool.so.2.0.0
    ├── libzpool.so.2 -> libzpool.so.2.0.0
    └── libzpool.so.2.0.0

4 directories, 34 files

When lxd runs, it eventually runs the script /snap/lxd/current/commands/lxd which adds zfs into the $PATH (the one in the snap, not the system $PATH):

# Make sure we have a ZFS binary on the path
export LD_LIBRARY_PATH="${SNAP_CURRENT}/zfs-0.7/lib/:${LD_LIBRARY_PATH}"
export PATH="${SNAP_CURRENT}/zfs-0.7/bin:${PATH}"

It is likely something specific to Debian and the packaging of the LXD snap.
Someone else can have a deeper look into this.

Did you perhaps install LXD before you installed the zfs kernel module on your system?

If so, that’d explain what you’re seeing. LXD detects the ZFS version on daemon startup and alters its PATH and LD_LIBRARY_PATH to pick the matching version of the tools.

That’d explain why lxd init's own detection finds zfs to be present but the daemon itself doesn’t. If that’s the case, running systemctl reload snap.lxd.daemon should solve it.

(This whole craziness is needed because userspace and kernelspace must match with ZFS… we can’t use the 0.7 tools with the 0.6 kernel or the other way around)

Aha! Good call, that was indeed the sequence of events now that I check my shell history, and restarting the daemon indeed fixes it. Thanks!

Glad that it was just that, sorry for the slightly suboptimal user experience there…