Lxd exports ZFS pools at shutdown, but does not import them properly at startup

LXD
1

When I shut down lxd, it exports the ZFS pools, but it does not reimport them at startup:

$ zpool status
no pools available
$ lxc list
Error: Get "http://unix.socket/1.0": dial unix /var/lib/lxd/unix.socket: connect: no such file or directory
$ rc-service lxd status
 * status: crashed
$ zpool import -d /var/lib/zfs_img/ zfs_lxd
$ zpool status
  pool: zfs_lxd
 state: ONLINE
  scan: resilvered 45K in 00:00:00 with 0 errors on Thu Aug 19 07:00:27 2021
config:

NAME                           STATE     READ WRITE CKSUM
zfs_lxd                        ONLINE       0     0     0
  mirror-0                     ONLINE       0     0     0
    /var/lib/zfs_img/zfs0.img  ONLINE       0     0     0
    /var/lib/zfs_img/zfs1.img  ONLINE       0     0     0
  mirror-1                     ONLINE       0     0     0
    /var/lib/zfs_img/zfs2.img  ONLINE       0     0     0
    /var/lib/zfs_img/zfs3.img  ONLINE       0     0     0

errors: No known data errors
$ rc-service lxd restart
 * Stopping lxd service (but not containers) ...
 * start-stop-daemon: no matching processes found                                                             [ ok ]
 * Starting lxd service ...                                                                                   [ ok ]
$ lxc list
+---------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|     NAME      |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+---------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| gentoo-PG-C01 | RUNNING | 10.248.20.110 (eth0) | fd42:9c78:69e0:463f:216:3eff:fe80:df1d (eth0) | CONTAINER | 0         |
+---------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| gentoo-WS-C01 | RUNNING | 10.248.20.100 (eth0) | fd42:9c78:69e0:463f:216:3eff:fea4:13e5 (eth0) | CONTAINER | 1         |
+---------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| ubuntu-PG-C01 | RUNNING | 10.248.20.120 (eth0) | fd42:9c78:69e0:463f:216:3eff:fe8f:3d93 (eth0) | CONTAINER | 0         |
+---------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
$ rc-service lxd stop
 * Stopping lxd service and containers, waiting 40s ...                                                       [ ok ]
$ lxc list
Error: Get "http://unix.socket/1.0": dial unix /var/lib/lxd/unix.socket: connect: no such file or directory
$ zpool status
no pools available
$ rc-service lxd start
 * Starting lxd service ...                                                                                   [ ok ]
$ rc-service lxd status
 * status: crashed
$ lxc list
Error: Get "http://unix.socket/1.0": dial unix /var/lib/lxd/unix.socket: connect: no such file or directory

Startup worked normally until the end of November, when I did some package updates (I’m running Gentoo).

2

Looks like we’ve had that logic for quite some time.

Basically if LXD is given a subset of an existing pool (say tank/lxd), it will not perform a zpool export on shutdown. But if LXD is given a full zpool to work with, either one it created itself or one provided by the user, it will run zpool export on shutdown to ensure everything is nice and consistent.

On startup, LXD runs zpool import -d /var/lib/lxd/disks/ as that’s the path that LXD created loop-backed zpools use for their disks, so zpool import will scan both that path and the regular block devices on the system.

Your setup is uncommon because you’re using a loop backed zpool backed by 4 files which weren’t created by LXD (so aren’t in the path we scan) and as you’re providing the entire pool to LXD, it does get exported on shutdown.

There are two easy ways out of this that I can think of:

  • Put an init job that runs prior to LXD and runs the correct zpool import for your setup
  • Relocate your zpool backing files to /var/lib/lxd/disks. Note that this option isn’t exactly ideal as LXD generally expects that directory to be used for loop-backed pools that it itself manages, so it may get confused by files that don’t line up.
1 Like
3

Is there a way, in configuration, to tell LXD what paths to search for pools?

If not, then that would be a useful feature, and I’ll file an issue for it.

EDIT: Filed lxd-4.0.8 hard codes the ZFS pool path so that it fails to find pools outside that path at startup #9739