Error: Storage pool "lxdpool" unavailable on this server after update to LXD 5.6

LXD
1

After snap updated to LXD 5.6 I get error message:

Error: Storage pool “lxdpool” unavailable on this server

Some background information about my (LXD) setup:
I had some years ago, trouble with my ubuntu installation and needed a new installation of ubuntu. I used LXD with zfs. I had some issues to reuse my existing ZFS-pool. So I created a subpool that is now used for all new created containers and I don’t know how i was also able to import old containers from original pool. But in the end I was able and everything works (sorry I’m far away from beeing a linux expert and it’s some years ago so I don’t know details how I was able).

I have now overall 5 container, 3 “old” on lxdpool (fhem, samba, webserver) and two on lxd-subpool (lxdtest1, lxdtest4)

 lxc storage list
 +-------------+--------+-----------------+-------------+---------+-------------+
 |    NAME     | DRIVER |     SOURCE      | DESCRIPTION | USED BY |    STATE    |
 +-------------+--------+-----------------+-------------+---------+-------------+
 | lxd-subpool | zfs    | lxdpool/subpool |             | 42      | CREATED     |
 +-------------+--------+-----------------+-------------+---------+-------------+
 | lxdpool     | zfs    | lxdpool         |             | 314     | UNAVAILABLE |
 +-------------+--------+-----------------+-------------+---------+-------------+

I can start lxdtest1 and lxdtest4 without issues. But I’m not able to start any container that is located on lxdpool. That make sense as lxdpool seems to be not availible for lxc (since today or yesterday). For me it looks like it happens after snap changed LXD version to LXD 5.6

So the the problem seems that the original pool is now not availible for lxd, but I can see it with e.g.

sudo zfs list
NAME                                                                                              USED  AVAIL     REFER  MOUNTPOINT
lxdpool                                                                                          3,80T  3,33T     95,5G  /var/lib/snapd/hostfs/lxdpool
lxdpool/containers                                                                               3,70T  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/containers
lxdpool/containers/apache                                                                         135M  3,33T      454M  /var/snap/lxd/common/lxd/storage-pools/lxdpool/containers/apache
lxdpool/containers/fhem                                                                          5,72G  3,33T     2,64G  legacy
lxdpool/containers/lxdtest5                                                                       135M  3,33T      711M  /var/snap/lxd/common/lxd/storage-pools/lxdpool/containers/lxdtest5
lxdpool/containers/samba                                                                         2,96T  3,33T     2,79T  legacy
lxdpool/containers/webserver                                                                      747G  3,33T      705G  legacy
lxdpool/custom                                                                                     96K  3,33T       96K  none
lxdpool/deleted                                                                                  1,45G  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/deleted
lxdpool/deleted/containers                                                                         96K  3,33T       96K  none
lxdpool/deleted/custom                                                                             96K  3,33T       96K  none
lxdpool/deleted/images                                                                           1,45G  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/deleted/images
lxdpool/deleted/images/51a86a09a79d1c88c1f9956c10a61c0e2be56a3a50a12cd6749fa47d987c2cba           418M  3,33T      418M  none
lxdpool/deleted/images/683cdd3938706deeb66f19854ee27ef0c75e2594ce60ad4525357c3ce46f9772           423M  3,33T      423M  none
lxdpool/deleted/images/a19c9ae2bd2e7bf99b0e2d31a0707cc534781a4eba47f44f172f486d2e01c96b           641M  3,33T      641M  none
lxdpool/deleted/virtual-machines                                                                   96K  3,33T       96K  none
lxdpool/images                                                                                   2,32G  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/images
lxdpool/images/745354737c38034c6133f2a4b9d7e40d703b8b497e2788d2cf8724e8532e8ddd                   361M  3,33T      361M  /var/snap/lxd/common/lxd/storage-pools/lxdpool/images/745354737c38034c6133f2a4b9d7e40d703b8b497e2788d2cf8724e8532e8ddd
lxdpool/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661                   371M  3,33T      371M  /var/snap/lxd/common/lxd/storage-pools/lxdpool/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661
lxdpool/images/c70385b7a7be39d7a78b354e4f619c63bfcebe7fcedf73a30d8dd471f1262f7e                  1,60G  3,33T     1,60G  /var/snap/lxd/common/lxd/storage-pools/lxdpool/images/c70385b7a7be39d7a78b354e4f619c63bfcebe7fcedf73a30d8dd471f1262f7e
lxdpool/snapshots                                                                                 488K  3,33T      104K  /var/lib/snapd/hostfs/lxdpool/snapshots
lxdpool/snapshots/fhem                                                                             96K  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/snapshots/fhem
lxdpool/snapshots/lxdtest4                                                                         96K  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/snapshots/lxdtest4
lxdpool/snapshots/samba                                                                            96K  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/snapshots/samba
lxdpool/snapshots/webserver                                                                        96K  3,33T       96K  /var/lib/snapd/hostfs/lxdpool/snapshots/webserver
lxdpool/subpool                                                                                  4,18G  3,33T       96K  legacy
lxdpool/subpool/buckets                                                                            96K  3,33T       96K  legacy
lxdpool/subpool/containers                                                                       3,33G  3,33T       96K  legacy
lxdpool/subpool/containers/lxdtest1                                                              1,15G  3,33T      631M  legacy
lxdpool/subpool/containers/lxdtest4                                                              2,18G  3,33T     1,43G  legacy
lxdpool/subpool/custom                                                                             96K  3,33T       96K  legacy
lxdpool/subpool/deleted                                                                           866M  3,33T       96K  legacy
lxdpool/subpool/deleted/buckets                                                                    96K  3,33T       96K  legacy
lxdpool/subpool/deleted/containers                                                                 96K  3,33T       96K  legacy
lxdpool/subpool/deleted/custom                                                                     96K  3,33T       96K  legacy
lxdpool/subpool/deleted/images                                                                    865M  3,33T       96K  legacy
lxdpool/subpool/deleted/images/028d045b1cfcfc8a69cc68674557bd86e015c0ba4bb5c3d6851043f785963728   421M  3,33T      421M  none
lxdpool/subpool/deleted/images/95c0e536d361eb5ac953ad343e0342c2f615e4aea714ca8a64126a228b809cae   444M  3,33T      444M  /var/snap/lxd/common/lxd/storage-pools/lxd-subpool/images/95c0e536d361eb5ac953ad343e0342c2f615e4aea714ca8a64126a228b809cae
lxdpool/subpool/deleted/virtual-machines                                                           96K  3,33T       96K  legacy
lxdpool/subpool/images                                                                             96K  3,33T       96K  legacy
lxdpool/subpool/snapshots                                                                          96K  3,33T       96K  none
lxdpool/subpool/virtual-machines                                                                   96K  3,33T       96K  legacy
lxdpool/virtual-machines

I hope I entered my issue in a correct way and someone can help me or give me some advice. Thanks in advance.

PS: I tried “snap revert lxd” but than I wasn’t able to start any container (with different failure message). So I switched back to latest LXD version.

2

Please can you show the contents of /var/snap/lxd/common/lxd/logs/lxd.log

3

I restarted the host and used command:

~$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
time="2022-09-29T18:18:47+02:00" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
time="2022-09-29T18:18:49+02:00" level=error msg="Failed mounting storage pool" err="Failed to run: zfs set setuid=on exec=on devices=on acltype=posixacl xattr=sa atime=off relatime=on mountpoint=legacy lxdpool: exit status 1 (cannot unmount '/var/lib/snapd/hostfs/var/lib/snapd/hostfs/lxdpool': pool or dataset is busy)" pool=lxdpool
time="2022-09-29T18:19:50+02:00" level=error msg="Failed mounting storage pool" err="Failed to run: zfs set exec=on devices=on acltype=posixacl xattr=sa atime=off relatime=on mountpoint=legacy setuid=on lxdpool: exit status 1 (cannot unmount '/var/lib/snapd/hostfs/var/lib/snapd/hostfs/lxdpool': pool or dataset is busy)" pool=lxdpool
time="2022-09-29T18:20:50+02:00" level=error msg="Failed mounting storage pool" err="Failed to run: zfs set relatime=on mountpoint=legacy setuid=on exec=on devices=on acltype=posixacl xattr=sa atime=off lxdpool: exit status 1 (cannot unmount '/var/lib/snapd/hostfs/var/lib/snapd/hostfs/lxdpool': pool or dataset is busy)" pool=lxdpool
time="2022-09-29T18:21:50+02:00" level=error msg="Failed mounting storage pool" err="Failed to run: zfs set acltype=posixacl xattr=sa atime=off relatime=on mountpoint=legacy setuid=on exec=on devices=on lxdpool: exit status 1 (cannot unmount '/var/lib/snapd/hostfs/var/lib/snapd/hostfs/lxdpool': pool or dataset is busy)" pool=lxdpool
4

Hello,

it seems changing mountpoint of lxdpool to “legacy” fixes the issue:

I wrote in starting post:

In difference, working subpool has mountpoint

So I changed mountpoint to legacy and my containers starts some moment later.

It was required to use the following commands to set lxdpool to legacy mountpoint:

sudo zfs unmount lxdpool
sudo zfs set mountpoint=legacy lxdpool

I’m not sure if this issue is somehow related to LXD 5.6 update, but I would expect not. On the other hand I do not know if or why this mountpoint was changed. Maybe a developer or more experienced user can explain.

5

So LXD 5.6 introduced this change:

Which initialises any missing datasets and sets up the pool’s top-level policy on mount (pool start up).

It sounds like because your pool didn’t have the matching policy of legacy (which disables the unnecessary auto mount of the top level dataset) that ZFS tried to unmount it and this seemed to causes issues (perhaps because you have running instances on that pool).

It may also be due to this snap/zfs issue:

In that sometimes after a snap refresh the mount tables inside the snap namespace can get out of sync with the host and this can confuse the ZFS tool run inside the snap to think something is/isn’t mounted when it infact is.

In these situations I would suggest rebooting the host to allow the snap mount table to be brought into sync.

I have tested various scenarios on mount the ZFS datasets in different orders, and it does appear you can get ZFS mounts confused if you, say, start a container, and then mount the zfs pool’s top level dataset and then try and stop the container and the container’s path no longer is available in the host’s mount namespace as its been over mounted.

So in general I would say you have done the correct thing by bringing all of your ZFS pool datasets into legacy mode (so there is no mounting except that done explicitly by LXD).

6

I’ve sent this patch in to help in situations where the ZFS pool has old mountpoints set but with active instances running: