Error upgrading zfs

candlerb · November 12, 2021, 10:31am

Doing a normal update on a 20.04 system (with lxd 4.20 from snap), I got the following error:

root@nuc2:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  alsa-ucm-conf cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf libasound2 libasound2-data libnetplan0 libnvpair1linux libtdb1 libuutil1linux libzfs2linux libzpool2linux
  linux-base linux-firmware netplan.io open-vm-tools overlayroot python3-software-properties software-properties-common ubuntu-advantage-tools ufw zfs-zed zfsutils-linux
23 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 standard security update
Need to get 115 MB of archives.
After this operation, 2,854 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
...
Setting up libzfs2linux (0.8.3-1ubuntu12.13) ...
Setting up cloud-init (21.3-1-g6803368d-0ubuntu1~20.04.4) ...
Setting up software-properties-common (0.99.9.8) ...
Setting up libzpool2linux (0.8.3-1ubuntu12.13) ...
Setting up zfsutils-linux (0.8.3-1ubuntu12.13) ...
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Job for zfs-mount.service failed because the control process exited with error code.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
invoke-rc.d: initscript zfs-mount, action "start" failed.
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2021-11-12 09:57:48 GMT; 19ms ago
       Docs: man:zfs(8)
    Process: 1997645 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
   Main PID: 1997645 (code=exited, status=1/FAILURE)

Nov 12 09:57:47 nuc2 systemd[1]: Starting Mount ZFS filesystems...
Nov 12 09:57:48 nuc2 zfs[1997645]: cannot mount '/var/snap/lxd/common/lxd/storage-pools/default': directory is not empty
Nov 12 09:57:48 nuc2 systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Nov 12 09:57:48 nuc2 systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Nov 12 09:57:48 nuc2 systemd[1]: Failed to start Mount ZFS filesystems.
dpkg: error processing package zfsutils-linux (--configure):
 installed zfsutils-linux package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of zfs-zed:
 zfs-zed depends on zfsutils-linux (>= 0.8.3-1ubuntu12.13); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-zed (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Processing triggers for rsyslog (8.2001.0-1ubuntu1.1) ...
Processing triggers for systemd (245.4-4ubuntu3.13) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Processing triggers for initramfs-tools (0.136ubuntu6.6) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-90-generic
I: The initramfs will attempt to resume from /dev/dm-2
I: (/dev/mapper/ubuntu--vg-swap)
I: Set the RESUME variable to override this.
Errors were encountered while processing:
 zfsutils-linux
 zfs-zed
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@nuc2:~#

This appears to be caused by the lxd storage pool.

root@nuc2:~# lxc storage list
+---------+--------+--------------------------------+-------------+---------+
|  NAME   | DRIVER |             SOURCE             | DESCRIPTION | USED BY |
+---------+--------+--------------------------------+-------------+---------+
| default | zfs    | zfs/lxd                        |             | 19      |
+---------+--------+--------------------------------+-------------+---------+
| plain   | dir    | /var/lib/snapd/hostfs/data/lxd |             | 0       |
+---------+--------+--------------------------------+-------------+---------+
root@nuc2:~# zfs list -o name,mountpoint,canmount,mounted
NAME                                              MOUNTPOINT                                                          CANMOUNT  MOUNTED
zfs                                               /zfs                                                                      on      yes
zfs/k8s                                           /zfs/k8s                                                                  on      yes
zfs/k8s/pvc-0885b2e2-b76e-4c00-ae4e-3df08df0f161  /zfs/k8s/pvc-0885b2e2-b76e-4c00-ae4e-3df08df0f161                         on      yes
zfs/lxd                                           /var/snap/lxd/common/lxd/storage-pools/default                            on       no
zfs/lxd/containers                                /var/snap/lxd/common/lxd/storage-pools/default/containers                 on       no
zfs/lxd/containers/apt-cacher                     none                                                                  noauto       no
zfs/lxd/containers/cache2                         none                                                                  noauto       no
zfs/lxd/containers/dmz1                           /var/snap/lxd/common/lxd/storage-pools/default/containers/dmz1            on       no
zfs/lxd/containers/netbox2                        /var/snap/lxd/common/lxd/storage-pools/default/containers/netbox2         on       no
zfs/lxd/containers/netbox3                        none                                                                  noauto       no
zfs/lxd/containers/ns-auth                        none                                                                  noauto       no
zfs/lxd/containers/oxidized                       none                                                                  noauto       no
zfs/lxd/containers/perf1                          none                                                                  noauto       no
zfs/lxd/containers/postgres                       none                                                                  noauto       no
zfs/lxd/containers/proxmox2                       /var/snap/lxd/common/lxd/storage-pools/default/containers/proxmox2        on       no
zfs/lxd/containers/selenium                       none                                                                  noauto       no
zfs/lxd/containers/smtp                           none                                                                  noauto       no
zfs/lxd/containers/sniproxy                       none                                                                  noauto       no
zfs/lxd/containers/unifi                          none                                                                  noauto       no
zfs/vm                                            /zfs/vm                                                                   on      yes
root@nuc2:~#

Most of those containers are running happily, despite the fact they have “MOUNTED no”.

I suspect this is a hangover from some earlier problems with lxd+zfs, and I might have tweaked mount points explicitly on some zfs filesystems in an attempt to fix them.

Any advice here?

candlerb · November 12, 2021, 10:40am

At least the problem is reproducible:

root@nuc2:~# dpkg --configure --pending
Setting up zfsutils-linux (0.8.3-1ubuntu12.13) ...
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Job for zfs-mount.service failed because the control process exited with error code.
See "systemctl status zfs-mount.service" and "journalctl -xe" for details.
invoke-rc.d: initscript zfs-mount, action "start" failed.
● zfs-mount.service - Mount ZFS filesystems
     Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2021-11-12 10:34:00 GMT; 16ms ago
       Docs: man:zfs(8)
    Process: 2061408 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
   Main PID: 2061408 (code=exited, status=1/FAILURE)

Nov 12 10:34:00 nuc2 systemd[1]: Starting Mount ZFS filesystems...
Nov 12 10:34:00 nuc2 zfs[2061408]: cannot mount '/var/snap/lxd/common/lxd/storage-pools/default': directory is not empty
Nov 12 10:34:00 nuc2 systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Nov 12 10:34:00 nuc2 systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Nov 12 10:34:00 nuc2 systemd[1]: Failed to start Mount ZFS filesystems.
dpkg: error processing package zfsutils-linux (--configure):
 installed zfsutils-linux package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of zfs-zed:
 zfs-zed depends on zfsutils-linux (>= 0.8.3-1ubuntu12.13); however:
  Package zfsutils-linux is not configured yet.

dpkg: error processing package zfs-zed (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 zfsutils-linux
 zfs-zed

And I think I was able to fix it by setting the mountpoints to “none”:

root@nuc2:~# zfs set mountpoint=none zfs/lxd/containers zfs/lxd/containers/dmz1 zfs/lxd/containers/netbox2 zfs/lxd/containers/proxmox2 zfs/lxd
root@nuc2:~# dpkg --configure --pending
Setting up zfsutils-linux (0.8.3-1ubuntu12.13) ...
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
Setting up zfs-zed (0.8.3-1ubuntu12.13) ...
root@nuc2:~#

However, I don’t have another zfs-based lxd host to compare with, so I’m not sure if that’s right.

tomp · November 12, 2021, 11:04am

This should be done automatically now each time an instance starts:

github.com/lxc/lxd

Storage: Set mountpoint=none for ZFS filesystem volumes

lxc:master ← tomponline:tp-zfs

opened 03:27PM - 06 Oct 21 UTC

tomponline

+15 -12

Now that we use standard mount syscall for mounting ZFS volumes (since https://g…ithub.com/lxc/lxd/pull/9349) lets try not setting a mount point in ZFS to see if that plays more nicely inside the snap mount namespace, as with `mountpoint` set to a path, we've seen that when mounting a zfs dataset inside the snap's mount namespace can cause it to be mounted on the host's mount namespace which isn't correct. Signed-off-by: Thomas Parrott <thomas.parrott@canonical.com>

candlerb · November 12, 2021, 12:26pm

That makes sense - the zfs filesystems which still had the ‘mountpoint’ setting are for containers which haven’t been started for a while.

It also confirms that setting mountpoint=none (and canmount=noauto) is the correct fix.

Thanks!

stgraber · November 12, 2021, 3:04pm

Right, the issue here is zfs/lxd having a mountpoint property set.
This gets inherited through the dataset tree and is causing a bunch of mount conflicts.

zfs set mountpoint=none zfs/lxd should take care of the issue.