Incus start fails: "Failed to mount ... using ZFS... permission denied" after upgrade

I recently upgraded my (Arch Linux) Incus server, which took incus from 6.1 to 6.5, and also upgraded the system itself from a months-old version[1].

So, not sure if it is Incus changes, or recent Arch changes, but now my user, who is a member of incus-admin group and could start/stop instances before this update, no longer can. (Log is below.)

What can I do to remedy this? The root user gets the same error.

➜  ~ incus ls
+------------------+---------+------+------+-----------------+-----------+
|       NAME       |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+------------------+---------+------+------+-----------------+-----------+
| butt             | STOPPED |      |      | CONTAINER       | 3         |
+------------------+---------+------+------+-----------------+-----------+
| discrete-mammal  | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+------------------+---------+------+------+-----------------+-----------+
| fed-40-container | STOPPED |      |      | CONTAINER       | 3         |
+------------------+---------+------+------+-----------------+-----------+
| ubuttnu-24-vm    | STOPPED |      |      | VIRTUAL-MACHINE | 1         |
+------------------+---------+------+------+-----------------+-----------+
| ubuttnu-24-vm2   | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+------------------+---------+------+------+-----------------+-----------+
➜  ~ incus start butt
Error: Failed to mount "tank/incus-encrypted-storage/containers/butt" on "/var/lib/incus/storage-pools/encrypted-pool/containers/butt" using "zfs": permission denied
Try `incus info --show-log butt` for more info
➜  ~ incus info --show-log butt
Name: butt
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/06/22 14:21 JST
Last Used: 2024/09/18 00:54 JST

Snapshots:
+-----------------------------------------------------------------+----------------------+------------+----------+
|                              NAME                               |       TAKEN AT       | EXPIRES AT | STATEFUL |
+-----------------------------------------------------------------+----------------------+------------+----------+
| 2024-06-22-virgin-incus-ubuttnu-24-04-image-container-with-btop | 2024/06/22 14:24 JST |            | NO       |
+-----------------------------------------------------------------+----------------------+------------+----------+
| 2024-08-11-before-desktop-GUI-install                           | 2024/08/11 16:27 JST |            | NO       |
+-----------------------------------------------------------------+----------------------+------------+----------+
| 2024-08-11-before-desktop-GUI-installuptime                     | 2024/09/05 10:26 JST |            | NO       |
+-----------------------------------------------------------------+----------------------+------------+----------+

Log:


➜  ~ uname -a
Linux ray 6.6.51-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 12 Sep 2024 11:23:10 +0000 x86_64 GNU/Linux
➜  ~ groups $(whoami)
wheel incus incus-admin libvirt mason
[root@ray ~]# incus start butt
Error: Failed to mount "tank/incus-encrypted-storage/containers/butt" on "/var/lib/incus/storage-pools/encrypted-pool/containers/butt" using "zfs": permission denied
Try `incus info --show-log butt` for more info
[root@ray ~]#

[1]: The reason I went months without updating this server is because I had been using the zfs-linux-lts package, and had been waiting to see if it would be updated, but it seems no longer maintained, so I switched to zfs-dkms. Then I could finally update Arch itself.

Oops, OK, the issue was that the mechanism to mount the encrypted ZFS filesystem at boot time was no longer working. This meant that the Incus storage pool (which was created using that existing ZFS filesystem) was not available.

Manually mounting the underlying ZFS filesystem fixed the issue; as soon as I did that, Incus works normally. So, nothing to do with Incus, actually. :sweat_smile:

1 Like