after a reboot of my production server a few of my instances failed to start. They reported the following error:
Error: Failed to run: zfs zone /proc/34650/ns/user sata-pool/incus/custom/default_data: exit status 1 (cannot add ‘sata-pool/incus/custom/default_data’ to namespace: dataset already exists)
All these instances have a similar device (mount) setup:
web:
path: /var/www/html
pool: data
source: data/web/data
type: disk
conf:
path: /etc/httpd/conf
pool: data
source: data/web/conf
type: disk
In short the same storage volume is mounted twice with different path locations.
Nothing shows up in the log:
incus info --show-log web
Name: web
Description:
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2025/06/24 21:45 AWST
Last Used: 2025/06/24 22:02 AWST
Log:
System log files shows sometimes:
incusd[5010]: time=“2025-06-24T21:57:33+08:00” level=error msg=“Error getting volume usage” err=“Storage volume "data/data/web" in project "default" of type "custom" does not exist on pool "data": Storage volume not found” instance=web instanceType=container project=default volume=data/data/web
By removing either of the devices the instance starts without any issues. As soon as both are added back it fails.
This configuration was working without any issues for a couple of month. Unfortunately I had to restart the server today and since then this issue persists.
incus version
Client version: 6.13
Server version: 6.13
It looks like something specific to this server as I wasn’t able to reproduce it on my test installations running same version of Incus.
This should be a pretty easy fix in the ZFS driver on our end, though that logic wouldn’t have changes since we first introduced support for the ZFS namespace stuff well over a year ago.
I wonder if maybe there’s a difference between adding those mounts to a running instance vs having them at startup which would have allowed you to start things initially.
In my case it doesn’t make a difference if I add both path mounts before I start the container or if I add them to a running container. In both cases I get the same error message.
To my surprise I just run another test:
creating a brand new storage volume on same custom data storage
created two directories in it
created a new instance and added both path to different location
instance starts, stops, etc.
No such error error reported. As soon as I try to perform the same with the existing storage volume it errors out. It smells like something has been left in an unknown status before / after the reboot?
I leave the system (volume) as is and work around it by creating a new volume and copy data over to bring the stack up and running.
Yeah, not sure about this. Just performed zfs get zoned call:
sata-pool/incus/custom/default_data zoned on local
sata-pool/incus/custom/default_data@2025-03-27-000000 zoned - -
sata-pool/incus/custom/default_data@2025-03-28-000000 zoned - -
sata-pool/incus/custom/default_data@2025-03-29-000000 zoned - -
sata-pool/incus/custom/default_data@2025-03-30-000000 zoned - -
As you can see it has zoned set to on. On the other hand it has plenty of snapshots too. Makes me wonder is this could be the related to the issue?
All my current tests I performed on a brand new dataset (volume) without snapshot nor zone=on. May be a hint for @stgraber will perform some more tests.
Can confirm that everything works as long as there is no snapshot added to the storage volume. As soon as you perform a few snapshots the magic error is returned and the instance fails to start.