Replacing a pool member missing at import fails: daemon passes vdev GUID as a by-id path

fred-bloggs · September 13, 2026, 9:35pm

Setup

IncusOS 202609120242
bare metal / 4x SATA HDD zfs-raidz2

What happend

I wanted to replace a drive that shows signs of failing with an new one.

What i did: poweroff, pulled the degrading member, fitted a new blank drive, powered on. storage show then reports the pool DEGRADED with the survivors in devices, the new drive with member_pool: "", and

devices_degraded:
- /dev/disk/by-id/138636987219773271

the vdev GUID, not the missing drive’s by-id path. zpool status (via a container with a delegated dataset):

138636987219773271  UNAVAIL  0 0 0  was /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC6E1PVATZ9-part1

Cause

ZFS names a leaf by GUID when it was absent at import (ZPOOL_CONFIG_NOT_PRESENT); a power-off swap always produces that. getZpoolMembersHelper (internal/storage/storage.go:504-506) prefixes the vdev name with /dev/disk/by-id/, so updateZpoolHelper, which otherwise handles the missing member correctly (zfs.go:782), ends up running

zpool replace <pool> /dev/disk/by-id/138636987219773271 /dev/disk/by-id/<new>

(zfs.go:834). libzfs treats a leading / as a path, finds no such path, and fails with no such device in pool. Nothing is applied; the documented flow cannot succeed. The upstream pull request #1025 test doesn’t hit this because it hot-detaches the disk from a running VM, so the path name survives; any reboot without the disk breaks it.

With a dead drive (mine luckily is only degraded) this would be terminal: the pool stays DEGRADED, the API cannot repair it, and there is no shell to run zpool replace <pool> <guid> <new> by hand.

Suggested fix

For non-ONLINE leaves use the path field from zpool status -j (present even when the name is a GUID; strip the partition suffix), or pass the bare GUID to zpool replace, libzfs should accept it.

Temporary Solution?

Is there a way I can offline replace a disk at the moment?

fred-bloggs · September 13, 2026, 9:40pm

github.com/lxc/incus-os

Replacing a pool member missing at import fails: daemon passes vdev GUID as a by-id path

opened 09:40PM - 13 Sep 26 UTC

fred-bloggs

# Setup IncusOS 202609120242 bare metal / 4x SATA HDD `zfs-raidz2` # What happe…nd I wanted to replace a drive that shows signs of failing with an new one. What i did: `poweroff`, pulled the degrading member, fitted a new blank drive, powered on. `storage show` then reports the pool `DEGRADED` with the survivors in `devices`, the new drive with `member_pool: ""`, and ``` devices_degraded: - /dev/disk/by-id/138636987219773271 ``` the vdev **GUID**, not the missing drive's by-id path. `zpool status` (via a container with a delegated dataset): ``` 138636987219773271 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC6E1PVATZ9-part1 ``` # Cause ZFS names a leaf by GUID when it was absent at import (`ZPOOL_CONFIG_NOT_PRESENT`); a power-off swap always produces that. `getZpoolMembersHelper` (`internal/storage/storage.go:504-506`) prefixes the vdev *name* with `/dev/disk/by-id/`, so `updateZpoolHelper`, which otherwise handles the missing member correctly (`zfs.go:782`), ends up running ``` zpool replace <pool> /dev/disk/by-id/138636987219773271 /dev/disk/by-id/<new> ``` (`zfs.go:834`). libzfs treats a leading `/` as a path, finds no such path, and fails with `no such device in pool`. Nothing is applied; the documented flow cannot succeed. The upsteram pull request #1025 test doesn't hit this because it hot-detaches the disk from a running VM, so the path name survives; any reboot without the disk breaks it. With a **dead** drive (mine luckily is only degraded) this would be terminal: the pool stays `DEGRADED`, the API cannot repair it, and there is no shell to run `zpool replace <pool> <guid> <new>` by hand. # Suggested fix For non-ONLINE leaves use the `path` field from `zpool status -j` (present even when the name is a GUID; strip the partition suffix), or pass the bare GUID to `zpool replace`, libzfs should accept it.

fred-bloggs · September 13, 2026, 9:48pm

github.com/lxc/incus-os

Replace-in-place wipes the old drive while ZFS is still resilvering from it

opened 09:48PM - 13 Sep 26 UTC

fred-bloggs

# Setup IncusOS 202609120242 bare metal / 4x SATA HDD `zfs-raidz2` ## Situation… The tutorial ("Recovering a failed non-system drive") shows replacing a member while the old drive is still attached. For a drive that is degrading but still readable this is also what ZFS recommends: `zpool replace` builds a `replacing-N` vdev and rebuilds the new member from the old one, keeping full redundancy throughout, instead of a parity resilver at reduced redundancy. ## Problem `updateZpoolHelper` calls `storage.WipeDrive` on the old device immediately after `zpool replace` returns, if its by-id path still exists (`internal/zfs/zfs.go:844-850`). `WipeDrive` refuses drives with a `member_pool`, but `getZpoolMembersHelper` does not descend into `replacing-N`, so the old drive, now a child of that vdev, is reported with `member_pool: ""` and the guard passes. `ClearBlock` writes its markers into a live pool member, and on a spinning disk (no TRIM) falls through to `blkdiscard -z`: a full zero-out of the drive ZFS is currently copying from, hours long on 4 TB, run without a context so the API request blocks until it finishes and a client timeout does not stop it. ZFS survives it (checksums fail on the zeroed reads, blocks are rebuilt from parity), but the point of keeping the old drive in, a full-redundancy rebuild, is lost, the controller carries a full-disk write stream on top of the resilver, and the operator sees a hung `storage edit`. On virtual disks (the tutorial's setting) the discard is instant, which is why it looks fine there. ## Suggested fix Don't wipe a device that is still part of the pool: either defer the wipe until `zpool status -j` shows no `replacing-N` containing it (ZFS detaches the old child automatically when the resilver completes), or don't wipe at all and leave it to an explicit `wipe-drive`. Either way the walker should descend into `replacing-N`/`spare-N` so `member_pool` stays correct during a replace. Related: [replacing a member that was *absent* at import fails for a different reason (GUID passed as a by-id path) ](https://github.com/lxc/incus-os/issues/1365) / separate issue.