Copying container from one server in an encrypted pool to another in an encrypted pool fails after performing a second copy/--refresh

1

Hello, I am trying to copy a container from one server to another, both using encrypted zfs backend pool (“encpool”), both running incus 6.0.2, both on ubuntu 22.04:

ubuntu@lxd-server:~$ incus launch ubuntu:24.04 c1 -s encpool
Creating c1
Starting c1
ubuntu@lxd-server:~$ incus stop c1
ubuntu@lxd-server:~$ incus copy c1 lxd-backup: -s encpool
ubuntu@lxd-server:~$ incus copy c1 lxd-backup: -s encpool --refresh
Error: Failed instance creation: Error transferring instance data: Failed migration on target: Failed creating instance on target: Failed receiving volume "c1": Problem with zfs receive: ([exit status 1 write |1: broken pipe]) cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one

At this point, c1 container’s storage on the backup server is completely lost.

I have seen this post: Encrypted ZFS dataset: incus copy --refresh fails

I think this is similar but at least in my situation, the zfs key is loaded on both source and destination already.

What is also strange is that I have another older container that does work perfectly fine. The only difference is that this is an old container running ubuntu 18 instance. However, creating a new ubuntu 18 instance fails.

Is there anything I can run on my end to help determine what or where the problem is?

(PS - I created a reddit /r/LXD post and posted an LXD github bug report, thinking that maybe it was specific to LXD. I was eventually going to migrate to Incus, and this bug motivated me to force the Incus migration and test the above scenario.)

2

Welcome!

The exact error message says

cannot receive new filesystem stream: 
       zfs receive -F cannot be used to 
           1. destroy an encrypted filesystem or 
           2. overwrite an unencrypted one with an encrypted one

When Incus invokes the zfs binary with the receive action, along with other parameters, gives out that specific error. It’s either one of the two cases, likely the first case. Then, see zfs receive -F cannot be used to destroy an encrypted filesystem · Issue #6793 · openzfs/zfs · GitHub

3

Thanks for the reply.

Maybe… I’m not super experienced with zfs but I’m not sure the linked issue relates to me. I’m using the same encryption key on both servers and both keys are already loaded.

What is really strange is it does work for some encrypted containers. But I can’t successfully copy any new containers to another server’s encrypted pool using --refresh.

Anyway I give up. I’ll set up LUKS encryption on the backup server and won’t use zfs native encryption. At least it seems to work that way. I don’t need any special features on a backup server but it does need to be reliable, and encrypted at rest, so the above should suffice for now.

1 Like