Lxc copy fails, possibly to do with ZFS?

lxd --version 2.21
running on snap on Debian Stretch (proxmox)

I have some test containers that I created which seem to run fine, and I managed to copy one over from other server to begin with.

However now I seem to get a possibly ZFS error when trying to copy them:

lvl=eror msg=“Problem with zfs send: internal error: Invalid argument\n.” t=2018-01-31T21:47:37+0000

I do have a firewall (with port 8443 open on the vpn interface) on the sending server but I’ve disabled it and get the same results.

on@hub2-proxhub2-prox:~$ lxc copy plex bramdc:plexBackup --stateless --verbose
error: Failed container creation:

  • https://10.55.0.1:8443: Error transferring container data: websocket: bad handshake
  • https://10.55.99.1:8443: Error transferring container data: websocket: bad handshake
  • https://[fd42:::1]:8443: Error transferring container data: Unable to connect to: [fd42:685b:504a:1c51::1]:8443

Just wondering if anyone has any idea what could be causing the copy to fail?

Both ends are using 2.21 snap version.

Cheers!

Jon. :sunglasses:

Does --mode=relay helps somehow?

I’ll try that now, cheers.

Hiya, no it didn’t help it just failed much faster, pretty much failed instantly. same zfs error in the log :frowning:

So I was chatting with one of our kernel engineers and apparently ZFS broke their send/receive kernel/userspace API at some point.

What version of the zfs module are you running?

Apparently there’s a problem when using the version of the tools we include (0.6.x) combined with a 0.7.x version of the kernel module. Unfortunately, bumping the version of the tools apparently will break when running on an older kernel… if that’s the case, we’re kinda stuck until the tools are corrected to work with both 0.6 and 0.7 kernel, then we can just bundle the 0.7 version of the tools.

the source server is a proxmox debian stretch, destination is ubuntu 17.10

proxmox / stretch: 4.13.13-5-pve
ubuntu: 4.13.0-32-generic

and yeah both snap 2.21 - had to recently upgrade the ubuntu from non snap.

looks like I don’t have zfs-dkms installed on either:

root@bramdc-src-01:/home/jon# apt install zfs-dkms
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.13.0-17 linux-headers-4.13.0-17-generic linux-headers-4.13.0-21 linux-headers-4.13.0-21-generic linux-headers-4.13.0-25 linux-headers-4.13.0-25-generic linux-image-4.13.0-17-generic linux-image-4.13.0-21-generic
  linux-image-4.13.0-25-generic linux-image-extra-4.13.0-17-generic linux-image-extra-4.13.0-21-generic linux-image-extra-4.13.0-25-generic xdelta3
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  dkms spl spl-dkms
Suggested packages:
  menu
The following NEW packages will be installed:
  dkms spl spl-dkms zfs-dkms
0 upgraded, 4 newly installed, 0 to remove and 3 not upgraded.
Need to get 1,554 kB of archives.
After this operation, 11.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.

Ah, so I guess the proxmox kernel comes with the zfs module then, interesting, so not quite a standard Debian then. Hopefully the error isn’t kernel related or that’ll make things a bit harder for us to reproduce.

Out of curiosity, does transferring the other way around works?

I’m doing some testing now to see just how true that claim is and whether we can get the newer tools to behave.

Oh, the forum didn’t post my previous comment. One of our kernel engineers said that there’s some known compatibility issue with the zfs 0.7.x kernel module when combined with the 0.6.x utils we ship in the snap.

That’s due to an unstable kernel/userspace API in zfsonlinux which is a bit annoying and is known to break send/receive. I’m hoping that it’s just a one way incompatibility and that we can use the newer tools everywhere.

Ah right!

Just tried the other way round, sending a container from ubuntu to proxmox and its working! :octopus:

Transferring container: backup: 98.30MB (1.29MB/s)

I also installed zfs-dkms on both.

Cheers,
Jon.

Just seems to be the proxmox side is unable to copy a container to ubuntu. Weirdness.

Then again I guess I am bending the laws of physics running lxd on a proxmox server!

Cheers,
Jon.

What does “modinfo -F version zfs” show on both systems?

Ubuntu Side:
root@bramdc-src-01:/home/jon# modinfo -F version zfs
0.6.5.11-1ubuntu3

Prox Side:

root@hub2-proxhub2-prox /home/jon # modinfo -F version zfs
0.7.4-1

Ok cool, that matches my suspicion. We have a fix coming to the snap package which will fix that.