Lxc copy: Error transferring instance data: exit status 22

@stgraber

I decided to try copying a container using the push mode:

$ lxc copy test-copy ceph: --mode push --verbose
Transferring container: test-copy: 142B (19B/s)

It just hangs at the above state indefinitely despite the following error showing up in ‘lxc monitor’ output on the target cluster:

metadata:
  class: websocket
  created_at: "2020-05-08T11:18:54.187185969-07:00"
  description: Creating container
  err: 'Error transferring instance data: exit status 2'
  id: 11809c5b-cae4-40a6-9c1e-95adfd5eb2ee
  location: node03
  may_cancel: false
  metadata:
    create_instance_from_image_unpack_progress: 'Unpack: 100% (3.90GB/s)'
    progress:
      percent: "100"
      speed: "3900497512"
      stage: create_instance_from_image_unpack
  resources:
    containers:
    - /1.0/containers/test-copy
    instances:
    - /1.0/instances/test-copy
  status: Failure
  status_code: 400
  updated_at: "2020-05-08T11:18:57.590873186-07:00"

Both push and pull modes encounter the same ‘exit status 2’ error. The above is at least a bug because the failure should have propagated out and terminated the hung command. However, I consider it a bug because rsyncing from LXD 3.0.3 on Ubuntu 18.04 LTS to LXD snap 4.0.1 on Ubuntu 18.04 LTS should have close enough rsync versions for this to work, right?

The source LXD daemon reports the following in its log:

lvl=eror msg=“Rsync send failed: /var/lib/lxd/containers/test-copy/: exit status 2: [Receiver] Invalid dir index: -1 (-101 - -101)\nrsync error: protocol incompatibility (code 2) at flist.c(2634) [Receiver=3.1.2]\n”

I tried searching for “[Receiver] Invalid dir index: -1 (-101 - -101)”. Maybe it is related to this?: