Lxc file pull from incus server returns status rather than files

Incus
1

We are migrating from LXD to Incus. The migration strategy is to move containers from LXD 5.0.3 hosts to Incus 6.0.0 hosts using LXC 5.0.3.

Everything else seems quite OK, but our orchestration relies on file pull --recursive for some daily operations. During the migration period, things need to continue working on both old LXD and new Incus servers.

However, when our orchestration runs the file pull, it gets different results:

  • From LXD returns the directory as expected
lxc file pull lxdh1:testc1/tags /tmp/ --recursive; cat /tmp/tags
cat: /tmp/tags/: Is a directory
  • From Incus it returns the status and metadata and puts it in a file
lxc file pull incush2:testc2/tags /tmp/ --recursive; cat /tmp/tags
{"type":"sync","status":"Success","status_code":200,"operation":"","error_code":0,"error":"","metadata":["nginx","portforwards","certbot","https","publicdns","ubuntu","type","http","letsencrypt"]}

It is a bit of a chicken-egg situation; our orchestration node would need to be rebuilt manually to use Incus before the orchestration can build Incus containers. Or, potentially having the Incus client installed from outside current repository on the current node. I am trying to figure out which caveats I can avoid.

Is there any way around the file pull discrepancy (not relying on rsync etc, needs to be lxc), or do I have to upgrade the orchestration node to Incus?

Will something else stop working instead between Incus 6.0.0 client and LXD 5.0.3 server?

2

The basic file transfer API relies on a bunch of HTTP headers, those headers are different between LXD and Incus, that’s what’s causing the issue here.

On the Incus side, we’re moving our file operations in the CLI to instead use the SFTP protocol rather than REST (still over our API), once that work is complete, it should actually improve compatibiltiy with recent LXD for those file operations.

In your case, you should be able to install both the LXD CLI and Incus CLI next to each other, either using the incus-client package to install just the CLI or even downloading a static binary of the client from our releases page on Github.

3

Thanks for the insight. Indeed it looks like we need to revise our strategy - our current migration strategy works only when not much more is needed than moving containers.

Installing Incus client for the orchestration will alone not solve the issue, since incompatibilities exist both ways.

Overall, I personally prefer a gradual and incremental approach over a everything-at-once approach.

Three options remain:

  1. Extend the orchestration to be aware of each host’s variant and the fact that it has two sets of remotes
  2. Rewrite incompatible functions in the orchestration
  3. Update the old hosts in-place using the Incus migration tool from external repo (although this does not allow working gradually)

Beyond this forum, I could not find documentation on migration and its caveats except some covering the in-place approach. Are there any further resources available to inform our decision?

4

After investigating a bit, the third option to update in-place may yet be on the table, since it wastes the least effort towards migration-specific workarounds such as for file pull.

One question remains though: the documentation mentions latest stable, whereas we intend to use LTS (since new hosts will use it from Ubuntu 24.04 repository). Are there any blockers or known issues with lxd-to-incus in 6.0.2?

5

Nope, lxd-to-incus in 6.0.2 is identical to that in 6.6.

6

Good to know. While the heading of this topic has been addressed, I am putting together the bigger picture of host-to-host migration in this thread as I progress through the overall migration for anyone attempting the same: