Fastest way to copy a container between two hosts?

I need to rapidly spin up new cloud servers running an LXC image for a CI/CD pipeline. However, the way I’ve currently implemented things, it takes too long:

  • Starting the underlying cloud server take about 20-30 sec
  • apt-get update && apt-get install -y --no-install-recommends lxd takes another 45-60 sec.
  • lxd init ... && lxd remote add ... takes another 30 sec
  • finally, lxd launch remote:... takes anywhere between 3-4 mins. This includes physically transferring a 4.5 GB image and then de-compressing/untarring it to get a 14 GB rootfs

I have the ability to parallelize these operations, so I was wondering if something like the following is possible. If yes, how?

  • While apt-get install lxd is running, can I scp the uncompressed rootfs from the remote server to some directory outside of /var/lib/lxd/storage-pools/default/containers/
  • When lxd is installed, can I start a container with the rootfs that I copied in the previous step?

The two machines have 3-4 Gbits/sec pipe (tested via iperf). So, theoretically, the uncompressed rootfs should get copied in about 40sec.

Is there any reason to download the image every time?