My goal is to transfer a file from instance c1 to instance c2 within Incus. I’m trying the file pull from c1 then push to c2 method.
I’m running incus latest stable and on a local network, I’ve set the instance c1 and c2 to direct attach to the network. The client is on the same network but on the WiFi.
If I run incus file pull c1/bigfile . the speeds I get are around 5 MB/s.
Using croc I get around 80 MB/s, closer to what I’d expect.
Out of curiosity, I added c2 as a trusted incus client to test a file pulling from c1 and the speeds were around 260 MB/s.
Is there a way to speed up this transfer or would it be a misconfiguration on my end?
Is there a better or best-practice way to transfer a file from one incus instance to the other directly?
It shouldn’t make any difference how the containers “attach to the network”: you’re talking to the incus daemon running on the physical host, not to the containers. Indeed, these pull/push operations should work even if the containers are stopped.
I have not come across “croc” before. Do you mean this? If you’re using it to transfer directly from c1 to c2 (both wired), then this is not a fair comparison to transfer from c1 to client device (wireless). Or are you running croc inside c1 and also on the wireless client?
To debug further, I suggest installing iperf3 on both the server host and the client. Run “iperf3 -s” on the server, and then “iperf3 -c x.x.x.x -R” on the client, to test the potential throughput available over wireless.
Also: you should specify the version of incus you’re running, and the underlying operating system.
Hi @candlerb! Thank you for enlightening me on how this works and for the pointers. You’re right I should’ve put more information before posting^^ I will gather my findings below:
Server IP: 192.168.0.10
Server OS: Debian 13 Stable
Incus Client version: 7.0.0 — (bin.macos.incus.aarch64)
Incus Server incus version: 7.0.0 — (zabbly incus repo)
Incus Storage driver: dir on ext4 (default admin init)
I’ve seen similar results on other clients on WiFi, I don’t have an ethernet cable to test with me right now but I’ll update if I do test and get different results with that. Thank you again!
Taking the middle case (incus pull, wired ethernet): running “top” on both client and server shows “kernel_task” at around 110% of a CPU on the client side. I wonder if too small a buffer is being used for I/O?
Update: tested on ethernet on a Windows laptop (also over remote in VPN).
I also reinstalled the server and retested (also with incus file push instead) with similar results.
The numbers vary wildly, but with the Windows laptop LAN I got e.g. 60% of the speed I got with scp/sftp command.
While e.g. over WireGuard VPN on another laptop I get 20% of the speed I got with SCP/SFTP command when pulling.
Here’s another test run on client where there’s a 4x difference in pull/push (it’s a 5GHz local WiFi client):
% dd if=/dev/urandom of=bigfile bs=1M count=200
% time incus file push bigfile c1/root/
Pushing bigfile to /root/bigfile: 100% (20.0MB/s)
incus file push bigfile c1/root/ 0.44s user 1.96s system 22% cpu 10.496 total
% time incus file pull c1/root/bigfile .
Pulling bigfile from /root/bigfile: 209.71MB (5.41MB/s)
incus file pull c1/root/bigfile . 1.24s user 2.21s system 8% cpu 38.761 total
% time sftp user:/var/lib/incus/containers/c1/rootfs/root/ <<< 'put bigfile'
Connected to user.
Changing to: /var/lib/incus/containers/c1/rootfs/root/
sftp> put bigfile
Uploading bigfile to /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
100% 200MB 66.9MB/s 00:02
2.70s user 0.49s system 64% cpu 4.964 total
% time sftp user <<< 'get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile'
Connected to user.
sftp> get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
Fetching /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile to bigfile
100% 200MB 81.1MB/s 00:02
2.70s user 0.44s system 69% cpu 4.509 total
Interesting to note the CPU on the incus file operations, makes it seems like it’s mostly idling around so I decided to prod around and test a bit more.
Now I do the same incus/sftp push/pull test as before from client:
% time incus file push bigfile c1/root/
Pushing bigfile to /root/bigfile: 100% (2.12MB/s)
0.58s user 2.60s system 3% cpu 1:39.46 total
% time incus file pull c1/root/bigfile .
Pulling bigfile from /root/bigfile: 209.71MB (534.73kB/s)
1.68s user 2.44s system 1% cpu 6:33.11 total
% time sftp user:/var/lib/incus/containers/c1/rootfs/root/ <<< 'put bigfile'
Connected to user.
Changing to: /var/lib/incus/containers/c1/rootfs/root/
sftp> put bigfile
Uploading bigfile to /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
100% 200MB 28.6MB/s 00:06
3.26s user 1.47s system 47% cpu 10.066 total
% time sftp user <<< 'get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile'
Connected to user.
sftp> get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
Fetching /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile to bigfile
100% 200MB 27.0MB/s 00:07
3.44s user 0.81s system 40% cpu 10.395 total
I see for incus file push/pull that the speed and CPU usage has dropped by roughly 10x. OTOH the speed and CPU usage for ‘native’ SFTP command decreases by only 1.5x roughly.
Looking at the lxc/incus repo in incus/client/incus_instances.go I see the following:
// GetInstanceFileSFTPConn returns a connection to the instance's SFTP endpoint.
func (r *ProtocolIncus) GetInstanceFileSFTPConn(instanceName string) (net.Conn, error) {
...
// Get a SFTP client.
client, err := sftp.NewClientPipe(conn, conn, sftp.MaxPacketUnchecked(128*1024))
if err != nil {
_ = conn.Close()
return nil, err
}
...
I don’t see MaxConcurrentRequests neither UseConcurrentReads set so I assume the defaults in the Go pkg/sftp should apply (which seem good). But it seems something’s not adding up with the performance and its sensitivity to RTT. I’m too
Now just out of curiosity, still with the ~60ms delay, I tried SFTP from client with a single request (no concurrency) and 128kiB & 32kiB buffer size comparisons:
With 6ms RTT the throughput expectedly bumped up by 10x (I won’t spam the numbers you’ll have to believe me^^).
Looking up speed issues with latency related to the Go sftp pkg I came across this issue:
And this PR:
Which seems to say sftp’s ReadFrom/WriteTo should be used to avoid performance degradation.
It looks like in incus code it’s using util.SafeCopy which uses io.CopyN with 4MiB chunks which wraps the reader in LimitReader which I think won’t use pkg/sftp WriteTo which if so wouldn’t keep the TCP window open and end up breaking the chunks into at most MaxPacket sized packets, e.g. with 32kiB size it’d be 128 packets, taking 7.7s to send them all on 60ms RTT (~500kB/s).
I’ve rambled enough and I’m not very familiar with how this works so I might’ve said some wrong things, I’ll raise an issue on GH soon to give it a bit more attention