Incus file pull/push Strange Slowness

Hi!

My goal is to transfer a file from instance c1 to instance c2 within Incus. I’m trying the file pull from c1 then push to c2 method.

I’m running incus latest stable and on a local network, I’ve set the instance c1 and c2 to direct attach to the network. The client is on the same network but on the WiFi.

If I run incus file pull c1/bigfile . the speeds I get are around 5 MB/s.

Using croc I get around 80 MB/s, closer to what I’d expect.

Out of curiosity, I added c2 as a trusted incus client to test a file pulling from c1 and the speeds were around 260 MB/s.

  1. Is there a way to speed up this transfer or would it be a misconfiguration on my end?
  2. Is there a better or best-practice way to transfer a file from one incus instance to the other directly?

Thank you for all your work

It shouldn’t make any difference how the containers “attach to the network”: you’re talking to the incus daemon running on the physical host, not to the containers. Indeed, these pull/push operations should work even if the containers are stopped.

I have not come across “croc” before. Do you mean this? If you’re using it to transfer directly from c1 to c2 (both wired), then this is not a fair comparison to transfer from c1 to client device (wireless). Or are you running croc inside c1 and also on the wireless client?

To debug further, I suggest installing iperf3 on both the server host and the client. Run “iperf3 -s” on the server, and then “iperf3 -c x.x.x.x -R” on the client, to test the potential throughput available over wireless.

Also: you should specify the version of incus you’re running, and the underlying operating system.

Hi @candlerb! Thank you for enlightening me on how this works and for the pointers. You’re right I should’ve put more information before posting^^ I will gather my findings below:

Server IP: 192.168.0.10
Server OS: Debian 13 Stable
Incus Client version: 7.0.0 — (bin.macos.incus.aarch64)
Incus Server incus version: 7.0.0 — (zabbly incus repo)
Incus Storage driver: dir on ext4 (default admin init)

Here are iperf3 results from client:

% iperf3 -c 192.168.0.10 -R
Connecting to host 192.168.0.10, port 5201
Reverse mode, remote host 192.168.0.10 is sending
[  5] local 192.168.0.55 port 49224 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  78.2 MBytes   653 Mbits/sec                  
[  5]   1.01-2.00   sec  84.1 MBytes   707 Mbits/sec                  
[  5]   2.00-3.00   sec  73.2 MBytes   615 Mbits/sec                  
[  5]   3.00-4.00   sec  71.2 MBytes   597 Mbits/sec                  
[  5]   4.00-5.00   sec  76.1 MBytes   639 Mbits/sec                  
[  5]   5.00-6.00   sec  78.0 MBytes   655 Mbits/sec                  
[  5]   6.00-7.00   sec  79.1 MBytes   664 Mbits/sec                  
[  5]   7.00-8.00   sec  76.8 MBytes   643 Mbits/sec                  
[  5]   8.00-9.00   sec  79.0 MBytes   662 Mbits/sec                  
[  5]   9.00-10.00  sec  81.0 MBytes   681 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   780 MBytes   654 Mbits/sec    0            sender
[  5]   0.00-10.00  sec   777 MBytes   652 Mbits/sec                  receiver

Then I run the following from client:

% incus stop c1

% incus file pull c1/root/bigfile .
Pulling bigfile from /root/bigfile: 59.54MB (5.40MB/s)^C

% sftp root@192.168.0.10:/var/lib/incus/containers/c1/rootfs/root/bigfile .
Fetching /var/lib/incus/containers/c1/rootfs/root/bigfile to ./bigfile
/var/lib/incus/containers/c1/r...  23%  956MB  78.0MB/s   00:40 ETA^C

I’ve seen similar results on other clients on WiFi, I don’t have an ethernet cable to test with me right now but I’ll update if I do test and get different results with that. Thank you again!

I don’t know what the problem is then, but if you can reproduce it over a cabled connection then it’s worth raising an issue for.

FWIW, I do see about the same as you over wireless (802.11ac = wifi 5, Macbook Pro M3 as the client, incus 6.23 on server):

% incus file pull kit:cndo/var/lib/GNS3/images/QEMU/ubuntu-22.04-server-cloudimg-amd64-20221214.img .
Pulling ubuntu-22.04-server-cloudimg-amd64-20221214.img from /var/lib/GNS3/images/QEMU/ubuntu-22.04-server-cloudimg-amd64-20221214.img: 218.92MB (5.45MB/s)^C

It’s 5 times faster on wired:

% incus file pull kit:cndo/var/lib/GNS3/images/QEMU/ubuntu-22.04-server-cloudimg-amd64-20221214.img .
Pulling ubuntu-22.04-server-cloudimg-amd64-20221214.img from /var/lib/GNS3/images/QEMU/ubuntu-22.04-server-cloudimg-amd64-20221214.img: 446.30MB (24.77MB/s)

but still not what gigabit can achieve, which is 4 times faster again:

% scp s1e:/var/lib/GNS3/images/QEMU/ubuntu-22.04-server-cloudimg-amd64-20221214.img .
ubuntu-22.04-server-cloudimg-amd64-20221214.img                                                                                    69%  445MB 110.2MB/s   00:01 ETA^C

Taking the middle case (incus pull, wired ethernet): running “top” on both client and server shows “kernel_task” at around 110% of a CPU on the client side. I wonder if too small a buffer is being used for I/O?

Update: tested on ethernet on a Windows laptop (also over remote in VPN).

I also reinstalled the server and retested (also with incus file push instead) with similar results.

The numbers vary wildly, but with the Windows laptop LAN I got e.g. 60% of the speed I got with scp/sftp command.

While e.g. over WireGuard VPN on another laptop I get 20% of the speed I got with SCP/SFTP command when pulling.

Here’s another test run on client where there’s a 4x difference in pull/push (it’s a 5GHz local WiFi client):

% dd if=/dev/urandom of=bigfile bs=1M count=200


% time incus file push bigfile c1/root/
Pushing bigfile to /root/bigfile: 100% (20.0MB/s)
incus file push bigfile c1/root/  0.44s user 1.96s system 22% cpu 10.496 total

% time incus file pull c1/root/bigfile .
Pulling bigfile from /root/bigfile: 209.71MB (5.41MB/s)
incus file pull c1/root/bigfile .  1.24s user 2.21s system 8% cpu 38.761 total


% time sftp user:/var/lib/incus/containers/c1/rootfs/root/ <<< 'put bigfile'
Connected to user.
Changing to: /var/lib/incus/containers/c1/rootfs/root/
sftp> put bigfile
Uploading bigfile to /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
100%  200MB  66.9MB/s   00:02    
2.70s user 0.49s system 64% cpu 4.964 total

% time sftp user <<< 'get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile'
Connected to user.
sftp> get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
Fetching /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile to bigfile
100%  200MB  81.1MB/s   00:02    
2.70s user 0.44s system 69% cpu 4.509 total

Interesting to note the CPU on the incus file operations, makes it seems like it’s mostly idling around so I decided to prod around and test a bit more.


On client:

% sudo hping3 -S -p 22 -c 20 192.168.0.10
20 packets tramitted, 20 packets received, 0% packet loss
round-trip min/avg/max = 3.5/6.3/17.4 ms

Now I add 54ms delay on server so I can get roughly 10x more latency:

$ tc qdisc add dev eno1 root netem delay 54ms

Now on client:

% sudo hping3 -S -p 22 -c 20 192.168.0.10
20 packets tramitted, 20 packets received, 0% packet loss
round-trip min/avg/max = 57.7/61.0/67.6 ms

Cool, RTT avg went from ~6ms to ~60ms.

Now I do the same incus/sftp push/pull test as before from client:

% time incus file push bigfile c1/root/
Pushing bigfile to /root/bigfile: 100% (2.12MB/s)
0.58s user 2.60s system 3% cpu 1:39.46 total

% time incus file pull c1/root/bigfile .
Pulling bigfile from /root/bigfile: 209.71MB (534.73kB/s)
1.68s user 2.44s system 1% cpu 6:33.11 total


% time sftp user:/var/lib/incus/containers/c1/rootfs/root/ <<< 'put bigfile'
Connected to user.
Changing to: /var/lib/incus/containers/c1/rootfs/root/
sftp> put bigfile
Uploading bigfile to /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
100%  200MB  28.6MB/s   00:06    
3.26s user 1.47s system 47% cpu 10.066 total

% time sftp user <<< 'get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile'
Connected to user.
sftp> get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile
Fetching /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile to bigfile
100%  200MB  27.0MB/s   00:07    
3.44s user 0.81s system 40% cpu 10.395 total

I see for incus file push/pull that the speed and CPU usage has dropped by roughly 10x. OTOH the speed and CPU usage for ‘native’ SFTP command decreases by only 1.5x roughly.

Looking at the lxc/incus repo in incus/client/incus_instances.go I see the following:

// GetInstanceFileSFTPConn returns a connection to the instance's SFTP endpoint.
func (r *ProtocolIncus) GetInstanceFileSFTPConn(instanceName string) (net.Conn, error) {
...
	// Get a SFTP client.
	client, err := sftp.NewClientPipe(conn, conn, sftp.MaxPacketUnchecked(128*1024))
	if err != nil {
		_ = conn.Close()
		return nil, err
	}
...

I don’t see MaxConcurrentRequests neither UseConcurrentReads set so I assume the defaults in the Go pkg/sftp should apply (which seem good). But it seems something’s not adding up with the performance and its sensitivity to RTT. I’m too

Now just out of curiosity, still with the ~60ms delay, I tried SFTP from client with a single request (no concurrency) and 128kiB & 32kiB buffer size comparisons:

# 128kiB PUSH
% sftp -B 131072 -R 1 user:/var/lib/incus/containers/c1/rootfs/root/ <<< 'put bigfile'
17%   34MB   3.3MB/s   00:50 ETA^C

# 32kiB PUSH
% sftp -B 32768 -R 1 user:/var/lib/incus/containers/c1/rootfs/root/ <<< 'put bigfile'
8%   17MB   1.0MB/s   03:02 ETA^C

# 128kiB PULL
% sftp -B 131072 -R 1 user <<< 'get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile'
8%   17MB   1.8MB/s   01:43 ETA^C

# 32kiB PULL
% sftp -B 32768 -R 1 user <<< 'get /var/lib/incus/storage-pools/default/containers/c1/rootfs/root/bigfile'
5%   11MB 511.3KB/s   06:19 ETA^C

With 6ms RTT the throughput expectedly bumped up by 10x (I won’t spam the numbers you’ll have to believe me^^).

Looking up speed issues with latency related to the Go sftp pkg I came across this issue:

And this PR:

Which seems to say sftp’s ReadFrom/WriteTo should be used to avoid performance degradation.

It looks like in incus code it’s using util.SafeCopy which uses io.CopyN with 4MiB chunks which wraps the reader in LimitReader which I think won’t use pkg/sftp WriteTo which if so wouldn’t keep the TCP window open and end up breaking the chunks into at most MaxPacket sized packets, e.g. with 32kiB size it’d be 128 packets, taking 7.7s to send them all on 60ms RTT (~500kB/s).

I’ve rambled enough and I’m not very familiar with how this works so I might’ve said some wrong things, I’ll raise an issue on GH soon to give it a bit more attention