Something weird happens when trying to copy image to own remote server

Incus
1

Cheers,

I have a remote Debian 11 server with Incus 6.10.1 installed. It listens on 0.0.0.0:8443, I think nftables has been set up correctly.

Also I have a local machine running Incus and above remote server has been set up as remote with trust token (server: https://server.com:8443). This local Incus has been set up to listen on 0.0.0.0:8443 as well. This local Linux utilizes a “br0” bridge which connects multiple Ethernet ports together and this “br0” bridge has an IP of 192.168.1.30.

So when I do this:
$ incus image list server:

it correctly lists all images on remote server, great.

Moreover, when I do a copy on this local machine from images (i.e. images.linuxcontainers.org) to server, it works perfectly correct:

$ incus image copy images:debian/10 server:
Image copied successfully!

BUT: when I try copying local image to server, e.g.:
$ incus image copy local:debian/10 server:

it says:
Error: Failed remote image download: Failed to connect to the server “https://192.168.1.30:8443”: Get “https://192.168.1.30:8443/1.0”: Unable to connect to: 192.168.1.30:8443 ([dial tcp 192.168.1.30:8443: i/o timeout])

(192.168.1.30:8443 is the local Incus API https listener)

So it seems everything is OK except when trying to copy image from local machine to server. Why is that? Maybe it’s the local “br0” bridge on which local Incus listens on? I just can’t understand what’s going on here.

I also tried “incus monitor” on local:

[user@local ~]$ incus monitor --type=logging --loglevel=debug --pretty

DEBUG [2025-03-14T22:16:56+01:00] Event listener server handler started id=f0c95ec9-1511-4860-a307-977605772f07 local=/var/lib/incus/unix.socket remote=@
DEBUG [2025-03-14T22:17:05+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=user
DEBUG [2025-03-14T22:17:05+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/images/aliases/a1 username=user
DEBUG [2025-03-14T22:17:05+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/images/a1 username= user
DEBUG [2025-03-14T22:17:05+01:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username= user
DEBUG [2025-03-14T22:17:05+01:00] Event listener server handler started id=b1493c73-d7af-4d7e-8e4a-8dc450ac428c local=/var/lib/incus/unix.socket remote=@
DEBUG [2025-03-14T22:17:05+01:00] Handling API request ip=@ method=POST protocol=unix url=/1.0/images/a19703d827052420fc2dc0cf592fa7aa2b2831041e8abec0d2ce671c0e418d41/secret username= user
DEBUG [2025-03-14T22:17:05+01:00] New operation class=token description=“Image download token” operation=31b40854-ee30-430f-9437-eb8f000b8b58 project=default
DEBUG [2025-03-14T22:17:05+01:00] Started operation class=token description=“Image download token” operation=31b40854-ee30-430f-9437-eb8f000b8b58 project=default
DEBUG [2025-03-14T22:17:25+01:00] Event listener server handler stopped listener=b1493c73-d7af-4d7e-8e4a-8dc450ac428c local=/var/lib/incus/unix.socket remote=@

(what does “@” mean here?)

So repeatedly: everything works except copying a local image to remote server. When I try to do that, it says it can’t connect to LOCAL Incus https API endpoint. So confused.
Thanks.

2

Checking on new local machines in several ways – same phenomenon. It seems the problem is at the abovementioned server (nftables perhaps). But Incus error reporting is still weird: it says there’s a problem at local connection, not at remote server – as such, I presume it lies as everything works except that “incus image copy” to that server. Investigating further.

3

Downgraded server Incus to 6.0.3 to no avail, same. Client is at 6.0.0.

Can even launch image on remote and CAN DO ANYTHING but simply can’t copy image to remote.

What’s on Earth is different with “incus image copy”?

4

The way incus image copy and incus copy work is that they instruct the target server to connect to the source for the download. Apparently there is some kind of network issue that’s preventing that target server from connecting to your local server.

You can reverse the flow with --mode=push or use --mode=relay for the CLI tool itself to relay the data. That latter option basically always works but causes more CPU and network usage as it needs to relay everything.

1 Like
5

OMG, with --mode=relay it works great!

Thank you a lot! Sorry for missing this option. I think I will check the source how that works.

Thanks again! <3