My incus lives behind a Traefik reverse proxy. The web UI is accessible no problem, but when I try to run the incus remote add incus.example.com https://incus.example.com:443 --token eyJjbGllbnRfbm…, I get the error message Error: http status not ok: 400 Bad Request {}.
I’ve tried setting up the reverse proxy either as an https terminator (incus.example.com), or a tcp forwarder (incus-cli.example.com). The server log is slightly different in either case (untrusted GET vs. matching trusted cert), but the error persists.
Adding the --debug flag shows the json response below, so I know this is actually the incus API responding.
You won’t be able to do anything related to TLS authentication when you have a proxy terminating TLS in the middle.
So you definitely need to use the TCP forwarder approach.
I’d recommend testing that the token works correctly without going through the proxy, if it does, then your proxy is doing something wrong somehow.
I’m not familiar with Traefik but all my production clusters use HAproxy in TCP forwarder/load-balancer and that works perfectly with HTTPS client certificates.
Thank you Stéphane. I also tried bypassing the reverse proxy, but couldn’t find out how to disable client-side the server cert, or generate a cert with an alternative name. If neither is possible, I can still test by forcing the hosts file, but an alt name cert would be preferable.
incus remote add should just prompt for the fingerprint in such cases and then trust the certificate based on that when that (this step is skipped when the certificate is found to be directly valid).
Indeed, and I’m still getting a 400 error with a direct connection:
❯ incus remote add incus.example.com https://100.111.228.5:8443 --token eyJjbGllbnRfbmFtZSI6ImJydWluZW4i…
Certificate fingerprint: 3acb8ba6a7025f1b7400655c646167a16afc0a5bf2e41aec2c91f9ade4230a4a
ok (y/n/[fingerprint])? y
Error: http status not ok: 400 Bad Request {}
Going back to my original message, where can I see the server-side error that is translated to an http error code on the client?
It was not… but when I created a new one and tried to connect the remote again, I still got the 400 error, both through the reverse proxy and using a direct connection.
Can you create one and then immediately use it with the direct connection, never try to use it through the proxy?
As soon as a token is seen by the server, it gets invalidated, so if your proxy is doing something weird, the token will get invalidated, then when you try to use it directly, you’re getting a failure because no such token exists.
You can try to run incus monitor --pretty on the server side to see all log messages as they happen. Though I don’t know how chatty this particular part of the code is.
We try not to log too much pre-authentication as otherwise someone can use that to DoS the machine by causing a lot of spurious logging.
Can you show env | grep -i proxy? Some kind of corporate HTTP proxy would also have the same effect of terminating TLS in a way that prevents the certificate from reaching the server, causing the same error.
Hmm, actually, I wonder if you’re not facing the opposite problem now, effectively tying to get Incus to add a new entry, using a token when it already trusts you
What happens if you don’t pass --token XYZ, just incus remote add incus.example.com https://100.111.228.5:8443
❯ incus remote add incus.example.com https://100.111.228.5:8443
Error: http status not ok: 400 Bad Request {}
I tried removing the token on the server incus config trust revoke-token bruinen and running the command again (with no token available on the server) → still 400.
I added a new token and ran remote add once more → still 400.
So the error must happen before the server checks the token.
I checked that my earlier remote does work on this client:
❯ incus remote ls
+-------------------------------+------------------------------------+---------------+-------------+--------+--------+--------+
| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC | GLOBAL |
+-------------------------------+------------------------------------+---------------+-------------+--------+--------+--------+
| images | https://images.linuxcontainers.org | simplestreams | none | YES | NO | NO |
+-------------------------------+------------------------------------+---------------+-------------+--------+--------+--------+
| incus.int.example.com (current) | https://incus.int.example.com:8443 | incus | tls | NO | NO | NO |
+-------------------------------+------------------------------------+---------------+-------------+--------+--------+--------+
| local | unix:// | incus | file access | NO | YES | NO |
+-------------------------------+------------------------------------+---------------+-------------+--------+--------+--------+
❯ incus ls
Error: Get "https://incus.int.example.com:8443/1.0": tls: failed to verify certificate: x509: certificate is valid for incus-cli.example.com, not incus.int.example.com
❯ subl /etc/hosts # add incus-cli.example.com to hosts
❯ doggo incus.int.example.com
NAME TYPE CLASS TTL ADDRESS NAMESERVER
incus.int.example.com. A IN 1s 192.168.3.5 100.100.100.100:53
❯ incus remote set-url incus.int.example.com https://incus-cli.example.com:8443
❯ incus ls
+-----------+---------+------------------------------+----------------------------------------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-----------+---------+------------------------------+----------------------------------------+-----------------+-----------+
| docker | STOPPED | | | CONTAINER | 0 |
+-----------+---------+------------------------------+----------------------------------------+-----------------+-----------+
| docker-vm | RUNNING | 192.168.3.105 (enp5s0) | fd7a:115c:… (tailscale0) | VIRTUAL-MACHINE | 0 |
… but it would be nice to be able to add a remote with the external name and IP.
stgraber@castiana:~$ incus remote add dakara https://dakara.lan:8443 --token='eyJjbGllbnRfbmFtZSI6ImZvbyIsImZpbmdlcnByaW50IjoiZDRjZjY1MjJlZjYzMzMwOTVhYWQ2MWE2ODY3NjdmNWUyNDRmMjYzMjkwODY0MDY0YTlhYWRjZmNjYmY3MmFlNyIsImFkZHJlc3NlcyI6WyIxNzIuMTcuMC4xMDA6ODQ0MyIsIlsyNjAyOmZjNjI6YzoxMDAwOmZhNTpjMTQ2OjYzNzU6ZWY0ZF06ODQ0MyIsIjEwLjAuMy4xOjg0NDMiLCJbZmM0Mjo1MDA5OmJhNGI6NWFiMDo6MV06ODQ0MyIsIltmZDQyOmI0YjA6ZDA3OToxMDhkOjoxXTo4NDQzIiwiMTcyLjMxLjI1NC4xOjg0NDMiLCJbZmQwMDoxZTRkOjYzN2Q6MTIzNDo6MV06ODQ0MyIsIjEwLjMwLjQuMTo4NDQzIiwiW2ZkNDI6MjlkNDoyMmYxOmE3MTQ6OjFdOjg0NDMiLCIxNzIuMTcuMjUwLjE6ODQ0MyIsIlsyNjAyOmZjNjI6YzoyNTA6OjFdOjg0NDMiLCJbMjYwMjpmYzYyOmVmOjEwMDA6Y2Y3OTo5NWZmOmZlZTE6ZjA4Yl06ODQ0MyIsIjEwLjEyNC4wLjE6ODQ0MyIsIltmZDQyOjEyMzQ6MTIzNDoxMjQ6NGY3MzpjNmZmOmZlZGM6MWVjMV06ODQ0MyJdLCJzZWNyZXQiOiIxZjJhZGIwZjA2NWUyYTg2NmE5MDM3ZGU3OTliNTczZDljODViYWE3Nzk2YWM2YzBlMzFlNWRlMmY5YzJlM2NhIiwiZXhwaXJlc19hdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIn0='
Certificate fingerprint: d4cf6522ef6333095aad61a686767f5e244f263290864064a9aadcfccbf72ae7
ok (y/n/[fingerprint])? y
Error: No matching certificate add operation found
Passing a valid token for the first time:
stgraber@castiana:~$ incus remote add dakara https://dakara.lan:8443 --token='eyJjbGllbnRfbmFtZSI6ImZvbyIsImZpbmdlcnByaW50IjoiZDRjZjY1MjJlZjYzMzMwOTVhYWQ2MWE2ODY3NjdmNWUyNDRmMjYzMjkwODY0MDY0YTlhYWRjZmNjYmY3MmFlNyIsImFkZHJlc3NlcyI6WyIxNzIuMTcuMC4xMDA6ODQ0MyIsIlsyNjAyOmZjNjI6YzoxMDAwOmZhNTpjMTQ2OjYzNzU6ZWY0ZF06ODQ0MyIsIjEwLjAuMy4xOjg0NDMiLCJbZmM0Mjo1MDA5OmJhNGI6NWFiMDo6MV06ODQ0MyIsIltmZDQyOmI0YjA6ZDA3OToxMDhkOjoxXTo4NDQzIiwiMTcyLjMxLjI1NC4xOjg0NDMiLCJbZmQwMDoxZTRkOjYzN2Q6MTIzNDo6MV06ODQ0MyIsIjEwLjMwLjQuMTo4NDQzIiwiW2ZkNDI6MjlkNDoyMmYxOmE3MTQ6OjFdOjg0NDMiLCIxNzIuMTcuMjUwLjE6ODQ0MyIsIlsyNjAyOmZjNjI6YzoyNTA6OjFdOjg0NDMiLCJbMjYwMjpmYzYyOmVmOjEwMDA6Y2Y3OTo5NWZmOmZlZTE6ZjA4Yl06ODQ0MyIsIjEwLjEyNC4wLjE6ODQ0MyIsIltmZDQyOjEyMzQ6MTIzNDoxMjQ6NGY3MzpjNmZmOmZlZGM6MWVjMV06ODQ0MyJdLCJzZWNyZXQiOiI3ODg2YjM1ZjY1OWJhMmU5NDM2MjZhOWUxZTQwNWU5OTlkMzQ5ZjczYTk5MmE5YzU3NDA1NTVmMDQyMjFhNzAzIiwiZXhwaXJlc19hdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIn0='
Certificate fingerprint: d4cf6522ef6333095aad61a686767f5e244f263290864064a9aadcfccbf72ae7
ok (y/n/[fingerprint])? y
Client certificate now trusted by server: dakara
Passing a valid or invalid token after the machine has already been trusted:
stgraber@castiana:~$ incus remote add dakara https://dakara.lan:8443 --token='eyJjbGllbnRfbmFtZSI6ImZvbyIsImZpbmdlcnByaW50IjoiZDRjZjY1MjJlZjYzMzMwOTVhYWQ2MWE2ODY3NjdmNWUyNDRmMjYzMjkwODY0MDY0YTlhYWRjZmNjYmY3MmFlNyIsImFkZHJlc3NlcyI6WyIxNzIuMTcuMC4xMDA6ODQ0MyIsIlsyNjAyOmZjNjI6YzoxMDAwOmZhNTpjMTQ2OjYzNzU6ZWY0ZF06ODQ0MyIsIjEwLjAuMy4xOjg0NDMiLCJbZmM0Mjo1MDA5OmJhNGI6NWFiMDo6MV06ODQ0MyIsIltmZDQyOmI0YjA6ZDA3OToxMDhkOjoxXTo4NDQzIiwiMTcyLjMxLjI1NC4xOjg0NDMiLCJbZmQwMDoxZTRkOjYzN2Q6MTIzNDo6MV06ODQ0MyIsIjEwLjMwLjQuMTo4NDQzIiwiW2ZkNDI6MjlkNDoyMmYxOmE3MTQ6OjFdOjg0NDMiLCIxNzIuMTcuMjUwLjE6ODQ0MyIsIlsyNjAyOmZjNjI6YzoyNTA6OjFdOjg0NDMiLCJbMjYwMjpmYzYyOmVmOjEwMDA6Y2Y3OTo5NWZmOmZlZTE6ZjA4Yl06ODQ0MyIsIjEwLjEyNC4wLjE6ODQ0MyIsIltmZDQyOjEyMzQ6MTIzNDoxMjQ6NGY3MzpjNmZmOmZlZGM6MWVjMV06ODQ0MyJdLCJzZWNyZXQiOiI3ODg2YjM1ZjY1OWJhMmU5NDM2MjZhOWUxZTQwNWU5OTlkMzQ5ZjczYTk5MmE5YzU3NDA1NTVmMDQyMjFhNzAzIiwiZXhwaXJlc19hdCI6IjAwMDEtMDEtMDFUMDA6MDA6MDBaIn0='
Certificate fingerprint: d4cf6522ef6333095aad61a686767f5e244f263290864064a9aadcfccbf72ae7
ok (y/n/[fingerprint])? y
Client certificate now trusted by server: dakara
We never have Incus log http status not ok so that’s not an error that’s coming from our side.
It is however an error that can come out of Zitadel OIDC’s implementation.
Is your server configured with some (potentially broken) OIDC configuration?
If so, the client will always prefer OIDC authentication unless told otherwise with --auth-type=tls, so that may be the problem here.
I do use Authelia for OIDC, and it’s working great, but despite the fact that the tcp forwarder for incus-cli doesn’t use forward auth, I understand that incus itself is trying to use oidc (although I’m having trouble understanding how that would work without a client-side oidc implementation).
A few follow-up questions if you’re not totally fed up with me:
(more of an observation) I was able to use incus remote add incus.example.com https://incus.example.com:443 --token eyJjbGllbnRfbmFtZSI6I… --auth-type=tls (the https reverse-proxy)
here’s the question: does this mean that my incus server now trusts my reverse-proxy, and that any query coming from that will be authorized (bad)? Or is the auth tied to my client (good)?
I still have the issue that the server doesn’t have an alternative name in its cert: incus ls incus.int.example.com: → Error: Get “https://incus.int.example.com:8443/1.0”: tls: failed to verify certificate: x509: certificate is valid for incus-cli.example.com, not incus.int.example.com
is that something I can work around client-side (–insecure flag to allow invalid server cert) or server-side (add alt name)?
Easiest is to check it. Run incus remote get-client-certificate on your client and go look at incus config trust list and incus config trust show on the server side, if the certificate matches, you’re good, it’s safe end to end. If it doesn’t match, then you’ve trusted an intermediate certificate like that of your proxy.
That’s confusing. Is that your old remote? If so, remove it and add it back, that should get you the certificate fingerprint prompt at which point it will trust that certificate regardless of what the CN is.
The Incus CLI does have a built-in OIDC client. I use that with all my production servers for authentication. Maybe it’s just a config issue on your IdP that’s getting in the way as the CLI requires support for device code authentication.
