The Incus client 6.16 cannot connect to a 6.15 remote server which has a let’s encrypt certificate.
$ incus list foo:
Error: Get "https://foo.example.com:8443/1.0": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-05-18TXX:XX:XXZ is before 2025-07-17TXX:XX:XXZ
$ date --utc
Fr 5. Sep 09:55:43 UTC 2025
When checking the certificate using testssl or curl the connection works.
I did write a small go test program using the net.http import like incus. This did also work without error.
I tested it using my two servers running arch with incus 6.15, and a cluster running 6.15. all with the same certificate (wildcard) which i also use for my other servers.
The clients are incus 6.16 (arch and debian,zabbly). When using incus 6.15 client (arch, debian) the connection has no problem.
One strange thing is the current time changes depending to which remote I connect. It seems to correlate when the remote was added to the client.
Yes that is why I included the date output.
The wrong “current" time” depends on the client and the config of the remote:
Today is 2025-09-05, but incus client thinks otherwise:
[user@clientA] $ incus list serverX:
… not yet valid: current time 2024-02-27TXX:XX:XXZ is before 2025-07-17TXX:XX:XXZ
[user@clientA] $ incus list serverY:
… not yet valid: current time 2023-02-12TXX:XX:XXZ is before 2025-07-17TXX:XX:XXZ
[user@clientA] $ incus list serverZ:
… not yet valid: current time 2024-01-26TXX:XX:XXZ is before 2025-07-17TXX:XX:XXZ
[user@clientB] $ incus list serverX:
… not yet valid: current time 2025-05-18TXX:XX:XXZ is before 2025-07-17TXX:XX:XXZ
But I think I know where the problem is, and how to get it working again. If you create a new remote on the client to the same server but with another name, then the new name works:
[user@clientA] $ incus remote list
| NAME | URL |
|:-----:|:----------------------------:|
| foo | https://foo.example.com:8443 | ** created several months/years ago **
| fnew | https://foo.example.com:8443 | ** created just now **
[user@clientA] $ incus list foo:
Error: Get "https://foo.example.com:8443/1.0": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-01-26TXX:XX:XXZ is before 2025-07-17TXX:XX:XXZ
[user@clientA] $ incus list fnew:
| NAME |
|:----------------:|
| barcontainer |
Just adding the new remote config, without a new token or anything.
Now I can confirm the “current time” in the error message is the “Not Before” date from the client-servercerts certificate:
[user@client]$ openssl x509 -in ~/.config/incus/servercerts/foo.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
xxxxxxxxx
Signature Algorithm: ecdsa-with-SHA384
Issuer: O=Linux Containers, CN=root@foo.example.com
Validity
Not Before: Jan 26 20:39:01 2024 GMT
Not After : Jan 23 20:39:01 2034 GMT
Subject: O=Linux Containers, CN=root@foo.example.com
[user@client]$ incus list foo:
Error: Get "https://foo.example.com:8443/1.0": tls: failed to verify
certificate: x509: certificate has expired or is not yet valid:
current time 2024-01-26T20:39:01Z is before 2025-07-17TXX:XX:XXZ
But then I don’t get it why the newly created remote connection does work? There is no new client certificate in the ~/.config/incus/servercerts/ folder.
The entries in the in the ~/.config/incus/config.yml are the same!!
When you run incus list on an Incus Server, it appears that the client is using the current datetime as provided by the server. That’s how I understand it.
When you run timedatectl or date --utcon each server, do you get a proper current time or is it skewed? The message is quite straightforward that the issue is with the current time.
the real current time on all servers and clients are now. Using the incus 6.15 on client works and is ok. When the time would be a problem, then both versions should have the same error.
Not sure if @masnax still gets notifications from the forum, but there’s a commit from him in 6.16 which tweaks the date check for certificates, so I’m assuming this issue is related to that
// Time returns the current time as the number of seconds since the epoch.
// If Time is nil, TLS uses time.Now. Time func() time.Time
So it’s now always evaluating the validity of the certificate at a time which is not the current time. In particular, in the case which returns time.Unix(0, 0), the certificate is being validated as if today were 1st January 1970.
In the case of the reported problem: I think it’s using the NotBefore time of the locally stored copy of the certificate as the evaluation time - but of course if the remote certificate has been renewed, then that time is prior to the NetBefore time of the remote certificate, so the certificate is considered invalid.
Thank you for the hint. This is indeed the code which breaks things. And I must say that I don’t like it to trust an expired certificate without consent of the user.
I did some debugging and I did find why it works with the newly created remote and not with the old ones. Because I have remote certificates in ~/.config/incus/servercerts/ it checks this folder if there is a file with the name of the remote.
| NAME | URL |
|:-----:|:----------------------------:|
| foo | https://foo.example.com:8443 | ** old, has a servercert foo.crt **
| fnew | https://foo.example.com:8443 | ** new, has no fnew.crt servercert**
In this example the is a remote certificate in the servercerts folder (foo.crt) but for the fnew remote there is no certificate. If there is no fnew.crt file then the code in question is not evaluated because in Line 85 the tlsRemoteCert is nil. Therefore there is no tinkering with time for the whole certificate pool.
But if there is a remote certificate, and this code in question is broken. So there is a certificate, it has a NotBefore AND a NotAfter time. NotBefore is before the real time, and NotAfter is after the real time. So there should be no tinkering with the time. But here in Line 96 the if block is ignored since both times are non zero.
95 tlsConfig.Time = func() time.Time {
96 if tlsRemoteCert.NotBefore.IsZero() && !tlsRemoteCert.NotAfter.IsZero() {
97 // If the value is zero, it will be overwritten with time.Now, so use the epoch time instead.
98 return time.Unix(0, 0)
99 }
100
101 return tlsRemoteCert.NotBefore
102 }
