Error: Certificate already in trust store

LXD
1

Hello,

I am trying to add a remote server for a container migration.

$ lxc remote add lko 192.168.1.111
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Admin password for lko: 
Error: Certificate already in trust store

The remote server ‘lko’ does not get added,

$ lxc remote list
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+

$ lxd sql global "SELECT * FROM certificates;"         
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+----+-------------+------+------+-------------+------------+
| id | fingerprint | type | name | certificate | restricted |
+----+-------------+------+------+-------------+------------+
+----+-------------+------+------+-------------+------------+

Is there any way to fix this?

2

Can you show output of lxd sql global "SELECT * FROM certificates;" from both servers involved?

3

Also what version of LXD are you running?

4

I am running LXD 4.13 on both ends.

There are some old entries when issuing lxd sql global "SELECT * FROM certificates;" but nothing for the ip address of interest.

Is it possible to empty the certificates?

5

On the server with old entries please can you run:

lxc config trust ls

And on your new client machine please run:

openssl x509 -noout -fingerprint -sha256 -inform pem -in ~/snap/lxd/current/.config/lxc/client.crt
6

On the server,

$ lxc config trust ls
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+--------------+-------------+------------------------------+------------------------------+
| FINGERPRINT  | COMMON NAME |          ISSUE DATE          |         EXPIRY DATE          |
+--------------+-------------+------------------------------+------------------------------+
| 96430c120df1 | usr@nas     | May 9, 2021 at 1:07am (UTC)  | May 7, 2031 at 1:07am (UTC)  |
+--------------+-------------+------------------------------+------------------------------+
| bef30861f69c | usr@nas     | Jun 2, 2019 at 2:25pm (UTC)  | May 30, 2029 at 2:25pm (UTC) |
+--------------+-------------+------------------------------+------------------------------+
| cd4c4e3ea708 | usr@nas     | Feb 9, 2019 at 5:01am (UTC)  | Feb 6, 2029 at 5:01am (UTC)  |
+--------------+-------------+------------------------------+------------------------------+
| fe17385b206e | root@nas    | May 23, 2020 at 2:52pm (UTC) | May 21, 2030 at 2:52pm (UTC) |
+--------------+-------------+------------------------------+------------------------------+

On the client,

$ openssl x509 -noout -fingerprint -sha256 -inform pem -in ~/snap/lxd/current/.config/lxc/client.crt

SHA256 Fingerprint=96:43:0C:12:0D:F1:1D:DA:A6:F2:8F:58:3C:E9:F4:07:31:49:F7:F6:7C:A3:F4:02:55:6E:59:44:90:CF:2C:54
7

So this one appears to be for the same certificate as your new client certificate:

If that is not in use by any existing client you can remove it using:

lxc config trust remove <fingerprint>
8

ok, tried this. Got a different message this time.

$ lxc remote add lko 192.168.1.111
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Admin password for lko: 
Error: Server doesn't trust us after authentication
9

Can you show output of lxc config trust ls on existing server again please.

Also please can you show output of lxc config show on existing server.

11

On the server (fingerprint entry gets added after a remote add request),

$ lxc config trust ls
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+--------------+-------------+------------------------------+------------------------------+
| FINGERPRINT  | COMMON NAME |          ISSUE DATE          |         EXPIRY DATE          |
+--------------+-------------+------------------------------+------------------------------+
| 96430c120df1 | usr@nas     | May 9, 2021 at 1:07am (UTC)  | May 7, 2031 at 1:07am (UTC)  |
+--------------+-------------+------------------------------+------------------------------+
| bef30861f69c | usr@nas     | Jun 2, 2019 at 2:25pm (UTC)  | May 30, 2029 at 2:25pm (UTC) |
+--------------+-------------+------------------------------+------------------------------+
| cd4c4e3ea708 | usr@nas     | Feb 9, 2019 at 5:01am (UTC)  | Feb 6, 2029 at 5:01am (UTC)  |
+--------------+-------------+------------------------------+------------------------------+
| fe17385b206e | root@nas    | May 23, 2020 at 2:52pm (UTC) | May 21, 2030 at 2:52pm (UTC) |
+--------------+-------------+------------------------------+------------------------------+

$ lxc config show 
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
config:
  core.https_address: 192.168.1.111:8443
  core.trust_password: true
12

Are you using those other usr@nas certs, if not I would remove them from the trust store too.
They might be other existing clients though so double check.

13

Ok, emptied the fingerprints on the server

$ lxc config trust ls
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+-------------+-------------+------------+-------------+
| FINGERPRINT | COMMON NAME | ISSUE DATE | EXPIRY DATE |
+-------------+-------------+------------+-------------+

On the client,

$ lxc remote add lko 192.168.1.111
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Admin password for lko: 
Error: Server doesn't trust us after authentication

Server now shows,

$ lxc config trust ls
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+--------------+-------------+-----------------------------+-----------------------------+
| FINGERPRINT  | COMMON NAME |         ISSUE DATE          |         EXPIRY DATE         |
+--------------+-------------+-----------------------------+-----------------------------+
| 96430c120df1 | usr@nas     | May 9, 2021 at 1:07am (UTC) | May 7, 2031 at 1:07am (UTC) |
+--------------+-------------+-----------------------------+-----------------------------+
14

Can you try again, but this time run this at the same time on the existing server and look for any messages that might indicate the problem:

lxc monitor --type=logging --pretty
15

Got this,

$ lxc monitor --type=logging --pretty
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
DBUG[05-09|03:18:25] New event listener: 9f5ea010-959f-41f5-96f2-768920133631  
DBUG[05-09|03:18:31] Allowing untrusted GET                   ip=172.16.1.5:49018 url=/1.0
DBUG[05-09|03:18:31] Allowing untrusted GET                   url=/1.0 ip=172.16.1.5:49020
DBUG[05-09|03:18:33] Allowing untrusted POST                  ip=172.16.1.5:49062 url=/1.0/certificates
DBUG[05-09|03:18:33] Database error: &errors.errorString{s:"No such object"}  
DBUG[05-09|03:18:33] Allowing untrusted GET                   ip=172.16.1.5:49064 url=/1.0

Also, not sure if it is related. On the ‘client’ i changed my pool today using,

lxd sql global "UPDATE storage_pools_config SET value='ssdpool3/lxd' WHERE value='ssdpool1/lxd';"

16

Are you going through any sort of proxy or load balancer between client and server? Something that might mess with the TLS certs?

17

No there is no proxy in between. The machines are connected over an ipsec tunnel.

18

Hrm, I’m not sure then.

@stgraber do you have any ideas what this TLS remote trust issue is (on 4.13)? Thanks

19

I just tested a fresh install of LXD 4.13 and adding remotes works fine. Also tested 4.14 as there have been some changes to that subsystem in there, and thats working fine also.

20

The Allowing untrusted POST bit is what your client does after entering the trust password to add its cert to the trust store (which we can see succeeding), and afterwards it does the request to GET /1.0 to check its authenticated. But at this stage it still isn’t despite its cert being in the trust store.

21

Ok. Let me take a step back. I have a firewall rule so that the client (172.16.1.5) can talk to the remote server (192.168.1.111), however the server is not allowed to initiate connection to 172.16.1.5.