Incus, PKI, and Step CA

Incus
1

Is there a way for Incus to be set up to get server and cluster certificates from Step CA? Ideally, the certificates would be short-lived and frequently updated.

Looking at the docs and a few other threads, I’ve seen information on using an existing PKI with Incus.

  • An earlier recommendation was to use an existing PKI to get the server cert and key before creating a cluster.
  • The server settings docs discuss integrating PKI but involve storing the server and client certs in directories (server-side and client-side). This seems problematic for short-lived and constantly-renewing certs.
    • The cluster certificate may be even more problematic since it probably needs to be the same on each cluster member. Perhaps the cluster cert should be longer-lived. (But what happens if it needs to be renewed?)
    • For client certs, getting frequently-renewed certs uploaded to all cluster members also seems like waves hands work.
  • There are server CA trust and ACME settings, but I’m not quite clear if those can be used for the server and cluster certs.
  • The docs didn’t seem to specify if the CA file on the server(s) should be in a specific format. I presume PEM but want to verify. It would also be good to know if it should be the full CA cert chain.

I started trying to set this up and tested but fear that Claude led me astray. :slight_smile: Before going further, I wanted to figure out if there are recommendations for a CA like Step CA.

Cluster vs. server certificates and PKI
2

Having this exact setup configured for year(s) (without the cluster setup) so you might inspire in my working setup.
Step-ca as my root and intermediate CA for infra part.
certificates for each machine is obtained and handled via certbot with deployment hook which restart the incus when certificate is renewed.

  1. I assume you have correctly configured step-ca root (and intermediate) ca to the system-wide CA store

  2. obtain a certificate from you step-ca with certbot or any other ACME client:

certbot certonly -n --standalone -d machine.hostna.me
–server https://your.step.ca/acme/acme/directory 
–non-interactive 
–agree-tos --email some@email.tld
–http-01-address ip.of.the.host
–deploy-hook “/path/to/deploy/hook.sh”

(create deploy hook script with some way to restart the incus systemctl restart incus)

  1. configure incus to have those certificates (symlink server certificates to the certbot obtained and your root cert:
root@your.machine:/var/lib/incus# ls -la |grep server.
lrwxrwxrwx  1 root root           50 Nov 24 22:48 server.ca → /usr/local/share/ca-certificates/infra-root-ca.crt
lrwxrwxrwx  1 root root           61 Nov 24 22:48 server.crt → /etc/letsencrypt/live/machine.host.name/fullchain.pem
lrwxrwxrwx  1 root root           59 Nov 24 22:48 server.key → /etc/letsencrypt/live/machine.host.name/privkey.pem
  1. restart the incus
  2. configure CA trust
incus config set core.trust_ca_certificates true
  1. optionally you can link the client certs to the certbot certs as well:
root@your.machine:~/.config/incus# ls -la |grep client.
lrwxrwxrwx 1 root root   50 Nov 24 22:48 client.ca → /usr/local/share/ca-certificates/infra-root-ca.crt
lrwxrwxrwx 1 root root   61 Nov 24 22:48 client.crt → /etc/letsencrypt/live/machine.host.name/fullchain.pem
lrwxrwxrwx 1 root root   59 Nov 24 22:48 client.key → /etc/letsencrypt/live/machine.host.name/privkey.pem

There might be some differences in incus-cluster setup. Your mileage may vary.

3

Cool, thanks for that real-world success info! I was looking at using step CLI directly, and renewing as per these docs from Smallstep.

I did incus init on the first server and had it start the cluster. I’m thinking I need to figure out how to reset that / start over there and just set up a single server first.

Doing client cert auth (which I may or may not actually do) and setting up the cluster with PKI definitely seems to add complications.

For the server cert, I did this after step ca bootstrap:

`sudo step ca certificate --san --san --san --san /var/lib/incus/server.crt /var/lib/incus/server.key`

This did put the server cert and key in the right place. However, since I had done incus init and set up clustering, I found that the API connections were using the cluster cert. I tested with:

sudo step certificate inspect /var/lib/incus/server.crt --fingerprint

sudo step certificate inspect /var/lib/incus/cluster.crt --fingerprint

step certificate inspect https://<server DNS name>:8443 --fingerprint

The HTTPS connection matched the cluster.crt certificate fingerprint.

4

If I reveal it correctly I had some problems with cluster/server certs back then so I went with the separate nodes (no cluster). But I’d try to just symlink cluster.* to the server.* files generated by step ca certificate and restart it and see what would happen.