I have a three-node cluster, provisioned via Operations Center. Manually triggered (via UI) a rolling update, and the first node in the cluster became unresponsive (step 1/27). When I connected a monitor there was a message along the lines of the system had updated and needed restarting. That was unexpected - I assumed the process would be automatic - but I restarted the node.
It didn’t come up properly, hanging at the ‘Starting application …’ step for around 15 minutes (see image) and then failed.
Before anyone comments on the recovery keys - this is a two-week old rebuild of a cluster, and everything deploys from Terraform so I’m treating it as transient.
Are you able to interact with that fallback listener? It would be good to be able to pull some more detailed logs from that system to see what’s going on exactly.
@gibmat the documentation on the fallback listener is a bit light on how exactly to use it? Feel like adding a paragraph to that doc page? I don’t recall if it looks sufficiently Incus-like that you can incus remote add it and use regular incus admin os commands against it, or if it only serves the API at its root and therefore requires curl.
@gibmat also, I thought we had a flag to force it to be always on? But now I’m not seeing it and just setting a listen address doesn’t cause it to come online either.
It looks like the fallback listener is using a client cert generated by Operations Center (OC). OC was seeded with my client certificate, and I’ve confirmed that this is beign used by its own fallback listener (at least in the config). But the nodes have OC’s self-signed fallback certificate, which I guess is why I’m getting a 403.
