Tailscale : Reset service configuration (state)

Hallo :waving_hand:

I configured the Tailscale service to connect to an account but I’d like to change that account.
Tried to change the auth-key, disable/enable the service, delete state; it is not working, still connecting to the first account.

Tried to reset the service but getting the error that this is not supported.

How can I proceed ?

Version	202607270148

It looks like my new auth-key is not being used/sent when reenabling the service.

state:
  backend_state: NeedsLogin
  self:
    public_key: nodekey:0000000000000000000000000000000000000000000000000000000000000000
  have_node_key: true

According to incus-os/incus-osd/internal/services/service_tailscale.go at main · lxc/incus-os · GitHub

A change in:

  • Enabled
  • LoginServer
  • AuthKey

Would trigger incus-os/incus-osd/internal/services/service_tailscale.go at main · lxc/incus-os · GitHub

Tried all of them, more than once, and nothing…
Most of the time, the update to one of these values never returns…
(incus config edit ... on save never returns)

There never seems to be any logout.

Did an analysis with my friend Claude based of my many tests;

# Tailscale service: failed `configure` leaves config unrecoverably wedged, and `up` can hang indefinitely

## Summary

When re-authenticating the Tailscale service to a different tailnet, a failed or
blocking `configure` run leaves the service in a state that cannot be recovered
through the API. Four issues in `incus-osd/internal/services/service_tailscale.go`
compound:

1. The new config is committed to state even when applying it fails.
2. `needsRejoin` is a pure config diff and ignores `BackendState`, so the
   now-committed config makes every subsequent apply a no-op.
3. `tailscale up` is invoked with no timeout and no `--timeout` flag, so a login
   that does not resolve hangs the API call indefinitely.
4. `configure` uses `tailscale down` rather than `tailscale logout`, which does not
   clear login state — the case `needsRejoin` exists to handle.

Because IncusOS has no shell and does not expose the `tailscale` CLI, there is no
workaround once this state is reached.

## Environment

- IncusOS: `<version>`
- Tailscale service enabled, previously working against tailnet A
- Goal: move the node to tailnet B (device removed from A, new auth key issued in B)

## Steps to reproduce

1. With the Tailscale service enabled and registered against tailnet A, remove the
   node from tailnet A's admin console.
2. Set `auth_key` to a key from tailnet B.
3. Observe that the apply does not return.
4. Interrupt the client. Read the service config: `auth_key` now holds the new key.
5. Re-apply the same config. It returns immediately and does nothing —
   `backend_state` stays `NeedsLogin`.

## Observed

- Step 3 blocks with no output, no error, and no `health` entries.
- Step 4 shows the key persisted despite the apply not succeeding.
- Step 5 silently performs no login attempt.
- Escaping step 5 requires changing `auth_key` or `login_server` to a *different*
  value, which for single-use auth keys means consuming a new key per attempt.
- There is no indication of which subprocess call is blocking.

## Expected

- A failed apply should not commit config to state.
- A node in `NeedsLogin` with a configured `auth_key` should retry the login.
- `up` should time out rather than block the API call forever.
- Re-applying an unchanged config should be safe to retry.

## Analysis

### 1. Config committed on failed apply

In `Update`, `n.state.Save()` is deferred at function entry, and
`n.state.Services.Tailscale.Config = newState.Config` happens *before* `Start` and
`configure` are called. When `configure` returns an error — including a context
cancellation from an interrupted client — the error propagates to the caller, but
the deferred save still commits the new config.

Suggested fix: assign and save only after `configure` succeeds, or snapshot and
roll back the previous config on error.

### 2. `needsRejoin` ignores backend state

`needsRejoin` returns true only when `Enabled`, `LoginServer`, or `AuthKey` differ
between old and new config. Combined with issue 1, a key that has been committed but
never successfully used can never be retried: the diff is empty, so `configure`
skips the `down`/`up` block and proceeds directly to `tailscale set`, which performs
no authentication.

`configure` is also reachable only from `Update` — the boot path consults
`ShouldStart` and calls `Start` only — so nothing reconciles this state over time,
and rebooting does not retry the login.

Suggested fix: treat `BackendState` of `NeedsLogin` (or `NoState`) with a non-empty
`AuthKey` as requiring a rejoin, independent of whether the config changed.

### 3. Unbounded `tailscale up`

`configure` passes the request context straight to `RunCommandContext` for `down`,
`up`, and `set`. `tailscale up` blocks until the backend reaches `Running`; if
registration does not resolve, it never returns. Tailscale's CLI provides
`--timeout` for exactly this, and it is not used.

Note the inconsistency: the `serve` call *is* wrapped in a 5s
`context.WithTimeout`, with a comment noting it goes interactive when the tailnet
admin has not provisioned HTTPS. The same hazard applies to `up` and is unguarded.

Suggested fix: pass `--timeout` to `up` and/or wrap these calls in a bounded
context, so the API returns a diagnosable error instead of hanging.

### 4. `down` does not clear login state

`configure` runs `tailscale down` before `up --reset --auth-key`. `down` only marks
the backend as not wanting to run; `--reset` resets preferences. Neither clears
login state. For the specific case `needsRejoin` is meant to cover — moving to a
different tailnet or login server — `tailscale logout` is the operation that
actually applies.

Additionally, `configure` returns early on a non-zero exit from `down`, so if `down`
fails or blocks, `up` is never reached. `Get` already special-cases
`Failed to connect to local Tailscale daemon`, acknowledging the daemon may be
unreachable; `configure` has no equivalent tolerance, and `Start` returns as soon as
systemd reports the unit started rather than waiting for the daemon's local socket.

Suggested fix: use `logout` when `needsRejoin` is set, and tolerate a failure from
the teardown step rather than aborting the rejoin.

### 5. No diagnosability

`configure` runs several blocking subprocess calls in sequence with no logging of
which one is executing. On a system with no shell and no access to the `tailscale`
CLI, a hang inside `configure` is entirely opaque — the operator cannot determine
which command is blocking or why. Logging each step, and surfacing subprocess
stderr, would make this class of problem self-diagnosing.

## Unresolved

I have not established why the login itself fails to resolve in my case. A refused
auth key would make `up` exit non-zero with a message; the observed behaviour is
silent indefinite blocking with an empty `health` array, which suggests the
registration request is not getting a definitive answer. That may be specific to my
network path to `controlplane.tailscale.com` and not an IncusOS problem at all.

The issues above are independent of that root cause: they are why a recoverable
authentication failure becomes an unrecoverable service state with no diagnostic
output.

Awesome, thanks