Incus exec failing for vms with a websocket: bad handshake

Incus
1

I’m running incus 6.15 on Fedora 42, installed from Zabbly repos.

All of a sudden, incus exec started giving me a websocket: bad handshake error when I use it with vm instances, while it works just fine when used with containers:

❯ incus exec fedora -- ls /
Error: websocket: bad handshake

❯ incus exec noble-lxc -- ls /
bin  bin.usr-is-merged	boot  dev  etc	home  lib  lib.usr-is-merged  lib64  media  mnt  opt  proc  root  run  sbin  sbin.usr-is-merged  srv  sys  tmp	usr  var

fedora is a vm that I created a while ago from the official fedora/42/cloud image. I can SSH to it and confirm that the agent is running. Below are the logs that I get from it:

journalctl -b -u incus-agent.service -f 
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="Matched trusted cert" fingerprint=0 subject="CN=root@mad-desktop.home.arpa,O=Linux Containers"
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="Matched trusted cert" fingerprint=0 subject="CN=root@mad-desktop.home.arpa,O=Linux Containers"
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="Event listener server handler started" id=962763f4-26f9-4577-b93e-2e5c94a1fd71 local="vm(4294967295):8443" remote="host(2):1942763303"
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="Matched trusted cert" fingerprint=0 subject="CN=root@mad-desktop.home.arpa,O=Linux Containers"
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="API Request\n\t{\n\t\t\"command\": [\n\t\t\t\"ls\",\n\t\t\t\"/\"\n\t\t],\n\t\t\"wait-for-websocket\": true,\n\t\t\"interactive\": true,\n\t\t\"environment\": {\n\t\t\t\"HOME\": \"/root\",\n\t\t\t\"LANG\": \"C.UTF-8\",\n\t\t\t\"PATH\": \"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n\t\t\t\"TERM\": \"xterm-256color\",\n\t\t\t\"USER\": \"root\"\n\t\t},\n\t\t\"width\": 184,\n\t\t\"height\": 39,\n\t\t\"record-output\": false,\n\t\t\"user\": 0,\n\t\t\"group\": 0,\n\t\t\"cwd\": \"\"\n\t}"
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="New operation" class=websocket description="Executing command" operation=9b4ab9ca-2742-4f23-a931-2e576e035242 project=
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="Started operation" class=websocket description="Executing command" operation=9b4ab9ca-2742-4f23-a931-2e576e035242 project=
Sep 26 14:28:10 fedora incus-agent[733]: time="2025-09-26T14:28:10Z" level=debug msg="Waiting for exec websockets to connect"

If I stop the agent in the VM I get a different error:

❯ incus exec fedora -- ls /
Error: VM agent isn't currently running

This leads me to a conclusion that incus can talk to the agent, but for some reason exec doesn’t work anymore.

Any new vms that I create are also affected by the same problem.

I would appreciate any help debugging and resolving this issue.

Trouble with debian/13 vm
2

Check that your incus-agent binary is a static binary.
We’ve seen issues like those when distros decide to ignore our instructions and build it dynamically instead.

3

It looks like it’s not a static binary:

❯ ssh fedora.incus.home.arpa sudo file /run/incus_agent/incus-agent
/run/incus_agent/incus-agent: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f8f7661e6fbcabac4cbac56e300f033a307f8828, for GNU/Linux 3.2.0, stripped

But that VM was created months ago, and it used to work back then. Is the agent injected from the host somehow instead of being installed into the VM upon the first boot?

4

The agent is loaded from the host on every VM boot.

5

dirty approach for now (especially true for Fedora distros :frowning: ):

```

set -e
ARCH=$(uname -m)

if [[ "$ARCH" == "x86_64" ]]; then
  ASSET="bin.linux.incus-agent.x86_64"
elif [[ "$ARCH" == "aarch64" ]]; then
  ASSET="bin.linux.incus-agent.aarch64"
else
  echo "Error: Unsupported architecture: $ARCH" >&2
  exit 1
fi

URL=$(curl -s "https://api.github.com/repos/lxc/incus/releases/latest" | \
      jq -r ".assets[] | select(.name == \"$ASSET\") | .browser_download_url")

if [ -z "$URL" ] || [ "$URL" == "null" ]; then
  echo "Error: Could not find download URL for $ASSET." >&2
  exit 1
fi

echo "Downloading $ASSET..."
curl -L -o /usr/bin/incus-agent "$URL"
chmod 500 /usr/bin/incus-agent
echo "Updated /usr/bin/incus-agent"
1 Like
6

This wasn’t intentionally ignored. I was aware of this but forgot to actually fix it properly.

This is done now, and updates have been proposed that need testing and karma to land:

2 Likes