Lxd-agent: No such file or directory

So I created an Ubuntu VM following Running virtual machines with LXD 4.0

sudo lxc init ubuntu:20.04 ubuntuvm --vm -c security.secureboot=false
(
cat << EOF
#cloud-config
apt_mirror: http://us.archive.ubuntu.com/ubuntu/
ssh_pwauth: yes
users:
  - name: user
    passwd: "\$6\$74P.JbeXGD1QDtaM\$hLGCPTMMC2ll6xRWxUR4I1ZaCrgFAopY57QG7Oz0XBODqEBppTbuviLWynI3yCXIguMHbhsmp9qKmREK0qrQn0"
    lock_passwd: false
    groups: lxd
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
EOF
) | sudo lxc config set ubuntuvm user.user-data -
sudo lxc config device add ubuntuvm config disk source=cloud-init:config
sudo lxc start ubuntuvm
sudo lxc console ubuntuvm

Now I’ve been having some trouble with lxd-agent not starting.

● lxd-agent.service - LXD - agent
     Loaded: loaded (/lib/systemd/system/lxd-agent.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2020-11-09 08:40:33 UTC; 28min ago
       Docs: https://linuxcontainers.org/lxd
    Process: 681 ExecStart=/run/lxd_config/9p/lxd-agent (code=exited, status=203/EXEC)
   Main PID: 681 (code=exited, status=203/EXEC)

lxd-agent.service: Scheduled restart job, restart counter is at 10.
Stopped LXD - agent.
lxd-agent.service: Start request repeated too quickly.
lxd-agent.service: Failed with result 'exit-code'.
Failed to start LXD - agent.

In the logs I see:

lxd-agent.service: Failed at step EXEC spawning /run/lxd_config/9p/lxd-agent: No such file or director
lxd-agent.service: Failed to execute command: No such file or directory

Looking at:

# ls -lah /run/lxd_config/9p/lxd-agent
-r-x------ 1 nobody root 14M Nov  9 08:37 /run/lxd_config/9p/lxd-agent

It does exist.

I did try:

# mount -t 9p config /mnt
9pnet_virtio: no channels available for device config
mount: /mnt: config already mounted on /run/lxd_config/9p

If I try to run it I get:

# ./run/lxd_config/9p/lxd-agent
bash: ./run/lxd_config/9p/lxd-agent: No such file or directory

Can you do:

  • file /run/lxd_config/9p/lxd-agent
  • ldd /run/lxd_config/9p/lxd-agent

Sure. I’m running LXD 4.7 on Alpine Linux 3.12:

This is from my ubuntu vm:

# file /run/lxd_config/9p/lxd-agent
/run/lxd_config/9p/lxd-agent: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=E6EmVpN0TZvNYTXWr4Jo/1hbSC-G7b9qjRqiI4udT/K2K342FkmGNq3PtXxl0R/e59AUSgMI1PzuL8Egmca, stripped

# ldd /run/lxd_config/9p/lxd-agent
	linux-vdso.so.1 (0x00007ffcd3754000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe2483ff000)
	libc.musl-x86_64.so.1 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe24820d000)
	/lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007fe249170000)

For anyone following this, the issue above is that lxd-agent is dynamically linked.
This should never be done as otherwise it would only work on VMs that use the same C library as the host system.

Instead lxd-agent should always be built statically so it can work in any VM.

I’ll work on testing/fixing my distributor’s package in a revision, so hopefully nobody else comes across this issue.