Hi,
I cannot start VMs with LXD whereas containers work fine.
Command lxc launch ubuntu:22.04 vm-test --vm is stuck forever at Starting vm-test. After stopping no info is in the log:
lxc info --show-log local:vm-test
Name: vm-test
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
PID: 53895
Created: 2023/06/15 09:50 CEST
Resources:
Processes: 0
Log:
I tried to find the qemu command that LXD is running and tried it directly and it failed with the following error message:
/snap/lxd/24918/bin/qemu-system-x86_64 -S -name vm-test -uuid deca09a3-5ebe-4fd5-b136-e8b3de7b9c28 -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/vm-test/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/vm-test/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/vm-test/qemu.pid -D /var/snap/lxd/common/lxd/logs/vm-test/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd
/snap/lxd/24918/bin/qemu-system-x86_64: error while loading shared libraries: liburing.so.2: cannot open shared object file: No such file or directory
I had some VMs running in April on the same machine and apart from the LXD update I cannot remember any changes.
What could be the issue and how can I resolve it?
tomp
June 15, 2023, 8:11am
2
Please can you stop all VMs, and then refresh latest/edge to try and see if this is an issue that will be resolved in LXD 5.15 next week:
sudo snap refresh lxd --channel=latest/edge
I think the io_uring error you’re quoting is a misnomer because running the literal command without being inside the snap’s mount namespace will result in shared libraries being not found.
stgraber
June 15, 2023, 11:26pm
3
Can you post your dmesg output?
A hang during startup could be cause by a kernel issue.
Thanks for the fast replies.
I stopped everything and used sudo snap refresh lxd --channel=latest/edge, however the results stay the same.
Here is the dmesg output:
[18659467.297685] audit: type=1400 audit(1686898667.863:15785): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxd-vm-test_</var/snap/lxd/common/lxd>" pid=150940 comm="apparmor_parser"
[18659467.319927] audit: type=1400 audit(1686898667.883:15786): apparmor="DENIED" operation="open" profile="lxd-vm-test_</var/snap/lxd/common/lxd>" name="/var/lib/snapd/hostfs/etc/resolv.conf" pid=150942 comm="lxd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
tomp
June 20, 2023, 10:00am
5
Can you restart, delete the VM if it exists, and then try separating the creation from the start using:
lxc init .... <instance>
lxc start <instance>
Also, when you do the lxc start command, before that in a separate window can you start a lxc monitor session:
lxc monitor --type=logging --pretty
```
Sure. That’s the output of the monitor command:
DEBUG [2023-06-21T21:33:31+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG [2023-06-21T21:33:31+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/instances/vm-ubuntu username=root
DEBUG [2023-06-21T21:33:31+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=root
DEBUG [2023-06-21T21:33:31+02:00] Event listener server handler started id=b3b9e9b0-713d-4c7b-bf7a-ad2d604952cd local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG [2023-06-21T21:33:31+02:00] Handling API request ip=@ method=PUT protocol=unix url=/1.0/instances/vm-ubuntu/state username=root
DEBUG [2023-06-21T21:33:31+02:00] Started operation class=task description="Starting instance" operation=9fe77695-1d26-4ae9-8742-0f3cc89b02df project=default
DEBUG [2023-06-21T21:33:31+02:00] New operation class=task description="Starting instance" operation=9fe77695-1d26-4ae9-8742-0f3cc89b02df project=default
DEBUG [2023-06-21T21:33:31+02:00] Start started instance=vm-ubuntu instanceType=virtual-machine project=default stateful=false
DEBUG [2023-06-21T21:33:31+02:00] Instance operation lock created action=start instance=vm-ubuntu project=default reusable=false
DEBUG [2023-06-21T21:33:31+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/9fe77695-1d26-4ae9-8742-0f3cc89b02df username=root
DEBUG [2023-06-21T21:33:31+02:00] MountInstance started driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] MountInstance finished driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] Installing lxd-agent installPath=/var/snap/lxd/common/lxd/virtual-machines/vm-ubuntu/config/lxd-agent instance=vm-ubuntu instanceType=virtual-machine project=default srcPath=/snap/lxd/25027/bin/lxd-agent
DEBUG [2023-06-21T21:33:31+02:00] MountInstance started driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] Generating NVRAM instance=vm-ubuntu instanceType=virtual-machine project=default
DEBUG [2023-06-21T21:33:31+02:00] MountInstance finished driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] UnmountInstance started driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] Skipping unmount as in use driver=dir pool=default refCount=1 volName=vm-ubuntu
DEBUG [2023-06-21T21:33:31+02:00] UnmountInstance finished driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] Starting device device=eth0 instance=vm-ubuntu instanceType=virtual-machine project=default type=nic
DEBUG [2023-06-21T21:33:31+02:00] Scheduler: network: tapbe3eb81d has been added: updating network priorities
DEBUG [2023-06-21T21:33:31+02:00] Starting device device=root instance=vm-ubuntu instanceType=virtual-machine project=default type=disk
DEBUG [2023-06-21T21:33:31+02:00] UpdateInstanceBackupFile started driver=dir instance=vm-ubuntu pool=default project=default
DEBUG [2023-06-21T21:33:31+02:00] UpdateInstanceBackupFile finished driver=dir instance=vm-ubuntu pool=default project=default
Stopping the start after some minutes:
DEBUG [2023-06-21T21:45:12+02:00] Handling API request ip=@ method=DELETE protocol=unix url=/1.0/operations/9fe77695-1d26-4ae9-8742-0f3cc89b02df username=root
DEBUG [2023-06-21T21:45:12+02:00] Handling API request ip=@ method=DELETE protocol=unix url=/1.0/operations/9fe77695-1d26-4ae9-8742-0f3cc89b02df username=root
DEBUG [2023-06-21T21:45:12+02:00] Event listener server handler stopped listener=b3b9e9b0-713d-4c7b-bf7a-ad2d604952cd local=/var/snap/lxd/common/lxd/unix.socket remote=@
tomp
June 22, 2023, 9:37pm
7
What host os and version is this running on?
tomp
June 22, 2023, 9:37pm
8
When it gets stuck please can you get the output of “ps auxf” thanks
Relevant output of ps auxf:
root 86136 0.0 0.0 3018276 71504 ? Ssl Jun20 0:56 /usr/lib/snapd/snapd
root 188799 0.0 0.0 2884 2012 ? Ss Jun21 0:00 /bin/sh /snap/lxd/25027/commands/daemon.start
root 189460 0.2 0.0 11050304 193492 ? Sl Jun21 8:19 \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
lxd 189651 0.0 0.0 10460 4252 ? Ss Jun21 0:02 \_ dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --inter
root 107921 0.0 0.0 147012 38436 ? Ssl Jun21 0:00 \_ /snap/lxd/25027/bin/qemu-system-x86_64 -S -name vm-ubuntu -uuid af20ccdc-8d91-461a-9d68-565c81f729d6 -daemonize -
root 107937 0.0 0.0 0 0 ? Zs Jun21 0:00 | \_ [qemu-system-x86] <defunct>
root 178789 0.0 0.0 147012 38072 ? Ssl 09:15 0:00 \_ /snap/lxd/25027/bin/qemu-system-x86_64 -S -name vm-test -uuid deca09a3-5ebe-4fd5-b136-e8b3de7b9c28 -daemonize -cp
root 178807 0.0 0.0 0 0 ? Zs 09:15 0:00 \_ [qemu-system-x86] <defunct>
root 104854 0.0 0.0 10168 5108 ? Ss Jun21 0:12 tmux
root 104855 0.0 0.0 8568 5344 pts/17 Ss+ Jun21 0:00 \_ -bash
root 105497 0.0 0.0 8568 5412 pts/25 Ss+ Jun21 0:00 \_ -bash
root 107939 0.0 0.0 1468912 43684 ? Sl Jun21 0:00 /snap/lxd/25027/bin/qemu-system-x86_64 -S -name vm-ubuntu -uuid af20ccdc-8d91-461a-9d68-565c81f729d6 -daemonize -cpu host
systemd+ 85125 0.0 0.0 93096 6576 ? Ssl 07:51 0:00 /lib/systemd/systemd-timesyncd
crawler 178631 0.0 0.0 10144 4916 ? Ss 09:14 0:00 tmux
crawler 178632 0.0 0.0 7960 4828 pts/28 Ss 09:14 0:00 \_ -bash
crawler 178729 0.1 0.0 1466836 20544 pts/28 Sl+ 09:15 0:00 | \_ /snap/lxd/25027/bin/lxc start vm-test
crawler 178935 0.0 0.0 7960 4792 pts/29 Ss 09:15 0:00 \_ -bash
crawler 180086 0.0 0.0 13068 5848 pts/29 R+ 09:16 0:00 \_ ps auxf
root 178809 0.0 0.0 1466864 42176 ? Sl 09:15 0:00 /snap/lxd/25027/bin/qemu-system-x86_64 -S -name vm-test -uuid deca09a3-5ebe-4fd5-b136-e8b3de7b9c28 -daemonize -cpu host
The host runs Debian 10.13
tomp
June 23, 2023, 7:35am
10
What does lxc info <instance> --show-log show?
tomp:
xc info --show-log
Name: vm-ubuntu
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
PID: 107939
Created: 2023/06/21 21:32 CEST
Resources:
Processes: 0
Log:
tomp
June 23, 2023, 7:50am
12
OK, im not sure about this, but it looks like qemu is crashing.
Suggest deleting the instances, doing a full reboot to get a fresh state, then try creating a new instance (using lxc init instead of lxc launch).
Then use lxc start <instance> --console and see if there is any useful output.
JannisBush
August 23, 2023, 2:16pm
13
After upgrading the host, VMs are working again.
1 Like
Hi there I am having this exact same problem
Here are the logs from starting the VM, after that last log it still hangs on the start command
I just checked did some of the steps above @JannisBush mentioned when trying to figure out what the problem is, im having the exact same problems
@tomp hey, is there any previous reports a side from this one of situations like this?
tomp
August 30, 2023, 11:00am
18
Not that I’ve heard, please can you create a thread over at Support - Ubuntu Community Hub with as much info as to the problem and your setup.
Thanks
