Running virtual machines with LXD 4.0

Thanks! Using macvlan.

nvm: works now… using root partition only.

I have a new test server available for LXD.
CPU: Intel Xeon W3520

egrep -c ‘(vmx|svm)’ /proc/cpuinfo

INFO: /dev/kvm exists
KVM acceleration can be used

Server OS: 20.04
2 disks in RAID1 with LVM
The VM starts after the launch (ipv6 only)

lxc launch ubuntu:20.04 test1 --vm
Then the VM stops (after a few seconds) and I can’t start the VM

Error: open /var/snap/lxd/common/lxd/logs/test1/qemu.log: no such file or directory

lxc info
driver: lxc | qemu
driver_version: 4.0.6 | 5.2.0
firewall: xtables
kernel: Linux
kernel_architecture: x86_64

Any idea @stgraber

Update: I cant delete or start the VM after this. Reboot required to delete it.

After a long wait… :wink:
user@node:~$ lxc delete test1
Error: Error deleting storage volume: Error removing LVM logical volume: Failed to run: lvremove -f /dev/LXD/virtual-machines_test1.block: Logical volume LXD/virtual-machines_test1.block in use.

lxc start test1
Error: virtiofsd failed to bind socket within 10s
Try lxc info --show-log test1 for more info

Anything scary looking in dmesg? Sounds like that system isn’t having a good day.

Nope, nothing. Reinstallation doesn’t help either

t=2021-03-12T20:47:26+0000 lvl=info msg="Started container" action=start created=2021-03-12T11:04:26+0000 ephemeral=false instance=test instanceType=container project=default stateful=false used=2021-03-12T11:20:09+0000

t=2021-03-12T20:53:31+0000 lvl=warn msg="Error getting disk usage" err="strconv.ParseFloat: parsing \"\": invalid syntax" instance=test1 instanceType=virtual-machine project=default

QEMU log:

KVM internal error. Suberror: 3
extra data[0]: 800000ec
extra data[1]: 31
extra data[2]: 81
extra data[3]: 30db0
RAX=0000000000000000 RBX=0000000000000001 RCX=0000000000000001 RDX=00000000000001ca
RSI=ffff9823b5d1cba8 RDI=0000000000000001 RBP=ffffa46d80077e90 RSP=ffffa46d80077e78
R8 =000000001327186b R9 =0000000000000000 R10=0000000000002000 R11=0000000000002000
R12=0000000000000001 R13=ffff9823401c8c80 R14=0000000000000000 R15=0000000000000000
RIP=ffffffffadf54c14 RFL=00000246 [—Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00c00000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =0000 0000000000000000 ffffffff 00c00000
FS =0000 0000000000000000 ffffffff 00c00000
GS =0000 ffff9823b5d00000 ffffffff 00c00000
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0040 fffffe0000036000 0000206f 00008b00 DPL=0 TSS64-busy
GDT= fffffe0000034000 0000007f
IDT= fffffe0000000000 00000fff
CR0=80050033 CR2=00000000ffffffff CR3=0000000029a0a000 CR4=000006a0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
Code=00 85 c0 7e 07 0f 00 2d b6 c1 4b 00 fb f4 8b 05 74 8f 78 00 <65> 44 8b 25 14 b5 0b 52 85 c0 0f 8f 85 00 00 00 5b 41 5c 41 5d 5d c3 65 8b 05 fe b4 0b 52

Tested with HWE (without success)
Tested with Proxmox (VM/LXC) instead of Ubuntu 20.04 with LXD and it works.

@stgraber I’ll use proxmox for now. I don’t know why it is going wrong on ubuntu 20.04.


distrobuilder is working like a magic.

I’ve updated the instructions to reference the work we did on distrobuilder and on LXD booting the resulting ISO, no more need for workarounds.

1 Like

Are there any pre-built/community images of win10 around which would spare me the effort of creating it as part of a small test?

I was hoping to see if I could use this with gitlab-runner and lxd executors…

No, unfortunately it’s not legal to re-distribute a Windows ISO image (or VM image) which is why we can only provide the tool that generates it for yourself and can’t just make the result available as we do for our other images.

1 Like

6 posts were split to a new topic: Unable to export/import Windows VM

3 posts were split to a new topic: VM not booting from Windows ISO

Is there or will be there a new tutorial for LXD version 4.0, as you have for LXD version 2.0?

I’m not sure what you mean. LXD 2.0 didn’t support VMs, so the existing support in LXD 4.0 remains.

Is it possible to get qemu to start with the qxl vga device instead of the virtio-gpu

If I add
raw.qemu: -device qxl
to the config then 2 vga devices are present on the machine.

I have is a windows10 vm running, and are remoting to it via a windows workstation with
lxc console u1:vm-2 --type=vga

2 windows appear for each of the devices.

raw.qemu: -vga qxl

causes it to not boot as there is a conflict with the (I asssume) default qemu.conf setting put out by lxd

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/vm-2/qemu.conf:79: PCI: slot 1 function 0 not available for pcie-root-port, in use by qxl-vga

Many Thanks

You can’t replace the default VGA adapter (virtio-vga), but you may indeed be able to add another one as QXL. Though you’ll most likely need to specify where on the PCIe bus it needs to sit to avoid getting into conflicts like you’re showing above.

Defining a new PCI bus dedicated for that device is the most likely way to avoid such issues.

Note that we’ve seen a fair bit of activity around a native virtio-gpu driver for Windows, so our hope is that in the near future this won’t be an issue anymore and we’ll get more than a plain VGA driver on Windows.

Thanks for the reply

I found the virtio-gpu binary driver here:

Windows installs the driver; the responsivness is similar to the default driver. In comparison a similar proxmox machine running the qxl driver with remote-viewer is quite usable.

Is the lagginess an issue with the virtio-gpu to spice or something with the display being routed through the lxc console command?

No complaints, just for information. I’ll use it with rdp.

RDP is always going to be the best experience, the QXL or virtio-gpu console is mostly really meant for installation. I need to test those new virtio-win drivers, last I played, you’d only get basic VGA, maybe that has now changed.

I recompiled lxd with qxl-vga instead virtio-vga.
Then I start the windows VM without console and connect the spice socket with:

spicy --uri=“spice+unix:///var/log/lxd/win10/qemu.spice”

I have only one display and everything seems working fine, clipping, mouse etc.

It worked for me also a test hex-eding the lxd binary :slightly_smiling_face:



Thanks for the tutorial!

I had two problems when following the instructions:

1) Secure Boot

I needed to set security.secureboot=false to get my VM to boot

When booting the VM, the console shows

BdsDxe: loading Boot0004 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0
,0x1)                                                                                                   BdsDxe: failed to load Boot0004 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/S
csi(0x0,0x1): Access Denied

2) Apt Repos for Arm64

The apt repos for arm64 are not available at the address in the instructions - I needed to find a mirror that hosted “ubuntu-ports”.


Host System : RPi 4, 8Gb RAM, Ubuntu 21.04
Host LXD: installed via snap
Storage: ZFS with 1 pool on external USB block device, 512GB
VM: Ubuntu 18.04 (but tried other versions and they showed the same symptoms)

Yeah, those two things are indeed specific to Arm64, though I’ve seen at least our own recent images starting to behave with SecureBoot on ARM recently.

How can I connect with public ip address (host) externally from Windows or Ubuntu 20.04 at home? And is it possible to specify a port? I can’t open virt-viewer on the terminal with lxc console win10 --type=vga remotely.