Running virtual machines with LXD 4.0

Thanks for the info. But why this strange requirement (uefi+gpt)?

We only emulate modern virtio hardware and want a stable, safe and maintained platform on multiple architectures, so we only support Q35 hardware definition and only OVMF for firmware.

UEFI requires GPT so that’s how that part of the requirement comes about.

For official Ubuntu images, cloud-init must be used along with a config drive to seed a default user into the VM and allow console access.

This is not needed for Ubuntu 20.04?

lxc init ubuntu:20.04 ubuntu --vm
lxc config device add ubuntu config disk source=cloud-init:config

This seems to be working fine without all the cloud-configs. Is that true? I can access the VM directly with lxc exec <VM> bash

Yes indeed, it appears the lxd-agent support has made it into official Focal images now.

Hi all, I tried the Windows machine building steps on Ubuntu Server 20.04 and I hit rock bottom when launching the machine. Maybe it’s related to the snap LXD installation, but it seems to have serious issues with qemu sandboxing.

root@shadow:~# echo -n '-device virtio-vga -vnc :1 -drive file=/data/virt/iso/WindowsServer2019/17763.737.190906-2324.rs5_release_svc_refresh_SERVERESSENTIALS_OEM_x64FRE_en-us_1.iso,index=0,media=cdrom,if=ide -drive file=/data/virt/iso/virtio-win-0.1.173.iso,index=1,media=cdrom,if=ide' | lxc config set wintest raw.qemu -
root@shadow:~# lxc start wintest 
Error: Failed to run: /snap/lxd/current/bin/lxd forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/16100/bin/qemu-system-x86_64 -S -name wintest -uuid 0aec4ad7-45f4-4bdb-a0de-a3268471d55a -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/wintest/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/wintest/ -D /var/snap/lxd/common/lxd/logs/wintest/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/wintest -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd -device virtio-vga -vnc :1 -drive file=/data/virt/iso/WindowsServer2019/17763.737.190906-2324.rs5_release_svc_refresh_SERVERESSENTIALS_OEM_x64FRE_en-us_1.iso,index=0,media=cdrom,if=ide -drive file=/data/virt/iso/virtio-win-0.1.173.iso,index=1,media=cdrom,if=ide: : exit status 1
Try `lxc info --show-log wintest` for more info

And then

root@shadow:~# ls /data/virt/iso/virtio-win-0.1.173.iso 
root@shadow:~# lxc info --show-log wintest
Name: wintest
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/07/28 16:45 UTC
Status: Stopped
Type: virtual-machine
Profiles: default


qemu-system-x86_64: -drive file=/data/virt/iso/WindowsServer2019/17763.737.190906-2324.rs5_release_svc_refresh_SERVERESSENTIALS_OEM_x64FRE_en-us_1.iso,index=0,media=cdrom,if=ide: Could not open '/data/virt/iso/WindowsServer2019/17763.737.190906-2324.rs5_release_svc_refresh_SERVERESSENTIALS_OEM_x64FRE_en-us_1.iso': No such file or directory

The CD images exist and are accessible by any user in the system. Is there any way to disable the sandboxing feature of qemu, or is there any other issue preventing this to work?


Hi, I’ve done a quick test copying the two ISO files in the same folder as the virtual machine and setting the configuration parameters to

root@shadow:~# echo -n '-device virtio-vga -vnc -drive file=/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/wintest/17763.737.190906-2324.rs5_release_svc_refresh_SERVERESSENTIALS_OEM_x64FRE_en-us_1.iso,index=0,media=cdrom,if=ide -drive file=/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/wintest/virtio-win-0.1.173.iso,index=1,media=cdrom,if=ide' | lxc config set wintest raw.qemu -

and then it seems to be booting correctly, so there is an issue with chroot/sandboxing in UbuntuServer 20.04 with LXD on snap.

Yes snap uses its own mount namespace so referencing files that exist outside of that mount namespace is not going to work directly. There is a path inside the mount namespace that gets you back to the host’s file system. However may I ask why you are not using the built in LXD disk device type to add your ISO drives? E.g. lxc config device add v1 myiso disk source=/path/to/my/iso?

@stgraber or @monstermunchkin might have a tip on why the manual cdrom ide devices are needed and how to break out of the snap sandbox.

If you prefix the path with /var/lib/snapd/hostfs it should work fine.

The example used /home which happens to always be mapped into the snap, that’s why it works fine for those following the exact instructions.

Note that this only affects raw.qemu, paths handled by LXD itself do know how to translate as needed.

1 Like

Thanks a lot, @stgraber , prefixing the paths with /var/lib/snapd/hostfs worked, and I can also confirm it worked cross-filesystems (my /data folder is a FS mount from a separate drive).

I also managed to export the image after sysprep and launch another VM instance based on that image, but I had to set the -c security.secureboot=false configuration value for it to start (WinSRV2019).


Yeah, security.secureboot=false is going to be required at least until someone gets a WHQL certified version of the virtio-scsi driver…

type exit at this shell, then you can choose the Boot Manager.

I tried Win10 and it works - only there is no network card.

| NAME  |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
| win10 | RUNNING |      |      | VIRTUAL-MACHINE | 0         |
  limits.cpu: "8"
  limits.memory: 10GB
  raw.qemu: -device virtio-vga -vnc :1 -drive file=/home/mgaerber/Downloads/Win10_2004_English_x64.iso,index=0,media=cdrom,if=ide -drive file=/home/mgaerber/Downloads/virtio-win-0.1.173.iso,index=1,media=cdrom,if=ide
  security.secureboot: "false"
  volatile.eth0.host_name: tap02192319
  volatile.eth0.hwaddr: 00:16:3e:3a:7e:4a
  volatile.last_state.power: RUNNING
  volatile.vm.uuid: 1fda3fd0-6e71-4575-99ad-0229a4800f76
    path: /
    pool: default
    size: 60GB
    type: disk
ephemeral: false
- default
stateful: false
description: ""

Normally your default profile will have a network card attached to some kind of bridge.

Can you show lxc config show --expanded win10? This may then show it there.

On the Windows side, the network card will be using virtio-net but that’s supported by the drivers on the ISO. If you haven’t already, make sure that all drivers are installed in the VM, this will make a big difference in the whole experience.

Is it expected/intentional that the official ubuntu images don’t have a default netplan to setup dhcp like the linuxcontainers images? Thanks.

$ lxc launch ubuntu:20.04 ubvm --vm
$ lxc launch images:ubuntu/focal ubvmlx --vm
$ lxc list
|  NAME  |  STATE  |          IPV4          |                      IPV6                       |      TYPE       | SNAPSHOTS |
| devenv | RUNNING | (eth0)   | fd42:2077:2dea:ddf0:216:3eff:fe56:cb96 (eth0)   | CONTAINER       | 0         |
| guienv | STOPPED |                        |                                                 | CONTAINER       | 0         |
| ubvm   | RUNNING |                        |                                                 | VIRTUAL-MACHINE | 0         |
| ubvmlx | RUNNING | (enp5s0) | fd42:2077:2dea:ddf0:216:3eff:fef7:4ef0 (enp5s0) | VIRTUAL-MACHINE | 0         |
$ lxc exec ubvm -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:16:3e:c9:3d:2b brd ff:ff:ff:ff:ff:ff

Yes they should I believe, I am looking into this now.

If you add the cloud-init:config disk to your VM it should allow cloud-init inside the VM to get access to the templates required for network configuration:

lxc config device add ubvm config disk source=cloud-init:config

@tomp Thanks! That works. I think an unintended consequence of the agent now being integrated in to the official images is that it’s easier to get in to your VM without any cloud-init config drive, and then run in to this issue

I’ve been studying the source and I can’t seem find why is it that the images:ubuntu/focal/cloud can work without a cidata disk, but the ubuntu official images need the drive attached.

If I boot the official image with cidata disk attached I get cloud-init log in /var/log… however without the volume there is no cloud-init log at all, not even showing cloud-init probing and not finding any nocloud data source. It seems cloud-init in this case is driven by systemd and there is some mechanism in systemd decided whether to run cloud-init at all.

Booting images:ubuntu/focal/cloud always finds data in /var/lib/cloud/seed. This is empty in the root.img. It appears to be populated by LXD via the agent early in the first boot based on this code.

Given the agent is now installed in official images and the metadata for templating out the seed directory is almost the same between images:ubuntu/focal/cloud and ubuntu:20.04, why is that agent-based template mechanism not being applied while booting the official images too?

@stgraber explained this to me today.

The issue is that by the time cloud-init runs on the official Ubuntu cloud images LXD has already tried to boot the VM once (and the VM panicked due to the generic kernel which comes in an initrdless config in those images).

Subsequently, LXD then removes the first-boot cloud-init templates and by the 2nd boot (which is the first time cloud-init actually gets to run) the templates are no longer there and so a netplan configuration isn’t generated.

The LXD images:ubuntu/focal image doesn’t have this problem because it comes with the correct kernel and is also built with a basic netplan config so it doesn’t need cloud-init at all.

The fix for this is to move to the uefi images instead which use the linux-kvm kernel instead. This is something we are working with the Ubuntu team to resolve.

  • Finally publish your VM as an image with:
    lxc publish win10 --alias win10

Anyone tried this on lxd cluster? I made windows10-image
normally this windows10 image in the making boots like this

BdsDxe: loading Boot0009 “Windows Boot Manager” from HD(1,GPT,64400D7F-0526-4A7A-B40F-7EC28CECECED,0x800,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi
BdsDxe: starting Boot0009 “Windows Boot Manager” from HD(1,GPT,64400D7F-0526-4A7A-B40F-7EC28CECECED,0x800,0x32000)/\EFI\Microsoft\Boot\bootmgfw.efi

after making and launching new virtual machine from windows10-image on same lxd node it fails to boot like this

BdsDxe: loading Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)
BdsDxe: starting Boot0001 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x1)

Something uefi/boot drivers related, I have same problems when trying to convert existing virtualbox image to lxd-qemu compatible.

I tried launching on different lxd node and output is completely empty and no errors. It just times out? after like 20s.

root@universe-linux:/var/lib# lxc start w10-rbsql; lxc console w10-rbsql
To detach from the console, press: +a q

I didint run sysprep tho.