Running virtual machines with LXD 4.0

tomp · June 16, 2021, 8:02am

I’m not sure what you mean. LXD 2.0 didn’t support VMs, so the existing support in LXD 4.0 remains.

willrat · August 3, 2021, 7:19am

Is it possible to get qemu to start with the qxl vga device instead of the virtio-gpu

If I add
raw.qemu: -device qxl
to the config then 2 vga devices are present on the machine.

I have is a windows10 vm running, and are remoting to it via a windows workstation with
lxc console u1:vm-2 --type=vga

2 windows appear for each of the devices.

adding
raw.qemu: -vga qxl

causes it to not boot as there is a conflict with the (I asssume) default qemu.conf setting put out by lxd

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/vm-2/qemu.conf:79: PCI: slot 1 function 0 not available for pcie-root-port, in use by qxl-vga

Many Thanks

stgraber · August 3, 2021, 6:00pm

You can’t replace the default VGA adapter (virtio-vga), but you may indeed be able to add another one as QXL. Though you’ll most likely need to specify where on the PCIe bus it needs to sit to avoid getting into conflicts like you’re showing above.

Defining a new PCI bus dedicated for that device is the most likely way to avoid such issues.

Note that we’ve seen a fair bit of activity around a native virtio-gpu driver for Windows, so our hope is that in the near future this won’t be an issue anymore and we’ll get more than a plain VGA driver on Windows.

willrat · August 3, 2021, 7:41pm

Thanks for the reply

I found the virtio-gpu binary driver here:
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.204-1/virtio-win-0.1.204.iso

Windows installs the driver; the responsivness is similar to the default driver. In comparison a similar proxmox machine running the qxl driver with remote-viewer is quite usable.

Is the lagginess an issue with the virtio-gpu to spice or something with the display being routed through the lxc console command?

No complaints, just for information. I’ll use it with rdp.

stgraber · August 3, 2021, 10:41pm

RDP is always going to be the best experience, the QXL or virtio-gpu console is mostly really meant for installation. I need to test those new virtio-win drivers, last I played, you’d only get basic VGA, maybe that has now changed.

massimo · August 5, 2021, 4:12pm

Hi,
I recompiled lxd with qxl-vga instead virtio-vga.
Then I start the windows VM without console and connect the spice socket with:

spicy --uri=“spice+unix:///var/log/lxd/win10/qemu.spice”

I have only one display and everything seems working fine, clipping, mouse etc.

It worked for me also a test hex-eding the lxd binary

Cheers,

massimo

phlipped · September 7, 2021, 12:17am

Thanks for the tutorial!

I had two problems when following the instructions:

1) Secure Boot

I needed to set security.secureboot=false to get my VM to boot

When booting the VM, the console shows

BdsDxe: loading Boot0004 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0
,0x1)                                                                                                   BdsDxe: failed to load Boot0004 "UEFI QEMU QEMU HARDDISK " from PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/S
csi(0x0,0x1): Access Denied

2) Apt Repos for Arm64

The apt repos for arm64 are not available at the http://us.archive.ubuntu.com/ubuntu/ address in the instructions - I needed to find a mirror that hosted “ubuntu-ports”.

Details

Host System : RPi 4, 8Gb RAM, Ubuntu 21.04
Host LXD: installed via snap
Storage: ZFS with 1 pool on external USB block device, 512GB
VM: Ubuntu 18.04 (but tried other versions and they showed the same symptoms)

stgraber · September 7, 2021, 12:55am

Yeah, those two things are indeed specific to Arm64, though I’ve seen at least our own recent images starting to behave with SecureBoot on ARM recently.

TomvB · September 7, 2021, 3:11pm

How can I connect with public ip address (host) externally from Windows or Ubuntu 20.04 at home? And is it possible to specify a port? I can’t open virt-viewer on the terminal with lxc console win10 --type=vga remotely.

stgraber · September 7, 2021, 3:45pm

The way you’d normally do that is by configuring LXD to listen on the network:

lxc config set core.https_address :8443
lxc config set core.trust_password some-password

Then on your Windows or Ubuntu 20.04 system, install LXD and remote-viewer.
For Windows that should be: choco install lxc virt-viewer
For Ubuntu: snap install lxd && apt install virt-viewer

Then add your remote server to LXD with:

lxc remote add my-server IP-ADDRESS
Accept the certificate and enter the trust password
lxc remote switch my-server (to have all commands sent to the server rather than local)

At which point you can issue lxc console win10 --type=vga from your Windows or Ubuntu system and that will talk to the remote LXD using its API and get you access to the VGA console.

phlipped · September 10, 2021, 5:07am

What is the default limits.memory setting for VMs?

It seems to be 1GB (because that’s what my VM seems to have available to it, and I haven’t configured it explicitly)

But the documentation suggests that the default setting is “-”, which means “all”, right?. Instances | LXD.

“all” seems an unlikely setting for a VM, so I’m guessing the documentation is wrong, but just wanted to check.

stgraber · September 10, 2021, 4:08pm

Looks like we need to improve the doc a tiny bit to cover the defaults for virtual machines.

For containers, it’s correct that no limit means you get to see whatever the host has to offer. For VMs it’s not something we can do, so it defaults to 1 vCPU and 1GB of RAM.

stgraber · September 10, 2021, 4:11pm

fanghm · October 4, 2021, 8:37am

I just started a VM and it failed with this error message:
~$ Lxc start win2 --console
Error: Failed to run: forklimits limit=memlock :unlimited :unlimited … /snap/lxd
/ 21545/bin/qemu-system-x86_ 64 -S -name win2 -uuid 8ee5118C- 7e6e -4a18- 9e59 - 36b1e2275dd1 - daemonize -cpu host - nogr aphic -serial chardev:console -nodefaults - no user-config -sandbox on，obsolete=deny，elevateprivileges=allow，spawn=deny，resourcecontrol=deny - readconfig /var /snap/lxd/ common/lxd/logs /win2/qemu. conf - spiceunix=ON，disable- ticketing=on，addr= /var / snap/lxd/ common/lxd/logs /win2/qemu . spice -pidfile /var /snap/lxd/ common/lxd/ Logs /win2/qemu.pid -D /var /snap/lxd/ common/Lxd/Logs /win2/qemu. log - smbios type=2 ，manufacturer=Canonical Ltd. ，product=LXD runas lxd: char device redirected to /dev/pts/0 (label console )
: Process exited with non-zero value 1
Try ‘lxc info --show-log win2’ for more info

~$ lxc info --show-log win2
Name: win2
Location: none
Remote: unix://
Architecture: x86_ 64
Created: 2021/10/04 07:30 UTC
Status: Stopped
Type: virtual-machine
Profiles: default
Log:
qemu-system-x86_ 64: error: failed to set MSR 0x48f to 0x7ffff f00036dfb
qemu-system-x86_ 64: . ./target/i386 /kvm/kvm.c:2833: kvm buf_ set_ msrs: Assertion
ret == cpu->kvm_ msr_ buf->nmsrs’ failed.

stgraber · October 4, 2021, 1:12pm

Hmm, that smells like a potential hardware issue.
Can you show:

uname -a
dmesg | grep -i kvm

Also what kind of environment is this, is it a physical system, a virtual machine, … ?

fanghm · October 4, 2021, 1:35pm

$ uname -a
Linux fhm- ubuntu 5. 11.0-37-generic #41~20. 04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_ 64 x86_ 64 x86_ 64 GNU/Linux

$ dmesg | grep -i kvm
I typed the above command, but nothing output.

It is a virtual machine, and the environment is Ubuntu 20.04 LTS

stgraber · October 4, 2021, 2:33pm

What VM platform is used? I suspect it’s something like VirtualBox or VMWare and that nested VMs don’t work properly.

fanghm · October 4, 2021, 2:54pm

VMware.

tomp · October 4, 2021, 2:58pm

Suggests that perhaps our qemu capability detection isn’t as thorough as it could be:

github.com

lxc/lxd/blob/master/lxd/instance/drivers/driver_qemu.go#L5700-L5747

    
      
          func (d *qemu) Info() instance.Info {
          	data := instance.Info{
          		Name:  "qemu",
          		Type:  instancetype.VM,
          		Error: fmt.Errorf("Unknown error"),
          	}
          
          
	if !shared.PathExists("/dev/kvm") {
          		data.Error = fmt.Errorf("KVM support is missing")
          		return data
          	}
          
          
	err := util.LoadModule("vhost_vsock")
          	if err != nil {
          		data.Error = fmt.Errorf("vhost_vsock kernel module not loaded: %v", err)
          		return data
          	}
          
          
	hostArch, err := osarch.ArchitectureGetLocalID()
          	if err != nil {

This file has been truncated. show original

tomp · October 4, 2021, 2:59pm

Do you have a file /dev/kvm?