Container or Virtual Machine

LXD
1

Virtual machine support is a pretty cool new feature that was introduced with LXD 4.0, although it also leads to a simple question: why?

The question is purely one of curiosity, to learn about the advantages of having the feature and possible disadvantages to not having it.

One of the more immediate thoughts about an advantage is that it allows for the use of LXD features - lxc client, API, speed, snapshots, migrations, efficiency, network & storage configurations, etc. - within the context of a VM, without losing the benefits of LXD containers due to the additional overhead and wastefulness of resources that at least have traditionally been associated with VMs.

Now that we can launch either a system container or a VM with LXD, some questions come to mind:

Why would someone want to use LXD to launch a VM instead of a container?

Why would someone want to use LXD to launch a container instead of a VM?

Is a LXD VM the same as any other VM?

Is the main advantage to using LXD VMs to allow for LXD to serve as a replacement for hypervisors?

Will it be possible to create nested VMs or even alternating nesting between containers and VMs?

What are some of the current limitations to LXD’s VM support?

What are some considerations for possible future features to incorporate into LXD’s VM support?

Thanks for reading! Looking forward to seeing more LXD developments!

1 Like
2

The driver for this was really that we saw many people run both LXD and something like libvirt or virtualbox on multiple of their systems. There’s nothing necessarily wrong with that, but it’s a bit annoying having to deal with two different set of commands and have duplicate storage and network configurations.

LXD with VMs lets you unify all of that, you can even manage a cluster that mixes virtual machines and containers, apply the same profiles, devices, … to both types and have them use the same storage and network.

VMs are obvious candidate if running a different operating system as containers are unable to do this. They’re also capable of doing things like netboot which containers cannot. And they’re the only option if you absolutely need a different kernel or the ability to load kernel modules.

Then there are corporate security policies that may often not consider a container to be a sufficient security boundary for untrusted workloads. Having builtin VM support lets you meet such policies too.

LXD VMs are QEMU based, so yes, very similar to other VMs like you’d get through libvirt. We’re pretty opinionated about the VM setup though, that’s why you get a modern Q35 layout with UEFI and SecureBoot by default, all devices are virtio-based, we don’t do any complex device emulation at the host level.

It’s possible to run LXD containers inside LXD VMs. Running LXD VMs inside a LXD container may be possible if you pass /dev/kvm into the container (though untested and vhost may be an issue). Nested LXD VMs will be possible with very modern kernels. I believe you need a 5.7 or 5.8 kernel to get vsock nesting, with that in place, should your CPU support it, nested LXD VMs should be fine.

For limitations, we’re missing a bunch of devices like USB, proxy. Ability to live update the limits, ability to live migrate the VM and GUI access. We’re planning on fixing all of those but it will take some time.

Currently we’re focused on letting you access the graphical VM console, better handling of Windows VMs and have planned work around things like USB, vTPM, CPU/memory hotplug, …

5 Likes
3

That’s excellent information, thank you very much! :sunglasses: