Running VM(s) inside LXD containers

[k4my4b]

Hi!

I intend to run LXD inside a (unprivileged) LXD container for the purpose of running VM(s) and other stuff inside said container, so far however I haven’t managed to get it working though and I keep getting the same error:

Error: Failed instance creation: Failed creating instance record: Instance type “virtual-machine” is not supported on this server: KVM support is missing (no /dev/kvm)

Is this even doable at the moment? or do I have to spawn the VMs on the host instead?

[ruskofd (Florian Bobin (old account))]

This was announced here, but I don’t know if this feature is live for now

Weekly status #251

[Weekly Update Templates(1)] Weekly status for the week of the 30th May to the 5th June. Introduction Last week we continued our focus on stability that we’ve had over the last few weeks by continuing to work through our issues backlog (and any other issues we find along the way) expanding our test suite as we go. LXD Improvements: Reworked the VM qemu.conf file generation to have less repetition and be unit testable. Continued DB generator refactor to make loading the UsedBy information…

[tomp (Thomas Parrott)]

Yes that feature is in the latest releases.

[k4my4b]

@Weekly status #251

• Add support for running LXD VMs inside LXD containers. This requires the following config:

lxc launch images:ubuntu/22.04 u22 -c security.nesting=true
lxc config device add u22 kvm unix-char source=/dev/kvm
lxc config device add u22 vhost-net unix-char source=/dev/vhost-net
lxc config device add u22 vhost-vsock unix-char source=/dev/vhost-vsock

Warning: Passing those character devices may significantly reduce or in some cases entirely nullify the normal security benefit of LXD containers. It’s fine for testing purposes but those devices generally expose you to system wide objects with no built-in namespacing support.

Is there anyway to mitigate the security risks mentioned here? perhaps something in the works?

[tomp (Thomas Parrott)]

Not really. The devices being passed in are not container aware by the kernel and so they can be used to do things at the system level (like launch a VM).

[k4my4b]

hmm, thanks for clearing that up.
Are there any known/major security risks such privilege escalation associated with doing this?
Do you know if there is/will be work done (in the kernel) on this or is this dead in the water as there is little to no reason for it?

PS. I also had to pass in /dev/vsock, is that new?

[tomp (Thomas Parrott)]

I’m not sure if /dev/vsock requirement is new, when @stgraber wrote that post he presumably only needed to pass /dev/vhost-vsock.

So by default on my Ubuntu system /dev/kvm has these permissions:

ls /dev/kvm -l
crw-rw----+ 1 root kvm 10, 232 Aug 22 08:29 /dev/kvm

So its only accessible by either the root user or a user in the kvm group.

So that suggests that if it were accessible to a wider set of users there may be security issues.

By passing it into an unprivileged container, the device is copied and configured so it is accessible by the unprivilged root user (or more if you set the mode, uid or gid settings on that LXD device, see Instance configuration - LXD documentation).

So at the very least you are implicitly granting the permissions that the root user (or kvm group users) on the host had to /dev/kvm to your container’s unprivileged root user.

I’m not sure if there are known security issues with that approach, its more that you need to be aware that those devices are not container aware and so it is at least possible that a process inside the container could do something they wouldn’t normally be able to do.