This post describes the minimal setup (for AMD/Intel CPUs) to run nested VMs with Incus.
There’s a bit of background on figuring out whether the firmware has been enabled for virtualization. Most mainstream Linux distros should have KVM working. This post was written in response to a question of a new member. I noticed that there was no document that talked about this.
3 Likes
Thank you for this great share!
Is there a way to disable nested virtualization for the Incus VM (outer VM)? From what I can see, by default, an Incus VM allow nested virtualization and there’s no config key-value pair to disable it.
Could this be possible by manually passing parameters to QEMU with incus config? Using raw.qemu?
Nested virtualization is controlled by KVM, which means that KVM should be configured to enable or disable it. There are different implementations of KVM depending on whether you have an Intel or AMD CPU.
To check whether nested virtualization is enabled or not, do the following. Replace kvm_amd with kvm_intel if you have an Intel CPU.
$ cat /sys/module/kvm_amd/parameters/nested
1
$
How do you disable them? Run the following. You see there’s a kernel module parameter, nested, that is of integer value (0: disabled, 1: enabled).
$ modinfo kvm_amd
...
parm: nested:int
...
$
Therefore, look into /etc/modprobe.d/ if there is any file related to KVM.
If not, create a file with a name like /etc/modprobe.d/kvm.conf
and add in there the following. This means that the kernel module is loaded with the parameter nested=0, disabling nested virtualization.
options kvm_amd nested=0
p.s.
I updated my blog post with this information.
Hmm I think I was wrong in what I asked.
I want to disable VM creation inside of an Incus VM
Host → Incus VM → a VM inside Incus VM
The “a VM Inside Incus VM” is what I want to disable. That means any Incus VM I create on the host should not be able to create VMs within it.
Simply put, I am looking for a “security.nesting=false” equivalent for a VM
security.nesting is special functionality for containers and I think that for VMs it should be a different setting, like security.vm.nesting.
However I am not sure whether the kernel module for KVM has a feature to disable nesting from within a VM. Because you are looking for an option that would disable nesting on VMs that were created on the host, so that no nested VMs can be created subsequently. Documentation, Running nested guests with KVM — The Linux Kernel documentation
Do you have a specific use-case that is not covered by the system-wide enable/disable for KVM nesting?
At our organization, we provide servers for customers (as a cloud hosting provider). So we want to know if disabling nested virtualization for KVMs is possible.
At our organization, we provide servers for customers (as a cloud hosting provider)
Do you provide containers or VMs to your customers? I’m going to assume that you provide kvm VMs.
If you disable nesting on the host, using the module parameters shown earlier, then kvm won’t be available in the guest VM (i.e. the customer’s VM).
This will apply to all kvm VMs you run on the host: none of them will have nested kvm. It doesn’t matter if they are started from incus, libvirt, or any other management tool. I don’t think it makes sense to have this as an incus-level setting, unless there’s some way to disable nested virtualization for individual VMs, which I’m unaware of.
The customers can still run qemu though, to do software emulation (which typically runs 10-20 times slower than kvm). There’s nothing you can do to stop that, since it’s just a userland application like any other piece of software.
The question though is, why do you want to disable nested kvm for them? Is it because you think there are security risks, or because you want to sell it as a separate value-add service?
We’re looking into this so that in the future, if we want to disable it, we know how. One possible reason for disabling it could be if some customers run Windows and engage in abusive activities.
They’ve done it in the past before we implemented some precautions to block these abusers from signing up on our platform. So knowing how to disable it will be beneficial for us, if we somehow have to deal with them again.
Is it a policy matter then?
- “You may not run Windows on our platform”? Or,
- “You may not run commercially licensed software on our platform”? Or,
- “You may not engage in abusive activities”?
I can’t see how running Windows by itself is necessarily abusive - I would tend to think of it more as masochism 
In any case, you can’t prevent all forms of “abuse”, but you can detect them as necessary, and kick off the abusers.
You should be aware that:
- Disabling kvm doesn’t prevent people running Windows (they can still run it under software emulation, which will burn a lot more of your CPU)
- Disabling kvm doesn’t prevent abuse in general
- Your customers may have legitimate uses for kvm (e.g. microkernels or unikernels for containerization)
I looked into the KVM and Qemu documentation.
I could not find a feature where you would disable nesting selectively, for specific VMs only.
It’s either enabled for all, or disabled for all.
If there is some obscure feature or trick to make a VM not being able to launch nested VMs (while nested virtualization is enabled), you would primarily look for that in KVM and Qemu. Those two projects are the upstream projects. If there is indeed such a feature, it would be easy for Incus to add support for that.
You may be able to do it with -cpu host,-svm or similar, so basically asking QEMU to hide the virtualization CPU extensions.