Can't get Incus container to start with NVIDIA MIG GPU attached

1

Could someone please help me get NVIDIA MIG GPUs working with incus? Is there a guide for this anywhere? I’d be happy to help write it if someone helps get me going. I have to say that I don’t think the incus docs for adding GPU devices are very good, mainly because no examples are given. Example commands would help a lot because the commands for adding devices to incus are long and not very intuitive.

I should also add that I have previously got NVIDIA MIG passthrough working on this same machine but using proxmox VE instead of incus and I’ve also had been able to get PCI passthrough of nvidia (non-MIG) GPUs working under incus so I’m not a total beginner here but there are still so many moving parts involved here, this really needs a guide writing. I’d prefer to use incus instead of proxmox-ve hence this post.

My host machine is running proxmox 9.1 but with proxmox VE removed and incus installed from the Debian 13 repos. The container I’m trying to get working with a MIG GPU is running Ubuntu 24.04.

On the host proxmox machine which has several A100s I have installed the latest CUDA libraries, the nvidia non-free driver, the cuda driver and the nvidia container toolkit. I have also install nvidia-mig-manager and I have several MIG devices showing as being configured under nvidia-smi.

I have run the commands

nvidia-smi -pm 1

and

nvidia-smi mig 1

On the host to make sure my mig GPUs are enabled. I am using the same mig-parted & mig-manager config as worked for me with proxmox-ve so that shouldn’t be the issue here.

I created my Ubuntu 24.04 container (which I had to disable apparmor on to get to start) using a command like this:

incus launch images:ubuntu/24.04 test -c nvidia.runtime=true

I configured its network interface, updated all the packages and then I installed the CUDA libraries but not the driver in the container. Do we need to install the cuda libraries into each container using CUDA or does -c nvidia.runtime=true make that unneccessary? I tried both and cannot get my container to start when it has a MIG GPU attached.

I checked I could start my container without the GPU attached first then I used:

nvidia-smi -L

To find out the UUID of one of my MIG GPUs and then I used a command like this to add it to my container:

incus config device add test gpu0 gpu gputype=mig mig.uuid=MIG-6d3ade01-c380-5acf-a5ac-4a83120f1c02

Device gpu0 added to test

incus start test

Error: Failed to start device “gpu0”: Card isn’t a NVIDIA GPU or driver isn’t properly setup
Try `incus info --show-log test` for more info

incus info --show-log test

Name: test
Description:
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2025/12/17 16:07 GMT
Last Used: 2025/12/18 17:47 GMT

Obviously there’s a lot more config details I could provide here but I wanted to try and keep this post as short as I could make it so I’ve only mentioned the key details. If any more details are needed then I’m happy to provide them.

Thanks

2

That’s the test we run daily (on a A100) to validate MIG: lxc-ci/bin/test-incus-gpu-mig at main · lxc/lxc-ci · GitHub

It’s obviously quite possible that NVIDIA changed some subtle details about how MIG works again, that’s happened a couple of times so far with various driver versions…

1 Like
3

Is there no need to specify the PCI address when using MIG? I’ll be trying that tomorrow.

Do I need to install the full CUDA Toolkit into each CUDA using container when I launch it with -c nvidia.runtime=true or not? Should the CUDA toolkit ideally be installed before you add the GPU to the container?

I should only need to install the CUDA driver package on the incus MIG server right?

Do I need to set any EVs or add anything to $PATH on either the server or in the containers to get CUDA working in MIG containers?

I just want my containers to start and nvidia-smi to recognize the MIG GPU its attached to.

4

The MIG UUIDs should be global, so there shouldn’t be a need to combine that with a PCI address. For the containers, using nvidia.runtime=true should pass in all the tools and libraries needed, avoiding having to install things in there.

The rest should generally just work as you can see in our test, basically, create the MIG instances, set the expected UUID in Incus, set nvidia.runtime=true and you should be getting a functional nvidia-smi in the container.

There are some other nvidia.XYZ config keys that can be used to configure the set of binaries and libraries being passed through.

1 Like
5

I’ve not tried deploying a Debian 13 or Ubuntu 22.04 container yet so that’ll be the first thing I try tomorrow

6

Have you searched the forum about how to configure nvidia?

Next to enable the runtime an other important setting is configure the right capabilities, see more at Incus Docket and Nvidia as one example or look at Install NVIDIA KVM driver on the host machine, how to use CUDA in MIG instance

Hope it helps

1 Like
7

Yes I’d searched and seen those topics but they seem to recommend opposing config options and it seems to be different for docker versus lxc containers, priv vs unpriv.

The project seems to lack one document or post it can reference that clearly states the key config settings required to get MIG and CUDA running under the two main container types and VM and I’ve also like to see more command examples in the docs.

I blame NVIDIA for this as they have more money than anyone right now. I bet everything I need to know about this could be summarised in less than 500 words or so, basically how to get nvidia-smi running with MIG in containers or in VMs for those wjho already know at least the basics of LXD/incus is something I’d like to see written.

8

Right, there is a lot of content out which give different directions. Most of it is related on which Incus version you are running. Assume you have latest stable release installed all what is required to have a working nvidia environment is to add two options to your instance:

nvidia.driver.capabilities: compute,graphics,utility,video 
nvidia.runtime: "true"

and add the device as described above.

There are additional options you can set as lined out here Nvidia options but they are no really a requirement.

For a VM if looks a bit different, you still need to passthrough the device but properly have to install the drivers as a VM doesn’t use the host runtime environment. It boots with its own kernel etc.

If you also want to install docker into a container you need to get Nvidia working in the container itself before configuring docker.

Just adding a device and add the two config lines above working very well for me to run Frigate or Immich in a container on latest Incus. Pretty much a standard setup like @stgraber explained.

1 Like
9

Thanks everyone for the suggestions, I got incus’ MIG containers working today so I thought I’d share my notes. They may not be ideal but they work for me.

It turns out that my problem was because I was not specifying the PCI address of the card I was using. Maybe this isn’t an issue on machines with only one GPU but all our compute nodes have at least two.

I’m sure there are several MIG cheat sheets but I referred to this one when I started using MIG:

https://www.seimaxim.com/kb/gpu/nvidia-a100-mig-cheat-sheat

Create a container with a MIG NVIDIA GPU attached

These commands will create an incus Ubuntu 24.04 container with one
Nvidia MIG GPU attached.

incus launch images:ubuntu/24.04 ubuntu24 -c nvidia.runtime=true

There should be no need to install the cuda toolkit or the cuda driver
in any incus containers when you use the incus -c nvidia.runtime=true
launch parameter.

Disable apparmor for the container or you probably won’t be able to start it:

incus config set ubuntu24 raw.lxc “lxc.apparmor.profile=unconfined”

You must specify both the mig.uuid and the PCI address when adding a
MIG GPU to an incus container. The mig.uuid for the GPU can be found
by running:

nvidia-smi -L

The pci address can be found in the Bus-Id column of the output of nvidia-smi.

To create an incus MIG GPU device and attach it to the ubuntu24 container:

incus config device add ubuntu24 gpu0 gpu gputype=mig
mig.uuid=MIG-6d3ade01-c380-5acf-a5ac-4a83120f1c02 pci=01:00.0