My initial, brief experiments with incus and cloud-init were unsuccessful and then I wanted to see how much I could automate when creating new containers on a single incus server with a single shell script without using any other automation tools. There were several steps involved in adding NVIDIA MIG GPUs to containers so I at least wanted to get that part automated, which I have, with some restrictions.
From the README:
"incus-mig.sh is a script to automate the creation of incus Ubuntu containers using NVIDIA GPUs, optionally attaching a NVIDIA MIG or a full NVIDIA PCI express attached GPU.
This script requires that you have incus installed and configured. You must have the CUDA toolkit installed on the incus server and you need to have configured at least one MIG device (using mig-manager) before running the script if you wish to use the -g option."
This script is pretty bespoke to my works GPU server and I don’t have any intention of trying to make it much more general purpose than it already is. Anyone wanting to use this script will need to modify the IP calculation code to suit their network, at least.
The only feature I currently plan to add is to replace the “Full CUDA install” code with code that fetches and installs the latest CUDA toolkit from NVIDIA’s site rather than installing the version from the Ubuntu repos. This feature is only needed for those wanting to run nvcc and build CUDA software within containers.
I’m not taking any feature requests as I’d prefer to keep it small and simple but I’ll probably accept sensible PR’s if they make sense and don’t risk breaking anything.
I’ve also added my nvitop GPU logging script (which has been submitted to the nvitop repo) and the cron command I use to log incus stats for those who think Prometheus is overkill.
Has anyone else written something like this already?
incus-mig no longer edits containers netplan config files, instead they are assigned via incus’ ipv4.address config setting. incus-mig now requires an incus managed network bridge and uses incus’ support for DHCP reservations to assign IP addresses to containers.
incus-mig can now provision Ubuntu 26.04, Ubuntu 24.04 and Debian 13 containers.
I updated incus-mig today by adding the -m option to specify a MIG GPU ID for a container to use (-m used to be used to define a containers RAM (memory) limit) and I added several example incus commands to a much improved README for my fun little set of incus MIG server management scripts.
incus-mig now has much better auto IP address assignment code, it now uses an IP that uses the first three octets of your br0 or incusbr0 bridge interface if either exists.
incus-mig is pretty much “finished” now as far as I’m concerned. I don’t plan on adding any more features to it, just fixing any bugs I might find.
It would be cool if someone added IPV6 support to it I suppose but I don’t use IPV6 so I won’t be adding that myself.
I thought I’d added all of the options I needed in incus-mig but that wasn’t the case as today I added the -x parameter to skip adding an expiry date to spare me manually removing expiry dates when not wanted.
I know I’ve said it a few times already but as far as I’m concerned these scripts are pretty much feature complete now
I’ll try to fix any bugs that people find that I can reproduce that are the fault of this script but this already does what I need apart from maybe automaing a full CUDA installation from NVIDIA’s repos instead of using the Ubuntu or Debian repo. No one has needed the CUDA dev tools yet here.
These scripts presume that all users already have a decent understanding of incus, enough to configure a managed bridge and optionally configure MIG because the docs don’t extend to getting these configured which are exercises for the user.
I should note that whilst incus-mig now supports creating VMs, you cannot attach a MIG GPU to a VM, only a container. This is a limitation inherited from incus that I have noted in the README.
It would be good if you could give incus-mig a quick test to see if it works for you on your MIG test box?
Clone it here:
Once you’ve downloaded it, run it with a command like:
bash incus-mig.sh -g migtest
This should create a Ubuntu container called migtest, assign it a suitable IPV4 address (based on the address of incusbr0) to it and attach a free MIG GPU to it, presuming there is at least one MIG GPU not in use.
It sets a few other things up too like snapshots etc but those are the key things it automates.
