I’ve been trying out the Terraform provider with my IncusOS instance running on my homelab server and wanted to share a bit of feedback, this seemed like the most appropriate place.
I’m coming from a cloud background so my opinions are probably biased in that direction.
I will add future feedback in this thread also.
incus_instance
The incus_instance resource which can be either a VM or a container is a bit confusing to be honest, and I think these would be best off as separate resources.
I can see why having a single resource for both would be seductive, especially since this is how it’s presented in the IncusOS UI/API. But, inevitably, you end up with fields that are only used for one or the other. There are concepts that don’t map cleanly from VMs to containers and vice versa.
At a glance, it’s not easy to tell in the Terraform code whether an incus_instance resource is referring to a VM or container.
Instance devices
The incus_instance block takes a device block which can be used to attach devices. This is OK but if you don’t know statically what devices you want to attach, you’re forced to use dynamic blocks, which are a bit hacky IMO. Maybe a separate incus_device resource that attaches to a incus_instancecould be nice, or it could be done like how some cloud providers do it, with an _attachment resource.
Disk attachment
On a similar note to above, I’ve been struggling to create an instance with an ISO and empty disk attached. I have two device blocks, one of which references a storage volume I’ve created. But when I try to apply this code, I get an error that tells me Root disk entry may not have a “source” property set
╷
│ Error: Failed to create instance "talos-master"
│
│ with incus_instance.talos_master,
│ on main.tf line 25, in resource "incus_instance" "talos_master":
│ 25: resource "incus_instance" "talos_master" {
│
│ Failed creating instance record: Failed initializing instance: Invalid devices: Device validation failed for "root": Root disk entry may not have a "source" property set
╵
{
...
device {
name = "root"
type = "disk"
properties = {
path = "/"
pool = "local"
source = incus_storage_volume.talos_master_root.name
}
}
}
But if I remove source, I get this:
╷
│ Error: Failed to create instance "talos-master"
│
│ with incus_instance.talos_master,
│ on main.tf line 25, in resource "incus_instance" "talos_master":
│ 25: resource "incus_instance" "talos_master" {
│
│ Failed creating instance record: Failed initializing instance: Failed add validation for device "talos-iso": Failed loading custom volume: Storage volume not found
╵
path doesn’t make much sense in this context either because it’s a VM managed by me and not Incus, so it shouldn’t be managing mounts. If I remove bring back source and remove path, then I get a new error:
╷
│ Error: Failed to create instance "talos-master"
│
│ with incus_instance.talos_master,
│ on main.tf line 25, in resource "incus_instance" "talos_master":
│ 25: resource "incus_instance" "talos_master" {
│
│ Failed creating instance record: Failed initializing instance: Invalid devices: Failed detecting root disk device: No root device could be found
╵
I’m sure there’s a solution for this but I want to explain my thought process as a user and why it can be confusing.
ISO management
I haven’t been able to find a way to upload an ISO to the Incus remote through Terraform. My instinct was to look for something like an incus_iso resource but there’s no such thing.
I see there’s an incus_image resource that can load from a local source_file, but it doesn’t say what file types it can load as an image source.
Confusing error from incorrect image
While I was trying to get an instance to boot from an ISO, I set the image field to something like this:
resource "incus_instance" "talos_worker" {
name = "talos-worker"
type = "virtual-machine"
image = "iso:talos.iso"
}
I put the iso: prefix in the image name because I had been playing with containers beforehand, and found that you have to put docker: as the prefix when wanting to run Docker images. So I thought maybe putting iso: would boot from an ISO, but instead I got this error:
╷
│ Error: Failed to retrieve Incus ImageServer
│
│ with incus_instance.talos_worker,
│ on main.tf line 60, in resource "incus_instance" "talos_worker":
│ 60: resource "incus_instance" "talos_worker" {
│
│ Incus socket with write permissions not found. Searched Incus socket paths: [/var/lib/incus/unix.socket /var/lib/incus/unix.socket.user /run/incus/unix.socket /run/incus/unix.socket.user]
╵
It’s probably not correct to try and boot from an ISO like this, but this error confused me. I believe I got this error at some other point as well and struggled to figure out why.
KV configuration
The configuration of instances and devices through KV pairs probably has a justification, but it’s a bit painful to work with from Terraform. Using strings for config means losing out on LSP warnings if there are typos, and with Terraform alone you cannot validate these strings without applying the code.
There’s not any documentation in the provider of the possible keys. I was able to find a few in the example code, but other than that I found that I had to dig through repo issues and other discussions.
Making the instance CPU/memory configurable through native Terraform fields at least would be nice.