Create a cluster bridge network with terraform

1

Ok, so I have a test cluster: 3 nodes, one drive (btrfs), one NIC each. No ceph or ovn, just Incus.

I’m trying to use TerraformOpenTofu to create projects and resources within it, and I’m struggling to create a bridge network. (My projects so far have features.networks set to false.)

terraform {
  required_providers {
    incus = {
      source  = "lxc/incus"
      version = "~> 1"
    }
  }
}

data "incus_cluster" "cluster" {
}

# Generate a random /24 subnet
module "netaddr4" {
  source  = "./randnet"
  subnet  = "172.16.0.0/12"
  netbits = 12
}

locals {
  # FIXME: Pull dynamically
  node_interfaces = {
    # Don't worry about it
  }
  defaultnet = {
    "ipv4.address" = module.netaddr4.incus_addr # eg 172.27.96.1/24
    "ipv4.nat"     = "true"
  }
}

variable "project" {
  description = "ID of current project"
  type        = string
}

variable "name" {
  description = "Name of network"
  type        = string
}

variable "config" {
  description = "Network configuration"
  type        = map(string)
  default     = {}
}


resource "incus_network" "net_node" {
  for_each = data.incus_cluster.cluster.members
  project  = var.project
  name     = "${var.project}_${var.name}"
  target   = each.key
  config = merge(var.config, {
    parent = local.node_interfaces[each.key]
  })
  type = "bridge"
  # FIXME: Destroy if net fails to create
}

resource "incus_network" "net" {
  project = var.project
  depends_on = [
    incus_network.net_node,
  ]

  name   = "${var.project}_${var.name}"
  type   = "bridge"
  config = merge(local.defaultnet, var.config)
}

output "id" {
  description = "Identifier to use elsewhere"
  value       = var.name
  depends_on  = [incus_network.net]
}

I think this does the right thing, where it’ll create the network on each node, and then create the cluster-wide network.

But, two things:

  • If I specify the network config in the node-specific network (not in the example), I get Config key "ipv4.address" may not be used as member-specific key, which is different from the cluster example
  • If I run the example as-is, I get Invalid option for network "forge2_default" option "parent" on the cluster-wide network
2

Actually, I’m starting to think that OpenTofu Registry is just wrong.

Comparing it to How to configure networks for a cluster - Incus documentation and How to create a network - Incus documentation , the suggested values are just red herring levels of wrong.

3

@maveonair though @breml may have some pointers on this one too as we’re doing something pretty similar in Operations Center I believe.

4

I think the missing piece is, that the config really needs to be defined once for each node with only config options, that are allowed per node and then once again with only config options allowed on cluster level. So I guess, you need var.config_per_node and var.config_cluster (or something similar) instead of just var.config.

In operations-center/internal/provisioning/adapter/terraform/testdata/resources_networks.tf at main · FuturFusion/operations-center · GitHub you can see a file, which is the network configuration “golden file” for the tests.

5

Indeed, the documentation is wrong… The correct way to configure is like this:

resource "incus_network" "my_network_node1" {
  name   = "my_network"
  target = "node-1"
}

resource "incus_network" "my_network_node2" {
  name   = "my_network"
  target = "node-2"
}

resource "incus_network" "my_network_node3" {
  name   = "my_network"
  target = "node-3"
}


resource "incus_network" "my_network" {
  depends_on = [
    incus_network.my_network_node1,
    incus_network.my_network_node2,
    incus_network.my_network_node3,
  ]

  name = "my_network"
                                  
  config = {
    "ipv4.address" = "10.150.19.1/24"
    "ipv4.nat"     = "true"
    "ipv6.address" = "fd42:474b:622d:259d::1/64"
    "ipv6.nat"     = "true"
  }
}

I will update the documentation.

6

Ok, removing the parent config on the individual node networks allowed it to be created.