Making files available on launch with opentofu on incusos remote

Incus
, ,
1

The initial command of many containers I try and run expect to have files available before execution.
I am running IncusOS on bare metal using the terraform provider + opentofu. I think the issue is, the file is on my local laptop, but not present on the remote filesystem?

This blocks me from adopting incus where I’d really like to be using it. Is there a workaround or some relevant things I should read up on and understand?

# traefik.tofu

terraform {
required_providers {
incus = {
source  = "lxc/incus"
version = "~> 1.2"
}
}
}

provider "incus" {
generate_client_certificates = true
accept_remote_certificate    = true
default_remote               = "incusos"

remote {
name    = "incusos"
address = "https://incusos.home.arpa:8443"
}

remote {
name     = "docker"
address  = "https://docker.io"
protocol = "oci"
public   = true
}
}

resource "incus_instance" "traefik" {
name  = "traefik"
image = "docker:library/traefik:v3.7"

device {
name = "config"
type = "disk"
properties = {
source = abspath("${path.root}/traefik.yaml")
path   = "/etc/traefik/traefik.yaml"
}
}
}

▲ tofu apply -auto-approve     

OpenTofu used the selected providers to generate the following execution plan. Resource actions
are indicated with the following symbols:

create

OpenTofu will perform the following actions:

incus_instance.traefik will be created

resource "incus_instance" "traefik" {

architecture = (known after apply)
config       = {}
ephemeral    = false
image        = "docker:library/traefik:v3.7"
interfaces   = (known after apply)
ipv4_address = (known after apply)
ipv6_address = (known after apply)
mac_address  = (known after apply)
name         = "traefik"
profiles     = [
"default",
]

running      = true
status       = (known after apply)
target       = (known after apply)
type         = "container"
device {
name       = "config"
properties = {
"path"   = "/etc/traefik/traefik.yaml"
"source" = "/home/micro/src/incus-launch/traefik.yaml"
}
type       = "disk"
}
}

Plan: 1 to add, 0 to change, 0 to destroy.
incus_instance.traefik: Creating...
╷
│ Error: Failed to create instance "traefik"
│
│   with incus_instance.traefik,
│   on traefik.tofu line 28, in resource "incus_instance" "traefik":
│   28: resource "incus_instance" "traefik" {
│
│ Failed instance creation: Failed creating instance record: Failed initializing instance: Failed
│ add validation for device "config": Missing source path
│ "/home/jedahan/src/incus-launch/traefik.yaml" for disk "config"
╵
2

Instead of creating a disk you could just push the file. Here is an example block I have used with the incus_instance resource.

  file {
    source_path = "./configuration.nix"
    target_path = "/etc/nixos/configuration.nix"
    mode        = "0644"
  }
3

My understanding is that file blocks get pushed after the instance is already running.

4

Here is the documentation you are looking for.

Your code is trying to pass a path on the host to the disk and the path does not exist.

You could create the volume first and pass the file into it. Then reference it in your instance configuration.

Here is an example of creating the volume with a file in it.

It is a bit more work than using the file block. Any reason the file block will not work for your use case? If you need to restart the instance after a configuration chance you could add an exec block to the instance.

  exec = {
    "10-reboot" = {
      command = ["shutdown", "-r", "0"]
      trigger = "once"
    }
  }
5

The file block in incus_instance mounts after boot. Testing the file block in a volume, and attaching the volume once I figure out permissions issues with the default pool and incus/incus-admin groups. Thanks for the patience and pointers.

6

It seems that mounting a volume with a single file actually creates a directory - what if I only want to mount a single file without overwriting the directory contents? For example, if /etc/somed has /etc/somed/default.conf and I want to add /etc/somed/cool.conf before launch?

First I looked at initial.copy but that only works if the volume is empty, and would snapshot what exists in that path already.

Then I saw that path= on disk devices should work, except there is a line “With VMs, this option supports mounting file system disk devices and paths within them. Mounting single files is not supported.” Is there a conceptual reason mounting single files are not supported, or is it ‘just’ work that hasn’t been asked for / is lower on priorities?

7

Restating the problem:

Incus remote runs IncusOS
Infrastructure defined in terraform
Instance requires config file to be present before command is run
Config file required to coexist with image provided files in the same directory

What is the most minimal way to achieve this?

8

What command are you referring to?

It would be great if you could provide a minimal reproducible example that demonstrates your situation. I have not run into a situation like this where I could not add a configuration file after initial boot. If the configs need to be there early during initial boot you could consider using cloud-init. At least on Debian and Ubuntu this is a common solution.

Without a minimal example, I am thinking you might need to build your own images. I used to do this when my infrastructure was Debian based. Then I would use cloud-init for small customizations.

I used distrobuilder to build the images but you could also do it manually with Incus commands.

I hope this helps. Maybe someone else has a better idea. Some of the details are different depending on what distribution your image is based on.

9 
# traefik.yml
api:
  insecure: true
entryPoints:
  web:
    address: ":80"

-- traefik.tofu
terraform {
  required_providers {
    incus = {
      source  = "lxc/incus"
      version = "~> 1.2"
    }
  }
}

provider "incus" {
  remote {
    name    = "local"
    address = "unix://"
  }

  remote {
    name     = "docker"
    address  = "https://docker.io"
    protocol = "oci"
    public   = true
  }
}

resource "incus_storage_pool" "default" {
  name    = "default"
  driver  = "dir"
}

import {
  id = "default"
  to = incus_storage_pool.default
}

resource "incus_storage_volume" "traefik-config" {
  name = "traefik-config"
  pool = incus_storage_pool.default.name

  file {
    source_path        = abspath("./traefik.yaml")
    target_path        = "traefik.yaml"
    uid                = 1000
    gid                = 1000
    mode               = "0644"
    create_directories = true
  }
}

resource "incus_instance" "traefik" {
  name  = "traefik"
  image = "docker:library/traefik:v3.7"

  device {
    name = "config"
    type = "disk"
    properties = {
      path   = "/etc/traefik/traefik.yaml"
      source = "${incus_storage_volume.traefik-config.name}/traefik.yaml"
      pool   = incus_storage_pool.default.name
    }
  }
}

This is what worked locally for me, testing with the incusos remote soon.

I think the difficult thing to think about, was putting a file on a instance has a few steps:

  1. creating a volume
  2. pushing a file to that volume
  3. mounting the volume/filepath

In hindsight, this looks fine, not sure I could point to any one thing that made it hard to find this solution.

Some things were trying out file blocks in the instance, not finding an example of mounting a single file from a volume in the docs, first time using terraform, and some wishful thinking that a file block alone in the resource would work.

Next papercut to solve is figuring out how to tell tofu to push the file when it changes…

Appreciate the responses, and would love for more folks to share minimal snippets/setups.

10

I think it is important to point out that OCI containers are a special case compared to regular Incus instances or VMs.

The Terraform provider is primarily responsible for creating and managing the instance itself, not for configuring the operating system or application inside it.

With OCI container images, this becomes a bit more complicated when files need to be available at instance launch. In this case, the intended approach is to create a storage volume containing the required files first, and then attach that volume to the instance.

11

My hope is to build out a terraform provider that works at a bit of a higher level, or some modules that serve as a template for communities to manage systems or applications. Either way, I think this reply has really nice information. Re-read about systems containers vs application containers, and I’ll look into what an application-container / incus native deploy of ‘cozy community infra’ might look like.

12

I am sorry for bringing this up again, but I don’t understand what is wrong with using something like this?

resource "incus_instance" "traefik" {
  name  = "traefik"
  image = "docker:library/traefik:v3.7"

  file {
    source_path = "./traefik.yaml"
    target_path = "/etc/traefik/traefik.yaml"
    uid         = 1000
    gid         = 1000
    mode        = "0644"
    create_directories = true
  }
}

It deploys and runs for me.

What error or mistake am I missing here?

13

There is nothing wrong with this approach.

What is going to happen is that first the image is downloaded, the instance is created and started and then the file is uploaded to the image. So if your OCI container requires the file to be there in order to be started then this approach will not work.

Kind of a chicken-egg problem with OCI containers…

14

I’ve managed to make a module called ‘coop’ that is some sorta porcelain that lets me define instances a bit more like compose would…

module "coop" {
  source = "./coop"

  name  = "traefik"
  image = "docker:library/traefik:v3.7"

  proxies = [
    {
      from = 8777
      to   = 80
    },
  ]

  files = [
    {
      source_path = abspath("./traefik.yaml")
      target_path = "traefik.yaml"
    },
  ]
}

Coop processes the files and creates the volume, pushes the file to the volume, and attaches to the instance.

Its my first time using tofu, so the module is awful, but I think the bones are there.

variable "name" {
  type = string
}

variable "image" {
  type = string
}

variable "proxies" {
  type = list(object({
    from = number
    to   = number
  }))
  default = []
}

variable "files" {
  type = list(object({
    source_path        = string
    target_path        = string
    uid                = optional(number, 1000)
    gid                = optional(number, 1000)
    mode               = optional(string, "0644")
    create_directories = optional(bool, true)
  }))
  default = []
}

variable "project" {
  type    = string
  default = "coop"
}

terraform {
  required_providers {
    incus = {
      source  = "lxc/incus"
      version = "~> 1.2"
    }
  }
}

provider "incus" {
  remote {
    name    = "local"
    address = "unix://"
  }

  remote {
    name     = "docker"
    address  = "https://docker.io"
    protocol = "oci"
    public   = true
  }
}

resource "incus_project" "coop" {
  name = var.project
}

resource "incus_profile" "default" {
  project     = incus_project.coop.name
  name        = "default"
  description = "default profile for project coop"

  device {
    name = "root"
    type = "disk"
    properties = {
      pool = "default"
      path = "/"
    }
  }

  device {
    name = "eth0"
    type = "nic"
    properties = {
      network = "incusbr0"
    }
  }
}

resource "incus_storage_pool" "coop" {
  name    = "coop"
  driver  = "dir"
  project = incus_project.coop.name
}

resource "incus_storage_volume" "coop" {
  name    = var.name
  pool    = incus_storage_pool.coop.name
  project = incus_project.coop.name

  dynamic "file" {
    for_each = var.files
    content {
      source_path        = file.value.source_path
      target_path        = file.value.target_path
      uid                = file.value.uid
      gid                = file.value.gid
      mode               = file.value.mode
      create_directories = file.value.create_directories
    }
  }
}

resource "incus_instance" "coop" {
  name    = var.name
  image   = var.image
  project = incus_project.coop.name

  # TODO: should this be one device or multiple?
  dynamic "device" {
    for_each = var.files
    content {
      # TODO: suffix with file basename? make sure to not double basename
      name = "config"
      type = "disk"
      properties = {
        path   = device.value["target_path"]
        source = "${incus_storage_volume.coop.name}/${device.value.target_path}"
        pool   = incus_storage_pool.coop.name
      }
    }
  }

  dynamic "device" {
    for_each = var.proxies
    content {
      # TODO: make sure these values are sanitized
      # TODO: only supports numbers for now
      name = "proxy-${device.value.from}-${device.value.to}"
      type = "proxy"

      properties = {
        listen  = "tcp:0.0.0.0:${device.value.from}"
        connect = "tcp:127.0.0.1:${device.value.to}"
      }
    }
  }
}

This might be a good candidate to try out OpenTofu Symbol Libraries

15

Cool. Thanks for sharing. If you are interested in creating modules you might also want to take a look at Terragrunt. It fixes a lot of the problems that arise over time when a many modules are used in several different projects.