Cloud-init configuration stopped working

Incus
1

Hello, I’ve been setting up some instances using incus and the terraform/opentofu provider. I used a template for cloud-init configurations like this

#cloud-config
groups:
  - admingroup: [root, sys]

users:
  - name: ${lxc_username}
    gecos: ${lxc_username_description}
    shell: /bin/bash
    homedir: /home/${lxc_username}
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users
    lock_passwd: false
    ssh_authorized_keys:
      - ${ssh_public_key}
    passwd: ${lxc_user_password}
  - name: ansible
    gecos: Ansible Automation Account
    shell: /bin/bash
    homedir: /home/ansible
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, sys
    lock_passwd: false
    ssh_authorized_keys:
      - ${ssh_public_key}
    passwd: ${ansible_password}
ssh_pwauth: False
packages:
  - podman
  - openssh-server
timezone: America/Bogota
write_files:
  - path: /etc/ssh/sshd_config
    permissions: "0600"
    content: |
      Port 22
      ListenAddress 0.0.0.0
      PermitRootLogin no
      PasswordAuthentication no
      ChallengeResponseAuthentication no
      UsePAM yes
      X11Forwarding no
      PrintMotd no
      Subsystem	sftp	/usr/lib/openssh/sftp-server

And my instances configuration look like the following


resource "incus_instance" "code" {
  name  = "code"
  image = "images:debian/trixie/cloud"

  config = {
    "boot.autostart"              = true
    "limits.cpu"                  = 2
    "limits.cpu.allowance"        = "20%"
    "limits.memory"               = "4GiB"
    "limits.memory.swap"          = "true"
    "limits.memory.swap.priority" = "0"
    "cloud-init.user-data" = templatefile(
      "${path.module}/cloud-init/generic.yml.tftpl",
      {
        ssh_public_key           = var.ssh_public_key
        lxc_user_password        = var.lxc_user_password
        lxc_username             = "code"
        lxc_username_description = "Coding Linux System Container"
        ansible_password         = var.ansible_password
      }
    )
    "security.nesting" = true
  }

  device {
    name = "eth0"
    type = "nic"
    properties = {
      network        = var.code_net.name
      "ipv4.address" = "10.0.7.10"
      "security.acls" = join(",",
        [
          incus_network_acl.code_svc.name
      ])
    }
  }

  device {
    name = "root"
    type = "disk"
    properties = {
      path = "/"
      pool = var.virt_main_pool
    }
  }
  device {
    name = "data"
    type = "disk"
    properties = {
      path   = "/home"
      source = incus_storage_volume.code_data.name
      pool   = var.virt_high_io_pool
    }
  }
}

I configured another instance named ‘container-host’ but I noticed there’s no ipv4 address when it is created

| container-host | RUNNING |                          | fd42:835c:7268:cb10:216:3eff:fe9d:213d (eth0) | CONTAINER | 0         |

I thought maybe it was a networking issue since I just setup OVN but when I create my instance without the cloud image…

| container-host | RUNNING | 10.0.12.10 (eth0)        | fd42:835c:7268:cb10:216:3eff:fed6:7ccf (eth0) | CONTAINER | 0         |

}

The container gets an IP address. So maybe the problem was my template so I made it extemely simple and placed it directly in my tofu file

  config = {
    "boot.autostart"              = true
    "limits.cpu"                  = 1
    "limits.cpu.allowance"        = "10%"
    "limits.memory"               = "8GiB"
    "limits.memory.swap"          = "true"
    "limits.memory.swap.priority" = "0"
    "cloud-init.user-data"        = <<-EOT
      #cloud-config
      network:
        config: disabled
      packages:
        - podman
    EOT
    "security.nesting"            = true
  }

It seems to be detected correctly when I plan and apply the configuration

  # module.system-containers.incus_instance.container_host will be created
  + resource "incus_instance" "container_host" {
      + config       = {
          + "boot.autostart"              = "true"
          + "cloud-init.user-data"        = <<-EOT
                #cloud-config
                packages:
                  - podman
            EOT

The configuration doesn’t seem to contain anything weird

hypervisor@hypervisor:~$ incus config show container-host
architecture: x86_64
config:
  boot.autostart: "true"
  cloud-init.user-data: |
    #cloud-config
    packages:
      - podman
  image.architecture: amd64
  image.description: Debian trixie amd64 (20250327_05:24)
  image.os: Debian
  image.release: trixie

When I check my code instance that was created months ago, I can see it got the configuration

root@code:~# cat /var/lib/cloud/instance/user-data.txt
#cloud-config
groups:
  - admingroup: [root, sys]

users:
  - name: code
    gecos: Coding Linux System Container
    shell: /bin/bash
    homedir: /home/code
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users
    lock_passwd: false
    ssh_authorized_keys:
      - 
    passwd:
  - name: ansible
    gecos: Ansible Automation Account
    shell: /bin/bash
    homedir: /home/ansible
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, sys
    lock_passwd: false
    ssh_authorized_keys:
      - 
    passwd: 
ssh_pwauth: False
packages:
  - podman
  - openssh-server
timezone: America/Bogota
write_files:
  - path: /etc/ssh/sshd_config
    permissions: "0600"
    content: |
      Port 22
      ListenAddress 0.0.0.0
      PermitRootLogin no
      PasswordAuthentication no
      ChallengeResponseAuthentication no
      UsePAM yes
      X11Forwarding no
      PrintMotd no
      Subsystem sftp    /usr/lib/openssh/sftp-server

But there’s nothing in my container-host

root@container-host:~# cat /var/lib/cloud/instance/user-data.txt
root@container-host:~#

This setup used to work in the past and I recently upgraded Incus and its provider so maybe something has changed on how the cloud-init configuration is sent or maybe I’m doing somerhing silly and cannot see it

Thanks :slightly_smiling_face:

2

After a lot of debugging, I noticed that I was setting up a disk that would be mounted on /var

  device {
    name = "var"
    type = "disk"
    properties = {
      path   = "/var"
      source = incus_storage_volume.container_host_var.name
      pool   = var.virt_high_io_pool
    }
  }

I did it because I want to persist some container volumes I’ll set up on Docker, I didn’t expect it to have such an effect on cloud-init but of course, its configurations end up in /var after all :slightly_smiling_face:

I removed this disk and now the machine works perfectly

| container-host | RUNNING | 10.0.12.10 (eth0)        | fd42:835c:7268:cb10:216:3eff:fe80:3508 (eth0) | CONTAINER | 0         |

So in the end, don’t mount the entire /var on custom volumes and instead mount specifically what I need :slightly_smiling_face:

2 Likes