Cloud-init not being run on second boot

Incus
1

I wonder if anyone can help here. cloud-init is correctly running on first start, but if I clean it all out with “cloud-init clean …” and restart the container then it’s not running a second time.

More specifically:

$ incus create images:ubuntu/22.04/cloud foo
$ sudo ls /var/lib/incus/containers/foo/rootfs/etc/netplan /var/lib/incus/containers/foo/rootfs/var/lib/cloud
... confirms those directories are not present
$ incus start foo
$ sudo ls /var/lib/incus/containers/foo/rootfs/etc/netplan /var/lib/incus/containers/foo/rootfs/var/lib/cloud
/var/lib/incus/containers/foo/rootfs/etc/netplan:
50-cloud-init.yaml

/var/lib/incus/containers/foo/rootfs/var/lib/cloud:
data  handlers	instance  instances  scripts  seed  sem

$ incus shell foo
root@foo:~# cloud-init clean --seed --logs --configs all
2024-04-19 14:38:46,174 - subp.py[WARNING]: skipping /etc/cloud/clean.d/README as its not executable or the underlying file system is mounted without executable permissions.
root@foo:~# ls /etc/netplan
root@foo:~# ls /var/lib/cloud
root@foo:~# exit
logout

$ incus stop foo
$ incus start foo

I find that /etc/netplan and /var/lib/cloud are not populated, and as a result, the instance does not pick up an (IPv4) address.

What’s different between the second startup and the first?

2

I think I found it. Before first boot, the container has volatile.apply_template: create. The image templates (in metadata.yaml and *.tpl) unpack various cloud-init files under /var/lib/cloud/seed/nocloud-net/

If I understand this correctly, it means there could be up to three different cloud-init mechanisms at play, depending on what you’re doing with incus:

  • nocloud files stuffed directly into the container
  • an ISO device that you attach to the VM
  • the lxd/incus data source via the incus agent (although I’m now not sure if or when that’s actually used)
3

I believe that’s correct, I did find somewhere you could reinitialize but it’s like creating a new instance. What are you trying to do?

Note

The cloud-init actions are run only once on the first start of the instance. Rebooting the instance does not re-trigger the actions

4

Yeah, for the case where cloud-init is fed configuration through on-disk files, which includes the default NoCloud files in containers and VMs (created by the agent in the VM case), then you need to set volatile.apply_template to create to have Incus re-generate those files.

The native LXD datasource is meant to better handle this but it’s not quite working with Incus nor is it available in all distributions yet.

And indeed, when attaching a cloud-init ISO to a VM, those files aren’t templated and so will not need a reset of the volatile key.

5

What are you trying to do?

It’s a multi-stage build which involves creating a bunch of different containers, and then assembling them into a master VM image (this is done as a separate stage, so that if I change one container I don’t have to rebuild all the others).

I was fully cleaning the containers, but when the VM started they weren’t picking up addresses via DHCP. For now I’ve just removed the cleaning step.

6

I thought that was referring to the behaviour of cloud-init itself - which maintains state between boots.

7

I have stumbled upon this exact problem and the volatile.apply_template=create indeed fixed it.

Here is an example opentofu/terraform snippet to add to your .tf file to automatically set this config, trigger cloud-init clean and then reboot the VMs:

# Forces a cloud-init reseed on the VM whenever its cloud-init
# config changes, without destroying/recreating the VM itself.
# Relevant documentation: https://discuss.linuxcontainers.org/t/cloud-init-not-being-run-on-second-boot/19740/4
resource "terraform_data" "cloudinit_reseed" {
  for_each = var.vms

  input = incus_instance.vm[each.key].config["cloud-init.user-data"]

  triggers_replace = [
    incus_instance.vm[each.key].config["cloud-init.user-data"],
  ]

  provisioner "local-exec" {
    command = <<-EOT
      set -euo pipefail
      NAME="${each.value}"

      echo "Waiting for incus-agent on $NAME..."
      incus wait "$NAME" agent --timeout 60

      incus config set "$NAME" volatile.apply_template=create
      incus exec "$NAME" -- cloud-init clean --logs
      incus stop "$NAME"
      incus start "$NAME"

      echo "Waiting for cloud-init to finish on $NAME..."
      incus wait "$NAME" agent --timeout 60
      incus exec "$NAME" -- cloud-init status --wait 2>/dev/null
    EOT
  }

  depends_on = [incus_instance.vm]
}

This assume the incus VMs are in incus_instance.vm in the .tf file.

8

FYI, you can now replace these loops with:

incus wait "$NAME" agent --timeout 60

or in the second case:

incus wait "$NAME" agent --timeout 60
incus exec "$NAME" -- cloud-init status --wait

without needing the loop, or having to grep the output.

Note also that cloud-init status can return values of 0 for success and 2 for “partial success”, so if you want to continue on partial success then you need to test for that value, e.g.

rc=0
incus exec "$NAME" -- cloud-init status --wait --format json || rc="$?"
if [ "$rc" -ne 0 -a "$rc" -ne 2 ]; then
  echo "*** cloud-init failed ($rc) ***"
  exit "$rc"   # or return "$rc", depending on context
fi

An example of a partial success:

  "recoverable_errors": {
    "WARNING": [
      "Device '/dev/a9b3ae14-6fde-4a11-826e-45f8d766ee67' did not exist. cannot resize: dev=a9b3ae14-6fde-4a11-826e-45f8d766ee67 mnt_point=/ path=rpool"
    ]
  },
9

TIL! Thanks I updated the example. Still kept non-zero as error as in my case I want to treat partial success as failure