I wonder if anyone can help here. cloud-init is correctly running on first start, but if I clean it all out with “cloud-init clean …” and restart the container then it’s not running a second time.
More specifically:
$ incus create images:ubuntu/22.04/cloud foo
$ sudo ls /var/lib/incus/containers/foo/rootfs/etc/netplan /var/lib/incus/containers/foo/rootfs/var/lib/cloud
... confirms those directories are not present
$ incus start foo
$ sudo ls /var/lib/incus/containers/foo/rootfs/etc/netplan /var/lib/incus/containers/foo/rootfs/var/lib/cloud
/var/lib/incus/containers/foo/rootfs/etc/netplan:
50-cloud-init.yaml
/var/lib/incus/containers/foo/rootfs/var/lib/cloud:
data handlers instance instances scripts seed sem
$ incus shell foo
root@foo:~# cloud-init clean --seed --logs --configs all
2024-04-19 14:38:46,174 - subp.py[WARNING]: skipping /etc/cloud/clean.d/README as its not executable or the underlying file system is mounted without executable permissions.
root@foo:~# ls /etc/netplan
root@foo:~# ls /var/lib/cloud
root@foo:~# exit
logout
$ incus stop foo
$ incus start foo
I find that /etc/netplan and /var/lib/cloud are not populated, and as a result, the instance does not pick up an (IPv4) address.
What’s different between the second startup and the first?
I think I found it. Before first boot, the container has volatile.apply_template: create. The image templates (in metadata.yaml and *.tpl) unpack various cloud-init files under /var/lib/cloud/seed/nocloud-net/
If I understand this correctly, it means there could be up to three different cloud-init mechanisms at play, depending on what you’re doing with incus:
nocloud files stuffed directly into the container
an ISO device that you attach to the VM
the lxd/incus data source via the incus agent (although I’m now not sure if or when that’s actually used)
Yeah, for the case where cloud-init is fed configuration through on-disk files, which includes the default NoCloud files in containers and VMs (created by the agent in the VM case), then you need to set volatile.apply_template to create to have Incus re-generate those files.
The native LXD datasource is meant to better handle this but it’s not quite working with Incus nor is it available in all distributions yet.
And indeed, when attaching a cloud-init ISO to a VM, those files aren’t templated and so will not need a reset of the volatile key.
It’s a multi-stage build which involves creating a bunch of different containers, and then assembling them into a master VM image (this is done as a separate stage, so that if I change one container I don’t have to rebuild all the others).
I was fully cleaning the containers, but when the VM started they weren’t picking up addresses via DHCP. For now I’ve just removed the cleaning step.