@rfruit gave me access to a testing VPS that exhibited this issue and I had a look.
This is the write-up.
There was a routed Incus profile that when it was applied to an Incus image, it would create an instance that did not have working DNS settings. You would have to configure manually the DNS. This would only happen to images from images.linuxcontainers.org, though there were some working old images from a few years ago (pre-Incus).
On my system the same routed configuration would work and produce instances with proper network configuration. What could be happening?
Is there any significance with the above innocuous indentation? Sure, there is.
$ incus exec TestDNS -- cat /var/log/cloud-init-output.log
Cloud-init v. 24.2-0ubuntu1~24.04.2 running 'init-local' at Mon, 16 Sep 2024 20:58:29 +0000. Up 0.57 seconds.
2024-09-16 20:58:29,931 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 13 column 16: "mapping values are not allowed here
in "<unicode string>", line 13, column 16:
via: 169.254.0.1
^"
...
$
So, there’s significance with the indentation at that part of the profile.
If they are not indented properly, then the network configuration fails. It does not fail completely because the specific routed has some configuration that is not part of cloud-init, which still works. Hence the difficulty in the troubleshooting.
When you use cloud-init instructions in a profile, those instructions are pushed by Incus into the container image. The instance is launched, and it runs whatever instructions are supplied. If there is an error, it can be found in /var/log/cloud-init-output.log.
But were those previous versions of cloud-init more relaxed in accepting bad-formed YAML configuration? Let’s test this with Incus.
First, we need an old container image from Ubuntu 20.04, from April 2024. Here it is.
Then, we download two of the files.
$ wget http://cloud-images-archive.ubuntu.com/releases/focal/release-20200423/ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz
...
$ wget http://cloud-images-archive.ubuntu.com/releases/focal/release-20200423/ubuntu-20.04-server-cloudimg-amd64-root.tar.xz
...
$ incus image import ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz ubuntu-20.04-server-cloudimg-amd64-root.tar.xz --alias ubuntu-from-canonical-20.04
Image imported with fingerprint: af34e9b8cb04c78250a4967306a45ca1dee482e91aca49760d22b070086d42fa
$ incus launch images:ubuntu/20.04/cloud TestDNS --profile default --profile routed
Launching TestDNS
$ incus exec TestDNS2 -- cat /var/log/cloud-init-output.log
Cloud-init v. 20.1-10-g71af48df-0ubuntu5 running 'init-local' at Mon, 16 Sep 2024 21:23:06 +0000. Up 0.74 seconds.
2024-09-16 21:23:06,063 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 13 column 16: "mapping values are not allowed here
in "<unicode string>", line 13, column 16:
via: 169.254.0.1
^"
2024-09-16 21:23:06,063 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'> failed
$
It is likely that the YAML format was not relaxed in that aspect at any time, it’s just the formatting detail is so innocuous that it is easy to not notice it.
Is there a validator for cloud-init files?
There’s cloud-init schema, though it does not appear to work with user-data.
$ cat routed.yaml
#cloud-config
version: 2
ethernets:
eth0:
addresses:
- 192.168.1.200/32
nameservers:
addresses:
- 8.8.8.8
search: []
routes:
- to: 0.0.0.0/0
via: 169.254.0.1
on-link: true
$ cloud-init schema --config-file routed.yaml
Invalid user-data routed.yaml
Error: Cloud config schema errors: ethernets: Additional properties are not allowed ('ethernets' was unexpected)
Error: Invalid schema: user-data
$
There’s supposed to be a validator with cloud-init devel schema on some forums, though it did not work for me. The cloud-init in Ubuntu 24.04 does not have devel schema.