Incus cloud-init & LXD datasource

Question :slight_smile:

Are there any thoughts/guidance on how to provision an incus VM using the cloud-init LXD Datasource? Is it a reasonable approach over nocloud . Advantages can be used to hot deploy cloud-init config changes with meta-data including config tags from incus.

I observed both ds -identity and cloud-init-local are unable to identity the LXD Datasource as they run earlier in the systemd boot process ahead of the incus-agent thus cannot detect the /dev/incus/sock; further ds-identity will also checks BOARD for “LXD” rather than “Incus” (cloud-init issue). But once systemd has finished boot , manually running cloud-init is fine and DatasourceLXD is detected.

I’ve been feeling my way using both Jammy and Noble cloud images; to enable DatasourceLXD I’ve deleted the cloud-init templates from the meta-data to avoid the creation of /var/lib/cloud/seed/nocloud-net/… files on first start.

Regarding, nocloud-net, I noticed in the cloud-init code that nocloud-net is being depreciated in favor of nocloud.

I am still learning so much by reading code and documentation, forgive me if I write with misunderstandings; I have so much respect for the Incus architecture, implementation plus community. Simply exemplary.

1 Like

Right, ideally, we’d want for cloud-init to either have the LXD datasource handle both Incus and LXD or have a separate Incus datasource. That should then use /dev/incus/sock as well as having the updated DMI board name.

After that, VM handling is indeed a bit tricky because /dev/incus/sock isn’t accessible until after the agent has started. That’s usually not an issue because our agent is designed to run before any cloud-init unit, but that doesn’t help with ds-identify which is run as a systemd generator and so runs before our unit.

The other issue we keep running into is just inconsistency in cloud-init versions available across distro and releases. It’d be great if we could have proper Incus support added to cloud-init now so that in a few years time we can fully do away with NoCloud and fully rely on cloud-init directly interacting with Incus.

There is one further potential snag too, security.guestapi=false will turn off /dev/incus/sock which then prevents the native datasource from working, the agent can still put the files through NoCloud in that case though.

So ideally, I think we’d want cloud-init to have a native Incus datasource (can be a shim around the LXD one). That should detect our DMI info and if on Incus, it should set Incus as the main datasource with a fallback on NoCloud/NoCloudNet and enable the cloud-init units.

Then incus-agent is setup to run before the cloud-init units, it will make /dev/incus/sock available if security.guestapi isn’t set to false. It should also be able to detect that cloud-init has the native datasource configured, if it does and /dev/incus/sock is available, it should refrain from populating NoCloudNet. But if cloud-init isn’t configured for the native datasource or /dev/incus/sock has been disabled, then it should write the fallback data to NoCloudNet so cloud-init can consume that.

1 Like

Dear Stéphane - thank you kindly for your timely and well thought out reply; clearly you state a considered ambition.

Without wanting to detract from your statement, can I just ask regarding ‘our agent is designed to run before any cloud-init unit’ how is this achieved as it appears incus-agent.service is fired by the udev rule which starts well after cloud-init-local and because of udev I am guessing the unit dependencies are not honored by systemd.

For the curious here’s the boot chart which shows the snag, we need incus-agent ahead of cloud-init-local + ds-identity (called via a generator). Sorry about this size of the image; if you zoom in and squint, note cloud-init-local.service starts well before incus-agent.service

Also unrelated, distrobuilder cloud-init generator use user.meta-data. What is the position regarding no-cloud and meta-data how should the nocloud cloud-init.meta-data file be configured?