I’d like to create an image of an instance I’ve set up, and be able to spawn instances of that image on various other hosts, supplying different user-data to each.
At first we tried:
lxc export my-container backup.tar.gz
# On another host
lxc import backup.tar.gz my-new-container
lxc stop my-new-container
lxc config set my-new-container user.user-data=...
This does indeed create a matching container on the other host, however the user-data never runs (or at least, the runcmd within).
I suspect this is either because cloud-init won’t run the runcmds again on an imported instance, or stopping and setting the config, and restarting the container won’t run those runcmds again.
Is there a workflow that can achieve what I need? I’ve done a bit of reading around and it looks like maybe I should rather be doing some combination of lxc publish, lxc image export and lxc image import. But I’m having trouble working out what the differences are between publish/export and also whether or not a published image will run a given user-data again.
Using lxc export is creating a backup of that particular instance, to be restored onto the same or different machine at a later point, rather than being intended for creating duplicates of the original instance.
Aside from the issue you describe of cloud-init not running again (I believe this is actually a behaviour of cloud-init rather than LXD as it only runs on first-boot), you may also experience problems with the machine-id and MAC address being the same for both source and imported instance, which could then go onto cause IP conflicts if the DHCP server thinks they are the same machine.
Instead you could consider using lxc publish to create a ‘pristine’ image from the existing instance, and then you can export and import that image using lxc image export and lxc image import, and then subsequenrly using lxc init <image alias> to create an instance from it (this will regenerate the MAC address and system-id in the process).
As for the cloud-init issue, this may still be an issue, as cloud-init may have written internal state data that stops it running again. This would need to be removed from the source instance before you do lxc image export.
I will give that a go and let you know how it goes.
I suspect you’re right about cloud-init, but I suspect there’s a way to clear that data before I publish it.