I normally use Docker, but I’m working on a gig that requires lxc. I’m used to configuring docker images with a Dockerfile: start with this image, run this command, copy that file, etc. It does simple dependency management so rebuilds go quickly.
What are my options for doing nice repeatable builds of lxc containers? Should I just start the image and run a chef/ansible/puppet recipe on it? Are there some examples, best-practices I should check out?
Cloud-init is what your looking for
Write your yaml to a file then a command like
lxc profile set PROFILE_NAME user.user-data - < path/to/file
Then lxc launch ubuntu: c1 -p PROFILE_NAME
Alternativly my web app https://github.com/turtle0x1/LxdMosaic has support for
This is something that is missing from LXD. I think third party tools like the ones you mention are the way to go, for now.
Launching and configuring a container with a single command + a configuration script would be nice. I have done something like it with my own tools, but I’m not happy with it yet. The general idea is to “push” configuration scripts, ssh keys, and other files from the host to the container and run them or install them in the container.
I reject cloud-init because it does not work with alpine and other images, nor would I want to bloat my small alpine images with cloud-init. I also find the clould-init configuration file too monolithic and different from everything else linux. I only use it for very simple things, like preventing the “ubuntu” user from being created on ubuntu images.
If you launch several containers with the same configuration, you can configure one “template” container and launch others by copying (cloning) the template container (or its snapshot). But there is still the problem of scripting the configuration of the template container.
It becomes even more complicated if you want to also configure devices with each container you launch. My goal is to have all my data in separate disk devices, and be able to swap in a new version of the container (the root filesystem) when I need to update any software. It requires a lot of housekeeping and it complicates taking snapshots.
I haven’t used Docker much, but I believe it is designed to do this. They say that LXD is for OS containers while Docker is for application containers, but in a production environment, you use OS containers to run applications, so I would like to see a convergence of the two approaches.
So I reject your rejection of cloud-init 
Given the following scenario:
You could do the following
Create a file containing
#cloud-config
# Apply updates using apt
package_update: true
package_upgrade: true
# Install packages
packages:
- apache2
- php-common
- php7.2
- libapache2-mod-php7.2
runcmd:
- timedatectl set-timezone Europe/London
- rm -rf /etc/apache2/sites-available/default-ssl.conf
- DO_A_BUNCH_OF_OTHER STUFF
then import this into a profile lxc profile PROFILE_NAME set user.user-data - < FILE_LOCATION
How this is less “monolithic and different from everything else linux.” than a DockerFile which may contain lines like the one below is beyond me
RUN apt-get update -y && apt-get install -y - apache2 php-common php7.2 libapache2-mod-php7.2
RUN timedatectl set-timezone Europe/London && rm -rf /etc/apache2/sites-available/default-ssl.conf
The lines can also get harder to read (if people don’t properly line break them) as you try to reduce the number of “layers” in your docker image
Now if your going to argue “id just make my own docker image” then just go and make your own LXD image and optionally don’t bother with cloud-init.
Create as many profiles as you need with different device & disk declarations (I’m not gonna repeat the documentation here)
You could even include the extra disk definitions in the same profile as the cloud-config instructions (as all containers of this “type” need the same disks right?)
Well now all our provisioning info is in profiles its as simple as
lxc launch ubuntu: c1 -p PROFILE_WITH_CLOUD_INIT -p PROFILE/S_WITH_DISKS_OR_DEVICES
Compared with docker-compose this is not as simple no, you would just run docker-composer and it would locate the file/s and run it.
But docker still requires a docker-composer.yml (disks & devices) & DockerFile (user.user-data) the only difference between LXD & Docker is that LXD stores this info in a database instead of flat files!
This used to be true, but there was a lot of work to bundle in cloud-init to the images
A lot of the provided images now have cloud-init variants and alpine was / is one of them (3.12 support coming soon?)
Thanks @turtle0x1 this is super helpful. I haven’t seen such a simple explanation of a cloud-init config file. Something like this should be in the docs!
Hi Gary,
I would advise you use Packer. It is a well done open-source project that is made to generate images for a lot of platforms. It has lots of builders to generate images (Docker, lxc/lxd, AMIs, etc) and multiple provisioners to configure those. You can for instance run the LXC builder to create an LXC container and run the shell provisioner to configure your image. Or change the provisioner to Ansible to do the same.
In your case, behind the scenes, it will indeed start an lxc container and run a shell script (in the case of the shell provisioner) or Ansible. This makes it easy to switch from one type of runtime to another.
Dockerfile parallel in lxd/lxc are distrobuilder yaml configs. https://github.com/lxc/distrobuilder
There is no such thing as a Dockerfile for lxc/lxd.
Yes the Answer with cloud-init is correct, however the image has to be configured to be compatible with cloud-init.
If you use the ubuntu: images you are fine with cloud-init, if you use any other image you can’t use cloud-init.
What works quite well is to use ansible, simply create a playbook and apply it to any container (yes the comma at the end is needed):
ansible-playbook playbook.yml -c lxd -i CONTAINER_NAME,
Recently, most container images from images: have a cloud-init version. Therefore, it is possible to use cloud-init on them too.
Run the following to get the full list.
lxc image list images:cloud
As noted, all container images at ubuntu: have cloud-init.
There should (or could) be some configuration transpiler to convert Dockerfile to something better usable by LXD. Something somewhat similar on Github is https://github.com/coreos/container-linux-config-transpiler There could be something readily available after some better searching.
Now thats an idea, bit complex as its probably full “bash” string extractions
You can add cloud-init compatibility by adding the following files to the metadata.yml:
/var/lib/cloud/seed/nocloud-net/meta-data:
when:
- create
- copy
template: cloud-init-meta.tpl
/var/lib/cloud/seed/nocloud-net/user-data:
when:
- create
- copy
template: cloud-init-user.tpl
properties:
default: |
#cloud-config
{}
/var/lib/cloud/seed/nocloud-net/vendor-data:
when:
- create
- copy
template: cloud-init-vendor.tpl
properties:
default: |
#cloud-config
{}
Where cloud-init-meta.tpl is:
instance-id: {{ container.name }}
local-hostname: {{ container.name }}
{{ config_get("user.meta-data", "") }}
cloud-init-user.tpl is:
{{ config_get("user.user-data", properties.default) }}
cloud-init-vendor.tpl is:
{{ config_get("user.vendor-data", properties.default) }}
And of course you need to install cloud-init on the target image 