My company https://greymatter.io uses LXD for running our CI/CD pipelines.
Architecture, Motivation
An LXD server (usually one big AWS instance) running ~9 LXD containers of different types. The LXD containers are built with Packer from a single file, which helps us keep them as consistent as possible, and only introduce differences where it’s necessary. For instance, we have Alma linux and two versions of Ubuntu to produce different builds for our customers, and to test our builds on different systems.
The agents are Buildkite. We migrated to Buildkite from CircleCI to get 1) major build speedups and 2) enforce a shared environment across our projects. The speedups come from pre-baking all the tools and packages we need into the containers, and from the better-performing server that we provision. We don’t put any CPU or memory constraints on our LXD containers. We just let the underlying Linux scheduler do its thing.
One of the artifacts we produce from these Buildkite agents are OCI containers. We use the buildah tool, and podman. These tools leave behind lots of layers that can fill up the disk, but since each environment is an LXD container, we can delete the agent’s container and the disk space problem is remediated. We want to introduce automation to “roll” these containers regularly automatically, but we haven’t yet.
Other Uses
We do run some internal tools and testing environments on other LXD servers, but CI/CD is our most serious use case, for now.