How are you using LXC?

Hello there,

I’m wondering how you, as users, are using LXC (bare LXC, not LXD) ?
What do you use, how do you automate stuff around it, how do you handle networking, etc.

Allow me to go first:

I’m using LXC for my personal stack of services (One container per bullet point):

  • A postfix server for SMTP
  • A dovecot server for IMAP
  • An OpenDKIM daemon
  • An OpenDMARC daemon
  • A spf-engine daemon
  • A Certbot client (Run periodically for updating certificates)
  • An nginx server (Used as a reverse-proxy for every other HTTP applications)
  • An nginx server to serve static files for my personal website
  • A radicale server for CalDAV/CardDAV

Every container rootfs and metadata was built from Dockerfiles. I automated part of the process with a Python script and some shell scripts. The process is as follow:

  • I write a Dockerfile
  • I build the Dockerfile (thanks to img)
  • I extract the OCI image (also through img)
  • I transform the OCI image to a -rootfs.tar and -meta.tar files (With my home-made Python script)
  • Create the container with the lxc-local template

For the root images, I’m relying on distrobuild to make the necessary “basic” system images.

For run-time, I’m essentially starting containers by hand for now. Everything is running on Debian 11, and I didn’t automated start/stop for now. All containers are unprivileged.

For networking, every container has one “private” interface and one “public” interface, which are veth devices in bridge mode, but assigned to the “none” bridge. The daemon is running as root and handle veth peer assignation and sets routing and bridge-level firewalling. The process is triggered via a hook script that communicate with the daemon via an UNIX socket.

Everything is a bit clunky. I mainly did what I did to learn about LXC and linux namespaces, but now that I “need” “production-grade” quality of service, I’m wondering whether I should continue with this setup, or start using LXD or libvirt. I’m starting to get frustrated with the networking setup. I’d like to have a firewall contained within the containers network namespace, but LXC won’t allow me to do that easily with scripting only. There is the start-host hook and the start hook, but the first doesn’t allow me to nsenter into the container’s network namespace, while the other forces me to have the hook script and nftables available withing the rootfs of the container.

I’d like to share some of the code I wrote for managing this stack. This is overall badly designed, but I’m alone, I don’t have a lot of time to write code, so I did what I could :face_holding_back_tears:

Why not rspam ?

I built a collection of ansible scripts to install lxc on a fresh machine and create containers that I define in my inventory file.

After a lot of struggle with unpriviledged users for me the coin dropped that I can create and manage the containers as root but still have unpriviledged containers with mapped uids/guids.

The nice thing about my setup is I can configure the complete environment in my ansible inventory file and its really lightweight, no docker stuff involved, everything is plain Debian.

I have one container acting as a nginx reverse proxy and defined some iptables firewall rules to forward the traffic from the public interface to the proxy container. On the proxy I have the letsencrypt stuff running and connect to the other containers with the name based web projects.

I wonder why you use docker images inside LXC?

I take it that you are running several services in a single container ? Are you following the “system in a container” rather than “application in a container” dogma ?

How are you achieving that ? Is your Ansible playbook configuring the reverse-proxy for you, or does it involve a local DNS and service-discovery protocol ?

I’m not. I’m only building the rootfs and metadata with a home-made script. I find it way simpler to build my rootfs with a Dockerfile builder rather than manually. For my needs, the “application in a container” dogma is the only sensible way.

Aaand I just discovered that the lxc-oci template is a thing… I’m kind of disappointed in myself for not looking for such a thing prior to writing my Python script. I yet have to see what lxc-oci is doing exactly, but given the name and what the issues are saying about it, I believe this does exactly what my script does.

EDIT: It does everything my script does. And it does even something my script doesn’t do yet, which is starting with specific uid/gid.

Yes I use “application in a container”. I view the containers more like little VMs. With the plugin GitHub - chifflier/ansible-lxc-ssh: Ansible connection plugin using ssh + lxc-attach Ansible can connect to a container withouth SSH so for my scripts its transperent wheter I configure a container or a “real” machine.

I do everything in the playbooks, my “LXC” playbook for the outer host defines iptables mappings to the internal proxy container.

Also in every container I use the vanilla debian image from LXC, so I can have the “stable” release thats valid for a few years and get easy security upgrades…

I believe this is “system in a container” then.

Understood. This is definitely “system in a container” then.

Debian is indeed very stable. I use it for my few system containers. My applications containers are based on Alpine, unless a software is giving me too much trouble being compiled with musl, or is a proprietary blob. I prefer very small containers sizes, since LXC doesn’t have the efficient layering system of docker. It can be emulated though, but better directly using docker rather than maintaining the overlayfs layers manually.