Readonly LXD containers?

Can an LXD container be readonly? I mean that the root (zfs) filesystem is readonly.
I tried it and I found that the answer is yes and no. Yes, because I was able to create a container that runs with a readonly root filesystem. No, because LXD fails to start or stop such a container.

First of all, I attached writable disk devices to the directories that the container might need to write to, such as /var/log, and /tmp.

Having done this, I made the zfs filesystem of the container readonly, while the container was running, and the container continued to run. I was able to start a shell in it and download a file from the internet (to a writable directory).

I used an images:alpine/3.17 image, with just the dhcpcd package added on top of the image.

But when I tried to start or stop such a container, the LXD operations failed. That’s because LXD modifies metadata files in the container filesystem, and also makes some other changes. These fail if the filesystem is readonly.

For the examples below, I use z/lxd as the zfs storage pool filesystem.

Stopping a container (b) with a readonly root filesystem:

sudo zfs set readonly=on z/lxd/containers/b
lxc stop b
  
Error: Failed clearing ownership: chown /var/snap/lxd/common/lxd/containers/b: read-only file system
Try `lxc info --show-log b` for more info

lxc list -f compact b
  NAME   STATE   IPV4  IPV6    TYPE     SNAPSHOTS  
  b     STOPPED              CONTAINER  0          

The container seems to have stopped.

Starting a container (b) with a readonly root filesystem:

sudo zfs set readonly=on z/lxd/containers/b
lxc start b

Error: saving config file for the container failed
Try `lxc info --show-log b` for more info

The container did not start.

Try setting “readonly: true” on the lxd root disk device config (rather than the underlying device) and see if that helps.

Yes, this works! I like it. I’ll see how I can incorporate this to my workflow.

I put readonly: true in a copy of the default profile, and swapped profiles after creating the container.

I noticed that:

  • In order for the setting to have effect, I had to stop the container and restart it. I cannot make a container readonly while it is running.
  • If a container with readonly root is stopped, “lxc file push” succeeds. If the container is running, “lxc file push” to the root filesystem fails with a “read-only file system” error:
lxc file push a.txt b/
Error: sftp: "open /a.txt: read-only file system" (SSH_FX_FAILURE)

Yes, this is because the effect is achieved using mount options, rather than using a feature in the underlying storage driver, as this offers consistency irrespective of the storage driver being used.

Hrm, yeah, I’m not sure what the correct behaviour should be there. Maybe open an issue here to discuss? Issues · lxc/incus · GitHub

I’m not sure either about “lxc file push” on a readonly root filesystem. I may form an opinion once I start using such filesystems.

My main goal is to prevent changes to the container root filesystem from container processes, including my own shell sessions. I sometimes make a configuration change to a container and forget to make sure that this change is applied again when I rebuild the container. A readonly root will prevent me from changing root filesystem files and will also give me peace of mind that I can rebuild a container safely, without losing any files.