Could a snapshot of a zfs volume be used as an immutable read-only container image, being used for several containers and being atomically updated with another snapshot, with only /etc and /var being writable in the container?
Did the team ever think about something in this direction?
We have some work planned on a new lxc rebuild command/API which will effectively let you throw away the root filesystem of the container and replace it with a new one.
In theory, this feature combined with readonly=true set on the root device and applicable custom volumes to provide the persistent storage, should match what you want, except that it won’t be automatic and you’ll need to issue a lxc rebuild when you want to switch to a new base image.
This would be very space efficient as most backends are copy-on-write and no writes would occur.
Now the issue is that most distributions just won’t work at all with something like this, so you’d need some pretty customized images to handle this nicely. Ubuntu Core effectively has that, a number of other similar initiatives likely would do that too. Generally those will still have a read-only /etc and /var but just have one specific path (/writable on Ubuntu Core) which holds everything that’s writable.
1 Like
Will lxc rebuild let me ditch the original 600MB ubuntu image clone in zfs/deleted/images in normal writable containers after it was running for a few years and internally updated to a newer LTS version anyways?
lxc rebuild is effectively going to be the same as lxc delete followed by lxc launch but conserving all LXD configuration on the instance (attached devices, config options, …).
So yeah, your old image will then go away as it won’t have any container using it, but lxc rebuild will obviously have to be used with care as all user data in the container will be gone too, unless it’s coming from attached devices.
I have been using a rebuild mechanism for containers with attached disk devices and have a tool to manage launching and rebuilding such containers:
https://github.com/melato/lxdops
I have also discussed it here: Lxd-ops: Separating container OS from user data But the project page is probably clearer.
The idea is to avoid using the root filesystem in the container, put all application data in attached disk devices (in places like /var/opt, /opt, /etc/opt, /usr/local/bin, /home, /var/log), and do a bit of repeatable minimal configuration to make your applications work this way. This minimal configuration needs to be done on launch and on every rebuild. For me it is not enough to replace /etc/ and /var/. I consider these to be replaceable too.
For example, for nginx and apache2, my initial repeatable configuration adds configuration files that read application configuration files from /etc/opt/apache2/ or /etc/opt/nginx/, which are in the /etc/opt attached device. So replacing the root filesystem, including /etc, does not lose my web server configuration files. /var/log is on an attached filesystem, so log files are preserved after a rebuild.
2 Likes