Inspiration for keeping a generic container based on changes from a main container

LXC
1

Hi,

I have been using a container as a mini system for a while (LXC 1.0.6) and it’s been really nice.

I actually use a squashfs as root and I have a script that repacks it using the changes and fires up the container again but I separate my /home and in this case /var/mysql and in general just use the “dir” storage, but because it is squashfs it is really custom now.

(I never knew about all these stores lol).

So backup is easy: take the squashfs, tar the home, tar any config files, any mysql, and tar the changes to the rootfs, put it all in one big tar and upload it somewhere.

I then currently use rdiff to generate rdiffs against a previous backup if I want to create a differential one and it is all a bit self-concocted but it works nicely.

However I would like to add another container but the complexity of the Linux filesystem makes it a bit hard.

I would want to have a generic container that is very close to my ‘first’ ‘production’ container except for customizations only pertinent to that host.

  • I can create a generic rootfs with a generic /etc and a generic /var that only contains /var/lib/apt and /var/lib/dpkg and whatever else is essential to getting the system ‘booted’, which is not a lot,

  • but to keep this generic I have to have a pristine /var, but a pristine /var can only be had from a fresh installed system containing all the packages that the main container has.

  • at that point I can overlay the / and the /etc and the /var with that of the main container but I cannot actually integrate any changes from the main container anymore apart from repeating package installs on the generic container, and even repeating configuration changes on the generic container, unless I explicitly put /etc files into the generic container that I need to have there, but this is still a problem for /var.

  • so if I were to do installations on the main container I would need to track changes to /etc and /var and move that to the generic container (filesystem) but this is a problem for /etc files I have customized.

Thus, the only true way to keep those changes pristine is to do it in the generic container but this precludes easy customization that is supposed to be container-system-wide and do it on the main container at the same time. I do not want to work on the generic container, I just want it to keep the results.

I realize that the best way to do this is probably to have everything automated and to just execute a playbook on all containers but I prefer merging for now.

IF I repeat my changes on the generic container by way of some script, or a repeat of some audit file, that are automatic (apt in this case) then I would have to create a difference of the main and generic container, and keep the changes in an overlay for the main container, while moving changes that are meant to be generic into the ‘underlay’ (so that there is no longer a diff) at which point the changes disappear from the overlay and have become generic indeed.

But honestly config file changes are usually not that many and it wouldn’t even be hard to grab any pristine files from the packages directly.

But /var/lib can be a bigger problem.

But since there is no individual /usr, /sbin, /bin, /lib, the primary container will have a rw overlay for these things, and all containers will have a rw overlay for /etc, there is barely any individual stuff there. So at this point I am working from overlays anyways, or changesets.

2

So to elucidate…

  • If I am going to have a single underlying image for /, /etc and /var, I will have to create this diffset based on a fresh installation that I install all the same packages into. At this point it will be pristine and manual changes into the generic will be correct.

  • But if at any later date I want to merge stuff into the underlay the only way to do this is to also repeat changes (package additions etc) on the underlay, and then repeat the difference creation.

  • So I guess the only way to do this is to keep repeating changes. Merging into the underlying rootfs then never happens, all overlays stay the same and only get stuff taken out of that is already present in the underlay, what we do is reduce the overlays, not integrate stuff into the underlay.

  • Then, the underlay image can be one image for /usr etc., and for /etc and /var, as long as I can overlay together existing directories and not just devices (bind mount stuff together), which works just fine.

  • In fact, I can keep a single monolithic overlay as well for each container. But this doesn’t work because I want some directories to be ro and others rw, so you still have rw /, /etc and /var overlays for the main container, and rw /etc/ and /var overlays for the other containers.

  • On the other hand, if you keep this in the same filesystem tree, it will still be one overlay for each container.

  • I mean also that we do integrate into the underlying /etc and /var, but only manually, however sometimes this also changes files in /var.

3

I think that for this case you want to keep a base image and apply the changes you need on the fly with automation with Ansible or Puppet which will give you the easy customization that you want. Every time you do a variation in your main image, you will have to either update manually all the other images that derived from that image or create them from scratch + automation to create those images. The last way of doing it ensures that you have a repeatable image generation and environment so that you know that if your system goes bonkers, you can always repeat what you have done previously giving the fact that you have a base image + the automation.

4

Aye, as I mentioned the playbook part was as yet a bit too hard for me to accomplish because well, I don’t like Ansible so much and it is the best alternative available.

Or in other words, I am still collecting my config which is not such a huge task but without a real descriptive language or a good script library it is a bit annoying to do. My current (single) container does not have that much config and all my Apache config is static but gets automatically deployed into the container already, I set it up that way. I just removed part of manual config and moved it into the deployed config.

Ansible would definitely make it easier. It would be … well, let’s not go there.

I actually used Git outside of the container to save my changed config files, but the .gitignore syntax is very annoying to use.

But it’s funny, I can run “git status” and see e.g. all of the exim4 files that changed.

That’s also difficult to automate for me because of debconf.

A “git ls-files” shows me all config files in my repository.

That are inside the container.

But, the more complicated thing I imagined here…

Would allow me to use the same base image for space savings.

5

See at this point I am mostly wondering about how can I reuse the base image not just by cloning, but directly.

6

So my issue is mostly on how to save space cq. keep the same base image.

My main issue is that my current main image is rather full-fledged.

I can easily with a bit of coding substract a new base image from my current main image and turn the main image into an overlay of the new base image.

But the problem I haven’t solved is apt.

Like I wrote at first, once I have substraction code I can “move” stuff into the base image.

How? Install it into the base image too and then run the substraction code. That way identical files get deleted from the main image and its overlay size is reduced.

Then the mounting code can use the generic image + individual rw-system for basically everything.

Changes to base image can be recorded and turned into playbook/scripts if needed.

Cloning container then implies starting fresh with empty rw.