I’m wondering what the best approach is to regularly backup specific files from a container.
Intuitively, I would do something like a “container mount” so that I can directly backup the files needed. I’m not sure how to do this, though.
lxc file mount does not seem to be the solution, or at least I don’t know how to run it in a script. The same goes for mounting the lxd namespace using nsenter. In both cases, a separate shell is needed to access the files.
So, I guess I could run lxc file pull as an alternative but this can generate quite some overhead because now every file will be copied twice, once for the pull, then again for the actual backup.
Seeing as how I use lvm for my storage pool, the better option would probably be to just create a snapshot and then mount the snapshot volume to access the files this way. But that wouldn’t be a universal LXD solution.
Another way I just thought of is to mount a host volume/directory in the container, then copy over the contents to be backed up and then run the backup routine from the host. But it still means I have the additional step of copying the data once more than really needed and having to clean up after it.
So, I guess I could mount the directory in such a way that the data that I want to backup is actually held on the host volume all the time. This way, I don’t have the additional step of copying data and clean-up. But since the data is already present in the potential mount points, some data shifting will be needed, making this a bit more complex than I hoped it would be.
Not sure I understand the question but I’ll elaborate.
So I have a container that is fulfilling some function, for example it’s a CRM or an ERP. There is a data part, such as the database and a file archive as well as the applications part.
The applications part is pretty much static so there is no real need to back it up every day. The data part changes all the time, though, so I want to make sure I have it backed up several times per day.
I could do this from within the container, of course, but it would be much cleaner and easier if I can run the backup from the host. But running lxc export multiple times per day is just not efficient, and deduplication of the backup becomes more difficult, I believe, with a compressed file.
I hope this makes sense. So, I’m just trying to figure out what more experienced people than me are doing to back up data within a container. And if you believe I should configure my containers differently in the first place, I’m all ears for advice.
I looked into the storage volume story yesterday but I’m not sure it solves my main issues.
My understanding is that a volume of the type file system can only be attached to one specific path in the container. If, for example, I have data in paths /var/x1 and /var/x2 on the container, I would have to work with two volumes. That will get messy very quickly. So, as an alternative, I guess I could work with a block volume and then, from within the container, mount folders on that volume to various paths within the system. The main downside here is that a block volume should only be mounted to one container, so I could not use the same volume to hold data for various containers at the same time.
I again have the problem that I cannot mount the volume on the host, meaning I can’t access the directory structure directly but instead, according to your link, should run an export first and then back up the exported archive. Yes, this helps me avoid copying unnecessary application files but I still have a copy process instead of simply having my backup software access the files directly. And because the export process creates a compressed archive, neither deduplication nor incremental backup will work properly, so, I assume the backup will take up more space than necessary.
Does that make sense or am I overlooking something?
LXD does have the concept of doing a --refresh copy of an instance to another LXD server too.
Combined with snapshots, this can make periodically taking an off-machine backup more efficient, as only the differences will be copied.
Yes I see. You could achieve the same effect by using a dir storage pool for each container’s custom volumes with a custom source=/cdata/c1 and then all of the custom volumes would be created inside that directory.
So, I changed my backup script, adding your lines to it, and it all works like a charm. And it’s perfect for my use case, so much cleaner and easier than all the other options we discussed. Thanks a lot for your support, @tomp
I realize one major downside of the solution using lxc file mount. Unless you’re shutting down the container before running the command (which works fine, by the way), you may end up accessing data that is currently in use, being changed.
When I used to work directly on a storage driver level, i.e. lvm, I was always working with snapshots so I could leave the instance running and still access every file safely. Quite convenient.
Seeing as how lxc file mount works on a shut down system, is there a way to get it to work on a snapshot? If not, is this a realistic feature request or would that be too complex?