First, the problem. I have 4 Servers with hundreds of containers. That are in production. I am not playing around here. I need to be able to backup my container without being an issue or causing problems.
Well, using LXC export does the following. On small containers under 10GB it seem to work ok, though it takes way too long over 15 minutes in some cases. I have all the small containers in one server and they seem to work best.
The problem is the bigger containers over 50G, not only do they take over a couple hour of hours but many never finish. The Export command seems to eat up local storage in massive amounts, and Zpool storage also. But the problem is local storage, worse after it finishes or doesn’t finish because it crashes it does not remove temp files somewhere.
If I start out with 200G free on local and 300g free on Zpool, after backup attempt I am left with 50G local and 150 free in Zpool. Basically I am in a critical situation because the only container on this server is 50G and it needs room to grow and I have found that without free space they crash.
On one server I was able to free up 100g by going into /var/snap/lxd/comon/lxd/backups and deleting a file from an aborted backup. It seems Lxc export unpacks the container into a local directory to then zip into a file. A very very time and storage intensive solution. There has to be a better way. Perhaps lxc export should be zipping container in zpool before it exports.
BTW, I am running these on massive dual cpu servers with plenty of ram and ssd drives, so performance is not a problem for any operation beside LXC export.
In a production environment, a system must not only run, but you have to back it up regularly. And yes you can do Raids, which I have. But you have to have physical backups that work. It is unacceptable that you have to take down a container (a client) to back it up and worse that it take hours. This is something that you have to be able to do every night hopefully, reliably and automatically without fear.
So far I have tried three methods.
- Stopping Container and doing Lxc export to a hard drive not the same as zpool drive or local. ssd. Does not work well with large containers.
- Stopping container and doing a lxc copy to another servver, takes a while but works, but problem is that if cluster crashes, I still dont have anything. Container is still unusable.
- Mounting container zpool directly and copy files directly to hard drive via Rsync. This is what I was doing before lxc export, but I am afraid it might damage container or other issues, with stability. But right now seems best way.
Backing up a container (no matter its size) should not be a big deal. and it is something that should be done regularly.
Is there a better way?