Backup containers from host

1

Hello,

I have a lxd v4.3 installed on Debian Buster with one container. I am using ZFS as storage backend. I use R1Soft to backup this server. This solution uses an agent installed on the server which searchs for modified blocks in the filesystem and copies them. I keep some archived copies to be able to recover old files. The problem is that the backup agent cannot access the container’s files.

These are the zfs datasets:

NAME                                                                                      USED  AVAIL  REFER  MOUNTPOINT
default                                                                                   488M  18.8G    24K  none
default/containers                                                                        128M  18.8G    24K  none
default/containers/c1                                                         128M  18.8G   303M  /var/snap/lxd/common/lxd/storage-pools/default/containers/c1
default/custom                                                                             24K  18.8G    24K  none
default/deleted                                                                           179M  18.8G    24K  none
default/deleted/containers                                                                 24K  18.8G    24K  none
default/deleted/custom                                                                     24K  18.8G    24K  none
default/deleted/images                                                                    179M  18.8G    24K  none
default/deleted/images/fa279e61e307e561930cf2febcf5526cd0956e5e104a7bca7f02efd4898c0b2c   179M  18.8G   179M  /var/snap/lxd/common/lxd/storage-pools/default/images/fa279e61e307e561930cf2febcf5526cd0956e5e104a7bca7f02efd4898c0b2c
default/deleted/virtual-machines                                                           24K  18.8G    24K  none
default/images                                                                            179M  18.8G    24K  none
default/images/b96bf7781d54253035bfa2414288ddcf872319fc6fc49b97de529fafbcb45c31           179M  18.8G   179M  /var/snap/lxd/common/lxd/storage-pools/default/images/b96bf7781d54253035bfa2414288ddcf872319fc6fc49b97de529fafbcb45c31
default/virtual-machines                                                                   24K  18.8G    24K  none

As you can see, the container’s files are mounted on /var/snap/lxd/common/lxd/storage-pools/default/containers/c1 but its namespace is different and the agent cannot see them.

Do you know how could I make visible the container’s files to the backup agent?

Kind regards,

2

You could have it look at /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd but another option would be to treat the containers as their own systems and run the agent inside them.

3

Hi @stgraber,

I really appreciate your answer. I took a look at /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd as you suggested. Inside that directory there is a containers one with links pointing to each of the containers:

$ sudo ls -l /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/
total 64
drwx------ 2 root root 4096 Jul  1 20:32 backups
drwx------ 3 root root 4096 Jul  1 20:32 cache
drwx--x--x 2 root root 4096 Jul  2 11:30 containers
drwx------ 4 root root 4096 Jul  3 05:51 database
drwx--x--x 3 root root 4096 Jul  3 05:51 devices
drwxr-xr-x 2 root root   60 Jul  3 05:51 devlxd
drwx------ 2 root root 4096 Jul  1 20:32 disks
drwx------ 2 root root 4096 Jul  5 11:51 images
drwx------ 3 root root 4096 Jul  4 05:51 logs
drwx--x--x 3 root root 4096 Jul  1 18:11 networks
drwx------ 4 root root 4096 Jul  1 18:36 security
-rw-r--r-- 1 root root  761 Jun 30 09:51 server.crt
-rw------- 1 root root  288 Jun 30 09:51 server.key
lrwxrwxrwx 1 root root   40 Jun 30 09:51 shmounts -> /var/snap/lxd/common/shmounts/containers
drwx------ 2 root root 4096 Jul  1 20:32 snapshots
drwx--x--x 3 root root 4096 Jul  1 18:11 storage-pools
srw-rw---- 1 root lxd     0 Jul  3 05:51 unix.socket
drwx--x--x 2 root root 4096 Jul  1 20:32 virtual-machines
drwx------ 2 root root 4096 Jul  1 20:32 virtual-machines-snapshots

$ sudo ls -l /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/containers/
total 4
lrwxrwxrwx 1 root root 72 Jul  1 18:36 c1-> /var/snap/lxd/common/lxd/storage-pools/default/containers/c1

But the problem is that the namespace of the mounted filesystems on /var/snap/lxd/common/lxd/storage-pools/default/containers/c1 prevents processes from the host from viewing that files.

Sure, I could install the backup agent in the containers, but this way I will have to use one license for each of the containers while doing the backup from the host will require just one license for all the containers in that host.

I have learned I can manually mount the ZFS datasets in the host:

$ sudo zfs list
NAME                                                                                      USED  AVAIL  REFER  MOUNTPOINT
default                                                                                   488M  18.8G    24K  none
default/containers                                                                        128M  18.8G    24K  none
default/containers/c1                                                                   128M  18.8G   304M  /var/snap/lxd/common/lxd/storage-pools/default/containers/c1
../..

$ sudo zfs get canmount
NAME                             PROPERTY  VALUE     SOURCE
default                          canmount  on        default
default/containers               canmount  on        default
default/containers/c1            canmount  noauto    local
../..

$ sudo zfs mount default/containers/c1
$ sudo mount
../..
$ default/containers/c1 on /var/snap/lxd/common/lxd/storage-pools/default/containers/c1 type zfs (rw,xattr,posixacl)
../..

So now the ZFS dataset default/containers/c1 is mounted twice, one using the container namespace and another one using the default namespace, which is accesible from the host:

$ sudo ls -l /var/snap/lxd/common/lxd/storage-pools/default/containers/c1
total 6
-r--------  1 root    root    3382 Jul  2 17:41 backup.yaml
-rw-r--r--  1 root    root    1283 Jul  1 07:34 metadata.yaml
drwxr-xr-x 21 1000000 1000000   21 Jul  1 07:34 rootfs
drwxr-xr-x  2 root    root       8 Jul  1 07:34 templates

I am not sure though if mounting the filesystem twice can be dangerous or if there are some drawbacks. Probably it would be safer mounting the container’s dataset as read-only from the host. After all, to back up the files I just need to be able to read them.

Kind regards,

4

Mounting it twice will cause you issues with ZFS… it’s not dangerous but ZFS is very much not mount namespace aware and gets very confused when this happens.

To access the actual data, look under /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools

5

Hello @stgraber,

Thank you. I can see the container’s files under /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/c1/rootfs. Unfortunately, the backup agent does not see that files. I will have to install the agent in both the host and the containers.

Regards,