Rsync from btrfs to zfs

LXD
,
1

Hi,

Seems like lxc copy/move gives problems when going from a btrfs pool to a zfs pool:

rsync: [receiver] rsync_xal_set: lsetxattr("/var/snap/lxd/common/lxd/storage-pools/lxdzfs/containers/lxd-move-of-2a2f61f2-40dd-4836-a1cc-313b5d76a23c/rootfs/var/snap/docker/common/var-lib-docker/overlay2/f8af683c002aee43ac7ac2a0ee2fb0e17cae2c422e262bcce708d9591543647a/work/work","trusted.overlay.opaque") failed: Operation not permitted (1)
rsync: [receiver] rsync_xal_set: lsetxattr("/var/snap/lxd/common/lxd/storage-pools/lxdzfs/containers/lxd-move-of-2a2f61f2-40dd-4836-a1cc-313b5d76a23c/rootfs/var/snap/docker/common/var-lib-docker/overlay2/fb46bb67489c96afafbcb8fc21febcb7976edee34bd741a552857f76b0fc3a7e/diff/docker-entrypoint-initdb.d","trusted.overlay.opaque") failed: Operation not permitted (1)
rsync: [receiver] rsync_xal_set: lsetxattr("/var/snap/lxd/common/lxd/storage-pools/lxdzfs/containers/lxd-move-of-2a2f61f2-40dd-4836-a1cc-313b5d76a23c/rootfs/var/snap/docker/common/var-lib-docker/overlay2/fb46bb67489c96afafbcb8fc21febcb7976edee34bd741a552857f76b0fc3a7e/work/work","trusted.overlay.opaque") failed: Operation not permitted (1)

Is user extended attributes supported in zfs?

LXD Version: lxd 5.14-7072c7b 24918 latest/stable canonicalâś“ -

Quinn

2

Hi,

Let me try and explain what is happening in more detail here.
I have two servers which both have the same lxd version installed.
Server3 uses btrfs pools
Server2 uses zfs pools
If I try to do a “lxc copy” or a “lxc move” of a container from server3 to server2, I then get the error messages in the first post and the operation fails. Let me fill in the commands here from server3:

lxc snapshot inventree01 snap0
lxc copy inventree01/snap0 server2:inventree01-2023-06-09

Any idea’s what is causing the rsync error?

Quinn

3

ok, I went through the syslogs on the destination server and it looks like apparmor is getting in the way:

I see these lines a lot:

Jun 9 19:46:23 server2 kernel: [611058.139799] audit: type=1400 audit(1686332783.177:93): apparmor=“DENIED” operation=“capable” profile="lxd_rsync-4b2f2468-20df-4243-a440-72413c4e4
eb3" pid=1835031 comm=“rsync” capability=21 capname="sys_admin"

And also these lines:

Jun 9 19:48:15 server2 lxd.daemon[6965]: time=“2023-06-09T19:48:15+02:00” level=error msg=“Failed migration on target” clusterMoveSourceName= err="Failed creating instance on targe
t: Rsync receive failed: /var/snap/lxd/common/lxd/storage-pools/lxdzfs/containers/inventree01-2023-06-09/: [exit status 23] (rsync: [receiver] rsync_xal_set: lsetxattr("/var/snap/l
xd/common/lxd/storage-pools/lxdzfs/containers/inventree01-2023-06-09/rootfs/var/snap/docker/common/var-lib-docker/overlay2/02350fe1cb4ae64b3b254f5693359ed70330bc301ac595d9c0d6927c52
9a8cdc/diff",“trusted.overlay.impure”) failed: Operation not permitted (1)

Quinn

4

Hi,

What LXD versions are you running here?

5

Hi,

LXD version 5.14-7072c7b rev 24918 on both servers.

Quinn

6

I did some testing.

It looks like it might be docker related. If I spin up a container on a btrfs pool, I can install docker without problems but doing the same on a zfs pool then docker installation fails. The container I was trying to move in the first post had docker running on it. Will try and research some more.

Quinn

7

I found a reference to zfs and docker saying docker will not run well on a default zfs pool.

https://ubuntu.com/tutorials/how-to-run-docker-inside-lxd-containers#2-create-lxd-container

Quinn

8

ok, I did some testing with docker by doing this:

  • On the first machine, I created a btrfs pool as default and launched a container in it with docker installed and running a simple docker container.
  • On the second machine, I created a zfs pool as default
  • On the first machine, I shutdown the container containing a working docker environment and then I did a “lxc copy” over to the second machine - as expected, the container containing docker stopped working but, I did not see the “rsync_xal_set” errors that I saw in the first post.

So yes, docker cannot be moved from one filesystem type to another, but this was not the cause of the “rsync_xal_set” errors seen in the first post.

Quinn

9

Btw, docker does run on both filesystem types as long as “security.nesting” is set to “true” for the container.

I think docker’s overlay system uses whatever is detected underneath during install. If it detects btrfs, is uses it’s btrfs driver for all docker containers.

Quinn

10

Hi,

After extensive testing, it is a docker related issue - mainly to do with docker’s storage drivers.
When installed for the first time, it automatically checks which storage driver to use together with the underlying file system. If your pool is btrfs, it will either use it’s btrfs driver together with subvolumes or the overlay2 driver - both work fine if the host file system is btrfs. If it is zfs, it will use the “vfs” storage driver.

But, if you have already run docker once in the container, then the storage driver will not change after a move. Also rsync cannot copy/move these directories used with either docker’s btrfs driver or the overlay2 driver. If you want to copy/move a lxd container that has docker running with a storage driver other than “vfs”, then you need to do the following first before the lxc copy/move operation:

  • lxc shell into the container
  • stop the docker system (snap: snap stop docker)
  • delete the contents of /var/lib/docker/* (snap: /var/snap/docker/common/var-lib-docker/*)
  • stop the lxc container

Then the lxc copy/move operation will succeed.

In order to force docker to always use the “vfs” driver, do this:

  • stop the docker service
  • make sure /var/lib/docker/* is empty
  • add the following to the daemon.json config file:
    {
    “storage-driver”: “vfs”
    ]

Now docker will always use vfs as the driver no matter which file system is used underneath and you can now copy/move the container without doing any changes to docker beforehand.

BIG NOTE: If you use snapshots, then all the snapshots will also need to be changed unless you use “–instance-only” option during the copy/move operation.

Quinn

2 Likes