Moving Instance with Docker - Operation Not Permitted

Incus
1

I’m trying to move an instance from one cluster member to another. This instance has security.nested set because it uses Docker containers. I’ve stopped the instance but when attempting to move it to another host it failed with lots of errors:

rsync: [receiver] rsync_xal_set: lsetxattr("/var/lib/incus/storage-pools/local/containers/media/rootfs/var/lib/docker/overlay2/98c4ff79f2202d45143c25b03ae9985a3c95846007b6db47594555313a135b36-init/diff/etc","trusted.overlay.origin") failed: Operation not permitted (1)

Any ideas? Thanks!

EDIT: Exporting the instance to a backup and importing it to the other cluster member worked without a problem. This is obviously not the preferred solution but it worked in a pinch.

2

What storage driver is used on the source and target servers?

3

Just the standard local directory storage on both. I’m also mounting an NFS volume from the host to the instance via this command:

incus config device add media-server photos disk source=/mnt/nas/photos path=/mnt/nas/photos

The NFS mount is on both hosts with the same directory location.

Could the NFS mount be the issue?

EDIT: The instance is also set as privileged so file permissions work correctly on the share. I’ll fix this later if I can figure out how to handle the shifting in a Debian instance.

Thanks for the assistance.

4

Okay, can you look for any sign of DENIED or similar log entries in dmesg on the target server when attempting such a transfer?

Basically the issue is around xattrs used by overlayfs inside of the container, those xattrs are set on a variety of directories and files within the Docker layers and need to be correctly re-applied on the target server. Rsync is reporting that it’s unable to do so, the question is whether that’s because of something Incus did or if that’s a kernel restriction.

I believe we may be a bit more lax on such errors when importing from a tarball, which would explain why that route worked, but it may also have resulted in that extended attribute going missing in the process, which could end up causing more issues (depending on exactly how critical that is to overlayfs operations).

5

This original issue is in my actual homelab “prod” environment. I spun up a test Debian 12 instance in my VM environment and installed Docker to test this out.

The target instance is showing some apparmor DENIED entries like the following:

audit: type=1400 audit(1707760440.829:55): apparmor="DENIED" operation="capable" profile="incus_rsync-d9a0f68a-bb0a-4b95-bd6e-e4a3b74c1ee1" pid=4627 comm="rsync" capability=21  capname="sys_admin"

I tested this by just creating a docker-compose.yml file for a basic nginx server like the following:

services:
  web:
    image: nginx
    volumes:
     - ./html:/var/www/html
    ports:
     - "8080:80"

I ran a docker-compose up -d to start the container. I then just did a docker-compose stop to stop the nginx container and then stopped the Incus instance. When I tried to perform a move it failed with the following errors (trimmed to only the first 3 lines):

Error: Instance move to destination failed: Error transferring instance data: Failed migration on target: Failed creating instance on target: Rsync receive failed: /var/lib/incus/storage-pools/local/containers/instance1/: [exit status 23 read |0: file already closed] (rsync: [receiver] rsync_xal_set: lsetxattr("/var/lib/incus/storage-pools/local/containers/instance1/rootfs/var/lib/docker/overlay2/527ca2546f4223b94a1c451b13647365b7cbd4f86928c174d2985fc61bd94421-init/diff","trusted.overlay.impure") failed: Operation not permitted (1)
rsync: [receiver] rsync_xal_set: lsetxattr("/var/lib/incus/storage-pools/local/containers/instance1/rootfs/var/lib/docker/overlay2/527ca2546f4223b94a1c451b13647365b7cbd4f86928c174d2985fc61bd94421-init/diff/dev","trusted.overlay.origin") failed: Operation not permitted (1)
rsync: [receiver] rsync_xal_set: lsetxattr("/var/lib/incus/storage-pools/local/containers/instance1/rootfs/var/lib/docker/overlay2/527ca2546f4223b94a1c451b13647365b7cbd4f86928c174d2985fc61bd94421-init/diff/etc","trusted.overlay.origin") failed: Operation not permitted (1)
6

Just quick update on this. I’ve been testing this more and I decided to try this with a different image. I launched an instance with alpine/3.19 instead of debian/12. I installed docker under Alpine and tried this same test with an nginx image and I had no problem moving the instance from one host to another.

I’m not sure why the debian/12 instance throws errors when moving the Docker overlay but apparently alpine/3.19 doesn’t have this same problem.

I prefer a basic Debian distro for my instances but if I can’t move them perhaps I’ll look for another minimal image to use for running Docker containers (not a big fan of Alpine+apk but if it works I’ll use it instead).

7

I spoke too soon. Running Alpine does not fix the problem.

I migrated my Debian instances over to Alpine thinking it would fix this move issue. I need to upgrade the ram on one of my hosts this weekend so I need to move instances off that host. I just tried moving one of them and I’m getting the failed: Operation not permitted (1) error again.

Quite frustrating. Ugh. I guess I’ll just stop the instances running on that host and shut it down and hope the ram upgrade doesn’t take long.

EDIT:

Many many thanks to this post from last year! This issue has to do with the Docker storage driver. That poster was moving from a host system with btrfs to zfs which I’m not doing. AFAIK both my systems should be using the same underlying filesystem type. The hosts are basic Debian 12 installs.

I followed the directions there and changed the Docker storage driver to vfs and started up all the containers. I then stopped the instance and successfully performed the move!

Here’s hoping this workaround continues to work.

EDIT 2: vfs is probably not the best option for a storage driver. So far it seems really slow and takes ages to move an instance.

The Docker storage driver I was using before that was failing is:

 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true

Both host systems are using ext4.