Snapshot fails due to wrong storage path

LXD
1

Command: lxc snapshot dse-t76548-dc1-1s test

Error: Create instance snapshot: Failed to run: rsync -a -HA --sparse --devices --delete --checksum --numeric-ids --xattrs --filter=-x security.selinux -q /var/snap/lxd/common/lxd/storage-pools/3tb/containers/dse-t76548-dc1-1s/ /var/snap/lxd/common/lxd/storage-pools/3tb/containers-snapshots/dse-t76548-dc1-1s/test: rsync: rsync_xal_set: lremovexattr("/var/snap/lxd/common/lxd/storage-pools/3tb/containers-snapshots/dse-t76548-dc1-1s/test/rootfs/var/log/journal/.system.journal.hcz3P1",“trusted.SGI_ACL_FILE”) failed: No data available (61) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]

Observation: The rsync command is using the source path of /var/snap/lxd/common/lxd/storage-pools/3tb which does not exist. The 3tb storage pool directory is: /data/lxd-storage. If I run the rsync command using the correct source path, the command works as expected. My hypothesis is that snapshots is incorrectly assembling the source path and the rsync command fails.

root@big-lab]# lxc storage list
+---------+--------+------------------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |                     SOURCE                     | DESCRIPTION | USED BY |  STATE  |
+---------+--------+------------------------------------------------+-------------+---------+---------+
| 3tb     | dir    | /data/lxd-storage                              |             | 11      | CREATED |
+---------+--------+------------------------------------------------+-------------+---------+---------+
| default | dir    | /var/snap/lxd/common/lxd/storage-pools/default |             | 0       | CREATED |
+---------+--------+------------------------------------------------+-------------+---------+---------+

When I run: lxd sql global .dump | egrep storage_pools | egrep "3tb|default"
It looks like the path from above the storage list did not make it to the storage_pools_config table (but default did).

INSERT INTO storage_pools VALUES(2,'default','dir','',1);
INSERT INTO storage_pools VALUES(4,'3tb','dir','3TB Drive',1);
INSERT INTO storage_pools_config VALUES(2,2,1,'source','/var/snap/lxd/common/lxd/storage-pools/default');

LXC/LXD versions (error in both versions: 5.2 & 5.3)

[root@big-lab]# cat /etc/redhat-release
Rocky Linux release 8.6 (Green Obsidian)

[root@big-lab]# rsync --version
rsync version 3.1.3 protocol version 31

/data is an xfs file system

2

Hi,

So I think there is a bit of a misunderstanding here. The path being used is not incorrect.
What happens is that the specified source path /data/lxd-storage is bind mounted (in the snap package’s mount namespace) to LXD’s /var/snap/lxd/common/lxd/storage-pools/3tb path. So the rsync behaviour you are seeing is correct.

The question that needs answering is why is rsync failing to transfer some of the file/attrs of your instance.

3

Perhaps the underlying filesystem does not support extended attributes or something like selinux is blocking it? Are you able to remove that file and try the snapshot again?

4

Sounds similar to this:

Which ended up having a fix for selinux:

But in this case it looks like that file has a different extended attribute trusted.SGI_ACL_FILE which is related to XFS. It would be interesting to see if this problem resolves itself by disabling selinux, at least then you’d know its related to selinux.

5

Thank you for the prompt reply.

I disabled selinux in my home lab:

[ken@big-lab ~]$ getenforce
Disabled

[ken@big-lab ~]$ cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

I put single quotes around --filter=-x security.selinux and the rsync command worked as expected with the non-snap file path.

If I understand correctly, the /var/snap/lxd/common/lxd/storage-pools/3tb directory should be “linked” to the 3tb storage pool? How can we test that the bind mount works as expected? I looked at the snap mount points in /etc/systemd/system, but that was not informative.

6

So does lxc snapshot work with SELINUX disabled (as it sounded like you were running the rsync command manually in your last reply).

You should be able to see the bind mount by running:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- mount

Also if it wasn’t working you would get a totally different error than the rsync one you’re getting.

7

It appears the container in question is the only container which errors when creating a snapshot. I created snapshots on every other container without issue. Snapshotting this container was for convenience and not critical.

Thank you very much for the prompt replies and pointers. I agree that the rsync error is a symptom of something other than the storage path being wrong.

1 Like