OK here we go, I’ve got closer, this is reproducible with all 15 of my containers.
It seems the error I was missing/dismissing is:
Error: Error transferring instance data: Got error reading migration source: websocket: close 1006 (abnormal closure): unexpected EOF
This appears to completely wipe out the container in the filesystem on the target but not the entry in the lxd database, thus triggering the lxc delete command to use the wrong path. [Edit I don’t think its actually wiping out the data just unmounting the BTRFS subvols from the filesystem]
Just to provide some detail…
Source Machine
⸾ snap list | grep lxd
lxd 5.0.0-b0287c1 22923 latest/stable canonical* -
⸾ lxc stop test
⸾ lxc copy test target:test --mode=relay --storage=default
Destination Machine
⸾ snap list | grep lxd
lxd 5.0.0-b0287c1 22923 latest/stable canonical* -
⸾ lxc storage list
+---------+--------+----------+-------------+---------+---------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY | STATE |
+---------+--------+----------+-------------+---------+---------+
| default | btrfs | /srv/lxd | | 16 | CREATED |
+---------+--------+----------+-------------+---------+---------+
⸾ lxc storage info default
info:
description: ""
driver: btrfs
name: default
space used: 4.17TiB
total space: 10.92TiB
used by:
instances:
- ..
- ..
- test
- ..
- ..
profiles:
- default
⸾ lxc info test
Name: test
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/04/27 19:04 BST
Snapshots:
+--------------------+----------------------+------------+----------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
+--------------------+----------------------+------------+----------+
| 2022-03-01-monthly | 2022/03/01 00:16 GMT | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-01-monthly | 2022/04/01 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-03-weekly | 2022/04/03 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-10-weekly | 2022/04/10 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-17-weekly | 2022/04/17 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-21-daily | 2022/04/21 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-22-daily | 2022/04/22 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-23-daily | 2022/04/23 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-24-daily | 2022/04/24 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-24-weekly | 2022/04/24 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-25-daily | 2022/04/25 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-26-daily | 2022/04/26 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-27-daily | 2022/04/27 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
⸾ ll /var/snap/lxd/common/lxd/storage-pools/default/
total 8.0K
drwx--x--x 2 root root 4.0K Mar 22 2021 ./
drwx--x--x 3 root root 4.0K Apr 13 09:18 ../
⸾ ll /srv/lxd/containers/test
total 44K
d--x------ 1 root root 78 Apr 27 21:19 ./
drwx--x--x 1 root root 180 Apr 27 21:19 ../
-r-------- 1 root root 38K Apr 27 20:50 backup.yaml
-rw-r--r-- 1 root root 1.1K Aug 4 2020 metadata.yaml
drwxr-xr-x 1 1000000 1000000 174 Aug 29 2020 rootfs/
drwxr-xr-x 1 root root 186 Aug 4 2020 templates/
Source Machine
⸾ lxc start test
⸾ lxc copy test target:test --mode=relay --storage=default --refresh
Error: Error transferring instance data: Got error reading migration source: websocket: close 1006 (abnormal closure): unexpected EOF
Destination Machine
⸾ ll /srv/lxd/containers/test
⸾ ls: cannot access '/srv/lxd/containers/test': No such file or directory
⸾ lxc info test
Name: test
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/04/27 20:54 BST
Snapshots:
+--------------------+----------------------+------------+----------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
+--------------------+----------------------+------------+----------+
| 2022-03-01-monthly | 2022/03/01 00:16 GMT | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-01-monthly | 2022/04/01 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-03-weekly | 2022/04/03 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-10-weekly | 2022/04/10 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-17-weekly | 2022/04/17 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-21-daily | 2022/04/21 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-22-daily | 2022/04/22 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-23-daily | 2022/04/23 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-24-daily | 2022/04/24 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-24-weekly | 2022/04/24 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-25-daily | 2022/04/25 00:17 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-26-daily | 2022/04/26 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
| 2022-04-27-daily | 2022/04/27 00:16 BST | | NO |
+--------------------+----------------------+------------+----------+
⸾ ll /srv/lxd/containers/test
⸾ ls: cannot access '/srv/lxd/containers/test': No such file or directory
⸾ lxc delete test
Error: Error deleting storage volume: Failed setting subvolume writable "/var/snap/lxd/common/lxd/storage-pools/default/containers/test": Failed to run: btrfs property set -ts /var/snap/lxd/common/lxd/storage-pools/default/containers/test ro false: ERROR: Could not get subvolume flags: Invalid argument
⸾ ll /srv/lxd/containers/test
total 4.0K
d--x------ 1 root root 22 Apr 27 21:29 ./
drwx--x--x 1 root root 180 Apr 27 21:29 ../
-r-------- 1 root root 2.6K Apr 27 21:30 backup.yaml
⸾ rm -fr /srv/lxd/containers/test
⸾ lxc delete test
@monstermunchkin it seems there are a few things going on here.
-
There is still an issue with btrfs --refresh which seems to not affect any new containers I make but affects remote copies of all 15 of my existing containers, perhaps related to this?
LXD init --preseed not working - Error: websocket: close 1006 (abnormal closure): unexpected EOF - #13 by stgraber -
There is an undesirable side effect the first time an attempt is made to delete a container which has no filesystem present (but lxc thinks it should be there) whereby lxc creates a folder in the correct storage pool location but then attempts to perform btrfs operations at the default path (likely because the metadata.yaml file is missing in the folder it just created) even if the container was never in the default location, there is no container currently in the default location and there’s not even a storage pool or btrfs filesystem at this location. I guess a recommendation would be to not create a folder in the path if its missing during delete operations, then it can just delete the container from the database as it does on the second attempt?
-
Its also possible to get the same issue with btrfs operations attempting to be performed on the default path (where there is no container, there never was a container and there’s not even a storage pool or btrfs filesystem) during a --refresh operation if the container has issues (I think maybe caused by point 1).
Hopefully this is helpful.
Sorry its taking so long to get decent info, the containers take ages to copy to my remote.