took very long copying and refreshing 13,2 G instance from source to destination starting 13:24 and ending 19:25. This was the first time to destination.
lxc copy bin00:bryssel bryssel --refresh
shortly after I issued ‘ps fauxww’ to follow process output on both source and destination hosts and noticed it’s doing zfs send/receive 13:24-17:xx for the last snapshot and zfs send/receive 17:xx - 19:XX for the older snapshot (see below).
it did not took that much time syncing the container (didn’t follow the process tree anymore could be rsync to remote or zfs send or both)
End results:
bin00:~$ lxc info bryssel
Name: bryssel
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/10/09 11:45 UTC
Status: Stopped
Type: container
Profiles: default, lanprofile
Snapshots:
bryssel_20201021 (taken at 2020/10/21 09:32 UTC) (stateless)
bryssel_20201228 (taken at 2020/12/28 18:20 UTC) (stateless)
server:~$ lxc info bryssel
Name: bryssel
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/01/04 11:24 UTC
Status: Stopped
Type: container
Profiles: default, lanprofile
Snapshots:
bryssel_20201021 (taken at 2020/10/21 09:32 UTC) (stateless)
bryssel_20201228 (taken at 2020/12/28 18:20 UTC) (stateless)
bin00:~$ lxc storage show lxd-pool
config:
source: pool1
volatile.initial_source: pool1
zfs.pool_name: pool1
description: ""
name: lxd-pool
driver: zfs
used_by:
- /1.0/instances/advanced-toad
- /1.0/instances/bryssel
- /1.0/instances/bryssel/bryssel_20201021
- /1.0/instances/bryssel/bryssel_20201228
- /1.0/instances/funny-reindeer
- /1.0/instances/funny-reindeer/funny-reindeer_20200911
- /1.0/profiles/default
status: Created
locations:
- none
server:~$ lxc storage show lxd-pool
config:
source: lxd-pool
volatile.initial_source: lxd-pool
zfs.pool_name: lxd-pool
description: ""
name: lxd-pool
driver: zfs
used_by:
- /1.0/images/95c0e536d361eb5ac953ad343e0342c2f615e4aea714ca8a64126a228b809cae
- /1.0/instances/bryssel
- /1.0/instances/thankful-skunk
- /1.0/profiles/default
status: Created
locations:
- none
I was concerned about the source lxd storage showing snapshots there (different from what I see in destination and other servers running lxd). I did
bin00:~$ sudo ls -ltr /var/snap/lxd/common/lxd/snapshots
lrwxrwxrwx 1 root root 71 joulu 28 20:09 funny-reindeer -> /var/lib/lxd/storage-pools/lxd-pool/containers-snapshots/funny-reindeer
lrwxrwxrwx 1 root root 64 joulu 28 20:09 bryssel -> /var/lib/lxd/storage-pools/lxd-pool/containers-snapshots/bryssel
server:~$ sudo ls -ltr /var/snap/lxd/common/lxd/snapshots
total 4
lrwxrwxrwx 1 root root 76 Jan 4 19:25 bryssel -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers-snapshots/bryssel
bin00:~$ lxc list
+----------------+---------+----------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------------+---------+----------------------+------+-----------+-----------+
| advanced-toad | STOPPED | | | CONTAINER | 0 |
+----------------+---------+----------------------+------+-----------+-----------+
| bryssel | STOPPED | | | CONTAINER | 2 |
+----------------+---------+----------------------+------+-----------+-----------+
| funny-reindeer | STOPPED | | | CONTAINER | 1 |
+----------------+---------+----------------------+------+-----------+-----------+
bin00:~$ lxc storage list
+----------+-------------+--------+--------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+----------+-------------+--------+--------+---------+
| lxd-pool | | zfs | pool1 | 9 |
+----------+-------------+--------+--------+---------+
bin00:~$ sudo ls -lh /var/snap/lxd/common/lxd/containers
total 16K
lrwxrwxrwx 1 root root 72 joulu 28 20:09 advanced-toad -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/advanced-toad
lrwxrwxrwx 1 root root 66 joulu 28 20:09 bryssel -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/bryssel
lrwxrwxrwx 1 root root 73 joulu 28 20:09 funny-reindeer -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/funny-reindeer
bin00:~$ sudo ls -lh /var/lib/lxd/containers
ls: cannot access '/var/lib/lxd/containers': No such file or directory
server:~$ sudo ls -lh /var/snap/lxd/common/lxd/containers
lrwxrwxrwx 1 root root 66 Jan 4 19:25 bryssel -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/bryssel
lrwxrwxrwx 1 root root 73 Dec 28 13:47 thankful-skunk -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/thankful-skunk
bin00:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool/
total 28K
drwx--x--x 6 root root 4,0K marra 24 09:29 containers
drwx------ 5 root root 4,0K joulu 28 15:00 containers-snapshots
drwx--x--x 2 root root 4,0K joulu 2 2019 custom
drwx--x--x 2 root root 4,0K joulu 29 01:54 custom-snapshots
drwx------ 2 root root 4,0K touko 21 2020 images
drwx--x--x 2 root root 4,0K joulu 29 01:54 virtual-machines
drwx--x--x 2 root root 4,0K joulu 29 01:54 virtual-machines-snapshots
server:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool
total 28K
drwx--x--x 5 root root 4.0K Jan 4 19:25 containers
drwx--x--x 3 root root 4.0K Jan 4 13:24 containers-snapshots
drwx--x--x 2 root root 4.0K Dec 28 13:44 custom
drwx--x--x 2 root root 4.0K Dec 28 13:44 custom-snapshots
drwx--x--x 3 root root 4.0K Dec 28 13:46 images
drwx--x--x 2 root root 4.0K Dec 28 13:44 virtual-machines
drwx--x--x 2 root root 4.0K Dec 28 13:44 virtual-machines-snapshots
bin00:~$ sudo ls -lh /var/lib/lxd/storage-pools/
ls: cannot access '/var/lib/lxd/storage-pools/': No such file or directory
bin00:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers
total 16K
drwx--x--x 2 root root 4,0K joulu 2 2019 advanced-toad
d--x------ 2 root root 4,0K loka 9 14:45 bryssel
drwx--x--x 2 root root 4,0K joulu 2 2019 funny-reindeer
server:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/
total 12K
d--x------ 2 root root 4.0K Jan 4 19:25 bryssel
d--x------ 2 root root 4.0K Dec 28 13:47 thankful-skunk
bin00:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers-snapshots/bryssel
total 8,0K
d--x------ 2 root root 4,0K loka 21 12:32 bryssel_20201021
d--x------ 2 root root 4,0K joulu 28 20:20 bryssel_20201228
server:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers-snapshots/bryssel
total 0
Why Is it empty in the destination after lxc copy ? Is it because of old container snapshot symlinks pointing to old location ?
My issue is similar as https://github.com/lxc/lxd/issues/5262
I tested taking new snapshots on the source server:
bin00:~$ lxc snapshot bryssel bryssel_20210104
bin00:~$ sudo ls -lh /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers-snapshots/bryssel
total 12K
d--x------ 2 root root 4,0K loka 21 12:32 bryssel_20201021
d--x------ 2 root root 4,0K joulu 28 20:20 bryssel_20201228
d--x------ 2 root root 4,0K tammi 4 22:56 bryssel_20210104
bin00:~$ sudo ls -ltr /var/snap/lxd/common/lxd/snapshots
lrwxrwxrwx 1 root root 71 joulu 28 20:09 funny-reindeer -> /var/lib/lxd/storage-pools/lxd-pool/containers-snapshots/funny-reindeer
lrwxrwxrwx 1 root root 76 tammi 4 22:56 bryssel -> /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers-snapshots/bryssel
bin00:~$ lxc info bryssel
Name: bryssel
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/10/09 11:45 UTC
Status: Stopped
Type: container
Profiles: default, lanprofile
Snapshots:
bryssel_20201021 (taken at 2020/10/21 09:32 UTC) (stateless)
bryssel_20201228 (taken at 2020/12/28 18:20 UTC) (stateless)
bryssel_20210104 (taken at 2021/01/04 20:56 UTC) (stateless)
bin00:~$
the new snapshot doesn’t show in the storage:
bin00:~$ lxc storage show lxd-pool
config:
source: pool1
volatile.initial_source: pool1
zfs.pool_name: pool1
description: ""
name: lxd-pool
driver: zfs
used_by:
- /1.0/instances/advanced-toad
- /1.0/instances/bryssel
- /1.0/instances/bryssel/bryssel_20201021
- /1.0/instances/bryssel/bryssel_20201228
- /1.0/instances/funny-reindeer
- /1.0/instances/funny-reindeer/funny-reindeer_20200911
- /1.0/profiles/default
status: Created
locations:
- none
bin00:~$
Here the snapshot is on the list
bin00:~$ lxc storage list
+----------+-------------+--------+--------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+----------+-------------+--------+--------+---------+
| lxd-pool | | zfs | pool1 | 9 |
+----------+-------------+--------+--------+---------+
bin00:~$ lxc storage volume list lxd-pool
+----------------------+----------------------------------------+-------------+--------------+---------+
| TYPE | NAME | DESCRIPTION | CONTENT TYPE | USED BY |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container | advanced-toad | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container | bryssel | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container (snapshot) | bryssel/bryssel_20201021 | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container (snapshot) | bryssel/bryssel_20201228 | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container (snapshot) | bryssel/bryssel_20210104 | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container | funny-reindeer | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
| container (snapshot) | funny-reindeer/funny-reindeer_20200911 | | filesystem | 1 |
+----------------------+----------------------------------------+-------------+--------------+---------+
bin00:~$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
pool1/containers/bryssel@snapshot-bryssel_20201021 1,55G - 11,1G -
pool1/containers/bryssel@snapshot-bryssel_20201228 26,6M - 11,6G -
pool1/containers/bryssel@snapshot-bryssel_20210104 120K - 11,6G -
pool1/containers/funny-reindeer@snapshot-funny-reindeer_20200911 229M - 9,77G -
pool1/deleted/images/028d045b1cfcfc8a69cc68674557bd86e015c0ba4bb5c3d6851043f785963728@readonly 0B - 660M -
bin00:~$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool1 44,5G 494G 24K none
pool1/containers 43,9G 494G 24K none
pool1/containers/advanced-toad 294M 494G 924M /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/advanced-toad
pool1/containers/bryssel 13,2G 494G 11,6G /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/bryssel
pool1/containers/funny-reindeer 9,81G 494G 9,84G /var/snap/lxd/common/lxd/storage-pools/lxd-pool/containers/funny-reindeer
pool1/custom 24K 494G 24K none
pool1/deleted 660M 494G 24K none
pool1/deleted/containers 24K 494G 24K none
pool1/deleted/custom 24K 494G 24K none
pool1/deleted/images 660M 494G 24K none
pool1/deleted/images/028d045b1cfcfc8a69cc68674557bd86e015c0ba4bb5c3d6851043f785963728 660M 494G 660M none
pool1/deleted/virtual-machines 24K 494G 24K none
pool1/images 24K 494G 24K none
pool1/snapshots 96K 494G 24K none
pool1/snapshots/bryssel 24K 494G 24K none
pool1/snapshots/funny-reindeer 24K 494G 24K none
pool1/virtual-machines 24K 494G 24K none
server:~$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
lxd-pool/containers/bryssel@snapshot-bryssel_20201021 1.69G - 11.7G -
lxd-pool/containers/bryssel@snapshot-bryssel_20201228 41.6M - 12.2G -
lxd-pool/containers/thankful-skunk@snapshot-snap0 84K - 973M -
lxd-pool/images/95c0e536d361eb5ac953ad343e0342c2f615e4aea714ca8a64126a228b809cae@readonly 8K - 787M -
server:~$
Then I found this:
I seem to have the same issue with the old/original snapshots turned in some point to storage volumes:
bin00:~$ lxd sql global .dump | grep storage_volumes
CREATE TABLE "storage_volumes" (
INSERT INTO "storage_volumes" VALUES(1,'advanced-toad',1,1,0,'',1,0);
INSERT INTO "storage_volumes" VALUES(3,'funny-reindeer',1,1,0,'',1,0);
INSERT INTO "storage_volumes" VALUES(6,'funny-reindeer/funny-reindeer_20200911',1,1,0,'',1,0);
INSERT INTO "storage_volumes" VALUES(11,'bryssel',1,1,0,'',1,0);
INSERT INTO "storage_volumes" VALUES(12,'bryssel/bryssel_20201021',1,1,0,'',1,0);
INSERT INTO "storage_volumes" VALUES(15,'bryssel/bryssel_20201228',1,1,0,'',1,0);
CREATE TABLE storage_volumes_backups (
FOREIGN KEY (storage_volume_id) REFERENCES "storage_volumes" (id) ON DELETE CASCADE,
CREATE TABLE storage_volumes_config (
FOREIGN KEY (storage_volume_id) REFERENCES storage_volumes (id) ON DELETE CASCADE
CREATE TABLE storage_volumes_snapshots (
FOREIGN KEY (storage_volume_id) REFERENCES storage_volumes (id) ON DELETE CASCADE
INSERT INTO storage_volumes_snapshots VALUES(16,11,'bryssel_20210104','',-62135596800);
CREATE TABLE storage_volumes_snapshots_config (
FOREIGN KEY (storage_volume_snapshot_id) REFERENCES storage_volumes_snapshots (id) ON DELETE CASCADE,
INSERT INTO sqlite_sequence VALUES('storage_volumes_config',0);
INSERT INTO sqlite_sequence VALUES('storage_volumes',16);
bin00:~$
Why the old snapshots are created there under storage volumes?