No space left on device while changing the BTRFS storage pool of an LXD container with docker

According to my tests, we should be able to see the effective disk usage (excluding BTRFS optimizations from snapshots) by doing:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- du -h /var/snap/lxd/common/lxd/storage-pools/default

This may show us where the problem is.

1 Like

Ok so actually I was wrong: it worked with zfs, but on the new storage bigstorage the space used is 50GB which is really high compared to the storage used by the moved container on the original default container.

lxc storage info bigstorage
info:
  description: ""
  driver: zfs
  name: bigstorage
  space used: 49.98GB
  total space: 96.74GB
used by:
  instances:
  - Mycontainer

I think the size usage your seeing on the source pool is somewhat misleading as its taking into account optimizations used by BTRFS snapshots. These will not necessarily be replicated as efficiently when moving pools.

I would have expected ZFS and LVM to be reasonably similar, but running du should help to see whats happening.

@tomp The debug logs are really long and I do not know if I am able to differentiate the various commands I launched :worried:

The lines output by
sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- du -h --threshold=1GB /var/snap/lxd/common/lxd/storage-pools/default (only greater than 1GB) are:

 1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt/freeswitch
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/usr/lib
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/usr/share
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/usr
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib/docker/btrfs/subvolumes
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib/docker/btrfs
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib/docker
4.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib
4.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var
9.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs
9.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1
9.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/usr/lib
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/usr/share
3.2G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/usr
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/var/cache/apt/archives
1.5G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/var/cache/apt
1.5G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/var/cache
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs/var
5.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021/rootfs
5.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-14-2021
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/usr/lib
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/usr/share
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/usr
1.5G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/cache/apt/archives
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/cache/apt
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/cache
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/lib/docker/btrfs/subvolumes
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/lib/docker/btrfs
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/lib/docker
4.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var/lib
5.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs/var
11G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021/rootfs
11G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-Jan-15-2021
16G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1
16G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots
26G     /var/snap/lxd/common/lxd/storage-pools/default

(container1 is the one that stayed on the default storage, now the other container has moved on bigstorage since it succeeded with zfs driver)

Already it is surprising since lxc storage info default is indicating only 10GB of used space… (which only corresponds to the space used by the container without snapshots). On the other hand sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- du -h /var/snap/lxd/common/lxd/storage-pools/bigstorage gives only:

4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/custom-snapshots
4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/virtual-machines-snapshots
4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots/Mycontainer/snap-Jan-17-2020
8.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots/Mycontainer
12K     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots
4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/custom
4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/virtual-machines
4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/images
4.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer
8.0K    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers
44K     /var/snap/lxd/common/lxd/storage-pools/bigstorage

Whereas lxc storage info default is indicating 49GB of used space. I am a bit lost to be honest :smile:

So you would run this in one terminal:

 sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log

Then run your lxc copy command in another until it failed, and capture everything that was in the other window.

I meant for you to run the lxc copy rather than lxc move when trying different storage pools as now you’ve successfully moved to ZFS the original BTRFS volume has been removed and so we cannot see its usage specifically on BTRFS. Do the docker containers still start OK, considering they were originally using BTRFS driver, and are now running on ZFS? What driver does it show them using now?

You cannot use the du command as easily on ZFS, as the volumes for each container are only visible when th container is running. Whereas with BTRFS the storage layer presents a single filesystem to the host OS, so there is no per-volume mounting needed.

1 Like

The moved container actually starts but there seem to be problems when starting the docker containers inside it… (docker-compose error). So I cannot run docker info, I don’t know which driver they are are using now…

Ah I understand I should have copied instead of moving… :neutral_face:

You were right about zfs, now that the container has started,the du works:

1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6-init
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52-init
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/1a1ad21c45f3b5e246696160f87637fa38491be987c0b56ec1a8204607c011f3
1.1G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/7cd1c2d84984db96c8514dfd92d5db84423439e38416b0da34f7cd0dbdbef5ef
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/79d5867ac039c435a812e889b79193476fc487353da3cc08180cbef042000a94
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9dda82ab13b4fa856d9891957df22fcbb577c8ee7452633e6ff9be2f783fa54e
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/8cae7c60c2e9969cf4f27d8b288e42216e53b100da5a0490b2d9a673c6990d38-init
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9454322f5dfcaad3230c1d58d4b8d8fe1a34b49c592785d044ef67e94ad4cab8
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/178a37a7d94c4037f4656fea51117cd5bda6197d7773fc7365c61fef953177ce
993M    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9cc8c42ad9cfbb3a3705e5b0748c40d64c046780fbe844d499a390eb534137ee
993M    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d01445dba3be41429402843e01c3dc7a6b7c0300d1f8f256f121705fa39b4b54
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115-init
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/8cae7c60c2e9969cf4f27d8b288e42216e53b100da5a0490b2d9a673c6990d38
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/01f9409ddfbbaf2069b712300d4fb75f7653b599a7ab3944501fea0f5bb5f48b
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52
1.4G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/4dc848c7d1defa0f16d39b7f2cfb23c6a2e7e4532296a493b2e35031a7369e73
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/cc2f83d692a70154de3a666da51c5d864c3f2db4e56be80ed44774f441f5bf77
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/63a927c28c8ca05463bc262eaa000dc948ba6f30202b977bb84674db8870f4bd
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c848e7420dd96aa80dfa28116b93ccc6c8e92dec763d3bcb7862f6576a48573f
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/a4f88ed9ac94a722f871f2bc5881ff0cf394519ec7bd25888326451f7936d2a9
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/839cad1441371e61d33e1e1106c0c96f0105b5dea0a64b23880de930c707fa70
1.3G    /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/285b9ed4147db7fd378392e84b7796395c98672c02c0b3d65bf9cb753d154b4c
47G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes
47G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker/btrfs
47G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib/docker
48G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var/lib
48G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs/var
49G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer/rootfs
49G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer
49G     /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers
49G     /var/snap/lxd/common/lxd/storage-pools/bigstorage

I think at this point I will just load a backup of my entire server (service provided by OVH which is hosting my server) to restart from scratch. I will try du before and after copying + check the debug logs. Do you think it will be helpful/informative? Do you have other recommendations?

I may also try the tarball approach…

OK great. Yes I think we can safely say that if your docker install is reliant on BTRFS then copying to a non-BTRFS pool or exporting it to a non-optimized tarball (which is the default without using --optimized-storage flag) will break your docker containers.

Doing lxc copy is a good way of testing without risking damaging the source container, and then once you’re happy you can delete the source (which is all that lxc move does anyway).

I suspect there is some issue with the way BTRFS subvolumes are being transferred.

Suggest trying the following with the debug mode enabled:

lxc copy Mycontainer Mycontainer2 -s bigstorage

And then separately, with a fresh debug log tail command:

lxc export Mycontainer /some/path/Mycontainer.tar.gz --optimized-storage

The BTRFS subvolume snapshots are likely masking the effective storage size. However I would expect an optimized migration or export to also benefit from these optimizations.

1 Like

I have restored my server as it was before any copying/moving/exporting operation.

Here is the output of du before any action:

$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- du -h --threshold=1GB /var/snap/lxd/common/lxd/storage-pools/default
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/usr
1004M   /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/67525a3625a447d58c4e21c44fdfb46b506c7094b38a055b5300a78d7a742d6b
1.1G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/f3cd52527396bd5d8fdb7318774bdc598ab0920cb727c112c46858beaf778480
1.3G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c3db7a977afcaddc1d8a7fa7dabd2fc4e4621b2f3a10fbe53740d7e01cae13b2
999M    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d01445dba3be41429402843e01c3dc7a6b7c0300d1f8f256f121705fa39b4b54/openedx
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d01445dba3be41429402843e01c3dc7a6b7c0300d1f8f256f121705fa39b4b54
999M    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9cc8c42ad9cfbb3a3705e5b0748c40d64c046780fbe844d499a390eb534137ee/openedx
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9cc8c42ad9cfbb3a3705e5b0748c40d64c046780fbe844d499a390eb534137ee
1.2G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/7cd1c2d84984db96c8514dfd92d5db84423439e38416b0da34f7cd0dbdbef5ef/openedx
2.1G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/7cd1c2d84984db96c8514dfd92d5db84423439e38416b0da34f7cd0dbdbef5ef
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/cc2f83d692a70154de3a666da51c5d864c3f2db4e56be80ed44774f441f5bf77/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/cc2f83d692a70154de3a666da51c5d864c3f2db4e56be80ed44774f441f5bf77
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9dda82ab13b4fa856d9891957df22fcbb577c8ee7452633e6ff9be2f783fa54e/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9dda82ab13b4fa856d9891957df22fcbb577c8ee7452633e6ff9be2f783fa54e
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/1a1ad21c45f3b5e246696160f87637fa38491be987c0b56ec1a8204607c011f3/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/1a1ad21c45f3b5e246696160f87637fa38491be987c0b56ec1a8204607c011f3
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/79d5867ac039c435a812e889b79193476fc487353da3cc08180cbef042000a94/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/79d5867ac039c435a812e889b79193476fc487353da3cc08180cbef042000a94
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/285b9ed4147db7fd378392e84b7796395c98672c02c0b3d65bf9cb753d154b4c/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/285b9ed4147db7fd378392e84b7796395c98672c02c0b3d65bf9cb753d154b4c
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/01f9409ddfbbaf2069b712300d4fb75f7653b599a7ab3944501fea0f5bb5f48b/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/01f9409ddfbbaf2069b712300d4fb75f7653b599a7ab3944501fea0f5bb5f48b
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/a4f88ed9ac94a722f871f2bc5881ff0cf394519ec7bd25888326451f7936d2a9/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/a4f88ed9ac94a722f871f2bc5881ff0cf394519ec7bd25888326451f7936d2a9
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9454322f5dfcaad3230c1d58d4b8d8fe1a34b49c592785d044ef67e94ad4cab8/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/9454322f5dfcaad3230c1d58d4b8d8fe1a34b49c592785d044ef67e94ad4cab8
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/839cad1441371e61d33e1e1106c0c96f0105b5dea0a64b23880de930c707fa70/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/839cad1441371e61d33e1e1106c0c96f0105b5dea0a64b23880de930c707fa70
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/178a37a7d94c4037f4656fea51117cd5bda6197d7773fc7365c61fef953177ce/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/178a37a7d94c4037f4656fea51117cd5bda6197d7773fc7365c61fef953177ce
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c848e7420dd96aa80dfa28116b93ccc6c8e92dec763d3bcb7862f6576a48573f/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c848e7420dd96aa80dfa28116b93ccc6c8e92dec763d3bcb7862f6576a48573f
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/63a927c28c8ca05463bc262eaa000dc948ba6f30202b977bb84674db8870f4bd/openedx
2.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/63a927c28c8ca05463bc262eaa000dc948ba6f30202b977bb84674db8870f4bd
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/4dc848c7d1defa0f16d39b7f2cfb23c6a2e7e4532296a493b2e35031a7369e73/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/4dc848c7d1defa0f16d39b7f2cfb23c6a2e7e4532296a493b2e35031a7369e73
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6-init/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6-init
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52-init/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52-init
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/8cae7c60c2e9969cf4f27d8b288e42216e53b100da5a0490b2d9a673c6990d38-init/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/8cae7c60c2e9969cf4f27d8b288e42216e53b100da5a0490b2d9a673c6990d38-init
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/8cae7c60c2e9969cf4f27d8b288e42216e53b100da5a0490b2d9a673c6990d38/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/8cae7c60c2e9969cf4f27d8b288e42216e53b100da5a0490b2d9a673c6990d38
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115-init/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115-init
1.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115/openedx
2.8G    /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115
95G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs/subvolumes
95G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker/btrfs
95G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib/docker
95G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var/lib
95G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs/var
97G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer/rootfs
97G     /var/snap/lxd/common/lxd/storage-pools/default/containers/Mycontainer
958M    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt/freeswitch/var/log/freeswitch
958M    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt/freeswitch/var/log
963M    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt/freeswitch/var
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt/freeswitch
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/opt
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/usr/lib
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/usr/share
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/usr
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib/docker/btrfs/subvolumes
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib/docker/btrfs
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib/docker
4.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var/lib
4.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs/var
9.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1/rootfs
9.9G    /var/snap/lxd/common/lxd/storage-pools/default/containers/container1
107G    /var/snap/lxd/common/lxd/storage-pools/default/containers
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/usr/lib
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/usr/share
3.2G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/usr
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/var/cache/apt/archives
1.5G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/var/cache/apt
1.5G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/var/cache
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs/var
5.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021/rootfs
5.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-demo-Jan-14-2021
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/usr/lib
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/usr/share
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/usr
1.5G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/cache/apt/archives
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/cache/apt
1.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/cache
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/lib/docker/btrfs/subvolumes
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/lib/docker/btrfs
3.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/lib/docker
4.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var/lib
5.6G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs/var
11G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021/rootfs
11G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1/snap-BBB-Greenlight-Jan-15-2021
16G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/container1
1.4G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/usr
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020
2.0G    /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer
18G     /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots
125G    /var/snap/lxd/common/lxd/storage-pools/default

We can clearly see that Mycontainer is taking 95GB of space and this is probably what is causing the No space left on device error when copying. What is weird is that Mycontainer only contains an installation of Tutor EDX (https://docs.tutor.overhang.io/), here are the installation requirements: Recommended configuration: 8 GB RAM, 4 CPU, 25 GB disk space (only 25GB…). Also, OVH which is hosting my server indicates that only 26GB of space is actually used for my server (2 containers: Mycontainer and container1). Also, the default storage pool has total space 30GB, with 24GB used according to lxc storage info default. Even if this command is not accurate, I am still confused about what is really happening here: I guess OVH cannot be wrong in the estimation of space used otherwise they would lose quite a lot of money or there would be a problem somewhere I guess (even though I am not completely sure, before running du, OVH was indicating 21GB of space used and after running this command it went up to 26GB… this is really hard to understand :sweat_smile:).

I will try to export Mycontainer to see what happens.

OK we are getting somewhere.

So the du command output is showing the total disk usage if all of the BTRFS subvolumes were ‘normal’ files (i.e on a normal ext4 filesystem for example). However what is likely going on is that docker is using BTRFS snapshots for each of the copies of a the base images it uses to make up each container (similar to what LXD does). You can see there is a lot of repetition in the docker volume sizes, suggesting these are snapshots of the same source volume.

LXD does a similar operation for its own snapshots.

This means the storage isn’t duplicated, until that is, it is copied to a non-BTRFS filesystem (or non-optimized backup), at which point the files are ‘expanded’ back to their normal presentation.

However the somewhat unexpected thing here is that when transferring to another BTRFS pool (or performing an optimized backup export) we use the native btrfs send tool to create a btrfs specific copy of the volumes with an understanding of the relationship between subvolumes and snapshots.

It is possible that the way we detect a subvolume’s parent is not correct and we are causing the btrfs send tool to send duplicated data.

I am interesting to see the Sending subvolume debug messages when doing an lxc copy as that should help to see what is happening.

2 Likes

Right now I am doing a storage-optimized export (it is taking some time). I will then do the copy and post the debug messages of both.

I never explicitly asked docker to take snapshots, do you think they could have been automatically generated or generated during the snapshot of the LXD container?

Its likely using snapshots and subvolumes as a way of building up the layers from each sub-image that a docker container is built from. Although that’s just a guess as am not familiar with Docker’s use of BTRFS.

Ok thanks, do you think that if Mycontainer had been created on a non-btfrs storage from the beginning (say lvm or ZFS), all these problems would not have appeared?

Yes I would think that it would use something else (probably overlayfs), although there are other issues to contend with (for example you cannot use overlay on ZFS see Docker with overlay driver in LXD cluster not working)

Depending on whether you don’t need efficient snapshots, then using a dir pool on ext4 might be a less problematic approach. Alternatively you could also try using an lvm pool using ext4 as that would then support ‘proper’ snapshots of the container itself.

1 Like

The docker vfs driver will work on all storage layers, but I would think this would be the most inefficient storage wise.

See https://docs.docker.com/storage/storagedriver/vfs-driver/

Debug messages of export with optimized storage (lxc export Mycontainer /home/ubuntu/Mycontainer.tar.gz --optimized-storage):

$ sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-01-28T08:45:23+0000 lvl=dbug msg="Processing image" alias=ubuntu/20.04 fp=a61920126a2c89d6d9f2d3bc4268db3be1e4200baea97927216f5766914afb19 protocol=simplestreams server=https://images.linuxcontainers.org
t=2021-01-28T08:45:23+0000 lvl=dbug msg="Connecting to a remote simplestreams server"
t=2021-01-28T08:45:23+0000 lvl=info msg="Done updating instance types"
t=2021-01-28T08:45:24+0000 lvl=dbug msg="Success for task operation: 228ba38e-eba6-47a5-b9a2-e7009076ec91"
t=2021-01-28T08:45:24+0000 lvl=dbug msg="Image \"a61920126a2c89d6d9f2d3bc4268db3be1e4200baea97927216f5766914afb19\" already exists in the DB"
t=2021-01-28T08:45:24+0000 lvl=dbug msg="Image already exists on storage pool \"default\""
t=2021-01-28T08:45:24+0000 lvl=dbug msg="Already up to date" fp=a61920126a2c89d6d9f2d3bc4268db3be1e4200baea97927216f5766914afb19
t=2021-01-28T08:45:24+0000 lvl=dbug msg="Success for task operation: 4f7ebfcf-f2ea-4a86-8e55-e7f9301510b7"
t=2021-01-28T08:45:24+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0 username=root
t=2021-01-28T08:45:24+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/internal/ready username=root
t=2021-01-28T08:46:44+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0 username=ubuntu
t=2021-01-28T08:46:44+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url="/1.0/instances?recursion=2" username=ubuntu
t=2021-01-28T08:46:44+0000 lvl=dbug msg="GetInstanceUsage started" driver=btrfs instance=container1 pool=default project=default
t=2021-01-28T08:46:44+0000 lvl=dbug msg="GetInstanceUsage started" driver=btrfs instance=Mycontainer pool=default project=default
t=2021-01-28T08:46:44+0000 lvl=dbug msg="GetInstanceUsage finished" driver=btrfs instance=Mycontainer pool=default project=default
t=2021-01-28T08:46:44+0000 lvl=dbug msg="GetInstanceUsage finished" driver=btrfs instance=container1 pool=default project=default
t=2021-01-28T08:47:35+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0 username=ubuntu
t=2021-01-28T08:47:35+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/events username=ubuntu
t=2021-01-28T08:47:35+0000 lvl=dbug msg="New event listener: bfe1ae0b-010d-4ccd-a3c9-4893267b186e"
t=2021-01-28T08:47:35+0000 lvl=dbug msg=Handling ip=@ method=POST protocol=unix url=/1.0/instances/Mycontainer/backups username=ubuntu
t=2021-01-28T08:47:35+0000 lvl=dbug msg="\n\t{\n\t\t\"name\": \"\",\n\t\t\"expires_at\": \"2021-01-29T08:47:35.805835911Z\",\n\t\t\"instance_only\": false,\n\t\t\"container_only\": false,\n\t\t\"optimized_storage\": true,\n\t\t\"compression_algorithm\": \"\"\n\t}"
t=2021-01-28T08:47:35+0000 lvl=dbug msg="New task Operation: 52b26311-7309-4693-a99f-c24ad2912981"
t=2021-01-28T08:47:35+0000 lvl=dbug msg="Started task operation: 52b26311-7309-4693-a99f-c24ad2912981"
t=2021-01-28T08:47:35+0000 lvl=dbug msg="Instance backup started" instance=Mycontainer name=Mycontainer/backup0 project=default
t=2021-01-28T08:47:35+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/52b26311-7309-4693-a99f-c24ad2912981\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"52b26311-7309-4693-a99f-c24ad2912981\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Backing up container\",\n\t\t\t\"created_at\": \"2021-01-28T08:47:35.815396335Z\",\n\t\t\t\"updated_at\": \"2021-01-28T08:47:35.815396335Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"backups\": [\n\t\t\t\t\t\"/1.0/backups/backup0\"\n\t\t\t\t],\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/Mycontainer\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/Mycontainer\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}"
t=2021-01-28T08:47:35+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/operations/52b26311-7309-4693-a99f-c24ad2912981 username=ubuntu
t=2021-01-28T08:47:35+0000 lvl=dbug msg="Opening backup tarball for writing" instance=Mycontainer name=Mycontainer/backup0 path=/var/snap/lxd/common/lxd/backups/Mycontainer/backup0 project=default
t=2021-01-28T08:47:35+0000 lvl=dbug msg="Adding backup index file" instance=Mycontainer name=Mycontainer/backup0 project=default
t=2021-01-28T08:47:35+0000 lvl=dbug msg="Started backup tarball writer" instance=Mycontainer name=Mycontainer/backup0 project=default
t=2021-01-28T08:47:35+0000 lvl=dbug msg="BackupInstance started" driver=btrfs instance=Mycontainer optimized=true pool=default project=default snapshots=true
t=2021-01-28T08:47:35+0000 lvl=dbug msg="UpdateInstanceBackupFile started" driver=btrfs instance=Mycontainer pool=default project=default
t=2021-01-28T08:47:35+0000 lvl=dbug msg="UpdateInstanceBackupFile finished" driver=btrfs instance=Mycontainer pool=default project=default
t=2021-01-28T08:49:45+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs839412570 name=backup/snapshots/snap-Tutor-Jan-17-2020.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020
t=2021-01-28T08:51:43+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs180651249 name=backup/snapshots/snap-Tutor-Jan-17-2020_rootfs-var-lib-docker-btrfs-subvolumes-381c436496c9289c6464401febbede4ff21dc0c5e45e8d79d73f678a5636e0d8.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/381c436496c9289c6464401febbede4ff21dc0c5e45e8d79d73f678a5636e0d8
t=2021-01-28T08:51:43+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs283818396 name=backup/snapshots/snap-Tutor-Jan-17-2020_rootfs-var-lib-docker-btrfs-subvolumes-83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92
t=2021-01-28T08:51:43+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs196749899 name=backup/snapshots/snap-Tutor-Jan-17-2020_rootfs-var-lib-docker-btrfs-subvolumes-83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92--init.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92-init
t=2021-01-28T08:53:42+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs730614965 name=backup/container.bin parent=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020 pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup
t=2021-01-28T08:54:08+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs143224464 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc
t=2021-01-28T08:54:55+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs799645359 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-01f9409ddfbbaf2069b712300d4fb75f7653b599a7ab3944501fea0f5bb5f48b.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/01f9409ddfbbaf2069b712300d4fb75f7653b599a7ab3944501fea0f5bb5f48b
t=2021-01-28T08:57:45+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs534228034 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-029296d7aa8498e4315eafd4dfc8a1f49f035e0b3f0e7e9454e14daf5ea52055.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/029296d7aa8498e4315eafd4dfc8a1f49f035e0b3f0e7e9454e14daf5ea52055
t=2021-01-28T08:57:51+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs982300857 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-034e387fc9229751738118c2241125aec88c8f2cba010753eafa0feef1869eba.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/034e387fc9229751738118c2241125aec88c8f2cba010753eafa0feef1869eba
t=2021-01-28T08:58:05+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs197586628 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-04d0f52b193526fdf9b99e8d3a22be9a5b43c8fa0ac6fe4e614e9f5152e0e26b.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/04d0f52b193526fdf9b99e8d3a22be9a5b43c8fa0ac6fe4e614e9f5152e0e26b
t=2021-01-28T08:58:13+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs983719251 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-07900b395dd5ab1d7396e7444ed4e9b2f7d8d6fbb78d2f0ba659dba528c84c56.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/07900b395dd5ab1d7396e7444ed4e9b2f7d8d6fbb78d2f0ba659dba528c84c56
t=2021-01-28T08:58:46+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs752347286 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-07ab0e5a0f8860c8879aa78081597af6b21419f42f861ed311ebbba8b9c58cfa.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/07ab0e5a0f8860c8879aa78081597af6b21419f42f861ed311ebbba8b9c58cfa
t=2021-01-28T08:58:53+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs294303997 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-08471973049c9578b4b5b3d04ac42796950a3a9ea79f44b31910857e059916ab.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/08471973049c9578b4b5b3d04ac42796950a3a9ea79f44b31910857e059916ab
t=2021-01-28T08:59:06+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs368697400 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-0bfa7cb7283c98aeee173638d00a87b2f921de7cfd59386c03c579b03709b0cc.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/0bfa7cb7283c98aeee173638d00a87b2f921de7cfd59386c03c579b03709b0cc
t=2021-01-28T08:59:45+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs011214391 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-100382bf437bfdaeceeb6870fafc13796ba816adfbb174988ceba32a2c1c0905.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/100382bf437bfdaeceeb6870fafc13796ba816adfbb174988ceba32a2c1c0905
...
t=2021-01-28T09:45:20+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs469711404 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-996f3fb2581236ac0367775e88a33041f7a21eab6f8e3174d359b44e2cd844a5.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/996f3fb2581236ac0367775e88a33041f7a21eab6f8e3174d359b44e2cd844a5
t=2021-01-28T09:45:23+0000 lvl=dbug msg="New task Operation: 2dc6734f-82f5-4010-b6ad-145348789ff0"
t=2021-01-28T09:45:23+0000 lvl=info msg="Pruning expired instance backups"
t=2021-01-28T09:45:23+0000 lvl=dbug msg="Started task operation: 2dc6734f-82f5-4010-b6ad-145348789ff0"
t=2021-01-28T09:45:23+0000 lvl=info msg="Done pruning expired instance backups"
t=2021-01-28T09:45:24+0000 lvl=dbug msg="Success for task operation: 2dc6734f-82f5-4010-b6ad-145348789ff0"
t=2021-01-28T09:45:27+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs132018075 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-9cc8c42ad9cfbb3a3705e5b0748c40d64c046780fbe844d499a390eb534137ee.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/9cc8c42ad9cfbb3a3705e5b0748c40d64c046780fbe844d499a390eb534137ee
t=2021-01-28T09:47:36+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs693461054 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-9dda82ab13b4fa856d9891957df22fcbb577c8ee7452633e6ff9be2f783fa54e.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/9dda82ab13b4fa856d9891957df22fcbb577c8ee7452633e6ff9be2f783fa54e
t=2021-01-28T09:50:26+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs224078981 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-a249cff840ee78909a7db60d1251f539eb070478b92c329ffe52265b147be7d5.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/a249cff840ee78909a7db60d1251f539eb070478b92c329ffe52265b147be7d5
t=2021-01-28T09:50:31+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs738368544 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-a4f88ed9ac94a722f871f2bc5881ff0cf394519ec7bd25888326451f7936d2a9.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/a4f88ed9ac94a722f871f2bc5881ff0cf394519ec7bd25888326451f7936d2a9
t=2021-01-28T09:53:23+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs166618879 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-a7d6f0480811c21968f4db15482afda880fb5a90990ce2e9b1e8aab9d26628b2.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/a7d6f0480811c21968f4db15482afda880fb5a90990ce2e9b1e8aab9d26628b2
t=2021-01-28T09:54:12+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs380267090 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-ab68743f7a365166e5c6e998368bcac3f180d493e5f5cbc3f13b16f4723882c0.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/ab68743f7a365166e5c6e998368bcac3f180d493e5f5cbc3f13b16f4723882c0
t=2021-01-28T09:54:19+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs085107593 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-ab68743f7a365166e5c6e998368bcac3f180d493e5f5cbc3f13b16f4723882c0--init.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/ab68743f7a365166e5c6e998368bcac3f180d493e5f5cbc3f13b16f4723882c0-init
t=2021-01-28T09:54:26+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs571886420 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56
t=2021-01-28T09:57:16+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs259572387 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-ae2622db75610796c85eda0ef5bc504e51eba0187a10ab5f32486cba9f4f74be.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/ae2622db75610796c85eda0ef5bc504e51eba0187a10ab5f32486cba9f4f74be
t=2021-01-28T09:57:37+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs621332390 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-af8bb684390c51e00a98161b6d18c75a1c5a3186ca1fc77685183a8c6798dd51.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/af8bb684390c51e00a98161b6d18c75a1c5a3186ca1fc77685183a8c6798dd51
t=2021-01-28T09:58:04+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs480459469 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-b4f8094cb551e2e2316c87465cb1b88d1b6d410be599175dbf35c5050b6bbc2c.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/b4f8094cb551e2e2316c87465cb1b88d1b6d410be599175dbf35c5050b6bbc2c
t=2021-01-28T09:58:31+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs016124360 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-b4f9c65edf561876af2343a0c346bbb75584c86ab790d1055dab9fba8a9482a7.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/b4f9c65edf561876af2343a0c346bbb75584c86ab790d1055dab9fba8a9482a7
t=2021-01-28T09:58:36+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs537634439 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-b6ddec9c2d725bc5ac6d72c6c4122507f6bd718b7170ec320397c5a433616c7c.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/b6ddec9c2d725bc5ac6d72c6c4122507f6bd718b7170ec320397c5a433616c7c
t=2021-01-28T09:58:43+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs297718842 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-b7531d46c76a0fd221bc1fdeea2c22870569f40f3538fc51b79f7f3160657076.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/b7531d46c76a0fd221bc1fdeea2c22870569f40f3538fc51b79f7f3160657076
t=2021-01-28T09:59:14+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs761371729 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-bb6e17ebfd62641cd03c981138fc12a24faa7eda8970df9f1a6fdea74a8eab44.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/bb6e17ebfd62641cd03c981138fc12a24faa7eda8970df9f1a6fdea74a8eab44
t=2021-01-28T09:59:28+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs996119932 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-bf795295e12cd843bec61d89ee477ba4188b5b0e68f11fe862690b6b3e6050af.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/bf795295e12cd843bec61d89ee477ba4188b5b0e68f11fe862690b6b3e6050af
t=2021-01-28T09:59:36+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs383064235 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c0fcca66627e0b77f523d0126966144a58d5d853a55284322aca2ae36c9a1958.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c0fcca66627e0b77f523d0126966144a58d5d853a55284322aca2ae36c9a1958
t=2021-01-28T10:00:15+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs258339854 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52
t=2021-01-28T10:03:22+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs781775893 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52--init.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c1afa32de063ad951f02507477b249fda63f580063ec358e47b650cb993dcc52-init
t=2021-01-28T10:06:18+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs635825264 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c3db7a977afcaddc1d8a7fa7dabd2fc4e4621b2f3a10fbe53740d7e01cae13b2.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c3db7a977afcaddc1d8a7fa7dabd2fc4e4621b2f3a10fbe53740d7e01cae13b2
t=2021-01-28T10:07:37+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs234753295 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c848e7420dd96aa80dfa28116b93ccc6c8e92dec763d3bcb7862f6576a48573f.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c848e7420dd96aa80dfa28116b93ccc6c8e92dec763d3bcb7862f6576a48573f
t=2021-01-28T10:10:25+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs747141922 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c87c74e70ead6e72298f1eef6379e7875432ff0c9114c00eb45475319223bf32.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c87c74e70ead6e72298f1eef6379e7875432ff0c9114c00eb45475319223bf32
t=2021-01-28T10:10:31+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs784827929 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-c8c00ac2f9a92ff58769831e345b42d6e8354bd7c500ef18a984e445e8e02f3d.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/c8c00ac2f9a92ff58769831e345b42d6e8354bd7c500ef18a984e445e8e02f3d
t=2021-01-28T10:10:37+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs879736996 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-ca1c9adafc5bbf4a8b19a522ba30b08a6ede7ccac0ec23c98a084bb2e031c00c.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/ca1c9adafc5bbf4a8b19a522ba30b08a6ede7ccac0ec23c98a084bb2e031c00c
t=2021-01-28T10:10:45+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs907403699 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-cbba83107aa784d981e2dbefc98dd6026760cc9cce07c68bb3a6cf07f15d2a4c.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/cbba83107aa784d981e2dbefc98dd6026760cc9cce07c68bb3a6cf07f15d2a4c
t=2021-01-28T10:10:47+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs726400374 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-cc2f83d692a70154de3a666da51c5d864c3f2db4e56be80ed44774f441f5bf77.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/cc2f83d692a70154de3a666da51c5d864c3f2db4e56be80ed44774f441f5bf77
t=2021-01-28T10:13:32+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs993201245 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-cd8cfdd6986935a6939e1b9b15e2d1b015173e43764db61627aab8d56b944d2f.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/cd8cfdd6986935a6939e1b9b15e2d1b015173e43764db61627aab8d56b944d2f
t=2021-01-28T10:14:14+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs929096728 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-cfa4ef476d17f6f93b8260e197f5540f06b436671e0c5df43d30628831ff0050.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/cfa4ef476d17f6f93b8260e197f5540f06b436671e0c5df43d30628831ff0050
t=2021-01-28T10:14:36+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs985905815 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-cff991596285494dc725fd37fb2523a649aa26c04eb707419f18dfa35ca85521.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/cff991596285494dc725fd37fb2523a649aa26c04eb707419f18dfa35ca85521
t=2021-01-28T10:14:50+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs045022986 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6
t=2021-01-28T10:17:52+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs223862497 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6--init.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/d003a8eea3a8a35ad0764a6e8406a07410e32e2eedaeed7c9b217b32e692a1f6-init
t=2021-01-28T10:20:42+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs503590092 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-d01445dba3be41429402843e01c3dc7a6b7c0300d1f8f256f121705fa39b4b54.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/d01445dba3be41429402843e01c3dc7a6b7c0300d1f8f256f121705fa39b4b54
t=2021-01-28T10:22:52+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs429532091 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115
t=2021-01-28T10:25:54+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs100314590 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115--init.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/d1f7f9f3761a7145d2b47ec7d8ec121233cb2499431bab661c9ba685f5a28115-init
t=2021-01-28T10:29:00+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs921791909 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-d4d74812c61896ce144fc098270dc6c6f16f955b1e2a00163c7cffb85b84a078.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/d4d74812c61896ce144fc098270dc6c6f16f955b1e2a00163c7cffb85b84a078
t=2021-01-28T10:29:27+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs358508736 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00
t=2021-01-28T10:32:07+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs925155615 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-df88744ff1b386b19d6282fd36a6e262c8dd883bf35dd076a22874873b56b81a.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/df88744ff1b386b19d6282fd36a6e262c8dd883bf35dd076a22874873b56b81a
t=2021-01-28T10:32:27+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs154351090 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-e294050a34998290b98237a6ca255fe9cca17e97deff7706c71469604a7229d4.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/e294050a34998290b98237a6ca255fe9cca17e97deff7706c71469604a7229d4
t=2021-01-28T10:32:59+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs361374889 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-e5640f70a225e1f0e35ee58cdd135454ef2957d5e1cf93dde9f768e918c64de3.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/e5640f70a225e1f0e35ee58cdd135454ef2957d5e1cf93dde9f768e918c64de3
t=2021-01-28T10:33:07+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs086244852 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-e853bfa7ce0d174ed666539e5b8233343e4c7fe30538bfde52fa30f1f4fef54d.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/e853bfa7ce0d174ed666539e5b8233343e4c7fe30538bfde52fa30f1f4fef54d
t=2021-01-28T10:33:21+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs487345859 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619
t=2021-01-28T10:36:11+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs151226694 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-eb4f60a4b07aaed19b185ebbe963d3a21bd931d0b5aa699f0c1a8190cf1df8ee.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/eb4f60a4b07aaed19b185ebbe963d3a21bd931d0b5aa699f0c1a8190cf1df8ee
t=2021-01-28T10:36:19+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs391985133 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-ec95df8b899ecf5b137097c8f981064a7fa02252d49f72fb2460d3b0ea8c8ffb.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/ec95df8b899ecf5b137097c8f981064a7fa02252d49f72fb2460d3b0ea8c8ffb
t=2021-01-28T10:36:27+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs319173736 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-f27945880a132acfc31c9d7cae9178acecaf0ce077d1519377dd42ed1330ce73.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/f27945880a132acfc31c9d7cae9178acecaf0ce077d1519377dd42ed1330ce73
t=2021-01-28T10:36:46+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs941706919 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-f3cd52527396bd5d8fdb7318774bdc598ab0920cb727c112c46858beaf778480.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/f3cd52527396bd5d8fdb7318774bdc598ab0920cb727c112c46858beaf778480
t=2021-01-28T10:37:55+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs163937242 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-f4557506ab1270079fcd51b222464dcad521083b5fd81e1f6b333750775821ce.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/f4557506ab1270079fcd51b222464dcad521083b5fd81e1f6b333750775821ce
t=2021-01-28T10:37:58+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs199460209 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-f4557506ab1270079fcd51b222464dcad521083b5fd81e1f6b333750775821ce--init.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/f4557506ab1270079fcd51b222464dcad521083b5fd81e1f6b333750775821ce-init
t=2021-01-28T10:38:01+0000 lvl=dbug msg="Generating optimized volume file" driver=btrfs file=/var/snap/lxd/common/lxd/backups/lxd_backup_btrfs625375772 name=backup/container_rootfs-var-lib-docker-btrfs-subvolumes-f56eb34a67557078e854b410f29898ba772e19799b446b3f7d6930caeb49a744.bin parent= pool=default sourcePath=/var/snap/lxd/common/lxd/storage-pools/default/containers/backup.796784430/.backup/rootfs/var/lib/docker/btrfs/subvolumes/f56eb34a67557078e854b410f29898ba772e19799b446b3f7d6930caeb49a744
t=2021-01-28T10:40:14+0000 lvl=dbug msg="BackupInstance finished" driver=btrfs instance=Mycontainer optimized=true pool=default project=default snapshots=true
t=2021-01-28T10:40:14+0000 lvl=dbug msg="Finished backup tarball writer" instance=Mycontainer name=Mycontainer/backup0 project=default
t=2021-01-28T10:40:14+0000 lvl=dbug msg="Instance backup finished" instance=Mycontainer name=Mycontainer/backup0 project=default
t=2021-01-28T10:40:14+0000 lvl=dbug msg="Success for task operation: 52b26311-7309-4693-a99f-c24ad2912981"
t=2021-01-28T10:40:15+0000 lvl=dbug msg=Handling ip=@ method=DELETE protocol=unix url=/1.0/instances/Mycontainer/backups/backup0 username=ubuntu
t=2021-01-28T10:40:15+0000 lvl=dbug msg="New task Operation: 5001f071-9a78-4649-aab3-edbfab24dcc0"
t=2021-01-28T10:40:15+0000 lvl=dbug msg="Started task operation: 5001f071-9a78-4649-aab3-edbfab24dcc0"
t=2021-01-28T10:40:15+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/5001f071-9a78-4649-aab3-edbfab24dcc0\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"5001f071-9a78-4649-aab3-edbfab24dcc0\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Removing container backup\",\n\t\t\t\"created_at\": \"2021-01-28T10:40:15.173870637Z\",\n\t\t\t\"updated_at\": \"2021-01-28T10:40:15.173870637Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"container\": [\n\t\t\t\t\t\"/1.0/container/Mycontainer\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}"
t=2021-01-28T10:40:15+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/operations/5001f071-9a78-4649-aab3-edbfab24dcc0 username=ubuntu
t=2021-01-28T10:40:15+0000 lvl=dbug msg="Success for task operation: 5001f071-9a78-4649-aab3-edbfab24dcc0"
t=2021-01-28T10:40:15+0000 lvl=dbug msg="Event listener finished: bfe1ae0b-010d-4ccd-a3c9-4893267b186e"
t=2021-01-28T10:40:15+0000 lvl=dbug msg="Disconnected event listener: bfe1ae0b-010d-4ccd-a3c9-4893267b186e"
t=2021-01-28T10:45:23+0000 lvl=dbug msg="New task Operation: 9d2a3d9a-682e-452b-bfc4-57850f4ee658"
t=2021-01-28T10:45:23+0000 lvl=info msg="Pruning expired instance backups"
t=2021-01-28T10:45:23+0000 lvl=dbug msg="Started task operation: 9d2a3d9a-682e-452b-bfc4-57850f4ee658"
t=2021-01-28T10:45:23+0000 lvl=info msg="Done pruning expired instance backups"
t=2021-01-28T10:45:23+0000 lvl=dbug msg="Success for task operation: 9d2a3d9a-682e-452b-bfc4-57850f4ee658"

The generated archive file Mycontainer.tar.gz is 31GB big.

Debug messages of copy to bigstorage (lxc copy Mycontainer Mycontainer2 -s bigstorage):

$ sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-01-28T11:41:46+0000 lvl=dbug msg="New event listener: a5e97619-5068-4252-ae90-773ba599d983"
t=2021-01-28T11:41:46+0000 lvl=dbug msg=Handling ip=@ method=POST protocol=unix url=/1.0/instances username=ubuntu
t=2021-01-28T11:41:46+0000 lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"x86_64\",\n\t\t\"config\": {\n\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\"image.description\": \"ubuntu 20.04 LTS amd64 (release) (20210105)\",\n\t\t\t\"image.label\": \"release\",\n\t\t\t\"image.os\": \"ubuntu\",\n\t\t\t\"image.release\": \"focal\",\n\t\t\t\"image.serial\": \"20210105\",\n\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\"image.version\": \"20.04\",\n\t\t\t\"limits.kernel.memlock\": \"unlimited\",\n\t\t\t\"security.nesting\": \"true\",\n\t\t\t\"security.privileged\": \"false\",\n\t\t\t\"volatile.base_image\": \"21da67063730fc446ca7fe090a7cf90ad9397ff4001f69907d7db690a30897c3\"\n\t\t},\n\t\t\"devices\": {\n\t\t\t\"root\": {\n\t\t\t\t\"path\": \"/\",\n\t\t\t\t\"pool\": \"bigstorage\",\n\t\t\t\t\"type\": \"disk\"\n\t\t\t}\n\t\t},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": [\n\t\t\t\"default\",\n\t\t\t\"routed_54.38.226.92\"\n\t\t],\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"Mycontainer2\",\n\t\t\"source\": {\n\t\t\t\"type\": \"copy\",\n\t\t\t\"certificate\": \"\",\n\t\t\t\"base-image\": \"21da67063730fc446ca7fe090a7cf90ad9397ff4001f69907d7db690a30897c3\",\n\t\t\t\"source\": \"Mycontainer\"\n\t\t},\n\t\t\"instance_type\": \"\",\n\t\t\"type\": \"container\"\n\t}"
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Responding to instance create"
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Skipping volatile key from copy source" key=volatile.eth0.hwaddr
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Skipping volatile key from copy source" key=volatile.last_state.power
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Skipping volatile key from copy source" key=volatile.eth0.name
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Skipping volatile key from copy source" key=volatile.idmap.next
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Skipping volatile key from copy source" key=volatile.idmap.base
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Skipping volatile key from copy source" key=volatile.idmap.current
t=2021-01-28T11:41:46+0000 lvl=dbug msg="New task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:41:46+0000 lvl=dbug msg="Started task operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:41:46+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/b95177fa-8e36-452c-8a5e-8f8b02af6cde\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"b95177fa-8e36-452c-8a5e-8f8b02af6cde\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Creating container\",\n\t\t\t\"created_at\": \"2021-01-28T11:41:46.752195012Z\",\n\t\t\t\"updated_at\": \"2021-01-28T11:41:46.752195012Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/Mycontainer2\",\n\t\t\t\t\t\"/1.0/containers/Mycontainer\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/Mycontainer2\",\n\t\t\t\t\t\"/1.0/instances/Mycontainer\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}"
t=2021-01-28T11:41:46+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/operations/b95177fa-8e36-452c-8a5e-8f8b02af6cde username=ubuntu
t=2021-01-28T11:41:46+0000 lvl=info msg="Creating container" ephemeral=false name=Mycontainer2 project=default
t=2021-01-28T11:41:46+0000 lvl=info msg="Created container" ephemeral=false name=Mycontainer2 project=default
t=2021-01-28T11:41:46+0000 lvl=info msg="Creating container" ephemeral=false name=Mycontainer2/snap-Tutor-Jan-17-2020 project=default
t=2021-01-28T11:41:46+0000 lvl=info msg="Created container" ephemeral=false name=Mycontainer2/snap-Tutor-Jan-17-2020 project=default
t=2021-01-28T11:41:46+0000 lvl=dbug msg="CreateInstanceFromCopy started" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default snapshots=true src=Mycontainer
t=2021-01-28T11:41:46+0000 lvl=dbug msg="CreateInstanceFromCopy cross-pool mode detected" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default snapshots=true src=Mycontainer
t=2021-01-28T11:41:46+0000 lvl=dbug msg="CreateInstanceFromMigration started" args="{Name:Mycontainer2 Description: Config:map[] Snapshots:[snap-Tutor-Jan-17-2020] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes]} TrackProgress:false Refresh:false Live:false VolumeSize:0}" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T11:41:46+0000 lvl=dbug msg="MigrateInstance started" args="&{Name:Mycontainer Snapshots:[snap-Tutor-Jan-17-2020] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil>}" driver=btrfs instance=Mycontainer pool=default project=default
t=2021-01-28T11:43:34+0000 lvl=dbug msg="Sent migration meta data header" driver=btrfs name=Mycontainer pool=default
t=2021-01-28T11:43:34+0000 lvl=dbug msg="Received migration meta data header" driver=btrfs name=Mycontainer2 pool=bigstorage
t=2021-01-28T11:43:34+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer/snap-Tutor-Jan-17-2020 parent= path=/ pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020
t=2021-01-28T11:43:34+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2/snap-Tutor-Jan-17-2020 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots/Mycontainer2/snap-Tutor-Jan-17-2020 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T11:43:35+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
...
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer/snap-Tutor-Jan-17-2020 parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/381c436496c9289c6464401febbede4ff21dc0c5e45e8d79d73f678a5636e0d8 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/381c436496c9289c6464401febbede4ff21dc0c5e45e8d79d73f678a5636e0d8
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer/snap-Tutor-Jan-17-2020 parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2/snap-Tutor-Jan-17-2020 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots/Mycontainer2/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/381c436496c9289c6464401febbede4ff21dc0c5e45e8d79d73f678a5636e0d8 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer/snap-Tutor-Jan-17-2020 parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92-init pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92-init
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2/snap-Tutor-Jan-17-2020 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots/Mycontainer2/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2/snap-Tutor-Jan-17-2020 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers-snapshots/Mycontainer2/snap-Tutor-Jan-17-2020/rootfs/var/lib/docker/btrfs/subvolumes/83db7cf8df7e4051eeec69c9ab9b3b4369ea0768de2b9d0ac5cd806c55960b92-init pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T11:44:02+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T11:45:23+0000 lvl=dbug msg="New task Operation: 46c73a27-d70a-4d95-ac51-941024f075ce"
t=2021-01-28T11:45:23+0000 lvl=info msg="Pruning expired instance backups"
t=2021-01-28T11:45:23+0000 lvl=dbug msg="Started task operation: 46c73a27-d70a-4d95-ac51-941024f075ce"
t=2021-01-28T11:45:23+0000 lvl=info msg="Done pruning expired instance backups"
t=2021-01-28T11:45:23+0000 lvl=dbug msg="Success for task operation: 46c73a27-d70a-4d95-ac51-941024f075ce"
t=2021-01-28T11:46:08+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020 path=/ pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send
t=2021-01-28T11:46:09+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:10+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:11+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:12+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:13+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:14+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:15+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:16+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:17+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:18+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:18+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc
t=2021-01-28T11:46:19+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T11:46:19+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T11:46:20+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
...
t=2021-01-28T12:08:29+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56
t=2021-01-28T12:08:29+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/ac60677aedcd77ecfa0b25bbc791fd1d62716121be0f4f0185d0d284d3027a56 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:08:29+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:08:30+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:08:31+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
...
t=2021-01-28T12:09:44+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:45+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/ae2622db75610796c85eda0ef5bc504e51eba0187a10ab5f32486cba9f4f74be pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:09:45+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/ae2622db75610796c85eda0ef5bc504e51eba0187a10ab5f32486cba9f4f74be pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/ae2622db75610796c85eda0ef5bc504e51eba0187a10ab5f32486cba9f4f74be
t=2021-01-28T12:09:45+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:46+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:47+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:48+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:49+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:50+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/af8bb684390c51e00a98161b6d18c75a1c5a3186ca1fc77685183a8c6798dd51 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/af8bb684390c51e00a98161b6d18c75a1c5a3186ca1fc77685183a8c6798dd51
t=2021-01-28T12:09:50+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/af8bb684390c51e00a98161b6d18c75a1c5a3186ca1fc77685183a8c6798dd51 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:09:50+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:51+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:52+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:53+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:54+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:55+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:56+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:57+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/b4f8094cb551e2e2316c87465cb1b88d1b6d410be599175dbf35c5050b6bbc2c pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/b4f8094cb551e2e2316c87465cb1b88d1b6d410be599175dbf35c5050b6bbc2c
t=2021-01-28T12:09:57+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:57+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/b4f8094cb551e2e2316c87465cb1b88d1b6d410be599175dbf35c5050b6bbc2c pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:09:58+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:09:59+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:10:00+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:10:01+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:10:02+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:10:03+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/b4f9c65edf561876af2343a0c346bbb75584c86ab790d1055dab9fba8a9482a7 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/b4f9c65edf561876af2343a0c346bbb75584c86ab790d1055dab9fba8a9482a7
t=2021-01-28T12:10:03+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/b4f9c65edf561876af2343a0c346bbb75584c86ab790d1055dab9fba8a9482a7 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:10:03+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:10:04+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:10:05+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/b6ddec9c2d725bc5ac6d72c6c4122507f6bd718b7170ec320397c5a433616c7c pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/b6ddec9c2d725bc5ac6d72c6c4122507f6bd718b7170ec320397c5a433616c7c
t=2021-01-28T12:10:05+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/b6ddec9c2d725bc5ac6d72c6c4122507f6bd718b7170ec320397c5a433616c7c pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:10:05+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
...
t=2021-01-28T12:22:30+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00
t=2021-01-28T12:22:30+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/de486ea39750a1678c2e786885896dd1e4362db71030998ebb3933f010f07a00 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:22:31+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:32+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:33+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:34+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:35+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:36+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:37+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:38+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:39+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:40+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:41+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:42+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:43+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:44+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:45+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:46+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:47+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:48+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:49+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:50+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:51+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:52+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:53+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:54+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:55+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:56+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:57+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:58+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:22:59+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:00+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:01+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:02+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:03+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:04+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:05+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:06+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:07+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:08+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:09+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:10+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:11+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:12+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:13+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:14+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:15+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:16+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:17+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:18+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:19+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:20+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:21+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:22+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:23+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:24+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:25+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:26+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:27+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:28+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:29+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:30+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:31+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:32+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:33+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:34+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:35+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:36+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:37+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:38+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:39+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:40+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:41+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:42+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:43+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:44+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:45+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:46+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:47+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:48+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:49+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:49+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/df88744ff1b386b19d6282fd36a6e262c8dd883bf35dd076a22874873b56b81a pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/df88744ff1b386b19d6282fd36a6e262c8dd883bf35dd076a22874873b56b81a
t=2021-01-28T12:23:49+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/df88744ff1b386b19d6282fd36a6e262c8dd883bf35dd076a22874873b56b81a pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:23:50+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:51+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:52+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:53+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:54+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:55+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:55+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/e294050a34998290b98237a6ca255fe9cca17e97deff7706c71469604a7229d4 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/e294050a34998290b98237a6ca255fe9cca17e97deff7706c71469604a7229d4
t=2021-01-28T12:23:55+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/e294050a34998290b98237a6ca255fe9cca17e97deff7706c71469604a7229d4 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:23:56+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:57+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:58+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:23:59+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:00+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:01+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:02+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:03+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:04+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:05+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:05+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/e5640f70a225e1f0e35ee58cdd135454ef2957d5e1cf93dde9f768e918c64de3 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/e5640f70a225e1f0e35ee58cdd135454ef2957d5e1cf93dde9f768e918c64de3
t=2021-01-28T12:24:06+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/e5640f70a225e1f0e35ee58cdd135454ef2957d5e1cf93dde9f768e918c64de3 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:24:06+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:07+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:08+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:08+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/e853bfa7ce0d174ed666539e5b8233343e4c7fe30538bfde52fa30f1f4fef54d pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/e853bfa7ce0d174ed666539e5b8233343e4c7fe30538bfde52fa30f1f4fef54d
t=2021-01-28T12:24:08+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/e853bfa7ce0d174ed666539e5b8233343e4c7fe30538bfde52fa30f1f4fef54d pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:24:09+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:10+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:11+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:12+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:12+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619 pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619
t=2021-01-28T12:24:12+0000 lvl=dbug msg="Receiving volume" driver=btrfs name=Mycontainer2 path=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/Mycontainer2/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619 pool=bigstorage receivePath=/var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795
t=2021-01-28T12:24:13+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:14+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:15+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:16+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:17+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:18+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:19+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:20+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:21+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:22+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:23+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:24+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:25+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:26+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:27+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:28+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:29+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:30+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:31+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:32+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:33+0000 lvl=dbug msg="Updated metadata for task Operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde"
t=2021-01-28T12:24:35+0000 lvl=dbug msg="DeleteInstance started" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:24:35+0000 lvl=dbug msg="DeleteInstance finished" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:24:35+0000 lvl=dbug msg="CreateInstanceFromMigration finished" args="{Name:Mycontainer2 Description: Config:map[] Snapshots:[snap-Tutor-Jan-17-2020] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes]} TrackProgress:false Refresh:false Live:false VolumeSize:0}" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:26:47+0000 lvl=dbug msg="MigrateInstance finished" args="&{Name:Mycontainer Snapshots:[snap-Tutor-Jan-17-2020] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil>}" driver=btrfs instance=Mycontainer pool=default project=default
t=2021-01-28T12:26:47+0000 lvl=dbug msg="DeleteInstance started" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:26:47+0000 lvl=dbug msg="DeleteInstance finished" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:26:47+0000 lvl=dbug msg="CreateInstanceFromCopy finished" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default snapshots=true src=Mycontainer
t=2021-01-28T12:26:47+0000 lvl=info msg="Deleting container" created=2021-01-28T11:41:46+0000 ephemeral=false name=Mycontainer2 project=default used=1970-01-01T00:00:00+0000
t=2021-01-28T12:26:47+0000 lvl=info msg="Deleting container" created=2021-01-17T18:49:18+0000 ephemeral=false name=Mycontainer2/snap-Tutor-Jan-17-2020 project=default used=0001-01-01T00:00:00+0000
t=2021-01-28T12:26:47+0000 lvl=dbug msg="DeleteInstanceSnapshot started" driver=btrfs instance=Mycontainer2/snap-Tutor-Jan-17-2020 pool=bigstorage project=default
t=2021-01-28T12:26:47+0000 lvl=dbug msg="Deleting instance snapshot volume" driver=btrfs instance=Mycontainer2/snap-Tutor-Jan-17-2020 pool=bigstorage project=default snapshotName=snap-Tutor-Jan-17-2020 volName=Mycontainer2
t=2021-01-28T12:26:48+0000 lvl=dbug msg="DeleteInstanceSnapshot finished" driver=btrfs instance=Mycontainer2/snap-Tutor-Jan-17-2020 pool=bigstorage project=default
t=2021-01-28T12:26:48+0000 lvl=info msg="Deleted container" created=2021-01-17T18:49:18+0000 ephemeral=false name=Mycontainer2/snap-Tutor-Jan-17-2020 project=default used=0001-01-01T00:00:00+0000
t=2021-01-28T12:26:48+0000 lvl=dbug msg="DeleteInstance started" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:26:48+0000 lvl=dbug msg="Deleting instance volume" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default volName=Mycontainer2
t=2021-01-28T12:29:30+0000 lvl=dbug msg="DeleteInstance finished" driver=btrfs instance=Mycontainer2 pool=bigstorage project=default
t=2021-01-28T12:29:31+0000 lvl=info msg="Deleted container" created=2021-01-28T11:41:46+0000 ephemeral=false name=Mycontainer2 project=default used=1970-01-01T00:00:00+0000
t=2021-01-28T12:29:31+0000 lvl=dbug msg="Failure for task operation: b95177fa-8e36-452c-8a5e-8f8b02af6cde: Create instance from copy: Create instance volume from copy failed: [Failed sending volume Mycontainer:/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619: Btrfs send failed: [signal: killed context canceled] (At subvol /var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619\n) Failed to run: btrfs receive -e /var/snap/lxd/common/lxd/storage-pools/bigstorage/containers/migration.411630795: At subvol e867eb55145b59ec709732c99bb73fd5d423b5133682d86006c452c938634619\nERROR: writing to openedx/edx-platform/conf/locale/ar/LC_MESSAGES/django.mo failed: No space left on device]"
t=2021-01-28T12:29:31+0000 lvl=dbug msg="Event listener finished: a5e97619-5068-4252-ae90-773ba599d983"
t=2021-01-28T12:29:31+0000 lvl=dbug msg="Disconnected event listener: a5e97619-5068-4252-ae90-773ba599d983"
t=2021-01-28T12:45:23+0000 lvl=dbug msg="New task Operation: 539ad785-2391-46f8-a3ab-a2ac01a364e4"
t=2021-01-28T12:45:23+0000 lvl=info msg="Pruning expired instance backups"
t=2021-01-28T12:45:23+0000 lvl=dbug msg="Started task operation: 539ad785-2391-46f8-a3ab-a2ac01a364e4"
t=2021-01-28T12:45:23+0000 lvl=info msg="Done pruning expired instance backups"
t=2021-01-28T12:45:23+0000 lvl=dbug msg="Success for task operation: 539ad785-2391-46f8-a3ab-a2ac01a364e4"

(Since the logs were too long to fit in the post, I took only an excerpt)

I tried creating a new lvm storage pool lvmpool with total space 25GB as well as a container Mycontainer2 using that pool. I configured Mycontainer2 in the same way as Mycontainer and installed the same software (Tutor EDX), docker containers, etc… After that I created a second lvm pool lvmpool2 with total space 25GB as well. I then copied Mycontainer2 to lvmpool2 and it worked (lxc copy Mycontainer2 Mycontainer2bis --storage=lvmpool2). The space used is around 9GB on both storage pools and there is no significant space expansion. I can also start Mycontainer2bis and everything works as in Mycontainer2. The storage driver used for docker inside Mycontainer2 and Mycontainer2bis is overlay2. I tried to export Mycontainer2 and the tarball is only 7GB big.

So I guess it is preferable that I stick to lvm storage driver for this container to avoid problems in the future. What are the advantages/drawbacks of lvm over btrfs when using LXD containers? (regarding snapshots for examples)

Glad to hear the ext4 on LVM is working better with docker (overlay2 driver is the preferred driver by docker so thats good too). It also means your container is exportable and will keep the docker containers working on import.

In your case both your BTRFS pools and your LVM pools are using a loop-back disk image on your host, in both cases you would get better performance if your pools were using a dedicated disk or partition rather than loop-back files, so no difference there. Equally both pool types are constrained by the size of the loopback files, so no difference there either.

One difference between LVM and BTRFS container volumes is that LVM always requires a hard volume size be specified (defaults to 10GB), whereas BTRFS being a filesystem rather than a volume manager, means that containers are free to use the entire pool space (unless you explicitly set a volume size quota). So be aware that if your LVM pool size is, say 30GB, your container will be limited to 10GB until you set the size property on the container’s root disk device (at which point the LVM volume and ext4 filesystem will be grown).

LVM supports snapshots, and these are at the volume level, so the docker system inside the container cannot take snapshots. In LXD, LVM pools by default use the LVM thinpool mode, which means that snapshots are efficient and can be long-lasting with no specific size. The other alternative is to use non-thin pool LVM pools (see lvm.use_thinpool at https://linuxcontainers.org/lxd/docs/master/storage), but these will use fixed size snapshots that can expire if too many changes occur in the parent volume.

An alternative would be to use the dir pool driver, this avoids having to set a hard disk size for the container, and instead just uses your host’s filesystem (assuming ext4). But then LXD container snapshots won’t be efficient as each snapshot will involve copying the container in its entirety not the difference.

I’ll take a look through the logs you sent and see if there is an issue with the way we using BTRFS to send the subvolumes.

1 Like

So I think I’ve figured out the problem here and its a limitation/bug of our current LXD BTRFS driver.

In your migration example above you have a container called Mycontainer with one snapshot called snap-Tutor-Jan-17-2020.

The way the optimized migration works is that it sends the snapshots first, in order from oldest to newest, and then references each previous snapshot as the parent to the next volume. This is so only the differences between the previous and current volumes are transferred by the BTRFS send/recv tools.

So the process should be:

  1. Send snapshot snap-Tutor-Jan-17-2020 with no parent (as it is the oldest snapshot).
  2. Send the main volume with the parent set to snap-Tutor-Jan-17-2020 so only differences between the snapshot and the main volume are transferred.

We can see this in action in the logs:

First the snapshot volume is transferred name=Mycontainer/snap-Tutor-Jan-17-2020 with parent= i.e no parent.

t=2021-01-28T11:43:34+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer/snap-Tutor-Jan-17-2020 parent= path=/ pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020

Then later the main volume is transferred name=Mycontainer with the parent set to the snapshot parent=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020:

t=2021-01-28T11:46:08+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/Mycontainer/snap-Tutor-Jan-17-2020 path=/ pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send

This is all good normally. However when using Docker with the BTRFS driver inside the container, it creates subvolumes (which are supported), but it does not look like they are transferred efficiently.

We can see this in progress. This is the first subvolume encountered in the main volume after the line above, and we can see the parent property is empty, unlike the line above that was using the snapshot as the parent.

t=2021-01-28T11:46:18+0000 lvl=dbug msg="Sending subvolume" driver=btrfs name=Mycontainer parent= path=/rootfs/var/lib/docker/btrfs/subvolumes/00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc pool=default source=/var/snap/lxd/common/lxd/storage-pools/default/containers/migration.594189230/.migration-send/rootfs/var/lib/docker/btrfs/subvolumes/00e678d666c6958379a5f602d56939922496e3d04efde776fbbd883cf4d7b9bc

My hypothesis is that we are only sending the main volume efficiently in relation to its snapshots, and that the full subvolumes are being sent, which would explain the size amplification on the target.

1 Like

I created this issue:

1 Like

Thanks a lot for your explanations and your help!