My situation is a 3 node LXD 3.19 cluster that each have a zfs pool ‘data’ for local storage.
I can create several containers on node2 and they will each only use up 20MB worth of data because of their connection to the base ubuntu:18.04 image like such:
root@node2# zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 2.33G 120G 96K none
data/containers 1.61G 120G 96K none
data/containers/accepted-mullet 21.8M 120G 747M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/accepted-mullet
data/containers/careful-longhorn 21.8M 120G 747M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/careful-longhorn
data/containers/ceph-mgr3 21.8M 120G 747M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/ceph-mgr3
data/containers/ceph-mon3 21.8M 120G 746M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/ceph-mon3
data/containers/closing-wolf 21.8M 120G 747M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/closing-wolf
data/containers/knowing-dinosaur 21.8M 120G 747M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/knowing-dinosaur
data/containers/normal-kingfish 21.8M 120G 747M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/normal-kingfish
...
data/images 736M 120G 96K none
data/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185 736M 120G 736M /var/snap/lxd/common/lxd/storage-pools/zfs/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185
But once I move my 5 containers to node 1 so that I can do maintenance/reboot node2 all of a sudden these 5 containers take up an extra 3.5GB worth of space.
root@node1# zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 3.97G 918G 24K none
data/containers 3.32G 918G 24K none
data/containers/accepted-mullet 673M 918G 673M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/accepted-mullet
data/containers/careful-longhorn 673M 918G 673M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/careful-longhorn
data/containers/closing-wolf 673M 918G 673M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/closing-wolf
data/containers/knowing-dinosaur 673M 918G 673M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/knowing-dinosaur
data/containers/normal-kingfish 673M 918G 673M /var/snap/lxd/common/lxd/storage-pools/zfs/containers/normal-kingfish
...
data/images 663M 918G 24K none
data/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185 663M 918G 663M /var/snap/lxd/common/lxd/storage-pools/zfs/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185
Their connection to the base image is now forever lost.
Is this expected behavior, and if so, what can I do to prevent this? Do I need to setup deduplication on my zfs pool to win back the lost storage?