Using Ubuntu 16.04 LTS as host and in LXD containers with ZFS storage.
I wanted to test out reclaiming unused space. In a container I created a 10gb file with dd. Saw the space taken up in both the container as well as in the host.
Upon deleting the 10gb file the container shows that space is no longer used/allocated. Host also shows that the space is no longer used by the lxd container mount, however, the host OS still shows that space in use.
I’ve tried a variety of ways to get that unused space released, but have only really found one way which is not really optimal. So far, the way I’ve figured out how to do this is basically to shutdown & delete all containers, delete images, unset zfs.storage_pool_name, destroy the zfs pool and then finally delete the zfs image file at /var/lib/lxd/zfs.img Doing that last step is what finally releases the unused space. But that’s basically restarting from scratch, which isn’t really optimal.
Is there another way to reclaim that unused space in the host OS that will not require deleting containers, images, zfs pool or the zfs.img file?
ZFS file based pools won't become sparse again after you clear space inside the pool. This may change with the TRIM feature being added to ZFS now, but this may take months/years before all the plumbing is done to make this possible.
The only trick I can think of for your situation is to:
Create a second sparse file of the exact same size as the original one
Add it as RAID1 into the zpool
Remove the old sparse file from the zpool
Rename the file back to the original name so LXD will find it after reboot
This will temporarily require you to have twice the amount of space as is actually used in your zpool available for this operation, but should result in the new sparse file only containing what's actually used in your zpool.
If this somehow doesn't work (I've never tested it), then the other option I can think of is to stop all containers, unmount all the zfs filesystems from your existing pool, create a new sparse file, then create a new zpool on that sparse file and use zfs send/receive to move all the datasets from the old pool to the new one. Then delete the old zpool, rename the new zpool to the name of the old one and do the same with the underlying sparse file before restarting LXD.
[quote=“stgraber, post:2, topic:338, full:true”]The only trick I can think of for your situation is to:
Create a second sparse file of the exact same size as the original one
Add it as RAID1 into the zpool
Remove the old sparse file from the zpool
Rename the file back to the original name so LXD will find it after reboot
[/quote]
Thank you very much for this suggestion. I can confirm that this appears to have worked just fine.
For anyone who comes across this, here is exactly what I did to reclaim the unused space:
$ cd /var/lib/lxd
Create second sparse file (57G since that was the size of my original zfs.img file)
$ sudo dd if=/dev/zero of=zfs2.img bs=1 count=0 seek=57G
Add second sparse file as RAID1 into the lxd zpool
$ sudo zpool attach lxd /var/lib/lxd/zfs.img /var/lib/lxd/zfs2.img
Remove the old sparse file, zfs.img, from the zpool:
$ sudo zpool detach lxd /var/lib/lxd/zfs.img
Delete old sparse file zfs.img; unused space was reclaimed after doing this.
$ sudo rm /var/lib/lxd/zfs.img
Rename the file back to the original name so LXD will find it after reboot. (I was surprised that renaming the file did not break anything; everything continued humming along pre and post reboot after renaming that file.)
$ sudo mv /var/lib/lxd/zfs2.img /var/lib/lxd/zfs.img
To all those that stumble upon this problem. TRIM feature, mentioned by stgraber, in zfsutils-linux has been already implemented, at least in Ubuntu 20.04, zfs-0.8.3-1ubuntu12.7 and zfs-kmod-0.8.4-1ubuntu11.2.
To reclaim disk space used by zfs sparse file just issue the command
zpool trim name-of-zpool