Reclaim Unused Space from /var/lib/lxd/zfs.img?


#1

Using Ubuntu 16.04 LTS as host and in LXD containers with ZFS storage.

I wanted to test out reclaiming unused space. In a container I created a 10gb file with dd. Saw the space taken up in both the container as well as in the host.

Upon deleting the 10gb file the container shows that space is no longer used/allocated. Host also shows that the space is no longer used by the lxd container mount, however, the host OS still shows that space in use.

I’ve tried a variety of ways to get that unused space released, but have only really found one way which is not really optimal. So far, the way I’ve figured out how to do this is basically to shutdown & delete all containers, delete images, unset zfs.storage_pool_name, destroy the zfs pool and then finally delete the zfs image file at /var/lib/lxd/zfs.img Doing that last step is what finally releases the unused space. But that’s basically restarting from scratch, which isn’t really optimal.

Is there another way to reclaim that unused space in the host OS that will not require deleting containers, images, zfs pool or the zfs.img file?

Thanks in advance for any help and advice.


(Stéphane Graber) #2

ZFS file based pools won't become sparse again after you clear space inside the pool.
This may change with the TRIM feature being added to ZFS now, but this may take months/years before all the plumbing is done to make this possible.

The only trick I can think of for your situation is to:

  • Create a second sparse file of the exact same size as the original one
  • Add it as RAID1 into the zpool
  • Remove the old sparse file from the zpool
  • Rename the file back to the original name so LXD will find it after reboot

This will temporarily require you to have twice the amount of space as is actually used in your zpool available for this operation, but should result in the new sparse file only containing what's actually used in your zpool.

If this somehow doesn't work (I've never tested it), then the other option I can think of is to stop all containers, unmount all the zfs filesystems from your existing pool, create a new sparse file, then create a new zpool on that sparse file and use zfs send/receive to move all the datasets from the old pool to the new one.
Then delete the old zpool, rename the new zpool to the name of the old one and do the same with the underlying sparse file before restarting LXD.


#3

Thank you very much for this suggestion. I can confirm that this appears to have worked just fine.

For anyone who comes across this, here is exactly what I did to reclaim the unused space:

$ cd /var/lib/lxd

Create second sparse file (57G since that was the size of my original zfs.img file)
$ sudo dd if=/dev/zero of=zfs2.img bs=1 count=0 seek=57G

Add second sparse file as RAID1 into the lxd zpool
$ sudo zpool attach lxd /var/lib/lxd/zfs.img /var/lib/lxd/zfs2.img

Remove the old sparse file, zfs.img, from the zpool:
$ sudo zpool detach lxd /var/lib/lxd/zfs.img

Delete old sparse file zfs.img; unused space was reclaimed after doing this.
$ sudo rm /var/lib/lxd/zfs.img

Rename the file back to the original name so LXD will find it after reboot. (I was surprised that renaming the file did not break anything; everything continued humming along pre and post reboot after renaming that file.)
$ sudo mv /var/lib/lxd/zfs2.img /var/lib/lxd/zfs.img

$ sudo reboot


Cannot restart LXD service after crash due to full disk
(Merlijn Sebrechts) #4

Thanks a lot @john, you’re a lifesaver! We tried these instructions and they worked perfectly.

Note that you can also put zfs2.img on a different disk, if there is not room on the main disk anymore.


#5

Another tip to create the 2nd sparse file: use truncate instead of dd. It costs no time and no physical space