Help me understand what's happening with my ZFS pool

1

Hi,

I am a relative novice at this stuff. I previously configured a disk backed ZFS pool but limited it to 50G in size (I was just trying ZFS). Recently, having 5 containers running in LXD (snap version) I started to run out of disk space.

When I deleted some unwanted files from the some of the containers I would see the Size and Used values drop in df within the container but not the Avail - eg:

root@dvr1:/tmp# df -h
Filesystem           Size  Used Avail Use% Mounted on
lxd/containers/dvr1  1.5G  1.1G  399M  73% /

Looking in zfs list, there for 4 of my 5 containers there are random entries that I don’t understand:

djwhyte@server1:/var/snap/lxd/common/lxd/storage-pools/lxd/containers$ sudo zfs list
NAME                                                                                  USED  AVAIL  REFER  MOUNTPOINT
lxd                                                                                  45.9G   358M    96K  none
lxd/containers                                                                       43.1G   358M    96K  none
lxd/containers/0a44f0cb-db69-45a8-b47f-e2aec201049e                                  28.1M   358M  4.69G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass035552651
lxd/containers/0b525e9b-315d-4d5f-973b-7b56e7439c44                                  28.7M   358M  5.11G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass554749637
lxd/containers/173ae9fc-ca4b-470c-abec-c498ded6be77                                  14.0M   358M  1.24G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1191624131
lxd/containers/219498ec-d581-4b8d-980a-8a93366bf9e0                                  73.2M   358M  15.7G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1717303870
lxd/containers/271fb553-95a5-4533-a54f-ddc778ec7145                                  13.7M   358M  1.21G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1594867246
lxd/containers/317479c8-e6cc-4b9e-b399-d497f0bb74da                                  13.7M   358M  1.21G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1025625384
lxd/containers/3bbcdf74-3b4d-4350-a70c-fff4c0903385                                  27.3M   358M  2.67G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass018082787
lxd/containers/43f807f1-1923-4839-9812-3cbc96abe69d                                  28.8M   358M  4.29G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass850153859
lxd/containers/45e13994-5c80-4ccb-83e7-c5a8c8772770                                  73.2M   358M  15.7G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1101613430
lxd/containers/5b31c557-0c27-4fd7-b02c-7f6c96d3f0f2                                  28.0M   358M  4.84G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass585734300
lxd/containers/64c8cf6a-e6f2-4c41-aff6-16550b4830fe                                  29.0M   358M  5.02G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass592371895
lxd/containers/713ae790-c21b-448b-9d39-1a43ae0c1b35                                  27.8M   358M  3.59G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass969443042
lxd/containers/741877da-f498-40be-934c-33268c0a2475                                  28.7M   358M  4.40G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass614097388
lxd/containers/77b56a71-3df7-4fff-a179-3c5eec823e33                                  13.8M   358M  1.19G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1507632008
lxd/containers/79ca1f7a-493d-4561-bc7a-d6c93681605c                                  27.3M   358M  2.72G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass804811689
lxd/containers/8553ad4b-40c6-42e3-8abe-d9e2ca74ac1b                                  18.6M   358M  1.09G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass352406120
lxd/containers/87307325-62fc-4784-b368-012e4297033b                                  13.7M   358M  1.21G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1024746997
lxd/containers/8c913682-21e9-4fb1-9841-40efda7d6987                                  28.0M   358M  4.94G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass198793062
lxd/containers/8da58310-af25-4b0e-8868-41092edf32cf                                  13.8M   358M  1.19G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1744592499
lxd/containers/ace62324-fb6b-4b89-a2c5-b6ff36d80d34                                  28.1M   358M  4.19G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass077438982
lxd/containers/af906ee5-5d04-418a-9d37-e0a843fe4b46                                  13.8M   358M  1.16G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1751432702
lxd/containers/b2285a61-94fb-460b-adec-eeec8077b00e                                  28.9M   358M  4.97G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass572585689
lxd/containers/b80a4b5d-44d2-42b7-97ee-3e7605d98d5e                                  13.9M   358M  1.16G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1210418861
lxd/containers/bd43719a-8336-4802-89f8-0f4f78a953c1                                  13.8M   358M  1.19G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1046126540
lxd/containers/cc7d50fa-c919-4eb6-87ee-bffd931e41bb                                  13.6M   358M  1.21G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1986119268
lxd/containers/db1                                                                    877M   358M  1.07G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/db1
lxd/containers/dedf60aa-446c-4a5b-aeeb-38b5159730be                                  48.7M   358M  15.8G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1720175480
lxd/containers/dvr1                                                                  17.1G   358M  1.01G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1
lxd/containers/f040b382-9fe1-4102-aea4-9be979eec676                                  13.9M   358M  1.16G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1332363071
lxd/containers/f4ae7592-e8de-4cb8-919a-ddc708871c5f                                  13.9M   358M  1.19G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1092427265
lxd/containers/f56af7e6-5494-45f9-9255-1c15a99c5605                                  14.0M   358M  1.24G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1618952205
lxd/containers/fdef72fb-fd26-4bf1-8b52-cef88fac4e67                                  28.5M   358M  4.85G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass841104906
lxd/containers/hass                                                                  9.92G   358M  2.51G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass
lxd/containers/plex                                                                  14.2G   358M  14.4G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/plex
lxd/containers/zabbix                                                                 264M   358M   535M  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/zabbix
lxd/deleted                                                                          2.54G   358M    96K  none
lxd/deleted/images                                                                   2.54G   358M    96K  none
lxd/deleted/images/84a71299044bc3c3563396bef153c0da83d494f6bf3d38fecc55d776b1e19bf9   414M   358M   414M  none
lxd/deleted/images/9879a79ac2b208c05af769089f0a6c3cbea8529571e056c82e96f1468cd1f610   411M   358M   411M  none
lxd/deleted/images/c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a   413M   358M   413M  none
lxd/deleted/images/db24c55f847e24dbe60fd665cf8f894ca1ace2d6419ba742f274966b94d4ca30   411M   358M   411M  none
lxd/deleted/images/e9239117192924904869ff135c1a795379af56e97128f54ba8e812d3831dee5e   954M   358M   954M  none
lxd/images                                                                            250M   358M    96K  none
lxd/images/4a89a29d4db20988ee3f0bcec5e054d1cde3d4255d2129208ef87d35cff8b464           250M   358M   250M  none

For my ‘plex’ container, there aren’t these random entries and deleting files in that container resulted in the Avail-able space to increase:

root@plex:~# df -h
Filesystem           Size  Used Avail Use% Mounted on
lxd/containers/plex   15G  7.5G  7.3G  51% /

Note also that not all of the mountpoints listed in zfs list are actually available on the filesystem:

djwhyte@server1:/var/snap/lxd/common/lxd/storage-pools/lxd/containers$ sudo ls -alh
total 32K
drwx--x--x 8 root    root    4.0K Oct  7 06:14 .
drwx--x--x 4 root    root    4.0K Sep 13  2018 ..
drwx--x--x 2  165536  165536 4.0K May 25  2018 db1
drwx--x--x 2 1000000 1000000 4.0K May 28  2018 dvr1
d--x------ 2  165536 root    4.0K May 25  2018 hass
drwx------ 2 root    root    4.0K Jul  9  2018 hass352406120
drwx--x--x 2 root    root    4.0K Sep 27  2018 plex
drwx--x--x 2 root    root    4.0K Jan 13  2019 zabbix

Can anyone explain what those random zfs list entries are - and whether I can remove them to recover space?

Thanks,
Whytey

3

That’s very weird, no idea what those UUID based entries are about.
Can you check that they’re not also listed in lxc storage volume list lxd?

If not, then those weird entries under lxd/containers/ should be safe to delete with zfs destroy.
If they’re somehow used as the parent of another entry, zfs destroy will complain loudly at that point.

zfs list -t all -o name,origin would also show any potential issue with such relationships.

4

@David_Whyte

Here are a couple links to help with ZFS management.

In this first one read the section under DF and DU as there is often confusion when using those commands with ZFS:

Confusion with used/free disk space in ZFS

https://www.oshogbo.vexillium.org/blog/65/

And this link has quite good information on ZFS management & use in general…

Use Your Storage Space More Effectively with ZFS: Exploring Datasets:

5

@bmullan

Thanks - those links help with understanding ZFS a bit more. It is kind of as I understand it, which is a bit akin to hard-links on a regular filesystem.

6

@stgraber

They were not listed under lxc storage volume list lxd

I have gone ahead and removed all the ‘dvr1’ entires and will monitor to ensure there is no erroneous behaviour before doing the same for the ‘hass’ ones.

Do you think there will be any issue with the zfs destroy command where the entry seems to be mounted in the filesystem?

djwhyte@server1:/var/snap/lxd/common/lxd/storage-pools/lxd/containers$ sudo zfs list 
NAME                                                                                  USED  AVAIL  REFER  MOUNTPOINT
lxd                                                                                  38.7G  7.56G    96K  none
lxd/containers                                                                       35.9G  7.56G    96K  none
lxd/containers/0a44f0cb-db69-45a8-b47f-e2aec201049e                                  28.1M  7.56G  4.69G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass035552651
lxd/containers/0b525e9b-315d-4d5f-973b-7b56e7439c44                                  28.7M  7.56G  5.11G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass554749637
lxd/containers/3bbcdf74-3b4d-4350-a70c-fff4c0903385                                  27.3M  7.56G  2.67G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass018082787
lxd/containers/43f807f1-1923-4839-9812-3cbc96abe69d                                  28.8M  7.56G  4.29G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass850153859
lxd/containers/5b31c557-0c27-4fd7-b02c-7f6c96d3f0f2                                  28.0M  7.56G  4.84G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass585734300
lxd/containers/64c8cf6a-e6f2-4c41-aff6-16550b4830fe                                  29.0M  7.56G  5.02G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass592371895
lxd/containers/713ae790-c21b-448b-9d39-1a43ae0c1b35                                  27.8M  7.56G  3.59G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass969443042
lxd/containers/741877da-f498-40be-934c-33268c0a2475                                  28.7M  7.56G  4.40G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass614097388
lxd/containers/79ca1f7a-493d-4561-bc7a-d6c93681605c                                  27.3M  7.56G  2.72G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass804811689
lxd/containers/8553ad4b-40c6-42e3-8abe-d9e2ca74ac1b                                  18.6M  7.56G  1.09G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass352406120
lxd/containers/8c913682-21e9-4fb1-9841-40efda7d6987                                  28.0M  7.56G  4.94G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass198793062
lxd/containers/ace62324-fb6b-4b89-a2c5-b6ff36d80d34                                  28.1M  7.56G  4.19G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass077438982
lxd/containers/b2285a61-94fb-460b-adec-eeec8077b00e                                  28.9M  7.56G  4.97G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass572585689
lxd/containers/db1                                                                    895M  7.56G  1.08G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/db1
lxd/containers/dvr1                                                                  17.1G  7.56G  1.01G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/dvr1
lxd/containers/fdef72fb-fd26-4bf1-8b52-cef88fac4e67                                  28.5M  7.56G  4.85G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass841104906
lxd/containers/hass                                                                  9.97G  7.56G  2.44G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/hass
lxd/containers/plex                                                                  7.22G  7.56G  7.44G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/plex
lxd/containers/zabbix                                                                 284M  7.56G   555M  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/zabbix
lxd/deleted                                                                          2.54G  7.56G    96K  none
lxd/deleted/images                                                                   2.54G  7.56G    96K  none
lxd/deleted/images/84a71299044bc3c3563396bef153c0da83d494f6bf3d38fecc55d776b1e19bf9   414M  7.56G   414M  none
lxd/deleted/images/9879a79ac2b208c05af769089f0a6c3cbea8529571e056c82e96f1468cd1f610   411M  7.56G   411M  none
lxd/deleted/images/c395a7105278712478ec1dbfaab1865593fc11292f99afe01d5b94f1c34a9a3a   413M  7.56G   413M  none
lxd/deleted/images/db24c55f847e24dbe60fd665cf8f894ca1ace2d6419ba742f274966b94d4ca30   411M  7.56G   411M  none
lxd/deleted/images/e9239117192924904869ff135c1a795379af56e97128f54ba8e812d3831dee5e   954M  7.56G   954M  none
lxd/images                                                                            250M  7.56G    96K  none
lxd/images/4a89a29d4db20988ee3f0bcec5e054d1cde3d4255d2129208ef87d35cff8b464           250M  7.56G   250M  none
djwhyte@server1:/var/snap/lxd/common/lxd/storage-pools/lxd/containers$ sudo ls -al
total 32
drwx--x--x 8 root    root    4096 Oct  7 06:14 .
drwx--x--x 4 root    root    4096 Sep 13  2018 ..
drwx--x--x 2  165536  165536 4096 May 25  2018 db1
drwx--x--x 2 1000000 1000000 4096 May 28  2018 dvr1
d--x------ 2  165536 root    4096 May 25  2018 hass
drwx------ 2 root    root    4096 Jul  9  2018 hass352406120
drwx--x--x 2 root    root    4096 Sep 27  2018 plex
drwx--x--x 2 root    root    4096 Jan 13  2019 zabbix

Thanks,
Whytey

7

Nope, those datasets are set with the noauto flag so even though they have a mountpoint set, they’re not expected to be mounted.