Data usage on ZFS storage doesn't add up

henning · April 6, 2023, 5:20pm

Hi there,

I’m using ZFS to as a storage back end for my containers. Here are the details:

zpool list -v
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
local      9.50G  7.66G  1.84G        -         -    85%    80%  1.00x    ONLINE  -
  sdb      9.50G  7.66G  1.84G        -         -    85%  80.6%      -    ONLINE

zfs list
NAME                                                                            USED  AVAIL     REFER  MOUNTPOINT
local                                                                          7.65G  1.55G       24K  legacy
local/buckets                                                                    24K  1.55G       24K  legacy
local/containers                                                               1.91G  1.55G       24K  legacy
local/containers/vps2196                                                       33.8M  1.55G      421M  legacy
local/containers/vps2211                                                       22.5M  1.55G      407M  legacy
local/containers/vps2226                                                       20.6M  1.55G      405M  legacy
local/containers/vps2240                                                       21.1M  1.55G      405M  legacy
local/containers/vps3005                                                       26.2M  1.55G      410M  legacy
local/containers/vps3025                                                       30.1M  1.55G      414M  legacy
local/containers/vps3036                                                       20.0M  1.55G      404M  legacy
local/containers/vps3060                                                       30.8M  1.55G      415M  legacy
local/containers/vps3070                                                       95.5M  1.55G      480M  legacy
local/containers/vps3089                                                       49.5M  1.55G      433M  legacy
local/containers/vps3101                                                       42.1M  1.55G      426M  legacy
local/containers/vps3116                                                       33.9M  1.55G      418M  legacy
local/containers/vps3127                                                       20.7M  1.55G      405M  legacy
local/containers/vps3141                                                       20.9M  1.55G      405M  legacy
local/containers/vps3159                                                       44.7M  1.55G      429M  legacy
local/containers/vps3175                                                       25.5M  1.55G      413M  legacy
local/containers/vps3194                                                       22.9M  1.55G      407M  legacy
local/containers/vps3219                                                       22.7M  1.55G      407M  legacy
local/containers/vps3238                                                       48.8M  1.55G      433M  legacy
local/containers/vps3248                                                       31.1M  1.55G      415M  legacy
local/containers/vps3260                                                       34.1M  1.55G      418M  legacy
local/containers/vps3270                                                       22.3M  1.55G      406M  legacy
local/containers/vps3291                                                       84.7M  1.55G      472M  legacy
local/containers/vps3299                                                       24.6M  1.55G      409M  legacy
local/containers/vps3310                                                       23.3M  1.55G      407M  legacy
local/containers/vps3320                                                       22.6M  1.55G      407M  legacy
local/containers/vps3334                                                       23.4M  1.55G      407M  legacy
local/containers/vps3348                                                       24.7M  1.55G      409M  legacy
local/containers/vps3384                                                       26.9M  1.55G      411M  legacy
local/containers/vps3399                                                       25.6M  1.55G      410M  legacy
local/containers/vps3403                                                       28.5M  1.55G      412M  legacy
local/containers/vps3408                                                       22.6M  1.55G      407M  legacy
local/containers/vps3426                                                       64.2M  1.55G      448M  legacy
local/containers/vps3437                                                       22.3M  1.55G      406M  legacy
local/containers/vps3451                                                       22.9M  1.55G      407M  legacy
local/containers/vps3465                                                       45.6M  1.55G      430M  legacy
local/containers/vps3477                                                       25.3M  1.55G      409M  legacy
local/containers/vps3488                                                       22.3M  1.55G      406M  legacy
local/containers/vps3500                                                       22.5M  1.55G      406M  legacy
local/containers/vps3506                                                       22.7M  1.55G      407M  legacy
local/containers/vps3515                                                       22.7M  1.55G      410M  legacy
local/containers/vps3538                                                       22.3M  1.55G      406M  legacy
local/containers/vps3547                                                       55.7M  1.55G      440M  legacy
local/containers/vps3561                                                       22.3M  1.55G      406M  legacy
local/containers/vps3584                                                       22.4M  1.55G      406M  legacy
local/containers/vps3623                                                       22.3M  1.55G      406M  legacy
local/containers/vps3638                                                       26.7M  1.55G      411M  legacy
local/containers/vps3666                                                       29.8M  1.55G      414M  legacy
local/containers/vps3692                                                       53.1M  1.55G      437M  legacy
local/containers/vps3932                                                       20.4M  1.55G      404M  legacy
local/containers/vps3941                                                       57.6M  1.55G      442M  legacy
local/containers/vps3952                                                       20.1M  1.55G      404M  legacy
local/containers/vps3961                                                       21.2M  1.55G      405M  legacy
local/containers/vps3976                                                       20.0M  1.55G      404M  legacy
local/containers/vps3985                                                       24.6M  1.55G      409M  legacy
local/containers/vps3991                                                       93.7M  1.55G      481M  legacy
local/containers/vps4015                                                       21.6M  1.55G      409M  legacy
local/containers/vps4048                                                       17.7M  1.55G      405M  legacy
local/containers/vps4061                                                       26.9M  1.55G      415M  legacy
local/containers/vps4066                                                       31.9M  1.55G      420M  legacy
local/containers/vps4081                                                       17.6M  1.55G      405M  legacy
local/containers/vps4097                                                       17.2M  1.55G      405M  legacy
local/containers/vps4107                                                       17.0M  1.55G      405M  legacy
local/custom                                                                     24K  1.55G       24K  legacy
local/deleted                                                                   144K  1.55G       24K  legacy
local/deleted/buckets                                                            24K  1.55G       24K  legacy
local/deleted/containers                                                         24K  1.55G       24K  legacy
local/deleted/custom                                                             24K  1.55G       24K  legacy
local/deleted/images                                                             24K  1.55G       24K  legacy
local/deleted/virtual-machines                                                   24K  1.55G       24K  legacy
local/images                                                                   1.01G  1.55G       24K  legacy
local/images/c481340385da83daaac66d4d707621787a859a2ad5bafdb8e2403fdc723881e4   403M  1.55G      403M  legacy
local/images/ed7509d7e83f29104ff6caa207140619a8b235f66b5997f1ed6c5e462617fb71   627M  1.55G      627M  legacy
local/virtual-machines                                                           24K  1.55G       24K  legacy

zfs list says that in local 7.66G are used but I don’t really see where that data is.

local/containers has 1.91G and local/images has 1.01G which adds up to around 3G.

Every other folder shown has barely any data and can be neglected.

Any idea where the missing ~4.65G of used data might be?

stgraber · April 6, 2023, 8:34pm

any snapshots?

henning · April 6, 2023, 10:27pm

Only the images

zfs list -t snapshot
NAME                                                                                     USED  AVAIL     REFER  MOUNTPOINT
local/images/c481340385da83daaac66d4d707621787a859a2ad5bafdb8e2403fdc723881e4@readonly     1K      -      403M  -
local/images/ed7509d7e83f29104ff6caa207140619a8b235f66b5997f1ed6c5e462617fb71@readonly     1K      -      627M  -

But those are shown by zfs list above and included in my calculation resulting in around 3G of data used.

henning · April 19, 2023, 8:07pm

One idea on this:

I’m creating all my containers based on a custom image (using lxc publish to create the image and lxc launch <image> <container> to create a new container). From time to time I create a new version of that image and new containers are created from that updated image after that.

I don’t know how that works exactly but I would a assume that all versions of that image are stored somewhere at least as long as one container is based on a certain version, right?

Where is that images and all of it’s versions stored? Aren’t all that files shown by zfs list?