Ghost ZFS images dataset clean up

LXD
1

Hi Guys,

I new at LXD and ZFS ecosystem, I notice and have several images dataset on my host, but I don’t have much images or containers running. It seems to be taken a lot of space.

# lxc list -f csv | wc -l
7

# lxc image list
+-------+--------------+--------+--------------------------------------+--------------+-----------+----------+-------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |             DESCRIPTION              | ARCHITECTURE |   TYPE    |   SIZE   |          UPLOAD DATE          |
+-------+--------------+--------+--------------------------------------+--------------+-----------+----------+-------------------------------+
|       | 9b2301ec680f | no     | Debian buster amd64 (20220224_06:50) | x86_64       | CONTAINER | 75.24MB  | Feb 24, 2022 at 10:42am (UTC) |
+-------+--------------+--------+--------------------------------------+--------------+-----------+----------+-------------------------------+
|       | 5025f5034c0a | no     | Centos 6 x86_64 (20201202_07:08)     | x86_64       | CONTAINER | 119.05MB | Apr 15, 2021 at 11:31am (UTC) |
+-------+--------------+--------+--------------------------------------+--------------+-----------+----------+-------------------------------+
|       | b031ef0d28a8 | no     | Centos 7 amd64 (20220224_07:08)      | x86_64       | CONTAINER | 84.49MB  | Feb 24, 2022 at 10:42am (UTC) |
+-------+--------------+--------+--------------------------------------+--------------+-----------+----------+-------------------------------+

# zfs list
zpool1/lxdpool1/images/010c79ad44b26b04456f2dbe716a18aa6030388b7a1bc19ed6012dd1ba649bd0                 415M  21.8G      415M  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/010c79ad44b26b04456f2dbe716a18aa6030388b7a1bc19ed6012dd1ba649bd0
zpool1/lxdpool1/images/01687841140c517eceef734a390a7f04c2972ad8078b3f21188f19a95322efa8                 415M  21.8G      415M  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/01687841140c517eceef734a390a7f04c2972ad8078b3f21188f19a95322efa8
zpool1/lxdpool1/images/01c29383240a18c93edbeeb9748e374572afcd3e0afe9c4fcdcac37fc025ac73                 415M  21.8G      415M  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/01c29383240a18c93edbeeb9748e374572afcd3e0afe9c4fcdcac37fc025ac73
zpool1/lxdpool1/images/02cd90bcc4670d6150d7565784f85816b1afc46e5c8d6301f77b87a01dca0722                 378M  21.8G      378M  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/02cd90bcc4670d6150d7565784f85816b1afc46e5c8d6301f77b87a01dca0722
zpool1/lxdpool1/images/031234bd21df2c33c307664703c075ed2e4c7aee2509feb157ed54b505a98c87                 268M  21.8G      268M  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/031234bd21df2c33c307664703c075ed2e4c7aee2509feb157ed54b505a98c87
zpool1/lxdpool1/images/0d04983fd90915e43037278ce4c3c79cbdfda7ef390bad8c794c879a4f6feaa0                  27K  95.3M       27K  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/0d04983fd90915e43037278ce4c3c79cbdfda7ef390bad8c794c879a4f6feaa0
zpool1/lxdpool1/images/0d04983fd90915e43037278ce4c3c79cbdfda7ef390bad8c794c879a4f6feaa0.block          4.01G  21.8G     4.01G  -
zpool1/lxdpool1/images/0dcbf4eae5f50dc6d7b844771141438108d3a78003fbfa4e256a13b23ca19598                 377M  21.8G      377M  /var/snap/lxd/common/lxd/storage-pools/lxdpool1/images/0dcbf4eae5f50dc6d7b844771141438108d3a78003fbfa4e256a13b23ca19598

# zfs list | grep images  | wc -l
242

I tried to add images.remote_cache_expiry: "1" and restart LXD but it doesn’t changed anything.

I would like know if someone can help me figure out what is causing this and how to do a clean up.

Some info:

OS: Ubuntu 20.04.1
LXD: 4.23

# lxc config show
config:
  core.https_address: 0.0.0.0
  core.trust_password: true
  images.remote_cache_expiry: "1"
2

When you create an instance on a storage pool from an image, the image is downloaded locally, and then unpacked into a volume on the storage pool (a so-called “image” volume). On storage pools like ZFS that support efficient snapshots, instances are then created by taking a snapshot of the image volume for the instance volume. This allows quick instance creation and efficient storage space, as only the differences between the instance and the image volume actually take up space.

So you maybe seeing image volumes because they are the basis of your instances.

3

Did you see that willie reported 425 zfs datasets when he only had 7 containers?

I had the same problem on several LXD servers. It turned out that these leftover zfs datasets had “zfs holds” from the backup software. That was why LXD could not delete them! After removing the holds (with zfs release) the question was: what images are still needed by lxd? I wrote the following bash function to find out:

show_active_lxd_images() {
    for project in . $(lxc project list -f csv | awk "-F |," '{print $1}'); do
        [ "$project" = . ] && continue
        for container in . $(lxc list -c n -f csv --project $project); do
            [ "$container" = . ] && continue
            echo "# $container (project $project)"
            lxc config show "$container" --project "$project" | grep base_image | sed -e 's/^.*: //'
        done
    done
    echo "# lxc image list -c F -f csv"
    lxc image list -c F -f csv
}

All I had to do was to destroy all zfs datasets for lxd images but those returned by this function.
Case closed. :slight_smile:

4

I had same issue with ghost images.
Local image cache cleanup - LXD - Linux Containers Forum

In your script, you can save two loops, by directly fetching base image:
output format: -f csv
—c, --columns: f for base image short, F for base image long

lxc ls --all-projects -f csv -c f

36acdd435250
47346cd1c1de
200424ff2f05
b3493d50c3da

lxc ls --all-projects -f csv -c F

36acdd435250cf6ec94cfefab3988ccd5fb642a9b297dfddd32f6e9ef17c0914
47346cd1c1def5e2a894518438c2d94f719baf37498f1bd86d1019cad94fb132
200424ff2f05b1c6a4c8a36b2394a6cc77f390222ade1bcad032df9982554518
b3493d50c3da36e38aa2268f91f66f64ad23c560e19998720faf270bda81adc8
f4c6939c4824ad15ffb9882464316ce3aa7f4ac95852ad7b0996a873eba818bf

5

Thanks @kamzar1 for your excellent suggestion. To just see the result my function can indeed be reduced to:

show_active_lxd_images() {
    (
        lxc image list -f csv -c F 
        lxc ls --all-projects -f csv -c F
    ) | sort -u
}
6