Removeing images, they are marked as deleted but still consumes Disk

Hi, we are using LDX 2.0.11 and want to free up some disk. When I run zfs list I see that I have 54.1G in lxd/deleted/images

lxd/deleted                                                                          54.1G   280G    19K  none
lxd/deleted/images                                                                   54.1G   280G    19K  none
lxd/deleted/images/11fc1b1d39b9f9cd7e9491871f1421ac4278e1d599ecf5d180f2a6e2483bd172   690M   280G   690M  none
lxd/deleted/images/315bedd32580c3fb79fd2003746245b9fe6a8863fc9dd990c3a2dc90f4930039   668M   280G   668M  none
lxd/deleted/images/641797338216a0b974cbf55e377d3b19aef7bcc94859c99233dfe333ef4a0385  25.2G   280G  25.2G  none
lxd/deleted/images/683cdd3938706deeb66f19854ee27ef0c75e2594ce60ad4525357c3ce46f9772   685M   280G   685M  none
lxd/deleted/images/87a1f1c305615024124b238c0fcbed99c11193cab4bbc5c340ca2500c99b1b1f   686M   280G   686M  none
lxd/deleted/images/b9cba741ef621f14e6e8c306c7f87637a59d7d09c1a3ec8ef717ab0d6e43430a   669M   280G   669M  none
lxd/deleted/images/d76440766d75d9883f0133defdfc0b411954b28ac2c202628548d6b1ada9f0e0  2.10G   280G  2.10G  none
lxd/deleted/images/f4c9feb3e4018ffbd793d4b80d54fb95835bb1461380f2f9b7976c7b10ac48b9   670M   280G   670M  none
lxd/deleted/images/f992294c683130520bfe50ccac5568e6278d7057b1fd559bb0bb7e70b4a7af0c  22.1G   280G  22.1G  none
lxd/deleted/images/fce0f27114d37f3e0827e0c629aefcc32dc1c99af293c5306d43f3d5d50c6f29   688M   280G   688M  none

is there a way I can free this space?

No, unfortunately you can’t, the image will be kept in your zpool until the last container that was created from it goes away.

That’s because ZFS’ model for dataset cloning keeps a reference to the entire source dataset, not to individual blocks within the dataset (as btrfs does for example). Meaning that even if your container no longer shares a single block with the image it was originally created from, you still cannot destroy that image without also causing your container to get deleted…

When doing a zfs destroy (without -r!),you’ll see which container it blocks. Then I promoted the container dataset (zfs promote ). After that I was able to destroy the deleted image dataset.
Is this procedure wrong? So far it doesn’t seem to harm at least :wink:

Hmm, I have bad memories of trying to get zfs promote to help in those cases but I think in the case where you only have a single container tied to a particular deleted image, it may be fine.

If you have more than one container though, then it’d make the other containers depend on the one you promoted without LXD knowing about it, so that’d likely get you into troubles when then trying to delete that first container.

The @readonly snapshot will also get moved from the image to the container which may confuse LXD if you ever try to copy/move your container over the network to another ZFS system (which then uses zfs send/receive and would result in an unknown snapshot that LXD doesn’t know what to do with).

So anyway, zfs promote may be an option if you’re indeed dealing with a single container that was created off that now deleted image and you either find a way to clean any extra snapshots that appear as a result or don’t plan on migrating that container to another system, in other cases, it’d likely cause more harm than good.

1 Like

Thanks both for your time.

How do I check which container is tied to a image?

If I try to do zfs destory I get

# zfs destroy lxd/deleted/images/11fc1b1d39b9f9cd7e9491871f1421ac4278e1d599ecf5d180f2a6e2483bd172
cannot destroy 'lxd/deleted/images/11fc1b1d39b9f9cd7e9491871f1421ac4278e1d599ecf5d180f2a6e2483bd172': filesystem has children
use '-r' to destroy the following datasets:
lxd/deleted/images/11fc1b1d39b9f9cd7e9491871f1421ac4278e1d599ecf5d180f2a6e2483bd172@readonly

Maybe I asking to stupid. Better that then taking down our production server :slight_smile:

And what is zfs promote?

I think it doesn’t have a container attached to it anymore, but as Stephane noted, better leave it.

About promoting: normaly when creating a container it clones the image dataset, which is not more then just a pointer, so it doesn’t take up extra space. So you can run 1000 containers based on the same image, which uses just de diskspace for the image. When you change a container, the container diskspace usage will start to grow (difference between base cloned image and current state). Create for example a container and look at zfs list what the diskspace usage is (under the column used). It’s almost nothing compared to the base image size. Look now under the number in the column REFER. That will match closely the base image size.
Anyway back to promoting… when you would promote the dataset belonging to the container, then you kind of copy the whole base image into the container dataset, so it matches the complete container. Just promote a containers dataset and look at zfs list for the changes. The used size will be now close to the base imae size too.

Anyway comparing sizes in zfs is a bit more complex then I just tried to explain, but I hope you got an idea now :wink:

disclaimer: I am not a ZFS expert :wink:

Try deleting the snapshot first as that’s what’s used by the containers, so zfs destroy lxd/deleted/images/11fc1b1d39b9f9cd7e9491871f1421ac4278e1d599ecf5d180f2a6e2483bd172@readonly should tell you exactly what’s using that image.

I don’t think that ZFS recurses in that warning, so deleting the image itself only warns about the @readonly snapshot rather than show you the snapshot and anything that was based on it.

Ok, so when I run zfs destroy lxd/deleted/images/11fc1b1d39b9f9cd7e9491871f1421ac4278e1d599ecf5d180f2a6e2483bd172@readonly it tells me exacly wich container.

Thanks. This helps a lot. And I learned a lot. I wonder if I export the container, deleted is from the server and then import it again would it recreate the deleted image?

is possible to start a new container from a deleted image?
And if this is the name what do I do next
lxd/deleted/images/641797338216a0b974cbf55e377d3b19aef7bcc94859c99233dfe333ef4a0385

It’s not possible to create a container from a delete image as LXD no longer has any of the image metadata.
However if you find a server that does have the image, LXD will re-import it pretty quickly as it won’t have to re-create the zfs data.

Exporting a container and re-importing it will indeed remove the link to its parent image, as would moving the container to another host and back.

1 Like

I have this same problem, but don’t see that “lxc export” is available to me. I’m running lxd version 3.03, which is later than the version listed in this thread.

Can you elaborate on how I can export/import a container to have it purge the deleted image from ZFS?