Can I access a container's files in a ZFS partition from host? (cannot run container due to error)

LXD
, ,
1

I have become locked out of my container as it has filled up its allotted disk space, giving me the following error:

Error: chmod /var/snap/lxd/common/lxd/containers/iotedgeserver: no space left on device

I have tried increasing the size with lxc volume config set size but this has not had any affect. I have also tried to restore an earlier snapshot, but received the annoying vague error message:

Error: In use

Is there any way I can access the ZFS partition to delete files and free up space?

2

Hi @esol-mike,

if you’re using ZFS, each container should have his own volume (you should see it when running zfs list on the lxd host). But those volumes are already mounted by LXD within its own namespace, so you’ll need to switch to this namespace to be able to work with the filesystem:

switch to the lxd namespace

nsenter -t $(cat /var/snap/lxd/common/lxd.pid) -m

work with the container’s rootfs directly

here for example, I’m using the default storage pool as ZFS:

cd /var/snap/lxd/common/lxd/storage-pools/default/containers/CONTAINER-NAME/rootfs
du -m --max-depth=1|sort -n

switch back to user namespace

just run exit or press CTRL+D

3

Running sudo systemctl reload snap.lxd.daemon will reset the mount counters that cause that In use error. Although it sounds like the operations you are attempting are not succeeding, causing the mount counter to be left in an invalid state.

Please can you reload LXD and then try setting the volume size again and capture any errors shown and the contents of /var/snap/lxd/common/lxd/logs/lxd.log

Please also show the output of lxc storage show <pool> and lxc storage volume show <pool> <volume>

4

Thank you for your help. This was exactly what I needed and it cleared up my problem (well, I couldn’t wait for du to run as I had 20GiB of 300k files in one directory thanks to idiot me’s bad code, but I could delete to directory)

1 Like
5

You may also have been able to do lxc exec <instance> -- rm ... and delete the files you wanted.

6

Thank you for your suggestion. If I have a similar problem in the future then I will try out your suggestion. This time I was able to use andrianaivo’s suggestion to solve my issue without increasing the space used by the container, which was the best possible solution for me.

As for pasting the lxc log files, this is a business computer and some of the information is sensitive. I was working on sanitizing the output and appending it to my post this morning, but andianaivo already replied before I got around to it, saving me the effort.

1 Like