Can't start container -> no space left on device

Hi,

today I’ve restarted my webserver LXC container, but now I can’t start it… I am getting this error

Error: Common start logic: Set volatile.last_state.idmap config key on container "webserver" (id 2): Failed to write backup file: open /var/lib/lxd/containers/webserver/backup.yaml: no space left on device
Try `lxc info --show-log webserver` for more info

So when I check disks capacity via df -h I see that it should be OK

/dev/loop5       38G  7.5G   29G  21% /var/lib/lxd/storage-pools/nextcloud
/dev/loop6       40G   22G   19G  54% /var/lib/lxd/storage-pools/webserver

So when I look to the log I can see this

lxc info --show-log webserver 

Name: webserver
Remote: unix://
Architecture: x86_64
Created: 2021/07/23 13:16 UTC
Status: Stopped
Type: persistent
Profiles: webserver
Snapshots:
  s1 (taken at 2021/08/19 06:55 UTC) (stateless)

Log:

lxc webserver 20210820072235.854 WARN     conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc webserver 20210820072925.799 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 12

I also tried remount the pool and restart the whole server. But still have the same error…

Thanks for ideas

Please show output of lxc config show <instance> --expanded

lxc config show webserver --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20210720)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20210720"
  image.version: "20.04"
  volatile.base_image: cea91a28441af36f088f3ae8b637499061518ea2d83c7560a2f7f33d74298968
  volatile.eth0.hwaddr: 00:16:3e:c1:19:e3
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: STOPPED
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: webserver
    type: disk
  vps-conf:
    path: /vps-conf
    source: /vps-conf
    type: disk
ephemeral: false
profiles:
- webserver
stateful: false
description: ""

I have also snapshot from yesterday but If I try to restore them, I get the similar err about the space.

Can you show lxc storage show webserver please

config:
  size: 17GB
  source: /var/lib/lxd/disks/webserver.img
description: ""
name: webserver
driver: btrfs
used_by:
- /1.0/containers/kmfire
- /1.0/containers/kmfire/snapshots/snap0
- /1.0/containers/mysql1
- /1.0/containers/webserver
- /1.0/containers/webserver/snapshots/s1
- /1.0/images/9c939141a94bbdcfd29b0b3be6dc135b9d3e8d73b19235f06de4593898e8b960
- /1.0/images/fab57376cf04b817d43804d079321241ce98d3b5c2296f1a41541de6c100ab09
- /1.0/profiles/default
- /1.0/profiles/kmfire
- /1.0/profiles/webserver
status: Created
locations:
- none

I see that the snapshot is also there, so it means that every snapshot has the same size as container itself?

Has the /var/lib/lxd/disks/webserver.img file been manually increased from its original 17GB in the past?

Can you show output of sudo ls -la /var/lib/lxd/disks/webserver.img

yes if I remember correct, it was… I have saved the steps for increasing the pool, so it was increased by this way

sudo truncate -s +5G /var/lib/lxd/disks/<POOL>.img
sudo losetup -c <LOOPDEV>
sudo btrfs filesystem resize max /var/lib/lxd/storage-pools/<POOL>/

Can you copy a file, say /var/lib/lxd/containers/webserver/backup.yaml to /var/lib/lxd/storage-pools/webserver? Just to test if it is a general pool problem or a specific subvolume issue.

sudo cp /var/lib/lxd/containers/webserver/backup.yaml /var/lib/lxd/storage-pools/webserver/
cp: cannot create regular file '/var/lib/lxd/storage-pools/webserver/backup.yaml': No space left on device

What filesystem is /var/lib/lxd/disks/ on and how much space (if any) does it have?

how do I check the filesystem of this dir?

the sizes are

/var/lib/lxd ----------------------------------------------------------------------------------------
   58.1 GiB [##########] /storage-pools
   32.2 GiB [#####     ] /disks
    1.7 GiB [          ] /devices

I am little bit confused by it, because it says, that there are files about 90GB and my host hdd has 90GB… but in df -h it seems OK…

Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           395M  1.1M  394M   1% /run
/dev/vda1        89G   48G   38G  56% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/loop0       33M   33M     0 100% /snap/snapd/12704
/dev/loop1       33M   33M     0 100% /snap/snapd/12398
/dev/loop2       91M   91M     0 100% /snap/go/7954
/dev/loop3       56M   56M     0 100% /snap/core18/2128
/dev/loop4       56M   56M     0 100% /snap/core18/2074
overlay          89G   48G   38G  56% /var/lib/docker/overlay2/d208fc0c7b2dc514ffa3883aa49b0628b3021d63d600c4106e5cc941a339acbc/merged
tmpfs           395M     0  395M   0% /run/user/1000
tmpfs           100K     0  100K   0% /var/lib/lxd/shmounts
tmpfs           100K     0  100K   0% /var/lib/lxd/devlxd
/dev/loop5       40G   22G   19G  54% /var/lib/lxd/storage-pools/webserver
/dev/loop6       38G  7.5G   29G  22% /var/lib/lxd/storage-pools/nextcloud
1 Like

Can you show full output of df -h please

Can you do this?

sudo ls -la /var/lib/lxd/disks/webserver.img
-rw------- 1 root root 42769803776 Aug 20 09:34 /var/lib/lxd/disks/webserver.img

ok, I see it… when I run ncdu on the root

93.6 GiB [##########] /var

So probably I dont understand the capacity around containers… because always check it with df -h which is probably not correct

What filesystem is / (and by extension /var) on btw?

how can I check it?

But now I understand, the problem happened when I’ve created some snapshots yesterday, so now the host hdd is full. I am trying to increase them with the provider of vps. I was just confused that other container did start but only the webserver did not.

cat /proc/mounts