“lxc launch” fails due to “no space left on device”

LXD
1

Hello,
I’m trying to launch migrated container image on a remote server.
On my local system container consumed around 30GB

$ lxc storage info localpool75G
info:
description: “”
driver: btrfs
name: localpool75G
space used: 31.24GB
total space: 75.00GB

Exported into tar archive it’s about 11GB big.

On a remote machine i have quite a limited space, but according to reported values it should be enough:

$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 1.7M 1.6G 1% /run
/dev/mapper/vg-root 53G 16G 36G 31% / ← 36GB free on root with already imported image
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/loop0 55M 55M 0 100% /snap/core18/1705
/dev/loop1 55M 55M 0 100% /snap/core18/1754
/dev/loop2 70M 70M 0 100% /snap/lxd/15223
/dev/loop4 28M 28M 0 100% /snap/snapd/7264
/dev/loop3 70M 70M 0 100% /snap/lxd/15161
/dev/sda1 511M 4.0K 511M 1% /boot/efi
tmpfs 1.6G 8.0K 1.6G 1% /run/user/123
tmpfs 1.0M 0 1.0M 0% /var/snap/lxd/common/ns
tmpfs 1.6G 836K 1.6G 1% /run/user/1001

and 45GiB big lvm volume for lxc containers:

~$ lxc storage info lxcpool45G
info:
description: “”
driver: lvm
name: lxcpool45G
space used: 734.12MB
total space: 46.17GB
used by:
profiles:

  • default

And 16GB of ram if it’s important.

Imported image:

$ lxc image list
±-------±-------------±-------±-----------------------------------------±-------------±----------±-----------±------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
±-------±-------------±-------±-----------------------------------------±-------------±----------±-----------±------------------------------+
| migbox | fa8f23ab1d8c | no | Archlinux x86_64 (20200325_04:18) | x86_64 | CONTAINER | 11231.65MB | May 28, 2020 at 7:41pm (UTC) |

Host os on remote machine differs from used in container (host ubuntu - container arch linux). On my local machine it matches container os (both arch linux).

So, when i try to launch imported migbox on remote machine i receive
No space left on device error and i don’t understand why.

Is there a way to precisely find out how much space do i need and at which point (root or lxcpool or maybe both)?

2

You’ll need to increase your volume.size on your pool so the LV created to receive your container can be large enough.

3

Thank you, but seems it didn’t helped.
Before setting i’ve checked it and by default value is empty.

$ lxc profile device get default root size

And by issuing this command i’ve set it nearly to the size of LV:

lxc profile device set smartex:default root size 46GB

However, after running launch i received the same error…

…
tar: rootfs/var: Cannot mkdir: No space left on device
tar: rootfs/var/log/btmp.1: Cannot open: No such file or directory
tar: rootfs/var: Cannot mkdir: No space left on device
tar: rootfs/var/log/journal: Cannot mkdir: No such file or directory
tar: rootfs/var: Cannot mkdir: No space left on device
tar: rootfs/var/log/journal/30e67dbb3d304b198d2c59c137e73a13: Cannot mkdir: No such file or directory
tar: rootfs/var: Cannot mkdir: No space left on device
…
tar: Exiting with failure status due to previous errors.

UPDATE Just tried to launch default container and received such errors:

$ lxc launch images:archlinux/current/amd64 xforward
Creating xforward
Error: Failed instance creation: Create instance from image: Invalid value: 46G
$ lxc profile device set default root size 46
$ lxc launch images:archlinux/current/amd64 xforward
Creating xforward
Error: Failed instance creation: Create instance from image: Failed to run: resize2fs /dev/lxcvg/containers_xforward 0K: resize2fs 1.44.1 (24-Mar-2018)
resize2fs: Invalid new size: 0K

Huh, GB is the right one, obviously )

$ lxc profile device set default root size 46GB

After fixing it, error haven’t changed, so i assume it’s related to host root size?
However, checking disk space while lxc launch still is running (and throwing errors) gives nothing useful as it shows free space like nothing is happening - same as before.-Same with pool-.
UPDATE2
Actually pool size changes while unpacking on launch.
Before error:

$ lxc storage info lxcpool45G
info:
description: “”
driver: lvm
name: lxcpool45G
space used: 10.90GB
total space: 46.17GB
used by:
containers:

  • migbox
    profiles:
  • default

And on error it flushes:

$ lxc storage info lxcpool45G
info:
description: “”
driver: lvm
name: lxcpool45G
space used: 734.12MB
total space: 46.17GB
used by:
containers:

  • migbox ← automatically gets remove afterwards
    profiles:
  • default
4

Can you show output of lxc storage show lxcpool45G please?

5

$ lxc storage show lxcpool45G
config:
lvm.thinpool_name: lxcvg_thinpool
lvm.use_thinpool: “true”
lvm.vg.force_reuse: “true”
lvm.vg_name: lxcvg
source: lxcvg
volatile.initial_source: lxcvg
description: “”
name: lxcpool45G
driver: lvm
used_by:

  • /1.0/containers/xforward
  • /1.0/images/13f288e5715fc6a7c97ed32e8378ff985898efb4a876163b32359eba7d2106e5
  • /1.0/profiles/default
    status: Created
    locations:
  • none
6

OK so you’re trying to launch a container onto a storage pool backed by LVM?

Please can you show me the following things:

  1. The command you are running that is failing (full command including arguments).
  2. The output of lvs command

Thanks

7

That’s right.

  1. lxc launch migbox testbox
  2. sudo lvs
    LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
    lxcvg_thinpool lxcvg twi-aotz-- 43.00g 0.00 1.59
    root vgroot -wi-ao---- <53.54g
    swap_1 vgroot -wi-ao---- 980.00m

p.s. i have removed unnecessary images from lxc

8

Please try as @stgraber suggested:

lxc storage set lxcpool45G volume.size 40GB

So that new LVM volumes are created with sufficient size to accommodate the image.

1 Like
9

Thank you @tomp for providing command example and @stgraber for pointing this out. I though that command i have found do that (lxc profile device set default root size 46GB), but i’ve made a mistake. Now container was launched with no problems.