Resize LXD BTRFS storage space

Hi all,

This has been asked before but I have been unsuccessful.

LXD Verson: 3.0.3
LXD Storage: BTRFS

Some info:

root@lxd:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  816K  797M   1% /run
/dev/xvda1       59G   17G   42G  29% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop2       18M   18M     0 100% /snap/amazon-ssm-agent/930
/dev/loop0       18M   18M     0 100% /snap/amazon-ssm-agent/1335
tmpfs           100K     0  100K   0% /var/lib/lxd/shmounts
tmpfs           100K     0  100K   0% /var/lib/lxd/devlxd
/dev/loop4       14G   11G  3.1G  79% /var/lib/lxd/storage-pools/default
/dev/loop5       89M   89M     0 100% /snap/core/7169
/dev/loop1       89M   89M     0 100% /snap/core/7270
tmpfs           798M     0  798M   0% /run/user/0

I want to resize the BTRFS space by 20G.

I have tried:

root@lxd:~# btrfs filesystem resize 10g /var/lib/lxd/storage-pools/default
Resize '/var/lib/lxd/storage-pools/default' of '10g'
ERROR: unable to resize '/var/lib/lxd/storage-pools/default': No space left on device

Which can’t be right because there is 42G available.

How can I increase the LXD container storage space by 20G?

what is the output of
lxc storage show default

Sure, more info:

root@lxd:~# lxc storage list default
+---------+-------------+--------+--------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |             SOURCE             | USED BY |
+---------+-------------+--------+--------------------------------+---------+
| default |             | btrfs  | /var/lib/lxd/disks/default.img | 8       |
+---------+-------------+--------+--------------------------------+---------+
root@lxd:~# lxc storage show default
config:
  size: 15GB
  source: /var/lib/lxd/disks/default.img
description: ""
name: default
driver: btrfs
used_by:
- /1.0/containers/chisel
- /1.0/containers/docker
- /1.0/containers/haproxy
- /1.0/containers/unms
- /1.0/containers/url-rewrite
- /1.0/containers/zabbix
- /1.0/images/6ae1c6e92017402f1aee655fa8d785ee9d2337a3369d76115cecad5e7a303e07
- /1.0/profiles/default
status: Created
locations:
- none

That command is telling btrfs to resize /var/lib/lxd/storage-pools/default to be 10g large, which isn’t possible as it currently holds 11g of data.

I suspect what you want instead is:

  • Grow /var/lib/lxd/disks/default.img to be 25GB large
  • Run btrfs filesystem resize max /var/lib/lxd/storage-pools/default

That would then grow it to be 25GB

sudo truncate -s +20G /var/lib/lxd/disks/default.img
sudo btrfs filesystem resize max /var/lib/lxd/storage-pools/default

Note that the btrfs resize would not work with snap lxd.

Also, using a sparse file is the default but not the preferred option (!) to store containers. It’s advised to manage containers in a dedicated partition or even dedicated disk.

Can’t seem to grow the BTRFS img:

root@lxd:~# truncate -s +20G /var/lib/lxd/disks/default.img
root@lxd:~# lxc storage show default
config:
  size: 15GB
  source: /var/lib/lxd/disks/default.img
description: ""
name: default
driver: btrfs
used_by:
- /1.0/containers/chisel
- /1.0/containers/haproxy
- /1.0/containers/url-rewrite
- /1.0/containers/zabbix
- /1.0/images/6ae1c6e92017402f1aee655fa8d785ee9d2337a3369d76115cecad5e7a303e07
- /1.0/profiles/default
status: Created
locations:
- none

Still says 15GB

That size property is the originally requested size for the pool.
Look at lxc storage info default instead for the usage data.

Still says 15GB

root@lxd:~# truncate -s +20G /var/lib/lxd/disks/default.img
root@lxd:~# btrfs filesystem resize max /var/lib/lxd/storage-pools/default
Resize '/var/lib/lxd/storage-pools/default' of 'max'
root@lxd:~# lxc storage info default
info:
  description: ""
  driver: btrfs
  name: default
  space used: 7.28GB
  total space: 15.00GB
used by:
  containers:
  - chisel
  - haproxy
  - url-rewrite
  - zabbix
  images:
  - 6ae1c6e92017402f1aee655fa8d785ee9d2337a3369d76115cecad5e7a303e07
  profiles:
  - default

Well, you have to understand that usually in a project when it’s said that some way of doing something is ‘not recommended’, it’s a code word for ‘hideously bugged’. That’s why I have stopped using sparse files.

Anyway, here is a way of doing it.

First stop all containers and unset any containers set to automatic start.
Then

gp@raspberry:~$ lxc storage info mybtrfs
info:
  description: ""
  driver: btrfs
  name: mybtrfs
  space used: 17.24MB
  total space: 5.00GB
used by: {}
gp@raspberry:~$ sudo truncate -s +5GB /var/lib/lxd/disks/mybtrfs.img
gp@raspberry:~$ sudo systemctl stop lxd
gp@raspberry:~$ sudo systemctl disable lxd
gp@raspberry:~$ sudo reboot

(…)
gp@raspberry:~$ sudo losetup /dev/loop0 /var/lib/lxd/disks/mybtrfs.img
gp@raspberry:~$ sudo mount /dev/loop0 /mnt/disk1
gp@raspberry:~$ sudo btrfs fi show /mnt/disk1
Label: ‘mybtrfs’ uuid: a8add0a3-e384-499d-b339-4e0a51a58669
Total devices 1 FS bytes used 256.00KiB
devid 1 size 4.66GiB used 500.75MiB path /dev/loop0
gp@raspberry:~$ sudo btrfs fi resize max /mnt/disk1
Resize ‘/mnt/disk1’ of ‘max’
gp@raspberry:~$ sudo btrfs fi show /mnt/disk1
Label: ‘mybtrfs’ uuid: a8add0a3-e384-499d-b339-4e0a51a58669
Total devices 1 FS bytes used 256.00KiB
devid 1 size 9.31GiB used 500.75MiB path /dev/loop0
gp@raspberry:~$ sudo reboot
(…)
gp@raspberry:~$ sudo systemctl enable lxd
Synchronizing state of lxd.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable lxd
insserv: warning: current start runlevel(s) (empty) of script lxd' overrides LSB defaults (2 3 4 5). insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of scriptlxd’ overrides LSB defaults (0 1 6).
Created symlink /etc/systemd/system/multi-user.target.wants/lxd-containers.service → /lib/systemd/system/lxd-containers.service.
Created symlink /etc/systemd/system/sockets.target.wants/lxd.socket → /lib/systemd/system/lxd.socket.
gp@raspberry:~$ sudo systemctl start lxd
gp@raspberry:~$ lxc storage info mybtrfs
info:
description: “”
driver: btrfs
name: mybtrfs
space used: 17.24MB
total space: 10.00GB
used by: {}

If you have no way of restarting your computer, you can temporarily create another storage with something like

lxc storage create mybtrfs btrfs size=5GB

and you can create containers in the secondary storage with the -s mystorage switch
Unfortunately (IIRC) I don’t think there is an easy way to copy a container from a storage to another with the stable LXD version.

Ah yeah, this makes sense. Resizing the backing file for the loop device doesn’t resize the loop device itself, so btrfs can’t actually grow.

There may be some ioctls that can be used to tell the loop device itself to grow, avoiding the need for the reboot, but it’s not something I’ve got experience with.

@morphis this may explain what you saw the other day too, assuming you were using a loop device.

It’s possible to do a losetup -c /dev/loopx but while resizing the loop device all right, it has always locked hard the computer when trying to resize the file system.

Thanks for all the help. I think I will build a new LXD server with BTRFS in a partition instead of a file, just in case I need to do this again. How do we setup BTRFS in a partition for LXD to use?

On an existing LXD, you’d do something like:

  • lxc storage create blah btrfs source=/dev/sda1

That assumes an unformatted partition at /dev/sda1 and will create a pool named blah.
LXD will format the partition as btrfs and mount it as needed.