Resize LXD container

I have a dedicated host running Ubuntu20 latest with LXD. I’m using LVM thin for containers pools.

I have tried to resize a running LXD container. Here’s some info:

root@s1 ~ # lvs
  LV                                                                      VG      Attr       LSize    Pool        Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LXDThinPool                                                             default twi-aotz--  789.62g                    1.18   2.29
  containers_vs1                                                          default Vwi-aotz-k <109.32g LXDThinPool        7.52
  images_d1cad2fbac21768f6ab2633a6e55c7fea118aba942dab0ab79c556ac5b1b149e default Vwi---tz-k   <9.32g LXDThinPool

root@s1 ~ # lvextend -L +125G default/containers_vs1
  Size of logical volume default/containers_vs1 changed from <109.32 GiB (27985 extents) to <234.32 GiB (59985 extents).
  Logical volume default/containers_vs1 successfully resized.

If I start the container I’m still seeing the initial disk size of 10G (the way the container was initially created)

How do I make that size available inside the container?!

1 Like

Don’t mess directly with LVM, instead just tell LXD what size you want on the disk device and LXD will resize the LV and filesystem for you.

Well I’ve already increased that LVM volume and using lvreduce is risky. How do I tell LXD exactly to do that since it’s not working… What I’ve tried so far:

lxc config device override vs1 root size=250G
lxc restart vs1

Inside the container I’m still seeing the 10G disk space.

And here’s the container’s info:

lxc config show vs1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20200921.1)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20200921.1"
  image.type: squashfs
  image.version: "20.04"
  volatile.base_image: 4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c
  volatile.eth0.host_name: vethe5a2ef7a
  volatile.eth0.hwaddr: 00:16:3e:1c:dc:f3
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices:
  root:
    path: /
    pool: default
    size: 250G
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Alternatively, would it be possible to create a new container, with a predefined disk size, create a full backup for the running container and then somehow restore the backup to the new container while keeping the disk size?

Look at the LXD logs, there may have been some kind of issue during the filesystem resize operation. What filesystem are you using?

Oh, also, if you’re using the snap package and aren’t running with lvm.external=true set on the snap, then what you did is a terrible idea :slight_smile:

That’s because LXD runs its own set of LVM tools with their own cache, so the LVM being used to run your container may not be aware of that resize operation at all and still see the old size…

In which case, I’d recommend snap set lxd lvm.external=true and then reboot the system to get into a clear LVM state and then investigate why things may still be the wrong size.

LVM, that’s how the server was initially created

root@s1 ~ # lxc storage list
+---------+-------------+--------+--------------------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |                   SOURCE                   | USED BY |
+---------+-------------+--------+--------------------------------------------+---------+
| default |             | lvm    | /var/snap/lxd/common/lxd/disks/default.img | 4       |
+---------+-------------+--------+--------------------------------------------+---------+
root@s1 ~ # lxc storage show default
config:
  lvm.thinpool_name: LXDThinPool
  lvm.vg_name: default
  size: 850GB
  source: /var/snap/lxd/common/lxd/disks/default.img
description: ""
name: default
driver: lvm
used_by:
- /1.0/images/7a88b8d840585bb405548c969cf5b60bbf03917ede20b64c5496b8406d6c7a99
- /1.0/images/d1cad2fbac21768f6ab2633a6e55c7fea118aba942dab0ab79c556ac5b1b149e
- /1.0/instances/vs1
- /1.0/profiles/default
status: Created
locations:
- none

root@s1 /var/log # lxc profile list
+---------+---------+
|  NAME   | USED BY |
+---------+---------+
| default | 1       |
+---------+---------+
root@s1 /var/log # lxc profile show
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/instances/vs1

Hopefully that helps!

Yes I’m using the snap packages, that’s correct

I have proceeded as instructed, executed snap set lxd lvm.external=true and rebooted the server. Not sure what I’m supposed to check now since the disk size inside the container is still 10G

lvs command is showing this:

root@s1 /var/log # lvdisplay
  --- Logical volume ---
  LV Name                LXDThinPool
  VG Name                default
  LV UUID                2zNSQN-c5cx-cvIE-9eBk-vd0x-NkkU-e9qpkS
  LV Write Access        read/write (activated read only)
  LV Creation host, time s1, 2020-10-05 11:45:30 +0200
  LV Pool metadata       LXDThinPool_tmeta
  LV Pool data           LXDThinPool_tdata
  LV Status              available
  # open                 2
  LV Size                789.62 GiB
  Allocated pool data    1.46%
  Allocated metadata     2.41%
  Current LE             202143
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/default/containers_vs1
  LV Name                containers_vs1
  VG Name                default
  LV UUID                bGrpbd-1P7e-RLHY-XtPT-vdmX-Xbma-SqgZlN
  LV Write Access        read/write
  LV Creation host, time s1, 2020-10-05 12:06:16 +0200
  LV Pool name           LXDThinPool
  LV Status              available
  # open                 1
  LV Size                <234.32 GiB
  Mapped size            3.51%
  Current LE             59985
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:4

  --- Logical volume ---
  LV Path                /dev/default/images_d1cad2fbac21768f6ab2633a6e55c7fea118aba942dab0ab79c556ac5b1b149e
  LV Name                images_d1cad2fbac21768f6ab2633a6e55c7fea118aba942dab0ab79c556ac5b1b149e
  VG Name                default
  LV UUID                hVBRUt-YTPi-UWwb-uDfd-grke-zZUv-yYpwrv
  LV Write Access        read/write
  LV Creation host, time s1, 2020-11-03 18:12:10 +0100
  LV Pool name           LXDThinPool
  LV Status              NOT available
  LV Size                <9.32 GiB
  Current LE             2385
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/default/images_7a88b8d840585bb405548c969cf5b60bbf03917ede20b64c5496b8406d6c7a99.block
  LV Name                images_7a88b8d840585bb405548c969cf5b60bbf03917ede20b64c5496b8406d6c7a99.block
  VG Name                default
  LV UUID                grWxPJ-PQQz-ZTLF-QA06-bF7y-Xvhd-9nJaXj
  LV Write Access        read/write
  LV Creation host, time s1, 2020-11-09 14:27:06 +0100
  LV Pool name           LXDThinPool
  LV Status              NOT available
  LV Size                <9.32 GiB
  Current LE             2385
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/default/images_7a88b8d840585bb405548c969cf5b60bbf03917ede20b64c5496b8406d6c7a99
  LV Name                images_7a88b8d840585bb405548c969cf5b60bbf03917ede20b64c5496b8406d6c7a99
  VG Name                default
  LV UUID                lccvnW-zdWV-wOti-uEXk-BIoO-hEfc-YfvGPC
  LV Write Access        read/write
  LV Creation host, time s1, 2020-11-09 14:27:07 +0100
  LV Pool name           LXDThinPool
  LV Status              NOT available
  LV Size                96.00 MiB
  Current LE             24
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

This is the default storage pool:
— Logical volume —
LV Name LXDThinPool
VG Name default

This seems to be assigned to the vs1 container:
— Logical volume —
LV Path /dev/default/containers_vs1
LV Name containers_vs1
VG Name default
LV UUID bGrpbd-1P7e-RLHY-XtPT-vdmX-Xbma-SqgZlN
LV Write Access read/write
LV Creation host, time s1, 2020-10-05 12:06:16 +0200
LV Pool name LXDThinPool
LV Status available
# open 1
LV Size <234.32 GiB

It beats me completely…

root@s1 /var/log # lvresize --resizefs --size +100G /dev/default/containers_vs1
  Size of logical volume default/containers_vs1 changed from <234.32 GiB (59985 extents) to <334.32 GiB (85585 extents).
  Logical volume default/containers_vs1 successfully resized.
resize2fs 1.45.5 (07-Jan-2020)
resize2fs: Device or resource busy while trying to open /dev/mapper/default-containers_vs1
Couldn't find valid filesystem superblock.
fsadm: Resize ext4 failed.
  /sbin/fsadm failed: 1
root@s1 /var/log # lvs
  LV                                                                            VG      Attr       LSize    Pool        Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LXDThinPool                                                                   default twi-aotz--  789.62g                    1.46   2.41
  containers_vs1                                                                default Vwi-aotz-k <334.32g LXDThinPool        2.46

Tried to alter the default profile and create a new container with a bigger disk size:

root@s1 /var/log # lxc profile device set default root size 100G

root@s1 /var/log # lxc profile show default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/instances/vs1-old

root@s1 /var/log # lxc launch -p default ubuntu:20.04 test
Creating test
Error: Failed instance creation: Create instance from image: Invalid value: 100G

Sorry but I don’t get it… Every documentation I read states pretty much the same stuff but applying, not working… What do I miss?!

100G is an invalid value, use 100GB or 100GiB

1 Like

Thanks for pointing that out. All the guides I have found were referring to 100G not 100GB or 100GiB. That was in fact the issue and the container was not starting.

I’ve managed to solve it.

Then only thing I had to type was:
lxc config device rm vs1 root size=100GB

and then restart the container. Size inside the container is the right one now.

Thank you for your help!

I see this repeated here on discuss all the time, but is there a document somewhere that explains why?

Cheers

When dealing with the snap at least, the version of LVM used by LXD inside of that environment may be different from the version of LVM you’re using on the host system.

Directly messing with the VG that LXD manages could lead to corruption of the LVM config.

So using the external tools is perfectly safe when it’s not installed from snap? (Our current one isn’t).

It’s just so confusing that there are so many people here stating “never use the OS tools” w/o an explanation, whereas if you e.g. stumble upon links to serverfault most people suggest that it’s the way to go.

Right, if not using the snap, then you don’t have the potential metadata corruption issue.

You will still hit a potential issue if you ever do migration or rely on project usage quotas as both of those require the size property to be set and match up with what’s in LVM. In the case of migration, that’s used to properly size the volume on the target, so it not matching will cause the migration to fail. In the project case, project quotas are evaluated against size information in the database, if you manually grow stuff, it won’t be represented in the usage.

2 Likes

Thank you for your patience with me and that explanation!

The other reason we generally say avoid using the OS tools for manipulating the storage volumes is because sometimes users remove/rename a storage volume without LXD’s knowledge (using the host OS’s tools), and this can prevent the instance from starting or even prevent LXD from starting, as LXD maintains a list of storage meta data in its database, and if they don’t match up it can cause problems that require manual database modifications.