Resizing custom Ceph Volumes

Okay, so, I created a pair of custom volumes to be attached to a container:

lxc storage volume create ceph_pool DBTestSource size=1000GB
lxc storage volume create ceph_pool DBTestStorage size=1000GB

I then attached them to the container. All is well. However, later I found that the volumes were too small so I tried to grow them thusly:
lxc storage volume set ceph_pool DBTestSource size=1500GB
lxc storage volume set ceph_pool DBTestStorage size=1500GB

Thing is, no matter what I’ve tried the size of the file systems do not change. The various info commands swear they’ve been changed to 1.5TB. However, when I go in the container, no change. Resize2fs does nothing as the container doesn’t see the /dev/rbd* entries to be changed. Trying the same from the host results in a bad file system error. So, I’m stuck. Surely there has to be a way to do this, or should I just be doing it from the ceph side in the first place with rbd commands?

Have you tried restarting the container, in some circumstances the resize can’t happen safely until the container is stopped, and we record the chance to occur on next start up.

Can you show output of lxc config show <container> --expanded before rebooting as that will show if the resize has been scheduled for next boot.

Unfortunately, I already did restart the container. No effect :frowning:

That said, here’s the requested output anyway:
architecture: x86_64
config:
volatile.base_image: 94300c0d9cc90aeaafa460cf9a21bb5a0d6b7555a4d58d081890e0ecc8c53af9
volatile.eth0.host_name: veth96b87b25
volatile.eth0.hwaddr: 00:16:3e:d7:07:bc
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: RUNNING
devices:
DBTestSTorage:
path: /dbstorage
pool: ceph_pool
source: DBTestSTorage
type: disk
DBTestSource:
path: /dbsource
pool: ceph_pool
source: DBTestSource
type: disk
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: ceph_pool
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: “”

What did you mean by “The various info commands swear they’ve been changed to 1.5TB”, can you elaborate a bit on that please?

For instance, this:
config:
block.filesystem: ext4
block.mount_options: discard
size: 1500GB
volatile.idmap.last: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
description: “”
name: DBTestSource
type: custom
used_by:
- /1.0/containers/DBUpgradeTest

However, going into the container shows this:
root@DBUpgradeTest:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/rbd2 9.2G 1.9G 6.8G 22% /
none 492K 4.0K 488K 1% /dev
udev 7.8G 0 7.8G 0% /dev/tty
tmpfs 100K 0 100K 0% /dev/lxd
/dev/rbd0 916G 869G 688M 100% /dbsource
/dev/rbd1 916G 23M 870G 1% /dbstorage
tmpfs 100K 0 100K 0% /dev/.lxd-mounts
tmpfs 7.9G 12K 7.9G 1% /dev/shm
tmpfs 7.9G 188K 7.9G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup

The ones in question being /dbsource and /dbstorage in the df output

So I just tried this on LXD 4.1 and it seems to be working OK, here’s my test:

lxc storage show default
config:
  ceph.cluster_name: ceph
  ceph.osd.pg_num: "32"
  ceph.osd.pool_name: persist-edfu
  ceph.user.name: admin
  volatile.pool.pristine: "true"
description: ""
name: default
driver: ceph

lxc stop c1 # Existing container
lxc storage volume create default cust1
lxc storage volume attach default cust1 c1 cust1
lxc start c1

lxc exec c1 -- df -h | grep cust1
/dev/rbd1       9.2G   37M  8.6G   1% /cust1

lxc storage volume set default cust1 size=15GB
lxc exec c1 -- df -h | grep cust1
/dev/rbd1        14G   41M   13G   1% /cust1

lxc stop c1
lxc storage volume set default cust1 size=5GB

lxc start c1
lxc exec c1 -- df -h | grep cust1
/dev/rbd1       4.6G   33M  4.3G   1% /cust1

So that seems to be working OK.

What version of LXD are you running?

Also 4.1. Interesting. This is all with my work cluster. I’ll try it with my internal one and see if it behaves any differently

Are you using the snap, as I’m using the “stable” branch which may have some additional revisions beyond the 4.1 release tarball.

I am using snap, yeah