Unset fixes the issue, however, quite odd.
(rbd14): bad geometry: block count 3906250 exceeds size of device (3906048 blocks)
Hmm, I have containers on CEPH running with a size limit (25GB) without running into problems, looks like some odd rounding/alignment issue?
Yeah, but what’s interesting is if you unset it and add a root device on an existing container or even a new container with the option unset it works fine.
Also, when I first setup the cluster I used it to change the size of new containers when I added them and it worked with the older version by just setting the default.
Start and size are divisible by block size on all disks, so pretty sure not alignment. At least not on the physical disk. I do notice that all rbds start at 10000000000 / 4096 = 2441406.25, slight alignment problem there .
May 3 13:35:23 host kernel: [2074545.369380] rbd14: detected capacity change from 10000000000 to 79999008768 / 4096 = 19531008
May 3 13:35:23 host kernel: [2074545.406244] EXT4-fs (rbd14): mounted filesystem with ordered data mode. Opts: discard
May 3 13:35:23 host kernel: [2074545.408635] EXT4-fs (rbd14): resizing filesystem from 2441406 / 4096 = 596.046386719 to 19531008 / 4096 = 4768.3125 blocks
May 3 13:35:24 host kernel: [2074546.005227] EXT4-fs (rbd14): resized filesystem to 19531008
May 3 13:35:31 host kernel: [2074552.848105] EXT4-fs (rbd14): mounted filesystem with ordered data mode. Opts: discard
May 3 13:35:31 host kernel: [2074552.995079] EXT4-fs (rbd14): mounted filesystem with ordered data mode. Opts: discard
Doesn’t seem to matter what size you specify either, perhaps I broke something by changing it back and forth. It does tend to cause a pause in the lxc output, and I did get an error about a container I didn’t use anymore. I just deleted it, probably should have made a note about the error.
t=2019-05-03T01:14:46-0500 lvl=eror msg=“Failed to mount RBD storage volume for container “container_name”: invalid argument”
lxc profile device set default root size 26843545600B
lxc launch minimal test
Creating test
Error: Configure container: Failed to run: resize2fs /dev/rbd14 26214400K: resize2fs 1.42.13 (17-May-2015)
The containing partition (or device) is only 2441406 (4k) blocks.
You requested a new size of 6553600 blocks.
It’s definitely trying to use a device that’s already in use for the new container.
lrwxrwxrwx 1 root root 11 May 3 13:09 container_ContainerName -> …/…/rbd14