Creating logical volume inside Fedora/CentOS LXC container

Hi all,

I really need to create a logical volume inside a linux container and with sufficient permissions have managed to create a volume group but encounter permission-related error on LV creation:

$ lvcreate -L 30G --thin ramdisk_vm1/thin_pool
setpriority -18 failed: Permission denied
/dev/ramdisk_vm1/lvol0: not found: device not cleared
Aborting. Failed to wipe start of new LV.
setpriority -18 failed: Permission denied

I have added container permissions

lxc.aa_profile = unconfined
lxc.cgroup.devices.allow = b 7:* rwm
lxc.cgroup.devices.allow = c 10:237 rwm
lxc.cgroup.devices.allow = c 10:236 rwm
lxc.cgroup.devices.allow = b 252:* rwm

and have loaded the kernel module

$ sudo modprobe dm-thin-pool
$ sudo lsmod | grep thin
dm_thin_pool           69632  1
dm_persistent_data     73728  1 dm_thin_pool
dm_bio_prison          20480  1 dm_thin_pool

in order to avoid problems with the thin provisioning.

I have tried this on both CentOS 7 and Fedora 25 containers with CentOS 7 and Fedora 25 host systems in order to rule out kernel version as a possible cause. The created containers are privileged and created in the most default way possible with the above parameter additions (and a few for networking). Scouring the internet only brought to one result where it was suggested that it could be LVM version mismatch. I obtained the error also in the cases of same container and host OS versions.

Any help is greatly appreciated!

Did you try to strace it to see exactly what’s throwing errors?

It looks like part of the problem is that LVM here is expecting device nodes to auto-appear through devtmpfs, which isn’t the case for containers as those used a normal tmpfs filesystems and so require direct device node creation.

I seem to remember there being a lot of settings in lvm.conf around device discovery so maybe turning off a bunch of that stuff will make it behave.

Thank you for the great hints! Setting the following in container’s /etc/lvm/lvm.conf solved part of the problem and can be used for future reference for everyone else that stumbles upon this:

udev_rules = 0

In this way LVM will handle the nodes internally and the above command will work but only sometimes. The actual output of the command changes to

$ lvcreate -L 30G --thin ramdisk_vm1/thin_pool
setpriority -18 failed: Permission denied
Aborting. Failed to wipe start of new LV.
setpriority -18 failed: Permission denied

The area around the setpriority -18 failed: Permission denied messages shows through strace

getpriority(PRIO_PROCESS, 0)            = 20
setpriority(PRIO_PROCESS, 0, -18)       = -1 EACCES (Permission denied)
write(2, "  ", 2)                       = 2
write(2, "setpriority -18 failed: Permissi"..., 41) = 41
write(2, "\n", 1)                       = 1

The process is trying to increase its priority from 20 to -18 and is denied this. Changing the setting

process_priority = -18

to 0 removed these messages although it is not something preferable and the error reduces to

$ lvcreate -L 30G --thin _vm1_ramdisk/thin_pool
File descriptor 6 (anon_inode:[signalfd]) leaked on lvcreate invocation. Parent PID 5: /bin/bash
  Aborting. Failed to wipe start of new LV.

which then can be straced to

stat("/dev/_vm1_ramdisk", 0x7ffec8b62a70) = -1 ENOENT (No such file or directory)
umask(022)                              = 077
mkdir("/dev/_vm1_ramdisk", 0777)        = 0
umask(077)                              = 022
lstat("/dev/_vm1_ramdisk/group", 0x7ffec8b629b0) = -1 ENOENT (No such file or directory)
lstat("/dev/_vm1_ramdisk/lvol0", 0x7ffec8b629b0) = -1 ENOENT (No such file or directory)
symlink("/dev/mapper/_vm1_ramdisk-lvol0", "/dev/_vm1_ramdisk/lvol0") = 0
stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(10, 236), ...}) = 0
open("/dev/mapper/control", O_RDWR)     = 4
ioctl(4, DM_DEV_STATUS, {version=4.0.0, data_size=16384, uuid="LVM-dtzs1fKrihUVQwkanojDA6UxNPSArz0Y8eAhnUpRh57aDe7zOs4PqfvqWxSd9Fo2", flags=DM_EXISTS_FLAG|DM_SKIP_BDGET_FLAG} => {version=4.33.0, data_size=312, dev=makedev(253, 8), name="_vm1_ramdisk-lvol0", uuid="LVM-dtzs1fKrihUVQwkanojDA6UxNPSArz0Y8eAhnUpRh57aDe7zOs4PqfvqWxSd9Fo2", target_count=1, open_count=0, event_nr=0, flags=DM_EXISTS_FLAG|DM_ACTIVE_PRESENT_FLAG|DM_SKIP_BDGET_FLAG}) = 0
stat("/dev/_vm1_ramdisk/lvol0", {st_mode=S_IFBLK|0660, st_rdev=makedev(253, 8), ...}) = 0
stat("/dev/_vm1_ramdisk/lvol0", {st_mode=S_IFBLK|0660, st_rdev=makedev(253, 8), ...}) = 0
open("/dev/_vm1_ramdisk/lvol0", O_RDWR|O_DIRECT|O_NOATIME) = -1 EPERM (Operation not permitted)
open("/dev/_vm1_ramdisk/lvol0", O_RDWR|O_DIRECT) = -1 EPERM (Operation not permitted)
open("/dev/_vm1_ramdisk/lvol0", O_RDWR) = -1 EPERM (Operation not permitted)
write(2, "  ", 2  )                       = 2
write(2, "Aborting. Failed to wipe start o"..., 41Aborting. Failed to wipe start of new LV.) = 41
write(2, "\n", 1
)                       = 1

There is some additional permission issue that i still can’t find after going through the config.

Ok, I solved it when I noticed that the major number used by LVM somehow changed from 252 to 253 and added the line

lxc.cgroup.devices.allow = b 253:* rwm

Since the major numbers of LVM are experimental it can change in this range a somewhat more universal solution would be

lxc.cgroup.devices.allow = all

even though I don’t like the security implications at all. There seems to be more devices involved since the second line prevents all remaining warnings while the first still has warnings like

File descriptor 6 (anon_inode:[signalfd]) leaked on lvcreate invocation. Parent PID 5: /bin/bash

and the set_priority if I don’t set the config value. Anyway, this is some solution for the time being (e.g. allowing 240-254 majors).