Loopback device refresh in an LXC container

LXC
1

Hi,

I’m attempting to partition a loop-mounted image file in a container and the dev/loop0pn device files do not appear.

fallocate -l 8G disk.img
losetup -P /dev/loop0 disk.img
fdisk /dev/loop0 # create partitions
ls /dev/loop0* # no devices representing subdevices appear

The same routine works fine on the host (/dev/loop0p1 and dev/loop0p2 dynamically appear - they don’t exist until fdisk write the partitions).

The container profile looks like:

config:
  raw.lxc: |-
    lxc.cgroup.devices.allow = c 10:237 rwm
    lxc.aa_profile = unconfined
    lxc.cgroup.devices.allow = b 7:* rwm
  security.privileged: "true"
description: ""
devices:
  loop0:
    path: /dev/loop0
    type: unix-block
  loop1:
    path: /dev/loop1
    type: unix-block
  loop2:
    path: /dev/loop2
    type: unix-block
  loop3:
    path: /dev/loop3
    type: unix-block
  loop4:
    path: /dev/loop4
    type: unix-block
  loop5:
    path: /dev/loop5
    type: unix-block
  loop6:
    path: /dev/loop6
    type: unix-block
  loop7:
    path: /dev/loop7
    type: unix-block
  root:
    path: /
    pool: default
    type: disk
  wlan0:
    nictype: bridged
    parent: br0
    type: nic

I’m using LXC 2.1.1 on Gentoo amd64

Any hints on how I might progress / workarounds / references / things to try would be great.
Also if someone could explain to me what is happening on the host after fdisk writes to the loopback device that causes the new /dev/loop0pn devices to be created that would be really excellent!

Cheers,

Gus.

2

The host uses devtmpfs for /dev which is a virtual filesystem from the kernel that dynamically creates devices for you based on uevents.

The containers can’t use that filesystem because it’s not namespaced and so instead use a regular tmpfs + manually created (or bind-mounted) device nodes.

When you run that losetup, the kernel will likely emit those uevents and the device may appear on the host, but in the container, nothing is there to react to those (if they even make it) and so nothing shows up.

One potential workaround is to bind-mount /dev/mapper into the container using something like:

lxc config device add NAME mapper disk source=/dev/mapper path=/dev/mapper recursive=true

There’s then a good chance that the uevent will trigger creation of /dev/mapper/XYZ devices that would then be visible in the container.

3

Ah excellent, thanks Stefan for your help. That’s really useful.

I bind-mount’d /dev/mapper as you suggested:

ls -alh /dev/mapper/control  
crw------- 1 root root 10, 236 Mar 30 18:05 /dev/mapper/control

I attempted to use kpartx to create devices for each partition explicitly - this method works on the host but not in the container. I strace'd it to see what is going on:

stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(10, 236), ...}) = 0
open("/dev/mapper/control", O_RDWR)     = -1 EPERM (Operation not permitted)
write(2, "/dev/mapper/control: open failed"..., 57/dev/mapper/control: open failed: Operation not permitted) = 57

The permissions for /dev/mapper/control are identical in the container and on the host, and the container is running in privileged mode - I can see its init process running as root.

Do you have any ideas as to why I’m seeing the O_RDWR open failure, or any further thoughts about how I might solve this issue?

Thanks again for your help,

Gus.

4

Did you update your lxc.cgroup.devices.allow to allow access to /dev/mapper/control and any block device it may create?

5

Hey Stefan,

Yes, good point, I added access to the following types of block device and kpartx ran without error:

lxc.cgroup.devices.allow = b 254:* rwm

However it still didn’t replicate precisely what was happening on the host. kpartx creates two symlinks inside /dev/mapper:

lrwxrwxrwx  1 root root       7 Apr  7 12:24 loop0p1 -> ../dm-0
lrwxrwxrwx  1 root root       7 Apr  7 12:24 loop0p2 -> ../dm-1

On the host these dm-* nodes appear correctly, but not in the container.

I compared the kpartx strace outputs on both the container and host but there was no discernible difference (aside from memory addresses and device numbers) - so it looks as though the dm-* files are created by something else (presumably udev on the host). I put udev into debug mode (udev_log="debug" in /etc/udev/udev.log - this showed that is was calling out to dmsetup so presumably that’s what is creating the device nodes.

Rather than try to replicate the udev setup in the container I opted to create the dm-* nodes using mknod inside the container:

mknod /dev/dm-0 b 254 0
mknod /dev/dm-1 b 254 1

In conjunction with running kpartx this workaround meets my needs, allowing me to continue on with formatting the loopback partitions.

Thank you for your help - I really appreciate it.

Gus.