Mapped folder doesn't show up in container

I want to map a folder from host to container. The folder doesn’t show up in the container not even as nobody. No change on ‘privileged’. The container has been initially created on a grsec environment (with newuidmap disabled thus the 1000000 mapping). Its on unrestricted kernel now and maprange has been reduced back to 65536.

Any idea why the folder is not showing up at all?

lxc config device add alpine collectd disk source=/var/lib/collectd path=/var/lib/collectd

LXD 3.0.0
Alpine Linux (host+guest) 4.14.33-0-vanilla

LXD daemon: root:lxd
/var/lib/collectd (host): 105:106
/var/lib/collectd (container): 102:103 (folder doesn’t exist)

Config:

architecture: i686
config:
  image.architecture: i386
  image.description: Alpine edge i386 (20180331_17:50)
  image.os: Alpine
  image.release: edge
  image.serial: "20180331_17:50"
  raw.idmap: |-
    uid 105 102
    gid 106 103
  security.privileged: "false"
  volatile.base_image: blahblahblah
  volatile.eth0.hwaddr: 00:16:3e:xx:xx:xx
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":102},{"Isuid":true,"Isgid":false,"Hostid":105,"Nsid":102,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000103,"Nsid":103,"Maprange":65433},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":103},{"Isuid":false,"Isgid":true,"Hostid":106,"Nsid":103,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000104,"Nsid":104,"Maprange":65432}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
devices:
  collectd:
    path: /var/lib/collectd
    source: /var/lib/collectd
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

subuid:

root:1000000:65536
lxd:1000000:65536

root:105:1
lxd:105:1

subgid

root:1000000:65536
lxd:1000000:65536

root:106:1
lxd:106:1

What’s in /var/log/lxd/logs/alpine/lxc.conf?

lxc.log.file = /var/log/lxd/alpine/lxc.log
lxc.log.level = warn
lxc.cap.drop = sys_time sys_module sys_rawio mac_admin mac_override
lxc.mount.auto = proc:mixed sys:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional
lxc.include = /usr/share/lxc/config/common.conf.d/
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = b *:* m
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.arch = linux32
lxc.hook.pre-start = /usr/sbin/lxd callhook /var/lib/lxd 11 start
lxc.hook.post-stop = /usr/sbin/lxd callhook /var/lib/lxd 11 stop
lxc.tty.max = 0
lxc.uts.name = alpine
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/alpine
lxc.rootfs.path = dir:/var/lib/lxd/containers/alpine/rootfs
lxc.mount.entry = /var/lib/lxd/devices/alpine/disk.collectd.var-lib-collectd var/lib/collectd none bind,create=dir
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxdbr0
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
lxc.net.0.name = eth0
lxc.mount.entry = /var/lib/lxd/shmounts/alpine dev/.lxd-mounts none bind,create=dir 0 0

lxd.log

lvl=info msg="LXD 3.0.0 is starting in normal mode" path=/var/lib/lxd t=2018-04-11T19:25:26+0200
lvl=info msg="Kernel uid/gid map:" t=2018-04-11T19:25:26+0200
lvl=info msg=" - u 0 0 4294967295" t=2018-04-11T19:25:26+0200
lvl=info msg=" - g 0 0 4294967295" t=2018-04-11T19:25:26+0200
lvl=info msg="Configured LXD uid/gid map:" t=2018-04-11T19:25:26+0200
lvl=info msg=" - u 0 1000000 65536" t=2018-04-11T19:25:26+0200
lvl=info msg=" - g 0 1000000 65536" t=2018-04-11T19:25:26+0200
lvl=warn msg="AppArmor support has been disabled because of lack of kernel support" t=2018-04-11T19:25:26+0200
lvl=warn msg="Couldn't find the CGroup CPUset controller, CPU pinning will be ignored." t=2018-04-11T19:25:26+0200
lvl=warn msg="Couldn't find the CGroup memory controller, memory limits will be ignored." t=2018-04-11T19:25:26+0200
lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-04-11T19:25:26+0200
lvl=info msg="Initializing database gateway" t=2018-04-11T19:25:26+0200
address= id=1 lvl=info msg="Start database node" t=2018-04-11T19:25:26+0200
lvl=info msg="Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]" t=2018-04-11T19:25:26+0200
lvl=info msg="Raft: Node at 0 [Leader] entering Leader state" t=2018-04-11T19:25:26+0200
lvl=info msg="LXD isn't socket activated" t=2018-04-11T19:25:26+0200
lvl=info msg="Starting /dev/lxd handler:" t=2018-04-11T19:25:26+0200
lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock t=2018-04-11T19:25:26+0200
lvl=info msg="REST API daemon:" t=2018-04-11T19:25:26+0200
lvl=info msg=" - binding Unix socket" socket=/var/lib/lxd/unix.socket t=2018-04-11T19:25:26+0200
lvl=info msg="Pruning expired images" t=2018-04-11T19:25:27+0200
lvl=info msg="Done pruning expired images" t=2018-04-11T19:25:27+0200
lvl=info msg="Updating instance types" t=2018-04-11T19:25:27+0200
lvl=info msg="Expiring log files" t=2018-04-11T19:25:27+0200
lvl=info msg="Done expiring log files" t=2018-04-11T19:25:27+0200
lvl=info msg="Done updating instance types" t=2018-04-11T19:26:09+0200

Kernel kconfig --> https://git.alpinelinux.org/cgit/aports/tree/main/linux-vanilla/config-vanilla.x86 <–

The actual UID/GID mapping seems to work though. Other folders/files with respective UID/GID show up mapped correctly. Its just the folder injection itself that fails.

Manually bind mounting an arbitrary folder works as well.

LXD mounts the folder to /var/lib/lxd/devices/alpine/disk.collectd.var-lib-collectd with correct mapping but not into the actual container rootfs.

Some more tests:
It does work if the device gets added when the container is running. But once stopped the mount is gone and doesn’t come back if the container gets restarted. The device itself remains listed in ‘lxc config device show alpine’ though. You need to remove and re-add it to get another one-time shot.
Can anyone confirm that?

lxc start alpine
lxc config device add alpine collectd disk source=/var/lib/collectd path=/var/lib/collectd
lxc exec alpine -- ls /var/lib/collectd
<folder content is present, folder uid=correct>
lxc restart alpine
lxc exec alpine -- ls /var/lib/collectd
<folder content is not present, folder uid=0>

@stgraber:
It works properly on ArchLinux so it might be AlpineLinux specific. But what could be missing there that it works partially only?

I’ve removed the disk device from lxc config and tried to bind mount it myself with:

mount --bind /var/lib/collectd /var/lib/lxd/storage-pools/default/containers/alpine/rootfs/var/lib/collectd

After restarting the container i can see the folder properly mapped within ssh but CGP (Collectd Graph Panel) on nginx says no hosts found in folder (it doesn’t seem to see the bind mounted data). If i copy the data to the container it does work properly.
From the scope of the ssh session both folders where completely identical. I guess nginx/php-fpm7 runs as ‘nobody’ and might see different mapping (than ssh).