Permission denied on /dev/hugepages

On Ubuntu 18.04 I can install LXD apt install lxc1 lxd lxd-client, create a container lxc launch images:centos/7 mycontainer and enter it using lxc exec mycontainer bash.

In the container I noticed that service dev-hugepages.mount has failed.

[root@mycontainer ~]# systemctl status dev-hugepages.mount 
● dev-hugepages.mount - Huge Pages File System
   Loaded: loaded (/usr/lib/systemd/system/dev-hugepages.mount; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-11-20 04:35:41 UTC; 33s ago
    Where: /dev/hugepages
     What: hugetlbfs
     Docs: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 35 ExecMount=/bin/mount hugetlbfs /dev/hugepages -t hugetlbfs (code=exited, status=32)

Nov 20 04:35:41 mycontainer mount[35]: mount: permission denied
Nov 20 04:35:41 mycontainer systemd[1]: dev-hugepages.mount mount process exited, code=exited status=32
Nov 20 04:35:41 mycontainer systemd[1]: Failed to mount Huge Pages File System.
Nov 20 04:35:41 mycontainer systemd[1]: Unit dev-hugepages.mount entered failed state.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[root@mycontainer ~]# 

The command that is making this service fails is /bin/mount hugetlbfs /dev/hugepages -t hugetlbfs -v

[root@mycontainer ~]# /bin/mount hugetlbfs /dev/hugepages -t hugetlbfs -v
mount: permission denied
[root@mycontainer ~]# 

This service causes other services to fail. For example nfs-server. How can I fix this issue?

When I configure the container to be unconfined, this does not fix the problem

lxc config set mycontainer raw.lxc "lxc.apparmor.profile=unconfined"

I have a similar container created with Vagrant and the Vagrant LXC plugin - which uses LXC only where this service does run properly.

You can try lxc config set mycontainer raw.apparmor="mount fstype=hugetlbfs,"

3d [user@ubuntulxchost:~] 130 $ lxc config set mycontainer raw.apparmor="mount fstype=hugetlbfs,"
Error: cannot set 'mycontainer' to 'raw.apparmor=mount fstype=hugetlbfs,': unknown key

I noticed btw that the lxc.con looks strange with two entries

3d [user@ubuntulxchost:~]1 $ sudo cat /var/log/lxd/mycontainer/lxc.conf | grep app
lxc.apparmor.profile = lxd-mycontainer_</var/lib/lxd>//&:lxd-mycontainer_<var-lib-lxd>:
lxc.apparmor.profile=unconfined
3d [user@ubuntulxchost:~]

Ah, you’re on an older LXD, so lxc config set mycontainer raw.apparmor "mount fstype=hugetlbfs," in that case.

I tried that before and after a restart of the container but the problem is still there.

journalctl -xe shows all kinds of permission problems

Nov 21 05:35:41 mycontainer systemd-udevd[40]: starting version 219
Nov 21 05:35:41 mycontainer systemd-sysctl[41]: Failed to write '0' to '/proc/sys/kernel/yama/ptrace_scope': Permiss
ion denied
Nov 21 05:35:41 mycontainer systemd-sysctl[41]: Failed to write '16' to '/proc/sys/kernel/sysrq': Permission denied

Nov 21 05:35:41 mycontainer systemd-sysctl[41]: Failed to write '1' to '/proc/sys/kernel/core_uses_pid': Permission 
denied
Nov 21 05:35:41 mycontainer mount[39]: mount: permission denied
Nov 21 05:35:41 mycontainer systemd-sysctl[41]: Failed to write '1' to '/proc/sys/kernel/kptr_restrict': Permission 
denied
Nov 21 05:35:41 mycontainer systemd-sysctl[41]: Failed to write '1' to '/proc/sys/fs/protected_hardlinks': Permissio
n denied
Nov 21 05:35:41 mycontainer systemd-sysctl[41]: Failed to write '1' to '/proc/sys/fs/protected_symlinks': Permission
 denied
Nov 21 05:35:41 mycontainer mount[42]: mount: permission denied
Nov 21 05:35:41 mycontainer systemd[1]: Started udev Kernel Device Manager.
-- Subject: Unit systemd-udevd.service has finished start-up

Ok, so the issue isn’t apparmor at all. I suspect it’s simply that hugetlbfs is a restricted filesystem which is never allowed to be mounted inside of an unprivileged container.

1 Like

This is a difference between LXD and LXC then I suppose because with LXC this does work. I used Vagrant with Vagrant LXC plugin and in that case the service works.

Checked this with the Vagrant LXD plugin and indeed. Same problem. When I searched filesystem I noticed many other permissions problems

[root@mycontainer ~]# find / -name 'dev-hugepages.mount'
find: '/sys/kernel/debug': Permission denied
find: '/sys/fs/pstore': Permission denied
find: '/sys/fs/fuse/connections/368': Permission denied
find: '/sys/fs/fuse/connections/230': Permission denied
find: '/sys/fs/fuse/connections/54': Permission denied
find: '/proc/tty/driver': Permission denied
/usr/lib/systemd/system/dev-hugepages.mount
/usr/lib/systemd/system/sysinit.target.wants/dev-hugepages.mount
find: '/dev/.lxd-mounts': Permission denied
[root@mycontainer ~]# 

So I suppose your conclusion is correct that this will not work with LXD. I was thinking about switching from LXC to LXD but as it turns out this would not make sense for me. In a LXD container it is not possible to run a NFS server or even mount a NFS folder. Those services depend on dev-hugepages.mount.

Your LXC container is almost certainly privileged.

1 Like

But when I launch the container as root, the service is still failing

[root@mylxdhost:~] # lxc launch images:centos/7 mycontainer

● dev-hugepages.mount - Huge Pages File System
   Loaded: loaded (/usr/lib/systemd/system/dev-hugepages.mount; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2020-11-22 02:12:34 UTC; 2min 11s ago
    Where: /dev/hugepages
     What: hugetlbfs
     Docs: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
--

● sys-kernel-config.mount - Configuration File System
   Loaded: loaded (/usr/lib/systemd/system/sys-kernel-config.mount; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2020-11-22 02:12:34 UTC; 2min 11s ago
    Where: /sys/kernel/config
     What: configfs
     Docs: https://www.kernel.org/doc/Documentation/filesystems/configfs/configfs.txt

The file /var/lib/lxd/security/apparmor/profiles/lxd-mycontainer also has the lines

  # allow hugetlbfs mounts everywhere
  mount fstype=hugetlbfs,

Is my understanding correct that if I run the container as root that the container is privileged?

Is my LXD / LXC too old? I have 3.0.3 installed on Ubuntu 18.04.

No, who starts the container doesn’t matter.
A privileged container would have security.privileged=true in its settings.

I managed to get nfs server and mounts working using profile from lxc project https://github.com/ubuntu/microk8s/blob/master/tests/lxc/microk8s.profile

I realize the settings are too relaxed for typical production use but for me for now it will do.