Permission denied for mount filesystem

LXD
1

I have a need to mount an encrypted external drive within a container.

LXD 4.0.5 LTS on Ubuntu 20.0.4 LTS installed using snap.

I’ve got through the luksOpen & AppArmor issues so the drive will decrypt, but now I can’t get the decrypted disk to mount. I’m getting permission denied against the target directory.

There are no apparmor or other issues showing in dmesg or journalctl, on either the container or host.

The mountpoint within the container was created inside the container. I’ve tried creating it in the host and mapping that as a disk device into the container, and I get the same result.

Is there a module I need to load or some other simple missing thing?
Would upgrading to the latest LXD fix this?

Thanks
David

in the container:

# /bin/mount --verbose /dev/mapper/disk
mount: /mnt/disk: permission denied.

# ls -l /mnt/
drwxr-xr-x 2 root root 4096 Mar  3 14:01 disk

# cat /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/disk  /mnt/disk  btrfs  defaults,rw,noauto,nodev,noexec,nosuid,relatime  0  2

container config:

config:
  security.idmap.isolated: "true"
  security.privileged: "true"
  linux.kernel_modules: dm_crypt,btrfs,dm_mod,cryptd,crypto_simd,dm_multipath,sha256,sha512,aes_ti,sha3_generic
  raw.apparmor: |
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /mnt/*,
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /mnt/*/,
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /dev/mapper/disk,
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /dev/mapper/disk/,
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /dev/dm*,

p.s. this has too many lines as I’ve been adding extra things while I try and get it to work - I’ll prune once it’s working.

pps. I can decrypt & mount the disk on the host without trouble.

2

What’s dmesg showing you, you’d normally get a DENIED for it.

3

Silence apart from logging the sudo commands. I was getting DENIED before I added the raw.apparmor lines to the profile.

here are the commands after startup:

HOST$ lxc exec saturn -- sudo --user ubuntu --login
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@saturn:~$ sudo /usr/sbin/cryptsetup luksOpen /dev/disk disk
Enter passphrase for /dev/disk: 
ubuntu@saturn:~$ sudo /bin/mount /dev/mapper/disk
mount: /mnt/disk: permission denied.

and here is end of dmesg:

[206536.070186] audit: type=1400 audit(1614796025.046:163): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-saturn_</var/snap/lxd/common/lxd>" pid=550957 comm="apparmor_parser"
[206536.183306] eth0: renamed from veth2f16c4a2
[206536.196323] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[206536.196356] milton: port 1(veth35883b68) entered blocking state
[206536.196358] milton: port 1(veth35883b68) entered forwarding state
[206538.389483] audit: type=1400 audit(1614796027.366:164): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-saturn_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/" pid=551137 comm="(networkd)" srcname="/" flags="rw, rbind"
[206538.518271] audit: type=1400 audit(1614796027.494:165): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-saturn_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/" pid=551139 comm="(resolved)" srcname="/" flags="rw, rbind"
[206540.498051] audit: type=1400 audit(1614796029.474:166): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-saturn_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/" pid=551208 comm="(d-logind)" srcname="/" flags="rw, rbind"
[206540.586837] audit: type=1400 audit(1614796029.562:167): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-saturn_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/" pid=551213 comm="(ostnamed)" srcname="/" flags="rw, rbind"
[206542.302117] [UFW AUDIT] IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.53 LEN=70 TOS=0x00 PREC=0x00 TTL=64 ID=14789 DF PROTO=UDP SPT=10943 DPT=53 LEN=50 
[206542.302131] [UFW AUDIT] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.53 LEN=70 TOS=0x00 PREC=0x00 TTL=64 ID=14789 DF PROTO=UDP SPT=10943 DPT=53 LEN=50 
[206542.302170] [UFW AUDIT] IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.53 LEN=70 TOS=0x00 PREC=0x00 TTL=64 ID=14790 DF PROTO=UDP SPT=1236 DPT=53 LEN=50 
[206542.302374] [UFW ALLOW] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=70 TOS=0x00 PREC=0x00 TTL=64 ID=38941 DF PROTO=UDP SPT=47520 DPT=53 LEN=50 
[206542.302513] [UFW ALLOW] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=70 TOS=0x00 PREC=0x00 TTL=64 ID=38942 DF PROTO=UDP SPT=58137 DPT=53 LEN=50 
[206546.620060] [UFW AUDIT] IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.53 LEN=87 TOS=0x00 PREC=0x00 TTL=64 ID=15480 DF PROTO=UDP SPT=51346 DPT=53 LEN=67 
[206546.620074] [UFW AUDIT] IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.53 LEN=87 TOS=0x00 PREC=0x00 TTL=64 ID=15480 DF PROTO=UDP SPT=51346 DPT=53 LEN=67 
[206546.620361] [UFW ALLOW] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=87 TOS=0x00 PREC=0x00 TTL=64 ID=39342 DF PROTO=UDP SPT=56203 DPT=53 LEN=67 
[206597.640401] [UFW AUDIT] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=76 TOS=0x00 PREC=0x00 TTL=64 ID=45958 DF PROTO=UDP SPT=44243 DPT=123 LEN=56 
[206597.640410] [UFW ALLOW] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=76 TOS=0x00 PREC=0x00 TTL=64 ID=45958 DF PROTO=UDP SPT=44243 DPT=123 LEN=56 
[206662.643162] [UFW AUDIT] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=76 TOS=0x00 PREC=0x00 TTL=64 ID=48482 DF PROTO=UDP SPT=56240 DPT=123 LEN=56 
[206662.643170] [UFW ALLOW] IN= OUT=eth0 SRC=192.168.57.215 DST=192.168.57.253 LEN=76 TOS=0x00 PREC=0x00 TTL=64 ID=48482 DF PROTO=UDP SPT=56240 DPT=123 LEN=56

and journalctl:

[206579.645385] saturn systemd[1]: Finished Execute cloud user/final scripts.
[206579.645771] saturn systemd[1]: Reached target Cloud-init target.
[206579.645853] saturn systemd[1]: Startup finished in 44.914s.
[206619.513588] saturn sudo[2923]:     root : TTY=console ; PWD=/root ; USER=ubuntu ; COMMAND=/bin/bash
[206619.515382] saturn sudo[2923]: pam_unix(sudo:session): session opened for user ubuntu by LOGIN(uid=0)
[206660.428575] saturn sudo[3008]:   ubuntu : TTY=console ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/sbin/cryptsetup luksOpen /dev/disk disk
[206660.430290] saturn sudo[3008]: pam_unix(sudo:session): session opened for user root by LOGIN(uid=0)
[206668.058006] saturn sudo[3008]: pam_unix(sudo:session): session closed for user root
[206681.374943] saturn sudo[3049]:   ubuntu : TTY=console ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/mount /dev/mapper/disk
[206681.376591] saturn sudo[3049]: pam_unix(sudo:session): session opened for user root by LOGIN(uid=0)
[206681.380880] saturn sudo[3049]: pam_unix(sudo:session): session closed for user root
[206690.432485] saturn sudo[3067]:   ubuntu : TTY=console ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/dmesg
[206690.433554] saturn sudo[3067]: pam_unix(sudo:session): session opened for user root by LOGIN(uid=0)
[206691.507599] saturn sudo[3067]: pam_unix(sudo:session): session closed for user root
[206884.005789] saturn PackageKit[484]: daemon quit
[206884.026550] saturn systemd[1]: packagekit.service: Succeeded.
[207104.625275] saturn sudo[3805]:   ubuntu : TTY=console ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/journalctl -o short-monotonic
[207104.626525] saturn sudo[3805]: pam_unix(sudo:session): session opened for user root by LOGIN(uid=0)
4

Hi Stephane

unfortunately, in this case, the logs are silent. I suspect it’s either a missing kernel module in the container (? which) or a control socket (or equivalent) used by mount for exclusivity or to record the state of what’s mounted and where. Again, which.

… I don’t have a detailed picture of how mount works and don’t know what tools would help trace the failure. Can you advise?

I’ll try again today without the isolated namespace, as I can’t think of anything else to experiment with.

Any advice or guidance would be appreciated.

Yours
David

5

Can you try putting just mount, in raw.apparmor to allow all mounts?
Ah, also, you’ll need some lxc.cgroup.devices.allow lines to allow read access from those block devices, that’s likely the main issue you’re hitting here.

6

Thanks @stgraber. I was missing the lxc.cgroup.devices.allow lines. For anyone finding this later, this is the config I used successfully:

config:
  security.idmap.isolated: "true"
  security.privileged: "true"
  linux.kernel_modules: dm_crypt,btrfs,dm_mod,cryptd,crypto_simd,dm_multipath,sha256,sha512,aes_ti,sha3_generic
  raw.apparmor: |-
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /mnt/test,
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /dev/mapper/test,
    mount fstype=btrfs options=(rw, nosuid, nodev, noexec, relatime) /dev/dm*,
  raw.lxc: |-
$( /bin/ls -l -d -b -w0 -U $( /bin/ls -l -d -b -w0 -U /dev/mapper/* | /bin/grep -Ev -e 'test' -e '/vgData-lvCrypt' | /bin/grep -E '^l' | /bin/sed -r -e 's|^.+\->\s+||' -e 's|^../|/dev/|' ) | /usr/bin/awk '{print $5 ":" $6}' | /bin/sed -r 's/,:/:/' | while read X ; do
  echo "    lxc.cgroup.devices.deny = b $X rwm"
done )
    lxc.cgroup.devices.allow = b 253:* rwm

devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
    security.mac_filtering: "true"
  root:
    path: /
    pool: local
    type: disk
  lvCrypt:
    path: /dev/test
    source: $( /usr/bin/printf '%q' "$( /usr/bin/realpath -e /dev/mapper/vgData-lvCrypt )")
    type: unix-block
  mapper.control:
    path: /dev/mapper/control
    source: /dev/mapper/control
    type: unix-char

The code denies access to all the dm nodes apart from those needed for the decrypt & mount.

/dev/mapper/control is needed for the device mapper to work.

Note that, unlike in @simos config in https://blog.simos.info/how-to-add-multi-line-raw-lxc-configuration-to-lxd/, in my system (ubuntu LTS 20.04) the device type identifiers are major:minor and it needs the modes (rwm for read write mknod) on the end.

How to autoprovision additional disks using profiles or Ansible in a cluster?