Operation not permitted on nvme devices

LXD
1

Hi,
“printf ‘lxc.cgroup.devices.allow = a\nlxc.mount.auto = proc:rw\nlxc.mount.auto = sys:rw\nlxc.mount.auto = cgroup-full:rw\nlxc.apparmor.profile = unconfined\n’ | lxc config set mycontainer raw.lxc -”
lxc config device add mycontainer nvme1 unix-block path=/dev/nvme2n1p1
When the application inside the container opens it for read/write, it gets ‘Operation not permitted’ error.
Any ideas?

lxc config show mycontainer

architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20200218)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20200218”
image.version: “18.04”
raw.lxc: |
lxc.cgroup.devices.allow = a
lxc.mount.auto = proc:rw
lxc.mount.auto = sys:rw
lxc.mount.auto = cgroup-full:rw
lxc.apparmor.profile = unconfined
volatile.base_image: 8c4e87e53c024e0449003350f0b0626b124b68060b73c0a7ad9547670e00d4b3
volatile.eth0.hwaddr: 00:16:3e:ed:c3:46
volatile.idmap.base: “0”
volatile.idmap.next: ‘[]’
volatile.last_state.idmap: ‘[]’
volatile.last_state.power: RUNNING
devices:
nvme1:
path: /dev/nvme2n1p1
type: unix-block
nvme2:
path: /dev/nvme2n1p2
type: unix-block
nvme3:
path: /dev/nvme2n1p3
type: unix-block
nvme4:
path: /dev/nvme2n1p4
type: unix-block
ephemeral: false
profiles:

  • my_profile
    stateful: false
    description: “”

#lxc profile show my_profile
config:
raw.lxc: lxc.apparmor.profile=unconfined
security.privileged: “true”
description: my LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: my_br
type: nic
root:
path: /
pool: btrfs_storage
type: disk
name: my_profile

2

That’s odd, are you getting the correct behavior on the host?

The error may make sense if the block device is somehow mounted on the host or in another container, but otherwise I’d expect this to work.

3

Thanks Stephane for quick reply.
block device is not mounted on host.
However, I am providing the same block device to all the containers. would that be an issue?

Also,
brw-rw---- 1 root root 259, 2 Mar 6 21:04 nvme2n1p1
brw-rw---- 1 root root 259, 3 Mar 6 21:04 nvme2n1p2
brw-rw---- 1 root root 259, 4 Mar 6 21:04 nvme2n1p3
brw-rw---- 1 root root 259, 5 Mar 6 21:04 nvme2n1p4
Also,
nvme list doesn’t show anything
df -k
/dev/sda5 14842581 11797234 2273251 84% /dev/nvme2n1p1
but it doesn’t show /dev/nme2n1p2
does this mean its treating this block device as sda(scsci disk)?

4

I tried not adding the same device to multiple containers as well. I can successfully add it to the container but application inside the container cannot access it.

5

Having it attached to multiple containers shouldn’t matter.

I’d expect nvme list to want to access the root devices for nvme which you don’t have in the container, so it not listing anything seems normal.

df -k's output is normal (though indeed looking confusing), that’s how injected device nodes look like.

The error you’re getting is what you get from cat /dev/nvme2n1p1?

7

Can you strace that software to see exactly what the final open call ends up being?

8

using strace did lead me to a conclusion that - when the process is started by systemd, then it can open the device with permission error. But if I try to run the application from command line as root, it can successfully open the file.
Anything to do with cgroup settings?
Below is what I shared earlier -
“printf ‘lxc.cgroup.devices.allow = a\nlxc.mount.auto = proc:rw\nlxc.mount.auto = sys:rw\nlxc.mount.auto = cgroup-full:rw\nlxc.apparmor.profile = unconfined\n’ | lxc config set mycontainer raw.lxc -”
lxc config device add mycontainer nvme1 unix-block path=/dev/nvme2n1p1

9

What’s the systemd unit for this thing like?

10

systemctl cat myservice

/usr/lib/systemd/system/myservice

[Unit]
Description= My Server
After=network.target
Wants=network.target

[Service]
LimitNOFILE=100000
TimeoutSec=15
User=root
Group=root
EnvironmentFile=/etc/sysconfig/myservice
PermissionsStartOnly=True
ExecStartPre=/usr/bin/myservice-systemd-helper
ExecStart=/usr/bin/myserviced $MYSERVICE_OPTIONS --config-file $MYSERVICE_CONFIG_FILE --fgdaemon
ExecStartPre=-/bin/systemctl start myservice_telemetry
ExecStopPost=-/bin/systemctl stop myservice_telemetry

[Install]
WantedBy=multi-user.target

/etc/systemd/system/myservice.service.d/myservice.conf

11

Hmm, okay, nothing looking particularly odd which would explain the privilege diference.