Error: open /dev/.incus-mounts: permission denied

I am getting an error message:

" Error: open /dev/.incus-mounts: permission denied "

when I try running podman inside a rockylinux9 container. And I get the same error message when I follow the same steps on ubuntu24.04 container as well.

Overall steps:

  1. Install podman and buildah inside the rockylinux container
  2. Use buildah to build the podman image passing a Dockerfile (I know! but yes - it is supported , though I don’t need to install docker)
  3. run podman with the newly built image

Here are the details:

step 1: build a ‘base’ image enabling systemd
step 2: build a second podman image based on the above ‘base’ image, with installing ‘htttpd’ in it

The goal is to build.a podman container running a httpd server, baked with all the required contents in it.

Building of both the podman images using buildah were successful

# podman images
REPOSITORY                       TAG         IMAGE ID      CREATED         SIZE
localhost/assets                 latest      dc9440eeaacb  24 minutes ago  611 MB
localhost/common-base            latest      fd05dc7f601d  13 hours ago    444 MB
docker.io/rockylinux/rockylinux  9           bb8a97547d22  2 months ago    241 MB

But when I try to run it, I get the above error message and the status of the podman is stuck at “Created”. nothing else happens.

# podman ps -a
CONTAINER ID  IMAGE                    COMMAND         CREATED        STATUS      PORTS       NAMES
2633be2806b1  localhost/assets:latest  /usr/sbin/init  5 minutes ago  Created     80/tcp      assets

And I cannot get inside the podman container either

# podman exec -ti assets /bin/bash
Error: can only create exec sessions on running containers: container state improper

Did some ‘perplexity’ search and there were suggestions to use options namely –userns=keep-id and/or –security-opt apparmor=unconfined passing them as parameter to podman but no luck :frowning:

Any suggestions are appreciated (though I am not sure how many out there use podman within incus). Thanks in advance .

Incus version and host OS version? Do you have security.nesting=true set on the container?

Thanks @candlerb for the response.

# incus version
Client version: 6.8
Server version: 6.8
# uname -a
Linux red-box 6.6.63-current-x86 #1 SMP PREEMPT_DYNAMIC Fri Nov 22 14:38:37 UTC 2024 x86_64 GNU/Linux
# cat /etc/os-release
PRETTY_NAME="Armbian 24.11.1 bookworm"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.armbian.com"
SUPPORT_URL="https://forum.armbian.com"
BUG_REPORT_URL="https://www.armbian.com/bugs"
ARMBIAN_PRETTY_NAME="Armbian 24.11.1 bookworm"

I use ‘incus webui’. I am running the rockylinux incus container (that runs the podman container) inside a Project. Since I set the container nesting on the project level, I believe it will be propagated to all the containers within that project. And I do see that seems to be the case (rl-assets is the name of my rockylinux incus container)

# incus config get rl-assets security.nesting
true

But when I try to set security.syscalls.intercept.mknod and security.syscalls.intercept.setxattr on the same rockylinux incus container, I am getting the following messages.

# incus config set rl-assets security.syscalls.intercept.mknod=true
Error: Failed checking if instance update allowed: Invalid value "true" for config "security.syscalls.intercept.mknod" on container "rl-assets" of project "handson-lab-1": Container syscall interception is forbidden

# incus config set rl-assets security.syscalls.intercept.setxattr=true
Error: Failed checking if instance update allowed: Invalid value "true" for config "security.syscalls.intercept.setxattr" on container "rl-assets" of project "handson-lab-1": Container syscall interception is forbidden

Remember reading the documentation that you need to set all the required properties to the Project first before running any incus containers/vms inside it. Guess I am running into that issue. Not sure.

I will re-do the exercise creating a new project, set all the security and nesting properties and see how it goes. And I will share my observations. But please let me know if you believe I am going in the wrong direction :slight_smile: Thanks again!

On the Project level, I do see that is set restricted=true , though restricted.containers.nesting=allow

# incus project show handson-lab-1
config:
  features.images: "true"
  features.networks: "false"
  features.networks.zones: "false"
  features.profiles: "true"
  features.storage.buckets: "true"
  features.storage.volumes: "true"
  limits.disk: 316GiB
  **restricted: "true"**
  restricted.containers.nesting: allow
  restricted.networks.uplinks: eno3
description: 'Handson Lab #1'
name: handson-lab-1

Instead of running inside.a project that I created, I tried with the default setting.

Spent hours on it but with no success so far:-( Running into the same error. Tried all these:

# incus config set rl-assets security.nesting=true
# incus config set rl-assets  security.privileged=true
# incus config set rl-assets  security.syscalls.intercept.mknod=true
# incus config set rl-assets  security.syscalls.intercept.setxattr=true

# incus shell rl-assets

# podman images
REPOSITORY                       TAG         IMAGE ID      CREATED        SIZE
localhost/pod-assets             latest      72dbb1e2e80f  3 minutes ago  779 MB
localhost/common-base            latest      42930ffd7b09  4 hours ago    444 MB
docker.io/rockylinux/rockylinux  9           bb8a97547d22  2 months ago   241 MB

# df -h
Filesystem                    Size  Used Avail Use% Mounted on
default/containers/rl-assets  5.2G  673M  4.5G  13% /
none                          492K  4.0K  488K   1% /dev
udev                           63G     0   63G   0% /dev/zfs
tmpfs                         100K     0  100K   0% /dev/incus
tmpfs                         100K     0  100K   0% /dev/.incus-mounts
tmpfs                          63G   84K   63G   1% /dev/shm
tmpfs                          26G  8.2M   26G   1% /run
tmpfs                          13G   12K   13G   1% /run/user/0

# id
uid=0(root) gid=0(root) groups=0(root)

# podman run --name pod-assets --privileged -d -p 9060:80 --userns=keep-id -v /sys/fs/cgroup:/sys/fs/cgroup:ro pod-assets 
**Error: open /dev/.incus-mounts: permission denied**

# podman ps -a
CONTAINER ID  IMAGE                        COMMAND         CREATED        STATUS      PORTS                         NAMES
7ea3d6f72cf8  localhost/pod-assets:latest  /usr/sbin/init  5 minutes ago  Created     0.0.0.0:9060->80/tcp, 80/tcp  pod-assets

# podman exec -ti pod-assets /bin/bash
Error: can only create exec sessions on running containers: container state improper

Wondering if anyone managed to run podman successfully inside an incus container ? Appreciate if you can share any details. Thanks in advance.

My reading is: project setting restricted.containers.nesting=allow will allow you to set security.nesting=true on a container within the project, not that it will be set.

I could be wrong of course, it’s just how I read it, but that’s what POLS implies to me.

But since later you said you’ve set security.nesting=true on the container, that’s not likely to be the problem.