I’m trying to run podman in an Incus container with security.nesting set to true.
podman run hello-world as a user produces this error:
cannot clone: Permission denied
Error: cannot re-exec process
Running sudo podman run hello-world works. Does anyone have any pointers whether this is something that can be fixed, or is it a known issue? Thank you!
Welcome!
I cannot replicate the issue (that is, it works for me). Here’s a complete set of steps. Let’s work on these to figure out where it breaks. The issue is to run podman as non-root in an Incus container.
The host in my case is Ubuntu.
$ incus launch images:ubuntu/24.04/cloud mycontainer --config security.nesting=true
Launching mycontainer
$ incus exec mycontainer -- su -l ubuntu
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@mycontainer:~$ sudo apt install -y podman
...
ubuntu@mycontainer:~$ podman run hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
...
Hello from Docker!
This message shows that your installation appears to be working correctly.
...
ubuntu@mycontainer:~$
Excited to be here, thanks for the warm welcome and the helping hand @simos and apologies for the late reply. I followed your steps and am getting the same error:
root@msch ~ # uname -a
Linux msch 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@msch ~ # dpkg -s incus | grep '^Version:'
Version: 1:6.14-ubuntu24.04-202507140410
root@msch ~ # incus launch images:ubuntu/24.04/cloud mycontainer --config security.nesting=true
Launching mycontainer
root@msch ~ # incus exec mycontainer -- su -l ubuntu
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@mycontainer:~$ sudo apt install -y podman
…
Unpacking podman (4.9.3+ds1-1ubuntu0.2) …
…
ubuntu@mycontainer:~$ podman run hello-world
cannot clone: Permission denied
Error: cannot re-exec process
simos’ reproducer works for me too (host ubuntu 22.04, incus 6.0.4 from zabbly)
Do you see any errors in dmesg output? (I am thinking about apparmor)
Also, what do you have in /etc/subuid and /etc/subgid on the host?
Your apparmor guess seems to have been right on the nose:
[641411.342937] audit: type=1400 audit(1753651524.990:971): apparmor="DENIED" operation="userns_create" class="namespace" info="Userns create restricted - failed to find unprivileged_userns profile" error=-13 namespace="root//incus-mycontainer_<var-lib-incus>" profile="unconfined" pid=1252590 comm="podman" requested="userns_create" denied="userns_create" target="unprivileged_userns"
/etc/subuid and /etc/subgid both are
msch:100000:65536
root:1000000:1000000000
What’s surprising to me is that after I (out of curiosity/desperation) ran apt install apparmor inside mycontainer rootless podman works.
Also, I tried incus launch images:fedora/42/cloud fedora --config security.nesting=true and in Fedora I get cannot clone: Permission denied as well, but there’s no apparmor package.
Are you sure you get that specific message with Fedora? Because I get instead the following:
Error: cannot set up namespace using "/usr/bin/newuidmap":
should have setuid or have filecaps setuid: exit status 1
The two workarounds work, Running podman rootless gives ERRO[0000] cannot setup namespace using newuidmap: exit status 1 · Issue #2788 · containers/podman · GitHub
What exactly is the OS and version running on the host?
I see you showed kernel 6.8.0-64-generic which makes me think it’s probably Ubuntu 24.04. (I am running Ubuntu 22.04 with HWE kernel, which is exactly the same kernel version).
Do you have any outstanding package updates?
apt update
apt list --upgradeable
Userns create restricted - failed to find unprivileged_userns profile
If you Google that error message you find some interesting pointers.
Maybe just restart apparmor on the host?
sudo systemctl enable apparmor
sudo systemctl start apparmor
I am not a Incus/Podman/Apparmor expert, but I also tried to run rootless Podman inside Incus containers on an Ubuntu 24.04 host recently, and also ran into this error message, so here’s my 2 cents.
The error is caused by the new default AppArmor settings in Ubuntu:
Restricted unprivileged user namespaces are coming to Ubuntu 23.10 | Ubuntu
As such, unprivileged processes will only be able to create user namespaces if they are confined and have the “userns,” rule in their AppArmor profile (or if they have CAP_SYS_ADMIN).
and some more background:
[spec] Unprivileged user namespace restrictions via AppArmor in Ubuntu 23.10 - Project Discussion / Security - Ubuntu Community Hub
The upstream AppArmor project recently introduced a mechanism to allow the use of unprivileged user namespaces to only those applications that both require the access and are appropriately confined by AppArmor (and to deny access to all other applications).
So it’s about kernel.apparmor_restrict_unprivileged_userns which is set to 1, so the restriction is enabled.
And as Incus creates a AppArmor namespace (something like ‘:incus-containername_<var-lib-incus>:’), there must be a AppArmor profile ‘unprivileged_userns’ in the AppArmor namespace, in order to let Podman create unprivileged user namespaces.
(like it says in the error message in you dmesg output:
info=“Userns create restricted - failed to find unprivileged_userns profile”
error=-13
namespace=“root//incus-mycontainer_<var-lib-incus>”
So there are options:
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0raw.lxc: "lxc.apparmor.profile=unconfined" in the container configsudo apparmor_parser -n 'incus-mycontainer_<var-lib-incus>' -r /etc/apparmor.d/unprivileged_usernsraw.apparmor = profileIn my opinion, based on what I’ve read, options 1 and 2 are insecure.
Option 3 and 4 are not really feasible, as it has to be done for every Incus container namespace (3 even after every start/restart)
(but nice for testing/understanding)
Option 5 is what I now use and I think is fine. I’ve disabled all other AppArmor profiles in the Incus container, to not introduce unwanted restrictions or unneeded permissions that might enable an attacker to escalate from regular user in the Incus container to root in the Incus container.
You have to have a container that supports installing AppArmor, but bonus is that you can use AppArmor in the Incus container to further restrict processes in the container.
Options 6 or any other solution that the Incus developers will come up with is of course the best in the long run.
There is also a setting kernel.apparmor_restrict_unprivileged_unconfined which might give problems, but that’s a whole other can of worms 
Running ps axZf or sudo aa-status on the host also helps to see what’s going on.
I did think about that, but I thought it was only affecting Ubuntu 25.04.
Anyway, there’s a simple way to test. Just read the current value:
# sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 0
If it comes back with 0, then this can’t be the problem. If it comes back with 1, then try changing it to 0. (But then the question becomes why the default is 0 on some systems but 1 on others)
I couldn’t find out where it’s explicitly turned on, but on all my Ubuntu 24.04 systems/VMs it’s turned on.
To reproduce:
incus launch images:ubuntu/noble mytest --vm
incus exec mytest apt update
incus exec mytest apt upgrade
incus exec mytest apt install apparmor
incus exec mytest -- cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
incus restart mytest
incus exec mytest -- cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
The first cat should give a 0, after the reboot, it should be set to 1
So I guess it depends on just having apparmor installed on the host, and no explicit persistent sysctl settings needed?
Yes, it seems to be to do with apparmor config in 24.04 which is not present in 22.04
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
$ sudo grep -R unprivileged_userns /etc 2>/dev/null
/etc/apparmor.d/unprivileged_userns:profile unprivileged_userns {
/etc/apparmor.d/unprivileged_userns: allow pix /** -> &unprivileged_userns ,
/etc/apparmor.d/unprivileged_userns: include if exists <local/unprivileged_userns>
If I launch an Ubuntu 22.04 VM, install apparmor and linux-image-generic-hwe-22.04, and reboot so I get the 6.8.0-65-generic kernel, I still see apparmor_restrict_unprivileged_userns=0
Sorry to necro this, but I’ve been struggling with AppArmor and Incus ever since AA v4 made it into Ubuntu 24.04. It’s really broken a lot of functionality with nested containers.
Can you elaborate on your option 5? AppArmor is already installed in my containers but I still run into issues.
“Option 5” solves the problem by making sure that the apparmor namespace that Incus creates for the container has a profile “unprivileged_userns” which Ubuntu 24.04 with the apparmor sysctl setting now requires.
Running aa-status inside the container (part of apparmor-utils package I believe) might help to see what’s going on, or aa-status on the Incus host, which will also show you all the profiles the kernel has loaded (including namespaced profiles).
Also note that in some cases apparmor denies doing something on the first try, but after that it will allow it. I wasn’t able to figure out why exactly, something with fork/exec, readable and executable permissions and the apparmor namespace transitioning.
Also note that depending on your setup profiles that are loaded in the host namespace may (depending on the path) be applied to binary paths inside the container, but this will be evident in the logging (on the host!)
Hope this helps!