The below script demonstrates that dnsmasq fails to run in a nested container on Ubuntu noble, preventing Incus to run inside an unprivileged container. The same setup works on, e.g. Debian trixie as the underlying host.
incus launch --vm images:ubuntu/noble noble
incus wait noble agent
incus exec noble -- sh -eu<<__EOF__
# install incus in the VM
apt-get -y install curl
curl https://pkgs.zabbly.com/get/incus-stable | sh
incus admin init --auto
# launch an unprivileged container with security.nesting=true
incus launch images:debian/trixie trixie
incus config set trixie security.nesting=true
# install incus and try to configure it
incus wait trixie ip
incus exec trixie -- sh -eu<<__EOF2__
apt-get -y install curl
curl https://pkgs.zabbly.com/get/incus-stable | sh
incus admin init --auto
# fail
__EOF2__
__EOF__
The error points at AppArmor. strace shows that dnsmasq fails on the socket() syscall with EACCES and audit.log shows `apparmor=”DENIED” operation=”create” class=”net” info=”failed type and protocol match” error=-13 […] family=”unix” sock_type=”dgram” protocol=0 […]`
Adding the raw.lxc=lxc.apparmor.profile=unconfined allows dnsmasq to run, but containers still fail to start with System doesn’t have a functional idmap setup despite ?idmap being setup.
incus exec noble -- sh -eu<<__EOF__
incus config set trixie raw.lxc=lxc.apparmor.profile=unconfined
incus restart trixie
incus wait trixie ip
incus exec trixie -- incus admin init --auto
__EOF__
Historically, Incus/LXD had a better integration with Ubuntu, but it doesn’t seem to be the case today. Any idea beside wiping my workstation?
I tried this and it worked (create a VM, then create a container with security.nesting enabled, enter that container, install Incus [successful], then launch a new container). In my case I used all Ubuntu images.
The typo is here. When you set security.nesting, you would need to restart the instance so that the change can take effect. For the change to take effect, Incus needs to recreate those namespaces, etc.
incus launch images:debian/trixie trixie
incus config set trixie security.nesting=true
Or, you would incus launch images:debian/trixie trixie -c security.nesting=true.
Good catch about the ordering. Indeed, it does work with all Ubuntu images, but fails with trixie in Ubuntu even with “-c”. On my Debian sid host, nested Incus works with any image inside the container.
incus launch --vm images:ubuntu/noble noble
incus wait noble agent
incus exec noble -- sh -eu<<__EOF__
# install incus in the VM
apt-get -y install curl
curl https://pkgs.zabbly.com/get/incus-stable | sh
incus admin init --auto
# launch an unprivileged container with security.nesting=true
incus launch images:debian/trixie trixie -c security.nesting=true
# install incus and try to configure it
incus wait trixie ip
incus exec trixie -- sh -eu<<__EOF2__
apt-get -y install curl
curl https://pkgs.zabbly.com/get/incus-stable | sh
incus admin init --auto
# fail
__EOF2__
__EOF__
I confirm the following error. Incus Ubuntu VM, install in there Incus, then create a trixie container with security.nesting=true, then try to initialize Incus in that container.
root@trixie:~# incus admin init --auto
Error: Failed to create local member network "incusbr0" in project "default": The DNS and DHCP service exited prematurely: exit status 3 ("dnsmasq: cannot open log : Permission denied")
root@trixie:~#
When you run incus admin init in trixie, you get this warning
We detected that you are running inside an unprivileged container.
This means that unless you manually configured your host otherwise,
you will not have enough uids and gids to allocate to your containers.
Your container's own allocation can be reused to avoid the problem.
Doing so makes your nested containers slightly less safe as they could
in theory attack their parent container and gain more privileges than
they otherwise would.
Would you like to have your containers share their parent's allocation? (yes/no) [default=yes]:
I do not know where the source of the issue is AppArmor, or whether the idmap needs special configuration.
Yes, the warning when running “incus admin init” interactively is expected due to the container being unprivileged with an idmap size of 65536 by default. In other words, the container running the nested Incus doesn’t have enough idmap to run unprivileged containers. If you choose “yes”, it sets “security.privileged=true” on the default profile. This last step needs to be done manually when using “init –auto”.
This would require to identify which AppArmor rule is causing the issue. At the moment, it affects nested containers when the inner container is Debian Trixie.
And the current workaround is to disable AppArmor on the outer container (not so good).
I do not know if you can try out with different versions so as to try to identify which version works, which does not, etc.