NixOS images broken on Incus running on Ubuntu with kernel 6.5+

Incus
1

I am unable to run containers with nixos images. The containers get created but there is no way to interact with them.

After seeing the release notes for incus 0.5.1, I suspect it might be related.
I have tried with 0.5.1 as well without success.

$ incus version
Client version: 0.5.1
Server version: 0.5.1
$ incus launch images:nixos/23.11 nixos
Launching nixos
$ incus exec nixos bash
Error: Command not found
$ incus exec nixos sh
sh-5.2# ls
sh: ls: command not found
2

I’m not aware of anything in Incus 0.5 that would really explain this.
Most likely, something has changed in the image itself.

@adamcstephens any ideas?

3

The current 23.11 container image works fine for me on 0.4. I’m preparing the 0.5.1 nixos upgrade and I’ll test again once that’s done.

It looks like the PATH is missing for some reason


─❯ incus launch images:nixos/23.11 test
Creating test
Starting test
─❯ incus exec test bash
[root@nixos:~]# echo $PATH
/run/wrappers/bin:/root/.nix-profile/bin:/nix/profile/bin:/root/.local/state/nix/profile/bin:/etc/profiles/per-user/root/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
4 
stgraber@dakara:~$ incus exec nixos sh
sh-5.2# env
PWD=/root
container=lxc
HOME=/root
LANG=C.UTF-8
TERM=xterm
USER=root
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/run/current-system/sw/bin
_=/usr/bin/env
sh-5.2#
5

/run/current-system appears to be missing

6

Oh, I bet it’s the new apparmor stupidity


@abiosoft what kernel are you running?

7

I tested on both Ubuntu 22.04 LTS (kernel 6.5) and Ubuntu 23.10 (kernel 6.6).

8

Yeah, that’s the issue, it’s the new apparmor “mediation” of the new VFS API which is breaking NixOS on recent kernels.

Basically AppArmor just straight up rejects every single attempt at performing bind-mounts through the new VFS API because they don’t have the logic to actually apply their policy.

I reported that kernel regression upstream here:
https://lore.kernel.org/all/CA+enf=u0UmgjKrd98EYkxFu7FYV8dR1SBYJn_1b0Naq=3twbbQ@mail.gmail.com/

The workaround is pertty ugly, basically to avoid this apparmor breakage, you need to disable the use of the new VFS API in your container. This can be done with:

incus stop NAME
printf "2\ndenylist\nreject_force_umount\n[all]\nfsconfig errno 38\nfsopen errno 38\n" | incus config set NAME raw.seccomp -
incus start NAME
9

I’ve confirmed 0.5.1 is not the issue, so it must be the host system. It works fine on the NixOS as a host system, but there is no apparmor.

─❯ incus version
Client version: 0.5.1
Server version: 0.5.1
─❯ incus launch images:nixos/23.11 test
Launching test
─❯ incus exec test bash
[root@nixos:~]# echo $PATH
/run/wrappers/bin:/root/.nix-profile/bin:/nix/profile/bin:/root/.local/state/nix/profile/bin:/etc/profiles/per-user/root/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin
[root@nixos:~]# grep BUILD_ID /etc/os-release
BUILD_ID="23.11.3640.56911ef3403a"
10

Another potentially easier workaround (less specific though) is to use security.nesting=true.

incus launch images:nixos/23.11 -c security.nesting=true seems to be working fine

11

@stgraber both of your suggested workarounds work fine, but I’m picking -c security.nesting=true as the solution due to it being the cleaner option.

Thanks.

FYI:
I’m currently running Incus on two Macs, with Asahi Linux on m1 Mac mini and an Ubuntu VM on m1 Macbook pro. Thanks for the hardwork.

12

That should take care of this out of the box once landed.
It still won’t work for privileged containers, for those I don’t know that there’s a lot we can do other than maybe automatically blocking those system calls.

But as we don’t really want folks running privileged containers, automatically handling those can wait :slight_smile:

1 Like
13

It is recommended to run NixOS nested containers regardless, as the Nix daemon relies on nesting to build anything.