Creating new containers/vms blocked by apparmor on NixOS

broizter · October 20, 2024, 12:33pm

When creating new containers or VMs on a NixOS host you first have to disable apparmor, otherwise you get errors and the creation fails. Is there any better workaround than disabling apparmor every time I wanna create something?

incus create images:debian/12 test
Creating test
Error: Failed instance creation: Failed creating instance from image: Unpack failed: Failed to run: tar --anchored --wildcards --exclude=dev/* --exclude=/dev/* --exclude=./dev/* --exclude=rootfs/dev/* --exclude=/rootfs/dev/* --exclude=./rootfs/dev/* --restrict --force-local -C /var/lib/incus/storage-pools/default/images/149e0ff3a1ca46db20336210704104db4c9409f2f33369dea26174d908757c29 --numeric-owner --xattrs-include=* -Jxf -: exit status 127 (tar: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory)

dmesg -T
[sön okt 20 14:32:58 2024] audit: type=1400 audit(1729427573.016:126): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus_archive-d6df5eea-1a10-4ad0-b163-43dbbc49e4b1" pid=2582270 comm="apparmor_parser"
[sön okt 20 14:32:58 2024] audit: type=1400 audit(1729427573.018:127): apparmor="DENIED" operation="open" class="file" profile="incus_archive-d6df5eea-1a10-4ad0-b163-43dbbc49e4b1" name="/nix/store/7px4n99mcmdzx8nygx59f28j8g7vj0kb-acl-2.3.2/lib/libacl.so.1.1.2302" pid=2582272 comm="tar" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[sön okt 20 14:32:58 2024] audit: type=1400 audit(1729427573.057:128): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_archive-d6df5eea-1a10-4ad0-b163-43dbbc49e4b1" pid=2582276 comm="apparmor_parser"

stgraber · October 21, 2024, 2:50am

So normally the AppArmor abstractions/base contains what’s needed for dynamic libraries to be loaded. All our profiles include that abstraction so we don’t need to specifically allow every single library.

There must be something odd going on with NixOS which causes that abstraction to not include a correct pattern for where Nix stores its libraries (/nix/storage/*/lib/so or something like that).

broizter · October 21, 2024, 9:28am

I see! Well I created an issue in NixOS/nixpkgs since I assume it’s something they need to fix with their packaging?

stgraber · October 21, 2024, 3:47pm

Yeah, you may be able to work around it with a /etc/apparmor.d/local/ file, though I’m not sure if that exists and/or work on Nix.

broizter · October 21, 2024, 3:54pm

It should be possible to implement a custom file. Do you mind helping me with what I should populate it with?

stgraber · October 21, 2024, 4:00pm

/nix/storage/*/lib/*so* mr,

That should do the trick, allows for reading and mapping of all shared objects.

broizter · October 21, 2024, 4:32pm

Looking into it I couldn’t find an easy to way to implement a custom file. It’s not a huge bug since I very rarely create new containers at this point. Hopefully it will be fixed upstream soon enough. Thanks for your help Stéphane!

MakiseKurisu · February 11, 2025, 2:31pm

I’m currently working on this issue when I tried to install the new OpenWrt 24.10 container today. Here is what I come up with:

  security.apparmor = {
    enable = true;
    policies = {
      "incusd".profile = ''
        # This profile allows everything and only exists to give the
        # application a name instead of having the label "unconfined"

        abi <abi/4.0>,
        include <tunables/global>

        profile incusd ${lib.getExe' config.virtualisation.incus.package "incusd"} flags=(unconfined) {
          userns,
          
          /nix/store/*/lib/*.so* mr,
          tar Ux,

          profile incusd-tar flags=(unconfined) {
            userns,
            /nix/store/*/lib/*.so* mr,
          }
        }
      '';
    };
  };

incusd was running unconfined on NixOS, so the basic structure is taken from /nix/store/lhsvb5ah5dphv6995dnhn8b0qxwr49z3-apparmor-profiles-4.0.3/etc/apparmor.d/Discord, which is an unconfined profile.

So far, the incusd profile is correctly applied, and I can see the profile name change if I change this value. However, the profile for tar is always incus_archive-GUID, and I simply can’t get it to use a custom profile.

Feb 11 22:09:57 nas kernel: audit: type=1400 audit(1739282997.839:385): apparmor="STATUS" operation="profile_load" profile="incusd" name="incus_archive-5d69d24a-9c44-46a2-9d77-7a28196f1c92" pid=8649 comm="apparmor_parser"
Feb 11 22:09:57 nas kernel: audit: type=1400 audit(1739282997.841:386): apparmor="DENIED" operation="open" class="file" profile="incus_archive-5d69d24a-9c44-46a2-9d77-7a28196f1c92" name="/nix/store/gmfwlnb6rwda4bwzih1cm4py494ld11r-acl-2.3.2/lib/libacl.so.1.1.2302" pid=8650 comm="tar" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Feb 11 22:09:57 nas kernel: audit: type=1400 audit(1739282997.878:387): apparmor="STATUS" operation="profile_remove" profile="incusd" name="incus_archive-5d69d24a-9c44-46a2-9d77-7a28196f1c92" pid=8652 comm="apparmor_parser"
Feb 11 22:09:57 nas incusd[8578]: time="2025-02-11T22:09:57+08:00" level=warning msg="Unpack failed" allowedCmds="[xz]" err="Failed to run: tar --anchored --wildcards --exclude=dev/* --exclude=/dev/* --exclude=./dev/* --exclude=rootfs/dev/* --exclude=/rootfs/dev/* --exclude=./rootfs/dev/* --restrict --force-local -C /var/lib/incus/storage-pools/default/containers/verified-bluejay --numeric-owner --xattrs-include=* -Jxf -: exit status 127 (tar: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory)" extension=.tar.xz file=/var/lib/incus/images/04419923a6788fd542356ab6fff19b776305d5593fb2f9219ac02b2e6c67343c path=/var/lib/incus/storage-pools/default/containers/verified-bluejay

So far, I have tried:

move tar profile to top level
match with Ux or Cx -> incusd-tar
match /**/tar, real full path, just tar
add the so rule in incusd and tar profiles

None of those worked.

MakiseKurisu · March 1, 2025, 9:07am

I think I figured out how this might work. An AppArmor execute wrapper was configured here with an embedded template. The wrapper is likely used by tar, as the reported profile name is incus_archive-27057a6f-6b34-4c13-96a5-79694e9de8d4, which should be created here.

Since the template have 2 include clauses, I checked how NixOS handled this, and come up with this:

  security.apparmor.includes."abstractions/base" = ''
    # Allow incusd to launch tar to extract image template
    # https://discuss.linuxcontainers.org/t/creating-new-containers-vms-blocked-by-apparmor-on-nixos/21908/6
    /nix/storage/*/lib/*so* mr,
  '';

I have confirmed the setting is now part of “abstractions/base”:

[excalibur@nas:~]$ cat /etc/apparmor.d/abstractions/base 
# Allow incusd to launch tar to extract image template
# https://discuss.linuxcontainers.org/t/creating-new-containers-vms-blocked-by-apparmor-on-nixos/21908/6
/nix/storage/*/lib/*so* mr,

include "/nix/store/lhsvb5ah5dphv6995dnhn8b0qxwr49z3-apparmor-profiles-4.0.3/etc/apparmor.d/abstractions/base"
r /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/share/locale/**,
r /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/share/locale.alias,
r /nix/store/yr4m7nrg1p1qbl7fr2n5h04qh3pnzbzh-glibc-locales-2.40-36/lib/locale/locale-archive,
r /etc/zoneinfo/Asia/Shanghai,
r /nix/store/78mhfhbhfhvx95hjv9hkjx8m0vadynjv-tzdata-2024b/share/zoneinfo/**,
r /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/share/i18n/**,

However, it is still not working:

Mar 01 17:05:32 nas audit[1678]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus_archive-a5c1fcea-f4b4-4ab1-99cd-d1509e6bb343" pid=1678 comm="apparmor_parser"
Mar 01 17:05:32 nas kernel: audit: type=1400 audit(1740819932.703:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus_archive-a5c1fcea-f4b4-4ab1-99cd-d1509e6bb343" pid=1678 comm="apparmor_parser"
Mar 01 17:05:32 nas kernel: audit: type=1400 audit(1740819932.705:11): apparmor="DENIED" operation="open" class="file" profile="incus_archive-a5c1fcea-f4b4-4ab1-99cd-d1509e6bb343" name="/nix/store/gmfwlnb6rwda4bwzih1cm4py494ld11r-acl-2.3.2/lib/libacl.so.1.1.2302" pid=1679 comm="tar" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Mar 01 17:05:32 nas audit[1679]: AVC apparmor="DENIED" operation="open" class="file" profile="incus_archive-a5c1fcea-f4b4-4ab1-99cd-d1509e6bb343" name="/nix/store/gmfwlnb6rwda4bwzih1cm4py494ld11r-acl-2.3.2/lib/libacl.so.1.1.2302" pid=1679 comm="tar" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Mar 01 17:05:32 nas audit[1681]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_archive-a5c1fcea-f4b4-4ab1-99cd-d1509e6bb343" pid=1681 comm="apparmor_parser"
Mar 01 17:05:32 nas incusd[1113]: time="2025-03-01T17:05:32+08:00" level=warning msg="Unpack failed" allowedCmds="[xz]" err="Failed to run: tar --anchored --wildcards --exclude=dev/* --exclude=/dev/* --exclude=./dev/* --exclude=rootfs/dev/* --exclude=/rootfs/dev/* --exclude=./rootfs/dev/* --restrict --force-local -C /var/lib/incus/storage-pools/default/containers/openwrt --numeric-owner --xattrs-include=* -Jxf -: exit status 127 (tar: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory)" extension=.tar.xz file=/var/lib/incus/images/d58643744101cc23519a37c969815e4ce3c03256366ff603323e9868bb1e3a6b path=/var/lib/incus/storage-pools/default/containers/openwrt
Mar 01 17:05:32 nas kernel: audit: type=1400 audit(1740819932.743:12): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_archive-a5c1fcea-f4b4-4ab1-99cd-d1509e6bb343" pid=1681 comm="apparmor_parser"

Maybe it is searching a different AA include path.

Also we might want to copy what libvirt does to have a abstractions/incusd file as a global override.

MakiseKurisu · March 1, 2025, 10:58am

Oh shit, I copied the wrong rule!

After fixing it, I got some new errors. The final ruleset can be found in nixpkgs PR #386060.