Issues in Ubuntu 26.04 - apt not working in unprivileged LXCs

I installed 26.04 on one of my Incus hosts yesterday. In some LXC instances (running 24.04.4) I’m seeing a problem I remember we saw with earlier Ubuntu releases, where apt update hangs even though other networking works fine. If the instance is privileged, then apt works.

This was related to AppArmor I believe?

That could be, yes. AppArmor is a powerfull Armor and bug creator :slight_smile:

Seems it’s easy to disable apparmor (on the host):

https://linuxconfig.org/how-to-disable-apparmor-on-ubuntu-26-04

For sure you can check dmsg / journalctl for apparmor messages.

Please report back

Also like previous issues, eventually all your RAM is consumed because parent processes in the instances can’t kill their children and the server becomes unresponsive.

When I get a chance (two week old baby!) I’ll see what’s in the logs.

1 Like

I get basically a continuous repeat of these two AppArmor issues for every instance :

Apr 24 18:49:09 librarian kernel: audit: type=1400 audit(1777052949.453:61435): apparmor=“DENIED” operation=“ptrace” class=“ptrace” profile=“incus-media_</var/lib/incus>” pid=4565 comm=“systemd-journal” requested_mask=“read” denied_mask=“read” peer=“incus-media_</var/lib/incus>//&unconfined”
Apr 24 18:49:09 librarian kernel: audit: type=1400 audit(1777052949.585:61436): apparmor=“DENIED” operation=“signal” class=“signal” profile=“incus-media_</var/lib/incus>” pid=260735 comm=2E4E455420424743 requested_mask=“send” denied_mask=“send” signal=rtmin+2 peer=“incus-media_</var/lib/incus>//&unconfined”

Did you try to disable apparmor - sudo aa-teardown; sudo aa-status. as you suspected apparmor seems the problem.

If you see incus working as is after that you might wanna add/modify the lxd/incus profile - I would need to have the create the same setup to give correct instructions.

Seems very similar to the issues we had with Noble, so I’d guess the fix is the same :

1 Like

Exasperating that the problem repeated across two distro upgrades. Perhaps no-one at Canonical runs test a system with LXD on it, since I’d assume the same issues would apply. :slight_smile:

Yeah apt update works again with aa off.

But … I have one instance whose only NIC is on a different VLAN and that will not compete Netplan apply.

The instance NIC has a static IPv4 address in Netplan, but instead the NIC tries to get a dhcp6 address. If I try to netplan apply the command hangs until I ctrl-C and nothing appears in the logs. After that though the instance is quite unhappy - Failed to connect to system scope bus via local transport : No such file or directory and I have to shut down the instance from the host.

journalctl is unhelpful, telling me AppArmor is stopping it even though AA is supposedly off.

22:42:39 librarian kernel: audit: type=1400 audit(1777066959.157:347942): apparmor=“DENIED” operation=“signal” class=“signal” profile=“incus-plex_</var/lib/incus>” pid=509119 comm=“systemd-udevd” requested_mask=“send” denied_mask=“send” signal=term peer=“incus-plex_</var/lib/incus>//&unconfined”

  1. Try to boot without the apparmor service at all.
  2. When enabling apparmor again make sure you do it the right order or you get lot’s of troubles (on my Arch WS kernel apparmor without service apparmor = blackscreen)
  3. If your plesk container works w/o apparmor on the host you might transfer apparmor profiles from the container (or any other older ubuntu) to the host … OR JUST keep apparmor disabled if you’r able to ignore security.

Easier is to follow the fix in the post I linked :

❯ incus profile create apparmor_fix
Profile apparmor_fix created
❯ incus profile set apparmor_fix raw.apparmor=“signal peer=@{profile_name}//&unconfined,”
❯ incus profile add problem_instance apparmor_fix
Profile apparmor_fix added to problem_instance
❯ incus restart problem_instance

1 Like

I’ve done a lot more digging.

The root cause seems to be that from 25.04 onwards Canonical has been enforcing AppArmor Profile Stacking (/sys/kernel/security/apparmor/features/domain/stack = yes) so we see other things breaking elsewhere too (eg AppArmor: signal rule does not match stacked profile peer on kernel 6.17 · Issue #12886 · containerd/containerd · GitHub ) and don’t get me started on Inception-like things like I had AppArmor complaining about something (error=-13 info="Failed name lookup - disconnected path") which turned out to be Ghostscript (!) inside the Paperless NGX container stack on my Docker LXC not being able to be properly contained by AppArmor.

I don’t know why Canonical hasn’t provided a fix but in the meantime rather than the profile route - I only had partial success with that and it was turning into whack-a-mole with AppArmor errors - I suggest instead to revert for now to AppArmor 24.04 behaviour where it no longer stacks AppArmor profiles :

sudo sysctl -w kernel.apparmor_restrict_unprivileged_unconfined=0 

echo "kernel.apparmor_restrict_unprivileged_unconfined=0" | sudo tee /etc/sysctl.d/99-apparmor-incus.conf

This seems to work but I am not a real sysadmin so please use with caution.

@stgraber I realise you said elsewhere you’re not interested in correcting problems in a brand new distro version, but maybe over time the general AppArmor stacked profile thing may become more of a general problem so I’ll just leave this here to help future travellers.

Edited to add : this removes some protection from kernel attacks from within the LXC, but was (as far as I can tell) the standard setup in 22.04 etc. so please do your own risk assessment.

2 Likes

And replying to myself - being a questioning type I wondered, why didn’t this break LXD in the same way? Why would Canonical shoot themselves in the foot with their own change?

The answer seems to be that it does, and so they seem to have hardcoded the exact workaround I provided above into the LXD snap. :frowning: Look for the euphamistic “manage_apparmor_restrictions” - line 439 for now - in the linked file.

Hence it looks like there will be no quick fix for this for Ubuntu users unless other things (eg the containerd issue I found) start to hurt.

1 Like