Symptoms and Preliminary Investigation
On an Ubuntu 25.04 host running Incus 6.13 from Zabbly repo I created an Ubuntu 24.04 guest container, opened a shell to it and ran apt update… and it froze.
From another host shell, I checked the container’s process tree, and it looked like this:
ihor@mad-desktop:~$ incus exec noble-lxc -- ps af
PID TTY STAT TIME COMMAND
245 pts/0 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud - 115200,38400,9600 linux
603 pts/2 Rs+ 0:00 ps af
267 pts/1 Ss 0:00 su -l
278 pts/1 S 0:00 \_ -bash
516 pts/1 S+ 0:00 \_ apt update
519 pts/1 S+ 0:00 \_ apt update
520 pts/1 S+ 0:00 \_ sh -c -- [ ! -e /run/systemd/system ] || [ $(id -u) -ne 0 ] || systemctl start --no-block apt-news.service esm-cache.service >/dev/null 2>&
522 pts/1 S+ 0:00 \_ systemctl start --no-block apt-news.service esm-cache.service
523 pts/1 S+ 0:00 \_ /usr/bin/systemd-tty-ask-password-agent --watch
So it looks like systemctl is blocked by systemd-tty-ask-password-agent for some reason.
There are many complaints on the Internet about systemd-tty-ask-password-agent being a blocker, but this time it wasn’t the case.
Here’s an interesting picture that I got from running strace on the host:
19596 19:07:45.041859 execve("/usr/bin/systemctl", ["systemctl", "start", "--no-block", "apt-news.service", "esm-cache.service"], 0x5f98d387fc30 /* 19 vars */) = 0
...
19596 19:07:45.120102 getpid() = 522 /* 19596 in strace's PID NS */
...
19596 19:07:45.194694 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x740668dbdb90) = 523 /* 19597 in strace's PID NS */
...
19597 19:07:45.343764 execve("/usr/bin/systemd-tty-ask-password-agent", ["/usr/bin/systemd-tty-ask-passwor"..., "--watch"], 0x7ffc3897b0a8 /* 19 vars */) = 0
...
19596 19:07:45.378738 kill(523 /* 19597 in strace's PID NS */, SIGTERM) = -1 EACCES (Permission denied)
19596 19:07:45.378819 waitid(P_PID, 523 /* 19597 in strace's PID NS */, <detached ...>
Here’s what’s happening:
systemctl(PID 522 in container, PID 19596 on host) spawnssystemd-tty-ask-password-agent(PID 523 in container, PID 19597 on host) just in case it would need to handle password input from the user;- it successfully starts apt-news and esm-cache services (not shown in this picture);
systemctlthen decides it no longer needssystemd-tty-ask-password-agentand tries tokillit, but is being forbidden from doing so (EACCESS);- it then starts to
waitidforsystemd-tty-ask-password-agentto die out of old age, but it never happens, hence the freeze we observe.
Steps to Reproduce
-
On an Ubuntu 25.04 host install incus 6.13 from Zabbly repo:
ihor@mad-desktop:~$ apt policy incus incus: Installed: 1:6.13-ubuntu24.04-202506030459 Candidate: 1:6.13-ubuntu24.04-202506030459 Version table: *** 1:6.13-ubuntu24.04-202506030459 500 500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages 100 /var/lib/dpkg/status 1:6.13-ubuntu24.04-202506021921 500 500 https://pkgs.zabbly.com/incus/stable noble/main amd64 Packages 6.0.3-4 500 500 http://pl.archive.ubuntu.com/ubuntu plucky/universe amd64 Packages -
Create an unprivileged Ubuntu 24.04 guest container:
ihor@mad-desktop:~$ incus launch images:ubuntu/noble noble-lxc --type=aws:t2.micro -
Enter the container:
ihor@mad-desktop:~$ incus shell noble-lxc -
Spawn a background process, then try to
killit:root@noble-lxc:~#yes > /dev/null & [1] 354 root@noble-lxc:~#kill $! -bash: kill: (354) - Permission denied
The very same steps when on an Ubuntu 24.04 do work: you can kill the yes process just fine.
This leads me to thinking that something has changed from 24.04 to 25.04.
Investigation
Ubuntu 25.04
After trial and error, I narrowed down the problem to AppArmor blocking the syscall:
ihor@mad-desktop:~$ journalctl --boot --grep=apparmor --since="-1 minute"
Jun 08 21:39:03 mad-desktop kernel: audit: type=1400 audit(1749411543.030:5475636): apparmor="DENIED" operation="signal" class="signal" profile="incus-noble-lxc_</var/lib/incus>" pid=233658 comm="bash" requested_mask="send" denied_mask="send" signal=term peer="incus-noble-lxc_</var/lib/incus>//&unconfined"
My understanding of this log message is that bash was denied to send a signal to a process with AppArmor label incus-noble-lxc_</var/lib/incus>//&unconfined.
So I checked the signal related rules in AppArmor profile for the container:
ihor@mad-desktop:~$ sudo grep -E '(^profile|signal)' /var/lib/incus/security/apparmor/profiles/incus-noble-lxc
profile "incus-noble-lxc_</var/lib/incus>" flags=(attach_disconnected,mediate_deleted) {
# Allow normal signal handling
signal (receive),
signal peer=@{profile_name},
The variable @{profile_name} should expand to incus-noble-lxc_</var/lib/incus> which does not equal to incus-noble-lxc_</var/lib/incus>//&unconfined I saw in the log.
If I understand correctly, //& means profile intersection, unconfined is an “everything is allowed” profile, so in theory both incus-noble-lxc_</var/lib/incus> and incus-noble-lxc_</var/lib/incus>//&unconfined should mean exactly the same, but it looks like AppArmor wants a literal string equality.
Ubuntu 24.04
Since it works on Ubuntu 24.04, I wanted to check what peer would be seen in the audit logs. So I adjusted AppArmor profile to deny sending SIGTERM:
root@incus-host:~# incus config set noble-lxc raw.apparmor="audit deny signal (send) set=("term"),"
Then I did the yes kill dance and looked at the logs:
root@incus-host:~# journalctl --boot --grep=apparmor --since="-1 minute"
Jun 08 20:58:28 incus-host kernel: audit: type=1400 audit(1749416308.401:131): apparmor="DENIED" operation="signal" class="signal" profile="incus-noble-lxc_</var/lib/incus>" pid=4840 comm="bash" requested_mask="send" denied_mask="send" signal=term peer="incus-noble-lxc_</var/lib/incus>"
Conclusion
On Ubuntu 24.04 the peer is $CONTAINER_NAME_</var/lib/incus>, on Ubuntu 25.04 the peer is $CONTAINER_NAME_</var/lib/incus>//&unconfined. The latter doesn’t match the rule in AppArmor profile, so the signal is denied.
Temporary Fix
I suppose not being able to kill your own children can cause a lot of software to misbehave, not just apt or systemd.
I don’t know what caused this change, but for now, I ended up with a profile containing an extra AppArmor profile entry:
incus profile create apparmor_fix
incus profile set apparmor_fix raw.apparmor="signal peer=@{profile_name}//&unconfined,"
Then it can either be used to attach to existing containers:
incus attach apparmor_fix noble-lxc
or to launch new containers:
incus launch images:ubuntu/noble noble-lxc2 --type=aws:t2.micro --profile default --profile apparmor_fix
In the long run, it would be nice to either have this entry in the official AppArmor template that Incus uses, or figure out why does Ubuntu 25.04 behave differently than Ubuntu 24.04.