Security.syscalls.intercept.mknod has no effect on Incus 7.0 (empty kernel_features/lxc_features) — works on 6.0.4, same kernel

Incus
, , ,
1

Summary

On an Incus 7.0.0 cluster, security.syscalls.intercept.mknod=true has no
effect:
mknod of even the whitelisted devices (/dev/null 1:3, overlayfs whiteout 0:0) fails with EPERM inside an unprivileged container. As a result, Docker (overlay2) inside an unprivileged system container cannot pull multi-layer images (failed to mknod('/home/ubuntu', S_IFCHR, 0): operation not permitted).

incus info reports empty kernel_features: {} and lxc_features: {}.

The exact same workload works on another host running Incus 6.0.4 with the identical kernel — where kernel_features/lxc_features are fully populated.

Environment (affected)

  • Incus: 7.0.0 (Zabbly lts-7.0, package 1:7.0-debian13-202605301824)
  • OS: Debian GNU/Linux 13 (trixie)
  • Kernel: 6.12.90+deb13.1-amd64
  • Storage: ZFS · 3-node cluster · unprivileged containers
  • driver_version (lxc): 7.0.0 (bundled liblxc)

incus info (relevant excerpt):
kernel_features: {}
lxc_features: {}

Working comparison

Different host, Incus 6.0.4, same kernel 6.12.90+deb13.1-amd64, Debian 13:
kernel_features:
idmapped_mounts: “true”
seccomp_listener: “true”
seccomp_listener_continue: “true”
…
lxc_features:
seccomp_notify: “true”
seccomp_proxy_send_notify_fd: “true”
…
Here interception works as expected.

Steps to reproduce (on the 7.0 host)

incus launch images:debian/13/cloud t1 -c security.syscalls.intercept.mknod=true
incus exec t1 – mknod /tmp/n c 1 3

→ mknod: /tmp/n: Operation not permitted (expected: success)

What I verified

  • The generated seccomp policy is correct — it contains mknod notify/mknodat notify for S_IFCHR/S_IFBLK, and the runtime config has lxc.seccomp.notify.proxy = unix:/run/incus/seccomp.socket.
  • incusd is listening on /run/incus/seccomp.socket.
  • With incusd --debug, the handler receives the notification:
    Handling mknodat syscall … syscall_args=“&{8630 259 /tmp/n}”
    Send seccomp notification for id(…) but the container still gets EPERM, and no error/warning is logged (even at debug).
  • The device is on the documented allow-list (both /dev/null 1:3 and whiteout 0:0).

Ruled out

  • Kernel: /proc/sys/kernel/seccomp/actions_avail includes user_notif; same kernel works on 6.0.4.
  • Not a boot-time race: empty maps persist after a clean reboot and after a manual systemctl restart incus in a settled environment.
  • No systemd hardening on the unit (MemoryDenyWriteExecute=no, no SystemCallFilter).
  • vm.memfd_noexec = 0.
  • Bundled liblxc version matches Incus (7.0.0).

Questions

  1. Is the empty kernel_features/lxc_features a known regression in the Incus 7.0 (Zabbly lts-7.0) build, and is it the cause of the interception handler returning EPERM for whitelisted devices?
  2. What determines these maps being empty at daemon startup (they populate on 6.0.4, same kernel), and how can I restore working syscall interception on 7.0?

Happy to provide full incus info, the generated seccomp policy, or --debug logs.

2

Did you try paths other than /tmp?

If /tmp is a tmpfs, then there is no way for the host to directly mknod the node there due to how kernel protection works. There used to be some pretty ugly fallback to try and address that through temporary bind-mounts, but I think we dropped some of that due to how fragile and ephemeral it is.

Would be interesting to try say /root/foo or something to see if that’s the problem.

There’s also an issue with relative mknodat syscalls which we fixed earlier this week, but your paths here are absolute so shouldn’t be the issue.

3

Thanks a lot, Stéphane — you were right on both counts, and it turned my original diagnosis on its head. Writing up what I found in case it’s useful to anyone hitting the same wall.

  1. The tmpfs point — spot on. My “manual mknod fails” test was indeed running in a tmpfs (/tmp), so the EPERM there was the expected kernel behaviour, nothing to do with interception. That artifact is what led me to the (wrong) “interception is dead / empty feature-maps is the cause” conclusion. My apologies for the noise on that.

Re-testing outside tmpfs — on the container’s ZFS rootfs and on a real ext4 mount — interception works perfectly on 7.0.0. Every variant I could throw at it succeeds:

Test (unprivileged container, intercept.mknod=true)

:white_check_mark: mknod /root/foo c 1 3 (whitelisted device, zfs rootfs)

:white_check_mark: mknod /root/wh c 0 0 (overlay whiteout, absolute)

:white_check_mark: mknod /var/lib/docker/testnode c 0 0 (whiteout on ext4)

:white_check_mark: mknodat(AT_FDCWD, “relnode”, c 0 0) (relative)

:white_check_mark: mknodat(dirfd, “home/ubuntu”, c 0 0) (dir-fd–relative, multi-component)

So the “empty kernel_features/lxc_features” I reported is a display/cosmetic difference from 6.0.4, not the cause
— interception is fully functional. Sorry for the red herring in the title.

  1. The relative mknodat fix — that’s the real one. The workload that actually fails is docker pull of a multi-layer image inside the unprivileged system container (Docker daemon nested, overlay2). It still fails, identically, on the current lts-7.0 builds:

failed to register layer: failed to mknod(‘/home/ubuntu’, S_IFCHR, 0): operation not permitted

I couldn’t reproduce it with a one-off mknodat() probe (all pass, see table), but the real containerd overlay applier reproduces it every time. And it lines up exactly with the commit you merged recently:

▎ 86395a28 — incusd/forksyscall: Fix mknod emulation for relative paths. The parent directory of the target is already opened and checked, so mknodat() must be passed the base name, not the full path. Closes #3613

containerd applies each layer with mknodat(rootfd, “home/ubuntu”, …) against an already-open directory fd — precisely the case that fix addresses. So my case is the same bug as #3613.

  1. Where I’m stuck: getting the fix onto lts-7.0. The fix is dated 2026-07-08. The newest build on the Zabbly lts-7.0 channel right now is 1:7.0-debian13-202606270707 (2026-06-27), so 86395a28 isn’t in any published lts-7.0 package yet.

Question: will 86395a28 be cherry-picked to the 7.0 LTS branch, and roughly when can I expect a Zabbly lts-7.0 build that includes it? I’d rather bump the LTS package and re-test than switch channels (stable is already at 7.2, and I want to stay on LTS). Once a build with the fix is available I’ll confirm the docker pull succeeds and report back here.

Thanks again for the quick and accurate steer — much appreciated.

4

Pretty sure the fix is in 7.0.1 released yesterday and so will show up soon when those packages are published over the next few days.