Systemd 247 with LXD 4.04 breaks systemd-networkd

benpro · December 5, 2020, 6:44am

I updated my Arch Linux container, rebooted it and systemd-networkd cannot start anymore.

[root@twt ~]# systemctl --version
systemd 247 (247.1-1-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

[root@twt ~]# systemctl status systemd-networkd
● systemd-networkd.service - Network Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/systemd-networkd.service.d
             └─lxc.conf
     Active: failed (Result: exit-code) since Sat 2020-12-05 06:27:09 UTC; 14min ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
    Process: 62 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=226/NAMESPACE)
   Main PID: 62 (code=exited, status=226/NAMESPACE)

Dec 05 06:27:09 twt systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE

Dec 05 06:27:09 twt systemd[55]: systemd-networkd.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Dec 05 06:27:09 twt systemd[55]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Permission denied
Dec 05 06:27:09 twt systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE

Dec 05 07:27:09 lxd00 audit[19428]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19428 comm="(resolved)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19423]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19423 comm="(resolved)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19420]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19420 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19411]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19411 comm="(resolved)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19407]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19407 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19403]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19403 comm="(resolved)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19395]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19395 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.774:72): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19379 comm="(resolved)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19379]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19379 comm="(resolved)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.694:71): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19384 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19384]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19384 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.578:70): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19370 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19370]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19370 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.522:69): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19365 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19365]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19365 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.366:68): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19361 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19361]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19361 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.334:67): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19359 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19359]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19359 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.270:66): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19358 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19358]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19358 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 kernel: audit: type=1400 audit(1607149629.242:65): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19357 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Dec 05 07:27:09 lxd00 audit[19357]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-twt_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/proc/" pid=19357 comm="(networkd)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"

Seems related to AppArmor profile.
Is it a know issue?

stgraber · December 5, 2020, 2:56pm

Looks like systemd is trying to mount a complete new copy of /proc in this case which would allow bypassing AppArmor restrictions…

The likely fix for this is to find what systemd feature is causing that and turn it off in the lxc.conf override file we already ship.

Alternatively you can allow such operations by setting security.nesting to true but that’s not exactly something we want to do by default.

benpro · December 6, 2020, 2:48am

Indeed.

So, I added BindReadOnlyPaths=/sys /proc in/etc/systemd/system/systemd-networkd.service.d/lxc.conf and it fixes the main issue.

Now I see this:

Dec 06 02:46:03 twt systemd-networkd[16511]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted
Dec 06 02:46:03 twt systemd-networkd[16511]: eth0: Cannot disable kernel IPv6 accept_ra for interface, ignoring: Read-only file system

I am not sure of what should be done.

Anyway it will be necessary to patch lxc.conf for images provided by LXD.

benpro · December 6, 2020, 3:29am

Actually, every service fails…

Dec 06 00:00:03 pad systemd[28586]: logrotate.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied

Dec 06 03:15:38 pad udevadm[48]: Failed to write 'add' to '/sys/bus/acpi/uevent': Permission denied

Dec 06 03:15:39 pad systemd[61]: systemd-resolved.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied

Dec 06 03:15:39 pad systemd[73]: systemd-logind.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied

Dec 06 03:15:39 pad systemd[83]: systemd-hostnamed.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied

Dec 06 03:15:39 pad systemd[85]: etherpad-lite.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied

Patching every service file is troublesome… I wonder if we can adjust some systemd settings…

benpro · December 6, 2020, 3:39am

I opened a systemd issue if it may help: https://github.com/systemd/systemd/issues/17866

stgraber · December 6, 2020, 6:13am

@brauner for awareness

jade · December 15, 2020, 4:10pm

Just wanted to say this fixed my issue. I seemed to have the same issue. No IPv4 address when I started an arch-linux container via lxdui. (Host was 5.8.18-1-MANJARO). Googled around and finally came across this. I already had “/sys” in the file so I added “/proc” and rebooted and I now have networking! Thanks

I had previously executed "sudo lxc profile set default security.syscalls.blacklist “keyctl errno 38"” as suggested in https://wiki.archlinux.org/index.php/User:Aimilius/LXD but that did not resolve my issue.

elgs · December 20, 2020, 10:05am

I have the same issue after a reboot of the container. Now I dare not reboot other containers.

elgs · December 20, 2020, 10:31am

My workaround:

yay -U /var/cache/pacman/pkg/systemd-246.6-1-x86_64.pkg.tar.zst
yay -U /var/cache/pacman/pkg/systemd-libs-246.6-1-x86_64.pkg.tar.zst

stfn · January 8, 2021, 4:04pm

I added BindReadOnlyPaths=/sys /proc in/etc/systemd/system/systemd-networkd.service.d/lxc.confon the host, rebooted the host, created a new container with lxc launch images:archlinux/current/amd64 radarr3

EDIT: Of course this should go in the container, but it doesn’t make a difference.

But it still gives the same error as OP:

Jan 08 15:46:15 radarr3 systemd[129]: systemd-networkd.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Jan 08 15:46:15 radarr3 systemd[129]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Permission denied

I also tried to enable dhcpcd on the eth0 interface (systemctl start dhcpcd@eth0.service) but that also fails:

Jan 08 15:42:20 radarr3 systemd[1]: Created slice system-dhcpcd.slice.
░░ Subject: A start job for unit system-dhcpcd.slice has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░  
░░ A start job for unit system-dhcpcd.slice has finished successfully.
░░  
░░ The job identifier is 482.
Jan 08 15:43:50 radarr3 systemd[1]: sys-subsystem-net-devices-eth0.device: Job sys-subsystem-net-devices-eth0.device/start timed out.
Jan 08 15:43:50 radarr3 systemd[1]: Timed out waiting for device /sys/subsystem/net/devices/eth0.
░░ Subject: A start job for unit sys-subsystem-net-devices-eth0.device has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░  
░░ A start job for unit sys-subsystem-net-devices-eth0.device has finished with a failure.
░░  
░░ The job identifier is 535 and the job result is timeout.
Jan 08 15:43:50 radarr3 systemd[1]: Dependency failed for dhcpcd on eth0.
░░ Subject: A start job for unit dhcpcd@eth0.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░  
░░ A start job for unit dhcpcd@eth0.service has finished with a failure.
░░  
░░ The job identifier is 481 and the job result is dependency.
Jan 08 15:43:50 radarr3 systemd[1]: dhcpcd@eth0.service: Job dhcpcd@eth0.service/start failed with result 'dependency'.
Jan 08 15:43:50 radarr3 systemd[1]: sys-subsystem-net-devices-eth0.device: Job sys-subsystem-net-devices-eth0.device/start failed with result 'timeout'.

Am I missing something?