3.0.1 - AppArmor buggy kernel patches causing problems in systemd based unpriviliged containers

LXC
1

@stgraber @brauner

testing systemd 239 in an unpriviliged arclinux container produces

Summary

systemd[58]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[58]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
systemd[63]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[63]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: systemd-networkd.service: Start request repeated too quickly.
systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
systemd[1]: systemd-networkd.socket: Failed with result ‘service-start-limit-hit’.

no such issue with systemd 238

Opened an issue networkd - device configuration never completes inside lxc container · Issue #9427 · systemd/systemd · GitHub and the systemd developers wondering whether DynamicUser= is supported in LXC or being a restriction of the unpriviliged environment

2

DynamicUser=yes works fine for me with systemd 239 in an ArchLinux container:

[root@arch1 ~]# systemctl status systemd-networkd
● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-27 13:20:43 UTC; 1min 4s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 54 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 1.3M
   CGroup: /system.slice/systemd-networkd.service
           └─54 /usr/lib/systemd/systemd-networkd

Jun 27 13:20:43 arch1 systemd[1]: Starting Network Service...
Jun 27 13:20:43 arch1 systemd-networkd[54]: Enumeration completed
Jun 27 13:20:43 arch1 systemd[1]: Started Network Service.
Jun 27 13:20:43 arch1 systemd-networkd[54]: request_name_destroy_callback n_ref=1
Jun 27 13:20:43 arch1 systemd-networkd[54]: eth0: DHCPv4 address 10.113.222.25/24 via 10.113.222.1
Jun 27 13:20:44 arch1 systemd-networkd[54]: eth0: Gained IPv6LL
Jun 27 13:20:45 arch1 systemd-networkd[54]: eth0: Configured

so I suspect that you somehow can’t create user namespace in your VPS.

3

Please don’t takes this wrong way but your test ran in an unpriviliged environment?

It does work with no such issue in 238 and creates the lxc.uts.name = just fine.
238 also does not exhibit

Failed to save link data to /run/systemd/netif/links/35: Permission denied

Latter the systemd developer just commented on github, suppose you saw that already

4

Right, I ran it in an unprivileged ArchLinux container. I’m surprised that DynamicUser=y would fail.

5

It does my end and I am surprised the same that it is working your end though.

Your tested with lxc 3.0.2 or 3.0.1?

6 
  driver: lxc
  driver_version: 3.0.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.17.0-rc4-brauner-uevent-filtering
  server: lxd
  server_pid: 28123
  server_version: "3.2"
7

Do you reckon a different kernel could make the difference? This end it is 4.15.0-23

8

Id would surprise me if it would but you can try. The error is weird. Are you dropping any specific capabilites in the unprivileged container? Is your systemd on the host preventing unprivileged namespace creation?

9

Nope

As it is wasted enough time of trying/debugging systemd-networkd and it has not converted me into a fan of it. The other network tools for archlinux are even worse 3.0.1 - udev / netctl / dhcpcd in archlinux unpriviliged container

raised awareness with the systemd disto’s maintainer anyway. perhaps he will sort it out. Will see what happens when systemd 239 hits the road.

Archlinux, whilst seemingly interesting in many aspects, holds no attraction for me for elaborate/expanded networking inside an unpriviliged container, and neither systemd-networkd in general.

Trying now Gentoo with netifrc which thus far looks like smooth going about netwroking. That it is of course until hitting a snag there…

10

Reading Units with DynamicUser=yes fail in lxc container · Issue #9493 · systemd/systemd · GitHub it seems you have reached the conclusion this being caused by AppArmor buggy patches applied in kernels (e.g. ubuntu 4.15) which are reverted from later kernels (4.18) however, thus it worked in your test but not this end.

then I started wondering why hostnamectl status kept on timing out in systemd based containers and looked into the host’s systemlog and surprise surprise it is loaded with

Summary

audit: type=1400 audit(1530715939.134:305): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=24666 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
server kernel: [56223.246444] audit: type=1400 audit(1530715939.134:306): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=24666 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
Jerver kernel: [56223.517342] audit: type=1400 audit(1530715939.406:307): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=24747 comm=“mount” fstype=“configfs” srcname=“configfs”
server kernel: [56223.517404] audit: type=1400 audit(1530715939.406:308): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=24747 comm=“mount” fstype=“configfs” srcname=“configfs” flags=“ro”
server kernel: [56223.587693] audit: type=1400 audit(1530715939.474:309): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name=“/” pid=24783 comm=“(networkd)” flags=“rw, rslave”
server kernel: [56223.867672] audit: type=1400 audit(1530715939.754:310): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [56223.867676] audit: type=1400 audit(1530715939.754:311): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [56223.867679] audit: type=1400 audit(1530715939.754:312): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [56223.867696] audit: type=1400 audit(1530715939.754:313): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084317] audit: type=1400 audit(1530717743.949:314): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084330] audit: type=1400 audit(1530717743.949:315): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084334] audit: type=1400 audit(1530717743.949:316): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084338] audit: type=1400 audit(1530717743.949:317): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server dbus-daemon[1006]: [system] Activating via systemd: service name=‘org.freedesktop.hostname1’ unit=‘dbus-org.freedesktop.hostname1.service’ requested by ‘:1.166’ (uid=0 pid=19907 comm=“hostnamectl status " label=“unconfined”)
server systemd[1]: Starting Hostname Service…
server dbus-daemon[1006]: [system] Successfully activated service ‘org.freedesktop.hostname1’
server systemd[1]: Started Hostname Service.
server kernel: [58119.314378] audit: type=1400 audit(1530717835.176:318): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=”(ostnamed)" family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58119.314394] audit: type=1400 audit(1530717835.176:319): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58119.314399] audit: type=1400 audit(1530717835.176:320): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58119.314403] audit: type=1400 audit(1530717835.176:321): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none

What would be best practice until AppArmor gets around to fix it and then make it downstream for backporting, testing and then distribution (couple of months from now?) - remove AppArmor?

11

Your best bet is to use lxc.apparmor.profile = unconfined unfortunately, I fear.

1 Like
12

That is kind defeating its purpose but what to do when it is creating this sort of havoc. :weary:

OpenRC seems to be suffering in its own right from AppArmor… :anguished: 3.0.1 - apparmor denied mount lxc-container-default-cgns

13

So, the good news is that this is fixed in upstream kernel starting with 4.17. The relevant socket mediatiojn patchset can be found here. I’ve requested that the patchset be backported to all Ubuntu LTS kernels but we need to see how feasible this is given that it is quite a large patchset.
The backport can be tracked at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1780227 .

1 Like
14

After having upgraded the lxc host’s kernel to 4.17.4 the locks placed on AF_UNIX sockets are no longer denied by AppArmor

yet there is still whole bunch of other DENIED by AppArmor popping up

Summary

server kernel: [ 792.772569] audit: type=1400 audit(1530791008.499:39): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=11901 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
server kernel: [ 792.772578] audit: type=1400 audit(1530791008.499:40): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=11901 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
server kernel: [ 792.897652] audit: type=1400 audit(1530791008.623:41): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=11978 comm=“mount” fstype=“configfs” srcname=“configfs”
server kernel: [ 792.897703] audit: type=1400 audit(1530791008.623:42): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=11978 comm=“mount” fstype=“configfs” srcname=“configfs” flags=“ro”
server kernel: [ 792.914358] audit: type=1400 audit(1530791008.639:43): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name=“/” pid=11990 comm=“(networkd)” flags=“rw, rslave”
server kernel: [ 793.042265] audit: type=1400 audit(1530791008.767:44): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name=“/” pid=12062 comm=“(ostnamed)” flags=“rw, rslave”