n8v8r
June 27, 2018, 12:51pm
1
@stgraber @brauner
testing systemd
239 in an unpriviliged arclinux container produces
Summary
systemd[58]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[58]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
systemd[63]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[63]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: systemd-networkd.service: Start request repeated too quickly.
systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
systemd[1]: systemd-networkd.socket: Failed with result ‘service-start-limit-hit’.
no such issue with systemd
238
Opened an issue networkd - device configuration never completes inside lxc container · Issue #9427 · systemd/systemd · GitHub and the systemd developers wondering whether DynamicUser=
is supported in LXC or being a restriction of the unpriviliged environment
brauner
(Christian Brauner)
June 27, 2018, 1:22pm
2
DynamicUser=yes
works fine for me with systemd 239
in an ArchLinux
container:
[root@arch1 ~]# systemctl status systemd-networkd
● systemd-networkd.service - Network Service
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-06-27 13:20:43 UTC; 1min 4s ago
Docs: man:systemd-networkd.service(8)
Main PID: 54 (systemd-network)
Status: "Processing requests..."
Tasks: 1 (limit: 4915)
Memory: 1.3M
CGroup: /system.slice/systemd-networkd.service
└─54 /usr/lib/systemd/systemd-networkd
Jun 27 13:20:43 arch1 systemd[1]: Starting Network Service...
Jun 27 13:20:43 arch1 systemd-networkd[54]: Enumeration completed
Jun 27 13:20:43 arch1 systemd[1]: Started Network Service.
Jun 27 13:20:43 arch1 systemd-networkd[54]: request_name_destroy_callback n_ref=1
Jun 27 13:20:43 arch1 systemd-networkd[54]: eth0: DHCPv4 address 10.113.222.25/24 via 10.113.222.1
Jun 27 13:20:44 arch1 systemd-networkd[54]: eth0: Gained IPv6LL
Jun 27 13:20:45 arch1 systemd-networkd[54]: eth0: Configured
so I suspect that you somehow can’t create user namespace in your VPS.
n8v8r
June 27, 2018, 1:44pm
3
Please don’t takes this wrong way but your test ran in an unpriviliged environment?
It does work with no such issue in 238 and creates the lxc.uts.name =
just fine.
238 also does not exhibit
Failed to save link data to /run/systemd/netif/links/35: Permission denied
Latter the systemd developer just commented on github, suppose you saw that already
brauner
(Christian Brauner)
June 27, 2018, 1:52pm
4
Right, I ran it in an unprivileged ArchLinux container. I’m surprised that DynamicUser=y
would fail.
n8v8r
June 27, 2018, 2:18pm
5
It does my end and I am surprised the same that it is working your end though.
Your tested with lxc 3.0.2 or 3.0.1?
brauner
(Christian Brauner)
June 27, 2018, 2:25pm
6
driver: lxc
driver_version: 3.0.1
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.17.0-rc4-brauner-uevent-filtering
server: lxd
server_pid: 28123
server_version: "3.2"
n8v8r
June 27, 2018, 2:35pm
7
Do you reckon a different kernel could make the difference? This end it is 4.15.0-23
brauner
(Christian Brauner)
June 28, 2018, 9:46am
8
Id would surprise me if it would but you can try. The error is weird. Are you dropping any specific capabilites in the unprivileged container? Is your systemd on the host preventing unprivileged namespace creation?
n8v8r
June 28, 2018, 10:11am
9
Nope
As it is wasted enough time of trying/debugging systemd-networkd
and it has not converted me into a fan of it. The other network tools for archlinux are even worse 3.0.1 - udev / netctl / dhcpcd in archlinux unpriviliged container
raised awareness with the systemd
disto’s maintainer anyway. perhaps he will sort it out. Will see what happens when systemd
239 hits the road.
Archlinux, whilst seemingly interesting in many aspects, holds no attraction for me for elaborate/expanded networking inside an unpriviliged container, and neither systemd-networkd
in general.
Trying now Gentoo with netifrc
which thus far looks like smooth going about netwroking. That it is of course until hitting a snag there…
n8v8r
July 4, 2018, 3:44pm
10
Reading Units with DynamicUser=yes fail in lxc container · Issue #9493 · systemd/systemd · GitHub it seems you have reached the conclusion this being caused by AppArmor buggy patches applied in kernels (e.g. ubuntu 4.15) which are reverted from later kernels (4.18) however, thus it worked in your test but not this end.
then I started wondering why hostnamectl status
kept on timing out in systemd
based containers and looked into the host’s systemlog and surprise surprise it is loaded with
Summary
audit: type=1400 audit(1530715939.134:305): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=24666 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
server kernel: [56223.246444] audit: type=1400 audit(1530715939.134:306): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=24666 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
Jerver kernel: [56223.517342] audit: type=1400 audit(1530715939.406:307): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=24747 comm=“mount” fstype=“configfs” srcname=“configfs”
server kernel: [56223.517404] audit: type=1400 audit(1530715939.406:308): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=24747 comm=“mount” fstype=“configfs” srcname=“configfs” flags=“ro”
server kernel: [56223.587693] audit: type=1400 audit(1530715939.474:309): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name=“/” pid=24783 comm=“(networkd)” flags=“rw, rslave”
server kernel: [56223.867672] audit: type=1400 audit(1530715939.754:310): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [56223.867676] audit: type=1400 audit(1530715939.754:311): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [56223.867679] audit: type=1400 audit(1530715939.754:312): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [56223.867696] audit: type=1400 audit(1530715939.754:313): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=24889 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084317] audit: type=1400 audit(1530717743.949:314): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084330] audit: type=1400 audit(1530717743.949:315): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084334] audit: type=1400 audit(1530717743.949:316): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58028.084338] audit: type=1400 audit(1530717743.949:317): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=19797 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server dbus-daemon[1006]: [system] Activating via systemd: service name=‘org.freedesktop.hostname1’ unit=‘dbus-org.freedesktop.hostname1.service’ requested by ‘:1.166’ (uid=0 pid=19907 comm=“hostnamectl status " label=“unconfined”)
server systemd[1]: Starting Hostname Service…
server dbus-daemon[1006]: [system] Successfully activated service ‘org.freedesktop.hostname1’
server systemd[1]: Started Hostname Service.
server kernel: [58119.314378] audit: type=1400 audit(1530717835.176:318): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=”(ostnamed)" family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58119.314394] audit: type=1400 audit(1530717835.176:319): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58119.314399] audit: type=1400 audit(1530717835.176:320): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
server kernel: [58119.314403] audit: type=1400 audit(1530717835.176:321): apparmor=“DENIED” operation=“file_lock” profile=“lxc-container-default-cgns” pid=21185 comm=“(ostnamed)” family=“unix” sock_type=“dgram” protocol=0 addr=none
What would be best practice until AppArmor gets around to fix it and then make it downstream for backporting, testing and then distribution (couple of months from now?) - remove AppArmor?
brauner
(Christian Brauner)
July 4, 2018, 4:05pm
11
Your best bet is to use lxc.apparmor.profile = unconfined
unfortunately, I fear.
1 Like
n8v8r
July 4, 2018, 4:25pm
12
That is kind defeating its purpose but what to do when it is creating this sort of havoc.
OpenRC seems to be suffering in its own right from AppArmor… 3.0.1 - apparmor denied mount lxc-container-default-cgns
brauner
(Christian Brauner)
July 5, 2018, 10:29am
13
So, the good news is that this is fixed in upstream kernel starting with 4.17. The relevant socket mediatiojn patchset can be found here . I’ve requested that the patchset be backported to all Ubuntu LTS kernels but we need to see how feasible this is given that it is quite a large patchset.
The backport can be tracked at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1780227 .
1 Like
n8v8r
July 5, 2018, 12:06pm
14
After having upgraded the lxc host’s kernel to 4.17.4 the locks placed on AF_UNIX sockets are no longer denied by AppArmor
yet there is still whole bunch of other DENIED by AppArmor popping up
Summary
server kernel: [ 792.772569] audit: type=1400 audit(1530791008.499:39): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=11901 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
server kernel: [ 792.772578] audit: type=1400 audit(1530791008.499:40): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/fs/cgroup/unified/” pid=11901 comm=“systemd” fstype=“cgroup2” srcname=“cgroup2” flags=“rw, nosuid, nodev, noexec”
server kernel: [ 792.897652] audit: type=1400 audit(1530791008.623:41): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=11978 comm=“mount” fstype=“configfs” srcname=“configfs”
server kernel: [ 792.897703] audit: type=1400 audit(1530791008.623:42): apparmor=“DENIED” operation=“mount” info=“failed type match” error=-13 profile=“lxc-container-default-cgns” name=“/sys/kernel/config/” pid=11978 comm=“mount” fstype=“configfs” srcname=“configfs” flags=“ro”
server kernel: [ 792.914358] audit: type=1400 audit(1530791008.639:43): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name=“/” pid=11990 comm=“(networkd)” flags=“rw, rslave”
server kernel: [ 793.042265] audit: type=1400 audit(1530791008.767:44): apparmor=“DENIED” operation=“mount” info=“failed flags match” error=-13 profile=“lxc-container-default-cgns” name=“/” pid=12062 comm=“(ostnamed)” flags=“rw, rslave”