n8v8r
June 27, 2018, 12:51pm
1
@stgraber @brauner
testing systemd
239 in an unpriviliged arclinux container produces
Summary
systemd[58]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[58]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result âexit-codeâ.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
systemd[63]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
systemd[63]: systemd-networkd.service: Failed at step USER spawning /usr/lib/systemd/systemd-networkd: Permission denied
systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
systemd[1]: systemd-networkd.service: Failed with result âexit-codeâ.
systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
systemd[1]: systemd-networkd.service: Start request repeated too quickly.
systemd[1]: systemd-networkd.service: Failed with result âexit-codeâ.
systemd[1]: systemd-networkd.socket: Failed with result âservice-start-limit-hitâ.
no such issue with systemd
238
Opened an issue networkd - device configuration never completes inside lxc container · Issue #9427 · systemd/systemd · GitHub and the systemd developers wondering whether DynamicUser=
is supported in LXC or being a restriction of the unpriviliged environment
brauner
(Christian Brauner)
June 27, 2018, 1:22pm
2
DynamicUser=yes
works fine for me with systemd 239
in an ArchLinux
container:
[root@arch1 ~]# systemctl status systemd-networkd
â systemd-networkd.service - Network Service
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-06-27 13:20:43 UTC; 1min 4s ago
Docs: man:systemd-networkd.service(8)
Main PID: 54 (systemd-network)
Status: "Processing requests..."
Tasks: 1 (limit: 4915)
Memory: 1.3M
CGroup: /system.slice/systemd-networkd.service
ââ54 /usr/lib/systemd/systemd-networkd
Jun 27 13:20:43 arch1 systemd[1]: Starting Network Service...
Jun 27 13:20:43 arch1 systemd-networkd[54]: Enumeration completed
Jun 27 13:20:43 arch1 systemd[1]: Started Network Service.
Jun 27 13:20:43 arch1 systemd-networkd[54]: request_name_destroy_callback n_ref=1
Jun 27 13:20:43 arch1 systemd-networkd[54]: eth0: DHCPv4 address 10.113.222.25/24 via 10.113.222.1
Jun 27 13:20:44 arch1 systemd-networkd[54]: eth0: Gained IPv6LL
Jun 27 13:20:45 arch1 systemd-networkd[54]: eth0: Configured
so I suspect that you somehow canât create user namespace in your VPS.
n8v8r
June 27, 2018, 1:44pm
3
Please donât takes this wrong way but your test ran in an unpriviliged environment?
It does work with no such issue in 238 and creates the lxc.uts.name =
just fine.
238 also does not exhibit
Failed to save link data to /run/systemd/netif/links/35: Permission denied
Latter the systemd developer just commented on github, suppose you saw that already
brauner
(Christian Brauner)
June 27, 2018, 1:52pm
4
Right, I ran it in an unprivileged ArchLinux container. Iâm surprised that DynamicUser=y
would fail.
n8v8r
June 27, 2018, 2:18pm
5
It does my end and I am surprised the same that it is working your end though.
Your tested with lxc 3.0.2 or 3.0.1?
brauner
(Christian Brauner)
June 27, 2018, 2:25pm
6
driver: lxc
driver_version: 3.0.1
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.17.0-rc4-brauner-uevent-filtering
server: lxd
server_pid: 28123
server_version: "3.2"
n8v8r
June 27, 2018, 2:35pm
7
Do you reckon a different kernel could make the difference? This end it is 4.15.0-23
brauner
(Christian Brauner)
June 28, 2018, 9:46am
8
Id would surprise me if it would but you can try. The error is weird. Are you dropping any specific capabilites in the unprivileged container? Is your systemd on the host preventing unprivileged namespace creation?
n8v8r
June 28, 2018, 10:11am
9
Nope
As it is wasted enough time of trying/debugging systemd-networkd
and it has not converted me into a fan of it. The other network tools for archlinux are even worse 3.0.1 - udev / netctl / dhcpcd in archlinux unpriviliged container
raised awareness with the systemd
distoâs maintainer anyway. perhaps he will sort it out. Will see what happens when systemd
239 hits the road.
Archlinux, whilst seemingly interesting in many aspects, holds no attraction for me for elaborate/expanded networking inside an unpriviliged container, and neither systemd-networkd
in general.
Trying now Gentoo with netifrc
which thus far looks like smooth going about netwroking. That it is of course until hitting a snag thereâŠ
n8v8r
July 4, 2018, 3:44pm
10
Reading Units with DynamicUser=yes fail in lxc container · Issue #9493 · systemd/systemd · GitHub it seems you have reached the conclusion this being caused by AppArmor buggy patches applied in kernels (e.g. ubuntu 4.15) which are reverted from later kernels (4.18) however, thus it worked in your test but not this end.
then I started wondering why hostnamectl status
kept on timing out in systemd
based containers and looked into the hostâs systemlog and surprise surprise it is loaded with
Summary
audit: type=1400 audit(1530715939.134:305): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/fs/cgroup/unified/â pid=24666 comm=âsystemdâ fstype=âcgroup2â srcname=âcgroup2â flags=ârw, nosuid, nodev, noexecâ
server kernel: [56223.246444] audit: type=1400 audit(1530715939.134:306): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/fs/cgroup/unified/â pid=24666 comm=âsystemdâ fstype=âcgroup2â srcname=âcgroup2â flags=ârw, nosuid, nodev, noexecâ
Jerver kernel: [56223.517342] audit: type=1400 audit(1530715939.406:307): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/kernel/config/â pid=24747 comm=âmountâ fstype=âconfigfsâ srcname=âconfigfsâ
server kernel: [56223.517404] audit: type=1400 audit(1530715939.406:308): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/kernel/config/â pid=24747 comm=âmountâ fstype=âconfigfsâ srcname=âconfigfsâ flags=âroâ
server kernel: [56223.587693] audit: type=1400 audit(1530715939.474:309): apparmor=âDENIEDâ operation=âmountâ info=âfailed flags matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/â pid=24783 comm=â(networkd)â flags=ârw, rslaveâ
server kernel: [56223.867672] audit: type=1400 audit(1530715939.754:310): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=24889 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [56223.867676] audit: type=1400 audit(1530715939.754:311): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=24889 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [56223.867679] audit: type=1400 audit(1530715939.754:312): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=24889 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [56223.867696] audit: type=1400 audit(1530715939.754:313): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=24889 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58028.084317] audit: type=1400 audit(1530717743.949:314): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=19797 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58028.084330] audit: type=1400 audit(1530717743.949:315): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=19797 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58028.084334] audit: type=1400 audit(1530717743.949:316): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=19797 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58028.084338] audit: type=1400 audit(1530717743.949:317): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=19797 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server dbus-daemon[1006]: [system] Activating via systemd: service name=âorg.freedesktop.hostname1â unit=âdbus-org.freedesktop.hostname1.serviceâ requested by â:1.166â (uid=0 pid=19907 comm=âhostnamectl status " label=âunconfinedâ)
server systemd[1]: Starting Hostname ServiceâŠ
server dbus-daemon[1006]: [system] Successfully activated service âorg.freedesktop.hostname1â
server systemd[1]: Started Hostname Service.
server kernel: [58119.314378] audit: type=1400 audit(1530717835.176:318): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=21185 comm=â(ostnamed)" family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58119.314394] audit: type=1400 audit(1530717835.176:319): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=21185 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58119.314399] audit: type=1400 audit(1530717835.176:320): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=21185 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
server kernel: [58119.314403] audit: type=1400 audit(1530717835.176:321): apparmor=âDENIEDâ operation=âfile_lockâ profile=âlxc-container-default-cgnsâ pid=21185 comm=â(ostnamed)â family=âunixâ sock_type=âdgramâ protocol=0 addr=none
What would be best practice until AppArmor gets around to fix it and then make it downstream for backporting, testing and then distribution (couple of months from now?) - remove AppArmor?
brauner
(Christian Brauner)
July 4, 2018, 4:05pm
11
Your best bet is to use lxc.apparmor.profile = unconfined
unfortunately, I fear.
1 Like
n8v8r
July 4, 2018, 4:25pm
12
That is kind defeating its purpose but what to do when it is creating this sort of havoc.
OpenRC seems to be suffering in its own right from AppArmor⊠3.0.1 - apparmor denied mount lxc-container-default-cgns
brauner
(Christian Brauner)
July 5, 2018, 10:29am
13
So, the good news is that this is fixed in upstream kernel starting with 4.17. The relevant socket mediatiojn patchset can be found here . Iâve requested that the patchset be backported to all Ubuntu LTS kernels but we need to see how feasible this is given that it is quite a large patchset.
The backport can be tracked at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1780227 .
1 Like
n8v8r
July 5, 2018, 12:06pm
14
After having upgraded the lxc hostâs kernel to 4.17.4 the locks placed on AF_UNIX sockets are no longer denied by AppArmor
yet there is still whole bunch of other DENIED by AppArmor popping up
Summary
server kernel: [ 792.772569] audit: type=1400 audit(1530791008.499:39): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/fs/cgroup/unified/â pid=11901 comm=âsystemdâ fstype=âcgroup2â srcname=âcgroup2â flags=ârw, nosuid, nodev, noexecâ
server kernel: [ 792.772578] audit: type=1400 audit(1530791008.499:40): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/fs/cgroup/unified/â pid=11901 comm=âsystemdâ fstype=âcgroup2â srcname=âcgroup2â flags=ârw, nosuid, nodev, noexecâ
server kernel: [ 792.897652] audit: type=1400 audit(1530791008.623:41): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/kernel/config/â pid=11978 comm=âmountâ fstype=âconfigfsâ srcname=âconfigfsâ
server kernel: [ 792.897703] audit: type=1400 audit(1530791008.623:42): apparmor=âDENIEDâ operation=âmountâ info=âfailed type matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/sys/kernel/config/â pid=11978 comm=âmountâ fstype=âconfigfsâ srcname=âconfigfsâ flags=âroâ
server kernel: [ 792.914358] audit: type=1400 audit(1530791008.639:43): apparmor=âDENIEDâ operation=âmountâ info=âfailed flags matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/â pid=11990 comm=â(networkd)â flags=ârw, rslaveâ
server kernel: [ 793.042265] audit: type=1400 audit(1530791008.767:44): apparmor=âDENIEDâ operation=âmountâ info=âfailed flags matchâ error=-13 profile=âlxc-container-default-cgnsâ name=â/â pid=12062 comm=â(ostnamed)â flags=ârw, rslaveâ