Bionic Containers on Xenial Host: systemd-hostnamed unable to start

LXD
1

Hi,

I’m still having problems likes

On my bionic Containers, systemd-hostnamed is unable to start

root@test:~# systemctl status systemd-hostnamed
● systemd-hostnamed.service - Hostname Service
   Loaded: loaded (/lib/systemd/system/systemd-hostnamed.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-05-07 23:23:23 CEST; 2min 30s ago
     Docs: man:systemd-hostnamed.service(8)
           man:hostname(5)
           man:machine-info(5)
           https://www.freedesktop.org/wiki/Software/systemd/hostnamed
  Process: 120 ExecStart=/lib/systemd/systemd-hostnamed (code=exited, status=225/NETWORK)
 Main PID: 120 (code=exited, status=225/NETWORK)

May 07 23:23:23 test systemd[1]: systemd-hostnamed.service: Failed to reset devices.list: Operation not permitted
May 07 23:23:23 test systemd[1]: Starting Hostname Service...
May 07 23:23:23 test systemd[120]: systemd-hostnamed.service: Failed to set up network namespacing: Permission denied
May 07 23:23:23 test systemd[120]: systemd-hostnamed.service: Failed at step NETWORK spawning /lib/systemd/systemd-hostnamed: Permission denied
May 07 23:23:23 test systemd[1]: systemd-hostnamed.service: Main process exited, code=exited, status=225/NETWORK
May 07 23:23:23 test systemd[1]: systemd-hostnamed.service: Failed with result 'exit-code'.
May 07 23:23:23 test systemd[1]: Failed to start Hostname Service.

I’m seeing DENIEDs from apparmor on dmesg

[ 1034.914235] audit: type=1400 audit(1525728203.691:82): apparmor="DENIED" operation="file_lock" profile="lxd-test_</var/lib/lxd>" pid=26852 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 1034.914239] audit: type=1400 audit(1525728203.691:83): apparmor="DENIED" operation="file_lock" profile="lxd-test_</var/lib/lxd>" pid=26852 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 1034.914240] audit: type=1400 audit(1525728203.691:84): apparmor="DENIED" operation="file_lock" profile="lxd-test_</var/lib/lxd>" pid=26852 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
[ 1034.914242] audit: type=1400 audit(1525728203.691:85): apparmor="DENIED" operation="file_lock" profile="lxd-test_</var/lib/lxd>" pid=26852 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none

Environment:
OS: Ubuntu 16.04
Kernel: 4.15.0-15-generic
LXD: 2.21
LXC: 2.0.8

According to https://github.com/lxc/lxd/issues/1603, installed a new HWE Kernel should resolved this problem which current doesn’t not it.

Is this behavior expected or do I need newer AppArmor profiles for example from LXD 3 or 18.04 itself?

2

Maybe

also related

3

Looks similar, yes. @brauner may have more context.
I thought the systemd version in bionic should work, but it could be apparmor related here.

4

PrivateNetwork should just create a new network namespace which should be possible. So this looks like AppArmor. Can you try to start the same container with:

lxc config set <container-name> raw.lxc "lxc.apparmor.profile ="

and report back, please?

5 
lxc config set test raw.lxc "lxc.apparmor.profile ="
error: Failed to load raw.lxc

But /var/log/lxd/test/lxc.conf is available.

cat /var/log/lxd/test/lxc.log
            lxc 20180509150508.425 ERROR    lxc_confile - confile.c:parse_line:2014 - unknown key lxc.apparmor.profile
            lxc 20180509150508.425 ERROR    lxc_parse - parse.c:lxc_file_for_each_line:57 - Failed to parse config: lxc.apparmor.profile =
6

Using the old keys does not result into a error:

lxc config set test raw.lxc "lxc.aa_profile="

But results into the same behavior. Here are the logs from host:

May 09 17:15:24 hades audit[21416]: AVC apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades audit[21416]: AVC apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades audit[21416]: AVC apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades audit[21416]: AVC apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades kernel: audit: type=1400 audit(1525878924.923:153): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades kernel: audit: type=1400 audit(1525878924.923:154): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades kernel: audit: type=1400 audit(1525878924.923:155): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none
May 09 17:15:24 hades kernel: audit: type=1400 audit(1525878924.923:156): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=21416 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 addr=none

The apparmor profile has changed into the audit error lxc-container-default-cgns in comparison to the first post.

7

Try setting lxc.aa_profile=unconfined instead

8

Thats works, but it disables AA for the container completly?

9

It does indeed, at least that suggests we’re dealing with an apparmor problem here.

@tyhicks does this ring a bell?

10

If AA doesn’t support it, there is a way to tell systemd that netnamespaces are not available?

Or there are just some missing AA profiles?

In case of AA doesn’t support this scenario systemd tries to detect if netnamespaces available. Looking at ns_type_supported show me that systemd checks if /proc/self/ns/net exists.

It’s maybe possible to hide /proc/self/ns/net with lxcfs?

11

@stgraber Do you have an update for this issue?

At this time, Bionic Containers are broken if they run unprivileged.

12

Have you restarted the container after setting AppArmor to unconfined?

13

Set the aa profile to unconfined helps. But it will disable the functionality of AppArmor for this Container (like SELinux permissive), not?

This souldn’t be the solution.

At this time, i’m override the systemd-hostnamd unit with PrivateNetwork=no. But its also a nasty hack expect its upstream.