3.0 - network service failing on archlinux unprivileged container

LXC
1

host Linux server 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018 x86_64 GNU/Linux

downloaded container image Archlinux Current amd64

amongst various failed messages in the log 3.0 - unprivileged containers failed message (un)explained are

[FAILED] Failed to start Network Service.
[FAILED] Failed to start Network Name Resolution.

output from systemctl status -l systemd-networkd.service

● systemd-networkd.service - Network Service
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2018-05-26 09:45:00 UTC; 15min ago
Docs: man:systemd-networkd.service(8)
Process: 52 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=237/KEYRING)
Main PID: 52 (code=exited, status=237/KEYRING)

May 26 09:45:00 dns systemd[1]: systemd-networkd.service: Service has no hold-off time, scheduling restart.
May 26 09:45:00 dns systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
May 26 09:45:00 dns systemd[1]: Stopped Network Service.
May 26 09:45:00 dns systemd[1]: systemd-networkd.service: Start request repeated too quickly.
May 26 09:45:00 dns systemd[1]: systemd-networkd.service: Failed with result ‘exit-code’.
May 26 09:45:00 dns systemd[1]: Failed to start Network Service.

the ran systemctl stop systemd-networkd.service and after 30 s waiting systemctl start systemd-networkd.service, however resulting with the same error.

output from journalctl -xe

the configured Restart= setting for the unit.
May 26 10:13:24 dns systemd[1]: Stopped Getty on lxc/tty6.
Subject: Unit getty@lxc-tty6.service has finished shutting down
Defined-By: systemd
Support: systemd-devel Info Page

Unit getty@lxc-tty6.service has finished shutting down.
May 26 10:13:24 dns systemd[1]: Started Getty on lxc/tty6.
Subject: Unit getty@lxc-tty6.service has finished start-up
Defined-By: systemd
Support: systemd-devel Info Page

Unit getty@lxc-tty6.service has finished starting up.

The start-up result is RESULT.
May 26 10:13:24 dns agetty[1143]: /dev/lxc/tty6: cannot open as standard input: No such file or directo>
May 26 10:13:24 dns systemd[1]: getty@lxc-tty3.service: Service has no hold-off time, scheduling restar>
May 26 10:13:24 dns systemd[1]: getty@lxc-tty3.service: Scheduled restart job, restart counter is at 17>
Subject: Automatic restarting of a unit has been scheduled
Defined-By: systemd
Support: systemd-devel Info Page

Automatic restarting of the unit getty@lxc-tty3.service has been scheduled, as the result for
the configured Restart= setting for the unit.
May 26 10:13:24 dns systemd[1]: Stopped Getty on lxc/tty3.
Subject: Unit getty@lxc-tty3.service has finished shutting down
Defined-By: systemd
Support: systemd-devel Info Page

Unit getty@lxc-tty3.service has finished shutting down.
May 26 10:13:24 dns systemd[1]: Started Getty on lxc/tty3.
Subject: Unit getty@lxc-tty3.service has finished start-up
Defined-By: systemd
Support: systemd-devel Info Page

Unit getty@lxc-tty3.service has finished starting up.

The start-up result is RESULT.
May 26 10:13:24 dns agetty[1144]: /dev/lxc/tty3: cannot open as standard input: No such file or directo>

there is a reported bug since Oct’17 but appears to be closed and referring to upstream instead

And it is also mentioned on the LXD github, however running lxc profile set default security.syscalls.blacklist "keyctl errno 38" prompts

Command ‘lxc’ not found, but can be installed with:
apt install lxd-client

Why it requires to install an LXD component to run a LXC command?

So basically, Archlinux is not working with LXC until wanting to install LXD. Or follow the suggestion stipulated here Failed to change ownership of session keyring: Permission denied (systemd 237) · Issue #2226 · lxc/lxc · GitHub

Append

keyctl_chown errno 38
keyctl errno 38

to /usr/share/lxc/config/common.seccomp. This will cause ENOSYS to be returned for those syscalls and thus systemd will skip over them.

2

LXD’s solution to this problem is the same as for LXC so there’s no requirement on LXD.

1 Like
3

It would be good that it shows somewhere in the offical LXC documentation rather then having to dig around for a while at various places to finally find the nugget.

4

That’s a very specific use-case though. This only affects systemd. Any non-systemd distro (Alpine, some CentOS, Gentoo) are not really affected by this. That’s why putting it in the official documentation is not a good solution.

5

But I understand that it’s annoying.

6

I am not sure I am getting this correctly (and please pardon my ignorance) - about sytemd. The host is Ubuntu Bionic and to my understanding is systemd based. Running another systemd unprivileged container off it does not casue the issue, except that is Archlinux. I thought that it is caused

by systemd making use of the keyring and the kernel keyring not being namespaced.

Well, of course it is not my call but from a user’s perspective it would be rather helpful to see this in the official documentation. But since it is now in this forum it might suffice to (faster) help others coming across the same issue.

7

keyctl_chown errno 38
keyctl errno 38

Is there a known reason why this workaround is no longer working with the latest LXC release? On 3.0 this works great, but on 3.0.1 I get this:

lxc-start -n arch -F --logpriority=DEBUG
lxc-start: arch: start.c: lxc_init: 740 Failed loading seccomp policy
lxc-start: arch: start.c: __lxc_start: 1853 Failed to initialize container “arch”
The container failed to start.

It seems 3.0.1 doesn’t like blacklisting keyctl_chown. If I comment it out, the Arch Linux container will start, but then systemd is not happy.

8

my ubuntu host with lxc has not yet received the update to 3.0.1 and thus I cannot reproduce, though I sincerely hope not and thus having the archlinux containers doomed…

@brauner any chance this being a regression bug in 3.0.1?

9

LXC 3.0.1 is stricter wrt to skipping syscalls it doesn’t know or that don’t exist. keyctl_chown is not a syscall so seccomp will not handle it anyway and never has even in prior versions. Simply uncomment it but leave keyctl errno 38 in there.
We have already restored this behavior in master and you’d need your distro to backport https://github.com/lxc/lxc/commit/24b9874f7920a19e4c5ad385405703c2ace98aac :slight_smile:

1 Like
10

Well… I thought I tried all possible combinations - uncommenting both, just one or neither. I was just about to build my own package with the backported commit, when the Arch container suddenly started working properly with 3.0.1. Don’t know if it was the systemd update today, but now it’s running great with just blacklisting keyctl.

Blacklisting keyctl_chown is not needed anymore. :smiley:

1 Like
11

We have already restored this behavior in master

It will be then in the next release of LXC? Will is stll require keyctl errno 38 or not even that anymore either?

12

but now it’s running great with just blacklisting keyctl

:+1: what a relief to hear

13

It will be then in the next release of LXC? Will is stll require keyctl errno 38 or not even that anymore either?

I tried starting starting the Arch container without blacklisting keyctl and it doesn’t work with LXC 3.0.1 and systemd 238. All sort of problems - systemd-networkd fails, database servers not starting etc.

This patch got merged in systemd and I guess it will be included in 239. Maybe it will help? :slight_smile:

https://github.com/systemd/systemd/pull/8447