Hmm, we’re not really involved with Ubuntu’s systemd and weren’t aware of any critical issue (just logging noise) in the variety of distros we build images for.
Indeed a bunch of things will fail in unprivileged containers but those do end up being softfail in most cases (ebpf, devices cgroup, some capabilities, some socket types, …).
Then systemctl status systemd-networkd.service gives an error about namespaces
Similar for nixOS without the patch. It’s clearly not softfail. Ubuntu works tho, because of the patches, specifically the “Revert 'namespace:…” which makes that exact error a softfail
× systemd-networkd.service - Network Service
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/systemd-networkd.service.d
└─lxc.conf
Active: failed (Result: exit-code) since Tue 2021-04-27 19:28:50 UTC; 48s ago
TriggeredBy: × systemd-networkd.socket
Docs: man:systemd-networkd.service(8)
Process: 43 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=226/NAMESPACE)
Main PID: 43 (code=exited, status=226/NAMESPACE)
Apr 27 19:28:50 arch-test systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Apr 27 19:28:50 arch-test systemd[1]: Stopped Network Service.
Apr 27 19:28:50 arch-test systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Apr 27 19:28:50 arch-test systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Apr 27 19:28:50 arch-test systemd[1]: Failed to start Network Service.
I’m not 100% sure if it’s nixOS way of packaging lxd (which looks fairly standard to me, but we could be missing a crucial workarround for this in our package) or if it really is the systemd patch ubuntu applies on their systemd.
Judging from the fact that 21.04 still has it, it doesn’t look like it got merged upstream.
It’s unmodified images:archlinux running unprivileged so you should be able to reproduce locally. Does it work for you with ubuntu lxd or is there an error, too? (just so I know if it’s the nixOS package or a general error)
I’m using lxd-4.13
[root@arch-test ~]# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-hostnamed.service loaded failed failed Hostname Service
● systemd-resolved.service loaded failed failed Network Name Resolution
● systemd-journald-audit.socket loaded failed failed Journal Audit Socket
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
3 loaded units listed.
With that, I only get systemd-journald-audit.socket which fails to start and that’s normal, that one will never be allowed to start in a container, might be worth getting upstream to add a container condition.
Yeah, no good reason for that one not to be upstream, audit sockets are specifically not allowed in user namespaces and because of how caps work in userns, the cap condition doesn’t actually help.
Feel free to send this to upstream systemd, I don’t see a reason why they wouldn’t easily pick it up.
Btw, for the future if I may I’d recommend a simple automatic test with all systemd-distros to check the output of systemctl --failed for any errors and fail the test if that happens.
The workaround so far was security.nesting=true, but that’s just a workaround.
The fixes were in distrobuilder and in lxc-ci, they’ve all been pushed so the next round of images should get the changes. Hopefully they don’t regress anything else.
