@bmullan I switched to podman 
Writing an init process for lxdocker turned out to be a lot of work if you need more features. Not that it was too much work but in the middle of doing it I noticed that podman made some changes that turned it into the perfect solution for my usecase(secure(r), declarative services on low-spec hardware). Specifically The netavark+aardvark network stack and podman-systemd-units(formerly quadlets) which will arrive with 4.4 but are very stable already. They had quite a few bugs related to user namespaces but I reported them and they fixed them quickly. There’s one left regarding volumes but my workaround is to use bind-mounted directories instead.
So in total this setup gives me:
- security through user-namespaces, idmapped mounts, selinux, seccomp, cgroups.
- very declarative containers with advanced dependency definition via systemd
- traffic-control(between containers and to the outside) via an nftables bridge filter. I had to use fixed mac addresses and put them all on the same bridge so I can use aardvark to resolve their IPs. For podman that’s okay since containers don’t have NET_ADMIN capabilities and thus can’t change them
- Thanks to the traffic control above, most of my containers don’t have internet or LAN access
- optionally auto update containers (can be enabled per-container)
caveats:
- I had to give up on using alpine and switch to fedora coreOS instead to get access to systemd though. That’s less volatile than my previous setup and as a slower boot-time (probably due to signature verification) but it’s worth it.
- I had to make my router and my homeserver be separate devices(I got two rk3399 devices now) because it’s not as easy to run openwrt inside podman as it is with LXD. 2W more power consumption aside, that’s better for both security and performance though given how big the impact on throughput was with those pesky realtek NICs(when using bridges and macvlan with them)
- I had to switch from ZFS to BTRFS because it’s not (properly) supported on fedora
This may sound sad for lxd but it’s actually awesome since now I can stop turning it into something that it’s not 
-
Full sourcecode for my homeserver setup (the container units are in
files/containers) - My reddit post on that topic (the information is now very outdated, but it shows you where I’m coming from)
Given that both podman and LXD are commercially backed: Sorry for doing this kind of “advertisement” through your own site, I just wanna make sure that people with a similar use case end up using the right tool for the job.