Lxdocker: convert docker images to LXD images

Okay nobody likes self-advertising but given the amount of docker-related questions both here and elsewhere I thought this might be useful to somebody :person_shrugging:

For various reasons stated in the README I didn’t find using docker(or podman) acceptable on my homeserver. It mostly boils down to LXD being superior in both security and device-forwarding.

While I agree that LXDs scope shouldn’t be extended to system containers, it can be useful. So I found this post, was disappointed that nobody had done it yet and finally implemented a tool for it myself :grin:

Feedback welcome :wink:

PS: Once I’m done with my setup as a whole it might also be a good post for the Community spotlight but I’m not quite there yet.

3 Likes

Hi, this looks very interesting, thanks! :slight_smile:

@m1cha

Any update status on your project?
You mentioned “finally implemented a tool for it myself” but then later “I’m not quite there yet”

thanks for any info

@bmullan I switched to podman :see_no_evil:

Writing an init process for lxdocker turned out to be a lot of work if you need more features. Not that it was too much work but in the middle of doing it I noticed that podman made some changes that turned it into the perfect solution for my usecase(secure(r), declarative services on low-spec hardware). Specifically The netavark+aardvark network stack and podman-systemd-units(formerly quadlets) which will arrive with 4.4 but are very stable already. They had quite a few bugs related to user namespaces but I reported them and they fixed them quickly. There’s one left regarding volumes but my workaround is to use bind-mounted directories instead.

So in total this setup gives me:

  • security through user-namespaces, idmapped mounts, selinux, seccomp, cgroups.
  • very declarative containers with advanced dependency definition via systemd
  • traffic-control(between containers and to the outside) via an nftables bridge filter. I had to use fixed mac addresses and put them all on the same bridge so I can use aardvark to resolve their IPs. For podman that’s okay since containers don’t have NET_ADMIN capabilities and thus can’t change them
  • Thanks to the traffic control above, most of my containers don’t have internet or LAN access
  • optionally auto update containers (can be enabled per-container)

caveats:

  • I had to give up on using alpine and switch to fedora coreOS instead to get access to systemd though. That’s less volatile than my previous setup and as a slower boot-time (probably due to signature verification) but it’s worth it.
  • I had to make my router and my homeserver be separate devices(I got two rk3399 devices now) because it’s not as easy to run openwrt inside podman as it is with LXD. 2W more power consumption aside, that’s better for both security and performance though given how big the impact on throughput was with those pesky realtek NICs(when using bridges and macvlan with them)
  • I had to switch from ZFS to BTRFS because it’s not (properly) supported on fedora

This may sound sad for lxd but it’s actually awesome since now I can stop turning it into something that it’s not :slight_smile:

Given that both podman and LXD are commercially backed: Sorry for doing this kind of “advertisement” through your own site, I just wanna make sure that people with a similar use case end up using the right tool for the job.

@m1cha
Don’t apologize for the change of plan. I am a big believer of using whatever tool satisfies my needs too. I’m going to try a roundabout method simply because I only really have 1 Docker application that I’d rather have running in an LXD system container.

So I plan on trying the following:
lets call the Docker application (app-1)

If it works great… if not… I may end up with another Docker app on my system :slight_smile:

@bmullan The go library lxdocker used to export the rootfs of a docker container as a .tar has a CLI. It’s called crane and was written by google employees: go-containerregistry/README.md at main · google/go-containerregistry · GitHub crane export ubuntu out.tar and you’re done :slight_smile:

for going from that rootfs to a working LXD-container you have many options but it really depends on the kind of container image you’re trying to use.

Most containers actually ship a full OS(e.g. debian) so you can try to replace the rootfs of your LXD container with it. Then call the original docker entrypoint manually on startup - e.g. with a systemd service unit.

Failing that you could extract the rootfs into a subdirectory of an existing LXD container and simply chroot into that and call it’s entrypoint.