I am writting/learning/looking for some documentation about Incus and its interactions with Docker. For example working with ZFS, BTRFS etc. Does anyone have relevant info on these topics? I did find some info on the documentation and previous treads. By all means please correct anything I state here as I am still learning.
Docker running along and Incus host
Docker inside a LXC container
– I am not sure if this information is still relevant but using docker in a container with VFS will run but can cause slowness and waste of resources due to dockers layering nature.
– Due to container`s unpriviledged nature cannot run as root.
---- Required to run inside a LXC ----
incus config set security.nesting true
security.syscalls.intercept.mknod=true
security.syscalls.intercept.setxattr=true
— with Zfs —
(For ZFS) zfs.delegate=true
Docker running inside an LXD VM
– As per documentation here Btrfs - btrfs - Incus documentation
Avoid running docker in a vm with BTRFS.
– Runs like a normal VM once configured properly. Has access to all modules and root.
Awesome thanks, noted. I was not aware of the zfs.delegate=true, maybe thats affecting me at the moment. I am also struggling with should I install docker alongside the host to run services that require root access? I wanted to keep my host very plain but I ran into trouble running homepage as root so made me question myself. I also considered running a normal VM and using that to host my dockers… so many options hehe.
I certainly do not want to hijack your thread and am curious if ciphermential guide would be any different if you want to to run podman in a virtual machine. I believe podman would benefit from ZFS 2.2 and overlayfs support. Would it be as simple as installing ZFS 2.2 and launching a Fedora40 container and installing podman? I know docker is the current standard, but podman rootless containers and direct integration with systemd seems like a better mouse trap at this point.
ZFS 2.2 introduced Linux container support for overlayfs.
Great to hear that you got it working. I wonder if the config key would also be needed with a podman/docker virtual machine?
incus storage volume set MyZpool container/Docker zfs.delegate=true
I know a lot of folks seem to run docker as a separate container as opposed to a virtual machine in incus. I assume because it is lighter than running a full vm and having to allocate CPU and RAM. From a security perspective it seems a full VM would be better. Again, Podman in a separate VM with rootless containers would seem to be more secure if I decided to open up some containers to the internet. I am not ready to open anything to the internet, but taking baby steps.
I really wonder if running an lxc container for each docker service you will use is not wasteful, in terms of memory and cpu, not so much storage. I mean I know the point of lXC are to be very lightweight but still having docker installed and doing os operations for each…
You’d usually run more than one Docker container in there.
But yeah, our plan over the next few months is to add native support for running OCI application containers alongside full system containers directly on Incus.
That’d use LXC’s native OCI support and would allow running most simple Docker workloads (image with some env variables and some devices/files passed through).
Not trying to full on compete with more complex Docker setups but definitely trying to cover the simpler cases where the extra level just adds needless complexity.
Excuse slight thread hijack! I’m using your Dockerfile as the basis for running incus under BalenaOS, with a few modifications. Just wanted to let you know I had to set the user.max_user_namespaces sysctl to something sensible (a large round number seems to be traditional) - that might something applicable to other OSes, too, so maybe worth documenting if nothing else.