Seeing as the recent topics from people looking for help troubleshooting issues similar to mine don’t have happy endings, I’m thinking it might be more fruitful to work backwards from success.
So any of y’all who have succeeded in getting Docker working in an Incus LXC, I want to bite your steez. What distro is your host system? Container distro? What storage driver does your Incus host use? Anyone using a different storage driver for the container than the host? Which security policies have you enabled? Nesting, I assume. Syscalls.intercept.mknod? Syscalls.intercept.setxattr? ZFS delegation?
Appreciate you in advance, and don’t worry, I’ll make your everyone knows that you’re the real Slim Shady.
Not 100% what you’re looking for, but I’ve been able to run Podman without issues in a fedora/42 container with just security.nesting=true.
ZFS delegation is still something I’m trying to look into for RHEL-based instances, but for Debian-based you won’t likely see any significant amount of trouble.
EDIT - Re: ZFS delegation: a tip if you try to go with this, don’t enable zfs.delegation on root disks. If you set it from the storage pool then set initial.zfs.delegate=false for your root disk in instance configuration or profile. I can’t nail this down to anything definitive, but it might be a upstream OpenZFS bug as the datasets get pinned to a non-existent namespace (or zone in OpenZFS terminology)
Hi, whereas I don’t use docker “directly”, I host a couple of discourse Forums inside Incus instances, and the deployment mechanism for discourse is to build docker images and deploy them inside the host environment, in this case an Incus container. So effectively I am hosting docker in a container and deploying docker images into it.
Typically works fine, production uptime, I want to say that all I needed to do at the time was to enable nesting … but on checking the instance now nesting isn’t enabled (nor is priv). The only customisation I seem to have is that swap is enabled, and I think that’s just because Discourse is a bit hungry when it rebuilds.
root@rpi3:~# incus shell linuxforums
root@linuxforums:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
36b1bceccc5e local_discourse/app "/sbin/boot" 3 months ago Up 3 days 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp app
Fairly standard low-power install, 8Gb RPi5, local three machine cluster, stock ZFS filesystem volume.
Typically works fine, production uptime, I want to say that all I needed to do at the time was to enable nesting … but on checking the instance now nesting isn’t enabled (nor is priv).
Thank you for this bit, I just did a test without security.nesting set and Podman worked just fine.
Fyi I’m using extensive use of docker images now directly in INCUS using the (relatively) new OCI format. Works great, no need for docker, at least not for the things I’ve tried.
The most challenging thus far has been getting Nginx Proxy Manager working in different configurations, I now have one straddling my OVN network and the Internet, and another straddling my local network and OVN. Still managed to get it all working using the stock docker images, despite needing multiple network ports inside the image. Also running standard docker images for Gitlab and unbound, all relatively straightforward, although maybe the UI could do with some OCI based enhancement…
Mm, not come across the authentication feature as a casual Docker user … I’m now wondering what the use-case is for authenticated images, will look it up
Mmm … well, just fyi; thus far I’ve not had any issues. Install seems to work and I’m setting up in such a way that rebuild also seems happy re; custom data volumes … but then I don’t update sufficiently often to hit any /hr limits. Once you’ve pulled an OCI image it’s cached locally so re-deploying and re-using doesn’t seem likely to incur any remote hits.
If someone does hit pull limits on Docker, they can setup a system container as a caching proxy for Docker packages. And configuration their Docker instances to use that instead.
Yeah this is my general use case as well excluding the local caching part, I spin up system instances to run a OCI runtime to do my experiments to get some degree of isolation for when something explodes.
@simos Thank you for this, I wasn’t aware you could host a local registry that can cache images. For some reason I thought this wasn’t possible with OCI without some form of MiTM proxy due to how apt and dnf handled proxy caches and HTTPS connections.
Mmm, I’m a little confused … I was under the impression that when Incus launched an instance, it first downloaded and stored the image, then deployed an instance from the image. If you then deploy another instance using the same image, it should use the already downloaded image. If it’s not doing this and instead downloading from docker (or wherever) each time, what’s the point of the Incus image repository … what am I missing ?
Incus does indeed cache the images, but this is project dependent because of how Incus projects features work, specifically features.images which is set to initial value of true.
Very broadly, when Incus pulls an OCI image, it gets manipulated by umoci to fit the system container format. IIRC it does not actually run it verbatim like how an OCI runtime does [1]
I have around three use cases: testing/deployment with pre-existing images on Docker Hub, using OCI images directly as a base (for development and such) from Docker Hub, and my own private Forgejo OCI Registry
In the case of the first two, it’s not too bad as I’m not constantly pulling and the use case ends up being the same as yours. So falling under the 10 pulls/hour rate limit is perfectly achievable, and if it isn’t then I can stagger my own rollouts.
But this generality conflicts with how I also use Incus projects. I create a new project (with defaults based on how Incus creates a new project for incus group members) for each test/deployment/upgrade to make sure I’m not hitting some unforeseen regression in my homelab; and once I consider the deployment good - for some arbitrary definition of good - I wipe the project and perform an upgrade on the instances in my master services project. I’m also fairly certain that umoci doesn’t cache images locally
Also compounded is that I’m pretty sure the pull limits is affected by both official image status (if it is then it doesn’t count) and how many layers an image has (I’ve heard others report that their CI has failed because of large number of layers).
I have found another “gotcha” recently, if you’re running the outer / docker container on an OVN network and you find you have to change the container MTU to accommodate, then you will also need to modify the docker network within the container.
I have to confess, I don’t actually remember posting this topic, not in the experiential/episodic sense typically understood in colloquial conversation. I do have memories of struggling with apparmor and socket permissions trying to get nested virtualization working in an LXC, but that was around midnight – by the post’s 4:30AM timestamp, it seems my present-day consciousness had crashed out, and oddly enough, based on the outdated slang and cultural reference point, a corrupt snapshot of myself from 25 years ago had taken over the keyboard.
In any case, thank you to everyone who replied. I switched all the nodes in cluster from Ubuntu to Debian, and I haven’t had any problems running docker in an LXC. Feels like Incus is running smoother in general in Debian, though that could easily be confirmation bias.
