Question: what are the differences between when incus runs containers launched from “normal” images (e.g. simplestreams) and OCI images?
Reason: I’m looking at building and distributing some incus system container images which are large, but mostly the same.
OCI has a layer capability, which means I could build a shared base layer, and then build the final images which are base + changes. Then downloading all the variants would be a lot quicker.
If I did this, and I set the entrypoint to /sbin/init, would incus run them in the same way as normal system containers? Or are there ways in which incus-managed containers launched from OCI images fundamentally differ from containers launched from normal images?
Looking in the source, I do see quite a lot of comparisons with instancetype.Container, so I’m wondering whether something will catch me out.
Incus doesn’t make use of the layer mechanism so that won’t really help
We support running OCI images, in that we know how to parse the OCI metadata, setup the right mounts and entrypoint and then have LXC run the container.
But when it comes to fetching stuff from the registries, we use skopeo to grab all the layers and we then use umoci to convert those layers into a final (flat) container rootfs suitable for immediate use within Incus.
We’ve been thinking about native support for layers on and off, but it would bring in a LOT of complexity to our image tracking logic for what seems to be extremely limited benefit in 90%+ of cases (folks don’t actually run containers that share a bunch of large layers all that often, exceptions being AI/ML type stuff).
The tool that fetches them (skopeo) presumably has to fetch and merge the layers. However, if it doesn’t cache the layers individually, and refetches them every time, then indeed that’s not going to help.
This SO question suggests that it can do some caching under .local/, and I see skopeo copy has an option --src-shared-blob-dir.
Ah yeah, that could help with the fetching/downloading aspect of it, wouldn’t make a different on the disk usage side though. Would also be a bit odd within a cluster as depending on the server performing the actual download, you may or may not have something useful in the cache. Then also comes having to figure out when to clear that thing as you don’t want layers to just keep accumulating.
