Since Incus does not plan to maintain backward compatibility with LXD, I would like to discuss removing the S3 API from the Incus server and moving it to an S3 image which can be launched to add S3 functionality to an Incus host. Along they way, it may help improve Incus so it can be used more easily to do these sort of things. For example, there should be a way to easily upgrade such an S3 container to a newer image of it.
If this encroaches on the use cases of Docker/Podman, so be it. Perhaps Incus should have a new APP image/instance type, in addition to CONTAINER and VIRTUAL MACHINE, which would behave more like a docker/podman image/container (perhaps use existing docker/podman images).
Putting an S3 server in a container is already possible. It is one of the use cases of LXC/LXD/Incus in general. The issue is to make it very easy to launch, use, and maintain such a container so there would no longer be a need to incorporate it directly into the Incus server.
What would be the benefit of this over the current Incus integration?
Given that Incus will not be deployed through snap, the minio binary will need to be deployed externally anyway. This means that its upgrade cycle will be separate from Incus itself already.
Providing S3 storage is completely independent from providing containers and Virtual Machines so it should not be packaged in the same product.
The main benefit of packaging S3 as an Incus guest image is that it would allow modular packaging of the S3 component, something that is at the core of using containers. I use containers to keep software modular and separate from other software so that it is safer to use, backup, and test, I can instantiate it multiple times, can move it easily to other systems, etc. Ideally I want the host to run the container system (LXD, Incus, or whatever) and minimal other software, and run everything else in containers. Note, that my topic is not restricted to S3. I am looking for better ways of deploying software as Incus containers. S3 could be one example of many.
Let me also ask the reverse question: Why should Incus provide S3 functionality? Does it help with running containers? If Incus provides S3 because it is useful, there are a lot of other useful things that it could also provide.
Removing S3 from Incus would have the following additional benefits:
It would “correct [some] mistakes that were made during LXD’s development which couldn’t be corrected without breaking backward compatibility”, as is stated in the “declaration of independence”.
It would keep Incus smaller and simpler. Incus developers would have less work and could focus on providing greater container/vm functionality instead of fixing S3 bugs.
It would separate the minio/S3 upgrade cycle from Incus, as you said. The fact that LXD was packaged as a snap is irrelevant. Incus could just as well include minio as an apt dependency so they are upgraded together.
It would give users a choice of not including S3 in their Incus installation, if they don’t need it.
It would give users a choice of using a different S3 implementation in an Incus system.
We don’t have any plans to remove the storage bucket capability.
Incus is designed to provide a cloud-like experience on systems of any size, it’s not just a container and VM manager, if you only care about running containers and nothing else, then LXC is for you, if you only care about virtual machines, then QEMU is for you.
Storage buckets through an S3-like API is a common characteristic of all public cloud offering and something which many cloud-native applications rely on, therefore having Incus provide that functionality makes sense.
Incus just drives minio or ceph-object for that particular feature, so the logic on the Incus side is minimal and minio is a soft dependency, in that Incus will work just fine if the binary isn’t present. Even when present, it’s spawned on-demand so it doesn’t cause any CPU/memory usage when it’s not in active use.
I can confirm that minio is not a hard requirement and that LXD will work just fine without it available. It was only recently that we added minio as a dependency to the Nix package, and LXD worked without it prior to that point. Obviously the storage buckets would not have worked, but the remaining functionality was not impacted without the minio binary.
I didn’t mean to suggest not having storage bucket functionality in an Incus system, but to consider implementing it in a container, thus taking advantage of the modular architecture that containers provide. Since storage bucket functionality is already implemented as it is, it was probably a mistake to suggest re-implementing it.
I still want to explore how one might use Incus to provide add-on functionality (software appliances), perhaps in another topic.
Imagine being able to add HTTP reverse proxy functionality to an Incus server by running a command like this:
incus launch app:haproxy haproxy
This would launch a container called “haproxy” using an image called “haproxy” from an application image remote called “app”. It would assign to it tcp proxy devices for ports 80 and 443. It would also somehow provide a REST API and a command-line-interface tool to configure haproxy by mapping domains to containers (and ports, etc.).
Or imagine being able to add S3 functionality by running a similar command:
incus launch app:minio s3
This would launch a container called “s3” that runs a minio server. It would be configured with a proxy device for a port that is connected to the minio server. It would also provide an additional REST API and a command-line-interface tool to do what you can do today with the “incus/lxc storage bucket” commands.
The two examples are similar, and we could discuss either one. Storage bucket functionality already exists in Incus/LXD, so in discussing it we know the functionality that it would need to provide in order to be equivalent with the current implementation. Haproxy functionality does not exist so we would argue more about what functionality it should provide.
My choice of “app” application image server is influenced from Hetzner’s “app” image type. These are images that Hetzner maintains, which can be used to launch servers that include a distribution + packages + configuration.