Providing S3 storage is completely independent from providing containers and Virtual Machines so it should not be packaged in the same product.
The main benefit of packaging S3 as an Incus guest image is that it would allow modular packaging of the S3 component, something that is at the core of using containers. I use containers to keep software modular and separate from other software so that it is safer to use, backup, and test, I can instantiate it multiple times, can move it easily to other systems, etc. Ideally I want the host to run the container system (LXD, Incus, or whatever) and minimal other software, and run everything else in containers. Note, that my topic is not restricted to S3. I am looking for better ways of deploying software as Incus containers. S3 could be one example of many.
Let me also ask the reverse question: Why should Incus provide S3 functionality? Does it help with running containers? If Incus provides S3 because it is useful, there are a lot of other useful things that it could also provide.
Removing S3 from Incus would have the following additional benefits:
- It would “correct [some] mistakes that were made during LXD’s development which couldn’t be corrected without breaking backward compatibility”, as is stated in the “declaration of independence”.
- It would keep Incus smaller and simpler. Incus developers would have less work and could focus on providing greater container/vm functionality instead of fixing S3 bugs.
- It would separate the minio/S3 upgrade cycle from Incus, as you said. The fact that LXD was packaged as a snap is irrelevant. Incus could just as well include minio as an apt dependency so they are upgraded together.
- It would give users a choice of not including S3 in their Incus installation, if they don’t need it.
- It would give users a choice of using a different S3 implementation in an Incus system.