I have been working on a proof of concept with the spirit of “docker-compose” but the soul of Incus. It is extremely rough, but the “up” and “rm” commands work. I’m posting it here, soliciting feedback on whether it’s a valuable idea in general, and how people might want to work with such a tool.
Here is the repo - seriously, it’s PoC code, so play on test systems.
I’m using this as-is already to manage my homelab Incus server. If it interests you, take a look at the README, the code, or even build it yourself and let me know if you think the idea has merit.
The code is what I would consider “early garbage”, there’s a mix of the Incus cli and api, which will certainly break usage against non-local servers. Not much error handling at all. Happy-path is incus-compose up and incus-compose down. incus-compose generate will create a sample compose file for you, but the one in the README is more complete.
Having an Incus compose is very close to my ideal setup tbh, I will be following this closely.
Incus is way easier than anything that followed up on Docker as soon as you have stateful workloads involved, and being able to just run OCI images for the trivial case of a stateless workload in 6.3 is great. This also makes composing straightforward.
Now using the actual compose-spec library instead of a made-up spec. This means for many cases incus-compose up will work with a regular docker-compose yaml file.
nearly all of the cli calls to incus have been removed and replaced with api calls
added compose extensions like x-incus-gpu to extend the docker-compose spec where additional features of incus can be used - storage locations, snapshots, etc
Next up is defining some defaults that don’t violate the Principle of Least Surprise.
Are you kidding me? This feature is amazing. The best thing after, well, making Incus capable of running OCI containers?
Create Incus/OCI containers with code (which made docker what it is now).
Bring to Incus the most extended, de facto standard for app packaging and distribution, especially in the FOSS world.
Wouldn’t support of Dockerfiles be required too for this to be fully compliant with compose definitions? Or are Dockerfiles just for building images and compose for mixing images together?
docker-compose supports building images from dockerfiles with a “build” directive. I have no intention to support that. That use case is most frequently used for developers who are building multi-container services.
This morning I started an unmodified compose file for Gitea+postgres. Exciting!
run and exec. If you’re here you can probably figure out how to do that with the incus cli.
ports, expose. Still don’t know how to handle these. What would you expect to happen?
See the samples folder for the tests we’ve been using, and some examples of using attributes to control incus resources (snapshots, pool names, networks)
However, this isn’t really required if you deploy a stack. If all instances are using the same Incus network they can communicate directly with each other using the their local instance ports. Only the instances which requires access from the outside would need a Incus proxy.
Might be rather complex to find the right strategy?
I made a change that makes incus-compose more like the docker version today. It creates a new managed bridged network named after the stack and puts the instances in that network. This sets the foundation for proxying, and also prevents instance naming collisions when stacks have services with the same name.
For the “expose” directive, I was considering adding an empty ACL to the network which would effectively prevent traffic from outside the instance’s network, then adding ACL rules for each exposed port. Still need to think that through more.
Hey Brian! I’m just getting started with my Incus journey, and I’m very excited for your work with incus-compose, as my personal opinion is that the Compose spec is my ideal format for defining an application stack. It’s concise enough to make a full stack easy to digest, but flexible enough to enable advanced configurations possible. Not to mention that it can take advantage of the wealth of reference Compose stacks out there for deploying applications quick and easy.
I’m curious how you intend on addressing labels. I’m currently working on building an Incus powered reference architecture, and have used Docker Labels on stacks to configure things like Traefik reverse proxies. Do you plan to map those 1:1 to user options? Knowing this will help with some middleware I plan to write to provide a dynamic provider for Traefik and Incus.
I think it’s reasonable to add labels as user options. It would have to be a comma-delimited string though, since user options can only be strings. Can you open an issue to track the request?
I think the compose specification fits nicely with the concept of applications, whereas the Terraform model is more suited to infrastructure. Terraform is lower level than compose, so you’d have more flexibility with terraform, but gain opinionated defaults from compose.
I haven’t started dev work, but I could easily share the high level design. I’ll start a new thread when I’m back at my desk that we can use to work together.
Having spent the morning writing scripts to generate reusable cloud-init configurations to help migrate my LAN services, I stumbled on this. It looks awesome and will address a lot of different use cases.
I’m particularly interested in rebuilding instances rather than maintaining them, i.e., rebuild and replace.
Can I just say @bketelsen, holy shit, thank you! I am gonna be diving into testing this shortly. This is really a very much desired piece of infrastructure you’re building!
Sorry double-post, but do we also have something in Incus world which is comparable to the swarmkit of Docker swarm? It does a lot but my main interest is in having some “zero downtime deployment” possibilities, e.g. if you re-deploy, it can understand how to switch over to the new containers if they’re up and healthy