Early development version of the resurrected `incus-compose` online

jochumdev · December 24, 2025, 9:35am

Project lives at René Jochum / incuscompose · GitLab

The current README as reference here:

Bring the familiar Docker Compose workflow to Incus containers. incus-compose implements the Compose specification for the Incus ecosystem, allowing you to define and run multi-container applications using the same docker-compose.yml files you already know.

Status

Early Development - This project is in its initial phase. APIs and behavior may change. Contributions and feedback are welcome!

It does “up, down and ps” those are well tested.

Compose projects get created with a incus project, storage pool volumes and as much bridge networks as you wish. Yes, it does also shift your Volumes transparently and it does bind mounts.

No specials included (caps and so on).

Why incus-compose?

Incus provides powerful system containers and virtual machines, but lacks the declarative multi-container orchestration that Docker Compose offers. This tool bridges that gap, letting you:

Use existing docker-compose.yml files with Incus containers
Leverage the superior security and isolation model of Incus
Run Docker/OCI images directly from registries like docker.io and ghcr.io
Manage complex multi-container applications with familiar commands

Goals

Specification Compliance

Parse and execute compose projects according to the Compose specification using compose-go
Support the latest compose file format features
Maintain compatibility with Docker Compose workflows

Incus Integration

Interact with Incus through its official Go client library
Leverage Incus’s native OCI registry support for image pulling
Support both system containers and VM instances where applicable

Command Compatibility

Implement core docker compose commands: up, down, start, stop, restart, logs, ps, exec, and more
Match Docker Compose CLI behavior and options where possible
Document all intentional differences from Docker Compose
Treat unexpected behavior differences as bugs

Container Building

Build container images using Podman (preferred) or Docker via their respective sockets
Support both local Dockerfiles and remote build contexts

Quality Assurance

Comprehensive unit test coverage for core functionality
End-to-end tests validating real-world compose scenarios
CI/CD integration for automated testing
Well-documented codebase with examples

Library Support

Expose a Go API (pkg/icclient/) for programmatic use
Enable embedding in other tools and workflows
API is unstable - will change without notice until this message is gone

Usage

docker.io and ghcr.io images

Simply add the remote as docker.io or ghcr.io to your incus server:

incus remote add --protocol oci docker.io https://docker.io
incus remote add --protocol oci ghcr.io https://ghcr.io

Now you can use incus-compose to pull and run images from those remotes, e.g.:

services:
  hello-world:
    image: docker.io/hello-world:latest

Credits

This is based on work done by @bketelsen
Some parts are replicated or copied from docker compose.
Im using AI to generate tests and to help me with reviews, real code is 90% hand written.

bketelsen · December 24, 2025, 6:28pm

This is awesome news. I stopped mostly because I knew how much work it would be to get real health-check based orchestration going (don’t start wordpress until mysql is alive and serving on port 3306) — but this was before I started using any AI assistance. I hope you’re able to carry the torch and make something great!

jochumdev · December 25, 2025, 7:11pm

Thanks Brian, also wanna public say sorry for not providing future work on my ugly patch, private stuff came in between. Sorry.

bketelsen · December 25, 2025, 7:32pm

Open source shouldn’t mean guilt
no obligations, no worries!

Sagi · December 28, 2025, 8:35am

Thank you for doing this work, I am really happy to see this continue.

I did a quick clone, followed by a go build -o incus-compose cmd/incus-compose/main.go and then found out API access through the unix socket is not currently supported. To be continued after I sort http API access w/ client certificates

jochumdev · December 28, 2025, 9:11am

Lovely! Did a big refactor of the whole app, it contains a compatibliity fix with the cli. Will push soon.

Btw, welcome!

jochumdev · December 28, 2025, 9:55am

Pushed my rewrite/refactor. Test as you wish, should work by default using the “local” remote of your incus cli.

Sagi · December 28, 2025, 10:12am

wow, that’s a lot of change in a single commit.

jochumdev · December 28, 2025, 10:15am

Ye, it’s “my” own project and in early development

Hope it’s the last big refactor.

jochumdev · December 28, 2025, 12:36pm

Having an issue with Networks on new projects, fixing it.

EDIT: Done.

Sagi · December 29, 2025, 10:12am

OK, some more testing. To take incus-compose for a spin, I’m trying the docker-compose files released by the Immich project:

wget -O docker-compose.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env

Then incus-compose -p immich-test up results in:

11:04 ERR Error(s) during up url=https://localhost:8443 error=“failed to create image "ghcr.io/immich-app/postgres:18-vectorchord0.5.3": Failed remote image download: Failed getting remote image info: Image not found”
11:04 ERR Command returned error=“failed to create image "ghcr.io/immich-app/postgres:18-vectorchord0.5.3": Failed remote image download: Failed getting remote image info: Image not found”

Checking incus monitor output for more details gives:

location: none
metadata:
context:
name: mmich-app/postgres:18-vectorchord0.5.3
stderr: ‘Failed to run: skopeo --insecure-policy inspect docker://ghcr.io/mmich-app/postgres:18-vectorchord0.5.3:
exit status 1 (time=“2025-12-29T11:04:03+01:00” level=fatal msg=“Error parsing
image name "docker://ghcr.io/mmich-app/postgres:18-vectorchord0.5.3": Requesting
bearer token: received unexpected HTTP status: 403 Forbidden”)’
stdout: “”
level: debug
message: Error getting image alias
timestamp: “2025-12-29T11:04:03.960371876+01:00”
type: logging

Curiously skopeo is passed docker://ghcr.io/mmich-app/ (notice the missing i), which is a path that does not exist.

I have yet to find where the first character of the path gets removed, it’s rather weird. Not sure if this a local issue, a bug in incus-compose, a bug in incus, or something with skopeo.

Version information:

skopeo version 1.18.0
incus 6.20
incuscompose main@82c3ce34193bf06a43ba5cacbd5ad3f49476c7f7

Sagi · December 29, 2025, 10:28am

This appears to be the offending line:

	r.Operation.ImageNoRemote = strings.TrimLeft(ref.String(), r.Operation.Remote+"/")

That function strips all leftmost occurences of whatever is in r.Operation.Remote+"/", which I’m guessing in this case includes an i.

I think this is what you had intended?

diff --git a/client/resources.go b/client/resources.go
index c8669ba..53150f6 100644
--- a/client/resources.go
+++ b/client/resources.go
@@ -358,7 +358,7 @@ func (r *Image) sanitize() error {
 
        r.Operation.Cache = r.project.imageCache
 
-       r.Operation.ImageNoRemote = strings.TrimLeft(ref.String(), r.Operation.Remote+"/")
+       r.Operation.ImageNoRemote, _ = strings.CutPrefix(ref.String(), r.Operation.Remote+"/")
 
        r.incusName = r.name

Sagi · December 29, 2025, 3:33pm

With that change, and some edits to the docker-compose.yml to turn binds into volumes (still using the API) , incus-compose succesfully brought up Immich’s docker-compose config.

To get things to work in my local setup, I patched in a --pool-name cli parameter to change from the default pool (see below). Then, exploring Immich some more, I discovered that I would actually like to have the ability to map docker-compose volumes to incus storage pools. I could see a --volume volname:poolname or something, or alternatively some config file that defines such mappings.

@jochumdev do you have any such needs, or appetite for such a change?

diff --git a/cmd/incus-compose/main.go b/cmd/incus-compose/main.go
index 85464e9..71cd71e 100644
--- a/cmd/incus-compose/main.go
+++ b/cmd/incus-compose/main.go
@@ -887,6 +887,12 @@ func main() {
                                Aliases: []string{"p"},
                                Usage:   `Project name`,
                        },
+                       // TODO: check for valid pool name; idk. what incus permits?
+                       &cli.StringFlag{
+                               Name:    "pool-name",
+                               Usage:   `Pool name`,
+                               Value:   "default",
+                       },
                        &cli.StringSliceFlag{
                                Name:    "file",
                                Aliases: []string{"f"},
@@ -937,6 +943,7 @@ func main() {
                                opts := []client.Option{
                                        client.URL(url),
                                        client.InsecureSkipVerify(),
+                                       client.DefaultStoragePool(cmd.String("pool-name")),
                                }
 
                                // Add TLS client certificate if provided
@@ -979,6 +986,7 @@ func main() {
 
                        opts := []client.Option{
                                client.ProvideConnection(instanceServer, imageCache),
+                               client.DefaultStoragePool(cmd.String("pool-name")),
                        }
 
                        slog.Debug("Using connection", "remote", remote)

jochumdev · December 30, 2025, 6:41am

Yes I’ll change that. How would you like to get mentioned in the README?

EDIT: Thinking about “Sagi did some testing and bugfixing”.

Yes, I do.

@bketelsen was exploring x-incus- attributes in compose files. Maybe we can do the same?

But there is another refactoring coming up. To archive “worker-pools” I need to make each Resource in client return an Operation like that one.

type Operation struct {
	Context context.Context
	handler func(error) error

	// Set in `Operation.Handle` or directly with `NewDoneOperation`
	done bool

	// Set in `Operation.Handle` or directly with `NewErrorOperation`
	Error error
}

func (o *Operation) Handle() error {
	if o.done {
		return o.Error
	}

	select {
	case <-o.Context.Done():
		return o.Context.Err()
	default:
		err := o.handler(o.Error)
		o.Error = err
		o.done = true
		return err
	}
}

This refactor is nearly done. After I pushed that I’m happy to accept PRs.

About my use of Gitlab, are you happy to use Gitlab?

jochumdev · December 30, 2025, 9:13am

I forgot to say: Thank you, very happy that you do tests. This helps a lot.

Sagi · December 30, 2025, 7:35pm

I appreciate you taking the time to credit people, though I don’t feel I have done much I’ll let you decide.

My use case would be to use (supported) docker-compose.yml from various projects, but run them on Incus. For upgrades, it would be simpler if no modifications would need to be made to such files. So personally, I would have a slight preference for some kind of Incus-specific config file on the side, a bit like a .env file.

I don’t think I understand what you are after, but I’ll probably find out in the future .

Sure, I think I can find my way and send a patch or two if I run into something.

jochumdev · December 30, 2025, 7:41pm

Ye, sure. Will do so after some more tests of you. Pick a name and link

Same use case we have, let’s test lots of those.

In short, I wanna have parallelism

Sagi · January 2, 2026, 9:49am

I’ve just noticed that I have zombie cgroups floating around that correspond to instances that incus-compose created. They persist when the instances are shut down and when incus is restarted. I am both unsure how this came about, ~~or how to resolve this situation~~ (edit: I can rmdir the cgroups manually).

# incus list --project immich-test
+-------------------------+---------+------+------+-----------------+-----------+
|          NAME           |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+-------------------------+---------+------+------+-----------------+-----------+
| database                | STOPPED |      |      | CONTAINER (APP) | 0         |
+-------------------------+---------+------+------+-----------------+-----------+
| immich-machine-learning | STOPPED |      |      | CONTAINER (APP) | 0         |
+-------------------------+---------+------+------+-----------------+-----------+
| immich-server           | STOPPED |      |      | CONTAINER (APP) | 0         |
+-------------------------+---------+------+------+-----------------+-----------+
| redis                   | STOPPED |      |      | CONTAINER (APP) | 0         |
+-------------------------+---------+------+------+-----------------+-----------+

# find /sys/fs/cgroup/ -type d -name 'lxc.monitor*immich*'
/sys/fs/cgroup/lxc.monitor.immich-test_immich-machine-learning-1
/sys/fs/cgroup/lxc.monitor.immich-test_redis
/sys/fs/cgroup/lxc.monitor.immich-test_immich-server-4
/sys/fs/cgroup/lxc.monitor.immich-test_database-2
/sys/fs/cgroup/lxc.monitor.immich-test_database
/sys/fs/cgroup/lxc.monitor.immich-test_immich-server-2
/sys/fs/cgroup/lxc.monitor.immich-test_redis-2
/sys/fs/cgroup/lxc.monitor.immich-test_immich-server-5
/sys/fs/cgroup/lxc.monitor.immich-test_immich-machine-learning
/sys/fs/cgroup/lxc.monitor.immich-test_immich-server-3
/sys/fs/cgroup/lxc.monitor.immich-test_database-1
/sys/fs/cgroup/lxc.monitor.immich-test_redis-3
/sys/fs/cgroup/lxc.monitor.immich-test_immich-server-1
/sys/fs/cgroup/lxc.monitor.immich-test_immich-server
/sys/fs/cgroup/lxc.monitor.immich-test_redis-1

Any idea if this is related to behaviour of incus-compose itself?

Sagi · January 2, 2026, 9:54am

Another observation that may more useful for direct improvement of incus-compose itself. Again going back to the immich docker-compose files; it seems the following bit of config for the immich-server component:

    ports:
      - '2283:2283'

gets translated in the following incus device config for the corresponding immich-server instance:

devices:
  proxy-0.0.0.0-0:
    connect: tcp:127.0.0.1:0
    listen: tcp:0.0.0.0:0
    type: proxy
  proxy-0.0.0.0-2283:
    connect: tcp:127.0.0.1:2283
    listen: tcp:0.0.0.0:2283
    type: proxy

The second one makes sense to me; the first does not. I have not yet looked at the code.

jochumdev · January 2, 2026, 12:39pm

I’m not, have to test/ play around with that.

If you like manualy create oci containers and check the behaviour, you can extract some commands here on howto do so https://github.com/nuttyb-community/server/blob/main/justfile.

It would be nice to know which features trigger that also the code itself does not enable any “debugging” options or the like.

I’ll make sure this bug is gone together with the the other 2 bugs you reported and fixed.

That will be part of the major architectural update I’ll push soon to incus-compose.

It will feature:

An extensible client library that wraps the incus go client library
Detailed unit tests
Parallel image downloads
A better ps with more incus related infos
down --project aliased to down --volumes = everything except images gone.
“hooks” for everything
Somehow good docs
The base layer for progress reports (they will be easy to implement after)

I write it with the pilosopy “Keep it simple stupid” and “boring”.