Incus-compose - the missing docker-compose for incus

, ,
1

A proper Docker-Compose workflow for Incus. The idea started with Brian’s proof-of-concept; this is a ground-up build, rewritten three times to get the core of it right.

asciicast

incus-compose runs multi-container apps on Incus from standard Compose files: point it at a compose.yaml and run incus-compose up. Incus pulls OCI images natively. Key bits:

  • Familiar commands: up, down, start, stop, restart, list (and ps), logs, exec, config, plus build, healthd, incus (pass-through), and self-update
  • Compose project parsing via compose-go, with automatic compose.incus.yaml overrides and x-incus / x-incus-compose extensions for raw Incus options
  • OCI image pulling from docker.io, ghcr.io, and other registries
  • Two-stage image cache in a dedicated Incus project (survives down/up, avoids registry rate limits)
  • Local image building via Podman/Docker
  • Bridge networks with automatic name sanitization
  • Static IPv4/IPv6 addresses with automatic DHCP ranges
  • Port forwarding via proxy devices or kernel NAT mode
  • Storage volumes with UID/GID shifting; bind mounts (pass-through by default, optional seeding)
  • Health checks, restart policies, and depends_on: service_healthy ordering via the ic-healthd sidecar
  • Service scaling with up --scale and orphan pruning
  • Incus project isolation
  • Resource limits and other advanced compose features (shm_size, container_name, etc.)
  • Configuration via INCUS_COMPOSE_* environment variables for every flag, with a configurable parallel worker count

The examples include a full Immich stack, so it handles real multi-service apps.

Requirements: An https Incus remote (needed for ic-healthd). See: incus-compose

Install:

curl -sSfL https://raw.githubusercontent.com/lxc/incus-compose/main/install.sh | sh -s -- -b ~/.local/bin

It’s still beta — feedback from real compose projects is exactly what I’m after.

René

Repo: GitHub - lxc/incus-compose: Bring the familiar Docker Compose workflow to Incus containers · GitHub
Releases: Releases · lxc/incus-compose · GitHub
Changelog: incus-compose/CHANGELOG.md at main · lxc/incus-compose · GitHub
Examples: incus-compose/examples at main · lxc/incus-compose · GitHub
Compatibility: incus-compose

8 Likes
Proof Of Concept: incus-compose
Help wanted: beta testers for incus-compose
Early development version of the resurrected `incus-compose` online
Configuring Multiple Managed NICs with OCI Containers
DNS resolution on a managed bridge doesn't work correctly sometimes
Early development version of the resurrected `incus-compose` online
2

Just released v0.0.1-beta3

New features:

Fixes:

2 Likes
3

As a user you might wanna have a look at the Fixtures for a quick start.

A complete compose file is immich (I run it in production with a manual docker.io/pcdummy/* mirror) and if you need nat-proxy is also very helpfull.

1 Like
4

New release v0.0.1-beta4

Features

  • Added support for x-incus options.

    • Compose services can now pass raw Incus instance config directly through to Incus.
    • Example use cases: memory/CPU limits, nesting, security flags.
    • Project configuration can now be applied consistently across CLI commands and Incus operations.

  • Added automatic loading of the default incus profile.

  • Added project-wide x-incus-compose.network-profile support.

    • Disables a per project default network/bridge.

  • Added healthd reload support on service changes.

    • The health daemon can now reload/reconfigure when service state changes.

Fixes

  • Fixed network creation race that could cause dnsmasq failures in CI.

    • Avoids immediately updating a newly-created Incus network, which could restart dnsmasq before the old process released its socket.

  • Fixed down so compose-managed networks are deleted when the project is brought down.

    • This fixes dangling networks…

  • Fixed up --no-pull behavior.

    • The no-pull flag is now respected correctly.
    • 2x faster test runs, 3min from 6min

Tests and tooling

4 Likes
5

v0.0.1-beta5 / v0.0.1-beta6

Compose Incus override files

incus-compose now automatically loads a compose.incus.yaml file when it is present next to the main Compose file.

This makes it easier to keep upstream Docker Compose files unchanged while adding Incus-specific configuration in a separate override file.

Example:

compose.yaml
compose.incus.yaml

Running:

incus-compose -f compose.yaml config

will also load:

compose.incus.yaml

when available.

Healthd image base changed

The ic-healthd image now uses busybox:glibc instead of scratch.

This gives the image a minimal runtime base while still keeping it lightweight.

3 Likes
6

That is ic-healthd (the healthchecker daemon that runs in a container):

$ incus --project=immich console --show-log ic-healthd
2026/06/04 07:15:43 debug mode enabled
2026/06/04 07:15:43 data dir: /var/lib/ic-healthd
2026/06/04 07:15:43 secrets dir: /run/secrets/ic-healthd
2026/06/04 07:15:43 incus: https://10.131.32.1:8443
2026/06/04 07:15:43 projects: [immich]
2026/06/04 07:15:43 no persisted cert; performing first-run registration
2026/06/04 07:15:43 certificate registered and persisted
2026/06/04 07:15:43 connected to incus, project=immich
2026/06/04 07:15:43 database-1, unless-stopped, ["CMD","/usr/local/bin/healthcheck.sh"]
2026/06/04 07:15:43 ic-healthd, ,
2026/06/04 07:15:43 immich-machine-learning-1, unless-stopped, ["CMD","python3","/usr/src/healthcheck.py"]
2026/06/04 07:15:43 immich-server-1, unless-stopped, ["CMD","/usr/src/app/server/bin/immich-healthcheck"]
2026/06/04 07:15:43 redis-1, unless-stopped, ["CMD","redis-cli","ping","||","exit 1"]
2026/06/04 07:15:43 started checker for immich-machine-learning-1 (interval=5s, retries=3, restart=false)
2026/06/04 07:15:43 started checker for immich-server-1 (interval=5s, retries=3, restart=false)
2026/06/04 07:15:43 started checker for redis-1 (interval=5s, retries=3, restart=false)
2026/06/04 07:15:43 started checker for database-1 (interval=10s, retries=3, restart=false)
2026/06/04 07:15:43 health daemon running, monitoring 4 services
2026/06/04 07:15:51 immich-machine-learning-1: check failed (1/3)
2026/06/04 07:15:56 immich-machine-learning-1: check failed (2/3)
2026/06/04 07:16:01 immich-machine-learning-1: check failed (3/3)
2026/06/04 07:16:01 immich-machine-learning-1: stopping the check
7

It’s amazing how fast you’re iterating and improving this - well done!

2 Likes
8

Same as @kgoetz, I’m awed by the breadth and depth of the changes and fixes you accomplished in so little time! Thanks a lot @jochumdev :folded_hands:t2:

It’s a quick and dirty fix, also I’m using unreliable image endpoints as well as one of my one uploads.

Are the fixtures services/apps requiring a little bit of tweaking to work with incus-compose, so you’ve cloned the upstream images to include those fixes and pull from those instead?

From a cursory look at https://hub.docker.com/layers/pcdummy/immich-server/v2/images/sha256-767bd708b3972bd5474403eb023a1dc4ffa7dd8d7b133a12b2cd45ea86d5b11b, it seems you’ve just added HEALTHCHECK &{["CMD-SHELL" "immich-healthcheck"] "0s" "0s" "0s" "0s" '\x00'} at the end of the Dockerfile, right?

In test/fixtures/immich/compose.incus.yaml · main · René Jochum / incus-compose · GitLab we see ports: !reset []. Would you mind explaining what it does and when we might need to do that for other compose projects?

Thanks a lot for all your work, I’ll report back when I make the time to test the changes properly :slight_smile:

9

For immich, I see you have moved the depends_on key from the upstream Compose file to compose.immich.yaml. Is this also something we should generally do?

(I’m just trying to learn the ins of immich-compose to make good use of it :slight_smile: )

10

Heya! :slight_smile:

Many thanks for all the thinking you/you guys do for the project, i’ll setup an authors file soon and give you credits, ok?

That’s maybe a left over from pull → tag → push.

I did that to test ports as i’m not using ports at all, I have a caddy pointing to the server’s ip.

Maybe a mistake, note that healthd isn’t feature complete yet it doesn’t “restart” and stuff.

1 Like
11

I forgot to answer that one, I’m doing the copy cause of ghcr.io beeing a bad registry for me with skopeo, want to move back using CNCF Distribution as a cache.

12

Could you maybe have been hit by the extended delay it took to pull some of the immich-related images from ghcr.io?

Several of us reported the issue in the previous thread and @kgoetz filed a nug report which has been fixed since (it should be in 7.1, although I haven’t tried it): Please use skopeo with --no-tags · Issue #3367 · lxc/incus · GitHub

1 Like
13

Have no clue anymore, it was just buggy :confused:

Now having this:

EDIT - new location: oci-registry-cache · main · René Jochum / incus-compose · GitLab

14

v0.0.1-beta7

New Features

incus-compose healthd subcommand group
Direct sidecar management without touching application services:

incus-compose healthd logs      # stream ic-healthd logs
incus-compose healthd reload    # reload health-check config
incus-compose healthd restart   # restart the sidecar
incus-compose healthd up        # recreate the sidecar (--recreate supported)
incus-compose healthd down      # remove the sidecar

External network name override via x-incus-compose.network
Networks can now declare their real Incus name independently of the compose key:

networks:
  backend:
    external: true
    x-incus-compose:
      network: my-actual-incus-net

Name resolution uses a 4-candidate probe (raw, sanitized — for both the override and compose name) and locks in the first match.

Bug Fixes

  • serviceName truncation regression — hyphenated service names (e.g. my-service) were incorrectly stripped. Only trailing -{n} integer suffixes (scaled instance convention) are now removed.
  • ic-healthd missing from list output — the healthd sidecar container now appears when running incus-compose list / ps.
  • Hardcoded default storage pool — ic-healthd resources now use the client’s configured DefaultStoragePool instead of always using "default".
  • Healthd re-create lost flags — up --recreate on a healthd container no longer loses --incus/--project OCI entrypoint flags due to a stale entry in the resource store. ResourceStore.Remove() is now called on every Delete().
1 Like
15

A big change, since incus-compose has healthd it requires https remotes:

https://gitlab.com/r3j0/incus-compose#prerequisites

EDIT: Updated prereqs, no more need for all interfaces listening :8443, see: docs/healthd.md · main · René Jochum / incus-compose · GitLab

16

0.0.1-beta8

Features

  • healthd restart — works as intended now, you kill a service healthd brings it back :slight_smile: .
  • list --healthd — opt-in flag to include the ic-healthd sidecar in list output.
  • up --detach|-d — detach after starting services (#38), logs are printed if not detached.
  • incus proxy — new incus-compose incus proxy pass-through command (incus --project={name} <xyz>) (#37).
  • Image name in ps / list — the IMAGE column is now populated for all instances, including the healthd sidecar, without requiring Image resources in read-only stacks. The image name is stored on the instance at creation time (user.image_alias) and resolved from volatile.base_image as a fallback.
  • healthd resource limits — the ic-healthd sidecar is now capped at 1 CPU and 50 MB RAM by default, this is required for project wide limits.
  • ic-healthd version — the sidecar binary now compiles in and prints its version.
  • OCI registry cache — oci-registry-cache promoted to a standalone helper project.

Fixes

  • down no longer deletes externally-managed networks.
  • healthd: fixed API endpoint resolution when connecting to the Incus socket (#39).
  • External network handling: various fixes for projects that attach to pre-existing networks.

Breaking changes

  • --no-pull replaced by --pull (string flag) for Docker Compose compatibility.
17

0.0.1-beta9

Image handling (major rework)

OCI config extraction at download time (38c4e13)
UID, GID, entrypoint, and cwd are now read from a temporary stopped container right after image download, stored as image properties (oci.uid, oci.gid, etc.) via UpdateImage. Subsequent runs read them back from properties — no extra container needed.

Two-stage cache restored (08df259, reverts e6a3dfa)
The image cache layer (source → cache project → instance project) was briefly removed, then restored. The revert also restored docs/architecture/client/image.md.

Deferred source/cache detection (e06355b)
GetImageServer and EnsureProject calls moved from newImage() into Ensure() — no network connections during the configuration phase, fixing CI slowness.

Resource deduplication (114818d)
ResourceStore.Get now compares by IncusName() so normalized references (docker.io/nginx:alpine vs docker.io/library/nginx:alpine) return the same object, preventing duplicate alias races.

Volume / bind-mount handling

Files via InstanceFile push, dirs via seeded volumes (717435e)
Bind-mount files are pushed post-start via InstanceFile; bind-mount directories become storage volumes with HostPath seeding on create. PostDevices and ActionPostEnsure are removed — volumes live in Devices and are ensured before CreateInstanceFromImage.

Healthd

Healthd resource removed (3c641f8)
client.Healthd wrapper replaced by healthdUp/healthdDown helpers in cmd/incus-compose/healthd.go. Cert management, network detection, and instance config now live in cmd. KindHealthd, HealthdConfig, and resource_healthd.go deleted.

Healthd instance prefixed with project name (f8fde58)
Prevents name collisions across projects.

Skip token registration when cert exists (0c4617f)
Prevents repeated re-registration on ic-healthd restart.

Bug fixes

down network error demoted to warn (9c2ac57)
Network listing failure during --project delete is now a warning; nil-check prevents panic when listing fails.

Tooling

justfile version tags (8186e0c)
Added --long --dirty to version tags for healthd builds.

1 Like
18

beta9 has a breaking change as I removed the project name from storage volume names.

To migrate:

incus-compose --debug down
# remove newly created one
incus-compose incus storage volume rm default postgres
# rename old
incus-compose incus storage volume rename default immich-postgres postgres

EDIT: Sorry the upcoming beta10 will need another rename as I have to prefix volumes with vol-.

19

@neitsab @pyrodogg @kgoetz @Sagi @edorgeville

You guys did some testing of incus-compose I would love to mention you as testers in the README and/or AUTHORS file with a link to the Gitlab and/or the discuss profile, what do you think?

20

0.0.1-beta10

The main theme of this release is making Compose workflows more complete: local image builds, better progress feedback, and improved healthcheck restart behavior.

BREAKING CHANGES

No more windows client support

As we now have build windows isn’t supported anymore, we can build windows clients without that if theres a request for that. Please fill an issue if you need a windows version.

Since beta8 storage volume names changed twice:

Old name: {project}-{name}
New name: vol-{name}

If the volume name is longer than 59 chars it will be hashed.

Existing volumes are not migrated automatically. If you used named volumes or bind-mounted directories with older beta releases, you may need to clean up or migrate old Incus storage volumes manually.

Highlights

Compose build: support

incus-compose can now build local service images from Compose build: definitions.

New commands and flags:

incus-compose build
incus-compose build web worker
incus-compose up --build
incus-compose up --no-build

Docs: docs/build.md.

Progress output

All operations now report progress in the CLI.

Better healthcheck restart behavior

ic-healthd now has restart backoff:

5s -> 10s -> 20s -> 40s -> 60s max

This avoids tight restart loops for unhealthy services.

restart: unless-stopped is also handled more correctly now:

  • incus-compose stop marks an instance with user.stopped=true
  • incus-compose start clears that flag
  • ic-healthd skips automatic restarts while the service was intentionally stopped

Bind mount and volume fixes

Bind mount handling got fixes around storage volume resolution and copied file metadata.

Notable changes:

  • bind-mounted directories now resolve through the actual Incus storage volume name
  • copied files/directories use more normal permissions: 0644 / 0755
  • a UID/GID copy bug was fixed
  • storage volume names are now prefixed with vol- to avoid naming collisions

Overall: beta10 is maybe the last step to a stable 1.0.0.