Incus-compose - the missing docker-compose for incus

, ,
62

Yes that IP is test.mosquitto.org, I put it there as part of testing to see if it was TLS/authentication related. That instance in the stack connects to an external (to the container) MQTT broker. The production target is a private address on the LAN. When set to that, the container dies with the same error…

63

v1.0.0-beta19

Mostly CLI and healthd fixes, plus event-driven log following.

https://asciinema.org/a/1259458

Update

From 1.0.0-beta.16 and higher:

incus-compose self-update

or update with the script:

curl -sSfL https://raw.githubusercontent.com/lxc/incus-compose/main/install.sh | sh -s -- -b ~/.local/bin

Highlights

Event-driven logs --follow – The logs -f command now uses the Incus events API to dynamically attach and detach log streams as instances start and stop. It no longer exits when instances go away and picks up new instances automatically. (#3)

down now cleans up all resources – Running down --project ic deletes instances, networks, volumes, and the healthd sidecar. Previously it relied on incus to do so.

healthd reliability – Fixed restart counting during the start period, fixed instance tracking after cancellation, and fixed healthd up/healthd down lifecycle. (#5)

CLI

  • logs -f reacts to instance lifecycle events instead of discovering instances once at startup (#3)
  • down deletes all resources including the healthd sidecar
  • --debug no longer shows progress bars (they interfered with debug output)
  • Removed --with-deps from logs (unnecessary flag)

Client

  • Removed automatic retry on client operations
  • DNSWwatcher: wait up to 5s after dnsmasq restart before starting the next instance

Healthd

  • Allow restarts during the start period; don’t recalculate retry budget
  • Reset running instance count after cancellation
  • Fix healthd up/healthd down commands (#5)

Other

  • Extended documentation with a Terms section (#4)
  • Updated example healthchecks with start_xxx directives
  • Immich example: wait for DNS readiness, drop tini
  • CI workflow refactor
  • Renamed slow e2e tests to TestSlowXXX for clearer filtering
64

v1.0.0-beta.20

Mostly an internal project/stack refactor, plus network-readiness and healthd reliability fixes.

Green across the board — E2E suite + ~50% coverage:

Highlights

Instances wait for the network before starting – Added raw.lxc=lxc.start.delay=1 so an instance no longer races ahead of its network being ready. Fixes flaky startups where services came up before DNS/networking was usable.

More reliable DNS and healthd updates – The DNS update now retries once on an ETag mismatch (concurrent-update race), and user.healthchecking.stopped updates went through a cleaner path – the hacky PATCH workaround is gone.

Update

From 1.0.0-beta.16 and higher:

incus-compose self-update

or update with the script:

curl -sSfL https://raw.githubusercontent.com/lxc/incus-compose/main/install.sh | sh -s -- -b ~/.local/bin

Please test (beta)

I reworked the ordering logic for up / down / start / stop, with and
without dependencies. Tested as best I can, but ordering bugs are easy to miss –
please watch this closely and report anything off.

Note the deliberate asymmetry in the dependency defaults:

  • up / down follow depends_on by default; --no-deps limits the
    command to the named service only.
  • start / stop / restart act on the named service only by default;
    --with-deps makes them follow depends_on like up/down.

Worth a careful look:

  • up brings deps up first; down tears down in reverse order
  • up <svc> --no-deps / down <svc> --no-deps touch only <svc>
  • start <svc> / stop <svc> touch only <svc>; add --with-deps to pull in its chain
  • Projects without depends_on behave the same with or without the flags

Removed

  • No more deb/rpm/apk packages – releases now ship the tarball/binary and install script only. If you installed via a distro package, switch to the install script above.

Project / internals

  • Project no longer returns a Stack; the CLI now owns stack assembly
  • Reworked how resources are added to a stack, with a new helper that adds them in priority order
  • Exported SanitizeNetworkName
65

v1.0.0-beta.21

Standalone and bugfixed healthd, more x-incus reach, a native exec, and an error-severity system so recoverable problems warn instead of aborting.

Green across the board — E2E suite + ~60% coverage:

Highlights

x-incus extensions reach further — raw Incus options now pass through on service networks, service volumes, and devices, plus direct tmpfs on services. Same verbatim key/value passthrough as instances/networks.

exec uses native incus exec — dropped our MVP terminal implementation in favor of the real thing (~250 lines gone). Better TTY handling and parity with the incus CLI.

lots of work on healthd again — ic-healthd got its own tests and is now able to run standalone. Env vars were renamed to the INCUS_COMPOSE_HEALTHD_* prefix and there’s a --token flag.

Errors can be demoted to warnings — new Clone() + IgnoreError() let commands treat non-fatal problems as warnings instead of hard failures, with a reworked severity system behind it. up/down/start/stop/restart in particular no longer abort on errors that don’t matter.

Please test (beta)

The CLI got a lot of small usability and functionality fixes across up / down / start / stop / restart — --with-deps handling, progress output, and error demotion all changed shape. Tested as best I can; please report anything off.

Other fixes

  • Instance volumes now land on the correct storage pool
  • security.shifted is left alone when you’ve set it yourself
  • Overridden network names are honored for normal networks too, not just special cases
  • progress.bypass() for all stdout/stderr — fixes garbled output (closes #37)
  • DNS watcher skipped when the service name equals the Incus name; no watcher for empty service names

Internals

  • StackFailFast() and Stack.SetOptions()
  • Exported SanitizeProjectName()
  • OCI config extracted after a build; dedup on both Name() and IncusName()

Update

From 1.0.0-beta.16 and higher:

incus-compose self-update

or with the script:

curl -sSfL https://raw.githubusercontent.com/lxc/incus-compose/main/install.sh | sh -s -- -b ~/.local/bin
66

This is the last per-release announcement in this thread. incus-compose self-update works reliably now, and the changelog these posts became lives in the repo: CHANGELOG.md.

1.0.0 will get its own post.

I’d love to see active discussion and feedback about incus-compose here — or start a new topic if you prefer, just tag it incus-compose.

67

Is incus-compose up --no-start --pull always the equivalent of docker compose pull ?

Normally to update a stack, I docker compose pull and then docker compose down && docker compose up

Regards.

68

So pull is for offline images? Yes up --no-start --pull=always does the same as a pull would do.

EDIT: Thanks for that one, updated the docs.

69

I might lied, a up --no-start --pull=always creates instances where a pull wouldn’t.

EDIT: I implement the pull command now, it’s a no brainer.

70

Having fun with 30 workers :slight_smile:

https://asciinema.org/a/1260145

71

pull will be in the next release: feat(cli): add the pull command · lxc/incus-compose@68e0524 · GitHub

72

Trying to move a Frigate compose instance over to incus-compose. It seems that incus-compose cannot bind /dev/usb and /dev/dri/renderD128 as with docker. (devices: not part of compose compatability)

I saw this Running Frigate NVR on IncusOS? - #3 by stgraber (even though that is specific to IncusOS) and was wondering if or how I can add these devices via x-incus or some other method.

devices:
  coral-early:
    productid: 089a
    required: "false"
    type: usb
    vendorid: 1a6e
  coral-late:
    productid: "9302"
    required: "false"
    type: usb
    vendorid: 18d1
  gpu:
    type: gpu
    vendorid: "8086"
73

We don’t have support for USB and GPU devices yet, also I don’t have the required Hardware to test that.

I’ll implement that and give you in a private Message either a binary or howto run it for testing, ok?

74

That wouold be great. Thanks!

75

Done.

76

Just wanted to let you know that adding devices via the x-incus-compose: extension is working well.

Hooray! Thank you!

77

Very happy I am :stuck_out_tongue:

78

I’m looking forward to it! The latest beta has been the most stable release candidate yet. Basic up/down/restart all working nicely. My bind mounts are working again (for a couple betas now). And healthd seems to actual work now :+1:

79

Thanks for the feedback, its important for me!

I’m testing compose files from the wild, found some stuff:

  • service.{name}.command overwrites → it should append to oci.entrypoint
  • Volumes where the target path don’t exist fail to mount → I have a workaround for that.
  • Healthd with beta.21 added a TOKEN over env instead a “secret” → i hate that and Stéphane told me about sFTP :slight_smile: so I reworked that.

“missing” features

I would like to bring Always use nat-proxy (requires incus 7.2) · Issue #36 · lxc/incus-compose · GitHub live in 1.0.0 it’s a very easy change but has a broad impact maybe - @blurry suggested that I fallback to proxy devices on < 7.2.

80

This is also important but I have to delay that to 1.1.x its to big of a change, also I have no design in mind yet.

EDIT: Now that I think about it, we could create a tmp instance (which we already do) and copy files from it to the volume. Another problem with that is that docker-compose users expect this as the default, incus users not.

81

Yea, bind mount support is in a good place right now for a 1.0 release. I think the concept of seeding volumes, as an option, could use more thought for a follow-up release, if use cases can actually be found.

I don’t have any more context for my earlier hypothetical example.