Incus compose v1.2 (rc) has been released

,
1

v1.2.0-rc.2

Three changes worth the version bump: health checking can be now one daemon for the whole server which is the default, builds are reusable across projects, and entrypoint: is supported. The rest is bugfixes.

This is a release candidate — please break it before I tag the final. The e2e
suite is green on every pass: four runs against four different Incus remotes,
~4 minutes each at 6 parallel threads. Getting it there was a lot of pain.

Update

Not with self-update this time — it is one of the things this release
fixes, so the copy you have cannot be trusted to replace itself. On 1.0.0 and
1.1.0 it ignores your platform and always downloads the macOS build, which on
Linux and Windows leaves you with a binary that will not run. Reinstall once
instead:

curl -sSfL https://raw.githubusercontent.com/lxc/incus-compose/main/install.sh | sh -s -- -b ~/.local/bin -p
incus-compose up                          # once per compose project

-p is new in that script: it installs rc releases.

This is a one-time detour. self-update --pre-release works again from 1.2.0
onward, because the fix is in the binary you just installed rather than the one
doing the installing.

That up per project is the migration. It stamps the two keys this release
introduced — the project’s healthd scope and the per-instance health-check
opt-in — in place: no --recreate, no downtime, containers keep running.

Skip a project and nothing breaks: it stays on its own sidecar and behaves like
1.1.0. It just doesn’t move.

You only pay this once. The instance opt-in key is what needs an up per
project, and from here on the daemon itself updates with a single
incus-compose healthd up on the server — it replaces a running sidecar in
place when the image you ask for is newer.

:small_blue_diamond: Key Changes & Features

1. One ic-healthd for the whole server

Every project used to get its own sidecar: ten projects meant ten containers,
ten event listeners, ten certificates. Now up creates a single ic-healthd in
the Incus default project and marks the project so that daemon picks it up. A
project costs a goroutine and a map entry instead of a container.

It also stops counting against you: the shared daemon lives in default, so its
limits.cpu/limits.memory no longer land in your project’s quota.

Keeping a sidecar per project, and how the scope is decided

Two reasons to keep one:

  • Least privilege. A project-scoped sidecar gets an Incus certificate
    restricted to its own project. The shared one cannot be restricted — a
    restricted cert carries a fixed project list, and the whole point of the shared
    daemon is to pick up projects that don’t exist yet. So a compromised shared
    ic-healthd is a compromised Incus server. It is one container running one
    binary from an image you control it has no external API.
  • Isolation. A wedged daemon takes down health checking for its own project
    only.

The cost is one container, one certificate and one event listener per project,
plus its limits.* back inside your quota.

x-incus-compose:
  healthd:
    scope: project

or incus-compose up --healthd-scope project the first time.

up writes the choice onto the Incus project as user.healthcheck.scope, and
that stored value then beats both the flag and the compose file — so a project
keeps the scope it was brought up with. Changing your mind later means changing
the key:

incus project set my-project user.healthcheck.scope=project
incus-compose up

up never leaves both daemons on one project: switching to global removes the
project’s own sidecar before marking the project, switching to project marks
it first so the shared daemon lets go before the sidecar appears.

Why untouched projects are safe

The shared daemon only watches projects that positively carry
user.healthcheck.scope=global. A project last brought up by 1.1.0 carries no
scope key at all, and a project-scoped one carries project — neither matches.
They are invisible to the shared daemon and keep running on their own sidecar,
with no window where two daemons watch the same project.

Matching on a value rather than on a key being present is the whole trick.

Managing it: healthd commands without a compose file

The healthd subcommands now run with no compose file in sight, acting on the
shared daemon. That is how you put one on a server before any project exists:

incus-compose healthd up      # create the shared daemon
incus-compose healthd logs -f # watch it

healthd up this way marks no project, so it watches nothing by itself —
projects opt in on their own up. The others fail with no ic-healthd is running instead of complaining about a missing compose.yaml.

healthd down on the shared daemon stops health checking for every project
using it, so it lists them and asks first. --force skips the question and is
required where there is no terminal to ask on (CI, scripts). incus-compose down never touches the shared daemon at all.

Sizing, for a daemon watching a lot of projects:

x-incus-compose:
  healthd:
    workers: 64 # concurrent health checks, default 32
    restart-workers: 8 # concurrent restarts, default 12
    x-incus:
      limits.cpu: 4
      limits.memory: 512MiB

Separate pools because a restart holds its worker far longer than a check does.
The first project to bring the daemon up supplies these; a later project whose
block differs is warned about and ignored, so one compose file can’t restart the
daemon everybody else is using.

Both healthd up and up upgrade a running daemon in place: when the image you
ask for is a newer release than the one it runs, the sidecar is stopped,
removed and recreated from the new image. The comparison is semver and
forward-only, so a machine on an older incus-compose cannot downgrade a daemon
shared with everyone else. Tags that aren’t release versions — moving tags like
latest, and git describe builds — aren’t comparable, so those replace on any
difference.

So updating healthd on a server is one command, and it doesn’t need a compose
file:

incus-compose self-update
incus-compose healthd up
New: --trace

A level below --debug (which it implies), for the lines the daemon emits per
Incus event and per check. Exactly what you want when a project is not being
watched and you need to see the events arrive — and exactly what you don’t want
otherwise, since on a busy server they bury everything else.

incus-compose healthd down --force
incus-compose --trace healthd up
incus-compose healthd logs --follow

Everything the daemon is configured with is injected as environment when the
container is created, so changing any of it is a recreate.

2. Health checking is opt-in

ic-healthd watches an instance only when it carries
user.healthcheck.enabled: "true". A healthcheck: block or a restart policy
alone is no longer enough — the instance has to say it wants watching.
incus-compose writes the key automatically, so you don’t set it by hand.

Opting a single service out 
services:
  sidecar-tool:
    image: docker.io/example/tool:latest
    x-incus:
      user.healthcheck.enabled: "false"

The service keeps its healthcheck: block, it simply isn’t watched.

Related: user.healthcheck.status is now written by ic-healthd and nothing
else. incus-compose no longer stamps starting/stopped on it, so the value
always says what a daemon actually saw. A fresh instance carries no status until
one reports — list shows Unknown for that moment.

3. Reusable builds

A service with build: no longer rebuilds in every project. The image cache is
checked before the builder runs, so the first up anywhere builds and every
project after that copies. --build forces a rebuild after changing a Dockerfile
or context.

This also means a machine with no local podman/docker/buildah — common on
Windows and macOS — can run a compose file with build: in it, as long as
someone has seeded the cache.

4. entrypoint:

Supported, and it follows the compose spec: it replaces the image’s entrypoint
and discards the image’s default command, so the container runs exactly
entrypoint: plus command: with nothing inherited.

services:
  web:
    image: docker.io/library/busybox:glibc
    entrypoint: ["httpd", "-f", "-v", "-p", "8080", "-h", "/www"]

This is the reliable way to control what a container runs — see the warning
about command: on its own, below.

5. --pull never

On up, build and pull (--policy never): never contacts a registry, fails
when the image isn’t already stored. For air-gapped setups. pull --policy is
honoured now instead of being ignored.

6. Parallel-safe

Concurrent up runs no longer collide creating the same volume, profile or
network, and image pulls and builds are serialized per image, so two projects
wanting the same image no longer fight over it.

7. up without --detach matches docker

Create, start, stream logs, and down on interrupt.

Bug fixes
  • ic-healthd reliability: stalled Incus API calls that silently killed an
    instance’s checker for good, checker cancellation races, a zero/negative
    interval crashing the daemon via time.NewTicker, unless-stopped misread
    as a deliberate stop after a transient API error, and per-instance state lost
    on an event-listener reconnect.
  • build.dockerfile is resolved relative to build.context as the compose spec
    requires — two services with different contexts but the same dockerfile: name
    were both built from whichever Dockerfile sat in the current directory.
  • config --format=yaml no longer nests everything under a project: key; it
    matches docker compose config exactly. --format=json keeps the x-incus
    blocks that docker drops, so parse it rather than diffing it against docker’s.
  • A static ipv4_address on a network with no explicit address now fails with an
    explanation instead of producing a broken NIC.
  • raw.dnsmasq entries are no longer appended twice.
  • Default storage pool detection checks the default profile’s root device first.
  • A configs: or secrets: entry aimed at a path the image already ships is
    written instead of being silently skipped — so a config can replace an
    application’s own default config file.
  • command: arguments containing spaces, quotes or $ are shell-quoted
    properly. They used to be wrapped in bare double quotes with no escaping, and a
    single-argument command was passed through unquoted, so either could be
    re-split into the wrong arguments.
  • Pushing directory content into a storage volume no longer closes each file twice.
Library consumers: breaking changes in client/ and project/
  • InstanceConfig.AppendEntrypoint became InstanceConfig.Entrypoint and
    InstanceConfig.Command, both []string.
  • ImageConfig.CacheServer became ImageConfig.CacheClient (a *client.Client).
  • Options.Pull is a PullMode, not a bool.
  • Instance.Ensure and GlobalClient.EnsureProject add declared config keys
    that are missing, comparing keys only — an existing key keeps its value. A
    changed value still needs --recreate.
  • New: project.ErrNoComposeFile, Client.HealthdRunning,
    GlobalClient.ProjectConfig, GlobalClient.ProjectsWithConfig,
    GlobalClient.AddMissingProjectConfig, InstanceConfig.NoRootDevice,
    StorageVolume.SFTP(), StorageVolume.Lock().

:warning: Housekeeping: the image cache

The image cache moved out of the Incus default project into its own
incus-compose-cache. Most people don’t use Incus projects, which meant our
cached images piled up in the one project they actually work in. They now live
somewhere you can inspect, or delete wholesale, without touching your own.

The catch: whatever earlier versions cached in default stays there, and
nothing will ever look at it again. down only removes the per-project copies;
the cache itself is never pruned. A one-time cleanup is worth doing.

A script to prune the old cache

There’s no prune command yet. Everything incus-compose caches is an OCI image,
so filter on that and your native Incus images stay put. Save as prune.sh:

#!/usr/bin/env bash
# Delete OCI images from an incus-compose image cache project.
# Usage: ./prune.sh [remote] [project]
set -euo pipefail

remote="${1:-${INCUS_REMOTE:-local-https}}"
project="${2:-default}"

images=$(incus image list "${remote}:" --project "${project}" -f json |
    jq -r '.[] | select(.properties.type == "oci") | .fingerprint')

if [[ -z "${images}" ]]; then
    echo "No OCI images in ${remote}:${project}."
    exit 0
fi

echo "Deleting OCI images in ${remote}:${project}:"
while IFS= read -r fingerprint; do
    echo "  ${fingerprint}"
    incus image delete --project "${project}" "${remote}:${fingerprint}"
done <<<"${images}"
echo "Done."

chmod +x prune.sh
./prune.sh local-https default

Needs jq. Both arguments are optional — the remote falls back to
$INCUS_REMOTE then local-https, the project to default — but pass them
explicitly, because this deletes.

It deletes every OCI image in that project, including ones you pulled
yourself — nothing marks an image as ours specifically. Check first if you keep
your own OCI images there:

incus image list local-https: --project default -f json \
  | jq -r '.[] | select(.properties.type == "oci") | .aliases[0].name'

Anything still needed gets re-pulled into the new cache on the next up, so the
cost of over-deleting is bandwidth, not breakage.

:warning: Heads-up: command: will change

Not in 1.2.0 — but it is coming, and it will be a breaking change, so here is the
warning ahead of time.

Today command: on its own appends to the image’s entrypoint instead of
replacing its CMD. That is not a choice we made: Incus derives oci.entrypoint
from the OCI runtime bundle’s resolved argv, which already has ENTRYPOINT and
CMD concatenated by the image-to-bundle conversion. The split is gone before
Incus ever sees it, so there is nothing to replace CMD against while keeping
ENTRYPOINT.

lxc/incus#3765 is the proposal to
expose the split upstream. If it lands, command: on its own starts substituting
like Docker does.

You can opt out of all of this today. entrypoint:, new in this release,
takes the image out of the equation — it replaces the entrypoint and discards the
image’s default command, so nothing is inherited and nothing is inferred. That is
exactly why it can be correct now while command: on its own cannot. An
entrypoint: means the same thing before and after #3765, and the same thing in
docker compose.

Who breaks how, when it lands
  • You get fixed. If you wrote a normal Docker-style command: against an
    image with a real ENTRYPOINT, it is broken today — caddy plus
    command: ["run", "--config", "X"] currently runs
    caddy run --config /etc/caddy/Caddyfile --adapter caddyfile run --config X.
    You can only improve.
  • You break loudly. If you reverse-engineered the append and wrote
    command: ["-c", "..."], that will try to exec -c and the instance will not
    start. Annoying, but obvious.
  • You break quietly. The one to watch. An image with
    ENTRYPOINT ["/entrypoint.sh"] and CMD ["serve"] plus command: ["debug"]
    runs /entrypoint.sh serve debug today and /entrypoint.sh debug afterwards.
    Both start. Only the behaviour differs.

One wrinkle: oci.entrypoint is written into the instance config when the
instance is created, and is not rewritten afterwards. So upgrading
incus-compose changes nothing for projects that are already up — it takes effect
on the next up --recreate or a fresh create. Good for uptime, bad for
attribution, because the change can surface weeks after the upgrade that caused
it.

If you use command: anywhere, this is a good moment to note where.

On collapsing ten daemons into one

The per-project sidecar was the honest first version: one project, one container,
one listener, no shared state to get wrong. It just doesn’t survive contact with
a server running fifteen compose projects — that’s fifteen containers whose only
job is watching, fifteen websockets, and fifteen certificates to rotate.

The part that took the longest was nothing to do with events. project-updated
is sent before the change is applied, and carries no config, so a daemon that
reacts by reading the project back races the write — and a read that wins sees
the old config, i.e. answers “not mine” for a project that just opted in. Losing
that race used to be permanent, because instance events for an unwatched project
are dropped and scope was only re-resolved per listener generation. A project
could opt in and simply never be picked up. The fix is that a negative is no
longer believed on the first look.

Also worth knowing, if you go reading the daemon
  • Every send inside the daemon is blocking and nothing is dropped for lack of
    room. One wedged scheduler eventually stalls routing for everyone — deliberately,
    because that is a visible failure the server recovers from (it drops a
    listener that stops reading, which triggers a reconnect and a resync). Dropping
    a stopped event instead would mean a crashed instance is never restarted, and
    nothing would say so.
  • The worker pools are the opposite: a full pool refuses an action rather than
    queueing it, and the instance is re-dued shortly after. A task waiting for a
    worker burns the deadline the watchdog reaps it by, which for a check would
    count as a failed probe.
  • Per-project schedulers hang off the daemon’s context, not the listener’s, so
    they survive a reconnect with their instances, failure counts, backoff and last
    reported status intact.

Full write-up: ic-healthd Internals.

First install

curl -sSfL https://raw.githubusercontent.com/lxc/incus-compose/main/install.sh | sh -s -- -b ~/.local/bin

Add -p to that to get this rc instead of 1.1.0.

Docs: Health Checking ·
docs.incus-compose.org ·
Full changelog: CHANGELOG.md

Thanks to @ishaan-jindal for working on backup, @blurry for testing and
feedback again, @tofil for planning and researching an upcoming DNS feature and
change, and of course @stgraber for all the consulting and the upstream fixes —
and to everyone else testing, reporting bugs, spreading the word, and just using
incus-compose.

It’s an rc: if you run something in production with it, the most useful thing you
can send me is the compose file that misbehaved.

René

Repo: GitHub - lxc/incus-compose: Bring the familiar Docker Compose workflow to Incus — run compose.yaml files natively on Incus · GitHub
Releases: Releases · lxc/incus-compose · GitHub
Changelog: incus-compose/CHANGELOG.md at main · lxc/incus-compose · GitHub
Previous threads: v1.0 · v1.1

1 Like