Been thinking, x-incus-* will be all optional. With them you will be able to create other types of nics than just bridge or use other pools than just the default it creates.
.env limits us to strings that brings complications.
Thining about a companion compose.incus.yaml that you might create next to a comose.yaml, that might be easy to implement or just works out-of-the-box with compose-go.
yeah, that sounds good. I would probably name it incus-compose.<ext> to go with the naming scheme, but that’s just bike shedding.
.incus.yaml is a profile for compose with a little trickery auto loaded by compose-go and ignored by docker-compose 
EDIT: It should be able to load other profiles as well.
@Sagi - Pushed the major update.
The refactor is done. Extensible client library, hooks for everything, ~70% test coverage on the core package. KISS and boring.
Your reported bugs should be gone - if you’re still testing, would appreciate verification.
Next is implementing the docker-compose commands.
Also it moved: René Jochum / incus-compose · GitLab
Chiming in here to share the AUR package I’ve created for incus-compose. Comments/suggestions welcome 
(I also created an issue in the repo before thinking of announcing it here instead)
Cheers
I’ve got health-checking done in: Health check support (#6) · Issues · René Jochum / incus-compose · GitLab
For anyone interested into the development, these issues needed to be closed for a first public beta: First Release · Milestones · René Jochum / incus-compose · GitLab
First beta release online!
Please check it out and give feedback, either here or on gitlab issues.
Thank you,
René
That’s damn impressive, congrats on the first beta! I’ll take time during this week to try it out and give you some feedback
I’m impressed by this beta! First impressions are very good, tested quickly with a random docker-compose.yml I had laying around from testing Ontime a few months back. Only had to change the bind-mount to a volume mount, add a `–remote` flag (possible improvement: use same remote as currently selected by incus CLI), and add a storage pool named “default” to my Incus cluster, otherwise I would get an `Failed creating instance record: Failed initializing instance: Failed loading storage pool: Storage pool not found” ` error.
Then, it just… worked!
Great work, I don’t have many currently active projects using docker-compose at the moment, but for my next “I’d like to use all the CPU cores of my Incus cluster instead of my laptop for this kernel build”, I’ll have to test it again 
Thank you!
Hmm, using the same remote as incus does as “default” sounds good, will add a issue for that.
Dang, that sound’s problematic - ofc. you don’t need a to create a default pool on init, I’m just used to create it with incus admin init.
Fun! That kernel-build use case is exactly what Incus can do that docker-compose can’t. Curious how it goes if you try it.
I’m still setting up IncusOS and hoping to try out this project for some simple self hosted container needs. This project looks very promising.
My initial testing ran into the default vs local storage pool issue mentioned above too though. (Ideally I’d be able to pick where the container’s root volume and data volumes come form. I’d like to put my container rootfs on local which is backed by SSD, but also mount some data volumes backed by a 2nd pool of mirrored HDDs.)
Been thinking,
x-incus-*will be all optional. With them you will be able to create other types ofnicsthan justbridgeor use other pools than just thedefaultit creates.
.envlimits us to strings that brings complications.Thining about a companion
compose.incus.yamlthat you might create next to acomose.yaml, that might be easy to implement or just works out-of-the-box withcompose-go.
From the official docker compose docs:
By default, Compose reads two files, a
compose.yamland an optionalcompose.override.yamlfile. By convention, thecompose.yamlcontains your base configuration. The override file can contain configuration overrides for existing services or entirely new services.
You could leverage the compose.override.yaml mechanism to let users use upstream compose.yaml files, but apply x-incus-* extensions via the override file? And apparently the docker compose CLI lets you even pick arbitrary files to merge. Maybe just specify that your default order of loaded and merged files is something like:
compose.yaml (docker-compose.yaml works too I think, not sure where it fits in the load order)compose.override.yamlcompose.incus.yamlcompose.incus.override.yaml (unsure if you need this one)Just tried my first incus-compose with Immich on a remote IncusOS server. Here’s the compose.yaml I used:
# docs: https://docs.immich.app/install/docker-compose
name: immich
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends:
# file: hwaccel.transcoding.yml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
volumes:
# Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
- ${UPLOAD_LOCATION}:/data
- /etc/localtime:/etc/localtime:ro
env_file:
- .env
ports:
- '2283:2283'
depends_on:
- redis
- database
restart: unless-stopped
healthcheck:
disable: false
immich-machine-learning:
container_name: immich_machine_learning
# For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
# Example tag: ${IMMICH_VERSION:-release}-cuda
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://docs.immich.app/features/ml-hardware-acceleration
# file: hwaccel.ml.yml
# service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
volumes:
- immich-model-cache:/cache
env_file:
- .env
restart: unless-stopped
healthcheck:
disable: false
redis:
container_name: immich_redis
image: docker.io/valkey/valkey:9@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9
healthcheck:
test: redis-cli ping || exit 1
restart: unless-stopped
database:
container_name: immich_postgres
image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
POSTGRES_INITDB_ARGS: '--data-checksums'
# Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
# DB_STORAGE_TYPE: 'HDD'
volumes:
# Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
- ${DB_DATA_LOCATION}:/var/lib/postgresql/data
shm_size: 128mb
restart: unless-stopped
healthcheck:
disable: false
volumes:
library:
immich-model-cache:
postgres:
I only changed restart: always to unless-stopped and the bind mounted volumes to regular ones declared at the bottom.
Accompanying .env file:
# You can find documentation for all the supported env variables at https://docs.immich.app/install/environment-variables
# The location where your uploaded files are stored
UPLOAD_LOCATION=library
# The location where your database files are stored. Network shares are not supported for the database
DB_DATA_LOCATION=postgres
# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Paris
# The Immich version to use. You can pin this to a specific version like "v2.1.0"
IMMICH_VERSION=v2
# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=supersecretpassword
# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
Doing incus-compose up trips up on the remote names:
error="image source error getting image server for docker.io: The remote \"docker.io\" doesn't exist"
error="image source error getting image server for ghcr.io: The remote \"ghcr.io\" doesn't exist"
Indeed, I’ve added those as docker and ghcr in my remotes 
After renaming those I got:
15:53 ERR Getting healthd image url=https://XXX.XXX.XXX.XXX:8443 project=immich image=registry.gitlab.com:r3j0/incus-compose/ic-healthd:latest error="invalid format: image(registry.gitlab.com:r3j0/incus-compose/ic-healthd:latest): invalid reference format"
I then added GitLab as a remote:
incus remote add registry.gitlab.com https://registry.gitlab.com --protocol=oci
And then it hangs on:
15:56 DBG Done url=https://XXX.XXX.XXX.XXX:8443 name=registry.gitlab.com:r3j0/incus-compose/ic-healthd:latest kind=image action=ensure
Full debug output:
15:56 DBG Using connection remote=my-remote
15:56 DBG Got project url=https://re.da.ct.ed:8443 name=default incus_name=default
15:56 DBG Got project url=https://re.da.ct.ed:8443 name=immich incus_name=immich
15:56 DBG Found healthchecks url=https://re.da.ct.ed:8443 project=immich incus_project=immich
15:56 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23 service=database
15:56 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=ghcr.io/immich-app/immich-server:v2 service=immich-server
15:56 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=ghcr.io/immich-app/immich-machine-learning:v2 service=immich-machine-learning
15:56 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=docker.io/valkey/valkey:9@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9 service=redis
15:56 DBG Prepared healthd sidecar image url=https://re.da.ct.ed:8443 project=immich incus_project=immich name=ic-healthd
15:56 DBG network:default
15:56 DBG network:default
15:56 DBG network:default
15:56 DBG Ensure url=https://re.da.ct.ed:8443 project=immich incus_project=immich resources="[network(ic-h3vxrnqwhm) instance(database-1) instance(immich-machine-learning-1) instance(redis-1) instance(immich-server-1) image(ghcr.io/immich-app/postgres@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23) image(ghcr.io/immich-app/immich-server:v2) image(ghcr.io/immich-app/immich-machine-learning:v2) image(docker.io/valkey/valkey@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9) image(registry.gitlab.com:r3j0/incus-compose/ic-healthd:latest) healthd(ic-healthd)]"
15:56 DBG Done url=https://re.da.ct.ed:8443 name=docker.io/valkey/valkey:9@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9 kind=image action=ensure
15:56 DBG Done url=https://re.da.ct.ed:8443 name=ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23 kind=image action=ensure
15:56 DBG Done url=https://re.da.ct.ed:8443 name=registry.gitlab.com:r3j0/incus-compose/ic-healthd:latest kind=image action=ensure
I tried adding --no-helathd but it remains stuck:
incus-compose up --remote re.da.ct.ed --no-healthd --debug
16:00 DBG Using connection remote=re.da.ct.ed
16:00 DBG Got project url=https://re.da.ct.ed:8443 name=default incus_name=default
16:00 DBG Got project url=https://re.da.ct.ed:8443 name=immich incus_name=immich
16:00 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=docker.io/valkey/valkey:9@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9 service=redis
16:00 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23 service=database
16:00 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=ghcr.io/immich-app/immich-server:v2 service=immich-server
16:00 DBG Getting image url=https://re.da.ct.ed:8443 project=immich incus_project=immich image=ghcr.io/immich-app/immich-machine-learning:v2 service=immich-machine-learning
16:00 DBG network:default
16:00 DBG network:default
16:00 DBG network:default
16:00 DBG Ensure url=https://re.da.ct.ed:8443 project=immich incus_project=immich resources="[network(ic-h3vxrnqwhm) instance(redis-1) instance(database-1) instance(immich-machine-learning-1) instance(immich-server-1) image(docker.io/valkey/valkey@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9) image(ghcr.io/immich-app/postgres@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23) image(ghcr.io/immich-app/immich-server:v2) image(ghcr.io/immich-app/immich-machine-learning:v2)]"
16:00 DBG Done url=https://re.da.ct.ed:8443 name=docker.io/valkey/valkey:9@sha256:3b55fbaa0cd93cf0d9d961f405e4dfcc70efe325e2d84da207a0a8e6d8fde4f9 kind=image action=ensure
16:00 DBG Done url=https://re.da.ct.ed:8443 name=ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23 kind=image action=ensure
16:08 DBG Result with error url=https://re.da.ct.ed:8443 name=ghcr.io/immich-app/immich-server:v2 kind=image action=ensure error="in an operation: Failed remote image download: Alias already exists: ghcr.io/immich-app/immich-server:v2"
So yeah, I’m currently stumped I suspect that the difference in default storage pool name between regular incus (default) and IncusOS (where the default storage is called local) may be the cause.
On another note, I agree with what @edorgeville said:
add a
–remoteflag (possible improvement: use same remote as currently selected by incus CLI),
This is a big one for me as well: I incus remote switch to my server so as not to have to add the remote name all the time, and I was expecting incus-compose to use the currently selected one. That would fit with how incus behaves, instead of hardcoding local unless overriden with --remote of $INCUS_REMOTE.
Meanwhile: thank you very much for bringing us this far, looking forward to using it more!
Have to add, those several failed attemtps where I’ve add to Ctrl+C my way out have left zombie processes on my remote, which manifest as Running operations that I cannot cancel…
incus os admin debug processes show at the bottom:
root 5013 0.7 0.3 2465052 98880 ? Sl 15:55 0:11 \_ skopeo --insecure-policy inspect docker://ghcr.io/immich-app/immich-machine-learning:v2
root 5179 0.7 0.3 2391064 100564 ? Sl 15:56 0:11 \_ skopeo --insecure-policy inspect docker://ghcr.io/immich-app/immich-machine-learning:v2
root 5265 0.8 0.2 2465052 94680 ? Sl 15:59 0:10 \_ skopeo --insecure-policy inspect docker://ghcr.io/immich-app/immich-machine-learning:v2
root 5316 0.6 0.2 2325200 80836 ? Sl 16:00 0:08 \_ skopeo --insecure-policy inspect docker://ghcr.io/immich-app/immich-machine-learning:v2
I’ll try rebooting the server.
In the case of Immich, the immich-server service is trying to access the database service at the network address “database”. This lookup fails because incus-compose starts the container as “database-1” and Incus appears to only provides a default DNS entry for the container name “database-1”.
For full compatibility, I think incus-compose would have to leverage Incus Network Zones and add DNS entries also for the service names in additional to the container names that get generated. This way the database service could be reached at either “database” or “database-1”. Docker supports both of these methods. I’ll leave links out for now.
One other feature request could help serve as a stopgap, If the “container_name” key were used, it could override the default container name generation and pin the name back to “database” instead of “database-1”. This is likely much easier to support than the deeper network integration.
Hey
Skopeo is for images, you could try to delete the incus project with incus project list and then incus project rm.
Many thanks for testing and playing with ic! I have also seen you’r gitlab issue @pyrodogg , thank you!
The workaround of implementing “container_name” sounds easy enough. Network zones might not as hard to implement too.
Hmm so it might be the gitlab registry in your area?
You could delete all “healthcheck:” entries to get rid of healthd the easy way
I recently tried Incus with skopeo + gitlab to deploy images from a private gitlab registry (unrelated to this cool looking project), and my investigation suggested skopeo does not follow the auth redirect from gitlab’s container registry like the docker-cli does (its not actually a skopeo issue, but the auth libs it uses).
The fix looked liked it required getting 1 open source lib to write a few lines of code + 1/2 OS libs to update + incus to update so I didn’t bother, but I probably should have done.
This might be unrelated to this issue, but thought id post publicly, beside the chat where I did before, someone might have more patience than I do
I’ve been trying to run up immich by hand (having seen here the corner cases in the compose file) and I noticed this output has your runs very close together.
I’ve been debugging incus image info taking 10 minutes to complete on immich-machine-learning and got some help in the general chat ( Message #14511 by kgoetz – #General ). It appears to be due to the tagging in that repository, other things return very quickly.
Is it possible here:
That its not gitlab hanging? the output says ‘Done’. My conjecture is that immich-machine-learning is not yet done, causing your “hang”.
Did you wait 15+ minutes on any of your tests to see if an error occurred?