Incus-caddy-config available

Hey :slight_smile:

Based on the ievent library in incus-compose, I made a new project:

It’s available as 1.0.0-beta.1.

Think of it like caddy-docker-proxy, but external, template-driven, and using Incus storage volume SFTP so Caddy doesn’t need API access.

All you do is label your service:

services:
  api:
    image: docker.io/library/busybox:latest
    command: httpd -f -p 8080
    labels:
      edge: "api.example.com,upstream=8080"

(Caddy configures the reverse proxy block automatically)

It also supports custom Go templates/flags per host, replica load balancing, and works either with incus-compose or directly on Incus instances (plus --os-path for host Caddy).

Currently running it in production across ~30 endpoints on my servers. Feedback is welcome!

The pictogram, the diagram, at https://caddy.incus-compose.org/ shows caddy server as container inside incus.

How to use (how to abuse) the code of incus-caddy-config for caddy on the host?

You might run

caddy-config \
   --os-path=my-caddy,path=/etc/caddy/Caddyfile \
   --incus="unix://" \
   --client-cert=/home/my-user/.config/incus/client.crt \
   --client-key=/home/my-user/.config/incus/client.key

Make sure to have a block like:

{
    admin localhost:2019 
}

in your caddyfile so it can reload.

You need to configure each service you wanna serve with:

config:
  user.label.my-caddy.service: my-service-group
  user.label.my-caddy: "<long-info-here>"

This is untested.


The label will get another format

Atm. we are using a single label “edge” / “my-caddy” whatever you name it, this produced stuff like:

services:
  wiki:
    labels:
      caddy-external: "incus-compose.org,upstream=8080,redirs='www.incus-compose.org,uri wiki.incus-compose.org,uri docs.incus-compose.org,uri'"
      caddy-internal: "incus-compose.org,upstream=8080,template=internal_acme.caddyfile,redirs='www.incus-compose.org,uri,template=internal_acme.caddyfile wiki.incus-compose.org,uri,template=internal_acme.caddyfile docs.incus-compose.org,uri,template=internal_acme.caddyfile'"

This is much to much info compressed in our own language :confused:

new will be:

services:
  wiki:
    labels:
      caddy-external.0.domain: incus-compose.org
      caddy-external.0.upstream: 8080
      caddy-external.0.redirs.0: www.incus-compose.org,uri
      caddy-external.0.redirs.1: wiki.incus-compose.org,uri
      caddy-external.0.redirs.2: docs.incus-compose.org,uri
      caddy-internal.0.domain: incus-compose.org
      caddy-internal.0.upstream: 8080
      caddy-internal.0.redirs.0: www.incus-compose.org,uri,template=internal_acme.caddyfile
      caddy-internal.0.redirs.1: wiki.incus-compose.org,uri,template=internal_acme.caddyfile
      caddy-internal.0.redirs.2: docs.incus-compose.org,uri,template=internal_acme.caddyfile

We will support both formats.


This is how my compose.yaml looks like:

name: caddy

services:
  external:
    image: docker.io/library/caddy:2.11.4-alpine
    container_name: external
    restart: unless-stopped
    volumes:
      - config-external:/config
      - logs-external:/var/log/caddy/
      - data:/data
      - ./sites:/var/www:ro
    command: |
      sh -c 'if [ ! -f /config/Caddyfile ]; then echo -e "{\n\tadmin localhost:2019\n}\n:80 {\n\trespond \"Caddy initializing...\" 503\n}\n" > /config/Caddyfile; fi; exec caddy run --config /config/Caddyfile --adapter caddyfile'
    ports:
      - published: 80
        target: 80
#        x-incus-compose:
#          nat: true
      - published: 443
        target: 443
#        x-incus-compose:
#          nat: true
    networks:
      default:
        ipv4_address: 10.179.214.5/23
        x-incus:
          ipv4.gateway: 10.179.214.1

  internal:
    image: docker.io/library/caddy:2.11.4-alpine
    container_name: internal
    restart: unless-stopped
    volumes:
      - config-internal:/config
      - logs-internal:/var/log/caddy/
      - data:/data
      - ./sites:/var/www:ro
    command: |
      sh -c 'if [ ! -f /config/Caddyfile ]; then echo -e "{\n\tadmin localhost:2019\n}\n:80 {\n\trespond \"Caddy initializing...\" 503\n}\n" > /config/Caddyfile; fi; exec caddy run --config /config/Caddyfile --adapter caddyfile'
    networks:
      default:
        ipv4_address: 10.179.214.6/23
        x-incus:
          ipv4.gateway: 10.179.214.1

  caddy-config:
    image: ghcr.io/jochumdev/incus-caddy-config/caddy-config:1.0.0-beta.1
    container_name: caddy-config
    restart: unless-stopped
    depends_on:
      external:
        condition: service_healthy
      internal:
        condition: service_healthy
    environment:
      INCUS_CADDY_INCUS: "${INCUS_CADDY_INCUS}"
      INCUS_CADDY_DATA_DIR: "/var/lib/caddy-config"
      INCUS_CADDY_INSTANCES: "caddy-external,project=caddy,instance=external,global_template=external.caddyfile caddy-internal,project=caddy,instance=internal,global_template=internal.caddyfile"
      INCUS_CADDY_HTTP_ADDRESS: ":9153"
      INCUS_CADDY_LOG: "INFO"
      INCUS_CADDY_TEMPLATES_DIR: "/etc/caddy/templates"
    secrets:
      - token
    volumes:
      - caddy-config:/var/lib/caddy-config
      - ./templates:/etc/caddy/templates:ro

secrets:
  token:
    environment: INCUS_TOKEN

volumes:
  config-external:
  logs-external:
  config-internal:
  logs-internal:
  data:
  caddy-config:

networks:
  default:
    external: true
    name: incusbr0

My internal_acme.caddyfile

{{ .Domain }} {
    tls /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/{{ .Domain }}/{{ .Domain }}.crt /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/{{ .Domain }}/{{ .Domain }}.key
    log {
        output file /var/log/caddy/{{ .Domain }}-access.log
    }
	{{- if .Redirect }}
		redir {{ .Redirect }} permanent
	{{- else if .Upstreams }}
		reverse_proxy {{ range $i, $u := .Upstreams }}{{ if $i }} {{ end }}{{ $u }}{{ end }}
	{{- end }}
}

This looks very interesting to me. :thinking:

I’ve been playing with incus for a couple/few years now and had never got on the docker bandwagon – odd I guess, but I’m a format + reinstall junkie and like to configure by hand to keep the muscle memory alive. Incus containers and vm are like etch-a-sketch for me. :laughing:

I also use caddy server for assorted basic things. I’m not a (current) hosting guru by any means and I don’t generally leave the lab environment which remains non-public, in as much as I can keep it private while all being on the public internet. :upside_down_face:

After a read of your site and here I wanted to ask if I’ve more or less got the gist of what incus-compose & incus-caddy-config provide.

  • spawn caddy reverse proxies for new incus instances more securely than other methods.
  • a more durable disassembly/reassembly of it all when instances/incus/host restart
  • QoL aspects to those which are accustomed to docker (attract docker users..?)

Is learning docker required? ..or at most a subset of the docker compose commands?

I imagine that I have radically simplified and probably didn’t understand fully the use and scope that incus-caddy-compose presents. Sorry, I’m extremely lacking on the commonly known lingo of hosting and related facets.

incus-compose is docker-compose a like, not docker-compose. I found docker-compose to be extremly simple for OCI stuff so I made it for incus, I disovered “incus-compose” from Brian first.

incus-caddy-config is a dynamically configured caddy for you, well not a caddy but it configures and reloads caddy on demand.

One big difference with incus I feel is that everything is statefull, as long as you don’t “down” something.

Learning docker is NOT required, but you have to learn the commands of incus-compose if you wanna host OCI stuff and at the same step you learn “docker compose” automatically as ic is just a clone with incus as engine.

For having used both and about 5 years of beeing a Kube Admin, incus is just SIMPLE and rock stable. It does more with less than any other solution.

I hope this is enough, kind regards,
René

Another great product you might look at is GitHub - abiosoft/incus-apply: Declarative configuration management for Incus · GitHub ,

incus-apply is something like “incus-compose” it orchestrates stuff but in this case “system” containers and vms and stuff.

Also keept very simple for the end user.

Thank you for the additional information René.

This ^ ,and partly why I never went further into docker.. virtual machines I found to be the most comfortable since long ago.

Thank you for the ref to incus-apply. So far I’ve stuck to vanilla incus. I hope to find the time to try out available supplemental projects!

Does incus-caddy-config work with multi service deployments? For example, deploying a web app where traffic flows app service → go-away/anubis → caddy, you may not want the app talking to caddy directly (or only doing it for some routes).

I had a quick look at Instance Labels & Routing · Incus Caddy Config , I think the answer should be yes, but couldn’t be certain :slight_smile:

You set labels on go-away/anbuis and route between them and app internally, then you’r fine.

You can see incus-caddy-config as “labels” → vars + ip’s → templates → reload caddy.