I have 3 identical servers with 2 SSDs. I finally got IncusOS installed with the bonded interface and the vlan tagging, however looking for some examples to get me over the hump w/ getting ceph and (optionally) ovn configured on this same cluster as well. I looked at the API specs for w/in the service_ceph.go however I’m no ceph ninja and lacking any examples on how to configure it is making it difficult to deploy it.
IncusOS acts as a Ceph or OVN client, it doesn’t run the control plane for either of those.
The way I’ve done it in my own environments is by running the control plane and the Ceph OSDs in containers:
stgraber@castiana:~ (incus:s-shf-cluster/internal)$ incus list -cnstL
+------------------+---------+-----------+----------+
| NAME | STATE | TYPE | LOCATION |
+------------------+---------+-----------+----------+
| ceph-central01 | RUNNING | CONTAINER | chulak |
+------------------+---------+-----------+----------+
| ceph-central02 | RUNNING | CONTAINER | sateda |
+------------------+---------+-----------+----------+
| ceph-central03 | RUNNING | CONTAINER | tollana |
+------------------+---------+-----------+----------+
| ceph-osd-chulak | RUNNING | CONTAINER | chulak |
+------------------+---------+-----------+----------+
| ceph-osd-delmak | RUNNING | CONTAINER | delmak |
+------------------+---------+-----------+----------+
| ceph-osd-sateda | RUNNING | CONTAINER | sateda |
+------------------+---------+-----------+----------+
| ceph-osd-tollana | RUNNING | CONTAINER | tollana |
+------------------+---------+-----------+----------+
| ovn-central01 | RUNNING | CONTAINER | chulak |
+------------------+---------+-----------+----------+
| ovn-central02 | RUNNING | CONTAINER | sateda |
+------------------+---------+-----------+----------+
| ovn-central03 | RUNNING | CONTAINER | tollana |
+------------------+---------+-----------+----------+
Those all run on a physical network which is directly reachable by the hosts as my routers are typically containers too and you obviously need to make sure that there’s no chicken and egg issue going on where on startup you need to go through your router to bring up the storage that this same router requires 
So far I’ve been running everything in simple Debian 13 containers but it’d be nice to be able to do this using OCI images instead. That’s the general plan behind Offer a full Incus deployment as part of Incus initialization · Issue #497 · lxc/incus-os · GitHub
1 Like
I’ll keep an eye out for updates for a full deployment. I’d love to have a deployment thats simple and stable. In the meantime I’ll take a look at the incus-deploy ansible scripts w/ debian 13 and see how that works.