IncusOS has been an amazing foundation for my homelab and I’m hoping to expand my setup to include Kubernetes nodes.
Does anybody have any recommendations for ways of setting up Kubernetes on Incus? I’m new to K8s and want to learn it, but I’m a bit confused / intimidated by the Cluster API and unsure if that’s the best way to go. I’ve heard good things about Talos Linux, but last I checked, there’s no Incus compatible Talos image that also gives the incus agent some amount of monitoring access to the host. Right now, setting up a K3s install on general purpose distro seems like the clearest path forward for me.
If you have any setups, opinions, or recommendations you’d like to share, I’d love to hear them!
This is the holy grail of Kubernetes on Incus, the project provides pre-built kubeadm machine images based on best practices. A hidden gem is the “kini” tool, it’s a lightweight cli that provides the same experience as kind (kubernetes inside docker) but instead of docker containers it reconciles your cluster nodes with either OCI app containers or LXC. Awesome project.
When you deploy your cluster you can even mix and match, so your control-plane can be lxc and your workers can be vms.
Does the incus cluster API provider allow one to create/manage a kubernetes cluster that spans multiple Incus instances? I have different machines so the incus part is not clustered.
I think Talos is pretty good, though it’s primarily intended for bare metal deployments.
Thanks for sharing! The only thing I’m a bit intimidates me about this, as a newcomer to K8s, is the setup and ‘cluster API’ part. What would setting up a three node cluster look like? If you have any tutorials or experience to share I’d love to hear more. Thanks!
I was curious about the Talos route, but hesitant since there’s no incus agent support similar to Proxmos (which uses the qemu guest agent). Supporting the incus agent is also currently not on their roadmap
Does the incus cluster API provider allow one to create/manage a kubernetes cluster that spans multiple Incus instances? I have different machines so the incus part is not clustered.
If the three machines are joined into an incus cluster, yes. The idea is that if an incus member goes away, the nodes of your cluster will get rescheduled
What would setting up a three node cluster look like? If you have any tutorials or experience to share I’d love to hear more. Thanks!
If you think about how kubernetes works, for example creating a deployment with 3 replicas, it’s done the same way if you have one node or three nodes. The scheduler takes care of running them across your workers.
cluster-api works the same way, if your incus cluster has one or three members, and you deploy a kubernetes cluster with 3 nodes or 30 nodes, CAPI handles scheduling them across your “infrakube”
Hmm, so the capi provider doesn’t support Kubernetes clusters across multiple Incus hosts unless those host are clustered?
I thought the whole idea of Kubernetes is to allow for graceful pod migration if a host goes down; having two layers of redundancy seems, well, redundant.
Hmm, so the capi provider doesn’t support Kubernetes clusters across multiple Incus hosts unless those host are clustered?
No, in the same way you can’t deploy a single kubernetes deployment across multiple clusters (out of the box at least)
I thought the whole idea of Kubernetes is to allow for graceful pod migration if a host goes down; having two layers of redundancy seems, well, redundant.
This is true, but CAPI will actually replace your failed node, just like when you delete a pod of a deployment, kubernetes will replace that pod.
To take it further, you can even scale up and down your cluster just like a deployment. It essentially connects the programmable infrastructure layer (the infrastructure provider) to be kubernetes aware. These concepts and cluster operations work the same way if you use the cluster-api-aws-provider or cluster-api-proxmox-provider. And everything is deployed with declarative manifests.
If you want to upgrade your cluster with zero downtime, you simply bump the version and machine-image in your cluster manifest. Your entire cluster then gets replaced one by one immutable infrastructure style.
I see. My disappointment is that Kubernetes is very flexible — I can join a Raspberry Pi, a Rackmount Server, and a laptop in the same cluster and have Kubernetes handle scheduling and availability.
Incus clustering is the opposite — every machine must have the same storage names, the same storage sizes, and capabilities are all assumed to be the same.
CAPI seems to be designed for the cloud, where the first point isn’t really relevant.
OK I see your motivation here, I agree with you for that is awesome benefit of k8s. The machines can even be different architectures and still kubeadm join which is great.
One thing that might interest you is a single cluster-api management cluster can operate multiple capi providers, but these clusters would be each be reconciled by their provider instead of a global pool of infrastructure which is your goal.
If your machines can all pxe on the same network, you can actually run k8s across them easily using canonical maas, which also has a cluster-api provider
