Kubernetes on incus system containers - longhorn volume issue

1

Hi all,

May be a bit offtopic. Is anyone here running k8s cluster on incus system contaienrs able to get longhorn working properly? (k8s nodes are system containers)

Made below post on LongHorn community/github discussion - (didn’t get any reposnse…hence trying here.)

My k8s nodes are incus system containers. (3 master, 6 workers). Incus contaienrs doesn’t support block device type volume. I have longhorn 1.7.1 version installed on this k8s cluster.

Whenever I create volume using yaml by kubectl, it creates volume with Frontend:Block Device. This is reason why my pods never comes up. They remain in pending/contaiener creating state as this volume can’t be attached to k8s worker node. And volume keep attaching/detaching stage

While doing basic testing from GUI, I observed that whenever I set frontend type to iSCSI while creating volume I could attach it to k8s node file. I then created pvc/pv from this volume in longhorn GUI. Then edited the yaml for POD to include new pvc/pv. From longhorn GUI I could see this volume attached to PODNAME on WORKERNODE. So from longhorn POV, this volume is healthy.

image
image2554×272 31.8 KB

Blockquote

But the pod still remain in containercreating state with kubectl describe POD showing

Events:
  Type     Reason              Age                 From                     Message
  ----     ------              ----                ----                     -------
  Warning  FailedAttachVolume  10s (x243 over 8h)  attachdetach-controller  AttachVolume.Attach failed for volume "webvol-pv" : rpc error: code = InvalidArgument desc = volume webvol invalid frontend type iscsi

Why is invalid frontend error coming here? Any idea?

2

Just to update this

The Longhorn CSI driver doesn’t support iSCSI mode.

Since Incus system containers don’t support attaching block devices directly, and Longhorn uses block devices as the default frontend for volume attachments, this creates a conflict. Longhorn cannot use iSCSI as the volume frontend for attaching and detaching volumes via Kubernetes

To get things working - one need to have incus VMs as worker nodes. Longhorn will work in that case mostly. Will test this soon.

3

Not directly related but I’ve had success with LXC on proxmox for workers but rather than use longhorn I used Kadalu which is based on GlusterFS GitHub - kadalu/kadalu: A lightweight Persistent storage solution for Kubernetes / OpenShift / Nomad using GlusterFS in background. More information at https://kadalu.tech

However I’m a bit worried that Gluster is a bit old and Kadalu doesn’t seem to be getting many updates anymore.

You can pass any plain old folder to Kadalu to use as the backing for storage.
I mounted folders into LXC to use like this.