I am trying to set up a K8s lab setup, with the nodes running on Incus system containers.
My host itself is a VM, running Ubuntu 22.04. Created a framework to spin up 1 master node and 2 worker nodes. So far the nodes can be started and configured using terraform and Ansible. But when I am running kubeadm init on the master node, it fails and in the error output there is “The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)”
The configs and scripts are here https://github.com/trifo75/kubernetes-playground.git
The profile used to spin up the hosts is like this:
$ incus profile show kubelab
config:
linux.kernel_modules: ip_tables,ip6_tables,nf_nat,overlay,br_netfilter
raw.lxc: "lxc.apparmor.profile=unconfined\n#lxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.sysctl.net.ipv4.ip_forward=1\nlxc.sysctl.net.bridge.bridge-nf-call-iptables=1\nlxc.sysctl.net.bridge.bridge-nf-call-ip6tables=1\nlxc.cgroup2.devices.allow=a\n#lxc.cgroup2.controllers=cpuset,cpu,io,memory,hugetlb,pids,rdma,misc,dmem\n\n"
security.nesting: "true"
security.privileged: "true"
security.syscalls.intercept.mknod: "true"
security.syscalls.intercept.mount: "true"
security.syscalls.intercept.setxattr: "true"
description: Kubernetes lab node
devices:
eth0:
name: eth0
network: kube_br0
type: nic
kmsg:
path: /dev/kmsg
source: /dev/kmsg
type: unix-char
root:
path: /
pool: kubepool
type: disk
name: kubelab
used_by:
- /1.0/instances/master
- /1.0/instances/node2
- /1.0/instances/node1
On the nodes I see that cgroup2 is mounted. The controllers and subtree_control files are like this:
root@master:~# cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc dmem
root@master:~# cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc dmem
What should I check/do to find out what is missing and how should I fix it?