Any recommendations as whether to use LXD containers or LXD VMs as hosts for Kubernetes nodes?
Or Maybe just don’t use LXD at all? (The setup I’m planning is permanent, not test nor dev, so maybe, even if I like the idea of it, it is best not to use LXD for such a setup).
I have found several blogs and videos explaining how to run K8S on top of LXD containers but not much about running it on top of LXD VMs
I’m guessing using VMs is so straight forward that nobody felt the need to make a blog post or a video…
When using LXD VMs, do you still have to take care about the special parameters required for LXD containers?
Specifically, can you use ZFS? And if not, what’s the recommended option, Ext4?
Thank you!
@tomp Sorry for the direct mention, but I have seen you responding to other “K8S on top of LXD” questions before.
I’m not sure of whether K8S works nested inside of LXD containers.
It is possible to use Docker, but using orchestrators like K8S or Docker Swarm seem to have mixed success.
Yes using LXD VMs to run K8S should be fine as its just like running it on a normal machine.
Just like a normal machine you get the choice of what guest filesystem to use, so for that I would suggest checking what K8S recommends.
As for the LXD storage driver underneath the guest, this will be dependent on workload and hardware being used. But using something that supports proper block volumes (which is what VMs use) such as ZFS or LVM would likely be more performant than using a block file ontop of a host filesystem (which is what happens with dir and BTRFS storage pools). Although your mileage/requirements may vary and make this still an option.
You also don’t have to select just one storage pool type, as its possible to have your VM guest running ontop of one storage pool type and then pass in custom block volumes from other different storage pool types for use with your guest applications.
I’m sorry this is rather vague, but it is quite a broad question. But my suggestion is to go with LXD VMs as you will avoid any container nesting issues, yet still get the flexibility that LXD offers around instance management (backups, snapshots, volumes, networking, host isolation etc).
LXD uses a “stripe” of 2 mirrors based on partitions.
K8s nodes use: ??? “using something that supports proper block volumes”
So I am thinking about surfacing some block devices from the underlying ZFS, not sure what exactly, yet.
I am trying a similar architecture with Kubernetes nodes on LXD vms…setting up one vm instance as the control node. Does LXD networking interoperate well with Kubernetes?