I would like to hear your opinion and tips about migrating and enhancing my setup (described below) to the new virtual machine feature. I read other threads, but got no answers
I operate 2 servers.
“One” is a dedicated (and my main) server hosted at Hetzner and on this one I use KVM to separate a virtual firewall (pfsense), an “internal” httpd server, a mail / groupware server (Centos) and my storage vm (with ~ 2TB data).
“Two” is a VPS on which I use lxd containers to separate various websites from each other and I do data backups here. This is mostly for security reasons, because websites get hacked often.
At the moment I think about implementing a high availability setup, mainly for the mailserver.
For this I would order another machine with the same specs as “One”.
I need pacemaker, corosync and drbd/gluster to create a HA cluster which is quite “heavy” stuff to work with.
And then it came to my mind that there is a new lxd virtual machine feature and I have some questions:
is it possible to create a HA cluster with LXD?
what pitfalls do I need to keep in mind?
Can I use pfsense as my firewall and openvpn entry point?
How would you operate a “network RAID”?
At Hetzner you can order an additional public IP which is bound to a MAC address. Can I move a vm around with a fixed MAC?
For network “security” I want to use multiple networks. Do I need this?
I know these are quite a lot questions, but any thoughts are appreciated.
Kinda. You can combine LXD clustering (HA database) with Ceph (distributed storage) to allow for the loss of servers without the loss of data. Instances running on a lost server will have to be moved (with lxc move --target) to another server and then started back up though, so this isn’t zero downtime or anything. One thing to note is that you need a minimum of three servers for that as both LXD and Ceph use quorum+consensus for state replication and for that to work you need an uneven number of active servers with a minimum of three.
Above should cover the most obvious ones
That should be fine, LXD isn’t particularly picky about what connects where, so you could totally have one VM or container on LXD act as a firewall/router.
So again, not zero downtime but if you’re allowed to move that MAC address from one machine to another, you could just move the entire firewall VM/container over to another host should one go down.
LXD is quite happy for you to do that, yes, but there’s a very good chance that Hetzner will not let you, effectively maintaining MAC tables on a per-switchport basis and so preventing you from using the MAC on another server. This is usually done to prevent you from stealing someone else’s traffic.
Creating multiple networks/bridges would be fine.
Ideally what you’d want for such a setup is for a private physical network that all your servers are attached to, you could then create VLANs on that physical network which would tie to your different networks. LXD can then bridge containers or VMs to those VLANs and your firewall/router VM would be connected to all of those as well as to the WAN.
But for this to work well you effectively need:
Have at least 3 servers so you can have consensus for both LXD and Ceph.
Your provider to allow that kind of internal/dedicated network (OVH has something called vRack which on paper matches that, I’ve never used it myself).
Your provider to allow the MAC address on the WAN side of your router VM to be moved around between your three servers.
