This is going to be a long ride, so please get your favorite beverage and relax.
We run our product’s CI on containers run on top of LXD. The product is tested on clusters of different sizes and, because of some features of the product, different amounts of NICs. This mean we can have 1, 3, 6, 12, 60 nodes with 1, 2 or 3 NICs configured. Technically we could have any amount, but I think we never used more than 3. All these node interfaces live in separate IP networks. These environments are created, some tests are run, and then destroyed, until the next CI run comes along.
We need a lot of flexibility because this same setup is used to reproduce setups at clients, which can have quite arbitrary network layouts, and sometimes we chase bugs related to that.
All this is run on a single host. We know of LXD clustering, but because of the way we lean on dsnmasq to do internal DNS (our product is quite sensible to this, and it’s needed to run the tests) it’s currently not possible to use them.
We also don’t do this ourselves, we use Terraform on top of LXD. We do this because we thought we could create a single description of clusters and run them on different backends, like LXD or Amazon EC2. We never go to the point where this happened, so currently we only run on LXD. We could remove Terraform, but since it’s interface is quite simple (just a text file that we can generate with a template), we’re keeping it. I’m in favor of lean products, but I still think we can get to the point we run on EC2.
Anyways…
For each cluster we define at least one LXD bridge network. All the interfaces of the nodes are connected here. We do this this way because so far we could, and requires less definitions.
The LXD hosts also have another LXD bridge network that we use to provision the nodes of the clusters through an extra interface. We also add an interface to each node where we run IPMI. That means that if we create a cluster with 2 declared interfaces per node, we add an extra one for provisioning and another one for IPMI, ending with 4 interfaces in total. These two extra interfaces are attached to the host’s bridge. We use the provisioning interface to run Ansible, but that’s almost irrelevant here 
That per cluster bridge is also the one handling the cluster’s internal DNS. We configure this via dnsmasq.raw like this:
"raw.dnsmasq" = <<EOF
no-hosts
address=/s3-admin.singlenode.cloudian.eu/10.11.1.151
address=/cmc.singlenode.cloudian.eu/10.11.1.151
address=/iam.singlenode.cloudian.eu/10.11.1.151
address=/s3-sqs.singlenode.cloudian.eu/10.11.1.151
address=/s3-eu-1.singlenode.cloudian.eu/10.11.1.151
address=/the-one.singlenode.cloudian.eu/10.11.1.151
address=/s3-website-eu-1.singlenode.cloudian.eu/10.11.1.151
address=/singlenode/10.11.1.151
That’s for a single node cluster; if we had more nodes, there would be an address for each. If the nodes have more than one interface, we only list one address for them.
This morning 5.7 landed on our CI machines and we started getting Error: failed to start container (singlenode): Failed start validation for device "ipmi": Instance DNS name "singlenode" already used on network. The node’s interfaces are declared like this in Terraform:
+ device {
+ name = "eth0"
+ properties = {
+ "name" = "eth0"
+ "nictype" = "bridged"
+ "parent" = "singlenode"
}
+ type = "nic"
}
+ device {
+ name = "ipmi"
+ properties = {
+ "name" = "ipmi"
+ "nictype" = "bridged"
+ "parent" = "lxd-provision"
}
+ type = "nic"
}
+ device {
+ name = "provision"
+ properties = {
+ "name" = "provision"
+ "nictype" = "bridged"
+ "parent" = "lxd-provision"
}
+ type = "nic"
}
Here we see a single interface eth0 linked to the per cluster network, and the IPMI and provision interfaces linked to the per host interface. LXD 5.7 is complaining about these ones. It’s nice that we can name them whatever we want and kinda make sure they won’t clash with real world interfaces.
Maybe the solution for us would be to start building one bridge per cluster interface, but @tomp asked me to post this, so here we are.
